Oct 31 04:03:21.127703 ip-10-0-130-57 systemd[1]: Starting Kubernetes Kubelet... Oct 31 04:03:21.581815 ip-10-0-130-57 kubenswrapper[2571]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 04:03:21.581815 ip-10-0-130-57 kubenswrapper[2571]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 04:03:21.581815 ip-10-0-130-57 kubenswrapper[2571]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 04:03:21.581815 ip-10-0-130-57 kubenswrapper[2571]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 04:03:21.587520 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587444 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 04:03:21.587520 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587516 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 04:03:21.587520 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587521 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 04:03:21.587520 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587525 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 04:03:21.587520 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587528 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 04:03:21.587721 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587531 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 04:03:21.587721 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587534 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 04:03:21.587721 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587537 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 04:03:21.587721 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587540 2571 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 04:03:21.587721 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587543 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 04:03:21.587721 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587545 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 04:03:21.587721 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587548 2571 feature_gate.go:386] unrecognized feature gate: Example Oct 31 04:03:21.587721 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587551 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 04:03:21.587721 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587554 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 04:03:21.587721 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587556 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 04:03:21.587721 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587559 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 04:03:21.587721 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587562 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 04:03:21.587721 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587565 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 04:03:21.587721 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587567 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 04:03:21.587721 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587570 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 04:03:21.587721 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587572 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 04:03:21.587721 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587574 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 04:03:21.587721 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587577 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 04:03:21.587721 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587580 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 04:03:21.587721 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587582 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 04:03:21.588179 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587590 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 04:03:21.588179 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587593 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 04:03:21.588179 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587595 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 04:03:21.588179 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587598 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 04:03:21.588179 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587601 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 04:03:21.588179 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587603 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 04:03:21.588179 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587605 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 04:03:21.588179 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587609 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 04:03:21.588179 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587611 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 04:03:21.588179 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587614 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 04:03:21.588179 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587616 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 04:03:21.588179 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587619 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 04:03:21.588179 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587622 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 04:03:21.588179 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587625 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 04:03:21.588179 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587628 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 04:03:21.588179 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587631 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 04:03:21.588179 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587633 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 04:03:21.588179 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587636 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 04:03:21.588179 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587639 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 04:03:21.588179 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587642 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 04:03:21.588668 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587644 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 04:03:21.588668 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587647 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 04:03:21.588668 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587649 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 04:03:21.588668 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587652 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 04:03:21.588668 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587655 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 04:03:21.588668 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587658 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 04:03:21.588668 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587660 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 04:03:21.588668 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587663 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 04:03:21.588668 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587666 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 04:03:21.588668 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587669 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 04:03:21.588668 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587672 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 04:03:21.588668 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587674 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 04:03:21.588668 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587677 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 04:03:21.588668 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587679 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 04:03:21.588668 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587684 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 04:03:21.588668 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587688 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 04:03:21.588668 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587692 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 04:03:21.588668 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587695 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 04:03:21.588668 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587697 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 04:03:21.589116 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587700 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 04:03:21.589116 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587703 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 04:03:21.589116 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587705 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 04:03:21.589116 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587708 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 04:03:21.589116 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587710 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 04:03:21.589116 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587714 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 04:03:21.589116 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587718 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 04:03:21.589116 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587721 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 04:03:21.589116 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587724 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 04:03:21.589116 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587727 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 04:03:21.589116 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587730 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 04:03:21.589116 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587733 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 04:03:21.589116 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587736 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 04:03:21.589116 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587740 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 04:03:21.589116 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587745 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 04:03:21.589116 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587748 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 04:03:21.589116 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587751 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 04:03:21.589116 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587754 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 04:03:21.589116 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587758 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 04:03:21.589610 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587761 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 04:03:21.589610 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587764 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 04:03:21.589610 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587767 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 04:03:21.589610 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587769 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 04:03:21.589610 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587772 2571 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 04:03:21.589610 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587775 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 04:03:21.589610 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587777 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 04:03:21.589610 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587780 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 04:03:21.589610 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587782 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 04:03:21.589610 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587784 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 04:03:21.589610 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587787 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 04:03:21.589610 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587789 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 04:03:21.589610 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587792 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 04:03:21.589610 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587794 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 04:03:21.589610 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587797 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 04:03:21.589610 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587799 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 04:03:21.589610 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587802 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 04:03:21.589610 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587805 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 04:03:21.589610 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587807 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 04:03:21.590050 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587810 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 04:03:21.590050 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587813 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 04:03:21.590050 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587815 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 04:03:21.590050 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.587818 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 04:03:21.590050 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588623 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 04:03:21.590050 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588632 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 04:03:21.590050 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588635 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 04:03:21.590050 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588638 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 04:03:21.590050 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588641 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 04:03:21.590050 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588644 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 04:03:21.590050 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588647 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 04:03:21.590050 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588650 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 04:03:21.590050 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588653 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 04:03:21.590050 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588655 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 04:03:21.590050 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588658 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 04:03:21.590050 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588661 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 04:03:21.590050 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588664 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 04:03:21.590050 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588666 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 04:03:21.590050 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588669 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 04:03:21.590050 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588672 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 04:03:21.590586 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588675 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 04:03:21.590586 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588677 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 04:03:21.590586 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588680 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 04:03:21.590586 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588682 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 04:03:21.590586 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588687 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 04:03:21.590586 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588691 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 04:03:21.590586 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588695 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 04:03:21.590586 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588699 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 04:03:21.590586 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588703 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 04:03:21.590586 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588707 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 04:03:21.590586 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588711 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 04:03:21.590586 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588715 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 04:03:21.590586 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588719 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 04:03:21.590586 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588722 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 04:03:21.590586 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588727 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 04:03:21.590586 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588731 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 04:03:21.590586 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588734 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 04:03:21.590586 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588738 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 04:03:21.590586 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588741 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 04:03:21.591033 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588746 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 04:03:21.591033 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588751 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 04:03:21.591033 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588755 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 04:03:21.591033 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588760 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 04:03:21.591033 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588766 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 04:03:21.591033 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588771 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 04:03:21.591033 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588777 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 04:03:21.591033 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588781 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 04:03:21.591033 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588784 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 04:03:21.591033 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588792 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 04:03:21.591033 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588798 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 04:03:21.591033 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588801 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 04:03:21.591033 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588805 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 04:03:21.591033 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588808 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 04:03:21.591033 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588810 2571 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 04:03:21.591033 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588813 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 04:03:21.591033 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588816 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 04:03:21.591033 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588819 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 04:03:21.591033 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588822 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 04:03:21.591033 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588824 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 04:03:21.591544 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588827 2571 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 04:03:21.591544 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588830 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 04:03:21.591544 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588834 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 04:03:21.591544 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588837 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 04:03:21.591544 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588841 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 04:03:21.591544 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588845 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 04:03:21.591544 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588849 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 04:03:21.591544 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588852 2571 feature_gate.go:386] unrecognized feature gate: Example Oct 31 04:03:21.591544 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588854 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 04:03:21.591544 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588857 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 04:03:21.591544 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588860 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 04:03:21.591544 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588863 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 04:03:21.591544 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588866 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 04:03:21.591544 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588869 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 04:03:21.591544 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588872 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 04:03:21.591544 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588875 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 04:03:21.591544 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588877 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 04:03:21.591544 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588880 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 04:03:21.591544 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588883 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 04:03:21.591544 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588886 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 04:03:21.592023 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588889 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 04:03:21.592023 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588892 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 04:03:21.592023 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588895 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 04:03:21.592023 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588897 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 04:03:21.592023 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588900 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 04:03:21.592023 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588903 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 04:03:21.592023 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588906 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 04:03:21.592023 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588909 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 04:03:21.592023 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588911 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 04:03:21.592023 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588914 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 04:03:21.592023 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588917 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 04:03:21.592023 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588920 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 04:03:21.592023 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588923 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 04:03:21.592023 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588928 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 04:03:21.592023 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588931 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 04:03:21.592023 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588933 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 04:03:21.592023 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588936 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 04:03:21.592023 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588938 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 04:03:21.592023 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588941 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 04:03:21.592581 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588943 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 04:03:21.592581 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588946 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 04:03:21.592581 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588948 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 04:03:21.592581 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588951 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 04:03:21.592581 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588954 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 04:03:21.592581 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588956 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 04:03:21.592581 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588960 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 04:03:21.592581 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588963 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 04:03:21.592581 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588965 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 04:03:21.592581 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588968 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 04:03:21.592581 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588970 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 04:03:21.592581 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.588973 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 04:03:21.592581 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589497 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 04:03:21.592581 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589503 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 04:03:21.592581 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589506 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 04:03:21.592581 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589510 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 04:03:21.592581 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589512 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 04:03:21.592581 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589515 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 04:03:21.592581 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589518 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 04:03:21.592581 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589521 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 04:03:21.593066 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589524 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 04:03:21.593066 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589527 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 04:03:21.593066 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589530 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 04:03:21.593066 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589532 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 04:03:21.593066 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589535 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 04:03:21.593066 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589538 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 04:03:21.593066 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589542 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 04:03:21.593066 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589544 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 04:03:21.593066 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589547 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 04:03:21.593066 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589550 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 04:03:21.593066 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589552 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 04:03:21.593066 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589555 2571 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 04:03:21.593066 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589557 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 04:03:21.593066 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589560 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 04:03:21.593066 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589562 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 04:03:21.593066 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589565 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 04:03:21.593066 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589567 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 04:03:21.593066 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589570 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 04:03:21.593066 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589572 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 04:03:21.593066 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589575 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 04:03:21.593066 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589577 2571 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 04:03:21.593583 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589580 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 04:03:21.593583 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589582 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 04:03:21.593583 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589585 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 04:03:21.593583 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589587 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 04:03:21.593583 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589590 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 04:03:21.593583 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589592 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 04:03:21.593583 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589595 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 04:03:21.593583 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589598 2571 feature_gate.go:386] unrecognized feature gate: Example Oct 31 04:03:21.593583 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589600 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 04:03:21.593583 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589603 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 04:03:21.593583 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589605 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 04:03:21.593583 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589608 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 04:03:21.593583 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589610 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 04:03:21.593583 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589613 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 04:03:21.593583 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589616 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 04:03:21.593583 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589618 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 04:03:21.593583 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589622 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 04:03:21.593583 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589625 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 04:03:21.593583 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589628 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 04:03:21.594034 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589630 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 04:03:21.594034 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589633 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 04:03:21.594034 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589635 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 04:03:21.594034 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589637 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 04:03:21.594034 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589640 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 04:03:21.594034 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589643 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 04:03:21.594034 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589645 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 04:03:21.594034 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589648 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 04:03:21.594034 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589650 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 04:03:21.594034 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589653 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 04:03:21.594034 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589657 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 04:03:21.594034 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589660 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 04:03:21.594034 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589664 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 04:03:21.594034 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589666 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 04:03:21.594034 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589669 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 04:03:21.594034 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589671 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 04:03:21.594034 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589673 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 04:03:21.594034 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589676 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 04:03:21.594034 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589678 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 04:03:21.594544 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589681 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 04:03:21.594544 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589683 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 04:03:21.594544 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589686 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 04:03:21.594544 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589689 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 04:03:21.594544 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589692 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 04:03:21.594544 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589694 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 04:03:21.594544 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589698 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 04:03:21.594544 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589703 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 04:03:21.594544 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589706 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 04:03:21.594544 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589709 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 04:03:21.594544 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589712 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 04:03:21.594544 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589715 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 04:03:21.594544 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589718 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 04:03:21.594544 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589720 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 04:03:21.594544 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589723 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 04:03:21.594544 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589725 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 04:03:21.594544 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589728 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 04:03:21.594544 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589730 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 04:03:21.594544 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589733 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 04:03:21.595004 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589735 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 04:03:21.595004 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589738 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 04:03:21.595004 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589741 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 04:03:21.595004 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589744 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 04:03:21.595004 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589746 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 04:03:21.595004 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589749 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 04:03:21.595004 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589752 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 04:03:21.595004 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589754 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 04:03:21.595004 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589756 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 04:03:21.595004 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589759 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 04:03:21.595004 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589761 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 04:03:21.595004 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589764 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 04:03:21.595004 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589766 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 04:03:21.595004 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589768 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 04:03:21.595004 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589771 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 04:03:21.595004 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589773 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 04:03:21.595004 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589776 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 04:03:21.595004 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589779 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 04:03:21.595004 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589781 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 04:03:21.595004 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.589784 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 04:03:21.595486 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.590929 2571 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 04:03:21.595486 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.590939 2571 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 04:03:21.595486 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.590947 2571 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 04:03:21.595486 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.590953 2571 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 04:03:21.595486 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.590958 2571 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 04:03:21.595486 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.590962 2571 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 04:03:21.595486 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.590966 2571 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 04:03:21.595486 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.590971 2571 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 04:03:21.595486 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.590974 2571 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 04:03:21.595486 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.590977 2571 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 04:03:21.595486 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.590981 2571 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 04:03:21.595486 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.590985 2571 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 04:03:21.595486 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.590988 2571 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 04:03:21.595486 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.590992 2571 flags.go:64] FLAG: --cgroup-root="" Oct 31 04:03:21.595486 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.590994 2571 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 04:03:21.595486 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.590998 2571 flags.go:64] FLAG: --client-ca-file="" Oct 31 04:03:21.595486 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591000 2571 flags.go:64] FLAG: --cloud-provider="external" Oct 31 04:03:21.595486 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591004 2571 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 04:03:21.595486 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591008 2571 flags.go:64] FLAG: --cluster-domain="" Oct 31 04:03:21.595486 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591011 2571 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 04:03:21.595486 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591014 2571 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 04:03:21.595486 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591017 2571 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 04:03:21.595486 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591020 2571 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 04:03:21.595486 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591024 2571 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 04:03:21.596035 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591027 2571 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 04:03:21.596035 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591030 2571 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 04:03:21.596035 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591033 2571 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 04:03:21.596035 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591037 2571 flags.go:64] FLAG: --contention-profiling="false" Oct 31 04:03:21.596035 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591040 2571 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 04:03:21.596035 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591044 2571 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 04:03:21.596035 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591047 2571 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 04:03:21.596035 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591050 2571 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 04:03:21.596035 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591055 2571 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 04:03:21.596035 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591059 2571 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 04:03:21.596035 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591061 2571 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 04:03:21.596035 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591066 2571 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 04:03:21.596035 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591069 2571 flags.go:64] FLAG: --enable-server="true" Oct 31 04:03:21.596035 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591072 2571 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 04:03:21.596035 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591077 2571 flags.go:64] FLAG: --event-burst="100" Oct 31 04:03:21.596035 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591080 2571 flags.go:64] FLAG: --event-qps="50" Oct 31 04:03:21.596035 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591083 2571 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 04:03:21.596035 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591087 2571 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 04:03:21.596035 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591089 2571 flags.go:64] FLAG: --eviction-hard="" Oct 31 04:03:21.596035 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591094 2571 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 04:03:21.596035 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591097 2571 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 04:03:21.596035 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591119 2571 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 04:03:21.596035 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591123 2571 flags.go:64] FLAG: --eviction-soft="" Oct 31 04:03:21.596035 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591127 2571 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 04:03:21.596035 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591130 2571 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 04:03:21.596700 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591133 2571 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 04:03:21.596700 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591137 2571 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 04:03:21.596700 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591140 2571 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 04:03:21.596700 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591143 2571 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 04:03:21.596700 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591146 2571 flags.go:64] FLAG: --feature-gates="" Oct 31 04:03:21.596700 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591150 2571 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 04:03:21.596700 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591154 2571 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 04:03:21.596700 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591157 2571 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 04:03:21.596700 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591160 2571 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 04:03:21.596700 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591164 2571 flags.go:64] FLAG: --healthz-port="10248" Oct 31 04:03:21.596700 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591167 2571 flags.go:64] FLAG: --help="false" Oct 31 04:03:21.596700 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591170 2571 flags.go:64] FLAG: --hostname-override="ip-10-0-130-57.ec2.internal" Oct 31 04:03:21.596700 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591173 2571 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 04:03:21.596700 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591176 2571 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 04:03:21.596700 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591179 2571 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 04:03:21.596700 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591183 2571 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 04:03:21.596700 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591186 2571 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 04:03:21.596700 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591189 2571 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 04:03:21.596700 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591193 2571 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 04:03:21.596700 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591196 2571 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 04:03:21.596700 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591199 2571 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 04:03:21.596700 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591202 2571 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 04:03:21.596700 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591206 2571 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 04:03:21.596700 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591209 2571 flags.go:64] FLAG: --kube-reserved="" Oct 31 04:03:21.597271 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591211 2571 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 04:03:21.597271 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591214 2571 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 04:03:21.597271 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591217 2571 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 04:03:21.597271 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591220 2571 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 04:03:21.597271 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591223 2571 flags.go:64] FLAG: --lock-file="" Oct 31 04:03:21.597271 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591226 2571 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 04:03:21.597271 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591229 2571 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 04:03:21.597271 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591232 2571 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 04:03:21.597271 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591238 2571 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 04:03:21.597271 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591241 2571 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 04:03:21.597271 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591244 2571 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 04:03:21.597271 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591247 2571 flags.go:64] FLAG: --logging-format="text" Oct 31 04:03:21.597271 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591250 2571 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 04:03:21.597271 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591254 2571 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 04:03:21.597271 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591257 2571 flags.go:64] FLAG: --manifest-url="" Oct 31 04:03:21.597271 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591260 2571 flags.go:64] FLAG: --manifest-url-header="" Oct 31 04:03:21.597271 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591265 2571 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 04:03:21.597271 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591268 2571 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 04:03:21.597271 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591272 2571 flags.go:64] FLAG: --max-pods="110" Oct 31 04:03:21.597271 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591275 2571 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 04:03:21.597271 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591278 2571 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 04:03:21.597271 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591281 2571 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 04:03:21.597271 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591285 2571 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 04:03:21.597271 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591288 2571 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 04:03:21.597271 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591291 2571 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 04:03:21.597977 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591293 2571 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 04:03:21.597977 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591302 2571 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 04:03:21.597977 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591305 2571 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 04:03:21.597977 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591308 2571 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 04:03:21.597977 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591311 2571 flags.go:64] FLAG: --pod-cidr="" Oct 31 04:03:21.597977 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591314 2571 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 04:03:21.597977 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591317 2571 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 04:03:21.597977 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591320 2571 flags.go:64] FLAG: --pods-per-core="0" Oct 31 04:03:21.597977 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591323 2571 flags.go:64] FLAG: --port="10250" Oct 31 04:03:21.597977 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591326 2571 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 04:03:21.597977 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591329 2571 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0260c631d3cc88ab1" Oct 31 04:03:21.597977 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591332 2571 flags.go:64] FLAG: --qos-reserved="" Oct 31 04:03:21.597977 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591335 2571 flags.go:64] FLAG: --read-only-port="10255" Oct 31 04:03:21.597977 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591338 2571 flags.go:64] FLAG: --register-node="true" Oct 31 04:03:21.597977 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591341 2571 flags.go:64] FLAG: --register-with-taints="" Oct 31 04:03:21.597977 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591345 2571 flags.go:64] FLAG: --registry-burst="10" Oct 31 04:03:21.597977 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591348 2571 flags.go:64] FLAG: --registry-qps="5" Oct 31 04:03:21.597977 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591351 2571 flags.go:64] FLAG: --reserved-cpus="" Oct 31 04:03:21.597977 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591353 2571 flags.go:64] FLAG: --reserved-memory="" Oct 31 04:03:21.597977 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591357 2571 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 04:03:21.597977 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591360 2571 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 04:03:21.597977 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591363 2571 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 04:03:21.597977 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591366 2571 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 04:03:21.597977 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591369 2571 flags.go:64] FLAG: --runonce="false" Oct 31 04:03:21.597977 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591372 2571 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 04:03:21.597977 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591376 2571 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 04:03:21.598973 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591379 2571 flags.go:64] FLAG: --seccomp-default="false" Oct 31 04:03:21.598973 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591396 2571 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 04:03:21.598973 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591400 2571 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 04:03:21.598973 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591403 2571 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 04:03:21.598973 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591407 2571 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 04:03:21.598973 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591410 2571 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 04:03:21.598973 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591413 2571 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 04:03:21.598973 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591417 2571 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 04:03:21.598973 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591420 2571 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 04:03:21.598973 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591423 2571 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 04:03:21.598973 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591426 2571 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 04:03:21.598973 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591429 2571 flags.go:64] FLAG: --system-cgroups="" Oct 31 04:03:21.598973 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591432 2571 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 04:03:21.598973 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591438 2571 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 04:03:21.598973 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591441 2571 flags.go:64] FLAG: --tls-cert-file="" Oct 31 04:03:21.598973 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591444 2571 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 04:03:21.598973 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591448 2571 flags.go:64] FLAG: --tls-min-version="" Oct 31 04:03:21.598973 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591450 2571 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 04:03:21.598973 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591453 2571 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 04:03:21.598973 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591456 2571 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 04:03:21.598973 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591459 2571 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 04:03:21.598973 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591462 2571 flags.go:64] FLAG: --v="2" Oct 31 04:03:21.598973 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591467 2571 flags.go:64] FLAG: --version="false" Oct 31 04:03:21.598973 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591471 2571 flags.go:64] FLAG: --vmodule="" Oct 31 04:03:21.598973 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591475 2571 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 04:03:21.599955 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591478 2571 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 04:03:21.599955 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591619 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 04:03:21.599955 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591622 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 04:03:21.599955 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591627 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 04:03:21.599955 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591630 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 04:03:21.599955 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591632 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 04:03:21.599955 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591635 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 04:03:21.599955 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591639 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 04:03:21.599955 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591641 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 04:03:21.599955 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591644 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 04:03:21.599955 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591647 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 04:03:21.599955 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591649 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 04:03:21.599955 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591652 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 04:03:21.599955 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591655 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 04:03:21.599955 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591659 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 04:03:21.599955 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591662 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 04:03:21.599955 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591664 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 04:03:21.599955 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591667 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 04:03:21.599955 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591669 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 04:03:21.599955 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591672 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 04:03:21.600751 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591675 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 04:03:21.600751 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591677 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 04:03:21.600751 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591680 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 04:03:21.600751 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591683 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 04:03:21.600751 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591685 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 04:03:21.600751 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591688 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 04:03:21.600751 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591691 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 04:03:21.600751 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591693 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 04:03:21.600751 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591696 2571 feature_gate.go:386] unrecognized feature gate: Example Oct 31 04:03:21.600751 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591698 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 04:03:21.600751 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591701 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 04:03:21.600751 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591704 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 04:03:21.600751 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591706 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 04:03:21.600751 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591709 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 04:03:21.600751 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591712 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 04:03:21.600751 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591715 2571 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 04:03:21.600751 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591718 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 04:03:21.600751 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591720 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 04:03:21.600751 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591723 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 04:03:21.600751 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591725 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 04:03:21.601217 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591728 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 04:03:21.601217 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591730 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 04:03:21.601217 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591733 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 04:03:21.601217 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591736 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 04:03:21.601217 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591738 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 04:03:21.601217 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591741 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 04:03:21.601217 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591745 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 04:03:21.601217 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591748 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 04:03:21.601217 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591750 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 04:03:21.601217 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591754 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 04:03:21.601217 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591758 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 04:03:21.601217 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591761 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 04:03:21.601217 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591764 2571 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 04:03:21.601217 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591767 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 04:03:21.601217 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591769 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 04:03:21.601217 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591772 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 04:03:21.601217 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591775 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 04:03:21.601217 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591777 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 04:03:21.601217 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591780 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 04:03:21.601217 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591782 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 04:03:21.601759 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591785 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 04:03:21.601759 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591788 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 04:03:21.601759 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591790 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 04:03:21.601759 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591793 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 04:03:21.601759 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591796 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 04:03:21.601759 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591798 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 04:03:21.601759 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591801 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 04:03:21.601759 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591804 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 04:03:21.601759 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591807 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 04:03:21.601759 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591809 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 04:03:21.601759 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591812 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 04:03:21.601759 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591814 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 04:03:21.601759 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591817 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 04:03:21.601759 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591820 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 04:03:21.601759 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591822 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 04:03:21.601759 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591825 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 04:03:21.601759 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591827 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 04:03:21.601759 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591830 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 04:03:21.601759 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591834 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 04:03:21.602207 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591838 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 04:03:21.602207 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591842 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 04:03:21.602207 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591845 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 04:03:21.602207 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591847 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 04:03:21.602207 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591850 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 04:03:21.602207 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591853 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 04:03:21.602207 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591855 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 04:03:21.602207 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591858 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 04:03:21.602207 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591860 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 04:03:21.602207 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591863 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 04:03:21.602207 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591866 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 04:03:21.602207 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591868 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 04:03:21.602207 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591871 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 04:03:21.602207 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591873 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 04:03:21.602207 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591876 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 04:03:21.602207 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591879 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 04:03:21.602207 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591881 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 04:03:21.602207 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591884 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 04:03:21.602207 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591887 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 04:03:21.602698 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591889 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 04:03:21.602698 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591893 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 04:03:21.602698 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591896 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 04:03:21.602698 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591898 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 04:03:21.602698 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591901 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 04:03:21.602698 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591903 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 04:03:21.602698 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591906 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 04:03:21.602698 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591908 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 04:03:21.602698 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591911 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 04:03:21.602698 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.591937 2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 04:03:21.602698 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591975 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 04:03:21.602698 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591979 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 04:03:21.602698 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591982 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 04:03:21.602698 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591986 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 04:03:21.602698 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591989 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 04:03:21.602698 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591992 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 04:03:21.603090 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591995 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 04:03:21.603090 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.591998 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 04:03:21.603090 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592000 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 04:03:21.603090 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592003 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 04:03:21.603090 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592005 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 04:03:21.603090 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592008 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 04:03:21.603090 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592010 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 04:03:21.603090 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592013 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 04:03:21.603090 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592017 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 04:03:21.603090 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592020 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 04:03:21.603090 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592023 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 04:03:21.603090 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592026 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 04:03:21.603090 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592029 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 04:03:21.603090 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592032 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 04:03:21.603090 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592035 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 04:03:21.603090 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592038 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 04:03:21.603090 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592041 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 04:03:21.603090 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592045 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 04:03:21.603090 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592048 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 04:03:21.603600 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592050 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 04:03:21.603600 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592053 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 04:03:21.603600 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592056 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 04:03:21.603600 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592058 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 04:03:21.603600 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592061 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 04:03:21.603600 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592064 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 04:03:21.603600 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592066 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 04:03:21.603600 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592069 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 04:03:21.603600 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592072 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 04:03:21.603600 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592074 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 04:03:21.603600 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592078 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 04:03:21.603600 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592081 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 04:03:21.603600 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592083 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 04:03:21.603600 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592086 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 04:03:21.603600 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592089 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 04:03:21.603600 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592091 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 04:03:21.603600 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592093 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 04:03:21.603600 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592096 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 04:03:21.603600 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592099 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 04:03:21.604039 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592101 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 04:03:21.604039 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592104 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 04:03:21.604039 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592106 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 04:03:21.604039 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592108 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 04:03:21.604039 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592112 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 04:03:21.604039 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592115 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 04:03:21.604039 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592118 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 04:03:21.604039 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592120 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 04:03:21.604039 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592123 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 04:03:21.604039 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592125 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 04:03:21.604039 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592128 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 04:03:21.604039 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592132 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 04:03:21.604039 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592134 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 04:03:21.604039 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592137 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 04:03:21.604039 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592139 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 04:03:21.604039 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592142 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 04:03:21.604039 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592144 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 04:03:21.604039 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592147 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 04:03:21.604039 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592149 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 04:03:21.604039 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592152 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 04:03:21.604528 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592155 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 04:03:21.604528 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592157 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 04:03:21.604528 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592160 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 04:03:21.604528 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592162 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 04:03:21.604528 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592165 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 04:03:21.604528 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592168 2571 feature_gate.go:386] unrecognized feature gate: Example Oct 31 04:03:21.604528 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592170 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 04:03:21.604528 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592173 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 04:03:21.604528 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592176 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 04:03:21.604528 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592178 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 04:03:21.604528 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592181 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 04:03:21.604528 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592183 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 04:03:21.604528 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592186 2571 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 04:03:21.604528 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592188 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 04:03:21.604528 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592191 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 04:03:21.604528 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592193 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 04:03:21.604528 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592196 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 04:03:21.604528 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592198 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 04:03:21.604528 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592201 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 04:03:21.604528 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592203 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 04:03:21.604992 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592206 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 04:03:21.604992 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592208 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 04:03:21.604992 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592211 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 04:03:21.604992 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592214 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 04:03:21.604992 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592217 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 04:03:21.604992 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592220 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 04:03:21.604992 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592223 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 04:03:21.604992 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592227 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 04:03:21.604992 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592230 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 04:03:21.604992 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592233 2571 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 04:03:21.604992 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592236 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 04:03:21.604992 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592238 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 04:03:21.604992 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592241 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 04:03:21.604992 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592244 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 04:03:21.604992 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592246 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 04:03:21.604992 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592249 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 04:03:21.604992 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592251 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 04:03:21.604992 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592255 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 04:03:21.604992 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592258 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 04:03:21.605466 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592260 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 04:03:21.605466 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592263 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 04:03:21.605466 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.592265 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 04:03:21.605466 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.601827 2571 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 04:03:21.605466 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.601844 2571 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 04:03:21.605466 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.602586 2571 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 04:03:21.605466 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.602594 2571 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 04:03:21.605466 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602658 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 04:03:21.605466 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602664 2571 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 04:03:21.605466 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602667 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 04:03:21.605466 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602671 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 04:03:21.605466 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602674 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 04:03:21.605466 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602677 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 04:03:21.605466 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602680 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 04:03:21.605466 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602683 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 04:03:21.605466 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602686 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 04:03:21.605466 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602689 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 04:03:21.605466 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602692 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 04:03:21.605466 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602695 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 04:03:21.605466 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602697 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 04:03:21.605934 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602700 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 04:03:21.605934 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602703 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 04:03:21.605934 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602706 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 04:03:21.605934 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602709 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 04:03:21.605934 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602712 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 04:03:21.605934 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602715 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 04:03:21.605934 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602717 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 04:03:21.605934 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602720 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 04:03:21.605934 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602731 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 04:03:21.605934 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602734 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 04:03:21.605934 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602737 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 04:03:21.605934 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602740 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 04:03:21.605934 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602742 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 04:03:21.605934 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602745 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 04:03:21.605934 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602747 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 04:03:21.605934 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602750 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 04:03:21.605934 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602752 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 04:03:21.605934 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602755 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 04:03:21.605934 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602757 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 04:03:21.605934 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602760 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 04:03:21.606419 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602763 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 04:03:21.606419 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602765 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 04:03:21.606419 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602768 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 04:03:21.606419 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602771 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 04:03:21.606419 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602773 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 04:03:21.606419 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602776 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 04:03:21.606419 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602779 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 04:03:21.606419 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602781 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 04:03:21.606419 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602784 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 04:03:21.606419 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602786 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 04:03:21.606419 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602789 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 04:03:21.606419 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602792 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 04:03:21.606419 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602794 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 04:03:21.606419 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602797 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 04:03:21.606419 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602799 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 04:03:21.606419 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602802 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 04:03:21.606419 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602804 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 04:03:21.606419 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602808 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 04:03:21.606419 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602812 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 04:03:21.606419 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602814 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 04:03:21.606943 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602817 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 04:03:21.606943 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602820 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 04:03:21.606943 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602823 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 04:03:21.606943 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602825 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 04:03:21.606943 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602828 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 04:03:21.606943 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602831 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 04:03:21.606943 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602834 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 04:03:21.606943 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602836 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 04:03:21.606943 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602839 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 04:03:21.606943 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602842 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 04:03:21.606943 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602847 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 04:03:21.606943 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602850 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 04:03:21.606943 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602853 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 04:03:21.606943 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602856 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 04:03:21.606943 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602859 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 04:03:21.606943 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602861 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 04:03:21.606943 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602864 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 04:03:21.606943 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602867 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 04:03:21.606943 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602869 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 04:03:21.607454 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602872 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 04:03:21.607454 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602874 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 04:03:21.607454 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602877 2571 feature_gate.go:386] unrecognized feature gate: Example Oct 31 04:03:21.607454 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602880 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 04:03:21.607454 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602883 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 04:03:21.607454 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602885 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 04:03:21.607454 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602888 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 04:03:21.607454 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602899 2571 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 04:03:21.607454 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602902 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 04:03:21.607454 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602905 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 04:03:21.607454 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602907 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 04:03:21.607454 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602910 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 04:03:21.607454 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602913 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 04:03:21.607454 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602916 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 04:03:21.607454 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602918 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 04:03:21.607454 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602921 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 04:03:21.607454 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602924 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 04:03:21.607454 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602927 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 04:03:21.607454 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602929 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 04:03:21.607924 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602932 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 04:03:21.607924 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602935 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 04:03:21.607924 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602937 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 04:03:21.607924 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602942 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 04:03:21.607924 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602945 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 04:03:21.607924 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602948 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 04:03:21.607924 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602951 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 04:03:21.607924 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602954 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 04:03:21.607924 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602957 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 04:03:21.607924 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602959 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 04:03:21.607924 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602962 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 04:03:21.607924 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602965 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 04:03:21.607924 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602967 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 04:03:21.607924 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602970 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 04:03:21.607924 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.602973 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 04:03:21.608275 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.603001 2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 04:03:21.608275 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603160 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 04:03:21.608275 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603165 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 04:03:21.608275 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603168 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 04:03:21.608275 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603172 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 04:03:21.608275 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603176 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 04:03:21.608275 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603185 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 04:03:21.608275 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603188 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 04:03:21.608275 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603191 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 04:03:21.608275 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603194 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 04:03:21.608275 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603197 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 04:03:21.608275 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603200 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 04:03:21.608275 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603202 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 04:03:21.608275 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603205 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 04:03:21.608275 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603208 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 04:03:21.608275 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603210 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 04:03:21.608776 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603213 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 04:03:21.608776 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603215 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 04:03:21.608776 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603218 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 04:03:21.608776 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603221 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 04:03:21.608776 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603223 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 04:03:21.608776 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603226 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 04:03:21.608776 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603228 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 04:03:21.608776 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603231 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 04:03:21.608776 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603234 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 04:03:21.608776 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603237 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 04:03:21.608776 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603239 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 04:03:21.608776 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603242 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 04:03:21.608776 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603244 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 04:03:21.608776 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603247 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 04:03:21.608776 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603249 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 04:03:21.608776 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603253 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 04:03:21.608776 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603255 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 04:03:21.608776 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603258 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 04:03:21.608776 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603260 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 04:03:21.609222 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603263 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 04:03:21.609222 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603265 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 04:03:21.609222 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603268 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 04:03:21.609222 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603270 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 04:03:21.609222 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603278 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 04:03:21.609222 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603281 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 04:03:21.609222 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603284 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 04:03:21.609222 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603286 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 04:03:21.609222 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603289 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 04:03:21.609222 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603291 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 04:03:21.609222 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603295 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 04:03:21.609222 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603299 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 04:03:21.609222 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603302 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 04:03:21.609222 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603305 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 04:03:21.609222 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603307 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 04:03:21.609222 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603310 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 04:03:21.609222 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603313 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 04:03:21.609222 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603315 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 04:03:21.609222 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603318 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 04:03:21.609707 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603320 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 04:03:21.609707 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603323 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 04:03:21.609707 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603325 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 04:03:21.609707 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603328 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 04:03:21.609707 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603330 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 04:03:21.609707 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603333 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 04:03:21.609707 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603335 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 04:03:21.609707 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603338 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 04:03:21.609707 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603340 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 04:03:21.609707 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603343 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 04:03:21.609707 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603345 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 04:03:21.609707 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603348 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 04:03:21.609707 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603350 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 04:03:21.609707 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603353 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 04:03:21.609707 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603355 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 04:03:21.609707 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603358 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 04:03:21.609707 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603360 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 04:03:21.609707 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603363 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 04:03:21.609707 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603367 2571 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 04:03:21.609707 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603370 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 04:03:21.610181 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603372 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 04:03:21.610181 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603375 2571 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 04:03:21.610181 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603377 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 04:03:21.610181 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603380 2571 feature_gate.go:386] unrecognized feature gate: Example Oct 31 04:03:21.610181 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603397 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 04:03:21.610181 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603400 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 04:03:21.610181 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603403 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 04:03:21.610181 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603405 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 04:03:21.610181 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603408 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 04:03:21.610181 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603411 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 04:03:21.610181 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603413 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 04:03:21.610181 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603416 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 04:03:21.610181 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603418 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 04:03:21.610181 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603421 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 04:03:21.610181 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603423 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 04:03:21.610181 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603426 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 04:03:21.610181 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603428 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 04:03:21.610181 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603431 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 04:03:21.610181 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603433 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 04:03:21.610181 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603436 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 04:03:21.610700 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603438 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 04:03:21.610700 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603441 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 04:03:21.610700 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603444 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 04:03:21.610700 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603446 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 04:03:21.610700 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603449 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 04:03:21.610700 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603451 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 04:03:21.610700 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603454 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 04:03:21.610700 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603456 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 04:03:21.610700 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603459 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 04:03:21.610700 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603461 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 04:03:21.610700 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603464 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 04:03:21.610700 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603467 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 04:03:21.610700 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603470 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 04:03:21.610700 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.603495 2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 04:03:21.610700 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603528 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 04:03:21.611042 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603533 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 04:03:21.611042 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603535 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 04:03:21.611042 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603538 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 04:03:21.611042 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603541 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 04:03:21.611042 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603543 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 04:03:21.611042 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603546 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 04:03:21.611042 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603549 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 04:03:21.611042 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603551 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 04:03:21.611042 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603554 2571 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 04:03:21.611042 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603556 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 04:03:21.611042 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603559 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 04:03:21.611042 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603561 2571 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 04:03:21.611042 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603564 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 04:03:21.611042 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603566 2571 feature_gate.go:386] unrecognized feature gate: Example Oct 31 04:03:21.611042 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603569 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 04:03:21.611042 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603572 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 04:03:21.611042 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603574 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 04:03:21.611042 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603576 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 04:03:21.611042 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603579 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 04:03:21.611042 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603582 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 04:03:21.611521 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603584 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 04:03:21.611521 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603587 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 04:03:21.611521 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603589 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 04:03:21.611521 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603592 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 04:03:21.611521 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603594 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 04:03:21.611521 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603597 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 04:03:21.611521 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603599 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 04:03:21.611521 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603602 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 04:03:21.611521 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603607 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 04:03:21.611521 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603610 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 04:03:21.611521 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603613 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 04:03:21.611521 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603616 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 04:03:21.611521 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603618 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 04:03:21.611521 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603621 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 04:03:21.611521 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603623 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 04:03:21.611521 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603626 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 04:03:21.611521 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603628 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 04:03:21.611521 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603631 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 04:03:21.611521 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603633 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 04:03:21.612042 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603636 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 04:03:21.612042 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603639 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 04:03:21.612042 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603641 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 04:03:21.612042 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603644 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 04:03:21.612042 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603647 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 04:03:21.612042 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603649 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 04:03:21.612042 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603652 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 04:03:21.612042 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603656 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 04:03:21.612042 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603660 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 04:03:21.612042 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603663 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 04:03:21.612042 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603666 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 04:03:21.612042 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603668 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 04:03:21.612042 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603671 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 04:03:21.612042 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603674 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 04:03:21.612042 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603676 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 04:03:21.612042 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603679 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 04:03:21.612042 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603681 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 04:03:21.612042 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603684 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 04:03:21.612042 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603686 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 04:03:21.612508 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603689 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 04:03:21.612508 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603691 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 04:03:21.612508 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603697 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 04:03:21.612508 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603700 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 04:03:21.612508 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603708 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 04:03:21.612508 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603711 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 04:03:21.612508 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603714 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 04:03:21.612508 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603716 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 04:03:21.612508 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603719 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 04:03:21.612508 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603721 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 04:03:21.612508 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603724 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 04:03:21.612508 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603727 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 04:03:21.612508 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603729 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 04:03:21.612508 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603731 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 04:03:21.612508 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603734 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 04:03:21.612508 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603737 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 04:03:21.612508 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603741 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 04:03:21.612508 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603743 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 04:03:21.612508 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603746 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 04:03:21.613087 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603749 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 04:03:21.613087 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603751 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 04:03:21.613087 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603754 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 04:03:21.613087 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603756 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 04:03:21.613087 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603759 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 04:03:21.613087 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603761 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 04:03:21.613087 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603764 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 04:03:21.613087 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603767 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 04:03:21.613087 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603769 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 04:03:21.613087 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603772 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 04:03:21.613087 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603775 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 04:03:21.613087 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603779 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 04:03:21.613087 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603782 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 04:03:21.613087 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603785 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 04:03:21.613087 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603788 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 04:03:21.613087 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603792 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 04:03:21.613087 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603795 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 04:03:21.613087 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603799 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 04:03:21.613087 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603802 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 04:03:21.613563 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603804 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 04:03:21.613563 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603807 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 04:03:21.613563 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603809 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 04:03:21.613563 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603812 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 04:03:21.613563 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603814 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 04:03:21.613563 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603817 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 04:03:21.613563 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603820 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 04:03:21.613563 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603822 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 04:03:21.613563 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:21.603825 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 04:03:21.613563 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.603883 2571 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 04:03:21.613563 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.606357 2571 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 31 04:03:21.613563 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.607409 2571 server.go:1014] "Starting client certificate rotation" Oct 31 04:03:21.613563 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.607504 2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 04:03:21.613563 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.607546 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 04:03:21.613563 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.609629 2571 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 04:03:21.613563 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.611725 2571 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 04:03:21.625231 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.625213 2571 log.go:25] "Validated CRI v1 runtime API" Oct 31 04:03:21.629584 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.629565 2571 log.go:25] "Validated CRI v1 image API" Oct 31 04:03:21.630483 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.630458 2571 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 04:03:21.634746 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.634725 2571 fs.go:135] Filesystem UUIDs: map[12a891d7-462b-47be-9115-93f445a1f6c2:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 dcde1bc0-f106-427f-b4b7-e4b21d62a9c3:/dev/nvme0n1p4] Oct 31 04:03:21.634819 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.634744 2571 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 31 04:03:21.640080 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.639478 2571 manager.go:217] Machine: {Timestamp:2026-10-31 04:03:21.638485655 +0000 UTC m=+0.379270653 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3087976 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec269f6bad3041abad834774c74ed6d0 SystemUUID:ec269f6b-ad30-41ab-ad83-4774c74ed6d0 BootID:827ad6cb-9cb9-459b-9e61-877869987660 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ff:da:55:17 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ff:da:55:17 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ce:24:68:47:16:b5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 04:03:21.640080 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.639695 2571 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 04:03:21.640080 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.639834 2571 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 04:03:21.641341 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.641308 2571 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 04:03:21.641528 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.641344 2571 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-57.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 04:03:21.641576 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.641533 2571 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 04:03:21.641576 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.641543 2571 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 04:03:21.641576 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.641572 2571 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 04:03:21.642629 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.642614 2571 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 04:03:21.643921 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.643909 2571 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 04:03:21.649092 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.649077 2571 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 04:03:21.649250 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.649239 2571 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 04:03:21.650105 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.650094 2571 kubelet.go:493] "Attempting to sync node with API server" Oct 31 04:03:21.650144 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.650112 2571 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 04:03:21.650144 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.650125 2571 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 04:03:21.650144 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.650137 2571 kubelet.go:405] "Adding apiserver pod source" Oct 31 04:03:21.650261 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.650148 2571 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 04:03:21.650561 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.650535 2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 04:03:21.655022 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.655002 2571 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 04:03:21.662257 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.662244 2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 04:03:21.663244 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.663222 2571 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 04:03:21.663318 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.663254 2571 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 04:03:21.663437 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.663426 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 04:03:21.663478 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.663443 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 04:03:21.663478 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.663449 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 04:03:21.663478 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.663455 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 04:03:21.663478 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.663465 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 04:03:21.663478 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.663472 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 04:03:21.664228 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.664219 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 04:03:21.664257 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.664229 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 04:03:21.664257 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.664237 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 04:03:21.664257 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.664243 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 04:03:21.664257 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.664251 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 04:03:21.664430 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.664262 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 04:03:21.665066 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.665054 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 04:03:21.665097 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.665071 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 04:03:21.669413 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.669398 2571 server.go:1263] "Started kubelet" Oct 31 04:03:21.669520 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.669443 2571 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 04:03:21.669597 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.669551 2571 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 04:03:21.669656 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.669620 2571 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 04:03:21.670117 ip-10-0-130-57 systemd[1]: Started Kubernetes Kubelet. Oct 31 04:03:21.671139 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.671120 2571 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 04:03:21.672436 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.672421 2571 server.go:317] "Adding debug handlers to kubelet server" Oct 31 04:03:21.676995 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:21.676975 2571 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 31 04:03:21.678253 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.678240 2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 04:03:21.678874 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.678858 2571 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 04:03:21.678972 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:21.678950 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-57.ec2.internal\" not found" Oct 31 04:03:21.679796 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.679644 2571 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 04:03:21.679796 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.679678 2571 factory.go:55] Registering systemd factory Oct 31 04:03:21.679796 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.679697 2571 factory.go:223] Registration of the systemd container factory successfully Oct 31 04:03:21.680044 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.679978 2571 factory.go:153] Registering CRI-O factory Oct 31 04:03:21.680044 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.679991 2571 factory.go:223] Registration of the crio container factory successfully Oct 31 04:03:21.680044 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.680040 2571 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 04:03:21.680188 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.680058 2571 factory.go:103] Registering Raw factory Oct 31 04:03:21.680188 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.680071 2571 manager.go:1199] Started watching for new ooms in manager Oct 31 04:03:21.680188 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.680084 2571 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 04:03:21.680188 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.680098 2571 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 04:03:21.680366 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.680279 2571 reconstruct.go:97] "Volume reconstruction finished" Oct 31 04:03:21.680366 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.680286 2571 reconciler.go:29] "Reconciler: start to sync state" Oct 31 04:03:21.681408 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.681377 2571 manager.go:319] Starting recovery of all containers Oct 31 04:03:21.690130 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.690105 2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 04:03:21.694248 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.694226 2571 manager.go:324] Recovery completed Oct 31 04:03:21.699252 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.699235 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 04:03:21.700863 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.700831 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasSufficientMemory" Oct 31 04:03:21.700946 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.700880 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 04:03:21.700946 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.700894 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasSufficientPID" Oct 31 04:03:21.701486 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.701472 2571 cpu_manager.go:226] "Starting" policy="none" Oct 31 04:03:21.701486 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.701484 2571 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 04:03:21.701652 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.701504 2571 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 04:03:21.701751 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:21.701724 2571 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-57.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 31 04:03:21.701831 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.701733 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-57.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 31 04:03:21.701831 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.701814 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 04:03:21.702853 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:21.701734 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-57.ec2.internal.18e38171a1c8bd9e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-57.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-57.ec2.internal,},FirstTimestamp:2026-10-31 04:03:21.669164446 +0000 UTC m=+0.409949444,LastTimestamp:2026-10-31 04:03:21.669164446 +0000 UTC m=+0.409949444,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-57.ec2.internal,}" Oct 31 04:03:21.703076 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.703063 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasSufficientMemory" Oct 31 04:03:21.703126 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.703091 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 04:03:21.703126 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.703101 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasSufficientPID" Oct 31 04:03:21.703753 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.703735 2571 policy_none.go:50] "Start" Oct 31 04:03:21.703816 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.703757 2571 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 04:03:21.703816 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.703770 2571 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 04:03:21.705864 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.705853 2571 policy_none.go:44] "Start" Oct 31 04:03:21.718047 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.718030 2571 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 04:03:21.718656 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:21.718634 2571 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 31 04:03:21.718656 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:21.718637 2571 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-57.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 31 04:03:21.718789 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:21.718688 2571 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 31 04:03:21.718979 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:21.718870 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-57.ec2.internal.18e38171a3ac685e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-57.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-57.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-57.ec2.internal,},FirstTimestamp:2026-10-31 04:03:21.700862046 +0000 UTC m=+0.441647045,LastTimestamp:2026-10-31 04:03:21.700862046 +0000 UTC m=+0.441647045,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-57.ec2.internal,}" Oct 31 04:03:21.763213 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.746975 2571 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 04:03:21.763213 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.749981 2571 manager.go:341] "Starting Device Plugin manager" Oct 31 04:03:21.763213 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:21.750027 2571 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 04:03:21.763213 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.750038 2571 server.go:87] "Starting device plugin registration server" Oct 31 04:03:21.763213 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.750204 2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5l7cm" Oct 31 04:03:21.763213 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.750285 2571 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 04:03:21.763213 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.750297 2571 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 04:03:21.763213 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.750396 2571 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 04:03:21.763213 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.750487 2571 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 04:03:21.763213 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.750495 2571 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 04:03:21.763213 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:21.750934 2571 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 04:03:21.763213 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:21.750983 2571 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-57.ec2.internal\" not found" Oct 31 04:03:21.763213 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:21.751102 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-57.ec2.internal.18e38171a3acc92d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-57.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-130-57.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-130-57.ec2.internal,},FirstTimestamp:2026-10-31 04:03:21.700886829 +0000 UTC m=+0.441671831,LastTimestamp:2026-10-31 04:03:21.700886829 +0000 UTC m=+0.441671831,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-57.ec2.internal,}" Oct 31 04:03:21.763213 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.758101 2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 04:03:21.763213 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.758122 2571 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 04:03:21.763213 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.758139 2571 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 04:03:21.763213 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:21.758169 2571 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 31 04:03:21.763213 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.761853 2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5l7cm" Oct 31 04:03:21.803986 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.803965 2571 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 04:03:21.851329 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.851252 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 04:03:21.852186 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.852170 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasSufficientMemory" Oct 31 04:03:21.852253 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.852201 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 04:03:21.852253 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.852210 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasSufficientPID" Oct 31 04:03:21.852253 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.852234 2571 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-57.ec2.internal" Oct 31 04:03:21.859243 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.859225 2571 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-57.ec2.internal"] Oct 31 04:03:21.859289 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.859282 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 04:03:21.860042 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.860026 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasSufficientMemory" Oct 31 04:03:21.860125 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.860052 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 04:03:21.860125 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.860063 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasSufficientPID" Oct 31 04:03:21.862337 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.862325 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 04:03:21.862518 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.862503 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal" Oct 31 04:03:21.862564 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.862538 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 04:03:21.863000 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.862985 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasSufficientMemory" Oct 31 04:03:21.863046 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.863000 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasSufficientMemory" Oct 31 04:03:21.863046 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.863016 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 04:03:21.863046 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.863030 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasSufficientPID" Oct 31 04:03:21.863128 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.863018 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 04:03:21.863128 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.863103 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasSufficientPID" Oct 31 04:03:21.865838 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.865823 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-57.ec2.internal" Oct 31 04:03:21.865889 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.865855 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 04:03:21.866624 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.866604 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasSufficientMemory" Oct 31 04:03:21.866733 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.866636 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 04:03:21.866733 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.866650 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasSufficientPID" Oct 31 04:03:21.874186 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.874169 2571 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-57.ec2.internal" Oct 31 04:03:21.874266 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:21.874191 2571 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-57.ec2.internal\": node \"ip-10-0-130-57.ec2.internal\" not found" Oct 31 04:03:21.881006 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.880985 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e8283c5a79822699286a7ae3ef8db4f6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal\" (UID: \"e8283c5a79822699286a7ae3ef8db4f6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal" Oct 31 04:03:21.881101 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.881012 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8283c5a79822699286a7ae3ef8db4f6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal\" (UID: \"e8283c5a79822699286a7ae3ef8db4f6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal" Oct 31 04:03:21.881101 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.881029 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6536fdc09c4b535fc805fe98673b8597-config\") pod \"kube-apiserver-proxy-ip-10-0-130-57.ec2.internal\" (UID: \"6536fdc09c4b535fc805fe98673b8597\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-57.ec2.internal" Oct 31 04:03:21.894006 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:21.893987 2571 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-57.ec2.internal\" not found" node="ip-10-0-130-57.ec2.internal" Oct 31 04:03:21.896508 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:21.896492 2571 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-57.ec2.internal\" not found" node="ip-10-0-130-57.ec2.internal" Oct 31 04:03:21.913723 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:21.913702 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-57.ec2.internal\" not found" Oct 31 04:03:21.981899 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.981740 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e8283c5a79822699286a7ae3ef8db4f6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal\" (UID: \"e8283c5a79822699286a7ae3ef8db4f6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal" Oct 31 04:03:21.981899 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.981903 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8283c5a79822699286a7ae3ef8db4f6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal\" (UID: \"e8283c5a79822699286a7ae3ef8db4f6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal" Oct 31 04:03:21.982071 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.981919 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6536fdc09c4b535fc805fe98673b8597-config\") pod \"kube-apiserver-proxy-ip-10-0-130-57.ec2.internal\" (UID: \"6536fdc09c4b535fc805fe98673b8597\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-57.ec2.internal" Oct 31 04:03:21.982071 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.981956 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6536fdc09c4b535fc805fe98673b8597-config\") pod \"kube-apiserver-proxy-ip-10-0-130-57.ec2.internal\" (UID: \"6536fdc09c4b535fc805fe98673b8597\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-57.ec2.internal" Oct 31 04:03:21.982071 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.981857 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e8283c5a79822699286a7ae3ef8db4f6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal\" (UID: \"e8283c5a79822699286a7ae3ef8db4f6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal" Oct 31 04:03:21.982071 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:21.982013 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8283c5a79822699286a7ae3ef8db4f6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal\" (UID: \"e8283c5a79822699286a7ae3ef8db4f6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal" Oct 31 04:03:22.014101 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:22.014065 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-57.ec2.internal\" not found" Oct 31 04:03:22.114602 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:22.114521 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-57.ec2.internal\" not found" Oct 31 04:03:23.840016 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:23.839988 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal" Oct 31 04:03:23.842991 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:23.842967 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-57.ec2.internal" Oct 31 04:03:23.856534 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:23.856515 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-57.ec2.internal\" not found" Oct 31 04:03:23.956820 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:23.956782 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-57.ec2.internal\" not found" Oct 31 04:03:24.057318 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:24.057239 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-57.ec2.internal\" not found" Oct 31 04:03:24.157840 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:24.157808 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-57.ec2.internal\" not found" Oct 31 04:03:24.248320 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:24.248291 2571 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 31 04:03:24.248755 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:24.248464 2571 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 04:03:24.258572 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:24.258553 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-57.ec2.internal\" not found" Oct 31 04:03:24.337032 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:24.336976 2571 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 04:03:24.358747 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:24.358719 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-57.ec2.internal\" not found" Oct 31 04:03:24.408600 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:24.408567 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 03:58:23 +0000 UTC" deadline="2028-05-15 07:02:29.50834468 +0000 UTC" Oct 31 04:03:24.408600 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:24.408595 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13490h59m5.099752965s" Oct 31 04:03:24.459061 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:24.459018 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-57.ec2.internal\" not found" Oct 31 04:03:24.479904 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:24.479880 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8283c5a79822699286a7ae3ef8db4f6.slice/crio-8a6619e28986b25f93c7224377a9cf5bac47e19970b0c0673ba89bff2552643f WatchSource:0}: Error finding container 8a6619e28986b25f93c7224377a9cf5bac47e19970b0c0673ba89bff2552643f: Status 404 returned error can't find the container with id 8a6619e28986b25f93c7224377a9cf5bac47e19970b0c0673ba89bff2552643f Oct 31 04:03:24.480600 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:24.480579 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6536fdc09c4b535fc805fe98673b8597.slice/crio-828cacb0290babc13ca9b7a25aebe8bc490f563d05ae84a04adfc94da818208f WatchSource:0}: Error finding container 828cacb0290babc13ca9b7a25aebe8bc490f563d05ae84a04adfc94da818208f: Status 404 returned error can't find the container with id 828cacb0290babc13ca9b7a25aebe8bc490f563d05ae84a04adfc94da818208f Oct 31 04:03:24.486392 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:24.486379 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 04:03:24.527742 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:24.527716 2571 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 04:03:24.620038 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:24.619961 2571 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal" Oct 31 04:03:24.648280 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:24.648246 2571 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 04:03:24.650521 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:24.650503 2571 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-57.ec2.internal" Oct 31 04:03:24.671674 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:24.671647 2571 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 04:03:24.685882 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:24.685857 2571 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 04:03:25.083374 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.083340 2571 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 04:03:25.292441 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.292407 2571 apiserver.go:52] "Watching apiserver" Oct 31 04:03:25.320228 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.320176 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 31 04:03:25.322634 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.322603 2571 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 04:03:25.322999 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.322978 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-bhgvp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sqbnh","openshift-cluster-node-tuning-operator/tuned-26vsf","openshift-image-registry/node-ca-h9sww","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal","openshift-multus/multus-additional-cni-plugins-6zzkx","kube-system/global-pull-secret-syncer-hnbll","kube-system/konnectivity-agent-hm5cr","openshift-network-diagnostics/network-check-target-4b7d5","openshift-ovn-kubernetes/ovnkube-node-tmt7k","kube-system/kube-apiserver-proxy-ip-10-0-130-57.ec2.internal","openshift-dns/node-resolver-nbl9z","openshift-multus/network-metrics-daemon-clnlz","openshift-multus/multus-6d7cp"] Oct 31 04:03:25.325628 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.325602 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hnbll" Oct 31 04:03:25.325774 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:25.325688 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hnbll" podUID="c1560abd-3afc-4f94-b0bb-ac7ef8b2464a" Oct 31 04:03:25.328305 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.327809 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hm5cr" Oct 31 04:03:25.331268 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.331245 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:03:25.333449 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.333381 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-h9sww" Oct 31 04:03:25.337478 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.336650 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bhgvp" Oct 31 04:03:25.338298 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.338280 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6zzkx" Oct 31 04:03:25.340210 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.340189 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4b7d5" Oct 31 04:03:25.340326 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:25.340257 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4b7d5" podUID="aef43d62-9761-4262-8183-01b9890462a7" Oct 31 04:03:25.343468 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.343447 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clnlz" Oct 31 04:03:25.343597 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:25.343542 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-clnlz" podUID="97c312b1-4ca5-49c8-8c8d-d247c0fa14ac" Oct 31 04:03:25.344263 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.344242 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xzg5s\"" Oct 31 04:03:25.344566 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.344546 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 04:03:25.344953 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.344922 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 04:03:25.347111 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.347095 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sqbnh" Oct 31 04:03:25.347299 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.347272 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 04:03:25.350000 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.349974 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 04:03:25.350105 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.350070 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 04:03:25.350159 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.350148 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 04:03:25.350245 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.350235 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9gl9k\"" Oct 31 04:03:25.350321 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.350257 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 04:03:25.350321 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.350298 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-45qpm\"" Oct 31 04:03:25.350581 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.350557 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.352433 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.352409 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 04:03:25.352552 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.352498 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 04:03:25.352598 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.352578 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 04:03:25.352635 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.352626 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-brbhc\"" Oct 31 04:03:25.352932 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.352911 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6d7cp" Oct 31 04:03:25.354024 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.353659 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 04:03:25.354505 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.354154 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 04:03:25.354505 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.354167 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kfpck\"" Oct 31 04:03:25.354505 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.354215 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 04:03:25.354505 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.354465 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 04:03:25.354505 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.354478 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 04:03:25.355399 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.355379 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nbl9z" Oct 31 04:03:25.372508 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.372476 2571 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 04:03:25.372654 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.372559 2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2k24r" Oct 31 04:03:25.379108 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.379074 2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2k24r" Oct 31 04:03:25.406859 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.406801 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-57.ec2.internal" event={"ID":"6536fdc09c4b535fc805fe98673b8597","Type":"ContainerStarted","Data":"828cacb0290babc13ca9b7a25aebe8bc490f563d05ae84a04adfc94da818208f"} Oct 31 04:03:25.408326 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.408293 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal" event={"ID":"e8283c5a79822699286a7ae3ef8db4f6","Type":"ContainerStarted","Data":"8a6619e28986b25f93c7224377a9cf5bac47e19970b0c0673ba89bff2552643f"} Oct 31 04:03:25.422772 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.422740 2571 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 04:03:25.432592 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.432556 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7f6cd22a-499a-4c35-b408-b2aa9b78620d-env-overrides\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.432592 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.432594 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-host-run-k8s-cni-cncf-io\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:03:25.432835 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.432617 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/31221235-f863-445b-a5ca-375aae905d38-host\") pod \"node-ca-h9sww\" (UID: \"31221235-f863-445b-a5ca-375aae905d38\") " pod="openshift-image-registry/node-ca-h9sww" Oct 31 04:03:25.432835 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.432675 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6d336e6e-feb0-40b2-a625-0e21d25bfac5-cni-binary-copy\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:03:25.432835 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.432733 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3a05c43f-e2db-44c0-b9bd-4bdabca92a98-hosts-file\") pod \"node-resolver-nbl9z\" (UID: \"3a05c43f-e2db-44c0-b9bd-4bdabca92a98\") " pod="openshift-dns/node-resolver-nbl9z" Oct 31 04:03:25.432835 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.432789 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aaae68b8-1654-474f-9e74-45d1e2025d20-cni-binary-copy\") pod \"multus-additional-cni-plugins-6zzkx\" (UID: \"aaae68b8-1654-474f-9e74-45d1e2025d20\") " pod="openshift-multus/multus-additional-cni-plugins-6zzkx" Oct 31 04:03:25.432835 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.432824 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-host-cni-netd\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.433014 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.432878 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7f6cd22a-499a-4c35-b408-b2aa9b78620d-ovn-node-metrics-cert\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.433014 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.432908 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.433014 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.432946 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c1560abd-3afc-4f94-b0bb-ac7ef8b2464a-kubelet-config\") pod \"global-pull-secret-syncer-hnbll\" (UID: \"c1560abd-3afc-4f94-b0bb-ac7ef8b2464a\") " pod="kube-system/global-pull-secret-syncer-hnbll" Oct 31 04:03:25.433014 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.432970 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd5qc\" (UniqueName: \"kubernetes.io/projected/aaae68b8-1654-474f-9e74-45d1e2025d20-kube-api-access-nd5qc\") pod \"multus-additional-cni-plugins-6zzkx\" (UID: \"aaae68b8-1654-474f-9e74-45d1e2025d20\") " pod="openshift-multus/multus-additional-cni-plugins-6zzkx" Oct 31 04:03:25.433014 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.433005 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7f6cd22a-499a-4c35-b408-b2aa9b78620d-ovnkube-script-lib\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.433197 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.433030 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f5pv\" (UniqueName: \"kubernetes.io/projected/97c312b1-4ca5-49c8-8c8d-d247c0fa14ac-kube-api-access-2f5pv\") pod \"network-metrics-daemon-clnlz\" (UID: \"97c312b1-4ca5-49c8-8c8d-d247c0fa14ac\") " pod="openshift-multus/network-metrics-daemon-clnlz" Oct 31 04:03:25.433197 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.433051 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5b6d69de-a860-4c82-a03f-23ab7905a5d7-registration-dir\") pod \"aws-ebs-csi-driver-node-sqbnh\" (UID: \"5b6d69de-a860-4c82-a03f-23ab7905a5d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sqbnh" Oct 31 04:03:25.433197 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.433066 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-multus-socket-dir-parent\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:03:25.433197 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.433083 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mpvl\" (UniqueName: \"kubernetes.io/projected/6d336e6e-feb0-40b2-a625-0e21d25bfac5-kube-api-access-9mpvl\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:03:25.433197 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.433107 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/89ea384b-705f-47a9-8df6-d3c266ed6761-konnectivity-ca\") pod \"konnectivity-agent-hm5cr\" (UID: \"89ea384b-705f-47a9-8df6-d3c266ed6761\") " pod="kube-system/konnectivity-agent-hm5cr" Oct 31 04:03:25.433197 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.433123 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-sys\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:03:25.433197 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.433139 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/31221235-f863-445b-a5ca-375aae905d38-serviceca\") pod \"node-ca-h9sww\" (UID: \"31221235-f863-445b-a5ca-375aae905d38\") " pod="openshift-image-registry/node-ca-h9sww" Oct 31 04:03:25.433197 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.433188 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-host-slash\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.433520 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.433204 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c1560abd-3afc-4f94-b0bb-ac7ef8b2464a-dbus\") pod \"global-pull-secret-syncer-hnbll\" (UID: \"c1560abd-3afc-4f94-b0bb-ac7ef8b2464a\") " pod="kube-system/global-pull-secret-syncer-hnbll" Oct 31 04:03:25.433520 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.433221 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-var-lib-openvswitch\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.433520 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.433241 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ce86be55-219f-46fb-bff6-6dcf87a40ef4-host-slash\") pod \"iptables-alerter-bhgvp\" (UID: \"ce86be55-219f-46fb-bff6-6dcf87a40ef4\") " pod="openshift-network-operator/iptables-alerter-bhgvp" Oct 31 04:03:25.433520 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.433255 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5b6d69de-a860-4c82-a03f-23ab7905a5d7-etc-selinux\") pod \"aws-ebs-csi-driver-node-sqbnh\" (UID: \"5b6d69de-a860-4c82-a03f-23ab7905a5d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sqbnh" Oct 31 04:03:25.433520 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.433270 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-multus-conf-dir\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:03:25.433520 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.433291 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-etc-modprobe-d\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:03:25.433520 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.433305 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-node-log\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.433520 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.433318 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-log-socket\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.433520 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.433351 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5b6d69de-a860-4c82-a03f-23ab7905a5d7-socket-dir\") pod \"aws-ebs-csi-driver-node-sqbnh\" (UID: \"5b6d69de-a860-4c82-a03f-23ab7905a5d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sqbnh" Oct 31 04:03:25.433520 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.433377 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5b6d69de-a860-4c82-a03f-23ab7905a5d7-sys-fs\") pod \"aws-ebs-csi-driver-node-sqbnh\" (UID: \"5b6d69de-a860-4c82-a03f-23ab7905a5d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sqbnh" Oct 31 04:03:25.433520 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.433392 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-var-lib-kubelet\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:03:25.433520 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.433405 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be0026cb-89bc-413a-bfe3-523d980397fa-tmp\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:03:25.433520 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.433420 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aaae68b8-1654-474f-9e74-45d1e2025d20-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6zzkx\" (UID: \"aaae68b8-1654-474f-9e74-45d1e2025d20\") " pod="openshift-multus/multus-additional-cni-plugins-6zzkx" Oct 31 04:03:25.433520 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.433475 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-host-var-lib-cni-multus\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:03:25.433520 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.433491 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-etc-sysctl-d\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:03:25.433520 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.433510 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-run-systemd\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.434048 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.433533 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-etc-openvswitch\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.434048 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.433548 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-run-openvswitch\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.434048 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.433564 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-run-ovn\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.434048 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.433579 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97c312b1-4ca5-49c8-8c8d-d247c0fa14ac-metrics-certs\") pod \"network-metrics-daemon-clnlz\" (UID: \"97c312b1-4ca5-49c8-8c8d-d247c0fa14ac\") " pod="openshift-multus/network-metrics-daemon-clnlz" Oct 31 04:03:25.434048 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.433596 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w6z5\" (UniqueName: \"kubernetes.io/projected/5b6d69de-a860-4c82-a03f-23ab7905a5d7-kube-api-access-7w6z5\") pod \"aws-ebs-csi-driver-node-sqbnh\" (UID: \"5b6d69de-a860-4c82-a03f-23ab7905a5d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sqbnh" Oct 31 04:03:25.434048 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.433611 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-host-run-multus-certs\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:03:25.434048 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.433626 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ce86be55-219f-46fb-bff6-6dcf87a40ef4-iptables-alerter-script\") pod \"iptables-alerter-bhgvp\" (UID: \"ce86be55-219f-46fb-bff6-6dcf87a40ef4\") " pod="openshift-network-operator/iptables-alerter-bhgvp" Oct 31 04:03:25.434048 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.433640 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-cnibin\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:03:25.434048 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.433655 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-host-run-netns\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:03:25.434048 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.433669 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-host-var-lib-kubelet\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:03:25.434048 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.433683 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6d336e6e-feb0-40b2-a625-0e21d25bfac5-multus-daemon-config\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:03:25.434048 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.433697 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-etc-sysctl-conf\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:03:25.434048 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.433712 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn7px\" (UniqueName: \"kubernetes.io/projected/be0026cb-89bc-413a-bfe3-523d980397fa-kube-api-access-dn7px\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:03:25.434048 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.433726 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-host-run-netns\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.434048 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.433742 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5b6d69de-a860-4c82-a03f-23ab7905a5d7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sqbnh\" (UID: \"5b6d69de-a860-4c82-a03f-23ab7905a5d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sqbnh" Oct 31 04:03:25.434048 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.433760 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-hostroot\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:03:25.434577 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.433779 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5nf8\" (UniqueName: \"kubernetes.io/projected/31221235-f863-445b-a5ca-375aae905d38-kube-api-access-x5nf8\") pod \"node-ca-h9sww\" (UID: \"31221235-f863-445b-a5ca-375aae905d38\") " pod="openshift-image-registry/node-ca-h9sww" Oct 31 04:03:25.434577 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.433795 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/aaae68b8-1654-474f-9e74-45d1e2025d20-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6zzkx\" (UID: \"aaae68b8-1654-474f-9e74-45d1e2025d20\") " pod="openshift-multus/multus-additional-cni-plugins-6zzkx" Oct 31 04:03:25.434577 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.433812 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/aaae68b8-1654-474f-9e74-45d1e2025d20-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6zzkx\" (UID: \"aaae68b8-1654-474f-9e74-45d1e2025d20\") " pod="openshift-multus/multus-additional-cni-plugins-6zzkx" Oct 31 04:03:25.434577 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.433827 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5hsv\" (UniqueName: \"kubernetes.io/projected/ce86be55-219f-46fb-bff6-6dcf87a40ef4-kube-api-access-x5hsv\") pod \"iptables-alerter-bhgvp\" (UID: \"ce86be55-219f-46fb-bff6-6dcf87a40ef4\") " pod="openshift-network-operator/iptables-alerter-bhgvp" Oct 31 04:03:25.434577 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.433841 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-etc-systemd\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:03:25.434577 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.433855 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-lib-modules\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:03:25.434577 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.433868 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-host\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:03:25.434577 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.433883 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aaae68b8-1654-474f-9e74-45d1e2025d20-cnibin\") pod \"multus-additional-cni-plugins-6zzkx\" (UID: \"aaae68b8-1654-474f-9e74-45d1e2025d20\") " pod="openshift-multus/multus-additional-cni-plugins-6zzkx" Oct 31 04:03:25.434577 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.433984 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-host-cni-bin\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.434577 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.434013 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjq6x\" (UniqueName: \"kubernetes.io/projected/aef43d62-9761-4262-8183-01b9890462a7-kube-api-access-kjq6x\") pod \"network-check-target-4b7d5\" (UID: \"aef43d62-9761-4262-8183-01b9890462a7\") " pod="openshift-network-diagnostics/network-check-target-4b7d5" Oct 31 04:03:25.434577 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.434030 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-system-cni-dir\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:03:25.434577 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.434047 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-etc-kubernetes\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:03:25.434577 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.434064 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhkfw\" (UniqueName: \"kubernetes.io/projected/3a05c43f-e2db-44c0-b9bd-4bdabca92a98-kube-api-access-vhkfw\") pod \"node-resolver-nbl9z\" (UID: \"3a05c43f-e2db-44c0-b9bd-4bdabca92a98\") " pod="openshift-dns/node-resolver-nbl9z" Oct 31 04:03:25.434577 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.434097 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/89ea384b-705f-47a9-8df6-d3c266ed6761-agent-certs\") pod \"konnectivity-agent-hm5cr\" (UID: \"89ea384b-705f-47a9-8df6-d3c266ed6761\") " pod="kube-system/konnectivity-agent-hm5cr" Oct 31 04:03:25.434577 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.434112 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-run\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:03:25.434577 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.434128 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phq2f\" (UniqueName: \"kubernetes.io/projected/7f6cd22a-499a-4c35-b408-b2aa9b78620d-kube-api-access-phq2f\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.435099 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.434144 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c1560abd-3afc-4f94-b0bb-ac7ef8b2464a-original-pull-secret\") pod \"global-pull-secret-syncer-hnbll\" (UID: \"c1560abd-3afc-4f94-b0bb-ac7ef8b2464a\") " pod="kube-system/global-pull-secret-syncer-hnbll" Oct 31 04:03:25.435099 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.434165 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5b6d69de-a860-4c82-a03f-23ab7905a5d7-device-dir\") pod \"aws-ebs-csi-driver-node-sqbnh\" (UID: \"5b6d69de-a860-4c82-a03f-23ab7905a5d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sqbnh" Oct 31 04:03:25.435099 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.434191 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-multus-cni-dir\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:03:25.435099 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.434208 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-etc-sysconfig\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:03:25.435099 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.434225 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-etc-kubernetes\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:03:25.435099 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.434247 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aaae68b8-1654-474f-9e74-45d1e2025d20-system-cni-dir\") pod \"multus-additional-cni-plugins-6zzkx\" (UID: \"aaae68b8-1654-474f-9e74-45d1e2025d20\") " pod="openshift-multus/multus-additional-cni-plugins-6zzkx" Oct 31 04:03:25.435099 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.434264 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-host-kubelet\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.435099 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.434286 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-systemd-units\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.435099 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.434311 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7f6cd22a-499a-4c35-b408-b2aa9b78620d-ovnkube-config\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.435099 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.434349 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-host-var-lib-cni-bin\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:03:25.435099 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.434366 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a05c43f-e2db-44c0-b9bd-4bdabca92a98-tmp-dir\") pod \"node-resolver-nbl9z\" (UID: \"3a05c43f-e2db-44c0-b9bd-4bdabca92a98\") " pod="openshift-dns/node-resolver-nbl9z" Oct 31 04:03:25.435099 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.434383 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/be0026cb-89bc-413a-bfe3-523d980397fa-etc-tuned\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:03:25.435099 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.434425 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-host-run-ovn-kubernetes\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.444895 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.444858 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 04:03:25.444895 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.444864 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 04:03:25.445141 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.444900 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 04:03:25.445141 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.444928 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 04:03:25.445141 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.444981 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 04:03:25.445141 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.444986 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 04:03:25.445141 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.445057 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 04:03:25.445141 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.445069 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 04:03:25.445419 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.445361 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ldktz\"" Oct 31 04:03:25.445419 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.445368 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pjn2p\"" Oct 31 04:03:25.445419 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.445397 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 04:03:25.445419 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.445409 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 04:03:25.445575 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.445420 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 04:03:25.445575 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.445426 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ctbtg\"" Oct 31 04:03:25.445575 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.445443 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-vc7dp\"" Oct 31 04:03:25.445575 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.445500 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 04:03:25.539178 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.535610 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-run\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:03:25.539178 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.535658 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phq2f\" (UniqueName: \"kubernetes.io/projected/7f6cd22a-499a-4c35-b408-b2aa9b78620d-kube-api-access-phq2f\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.539178 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.535678 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c1560abd-3afc-4f94-b0bb-ac7ef8b2464a-original-pull-secret\") pod \"global-pull-secret-syncer-hnbll\" (UID: \"c1560abd-3afc-4f94-b0bb-ac7ef8b2464a\") " pod="kube-system/global-pull-secret-syncer-hnbll" Oct 31 04:03:25.539178 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.535700 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5b6d69de-a860-4c82-a03f-23ab7905a5d7-device-dir\") pod \"aws-ebs-csi-driver-node-sqbnh\" (UID: \"5b6d69de-a860-4c82-a03f-23ab7905a5d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sqbnh" Oct 31 04:03:25.539178 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.535720 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-multus-cni-dir\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:03:25.539178 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.535741 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-etc-sysconfig\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:03:25.539178 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.535760 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-etc-kubernetes\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:03:25.539178 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.535837 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-etc-kubernetes\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:03:25.539178 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.535954 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-run\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:03:25.539178 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:25.536312 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 04:03:25.539178 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:25.536411 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1560abd-3afc-4f94-b0bb-ac7ef8b2464a-original-pull-secret podName:c1560abd-3afc-4f94-b0bb-ac7ef8b2464a nodeName:}" failed. No retries permitted until 2026-10-31 04:03:26.036379354 +0000 UTC m=+3.136217239 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c1560abd-3afc-4f94-b0bb-ac7ef8b2464a-original-pull-secret") pod "global-pull-secret-syncer-hnbll" (UID: "c1560abd-3afc-4f94-b0bb-ac7ef8b2464a") : object "kube-system"/"original-pull-secret" not registered Oct 31 04:03:25.539178 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.536646 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5b6d69de-a860-4c82-a03f-23ab7905a5d7-device-dir\") pod \"aws-ebs-csi-driver-node-sqbnh\" (UID: \"5b6d69de-a860-4c82-a03f-23ab7905a5d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sqbnh" Oct 31 04:03:25.539178 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.536721 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-multus-cni-dir\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:03:25.539178 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.536763 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-etc-sysconfig\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:03:25.539178 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.536854 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aaae68b8-1654-474f-9e74-45d1e2025d20-system-cni-dir\") pod \"multus-additional-cni-plugins-6zzkx\" (UID: \"aaae68b8-1654-474f-9e74-45d1e2025d20\") " pod="openshift-multus/multus-additional-cni-plugins-6zzkx" Oct 31 04:03:25.539178 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.536878 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-host-kubelet\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.539178 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.536906 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-systemd-units\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.540129 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.536936 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7f6cd22a-499a-4c35-b408-b2aa9b78620d-ovnkube-config\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.540129 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.536960 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-host-var-lib-cni-bin\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:03:25.540129 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.536992 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a05c43f-e2db-44c0-b9bd-4bdabca92a98-tmp-dir\") pod \"node-resolver-nbl9z\" (UID: \"3a05c43f-e2db-44c0-b9bd-4bdabca92a98\") " pod="openshift-dns/node-resolver-nbl9z" Oct 31 04:03:25.540129 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.537014 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/be0026cb-89bc-413a-bfe3-523d980397fa-etc-tuned\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:03:25.540129 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.537034 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-host-run-ovn-kubernetes\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.540129 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.537052 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7f6cd22a-499a-4c35-b408-b2aa9b78620d-env-overrides\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.540129 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.537069 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-host-run-k8s-cni-cncf-io\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:03:25.540129 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.537087 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/31221235-f863-445b-a5ca-375aae905d38-host\") pod \"node-ca-h9sww\" (UID: \"31221235-f863-445b-a5ca-375aae905d38\") " pod="openshift-image-registry/node-ca-h9sww" Oct 31 04:03:25.540129 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.537104 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6d336e6e-feb0-40b2-a625-0e21d25bfac5-cni-binary-copy\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:03:25.540129 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.537123 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3a05c43f-e2db-44c0-b9bd-4bdabca92a98-hosts-file\") pod \"node-resolver-nbl9z\" (UID: \"3a05c43f-e2db-44c0-b9bd-4bdabca92a98\") " pod="openshift-dns/node-resolver-nbl9z" Oct 31 04:03:25.540129 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.537142 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aaae68b8-1654-474f-9e74-45d1e2025d20-cni-binary-copy\") pod \"multus-additional-cni-plugins-6zzkx\" (UID: \"aaae68b8-1654-474f-9e74-45d1e2025d20\") " pod="openshift-multus/multus-additional-cni-plugins-6zzkx" Oct 31 04:03:25.540129 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.537167 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-host-cni-netd\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.540129 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.537194 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7f6cd22a-499a-4c35-b408-b2aa9b78620d-ovn-node-metrics-cert\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.540129 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.537219 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.540129 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.537249 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c1560abd-3afc-4f94-b0bb-ac7ef8b2464a-kubelet-config\") pod \"global-pull-secret-syncer-hnbll\" (UID: \"c1560abd-3afc-4f94-b0bb-ac7ef8b2464a\") " pod="kube-system/global-pull-secret-syncer-hnbll" Oct 31 04:03:25.540129 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.537279 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd5qc\" (UniqueName: \"kubernetes.io/projected/aaae68b8-1654-474f-9e74-45d1e2025d20-kube-api-access-nd5qc\") pod \"multus-additional-cni-plugins-6zzkx\" (UID: \"aaae68b8-1654-474f-9e74-45d1e2025d20\") " pod="openshift-multus/multus-additional-cni-plugins-6zzkx" Oct 31 04:03:25.540129 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.537312 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7f6cd22a-499a-4c35-b408-b2aa9b78620d-ovnkube-script-lib\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.540926 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.537363 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f5pv\" (UniqueName: \"kubernetes.io/projected/97c312b1-4ca5-49c8-8c8d-d247c0fa14ac-kube-api-access-2f5pv\") pod \"network-metrics-daemon-clnlz\" (UID: \"97c312b1-4ca5-49c8-8c8d-d247c0fa14ac\") " pod="openshift-multus/network-metrics-daemon-clnlz" Oct 31 04:03:25.540926 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.537397 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5b6d69de-a860-4c82-a03f-23ab7905a5d7-registration-dir\") pod \"aws-ebs-csi-driver-node-sqbnh\" (UID: \"5b6d69de-a860-4c82-a03f-23ab7905a5d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sqbnh" Oct 31 04:03:25.540926 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.537422 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-multus-socket-dir-parent\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:03:25.540926 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.537452 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mpvl\" (UniqueName: \"kubernetes.io/projected/6d336e6e-feb0-40b2-a625-0e21d25bfac5-kube-api-access-9mpvl\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:03:25.540926 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.537481 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/89ea384b-705f-47a9-8df6-d3c266ed6761-konnectivity-ca\") pod \"konnectivity-agent-hm5cr\" (UID: \"89ea384b-705f-47a9-8df6-d3c266ed6761\") " pod="kube-system/konnectivity-agent-hm5cr" Oct 31 04:03:25.540926 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.537509 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-sys\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:03:25.540926 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.537539 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/31221235-f863-445b-a5ca-375aae905d38-serviceca\") pod \"node-ca-h9sww\" (UID: \"31221235-f863-445b-a5ca-375aae905d38\") " pod="openshift-image-registry/node-ca-h9sww" Oct 31 04:03:25.540926 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.537563 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-host-slash\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.540926 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.537590 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c1560abd-3afc-4f94-b0bb-ac7ef8b2464a-dbus\") pod \"global-pull-secret-syncer-hnbll\" (UID: \"c1560abd-3afc-4f94-b0bb-ac7ef8b2464a\") " pod="kube-system/global-pull-secret-syncer-hnbll" Oct 31 04:03:25.540926 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.537622 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-var-lib-openvswitch\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.540926 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.537653 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ce86be55-219f-46fb-bff6-6dcf87a40ef4-host-slash\") pod \"iptables-alerter-bhgvp\" (UID: \"ce86be55-219f-46fb-bff6-6dcf87a40ef4\") " pod="openshift-network-operator/iptables-alerter-bhgvp" Oct 31 04:03:25.540926 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.537682 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5b6d69de-a860-4c82-a03f-23ab7905a5d7-etc-selinux\") pod \"aws-ebs-csi-driver-node-sqbnh\" (UID: \"5b6d69de-a860-4c82-a03f-23ab7905a5d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sqbnh" Oct 31 04:03:25.540926 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.537707 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-multus-conf-dir\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:03:25.540926 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.537738 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-etc-modprobe-d\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:03:25.540926 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.537769 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-node-log\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.540926 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.537800 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-log-socket\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.540926 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.537825 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5b6d69de-a860-4c82-a03f-23ab7905a5d7-socket-dir\") pod \"aws-ebs-csi-driver-node-sqbnh\" (UID: \"5b6d69de-a860-4c82-a03f-23ab7905a5d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sqbnh" Oct 31 04:03:25.541632 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.537858 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5b6d69de-a860-4c82-a03f-23ab7905a5d7-sys-fs\") pod \"aws-ebs-csi-driver-node-sqbnh\" (UID: \"5b6d69de-a860-4c82-a03f-23ab7905a5d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sqbnh" Oct 31 04:03:25.541632 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.537888 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-var-lib-kubelet\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:03:25.541632 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.537917 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be0026cb-89bc-413a-bfe3-523d980397fa-tmp\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:03:25.541632 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.537987 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aaae68b8-1654-474f-9e74-45d1e2025d20-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6zzkx\" (UID: \"aaae68b8-1654-474f-9e74-45d1e2025d20\") " pod="openshift-multus/multus-additional-cni-plugins-6zzkx" Oct 31 04:03:25.541632 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.538017 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-host-var-lib-cni-multus\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:03:25.541632 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.538049 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-etc-sysctl-d\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:03:25.541632 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.538080 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-run-systemd\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.541632 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.538113 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-etc-openvswitch\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.541632 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.538144 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-run-openvswitch\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.541632 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.538170 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-run-ovn\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.541632 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.538201 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97c312b1-4ca5-49c8-8c8d-d247c0fa14ac-metrics-certs\") pod \"network-metrics-daemon-clnlz\" (UID: \"97c312b1-4ca5-49c8-8c8d-d247c0fa14ac\") " pod="openshift-multus/network-metrics-daemon-clnlz" Oct 31 04:03:25.541632 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.538212 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aaae68b8-1654-474f-9e74-45d1e2025d20-system-cni-dir\") pod \"multus-additional-cni-plugins-6zzkx\" (UID: \"aaae68b8-1654-474f-9e74-45d1e2025d20\") " pod="openshift-multus/multus-additional-cni-plugins-6zzkx" Oct 31 04:03:25.541632 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.538232 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w6z5\" (UniqueName: \"kubernetes.io/projected/5b6d69de-a860-4c82-a03f-23ab7905a5d7-kube-api-access-7w6z5\") pod \"aws-ebs-csi-driver-node-sqbnh\" (UID: \"5b6d69de-a860-4c82-a03f-23ab7905a5d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sqbnh" Oct 31 04:03:25.541632 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.538269 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-host-kubelet\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.541632 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.538304 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-systemd-units\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.541632 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.538395 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-host-cni-netd\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.541632 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.538414 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-multus-socket-dir-parent\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:03:25.542160 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.538444 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-host-run-multus-certs\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:03:25.542160 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.538485 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ce86be55-219f-46fb-bff6-6dcf87a40ef4-iptables-alerter-script\") pod \"iptables-alerter-bhgvp\" (UID: \"ce86be55-219f-46fb-bff6-6dcf87a40ef4\") " pod="openshift-network-operator/iptables-alerter-bhgvp" Oct 31 04:03:25.542160 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.538515 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-cnibin\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:03:25.542160 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.538546 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-host-run-netns\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:03:25.542160 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.538577 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-host-var-lib-kubelet\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:03:25.542160 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.538586 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-multus-conf-dir\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:03:25.542160 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.538609 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6d336e6e-feb0-40b2-a625-0e21d25bfac5-multus-daemon-config\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:03:25.542160 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.538641 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-etc-sysctl-conf\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:03:25.542160 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.538671 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn7px\" (UniqueName: \"kubernetes.io/projected/be0026cb-89bc-413a-bfe3-523d980397fa-kube-api-access-dn7px\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:03:25.542160 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.538703 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-host-run-netns\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.542160 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.538736 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5b6d69de-a860-4c82-a03f-23ab7905a5d7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sqbnh\" (UID: \"5b6d69de-a860-4c82-a03f-23ab7905a5d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sqbnh" Oct 31 04:03:25.542160 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.538755 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-etc-modprobe-d\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:03:25.542160 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.538772 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-hostroot\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:03:25.542160 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.538758 2571 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 04:03:25.542160 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.538804 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5nf8\" (UniqueName: \"kubernetes.io/projected/31221235-f863-445b-a5ca-375aae905d38-kube-api-access-x5nf8\") pod \"node-ca-h9sww\" (UID: \"31221235-f863-445b-a5ca-375aae905d38\") " pod="openshift-image-registry/node-ca-h9sww" Oct 31 04:03:25.542160 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.538824 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-node-log\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.542160 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.538834 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/aaae68b8-1654-474f-9e74-45d1e2025d20-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6zzkx\" (UID: \"aaae68b8-1654-474f-9e74-45d1e2025d20\") " pod="openshift-multus/multus-additional-cni-plugins-6zzkx" Oct 31 04:03:25.542160 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.538889 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-log-socket\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.544218 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.538941 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/aaae68b8-1654-474f-9e74-45d1e2025d20-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6zzkx\" (UID: \"aaae68b8-1654-474f-9e74-45d1e2025d20\") " pod="openshift-multus/multus-additional-cni-plugins-6zzkx" Oct 31 04:03:25.544218 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.538985 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5hsv\" (UniqueName: \"kubernetes.io/projected/ce86be55-219f-46fb-bff6-6dcf87a40ef4-kube-api-access-x5hsv\") pod \"iptables-alerter-bhgvp\" (UID: \"ce86be55-219f-46fb-bff6-6dcf87a40ef4\") " pod="openshift-network-operator/iptables-alerter-bhgvp" Oct 31 04:03:25.544218 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.539016 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-etc-systemd\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:03:25.544218 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.539040 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-lib-modules\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:03:25.544218 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.539072 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-host\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:03:25.544218 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.539104 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aaae68b8-1654-474f-9e74-45d1e2025d20-cnibin\") pod \"multus-additional-cni-plugins-6zzkx\" (UID: \"aaae68b8-1654-474f-9e74-45d1e2025d20\") " pod="openshift-multus/multus-additional-cni-plugins-6zzkx" Oct 31 04:03:25.544218 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.539138 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-host-cni-bin\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.544218 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.539197 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-host-cni-bin\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.544218 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.539222 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c1560abd-3afc-4f94-b0bb-ac7ef8b2464a-kubelet-config\") pod \"global-pull-secret-syncer-hnbll\" (UID: \"c1560abd-3afc-4f94-b0bb-ac7ef8b2464a\") " pod="kube-system/global-pull-secret-syncer-hnbll" Oct 31 04:03:25.544218 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.539237 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjq6x\" (UniqueName: \"kubernetes.io/projected/aef43d62-9761-4262-8183-01b9890462a7-kube-api-access-kjq6x\") pod \"network-check-target-4b7d5\" (UID: \"aef43d62-9761-4262-8183-01b9890462a7\") " pod="openshift-network-diagnostics/network-check-target-4b7d5" Oct 31 04:03:25.544218 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.539274 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-system-cni-dir\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:03:25.544218 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.539306 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/aaae68b8-1654-474f-9e74-45d1e2025d20-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6zzkx\" (UID: \"aaae68b8-1654-474f-9e74-45d1e2025d20\") " pod="openshift-multus/multus-additional-cni-plugins-6zzkx" Oct 31 04:03:25.544218 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.539304 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-etc-kubernetes\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:03:25.544218 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.539397 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhkfw\" (UniqueName: \"kubernetes.io/projected/3a05c43f-e2db-44c0-b9bd-4bdabca92a98-kube-api-access-vhkfw\") pod \"node-resolver-nbl9z\" (UID: \"3a05c43f-e2db-44c0-b9bd-4bdabca92a98\") " pod="openshift-dns/node-resolver-nbl9z" Oct 31 04:03:25.544218 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.539433 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/89ea384b-705f-47a9-8df6-d3c266ed6761-agent-certs\") pod \"konnectivity-agent-hm5cr\" (UID: \"89ea384b-705f-47a9-8df6-d3c266ed6761\") " pod="kube-system/konnectivity-agent-hm5cr" Oct 31 04:03:25.544218 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.539762 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7f6cd22a-499a-4c35-b408-b2aa9b78620d-env-overrides\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.544218 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.539892 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-host-slash\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.544890 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.539908 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-host\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:03:25.544890 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.539966 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aaae68b8-1654-474f-9e74-45d1e2025d20-cnibin\") pod \"multus-additional-cni-plugins-6zzkx\" (UID: \"aaae68b8-1654-474f-9e74-45d1e2025d20\") " pod="openshift-multus/multus-additional-cni-plugins-6zzkx" Oct 31 04:03:25.544890 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:25.539986 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 04:03:25.544890 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:25.540054 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97c312b1-4ca5-49c8-8c8d-d247c0fa14ac-metrics-certs podName:97c312b1-4ca5-49c8-8c8d-d247c0fa14ac nodeName:}" failed. No retries permitted until 2026-10-31 04:03:26.040038149 +0000 UTC m=+3.139876013 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/97c312b1-4ca5-49c8-8c8d-d247c0fa14ac-metrics-certs") pod "network-metrics-daemon-clnlz" (UID: "97c312b1-4ca5-49c8-8c8d-d247c0fa14ac") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 04:03:25.544890 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.540124 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c1560abd-3afc-4f94-b0bb-ac7ef8b2464a-dbus\") pod \"global-pull-secret-syncer-hnbll\" (UID: \"c1560abd-3afc-4f94-b0bb-ac7ef8b2464a\") " pod="kube-system/global-pull-secret-syncer-hnbll" Oct 31 04:03:25.544890 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.540186 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ce86be55-219f-46fb-bff6-6dcf87a40ef4-host-slash\") pod \"iptables-alerter-bhgvp\" (UID: \"ce86be55-219f-46fb-bff6-6dcf87a40ef4\") " pod="openshift-network-operator/iptables-alerter-bhgvp" Oct 31 04:03:25.544890 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.540209 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-system-cni-dir\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:03:25.544890 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.540252 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-etc-kubernetes\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:03:25.544890 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.540379 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5b6d69de-a860-4c82-a03f-23ab7905a5d7-registration-dir\") pod \"aws-ebs-csi-driver-node-sqbnh\" (UID: \"5b6d69de-a860-4c82-a03f-23ab7905a5d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sqbnh" Oct 31 04:03:25.544890 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.540796 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/31221235-f863-445b-a5ca-375aae905d38-serviceca\") pod \"node-ca-h9sww\" (UID: \"31221235-f863-445b-a5ca-375aae905d38\") " pod="openshift-image-registry/node-ca-h9sww" Oct 31 04:03:25.544890 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.538170 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aaae68b8-1654-474f-9e74-45d1e2025d20-cni-binary-copy\") pod \"multus-additional-cni-plugins-6zzkx\" (UID: \"aaae68b8-1654-474f-9e74-45d1e2025d20\") " pod="openshift-multus/multus-additional-cni-plugins-6zzkx" Oct 31 04:03:25.544890 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.540311 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-etc-systemd\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:03:25.544890 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.541065 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/aaae68b8-1654-474f-9e74-45d1e2025d20-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6zzkx\" (UID: \"aaae68b8-1654-474f-9e74-45d1e2025d20\") " pod="openshift-multus/multus-additional-cni-plugins-6zzkx" Oct 31 04:03:25.544890 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.540836 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-host-run-multus-certs\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:03:25.544890 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.541128 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/89ea384b-705f-47a9-8df6-d3c266ed6761-konnectivity-ca\") pod \"konnectivity-agent-hm5cr\" (UID: \"89ea384b-705f-47a9-8df6-d3c266ed6761\") " pod="kube-system/konnectivity-agent-hm5cr" Oct 31 04:03:25.544890 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.541251 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/31221235-f863-445b-a5ca-375aae905d38-host\") pod \"node-ca-h9sww\" (UID: \"31221235-f863-445b-a5ca-375aae905d38\") " pod="openshift-image-registry/node-ca-h9sww" Oct 31 04:03:25.544890 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.541308 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5b6d69de-a860-4c82-a03f-23ab7905a5d7-socket-dir\") pod \"aws-ebs-csi-driver-node-sqbnh\" (UID: \"5b6d69de-a860-4c82-a03f-23ab7905a5d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sqbnh" Oct 31 04:03:25.545447 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.541310 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-host-run-k8s-cni-cncf-io\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:03:25.545447 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.541380 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3a05c43f-e2db-44c0-b9bd-4bdabca92a98-hosts-file\") pod \"node-resolver-nbl9z\" (UID: \"3a05c43f-e2db-44c0-b9bd-4bdabca92a98\") " pod="openshift-dns/node-resolver-nbl9z" Oct 31 04:03:25.545447 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.541465 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5b6d69de-a860-4c82-a03f-23ab7905a5d7-etc-selinux\") pod \"aws-ebs-csi-driver-node-sqbnh\" (UID: \"5b6d69de-a860-4c82-a03f-23ab7905a5d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sqbnh" Oct 31 04:03:25.545447 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.541472 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-lib-modules\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:03:25.545447 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.541928 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6d336e6e-feb0-40b2-a625-0e21d25bfac5-cni-binary-copy\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:03:25.545447 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.542049 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ce86be55-219f-46fb-bff6-6dcf87a40ef4-iptables-alerter-script\") pod \"iptables-alerter-bhgvp\" (UID: \"ce86be55-219f-46fb-bff6-6dcf87a40ef4\") " pod="openshift-network-operator/iptables-alerter-bhgvp" Oct 31 04:03:25.545447 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.542073 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-cnibin\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:03:25.545447 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.542087 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7f6cd22a-499a-4c35-b408-b2aa9b78620d-ovnkube-script-lib\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.545447 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.542117 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.545447 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.542152 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-host-var-lib-kubelet\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:03:25.545447 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.542230 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-host-run-netns\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.545447 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.542245 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-etc-sysctl-conf\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:03:25.545447 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.542284 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5b6d69de-a860-4c82-a03f-23ab7905a5d7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sqbnh\" (UID: \"5b6d69de-a860-4c82-a03f-23ab7905a5d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sqbnh" Oct 31 04:03:25.545447 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.542352 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-hostroot\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:03:25.545447 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.542409 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-sys\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:03:25.545447 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.542558 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5b6d69de-a860-4c82-a03f-23ab7905a5d7-sys-fs\") pod \"aws-ebs-csi-driver-node-sqbnh\" (UID: \"5b6d69de-a860-4c82-a03f-23ab7905a5d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sqbnh" Oct 31 04:03:25.545447 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.542605 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-var-lib-kubelet\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:03:25.545447 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.542769 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6d336e6e-feb0-40b2-a625-0e21d25bfac5-multus-daemon-config\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:03:25.546168 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.542921 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aaae68b8-1654-474f-9e74-45d1e2025d20-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6zzkx\" (UID: \"aaae68b8-1654-474f-9e74-45d1e2025d20\") " pod="openshift-multus/multus-additional-cni-plugins-6zzkx" Oct 31 04:03:25.546168 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.542943 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-host-var-lib-cni-multus\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:03:25.546168 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.543031 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-etc-sysctl-d\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:03:25.546168 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.543067 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-run-systemd\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.546168 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.543203 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-run-openvswitch\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.546168 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.543254 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-run-ovn\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.546168 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.543301 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-host-var-lib-cni-bin\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:03:25.546168 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.543397 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-etc-openvswitch\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.546168 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.543645 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-host-run-ovn-kubernetes\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.546168 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.543882 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-var-lib-openvswitch\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.546168 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.544139 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-host-run-netns\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:03:25.546168 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.544954 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7f6cd22a-499a-4c35-b408-b2aa9b78620d-ovn-node-metrics-cert\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.546168 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.545362 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/89ea384b-705f-47a9-8df6-d3c266ed6761-agent-certs\") pod \"konnectivity-agent-hm5cr\" (UID: \"89ea384b-705f-47a9-8df6-d3c266ed6761\") " pod="kube-system/konnectivity-agent-hm5cr" Oct 31 04:03:25.546168 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.546026 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7f6cd22a-499a-4c35-b408-b2aa9b78620d-ovnkube-config\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.547619 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.547596 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a05c43f-e2db-44c0-b9bd-4bdabca92a98-tmp-dir\") pod \"node-resolver-nbl9z\" (UID: \"3a05c43f-e2db-44c0-b9bd-4bdabca92a98\") " pod="openshift-dns/node-resolver-nbl9z" Oct 31 04:03:25.549956 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.549921 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be0026cb-89bc-413a-bfe3-523d980397fa-tmp\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:03:25.550112 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.550091 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/be0026cb-89bc-413a-bfe3-523d980397fa-etc-tuned\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:03:25.564035 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.563999 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phq2f\" (UniqueName: \"kubernetes.io/projected/7f6cd22a-499a-4c35-b408-b2aa9b78620d-kube-api-access-phq2f\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.564824 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.564784 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5hsv\" (UniqueName: \"kubernetes.io/projected/ce86be55-219f-46fb-bff6-6dcf87a40ef4-kube-api-access-x5hsv\") pod \"iptables-alerter-bhgvp\" (UID: \"ce86be55-219f-46fb-bff6-6dcf87a40ef4\") " pod="openshift-network-operator/iptables-alerter-bhgvp" Oct 31 04:03:25.565974 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.565917 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhkfw\" (UniqueName: \"kubernetes.io/projected/3a05c43f-e2db-44c0-b9bd-4bdabca92a98-kube-api-access-vhkfw\") pod \"node-resolver-nbl9z\" (UID: \"3a05c43f-e2db-44c0-b9bd-4bdabca92a98\") " pod="openshift-dns/node-resolver-nbl9z" Oct 31 04:03:25.566078 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.566052 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd5qc\" (UniqueName: \"kubernetes.io/projected/aaae68b8-1654-474f-9e74-45d1e2025d20-kube-api-access-nd5qc\") pod \"multus-additional-cni-plugins-6zzkx\" (UID: \"aaae68b8-1654-474f-9e74-45d1e2025d20\") " pod="openshift-multus/multus-additional-cni-plugins-6zzkx" Oct 31 04:03:25.566292 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.566270 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w6z5\" (UniqueName: \"kubernetes.io/projected/5b6d69de-a860-4c82-a03f-23ab7905a5d7-kube-api-access-7w6z5\") pod \"aws-ebs-csi-driver-node-sqbnh\" (UID: \"5b6d69de-a860-4c82-a03f-23ab7905a5d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sqbnh" Oct 31 04:03:25.566560 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:25.566542 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 04:03:25.566641 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:25.566564 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 04:03:25.566641 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:25.566578 2571 projected.go:196] Error preparing data for projected volume kube-api-access-kjq6x for pod openshift-network-diagnostics/network-check-target-4b7d5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 04:03:25.566740 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:25.566649 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aef43d62-9761-4262-8183-01b9890462a7-kube-api-access-kjq6x podName:aef43d62-9761-4262-8183-01b9890462a7 nodeName:}" failed. No retries permitted until 2026-10-31 04:03:26.066629712 +0000 UTC m=+3.166467591 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-kjq6x" (UniqueName: "kubernetes.io/projected/aef43d62-9761-4262-8183-01b9890462a7-kube-api-access-kjq6x") pod "network-check-target-4b7d5" (UID: "aef43d62-9761-4262-8183-01b9890462a7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 04:03:25.568181 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.568159 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f5pv\" (UniqueName: \"kubernetes.io/projected/97c312b1-4ca5-49c8-8c8d-d247c0fa14ac-kube-api-access-2f5pv\") pod \"network-metrics-daemon-clnlz\" (UID: \"97c312b1-4ca5-49c8-8c8d-d247c0fa14ac\") " pod="openshift-multus/network-metrics-daemon-clnlz" Oct 31 04:03:25.568269 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.568211 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5nf8\" (UniqueName: \"kubernetes.io/projected/31221235-f863-445b-a5ca-375aae905d38-kube-api-access-x5nf8\") pod \"node-ca-h9sww\" (UID: \"31221235-f863-445b-a5ca-375aae905d38\") " pod="openshift-image-registry/node-ca-h9sww" Oct 31 04:03:25.568360 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.568324 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn7px\" (UniqueName: \"kubernetes.io/projected/be0026cb-89bc-413a-bfe3-523d980397fa-kube-api-access-dn7px\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:03:25.569998 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.569966 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mpvl\" (UniqueName: \"kubernetes.io/projected/6d336e6e-feb0-40b2-a625-0e21d25bfac5-kube-api-access-9mpvl\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:03:25.642250 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.642173 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hm5cr" Oct 31 04:03:25.652936 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.652907 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:03:25.653636 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:25.653582 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89ea384b_705f_47a9_8df6_d3c266ed6761.slice/crio-2c9ecfdfa891ea5b3d95c594421524a9d34def2e9acce4be2264b5af5d8c2d3b WatchSource:0}: Error finding container 2c9ecfdfa891ea5b3d95c594421524a9d34def2e9acce4be2264b5af5d8c2d3b: Status 404 returned error can't find the container with id 2c9ecfdfa891ea5b3d95c594421524a9d34def2e9acce4be2264b5af5d8c2d3b Oct 31 04:03:25.663722 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:25.663698 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe0026cb_89bc_413a_bfe3_523d980397fa.slice/crio-09db7817717a0cb4ceab0532753de0c4e26cad1c65b7346947c5c1ac4c0dac92 WatchSource:0}: Error finding container 09db7817717a0cb4ceab0532753de0c4e26cad1c65b7346947c5c1ac4c0dac92: Status 404 returned error can't find the container with id 09db7817717a0cb4ceab0532753de0c4e26cad1c65b7346947c5c1ac4c0dac92 Oct 31 04:03:25.664520 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.664498 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-h9sww" Oct 31 04:03:25.671395 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.671374 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bhgvp" Oct 31 04:03:25.682967 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:25.682945 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31221235_f863_445b_a5ca_375aae905d38.slice/crio-7b7497670421e46a8e33c6408131a15a55c8bd6d7eee75bb68d86379ca727a32 WatchSource:0}: Error finding container 7b7497670421e46a8e33c6408131a15a55c8bd6d7eee75bb68d86379ca727a32: Status 404 returned error can't find the container with id 7b7497670421e46a8e33c6408131a15a55c8bd6d7eee75bb68d86379ca727a32 Oct 31 04:03:25.683440 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:25.683406 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce86be55_219f_46fb_bff6_6dcf87a40ef4.slice/crio-a6569749975018a518d8014ad4482d3304d6a757a26850177414e8d03cd3f031 WatchSource:0}: Error finding container a6569749975018a518d8014ad4482d3304d6a757a26850177414e8d03cd3f031: Status 404 returned error can't find the container with id a6569749975018a518d8014ad4482d3304d6a757a26850177414e8d03cd3f031 Oct 31 04:03:25.696206 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.696177 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6zzkx" Oct 31 04:03:25.704468 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.704443 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sqbnh" Oct 31 04:03:25.706430 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:25.706403 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaae68b8_1654_474f_9e74_45d1e2025d20.slice/crio-8b5a131b67071cd06f8262e388f9074e8564dcdcb559a77ab17851be4682e542 WatchSource:0}: Error finding container 8b5a131b67071cd06f8262e388f9074e8564dcdcb559a77ab17851be4682e542: Status 404 returned error can't find the container with id 8b5a131b67071cd06f8262e388f9074e8564dcdcb559a77ab17851be4682e542 Oct 31 04:03:25.707102 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.707078 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6d7cp" Oct 31 04:03:25.714130 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.714107 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nbl9z" Oct 31 04:03:25.715549 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:25.715523 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b6d69de_a860_4c82_a03f_23ab7905a5d7.slice/crio-ac6d070cb1c7f393c8567c4abe16bfbee71c7ddd0023ec00c4e9ed438d34fefb WatchSource:0}: Error finding container ac6d070cb1c7f393c8567c4abe16bfbee71c7ddd0023ec00c4e9ed438d34fefb: Status 404 returned error can't find the container with id ac6d070cb1c7f393c8567c4abe16bfbee71c7ddd0023ec00c4e9ed438d34fefb Oct 31 04:03:25.719714 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:25.719318 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d336e6e_feb0_40b2_a625_0e21d25bfac5.slice/crio-674c8c743183843c2e2fee5053ae52e4d2519342fefad52eadb490e1c7f6aba9 WatchSource:0}: Error finding container 674c8c743183843c2e2fee5053ae52e4d2519342fefad52eadb490e1c7f6aba9: Status 404 returned error can't find the container with id 674c8c743183843c2e2fee5053ae52e4d2519342fefad52eadb490e1c7f6aba9 Oct 31 04:03:25.723161 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:25.722920 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:25.725603 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:25.725577 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a05c43f_e2db_44c0_b9bd_4bdabca92a98.slice/crio-a865c4d77ec5eae81f238b3cbe27af91d7b1541db6173ecd62ad362bd8c7908f WatchSource:0}: Error finding container a865c4d77ec5eae81f238b3cbe27af91d7b1541db6173ecd62ad362bd8c7908f: Status 404 returned error can't find the container with id a865c4d77ec5eae81f238b3cbe27af91d7b1541db6173ecd62ad362bd8c7908f Oct 31 04:03:25.733832 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:25.733809 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f6cd22a_499a_4c35_b408_b2aa9b78620d.slice/crio-cdd3ae8de99afb7e468a104958df90918f2055ac5de8be3f64191f0dca24df0b WatchSource:0}: Error finding container cdd3ae8de99afb7e468a104958df90918f2055ac5de8be3f64191f0dca24df0b: Status 404 returned error can't find the container with id cdd3ae8de99afb7e468a104958df90918f2055ac5de8be3f64191f0dca24df0b Oct 31 04:03:26.044241 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:26.044202 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97c312b1-4ca5-49c8-8c8d-d247c0fa14ac-metrics-certs\") pod \"network-metrics-daemon-clnlz\" (UID: \"97c312b1-4ca5-49c8-8c8d-d247c0fa14ac\") " pod="openshift-multus/network-metrics-daemon-clnlz" Oct 31 04:03:26.044422 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:26.044285 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c1560abd-3afc-4f94-b0bb-ac7ef8b2464a-original-pull-secret\") pod \"global-pull-secret-syncer-hnbll\" (UID: \"c1560abd-3afc-4f94-b0bb-ac7ef8b2464a\") " pod="kube-system/global-pull-secret-syncer-hnbll" Oct 31 04:03:26.044422 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:26.044354 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 04:03:26.044422 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:26.044414 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97c312b1-4ca5-49c8-8c8d-d247c0fa14ac-metrics-certs podName:97c312b1-4ca5-49c8-8c8d-d247c0fa14ac nodeName:}" failed. No retries permitted until 2026-10-31 04:03:27.044397952 +0000 UTC m=+4.144235835 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/97c312b1-4ca5-49c8-8c8d-d247c0fa14ac-metrics-certs") pod "network-metrics-daemon-clnlz" (UID: "97c312b1-4ca5-49c8-8c8d-d247c0fa14ac") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 04:03:26.044587 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:26.044438 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 04:03:26.044587 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:26.044499 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1560abd-3afc-4f94-b0bb-ac7ef8b2464a-original-pull-secret podName:c1560abd-3afc-4f94-b0bb-ac7ef8b2464a nodeName:}" failed. No retries permitted until 2026-10-31 04:03:27.044481504 +0000 UTC m=+4.144319382 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c1560abd-3afc-4f94-b0bb-ac7ef8b2464a-original-pull-secret") pod "global-pull-secret-syncer-hnbll" (UID: "c1560abd-3afc-4f94-b0bb-ac7ef8b2464a") : object "kube-system"/"original-pull-secret" not registered Oct 31 04:03:26.145595 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:26.145479 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjq6x\" (UniqueName: \"kubernetes.io/projected/aef43d62-9761-4262-8183-01b9890462a7-kube-api-access-kjq6x\") pod \"network-check-target-4b7d5\" (UID: \"aef43d62-9761-4262-8183-01b9890462a7\") " pod="openshift-network-diagnostics/network-check-target-4b7d5" Oct 31 04:03:26.145772 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:26.145646 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 04:03:26.145772 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:26.145677 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 04:03:26.145772 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:26.145691 2571 projected.go:196] Error preparing data for projected volume kube-api-access-kjq6x for pod openshift-network-diagnostics/network-check-target-4b7d5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 04:03:26.145772 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:26.145751 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aef43d62-9761-4262-8183-01b9890462a7-kube-api-access-kjq6x podName:aef43d62-9761-4262-8183-01b9890462a7 nodeName:}" failed. No retries permitted until 2026-10-31 04:03:27.145732737 +0000 UTC m=+4.245570606 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-kjq6x" (UniqueName: "kubernetes.io/projected/aef43d62-9761-4262-8183-01b9890462a7-kube-api-access-kjq6x") pod "network-check-target-4b7d5" (UID: "aef43d62-9761-4262-8183-01b9890462a7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 04:03:26.383325 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:26.383257 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 03:58:25 +0000 UTC" deadline="2028-07-24 18:18:51.030054504 +0000 UTC" Oct 31 04:03:26.383325 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:26.383323 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15182h15m24.646736178s" Oct 31 04:03:26.399934 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:26.399905 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hnbll" Oct 31 04:03:26.400074 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:26.400038 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hnbll" podUID="c1560abd-3afc-4f94-b0bb-ac7ef8b2464a" Oct 31 04:03:26.420202 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:26.420128 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bhgvp" event={"ID":"ce86be55-219f-46fb-bff6-6dcf87a40ef4","Type":"ContainerStarted","Data":"a6569749975018a518d8014ad4482d3304d6a757a26850177414e8d03cd3f031"} Oct 31 04:03:26.426102 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:26.426041 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6d7cp" event={"ID":"6d336e6e-feb0-40b2-a625-0e21d25bfac5","Type":"ContainerStarted","Data":"674c8c743183843c2e2fee5053ae52e4d2519342fefad52eadb490e1c7f6aba9"} Oct 31 04:03:26.427635 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:26.427565 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sqbnh" event={"ID":"5b6d69de-a860-4c82-a03f-23ab7905a5d7","Type":"ContainerStarted","Data":"ac6d070cb1c7f393c8567c4abe16bfbee71c7ddd0023ec00c4e9ed438d34fefb"} Oct 31 04:03:26.429460 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:26.429404 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-h9sww" event={"ID":"31221235-f863-445b-a5ca-375aae905d38","Type":"ContainerStarted","Data":"7b7497670421e46a8e33c6408131a15a55c8bd6d7eee75bb68d86379ca727a32"} Oct 31 04:03:26.431981 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:26.431922 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-26vsf" event={"ID":"be0026cb-89bc-413a-bfe3-523d980397fa","Type":"ContainerStarted","Data":"09db7817717a0cb4ceab0532753de0c4e26cad1c65b7346947c5c1ac4c0dac92"} Oct 31 04:03:26.434115 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:26.434038 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hm5cr" event={"ID":"89ea384b-705f-47a9-8df6-d3c266ed6761","Type":"ContainerStarted","Data":"2c9ecfdfa891ea5b3d95c594421524a9d34def2e9acce4be2264b5af5d8c2d3b"} Oct 31 04:03:26.438973 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:26.438394 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-57.ec2.internal" event={"ID":"6536fdc09c4b535fc805fe98673b8597","Type":"ContainerStarted","Data":"713d3ff48543c2aa7c9049d7b0ff4e663af284fb92d00922e6e76919f5a8727d"} Oct 31 04:03:26.444356 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:26.444228 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" event={"ID":"7f6cd22a-499a-4c35-b408-b2aa9b78620d","Type":"ContainerStarted","Data":"cdd3ae8de99afb7e468a104958df90918f2055ac5de8be3f64191f0dca24df0b"} Oct 31 04:03:26.448904 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:26.448882 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nbl9z" event={"ID":"3a05c43f-e2db-44c0-b9bd-4bdabca92a98","Type":"ContainerStarted","Data":"a865c4d77ec5eae81f238b3cbe27af91d7b1541db6173ecd62ad362bd8c7908f"} Oct 31 04:03:26.451247 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:26.451210 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zzkx" event={"ID":"aaae68b8-1654-474f-9e74-45d1e2025d20","Type":"ContainerStarted","Data":"8b5a131b67071cd06f8262e388f9074e8564dcdcb559a77ab17851be4682e542"} Oct 31 04:03:27.052515 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:27.051998 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97c312b1-4ca5-49c8-8c8d-d247c0fa14ac-metrics-certs\") pod \"network-metrics-daemon-clnlz\" (UID: \"97c312b1-4ca5-49c8-8c8d-d247c0fa14ac\") " pod="openshift-multus/network-metrics-daemon-clnlz" Oct 31 04:03:27.052515 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:27.052233 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 04:03:27.052515 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:27.052378 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97c312b1-4ca5-49c8-8c8d-d247c0fa14ac-metrics-certs podName:97c312b1-4ca5-49c8-8c8d-d247c0fa14ac nodeName:}" failed. No retries permitted until 2026-10-31 04:03:29.052359802 +0000 UTC m=+6.152197678 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/97c312b1-4ca5-49c8-8c8d-d247c0fa14ac-metrics-certs") pod "network-metrics-daemon-clnlz" (UID: "97c312b1-4ca5-49c8-8c8d-d247c0fa14ac") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 04:03:27.052515 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:27.052380 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 04:03:27.052515 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:27.052419 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1560abd-3afc-4f94-b0bb-ac7ef8b2464a-original-pull-secret podName:c1560abd-3afc-4f94-b0bb-ac7ef8b2464a nodeName:}" failed. No retries permitted until 2026-10-31 04:03:29.052407917 +0000 UTC m=+6.152245781 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c1560abd-3afc-4f94-b0bb-ac7ef8b2464a-original-pull-secret") pod "global-pull-secret-syncer-hnbll" (UID: "c1560abd-3afc-4f94-b0bb-ac7ef8b2464a") : object "kube-system"/"original-pull-secret" not registered Oct 31 04:03:27.052515 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:27.052303 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c1560abd-3afc-4f94-b0bb-ac7ef8b2464a-original-pull-secret\") pod \"global-pull-secret-syncer-hnbll\" (UID: \"c1560abd-3afc-4f94-b0bb-ac7ef8b2464a\") " pod="kube-system/global-pull-secret-syncer-hnbll" Oct 31 04:03:27.154252 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:27.153283 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjq6x\" (UniqueName: \"kubernetes.io/projected/aef43d62-9761-4262-8183-01b9890462a7-kube-api-access-kjq6x\") pod \"network-check-target-4b7d5\" (UID: \"aef43d62-9761-4262-8183-01b9890462a7\") " pod="openshift-network-diagnostics/network-check-target-4b7d5" Oct 31 04:03:27.154252 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:27.153513 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 04:03:27.154252 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:27.153534 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 04:03:27.154252 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:27.153548 2571 projected.go:196] Error preparing data for projected volume kube-api-access-kjq6x for pod openshift-network-diagnostics/network-check-target-4b7d5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 04:03:27.154252 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:27.153605 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aef43d62-9761-4262-8183-01b9890462a7-kube-api-access-kjq6x podName:aef43d62-9761-4262-8183-01b9890462a7 nodeName:}" failed. No retries permitted until 2026-10-31 04:03:29.153585426 +0000 UTC m=+6.253423291 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-kjq6x" (UniqueName: "kubernetes.io/projected/aef43d62-9761-4262-8183-01b9890462a7-kube-api-access-kjq6x") pod "network-check-target-4b7d5" (UID: "aef43d62-9761-4262-8183-01b9890462a7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 04:03:27.384519 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:27.384397 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 03:58:25 +0000 UTC" deadline="2028-08-04 22:09:53.055460612 +0000 UTC" Oct 31 04:03:27.384519 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:27.384441 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15450h6m25.671024313s" Oct 31 04:03:27.402959 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:27.402264 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4b7d5" Oct 31 04:03:27.402959 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:27.402406 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4b7d5" podUID="aef43d62-9761-4262-8183-01b9890462a7" Oct 31 04:03:27.402959 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:27.402789 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clnlz" Oct 31 04:03:27.402959 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:27.402899 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-clnlz" podUID="97c312b1-4ca5-49c8-8c8d-d247c0fa14ac" Oct 31 04:03:27.468966 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:27.468884 2571 generic.go:356] "Generic (PLEG): container finished" podID="e8283c5a79822699286a7ae3ef8db4f6" containerID="7262c355488431c1bf772ef311d19308883aa44dd7e85053603a15d191d212c2" exitCode=0 Oct 31 04:03:27.469943 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:27.469918 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal" event={"ID":"e8283c5a79822699286a7ae3ef8db4f6","Type":"ContainerDied","Data":"7262c355488431c1bf772ef311d19308883aa44dd7e85053603a15d191d212c2"} Oct 31 04:03:27.517567 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:27.517510 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-57.ec2.internal" podStartSLOduration=3.517487026 podStartE2EDuration="3.517487026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 04:03:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:03:26.481071657 +0000 UTC m=+3.580909543" watchObservedRunningTime="2026-10-31 04:03:27.517487026 +0000 UTC m=+4.617324912" Oct 31 04:03:28.400233 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:28.399599 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hnbll" Oct 31 04:03:28.400233 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:28.399811 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hnbll" podUID="c1560abd-3afc-4f94-b0bb-ac7ef8b2464a" Oct 31 04:03:28.486990 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:28.486584 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal" event={"ID":"e8283c5a79822699286a7ae3ef8db4f6","Type":"ContainerStarted","Data":"e67453bb6c81cad759194fcee9d387ecd699a14eaa6da143f1d52e1ae4c948c5"} Oct 31 04:03:28.492377 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:28.492309 2571 generic.go:356] "Generic (PLEG): container finished" podID="aaae68b8-1654-474f-9e74-45d1e2025d20" containerID="b1ccb1c6bcd958ea3f55f4e3d437d0a2662df72c1a3471a19b13e8bca8a54d09" exitCode=0 Oct 31 04:03:28.492545 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:28.492388 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zzkx" event={"ID":"aaae68b8-1654-474f-9e74-45d1e2025d20","Type":"ContainerDied","Data":"b1ccb1c6bcd958ea3f55f4e3d437d0a2662df72c1a3471a19b13e8bca8a54d09"} Oct 31 04:03:28.590643 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:28.590530 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal" podStartSLOduration=4.5905096610000005 podStartE2EDuration="4.590509661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 04:03:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:03:28.538559489 +0000 UTC m=+5.638397373" watchObservedRunningTime="2026-10-31 04:03:28.590509661 +0000 UTC m=+5.690347546" Oct 31 04:03:29.069272 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:29.069239 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97c312b1-4ca5-49c8-8c8d-d247c0fa14ac-metrics-certs\") pod \"network-metrics-daemon-clnlz\" (UID: \"97c312b1-4ca5-49c8-8c8d-d247c0fa14ac\") " pod="openshift-multus/network-metrics-daemon-clnlz" Oct 31 04:03:29.069421 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:29.069312 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c1560abd-3afc-4f94-b0bb-ac7ef8b2464a-original-pull-secret\") pod \"global-pull-secret-syncer-hnbll\" (UID: \"c1560abd-3afc-4f94-b0bb-ac7ef8b2464a\") " pod="kube-system/global-pull-secret-syncer-hnbll" Oct 31 04:03:29.069534 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:29.069476 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 04:03:29.069575 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:29.069547 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1560abd-3afc-4f94-b0bb-ac7ef8b2464a-original-pull-secret podName:c1560abd-3afc-4f94-b0bb-ac7ef8b2464a nodeName:}" failed. No retries permitted until 2026-10-31 04:03:33.069529074 +0000 UTC m=+10.169366950 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c1560abd-3afc-4f94-b0bb-ac7ef8b2464a-original-pull-secret") pod "global-pull-secret-syncer-hnbll" (UID: "c1560abd-3afc-4f94-b0bb-ac7ef8b2464a") : object "kube-system"/"original-pull-secret" not registered Oct 31 04:03:29.069948 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:29.069932 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 04:03:29.070004 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:29.069987 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97c312b1-4ca5-49c8-8c8d-d247c0fa14ac-metrics-certs podName:97c312b1-4ca5-49c8-8c8d-d247c0fa14ac nodeName:}" failed. No retries permitted until 2026-10-31 04:03:33.069971956 +0000 UTC m=+10.169809823 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/97c312b1-4ca5-49c8-8c8d-d247c0fa14ac-metrics-certs") pod "network-metrics-daemon-clnlz" (UID: "97c312b1-4ca5-49c8-8c8d-d247c0fa14ac") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 04:03:29.170887 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:29.170148 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjq6x\" (UniqueName: \"kubernetes.io/projected/aef43d62-9761-4262-8183-01b9890462a7-kube-api-access-kjq6x\") pod \"network-check-target-4b7d5\" (UID: \"aef43d62-9761-4262-8183-01b9890462a7\") " pod="openshift-network-diagnostics/network-check-target-4b7d5" Oct 31 04:03:29.170887 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:29.170391 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 04:03:29.170887 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:29.170412 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 04:03:29.170887 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:29.170425 2571 projected.go:196] Error preparing data for projected volume kube-api-access-kjq6x for pod openshift-network-diagnostics/network-check-target-4b7d5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 04:03:29.170887 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:29.170487 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aef43d62-9761-4262-8183-01b9890462a7-kube-api-access-kjq6x podName:aef43d62-9761-4262-8183-01b9890462a7 nodeName:}" failed. No retries permitted until 2026-10-31 04:03:33.170468663 +0000 UTC m=+10.270306535 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-kjq6x" (UniqueName: "kubernetes.io/projected/aef43d62-9761-4262-8183-01b9890462a7-kube-api-access-kjq6x") pod "network-check-target-4b7d5" (UID: "aef43d62-9761-4262-8183-01b9890462a7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 04:03:29.402470 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:29.402437 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4b7d5" Oct 31 04:03:29.402937 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:29.402566 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4b7d5" podUID="aef43d62-9761-4262-8183-01b9890462a7" Oct 31 04:03:29.402937 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:29.402653 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clnlz" Oct 31 04:03:29.402937 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:29.402748 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-clnlz" podUID="97c312b1-4ca5-49c8-8c8d-d247c0fa14ac" Oct 31 04:03:29.509185 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:29.509085 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hm5cr" event={"ID":"89ea384b-705f-47a9-8df6-d3c266ed6761","Type":"ContainerStarted","Data":"2a8a38f99029a2221b9cbd67f28ef39a2ff379ecb6a0cd5596acb0936da1b4e3"} Oct 31 04:03:30.400640 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:30.400091 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hnbll" Oct 31 04:03:30.400640 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:30.400231 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hnbll" podUID="c1560abd-3afc-4f94-b0bb-ac7ef8b2464a" Oct 31 04:03:31.401826 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:31.401790 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4b7d5" Oct 31 04:03:31.402347 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:31.401939 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4b7d5" podUID="aef43d62-9761-4262-8183-01b9890462a7" Oct 31 04:03:31.402403 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:31.402348 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clnlz" Oct 31 04:03:31.402486 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:31.402464 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-clnlz" podUID="97c312b1-4ca5-49c8-8c8d-d247c0fa14ac" Oct 31 04:03:31.517446 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:31.517401 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sqbnh" event={"ID":"5b6d69de-a860-4c82-a03f-23ab7905a5d7","Type":"ContainerStarted","Data":"58f693490a48d0b2a39e590e26678be4b41cd50ed5feb056e8554a0202f35641"} Oct 31 04:03:31.520291 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:31.520257 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-h9sww" event={"ID":"31221235-f863-445b-a5ca-375aae905d38","Type":"ContainerStarted","Data":"c0f548e5a002009e5005620027ba7af82c236741ea3143ab563b0fabf1158af3"} Oct 31 04:03:31.562573 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:31.562459 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-hm5cr" podStartSLOduration=5.310761726 podStartE2EDuration="8.562438878s" totalImagesPullingTime="3.251677152s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:03:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:03:29.551025694 +0000 UTC m=+6.650863591" watchObservedRunningTime="2026-10-31 04:03:31.562438878 +0000 UTC m=+8.662276764" Oct 31 04:03:32.100379 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:32.100345 2571 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-hm5cr" Oct 31 04:03:32.101658 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:32.101638 2571 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-hm5cr" Oct 31 04:03:32.143652 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:32.143546 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-h9sww" podStartSLOduration=4.286510142 podStartE2EDuration="9.143526035s" totalImagesPullingTime="4.857015893s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 04:03:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:03:31.562997005 +0000 UTC m=+8.662834890" watchObservedRunningTime="2026-10-31 04:03:32.143526035 +0000 UTC m=+9.243363923" Oct 31 04:03:32.400006 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:32.399935 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hnbll" Oct 31 04:03:32.400162 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:32.400057 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hnbll" podUID="c1560abd-3afc-4f94-b0bb-ac7ef8b2464a" Oct 31 04:03:32.854709 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:32.854624 2571 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 04:03:33.105015 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:33.104985 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97c312b1-4ca5-49c8-8c8d-d247c0fa14ac-metrics-certs\") pod \"network-metrics-daemon-clnlz\" (UID: \"97c312b1-4ca5-49c8-8c8d-d247c0fa14ac\") " pod="openshift-multus/network-metrics-daemon-clnlz" Oct 31 04:03:33.105121 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:33.105061 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c1560abd-3afc-4f94-b0bb-ac7ef8b2464a-original-pull-secret\") pod \"global-pull-secret-syncer-hnbll\" (UID: \"c1560abd-3afc-4f94-b0bb-ac7ef8b2464a\") " pod="kube-system/global-pull-secret-syncer-hnbll" Oct 31 04:03:33.105220 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:33.105196 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 04:03:33.105271 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:33.105255 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1560abd-3afc-4f94-b0bb-ac7ef8b2464a-original-pull-secret podName:c1560abd-3afc-4f94-b0bb-ac7ef8b2464a nodeName:}" failed. No retries permitted until 2026-10-31 04:03:41.105238057 +0000 UTC m=+18.205075935 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c1560abd-3afc-4f94-b0bb-ac7ef8b2464a-original-pull-secret") pod "global-pull-secret-syncer-hnbll" (UID: "c1560abd-3afc-4f94-b0bb-ac7ef8b2464a") : object "kube-system"/"original-pull-secret" not registered Oct 31 04:03:33.105350 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:33.105323 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 04:03:33.105408 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:33.105379 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97c312b1-4ca5-49c8-8c8d-d247c0fa14ac-metrics-certs podName:97c312b1-4ca5-49c8-8c8d-d247c0fa14ac nodeName:}" failed. No retries permitted until 2026-10-31 04:03:41.105364942 +0000 UTC m=+18.205202823 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/97c312b1-4ca5-49c8-8c8d-d247c0fa14ac-metrics-certs") pod "network-metrics-daemon-clnlz" (UID: "97c312b1-4ca5-49c8-8c8d-d247c0fa14ac") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 04:03:33.206265 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:33.206234 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjq6x\" (UniqueName: \"kubernetes.io/projected/aef43d62-9761-4262-8183-01b9890462a7-kube-api-access-kjq6x\") pod \"network-check-target-4b7d5\" (UID: \"aef43d62-9761-4262-8183-01b9890462a7\") " pod="openshift-network-diagnostics/network-check-target-4b7d5" Oct 31 04:03:33.206468 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:33.206450 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 04:03:33.206541 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:33.206480 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 04:03:33.206541 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:33.206494 2571 projected.go:196] Error preparing data for projected volume kube-api-access-kjq6x for pod openshift-network-diagnostics/network-check-target-4b7d5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 04:03:33.206635 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:33.206551 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aef43d62-9761-4262-8183-01b9890462a7-kube-api-access-kjq6x podName:aef43d62-9761-4262-8183-01b9890462a7 nodeName:}" failed. No retries permitted until 2026-10-31 04:03:41.206531268 +0000 UTC m=+18.306369136 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-kjq6x" (UniqueName: "kubernetes.io/projected/aef43d62-9761-4262-8183-01b9890462a7-kube-api-access-kjq6x") pod "network-check-target-4b7d5" (UID: "aef43d62-9761-4262-8183-01b9890462a7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 04:03:33.398061 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:33.397413 2571 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T04:03:32.854645361Z","UUID":"eac04352-fac3-4c14-9830-d4765f21c1a4","Handler":null,"Name":"","Endpoint":""} Oct 31 04:03:33.399613 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:33.399140 2571 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 04:03:33.399613 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:33.399172 2571 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 04:03:33.400304 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:33.400241 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4b7d5" Oct 31 04:03:33.400422 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:33.400371 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4b7d5" podUID="aef43d62-9761-4262-8183-01b9890462a7" Oct 31 04:03:33.400534 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:33.400508 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clnlz" Oct 31 04:03:33.400679 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:33.400655 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-clnlz" podUID="97c312b1-4ca5-49c8-8c8d-d247c0fa14ac" Oct 31 04:03:33.527422 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:33.527390 2571 generic.go:356] "Generic (PLEG): container finished" podID="aaae68b8-1654-474f-9e74-45d1e2025d20" containerID="17ae925742d4d05dacaf9d32f8e98e57400e246d5d50542f943298ebd162126f" exitCode=0 Oct 31 04:03:33.527538 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:33.527479 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zzkx" event={"ID":"aaae68b8-1654-474f-9e74-45d1e2025d20","Type":"ContainerDied","Data":"17ae925742d4d05dacaf9d32f8e98e57400e246d5d50542f943298ebd162126f"} Oct 31 04:03:33.531641 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:33.531580 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sqbnh" event={"ID":"5b6d69de-a860-4c82-a03f-23ab7905a5d7","Type":"ContainerStarted","Data":"87c5c1bf54a337519cf7d576ae12c2e52d1f76f2accbb1a1ef6c21c64ba4da70"} Oct 31 04:03:34.399977 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:34.399947 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hnbll" Oct 31 04:03:34.400450 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:34.400070 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hnbll" podUID="c1560abd-3afc-4f94-b0bb-ac7ef8b2464a" Oct 31 04:03:34.536046 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:34.536011 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nbl9z" event={"ID":"3a05c43f-e2db-44c0-b9bd-4bdabca92a98","Type":"ContainerStarted","Data":"6abf97fbba291a0bb8d82a8a9ff737271a550268d77327d8fc42113079775bc8"} Oct 31 04:03:34.540774 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:34.540707 2571 generic.go:356] "Generic (PLEG): container finished" podID="aaae68b8-1654-474f-9e74-45d1e2025d20" containerID="cce4cc3d11b1f597e1ffc5ca34b38255c353c1f9da1d65a8abe12219ac5c942a" exitCode=0 Oct 31 04:03:34.540877 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:34.540794 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zzkx" event={"ID":"aaae68b8-1654-474f-9e74-45d1e2025d20","Type":"ContainerDied","Data":"cce4cc3d11b1f597e1ffc5ca34b38255c353c1f9da1d65a8abe12219ac5c942a"} Oct 31 04:03:34.542734 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:34.542696 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6d7cp" event={"ID":"6d336e6e-feb0-40b2-a625-0e21d25bfac5","Type":"ContainerStarted","Data":"a68eed8fcfcf392381f53a4d6842000a51688ec5bfe9439bc5d63cd454a3c3ec"} Oct 31 04:03:34.546954 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:34.546891 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sqbnh" event={"ID":"5b6d69de-a860-4c82-a03f-23ab7905a5d7","Type":"ContainerStarted","Data":"0a0ec85040426f6e7282da6c1549250523ba652b4e57aad0e076cb155578de89"} Oct 31 04:03:34.577867 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:34.577806 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-nbl9z" podStartSLOduration=3.875909393 podStartE2EDuration="11.577784688s" totalImagesPullingTime="7.701875295s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 04:03:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:03:34.577373916 +0000 UTC m=+11.677212009" watchObservedRunningTime="2026-10-31 04:03:34.577784688 +0000 UTC m=+11.677622578" Oct 31 04:03:34.714301 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:34.714117 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-6d7cp" podStartSLOduration=3.913786366 podStartE2EDuration="11.714102123s" totalImagesPullingTime="7.800315757s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 04:03:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:03:34.713740882 +0000 UTC m=+11.813578768" watchObservedRunningTime="2026-10-31 04:03:34.714102123 +0000 UTC m=+11.813940008" Oct 31 04:03:34.714505 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:34.714422 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sqbnh" podStartSLOduration=3.893743478 podStartE2EDuration="11.71441424s" totalImagesPullingTime="7.820670762s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 04:03:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:03:34.669576411 +0000 UTC m=+11.769414298" watchObservedRunningTime="2026-10-31 04:03:34.71441424 +0000 UTC m=+11.814252185" Oct 31 04:03:35.400850 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:35.400738 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4b7d5" Oct 31 04:03:35.401237 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:35.400851 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4b7d5" podUID="aef43d62-9761-4262-8183-01b9890462a7" Oct 31 04:03:35.401959 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:35.401930 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clnlz" Oct 31 04:03:35.402082 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:35.402059 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-clnlz" podUID="97c312b1-4ca5-49c8-8c8d-d247c0fa14ac" Oct 31 04:03:35.552574 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:35.552540 2571 generic.go:356] "Generic (PLEG): container finished" podID="aaae68b8-1654-474f-9e74-45d1e2025d20" containerID="3dcb83a5957be86beeda6b0b18d2934b93f711c313a9b3e5f87cc061cd49a232" exitCode=0 Oct 31 04:03:35.552746 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:35.552644 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zzkx" event={"ID":"aaae68b8-1654-474f-9e74-45d1e2025d20","Type":"ContainerDied","Data":"3dcb83a5957be86beeda6b0b18d2934b93f711c313a9b3e5f87cc061cd49a232"} Oct 31 04:03:35.554773 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:35.554456 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bhgvp" event={"ID":"ce86be55-219f-46fb-bff6-6dcf87a40ef4","Type":"ContainerStarted","Data":"847f9e87f1f8699f1ad5ed7bf45d36209c00680929ec48744ef1d68ed4787417"} Oct 31 04:03:35.643386 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:35.643312 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bhgvp" podStartSLOduration=4.542018273 podStartE2EDuration="12.643293804s" totalImagesPullingTime="8.101275531s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 04:03:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:03:35.643085089 +0000 UTC m=+12.742922973" watchObservedRunningTime="2026-10-31 04:03:35.643293804 +0000 UTC m=+12.743131693" Oct 31 04:03:36.400256 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:36.400222 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hnbll" Oct 31 04:03:36.400469 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:36.400371 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hnbll" podUID="c1560abd-3afc-4f94-b0bb-ac7ef8b2464a" Oct 31 04:03:36.559075 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:36.559034 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-26vsf" event={"ID":"be0026cb-89bc-413a-bfe3-523d980397fa","Type":"ContainerStarted","Data":"c84953f46905e6a3b584655bc12984489ceb992827d2dd296869fc24dfa02dec"} Oct 31 04:03:36.603475 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:36.603414 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-26vsf" podStartSLOduration=3.386726349 podStartE2EDuration="13.603395754s" totalImagesPullingTime="10.216669405s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 04:03:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:03:36.603183303 +0000 UTC m=+13.703021188" watchObservedRunningTime="2026-10-31 04:03:36.603395754 +0000 UTC m=+13.703233656" Oct 31 04:03:37.400090 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:37.400054 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4b7d5" Oct 31 04:03:37.400254 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:37.400188 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4b7d5" podUID="aef43d62-9761-4262-8183-01b9890462a7" Oct 31 04:03:37.400663 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:37.400497 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clnlz" Oct 31 04:03:37.400663 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:37.400624 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-clnlz" podUID="97c312b1-4ca5-49c8-8c8d-d247c0fa14ac" Oct 31 04:03:37.564525 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:37.564487 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zzkx" event={"ID":"aaae68b8-1654-474f-9e74-45d1e2025d20","Type":"ContainerStarted","Data":"f96e5fcf2fb0ea7c35958096eca0af14cfd77a2d0f79a4a4eb59d6ba1255190d"} Oct 31 04:03:38.399305 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:38.399273 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hnbll" Oct 31 04:03:38.399485 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:38.399416 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hnbll" podUID="c1560abd-3afc-4f94-b0bb-ac7ef8b2464a" Oct 31 04:03:38.570167 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:38.570134 2571 generic.go:356] "Generic (PLEG): container finished" podID="aaae68b8-1654-474f-9e74-45d1e2025d20" containerID="f96e5fcf2fb0ea7c35958096eca0af14cfd77a2d0f79a4a4eb59d6ba1255190d" exitCode=0 Oct 31 04:03:38.570734 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:38.570188 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zzkx" event={"ID":"aaae68b8-1654-474f-9e74-45d1e2025d20","Type":"ContainerDied","Data":"f96e5fcf2fb0ea7c35958096eca0af14cfd77a2d0f79a4a4eb59d6ba1255190d"} Oct 31 04:03:39.400165 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:39.399962 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4b7d5" Oct 31 04:03:39.400346 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:39.400013 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clnlz" Oct 31 04:03:39.400346 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:39.400263 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4b7d5" podUID="aef43d62-9761-4262-8183-01b9890462a7" Oct 31 04:03:39.400477 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:39.400373 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-clnlz" podUID="97c312b1-4ca5-49c8-8c8d-d247c0fa14ac" Oct 31 04:03:39.575182 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:39.575150 2571 generic.go:356] "Generic (PLEG): container finished" podID="aaae68b8-1654-474f-9e74-45d1e2025d20" containerID="f01e6b79ca5c6f0780f8e064f46ecd725b95bf4bb50dc6fee86af2717d108159" exitCode=0 Oct 31 04:03:39.575182 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:39.575188 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zzkx" event={"ID":"aaae68b8-1654-474f-9e74-45d1e2025d20","Type":"ContainerDied","Data":"f01e6b79ca5c6f0780f8e064f46ecd725b95bf4bb50dc6fee86af2717d108159"} Oct 31 04:03:40.399644 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:40.399621 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hnbll" Oct 31 04:03:40.399785 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:40.399758 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hnbll" podUID="c1560abd-3afc-4f94-b0bb-ac7ef8b2464a" Oct 31 04:03:40.580745 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:40.580654 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" event={"ID":"7f6cd22a-499a-4c35-b408-b2aa9b78620d","Type":"ContainerStarted","Data":"6522405b0452faa280fd9379d6d43189ba57f56f36b642f90ce216daf83cd804"} Oct 31 04:03:40.580745 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:40.580737 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" event={"ID":"7f6cd22a-499a-4c35-b408-b2aa9b78620d","Type":"ContainerStarted","Data":"cb8c5067dc706476cefec56ba2beeda97fb145d575163a09ff97bfc5fbe12105"} Oct 31 04:03:40.581254 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:40.580782 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" event={"ID":"7f6cd22a-499a-4c35-b408-b2aa9b78620d","Type":"ContainerStarted","Data":"73033cd6ab969dc5ef64ecba0c4e58b2c0c948fd10b148553a0cafbcfe77a5a5"} Oct 31 04:03:40.581254 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:40.580824 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" event={"ID":"7f6cd22a-499a-4c35-b408-b2aa9b78620d","Type":"ContainerStarted","Data":"57202739abcfadd8ca282ceda97186ac8ff4cf6aa2af89f1aadd1854e144d8c0"} Oct 31 04:03:40.581254 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:40.580871 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" event={"ID":"7f6cd22a-499a-4c35-b408-b2aa9b78620d","Type":"ContainerStarted","Data":"b2335c821273714691c800bfacc955e88b39ab0098f487377e04e2140eb3e152"} Oct 31 04:03:40.581254 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:40.580921 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" event={"ID":"7f6cd22a-499a-4c35-b408-b2aa9b78620d","Type":"ContainerStarted","Data":"8c86a597ba30ee5cfb183ddbee91999f9961514963d363734035a0c5ff480aed"} Oct 31 04:03:40.583974 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:40.583813 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zzkx" event={"ID":"aaae68b8-1654-474f-9e74-45d1e2025d20","Type":"ContainerStarted","Data":"67c48c61b0f8708ea83596d757e78b8453291882ae9a4d8d8f6140b8bf238d5c"} Oct 31 04:03:40.632747 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:40.632699 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6zzkx" podStartSLOduration=7.4395270700000005 podStartE2EDuration="17.632684675s" totalImagesPullingTime="10.193157605s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 04:03:23 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:03:40.632457972 +0000 UTC m=+17.732295857" watchObservedRunningTime="2026-10-31 04:03:40.632684675 +0000 UTC m=+17.732522557" Oct 31 04:03:40.851363 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:40.851250 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-hm5cr" Oct 31 04:03:40.851516 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:40.851409 2571 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 04:03:40.851984 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:40.851965 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hm5cr" Oct 31 04:03:41.165979 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:41.165901 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97c312b1-4ca5-49c8-8c8d-d247c0fa14ac-metrics-certs\") pod \"network-metrics-daemon-clnlz\" (UID: \"97c312b1-4ca5-49c8-8c8d-d247c0fa14ac\") " pod="openshift-multus/network-metrics-daemon-clnlz" Oct 31 04:03:41.165979 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:41.165956 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c1560abd-3afc-4f94-b0bb-ac7ef8b2464a-original-pull-secret\") pod \"global-pull-secret-syncer-hnbll\" (UID: \"c1560abd-3afc-4f94-b0bb-ac7ef8b2464a\") " pod="kube-system/global-pull-secret-syncer-hnbll" Oct 31 04:03:41.166145 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:41.166061 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 04:03:41.166145 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:41.166063 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 04:03:41.166145 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:41.166111 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1560abd-3afc-4f94-b0bb-ac7ef8b2464a-original-pull-secret podName:c1560abd-3afc-4f94-b0bb-ac7ef8b2464a nodeName:}" failed. No retries permitted until 2026-10-31 04:03:57.166100347 +0000 UTC m=+34.265938210 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c1560abd-3afc-4f94-b0bb-ac7ef8b2464a-original-pull-secret") pod "global-pull-secret-syncer-hnbll" (UID: "c1560abd-3afc-4f94-b0bb-ac7ef8b2464a") : object "kube-system"/"original-pull-secret" not registered Oct 31 04:03:41.166145 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:41.166125 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97c312b1-4ca5-49c8-8c8d-d247c0fa14ac-metrics-certs podName:97c312b1-4ca5-49c8-8c8d-d247c0fa14ac nodeName:}" failed. No retries permitted until 2026-10-31 04:03:57.166118754 +0000 UTC m=+34.265956618 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/97c312b1-4ca5-49c8-8c8d-d247c0fa14ac-metrics-certs") pod "network-metrics-daemon-clnlz" (UID: "97c312b1-4ca5-49c8-8c8d-d247c0fa14ac") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 04:03:41.267083 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:41.267034 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjq6x\" (UniqueName: \"kubernetes.io/projected/aef43d62-9761-4262-8183-01b9890462a7-kube-api-access-kjq6x\") pod \"network-check-target-4b7d5\" (UID: \"aef43d62-9761-4262-8183-01b9890462a7\") " pod="openshift-network-diagnostics/network-check-target-4b7d5" Oct 31 04:03:41.267262 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:41.267216 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 04:03:41.267262 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:41.267237 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 04:03:41.267262 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:41.267248 2571 projected.go:196] Error preparing data for projected volume kube-api-access-kjq6x for pod openshift-network-diagnostics/network-check-target-4b7d5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 04:03:41.267398 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:41.267301 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aef43d62-9761-4262-8183-01b9890462a7-kube-api-access-kjq6x podName:aef43d62-9761-4262-8183-01b9890462a7 nodeName:}" failed. No retries permitted until 2026-10-31 04:03:57.267287015 +0000 UTC m=+34.367124877 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-kjq6x" (UniqueName: "kubernetes.io/projected/aef43d62-9761-4262-8183-01b9890462a7-kube-api-access-kjq6x") pod "network-check-target-4b7d5" (UID: "aef43d62-9761-4262-8183-01b9890462a7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 04:03:41.399657 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:41.399625 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4b7d5" Oct 31 04:03:41.399799 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:41.399738 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4b7d5" podUID="aef43d62-9761-4262-8183-01b9890462a7" Oct 31 04:03:41.399799 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:41.399769 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clnlz" Oct 31 04:03:41.399928 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:41.399910 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-clnlz" podUID="97c312b1-4ca5-49c8-8c8d-d247c0fa14ac" Oct 31 04:03:42.399831 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:42.399790 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hnbll" Oct 31 04:03:42.400301 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:42.399914 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hnbll" podUID="c1560abd-3afc-4f94-b0bb-ac7ef8b2464a" Oct 31 04:03:42.591416 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:42.591383 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" event={"ID":"7f6cd22a-499a-4c35-b408-b2aa9b78620d","Type":"ContainerStarted","Data":"bfa3611771c2e3ffbccdbfc2efb58cae6ef82908e4e561f737069fc3b09701c2"} Oct 31 04:03:43.400117 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:43.400087 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clnlz" Oct 31 04:03:43.400647 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:43.400127 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4b7d5" Oct 31 04:03:43.400647 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:43.400205 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-clnlz" podUID="97c312b1-4ca5-49c8-8c8d-d247c0fa14ac" Oct 31 04:03:43.400647 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:43.400251 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4b7d5" podUID="aef43d62-9761-4262-8183-01b9890462a7" Oct 31 04:03:44.399881 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:44.399848 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hnbll" Oct 31 04:03:44.400045 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:44.399978 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hnbll" podUID="c1560abd-3afc-4f94-b0bb-ac7ef8b2464a" Oct 31 04:03:45.399903 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:45.399856 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4b7d5" Oct 31 04:03:45.400718 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:45.399971 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4b7d5" podUID="aef43d62-9761-4262-8183-01b9890462a7" Oct 31 04:03:45.400718 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:45.400033 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clnlz" Oct 31 04:03:45.400718 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:45.400144 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-clnlz" podUID="97c312b1-4ca5-49c8-8c8d-d247c0fa14ac" Oct 31 04:03:45.600691 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:45.600653 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" event={"ID":"7f6cd22a-499a-4c35-b408-b2aa9b78620d","Type":"ContainerStarted","Data":"318e1d7a3df84f9fa1a108e7e170856b9e38ea9d960512eecc88e29e5a68d522"} Oct 31 04:03:45.600950 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:45.600932 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:45.616970 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:45.616944 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:45.696271 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:45.696168 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" podStartSLOduration=8.369936814 podStartE2EDuration="22.696155533s" totalImagesPullingTime="14.326218719s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 04:03:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:03:45.695912398 +0000 UTC m=+22.795750284" watchObservedRunningTime="2026-10-31 04:03:45.696155533 +0000 UTC m=+22.795993418" Oct 31 04:03:46.400133 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:46.400106 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hnbll" Oct 31 04:03:46.400509 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:46.400220 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hnbll" podUID="c1560abd-3afc-4f94-b0bb-ac7ef8b2464a" Oct 31 04:03:46.603560 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:46.603529 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:46.603679 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:46.603658 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:46.621992 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:46.621959 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:03:46.673152 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:46.673082 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hnbll"] Oct 31 04:03:46.673392 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:46.673172 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hnbll" Oct 31 04:03:46.673392 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:46.673247 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hnbll" podUID="c1560abd-3afc-4f94-b0bb-ac7ef8b2464a" Oct 31 04:03:46.676440 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:46.676413 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4b7d5"] Oct 31 04:03:46.676552 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:46.676512 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4b7d5" Oct 31 04:03:46.676617 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:46.676603 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4b7d5" podUID="aef43d62-9761-4262-8183-01b9890462a7" Oct 31 04:03:46.678995 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:46.678974 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-clnlz"] Oct 31 04:03:46.679107 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:46.679063 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clnlz" Oct 31 04:03:46.679158 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:46.679142 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-clnlz" podUID="97c312b1-4ca5-49c8-8c8d-d247c0fa14ac" Oct 31 04:03:48.399450 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:48.399274 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hnbll" Oct 31 04:03:48.399877 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:48.399272 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clnlz" Oct 31 04:03:48.399877 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:48.399539 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hnbll" podUID="c1560abd-3afc-4f94-b0bb-ac7ef8b2464a" Oct 31 04:03:48.399877 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:48.399305 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4b7d5" Oct 31 04:03:48.399877 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:48.399590 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-clnlz" podUID="97c312b1-4ca5-49c8-8c8d-d247c0fa14ac" Oct 31 04:03:48.399877 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:48.399678 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4b7d5" podUID="aef43d62-9761-4262-8183-01b9890462a7" Oct 31 04:03:50.400326 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:50.400281 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hnbll" Oct 31 04:03:50.400326 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:50.400322 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4b7d5" Oct 31 04:03:50.400832 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:50.400402 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clnlz" Oct 31 04:03:50.400832 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:50.400420 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hnbll" podUID="c1560abd-3afc-4f94-b0bb-ac7ef8b2464a" Oct 31 04:03:50.400832 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:50.400492 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4b7d5" podUID="aef43d62-9761-4262-8183-01b9890462a7" Oct 31 04:03:50.400832 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:50.400574 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-clnlz" podUID="97c312b1-4ca5-49c8-8c8d-d247c0fa14ac" Oct 31 04:03:52.399924 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:52.399882 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4b7d5" Oct 31 04:03:52.400457 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:52.400000 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hnbll" Oct 31 04:03:52.400457 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:52.400012 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clnlz" Oct 31 04:03:52.400457 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:52.400021 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4b7d5" podUID="aef43d62-9761-4262-8183-01b9890462a7" Oct 31 04:03:52.400457 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:52.400078 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hnbll" podUID="c1560abd-3afc-4f94-b0bb-ac7ef8b2464a" Oct 31 04:03:52.400457 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:52.400181 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-clnlz" podUID="97c312b1-4ca5-49c8-8c8d-d247c0fa14ac" Oct 31 04:03:53.168112 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.168039 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeReady" Oct 31 04:03:53.168255 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.168213 2571 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 31 04:03:53.220158 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.220126 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-z75pq"] Oct 31 04:03:53.223172 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.223145 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-z75pq" Oct 31 04:03:53.224654 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.224618 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xgjnm"] Oct 31 04:03:53.227728 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.227703 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xqd9k"] Oct 31 04:03:53.227872 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.227784 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-xgjnm" Oct 31 04:03:53.232088 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.231656 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6f69bcc59c-w96bl"] Oct 31 04:03:53.232088 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.231991 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-xqd9k" Oct 31 04:03:53.243602 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.243577 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-pnwds"] Oct 31 04:03:53.243719 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.243698 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f69bcc59c-w96bl" Oct 31 04:03:53.246124 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.246101 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-zqvkx"] Oct 31 04:03:53.246258 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.246230 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-pnwds" Oct 31 04:03:53.248905 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.248867 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hntxx"] Oct 31 04:03:53.249268 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.249202 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 31 04:03:53.249268 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.249206 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zqvkx" Oct 31 04:03:53.249268 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.249249 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 31 04:03:53.252880 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.252854 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-ld96l"] Oct 31 04:03:53.255449 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.254106 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hntxx" Oct 31 04:03:53.255586 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.255563 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-678d747c5f-bgj92"] Oct 31 04:03:53.255722 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.255700 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-ld96l" Oct 31 04:03:53.258690 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.258571 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c64cl"] Oct 31 04:03:53.258841 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.258693 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-678d747c5f-bgj92" Oct 31 04:03:53.262217 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.262190 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 31 04:03:53.263011 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.262878 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-s88kp"] Oct 31 04:03:53.263239 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.263213 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c64cl" Oct 31 04:03:53.265960 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.265906 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 31 04:03:53.267039 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.266768 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-gmtbq\"" Oct 31 04:03:53.267039 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.266796 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 31 04:03:53.271555 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.271403 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-khdq4"] Oct 31 04:03:53.278827 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.278796 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-s88kp" Oct 31 04:03:53.286981 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.286951 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bt9c9"] Oct 31 04:03:53.287717 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.287691 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-khdq4" Oct 31 04:03:53.291709 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.291680 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7f78cfdfd8-w98g7"] Oct 31 04:03:53.292273 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.292249 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bt9c9" Oct 31 04:03:53.297514 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.297486 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-rf8vj"] Oct 31 04:03:53.298537 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.298515 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" Oct 31 04:03:53.309103 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.309071 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zjpdv"] Oct 31 04:03:53.310753 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.310580 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rf8vj" Oct 31 04:03:53.313092 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.313072 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vqsgm"] Oct 31 04:03:53.313226 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.313206 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zjpdv" Oct 31 04:03:53.318002 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.317977 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-z75pq"] Oct 31 04:03:53.318152 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.318021 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xqd9k"] Oct 31 04:03:53.318152 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.318046 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xgjnm"] Oct 31 04:03:53.318152 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.318068 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-pnwds"] Oct 31 04:03:53.318152 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.318090 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-zqvkx"] Oct 31 04:03:53.318152 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.318111 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-s88kp"] Oct 31 04:03:53.318152 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.318130 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-rf8vj"] Oct 31 04:03:53.318152 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.318147 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-ld96l"] Oct 31 04:03:53.318491 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.318163 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hntxx"] Oct 31 04:03:53.318491 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.318181 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f78cfdfd8-w98g7"] Oct 31 04:03:53.318491 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.318204 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c64cl"] Oct 31 04:03:53.318491 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.318235 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f69bcc59c-w96bl"] Oct 31 04:03:53.318491 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.318256 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-678d747c5f-bgj92"] Oct 31 04:03:53.318491 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.318279 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vqsgm"] Oct 31 04:03:53.318491 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.318299 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zjpdv"] Oct 31 04:03:53.318491 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.318325 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bt9c9"] Oct 31 04:03:53.318491 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.318356 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vqsgm" Oct 31 04:03:53.318491 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.318361 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-khdq4"] Oct 31 04:03:53.343386 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.343324 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/197dee13-748f-4c6d-acc5-1793b5cee80a-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-c64cl\" (UID: \"197dee13-748f-4c6d-acc5-1793b5cee80a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c64cl" Oct 31 04:03:53.343539 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.343432 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/93beb8b8-c9e7-4cce-8baf-8497b971e193-ca-trust-extracted\") pod \"image-registry-7f78cfdfd8-w98g7\" (UID: \"93beb8b8-c9e7-4cce-8baf-8497b971e193\") " pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" Oct 31 04:03:53.343539 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.343464 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/93beb8b8-c9e7-4cce-8baf-8497b971e193-installation-pull-secrets\") pod \"image-registry-7f78cfdfd8-w98g7\" (UID: \"93beb8b8-c9e7-4cce-8baf-8497b971e193\") " pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" Oct 31 04:03:53.343627 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.343534 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-trusted-ca\") pod \"image-registry-6f69bcc59c-w96bl\" (UID: \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\") " pod="openshift-image-registry/image-registry-6f69bcc59c-w96bl" Oct 31 04:03:53.343627 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.343594 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e667439c-7121-4c5f-84fe-b36f2391dc66-tmp\") pod \"insights-operator-7c4986d449-ld96l\" (UID: \"e667439c-7121-4c5f-84fe-b36f2391dc66\") " pod="openshift-insights/insights-operator-7c4986d449-ld96l" Oct 31 04:03:53.343627 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.343614 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e667439c-7121-4c5f-84fe-b36f2391dc66-snapshots\") pod \"insights-operator-7c4986d449-ld96l\" (UID: \"e667439c-7121-4c5f-84fe-b36f2391dc66\") " pod="openshift-insights/insights-operator-7c4986d449-ld96l" Oct 31 04:03:53.343736 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.343631 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqx72\" (UniqueName: \"kubernetes.io/projected/3945ff65-e877-4372-9334-3968c68b7295-kube-api-access-fqx72\") pod \"console-operator-7cd56b8dbc-s88kp\" (UID: \"3945ff65-e877-4372-9334-3968c68b7295\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-s88kp" Oct 31 04:03:53.343736 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.343651 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghxbr\" (UniqueName: \"kubernetes.io/projected/944c9831-b2fa-4f58-bba2-93288793b125-kube-api-access-ghxbr\") pod \"service-ca-operator-544d749c8c-khdq4\" (UID: \"944c9831-b2fa-4f58-bba2-93288793b125\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-khdq4" Oct 31 04:03:53.343736 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.343670 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6xnt\" (UniqueName: \"kubernetes.io/projected/d22707c7-21fa-4511-b0b1-869b41fb2351-kube-api-access-v6xnt\") pod \"router-default-678d747c5f-bgj92\" (UID: \"d22707c7-21fa-4511-b0b1-869b41fb2351\") " pod="openshift-ingress/router-default-678d747c5f-bgj92" Oct 31 04:03:53.343736 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.343713 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c3c90e6f-df5e-4c17-bec4-554280a8e8e7-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bt9c9\" (UID: \"c3c90e6f-df5e-4c17-bec4-554280a8e8e7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bt9c9" Oct 31 04:03:53.343944 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.343749 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c3c90e6f-df5e-4c17-bec4-554280a8e8e7-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bt9c9\" (UID: \"c3c90e6f-df5e-4c17-bec4-554280a8e8e7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bt9c9" Oct 31 04:03:53.343944 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.343838 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlg9n\" (UniqueName: \"kubernetes.io/projected/b541554a-0460-4716-9f14-f7a46567465c-kube-api-access-qlg9n\") pod \"kas-connection-checker-685895c7c9-xgjnm\" (UID: \"b541554a-0460-4716-9f14-f7a46567465c\") " pod="kube-system/kas-connection-checker-685895c7c9-xgjnm" Oct 31 04:03:53.343944 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.343901 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a173790b-6c13-45a1-a361-3fe51844b4c2-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-zqvkx\" (UID: \"a173790b-6c13-45a1-a361-3fe51844b4c2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zqvkx" Oct 31 04:03:53.343944 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.343937 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e667439c-7121-4c5f-84fe-b36f2391dc66-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-ld96l\" (UID: \"e667439c-7121-4c5f-84fe-b36f2391dc66\") " pod="openshift-insights/insights-operator-7c4986d449-ld96l" Oct 31 04:03:53.344096 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.343961 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d22707c7-21fa-4511-b0b1-869b41fb2351-default-certificate\") pod \"router-default-678d747c5f-bgj92\" (UID: \"d22707c7-21fa-4511-b0b1-869b41fb2351\") " pod="openshift-ingress/router-default-678d747c5f-bgj92" Oct 31 04:03:53.344096 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.344029 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wddq5\" (UniqueName: \"kubernetes.io/projected/e667439c-7121-4c5f-84fe-b36f2391dc66-kube-api-access-wddq5\") pod \"insights-operator-7c4986d449-ld96l\" (UID: \"e667439c-7121-4c5f-84fe-b36f2391dc66\") " pod="openshift-insights/insights-operator-7c4986d449-ld96l" Oct 31 04:03:53.344096 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.344047 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3945ff65-e877-4372-9334-3968c68b7295-config\") pod \"console-operator-7cd56b8dbc-s88kp\" (UID: \"3945ff65-e877-4372-9334-3968c68b7295\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-s88kp" Oct 31 04:03:53.344275 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.344108 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/93beb8b8-c9e7-4cce-8baf-8497b971e193-trusted-ca\") pod \"image-registry-7f78cfdfd8-w98g7\" (UID: \"93beb8b8-c9e7-4cce-8baf-8497b971e193\") " pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" Oct 31 04:03:53.344275 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.344139 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/944c9831-b2fa-4f58-bba2-93288793b125-serving-cert\") pod \"service-ca-operator-544d749c8c-khdq4\" (UID: \"944c9831-b2fa-4f58-bba2-93288793b125\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-khdq4" Oct 31 04:03:53.344275 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.344155 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/93beb8b8-c9e7-4cce-8baf-8497b971e193-bound-sa-token\") pod \"image-registry-7f78cfdfd8-w98g7\" (UID: \"93beb8b8-c9e7-4cce-8baf-8497b971e193\") " pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" Oct 31 04:03:53.344275 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.344179 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc2vq\" (UniqueName: \"kubernetes.io/projected/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-kube-api-access-xc2vq\") pod \"image-registry-6f69bcc59c-w96bl\" (UID: \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\") " pod="openshift-image-registry/image-registry-6f69bcc59c-w96bl" Oct 31 04:03:53.344275 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.344208 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/93beb8b8-c9e7-4cce-8baf-8497b971e193-registry-tls\") pod \"image-registry-7f78cfdfd8-w98g7\" (UID: \"93beb8b8-c9e7-4cce-8baf-8497b971e193\") " pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" Oct 31 04:03:53.344275 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.344241 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-installation-pull-secrets\") pod \"image-registry-6f69bcc59c-w96bl\" (UID: \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\") " pod="openshift-image-registry/image-registry-6f69bcc59c-w96bl" Oct 31 04:03:53.344568 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.344285 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/183f4178-99c0-4945-b490-c71cbd2780f3-nginx-conf\") pod \"networking-console-plugin-86c478bb88-pnwds\" (UID: \"183f4178-99c0-4945-b490-c71cbd2780f3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pnwds" Oct 31 04:03:53.344568 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.344359 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mtpj\" (UniqueName: \"kubernetes.io/projected/c3c90e6f-df5e-4c17-bec4-554280a8e8e7-kube-api-access-7mtpj\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bt9c9\" (UID: \"c3c90e6f-df5e-4c17-bec4-554280a8e8e7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bt9c9" Oct 31 04:03:53.344568 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.344410 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/197dee13-748f-4c6d-acc5-1793b5cee80a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-c64cl\" (UID: \"197dee13-748f-4c6d-acc5-1793b5cee80a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c64cl" Oct 31 04:03:53.344568 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.344441 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cq9b\" (UniqueName: \"kubernetes.io/projected/197dee13-748f-4c6d-acc5-1793b5cee80a-kube-api-access-7cq9b\") pod \"cluster-samples-operator-6ddb6c8b94-c64cl\" (UID: \"197dee13-748f-4c6d-acc5-1793b5cee80a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c64cl" Oct 31 04:03:53.344568 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.344468 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d22707c7-21fa-4511-b0b1-869b41fb2351-service-ca-bundle\") pod \"router-default-678d747c5f-bgj92\" (UID: \"d22707c7-21fa-4511-b0b1-869b41fb2351\") " pod="openshift-ingress/router-default-678d747c5f-bgj92" Oct 31 04:03:53.344568 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.344502 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d22707c7-21fa-4511-b0b1-869b41fb2351-stats-auth\") pod \"router-default-678d747c5f-bgj92\" (UID: \"d22707c7-21fa-4511-b0b1-869b41fb2351\") " pod="openshift-ingress/router-default-678d747c5f-bgj92" Oct 31 04:03:53.344852 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.344585 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxlpp\" (UniqueName: \"kubernetes.io/projected/c4a091a2-58d2-4b05-a968-d16fe1bd9d40-kube-api-access-nxlpp\") pod \"volume-data-source-validator-58c9f5f46d-hntxx\" (UID: \"c4a091a2-58d2-4b05-a968-d16fe1bd9d40\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hntxx" Oct 31 04:03:53.344852 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.344612 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-registry-tls\") pod \"image-registry-6f69bcc59c-w96bl\" (UID: \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\") " pod="openshift-image-registry/image-registry-6f69bcc59c-w96bl" Oct 31 04:03:53.344852 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.344659 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k52nk\" (UniqueName: \"kubernetes.io/projected/2050cf2e-0f83-43b3-bb85-7bd4ac42a473-kube-api-access-k52nk\") pod \"kas-connection-checker-685895c7c9-z75pq\" (UID: \"2050cf2e-0f83-43b3-bb85-7bd4ac42a473\") " pod="kube-system/kas-connection-checker-685895c7c9-z75pq" Oct 31 04:03:53.344852 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.344684 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-registry-certificates\") pod \"image-registry-6f69bcc59c-w96bl\" (UID: \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\") " pod="openshift-image-registry/image-registry-6f69bcc59c-w96bl" Oct 31 04:03:53.344852 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.344709 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/944c9831-b2fa-4f58-bba2-93288793b125-config\") pod \"service-ca-operator-544d749c8c-khdq4\" (UID: \"944c9831-b2fa-4f58-bba2-93288793b125\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-khdq4" Oct 31 04:03:53.344852 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.344741 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/93beb8b8-c9e7-4cce-8baf-8497b971e193-registry-certificates\") pod \"image-registry-7f78cfdfd8-w98g7\" (UID: \"93beb8b8-c9e7-4cce-8baf-8497b971e193\") " pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" Oct 31 04:03:53.344852 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.344772 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/93beb8b8-c9e7-4cce-8baf-8497b971e193-image-registry-private-configuration\") pod \"image-registry-7f78cfdfd8-w98g7\" (UID: \"93beb8b8-c9e7-4cce-8baf-8497b971e193\") " pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" Oct 31 04:03:53.344852 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.344821 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25x8r\" (UniqueName: \"kubernetes.io/projected/93beb8b8-c9e7-4cce-8baf-8497b971e193-kube-api-access-25x8r\") pod \"image-registry-7f78cfdfd8-w98g7\" (UID: \"93beb8b8-c9e7-4cce-8baf-8497b971e193\") " pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" Oct 31 04:03:53.344852 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.344842 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/183f4178-99c0-4945-b490-c71cbd2780f3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pnwds\" (UID: \"183f4178-99c0-4945-b490-c71cbd2780f3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pnwds" Oct 31 04:03:53.345175 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.344881 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-ca-trust-extracted\") pod \"image-registry-6f69bcc59c-w96bl\" (UID: \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\") " pod="openshift-image-registry/image-registry-6f69bcc59c-w96bl" Oct 31 04:03:53.345175 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.344907 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e667439c-7121-4c5f-84fe-b36f2391dc66-serving-cert\") pod \"insights-operator-7c4986d449-ld96l\" (UID: \"e667439c-7121-4c5f-84fe-b36f2391dc66\") " pod="openshift-insights/insights-operator-7c4986d449-ld96l" Oct 31 04:03:53.345175 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.344925 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3945ff65-e877-4372-9334-3968c68b7295-trusted-ca\") pod \"console-operator-7cd56b8dbc-s88kp\" (UID: \"3945ff65-e877-4372-9334-3968c68b7295\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-s88kp" Oct 31 04:03:53.345175 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.345020 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e667439c-7121-4c5f-84fe-b36f2391dc66-service-ca-bundle\") pod \"insights-operator-7c4986d449-ld96l\" (UID: \"e667439c-7121-4c5f-84fe-b36f2391dc66\") " pod="openshift-insights/insights-operator-7c4986d449-ld96l" Oct 31 04:03:53.345175 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.345048 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d22707c7-21fa-4511-b0b1-869b41fb2351-metrics-certs\") pod \"router-default-678d747c5f-bgj92\" (UID: \"d22707c7-21fa-4511-b0b1-869b41fb2351\") " pod="openshift-ingress/router-default-678d747c5f-bgj92" Oct 31 04:03:53.345175 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.345095 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a173790b-6c13-45a1-a361-3fe51844b4c2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-zqvkx\" (UID: \"a173790b-6c13-45a1-a361-3fe51844b4c2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zqvkx" Oct 31 04:03:53.345175 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.345116 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3945ff65-e877-4372-9334-3968c68b7295-serving-cert\") pod \"console-operator-7cd56b8dbc-s88kp\" (UID: \"3945ff65-e877-4372-9334-3968c68b7295\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-s88kp" Oct 31 04:03:53.345175 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.345170 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-bound-sa-token\") pod \"image-registry-6f69bcc59c-w96bl\" (UID: \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\") " pod="openshift-image-registry/image-registry-6f69bcc59c-w96bl" Oct 31 04:03:53.345441 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.345189 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj5t7\" (UniqueName: \"kubernetes.io/projected/57b47c7b-9c84-4888-be69-64afca28a8c0-kube-api-access-jj5t7\") pod \"kas-connection-checker-685895c7c9-xqd9k\" (UID: \"57b47c7b-9c84-4888-be69-64afca28a8c0\") " pod="kube-system/kas-connection-checker-685895c7c9-xqd9k" Oct 31 04:03:53.345441 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.345207 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggq72\" (UniqueName: \"kubernetes.io/projected/a173790b-6c13-45a1-a361-3fe51844b4c2-kube-api-access-ggq72\") pod \"cluster-monitoring-operator-884576c57-zqvkx\" (UID: \"a173790b-6c13-45a1-a361-3fe51844b4c2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zqvkx" Oct 31 04:03:53.345441 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.345240 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-image-registry-private-configuration\") pod \"image-registry-6f69bcc59c-w96bl\" (UID: \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\") " pod="openshift-image-registry/image-registry-6f69bcc59c-w96bl" Oct 31 04:03:53.349240 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.349219 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 31 04:03:53.349240 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.349233 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 04:03:53.349448 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.349344 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 04:03:53.349448 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.349387 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 31 04:03:53.349448 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.349397 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 04:03:53.349853 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.349830 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 31 04:03:53.349853 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.349845 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 31 04:03:53.349931 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.349919 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 31 04:03:53.350107 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.350093 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 31 04:03:53.350154 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.350095 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 31 04:03:53.350268 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.350199 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 04:03:53.350268 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.350253 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 31 04:03:53.350268 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.350254 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 31 04:03:53.350463 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.350279 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 31 04:03:53.350463 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.350287 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 31 04:03:53.350463 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.350325 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 31 04:03:53.350638 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.350617 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 04:03:53.350734 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.350666 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 04:03:53.350734 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.350677 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 04:03:53.350734 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.350714 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 04:03:53.350888 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.350870 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 31 04:03:53.350941 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.350903 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 04:03:53.350996 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.350940 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 31 04:03:53.350996 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.350929 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 31 04:03:53.351069 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.351000 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 31 04:03:53.351069 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.351039 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 31 04:03:53.364242 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.364204 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 31 04:03:53.364242 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.364224 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 31 04:03:53.377652 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.377628 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 31 04:03:53.379659 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.379636 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 31 04:03:53.379781 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.379769 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 31 04:03:53.379862 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.379843 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 31 04:03:53.380375 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.380345 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 31 04:03:53.380567 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.380538 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-c8446\"" Oct 31 04:03:53.380972 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.379846 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 31 04:03:53.381260 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.381239 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-s6w7q\"" Oct 31 04:03:53.381799 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.381404 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-gw6dk\"" Oct 31 04:03:53.381799 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.380695 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 31 04:03:53.381799 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.381008 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-677wb\"" Oct 31 04:03:53.381799 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.381044 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-jnm7g\"" Oct 31 04:03:53.381799 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.381641 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-np8k4\"" Oct 31 04:03:53.381799 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.381392 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 31 04:03:53.382125 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.380675 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 04:03:53.382125 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.381409 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 31 04:03:53.382125 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.381988 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 31 04:03:53.382125 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.382090 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 31 04:03:53.382125 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.382097 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 31 04:03:53.382354 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.380680 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-7brl8\"" Oct 31 04:03:53.382489 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.382327 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-bqqdt\"" Oct 31 04:03:53.382770 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.382751 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qzkqw\"" Oct 31 04:03:53.382841 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.382772 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 31 04:03:53.382841 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.382802 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-wtrm9\"" Oct 31 04:03:53.382841 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.382756 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-j9kgn\"" Oct 31 04:03:53.383035 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.383020 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 04:03:53.383185 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.383169 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-fl6gf\"" Oct 31 04:03:53.383251 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.383204 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 31 04:03:53.445663 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.445578 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/93beb8b8-c9e7-4cce-8baf-8497b971e193-image-registry-private-configuration\") pod \"image-registry-7f78cfdfd8-w98g7\" (UID: \"93beb8b8-c9e7-4cce-8baf-8497b971e193\") " pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" Oct 31 04:03:53.445663 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.445616 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25x8r\" (UniqueName: \"kubernetes.io/projected/93beb8b8-c9e7-4cce-8baf-8497b971e193-kube-api-access-25x8r\") pod \"image-registry-7f78cfdfd8-w98g7\" (UID: \"93beb8b8-c9e7-4cce-8baf-8497b971e193\") " pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" Oct 31 04:03:53.445663 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.445644 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/183f4178-99c0-4945-b490-c71cbd2780f3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pnwds\" (UID: \"183f4178-99c0-4945-b490-c71cbd2780f3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pnwds" Oct 31 04:03:53.446243 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.445669 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-ca-trust-extracted\") pod \"image-registry-6f69bcc59c-w96bl\" (UID: \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\") " pod="openshift-image-registry/image-registry-6f69bcc59c-w96bl" Oct 31 04:03:53.446243 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.445694 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e667439c-7121-4c5f-84fe-b36f2391dc66-serving-cert\") pod \"insights-operator-7c4986d449-ld96l\" (UID: \"e667439c-7121-4c5f-84fe-b36f2391dc66\") " pod="openshift-insights/insights-operator-7c4986d449-ld96l" Oct 31 04:03:53.446243 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.445719 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3945ff65-e877-4372-9334-3968c68b7295-trusted-ca\") pod \"console-operator-7cd56b8dbc-s88kp\" (UID: \"3945ff65-e877-4372-9334-3968c68b7295\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-s88kp" Oct 31 04:03:53.446243 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.445748 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ll7m\" (UniqueName: \"kubernetes.io/projected/a2ce92f3-5b05-4c89-9d53-a57bc85b17ef-kube-api-access-7ll7m\") pod \"ingress-canary-vqsgm\" (UID: \"a2ce92f3-5b05-4c89-9d53-a57bc85b17ef\") " pod="openshift-ingress-canary/ingress-canary-vqsgm" Oct 31 04:03:53.446243 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.445783 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e667439c-7121-4c5f-84fe-b36f2391dc66-service-ca-bundle\") pod \"insights-operator-7c4986d449-ld96l\" (UID: \"e667439c-7121-4c5f-84fe-b36f2391dc66\") " pod="openshift-insights/insights-operator-7c4986d449-ld96l" Oct 31 04:03:53.446243 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.445807 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d22707c7-21fa-4511-b0b1-869b41fb2351-metrics-certs\") pod \"router-default-678d747c5f-bgj92\" (UID: \"d22707c7-21fa-4511-b0b1-869b41fb2351\") " pod="openshift-ingress/router-default-678d747c5f-bgj92" Oct 31 04:03:53.446243 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.445838 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a173790b-6c13-45a1-a361-3fe51844b4c2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-zqvkx\" (UID: \"a173790b-6c13-45a1-a361-3fe51844b4c2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zqvkx" Oct 31 04:03:53.446243 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.445866 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3945ff65-e877-4372-9334-3968c68b7295-serving-cert\") pod \"console-operator-7cd56b8dbc-s88kp\" (UID: \"3945ff65-e877-4372-9334-3968c68b7295\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-s88kp" Oct 31 04:03:53.446243 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.445895 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-bound-sa-token\") pod \"image-registry-6f69bcc59c-w96bl\" (UID: \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\") " pod="openshift-image-registry/image-registry-6f69bcc59c-w96bl" Oct 31 04:03:53.446243 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.445917 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2ce92f3-5b05-4c89-9d53-a57bc85b17ef-cert\") pod \"ingress-canary-vqsgm\" (UID: \"a2ce92f3-5b05-4c89-9d53-a57bc85b17ef\") " pod="openshift-ingress-canary/ingress-canary-vqsgm" Oct 31 04:03:53.446243 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.445943 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj5t7\" (UniqueName: \"kubernetes.io/projected/57b47c7b-9c84-4888-be69-64afca28a8c0-kube-api-access-jj5t7\") pod \"kas-connection-checker-685895c7c9-xqd9k\" (UID: \"57b47c7b-9c84-4888-be69-64afca28a8c0\") " pod="kube-system/kas-connection-checker-685895c7c9-xqd9k" Oct 31 04:03:53.446243 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.445965 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggq72\" (UniqueName: \"kubernetes.io/projected/a173790b-6c13-45a1-a361-3fe51844b4c2-kube-api-access-ggq72\") pod \"cluster-monitoring-operator-884576c57-zqvkx\" (UID: \"a173790b-6c13-45a1-a361-3fe51844b4c2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zqvkx" Oct 31 04:03:53.446243 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.445988 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-image-registry-private-configuration\") pod \"image-registry-6f69bcc59c-w96bl\" (UID: \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\") " pod="openshift-image-registry/image-registry-6f69bcc59c-w96bl" Oct 31 04:03:53.446243 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.446026 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/197dee13-748f-4c6d-acc5-1793b5cee80a-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-c64cl\" (UID: \"197dee13-748f-4c6d-acc5-1793b5cee80a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c64cl" Oct 31 04:03:53.446243 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:53.446045 2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 04:03:53.446243 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.446059 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/93beb8b8-c9e7-4cce-8baf-8497b971e193-ca-trust-extracted\") pod \"image-registry-7f78cfdfd8-w98g7\" (UID: \"93beb8b8-c9e7-4cce-8baf-8497b971e193\") " pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" Oct 31 04:03:53.447031 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.446086 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/93beb8b8-c9e7-4cce-8baf-8497b971e193-installation-pull-secrets\") pod \"image-registry-7f78cfdfd8-w98g7\" (UID: \"93beb8b8-c9e7-4cce-8baf-8497b971e193\") " pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" Oct 31 04:03:53.447031 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:53.446124 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d22707c7-21fa-4511-b0b1-869b41fb2351-metrics-certs podName:d22707c7-21fa-4511-b0b1-869b41fb2351 nodeName:}" failed. No retries permitted until 2026-10-31 04:03:53.946100373 +0000 UTC m=+31.045938252 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d22707c7-21fa-4511-b0b1-869b41fb2351-metrics-certs") pod "router-default-678d747c5f-bgj92" (UID: "d22707c7-21fa-4511-b0b1-869b41fb2351") : secret "router-metrics-certs-default" not found Oct 31 04:03:53.447031 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.446152 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/48309c9d-5f34-49bf-b55b-f74959984e0a-config-volume\") pod \"dns-default-zjpdv\" (UID: \"48309c9d-5f34-49bf-b55b-f74959984e0a\") " pod="openshift-dns/dns-default-zjpdv" Oct 31 04:03:53.447031 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.446195 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-trusted-ca\") pod \"image-registry-6f69bcc59c-w96bl\" (UID: \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\") " pod="openshift-image-registry/image-registry-6f69bcc59c-w96bl" Oct 31 04:03:53.447031 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.446226 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48309c9d-5f34-49bf-b55b-f74959984e0a-tmp-dir\") pod \"dns-default-zjpdv\" (UID: \"48309c9d-5f34-49bf-b55b-f74959984e0a\") " pod="openshift-dns/dns-default-zjpdv" Oct 31 04:03:53.447031 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.446262 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e667439c-7121-4c5f-84fe-b36f2391dc66-tmp\") pod \"insights-operator-7c4986d449-ld96l\" (UID: \"e667439c-7121-4c5f-84fe-b36f2391dc66\") " pod="openshift-insights/insights-operator-7c4986d449-ld96l" Oct 31 04:03:53.447031 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.446289 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e667439c-7121-4c5f-84fe-b36f2391dc66-snapshots\") pod \"insights-operator-7c4986d449-ld96l\" (UID: \"e667439c-7121-4c5f-84fe-b36f2391dc66\") " pod="openshift-insights/insights-operator-7c4986d449-ld96l" Oct 31 04:03:53.447031 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.446318 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqx72\" (UniqueName: \"kubernetes.io/projected/3945ff65-e877-4372-9334-3968c68b7295-kube-api-access-fqx72\") pod \"console-operator-7cd56b8dbc-s88kp\" (UID: \"3945ff65-e877-4372-9334-3968c68b7295\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-s88kp" Oct 31 04:03:53.447031 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.446431 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghxbr\" (UniqueName: \"kubernetes.io/projected/944c9831-b2fa-4f58-bba2-93288793b125-kube-api-access-ghxbr\") pod \"service-ca-operator-544d749c8c-khdq4\" (UID: \"944c9831-b2fa-4f58-bba2-93288793b125\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-khdq4" Oct 31 04:03:53.447031 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.446460 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6xnt\" (UniqueName: \"kubernetes.io/projected/d22707c7-21fa-4511-b0b1-869b41fb2351-kube-api-access-v6xnt\") pod \"router-default-678d747c5f-bgj92\" (UID: \"d22707c7-21fa-4511-b0b1-869b41fb2351\") " pod="openshift-ingress/router-default-678d747c5f-bgj92" Oct 31 04:03:53.447031 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.446487 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c3c90e6f-df5e-4c17-bec4-554280a8e8e7-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bt9c9\" (UID: \"c3c90e6f-df5e-4c17-bec4-554280a8e8e7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bt9c9" Oct 31 04:03:53.447031 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.446515 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c3c90e6f-df5e-4c17-bec4-554280a8e8e7-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bt9c9\" (UID: \"c3c90e6f-df5e-4c17-bec4-554280a8e8e7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bt9c9" Oct 31 04:03:53.447031 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.446548 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlg9n\" (UniqueName: \"kubernetes.io/projected/b541554a-0460-4716-9f14-f7a46567465c-kube-api-access-qlg9n\") pod \"kas-connection-checker-685895c7c9-xgjnm\" (UID: \"b541554a-0460-4716-9f14-f7a46567465c\") " pod="kube-system/kas-connection-checker-685895c7c9-xgjnm" Oct 31 04:03:53.447031 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.446577 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsrbd\" (UniqueName: \"kubernetes.io/projected/fb656990-e663-4fea-8733-18ded9222a79-kube-api-access-hsrbd\") pod \"network-check-source-6f78d59f99-rf8vj\" (UID: \"fb656990-e663-4fea-8733-18ded9222a79\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rf8vj" Oct 31 04:03:53.447031 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.446608 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a173790b-6c13-45a1-a361-3fe51844b4c2-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-zqvkx\" (UID: \"a173790b-6c13-45a1-a361-3fe51844b4c2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zqvkx" Oct 31 04:03:53.447792 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.446647 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e667439c-7121-4c5f-84fe-b36f2391dc66-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-ld96l\" (UID: \"e667439c-7121-4c5f-84fe-b36f2391dc66\") " pod="openshift-insights/insights-operator-7c4986d449-ld96l" Oct 31 04:03:53.447792 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.446673 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d22707c7-21fa-4511-b0b1-869b41fb2351-default-certificate\") pod \"router-default-678d747c5f-bgj92\" (UID: \"d22707c7-21fa-4511-b0b1-869b41fb2351\") " pod="openshift-ingress/router-default-678d747c5f-bgj92" Oct 31 04:03:53.447792 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.446720 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt6hk\" (UniqueName: \"kubernetes.io/projected/48309c9d-5f34-49bf-b55b-f74959984e0a-kube-api-access-kt6hk\") pod \"dns-default-zjpdv\" (UID: \"48309c9d-5f34-49bf-b55b-f74959984e0a\") " pod="openshift-dns/dns-default-zjpdv" Oct 31 04:03:53.447792 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.446749 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wddq5\" (UniqueName: \"kubernetes.io/projected/e667439c-7121-4c5f-84fe-b36f2391dc66-kube-api-access-wddq5\") pod \"insights-operator-7c4986d449-ld96l\" (UID: \"e667439c-7121-4c5f-84fe-b36f2391dc66\") " pod="openshift-insights/insights-operator-7c4986d449-ld96l" Oct 31 04:03:53.447792 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.446778 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3945ff65-e877-4372-9334-3968c68b7295-config\") pod \"console-operator-7cd56b8dbc-s88kp\" (UID: \"3945ff65-e877-4372-9334-3968c68b7295\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-s88kp" Oct 31 04:03:53.447792 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.446826 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/93beb8b8-c9e7-4cce-8baf-8497b971e193-trusted-ca\") pod \"image-registry-7f78cfdfd8-w98g7\" (UID: \"93beb8b8-c9e7-4cce-8baf-8497b971e193\") " pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" Oct 31 04:03:53.447792 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.446873 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/944c9831-b2fa-4f58-bba2-93288793b125-serving-cert\") pod \"service-ca-operator-544d749c8c-khdq4\" (UID: \"944c9831-b2fa-4f58-bba2-93288793b125\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-khdq4" Oct 31 04:03:53.447792 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.446898 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/93beb8b8-c9e7-4cce-8baf-8497b971e193-bound-sa-token\") pod \"image-registry-7f78cfdfd8-w98g7\" (UID: \"93beb8b8-c9e7-4cce-8baf-8497b971e193\") " pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" Oct 31 04:03:53.447792 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.447048 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3945ff65-e877-4372-9334-3968c68b7295-trusted-ca\") pod \"console-operator-7cd56b8dbc-s88kp\" (UID: \"3945ff65-e877-4372-9334-3968c68b7295\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-s88kp" Oct 31 04:03:53.447792 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.447118 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc2vq\" (UniqueName: \"kubernetes.io/projected/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-kube-api-access-xc2vq\") pod \"image-registry-6f69bcc59c-w96bl\" (UID: \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\") " pod="openshift-image-registry/image-registry-6f69bcc59c-w96bl" Oct 31 04:03:53.447792 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.447156 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/93beb8b8-c9e7-4cce-8baf-8497b971e193-registry-tls\") pod \"image-registry-7f78cfdfd8-w98g7\" (UID: \"93beb8b8-c9e7-4cce-8baf-8497b971e193\") " pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" Oct 31 04:03:53.447792 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.447176 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-installation-pull-secrets\") pod \"image-registry-6f69bcc59c-w96bl\" (UID: \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\") " pod="openshift-image-registry/image-registry-6f69bcc59c-w96bl" Oct 31 04:03:53.447792 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.447195 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/183f4178-99c0-4945-b490-c71cbd2780f3-nginx-conf\") pod \"networking-console-plugin-86c478bb88-pnwds\" (UID: \"183f4178-99c0-4945-b490-c71cbd2780f3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pnwds" Oct 31 04:03:53.447792 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.447211 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mtpj\" (UniqueName: \"kubernetes.io/projected/c3c90e6f-df5e-4c17-bec4-554280a8e8e7-kube-api-access-7mtpj\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bt9c9\" (UID: \"c3c90e6f-df5e-4c17-bec4-554280a8e8e7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bt9c9" Oct 31 04:03:53.447792 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.447234 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/48309c9d-5f34-49bf-b55b-f74959984e0a-metrics-tls\") pod \"dns-default-zjpdv\" (UID: \"48309c9d-5f34-49bf-b55b-f74959984e0a\") " pod="openshift-dns/dns-default-zjpdv" Oct 31 04:03:53.447792 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.447253 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/197dee13-748f-4c6d-acc5-1793b5cee80a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-c64cl\" (UID: \"197dee13-748f-4c6d-acc5-1793b5cee80a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c64cl" Oct 31 04:03:53.448589 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.447269 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cq9b\" (UniqueName: \"kubernetes.io/projected/197dee13-748f-4c6d-acc5-1793b5cee80a-kube-api-access-7cq9b\") pod \"cluster-samples-operator-6ddb6c8b94-c64cl\" (UID: \"197dee13-748f-4c6d-acc5-1793b5cee80a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c64cl" Oct 31 04:03:53.448589 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.447285 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d22707c7-21fa-4511-b0b1-869b41fb2351-service-ca-bundle\") pod \"router-default-678d747c5f-bgj92\" (UID: \"d22707c7-21fa-4511-b0b1-869b41fb2351\") " pod="openshift-ingress/router-default-678d747c5f-bgj92" Oct 31 04:03:53.448589 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.447300 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d22707c7-21fa-4511-b0b1-869b41fb2351-stats-auth\") pod \"router-default-678d747c5f-bgj92\" (UID: \"d22707c7-21fa-4511-b0b1-869b41fb2351\") " pod="openshift-ingress/router-default-678d747c5f-bgj92" Oct 31 04:03:53.448589 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.447347 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxlpp\" (UniqueName: \"kubernetes.io/projected/c4a091a2-58d2-4b05-a968-d16fe1bd9d40-kube-api-access-nxlpp\") pod \"volume-data-source-validator-58c9f5f46d-hntxx\" (UID: \"c4a091a2-58d2-4b05-a968-d16fe1bd9d40\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hntxx" Oct 31 04:03:53.448589 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.447372 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-registry-tls\") pod \"image-registry-6f69bcc59c-w96bl\" (UID: \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\") " pod="openshift-image-registry/image-registry-6f69bcc59c-w96bl" Oct 31 04:03:53.448589 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.447397 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k52nk\" (UniqueName: \"kubernetes.io/projected/2050cf2e-0f83-43b3-bb85-7bd4ac42a473-kube-api-access-k52nk\") pod \"kas-connection-checker-685895c7c9-z75pq\" (UID: \"2050cf2e-0f83-43b3-bb85-7bd4ac42a473\") " pod="kube-system/kas-connection-checker-685895c7c9-z75pq" Oct 31 04:03:53.448589 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.447417 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-registry-certificates\") pod \"image-registry-6f69bcc59c-w96bl\" (UID: \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\") " pod="openshift-image-registry/image-registry-6f69bcc59c-w96bl" Oct 31 04:03:53.448589 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.447434 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/944c9831-b2fa-4f58-bba2-93288793b125-config\") pod \"service-ca-operator-544d749c8c-khdq4\" (UID: \"944c9831-b2fa-4f58-bba2-93288793b125\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-khdq4" Oct 31 04:03:53.448589 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.447450 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/93beb8b8-c9e7-4cce-8baf-8497b971e193-registry-certificates\") pod \"image-registry-7f78cfdfd8-w98g7\" (UID: \"93beb8b8-c9e7-4cce-8baf-8497b971e193\") " pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" Oct 31 04:03:53.448589 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:53.447940 2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 04:03:53.448589 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.447984 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/93beb8b8-c9e7-4cce-8baf-8497b971e193-registry-certificates\") pod \"image-registry-7f78cfdfd8-w98g7\" (UID: \"93beb8b8-c9e7-4cce-8baf-8497b971e193\") " pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" Oct 31 04:03:53.448589 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:53.448007 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/197dee13-748f-4c6d-acc5-1793b5cee80a-samples-operator-tls podName:197dee13-748f-4c6d-acc5-1793b5cee80a nodeName:}" failed. No retries permitted until 2026-10-31 04:03:53.947988822 +0000 UTC m=+31.047826702 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/197dee13-748f-4c6d-acc5-1793b5cee80a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-c64cl" (UID: "197dee13-748f-4c6d-acc5-1793b5cee80a") : secret "samples-operator-tls" not found Oct 31 04:03:53.448589 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:53.448009 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 04:03:53.448589 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:53.448027 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f78cfdfd8-w98g7: secret "image-registry-tls" not found Oct 31 04:03:53.448589 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:53.448084 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93beb8b8-c9e7-4cce-8baf-8497b971e193-registry-tls podName:93beb8b8-c9e7-4cce-8baf-8497b971e193 nodeName:}" failed. No retries permitted until 2026-10-31 04:03:53.94806839 +0000 UTC m=+31.047906258 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/93beb8b8-c9e7-4cce-8baf-8497b971e193-registry-tls") pod "image-registry-7f78cfdfd8-w98g7" (UID: "93beb8b8-c9e7-4cce-8baf-8497b971e193") : secret "image-registry-tls" not found Oct 31 04:03:53.449281 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:53.448682 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 04:03:53.449281 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:53.448695 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f69bcc59c-w96bl: secret "image-registry-tls" not found Oct 31 04:03:53.449281 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:53.448749 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-registry-tls podName:e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22 nodeName:}" failed. No retries permitted until 2026-10-31 04:03:53.948733999 +0000 UTC m=+31.048571880 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-registry-tls") pod "image-registry-6f69bcc59c-w96bl" (UID: "e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22") : secret "image-registry-tls" not found Oct 31 04:03:53.449281 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.449269 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-registry-certificates\") pod \"image-registry-6f69bcc59c-w96bl\" (UID: \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\") " pod="openshift-image-registry/image-registry-6f69bcc59c-w96bl" Oct 31 04:03:53.449673 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.449642 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/944c9831-b2fa-4f58-bba2-93288793b125-config\") pod \"service-ca-operator-544d749c8c-khdq4\" (UID: \"944c9831-b2fa-4f58-bba2-93288793b125\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-khdq4" Oct 31 04:03:53.450632 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:53.449716 2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 04:03:53.450632 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:53.449757 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a173790b-6c13-45a1-a361-3fe51844b4c2-cluster-monitoring-operator-tls podName:a173790b-6c13-45a1-a361-3fe51844b4c2 nodeName:}" failed. No retries permitted until 2026-10-31 04:03:53.949747485 +0000 UTC m=+31.049585352 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a173790b-6c13-45a1-a361-3fe51844b4c2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-zqvkx" (UID: "a173790b-6c13-45a1-a361-3fe51844b4c2") : secret "cluster-monitoring-operator-tls" not found Oct 31 04:03:53.450632 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:53.449798 2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 04:03:53.450632 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:53.449830 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/183f4178-99c0-4945-b490-c71cbd2780f3-networking-console-plugin-cert podName:183f4178-99c0-4945-b490-c71cbd2780f3 nodeName:}" failed. No retries permitted until 2026-10-31 04:03:53.949819213 +0000 UTC m=+31.049657080 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/183f4178-99c0-4945-b490-c71cbd2780f3-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-pnwds" (UID: "183f4178-99c0-4945-b490-c71cbd2780f3") : secret "networking-console-plugin-cert" not found Oct 31 04:03:53.450632 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.450386 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/197dee13-748f-4c6d-acc5-1793b5cee80a-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-c64cl\" (UID: \"197dee13-748f-4c6d-acc5-1793b5cee80a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c64cl" Oct 31 04:03:53.450941 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.450677 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-ca-trust-extracted\") pod \"image-registry-6f69bcc59c-w96bl\" (UID: \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\") " pod="openshift-image-registry/image-registry-6f69bcc59c-w96bl" Oct 31 04:03:53.451780 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.451750 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/93beb8b8-c9e7-4cce-8baf-8497b971e193-ca-trust-extracted\") pod \"image-registry-7f78cfdfd8-w98g7\" (UID: \"93beb8b8-c9e7-4cce-8baf-8497b971e193\") " pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" Oct 31 04:03:53.451878 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.451844 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-installation-pull-secrets\") pod \"image-registry-6f69bcc59c-w96bl\" (UID: \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\") " pod="openshift-image-registry/image-registry-6f69bcc59c-w96bl" Oct 31 04:03:53.451944 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.451760 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3945ff65-e877-4372-9334-3968c68b7295-serving-cert\") pod \"console-operator-7cd56b8dbc-s88kp\" (UID: \"3945ff65-e877-4372-9334-3968c68b7295\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-s88kp" Oct 31 04:03:53.452262 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.452238 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/183f4178-99c0-4945-b490-c71cbd2780f3-nginx-conf\") pod \"networking-console-plugin-86c478bb88-pnwds\" (UID: \"183f4178-99c0-4945-b490-c71cbd2780f3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pnwds" Oct 31 04:03:53.452469 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.452264 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c3c90e6f-df5e-4c17-bec4-554280a8e8e7-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bt9c9\" (UID: \"c3c90e6f-df5e-4c17-bec4-554280a8e8e7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bt9c9" Oct 31 04:03:53.452958 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.452932 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e667439c-7121-4c5f-84fe-b36f2391dc66-service-ca-bundle\") pod \"insights-operator-7c4986d449-ld96l\" (UID: \"e667439c-7121-4c5f-84fe-b36f2391dc66\") " pod="openshift-insights/insights-operator-7c4986d449-ld96l" Oct 31 04:03:53.454277 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.453461 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d22707c7-21fa-4511-b0b1-869b41fb2351-stats-auth\") pod \"router-default-678d747c5f-bgj92\" (UID: \"d22707c7-21fa-4511-b0b1-869b41fb2351\") " pod="openshift-ingress/router-default-678d747c5f-bgj92" Oct 31 04:03:53.454277 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.453508 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e667439c-7121-4c5f-84fe-b36f2391dc66-tmp\") pod \"insights-operator-7c4986d449-ld96l\" (UID: \"e667439c-7121-4c5f-84fe-b36f2391dc66\") " pod="openshift-insights/insights-operator-7c4986d449-ld96l" Oct 31 04:03:53.454277 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.454054 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/93beb8b8-c9e7-4cce-8baf-8497b971e193-image-registry-private-configuration\") pod \"image-registry-7f78cfdfd8-w98g7\" (UID: \"93beb8b8-c9e7-4cce-8baf-8497b971e193\") " pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" Oct 31 04:03:53.454277 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.454094 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a173790b-6c13-45a1-a361-3fe51844b4c2-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-zqvkx\" (UID: \"a173790b-6c13-45a1-a361-3fe51844b4c2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zqvkx" Oct 31 04:03:53.454554 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.454432 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-image-registry-private-configuration\") pod \"image-registry-6f69bcc59c-w96bl\" (UID: \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\") " pod="openshift-image-registry/image-registry-6f69bcc59c-w96bl" Oct 31 04:03:53.456558 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.454989 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3945ff65-e877-4372-9334-3968c68b7295-config\") pod \"console-operator-7cd56b8dbc-s88kp\" (UID: \"3945ff65-e877-4372-9334-3968c68b7295\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-s88kp" Oct 31 04:03:53.456558 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.455195 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-trusted-ca\") pod \"image-registry-6f69bcc59c-w96bl\" (UID: \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\") " pod="openshift-image-registry/image-registry-6f69bcc59c-w96bl" Oct 31 04:03:53.456558 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.455194 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d22707c7-21fa-4511-b0b1-869b41fb2351-default-certificate\") pod \"router-default-678d747c5f-bgj92\" (UID: \"d22707c7-21fa-4511-b0b1-869b41fb2351\") " pod="openshift-ingress/router-default-678d747c5f-bgj92" Oct 31 04:03:53.456558 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.455348 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e667439c-7121-4c5f-84fe-b36f2391dc66-snapshots\") pod \"insights-operator-7c4986d449-ld96l\" (UID: \"e667439c-7121-4c5f-84fe-b36f2391dc66\") " pod="openshift-insights/insights-operator-7c4986d449-ld96l" Oct 31 04:03:53.456558 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:53.455501 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d22707c7-21fa-4511-b0b1-869b41fb2351-service-ca-bundle podName:d22707c7-21fa-4511-b0b1-869b41fb2351 nodeName:}" failed. No retries permitted until 2026-10-31 04:03:53.955472861 +0000 UTC m=+31.055310745 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d22707c7-21fa-4511-b0b1-869b41fb2351-service-ca-bundle") pod "router-default-678d747c5f-bgj92" (UID: "d22707c7-21fa-4511-b0b1-869b41fb2351") : configmap references non-existent config key: service-ca.crt Oct 31 04:03:53.456558 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.455822 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e667439c-7121-4c5f-84fe-b36f2391dc66-serving-cert\") pod \"insights-operator-7c4986d449-ld96l\" (UID: \"e667439c-7121-4c5f-84fe-b36f2391dc66\") " pod="openshift-insights/insights-operator-7c4986d449-ld96l" Oct 31 04:03:53.456558 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.455850 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/93beb8b8-c9e7-4cce-8baf-8497b971e193-trusted-ca\") pod \"image-registry-7f78cfdfd8-w98g7\" (UID: \"93beb8b8-c9e7-4cce-8baf-8497b971e193\") " pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" Oct 31 04:03:53.456558 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.456506 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e667439c-7121-4c5f-84fe-b36f2391dc66-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-ld96l\" (UID: \"e667439c-7121-4c5f-84fe-b36f2391dc66\") " pod="openshift-insights/insights-operator-7c4986d449-ld96l" Oct 31 04:03:53.457034 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.456630 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/93beb8b8-c9e7-4cce-8baf-8497b971e193-installation-pull-secrets\") pod \"image-registry-7f78cfdfd8-w98g7\" (UID: \"93beb8b8-c9e7-4cce-8baf-8497b971e193\") " pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" Oct 31 04:03:53.457661 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.457638 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/944c9831-b2fa-4f58-bba2-93288793b125-serving-cert\") pod \"service-ca-operator-544d749c8c-khdq4\" (UID: \"944c9831-b2fa-4f58-bba2-93288793b125\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-khdq4" Oct 31 04:03:53.458062 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.458042 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c3c90e6f-df5e-4c17-bec4-554280a8e8e7-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bt9c9\" (UID: \"c3c90e6f-df5e-4c17-bec4-554280a8e8e7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bt9c9" Oct 31 04:03:53.474300 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.474265 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj5t7\" (UniqueName: \"kubernetes.io/projected/57b47c7b-9c84-4888-be69-64afca28a8c0-kube-api-access-jj5t7\") pod \"kas-connection-checker-685895c7c9-xqd9k\" (UID: \"57b47c7b-9c84-4888-be69-64afca28a8c0\") " pod="kube-system/kas-connection-checker-685895c7c9-xqd9k" Oct 31 04:03:53.479595 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.479562 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cq9b\" (UniqueName: \"kubernetes.io/projected/197dee13-748f-4c6d-acc5-1793b5cee80a-kube-api-access-7cq9b\") pod \"cluster-samples-operator-6ddb6c8b94-c64cl\" (UID: \"197dee13-748f-4c6d-acc5-1793b5cee80a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c64cl" Oct 31 04:03:53.479822 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.479796 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/93beb8b8-c9e7-4cce-8baf-8497b971e193-bound-sa-token\") pod \"image-registry-7f78cfdfd8-w98g7\" (UID: \"93beb8b8-c9e7-4cce-8baf-8497b971e193\") " pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" Oct 31 04:03:53.480881 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.480854 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggq72\" (UniqueName: \"kubernetes.io/projected/a173790b-6c13-45a1-a361-3fe51844b4c2-kube-api-access-ggq72\") pod \"cluster-monitoring-operator-884576c57-zqvkx\" (UID: \"a173790b-6c13-45a1-a361-3fe51844b4c2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zqvkx" Oct 31 04:03:53.480881 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.480874 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc2vq\" (UniqueName: \"kubernetes.io/projected/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-kube-api-access-xc2vq\") pod \"image-registry-6f69bcc59c-w96bl\" (UID: \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\") " pod="openshift-image-registry/image-registry-6f69bcc59c-w96bl" Oct 31 04:03:53.481071 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.480854 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxlpp\" (UniqueName: \"kubernetes.io/projected/c4a091a2-58d2-4b05-a968-d16fe1bd9d40-kube-api-access-nxlpp\") pod \"volume-data-source-validator-58c9f5f46d-hntxx\" (UID: \"c4a091a2-58d2-4b05-a968-d16fe1bd9d40\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hntxx" Oct 31 04:03:53.481071 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.480977 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mtpj\" (UniqueName: \"kubernetes.io/projected/c3c90e6f-df5e-4c17-bec4-554280a8e8e7-kube-api-access-7mtpj\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bt9c9\" (UID: \"c3c90e6f-df5e-4c17-bec4-554280a8e8e7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bt9c9" Oct 31 04:03:53.481227 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.481208 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-bound-sa-token\") pod \"image-registry-6f69bcc59c-w96bl\" (UID: \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\") " pod="openshift-image-registry/image-registry-6f69bcc59c-w96bl" Oct 31 04:03:53.481518 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.481493 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqx72\" (UniqueName: \"kubernetes.io/projected/3945ff65-e877-4372-9334-3968c68b7295-kube-api-access-fqx72\") pod \"console-operator-7cd56b8dbc-s88kp\" (UID: \"3945ff65-e877-4372-9334-3968c68b7295\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-s88kp" Oct 31 04:03:53.482448 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.482414 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k52nk\" (UniqueName: \"kubernetes.io/projected/2050cf2e-0f83-43b3-bb85-7bd4ac42a473-kube-api-access-k52nk\") pod \"kas-connection-checker-685895c7c9-z75pq\" (UID: \"2050cf2e-0f83-43b3-bb85-7bd4ac42a473\") " pod="kube-system/kas-connection-checker-685895c7c9-z75pq" Oct 31 04:03:53.483624 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.483600 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlg9n\" (UniqueName: \"kubernetes.io/projected/b541554a-0460-4716-9f14-f7a46567465c-kube-api-access-qlg9n\") pod \"kas-connection-checker-685895c7c9-xgjnm\" (UID: \"b541554a-0460-4716-9f14-f7a46567465c\") " pod="kube-system/kas-connection-checker-685895c7c9-xgjnm" Oct 31 04:03:53.483785 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.483769 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wddq5\" (UniqueName: \"kubernetes.io/projected/e667439c-7121-4c5f-84fe-b36f2391dc66-kube-api-access-wddq5\") pod \"insights-operator-7c4986d449-ld96l\" (UID: \"e667439c-7121-4c5f-84fe-b36f2391dc66\") " pod="openshift-insights/insights-operator-7c4986d449-ld96l" Oct 31 04:03:53.484308 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.484286 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghxbr\" (UniqueName: \"kubernetes.io/projected/944c9831-b2fa-4f58-bba2-93288793b125-kube-api-access-ghxbr\") pod \"service-ca-operator-544d749c8c-khdq4\" (UID: \"944c9831-b2fa-4f58-bba2-93288793b125\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-khdq4" Oct 31 04:03:53.484435 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.484418 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25x8r\" (UniqueName: \"kubernetes.io/projected/93beb8b8-c9e7-4cce-8baf-8497b971e193-kube-api-access-25x8r\") pod \"image-registry-7f78cfdfd8-w98g7\" (UID: \"93beb8b8-c9e7-4cce-8baf-8497b971e193\") " pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" Oct 31 04:03:53.488264 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.488243 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6xnt\" (UniqueName: \"kubernetes.io/projected/d22707c7-21fa-4511-b0b1-869b41fb2351-kube-api-access-v6xnt\") pod \"router-default-678d747c5f-bgj92\" (UID: \"d22707c7-21fa-4511-b0b1-869b41fb2351\") " pod="openshift-ingress/router-default-678d747c5f-bgj92" Oct 31 04:03:53.545464 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.545420 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-z75pq" Oct 31 04:03:53.548795 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.548757 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ll7m\" (UniqueName: \"kubernetes.io/projected/a2ce92f3-5b05-4c89-9d53-a57bc85b17ef-kube-api-access-7ll7m\") pod \"ingress-canary-vqsgm\" (UID: \"a2ce92f3-5b05-4c89-9d53-a57bc85b17ef\") " pod="openshift-ingress-canary/ingress-canary-vqsgm" Oct 31 04:03:53.548949 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.548832 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2ce92f3-5b05-4c89-9d53-a57bc85b17ef-cert\") pod \"ingress-canary-vqsgm\" (UID: \"a2ce92f3-5b05-4c89-9d53-a57bc85b17ef\") " pod="openshift-ingress-canary/ingress-canary-vqsgm" Oct 31 04:03:53.548949 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.548871 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/48309c9d-5f34-49bf-b55b-f74959984e0a-config-volume\") pod \"dns-default-zjpdv\" (UID: \"48309c9d-5f34-49bf-b55b-f74959984e0a\") " pod="openshift-dns/dns-default-zjpdv" Oct 31 04:03:53.548949 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.548897 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48309c9d-5f34-49bf-b55b-f74959984e0a-tmp-dir\") pod \"dns-default-zjpdv\" (UID: \"48309c9d-5f34-49bf-b55b-f74959984e0a\") " pod="openshift-dns/dns-default-zjpdv" Oct 31 04:03:53.548949 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.548934 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsrbd\" (UniqueName: \"kubernetes.io/projected/fb656990-e663-4fea-8733-18ded9222a79-kube-api-access-hsrbd\") pod \"network-check-source-6f78d59f99-rf8vj\" (UID: \"fb656990-e663-4fea-8733-18ded9222a79\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rf8vj" Oct 31 04:03:53.549171 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.548979 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt6hk\" (UniqueName: \"kubernetes.io/projected/48309c9d-5f34-49bf-b55b-f74959984e0a-kube-api-access-kt6hk\") pod \"dns-default-zjpdv\" (UID: \"48309c9d-5f34-49bf-b55b-f74959984e0a\") " pod="openshift-dns/dns-default-zjpdv" Oct 31 04:03:53.549171 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.549050 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/48309c9d-5f34-49bf-b55b-f74959984e0a-metrics-tls\") pod \"dns-default-zjpdv\" (UID: \"48309c9d-5f34-49bf-b55b-f74959984e0a\") " pod="openshift-dns/dns-default-zjpdv" Oct 31 04:03:53.549272 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:53.549201 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 04:03:53.549272 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.549247 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48309c9d-5f34-49bf-b55b-f74959984e0a-tmp-dir\") pod \"dns-default-zjpdv\" (UID: \"48309c9d-5f34-49bf-b55b-f74959984e0a\") " pod="openshift-dns/dns-default-zjpdv" Oct 31 04:03:53.549359 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:53.549255 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48309c9d-5f34-49bf-b55b-f74959984e0a-metrics-tls podName:48309c9d-5f34-49bf-b55b-f74959984e0a nodeName:}" failed. No retries permitted until 2026-10-31 04:03:54.049237405 +0000 UTC m=+31.149075275 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/48309c9d-5f34-49bf-b55b-f74959984e0a-metrics-tls") pod "dns-default-zjpdv" (UID: "48309c9d-5f34-49bf-b55b-f74959984e0a") : secret "dns-default-metrics-tls" not found Oct 31 04:03:53.549511 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.549490 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/48309c9d-5f34-49bf-b55b-f74959984e0a-config-volume\") pod \"dns-default-zjpdv\" (UID: \"48309c9d-5f34-49bf-b55b-f74959984e0a\") " pod="openshift-dns/dns-default-zjpdv" Oct 31 04:03:53.550366 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.550345 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-xgjnm" Oct 31 04:03:53.551304 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.551287 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2ce92f3-5b05-4c89-9d53-a57bc85b17ef-cert\") pod \"ingress-canary-vqsgm\" (UID: \"a2ce92f3-5b05-4c89-9d53-a57bc85b17ef\") " pod="openshift-ingress-canary/ingress-canary-vqsgm" Oct 31 04:03:53.562940 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.562720 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-xqd9k" Oct 31 04:03:53.574123 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.574095 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ll7m\" (UniqueName: \"kubernetes.io/projected/a2ce92f3-5b05-4c89-9d53-a57bc85b17ef-kube-api-access-7ll7m\") pod \"ingress-canary-vqsgm\" (UID: \"a2ce92f3-5b05-4c89-9d53-a57bc85b17ef\") " pod="openshift-ingress-canary/ingress-canary-vqsgm" Oct 31 04:03:53.574226 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.574146 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt6hk\" (UniqueName: \"kubernetes.io/projected/48309c9d-5f34-49bf-b55b-f74959984e0a-kube-api-access-kt6hk\") pod \"dns-default-zjpdv\" (UID: \"48309c9d-5f34-49bf-b55b-f74959984e0a\") " pod="openshift-dns/dns-default-zjpdv" Oct 31 04:03:53.574346 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.574313 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsrbd\" (UniqueName: \"kubernetes.io/projected/fb656990-e663-4fea-8733-18ded9222a79-kube-api-access-hsrbd\") pod \"network-check-source-6f78d59f99-rf8vj\" (UID: \"fb656990-e663-4fea-8733-18ded9222a79\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rf8vj" Oct 31 04:03:53.612960 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.612253 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hntxx" Oct 31 04:03:53.620726 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.619105 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-ld96l" Oct 31 04:03:53.637603 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.637566 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-s88kp" Oct 31 04:03:53.648640 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.648208 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-khdq4" Oct 31 04:03:53.655190 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.652983 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bt9c9" Oct 31 04:03:53.668774 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.667604 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rf8vj" Oct 31 04:03:53.684359 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.680823 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vqsgm" Oct 31 04:03:53.872004 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.871977 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-z75pq"] Oct 31 04:03:53.873531 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:53.873502 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb541554a_0460_4716_9f14_f7a46567465c.slice/crio-b238274377c0617b219aa1379467d63523e3e1d10047202c1b0e8e66268b84e9 WatchSource:0}: Error finding container b238274377c0617b219aa1379467d63523e3e1d10047202c1b0e8e66268b84e9: Status 404 returned error can't find the container with id b238274377c0617b219aa1379467d63523e3e1d10047202c1b0e8e66268b84e9 Oct 31 04:03:53.873736 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.873717 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xgjnm"] Oct 31 04:03:53.874184 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:53.874164 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2050cf2e_0f83_43b3_bb85_7bd4ac42a473.slice/crio-2df9b248146c1ec79d4800aadd14a9443d89d935854efcaaaf398ce7d67e4e2b WatchSource:0}: Error finding container 2df9b248146c1ec79d4800aadd14a9443d89d935854efcaaaf398ce7d67e4e2b: Status 404 returned error can't find the container with id 2df9b248146c1ec79d4800aadd14a9443d89d935854efcaaaf398ce7d67e4e2b Oct 31 04:03:53.878286 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.878227 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xqd9k"] Oct 31 04:03:53.881598 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:53.881575 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57b47c7b_9c84_4888_be69_64afca28a8c0.slice/crio-a582b14d9cf72583d5e2976716d25832aab83f4ac2ffd21642c4c87ca9dab847 WatchSource:0}: Error finding container a582b14d9cf72583d5e2976716d25832aab83f4ac2ffd21642c4c87ca9dab847: Status 404 returned error can't find the container with id a582b14d9cf72583d5e2976716d25832aab83f4ac2ffd21642c4c87ca9dab847 Oct 31 04:03:53.943580 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.943536 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-ld96l"] Oct 31 04:03:53.957001 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.955986 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/93beb8b8-c9e7-4cce-8baf-8497b971e193-registry-tls\") pod \"image-registry-7f78cfdfd8-w98g7\" (UID: \"93beb8b8-c9e7-4cce-8baf-8497b971e193\") " pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" Oct 31 04:03:53.957001 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.956045 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/197dee13-748f-4c6d-acc5-1793b5cee80a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-c64cl\" (UID: \"197dee13-748f-4c6d-acc5-1793b5cee80a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c64cl" Oct 31 04:03:53.957001 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.956074 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d22707c7-21fa-4511-b0b1-869b41fb2351-service-ca-bundle\") pod \"router-default-678d747c5f-bgj92\" (UID: \"d22707c7-21fa-4511-b0b1-869b41fb2351\") " pod="openshift-ingress/router-default-678d747c5f-bgj92" Oct 31 04:03:53.957001 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.956107 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-registry-tls\") pod \"image-registry-6f69bcc59c-w96bl\" (UID: \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\") " pod="openshift-image-registry/image-registry-6f69bcc59c-w96bl" Oct 31 04:03:53.957001 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.956142 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/183f4178-99c0-4945-b490-c71cbd2780f3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pnwds\" (UID: \"183f4178-99c0-4945-b490-c71cbd2780f3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pnwds" Oct 31 04:03:53.957001 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.956177 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d22707c7-21fa-4511-b0b1-869b41fb2351-metrics-certs\") pod \"router-default-678d747c5f-bgj92\" (UID: \"d22707c7-21fa-4511-b0b1-869b41fb2351\") " pod="openshift-ingress/router-default-678d747c5f-bgj92" Oct 31 04:03:53.957001 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.956206 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a173790b-6c13-45a1-a361-3fe51844b4c2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-zqvkx\" (UID: \"a173790b-6c13-45a1-a361-3fe51844b4c2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zqvkx" Oct 31 04:03:53.957001 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:53.956348 2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 04:03:53.957001 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:53.956405 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a173790b-6c13-45a1-a361-3fe51844b4c2-cluster-monitoring-operator-tls podName:a173790b-6c13-45a1-a361-3fe51844b4c2 nodeName:}" failed. No retries permitted until 2026-10-31 04:03:54.956386591 +0000 UTC m=+32.056224477 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a173790b-6c13-45a1-a361-3fe51844b4c2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-zqvkx" (UID: "a173790b-6c13-45a1-a361-3fe51844b4c2") : secret "cluster-monitoring-operator-tls" not found Oct 31 04:03:53.957001 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:53.956481 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 04:03:53.957001 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:53.956493 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f78cfdfd8-w98g7: secret "image-registry-tls" not found Oct 31 04:03:53.957001 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:53.956525 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93beb8b8-c9e7-4cce-8baf-8497b971e193-registry-tls podName:93beb8b8-c9e7-4cce-8baf-8497b971e193 nodeName:}" failed. No retries permitted until 2026-10-31 04:03:54.956514237 +0000 UTC m=+32.056352104 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/93beb8b8-c9e7-4cce-8baf-8497b971e193-registry-tls") pod "image-registry-7f78cfdfd8-w98g7" (UID: "93beb8b8-c9e7-4cce-8baf-8497b971e193") : secret "image-registry-tls" not found Oct 31 04:03:53.957001 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:53.956586 2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 04:03:53.957001 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:53.956617 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/197dee13-748f-4c6d-acc5-1793b5cee80a-samples-operator-tls podName:197dee13-748f-4c6d-acc5-1793b5cee80a nodeName:}" failed. No retries permitted until 2026-10-31 04:03:54.956607299 +0000 UTC m=+32.056445181 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/197dee13-748f-4c6d-acc5-1793b5cee80a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-c64cl" (UID: "197dee13-748f-4c6d-acc5-1793b5cee80a") : secret "samples-operator-tls" not found Oct 31 04:03:53.957001 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:53.956680 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d22707c7-21fa-4511-b0b1-869b41fb2351-service-ca-bundle podName:d22707c7-21fa-4511-b0b1-869b41fb2351 nodeName:}" failed. No retries permitted until 2026-10-31 04:03:54.956670082 +0000 UTC m=+32.056507958 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d22707c7-21fa-4511-b0b1-869b41fb2351-service-ca-bundle") pod "router-default-678d747c5f-bgj92" (UID: "d22707c7-21fa-4511-b0b1-869b41fb2351") : configmap references non-existent config key: service-ca.crt Oct 31 04:03:53.957001 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:53.956754 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 04:03:53.957867 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:53.956765 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f69bcc59c-w96bl: secret "image-registry-tls" not found Oct 31 04:03:53.957867 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:53.956793 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-registry-tls podName:e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22 nodeName:}" failed. No retries permitted until 2026-10-31 04:03:54.956782962 +0000 UTC m=+32.056620837 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-registry-tls") pod "image-registry-6f69bcc59c-w96bl" (UID: "e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22") : secret "image-registry-tls" not found Oct 31 04:03:53.957867 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:53.956851 2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 04:03:53.957867 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:53.956881 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/183f4178-99c0-4945-b490-c71cbd2780f3-networking-console-plugin-cert podName:183f4178-99c0-4945-b490-c71cbd2780f3 nodeName:}" failed. No retries permitted until 2026-10-31 04:03:54.956871691 +0000 UTC m=+32.056709569 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/183f4178-99c0-4945-b490-c71cbd2780f3-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-pnwds" (UID: "183f4178-99c0-4945-b490-c71cbd2780f3") : secret "networking-console-plugin-cert" not found Oct 31 04:03:53.957867 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:53.956935 2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 04:03:53.957867 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:53.956962 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d22707c7-21fa-4511-b0b1-869b41fb2351-metrics-certs podName:d22707c7-21fa-4511-b0b1-869b41fb2351 nodeName:}" failed. No retries permitted until 2026-10-31 04:03:54.956953243 +0000 UTC m=+32.056791121 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d22707c7-21fa-4511-b0b1-869b41fb2351-metrics-certs") pod "router-default-678d747c5f-bgj92" (UID: "d22707c7-21fa-4511-b0b1-869b41fb2351") : secret "router-metrics-certs-default" not found Oct 31 04:03:53.967602 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.967219 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hntxx"] Oct 31 04:03:53.971438 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:53.970561 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4a091a2_58d2_4b05_a968_d16fe1bd9d40.slice/crio-b470b2151e3fb3301b0be2d05d98b6088a17b10108862e8c5c769f40f4023df2 WatchSource:0}: Error finding container b470b2151e3fb3301b0be2d05d98b6088a17b10108862e8c5c769f40f4023df2: Status 404 returned error can't find the container with id b470b2151e3fb3301b0be2d05d98b6088a17b10108862e8c5c769f40f4023df2 Oct 31 04:03:53.976548 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:53.976520 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-s88kp"] Oct 31 04:03:53.986961 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:53.986927 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3945ff65_e877_4372_9334_3968c68b7295.slice/crio-d7f8a4628c1cc4450b1ba7ae4188a8c9d68202dd4522ab53d6fe4936506ecd6f WatchSource:0}: Error finding container d7f8a4628c1cc4450b1ba7ae4188a8c9d68202dd4522ab53d6fe4936506ecd6f: Status 404 returned error can't find the container with id d7f8a4628c1cc4450b1ba7ae4188a8c9d68202dd4522ab53d6fe4936506ecd6f Oct 31 04:03:54.035013 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:54.034983 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-khdq4"] Oct 31 04:03:54.038217 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:54.038178 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-rf8vj"] Oct 31 04:03:54.038858 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:54.038833 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod944c9831_b2fa_4f58_bba2_93288793b125.slice/crio-12f85ecffbe0e7f649e1d9be59d8f02d7aa3cb47bc0552465a340b6878f0c950 WatchSource:0}: Error finding container 12f85ecffbe0e7f649e1d9be59d8f02d7aa3cb47bc0552465a340b6878f0c950: Status 404 returned error can't find the container with id 12f85ecffbe0e7f649e1d9be59d8f02d7aa3cb47bc0552465a340b6878f0c950 Oct 31 04:03:54.039166 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:54.039146 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bt9c9"] Oct 31 04:03:54.040296 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:54.040276 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3c90e6f_df5e_4c17_bec4_554280a8e8e7.slice/crio-f033db6f8846c03553449c030e1787fcd20f1d39dd5254fbe9b4c9ca77a535f1 WatchSource:0}: Error finding container f033db6f8846c03553449c030e1787fcd20f1d39dd5254fbe9b4c9ca77a535f1: Status 404 returned error can't find the container with id f033db6f8846c03553449c030e1787fcd20f1d39dd5254fbe9b4c9ca77a535f1 Oct 31 04:03:54.041468 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:54.041450 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb656990_e663_4fea_8733_18ded9222a79.slice/crio-071ca9d03626e2d4a643c2e58587db9b4c38cbcd00e223f98624547c9b878714 WatchSource:0}: Error finding container 071ca9d03626e2d4a643c2e58587db9b4c38cbcd00e223f98624547c9b878714: Status 404 returned error can't find the container with id 071ca9d03626e2d4a643c2e58587db9b4c38cbcd00e223f98624547c9b878714 Oct 31 04:03:54.057519 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:54.057431 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/48309c9d-5f34-49bf-b55b-f74959984e0a-metrics-tls\") pod \"dns-default-zjpdv\" (UID: \"48309c9d-5f34-49bf-b55b-f74959984e0a\") " pod="openshift-dns/dns-default-zjpdv" Oct 31 04:03:54.057622 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:54.057602 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 04:03:54.057683 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:54.057672 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48309c9d-5f34-49bf-b55b-f74959984e0a-metrics-tls podName:48309c9d-5f34-49bf-b55b-f74959984e0a nodeName:}" failed. No retries permitted until 2026-10-31 04:03:55.057651978 +0000 UTC m=+32.157489859 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/48309c9d-5f34-49bf-b55b-f74959984e0a-metrics-tls") pod "dns-default-zjpdv" (UID: "48309c9d-5f34-49bf-b55b-f74959984e0a") : secret "dns-default-metrics-tls" not found Oct 31 04:03:54.086259 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:54.086233 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vqsgm"] Oct 31 04:03:54.088617 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:54.088586 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2ce92f3_5b05_4c89_9d53_a57bc85b17ef.slice/crio-eed219fb3e1a90717b9fa9c013bab0e08920a8e517198614557b82a63f6a048d WatchSource:0}: Error finding container eed219fb3e1a90717b9fa9c013bab0e08920a8e517198614557b82a63f6a048d: Status 404 returned error can't find the container with id eed219fb3e1a90717b9fa9c013bab0e08920a8e517198614557b82a63f6a048d Oct 31 04:03:54.400264 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:54.399474 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hnbll" Oct 31 04:03:54.400433 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:54.399474 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4b7d5" Oct 31 04:03:54.400514 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:54.399490 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clnlz" Oct 31 04:03:54.417020 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:54.416993 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 04:03:54.417020 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:54.417002 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lwzcw\"" Oct 31 04:03:54.417020 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:54.417006 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 04:03:54.417225 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:54.417035 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9876q\"" Oct 31 04:03:54.417225 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:54.417051 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-brpbt\"" Oct 31 04:03:54.440773 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:54.440699 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm5cr_89ea384b-705f-47a9-8df6-d3c266ed6761/konnectivity-agent/0.log" limit=1024 Oct 31 04:03:54.624828 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:54.624780 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vqsgm" event={"ID":"a2ce92f3-5b05-4c89-9d53-a57bc85b17ef","Type":"ContainerStarted","Data":"eed219fb3e1a90717b9fa9c013bab0e08920a8e517198614557b82a63f6a048d"} Oct 31 04:03:54.627945 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:54.627906 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hntxx" event={"ID":"c4a091a2-58d2-4b05-a968-d16fe1bd9d40","Type":"ContainerStarted","Data":"b470b2151e3fb3301b0be2d05d98b6088a17b10108862e8c5c769f40f4023df2"} Oct 31 04:03:54.631092 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:54.631052 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bt9c9" event={"ID":"c3c90e6f-df5e-4c17-bec4-554280a8e8e7","Type":"ContainerStarted","Data":"f033db6f8846c03553449c030e1787fcd20f1d39dd5254fbe9b4c9ca77a535f1"} Oct 31 04:03:54.632856 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:54.632779 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-khdq4" event={"ID":"944c9831-b2fa-4f58-bba2-93288793b125","Type":"ContainerStarted","Data":"12f85ecffbe0e7f649e1d9be59d8f02d7aa3cb47bc0552465a340b6878f0c950"} Oct 31 04:03:54.635244 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:54.635163 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-s88kp" event={"ID":"3945ff65-e877-4372-9334-3968c68b7295","Type":"ContainerStarted","Data":"d7f8a4628c1cc4450b1ba7ae4188a8c9d68202dd4522ab53d6fe4936506ecd6f"} Oct 31 04:03:54.638896 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:54.638577 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-xgjnm" event={"ID":"b541554a-0460-4716-9f14-f7a46567465c","Type":"ContainerStarted","Data":"65f0981960066234108485cea26b39cd1b81c68d090d42b2b42c85dbae00bd32"} Oct 31 04:03:54.638896 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:54.638627 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-xgjnm" event={"ID":"b541554a-0460-4716-9f14-f7a46567465c","Type":"ContainerStarted","Data":"b238274377c0617b219aa1379467d63523e3e1d10047202c1b0e8e66268b84e9"} Oct 31 04:03:54.644028 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:54.643927 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-z75pq" event={"ID":"2050cf2e-0f83-43b3-bb85-7bd4ac42a473","Type":"ContainerStarted","Data":"2f811db632123276ba74356941af2f092bce1d61c2feb72decc7f3422b010aca"} Oct 31 04:03:54.644028 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:54.643993 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-z75pq" event={"ID":"2050cf2e-0f83-43b3-bb85-7bd4ac42a473","Type":"ContainerStarted","Data":"2df9b248146c1ec79d4800aadd14a9443d89d935854efcaaaf398ce7d67e4e2b"} Oct 31 04:03:54.645567 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:54.645534 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rf8vj" event={"ID":"fb656990-e663-4fea-8733-18ded9222a79","Type":"ContainerStarted","Data":"071ca9d03626e2d4a643c2e58587db9b4c38cbcd00e223f98624547c9b878714"} Oct 31 04:03:54.647035 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:54.646980 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-ld96l" event={"ID":"e667439c-7121-4c5f-84fe-b36f2391dc66","Type":"ContainerStarted","Data":"dc34e5c3243961a8722af6ca8afcb8a4e8cfbed45618bc22019ed5bb7c849c95"} Oct 31 04:03:54.649105 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:54.649073 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-xqd9k" event={"ID":"57b47c7b-9c84-4888-be69-64afca28a8c0","Type":"ContainerStarted","Data":"1eed3106ef22be8091e8ab72c6902815d28cd73a186406c4a3f390a2ed900843"} Oct 31 04:03:54.649228 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:54.649127 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-xqd9k" event={"ID":"57b47c7b-9c84-4888-be69-64afca28a8c0","Type":"ContainerStarted","Data":"a582b14d9cf72583d5e2976716d25832aab83f4ac2ffd21642c4c87ca9dab847"} Oct 31 04:03:54.688112 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:54.687923 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-xgjnm" podStartSLOduration=393.687902214 podStartE2EDuration="6m33.687902214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:57:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:03:54.685580743 +0000 UTC m=+31.785418629" watchObservedRunningTime="2026-10-31 04:03:54.687902214 +0000 UTC m=+31.787740099" Oct 31 04:03:54.714353 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:54.714297 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm5cr_89ea384b-705f-47a9-8df6-d3c266ed6761/konnectivity-agent/0.log" limit=1024 Oct 31 04:03:54.729195 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:54.729119 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-xqd9k" podStartSLOduration=393.729098026 podStartE2EDuration="6m33.729098026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:57:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:03:54.727824822 +0000 UTC m=+31.827662710" watchObservedRunningTime="2026-10-31 04:03:54.729098026 +0000 UTC m=+31.828935909" Oct 31 04:03:54.774768 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:54.774707 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-z75pq" podStartSLOduration=393.774686807 podStartE2EDuration="6m33.774686807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:57:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:03:54.773975843 +0000 UTC m=+31.873813758" watchObservedRunningTime="2026-10-31 04:03:54.774686807 +0000 UTC m=+31.874524693" Oct 31 04:03:54.966925 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:54.966879 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/93beb8b8-c9e7-4cce-8baf-8497b971e193-registry-tls\") pod \"image-registry-7f78cfdfd8-w98g7\" (UID: \"93beb8b8-c9e7-4cce-8baf-8497b971e193\") " pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" Oct 31 04:03:54.967106 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:54.966959 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/197dee13-748f-4c6d-acc5-1793b5cee80a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-c64cl\" (UID: \"197dee13-748f-4c6d-acc5-1793b5cee80a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c64cl" Oct 31 04:03:54.967106 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:54.966992 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d22707c7-21fa-4511-b0b1-869b41fb2351-service-ca-bundle\") pod \"router-default-678d747c5f-bgj92\" (UID: \"d22707c7-21fa-4511-b0b1-869b41fb2351\") " pod="openshift-ingress/router-default-678d747c5f-bgj92" Oct 31 04:03:54.967106 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:54.967030 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-registry-tls\") pod \"image-registry-6f69bcc59c-w96bl\" (UID: \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\") " pod="openshift-image-registry/image-registry-6f69bcc59c-w96bl" Oct 31 04:03:54.967106 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:54.967067 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/183f4178-99c0-4945-b490-c71cbd2780f3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pnwds\" (UID: \"183f4178-99c0-4945-b490-c71cbd2780f3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pnwds" Oct 31 04:03:54.967234 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:54.967172 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d22707c7-21fa-4511-b0b1-869b41fb2351-metrics-certs\") pod \"router-default-678d747c5f-bgj92\" (UID: \"d22707c7-21fa-4511-b0b1-869b41fb2351\") " pod="openshift-ingress/router-default-678d747c5f-bgj92" Oct 31 04:03:54.967234 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:54.967206 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a173790b-6c13-45a1-a361-3fe51844b4c2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-zqvkx\" (UID: \"a173790b-6c13-45a1-a361-3fe51844b4c2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zqvkx" Oct 31 04:03:54.967451 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:54.967433 2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 04:03:54.967547 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:54.967520 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a173790b-6c13-45a1-a361-3fe51844b4c2-cluster-monitoring-operator-tls podName:a173790b-6c13-45a1-a361-3fe51844b4c2 nodeName:}" failed. No retries permitted until 2026-10-31 04:03:56.967487778 +0000 UTC m=+34.067325646 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a173790b-6c13-45a1-a361-3fe51844b4c2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-zqvkx" (UID: "a173790b-6c13-45a1-a361-3fe51844b4c2") : secret "cluster-monitoring-operator-tls" not found Oct 31 04:03:54.967969 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:54.967951 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 04:03:54.968027 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:54.967974 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f78cfdfd8-w98g7: secret "image-registry-tls" not found Oct 31 04:03:54.968027 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:54.968025 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93beb8b8-c9e7-4cce-8baf-8497b971e193-registry-tls podName:93beb8b8-c9e7-4cce-8baf-8497b971e193 nodeName:}" failed. No retries permitted until 2026-10-31 04:03:56.968009705 +0000 UTC m=+34.067847572 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/93beb8b8-c9e7-4cce-8baf-8497b971e193-registry-tls") pod "image-registry-7f78cfdfd8-w98g7" (UID: "93beb8b8-c9e7-4cce-8baf-8497b971e193") : secret "image-registry-tls" not found Oct 31 04:03:54.968137 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:54.968092 2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 04:03:54.968137 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:54.968127 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/197dee13-748f-4c6d-acc5-1793b5cee80a-samples-operator-tls podName:197dee13-748f-4c6d-acc5-1793b5cee80a nodeName:}" failed. No retries permitted until 2026-10-31 04:03:56.968116535 +0000 UTC m=+34.067954403 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/197dee13-748f-4c6d-acc5-1793b5cee80a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-c64cl" (UID: "197dee13-748f-4c6d-acc5-1793b5cee80a") : secret "samples-operator-tls" not found Oct 31 04:03:54.968234 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:54.968191 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d22707c7-21fa-4511-b0b1-869b41fb2351-service-ca-bundle podName:d22707c7-21fa-4511-b0b1-869b41fb2351 nodeName:}" failed. No retries permitted until 2026-10-31 04:03:56.96818095 +0000 UTC m=+34.068018819 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d22707c7-21fa-4511-b0b1-869b41fb2351-service-ca-bundle") pod "router-default-678d747c5f-bgj92" (UID: "d22707c7-21fa-4511-b0b1-869b41fb2351") : configmap references non-existent config key: service-ca.crt Oct 31 04:03:54.968292 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:54.968252 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 04:03:54.968292 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:54.968261 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f69bcc59c-w96bl: secret "image-registry-tls" not found Oct 31 04:03:54.968292 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:54.968292 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-registry-tls podName:e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22 nodeName:}" failed. No retries permitted until 2026-10-31 04:03:56.968281167 +0000 UTC m=+34.068119031 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-registry-tls") pod "image-registry-6f69bcc59c-w96bl" (UID: "e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22") : secret "image-registry-tls" not found Oct 31 04:03:54.968447 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:54.968365 2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 04:03:54.968447 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:54.968398 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/183f4178-99c0-4945-b490-c71cbd2780f3-networking-console-plugin-cert podName:183f4178-99c0-4945-b490-c71cbd2780f3 nodeName:}" failed. No retries permitted until 2026-10-31 04:03:56.968388378 +0000 UTC m=+34.068226247 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/183f4178-99c0-4945-b490-c71cbd2780f3-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-pnwds" (UID: "183f4178-99c0-4945-b490-c71cbd2780f3") : secret "networking-console-plugin-cert" not found Oct 31 04:03:54.968545 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:54.968456 2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 04:03:54.968545 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:54.968484 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d22707c7-21fa-4511-b0b1-869b41fb2351-metrics-certs podName:d22707c7-21fa-4511-b0b1-869b41fb2351 nodeName:}" failed. No retries permitted until 2026-10-31 04:03:56.9684755 +0000 UTC m=+34.068313366 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d22707c7-21fa-4511-b0b1-869b41fb2351-metrics-certs") pod "router-default-678d747c5f-bgj92" (UID: "d22707c7-21fa-4511-b0b1-869b41fb2351") : secret "router-metrics-certs-default" not found Oct 31 04:03:55.068764 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:55.068723 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/48309c9d-5f34-49bf-b55b-f74959984e0a-metrics-tls\") pod \"dns-default-zjpdv\" (UID: \"48309c9d-5f34-49bf-b55b-f74959984e0a\") " pod="openshift-dns/dns-default-zjpdv" Oct 31 04:03:55.068950 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:55.068897 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 04:03:55.069194 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:55.069084 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48309c9d-5f34-49bf-b55b-f74959984e0a-metrics-tls podName:48309c9d-5f34-49bf-b55b-f74959984e0a nodeName:}" failed. No retries permitted until 2026-10-31 04:03:57.069030861 +0000 UTC m=+34.168868762 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/48309c9d-5f34-49bf-b55b-f74959984e0a-metrics-tls") pod "dns-default-zjpdv" (UID: "48309c9d-5f34-49bf-b55b-f74959984e0a") : secret "dns-default-metrics-tls" not found Oct 31 04:03:56.395213 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:56.395158 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm5cr_89ea384b-705f-47a9-8df6-d3c266ed6761/konnectivity-agent/0.log" limit=1024 Oct 31 04:03:56.667828 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:56.667786 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hntxx" event={"ID":"c4a091a2-58d2-4b05-a968-d16fe1bd9d40","Type":"ContainerStarted","Data":"420dc298df5bb190c0376798723d2b367eaa9ede67c7f1f77a7fc0661c8a5544"} Oct 31 04:03:56.712530 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:56.711847 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hntxx" podStartSLOduration=361.367456324 podStartE2EDuration="6m3.711832089s" totalImagesPullingTime="2.344375765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:57:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:03:56.711055954 +0000 UTC m=+33.810893841" watchObservedRunningTime="2026-10-31 04:03:56.711832089 +0000 UTC m=+33.811669974" Oct 31 04:03:56.914131 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:56.914040 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm5cr_89ea384b-705f-47a9-8df6-d3c266ed6761/konnectivity-agent/0.log" limit=1024 Oct 31 04:03:56.989584 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:56.989534 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/93beb8b8-c9e7-4cce-8baf-8497b971e193-registry-tls\") pod \"image-registry-7f78cfdfd8-w98g7\" (UID: \"93beb8b8-c9e7-4cce-8baf-8497b971e193\") " pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" Oct 31 04:03:56.989760 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:56.989606 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/197dee13-748f-4c6d-acc5-1793b5cee80a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-c64cl\" (UID: \"197dee13-748f-4c6d-acc5-1793b5cee80a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c64cl" Oct 31 04:03:56.989760 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:56.989650 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d22707c7-21fa-4511-b0b1-869b41fb2351-service-ca-bundle\") pod \"router-default-678d747c5f-bgj92\" (UID: \"d22707c7-21fa-4511-b0b1-869b41fb2351\") " pod="openshift-ingress/router-default-678d747c5f-bgj92" Oct 31 04:03:56.989760 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:56.989688 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-registry-tls\") pod \"image-registry-6f69bcc59c-w96bl\" (UID: \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\") " pod="openshift-image-registry/image-registry-6f69bcc59c-w96bl" Oct 31 04:03:56.989760 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:56.989727 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/183f4178-99c0-4945-b490-c71cbd2780f3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pnwds\" (UID: \"183f4178-99c0-4945-b490-c71cbd2780f3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pnwds" Oct 31 04:03:56.989945 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:56.989765 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d22707c7-21fa-4511-b0b1-869b41fb2351-metrics-certs\") pod \"router-default-678d747c5f-bgj92\" (UID: \"d22707c7-21fa-4511-b0b1-869b41fb2351\") " pod="openshift-ingress/router-default-678d747c5f-bgj92" Oct 31 04:03:56.989945 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:56.989795 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a173790b-6c13-45a1-a361-3fe51844b4c2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-zqvkx\" (UID: \"a173790b-6c13-45a1-a361-3fe51844b4c2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zqvkx" Oct 31 04:03:56.990038 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:56.989946 2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 04:03:56.990038 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:56.990014 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a173790b-6c13-45a1-a361-3fe51844b4c2-cluster-monitoring-operator-tls podName:a173790b-6c13-45a1-a361-3fe51844b4c2 nodeName:}" failed. No retries permitted until 2026-10-31 04:04:00.989995677 +0000 UTC m=+38.089833545 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a173790b-6c13-45a1-a361-3fe51844b4c2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-zqvkx" (UID: "a173790b-6c13-45a1-a361-3fe51844b4c2") : secret "cluster-monitoring-operator-tls" not found Oct 31 04:03:56.990479 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:56.990457 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 04:03:56.990479 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:56.990477 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f78cfdfd8-w98g7: secret "image-registry-tls" not found Oct 31 04:03:56.990603 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:56.990529 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93beb8b8-c9e7-4cce-8baf-8497b971e193-registry-tls podName:93beb8b8-c9e7-4cce-8baf-8497b971e193 nodeName:}" failed. No retries permitted until 2026-10-31 04:04:00.990504161 +0000 UTC m=+38.090342024 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/93beb8b8-c9e7-4cce-8baf-8497b971e193-registry-tls") pod "image-registry-7f78cfdfd8-w98g7" (UID: "93beb8b8-c9e7-4cce-8baf-8497b971e193") : secret "image-registry-tls" not found Oct 31 04:03:56.990603 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:56.990582 2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 04:03:56.990696 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:56.990606 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/197dee13-748f-4c6d-acc5-1793b5cee80a-samples-operator-tls podName:197dee13-748f-4c6d-acc5-1793b5cee80a nodeName:}" failed. No retries permitted until 2026-10-31 04:04:00.990599345 +0000 UTC m=+38.090437208 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/197dee13-748f-4c6d-acc5-1793b5cee80a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-c64cl" (UID: "197dee13-748f-4c6d-acc5-1793b5cee80a") : secret "samples-operator-tls" not found Oct 31 04:03:56.990696 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:56.990649 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d22707c7-21fa-4511-b0b1-869b41fb2351-service-ca-bundle podName:d22707c7-21fa-4511-b0b1-869b41fb2351 nodeName:}" failed. No retries permitted until 2026-10-31 04:04:00.990643518 +0000 UTC m=+38.090481381 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d22707c7-21fa-4511-b0b1-869b41fb2351-service-ca-bundle") pod "router-default-678d747c5f-bgj92" (UID: "d22707c7-21fa-4511-b0b1-869b41fb2351") : configmap references non-existent config key: service-ca.crt Oct 31 04:03:56.990696 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:56.990684 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 04:03:56.990696 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:56.990692 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f69bcc59c-w96bl: secret "image-registry-tls" not found Oct 31 04:03:56.990841 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:56.990708 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-registry-tls podName:e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22 nodeName:}" failed. No retries permitted until 2026-10-31 04:04:00.990702968 +0000 UTC m=+38.090540831 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-registry-tls") pod "image-registry-6f69bcc59c-w96bl" (UID: "e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22") : secret "image-registry-tls" not found Oct 31 04:03:56.990841 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:56.990743 2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 04:03:56.990841 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:56.990765 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/183f4178-99c0-4945-b490-c71cbd2780f3-networking-console-plugin-cert podName:183f4178-99c0-4945-b490-c71cbd2780f3 nodeName:}" failed. No retries permitted until 2026-10-31 04:04:00.990757543 +0000 UTC m=+38.090595405 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/183f4178-99c0-4945-b490-c71cbd2780f3-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-pnwds" (UID: "183f4178-99c0-4945-b490-c71cbd2780f3") : secret "networking-console-plugin-cert" not found Oct 31 04:03:56.990841 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:56.990801 2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 04:03:56.990841 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:56.990819 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d22707c7-21fa-4511-b0b1-869b41fb2351-metrics-certs podName:d22707c7-21fa-4511-b0b1-869b41fb2351 nodeName:}" failed. No retries permitted until 2026-10-31 04:04:00.990813762 +0000 UTC m=+38.090651625 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d22707c7-21fa-4511-b0b1-869b41fb2351-metrics-certs") pod "router-default-678d747c5f-bgj92" (UID: "d22707c7-21fa-4511-b0b1-869b41fb2351") : secret "router-metrics-certs-default" not found Oct 31 04:03:57.092067 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:57.092019 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/48309c9d-5f34-49bf-b55b-f74959984e0a-metrics-tls\") pod \"dns-default-zjpdv\" (UID: \"48309c9d-5f34-49bf-b55b-f74959984e0a\") " pod="openshift-dns/dns-default-zjpdv" Oct 31 04:03:57.092500 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:57.092477 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 04:03:57.092582 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:57.092569 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48309c9d-5f34-49bf-b55b-f74959984e0a-metrics-tls podName:48309c9d-5f34-49bf-b55b-f74959984e0a nodeName:}" failed. No retries permitted until 2026-10-31 04:04:01.092545972 +0000 UTC m=+38.192383858 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/48309c9d-5f34-49bf-b55b-f74959984e0a-metrics-tls") pod "dns-default-zjpdv" (UID: "48309c9d-5f34-49bf-b55b-f74959984e0a") : secret "dns-default-metrics-tls" not found Oct 31 04:03:57.139041 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:57.139002 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hm5cr_89ea384b-705f-47a9-8df6-d3c266ed6761/konnectivity-agent/0.log" limit=1024 Oct 31 04:03:57.194013 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:57.193919 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97c312b1-4ca5-49c8-8c8d-d247c0fa14ac-metrics-certs\") pod \"network-metrics-daemon-clnlz\" (UID: \"97c312b1-4ca5-49c8-8c8d-d247c0fa14ac\") " pod="openshift-multus/network-metrics-daemon-clnlz" Oct 31 04:03:57.194013 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:57.194004 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c1560abd-3afc-4f94-b0bb-ac7ef8b2464a-original-pull-secret\") pod \"global-pull-secret-syncer-hnbll\" (UID: \"c1560abd-3afc-4f94-b0bb-ac7ef8b2464a\") " pod="kube-system/global-pull-secret-syncer-hnbll" Oct 31 04:03:57.194457 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:57.194409 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 04:03:57.194545 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:57.194503 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97c312b1-4ca5-49c8-8c8d-d247c0fa14ac-metrics-certs podName:97c312b1-4ca5-49c8-8c8d-d247c0fa14ac nodeName:}" failed. No retries permitted until 2026-10-31 04:04:29.194480058 +0000 UTC m=+66.294317920 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/97c312b1-4ca5-49c8-8c8d-d247c0fa14ac-metrics-certs") pod "network-metrics-daemon-clnlz" (UID: "97c312b1-4ca5-49c8-8c8d-d247c0fa14ac") : secret "metrics-daemon-secret" not found Oct 31 04:03:57.197576 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:57.197539 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c1560abd-3afc-4f94-b0bb-ac7ef8b2464a-original-pull-secret\") pod \"global-pull-secret-syncer-hnbll\" (UID: \"c1560abd-3afc-4f94-b0bb-ac7ef8b2464a\") " pod="kube-system/global-pull-secret-syncer-hnbll" Oct 31 04:03:57.295320 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:57.295247 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjq6x\" (UniqueName: \"kubernetes.io/projected/aef43d62-9761-4262-8183-01b9890462a7-kube-api-access-kjq6x\") pod \"network-check-target-4b7d5\" (UID: \"aef43d62-9761-4262-8183-01b9890462a7\") " pod="openshift-network-diagnostics/network-check-target-4b7d5" Oct 31 04:03:57.299062 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:57.299030 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjq6x\" (UniqueName: \"kubernetes.io/projected/aef43d62-9761-4262-8183-01b9890462a7-kube-api-access-kjq6x\") pod \"network-check-target-4b7d5\" (UID: \"aef43d62-9761-4262-8183-01b9890462a7\") " pod="openshift-network-diagnostics/network-check-target-4b7d5" Oct 31 04:03:57.437393 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:57.437284 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4b7d5" Oct 31 04:03:57.470881 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:57.470847 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hnbll" Oct 31 04:03:57.798818 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:57.798726 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4b7d5"] Oct 31 04:03:57.888092 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:57.888001 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hnbll"] Oct 31 04:03:57.915452 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:57.915408 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaef43d62_9761_4262_8183_01b9890462a7.slice/crio-c04e1f5e890b4fc9f734848a734ebcb4a6352ad54b9fba27f92545850d9f863c WatchSource:0}: Error finding container c04e1f5e890b4fc9f734848a734ebcb4a6352ad54b9fba27f92545850d9f863c: Status 404 returned error can't find the container with id c04e1f5e890b4fc9f734848a734ebcb4a6352ad54b9fba27f92545850d9f863c Oct 31 04:03:57.917466 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:03:57.917304 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1560abd_3afc_4f94_b0bb_ac7ef8b2464a.slice/crio-22e82066481b6d476e82631fdd12228718bd88c970a6bc27a91876d0d46f26e5 WatchSource:0}: Error finding container 22e82066481b6d476e82631fdd12228718bd88c970a6bc27a91876d0d46f26e5: Status 404 returned error can't find the container with id 22e82066481b6d476e82631fdd12228718bd88c970a6bc27a91876d0d46f26e5 Oct 31 04:03:58.678309 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:58.678275 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s88kp_3945ff65-e877-4372-9334-3968c68b7295/console-operator/0.log" Oct 31 04:03:58.678821 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:58.678327 2571 generic.go:356] "Generic (PLEG): container finished" podID="3945ff65-e877-4372-9334-3968c68b7295" containerID="3c7035b5b7013fd8e43e001803fbfd2f9da4f31219ff66c6ec0df6834152ed7d" exitCode=255 Oct 31 04:03:58.678821 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:58.678667 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-s88kp" event={"ID":"3945ff65-e877-4372-9334-3968c68b7295","Type":"ContainerDied","Data":"3c7035b5b7013fd8e43e001803fbfd2f9da4f31219ff66c6ec0df6834152ed7d"} Oct 31 04:03:58.678929 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:58.678869 2571 scope.go:122] "RemoveContainer" containerID="3c7035b5b7013fd8e43e001803fbfd2f9da4f31219ff66c6ec0df6834152ed7d" Oct 31 04:03:58.680433 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:58.680366 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hnbll" event={"ID":"c1560abd-3afc-4f94-b0bb-ac7ef8b2464a","Type":"ContainerStarted","Data":"22e82066481b6d476e82631fdd12228718bd88c970a6bc27a91876d0d46f26e5"} Oct 31 04:03:58.681908 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:58.681868 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4b7d5" event={"ID":"aef43d62-9761-4262-8183-01b9890462a7","Type":"ContainerStarted","Data":"c04e1f5e890b4fc9f734848a734ebcb4a6352ad54b9fba27f92545850d9f863c"} Oct 31 04:03:59.697610 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:59.697544 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-ld96l" event={"ID":"e667439c-7121-4c5f-84fe-b36f2391dc66","Type":"ContainerStarted","Data":"284e9cfc7e02f8f1e278322b51bbe397a68e25d935271bc660bfb0b55f60f5d2"} Oct 31 04:03:59.700672 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:59.700521 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vqsgm" event={"ID":"a2ce92f3-5b05-4c89-9d53-a57bc85b17ef","Type":"ContainerStarted","Data":"27b654a06654416519d23f5cd69d72063184f65f58c6931f4aa977c5270ccca8"} Oct 31 04:03:59.704243 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:59.703650 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bt9c9" event={"ID":"c3c90e6f-df5e-4c17-bec4-554280a8e8e7","Type":"ContainerStarted","Data":"646dc8afa76d49bdf223f7226aadcaaa754fcf0a10e173603e622a072a919d98"} Oct 31 04:03:59.708362 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:59.707191 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-khdq4" event={"ID":"944c9831-b2fa-4f58-bba2-93288793b125","Type":"ContainerStarted","Data":"11e7117bd4a15f93d0d609b64f88edc2a6d3d0cd2726c8e903546716dc0b63c2"} Oct 31 04:03:59.709842 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:59.709758 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s88kp_3945ff65-e877-4372-9334-3968c68b7295/console-operator/1.log" Oct 31 04:03:59.710368 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:59.710191 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s88kp_3945ff65-e877-4372-9334-3968c68b7295/console-operator/0.log" Oct 31 04:03:59.710368 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:59.710228 2571 generic.go:356] "Generic (PLEG): container finished" podID="3945ff65-e877-4372-9334-3968c68b7295" containerID="e11078a797b967674e01f514aab383ae2b67709c9bc64c60177a719670d63040" exitCode=255 Oct 31 04:03:59.710368 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:59.710273 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-s88kp" event={"ID":"3945ff65-e877-4372-9334-3968c68b7295","Type":"ContainerDied","Data":"e11078a797b967674e01f514aab383ae2b67709c9bc64c60177a719670d63040"} Oct 31 04:03:59.710368 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:59.710320 2571 scope.go:122] "RemoveContainer" containerID="3c7035b5b7013fd8e43e001803fbfd2f9da4f31219ff66c6ec0df6834152ed7d" Oct 31 04:03:59.710895 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:59.710876 2571 scope.go:122] "RemoveContainer" containerID="e11078a797b967674e01f514aab383ae2b67709c9bc64c60177a719670d63040" Oct 31 04:03:59.711226 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:03:59.711162 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-s88kp_openshift-console-operator(3945ff65-e877-4372-9334-3968c68b7295)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-s88kp" podUID="3945ff65-e877-4372-9334-3968c68b7295" Oct 31 04:03:59.760391 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:59.746586 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-ld96l" podStartSLOduration=361.652765147 podStartE2EDuration="6m6.746561122s" totalImagesPullingTime="5.093795975s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:57:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:03:59.744933287 +0000 UTC m=+36.844771175" watchObservedRunningTime="2026-10-31 04:03:59.746561122 +0000 UTC m=+36.846399010" Oct 31 04:03:59.829403 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:59.792782 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vqsgm" podStartSLOduration=1.349477026 podStartE2EDuration="6.792760265s" totalImagesPullingTime="5.443283239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:03:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:03:59.792116834 +0000 UTC m=+36.891954721" watchObservedRunningTime="2026-10-31 04:03:59.792760265 +0000 UTC m=+36.892598151" Oct 31 04:03:59.837661 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:59.837076 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bt9c9" podStartSLOduration=361.830824423 podStartE2EDuration="6m6.837053425s" totalImagesPullingTime="5.006229002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:57:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:03:59.835750913 +0000 UTC m=+36.935588800" watchObservedRunningTime="2026-10-31 04:03:59.837053425 +0000 UTC m=+36.936891311" Oct 31 04:03:59.926010 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:03:59.925945 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-khdq4" podStartSLOduration=361.890195265 podStartE2EDuration="6m6.925928449s" totalImagesPullingTime="5.035733184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:57:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:03:59.924483983 +0000 UTC m=+37.024321870" watchObservedRunningTime="2026-10-31 04:03:59.925928449 +0000 UTC m=+37.025766334" Oct 31 04:04:00.102238 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.102128 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-c6gcg"] Oct 31 04:04:00.126865 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.126764 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-c6gcg"] Oct 31 04:04:00.127023 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.126920 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-c6gcg" Oct 31 04:04:00.162941 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.162885 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 04:04:00.177709 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.177656 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-wh6x6-t7s8v"] Oct 31 04:04:00.192228 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.192085 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wh6x6-t7s8v" Oct 31 04:04:00.195905 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.195842 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-wh6x6-t7s8v"] Oct 31 04:04:00.211354 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.211297 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 04:04:00.211717 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.211675 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8w2pm\"" Oct 31 04:04:00.233911 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.233833 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c6gcg\" (UID: \"9b7bf8e5-7877-46e0-88dd-a3503b7639c5\") " pod="openshift-insights/insights-runtime-extractor-c6gcg" Oct 31 04:04:00.234082 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.233935 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c6gcg\" (UID: \"9b7bf8e5-7877-46e0-88dd-a3503b7639c5\") " pod="openshift-insights/insights-runtime-extractor-c6gcg" Oct 31 04:04:00.234082 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.233973 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frshf\" (UniqueName: \"kubernetes.io/projected/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-kube-api-access-frshf\") pod \"insights-runtime-extractor-c6gcg\" (UID: \"9b7bf8e5-7877-46e0-88dd-a3503b7639c5\") " pod="openshift-insights/insights-runtime-extractor-c6gcg" Oct 31 04:04:00.234082 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.234042 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-data-volume\") pod \"insights-runtime-extractor-c6gcg\" (UID: \"9b7bf8e5-7877-46e0-88dd-a3503b7639c5\") " pod="openshift-insights/insights-runtime-extractor-c6gcg" Oct 31 04:04:00.234251 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.234129 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-crio-socket\") pod \"insights-runtime-extractor-c6gcg\" (UID: \"9b7bf8e5-7877-46e0-88dd-a3503b7639c5\") " pod="openshift-insights/insights-runtime-extractor-c6gcg" Oct 31 04:04:00.335210 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.335172 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-crio-socket\") pod \"insights-runtime-extractor-c6gcg\" (UID: \"9b7bf8e5-7877-46e0-88dd-a3503b7639c5\") " pod="openshift-insights/insights-runtime-extractor-c6gcg" Oct 31 04:04:00.335400 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.335232 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07320d54-b911-4157-aaf5-2517187d745c-trusted-ca-bundle\") pod \"periodic-gathering-wh6x6-t7s8v\" (UID: \"07320d54-b911-4157-aaf5-2517187d745c\") " pod="openshift-insights/periodic-gathering-wh6x6-t7s8v" Oct 31 04:04:00.335400 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.335311 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c6gcg\" (UID: \"9b7bf8e5-7877-46e0-88dd-a3503b7639c5\") " pod="openshift-insights/insights-runtime-extractor-c6gcg" Oct 31 04:04:00.335400 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.335323 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-crio-socket\") pod \"insights-runtime-extractor-c6gcg\" (UID: \"9b7bf8e5-7877-46e0-88dd-a3503b7639c5\") " pod="openshift-insights/insights-runtime-extractor-c6gcg" Oct 31 04:04:00.335591 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.335406 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4pc9\" (UniqueName: \"kubernetes.io/projected/07320d54-b911-4157-aaf5-2517187d745c-kube-api-access-x4pc9\") pod \"periodic-gathering-wh6x6-t7s8v\" (UID: \"07320d54-b911-4157-aaf5-2517187d745c\") " pod="openshift-insights/periodic-gathering-wh6x6-t7s8v" Oct 31 04:04:00.335591 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:04:00.335448 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 04:04:00.335591 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.335451 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07320d54-b911-4157-aaf5-2517187d745c-service-ca-bundle\") pod \"periodic-gathering-wh6x6-t7s8v\" (UID: \"07320d54-b911-4157-aaf5-2517187d745c\") " pod="openshift-insights/periodic-gathering-wh6x6-t7s8v" Oct 31 04:04:00.335591 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:04:00.335508 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-kube-rbac-proxy-cm podName:9b7bf8e5-7877-46e0-88dd-a3503b7639c5 nodeName:}" failed. No retries permitted until 2026-10-31 04:04:00.835484011 +0000 UTC m=+37.935321879 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c6gcg" (UID: "9b7bf8e5-7877-46e0-88dd-a3503b7639c5") : configmap "kube-rbac-proxy" not found Oct 31 04:04:00.335591 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.335521 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c6gcg\" (UID: \"9b7bf8e5-7877-46e0-88dd-a3503b7639c5\") " pod="openshift-insights/insights-runtime-extractor-c6gcg" Oct 31 04:04:00.335591 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.335573 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frshf\" (UniqueName: \"kubernetes.io/projected/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-kube-api-access-frshf\") pod \"insights-runtime-extractor-c6gcg\" (UID: \"9b7bf8e5-7877-46e0-88dd-a3503b7639c5\") " pod="openshift-insights/insights-runtime-extractor-c6gcg" Oct 31 04:04:00.335873 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.335649 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-data-volume\") pod \"insights-runtime-extractor-c6gcg\" (UID: \"9b7bf8e5-7877-46e0-88dd-a3503b7639c5\") " pod="openshift-insights/insights-runtime-extractor-c6gcg" Oct 31 04:04:00.335873 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.335703 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/07320d54-b911-4157-aaf5-2517187d745c-archives-path\") pod \"periodic-gathering-wh6x6-t7s8v\" (UID: \"07320d54-b911-4157-aaf5-2517187d745c\") " pod="openshift-insights/periodic-gathering-wh6x6-t7s8v" Oct 31 04:04:00.335966 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:04:00.335913 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 04:04:00.336013 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:04:00.335975 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-insights-runtime-extractor-tls podName:9b7bf8e5-7877-46e0-88dd-a3503b7639c5 nodeName:}" failed. No retries permitted until 2026-10-31 04:04:00.835958972 +0000 UTC m=+37.935796853 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c6gcg" (UID: "9b7bf8e5-7877-46e0-88dd-a3503b7639c5") : secret "insights-runtime-extractor-tls" not found Oct 31 04:04:00.336187 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.336170 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-data-volume\") pod \"insights-runtime-extractor-c6gcg\" (UID: \"9b7bf8e5-7877-46e0-88dd-a3503b7639c5\") " pod="openshift-insights/insights-runtime-extractor-c6gcg" Oct 31 04:04:00.364102 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.364003 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frshf\" (UniqueName: \"kubernetes.io/projected/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-kube-api-access-frshf\") pod \"insights-runtime-extractor-c6gcg\" (UID: \"9b7bf8e5-7877-46e0-88dd-a3503b7639c5\") " pod="openshift-insights/insights-runtime-extractor-c6gcg" Oct 31 04:04:00.439683 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.437019 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/07320d54-b911-4157-aaf5-2517187d745c-archives-path\") pod \"periodic-gathering-wh6x6-t7s8v\" (UID: \"07320d54-b911-4157-aaf5-2517187d745c\") " pod="openshift-insights/periodic-gathering-wh6x6-t7s8v" Oct 31 04:04:00.439683 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.437124 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07320d54-b911-4157-aaf5-2517187d745c-trusted-ca-bundle\") pod \"periodic-gathering-wh6x6-t7s8v\" (UID: \"07320d54-b911-4157-aaf5-2517187d745c\") " pod="openshift-insights/periodic-gathering-wh6x6-t7s8v" Oct 31 04:04:00.439683 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.437201 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4pc9\" (UniqueName: \"kubernetes.io/projected/07320d54-b911-4157-aaf5-2517187d745c-kube-api-access-x4pc9\") pod \"periodic-gathering-wh6x6-t7s8v\" (UID: \"07320d54-b911-4157-aaf5-2517187d745c\") " pod="openshift-insights/periodic-gathering-wh6x6-t7s8v" Oct 31 04:04:00.439683 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.437236 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07320d54-b911-4157-aaf5-2517187d745c-service-ca-bundle\") pod \"periodic-gathering-wh6x6-t7s8v\" (UID: \"07320d54-b911-4157-aaf5-2517187d745c\") " pod="openshift-insights/periodic-gathering-wh6x6-t7s8v" Oct 31 04:04:00.439683 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.437900 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07320d54-b911-4157-aaf5-2517187d745c-service-ca-bundle\") pod \"periodic-gathering-wh6x6-t7s8v\" (UID: \"07320d54-b911-4157-aaf5-2517187d745c\") " pod="openshift-insights/periodic-gathering-wh6x6-t7s8v" Oct 31 04:04:00.439683 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.438292 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/07320d54-b911-4157-aaf5-2517187d745c-archives-path\") pod \"periodic-gathering-wh6x6-t7s8v\" (UID: \"07320d54-b911-4157-aaf5-2517187d745c\") " pod="openshift-insights/periodic-gathering-wh6x6-t7s8v" Oct 31 04:04:00.439683 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.439274 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07320d54-b911-4157-aaf5-2517187d745c-trusted-ca-bundle\") pod \"periodic-gathering-wh6x6-t7s8v\" (UID: \"07320d54-b911-4157-aaf5-2517187d745c\") " pod="openshift-insights/periodic-gathering-wh6x6-t7s8v" Oct 31 04:04:00.466442 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.466400 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4pc9\" (UniqueName: \"kubernetes.io/projected/07320d54-b911-4157-aaf5-2517187d745c-kube-api-access-x4pc9\") pod \"periodic-gathering-wh6x6-t7s8v\" (UID: \"07320d54-b911-4157-aaf5-2517187d745c\") " pod="openshift-insights/periodic-gathering-wh6x6-t7s8v" Oct 31 04:04:00.498964 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.498916 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-fvqtv"] Oct 31 04:04:00.528798 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.528757 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wh6x6-t7s8v" Oct 31 04:04:00.529089 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.528812 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-zg2xl"] Oct 31 04:04:00.529089 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.528951 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fvqtv" Oct 31 04:04:00.545522 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.545491 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-mqs6m\"" Oct 31 04:04:00.549514 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.548765 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-fvqtv"] Oct 31 04:04:00.549514 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.548829 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-zg2xl"] Oct 31 04:04:00.549514 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.548954 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zg2xl" Oct 31 04:04:00.590134 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.589349 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 31 04:04:00.590134 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.589420 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 31 04:04:00.640595 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.640057 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jbhv\" (UniqueName: \"kubernetes.io/projected/9c8cb360-ff57-41b1-8244-47588703cba5-kube-api-access-9jbhv\") pod \"migrator-66df5d496b-zg2xl\" (UID: \"9c8cb360-ff57-41b1-8244-47588703cba5\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zg2xl" Oct 31 04:04:00.640595 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.640268 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vrvs\" (UniqueName: \"kubernetes.io/projected/49140fa3-6ce2-41ea-8b6b-e25257efaf3e-kube-api-access-6vrvs\") pod \"migrator-66df5d496b-fvqtv\" (UID: \"49140fa3-6ce2-41ea-8b6b-e25257efaf3e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fvqtv" Oct 31 04:04:00.665932 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.664458 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-l25xj"] Oct 31 04:04:00.680595 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.680552 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-l25xj" Oct 31 04:04:00.686190 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.686150 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-l25xj"] Oct 31 04:04:00.696725 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.696502 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 31 04:04:00.696725 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.696579 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 31 04:04:00.696725 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.696599 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 31 04:04:00.696725 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.696664 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-dt5cz\"" Oct 31 04:04:00.696725 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.696687 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 31 04:04:00.717548 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.717514 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s88kp_3945ff65-e877-4372-9334-3968c68b7295/console-operator/1.log" Oct 31 04:04:00.718003 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.717984 2571 scope.go:122] "RemoveContainer" containerID="e11078a797b967674e01f514aab383ae2b67709c9bc64c60177a719670d63040" Oct 31 04:04:00.718263 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:04:00.718234 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-s88kp_openshift-console-operator(3945ff65-e877-4372-9334-3968c68b7295)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-s88kp" podUID="3945ff65-e877-4372-9334-3968c68b7295" Oct 31 04:04:00.720824 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.720792 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rf8vj" event={"ID":"fb656990-e663-4fea-8733-18ded9222a79","Type":"ContainerStarted","Data":"08f391d46fcc7e76dbd28d97bffa852148f443280431424e0661bc4dbc4968ad"} Oct 31 04:04:00.742669 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.741957 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vrvs\" (UniqueName: \"kubernetes.io/projected/49140fa3-6ce2-41ea-8b6b-e25257efaf3e-kube-api-access-6vrvs\") pod \"migrator-66df5d496b-fvqtv\" (UID: \"49140fa3-6ce2-41ea-8b6b-e25257efaf3e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fvqtv" Oct 31 04:04:00.742669 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.742065 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jbhv\" (UniqueName: \"kubernetes.io/projected/9c8cb360-ff57-41b1-8244-47588703cba5-kube-api-access-9jbhv\") pod \"migrator-66df5d496b-zg2xl\" (UID: \"9c8cb360-ff57-41b1-8244-47588703cba5\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zg2xl" Oct 31 04:04:00.768871 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.768838 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vrvs\" (UniqueName: \"kubernetes.io/projected/49140fa3-6ce2-41ea-8b6b-e25257efaf3e-kube-api-access-6vrvs\") pod \"migrator-66df5d496b-fvqtv\" (UID: \"49140fa3-6ce2-41ea-8b6b-e25257efaf3e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fvqtv" Oct 31 04:04:00.776979 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.776924 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jbhv\" (UniqueName: \"kubernetes.io/projected/9c8cb360-ff57-41b1-8244-47588703cba5-kube-api-access-9jbhv\") pod \"migrator-66df5d496b-zg2xl\" (UID: \"9c8cb360-ff57-41b1-8244-47588703cba5\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zg2xl" Oct 31 04:04:00.809479 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.809413 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rf8vj" podStartSLOduration=361.879345067 podStartE2EDuration="6m7.809390265s" totalImagesPullingTime="5.930045198s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:57:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:04:00.808799147 +0000 UTC m=+37.908637033" watchObservedRunningTime="2026-10-31 04:04:00.809390265 +0000 UTC m=+37.909228153" Oct 31 04:04:00.847160 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.843600 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c9eeaad0-e2e5-4b90-a0af-2d254c787fac-signing-cabundle\") pod \"service-ca-5fffc97bdf-l25xj\" (UID: \"c9eeaad0-e2e5-4b90-a0af-2d254c787fac\") " pod="openshift-service-ca/service-ca-5fffc97bdf-l25xj" Oct 31 04:04:00.847160 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.843758 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c9eeaad0-e2e5-4b90-a0af-2d254c787fac-signing-key\") pod \"service-ca-5fffc97bdf-l25xj\" (UID: \"c9eeaad0-e2e5-4b90-a0af-2d254c787fac\") " pod="openshift-service-ca/service-ca-5fffc97bdf-l25xj" Oct 31 04:04:00.847160 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.843808 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57gkt\" (UniqueName: \"kubernetes.io/projected/c9eeaad0-e2e5-4b90-a0af-2d254c787fac-kube-api-access-57gkt\") pod \"service-ca-5fffc97bdf-l25xj\" (UID: \"c9eeaad0-e2e5-4b90-a0af-2d254c787fac\") " pod="openshift-service-ca/service-ca-5fffc97bdf-l25xj" Oct 31 04:04:00.847160 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.843926 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c6gcg\" (UID: \"9b7bf8e5-7877-46e0-88dd-a3503b7639c5\") " pod="openshift-insights/insights-runtime-extractor-c6gcg" Oct 31 04:04:00.847160 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.844070 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c6gcg\" (UID: \"9b7bf8e5-7877-46e0-88dd-a3503b7639c5\") " pod="openshift-insights/insights-runtime-extractor-c6gcg" Oct 31 04:04:00.847160 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:04:00.846482 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 04:04:00.847160 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:04:00.846566 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-insights-runtime-extractor-tls podName:9b7bf8e5-7877-46e0-88dd-a3503b7639c5 nodeName:}" failed. No retries permitted until 2026-10-31 04:04:01.846544601 +0000 UTC m=+38.946382479 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c6gcg" (UID: "9b7bf8e5-7877-46e0-88dd-a3503b7639c5") : secret "insights-runtime-extractor-tls" not found Oct 31 04:04:00.847160 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:04:00.846482 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 04:04:00.847160 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:04:00.846680 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-kube-rbac-proxy-cm podName:9b7bf8e5-7877-46e0-88dd-a3503b7639c5 nodeName:}" failed. No retries permitted until 2026-10-31 04:04:01.846658268 +0000 UTC m=+38.946496140 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c6gcg" (UID: "9b7bf8e5-7877-46e0-88dd-a3503b7639c5") : configmap "kube-rbac-proxy" not found Oct 31 04:04:00.862816 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.862732 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-wh6x6-t7s8v"] Oct 31 04:04:00.865737 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.865707 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fvqtv" Oct 31 04:04:00.867143 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:04:00.867074 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07320d54_b911_4157_aaf5_2517187d745c.slice/crio-a599fe68f82ea190e75d07f1f172d52fc495af94b489a72947ed0658320f0e77 WatchSource:0}: Error finding container a599fe68f82ea190e75d07f1f172d52fc495af94b489a72947ed0658320f0e77: Status 404 returned error can't find the container with id a599fe68f82ea190e75d07f1f172d52fc495af94b489a72947ed0658320f0e77 Oct 31 04:04:00.898793 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.898759 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zg2xl" Oct 31 04:04:00.945102 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.945061 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c9eeaad0-e2e5-4b90-a0af-2d254c787fac-signing-cabundle\") pod \"service-ca-5fffc97bdf-l25xj\" (UID: \"c9eeaad0-e2e5-4b90-a0af-2d254c787fac\") " pod="openshift-service-ca/service-ca-5fffc97bdf-l25xj" Oct 31 04:04:00.945259 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.945182 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c9eeaad0-e2e5-4b90-a0af-2d254c787fac-signing-key\") pod \"service-ca-5fffc97bdf-l25xj\" (UID: \"c9eeaad0-e2e5-4b90-a0af-2d254c787fac\") " pod="openshift-service-ca/service-ca-5fffc97bdf-l25xj" Oct 31 04:04:00.945259 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.945228 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57gkt\" (UniqueName: \"kubernetes.io/projected/c9eeaad0-e2e5-4b90-a0af-2d254c787fac-kube-api-access-57gkt\") pod \"service-ca-5fffc97bdf-l25xj\" (UID: \"c9eeaad0-e2e5-4b90-a0af-2d254c787fac\") " pod="openshift-service-ca/service-ca-5fffc97bdf-l25xj" Oct 31 04:04:00.946297 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.946273 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c9eeaad0-e2e5-4b90-a0af-2d254c787fac-signing-cabundle\") pod \"service-ca-5fffc97bdf-l25xj\" (UID: \"c9eeaad0-e2e5-4b90-a0af-2d254c787fac\") " pod="openshift-service-ca/service-ca-5fffc97bdf-l25xj" Oct 31 04:04:00.950371 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.949741 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c9eeaad0-e2e5-4b90-a0af-2d254c787fac-signing-key\") pod \"service-ca-5fffc97bdf-l25xj\" (UID: \"c9eeaad0-e2e5-4b90-a0af-2d254c787fac\") " pod="openshift-service-ca/service-ca-5fffc97bdf-l25xj" Oct 31 04:04:00.973828 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:00.973753 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57gkt\" (UniqueName: \"kubernetes.io/projected/c9eeaad0-e2e5-4b90-a0af-2d254c787fac-kube-api-access-57gkt\") pod \"service-ca-5fffc97bdf-l25xj\" (UID: \"c9eeaad0-e2e5-4b90-a0af-2d254c787fac\") " pod="openshift-service-ca/service-ca-5fffc97bdf-l25xj" Oct 31 04:04:01.006692 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:01.006436 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-l25xj" Oct 31 04:04:01.047918 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:01.046189 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/93beb8b8-c9e7-4cce-8baf-8497b971e193-registry-tls\") pod \"image-registry-7f78cfdfd8-w98g7\" (UID: \"93beb8b8-c9e7-4cce-8baf-8497b971e193\") " pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" Oct 31 04:04:01.047918 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:01.046239 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/197dee13-748f-4c6d-acc5-1793b5cee80a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-c64cl\" (UID: \"197dee13-748f-4c6d-acc5-1793b5cee80a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c64cl" Oct 31 04:04:01.047918 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:01.046262 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d22707c7-21fa-4511-b0b1-869b41fb2351-service-ca-bundle\") pod \"router-default-678d747c5f-bgj92\" (UID: \"d22707c7-21fa-4511-b0b1-869b41fb2351\") " pod="openshift-ingress/router-default-678d747c5f-bgj92" Oct 31 04:04:01.047918 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:01.046286 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-registry-tls\") pod \"image-registry-6f69bcc59c-w96bl\" (UID: \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\") " pod="openshift-image-registry/image-registry-6f69bcc59c-w96bl" Oct 31 04:04:01.047918 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:01.046309 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/183f4178-99c0-4945-b490-c71cbd2780f3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pnwds\" (UID: \"183f4178-99c0-4945-b490-c71cbd2780f3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pnwds" Oct 31 04:04:01.047918 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:01.046391 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d22707c7-21fa-4511-b0b1-869b41fb2351-metrics-certs\") pod \"router-default-678d747c5f-bgj92\" (UID: \"d22707c7-21fa-4511-b0b1-869b41fb2351\") " pod="openshift-ingress/router-default-678d747c5f-bgj92" Oct 31 04:04:01.047918 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:01.046420 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a173790b-6c13-45a1-a361-3fe51844b4c2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-zqvkx\" (UID: \"a173790b-6c13-45a1-a361-3fe51844b4c2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zqvkx" Oct 31 04:04:01.047918 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:04:01.046491 2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 04:04:01.047918 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:04:01.046527 2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 04:04:01.047918 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:04:01.046577 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/197dee13-748f-4c6d-acc5-1793b5cee80a-samples-operator-tls podName:197dee13-748f-4c6d-acc5-1793b5cee80a nodeName:}" failed. No retries permitted until 2026-10-31 04:04:09.046540551 +0000 UTC m=+46.146378419 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/197dee13-748f-4c6d-acc5-1793b5cee80a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-c64cl" (UID: "197dee13-748f-4c6d-acc5-1793b5cee80a") : secret "samples-operator-tls" not found Oct 31 04:04:01.047918 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:04:01.046596 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a173790b-6c13-45a1-a361-3fe51844b4c2-cluster-monitoring-operator-tls podName:a173790b-6c13-45a1-a361-3fe51844b4c2 nodeName:}" failed. No retries permitted until 2026-10-31 04:04:09.046586986 +0000 UTC m=+46.146424849 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a173790b-6c13-45a1-a361-3fe51844b4c2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-zqvkx" (UID: "a173790b-6c13-45a1-a361-3fe51844b4c2") : secret "cluster-monitoring-operator-tls" not found Oct 31 04:04:01.047918 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:04:01.046653 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 04:04:01.047918 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:04:01.046664 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f78cfdfd8-w98g7: secret "image-registry-tls" not found Oct 31 04:04:01.047918 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:04:01.046699 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93beb8b8-c9e7-4cce-8baf-8497b971e193-registry-tls podName:93beb8b8-c9e7-4cce-8baf-8497b971e193 nodeName:}" failed. No retries permitted until 2026-10-31 04:04:09.046689697 +0000 UTC m=+46.146527560 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/93beb8b8-c9e7-4cce-8baf-8497b971e193-registry-tls") pod "image-registry-7f78cfdfd8-w98g7" (UID: "93beb8b8-c9e7-4cce-8baf-8497b971e193") : secret "image-registry-tls" not found Oct 31 04:04:01.047918 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:04:01.047131 2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 04:04:01.048592 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:04:01.047191 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/183f4178-99c0-4945-b490-c71cbd2780f3-networking-console-plugin-cert podName:183f4178-99c0-4945-b490-c71cbd2780f3 nodeName:}" failed. No retries permitted until 2026-10-31 04:04:09.047175058 +0000 UTC m=+46.147012934 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/183f4178-99c0-4945-b490-c71cbd2780f3-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-pnwds" (UID: "183f4178-99c0-4945-b490-c71cbd2780f3") : secret "networking-console-plugin-cert" not found Oct 31 04:04:01.048592 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:04:01.047269 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 04:04:01.048592 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:04:01.047280 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f69bcc59c-w96bl: secret "image-registry-tls" not found Oct 31 04:04:01.048592 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:04:01.047310 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-registry-tls podName:e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22 nodeName:}" failed. No retries permitted until 2026-10-31 04:04:09.047300449 +0000 UTC m=+46.147138325 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-registry-tls") pod "image-registry-6f69bcc59c-w96bl" (UID: "e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22") : secret "image-registry-tls" not found Oct 31 04:04:01.048592 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:04:01.047392 2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 04:04:01.048592 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:04:01.047422 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d22707c7-21fa-4511-b0b1-869b41fb2351-metrics-certs podName:d22707c7-21fa-4511-b0b1-869b41fb2351 nodeName:}" failed. No retries permitted until 2026-10-31 04:04:09.047413153 +0000 UTC m=+46.147251030 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d22707c7-21fa-4511-b0b1-869b41fb2351-metrics-certs") pod "router-default-678d747c5f-bgj92" (UID: "d22707c7-21fa-4511-b0b1-869b41fb2351") : secret "router-metrics-certs-default" not found Oct 31 04:04:01.048592 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:04:01.047884 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d22707c7-21fa-4511-b0b1-869b41fb2351-service-ca-bundle podName:d22707c7-21fa-4511-b0b1-869b41fb2351 nodeName:}" failed. No retries permitted until 2026-10-31 04:04:09.047866338 +0000 UTC m=+46.147704213 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d22707c7-21fa-4511-b0b1-869b41fb2351-service-ca-bundle") pod "router-default-678d747c5f-bgj92" (UID: "d22707c7-21fa-4511-b0b1-869b41fb2351") : configmap references non-existent config key: service-ca.crt Oct 31 04:04:01.147379 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:01.147270 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/48309c9d-5f34-49bf-b55b-f74959984e0a-metrics-tls\") pod \"dns-default-zjpdv\" (UID: \"48309c9d-5f34-49bf-b55b-f74959984e0a\") " pod="openshift-dns/dns-default-zjpdv" Oct 31 04:04:01.147560 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:04:01.147478 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 04:04:01.147618 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:04:01.147560 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48309c9d-5f34-49bf-b55b-f74959984e0a-metrics-tls podName:48309c9d-5f34-49bf-b55b-f74959984e0a nodeName:}" failed. No retries permitted until 2026-10-31 04:04:09.14753866 +0000 UTC m=+46.247376529 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/48309c9d-5f34-49bf-b55b-f74959984e0a-metrics-tls") pod "dns-default-zjpdv" (UID: "48309c9d-5f34-49bf-b55b-f74959984e0a") : secret "dns-default-metrics-tls" not found Oct 31 04:04:01.197001 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:01.196968 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-fvqtv"] Oct 31 04:04:01.198242 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:04:01.198208 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49140fa3_6ce2_41ea_8b6b_e25257efaf3e.slice/crio-0a5cf0a0e02d01bfcf84d57721f549255cfa89ccd68b088a32c3604ad84eb68c WatchSource:0}: Error finding container 0a5cf0a0e02d01bfcf84d57721f549255cfa89ccd68b088a32c3604ad84eb68c: Status 404 returned error can't find the container with id 0a5cf0a0e02d01bfcf84d57721f549255cfa89ccd68b088a32c3604ad84eb68c Oct 31 04:04:01.262288 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:01.262196 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-zg2xl"] Oct 31 04:04:01.265664 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:04:01.265616 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c8cb360_ff57_41b1_8244_47588703cba5.slice/crio-957817c30f358892a8c5e80b384def14d7201c419b27bbef2a4e0f9beb0c247d WatchSource:0}: Error finding container 957817c30f358892a8c5e80b384def14d7201c419b27bbef2a4e0f9beb0c247d: Status 404 returned error can't find the container with id 957817c30f358892a8c5e80b384def14d7201c419b27bbef2a4e0f9beb0c247d Oct 31 04:04:01.346348 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:01.346304 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-l25xj"] Oct 31 04:04:01.347322 ip-10-0-130-57 kubenswrapper[2571]: W1031 04:04:01.347288 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9eeaad0_e2e5_4b90_a0af_2d254c787fac.slice/crio-46f399d545a61f1c153a8737b0478561959296851004ac253536beb401acd06a WatchSource:0}: Error finding container 46f399d545a61f1c153a8737b0478561959296851004ac253536beb401acd06a: Status 404 returned error can't find the container with id 46f399d545a61f1c153a8737b0478561959296851004ac253536beb401acd06a Oct 31 04:04:01.727395 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:01.727355 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zg2xl" event={"ID":"9c8cb360-ff57-41b1-8244-47588703cba5","Type":"ContainerStarted","Data":"957817c30f358892a8c5e80b384def14d7201c419b27bbef2a4e0f9beb0c247d"} Oct 31 04:04:01.728770 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:01.728740 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fvqtv" event={"ID":"49140fa3-6ce2-41ea-8b6b-e25257efaf3e","Type":"ContainerStarted","Data":"0a5cf0a0e02d01bfcf84d57721f549255cfa89ccd68b088a32c3604ad84eb68c"} Oct 31 04:04:01.730738 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:01.730641 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wh6x6-t7s8v" event={"ID":"07320d54-b911-4157-aaf5-2517187d745c","Type":"ContainerStarted","Data":"eab65427a89c6dce6446b410bfe76a21561a7d8a57d4d33db299596819ab954e"} Oct 31 04:04:01.730738 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:01.730709 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wh6x6-t7s8v" event={"ID":"07320d54-b911-4157-aaf5-2517187d745c","Type":"ContainerStarted","Data":"a599fe68f82ea190e75d07f1f172d52fc495af94b489a72947ed0658320f0e77"} Oct 31 04:04:01.732503 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:01.732479 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4b7d5" event={"ID":"aef43d62-9761-4262-8183-01b9890462a7","Type":"ContainerStarted","Data":"295c1d995013bb6133cd13b111e10b0d8f39d2998d5220f289263e96676e0984"} Oct 31 04:04:01.732601 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:01.732549 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4b7d5" Oct 31 04:04:01.734938 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:01.734724 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-l25xj" event={"ID":"c9eeaad0-e2e5-4b90-a0af-2d254c787fac","Type":"ContainerStarted","Data":"a595dba561d98206e3324054fa17f3c080be2aae427722288bb2903717d6d74d"} Oct 31 04:04:01.734938 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:01.734765 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-l25xj" event={"ID":"c9eeaad0-e2e5-4b90-a0af-2d254c787fac","Type":"ContainerStarted","Data":"46f399d545a61f1c153a8737b0478561959296851004ac253536beb401acd06a"} Oct 31 04:04:01.776056 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:01.775985 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-wh6x6-t7s8v" podStartSLOduration=1.7759641510000002 podStartE2EDuration="1.775964151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:04:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:04:01.773873441 +0000 UTC m=+38.873711326" watchObservedRunningTime="2026-10-31 04:04:01.775964151 +0000 UTC m=+38.875802037" Oct 31 04:04:01.818620 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:01.818373 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-l25xj" podStartSLOduration=1.818350872 podStartE2EDuration="1.818350872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:04:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:04:01.815921882 +0000 UTC m=+38.915759802" watchObservedRunningTime="2026-10-31 04:04:01.818350872 +0000 UTC m=+38.918188758" Oct 31 04:04:01.856006 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:01.855875 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c6gcg\" (UID: \"9b7bf8e5-7877-46e0-88dd-a3503b7639c5\") " pod="openshift-insights/insights-runtime-extractor-c6gcg" Oct 31 04:04:01.856967 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:04:01.856929 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 04:04:01.857619 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:01.857586 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c6gcg\" (UID: \"9b7bf8e5-7877-46e0-88dd-a3503b7639c5\") " pod="openshift-insights/insights-runtime-extractor-c6gcg" Oct 31 04:04:01.858314 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:04:01.858294 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-insights-runtime-extractor-tls podName:9b7bf8e5-7877-46e0-88dd-a3503b7639c5 nodeName:}" failed. No retries permitted until 2026-10-31 04:04:03.858270361 +0000 UTC m=+40.958108231 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c6gcg" (UID: "9b7bf8e5-7877-46e0-88dd-a3503b7639c5") : secret "insights-runtime-extractor-tls" not found Oct 31 04:04:01.859070 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:04:01.858969 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 04:04:01.859070 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:04:01.859036 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-kube-rbac-proxy-cm podName:9b7bf8e5-7877-46e0-88dd-a3503b7639c5 nodeName:}" failed. No retries permitted until 2026-10-31 04:04:03.85902109 +0000 UTC m=+40.958858970 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c6gcg" (UID: "9b7bf8e5-7877-46e0-88dd-a3503b7639c5") : configmap "kube-rbac-proxy" not found Oct 31 04:04:01.862564 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:01.861232 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-4b7d5" podStartSLOduration=35.257255467 podStartE2EDuration="38.861161564s" totalImagesPullingTime="3.603906097s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:03:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:04:01.860413428 +0000 UTC m=+38.960251314" watchObservedRunningTime="2026-10-31 04:04:01.861161564 +0000 UTC m=+38.960999452" Oct 31 04:04:03.638128 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:03.638086 2571 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-s88kp" Oct 31 04:04:03.638801 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:03.638170 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-s88kp" Oct 31 04:04:03.638801 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:03.638615 2571 scope.go:122] "RemoveContainer" containerID="e11078a797b967674e01f514aab383ae2b67709c9bc64c60177a719670d63040" Oct 31 04:04:03.638930 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:04:03.638857 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-s88kp_openshift-console-operator(3945ff65-e877-4372-9334-3968c68b7295)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-s88kp" podUID="3945ff65-e877-4372-9334-3968c68b7295" Oct 31 04:04:03.745223 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:03.745170 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zg2xl" event={"ID":"9c8cb360-ff57-41b1-8244-47588703cba5","Type":"ContainerStarted","Data":"1dad1e6bd265eadad1d13779c82c8fc6a39397b1f8d4f1189d92a4554c04ea87"} Oct 31 04:04:03.747123 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:03.747086 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fvqtv" event={"ID":"49140fa3-6ce2-41ea-8b6b-e25257efaf3e","Type":"ContainerStarted","Data":"03cfa5ca9691e5cfc724efe2cb7414f1ba692f35f74c97d3f9128938abe6a8fd"} Oct 31 04:04:03.790896 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:03.790841 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zg2xl" podStartSLOduration=1.9124995089999999 podStartE2EDuration="3.790826697s" totalImagesPullingTime="1.878327188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:04:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:04:03.790128698 +0000 UTC m=+40.889966585" watchObservedRunningTime="2026-10-31 04:04:03.790826697 +0000 UTC m=+40.890664581" Oct 31 04:04:03.833973 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:03.833891 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fvqtv" podStartSLOduration=1.8838733840000002 podStartE2EDuration="3.833866691s" totalImagesPullingTime="1.949993307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:04:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:04:03.831897365 +0000 UTC m=+40.931735253" watchObservedRunningTime="2026-10-31 04:04:03.833866691 +0000 UTC m=+40.933704576" Oct 31 04:04:03.888202 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:03.888105 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c6gcg\" (UID: \"9b7bf8e5-7877-46e0-88dd-a3503b7639c5\") " pod="openshift-insights/insights-runtime-extractor-c6gcg" Oct 31 04:04:03.888406 ip-10-0-130-57 kubenswrapper[2571]: I1031 04:04:03.888221 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c6gcg\" (UID: \"9b7bf8e5-7877-46e0-88dd-a3503b7639c5\") " pod="openshift-insights/insights-runtime-extractor-c6gcg" Oct 31 04:04:03.888406 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:04:03.888254 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 04:04:03.888406 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:04:03.888346 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-kube-rbac-proxy-cm podName:9b7bf8e5-7877-46e0-88dd-a3503b7639c5 nodeName:}" failed. No retries permitted until 2026-10-31 04:04:07.888311289 +0000 UTC m=+44.988149153 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c6gcg" (UID: "9b7bf8e5-7877-46e0-88dd-a3503b7639c5") : configmap "kube-rbac-proxy" not found Oct 31 04:04:03.888582 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:04:03.888414 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 04:04:03.888582 ip-10-0-130-57 kubenswrapper[2571]: E1031 04:04:03.888489 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-insights-runtime-extractor-tls podName:9b7bf8e5-7877-46e0-88dd-a3503b7639c5 nodeName:}" failed. No retries permitted until 2026-10-31 04:04:07.888471574 +0000 UTC m=+44.988309461 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c6gcg" (UID: "9b7bf8e5-7877-46e0-88dd-a3503b7639c5") : secret "insights-runtime-extractor-tls" not found Oct 31 04:04:05.161554 ip-10-0-130-57 systemd[1]: Stopping Kubernetes Kubelet... Oct 31 04:04:05.177814 ip-10-0-130-57 systemd[1]: kubelet.service: Deactivated successfully. Oct 31 04:04:05.178047 ip-10-0-130-57 systemd[1]: Stopped Kubernetes Kubelet. Oct 31 04:04:05.178117 ip-10-0-130-57 systemd[1]: kubelet.service: Consumed 3.484s CPU time. Oct 31 04:04:05.194194 ip-10-0-130-57 systemd[1]: Starting Kubernetes Kubelet... Oct 31 04:04:05.315666 ip-10-0-130-57 kubenswrapper[5019]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 04:04:05.315666 ip-10-0-130-57 kubenswrapper[5019]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 04:04:05.315666 ip-10-0-130-57 kubenswrapper[5019]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 04:04:05.315666 ip-10-0-130-57 kubenswrapper[5019]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 04:04:05.317907 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.317806 5019 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 04:04:05.317907 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.317904 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 04:04:05.317907 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.317909 5019 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 04:04:05.317907 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.317915 5019 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 04:04:05.318162 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.317918 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 04:04:05.318162 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.317921 5019 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 04:04:05.318162 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.317924 5019 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 04:04:05.318162 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.317927 5019 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 04:04:05.318162 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.317930 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 04:04:05.318162 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.317933 5019 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 04:04:05.318162 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.317935 5019 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 04:04:05.318162 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.317938 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 04:04:05.318162 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.317941 5019 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 04:04:05.318162 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.317944 5019 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 04:04:05.318162 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.317947 5019 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 04:04:05.318162 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.317950 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 04:04:05.318162 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.317953 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 04:04:05.318162 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.317956 5019 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 04:04:05.318162 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.317959 5019 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 04:04:05.318162 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.317961 5019 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 04:04:05.318162 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.317968 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 04:04:05.318162 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.317972 5019 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 04:04:05.318162 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.317975 5019 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 04:04:05.318691 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.317979 5019 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 04:04:05.318691 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.317982 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 04:04:05.318691 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.317985 5019 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 04:04:05.318691 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.317988 5019 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 04:04:05.318691 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.317991 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 04:04:05.318691 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.317994 5019 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 04:04:05.318691 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.317996 5019 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 04:04:05.318691 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.317999 5019 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 04:04:05.318691 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318002 5019 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 04:04:05.318691 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318004 5019 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 04:04:05.318691 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318007 5019 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 04:04:05.318691 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318010 5019 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 04:04:05.318691 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318013 5019 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 04:04:05.318691 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318015 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 04:04:05.318691 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318018 5019 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 04:04:05.318691 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318021 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 04:04:05.318691 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318024 5019 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 04:04:05.318691 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318026 5019 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 04:04:05.318691 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318029 5019 feature_gate.go:386] unrecognized feature gate: Example Oct 31 04:04:05.318691 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318032 5019 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 04:04:05.319192 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318034 5019 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 04:04:05.319192 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318037 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 04:04:05.319192 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318040 5019 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 04:04:05.319192 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318043 5019 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 04:04:05.319192 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318048 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 04:04:05.319192 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318051 5019 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 04:04:05.319192 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318053 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 04:04:05.319192 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318056 5019 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 04:04:05.319192 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318059 5019 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 04:04:05.319192 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318061 5019 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 04:04:05.319192 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318065 5019 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 04:04:05.319192 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318067 5019 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 04:04:05.319192 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318070 5019 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 04:04:05.319192 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318072 5019 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 04:04:05.319192 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318075 5019 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 04:04:05.319192 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318078 5019 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 04:04:05.319192 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318081 5019 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 04:04:05.319192 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318084 5019 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 04:04:05.319192 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318086 5019 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 04:04:05.319192 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318089 5019 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 04:04:05.319752 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318094 5019 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 04:04:05.319752 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318098 5019 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 04:04:05.319752 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318102 5019 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 04:04:05.319752 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318108 5019 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 04:04:05.319752 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318112 5019 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 04:04:05.319752 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318116 5019 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 04:04:05.319752 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318121 5019 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 04:04:05.319752 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318125 5019 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 04:04:05.319752 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318128 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 04:04:05.319752 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318132 5019 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 04:04:05.319752 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318137 5019 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 04:04:05.319752 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318142 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 04:04:05.319752 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318145 5019 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 04:04:05.319752 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318149 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 04:04:05.319752 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318154 5019 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 04:04:05.319752 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318158 5019 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 04:04:05.319752 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318163 5019 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 04:04:05.319752 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318168 5019 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 04:04:05.319752 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318172 5019 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 04:04:05.320229 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318176 5019 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 04:04:05.320229 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318181 5019 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 04:04:05.320229 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318187 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 04:04:05.320229 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318192 5019 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 04:04:05.320229 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318195 5019 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 04:04:05.320229 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318197 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 04:04:05.320229 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318201 5019 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 04:04:05.320229 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318204 5019 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 04:04:05.320229 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318206 5019 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 04:04:05.320229 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318209 5019 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 04:04:05.320229 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318212 5019 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 04:04:05.320229 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318217 5019 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 04:04:05.320229 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318221 5019 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 04:04:05.320229 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318224 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 04:04:05.320229 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318227 5019 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 04:04:05.320229 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318230 5019 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 04:04:05.320229 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318234 5019 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 04:04:05.320229 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318237 5019 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 04:04:05.320229 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318240 5019 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 04:04:05.320754 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318242 5019 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 04:04:05.320754 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318245 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 04:04:05.320754 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318247 5019 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 04:04:05.320754 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318250 5019 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 04:04:05.320754 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.318253 5019 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 04:04:05.320754 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319158 5019 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 04:04:05.320754 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319167 5019 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 04:04:05.320754 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319170 5019 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 04:04:05.320754 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319174 5019 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 04:04:05.320754 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319177 5019 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 04:04:05.320754 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319181 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 04:04:05.320754 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319184 5019 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 04:04:05.320754 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319187 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 04:04:05.320754 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319190 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 04:04:05.320754 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319193 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 04:04:05.320754 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319196 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 04:04:05.320754 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319199 5019 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 04:04:05.320754 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319202 5019 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 04:04:05.320754 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319206 5019 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 04:04:05.321267 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319211 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 04:04:05.321267 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319214 5019 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 04:04:05.321267 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319217 5019 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 04:04:05.321267 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319220 5019 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 04:04:05.321267 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319223 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 04:04:05.321267 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319226 5019 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 04:04:05.321267 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319228 5019 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 04:04:05.321267 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319231 5019 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 04:04:05.321267 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319233 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 04:04:05.321267 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319236 5019 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 04:04:05.321267 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319239 5019 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 04:04:05.321267 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319245 5019 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 04:04:05.321267 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319249 5019 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 04:04:05.321267 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319251 5019 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 04:04:05.321267 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319254 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 04:04:05.321267 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319257 5019 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 04:04:05.321267 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319259 5019 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 04:04:05.321267 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319262 5019 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 04:04:05.321267 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319264 5019 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 04:04:05.321775 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319267 5019 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 04:04:05.321775 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319269 5019 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 04:04:05.321775 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319272 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 04:04:05.321775 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319274 5019 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 04:04:05.321775 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319278 5019 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 04:04:05.321775 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319282 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 04:04:05.321775 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319286 5019 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 04:04:05.321775 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319288 5019 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 04:04:05.321775 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319291 5019 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 04:04:05.321775 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319294 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 04:04:05.321775 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319297 5019 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 04:04:05.321775 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319300 5019 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 04:04:05.321775 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319302 5019 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 04:04:05.321775 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319305 5019 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 04:04:05.321775 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319308 5019 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 04:04:05.321775 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319310 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 04:04:05.321775 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319313 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 04:04:05.321775 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319316 5019 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 04:04:05.321775 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319319 5019 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 04:04:05.322256 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319321 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 04:04:05.322256 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319324 5019 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 04:04:05.322256 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319326 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 04:04:05.322256 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319343 5019 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 04:04:05.322256 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319346 5019 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 04:04:05.322256 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319348 5019 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 04:04:05.322256 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319357 5019 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 04:04:05.322256 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319360 5019 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 04:04:05.322256 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319363 5019 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 04:04:05.322256 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319365 5019 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 04:04:05.322256 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319368 5019 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 04:04:05.322256 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319370 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 04:04:05.322256 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319373 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 04:04:05.322256 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319376 5019 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 04:04:05.322256 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319378 5019 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 04:04:05.322256 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319381 5019 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 04:04:05.322256 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319383 5019 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 04:04:05.322256 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319386 5019 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 04:04:05.322256 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319389 5019 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 04:04:05.322256 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319392 5019 feature_gate.go:386] unrecognized feature gate: Example Oct 31 04:04:05.322868 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319394 5019 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 04:04:05.322868 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319397 5019 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 04:04:05.322868 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319399 5019 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 04:04:05.322868 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319402 5019 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 04:04:05.322868 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319405 5019 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 04:04:05.322868 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319408 5019 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 04:04:05.322868 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319410 5019 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 04:04:05.322868 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319413 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 04:04:05.322868 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319416 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 04:04:05.322868 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319418 5019 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 04:04:05.322868 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319420 5019 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 04:04:05.322868 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319423 5019 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 04:04:05.322868 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319425 5019 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 04:04:05.322868 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319428 5019 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 04:04:05.322868 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319430 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 04:04:05.322868 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319433 5019 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 04:04:05.322868 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319436 5019 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 04:04:05.322868 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319438 5019 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 04:04:05.322868 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319441 5019 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 04:04:05.322868 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319449 5019 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 04:04:05.323388 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319452 5019 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 04:04:05.323388 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319455 5019 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 04:04:05.323388 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319458 5019 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 04:04:05.323388 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319460 5019 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 04:04:05.323388 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319463 5019 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 04:04:05.323388 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319465 5019 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 04:04:05.323388 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319468 5019 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 04:04:05.323388 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319470 5019 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 04:04:05.323388 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319473 5019 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 04:04:05.323388 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319475 5019 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 04:04:05.323388 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319478 5019 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 04:04:05.323388 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319482 5019 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 04:04:05.323388 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319484 5019 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 04:04:05.323388 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.319487 5019 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 04:04:05.323388 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320114 5019 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 04:04:05.323388 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320119 5019 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 04:04:05.323388 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320123 5019 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 04:04:05.323388 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320126 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 04:04:05.323388 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320129 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 04:04:05.323388 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320131 5019 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 04:04:05.323950 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320134 5019 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 04:04:05.323950 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320137 5019 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 04:04:05.323950 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320139 5019 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 04:04:05.323950 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320142 5019 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 04:04:05.323950 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320144 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 04:04:05.323950 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320147 5019 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 04:04:05.323950 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320150 5019 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 04:04:05.323950 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320152 5019 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 04:04:05.323950 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320155 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 04:04:05.323950 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320157 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 04:04:05.323950 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320160 5019 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 04:04:05.323950 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320168 5019 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 04:04:05.323950 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320171 5019 feature_gate.go:386] unrecognized feature gate: Example Oct 31 04:04:05.323950 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320174 5019 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 04:04:05.323950 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320177 5019 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 04:04:05.323950 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320179 5019 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 04:04:05.323950 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320185 5019 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 04:04:05.323950 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320189 5019 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 04:04:05.323950 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320194 5019 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 04:04:05.324481 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320197 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 04:04:05.324481 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320200 5019 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 04:04:05.324481 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320203 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 04:04:05.324481 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320205 5019 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 04:04:05.324481 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320208 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 04:04:05.324481 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320211 5019 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 04:04:05.324481 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320213 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 04:04:05.324481 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320216 5019 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 04:04:05.324481 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320219 5019 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 04:04:05.324481 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320221 5019 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 04:04:05.324481 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320224 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 04:04:05.324481 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320226 5019 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 04:04:05.324481 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320229 5019 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 04:04:05.324481 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320232 5019 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 04:04:05.324481 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320235 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 04:04:05.324481 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320238 5019 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 04:04:05.324481 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320240 5019 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 04:04:05.324481 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320243 5019 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 04:04:05.324481 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320246 5019 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 04:04:05.324481 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320249 5019 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 04:04:05.325085 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320251 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 04:04:05.325085 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320254 5019 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 04:04:05.325085 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320257 5019 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 04:04:05.325085 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320259 5019 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 04:04:05.325085 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320262 5019 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 04:04:05.325085 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320266 5019 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 04:04:05.325085 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320269 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 04:04:05.325085 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320271 5019 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 04:04:05.325085 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320274 5019 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 04:04:05.325085 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320278 5019 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 04:04:05.325085 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320281 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 04:04:05.325085 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320284 5019 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 04:04:05.325085 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320286 5019 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 04:04:05.325085 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320289 5019 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 04:04:05.325085 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320294 5019 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 04:04:05.325085 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320297 5019 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 04:04:05.325085 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320300 5019 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 04:04:05.325085 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320302 5019 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 04:04:05.325085 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320305 5019 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 04:04:05.325907 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320308 5019 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 04:04:05.325907 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320310 5019 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 04:04:05.325907 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320313 5019 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 04:04:05.325907 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320316 5019 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 04:04:05.325907 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320319 5019 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 04:04:05.325907 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320321 5019 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 04:04:05.325907 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320324 5019 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 04:04:05.325907 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320327 5019 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 04:04:05.325907 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320342 5019 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 04:04:05.325907 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320345 5019 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 04:04:05.325907 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320349 5019 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 04:04:05.325907 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320351 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 04:04:05.325907 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320354 5019 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 04:04:05.325907 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320357 5019 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 04:04:05.325907 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320360 5019 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 04:04:05.325907 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320363 5019 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 04:04:05.325907 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320365 5019 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 04:04:05.325907 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320368 5019 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 04:04:05.325907 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320370 5019 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 04:04:05.326435 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320374 5019 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 04:04:05.326435 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320378 5019 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 04:04:05.326435 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320382 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 04:04:05.326435 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320388 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 04:04:05.326435 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320391 5019 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 04:04:05.326435 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320393 5019 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 04:04:05.326435 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320396 5019 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 04:04:05.326435 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320399 5019 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 04:04:05.326435 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320405 5019 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 04:04:05.326435 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320407 5019 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 04:04:05.326435 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320411 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 04:04:05.326435 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320413 5019 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 04:04:05.326435 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320416 5019 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 04:04:05.326435 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320419 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 04:04:05.326435 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320422 5019 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 04:04:05.326435 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320424 5019 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 04:04:05.326435 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320427 5019 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 04:04:05.326435 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320430 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 04:04:05.326435 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320433 5019 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 04:04:05.326938 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320435 5019 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 04:04:05.326938 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320438 5019 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 04:04:05.326938 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320441 5019 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 04:04:05.326938 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.320444 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 04:04:05.326938 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320543 5019 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 04:04:05.326938 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320551 5019 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 04:04:05.326938 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320563 5019 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 04:04:05.326938 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320569 5019 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 04:04:05.326938 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320573 5019 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 04:04:05.326938 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320577 5019 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 04:04:05.326938 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320582 5019 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 04:04:05.326938 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320586 5019 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 04:04:05.326938 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320589 5019 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 04:04:05.326938 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320592 5019 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 04:04:05.326938 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320597 5019 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 04:04:05.326938 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320601 5019 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 04:04:05.326938 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320606 5019 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 04:04:05.326938 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320609 5019 flags.go:64] FLAG: --cgroup-root="" Oct 31 04:04:05.326938 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320612 5019 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 04:04:05.326938 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320615 5019 flags.go:64] FLAG: --client-ca-file="" Oct 31 04:04:05.326938 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320618 5019 flags.go:64] FLAG: --cloud-provider="external" Oct 31 04:04:05.326938 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320624 5019 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 04:04:05.326938 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320632 5019 flags.go:64] FLAG: --cluster-domain="" Oct 31 04:04:05.327552 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320635 5019 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 04:04:05.327552 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320639 5019 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 04:04:05.327552 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320642 5019 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 04:04:05.327552 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320646 5019 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 04:04:05.327552 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320651 5019 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 04:04:05.327552 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320654 5019 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 04:04:05.327552 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320658 5019 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 04:04:05.327552 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320662 5019 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 04:04:05.327552 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320665 5019 flags.go:64] FLAG: --contention-profiling="false" Oct 31 04:04:05.327552 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320669 5019 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 04:04:05.327552 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320672 5019 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 04:04:05.327552 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320675 5019 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 04:04:05.327552 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320678 5019 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 04:04:05.327552 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320683 5019 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 04:04:05.327552 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320686 5019 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 04:04:05.327552 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320689 5019 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 04:04:05.327552 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320692 5019 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 04:04:05.327552 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320695 5019 flags.go:64] FLAG: --enable-server="true" Oct 31 04:04:05.327552 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320699 5019 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 04:04:05.327552 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320703 5019 flags.go:64] FLAG: --event-burst="100" Oct 31 04:04:05.327552 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320707 5019 flags.go:64] FLAG: --event-qps="50" Oct 31 04:04:05.327552 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320710 5019 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 04:04:05.327552 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320713 5019 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 04:04:05.327552 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320716 5019 flags.go:64] FLAG: --eviction-hard="" Oct 31 04:04:05.328150 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320721 5019 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 04:04:05.328150 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320729 5019 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 04:04:05.328150 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320733 5019 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 04:04:05.328150 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320736 5019 flags.go:64] FLAG: --eviction-soft="" Oct 31 04:04:05.328150 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320739 5019 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 04:04:05.328150 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320742 5019 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 04:04:05.328150 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320747 5019 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 04:04:05.328150 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320750 5019 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 04:04:05.328150 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320753 5019 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 04:04:05.328150 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320756 5019 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 04:04:05.328150 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320759 5019 flags.go:64] FLAG: --feature-gates="" Oct 31 04:04:05.328150 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320764 5019 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 04:04:05.328150 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320767 5019 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 04:04:05.328150 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320770 5019 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 04:04:05.328150 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320774 5019 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 04:04:05.328150 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320777 5019 flags.go:64] FLAG: --healthz-port="10248" Oct 31 04:04:05.328150 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320781 5019 flags.go:64] FLAG: --help="false" Oct 31 04:04:05.328150 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320784 5019 flags.go:64] FLAG: --hostname-override="ip-10-0-130-57.ec2.internal" Oct 31 04:04:05.328150 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320788 5019 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 04:04:05.328150 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320791 5019 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 04:04:05.328150 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320795 5019 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 04:04:05.328150 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320798 5019 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 04:04:05.328150 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320802 5019 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 04:04:05.328758 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320805 5019 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 04:04:05.328758 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320808 5019 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 04:04:05.328758 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320811 5019 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 04:04:05.328758 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320814 5019 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 04:04:05.328758 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320817 5019 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 04:04:05.328758 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320820 5019 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 04:04:05.328758 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320824 5019 flags.go:64] FLAG: --kube-reserved="" Oct 31 04:04:05.328758 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320827 5019 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 04:04:05.328758 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320830 5019 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 04:04:05.328758 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320833 5019 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 04:04:05.328758 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320837 5019 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 04:04:05.328758 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320840 5019 flags.go:64] FLAG: --lock-file="" Oct 31 04:04:05.328758 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320843 5019 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 04:04:05.328758 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320846 5019 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 04:04:05.328758 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320849 5019 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 04:04:05.328758 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320857 5019 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 04:04:05.328758 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320860 5019 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 04:04:05.328758 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320863 5019 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 04:04:05.328758 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320866 5019 flags.go:64] FLAG: --logging-format="text" Oct 31 04:04:05.328758 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320869 5019 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 04:04:05.328758 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320873 5019 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 04:04:05.328758 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320877 5019 flags.go:64] FLAG: --manifest-url="" Oct 31 04:04:05.328758 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320880 5019 flags.go:64] FLAG: --manifest-url-header="" Oct 31 04:04:05.328758 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320885 5019 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 04:04:05.328758 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320888 5019 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 04:04:05.329434 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320892 5019 flags.go:64] FLAG: --max-pods="110" Oct 31 04:04:05.329434 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320896 5019 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 04:04:05.329434 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320899 5019 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 04:04:05.329434 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320902 5019 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 04:04:05.329434 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320905 5019 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 04:04:05.329434 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320908 5019 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 04:04:05.329434 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320912 5019 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 04:04:05.329434 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320915 5019 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 04:04:05.329434 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320922 5019 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 04:04:05.329434 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320926 5019 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 04:04:05.329434 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320929 5019 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 04:04:05.329434 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320932 5019 flags.go:64] FLAG: --pod-cidr="" Oct 31 04:04:05.329434 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320935 5019 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 04:04:05.329434 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320938 5019 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 04:04:05.329434 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320942 5019 flags.go:64] FLAG: --pods-per-core="0" Oct 31 04:04:05.329434 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320945 5019 flags.go:64] FLAG: --port="10250" Oct 31 04:04:05.329434 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320948 5019 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 04:04:05.329434 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320951 5019 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0260c631d3cc88ab1" Oct 31 04:04:05.329434 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320955 5019 flags.go:64] FLAG: --qos-reserved="" Oct 31 04:04:05.329434 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320958 5019 flags.go:64] FLAG: --read-only-port="10255" Oct 31 04:04:05.329434 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320961 5019 flags.go:64] FLAG: --register-node="true" Oct 31 04:04:05.329434 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320964 5019 flags.go:64] FLAG: --register-with-taints="" Oct 31 04:04:05.329434 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320969 5019 flags.go:64] FLAG: --registry-burst="10" Oct 31 04:04:05.329434 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320973 5019 flags.go:64] FLAG: --registry-qps="5" Oct 31 04:04:05.329434 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320976 5019 flags.go:64] FLAG: --reserved-cpus="" Oct 31 04:04:05.329434 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320979 5019 flags.go:64] FLAG: --reserved-memory="" Oct 31 04:04:05.330128 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320983 5019 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 04:04:05.330128 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320986 5019 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 04:04:05.330128 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320990 5019 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 04:04:05.330128 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320994 5019 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 04:04:05.330128 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.320997 5019 flags.go:64] FLAG: --runonce="false" Oct 31 04:04:05.330128 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.321000 5019 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 04:04:05.330128 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.321003 5019 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 04:04:05.330128 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.321007 5019 flags.go:64] FLAG: --seccomp-default="false" Oct 31 04:04:05.330128 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.321010 5019 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 04:04:05.330128 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.321013 5019 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 04:04:05.330128 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.321016 5019 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 04:04:05.330128 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.321020 5019 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 04:04:05.330128 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.321023 5019 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 04:04:05.330128 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.321026 5019 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 04:04:05.330128 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.321029 5019 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 04:04:05.330128 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.321032 5019 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 04:04:05.330128 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.321035 5019 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 04:04:05.330128 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.321038 5019 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 04:04:05.330128 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.321041 5019 flags.go:64] FLAG: --system-cgroups="" Oct 31 04:04:05.330128 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.321044 5019 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 04:04:05.330128 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.321049 5019 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 04:04:05.330128 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.321052 5019 flags.go:64] FLAG: --tls-cert-file="" Oct 31 04:04:05.330128 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.321055 5019 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 04:04:05.330128 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.321060 5019 flags.go:64] FLAG: --tls-min-version="" Oct 31 04:04:05.330128 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.321063 5019 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 04:04:05.330777 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.321067 5019 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 04:04:05.330777 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.321070 5019 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 04:04:05.330777 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.321072 5019 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 04:04:05.330777 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.321077 5019 flags.go:64] FLAG: --v="2" Oct 31 04:04:05.330777 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.321082 5019 flags.go:64] FLAG: --version="false" Oct 31 04:04:05.330777 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.321086 5019 flags.go:64] FLAG: --vmodule="" Oct 31 04:04:05.330777 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.321090 5019 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 04:04:05.330777 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.321094 5019 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 04:04:05.330777 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321252 5019 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 04:04:05.330777 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321256 5019 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 04:04:05.330777 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321260 5019 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 04:04:05.330777 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321264 5019 feature_gate.go:386] unrecognized feature gate: Example Oct 31 04:04:05.330777 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321267 5019 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 04:04:05.330777 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321270 5019 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 04:04:05.330777 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321273 5019 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 04:04:05.330777 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321276 5019 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 04:04:05.330777 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321279 5019 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 04:04:05.330777 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321282 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 04:04:05.330777 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321285 5019 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 04:04:05.330777 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321287 5019 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 04:04:05.330777 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321290 5019 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 04:04:05.331302 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321292 5019 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 04:04:05.331302 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321295 5019 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 04:04:05.331302 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321297 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 04:04:05.331302 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321300 5019 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 04:04:05.331302 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321303 5019 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 04:04:05.331302 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321305 5019 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 04:04:05.331302 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321308 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 04:04:05.331302 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321311 5019 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 04:04:05.331302 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321313 5019 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 04:04:05.331302 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321316 5019 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 04:04:05.331302 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321319 5019 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 04:04:05.331302 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321321 5019 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 04:04:05.331302 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321324 5019 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 04:04:05.331302 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321327 5019 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 04:04:05.331302 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321345 5019 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 04:04:05.331302 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321349 5019 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 04:04:05.331302 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321351 5019 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 04:04:05.331302 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321354 5019 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 04:04:05.331302 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321356 5019 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 04:04:05.331795 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321360 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 04:04:05.331795 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321363 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 04:04:05.331795 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321366 5019 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 04:04:05.331795 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321368 5019 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 04:04:05.331795 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321371 5019 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 04:04:05.331795 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321374 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 04:04:05.331795 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321377 5019 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 04:04:05.331795 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321380 5019 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 04:04:05.331795 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321383 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 04:04:05.331795 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321386 5019 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 04:04:05.331795 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321390 5019 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 04:04:05.331795 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321394 5019 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 04:04:05.331795 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321397 5019 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 04:04:05.331795 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321400 5019 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 04:04:05.331795 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321403 5019 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 04:04:05.331795 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321405 5019 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 04:04:05.331795 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321408 5019 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 04:04:05.331795 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321410 5019 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 04:04:05.331795 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321413 5019 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 04:04:05.331795 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321416 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 04:04:05.332356 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321418 5019 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 04:04:05.332356 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321421 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 04:04:05.332356 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321423 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 04:04:05.332356 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321426 5019 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 04:04:05.332356 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321428 5019 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 04:04:05.332356 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321431 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 04:04:05.332356 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321434 5019 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 04:04:05.332356 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321438 5019 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 04:04:05.332356 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321442 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 04:04:05.332356 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321444 5019 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 04:04:05.332356 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321447 5019 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 04:04:05.332356 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321449 5019 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 04:04:05.332356 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321453 5019 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 04:04:05.332356 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321456 5019 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 04:04:05.332356 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321459 5019 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 04:04:05.332356 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321462 5019 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 04:04:05.332356 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321464 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 04:04:05.332356 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321467 5019 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 04:04:05.332356 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321469 5019 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 04:04:05.332918 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321472 5019 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 04:04:05.332918 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321475 5019 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 04:04:05.332918 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321477 5019 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 04:04:05.332918 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321480 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 04:04:05.332918 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321482 5019 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 04:04:05.332918 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321485 5019 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 04:04:05.332918 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321488 5019 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 04:04:05.332918 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321490 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 04:04:05.332918 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321493 5019 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 04:04:05.332918 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321495 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 04:04:05.332918 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321498 5019 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 04:04:05.332918 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321500 5019 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 04:04:05.332918 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321503 5019 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 04:04:05.332918 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321506 5019 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 04:04:05.332918 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321508 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 04:04:05.332918 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321511 5019 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 04:04:05.332918 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321513 5019 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 04:04:05.332918 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321516 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 04:04:05.332918 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321519 5019 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 04:04:05.332918 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321521 5019 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 04:04:05.333619 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321526 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 04:04:05.333619 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321530 5019 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 04:04:05.333619 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321534 5019 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 04:04:05.333619 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321537 5019 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 04:04:05.333619 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321540 5019 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 04:04:05.333619 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321544 5019 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 04:04:05.333619 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321546 5019 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 04:04:05.333619 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321549 5019 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 04:04:05.333619 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321552 5019 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 04:04:05.333619 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321554 5019 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 04:04:05.333619 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321557 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 04:04:05.333619 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321560 5019 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 04:04:05.333619 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321563 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 04:04:05.333619 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321566 5019 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 04:04:05.333619 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321568 5019 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 04:04:05.333619 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.321595 5019 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 04:04:05.334087 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321634 5019 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 04:04:05.334087 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321640 5019 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 04:04:05.334087 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321644 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 04:04:05.334087 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321647 5019 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 04:04:05.334087 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321650 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 04:04:05.334087 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321653 5019 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 04:04:05.334087 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321655 5019 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 04:04:05.334087 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321658 5019 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 04:04:05.334087 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321661 5019 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 04:04:05.334087 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321664 5019 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 04:04:05.334087 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321666 5019 feature_gate.go:386] unrecognized feature gate: Example Oct 31 04:04:05.334087 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321669 5019 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 04:04:05.334087 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321671 5019 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 04:04:05.334087 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321674 5019 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 04:04:05.334087 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321677 5019 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 04:04:05.334087 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321680 5019 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 04:04:05.334087 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321683 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 04:04:05.334087 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321687 5019 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 04:04:05.334087 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321690 5019 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 04:04:05.334087 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321693 5019 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 04:04:05.334680 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321696 5019 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 04:04:05.334680 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321700 5019 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 04:04:05.334680 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321702 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 04:04:05.334680 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321705 5019 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 04:04:05.334680 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321708 5019 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 04:04:05.334680 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321711 5019 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 04:04:05.334680 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321714 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 04:04:05.334680 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321717 5019 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 04:04:05.334680 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321719 5019 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 04:04:05.334680 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321722 5019 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 04:04:05.334680 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321724 5019 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 04:04:05.334680 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321727 5019 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 04:04:05.334680 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321730 5019 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 04:04:05.334680 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321732 5019 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 04:04:05.334680 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321735 5019 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 04:04:05.334680 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321738 5019 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 04:04:05.334680 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321740 5019 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 04:04:05.334680 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321743 5019 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 04:04:05.334680 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321745 5019 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 04:04:05.335294 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321748 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 04:04:05.335294 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321751 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 04:04:05.335294 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321755 5019 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 04:04:05.335294 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321757 5019 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 04:04:05.335294 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321760 5019 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 04:04:05.335294 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321762 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 04:04:05.335294 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321765 5019 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 04:04:05.335294 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321768 5019 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 04:04:05.335294 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321770 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 04:04:05.335294 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321774 5019 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 04:04:05.335294 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321779 5019 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 04:04:05.335294 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321783 5019 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 04:04:05.335294 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321786 5019 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 04:04:05.335294 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321789 5019 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 04:04:05.335294 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321793 5019 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 04:04:05.335294 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321795 5019 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 04:04:05.335294 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321798 5019 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 04:04:05.335294 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321801 5019 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 04:04:05.335294 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321804 5019 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 04:04:05.335294 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321806 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 04:04:05.336041 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321809 5019 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 04:04:05.336041 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321811 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 04:04:05.336041 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321814 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 04:04:05.336041 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321816 5019 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 04:04:05.336041 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321819 5019 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 04:04:05.336041 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321822 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 04:04:05.336041 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321824 5019 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 04:04:05.336041 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321827 5019 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 04:04:05.336041 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321830 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 04:04:05.336041 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321832 5019 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 04:04:05.336041 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321835 5019 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 04:04:05.336041 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321837 5019 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 04:04:05.336041 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321840 5019 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 04:04:05.336041 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321843 5019 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 04:04:05.336041 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321846 5019 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 04:04:05.336041 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321848 5019 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 04:04:05.336041 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321851 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 04:04:05.336041 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321854 5019 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 04:04:05.336041 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321856 5019 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 04:04:05.336608 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321859 5019 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 04:04:05.336608 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321861 5019 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 04:04:05.336608 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321864 5019 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 04:04:05.336608 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321867 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 04:04:05.336608 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321870 5019 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 04:04:05.336608 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321873 5019 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 04:04:05.336608 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321876 5019 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 04:04:05.336608 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321880 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 04:04:05.336608 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321883 5019 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 04:04:05.336608 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321885 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 04:04:05.336608 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321888 5019 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 04:04:05.336608 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321890 5019 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 04:04:05.336608 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321893 5019 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 04:04:05.336608 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321895 5019 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 04:04:05.336608 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321898 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 04:04:05.336608 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321901 5019 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 04:04:05.336608 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321903 5019 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 04:04:05.336608 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321906 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 04:04:05.336608 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321908 5019 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 04:04:05.336608 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321911 5019 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 04:04:05.337108 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321913 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 04:04:05.337108 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321918 5019 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 04:04:05.337108 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321922 5019 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 04:04:05.337108 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321925 5019 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 04:04:05.337108 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321928 5019 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 04:04:05.337108 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321930 5019 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 04:04:05.337108 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321933 5019 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 04:04:05.337108 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.321935 5019 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 04:04:05.337108 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.326998 5019 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 04:04:05.337108 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.327020 5019 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 04:04:05.337108 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.327045 5019 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 04:04:05.337108 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.327051 5019 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 04:04:05.337108 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327120 5019 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 04:04:05.337108 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327127 5019 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 04:04:05.337108 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327131 5019 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 04:04:05.337108 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327135 5019 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 04:04:05.337108 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327138 5019 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 04:04:05.337108 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327142 5019 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 04:04:05.337720 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327145 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 04:04:05.337720 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327148 5019 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 04:04:05.337720 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327151 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 04:04:05.337720 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327154 5019 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 04:04:05.337720 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327157 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 04:04:05.337720 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327160 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 04:04:05.337720 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327163 5019 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 04:04:05.337720 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327166 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 04:04:05.337720 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327169 5019 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 04:04:05.337720 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327172 5019 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 04:04:05.337720 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327175 5019 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 04:04:05.337720 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327178 5019 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 04:04:05.337720 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327180 5019 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 04:04:05.337720 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327183 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 04:04:05.337720 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327186 5019 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 04:04:05.337720 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327188 5019 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 04:04:05.337720 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327192 5019 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 04:04:05.337720 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327195 5019 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 04:04:05.337720 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327198 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 04:04:05.338196 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327200 5019 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 04:04:05.338196 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327203 5019 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 04:04:05.338196 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327211 5019 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 04:04:05.338196 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327214 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 04:04:05.338196 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327217 5019 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 04:04:05.338196 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327220 5019 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 04:04:05.338196 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327223 5019 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 04:04:05.338196 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327225 5019 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 04:04:05.338196 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327228 5019 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 04:04:05.338196 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327231 5019 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 04:04:05.338196 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327233 5019 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 04:04:05.338196 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327236 5019 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 04:04:05.338196 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327239 5019 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 04:04:05.338196 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327243 5019 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 04:04:05.338196 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327246 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 04:04:05.338196 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327249 5019 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 04:04:05.338196 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327252 5019 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 04:04:05.338196 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327255 5019 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 04:04:05.338196 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327257 5019 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 04:04:05.338681 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327260 5019 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 04:04:05.338681 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327263 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 04:04:05.338681 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327266 5019 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 04:04:05.338681 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327269 5019 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 04:04:05.338681 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327272 5019 feature_gate.go:386] unrecognized feature gate: Example Oct 31 04:04:05.338681 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327275 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 04:04:05.338681 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327277 5019 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 04:04:05.338681 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327280 5019 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 04:04:05.338681 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327282 5019 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 04:04:05.338681 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327285 5019 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 04:04:05.338681 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327288 5019 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 04:04:05.338681 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327291 5019 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 04:04:05.338681 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327294 5019 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 04:04:05.338681 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327297 5019 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 04:04:05.338681 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327300 5019 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 04:04:05.338681 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327303 5019 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 04:04:05.338681 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327306 5019 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 04:04:05.338681 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327309 5019 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 04:04:05.338681 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327312 5019 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 04:04:05.338681 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327315 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 04:04:05.338681 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327317 5019 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 04:04:05.339219 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327320 5019 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 04:04:05.339219 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327323 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 04:04:05.339219 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327325 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 04:04:05.339219 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327328 5019 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 04:04:05.339219 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327342 5019 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 04:04:05.339219 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327345 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 04:04:05.339219 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327348 5019 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 04:04:05.339219 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327351 5019 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 04:04:05.339219 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327354 5019 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 04:04:05.339219 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327357 5019 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 04:04:05.339219 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327360 5019 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 04:04:05.339219 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327363 5019 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 04:04:05.339219 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327366 5019 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 04:04:05.339219 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327369 5019 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 04:04:05.339219 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327372 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 04:04:05.339219 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327375 5019 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 04:04:05.339219 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327378 5019 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 04:04:05.339219 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327380 5019 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 04:04:05.339219 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327383 5019 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 04:04:05.339739 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327386 5019 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 04:04:05.339739 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327389 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 04:04:05.339739 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327392 5019 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 04:04:05.339739 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327395 5019 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 04:04:05.339739 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327398 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 04:04:05.339739 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327401 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 04:04:05.339739 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327404 5019 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 04:04:05.339739 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327406 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 04:04:05.339739 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327409 5019 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 04:04:05.339739 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327411 5019 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 04:04:05.339739 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327414 5019 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 04:04:05.339739 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327417 5019 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 04:04:05.339739 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327420 5019 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 04:04:05.339739 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327422 5019 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 04:04:05.339739 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327425 5019 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 04:04:05.339739 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327427 5019 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 04:04:05.339739 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327430 5019 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 04:04:05.339739 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327432 5019 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 04:04:05.339739 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327435 5019 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 04:04:05.339739 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327438 5019 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 04:04:05.340235 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327441 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 04:04:05.340235 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327444 5019 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 04:04:05.340235 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.327477 5019 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 04:04:05.340235 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327645 5019 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 04:04:05.340235 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327652 5019 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 04:04:05.340235 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327656 5019 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 04:04:05.340235 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327660 5019 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 04:04:05.340235 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327663 5019 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 04:04:05.340235 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327668 5019 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 04:04:05.340235 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327671 5019 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 04:04:05.340235 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327674 5019 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 04:04:05.340235 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327677 5019 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 04:04:05.340235 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327680 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 04:04:05.340235 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327683 5019 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 04:04:05.340235 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327686 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 04:04:05.340235 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327689 5019 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 04:04:05.340641 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327691 5019 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 04:04:05.340641 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327694 5019 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 04:04:05.340641 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327697 5019 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 04:04:05.340641 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327700 5019 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 04:04:05.340641 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327703 5019 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 04:04:05.340641 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327706 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 04:04:05.340641 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327708 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 04:04:05.340641 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327712 5019 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 04:04:05.340641 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327714 5019 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 04:04:05.340641 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327717 5019 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 04:04:05.340641 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327720 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 04:04:05.340641 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327723 5019 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 04:04:05.340641 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327725 5019 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 04:04:05.340641 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327728 5019 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 04:04:05.340641 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327730 5019 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 04:04:05.340641 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327733 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 04:04:05.340641 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327736 5019 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 04:04:05.340641 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327739 5019 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 04:04:05.340641 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327741 5019 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 04:04:05.341187 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327744 5019 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 04:04:05.341187 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327747 5019 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 04:04:05.341187 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327750 5019 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 04:04:05.341187 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327752 5019 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 04:04:05.341187 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327755 5019 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 04:04:05.341187 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327759 5019 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 04:04:05.341187 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327765 5019 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 04:04:05.341187 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327768 5019 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 04:04:05.341187 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327771 5019 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 04:04:05.341187 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327774 5019 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 04:04:05.341187 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327777 5019 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 04:04:05.341187 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327780 5019 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 04:04:05.341187 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327783 5019 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 04:04:05.341187 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327785 5019 feature_gate.go:386] unrecognized feature gate: Example Oct 31 04:04:05.341187 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327788 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 04:04:05.341187 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327791 5019 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 04:04:05.341187 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327793 5019 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 04:04:05.341187 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327796 5019 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 04:04:05.341187 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327799 5019 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 04:04:05.341187 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327801 5019 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 04:04:05.341697 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327804 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 04:04:05.341697 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327807 5019 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 04:04:05.341697 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327810 5019 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 04:04:05.341697 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327812 5019 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 04:04:05.341697 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327815 5019 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 04:04:05.341697 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327818 5019 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 04:04:05.341697 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327820 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 04:04:05.341697 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327823 5019 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 04:04:05.341697 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327825 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 04:04:05.341697 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327828 5019 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 04:04:05.341697 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327830 5019 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 04:04:05.341697 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327833 5019 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 04:04:05.341697 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327836 5019 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 04:04:05.341697 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327838 5019 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 04:04:05.341697 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327841 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 04:04:05.341697 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327843 5019 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 04:04:05.341697 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327846 5019 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 04:04:05.341697 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327849 5019 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 04:04:05.341697 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327852 5019 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 04:04:05.341697 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327855 5019 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 04:04:05.342212 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327857 5019 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 04:04:05.342212 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327860 5019 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 04:04:05.342212 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327866 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 04:04:05.342212 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327869 5019 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 04:04:05.342212 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327871 5019 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 04:04:05.342212 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327875 5019 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 04:04:05.342212 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327878 5019 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 04:04:05.342212 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327881 5019 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 04:04:05.342212 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327883 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 04:04:05.342212 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327886 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 04:04:05.342212 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327889 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 04:04:05.342212 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327892 5019 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 04:04:05.342212 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327894 5019 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 04:04:05.342212 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327897 5019 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 04:04:05.342212 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327900 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 04:04:05.342212 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327902 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 04:04:05.342212 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327905 5019 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 04:04:05.342212 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327907 5019 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 04:04:05.342212 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327910 5019 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 04:04:05.342829 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327913 5019 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 04:04:05.342829 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327916 5019 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 04:04:05.342829 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327918 5019 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 04:04:05.342829 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327921 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 04:04:05.342829 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327923 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 04:04:05.342829 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327926 5019 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 04:04:05.342829 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327929 5019 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 04:04:05.342829 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327933 5019 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 04:04:05.342829 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327937 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 04:04:05.342829 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327940 5019 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 04:04:05.342829 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327943 5019 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 04:04:05.342829 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327946 5019 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 04:04:05.342829 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327948 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 04:04:05.342829 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327951 5019 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 04:04:05.342829 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.327954 5019 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 04:04:05.342829 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.327986 5019 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 04:04:05.343234 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328019 5019 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 04:04:05.343234 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328024 5019 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 04:04:05.343234 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328026 5019 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 04:04:05.343234 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328031 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 04:04:05.343234 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328034 5019 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 04:04:05.343234 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328037 5019 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 04:04:05.343234 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328039 5019 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 04:04:05.343234 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328042 5019 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 04:04:05.343234 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328045 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 04:04:05.343234 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328047 5019 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 04:04:05.343234 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328050 5019 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 04:04:05.343234 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328053 5019 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 04:04:05.343234 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328056 5019 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 04:04:05.343234 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328059 5019 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 04:04:05.343234 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328062 5019 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 04:04:05.343234 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328064 5019 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 04:04:05.343234 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328067 5019 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 04:04:05.343234 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328071 5019 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 04:04:05.343234 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328075 5019 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 04:04:05.343729 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328078 5019 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 04:04:05.343729 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328081 5019 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 04:04:05.343729 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328084 5019 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 04:04:05.343729 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328087 5019 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 04:04:05.343729 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328089 5019 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 04:04:05.343729 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328092 5019 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 04:04:05.343729 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328094 5019 feature_gate.go:386] unrecognized feature gate: Example Oct 31 04:04:05.343729 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328097 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 04:04:05.343729 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328100 5019 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 04:04:05.343729 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328102 5019 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 04:04:05.343729 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328105 5019 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 04:04:05.343729 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328110 5019 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 04:04:05.343729 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328114 5019 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 04:04:05.343729 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328117 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 04:04:05.343729 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328120 5019 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 04:04:05.343729 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328123 5019 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 04:04:05.343729 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328126 5019 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 04:04:05.343729 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328129 5019 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 04:04:05.343729 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328132 5019 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 04:04:05.343729 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328135 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 04:04:05.344322 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328137 5019 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 04:04:05.344322 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328140 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 04:04:05.344322 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328144 5019 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 04:04:05.344322 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328148 5019 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 04:04:05.344322 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328152 5019 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 04:04:05.344322 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328156 5019 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 04:04:05.344322 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328161 5019 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 04:04:05.344322 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328165 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 04:04:05.344322 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328169 5019 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 04:04:05.344322 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328174 5019 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 04:04:05.344322 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328178 5019 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 04:04:05.344322 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328181 5019 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 04:04:05.344322 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328184 5019 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 04:04:05.344322 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328187 5019 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 04:04:05.344322 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328189 5019 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 04:04:05.344322 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328192 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 04:04:05.344322 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328195 5019 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 04:04:05.344322 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328197 5019 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 04:04:05.344322 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328200 5019 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 04:04:05.345096 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328203 5019 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 04:04:05.345096 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328206 5019 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 04:04:05.345096 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328208 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 04:04:05.345096 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328211 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 04:04:05.345096 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328214 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 04:04:05.345096 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328218 5019 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 04:04:05.345096 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328221 5019 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 04:04:05.345096 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328224 5019 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 04:04:05.345096 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328227 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 04:04:05.345096 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328230 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 04:04:05.345096 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328233 5019 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 04:04:05.345096 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328235 5019 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 04:04:05.345096 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328238 5019 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 04:04:05.345096 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328240 5019 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 04:04:05.345096 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328243 5019 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 04:04:05.345096 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328246 5019 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 04:04:05.345096 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328248 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 04:04:05.345096 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328251 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 04:04:05.345096 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328253 5019 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 04:04:05.345842 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328256 5019 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 04:04:05.345842 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328260 5019 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 04:04:05.345842 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328263 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 04:04:05.345842 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328267 5019 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 04:04:05.345842 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328270 5019 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 04:04:05.345842 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328273 5019 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 04:04:05.345842 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328275 5019 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 04:04:05.345842 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328278 5019 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 04:04:05.345842 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328280 5019 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 04:04:05.345842 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328283 5019 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 04:04:05.345842 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328285 5019 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 04:04:05.345842 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328288 5019 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 04:04:05.345842 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328291 5019 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 04:04:05.345842 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328293 5019 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 04:04:05.345842 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328296 5019 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 04:04:05.345842 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328299 5019 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 04:04:05.345842 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328301 5019 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 04:04:05.345842 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328304 5019 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 04:04:05.345842 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328307 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 04:04:05.346641 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328310 5019 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 04:04:05.346641 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328312 5019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 04:04:05.346641 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328315 5019 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 04:04:05.346641 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328318 5019 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 04:04:05.346641 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328320 5019 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 04:04:05.346641 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328323 5019 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 04:04:05.346641 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328326 5019 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 04:04:05.346641 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328344 5019 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 04:04:05.346641 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328348 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 04:04:05.346641 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:05.328352 5019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 04:04:05.346641 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.328417 5019 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 04:04:05.346641 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.329497 5019 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 31 04:04:05.346641 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.329571 5019 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 31 04:04:05.346641 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.329751 5019 server.go:1014] "Starting client certificate rotation" Oct 31 04:04:05.346641 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.329912 5019 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 04:04:05.346641 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.329982 5019 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 03:58:23 +0000 UTC" deadline="2028-04-09 13:38:39.406275696 +0000 UTC" Oct 31 04:04:05.347415 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.330019 5019 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12633h34m34.076259148s" Oct 31 04:04:05.347415 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.330364 5019 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 04:04:05.347415 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.331543 5019 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 04:04:05.347415 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.333118 5019 log.go:25] "Validated CRI v1 runtime API" Oct 31 04:04:05.347415 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.335009 5019 log.go:25] "Validated CRI v1 image API" Oct 31 04:04:05.347415 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.335789 5019 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 04:04:05.347415 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.338075 5019 fs.go:135] Filesystem UUIDs: map[12a891d7-462b-47be-9115-93f445a1f6c2:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 dcde1bc0-f106-427f-b4b7-e4b21d62a9c3:/dev/nvme0n1p4] Oct 31 04:04:05.347812 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.338098 5019 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/071ca9d03626e2d4a643c2e58587db9b4c38cbcd00e223f98624547c9b878714/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/071ca9d03626e2d4a643c2e58587db9b4c38cbcd00e223f98624547c9b878714/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0a5cf0a0e02d01bfcf84d57721f549255cfa89ccd68b088a32c3604ad84eb68c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0a5cf0a0e02d01bfcf84d57721f549255cfa89ccd68b088a32c3604ad84eb68c/userdata/shm major:0 minor:342 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/12f85ecffbe0e7f649e1d9be59d8f02d7aa3cb47bc0552465a340b6878f0c950/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/12f85ecffbe0e7f649e1d9be59d8f02d7aa3cb47bc0552465a340b6878f0c950/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/22e82066481b6d476e82631fdd12228718bd88c970a6bc27a91876d0d46f26e5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/22e82066481b6d476e82631fdd12228718bd88c970a6bc27a91876d0d46f26e5/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2c9ecfdfa891ea5b3d95c594421524a9d34def2e9acce4be2264b5af5d8c2d3b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2c9ecfdfa891ea5b3d95c594421524a9d34def2e9acce4be2264b5af5d8c2d3b/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2df9b248146c1ec79d4800aadd14a9443d89d935854efcaaaf398ce7d67e4e2b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2df9b248146c1ec79d4800aadd14a9443d89d935854efcaaaf398ce7d67e4e2b/userdata/shm major:0 minor:225 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/46f399d545a61f1c153a8737b0478561959296851004ac253536beb401acd06a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/46f399d545a61f1c153a8737b0478561959296851004ac253536beb401acd06a/userdata/shm major:0 minor:344 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/674c8c743183843c2e2fee5053ae52e4d2519342fefad52eadb490e1c7f6aba9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/674c8c743183843c2e2fee5053ae52e4d2519342fefad52eadb490e1c7f6aba9/userdata/shm major:0 minor:89 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7b7497670421e46a8e33c6408131a15a55c8bd6d7eee75bb68d86379ca727a32/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7b7497670421e46a8e33c6408131a15a55c8bd6d7eee75bb68d86379ca727a32/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/828cacb0290babc13ca9b7a25aebe8bc490f563d05ae84a04adfc94da818208f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/828cacb0290babc13ca9b7a25aebe8bc490f563d05ae84a04adfc94da818208f/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8a6619e28986b25f93c7224377a9cf5bac47e19970b0c0673ba89bff2552643f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8a6619e28986b25f93c7224377a9cf5bac47e19970b0c0673ba89bff2552643f/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8b5a131b67071cd06f8262e388f9074e8564dcdcb559a77ab17851be4682e542/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8b5a131b67071cd06f8262e388f9074e8564dcdcb559a77ab17851be4682e542/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/957817c30f358892a8c5e80b384def14d7201c419b27bbef2a4e0f9beb0c247d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/957817c30f358892a8c5e80b384def14d7201c419b27bbef2a4e0f9beb0c247d/userdata/shm major:0 minor:349 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a582b14d9cf72583d5e2976716d25832aab83f4ac2ffd21642c4c87ca9dab847/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a582b14d9cf72583d5e2976716d25832aab83f4ac2ffd21642c4c87ca9dab847/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a599fe68f82ea190e75d07f1f172d52fc495af94b489a72947ed0658320f0e77/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a599fe68f82ea190e75d07f1f172d52fc495af94b489a72947ed0658320f0e77/userdata/shm major:0 minor:334 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a6569749975018a518d8014ad4482d3304d6a757a26850177414e8d03cd3f031/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a6569749975018a518d8014ad4482d3304d6a757a26850177414e8d03cd3f031/userdata/shm major:0 minor:75 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a865c4d77ec5eae81f238b3cbe27af91d7b1541db6173ecd62ad362bd8c7908f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a865c4d77ec5eae81f238b3cbe27af91d7b1541db6173ecd62ad362bd8c7908f/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ac6d070cb1c7f393c8567c4abe16bfbee71c7ddd0023ec00c4e9ed438d34fefb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ac6d070cb1c7f393c8567c4abe16bfbee71c7ddd0023ec00c4e9ed438d34fefb/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b238274377c0617b219aa1379467d63523e3e1d10047202c1b0e8e66268b84e9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b238274377c0617b219aa1379467d63523e3e1d10047202c1b0e8e66268b84e9/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b470b2151e3fb3301b0be2d05d98b6088a17b10108862e8c5c769f40f4023df2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b470b2151e3fb3301b0be2d05d98b6088a17b10108862e8c5c769f40f4023df2/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c04e1f5e890b4fc9f734848a734ebcb4a6352ad54b9fba27f92545850d9f863c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c04e1f5e890b4fc9f734848a734ebcb4a6352ad54b9fba27f92545850d9f863c/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cdd3ae8de99afb7e468a104958df90918f2055ac5de8be3f64191f0dca24df0b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cdd3ae8de99afb7e468a104958df90918f2055ac5de8be3f64191f0dca24df0b/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d7f8a4628c1cc4450b1ba7ae4188a8c9d68202dd4522ab53d6fe4936506ecd6f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d7f8a4628c1cc4450b1ba7ae4188a8c9d68202dd4522ab53d6fe4936506ecd6f/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dc34e5c3243961a8722af6ca8afcb8a4e8cfbed45618bc22019ed5bb7c849c95/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dc34e5c3243961a8722af6ca8afcb8a4e8cfbed45618bc22019ed5bb7c849c95/userdata/shm major:0 minor:235 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eed219fb3e1a90717b9fa9c013bab0e08920a8e517198614557b82a63f6a048d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eed219fb3e1a90717b9fa9c013bab0e08920a8e517198614557b82a63f6a048d/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f033db6f8846c03553449c030e1787fcd20f1d39dd5254fbe9b4c9ca77a535f1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f033db6f8846c03553449c030e1787fcd20f1d39dd5254fbe9b4c9ca77a535f1/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/07320d54-b911-4157-aaf5-2517187d745c/volumes/kubernetes.io~projected/kube-api-access-x4pc9:{mountpoint:/var/lib/kubelet/pods/07320d54-b911-4157-aaf5-2517187d745c/volumes/kubernetes.io~projected/kube-api-access-x4pc9 major:0 minor:333 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/197dee13-748f-4c6d-acc5-1793b5cee80a/volumes/kubernetes.io~projected/kube-api-access-7cq9b:{mountpoint:/var/lib/kubelet/pods/197dee13-748f-4c6d-acc5-1793b5cee80a/volumes/kubernetes.io~projected/kube-api-access-7cq9b major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2050cf2e-0f83-43b3-bb85-7bd4ac42a473/volumes/kubernetes.io~projected/kube-api-access-k52nk:{mountpoint:/var/lib/kubelet/pods/2050cf2e-0f83-43b3-bb85-7bd4ac42a473/volumes/kubernetes.io~projected/kube-api-access-k52nk major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31221235-f863-445b-a5ca-375aae905d38/volumes/kubernetes.io~projected/kube-api-access-x5nf8:{mountpoint:/var/lib/kubelet/pods/31221235-f863-445b-a5ca-375aae905d38/volumes/kubernetes.io~projected/kube-api-access-x5nf8 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3945ff65-e877-4372-9334-3968c68b7295/volumes/kubernetes.io~projected/kube-api-access-fqx72:{mountpoint:/var/lib/kubelet/pods/3945ff65-e877-4372-9334-3968c68b7295/volumes/kubernetes.io~projected/kube-api-access-fqx72 major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3945ff65-e877-4372-9334-3968c68b7295/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/3945ff65-e877-4372-9334-3968c68b7295/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a05c43f-e2db-44c0-b9bd-4bdabca92a98/volumes/kubernetes.io~projected/kube-api-access-vhkfw:{mountpoint:/var/lib/kubelet/pods/3a05c43f-e2db-44c0-b9bd-4bdabca92a98/volumes/kubernetes.io~projected/kube-api-access-vhkfw major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/48309c9d-5f34-49bf-b55b-f74959984e0a/volumes/kubernetes.io~projected/kube-api-access-kt6hk:{mountpoint:/var/lib/kubelet/pods/48309c9d-5f34-49bf-b55b-f74959984e0a/volumes/kubernetes.io~projected/kube-api-access-kt6hk major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/49140fa3-6ce2-41ea-8b6b-e25257efaf3e/volumes/kubernetes.io~projected/kube-api-access-6vrvs:{mountpoint:/var/lib/kubelet/pods/49140fa3-6ce2-41ea-8b6b-e25257efaf3e/volumes/kubernetes.io~projected/kube-api-access-6vrvs major:0 minor:337 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/57b47c7b-9c84-4888-be69-64afca28a8c0/volumes/kubernetes.io~projected/kube-api-access-jj5t7:{mountpoint:/var/lib/kubelet/pods/57b47c7b-9c84-4888-be69-64afca28a8c0/volumes/kubernetes.io~projected/kube-api-access-jj5t7 major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b6d69de-a860-4c82-a03f-23ab7905a5d7/volumes/kubernetes.io~projected/kube-api-access-7w6z5:{mountpoint:/var/lib/kubelet/pods/5b6d69de-a860-4c82-a03f-23ab7905a5d7/volumes/kubernetes.io~projected/kube-api-access-7w6z5 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6d336e6e-feb0-40b2-a625-0e21d25bfac5/volumes/kubernetes.io~projected/kube-api-access-9mpvl:{mountpoint:/var/lib/kubelet/pods/6d336e6e-feb0-40b2-a625-0e21d25bfac5/volumes/kubernetes.io~projected/kube-api-access-9mpvl major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7f6cd22a-499a-4c35-b408-b2aa9b78620d/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/7f6cd22a-499a-4c35-b408-b2aa9b78620d/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7f6cd22a-499a-4c35-b408-b2aa9b78620d/volumes/kubernetes.io~projected/kube-api-access-phq2f:{mountpoint:/var/lib/kubelet/pods/7f6cd22a-499a-4c35-b408-b2aa9b78620d/volumes/kubernetes.io~projected/kube-api-access-phq2f major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7f6cd22a-499a-4c35-b408-b2aa9b78620d/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/7f6cd22a-499a-4c35-b408-b2aa9b78620d/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/89ea384b-705f-47a9-8df6-d3c266ed6761/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/89ea384b-705f-47a9-8df6-d3c266ed6761/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93beb8b8-c9e7-4cce-8baf-8497b971e193/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/93beb8b8-c9e7-4cce-8baf-8497b971e193/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93beb8b8-c9e7-4cce-8baf-8497b971e193/volumes/kubernetes.io~projected/kube-api-access-25x8r:{mountpoint:/var/lib/kubelet/pods/93beb8b8-c9e7-4cce-8baf-8497b971e193/volumes/kubernetes.io~projected/kube-api-access-25x8r major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93beb8b8-c9e7-4cce-8baf-8497b971e193/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/93beb8b8-c9e7-4cce-8baf-8497b971e193/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93beb8b8-c9e7-4cce-8baf-8497b971e193/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/93beb8b8-c9e7-4cce-8baf-8497b971e193/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/944c9831-b2fa-4f58-bba2-93288793b125/volumes/kubernetes.io~projected/kube-api-access-ghxbr:{mountpoint:/var/lib/kubelet/pods/944c9831-b2fa-4f58-bba2-93288793b125/volumes/kubernetes.io~projected/kube-api-access-ghxbr major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/944c9831-b2fa-4f58-bba2-93288793b125/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/944c9831-b2fa-4f58-bba2-93288793b125/volumes/kubernetes.io~secret/serving-cert major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/97c312b1-4ca5-49c8-8c8d-d247c0fa14ac/volumes/kubernetes.io~projected/kube-api-access-2f5pv:{mountpoint:/var/lib/kubelet/pods/97c312b1-4ca5-49c8-8c8d-d247c0fa14ac/volumes/kubernetes.io~projected/kube-api-access-2f5pv major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b7bf8e5-7877-46e0-88dd-a3503b7639c5/volumes/kubernetes.io~projected/kube-api-access-frshf:{mountpoint:/var/lib/kubelet/pods/9b7bf8e5-7877-46e0-88dd-a3503b7639c5/volumes/kubernetes.io~projected/kube-api-access-frshf major:0 minor:320 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c8cb360-ff57-41b1-8244-47588703cba5/volumes/kubernetes.io~projected/kube-api-access-9jbhv:{mountpoint:/var/lib/kubelet/pods/9c8cb360-ff57-41b1-8244-47588703cba5/volumes/kubernetes.io~projected/kube-api-access-9jbhv major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a173790b-6c13-45a1-a361-3fe51844b4c2/volumes/kubernetes.io~projected/kube-api-access-ggq72:{mountpoint:/var/lib/kubelet/pods/a173790b-6c13-45a1-a361-3fe51844b4c2/volumes/kubernetes.io~projected/kube-api-access-ggq72 major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2ce92f3-5b05-4c89-9d53-a57bc85b17ef/volumes/kubernetes.io~projected/kube-api-access-7ll7m:{mountpoint:/var/lib/kubelet/pods/a2ce92f3-5b05-4c89-9d53-a57bc85b17ef/volumes/kubernetes.io~projected/kube-api-access-7ll7m major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2ce92f3-5b05-4c89-9d53-a57bc85b17ef/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/a2ce92f3-5b05-4c89-9d53-a57bc85b17ef/volumes/kubernetes.io~secret/cert major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aaae68b8-1654-474f-9e74-45d1e2025d20/volumes/kubernetes.io~projected/kube-api-access-nd5qc:{mountpoint:/var/lib/kubelet/pods/aaae68b8-1654-474f-9e74-45d1e2025d20/volumes/kubernetes.io~projected/kube-api-access-nd5qc major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aef43d62-9761-4262-8183-01b9890462a7/volumes/kubernetes.io~projected/kube-api-access-kjq6x:{mountpoint:/var/lib/kubelet/pods/aef43d62-9761-4262-8183-01b9890462a7/volumes/kubernetes.io~projected/kube-api-access-kjq6x major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b541554a-0460-4716-9f14-f7a46567465c/volumes/kubernetes.io~projected/kube-api-access-qlg9n:{mountpoint:/var/lib/kubelet/pods/b541554a-0460-4716-9f14-f7a46567465c/volumes/kubernetes.io~projected/kube-api-access-qlg9n major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be0026cb-89bc-413a-bfe3-523d980397fa/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/be0026cb-89bc-413a-bfe3-523d980397fa/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be0026cb-89bc-413a-bfe3-523d980397fa/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/be0026cb-89bc-413a-bfe3-523d980397fa/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be0026cb-89bc-413a-bfe3-523d980397fa/volumes/kubernetes.io~projected/kube-api-access-dn7px:{mountpoint:/var/lib/kubelet/pods/be0026cb-89bc-413a-bfe3-523d980397fa/volumes/kubernetes.io~projected/kube-api-access-dn7px major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1560abd-3afc-4f94-b0bb-ac7ef8b2464a/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/c1560abd-3afc-4f94-b0bb-ac7ef8b2464a/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c3c90e6f-df5e-4c17-bec4-554280a8e8e7/volumes/kubernetes.io~projected/kube-api-access-7mtpj:{mountpoint:/var/lib/kubelet/pods/c3c90e6f-df5e-4c17-bec4-554280a8e8e7/volumes/kubernetes.io~projected/kube-api-access-7mtpj major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c3c90e6f-df5e-4c17-bec4-554280a8e8e7/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/c3c90e6f-df5e-4c17-bec4-554280a8e8e7/volumes/kubernetes.io~secret/serving-cert major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c4a091a2-58d2-4b05-a968-d16fe1bd9d40/volumes/kubernetes.io~projected/kube-api-access-nxlpp:{mountpoint:/var/lib/kubelet/pods/c4a091a2-58d2-4b05-a968-d16fe1bd9d40/volumes/kubernetes.io~projected/kube-api-access-nxlpp major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c9eeaad0-e2e5-4b90-a0af-2d254c787fac/volumes/kubernetes.io~projected/kube-api-access-57gkt:{mountpoint:/var/lib/kubelet/pods/c9eeaad0-e2e5-4b90-a0af-2d254c787fac/volumes/kubernetes.io~projected/kube-api-access-57gkt major:0 minor:352 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c9eeaad0-e2e5-4b90-a0af-2d254c787fac/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/c9eeaad0-e2e5-4b90-a0af-2d254c787fac/volumes/kubernetes.io~secret/signing-key major:0 minor:351 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce86be55-219f-46fb-bff6-6dcf87a40ef4/volumes/kubernetes.io~projected/kube-api-access-x5hsv:{mountpoint:/var/lib/kubelet/pods/ce86be55-219f-46fb-bff6-6dcf87a40ef4/volumes/kubernetes.io~projected/kube-api-access-x5hsv major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d22707c7-21fa-4511-b0b1-869b41fb2351/volumes/kubernetes.io~projected/kube-api-access-v6xnt:{mountpoint:/var/lib/kubelet/pods/d22707c7-21fa-4511-b0b1-869b41fb2351/volumes/kubernetes.io~projected/kube-api-access-v6xnt major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d22707c7-21fa-4511-b0b1-869b41fb2351/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/d22707c7-21fa-4511-b0b1-869b41fb2351/volumes/kubernetes.io~secret/default-certificate major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d22707c7-21fa-4511-b0b1-869b41fb2351/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/d22707c7-21fa-4511-b0b1-869b41fb2351/volumes/kubernetes.io~secret/stats-auth major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e667439c-7121-4c5f-84fe-b36f2391dc66/volumes/kubernetes.io~projected/kube-api-access-wddq5:{mountpoint:/var/lib/kubelet/pods/e667439c-7121-4c5f-84fe-b36f2391dc66/volumes/kubernetes.io~projected/kube-api-access-wddq5 major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e667439c-7121-4c5f-84fe-b36f2391dc66/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/e667439c-7121-4c5f-84fe-b36f2391dc66/volumes/kubernetes.io~secret/serving-cert major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22/volumes/kubernetes.io~projected/kube-api-access-xc2vq:{mountpoint:/var/lib/kubelet/pods/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22/volumes/kubernetes.io~projected/kube-api-access-xc2vq major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb656990-e663-4fea-8733-18ded9222a79/volumes/kubernetes.io~projected/kube-api-access-hsrbd:{mountpoint:/var/lib/kubelet/pods/fb656990-e663-4fea-8733-18ded9222a79/volumes/kubernetes.io~projected/kube-api-access-hsrbd major:0 minor:233 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/b77b040d83ac260fcabc48642a6e33da2467a84f920e6d32594216516462f10f/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/62723a5f25b305464bd5efa6d6ee76bb2c0428857810eba626d739814ec64b77/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/7c0d510d58d310002b235203064f0b10ceee89d45570dcffa24bb4ab8e255dab/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/48b9b84775190b9202c210bf6c081afa5ed8d5345b02b53b0563c1e8130209aa/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/df867c86a1c8abe540b5029be68799719447e61c15bb59cfb72e1be0cc03abda/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/889706c160bcd337546d79522dc021df723e41314d00d71a78ea9cc7d24e726c/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/5a639399923430013cb35a51655d4146e8212fa96c54b33706b700e90e4ec391/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/75e89f1c4c821e998ee5b089f98d2601d767660b4542feb67d569967116b5438/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/b0d66e865a3d284c761afdb22610fb42a7b9af912a4cb62781655a4e79dd08e8/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/f4ccd14373ffaa7cdc228e661bd5a46bf7faf644483c0e7dd859f06ec9a90a60/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/224aefcb17be2d823470f79e07fde44687e644d0d7dc330feee08c79b999632a/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/ff47aad6ace8680ebd08d6526a6354600ea41592b64e68a1728548b6c0544ccd/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/f67491bb99a66e29a3afdbf27bf4ce7072894e8a8c591d122192a8c12074e53a/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/97aaa0c22fd1134220d21d372898fb4cc61435145c1ef30b86c929adeafa2a3d/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/db63bee56dd4b2ddbba5489dd8cb030cb02b6420fc919fb6b9d682fd5c173d06/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/470276f37aeb0c1ba801cf183a1226b87915f85037fd76ae5283187c89866063/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/27c35803a0bda9153ac77dcdce21ff04825696dd41f3c9b09d186b3918eadb0e/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/7272ce7c75945f50e1c96fa2eb6b17b33836eacec8ce40700cdff427be2b9d84/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/7a86beb51b00ac7516b9abd9e8be9b114b1d0bc691cc765327119aed6672dc1f/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/0b2a8c3406d21432141ad428b98a2a3467f191ecb15c6f770fafc1b4cd5e4635/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/d37db847d4150bb65f8e5160ba026c6a809037bd0fbeb1e5fdfa8006371fce24/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/6412df2e463f08f014bd1a7c9c456dd084f3dd8064b1163bbdf924886a79bf7c/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/e4c0620bcb26bc489c8de9f068ef12841cf62f5de198517cba8733343e362e3a/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/4a3fb7bc242f5ea01f96aa7b38dfc92ec6f48d74d30cc4353d5553d16b475579/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/5d676a70a29f242cf8f1b48eaf04d5c8bd22de5a99ff53b7d35fecc4aee81d5b/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/a0e76b18df3ddcb5c0cc5361ef2674a2183d87fc594d59054b65bc940277f267/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/764085af857ce7ffcec62a630694a32c57dd3fa3b13d76845b7fedd9961b3392/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/5b8c687c51ef6cf471dc4e8ad5bacfcb9b64ba9bc408c524e5edf3893f0e9b76/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/45365b2e1a383877f876090a8ef72c077e535667d3507072d6704924f32e6337/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/1dfc756694c70b030ab82d9d17119ac2f7e4401d06d9d4909c965e57e68eded3/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/e07ef9539900d81b2185bdd476014d7b7a3e5e327a1f33457f6e07de7f5e4831/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/3faee3d2f0b4caf776f3dd2cd7b5f96efd3347e682507e4fb1e2b4233a1ea1fc/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-281:{mountpoint:/var/lib/containers/storage/overlay/b0b58c1b646f60a292c384c2d7da5675a6dd118b92b2808c4287437f7a83efdb/merged major:0 minor:281 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/79cd2a24abe6cb62e445de7b61d2fe9ce260fd73c3d2a478e2aaf0b49b22aca6/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/27e87206fbae1837678c74ad93da355010a50dfdc842f49d269358e267b91148/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/ea9d2a08ff0c091dfd836e766eeeb7c4cbc4a6e5f946429948cd529ea2716e0e/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/5b7560a3b788dce882b30023c23330366575c9aaabead4f8a81435c6e72c8712/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/0b2d168cc130479975511488603b1bdd3da49519c0de1ea77f46b56fd5b4b6d8/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/34b20ee6dd0e9a1a32d073dbf2423d1717289092d7804ca4f6f2b3cdd57eeebe/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/eae95454b412b2e245f06272b62411f7ff8b2cd26e1c2209c8aa0d8d36ac104c/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/9dad840ecd1dccd4378e9bc85262e200649722b4d78d356ee576f5b18c8cec77/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/6e1474c0a19cdb0e4385ee307627982d213711ea0759ee8fa7f9c0a8ca86bd66/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/63f9cfc5ed4e16af25be9294e7f40e04da354b61e6da64498403ad2dfee64c74/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/6cea23341aa0dc5edd9e55a9b9a6a45b08f0c56f118d4f5bdf9481dd6273d9cd/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/7cd419583b274726e480ff168d4fb2bdb570d283accf3c18a21b0d6fdc75f042/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/d2effe71bc528eb4dc2be0f4f5c893d4ed95f509558697235ca0344ae48ac8f3/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/0d8ea2a147df6bae81ae3ca4824065525daa8031bafc2c21134dcf1c6d7e7263/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/a197b32e860bd0c141961d043970ae61e2f7d571809f86a7d150a44ec15c9eb1/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/c117dab68e0a1c6bb66b35c5983fbf5a577c73b2acde0ad75cab5b95dd6d606b/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/17c7d19c92b10a65eeeb059dae0ef0876bbc4479082e91b1e7f3b97d969f9897/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/90ac162de7cefe96e006d54bda0a61dbebfa871567982e626d5dca8454419fc1/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/1ec381b0f648fe328d23d9405aec4e6a9fc1d9b6c12e64bfd62de0b0b39a117f/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/5517ed1fd5f063bceda4d08d1f3c146153b2b67a48004e65d0ab3c4bcd65896b/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/370cbf33570f3b06e4e11817d3825d784ee1fc3af2047064b5be5bf4875ee34a/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/37b6eb802607baa32c86592ab0f28eba4628ca9ee660e14a9aecccc77c7f121c/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/3db2b7a1fb30a91e11e7013ccf20814479e28f06214098588015dffbae4ca4e6/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-77:{mountpoint:/var/lib/containers/storage/overlay/388399692a634321232b07fd80c6b3f07482adaa7c7aea909a96f3827a11b597/merged major:0 minor:77 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/11c86420e71a415c49021724a85d6cd663787b3827b3989a383d917f551f5cc3/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/2745683c34b5998547710cde3cf291a7b3ffda3a4e6f2dd5900907338e183edf/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/3ec416ae60d0cbf2d51b7204bc246aa66f7f00d6a4af82693d0fd50e18b86a30/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/3f57284410bead4a030908d398e0b15535a0d6e7b4b7a5508fff290c569c08fe/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/cb2581375b2697814f0c1d1fd52df96fe2678b5f972c98e88a6f75c66c267e1e/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 31 04:04:05.349676 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.349247 5019 manager.go:217] Machine: {Timestamp:2026-10-31 04:04:05.348768394 +0000 UTC m=+0.105106426 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3106853 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec269f6bad3041abad834774c74ed6d0 SystemUUID:ec269f6b-ad30-41ab-ad83-4774c74ed6d0 BootID:827ad6cb-9cb9-459b-9e61-877869987660 Filesystems:[{Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0a5cf0a0e02d01bfcf84d57721f549255cfa89ccd68b088a32c3604ad84eb68c/userdata/shm DeviceMajor:0 DeviceMinor:342 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/aaae68b8-1654-474f-9e74-45d1e2025d20/volumes/kubernetes.io~projected/kube-api-access-nd5qc DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/97c312b1-4ca5-49c8-8c8d-d247c0fa14ac/volumes/kubernetes.io~projected/kube-api-access-2f5pv DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/674c8c743183843c2e2fee5053ae52e4d2519342fefad52eadb490e1c7f6aba9/userdata/shm DeviceMajor:0 DeviceMinor:89 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/93beb8b8-c9e7-4cce-8baf-8497b971e193/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e667439c-7121-4c5f-84fe-b36f2391dc66/volumes/kubernetes.io~projected/kube-api-access-wddq5 DeviceMajor:0 DeviceMinor:220 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/93beb8b8-c9e7-4cce-8baf-8497b971e193/volumes/kubernetes.io~projected/kube-api-access-25x8r DeviceMajor:0 DeviceMinor:222 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/22e82066481b6d476e82631fdd12228718bd88c970a6bc27a91876d0d46f26e5/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2df9b248146c1ec79d4800aadd14a9443d89d935854efcaaaf398ce7d67e4e2b/userdata/shm DeviceMajor:0 DeviceMinor:225 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/957817c30f358892a8c5e80b384def14d7201c419b27bbef2a4e0f9beb0c247d/userdata/shm DeviceMajor:0 DeviceMinor:349 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/197dee13-748f-4c6d-acc5-1793b5cee80a/volumes/kubernetes.io~projected/kube-api-access-7cq9b DeviceMajor:0 DeviceMinor:210 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22/volumes/kubernetes.io~projected/kube-api-access-xc2vq DeviceMajor:0 DeviceMinor:213 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c3c90e6f-df5e-4c17-bec4-554280a8e8e7/volumes/kubernetes.io~projected/kube-api-access-7mtpj DeviceMajor:0 DeviceMinor:215 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9b7bf8e5-7877-46e0-88dd-a3503b7639c5/volumes/kubernetes.io~projected/kube-api-access-frshf DeviceMajor:0 DeviceMinor:320 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/5b6d69de-a860-4c82-a03f-23ab7905a5d7/volumes/kubernetes.io~projected/kube-api-access-7w6z5 DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8b5a131b67071cd06f8262e388f9074e8564dcdcb559a77ab17851be4682e542/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6d336e6e-feb0-40b2-a625-0e21d25bfac5/volumes/kubernetes.io~projected/kube-api-access-9mpvl DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a2ce92f3-5b05-4c89-9d53-a57bc85b17ef/volumes/kubernetes.io~projected/kube-api-access-7ll7m DeviceMajor:0 DeviceMinor:231 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7b7497670421e46a8e33c6408131a15a55c8bd6d7eee75bb68d86379ca727a32/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3945ff65-e877-4372-9334-3968c68b7295/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d22707c7-21fa-4511-b0b1-869b41fb2351/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b470b2151e3fb3301b0be2d05d98b6088a17b10108862e8c5c769f40f4023df2/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/828cacb0290babc13ca9b7a25aebe8bc490f563d05ae84a04adfc94da818208f/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/071ca9d03626e2d4a643c2e58587db9b4c38cbcd00e223f98624547c9b878714/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c4a091a2-58d2-4b05-a968-d16fe1bd9d40/volumes/kubernetes.io~projected/kube-api-access-nxlpp DeviceMajor:0 DeviceMinor:214 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c04e1f5e890b4fc9f734848a734ebcb4a6352ad54b9fba27f92545850d9f863c/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aef43d62-9761-4262-8183-01b9890462a7/volumes/kubernetes.io~projected/kube-api-access-kjq6x DeviceMajor:0 DeviceMinor:293 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c9eeaad0-e2e5-4b90-a0af-2d254c787fac/volumes/kubernetes.io~projected/kube-api-access-57gkt DeviceMajor:0 DeviceMinor:352 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2050cf2e-0f83-43b3-bb85-7bd4ac42a473/volumes/kubernetes.io~projected/kube-api-access-k52nk DeviceMajor:0 DeviceMinor:218 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a6569749975018a518d8014ad4482d3304d6a757a26850177414e8d03cd3f031/userdata/shm DeviceMajor:0 DeviceMinor:75 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a865c4d77ec5eae81f238b3cbe27af91d7b1541db6173ecd62ad362bd8c7908f/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a582b14d9cf72583d5e2976716d25832aab83f4ac2ffd21642c4c87ca9dab847/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f033db6f8846c03553449c030e1787fcd20f1d39dd5254fbe9b4c9ca77a535f1/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/c1560abd-3afc-4f94-b0bb-ac7ef8b2464a/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c9eeaad0-e2e5-4b90-a0af-2d254c787fac/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:351 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/89ea384b-705f-47a9-8df6-d3c266ed6761/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/944c9831-b2fa-4f58-bba2-93288793b125/volumes/kubernetes.io~projected/kube-api-access-ghxbr DeviceMajor:0 DeviceMinor:221 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3a05c43f-e2db-44c0-b9bd-4bdabca92a98/volumes/kubernetes.io~projected/kube-api-access-vhkfw DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a173790b-6c13-45a1-a361-3fe51844b4c2/volumes/kubernetes.io~projected/kube-api-access-ggq72 DeviceMajor:0 DeviceMinor:211 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fb656990-e663-4fea-8733-18ded9222a79/volumes/kubernetes.io~projected/kube-api-access-hsrbd DeviceMajor:0 DeviceMinor:233 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3945ff65-e877-4372-9334-3968c68b7295/volumes/kubernetes.io~projected/kube-api-access-fqx72 DeviceMajor:0 DeviceMinor:217 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dc34e5c3243961a8722af6ca8afcb8a4e8cfbed45618bc22019ed5bb7c849c95/userdata/shm DeviceMajor:0 DeviceMinor:235 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7f6cd22a-499a-4c35-b408-b2aa9b78620d/volumes/kubernetes.io~projected/kube-api-access-phq2f DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/57b47c7b-9c84-4888-be69-64afca28a8c0/volumes/kubernetes.io~projected/kube-api-access-jj5t7 DeviceMajor:0 DeviceMinor:209 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b238274377c0617b219aa1379467d63523e3e1d10047202c1b0e8e66268b84e9/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2c9ecfdfa891ea5b3d95c594421524a9d34def2e9acce4be2264b5af5d8c2d3b/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/be0026cb-89bc-413a-bfe3-523d980397fa/volumes/kubernetes.io~projected/kube-api-access-dn7px DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-281 DeviceMajor:0 DeviceMinor:281 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-77 DeviceMajor:0 DeviceMinor:77 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cdd3ae8de99afb7e468a104958df90918f2055ac5de8be3f64191f0dca24df0b/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e667439c-7121-4c5f-84fe-b36f2391dc66/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:206 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c3c90e6f-df5e-4c17-bec4-554280a8e8e7/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:208 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a599fe68f82ea190e75d07f1f172d52fc495af94b489a72947ed0658320f0e77/userdata/shm DeviceMajor:0 DeviceMinor:334 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7f6cd22a-499a-4c35-b408-b2aa9b78620d/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ac6d070cb1c7f393c8567c4abe16bfbee71c7ddd0023ec00c4e9ed438d34fefb/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d22707c7-21fa-4511-b0b1-869b41fb2351/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:205 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:216 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/be0026cb-89bc-413a-bfe3-523d980397fa/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/be0026cb-89bc-413a-bfe3-523d980397fa/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ce86be55-219f-46fb-bff6-6dcf87a40ef4/volumes/kubernetes.io~projected/kube-api-access-x5hsv DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b541554a-0460-4716-9f14-f7a46567465c/volumes/kubernetes.io~projected/kube-api-access-qlg9n DeviceMajor:0 DeviceMinor:219 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eed219fb3e1a90717b9fa9c013bab0e08920a8e517198614557b82a63f6a048d/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9c8cb360-ff57-41b1-8244-47588703cba5/volumes/kubernetes.io~projected/kube-api-access-9jbhv DeviceMajor:0 DeviceMinor:336 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7f6cd22a-499a-4c35-b408-b2aa9b78620d/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/07320d54-b911-4157-aaf5-2517187d745c/volumes/kubernetes.io~projected/kube-api-access-x4pc9 DeviceMajor:0 DeviceMinor:333 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/46f399d545a61f1c153a8737b0478561959296851004ac253536beb401acd06a/userdata/shm DeviceMajor:0 DeviceMinor:344 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/93beb8b8-c9e7-4cce-8baf-8497b971e193/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:212 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/12f85ecffbe0e7f649e1d9be59d8f02d7aa3cb47bc0552465a340b6878f0c950/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/944c9831-b2fa-4f58-bba2-93288793b125/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:207 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d7f8a4628c1cc4450b1ba7ae4188a8c9d68202dd4522ab53d6fe4936506ecd6f/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8a6619e28986b25f93c7224377a9cf5bac47e19970b0c0673ba89bff2552643f/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/31221235-f863-445b-a5ca-375aae905d38/volumes/kubernetes.io~projected/kube-api-access-x5nf8 DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/93beb8b8-c9e7-4cce-8baf-8497b971e193/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d22707c7-21fa-4511-b0b1-869b41fb2351/volumes/kubernetes.io~projected/kube-api-access-v6xnt DeviceMajor:0 DeviceMinor:223 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a2ce92f3-5b05-4c89-9d53-a57bc85b17ef/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:224 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/48309c9d-5f34-49bf-b55b-f74959984e0a/volumes/kubernetes.io~projected/kube-api-access-kt6hk DeviceMajor:0 DeviceMinor:232 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/49140fa3-6ce2-41ea-8b6b-e25257efaf3e/volumes/kubernetes.io~projected/kube-api-access-6vrvs DeviceMajor:0 DeviceMinor:337 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:071ca9d03626e2d MacAddress:6a:66:10:c0:df:6b Speed:10000 Mtu:8901} {Name:0a5cf0a0e02d01b MacAddress:d2:71:da:5a:a0:86 Speed:10000 Mtu:8901} {Name:12f85ecffbe0e7f MacAddress:96:be:c0:dd:81:de Speed:10000 Mtu:8901} {Name:22e82066481b6d4 MacAddress:9a:36:98:9b:00:eb Speed:10000 Mtu:8901} {Name:2df9b248146c1ec MacAddress:ba:36:9d:c6:87:ef Speed:10000 Mtu:8901} {Name:46f399d545a61f1 MacAddress:62:9d:09:13:65:6e Speed:10000 Mtu:8901} {Name:957817c30f35889 MacAddress:42:12:8d:9d:d5:89 Speed:10000 Mtu:8901} {Name:a582b14d9cf7258 MacAddress:e2:8c:d6:9d:8d:c0 Speed:10000 Mtu:8901} {Name:a599fe68f82ea19 MacAddress:8a:be:03:e9:ef:33 Speed:10000 Mtu:8901} {Name:b238274377c0617 MacAddress:aa:40:d8:bb:03:9d Speed:10000 Mtu:8901} {Name:b470b2151e3fb33 MacAddress:5a:b8:99:21:74:a1 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:ff:da:55:17 Speed:0 Mtu:9001} {Name:br-int MacAddress:aa:b1:75:e6:eb:c8 Speed:0 Mtu:8901} {Name:c04e1f5e890b4fc MacAddress:5a:55:3c:f7:a8:05 Speed:10000 Mtu:8901} {Name:d7f8a4628c1cc44 MacAddress:c2:12:39:29:41:78 Speed:10000 Mtu:8901} {Name:dc34e5c3243961a MacAddress:a6:ea:4b:1c:fc:6b Speed:10000 Mtu:8901} {Name:eed219fb3e1a907 MacAddress:72:a6:a6:4b:30:c4 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:ff:da:55:17 Speed:0 Mtu:9001} {Name:f033db6f8846c03 MacAddress:72:76:fe:78:0b:89 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:c2:23:5b:5e:2d:22 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:ce:24:68:47:16:b5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 04:04:05.350034 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.349682 5019 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 04:04:05.350034 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.349754 5019 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 04:04:05.350142 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.350112 5019 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 04:04:05.350463 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.350145 5019 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-57.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 04:04:05.350463 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.350376 5019 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 04:04:05.350463 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.350389 5019 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 04:04:05.350463 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.350404 5019 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 04:04:05.350463 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.350434 5019 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 04:04:05.350463 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.350464 5019 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 04:04:05.350762 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.350706 5019 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 04:04:05.350869 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.350856 5019 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 04:04:05.350920 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.350908 5019 kubelet.go:493] "Attempting to sync node with API server" Oct 31 04:04:05.350971 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.350924 5019 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 04:04:05.350971 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.350939 5019 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 04:04:05.350971 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.350958 5019 kubelet.go:405] "Adding apiserver pod source" Oct 31 04:04:05.350971 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.350970 5019 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 04:04:05.351707 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.351691 5019 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 04:04:05.351707 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.351706 5019 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 04:04:05.352312 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.352293 5019 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 04:04:05.354678 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.354661 5019 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 04:04:05.354678 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.354677 5019 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 04:04:05.354795 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.354785 5019 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 31 04:04:05.355003 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.354992 5019 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 04:04:05.355037 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.355024 5019 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 04:04:05.355136 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.355127 5019 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 04:04:05.355169 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.355141 5019 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 04:04:05.355169 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.355148 5019 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 04:04:05.355169 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.355154 5019 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 04:04:05.355169 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.355160 5019 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 04:04:05.355169 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.355166 5019 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 04:04:05.355319 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.355172 5019 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 04:04:05.355319 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.355178 5019 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 04:04:05.355319 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.355185 5019 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 04:04:05.355319 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.355191 5019 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 04:04:05.355319 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.355205 5019 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 04:04:05.355319 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.355215 5019 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 04:04:05.355319 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.355241 5019 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 04:04:05.355319 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.355247 5019 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 04:04:05.359019 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.358998 5019 server.go:1263] "Started kubelet" Oct 31 04:04:05.359121 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.359054 5019 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 04:04:05.359842 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.359131 5019 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 04:04:05.359842 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.359222 5019 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 04:04:05.359842 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.359580 5019 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 04:04:05.359659 ip-10-0-130-57 systemd[1]: Started Kubernetes Kubelet. Oct 31 04:04:05.365997 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.365964 5019 server.go:317] "Adding debug handlers to kubelet server" Oct 31 04:04:05.366311 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.366293 5019 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 04:04:05.367553 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.367523 5019 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 04:04:05.367681 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.367604 5019 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 03:58:25 +0000 UTC" deadline="2028-04-19 08:54:56.286548087 +0000 UTC" Oct 31 04:04:05.367681 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.367628 5019 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12868h50m50.91892359s" Oct 31 04:04:05.368412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.367912 5019 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 04:04:05.368412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.367930 5019 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 04:04:05.368412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.368003 5019 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 04:04:05.368651 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:04:05.367840 5019 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-57.ec2.internal\" not found" Oct 31 04:04:05.371268 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.368814 5019 factory.go:55] Registering systemd factory Oct 31 04:04:05.371382 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.371277 5019 factory.go:223] Registration of the systemd container factory successfully Oct 31 04:04:05.371382 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.370713 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2050cf2e-0f83-43b3-bb85-7bd4ac42a473" volumeName="kubernetes.io/projected/2050cf2e-0f83-43b3-bb85-7bd4ac42a473-kube-api-access-k52nk" seLinuxMountContext="" Oct 31 04:04:05.371382 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.371376 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49140fa3-6ce2-41ea-8b6b-e25257efaf3e" volumeName="kubernetes.io/projected/49140fa3-6ce2-41ea-8b6b-e25257efaf3e-kube-api-access-6vrvs" seLinuxMountContext="" Oct 31 04:04:05.371545 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.371392 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89ea384b-705f-47a9-8df6-d3c266ed6761" volumeName="kubernetes.io/secret/89ea384b-705f-47a9-8df6-d3c266ed6761-agent-certs" seLinuxMountContext="" Oct 31 04:04:05.371545 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.371404 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93beb8b8-c9e7-4cce-8baf-8497b971e193" volumeName="kubernetes.io/secret/93beb8b8-c9e7-4cce-8baf-8497b971e193-image-registry-private-configuration" seLinuxMountContext="" Oct 31 04:04:05.371545 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.371418 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a173790b-6c13-45a1-a361-3fe51844b4c2" volumeName="kubernetes.io/projected/a173790b-6c13-45a1-a361-3fe51844b4c2-kube-api-access-ggq72" seLinuxMountContext="" Oct 31 04:04:05.371545 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.371429 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22" volumeName="kubernetes.io/empty-dir/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-ca-trust-extracted" seLinuxMountContext="" Oct 31 04:04:05.371545 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.371440 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07320d54-b911-4157-aaf5-2517187d745c" volumeName="kubernetes.io/configmap/07320d54-b911-4157-aaf5-2517187d745c-service-ca-bundle" seLinuxMountContext="" Oct 31 04:04:05.371545 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.371452 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48309c9d-5f34-49bf-b55b-f74959984e0a" volumeName="kubernetes.io/empty-dir/48309c9d-5f34-49bf-b55b-f74959984e0a-tmp-dir" seLinuxMountContext="" Oct 31 04:04:05.371545 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.371466 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48309c9d-5f34-49bf-b55b-f74959984e0a" volumeName="kubernetes.io/projected/48309c9d-5f34-49bf-b55b-f74959984e0a-kube-api-access-kt6hk" seLinuxMountContext="" Oct 31 04:04:05.371545 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.371478 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93beb8b8-c9e7-4cce-8baf-8497b971e193" volumeName="kubernetes.io/configmap/93beb8b8-c9e7-4cce-8baf-8497b971e193-registry-certificates" seLinuxMountContext="" Oct 31 04:04:05.371545 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.371502 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97c312b1-4ca5-49c8-8c8d-d247c0fa14ac" volumeName="kubernetes.io/projected/97c312b1-4ca5-49c8-8c8d-d247c0fa14ac-kube-api-access-2f5pv" seLinuxMountContext="" Oct 31 04:04:05.371545 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.371516 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a173790b-6c13-45a1-a361-3fe51844b4c2" volumeName="kubernetes.io/configmap/a173790b-6c13-45a1-a361-3fe51844b4c2-telemetry-config" seLinuxMountContext="" Oct 31 04:04:05.371545 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.371530 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aef43d62-9761-4262-8183-01b9890462a7" volumeName="kubernetes.io/projected/aef43d62-9761-4262-8183-01b9890462a7-kube-api-access-kjq6x" seLinuxMountContext="" Oct 31 04:04:05.371545 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.371547 5019 factory.go:153] Registering CRI-O factory Oct 31 04:04:05.371982 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.371564 5019 factory.go:223] Registration of the crio container factory successfully Oct 31 04:04:05.371982 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.371543 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c3c90e6f-df5e-4c17-bec4-554280a8e8e7" volumeName="kubernetes.io/configmap/c3c90e6f-df5e-4c17-bec4-554280a8e8e7-config" seLinuxMountContext="" Oct 31 04:04:05.371982 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.371587 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="183f4178-99c0-4945-b490-c71cbd2780f3" volumeName="kubernetes.io/configmap/183f4178-99c0-4945-b490-c71cbd2780f3-nginx-conf" seLinuxMountContext="" Oct 31 04:04:05.371982 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.371611 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f6cd22a-499a-4c35-b408-b2aa9b78620d" volumeName="kubernetes.io/secret/7f6cd22a-499a-4c35-b408-b2aa9b78620d-ovn-node-metrics-cert" seLinuxMountContext="" Oct 31 04:04:05.371982 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.371619 5019 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 04:04:05.371982 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.371627 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93beb8b8-c9e7-4cce-8baf-8497b971e193" volumeName="kubernetes.io/empty-dir/93beb8b8-c9e7-4cce-8baf-8497b971e193-ca-trust-extracted" seLinuxMountContext="" Oct 31 04:04:05.371982 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.371641 5019 factory.go:103] Registering Raw factory Oct 31 04:04:05.371982 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.371658 5019 manager.go:1199] Started watching for new ooms in manager Oct 31 04:04:05.371982 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.371641 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c8cb360-ff57-41b1-8244-47588703cba5" volumeName="kubernetes.io/projected/9c8cb360-ff57-41b1-8244-47588703cba5-kube-api-access-9jbhv" seLinuxMountContext="" Oct 31 04:04:05.371982 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.371715 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2ce92f3-5b05-4c89-9d53-a57bc85b17ef" volumeName="kubernetes.io/secret/a2ce92f3-5b05-4c89-9d53-a57bc85b17ef-cert" seLinuxMountContext="" Oct 31 04:04:05.371982 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.371740 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be0026cb-89bc-413a-bfe3-523d980397fa" volumeName="kubernetes.io/empty-dir/be0026cb-89bc-413a-bfe3-523d980397fa-etc-tuned" seLinuxMountContext="" Oct 31 04:04:05.371982 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.371757 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1560abd-3afc-4f94-b0bb-ac7ef8b2464a" volumeName="kubernetes.io/secret/c1560abd-3afc-4f94-b0bb-ac7ef8b2464a-original-pull-secret" seLinuxMountContext="" Oct 31 04:04:05.371982 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.371771 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c3c90e6f-df5e-4c17-bec4-554280a8e8e7" volumeName="kubernetes.io/projected/c3c90e6f-df5e-4c17-bec4-554280a8e8e7-kube-api-access-7mtpj" seLinuxMountContext="" Oct 31 04:04:05.371982 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.371785 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07320d54-b911-4157-aaf5-2517187d745c" volumeName="kubernetes.io/projected/07320d54-b911-4157-aaf5-2517187d745c-kube-api-access-x4pc9" seLinuxMountContext="" Oct 31 04:04:05.371982 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.371798 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="197dee13-748f-4c6d-acc5-1793b5cee80a" volumeName="kubernetes.io/configmap/197dee13-748f-4c6d-acc5-1793b5cee80a-samples-operator-config" seLinuxMountContext="" Oct 31 04:04:05.371982 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.371814 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3945ff65-e877-4372-9334-3968c68b7295" volumeName="kubernetes.io/configmap/3945ff65-e877-4372-9334-3968c68b7295-config" seLinuxMountContext="" Oct 31 04:04:05.371982 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.371827 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d22707c7-21fa-4511-b0b1-869b41fb2351" volumeName="kubernetes.io/secret/d22707c7-21fa-4511-b0b1-869b41fb2351-stats-auth" seLinuxMountContext="" Oct 31 04:04:05.371982 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.371840 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e667439c-7121-4c5f-84fe-b36f2391dc66" volumeName="kubernetes.io/projected/e667439c-7121-4c5f-84fe-b36f2391dc66-kube-api-access-wddq5" seLinuxMountContext="" Oct 31 04:04:05.371982 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.371852 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22" volumeName="kubernetes.io/projected/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-kube-api-access-xc2vq" seLinuxMountContext="" Oct 31 04:04:05.371982 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.371872 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07320d54-b911-4157-aaf5-2517187d745c" volumeName="kubernetes.io/empty-dir/07320d54-b911-4157-aaf5-2517187d745c-archives-path" seLinuxMountContext="" Oct 31 04:04:05.371982 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.371886 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3945ff65-e877-4372-9334-3968c68b7295" volumeName="kubernetes.io/configmap/3945ff65-e877-4372-9334-3968c68b7295-trusted-ca" seLinuxMountContext="" Oct 31 04:04:05.371982 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.371901 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93beb8b8-c9e7-4cce-8baf-8497b971e193" volumeName="kubernetes.io/configmap/93beb8b8-c9e7-4cce-8baf-8497b971e193-trusted-ca" seLinuxMountContext="" Oct 31 04:04:05.371982 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.371915 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aaae68b8-1654-474f-9e74-45d1e2025d20" volumeName="kubernetes.io/projected/aaae68b8-1654-474f-9e74-45d1e2025d20-kube-api-access-nd5qc" seLinuxMountContext="" Oct 31 04:04:05.371982 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.371928 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9eeaad0-e2e5-4b90-a0af-2d254c787fac" volumeName="kubernetes.io/secret/c9eeaad0-e2e5-4b90-a0af-2d254c787fac-signing-key" seLinuxMountContext="" Oct 31 04:04:05.371982 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.371943 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce86be55-219f-46fb-bff6-6dcf87a40ef4" volumeName="kubernetes.io/configmap/ce86be55-219f-46fb-bff6-6dcf87a40ef4-iptables-alerter-script" seLinuxMountContext="" Oct 31 04:04:05.371982 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.371955 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e667439c-7121-4c5f-84fe-b36f2391dc66" volumeName="kubernetes.io/empty-dir/e667439c-7121-4c5f-84fe-b36f2391dc66-snapshots" seLinuxMountContext="" Oct 31 04:04:05.371982 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.371970 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07320d54-b911-4157-aaf5-2517187d745c" volumeName="kubernetes.io/configmap/07320d54-b911-4157-aaf5-2517187d745c-trusted-ca-bundle" seLinuxMountContext="" Oct 31 04:04:05.371982 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.371984 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3945ff65-e877-4372-9334-3968c68b7295" volumeName="kubernetes.io/projected/3945ff65-e877-4372-9334-3968c68b7295-kube-api-access-fqx72" seLinuxMountContext="" Oct 31 04:04:05.372759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.372013 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a05c43f-e2db-44c0-b9bd-4bdabca92a98" volumeName="kubernetes.io/projected/3a05c43f-e2db-44c0-b9bd-4bdabca92a98-kube-api-access-vhkfw" seLinuxMountContext="" Oct 31 04:04:05.372759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.372027 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f6cd22a-499a-4c35-b408-b2aa9b78620d" volumeName="kubernetes.io/projected/7f6cd22a-499a-4c35-b408-b2aa9b78620d-kube-api-access-phq2f" seLinuxMountContext="" Oct 31 04:04:05.372759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.372040 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93beb8b8-c9e7-4cce-8baf-8497b971e193" volumeName="kubernetes.io/secret/93beb8b8-c9e7-4cce-8baf-8497b971e193-installation-pull-secrets" seLinuxMountContext="" Oct 31 04:04:05.372759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.372054 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b541554a-0460-4716-9f14-f7a46567465c" volumeName="kubernetes.io/projected/b541554a-0460-4716-9f14-f7a46567465c-kube-api-access-qlg9n" seLinuxMountContext="" Oct 31 04:04:05.372759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.372067 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aaae68b8-1654-474f-9e74-45d1e2025d20" volumeName="kubernetes.io/configmap/aaae68b8-1654-474f-9e74-45d1e2025d20-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 31 04:04:05.372759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.372090 5019 manager.go:319] Starting recovery of all containers Oct 31 04:04:05.372759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.372093 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c3c90e6f-df5e-4c17-bec4-554280a8e8e7" volumeName="kubernetes.io/secret/c3c90e6f-df5e-4c17-bec4-554280a8e8e7-serving-cert" seLinuxMountContext="" Oct 31 04:04:05.372759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.372169 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="197dee13-748f-4c6d-acc5-1793b5cee80a" volumeName="kubernetes.io/projected/197dee13-748f-4c6d-acc5-1793b5cee80a-kube-api-access-7cq9b" seLinuxMountContext="" Oct 31 04:04:05.372759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.372182 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b6d69de-a860-4c82-a03f-23ab7905a5d7" volumeName="kubernetes.io/projected/5b6d69de-a860-4c82-a03f-23ab7905a5d7-kube-api-access-7w6z5" seLinuxMountContext="" Oct 31 04:04:05.372759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.372196 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d336e6e-feb0-40b2-a625-0e21d25bfac5" volumeName="kubernetes.io/configmap/6d336e6e-feb0-40b2-a625-0e21d25bfac5-multus-daemon-config" seLinuxMountContext="" Oct 31 04:04:05.372759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.372209 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93beb8b8-c9e7-4cce-8baf-8497b971e193" volumeName="kubernetes.io/projected/93beb8b8-c9e7-4cce-8baf-8497b971e193-kube-api-access-25x8r" seLinuxMountContext="" Oct 31 04:04:05.372759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.372221 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="944c9831-b2fa-4f58-bba2-93288793b125" volumeName="kubernetes.io/projected/944c9831-b2fa-4f58-bba2-93288793b125-kube-api-access-ghxbr" seLinuxMountContext="" Oct 31 04:04:05.372759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.372236 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2ce92f3-5b05-4c89-9d53-a57bc85b17ef" volumeName="kubernetes.io/projected/a2ce92f3-5b05-4c89-9d53-a57bc85b17ef-kube-api-access-7ll7m" seLinuxMountContext="" Oct 31 04:04:05.372759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.372250 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4a091a2-58d2-4b05-a968-d16fe1bd9d40" volumeName="kubernetes.io/projected/c4a091a2-58d2-4b05-a968-d16fe1bd9d40-kube-api-access-nxlpp" seLinuxMountContext="" Oct 31 04:04:05.372759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.372261 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce86be55-219f-46fb-bff6-6dcf87a40ef4" volumeName="kubernetes.io/projected/ce86be55-219f-46fb-bff6-6dcf87a40ef4-kube-api-access-x5hsv" seLinuxMountContext="" Oct 31 04:04:05.372759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.372273 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3945ff65-e877-4372-9334-3968c68b7295" volumeName="kubernetes.io/secret/3945ff65-e877-4372-9334-3968c68b7295-serving-cert" seLinuxMountContext="" Oct 31 04:04:05.372759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.372285 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a05c43f-e2db-44c0-b9bd-4bdabca92a98" volumeName="kubernetes.io/empty-dir/3a05c43f-e2db-44c0-b9bd-4bdabca92a98-tmp-dir" seLinuxMountContext="" Oct 31 04:04:05.372759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.372297 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e667439c-7121-4c5f-84fe-b36f2391dc66" volumeName="kubernetes.io/empty-dir/e667439c-7121-4c5f-84fe-b36f2391dc66-tmp" seLinuxMountContext="" Oct 31 04:04:05.372759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.372310 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e667439c-7121-4c5f-84fe-b36f2391dc66" volumeName="kubernetes.io/secret/e667439c-7121-4c5f-84fe-b36f2391dc66-serving-cert" seLinuxMountContext="" Oct 31 04:04:05.372759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.372322 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22" volumeName="kubernetes.io/projected/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-bound-sa-token" seLinuxMountContext="" Oct 31 04:04:05.372759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.372372 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f6cd22a-499a-4c35-b408-b2aa9b78620d" volumeName="kubernetes.io/configmap/7f6cd22a-499a-4c35-b408-b2aa9b78620d-ovnkube-script-lib" seLinuxMountContext="" Oct 31 04:04:05.372759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.372385 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b7bf8e5-7877-46e0-88dd-a3503b7639c5" volumeName="kubernetes.io/projected/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-kube-api-access-frshf" seLinuxMountContext="" Oct 31 04:04:05.372759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.372398 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22" volumeName="kubernetes.io/configmap/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-registry-certificates" seLinuxMountContext="" Oct 31 04:04:05.372759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.372410 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22" volumeName="kubernetes.io/secret/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-installation-pull-secrets" seLinuxMountContext="" Oct 31 04:04:05.372759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.372423 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93beb8b8-c9e7-4cce-8baf-8497b971e193" volumeName="kubernetes.io/projected/93beb8b8-c9e7-4cce-8baf-8497b971e193-bound-sa-token" seLinuxMountContext="" Oct 31 04:04:05.372759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.372434 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9eeaad0-e2e5-4b90-a0af-2d254c787fac" volumeName="kubernetes.io/projected/c9eeaad0-e2e5-4b90-a0af-2d254c787fac-kube-api-access-57gkt" seLinuxMountContext="" Oct 31 04:04:05.372759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.372446 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d336e6e-feb0-40b2-a625-0e21d25bfac5" volumeName="kubernetes.io/configmap/6d336e6e-feb0-40b2-a625-0e21d25bfac5-cni-binary-copy" seLinuxMountContext="" Oct 31 04:04:05.372759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.372457 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="944c9831-b2fa-4f58-bba2-93288793b125" volumeName="kubernetes.io/secret/944c9831-b2fa-4f58-bba2-93288793b125-serving-cert" seLinuxMountContext="" Oct 31 04:04:05.372759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.372471 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb656990-e663-4fea-8733-18ded9222a79" volumeName="kubernetes.io/projected/fb656990-e663-4fea-8733-18ded9222a79-kube-api-access-hsrbd" seLinuxMountContext="" Oct 31 04:04:05.372759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.372484 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aaae68b8-1654-474f-9e74-45d1e2025d20" volumeName="kubernetes.io/configmap/aaae68b8-1654-474f-9e74-45d1e2025d20-cni-sysctl-allowlist" seLinuxMountContext="" Oct 31 04:04:05.372759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.372497 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31221235-f863-445b-a5ca-375aae905d38" volumeName="kubernetes.io/projected/31221235-f863-445b-a5ca-375aae905d38-kube-api-access-x5nf8" seLinuxMountContext="" Oct 31 04:04:05.372759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.372525 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f6cd22a-499a-4c35-b408-b2aa9b78620d" volumeName="kubernetes.io/configmap/7f6cd22a-499a-4c35-b408-b2aa9b78620d-ovnkube-config" seLinuxMountContext="" Oct 31 04:04:05.372759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.372538 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89ea384b-705f-47a9-8df6-d3c266ed6761" volumeName="kubernetes.io/configmap/89ea384b-705f-47a9-8df6-d3c266ed6761-konnectivity-ca" seLinuxMountContext="" Oct 31 04:04:05.372759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.372550 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="944c9831-b2fa-4f58-bba2-93288793b125" volumeName="kubernetes.io/configmap/944c9831-b2fa-4f58-bba2-93288793b125-config" seLinuxMountContext="" Oct 31 04:04:05.372759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.372564 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aaae68b8-1654-474f-9e74-45d1e2025d20" volumeName="kubernetes.io/configmap/aaae68b8-1654-474f-9e74-45d1e2025d20-cni-binary-copy" seLinuxMountContext="" Oct 31 04:04:05.372759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.372579 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d22707c7-21fa-4511-b0b1-869b41fb2351" volumeName="kubernetes.io/projected/d22707c7-21fa-4511-b0b1-869b41fb2351-kube-api-access-v6xnt" seLinuxMountContext="" Oct 31 04:04:05.372759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.372592 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22" volumeName="kubernetes.io/secret/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-image-registry-private-configuration" seLinuxMountContext="" Oct 31 04:04:05.372759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.372604 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31221235-f863-445b-a5ca-375aae905d38" volumeName="kubernetes.io/configmap/31221235-f863-445b-a5ca-375aae905d38-serviceca" seLinuxMountContext="" Oct 31 04:04:05.372759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.372616 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be0026cb-89bc-413a-bfe3-523d980397fa" volumeName="kubernetes.io/empty-dir/be0026cb-89bc-413a-bfe3-523d980397fa-tmp" seLinuxMountContext="" Oct 31 04:04:05.372759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.372628 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be0026cb-89bc-413a-bfe3-523d980397fa" volumeName="kubernetes.io/projected/be0026cb-89bc-413a-bfe3-523d980397fa-kube-api-access-dn7px" seLinuxMountContext="" Oct 31 04:04:05.372759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.372641 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9eeaad0-e2e5-4b90-a0af-2d254c787fac" volumeName="kubernetes.io/configmap/c9eeaad0-e2e5-4b90-a0af-2d254c787fac-signing-cabundle" seLinuxMountContext="" Oct 31 04:04:05.372759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.372655 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48309c9d-5f34-49bf-b55b-f74959984e0a" volumeName="kubernetes.io/configmap/48309c9d-5f34-49bf-b55b-f74959984e0a-config-volume" seLinuxMountContext="" Oct 31 04:04:05.372759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.372668 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57b47c7b-9c84-4888-be69-64afca28a8c0" volumeName="kubernetes.io/projected/57b47c7b-9c84-4888-be69-64afca28a8c0-kube-api-access-jj5t7" seLinuxMountContext="" Oct 31 04:04:05.372759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.372689 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d336e6e-feb0-40b2-a625-0e21d25bfac5" volumeName="kubernetes.io/projected/6d336e6e-feb0-40b2-a625-0e21d25bfac5-kube-api-access-9mpvl" seLinuxMountContext="" Oct 31 04:04:05.372759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.372705 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f6cd22a-499a-4c35-b408-b2aa9b78620d" volumeName="kubernetes.io/configmap/7f6cd22a-499a-4c35-b408-b2aa9b78620d-env-overrides" seLinuxMountContext="" Oct 31 04:04:05.372759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.372719 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22" volumeName="kubernetes.io/configmap/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-trusted-ca" seLinuxMountContext="" Oct 31 04:04:05.372759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.372732 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b7bf8e5-7877-46e0-88dd-a3503b7639c5" volumeName="kubernetes.io/empty-dir/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-data-volume" seLinuxMountContext="" Oct 31 04:04:05.372759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.372745 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d22707c7-21fa-4511-b0b1-869b41fb2351" volumeName="kubernetes.io/secret/d22707c7-21fa-4511-b0b1-869b41fb2351-default-certificate" seLinuxMountContext="" Oct 31 04:04:05.372759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.372757 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e667439c-7121-4c5f-84fe-b36f2391dc66" volumeName="kubernetes.io/configmap/e667439c-7121-4c5f-84fe-b36f2391dc66-service-ca-bundle" seLinuxMountContext="" Oct 31 04:04:05.372759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.372772 5019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e667439c-7121-4c5f-84fe-b36f2391dc66" volumeName="kubernetes.io/configmap/e667439c-7121-4c5f-84fe-b36f2391dc66-trusted-ca-bundle" seLinuxMountContext="" Oct 31 04:04:05.372759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.372784 5019 reconstruct.go:97] "Volume reconstruction finished" Oct 31 04:04:05.374112 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.372792 5019 reconciler.go:29] "Reconciler: start to sync state" Oct 31 04:04:05.386557 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.386524 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s88kp_3945ff65-e877-4372-9334-3968c68b7295/console-operator/1.log" Oct 31 04:04:05.387097 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.387077 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s88kp_3945ff65-e877-4372-9334-3968c68b7295/console-operator/1.log" Oct 31 04:04:05.397377 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.397314 5019 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 04:04:05.399144 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.399021 5019 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 04:04:05.399144 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.399051 5019 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 04:04:05.399144 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.399073 5019 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 04:04:05.399144 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:04:05.399117 5019 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 31 04:04:05.416653 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.416581 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s88kp_3945ff65-e877-4372-9334-3968c68b7295/console-operator/1.log" Oct 31 04:04:05.416653 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.416613 5019 generic.go:356] "Generic (PLEG): container finished" podID="3945ff65-e877-4372-9334-3968c68b7295" containerID="e11078a797b967674e01f514aab383ae2b67709c9bc64c60177a719670d63040" exitCode=255 Oct 31 04:04:05.419547 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.419518 5019 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 04:04:05.419708 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.419560 5019 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 31 04:04:05.421125 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.421104 5019 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 04:04:05.434136 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.434099 5019 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 04:04:05.434388 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.434367 5019 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 04:04:05.437438 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.437392 5019 generic.go:356] "Generic (PLEG): container finished" podID="e8283c5a79822699286a7ae3ef8db4f6" containerID="7262c355488431c1bf772ef311d19308883aa44dd7e85053603a15d191d212c2" exitCode=0 Oct 31 04:04:05.450540 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.450509 5019 generic.go:356] "Generic (PLEG): container finished" podID="aaae68b8-1654-474f-9e74-45d1e2025d20" containerID="f01e6b79ca5c6f0780f8e064f46ecd725b95bf4bb50dc6fee86af2717d108159" exitCode=0 Oct 31 04:04:05.450540 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.450534 5019 generic.go:356] "Generic (PLEG): container finished" podID="aaae68b8-1654-474f-9e74-45d1e2025d20" containerID="f96e5fcf2fb0ea7c35958096eca0af14cfd77a2d0f79a4a4eb59d6ba1255190d" exitCode=0 Oct 31 04:04:05.450540 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.450541 5019 generic.go:356] "Generic (PLEG): container finished" podID="aaae68b8-1654-474f-9e74-45d1e2025d20" containerID="3dcb83a5957be86beeda6b0b18d2934b93f711c313a9b3e5f87cc061cd49a232" exitCode=0 Oct 31 04:04:05.450540 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.450546 5019 generic.go:356] "Generic (PLEG): container finished" podID="aaae68b8-1654-474f-9e74-45d1e2025d20" containerID="cce4cc3d11b1f597e1ffc5ca34b38255c353c1f9da1d65a8abe12219ac5c942a" exitCode=0 Oct 31 04:04:05.450540 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.450551 5019 generic.go:356] "Generic (PLEG): container finished" podID="aaae68b8-1654-474f-9e74-45d1e2025d20" containerID="17ae925742d4d05dacaf9d32f8e98e57400e246d5d50542f943298ebd162126f" exitCode=0 Oct 31 04:04:05.450876 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.450556 5019 generic.go:356] "Generic (PLEG): container finished" podID="aaae68b8-1654-474f-9e74-45d1e2025d20" containerID="b1ccb1c6bcd958ea3f55f4e3d437d0a2662df72c1a3471a19b13e8bca8a54d09" exitCode=0 Oct 31 04:04:05.478154 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.478123 5019 manager.go:324] Recovery completed Oct 31 04:04:05.499244 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:04:05.499208 5019 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 31 04:04:05.501716 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.501694 5019 cpu_manager.go:226] "Starting" policy="none" Oct 31 04:04:05.501716 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.501709 5019 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 04:04:05.501916 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.501733 5019 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 04:04:05.501916 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.501887 5019 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 31 04:04:05.502422 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.501898 5019 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 31 04:04:05.502480 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.502426 5019 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 31 04:04:05.502480 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.502433 5019 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 31 04:04:05.502480 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.502441 5019 policy_none.go:50] "Start" Oct 31 04:04:05.502480 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.502451 5019 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 04:04:05.502480 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.502466 5019 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 04:04:05.502634 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.502624 5019 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 31 04:04:05.502668 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.502635 5019 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 31 04:04:05.502668 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.502641 5019 policy_none.go:44] "Start" Oct 31 04:04:05.507497 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.507451 5019 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 04:04:05.510107 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.510092 5019 manager.go:341] "Starting Device Plugin manager" Oct 31 04:04:05.510191 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:04:05.510123 5019 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 04:04:05.510191 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.510136 5019 server.go:87] "Starting device plugin registration server" Oct 31 04:04:05.510536 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.510518 5019 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 04:04:05.510643 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.510534 5019 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 04:04:05.510643 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.510600 5019 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 04:04:05.510745 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.510693 5019 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 04:04:05.510745 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.510728 5019 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 04:04:05.510745 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.510735 5019 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 04:04:05.510885 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.510776 5019 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T04:04:05.510718574Z","UUID":"713357e0-a247-46be-b71d-d2d3167dd102","Handler":null,"Name":"","Endpoint":""} Oct 31 04:04:05.512508 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.512489 5019 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 04:04:05.512659 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.512530 5019 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 04:04:05.513184 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:04:05.513159 5019 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 04:04:05.611610 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.611566 5019 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 04:04:05.613057 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.613028 5019 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasSufficientMemory" Oct 31 04:04:05.613057 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.613063 5019 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 04:04:05.613205 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.613074 5019 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasSufficientPID" Oct 31 04:04:05.613205 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.613120 5019 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-57.ec2.internal" Oct 31 04:04:05.649960 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.649927 5019 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-130-57.ec2.internal" Oct 31 04:04:05.650108 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.650018 5019 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-57.ec2.internal" Oct 31 04:04:05.699898 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.699793 5019 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-57.ec2.internal"] Oct 31 04:04:05.700564 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.700542 5019 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal" Oct 31 04:04:05.700671 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.700531 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-57.ec2.internal" event={"ID":"6536fdc09c4b535fc805fe98673b8597","Type":"ContainerStarted","Data":"713d3ff48543c2aa7c9049d7b0ff4e663af284fb92d00922e6e76919f5a8727d"} Oct 31 04:04:05.700744 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.700669 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-57.ec2.internal" event={"ID":"6536fdc09c4b535fc805fe98673b8597","Type":"ContainerStarted","Data":"828cacb0290babc13ca9b7a25aebe8bc490f563d05ae84a04adfc94da818208f"} Oct 31 04:04:05.700744 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.700696 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal" event={"ID":"e8283c5a79822699286a7ae3ef8db4f6","Type":"ContainerStarted","Data":"e67453bb6c81cad759194fcee9d387ecd699a14eaa6da143f1d52e1ae4c948c5"} Oct 31 04:04:05.700744 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.700713 5019 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-57.ec2.internal" Oct 31 04:04:05.700744 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.700721 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal" event={"ID":"e8283c5a79822699286a7ae3ef8db4f6","Type":"ContainerDied","Data":"7262c355488431c1bf772ef311d19308883aa44dd7e85053603a15d191d212c2"} Oct 31 04:04:05.700744 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.700737 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal" event={"ID":"e8283c5a79822699286a7ae3ef8db4f6","Type":"ContainerStarted","Data":"8a6619e28986b25f93c7224377a9cf5bac47e19970b0c0673ba89bff2552643f"} Oct 31 04:04:05.719327 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.719296 5019 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 04:04:05.719460 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:04:05.719372 5019 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-57.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-57.ec2.internal" Oct 31 04:04:05.724453 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.724429 5019 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 04:04:05.724567 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:04:05.724487 5019 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal" Oct 31 04:04:05.820693 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.820661 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e8283c5a79822699286a7ae3ef8db4f6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal\" (UID: \"e8283c5a79822699286a7ae3ef8db4f6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal" Oct 31 04:04:05.820693 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.820694 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8283c5a79822699286a7ae3ef8db4f6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal\" (UID: \"e8283c5a79822699286a7ae3ef8db4f6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal" Oct 31 04:04:05.820894 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.820716 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6536fdc09c4b535fc805fe98673b8597-config\") pod \"kube-apiserver-proxy-ip-10-0-130-57.ec2.internal\" (UID: \"6536fdc09c4b535fc805fe98673b8597\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-57.ec2.internal" Oct 31 04:04:05.921309 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.921272 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e8283c5a79822699286a7ae3ef8db4f6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal\" (UID: \"e8283c5a79822699286a7ae3ef8db4f6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal" Oct 31 04:04:05.921309 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.921308 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8283c5a79822699286a7ae3ef8db4f6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal\" (UID: \"e8283c5a79822699286a7ae3ef8db4f6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal" Oct 31 04:04:05.921517 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.921361 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6536fdc09c4b535fc805fe98673b8597-config\") pod \"kube-apiserver-proxy-ip-10-0-130-57.ec2.internal\" (UID: \"6536fdc09c4b535fc805fe98673b8597\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-57.ec2.internal" Oct 31 04:04:05.921517 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.921402 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6536fdc09c4b535fc805fe98673b8597-config\") pod \"kube-apiserver-proxy-ip-10-0-130-57.ec2.internal\" (UID: \"6536fdc09c4b535fc805fe98673b8597\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-57.ec2.internal" Oct 31 04:04:05.921517 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.921400 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e8283c5a79822699286a7ae3ef8db4f6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal\" (UID: \"e8283c5a79822699286a7ae3ef8db4f6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal" Oct 31 04:04:05.921517 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:05.921402 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8283c5a79822699286a7ae3ef8db4f6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal\" (UID: \"e8283c5a79822699286a7ae3ef8db4f6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal" Oct 31 04:04:06.352132 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.352091 5019 apiserver.go:52] "Watching apiserver" Oct 31 04:04:06.417780 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.417740 5019 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 04:04:06.418594 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.418561 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-s88kp","openshift-kube-storage-version-migrator/migrator-66df5d496b-zg2xl","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bt9c9","kube-system/global-pull-secret-syncer-hnbll","openshift-ovn-kubernetes/ovnkube-node-tmt7k","openshift-service-ca/service-ca-5fffc97bdf-l25xj","openshift-dns/node-resolver-nbl9z","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hntxx","openshift-ingress-canary/ingress-canary-vqsgm","openshift-network-console/networking-console-plugin-86c478bb88-pnwds","kube-system/kas-connection-checker-685895c7c9-xgjnm","kube-system/kas-connection-checker-685895c7c9-z75pq","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal","openshift-image-registry/image-registry-7f78cfdfd8-w98g7","kube-system/kube-apiserver-proxy-ip-10-0-130-57.ec2.internal","openshift-kube-storage-version-migrator/migrator-66df5d496b-fvqtv","openshift-insights/insights-operator-7c4986d449-ld96l","openshift-network-operator/iptables-alerter-bhgvp","openshift-insights/periodic-gathering-wh6x6-t7s8v","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c64cl","openshift-cluster-node-tuning-operator/tuned-26vsf","openshift-dns/dns-default-zjpdv","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sqbnh","openshift-monitoring/cluster-monitoring-operator-884576c57-zqvkx","kube-system/konnectivity-agent-hm5cr","openshift-service-ca-operator/service-ca-operator-544d749c8c-khdq4","openshift-image-registry/node-ca-h9sww","openshift-multus/network-metrics-daemon-clnlz","kube-system/kas-connection-checker-685895c7c9-xqd9k","openshift-network-diagnostics/network-check-target-4b7d5","openshift-multus/multus-6d7cp","openshift-image-registry/image-registry-6f69bcc59c-w96bl","openshift-multus/multus-additional-cni-plugins-6zzkx","openshift-ingress/router-default-678d747c5f-bgj92","openshift-insights/insights-runtime-extractor-c6gcg","openshift-network-diagnostics/network-check-source-6f78d59f99-rf8vj"] Oct 31 04:04:06.420470 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.420435 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-678d747c5f-bgj92" Oct 31 04:04:06.425241 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.425206 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zqvkx" Oct 31 04:04:06.425408 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.425272 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c64cl" Oct 31 04:04:06.426522 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.426498 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f69bcc59c-w96bl" Oct 31 04:04:06.426923 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.426908 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-pnwds" Oct 31 04:04:06.427285 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.427272 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clnlz" Oct 31 04:04:06.433675 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.433641 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" Oct 31 04:04:06.434130 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.434115 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 31 04:04:06.434774 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.434755 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zjpdv" Oct 31 04:04:06.435169 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.435153 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 31 04:04:06.435264 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.435152 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 31 04:04:06.435264 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.435171 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 31 04:04:06.435264 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.435214 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 31 04:04:06.435999 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.435978 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 31 04:04:06.436149 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.436130 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-677wb\"" Oct 31 04:04:06.436439 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.436243 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 31 04:04:06.436439 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.436366 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 31 04:04:06.436609 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.436590 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 31 04:04:06.436673 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.436591 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 31 04:04:06.437616 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.437599 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-c6gcg" Oct 31 04:04:06.446935 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.446902 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 31 04:04:06.447071 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.446947 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-c8446\"" Oct 31 04:04:06.447071 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.446971 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 31 04:04:06.447800 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.447778 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 04:04:06.447800 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.447778 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 31 04:04:06.448570 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.448547 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 31 04:04:06.448654 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.448603 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 31 04:04:06.448654 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.448633 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 31 04:04:06.448740 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.448681 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 31 04:04:06.450605 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.450577 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-mqs6m\"" Oct 31 04:04:06.450715 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.450696 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 31 04:04:06.450790 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.450776 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 04:04:06.450871 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.450858 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 04:04:06.450990 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.450976 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-j9kgn\"" Oct 31 04:04:06.451072 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.451059 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-np8k4\"" Oct 31 04:04:06.451140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.451128 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 04:04:06.451204 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.451191 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 31 04:04:06.451250 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.451231 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 31 04:04:06.451292 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.451281 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 31 04:04:06.451373 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.451358 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-gmtbq\"" Oct 31 04:04:06.451467 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.451454 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lwzcw\"" Oct 31 04:04:06.451537 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.451498 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-s6w7q\"" Oct 31 04:04:06.451537 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.451514 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 04:04:06.451646 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.451632 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 31 04:04:06.451779 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.451760 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 31 04:04:06.451868 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.451847 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pjn2p\"" Oct 31 04:04:06.452065 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.452046 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-gw6dk\"" Oct 31 04:04:06.452122 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.452075 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 04:04:06.452241 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.452225 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 04:04:06.452409 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.452393 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 04:04:06.452457 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.452424 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ldktz\"" Oct 31 04:04:06.452510 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.452495 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-vc7dp\"" Oct 31 04:04:06.452676 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.452658 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xzg5s\"" Oct 31 04:04:06.452718 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.452682 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9gl9k\"" Oct 31 04:04:06.452809 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.452791 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-bqqdt\"" Oct 31 04:04:06.452854 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.452842 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9876q\"" Oct 31 04:04:06.453021 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.451817 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-wtrm9\"" Oct 31 04:04:06.453086 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.451360 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 31 04:04:06.453086 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.453013 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 31 04:04:06.453176 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.453107 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 04:04:06.453499 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.453478 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 31 04:04:06.453837 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.453820 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ctbtg\"" Oct 31 04:04:06.453900 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.453822 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 04:04:06.453900 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.453849 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 04:04:06.454016 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.453898 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kfpck\"" Oct 31 04:04:06.454016 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.453913 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 04:04:06.454016 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.453970 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 31 04:04:06.454016 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.453992 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-7brl8\"" Oct 31 04:04:06.454016 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.454016 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 31 04:04:06.454245 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.454049 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qzkqw\"" Oct 31 04:04:06.454245 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.454096 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 04:04:06.454245 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.454118 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-brbhc\"" Oct 31 04:04:06.454245 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.454159 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-jnm7g\"" Oct 31 04:04:06.454245 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.454180 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 04:04:06.454245 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.454215 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-dt5cz\"" Oct 31 04:04:06.454457 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.454277 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 31 04:04:06.454457 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.454436 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-45qpm\"" Oct 31 04:04:06.454521 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.454489 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-fl6gf\"" Oct 31 04:04:06.454566 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.454556 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 04:04:06.454740 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.454722 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-brpbt\"" Oct 31 04:04:06.459539 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.459369 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 04:04:06.459539 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.459389 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 31 04:04:06.459539 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.459407 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 04:04:06.459539 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.459369 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 04:04:06.459539 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.459438 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 04:04:06.459539 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.459459 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 04:04:06.459539 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.459482 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 04:04:06.459539 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.459483 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 04:04:06.459539 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.459512 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 31 04:04:06.459539 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.459521 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 04:04:06.459539 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.459533 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 31 04:04:06.460100 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.459594 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 04:04:06.460100 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.459620 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 04:04:06.460215 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.460101 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 31 04:04:06.460699 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.460682 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 04:04:06.460777 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.460737 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 04:04:06.460841 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.460819 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 04:04:06.460899 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.460835 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 31 04:04:06.460899 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.460860 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 04:04:06.460899 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.460837 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 04:04:06.460899 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.460875 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 04:04:06.461090 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.460908 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 04:04:06.461090 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.460908 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 31 04:04:06.461090 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.460925 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 31 04:04:06.461090 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.460945 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8w2pm\"" Oct 31 04:04:06.461090 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.460963 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 31 04:04:06.461090 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.460988 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 04:04:06.461090 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.461069 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 04:04:06.461090 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.461073 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 31 04:04:06.461090 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.461109 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 31 04:04:06.461586 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.461168 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 31 04:04:06.461586 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.461179 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 31 04:04:06.461586 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.461200 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 04:04:06.461586 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.461208 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 04:04:06.461586 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.461234 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 04:04:06.461586 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.461243 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 04:04:06.461586 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.461250 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 04:04:06.461586 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.461256 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 04:04:06.461586 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.461311 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 31 04:04:06.461586 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.461405 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 31 04:04:06.461586 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.461441 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 31 04:04:06.461586 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.461549 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 04:04:06.461950 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.461610 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 04:04:06.465111 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.465093 5019 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-57.ec2.internal" Oct 31 04:04:06.465232 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.465219 5019 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal" Oct 31 04:04:06.470378 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.470345 5019 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 04:04:06.488046 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.488021 5019 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 04:04:06.488134 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:04:06.488080 5019 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-57.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-57.ec2.internal" Oct 31 04:04:06.488479 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.488466 5019 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 04:04:06.488539 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:04:06.488503 5019 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal" Oct 31 04:04:06.526177 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.526136 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e667439c-7121-4c5f-84fe-b36f2391dc66-serving-cert\") pod \"insights-operator-7c4986d449-ld96l\" (UID: \"e667439c-7121-4c5f-84fe-b36f2391dc66\") " pod="openshift-insights/insights-operator-7c4986d449-ld96l" Oct 31 04:04:06.526296 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.526184 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-cnibin\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:04:06.526296 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.526208 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6d336e6e-feb0-40b2-a625-0e21d25bfac5-cni-binary-copy\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:04:06.526296 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.526233 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-host-var-lib-cni-bin\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:04:06.526509 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.526423 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-etc-sysconfig\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:04:06.526509 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.526494 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/93beb8b8-c9e7-4cce-8baf-8497b971e193-ca-trust-extracted\") pod \"image-registry-7f78cfdfd8-w98g7\" (UID: \"93beb8b8-c9e7-4cce-8baf-8497b971e193\") " pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" Oct 31 04:04:06.526592 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.526544 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3945ff65-e877-4372-9334-3968c68b7295-trusted-ca\") pod \"console-operator-7cd56b8dbc-s88kp\" (UID: \"3945ff65-e877-4372-9334-3968c68b7295\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-s88kp" Oct 31 04:04:06.526592 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.526586 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aaae68b8-1654-474f-9e74-45d1e2025d20-cnibin\") pod \"multus-additional-cni-plugins-6zzkx\" (UID: \"aaae68b8-1654-474f-9e74-45d1e2025d20\") " pod="openshift-multus/multus-additional-cni-plugins-6zzkx" Oct 31 04:04:06.526669 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.526586 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6d336e6e-feb0-40b2-a625-0e21d25bfac5-cni-binary-copy\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:04:06.526723 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.526673 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-multus-conf-dir\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:04:06.526723 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.526698 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/93beb8b8-c9e7-4cce-8baf-8497b971e193-ca-trust-extracted\") pod \"image-registry-7f78cfdfd8-w98g7\" (UID: \"93beb8b8-c9e7-4cce-8baf-8497b971e193\") " pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" Oct 31 04:04:06.526795 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.526717 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/89ea384b-705f-47a9-8df6-d3c266ed6761-agent-certs\") pod \"konnectivity-agent-hm5cr\" (UID: \"89ea384b-705f-47a9-8df6-d3c266ed6761\") " pod="kube-system/konnectivity-agent-hm5cr" Oct 31 04:04:06.526795 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.526749 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/89ea384b-705f-47a9-8df6-d3c266ed6761-konnectivity-ca\") pod \"konnectivity-agent-hm5cr\" (UID: \"89ea384b-705f-47a9-8df6-d3c266ed6761\") " pod="kube-system/konnectivity-agent-hm5cr" Oct 31 04:04:06.526795 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.526750 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e667439c-7121-4c5f-84fe-b36f2391dc66-serving-cert\") pod \"insights-operator-7c4986d449-ld96l\" (UID: \"e667439c-7121-4c5f-84fe-b36f2391dc66\") " pod="openshift-insights/insights-operator-7c4986d449-ld96l" Oct 31 04:04:06.526893 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.526812 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/31221235-f863-445b-a5ca-375aae905d38-host\") pod \"node-ca-h9sww\" (UID: \"31221235-f863-445b-a5ca-375aae905d38\") " pod="openshift-image-registry/node-ca-h9sww" Oct 31 04:04:06.526893 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.526846 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3945ff65-e877-4372-9334-3968c68b7295-config\") pod \"console-operator-7cd56b8dbc-s88kp\" (UID: \"3945ff65-e877-4372-9334-3968c68b7295\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-s88kp" Oct 31 04:04:06.526893 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.526869 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c1560abd-3afc-4f94-b0bb-ac7ef8b2464a-dbus\") pod \"global-pull-secret-syncer-hnbll\" (UID: \"c1560abd-3afc-4f94-b0bb-ac7ef8b2464a\") " pod="kube-system/global-pull-secret-syncer-hnbll" Oct 31 04:04:06.527004 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.526892 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5b6d69de-a860-4c82-a03f-23ab7905a5d7-sys-fs\") pod \"aws-ebs-csi-driver-node-sqbnh\" (UID: \"5b6d69de-a860-4c82-a03f-23ab7905a5d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sqbnh" Oct 31 04:04:06.527004 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.526920 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cq9b\" (UniqueName: \"kubernetes.io/projected/197dee13-748f-4c6d-acc5-1793b5cee80a-kube-api-access-7cq9b\") pod \"cluster-samples-operator-6ddb6c8b94-c64cl\" (UID: \"197dee13-748f-4c6d-acc5-1793b5cee80a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c64cl" Oct 31 04:04:06.527004 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.526922 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3945ff65-e877-4372-9334-3968c68b7295-trusted-ca\") pod \"console-operator-7cd56b8dbc-s88kp\" (UID: \"3945ff65-e877-4372-9334-3968c68b7295\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-s88kp" Oct 31 04:04:06.527004 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.526980 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7f6cd22a-499a-4c35-b408-b2aa9b78620d-ovnkube-script-lib\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:04:06.527188 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.527026 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/89ea384b-705f-47a9-8df6-d3c266ed6761-konnectivity-ca\") pod \"konnectivity-agent-hm5cr\" (UID: \"89ea384b-705f-47a9-8df6-d3c266ed6761\") " pod="kube-system/konnectivity-agent-hm5cr" Oct 31 04:04:06.527188 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.527048 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c3c90e6f-df5e-4c17-bec4-554280a8e8e7-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bt9c9\" (UID: \"c3c90e6f-df5e-4c17-bec4-554280a8e8e7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bt9c9" Oct 31 04:04:06.527188 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.527057 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/89ea384b-705f-47a9-8df6-d3c266ed6761-agent-certs\") pod \"konnectivity-agent-hm5cr\" (UID: \"89ea384b-705f-47a9-8df6-d3c266ed6761\") " pod="kube-system/konnectivity-agent-hm5cr" Oct 31 04:04:06.527188 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.527061 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3945ff65-e877-4372-9334-3968c68b7295-config\") pod \"console-operator-7cd56b8dbc-s88kp\" (UID: \"3945ff65-e877-4372-9334-3968c68b7295\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-s88kp" Oct 31 04:04:06.527188 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.527090 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-host-run-netns\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:04:06.527188 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.527118 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-etc-kubernetes\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:04:06.527188 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.527174 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-crio-socket\") pod \"insights-runtime-extractor-c6gcg\" (UID: \"9b7bf8e5-7877-46e0-88dd-a3503b7639c5\") " pod="openshift-insights/insights-runtime-extractor-c6gcg" Oct 31 04:04:06.527514 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.527217 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a173790b-6c13-45a1-a361-3fe51844b4c2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-zqvkx\" (UID: \"a173790b-6c13-45a1-a361-3fe51844b4c2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zqvkx" Oct 31 04:04:06.527514 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.527236 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c3c90e6f-df5e-4c17-bec4-554280a8e8e7-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bt9c9\" (UID: \"c3c90e6f-df5e-4c17-bec4-554280a8e8e7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bt9c9" Oct 31 04:04:06.527514 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.527255 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97c312b1-4ca5-49c8-8c8d-d247c0fa14ac-metrics-certs\") pod \"network-metrics-daemon-clnlz\" (UID: \"97c312b1-4ca5-49c8-8c8d-d247c0fa14ac\") " pod="openshift-multus/network-metrics-daemon-clnlz" Oct 31 04:04:06.527514 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.527280 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5hsv\" (UniqueName: \"kubernetes.io/projected/ce86be55-219f-46fb-bff6-6dcf87a40ef4-kube-api-access-x5hsv\") pod \"iptables-alerter-bhgvp\" (UID: \"ce86be55-219f-46fb-bff6-6dcf87a40ef4\") " pod="openshift-network-operator/iptables-alerter-bhgvp" Oct 31 04:04:06.527514 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.527295 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-etc-sysctl-d\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:04:06.527514 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.527297 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7f6cd22a-499a-4c35-b408-b2aa9b78620d-ovnkube-script-lib\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:04:06.527514 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.527315 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ce86be55-219f-46fb-bff6-6dcf87a40ef4-host-slash\") pod \"iptables-alerter-bhgvp\" (UID: \"ce86be55-219f-46fb-bff6-6dcf87a40ef4\") " pod="openshift-network-operator/iptables-alerter-bhgvp" Oct 31 04:04:06.527514 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.527482 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c6gcg\" (UID: \"9b7bf8e5-7877-46e0-88dd-a3503b7639c5\") " pod="openshift-insights/insights-runtime-extractor-c6gcg" Oct 31 04:04:06.527862 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.527536 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k52nk\" (UniqueName: \"kubernetes.io/projected/2050cf2e-0f83-43b3-bb85-7bd4ac42a473-kube-api-access-k52nk\") pod \"kas-connection-checker-685895c7c9-z75pq\" (UID: \"2050cf2e-0f83-43b3-bb85-7bd4ac42a473\") " pod="kube-system/kas-connection-checker-685895c7c9-z75pq" Oct 31 04:04:06.527862 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.527574 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/197dee13-748f-4c6d-acc5-1793b5cee80a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-c64cl\" (UID: \"197dee13-748f-4c6d-acc5-1793b5cee80a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c64cl" Oct 31 04:04:06.527862 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.527599 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f5pv\" (UniqueName: \"kubernetes.io/projected/97c312b1-4ca5-49c8-8c8d-d247c0fa14ac-kube-api-access-2f5pv\") pod \"network-metrics-daemon-clnlz\" (UID: \"97c312b1-4ca5-49c8-8c8d-d247c0fa14ac\") " pod="openshift-multus/network-metrics-daemon-clnlz" Oct 31 04:04:06.527862 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.527616 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-var-lib-openvswitch\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:04:06.527862 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.527634 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7f6cd22a-499a-4c35-b408-b2aa9b78620d-ovnkube-config\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:04:06.527862 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.527651 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjq6x\" (UniqueName: \"kubernetes.io/projected/aef43d62-9761-4262-8183-01b9890462a7-kube-api-access-kjq6x\") pod \"network-check-target-4b7d5\" (UID: \"aef43d62-9761-4262-8183-01b9890462a7\") " pod="openshift-network-diagnostics/network-check-target-4b7d5" Oct 31 04:04:06.527862 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.527694 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4pc9\" (UniqueName: \"kubernetes.io/projected/07320d54-b911-4157-aaf5-2517187d745c-kube-api-access-x4pc9\") pod \"periodic-gathering-wh6x6-t7s8v\" (UID: \"07320d54-b911-4157-aaf5-2517187d745c\") " pod="openshift-insights/periodic-gathering-wh6x6-t7s8v" Oct 31 04:04:06.527862 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.527856 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7f6cd22a-499a-4c35-b408-b2aa9b78620d-ovnkube-config\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:04:06.528279 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.527870 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frshf\" (UniqueName: \"kubernetes.io/projected/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-kube-api-access-frshf\") pod \"insights-runtime-extractor-c6gcg\" (UID: \"9b7bf8e5-7877-46e0-88dd-a3503b7639c5\") " pod="openshift-insights/insights-runtime-extractor-c6gcg" Oct 31 04:04:06.528279 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.527902 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d22707c7-21fa-4511-b0b1-869b41fb2351-metrics-certs\") pod \"router-default-678d747c5f-bgj92\" (UID: \"d22707c7-21fa-4511-b0b1-869b41fb2351\") " pod="openshift-ingress/router-default-678d747c5f-bgj92" Oct 31 04:04:06.528279 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.527928 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-registry-tls\") pod \"image-registry-6f69bcc59c-w96bl\" (UID: \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\") " pod="openshift-image-registry/image-registry-6f69bcc59c-w96bl" Oct 31 04:04:06.528279 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.527980 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-run-ovn\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:04:06.528279 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.528002 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-node-log\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:04:06.528279 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.528017 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5b6d69de-a860-4c82-a03f-23ab7905a5d7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sqbnh\" (UID: \"5b6d69de-a860-4c82-a03f-23ab7905a5d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sqbnh" Oct 31 04:04:06.528279 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.528033 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5b6d69de-a860-4c82-a03f-23ab7905a5d7-socket-dir\") pod \"aws-ebs-csi-driver-node-sqbnh\" (UID: \"5b6d69de-a860-4c82-a03f-23ab7905a5d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sqbnh" Oct 31 04:04:06.528279 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.528086 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48309c9d-5f34-49bf-b55b-f74959984e0a-tmp-dir\") pod \"dns-default-zjpdv\" (UID: \"48309c9d-5f34-49bf-b55b-f74959984e0a\") " pod="openshift-dns/dns-default-zjpdv" Oct 31 04:04:06.528279 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.528138 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/93beb8b8-c9e7-4cce-8baf-8497b971e193-trusted-ca\") pod \"image-registry-7f78cfdfd8-w98g7\" (UID: \"93beb8b8-c9e7-4cce-8baf-8497b971e193\") " pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" Oct 31 04:04:06.528279 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.528178 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/944c9831-b2fa-4f58-bba2-93288793b125-serving-cert\") pod \"service-ca-operator-544d749c8c-khdq4\" (UID: \"944c9831-b2fa-4f58-bba2-93288793b125\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-khdq4" Oct 31 04:04:06.528279 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.528180 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48309c9d-5f34-49bf-b55b-f74959984e0a-tmp-dir\") pod \"dns-default-zjpdv\" (UID: \"48309c9d-5f34-49bf-b55b-f74959984e0a\") " pod="openshift-dns/dns-default-zjpdv" Oct 31 04:04:06.528279 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.528207 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-registry-certificates\") pod \"image-registry-6f69bcc59c-w96bl\" (UID: \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\") " pod="openshift-image-registry/image-registry-6f69bcc59c-w96bl" Oct 31 04:04:06.528725 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.528316 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/944c9831-b2fa-4f58-bba2-93288793b125-serving-cert\") pod \"service-ca-operator-544d749c8c-khdq4\" (UID: \"944c9831-b2fa-4f58-bba2-93288793b125\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-khdq4" Oct 31 04:04:06.528725 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.528322 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-system-cni-dir\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:04:06.528725 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.528367 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07320d54-b911-4157-aaf5-2517187d745c-service-ca-bundle\") pod \"periodic-gathering-wh6x6-t7s8v\" (UID: \"07320d54-b911-4157-aaf5-2517187d745c\") " pod="openshift-insights/periodic-gathering-wh6x6-t7s8v" Oct 31 04:04:06.528725 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.528420 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/93beb8b8-c9e7-4cce-8baf-8497b971e193-registry-certificates\") pod \"image-registry-7f78cfdfd8-w98g7\" (UID: \"93beb8b8-c9e7-4cce-8baf-8497b971e193\") " pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" Oct 31 04:04:06.528725 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.528512 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/93beb8b8-c9e7-4cce-8baf-8497b971e193-trusted-ca\") pod \"image-registry-7f78cfdfd8-w98g7\" (UID: \"93beb8b8-c9e7-4cce-8baf-8497b971e193\") " pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" Oct 31 04:04:06.528725 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.528512 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/183f4178-99c0-4945-b490-c71cbd2780f3-nginx-conf\") pod \"networking-console-plugin-86c478bb88-pnwds\" (UID: \"183f4178-99c0-4945-b490-c71cbd2780f3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pnwds" Oct 31 04:04:06.528725 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.528552 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07320d54-b911-4157-aaf5-2517187d745c-service-ca-bundle\") pod \"periodic-gathering-wh6x6-t7s8v\" (UID: \"07320d54-b911-4157-aaf5-2517187d745c\") " pod="openshift-insights/periodic-gathering-wh6x6-t7s8v" Oct 31 04:04:06.528725 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.528600 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/183f4178-99c0-4945-b490-c71cbd2780f3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pnwds\" (UID: \"183f4178-99c0-4945-b490-c71cbd2780f3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pnwds" Oct 31 04:04:06.528725 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.528627 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-etc-systemd\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:04:06.528725 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.528652 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj5t7\" (UniqueName: \"kubernetes.io/projected/57b47c7b-9c84-4888-be69-64afca28a8c0-kube-api-access-jj5t7\") pod \"kas-connection-checker-685895c7c9-xqd9k\" (UID: \"57b47c7b-9c84-4888-be69-64afca28a8c0\") " pod="kube-system/kas-connection-checker-685895c7c9-xqd9k" Oct 31 04:04:06.528725 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.528676 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5b6d69de-a860-4c82-a03f-23ab7905a5d7-etc-selinux\") pod \"aws-ebs-csi-driver-node-sqbnh\" (UID: \"5b6d69de-a860-4c82-a03f-23ab7905a5d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sqbnh" Oct 31 04:04:06.528725 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.528693 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/183f4178-99c0-4945-b490-c71cbd2780f3-nginx-conf\") pod \"networking-console-plugin-86c478bb88-pnwds\" (UID: \"183f4178-99c0-4945-b490-c71cbd2780f3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pnwds" Oct 31 04:04:06.528725 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.528720 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ce86be55-219f-46fb-bff6-6dcf87a40ef4-iptables-alerter-script\") pod \"iptables-alerter-bhgvp\" (UID: \"ce86be55-219f-46fb-bff6-6dcf87a40ef4\") " pod="openshift-network-operator/iptables-alerter-bhgvp" Oct 31 04:04:06.529350 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.528760 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-host-slash\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:04:06.529350 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.528778 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-host-cni-bin\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:04:06.529350 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.528793 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3a05c43f-e2db-44c0-b9bd-4bdabca92a98-hosts-file\") pod \"node-resolver-nbl9z\" (UID: \"3a05c43f-e2db-44c0-b9bd-4bdabca92a98\") " pod="openshift-dns/node-resolver-nbl9z" Oct 31 04:04:06.529350 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.528844 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jbhv\" (UniqueName: \"kubernetes.io/projected/9c8cb360-ff57-41b1-8244-47588703cba5-kube-api-access-9jbhv\") pod \"migrator-66df5d496b-zg2xl\" (UID: \"9c8cb360-ff57-41b1-8244-47588703cba5\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zg2xl" Oct 31 04:04:06.529350 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.528985 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c3c90e6f-df5e-4c17-bec4-554280a8e8e7-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bt9c9\" (UID: \"c3c90e6f-df5e-4c17-bec4-554280a8e8e7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bt9c9" Oct 31 04:04:06.529350 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.529007 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5b6d69de-a860-4c82-a03f-23ab7905a5d7-registration-dir\") pod \"aws-ebs-csi-driver-node-sqbnh\" (UID: \"5b6d69de-a860-4c82-a03f-23ab7905a5d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sqbnh" Oct 31 04:04:06.529350 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.529024 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/be0026cb-89bc-413a-bfe3-523d980397fa-etc-tuned\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:04:06.529350 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.529021 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ce86be55-219f-46fb-bff6-6dcf87a40ef4-iptables-alerter-script\") pod \"iptables-alerter-bhgvp\" (UID: \"ce86be55-219f-46fb-bff6-6dcf87a40ef4\") " pod="openshift-network-operator/iptables-alerter-bhgvp" Oct 31 04:04:06.529350 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.529045 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/93beb8b8-c9e7-4cce-8baf-8497b971e193-registry-tls\") pod \"image-registry-7f78cfdfd8-w98g7\" (UID: \"93beb8b8-c9e7-4cce-8baf-8497b971e193\") " pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" Oct 31 04:04:06.529350 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.529101 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wddq5\" (UniqueName: \"kubernetes.io/projected/e667439c-7121-4c5f-84fe-b36f2391dc66-kube-api-access-wddq5\") pod \"insights-operator-7c4986d449-ld96l\" (UID: \"e667439c-7121-4c5f-84fe-b36f2391dc66\") " pod="openshift-insights/insights-operator-7c4986d449-ld96l" Oct 31 04:04:06.529350 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.529113 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-registry-certificates\") pod \"image-registry-6f69bcc59c-w96bl\" (UID: \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\") " pod="openshift-image-registry/image-registry-6f69bcc59c-w96bl" Oct 31 04:04:06.529350 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.529152 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/be0026cb-89bc-413a-bfe3-523d980397fa-etc-tuned\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:04:06.529350 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.529170 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqx72\" (UniqueName: \"kubernetes.io/projected/3945ff65-e877-4372-9334-3968c68b7295-kube-api-access-fqx72\") pod \"console-operator-7cd56b8dbc-s88kp\" (UID: \"3945ff65-e877-4372-9334-3968c68b7295\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-s88kp" Oct 31 04:04:06.529350 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.529231 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/93beb8b8-c9e7-4cce-8baf-8497b971e193-registry-certificates\") pod \"image-registry-7f78cfdfd8-w98g7\" (UID: \"93beb8b8-c9e7-4cce-8baf-8497b971e193\") " pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" Oct 31 04:04:06.529350 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.529270 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhkfw\" (UniqueName: \"kubernetes.io/projected/3a05c43f-e2db-44c0-b9bd-4bdabca92a98-kube-api-access-vhkfw\") pod \"node-resolver-nbl9z\" (UID: \"3a05c43f-e2db-44c0-b9bd-4bdabca92a98\") " pod="openshift-dns/node-resolver-nbl9z" Oct 31 04:04:06.529350 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.529308 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c3c90e6f-df5e-4c17-bec4-554280a8e8e7-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bt9c9\" (UID: \"c3c90e6f-df5e-4c17-bec4-554280a8e8e7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bt9c9" Oct 31 04:04:06.530164 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.529460 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a173790b-6c13-45a1-a361-3fe51844b4c2-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-zqvkx\" (UID: \"a173790b-6c13-45a1-a361-3fe51844b4c2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zqvkx" Oct 31 04:04:06.530164 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.529483 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25x8r\" (UniqueName: \"kubernetes.io/projected/93beb8b8-c9e7-4cce-8baf-8497b971e193-kube-api-access-25x8r\") pod \"image-registry-7f78cfdfd8-w98g7\" (UID: \"93beb8b8-c9e7-4cce-8baf-8497b971e193\") " pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" Oct 31 04:04:06.530164 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.529499 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3945ff65-e877-4372-9334-3968c68b7295-serving-cert\") pod \"console-operator-7cd56b8dbc-s88kp\" (UID: \"3945ff65-e877-4372-9334-3968c68b7295\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-s88kp" Oct 31 04:04:06.530164 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.529517 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d22707c7-21fa-4511-b0b1-869b41fb2351-service-ca-bundle\") pod \"router-default-678d747c5f-bgj92\" (UID: \"d22707c7-21fa-4511-b0b1-869b41fb2351\") " pod="openshift-ingress/router-default-678d747c5f-bgj92" Oct 31 04:04:06.530164 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.529539 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghxbr\" (UniqueName: \"kubernetes.io/projected/944c9831-b2fa-4f58-bba2-93288793b125-kube-api-access-ghxbr\") pod \"service-ca-operator-544d749c8c-khdq4\" (UID: \"944c9831-b2fa-4f58-bba2-93288793b125\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-khdq4" Oct 31 04:04:06.530164 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.529557 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd5qc\" (UniqueName: \"kubernetes.io/projected/aaae68b8-1654-474f-9e74-45d1e2025d20-kube-api-access-nd5qc\") pod \"multus-additional-cni-plugins-6zzkx\" (UID: \"aaae68b8-1654-474f-9e74-45d1e2025d20\") " pod="openshift-multus/multus-additional-cni-plugins-6zzkx" Oct 31 04:04:06.530164 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.529572 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-host-cni-netd\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:04:06.530164 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.529595 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxlpp\" (UniqueName: \"kubernetes.io/projected/c4a091a2-58d2-4b05-a968-d16fe1bd9d40-kube-api-access-nxlpp\") pod \"volume-data-source-validator-58c9f5f46d-hntxx\" (UID: \"c4a091a2-58d2-4b05-a968-d16fe1bd9d40\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hntxx" Oct 31 04:04:06.530164 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.529664 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3945ff65-e877-4372-9334-3968c68b7295-serving-cert\") pod \"console-operator-7cd56b8dbc-s88kp\" (UID: \"3945ff65-e877-4372-9334-3968c68b7295\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-s88kp" Oct 31 04:04:06.530164 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.529708 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vrvs\" (UniqueName: \"kubernetes.io/projected/49140fa3-6ce2-41ea-8b6b-e25257efaf3e-kube-api-access-6vrvs\") pod \"migrator-66df5d496b-fvqtv\" (UID: \"49140fa3-6ce2-41ea-8b6b-e25257efaf3e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fvqtv" Oct 31 04:04:06.530164 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.529772 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aaae68b8-1654-474f-9e74-45d1e2025d20-cni-binary-copy\") pod \"multus-additional-cni-plugins-6zzkx\" (UID: \"aaae68b8-1654-474f-9e74-45d1e2025d20\") " pod="openshift-multus/multus-additional-cni-plugins-6zzkx" Oct 31 04:04:06.530164 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.529803 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-run\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:04:06.530164 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.529842 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn7px\" (UniqueName: \"kubernetes.io/projected/be0026cb-89bc-413a-bfe3-523d980397fa-kube-api-access-dn7px\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:04:06.530164 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.529861 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-data-volume\") pod \"insights-runtime-extractor-c6gcg\" (UID: \"9b7bf8e5-7877-46e0-88dd-a3503b7639c5\") " pod="openshift-insights/insights-runtime-extractor-c6gcg" Oct 31 04:04:06.530164 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.529878 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/aaae68b8-1654-474f-9e74-45d1e2025d20-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6zzkx\" (UID: \"aaae68b8-1654-474f-9e74-45d1e2025d20\") " pod="openshift-multus/multus-additional-cni-plugins-6zzkx" Oct 31 04:04:06.530164 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.529905 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-systemd-units\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:04:06.530164 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.529950 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-host-var-lib-kubelet\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:04:06.530164 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.529956 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aaae68b8-1654-474f-9e74-45d1e2025d20-cni-binary-copy\") pod \"multus-additional-cni-plugins-6zzkx\" (UID: \"aaae68b8-1654-474f-9e74-45d1e2025d20\") " pod="openshift-multus/multus-additional-cni-plugins-6zzkx" Oct 31 04:04:06.530164 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.529951 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-data-volume\") pod \"insights-runtime-extractor-c6gcg\" (UID: \"9b7bf8e5-7877-46e0-88dd-a3503b7639c5\") " pod="openshift-insights/insights-runtime-extractor-c6gcg" Oct 31 04:04:06.530164 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.529940 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a173790b-6c13-45a1-a361-3fe51844b4c2-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-zqvkx\" (UID: \"a173790b-6c13-45a1-a361-3fe51844b4c2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zqvkx" Oct 31 04:04:06.530164 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.529994 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6d336e6e-feb0-40b2-a625-0e21d25bfac5-multus-daemon-config\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:04:06.530164 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.530042 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5nf8\" (UniqueName: \"kubernetes.io/projected/31221235-f863-445b-a5ca-375aae905d38-kube-api-access-x5nf8\") pod \"node-ca-h9sww\" (UID: \"31221235-f863-445b-a5ca-375aae905d38\") " pod="openshift-image-registry/node-ca-h9sww" Oct 31 04:04:06.530164 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.530070 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/48309c9d-5f34-49bf-b55b-f74959984e0a-metrics-tls\") pod \"dns-default-zjpdv\" (UID: \"48309c9d-5f34-49bf-b55b-f74959984e0a\") " pod="openshift-dns/dns-default-zjpdv" Oct 31 04:04:06.530164 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.530079 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/aaae68b8-1654-474f-9e74-45d1e2025d20-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6zzkx\" (UID: \"aaae68b8-1654-474f-9e74-45d1e2025d20\") " pod="openshift-multus/multus-additional-cni-plugins-6zzkx" Oct 31 04:04:06.530164 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.530095 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-run-openvswitch\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:04:06.530164 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.530143 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-host-run-k8s-cni-cncf-io\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:04:06.530164 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.530161 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6d336e6e-feb0-40b2-a625-0e21d25bfac5-multus-daemon-config\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:04:06.530164 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.530167 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c9eeaad0-e2e5-4b90-a0af-2d254c787fac-signing-cabundle\") pod \"service-ca-5fffc97bdf-l25xj\" (UID: \"c9eeaad0-e2e5-4b90-a0af-2d254c787fac\") " pod="openshift-service-ca/service-ca-5fffc97bdf-l25xj" Oct 31 04:04:06.531204 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.530188 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2ce92f3-5b05-4c89-9d53-a57bc85b17ef-cert\") pod \"ingress-canary-vqsgm\" (UID: \"a2ce92f3-5b05-4c89-9d53-a57bc85b17ef\") " pod="openshift-ingress-canary/ingress-canary-vqsgm" Oct 31 04:04:06.531204 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.530206 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc2vq\" (UniqueName: \"kubernetes.io/projected/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-kube-api-access-xc2vq\") pod \"image-registry-6f69bcc59c-w96bl\" (UID: \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\") " pod="openshift-image-registry/image-registry-6f69bcc59c-w96bl" Oct 31 04:04:06.531204 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.530222 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-log-socket\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:04:06.531204 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.530312 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-host-run-multus-certs\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:04:06.531204 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.530316 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c9eeaad0-e2e5-4b90-a0af-2d254c787fac-signing-cabundle\") pod \"service-ca-5fffc97bdf-l25xj\" (UID: \"c9eeaad0-e2e5-4b90-a0af-2d254c787fac\") " pod="openshift-service-ca/service-ca-5fffc97bdf-l25xj" Oct 31 04:04:06.531204 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.530396 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-host-run-netns\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:04:06.531204 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.530404 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2ce92f3-5b05-4c89-9d53-a57bc85b17ef-cert\") pod \"ingress-canary-vqsgm\" (UID: \"a2ce92f3-5b05-4c89-9d53-a57bc85b17ef\") " pod="openshift-ingress-canary/ingress-canary-vqsgm" Oct 31 04:04:06.531204 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.530421 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mpvl\" (UniqueName: \"kubernetes.io/projected/6d336e6e-feb0-40b2-a625-0e21d25bfac5-kube-api-access-9mpvl\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:04:06.531204 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.530452 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5b6d69de-a860-4c82-a03f-23ab7905a5d7-device-dir\") pod \"aws-ebs-csi-driver-node-sqbnh\" (UID: \"5b6d69de-a860-4c82-a03f-23ab7905a5d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sqbnh" Oct 31 04:04:06.531204 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.530514 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/07320d54-b911-4157-aaf5-2517187d745c-archives-path\") pod \"periodic-gathering-wh6x6-t7s8v\" (UID: \"07320d54-b911-4157-aaf5-2517187d745c\") " pod="openshift-insights/periodic-gathering-wh6x6-t7s8v" Oct 31 04:04:06.531204 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.530582 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c6gcg\" (UID: \"9b7bf8e5-7877-46e0-88dd-a3503b7639c5\") " pod="openshift-insights/insights-runtime-extractor-c6gcg" Oct 31 04:04:06.531204 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.530606 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/197dee13-748f-4c6d-acc5-1793b5cee80a-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-c64cl\" (UID: \"197dee13-748f-4c6d-acc5-1793b5cee80a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c64cl" Oct 31 04:04:06.531204 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.530625 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/48309c9d-5f34-49bf-b55b-f74959984e0a-config-volume\") pod \"dns-default-zjpdv\" (UID: \"48309c9d-5f34-49bf-b55b-f74959984e0a\") " pod="openshift-dns/dns-default-zjpdv" Oct 31 04:04:06.531204 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.530627 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/07320d54-b911-4157-aaf5-2517187d745c-archives-path\") pod \"periodic-gathering-wh6x6-t7s8v\" (UID: \"07320d54-b911-4157-aaf5-2517187d745c\") " pod="openshift-insights/periodic-gathering-wh6x6-t7s8v" Oct 31 04:04:06.531204 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.530659 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-host-var-lib-cni-multus\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:04:06.531204 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.530698 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d22707c7-21fa-4511-b0b1-869b41fb2351-default-certificate\") pod \"router-default-678d747c5f-bgj92\" (UID: \"d22707c7-21fa-4511-b0b1-869b41fb2351\") " pod="openshift-ingress/router-default-678d747c5f-bgj92" Oct 31 04:04:06.531204 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.530750 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a05c43f-e2db-44c0-b9bd-4bdabca92a98-tmp-dir\") pod \"node-resolver-nbl9z\" (UID: \"3a05c43f-e2db-44c0-b9bd-4bdabca92a98\") " pod="openshift-dns/node-resolver-nbl9z" Oct 31 04:04:06.531204 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.530781 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w6z5\" (UniqueName: \"kubernetes.io/projected/5b6d69de-a860-4c82-a03f-23ab7905a5d7-kube-api-access-7w6z5\") pod \"aws-ebs-csi-driver-node-sqbnh\" (UID: \"5b6d69de-a860-4c82-a03f-23ab7905a5d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sqbnh" Oct 31 04:04:06.531204 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.530808 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggq72\" (UniqueName: \"kubernetes.io/projected/a173790b-6c13-45a1-a361-3fe51844b4c2-kube-api-access-ggq72\") pod \"cluster-monitoring-operator-884576c57-zqvkx\" (UID: \"a173790b-6c13-45a1-a361-3fe51844b4c2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zqvkx" Oct 31 04:04:06.531204 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.530837 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c1560abd-3afc-4f94-b0bb-ac7ef8b2464a-kubelet-config\") pod \"global-pull-secret-syncer-hnbll\" (UID: \"c1560abd-3afc-4f94-b0bb-ac7ef8b2464a\") " pod="kube-system/global-pull-secret-syncer-hnbll" Oct 31 04:04:06.531204 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.530844 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/197dee13-748f-4c6d-acc5-1793b5cee80a-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-c64cl\" (UID: \"197dee13-748f-4c6d-acc5-1793b5cee80a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c64cl" Oct 31 04:04:06.531204 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.530858 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a05c43f-e2db-44c0-b9bd-4bdabca92a98-tmp-dir\") pod \"node-resolver-nbl9z\" (UID: \"3a05c43f-e2db-44c0-b9bd-4bdabca92a98\") " pod="openshift-dns/node-resolver-nbl9z" Oct 31 04:04:06.531204 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.530863 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-hostroot\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:04:06.531204 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.530947 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d22707c7-21fa-4511-b0b1-869b41fb2351-default-certificate\") pod \"router-default-678d747c5f-bgj92\" (UID: \"d22707c7-21fa-4511-b0b1-869b41fb2351\") " pod="openshift-ingress/router-default-678d747c5f-bgj92" Oct 31 04:04:06.531204 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.530957 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/48309c9d-5f34-49bf-b55b-f74959984e0a-config-volume\") pod \"dns-default-zjpdv\" (UID: \"48309c9d-5f34-49bf-b55b-f74959984e0a\") " pod="openshift-dns/dns-default-zjpdv" Oct 31 04:04:06.531204 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.530960 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-etc-sysctl-conf\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:04:06.531204 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.531011 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/aaae68b8-1654-474f-9e74-45d1e2025d20-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6zzkx\" (UID: \"aaae68b8-1654-474f-9e74-45d1e2025d20\") " pod="openshift-multus/multus-additional-cni-plugins-6zzkx" Oct 31 04:04:06.531204 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.531040 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7f6cd22a-499a-4c35-b408-b2aa9b78620d-ovn-node-metrics-cert\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:04:06.531204 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.531133 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c9eeaad0-e2e5-4b90-a0af-2d254c787fac-signing-key\") pod \"service-ca-5fffc97bdf-l25xj\" (UID: \"c9eeaad0-e2e5-4b90-a0af-2d254c787fac\") " pod="openshift-service-ca/service-ca-5fffc97bdf-l25xj" Oct 31 04:04:06.531204 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.531151 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57gkt\" (UniqueName: \"kubernetes.io/projected/c9eeaad0-e2e5-4b90-a0af-2d254c787fac-kube-api-access-57gkt\") pod \"service-ca-5fffc97bdf-l25xj\" (UID: \"c9eeaad0-e2e5-4b90-a0af-2d254c787fac\") " pod="openshift-service-ca/service-ca-5fffc97bdf-l25xj" Oct 31 04:04:06.531204 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.531156 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/aaae68b8-1654-474f-9e74-45d1e2025d20-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6zzkx\" (UID: \"aaae68b8-1654-474f-9e74-45d1e2025d20\") " pod="openshift-multus/multus-additional-cni-plugins-6zzkx" Oct 31 04:04:06.531204 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.531168 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt6hk\" (UniqueName: \"kubernetes.io/projected/48309c9d-5f34-49bf-b55b-f74959984e0a-kube-api-access-kt6hk\") pod \"dns-default-zjpdv\" (UID: \"48309c9d-5f34-49bf-b55b-f74959984e0a\") " pod="openshift-dns/dns-default-zjpdv" Oct 31 04:04:06.531204 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.531184 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/93beb8b8-c9e7-4cce-8baf-8497b971e193-image-registry-private-configuration\") pod \"image-registry-7f78cfdfd8-w98g7\" (UID: \"93beb8b8-c9e7-4cce-8baf-8497b971e193\") " pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" Oct 31 04:04:06.531204 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.531201 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/93beb8b8-c9e7-4cce-8baf-8497b971e193-installation-pull-secrets\") pod \"image-registry-7f78cfdfd8-w98g7\" (UID: \"93beb8b8-c9e7-4cce-8baf-8497b971e193\") " pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" Oct 31 04:04:06.531204 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.531218 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/944c9831-b2fa-4f58-bba2-93288793b125-config\") pod \"service-ca-operator-544d749c8c-khdq4\" (UID: \"944c9831-b2fa-4f58-bba2-93288793b125\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-khdq4" Oct 31 04:04:06.532551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.531234 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-run-systemd\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:04:06.532551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.531256 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-ca-trust-extracted\") pod \"image-registry-6f69bcc59c-w96bl\" (UID: \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\") " pod="openshift-image-registry/image-registry-6f69bcc59c-w96bl" Oct 31 04:04:06.532551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.531273 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-trusted-ca\") pod \"image-registry-6f69bcc59c-w96bl\" (UID: \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\") " pod="openshift-image-registry/image-registry-6f69bcc59c-w96bl" Oct 31 04:04:06.532551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.531290 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aaae68b8-1654-474f-9e74-45d1e2025d20-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6zzkx\" (UID: \"aaae68b8-1654-474f-9e74-45d1e2025d20\") " pod="openshift-multus/multus-additional-cni-plugins-6zzkx" Oct 31 04:04:06.532551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.531299 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c9eeaad0-e2e5-4b90-a0af-2d254c787fac-signing-key\") pod \"service-ca-5fffc97bdf-l25xj\" (UID: \"c9eeaad0-e2e5-4b90-a0af-2d254c787fac\") " pod="openshift-service-ca/service-ca-5fffc97bdf-l25xj" Oct 31 04:04:06.532551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.531359 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e667439c-7121-4c5f-84fe-b36f2391dc66-tmp\") pod \"insights-operator-7c4986d449-ld96l\" (UID: \"e667439c-7121-4c5f-84fe-b36f2391dc66\") " pod="openshift-insights/insights-operator-7c4986d449-ld96l" Oct 31 04:04:06.532551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.531384 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e667439c-7121-4c5f-84fe-b36f2391dc66-snapshots\") pod \"insights-operator-7c4986d449-ld96l\" (UID: \"e667439c-7121-4c5f-84fe-b36f2391dc66\") " pod="openshift-insights/insights-operator-7c4986d449-ld96l" Oct 31 04:04:06.532551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.531409 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-etc-kubernetes\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:04:06.532551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.531425 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7f6cd22a-499a-4c35-b408-b2aa9b78620d-ovn-node-metrics-cert\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:04:06.532551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.531447 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-multus-cni-dir\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:04:06.532551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.531473 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be0026cb-89bc-413a-bfe3-523d980397fa-tmp\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:04:06.532551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.531501 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/93beb8b8-c9e7-4cce-8baf-8497b971e193-installation-pull-secrets\") pod \"image-registry-7f78cfdfd8-w98g7\" (UID: \"93beb8b8-c9e7-4cce-8baf-8497b971e193\") " pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" Oct 31 04:04:06.532551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.531518 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsrbd\" (UniqueName: \"kubernetes.io/projected/fb656990-e663-4fea-8733-18ded9222a79-kube-api-access-hsrbd\") pod \"network-check-source-6f78d59f99-rf8vj\" (UID: \"fb656990-e663-4fea-8733-18ded9222a79\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rf8vj" Oct 31 04:04:06.532551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.531579 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-ca-trust-extracted\") pod \"image-registry-6f69bcc59c-w96bl\" (UID: \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\") " pod="openshift-image-registry/image-registry-6f69bcc59c-w96bl" Oct 31 04:04:06.532551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.531588 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e667439c-7121-4c5f-84fe-b36f2391dc66-tmp\") pod \"insights-operator-7c4986d449-ld96l\" (UID: \"e667439c-7121-4c5f-84fe-b36f2391dc66\") " pod="openshift-insights/insights-operator-7c4986d449-ld96l" Oct 31 04:04:06.532551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.531578 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be0026cb-89bc-413a-bfe3-523d980397fa-tmp\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:04:06.532551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.531601 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/93beb8b8-c9e7-4cce-8baf-8497b971e193-image-registry-private-configuration\") pod \"image-registry-7f78cfdfd8-w98g7\" (UID: \"93beb8b8-c9e7-4cce-8baf-8497b971e193\") " pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" Oct 31 04:04:06.532551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.531643 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e667439c-7121-4c5f-84fe-b36f2391dc66-snapshots\") pod \"insights-operator-7c4986d449-ld96l\" (UID: \"e667439c-7121-4c5f-84fe-b36f2391dc66\") " pod="openshift-insights/insights-operator-7c4986d449-ld96l" Oct 31 04:04:06.532551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.531690 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-host-kubelet\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:04:06.532551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.531724 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phq2f\" (UniqueName: \"kubernetes.io/projected/7f6cd22a-499a-4c35-b408-b2aa9b78620d-kube-api-access-phq2f\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:04:06.532551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.531749 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-trusted-ca\") pod \"image-registry-6f69bcc59c-w96bl\" (UID: \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\") " pod="openshift-image-registry/image-registry-6f69bcc59c-w96bl" Oct 31 04:04:06.532551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.531755 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/944c9831-b2fa-4f58-bba2-93288793b125-config\") pod \"service-ca-operator-544d749c8c-khdq4\" (UID: \"944c9831-b2fa-4f58-bba2-93288793b125\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-khdq4" Oct 31 04:04:06.532551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.531769 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e667439c-7121-4c5f-84fe-b36f2391dc66-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-ld96l\" (UID: \"e667439c-7121-4c5f-84fe-b36f2391dc66\") " pod="openshift-insights/insights-operator-7c4986d449-ld96l" Oct 31 04:04:06.532551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.531828 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-multus-socket-dir-parent\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:04:06.532551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.531879 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07320d54-b911-4157-aaf5-2517187d745c-trusted-ca-bundle\") pod \"periodic-gathering-wh6x6-t7s8v\" (UID: \"07320d54-b911-4157-aaf5-2517187d745c\") " pod="openshift-insights/periodic-gathering-wh6x6-t7s8v" Oct 31 04:04:06.532551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.531931 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-lib-modules\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:04:06.532551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.531958 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-var-lib-kubelet\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:04:06.532551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.532010 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d22707c7-21fa-4511-b0b1-869b41fb2351-stats-auth\") pod \"router-default-678d747c5f-bgj92\" (UID: \"d22707c7-21fa-4511-b0b1-869b41fb2351\") " pod="openshift-ingress/router-default-678d747c5f-bgj92" Oct 31 04:04:06.532551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.532085 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-bound-sa-token\") pod \"image-registry-6f69bcc59c-w96bl\" (UID: \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\") " pod="openshift-image-registry/image-registry-6f69bcc59c-w96bl" Oct 31 04:04:06.532551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.532104 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-etc-modprobe-d\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:04:06.532551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.532120 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-host\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:04:06.532551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.532121 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e667439c-7121-4c5f-84fe-b36f2391dc66-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-ld96l\" (UID: \"e667439c-7121-4c5f-84fe-b36f2391dc66\") " pod="openshift-insights/insights-operator-7c4986d449-ld96l" Oct 31 04:04:06.532551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.532139 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ll7m\" (UniqueName: \"kubernetes.io/projected/a2ce92f3-5b05-4c89-9d53-a57bc85b17ef-kube-api-access-7ll7m\") pod \"ingress-canary-vqsgm\" (UID: \"a2ce92f3-5b05-4c89-9d53-a57bc85b17ef\") " pod="openshift-ingress-canary/ingress-canary-vqsgm" Oct 31 04:04:06.532551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.532167 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d22707c7-21fa-4511-b0b1-869b41fb2351-stats-auth\") pod \"router-default-678d747c5f-bgj92\" (UID: \"d22707c7-21fa-4511-b0b1-869b41fb2351\") " pod="openshift-ingress/router-default-678d747c5f-bgj92" Oct 31 04:04:06.532551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.532198 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/31221235-f863-445b-a5ca-375aae905d38-serviceca\") pod \"node-ca-h9sww\" (UID: \"31221235-f863-445b-a5ca-375aae905d38\") " pod="openshift-image-registry/node-ca-h9sww" Oct 31 04:04:06.532551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.532285 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07320d54-b911-4157-aaf5-2517187d745c-trusted-ca-bundle\") pod \"periodic-gathering-wh6x6-t7s8v\" (UID: \"07320d54-b911-4157-aaf5-2517187d745c\") " pod="openshift-insights/periodic-gathering-wh6x6-t7s8v" Oct 31 04:04:06.532551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.532313 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/93beb8b8-c9e7-4cce-8baf-8497b971e193-bound-sa-token\") pod \"image-registry-7f78cfdfd8-w98g7\" (UID: \"93beb8b8-c9e7-4cce-8baf-8497b971e193\") " pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" Oct 31 04:04:06.532551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.532360 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aaae68b8-1654-474f-9e74-45d1e2025d20-system-cni-dir\") pod \"multus-additional-cni-plugins-6zzkx\" (UID: \"aaae68b8-1654-474f-9e74-45d1e2025d20\") " pod="openshift-multus/multus-additional-cni-plugins-6zzkx" Oct 31 04:04:06.532551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.532386 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-sys\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:04:06.532551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.532409 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-host-run-ovn-kubernetes\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:04:06.532551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.532445 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7f6cd22a-499a-4c35-b408-b2aa9b78620d-env-overrides\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:04:06.532551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.532473 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-image-registry-private-configuration\") pod \"image-registry-6f69bcc59c-w96bl\" (UID: \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\") " pod="openshift-image-registry/image-registry-6f69bcc59c-w96bl" Oct 31 04:04:06.532551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.532500 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c1560abd-3afc-4f94-b0bb-ac7ef8b2464a-original-pull-secret\") pod \"global-pull-secret-syncer-hnbll\" (UID: \"c1560abd-3afc-4f94-b0bb-ac7ef8b2464a\") " pod="kube-system/global-pull-secret-syncer-hnbll" Oct 31 04:04:06.532551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.532527 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:04:06.532551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.532557 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e667439c-7121-4c5f-84fe-b36f2391dc66-service-ca-bundle\") pod \"insights-operator-7c4986d449-ld96l\" (UID: \"e667439c-7121-4c5f-84fe-b36f2391dc66\") " pod="openshift-insights/insights-operator-7c4986d449-ld96l" Oct 31 04:04:06.532551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.532573 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7f6cd22a-499a-4c35-b408-b2aa9b78620d-env-overrides\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:04:06.533964 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.532582 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mtpj\" (UniqueName: \"kubernetes.io/projected/c3c90e6f-df5e-4c17-bec4-554280a8e8e7-kube-api-access-7mtpj\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bt9c9\" (UID: \"c3c90e6f-df5e-4c17-bec4-554280a8e8e7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bt9c9" Oct 31 04:04:06.533964 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.532700 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-image-registry-private-configuration\") pod \"image-registry-6f69bcc59c-w96bl\" (UID: \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\") " pod="openshift-image-registry/image-registry-6f69bcc59c-w96bl" Oct 31 04:04:06.533964 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.532751 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e667439c-7121-4c5f-84fe-b36f2391dc66-service-ca-bundle\") pod \"insights-operator-7c4986d449-ld96l\" (UID: \"e667439c-7121-4c5f-84fe-b36f2391dc66\") " pod="openshift-insights/insights-operator-7c4986d449-ld96l" Oct 31 04:04:06.533964 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.532755 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlg9n\" (UniqueName: \"kubernetes.io/projected/b541554a-0460-4716-9f14-f7a46567465c-kube-api-access-qlg9n\") pod \"kas-connection-checker-685895c7c9-xgjnm\" (UID: \"b541554a-0460-4716-9f14-f7a46567465c\") " pod="kube-system/kas-connection-checker-685895c7c9-xgjnm" Oct 31 04:04:06.533964 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.532806 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6xnt\" (UniqueName: \"kubernetes.io/projected/d22707c7-21fa-4511-b0b1-869b41fb2351-kube-api-access-v6xnt\") pod \"router-default-678d747c5f-bgj92\" (UID: \"d22707c7-21fa-4511-b0b1-869b41fb2351\") " pod="openshift-ingress/router-default-678d747c5f-bgj92" Oct 31 04:04:06.533964 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.532756 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c1560abd-3afc-4f94-b0bb-ac7ef8b2464a-original-pull-secret\") pod \"global-pull-secret-syncer-hnbll\" (UID: \"c1560abd-3afc-4f94-b0bb-ac7ef8b2464a\") " pod="kube-system/global-pull-secret-syncer-hnbll" Oct 31 04:04:06.533964 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.532836 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-installation-pull-secrets\") pod \"image-registry-6f69bcc59c-w96bl\" (UID: \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\") " pod="openshift-image-registry/image-registry-6f69bcc59c-w96bl" Oct 31 04:04:06.533964 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.532864 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-etc-openvswitch\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:04:06.533964 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.532997 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-installation-pull-secrets\") pod \"image-registry-6f69bcc59c-w96bl\" (UID: \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\") " pod="openshift-image-registry/image-registry-6f69bcc59c-w96bl" Oct 31 04:04:06.533964 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.533252 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/31221235-f863-445b-a5ca-375aae905d38-serviceca\") pod \"node-ca-h9sww\" (UID: \"31221235-f863-445b-a5ca-375aae905d38\") " pod="openshift-image-registry/node-ca-h9sww" Oct 31 04:04:06.547916 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.547885 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5hsv\" (UniqueName: \"kubernetes.io/projected/ce86be55-219f-46fb-bff6-6dcf87a40ef4-kube-api-access-x5hsv\") pod \"iptables-alerter-bhgvp\" (UID: \"ce86be55-219f-46fb-bff6-6dcf87a40ef4\") " pod="openshift-network-operator/iptables-alerter-bhgvp" Oct 31 04:04:06.548948 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.548921 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f5pv\" (UniqueName: \"kubernetes.io/projected/97c312b1-4ca5-49c8-8c8d-d247c0fa14ac-kube-api-access-2f5pv\") pod \"network-metrics-daemon-clnlz\" (UID: \"97c312b1-4ca5-49c8-8c8d-d247c0fa14ac\") " pod="openshift-multus/network-metrics-daemon-clnlz" Oct 31 04:04:06.549074 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.548996 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k52nk\" (UniqueName: \"kubernetes.io/projected/2050cf2e-0f83-43b3-bb85-7bd4ac42a473-kube-api-access-k52nk\") pod \"kas-connection-checker-685895c7c9-z75pq\" (UID: \"2050cf2e-0f83-43b3-bb85-7bd4ac42a473\") " pod="kube-system/kas-connection-checker-685895c7c9-z75pq" Oct 31 04:04:06.549163 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.549143 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cq9b\" (UniqueName: \"kubernetes.io/projected/197dee13-748f-4c6d-acc5-1793b5cee80a-kube-api-access-7cq9b\") pod \"cluster-samples-operator-6ddb6c8b94-c64cl\" (UID: \"197dee13-748f-4c6d-acc5-1793b5cee80a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c64cl" Oct 31 04:04:06.549245 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.549229 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4pc9\" (UniqueName: \"kubernetes.io/projected/07320d54-b911-4157-aaf5-2517187d745c-kube-api-access-x4pc9\") pod \"periodic-gathering-wh6x6-t7s8v\" (UID: \"07320d54-b911-4157-aaf5-2517187d745c\") " pod="openshift-insights/periodic-gathering-wh6x6-t7s8v" Oct 31 04:04:06.549316 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.549296 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjq6x\" (UniqueName: \"kubernetes.io/projected/aef43d62-9761-4262-8183-01b9890462a7-kube-api-access-kjq6x\") pod \"network-check-target-4b7d5\" (UID: \"aef43d62-9761-4262-8183-01b9890462a7\") " pod="openshift-network-diagnostics/network-check-target-4b7d5" Oct 31 04:04:06.550366 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.550320 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frshf\" (UniqueName: \"kubernetes.io/projected/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-kube-api-access-frshf\") pod \"insights-runtime-extractor-c6gcg\" (UID: \"9b7bf8e5-7877-46e0-88dd-a3503b7639c5\") " pod="openshift-insights/insights-runtime-extractor-c6gcg" Oct 31 04:04:06.554319 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.554298 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5nf8\" (UniqueName: \"kubernetes.io/projected/31221235-f863-445b-a5ca-375aae905d38-kube-api-access-x5nf8\") pod \"node-ca-h9sww\" (UID: \"31221235-f863-445b-a5ca-375aae905d38\") " pod="openshift-image-registry/node-ca-h9sww" Oct 31 04:04:06.554434 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.554413 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghxbr\" (UniqueName: \"kubernetes.io/projected/944c9831-b2fa-4f58-bba2-93288793b125-kube-api-access-ghxbr\") pod \"service-ca-operator-544d749c8c-khdq4\" (UID: \"944c9831-b2fa-4f58-bba2-93288793b125\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-khdq4" Oct 31 04:04:06.554903 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.554885 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w6z5\" (UniqueName: \"kubernetes.io/projected/5b6d69de-a860-4c82-a03f-23ab7905a5d7-kube-api-access-7w6z5\") pod \"aws-ebs-csi-driver-node-sqbnh\" (UID: \"5b6d69de-a860-4c82-a03f-23ab7905a5d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sqbnh" Oct 31 04:04:06.559242 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.559192 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jbhv\" (UniqueName: \"kubernetes.io/projected/9c8cb360-ff57-41b1-8244-47588703cba5-kube-api-access-9jbhv\") pod \"migrator-66df5d496b-zg2xl\" (UID: \"9c8cb360-ff57-41b1-8244-47588703cba5\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zg2xl" Oct 31 04:04:06.559437 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.559304 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc2vq\" (UniqueName: \"kubernetes.io/projected/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-kube-api-access-xc2vq\") pod \"image-registry-6f69bcc59c-w96bl\" (UID: \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\") " pod="openshift-image-registry/image-registry-6f69bcc59c-w96bl" Oct 31 04:04:06.559529 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.559432 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxlpp\" (UniqueName: \"kubernetes.io/projected/c4a091a2-58d2-4b05-a968-d16fe1bd9d40-kube-api-access-nxlpp\") pod \"volume-data-source-validator-58c9f5f46d-hntxx\" (UID: \"c4a091a2-58d2-4b05-a968-d16fe1bd9d40\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hntxx" Oct 31 04:04:06.559638 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.559602 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25x8r\" (UniqueName: \"kubernetes.io/projected/93beb8b8-c9e7-4cce-8baf-8497b971e193-kube-api-access-25x8r\") pod \"image-registry-7f78cfdfd8-w98g7\" (UID: \"93beb8b8-c9e7-4cce-8baf-8497b971e193\") " pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" Oct 31 04:04:06.559778 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.559754 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsrbd\" (UniqueName: \"kubernetes.io/projected/fb656990-e663-4fea-8733-18ded9222a79-kube-api-access-hsrbd\") pod \"network-check-source-6f78d59f99-rf8vj\" (UID: \"fb656990-e663-4fea-8733-18ded9222a79\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rf8vj" Oct 31 04:04:06.559930 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.559908 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqx72\" (UniqueName: \"kubernetes.io/projected/3945ff65-e877-4372-9334-3968c68b7295-kube-api-access-fqx72\") pod \"console-operator-7cd56b8dbc-s88kp\" (UID: \"3945ff65-e877-4372-9334-3968c68b7295\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-s88kp" Oct 31 04:04:06.560624 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.560428 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mpvl\" (UniqueName: \"kubernetes.io/projected/6d336e6e-feb0-40b2-a625-0e21d25bfac5-kube-api-access-9mpvl\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:04:06.560624 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.560449 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wddq5\" (UniqueName: \"kubernetes.io/projected/e667439c-7121-4c5f-84fe-b36f2391dc66-kube-api-access-wddq5\") pod \"insights-operator-7c4986d449-ld96l\" (UID: \"e667439c-7121-4c5f-84fe-b36f2391dc66\") " pod="openshift-insights/insights-operator-7c4986d449-ld96l" Oct 31 04:04:06.560624 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.560565 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vrvs\" (UniqueName: \"kubernetes.io/projected/49140fa3-6ce2-41ea-8b6b-e25257efaf3e-kube-api-access-6vrvs\") pod \"migrator-66df5d496b-fvqtv\" (UID: \"49140fa3-6ce2-41ea-8b6b-e25257efaf3e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fvqtv" Oct 31 04:04:06.562471 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.560889 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj5t7\" (UniqueName: \"kubernetes.io/projected/57b47c7b-9c84-4888-be69-64afca28a8c0-kube-api-access-jj5t7\") pod \"kas-connection-checker-685895c7c9-xqd9k\" (UID: \"57b47c7b-9c84-4888-be69-64afca28a8c0\") " pod="kube-system/kas-connection-checker-685895c7c9-xqd9k" Oct 31 04:04:06.562471 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.560948 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57gkt\" (UniqueName: \"kubernetes.io/projected/c9eeaad0-e2e5-4b90-a0af-2d254c787fac-kube-api-access-57gkt\") pod \"service-ca-5fffc97bdf-l25xj\" (UID: \"c9eeaad0-e2e5-4b90-a0af-2d254c787fac\") " pod="openshift-service-ca/service-ca-5fffc97bdf-l25xj" Oct 31 04:04:06.562471 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.561019 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn7px\" (UniqueName: \"kubernetes.io/projected/be0026cb-89bc-413a-bfe3-523d980397fa-kube-api-access-dn7px\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:04:06.562471 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.561077 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhkfw\" (UniqueName: \"kubernetes.io/projected/3a05c43f-e2db-44c0-b9bd-4bdabca92a98-kube-api-access-vhkfw\") pod \"node-resolver-nbl9z\" (UID: \"3a05c43f-e2db-44c0-b9bd-4bdabca92a98\") " pod="openshift-dns/node-resolver-nbl9z" Oct 31 04:04:06.562471 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.561531 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggq72\" (UniqueName: \"kubernetes.io/projected/a173790b-6c13-45a1-a361-3fe51844b4c2-kube-api-access-ggq72\") pod \"cluster-monitoring-operator-884576c57-zqvkx\" (UID: \"a173790b-6c13-45a1-a361-3fe51844b4c2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zqvkx" Oct 31 04:04:06.564774 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.564057 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd5qc\" (UniqueName: \"kubernetes.io/projected/aaae68b8-1654-474f-9e74-45d1e2025d20-kube-api-access-nd5qc\") pod \"multus-additional-cni-plugins-6zzkx\" (UID: \"aaae68b8-1654-474f-9e74-45d1e2025d20\") " pod="openshift-multus/multus-additional-cni-plugins-6zzkx" Oct 31 04:04:06.564774 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.564416 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/93beb8b8-c9e7-4cce-8baf-8497b971e193-bound-sa-token\") pod \"image-registry-7f78cfdfd8-w98g7\" (UID: \"93beb8b8-c9e7-4cce-8baf-8497b971e193\") " pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" Oct 31 04:04:06.564774 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.564596 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlg9n\" (UniqueName: \"kubernetes.io/projected/b541554a-0460-4716-9f14-f7a46567465c-kube-api-access-qlg9n\") pod \"kas-connection-checker-685895c7c9-xgjnm\" (UID: \"b541554a-0460-4716-9f14-f7a46567465c\") " pod="kube-system/kas-connection-checker-685895c7c9-xgjnm" Oct 31 04:04:06.565232 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.565213 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-bound-sa-token\") pod \"image-registry-6f69bcc59c-w96bl\" (UID: \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\") " pod="openshift-image-registry/image-registry-6f69bcc59c-w96bl" Oct 31 04:04:06.565232 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.565220 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mtpj\" (UniqueName: \"kubernetes.io/projected/c3c90e6f-df5e-4c17-bec4-554280a8e8e7-kube-api-access-7mtpj\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bt9c9\" (UID: \"c3c90e6f-df5e-4c17-bec4-554280a8e8e7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bt9c9" Oct 31 04:04:06.565378 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.565221 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phq2f\" (UniqueName: \"kubernetes.io/projected/7f6cd22a-499a-4c35-b408-b2aa9b78620d-kube-api-access-phq2f\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:04:06.565446 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.565420 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt6hk\" (UniqueName: \"kubernetes.io/projected/48309c9d-5f34-49bf-b55b-f74959984e0a-kube-api-access-kt6hk\") pod \"dns-default-zjpdv\" (UID: \"48309c9d-5f34-49bf-b55b-f74959984e0a\") " pod="openshift-dns/dns-default-zjpdv" Oct 31 04:04:06.565516 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.565490 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ll7m\" (UniqueName: \"kubernetes.io/projected/a2ce92f3-5b05-4c89-9d53-a57bc85b17ef-kube-api-access-7ll7m\") pod \"ingress-canary-vqsgm\" (UID: \"a2ce92f3-5b05-4c89-9d53-a57bc85b17ef\") " pod="openshift-ingress-canary/ingress-canary-vqsgm" Oct 31 04:04:06.566089 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.566070 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6xnt\" (UniqueName: \"kubernetes.io/projected/d22707c7-21fa-4511-b0b1-869b41fb2351-kube-api-access-v6xnt\") pod \"router-default-678d747c5f-bgj92\" (UID: \"d22707c7-21fa-4511-b0b1-869b41fb2351\") " pod="openshift-ingress/router-default-678d747c5f-bgj92" Oct 31 04:04:06.634383 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.634268 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-host-var-lib-kubelet\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:04:06.634383 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.634317 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/48309c9d-5f34-49bf-b55b-f74959984e0a-metrics-tls\") pod \"dns-default-zjpdv\" (UID: \"48309c9d-5f34-49bf-b55b-f74959984e0a\") " pod="openshift-dns/dns-default-zjpdv" Oct 31 04:04:06.634383 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.634350 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-run-openvswitch\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:04:06.634654 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.634392 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-run-openvswitch\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:04:06.634654 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.634414 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-host-var-lib-kubelet\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:04:06.634654 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.634476 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-host-run-k8s-cni-cncf-io\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:04:06.634654 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:04:06.634502 5019 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 04:04:06.634654 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.634521 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-log-socket\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:04:06.634654 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.634545 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-host-run-multus-certs\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:04:06.634654 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.634560 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-host-run-k8s-cni-cncf-io\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:04:06.634654 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:04:06.634589 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48309c9d-5f34-49bf-b55b-f74959984e0a-metrics-tls podName:48309c9d-5f34-49bf-b55b-f74959984e0a nodeName:}" failed. No retries permitted until 2026-10-31 04:04:07.134565225 +0000 UTC m=+1.890903262 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/48309c9d-5f34-49bf-b55b-f74959984e0a-metrics-tls") pod "dns-default-zjpdv" (UID: "48309c9d-5f34-49bf-b55b-f74959984e0a") : secret "dns-default-metrics-tls" not found Oct 31 04:04:06.634654 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.634591 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-host-run-multus-certs\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:04:06.634654 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.634588 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-log-socket\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:04:06.634654 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.634611 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-host-run-netns\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:04:06.634654 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.634639 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5b6d69de-a860-4c82-a03f-23ab7905a5d7-device-dir\") pod \"aws-ebs-csi-driver-node-sqbnh\" (UID: \"5b6d69de-a860-4c82-a03f-23ab7905a5d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sqbnh" Oct 31 04:04:06.634654 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.634651 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-host-run-netns\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:04:06.635233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.634662 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c6gcg\" (UID: \"9b7bf8e5-7877-46e0-88dd-a3503b7639c5\") " pod="openshift-insights/insights-runtime-extractor-c6gcg" Oct 31 04:04:06.635233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.634687 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-host-var-lib-cni-multus\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:04:06.635233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.634694 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5b6d69de-a860-4c82-a03f-23ab7905a5d7-device-dir\") pod \"aws-ebs-csi-driver-node-sqbnh\" (UID: \"5b6d69de-a860-4c82-a03f-23ab7905a5d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sqbnh" Oct 31 04:04:06.635233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.634713 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c1560abd-3afc-4f94-b0bb-ac7ef8b2464a-kubelet-config\") pod \"global-pull-secret-syncer-hnbll\" (UID: \"c1560abd-3afc-4f94-b0bb-ac7ef8b2464a\") " pod="kube-system/global-pull-secret-syncer-hnbll" Oct 31 04:04:06.635233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.634729 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-host-var-lib-cni-multus\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:04:06.635233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.634734 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-hostroot\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:04:06.635233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.634763 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-etc-sysctl-conf\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:04:06.635233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.634787 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c1560abd-3afc-4f94-b0bb-ac7ef8b2464a-kubelet-config\") pod \"global-pull-secret-syncer-hnbll\" (UID: \"c1560abd-3afc-4f94-b0bb-ac7ef8b2464a\") " pod="kube-system/global-pull-secret-syncer-hnbll" Oct 31 04:04:06.635233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.634791 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-hostroot\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:04:06.635233 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:04:06.634764 5019 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 04:04:06.635233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.634815 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-run-systemd\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:04:06.635233 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:04:06.634865 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-kube-rbac-proxy-cm podName:9b7bf8e5-7877-46e0-88dd-a3503b7639c5 nodeName:}" failed. No retries permitted until 2026-10-31 04:04:07.134852892 +0000 UTC m=+1.891190910 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c6gcg" (UID: "9b7bf8e5-7877-46e0-88dd-a3503b7639c5") : configmap "kube-rbac-proxy" not found Oct 31 04:04:06.635233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.634902 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-run-systemd\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:04:06.635233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.634912 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aaae68b8-1654-474f-9e74-45d1e2025d20-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6zzkx\" (UID: \"aaae68b8-1654-474f-9e74-45d1e2025d20\") " pod="openshift-multus/multus-additional-cni-plugins-6zzkx" Oct 31 04:04:06.635233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.634913 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-etc-sysctl-conf\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:04:06.635233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.634933 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-etc-kubernetes\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:04:06.635233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.634948 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-multus-cni-dir\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:04:06.635233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.634966 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-host-kubelet\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:04:06.635233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.634982 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-multus-socket-dir-parent\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:04:06.635233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.634997 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-lib-modules\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:04:06.635233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635009 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-etc-kubernetes\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:04:06.635233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635015 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-var-lib-kubelet\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:04:06.635233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635025 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-host-kubelet\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:04:06.635233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635033 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-multus-cni-dir\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:04:06.635233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635057 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-etc-modprobe-d\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:04:06.635233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635057 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-multus-socket-dir-parent\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:04:06.635233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635066 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aaae68b8-1654-474f-9e74-45d1e2025d20-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6zzkx\" (UID: \"aaae68b8-1654-474f-9e74-45d1e2025d20\") " pod="openshift-multus/multus-additional-cni-plugins-6zzkx" Oct 31 04:04:06.635233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635072 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-var-lib-kubelet\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:04:06.635233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635085 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-host\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:04:06.635233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635116 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aaae68b8-1654-474f-9e74-45d1e2025d20-system-cni-dir\") pod \"multus-additional-cni-plugins-6zzkx\" (UID: \"aaae68b8-1654-474f-9e74-45d1e2025d20\") " pod="openshift-multus/multus-additional-cni-plugins-6zzkx" Oct 31 04:04:06.635233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635118 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-lib-modules\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:04:06.635233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635124 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-host\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:04:06.635233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635138 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-sys\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:04:06.635233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635152 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aaae68b8-1654-474f-9e74-45d1e2025d20-system-cni-dir\") pod \"multus-additional-cni-plugins-6zzkx\" (UID: \"aaae68b8-1654-474f-9e74-45d1e2025d20\") " pod="openshift-multus/multus-additional-cni-plugins-6zzkx" Oct 31 04:04:06.635233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635161 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-host-run-ovn-kubernetes\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:04:06.635233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635172 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-sys\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:04:06.635233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635180 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:04:06.635233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635151 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-etc-modprobe-d\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:04:06.635233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635200 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-etc-openvswitch\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:04:06.635233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635208 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-host-run-ovn-kubernetes\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:04:06.635233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635216 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-cnibin\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:04:06.635233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635223 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:04:06.635233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635251 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-cnibin\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:04:06.635233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635253 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-etc-openvswitch\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635281 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-host-var-lib-cni-bin\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635249 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-host-var-lib-cni-bin\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635308 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-etc-sysconfig\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635325 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aaae68b8-1654-474f-9e74-45d1e2025d20-cnibin\") pod \"multus-additional-cni-plugins-6zzkx\" (UID: \"aaae68b8-1654-474f-9e74-45d1e2025d20\") " pod="openshift-multus/multus-additional-cni-plugins-6zzkx" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635357 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-multus-conf-dir\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635359 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-etc-sysconfig\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635374 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/31221235-f863-445b-a5ca-375aae905d38-host\") pod \"node-ca-h9sww\" (UID: \"31221235-f863-445b-a5ca-375aae905d38\") " pod="openshift-image-registry/node-ca-h9sww" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635389 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c1560abd-3afc-4f94-b0bb-ac7ef8b2464a-dbus\") pod \"global-pull-secret-syncer-hnbll\" (UID: \"c1560abd-3afc-4f94-b0bb-ac7ef8b2464a\") " pod="kube-system/global-pull-secret-syncer-hnbll" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635403 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aaae68b8-1654-474f-9e74-45d1e2025d20-cnibin\") pod \"multus-additional-cni-plugins-6zzkx\" (UID: \"aaae68b8-1654-474f-9e74-45d1e2025d20\") " pod="openshift-multus/multus-additional-cni-plugins-6zzkx" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635410 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5b6d69de-a860-4c82-a03f-23ab7905a5d7-sys-fs\") pod \"aws-ebs-csi-driver-node-sqbnh\" (UID: \"5b6d69de-a860-4c82-a03f-23ab7905a5d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sqbnh" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635434 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-multus-conf-dir\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635438 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-host-run-netns\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635437 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/31221235-f863-445b-a5ca-375aae905d38-host\") pod \"node-ca-h9sww\" (UID: \"31221235-f863-445b-a5ca-375aae905d38\") " pod="openshift-image-registry/node-ca-h9sww" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635458 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c1560abd-3afc-4f94-b0bb-ac7ef8b2464a-dbus\") pod \"global-pull-secret-syncer-hnbll\" (UID: \"c1560abd-3afc-4f94-b0bb-ac7ef8b2464a\") " pod="kube-system/global-pull-secret-syncer-hnbll" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635463 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-etc-kubernetes\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635467 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5b6d69de-a860-4c82-a03f-23ab7905a5d7-sys-fs\") pod \"aws-ebs-csi-driver-node-sqbnh\" (UID: \"5b6d69de-a860-4c82-a03f-23ab7905a5d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sqbnh" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635486 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-host-run-netns\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635505 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-etc-kubernetes\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635509 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-crio-socket\") pod \"insights-runtime-extractor-c6gcg\" (UID: \"9b7bf8e5-7877-46e0-88dd-a3503b7639c5\") " pod="openshift-insights/insights-runtime-extractor-c6gcg" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635536 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a173790b-6c13-45a1-a361-3fe51844b4c2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-zqvkx\" (UID: \"a173790b-6c13-45a1-a361-3fe51844b4c2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zqvkx" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635562 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97c312b1-4ca5-49c8-8c8d-d247c0fa14ac-metrics-certs\") pod \"network-metrics-daemon-clnlz\" (UID: \"97c312b1-4ca5-49c8-8c8d-d247c0fa14ac\") " pod="openshift-multus/network-metrics-daemon-clnlz" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635589 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-crio-socket\") pod \"insights-runtime-extractor-c6gcg\" (UID: \"9b7bf8e5-7877-46e0-88dd-a3503b7639c5\") " pod="openshift-insights/insights-runtime-extractor-c6gcg" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635597 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-etc-sysctl-d\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635616 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ce86be55-219f-46fb-bff6-6dcf87a40ef4-host-slash\") pod \"iptables-alerter-bhgvp\" (UID: \"ce86be55-219f-46fb-bff6-6dcf87a40ef4\") " pod="openshift-network-operator/iptables-alerter-bhgvp" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635632 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c6gcg\" (UID: \"9b7bf8e5-7877-46e0-88dd-a3503b7639c5\") " pod="openshift-insights/insights-runtime-extractor-c6gcg" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635659 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/197dee13-748f-4c6d-acc5-1793b5cee80a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-c64cl\" (UID: \"197dee13-748f-4c6d-acc5-1793b5cee80a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c64cl" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635680 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ce86be55-219f-46fb-bff6-6dcf87a40ef4-host-slash\") pod \"iptables-alerter-bhgvp\" (UID: \"ce86be55-219f-46fb-bff6-6dcf87a40ef4\") " pod="openshift-network-operator/iptables-alerter-bhgvp" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635724 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-var-lib-openvswitch\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635729 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-etc-sysctl-d\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:04:06.635739 5019 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635756 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d22707c7-21fa-4511-b0b1-869b41fb2351-metrics-certs\") pod \"router-default-678d747c5f-bgj92\" (UID: \"d22707c7-21fa-4511-b0b1-869b41fb2351\") " pod="openshift-ingress/router-default-678d747c5f-bgj92" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635770 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-var-lib-openvswitch\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635777 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-registry-tls\") pod \"image-registry-6f69bcc59c-w96bl\" (UID: \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\") " pod="openshift-image-registry/image-registry-6f69bcc59c-w96bl" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:04:06.635739 5019 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:04:06.635803 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a173790b-6c13-45a1-a361-3fe51844b4c2-cluster-monitoring-operator-tls podName:a173790b-6c13-45a1-a361-3fe51844b4c2 nodeName:}" failed. No retries permitted until 2026-10-31 04:04:07.135786972 +0000 UTC m=+1.892125013 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a173790b-6c13-45a1-a361-3fe51844b4c2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-zqvkx" (UID: "a173790b-6c13-45a1-a361-3fe51844b4c2") : secret "cluster-monitoring-operator-tls" not found Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:04:06.635740 5019 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:04:06.635843 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-insights-runtime-extractor-tls podName:9b7bf8e5-7877-46e0-88dd-a3503b7639c5 nodeName:}" failed. No retries permitted until 2026-10-31 04:04:07.135826524 +0000 UTC m=+1.892164548 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c6gcg" (UID: "9b7bf8e5-7877-46e0-88dd-a3503b7639c5") : secret "insights-runtime-extractor-tls" not found Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635860 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-run-ovn\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635885 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-node-log\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:04:06.635893 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/197dee13-748f-4c6d-acc5-1793b5cee80a-samples-operator-tls podName:197dee13-748f-4c6d-acc5-1793b5cee80a nodeName:}" failed. No retries permitted until 2026-10-31 04:04:07.135883543 +0000 UTC m=+1.892221563 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/197dee13-748f-4c6d-acc5-1793b5cee80a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-c64cl" (UID: "197dee13-748f-4c6d-acc5-1793b5cee80a") : secret "samples-operator-tls" not found Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635908 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-run-ovn\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635915 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5b6d69de-a860-4c82-a03f-23ab7905a5d7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sqbnh\" (UID: \"5b6d69de-a860-4c82-a03f-23ab7905a5d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sqbnh" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:04:06.635931 5019 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:04:06.635944 5019 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f69bcc59c-w96bl: secret "image-registry-tls" not found Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635940 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5b6d69de-a860-4c82-a03f-23ab7905a5d7-socket-dir\") pod \"aws-ebs-csi-driver-node-sqbnh\" (UID: \"5b6d69de-a860-4c82-a03f-23ab7905a5d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sqbnh" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635917 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-node-log\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635976 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-system-cni-dir\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.635984 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5b6d69de-a860-4c82-a03f-23ab7905a5d7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sqbnh\" (UID: \"5b6d69de-a860-4c82-a03f-23ab7905a5d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sqbnh" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:04:06.636001 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-registry-tls podName:e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22 nodeName:}" failed. No retries permitted until 2026-10-31 04:04:07.135987576 +0000 UTC m=+1.892325614 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-registry-tls") pod "image-registry-6f69bcc59c-w96bl" (UID: "e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22") : secret "image-registry-tls" not found Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.636004 5019 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.636021 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d336e6e-feb0-40b2-a625-0e21d25bfac5-system-cni-dir\") pod \"multus-6d7cp\" (UID: \"6d336e6e-feb0-40b2-a625-0e21d25bfac5\") " pod="openshift-multus/multus-6d7cp" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.636026 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5b6d69de-a860-4c82-a03f-23ab7905a5d7-socket-dir\") pod \"aws-ebs-csi-driver-node-sqbnh\" (UID: \"5b6d69de-a860-4c82-a03f-23ab7905a5d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sqbnh" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.636029 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/183f4178-99c0-4945-b490-c71cbd2780f3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pnwds\" (UID: \"183f4178-99c0-4945-b490-c71cbd2780f3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pnwds" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.636049 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-etc-systemd\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.636073 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5b6d69de-a860-4c82-a03f-23ab7905a5d7-etc-selinux\") pod \"aws-ebs-csi-driver-node-sqbnh\" (UID: \"5b6d69de-a860-4c82-a03f-23ab7905a5d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sqbnh" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.636098 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-host-slash\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.636110 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-etc-systemd\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.636122 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-host-cni-bin\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.636146 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3a05c43f-e2db-44c0-b9bd-4bdabca92a98-hosts-file\") pod \"node-resolver-nbl9z\" (UID: \"3a05c43f-e2db-44c0-b9bd-4bdabca92a98\") " pod="openshift-dns/node-resolver-nbl9z" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.636154 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-host-slash\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.636170 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5b6d69de-a860-4c82-a03f-23ab7905a5d7-etc-selinux\") pod \"aws-ebs-csi-driver-node-sqbnh\" (UID: \"5b6d69de-a860-4c82-a03f-23ab7905a5d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sqbnh" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.636180 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5b6d69de-a860-4c82-a03f-23ab7905a5d7-registration-dir\") pod \"aws-ebs-csi-driver-node-sqbnh\" (UID: \"5b6d69de-a860-4c82-a03f-23ab7905a5d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sqbnh" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.636203 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3a05c43f-e2db-44c0-b9bd-4bdabca92a98-hosts-file\") pod \"node-resolver-nbl9z\" (UID: \"3a05c43f-e2db-44c0-b9bd-4bdabca92a98\") " pod="openshift-dns/node-resolver-nbl9z" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.636208 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/93beb8b8-c9e7-4cce-8baf-8497b971e193-registry-tls\") pod \"image-registry-7f78cfdfd8-w98g7\" (UID: \"93beb8b8-c9e7-4cce-8baf-8497b971e193\") " pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.636225 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-host-cni-bin\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.636243 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d22707c7-21fa-4511-b0b1-869b41fb2351-service-ca-bundle\") pod \"router-default-678d747c5f-bgj92\" (UID: \"d22707c7-21fa-4511-b0b1-869b41fb2351\") " pod="openshift-ingress/router-default-678d747c5f-bgj92" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:04:06.636271 5019 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:04:06.636283 5019 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f78cfdfd8-w98g7: secret "image-registry-tls" not found Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.636283 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5b6d69de-a860-4c82-a03f-23ab7905a5d7-registration-dir\") pod \"aws-ebs-csi-driver-node-sqbnh\" (UID: \"5b6d69de-a860-4c82-a03f-23ab7905a5d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sqbnh" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.636305 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-host-cni-netd\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.636271 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-host-cni-netd\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:04:06.636354 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93beb8b8-c9e7-4cce-8baf-8497b971e193-registry-tls podName:93beb8b8-c9e7-4cce-8baf-8497b971e193 nodeName:}" failed. No retries permitted until 2026-10-31 04:04:07.13631731 +0000 UTC m=+1.892655346 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/93beb8b8-c9e7-4cce-8baf-8497b971e193-registry-tls") pod "image-registry-7f78cfdfd8-w98g7" (UID: "93beb8b8-c9e7-4cce-8baf-8497b971e193") : secret "image-registry-tls" not found Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.636387 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-run\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.636411 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-systemd-units\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.636452 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/be0026cb-89bc-413a-bfe3-523d980397fa-run\") pod \"tuned-26vsf\" (UID: \"be0026cb-89bc-413a-bfe3-523d980397fa\") " pod="openshift-cluster-node-tuning-operator/tuned-26vsf" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.636471 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7f6cd22a-499a-4c35-b408-b2aa9b78620d-systemd-units\") pod \"ovnkube-node-tmt7k\" (UID: \"7f6cd22a-499a-4c35-b408-b2aa9b78620d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:04:06.637140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.636921 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d22707c7-21fa-4511-b0b1-869b41fb2351-service-ca-bundle\") pod \"router-default-678d747c5f-bgj92\" (UID: \"d22707c7-21fa-4511-b0b1-869b41fb2351\") " pod="openshift-ingress/router-default-678d747c5f-bgj92" Oct 31 04:04:06.639646 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.638627 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/183f4178-99c0-4945-b490-c71cbd2780f3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pnwds\" (UID: \"183f4178-99c0-4945-b490-c71cbd2780f3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pnwds" Oct 31 04:04:06.639646 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.638755 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d22707c7-21fa-4511-b0b1-869b41fb2351-metrics-certs\") pod \"router-default-678d747c5f-bgj92\" (UID: \"d22707c7-21fa-4511-b0b1-869b41fb2351\") " pod="openshift-ingress/router-default-678d747c5f-bgj92" Oct 31 04:04:06.640626 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.640600 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97c312b1-4ca5-49c8-8c8d-d247c0fa14ac-metrics-certs\") pod \"network-metrics-daemon-clnlz\" (UID: \"97c312b1-4ca5-49c8-8c8d-d247c0fa14ac\") " pod="openshift-multus/network-metrics-daemon-clnlz" Oct 31 04:04:06.720736 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.720699 5019 scope.go:122] "RemoveContainer" containerID="e11078a797b967674e01f514aab383ae2b67709c9bc64c60177a719670d63040" Oct 31 04:04:06.720736 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.720738 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-678d747c5f-bgj92" Oct 31 04:04:06.727918 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.727894 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clnlz" Oct 31 04:04:06.728077 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:06.727982 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-pnwds" Oct 31 04:04:07.021065 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:07.021033 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vqsgm_a2ce92f3-5b05-4c89-9d53-a57bc85b17ef/serve-healthcheck-canary/0.log" Oct 31 04:04:07.050269 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:07.050233 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-678d747c5f-bgj92"] Oct 31 04:04:07.053602 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:07.053572 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd22707c7_21fa_4511_b0b1_869b41fb2351.slice/crio-9021322515eae83cfc0a38437570f07c1739923cb87e7fc9ea6561e254bfc218 WatchSource:0}: Error finding container 9021322515eae83cfc0a38437570f07c1739923cb87e7fc9ea6561e254bfc218: Status 404 returned error can't find the container with id 9021322515eae83cfc0a38437570f07c1739923cb87e7fc9ea6561e254bfc218 Oct 31 04:04:07.059060 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:07.058999 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-clnlz"] Oct 31 04:04:07.064275 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:07.064239 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97c312b1_4ca5_49c8_8c8d_d247c0fa14ac.slice/crio-01c56e071d428f3025a47782ea70fb5ba6a6ebeba9ff10410e8580ab72f1bca0 WatchSource:0}: Error finding container 01c56e071d428f3025a47782ea70fb5ba6a6ebeba9ff10410e8580ab72f1bca0: Status 404 returned error can't find the container with id 01c56e071d428f3025a47782ea70fb5ba6a6ebeba9ff10410e8580ab72f1bca0 Oct 31 04:04:07.066595 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:07.066574 5019 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 04:04:07.078973 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:07.078950 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-pnwds"] Oct 31 04:04:07.082910 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:07.082882 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod183f4178_99c0_4945_b490_c71cbd2780f3.slice/crio-262af931ef5c6d9eb0c83ccbbddac0f0fbedc47792a7e202501f095814e5194e WatchSource:0}: Error finding container 262af931ef5c6d9eb0c83ccbbddac0f0fbedc47792a7e202501f095814e5194e: Status 404 returned error can't find the container with id 262af931ef5c6d9eb0c83ccbbddac0f0fbedc47792a7e202501f095814e5194e Oct 31 04:04:07.140590 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:07.140553 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c6gcg\" (UID: \"9b7bf8e5-7877-46e0-88dd-a3503b7639c5\") " pod="openshift-insights/insights-runtime-extractor-c6gcg" Oct 31 04:04:07.140743 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:07.140652 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a173790b-6c13-45a1-a361-3fe51844b4c2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-zqvkx\" (UID: \"a173790b-6c13-45a1-a361-3fe51844b4c2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zqvkx" Oct 31 04:04:07.140743 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:07.140686 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c6gcg\" (UID: \"9b7bf8e5-7877-46e0-88dd-a3503b7639c5\") " pod="openshift-insights/insights-runtime-extractor-c6gcg" Oct 31 04:04:07.140743 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:07.140725 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/197dee13-748f-4c6d-acc5-1793b5cee80a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-c64cl\" (UID: \"197dee13-748f-4c6d-acc5-1793b5cee80a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c64cl" Oct 31 04:04:07.140871 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:07.140766 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-registry-tls\") pod \"image-registry-6f69bcc59c-w96bl\" (UID: \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\") " pod="openshift-image-registry/image-registry-6f69bcc59c-w96bl" Oct 31 04:04:07.140871 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:04:07.140809 5019 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 04:04:07.140871 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:07.140809 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/93beb8b8-c9e7-4cce-8baf-8497b971e193-registry-tls\") pod \"image-registry-7f78cfdfd8-w98g7\" (UID: \"93beb8b8-c9e7-4cce-8baf-8497b971e193\") " pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" Oct 31 04:04:07.141002 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:04:07.140864 5019 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 04:04:07.141002 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:04:07.140886 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-insights-runtime-extractor-tls podName:9b7bf8e5-7877-46e0-88dd-a3503b7639c5 nodeName:}" failed. No retries permitted until 2026-10-31 04:04:08.14086647 +0000 UTC m=+2.897204493 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c6gcg" (UID: "9b7bf8e5-7877-46e0-88dd-a3503b7639c5") : secret "insights-runtime-extractor-tls" not found Oct 31 04:04:07.141002 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:04:07.140890 5019 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f69bcc59c-w96bl: secret "image-registry-tls" not found Oct 31 04:04:07.141002 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:04:07.140895 5019 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 04:04:07.141002 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:04:07.140909 5019 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f78cfdfd8-w98g7: secret "image-registry-tls" not found Oct 31 04:04:07.141002 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:07.140927 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/48309c9d-5f34-49bf-b55b-f74959984e0a-metrics-tls\") pod \"dns-default-zjpdv\" (UID: \"48309c9d-5f34-49bf-b55b-f74959984e0a\") " pod="openshift-dns/dns-default-zjpdv" Oct 31 04:04:07.141002 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:04:07.140950 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-registry-tls podName:e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22 nodeName:}" failed. No retries permitted until 2026-10-31 04:04:08.140931986 +0000 UTC m=+2.897270021 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-registry-tls") pod "image-registry-6f69bcc59c-w96bl" (UID: "e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22") : secret "image-registry-tls" not found Oct 31 04:04:07.141002 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:04:07.140809 5019 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 04:04:07.141002 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:04:07.140979 5019 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 04:04:07.141002 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:04:07.141003 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93beb8b8-c9e7-4cce-8baf-8497b971e193-registry-tls podName:93beb8b8-c9e7-4cce-8baf-8497b971e193 nodeName:}" failed. No retries permitted until 2026-10-31 04:04:08.140992927 +0000 UTC m=+2.897330949 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/93beb8b8-c9e7-4cce-8baf-8497b971e193-registry-tls") pod "image-registry-7f78cfdfd8-w98g7" (UID: "93beb8b8-c9e7-4cce-8baf-8497b971e193") : secret "image-registry-tls" not found Oct 31 04:04:07.141002 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:04:07.140871 5019 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 04:04:07.141377 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:04:07.141021 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a173790b-6c13-45a1-a361-3fe51844b4c2-cluster-monitoring-operator-tls podName:a173790b-6c13-45a1-a361-3fe51844b4c2 nodeName:}" failed. No retries permitted until 2026-10-31 04:04:08.141012831 +0000 UTC m=+2.897350849 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a173790b-6c13-45a1-a361-3fe51844b4c2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-zqvkx" (UID: "a173790b-6c13-45a1-a361-3fe51844b4c2") : secret "cluster-monitoring-operator-tls" not found Oct 31 04:04:07.141377 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:04:07.141040 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48309c9d-5f34-49bf-b55b-f74959984e0a-metrics-tls podName:48309c9d-5f34-49bf-b55b-f74959984e0a nodeName:}" failed. No retries permitted until 2026-10-31 04:04:08.141032392 +0000 UTC m=+2.897370411 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/48309c9d-5f34-49bf-b55b-f74959984e0a-metrics-tls") pod "dns-default-zjpdv" (UID: "48309c9d-5f34-49bf-b55b-f74959984e0a") : secret "dns-default-metrics-tls" not found Oct 31 04:04:07.141377 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:04:07.141067 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/197dee13-748f-4c6d-acc5-1793b5cee80a-samples-operator-tls podName:197dee13-748f-4c6d-acc5-1793b5cee80a nodeName:}" failed. No retries permitted until 2026-10-31 04:04:08.141051647 +0000 UTC m=+2.897389668 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/197dee13-748f-4c6d-acc5-1793b5cee80a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-c64cl" (UID: "197dee13-748f-4c6d-acc5-1793b5cee80a") : secret "samples-operator-tls" not found Oct 31 04:04:07.141377 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:07.141210 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c6gcg\" (UID: \"9b7bf8e5-7877-46e0-88dd-a3503b7639c5\") " pod="openshift-insights/insights-runtime-extractor-c6gcg" Oct 31 04:04:07.222671 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:07.222639 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-fvqtv_49140fa3-6ce2-41ea-8b6b-e25257efaf3e/migrator/0.log" Oct 31 04:04:07.422656 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:07.422558 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-zg2xl_9c8cb360-ff57-41b1-8244-47588703cba5/migrator/0.log" Oct 31 04:04:07.470844 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:07.470809 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-clnlz" event={"ID":"97c312b1-4ca5-49c8-8c8d-d247c0fa14ac","Type":"ContainerStarted","Data":"01c56e071d428f3025a47782ea70fb5ba6a6ebeba9ff10410e8580ab72f1bca0"} Oct 31 04:04:07.472191 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:07.472156 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-pnwds" event={"ID":"183f4178-99c0-4945-b490-c71cbd2780f3","Type":"ContainerStarted","Data":"262af931ef5c6d9eb0c83ccbbddac0f0fbedc47792a7e202501f095814e5194e"} Oct 31 04:04:07.474073 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:07.474015 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-678d747c5f-bgj92" event={"ID":"d22707c7-21fa-4511-b0b1-869b41fb2351","Type":"ContainerStarted","Data":"c4244ce100f2302d95fcd1e4e3dbf633b8b367324ce1593bda2612cc4bf899af"} Oct 31 04:04:07.474073 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:07.474079 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-678d747c5f-bgj92" event={"ID":"d22707c7-21fa-4511-b0b1-869b41fb2351","Type":"ContainerStarted","Data":"9021322515eae83cfc0a38437570f07c1739923cb87e7fc9ea6561e254bfc218"} Oct 31 04:04:07.475857 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:07.475833 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s88kp_3945ff65-e877-4372-9334-3968c68b7295/console-operator/2.log" Oct 31 04:04:07.476308 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:07.476293 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s88kp_3945ff65-e877-4372-9334-3968c68b7295/console-operator/1.log" Oct 31 04:04:07.476417 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:07.476325 5019 generic.go:356] "Generic (PLEG): container finished" podID="3945ff65-e877-4372-9334-3968c68b7295" containerID="ef40d036b07738b2d805b25bb5ec7917ab3f073f96ba04d020b998b639161f1d" exitCode=255 Oct 31 04:04:07.476486 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:07.476447 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-s88kp" event={"ID":"3945ff65-e877-4372-9334-3968c68b7295","Type":"ContainerDied","Data":"ef40d036b07738b2d805b25bb5ec7917ab3f073f96ba04d020b998b639161f1d"} Oct 31 04:04:07.476541 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:07.476500 5019 scope.go:122] "RemoveContainer" containerID="e11078a797b967674e01f514aab383ae2b67709c9bc64c60177a719670d63040" Oct 31 04:04:07.476986 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:07.476957 5019 scope.go:122] "RemoveContainer" containerID="ef40d036b07738b2d805b25bb5ec7917ab3f073f96ba04d020b998b639161f1d" Oct 31 04:04:07.477169 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:04:07.477140 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-s88kp_openshift-console-operator(3945ff65-e877-4372-9334-3968c68b7295)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-s88kp" podUID="3945ff65-e877-4372-9334-3968c68b7295" Oct 31 04:04:07.623900 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:07.623862 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-bt9c9_c3c90e6f-df5e-4c17-bec4-554280a8e8e7/kube-storage-version-migrator-operator/0.log" Oct 31 04:04:07.721439 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:07.721239 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-678d747c5f-bgj92" Oct 31 04:04:07.724608 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:07.724578 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-678d747c5f-bgj92" Oct 31 04:04:08.151116 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:08.151058 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c6gcg\" (UID: \"9b7bf8e5-7877-46e0-88dd-a3503b7639c5\") " pod="openshift-insights/insights-runtime-extractor-c6gcg" Oct 31 04:04:08.151116 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:08.151099 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/197dee13-748f-4c6d-acc5-1793b5cee80a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-c64cl\" (UID: \"197dee13-748f-4c6d-acc5-1793b5cee80a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c64cl" Oct 31 04:04:08.151454 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:08.151127 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-registry-tls\") pod \"image-registry-6f69bcc59c-w96bl\" (UID: \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\") " pod="openshift-image-registry/image-registry-6f69bcc59c-w96bl" Oct 31 04:04:08.151454 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:08.151175 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/93beb8b8-c9e7-4cce-8baf-8497b971e193-registry-tls\") pod \"image-registry-7f78cfdfd8-w98g7\" (UID: \"93beb8b8-c9e7-4cce-8baf-8497b971e193\") " pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" Oct 31 04:04:08.151454 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:08.151209 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/48309c9d-5f34-49bf-b55b-f74959984e0a-metrics-tls\") pod \"dns-default-zjpdv\" (UID: \"48309c9d-5f34-49bf-b55b-f74959984e0a\") " pod="openshift-dns/dns-default-zjpdv" Oct 31 04:04:08.151454 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:08.151273 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a173790b-6c13-45a1-a361-3fe51844b4c2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-zqvkx\" (UID: \"a173790b-6c13-45a1-a361-3fe51844b4c2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zqvkx" Oct 31 04:04:08.151912 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:04:08.151887 5019 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 04:04:08.151985 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:04:08.151972 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-insights-runtime-extractor-tls podName:9b7bf8e5-7877-46e0-88dd-a3503b7639c5 nodeName:}" failed. No retries permitted until 2026-10-31 04:04:10.151949791 +0000 UTC m=+4.908287828 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c6gcg" (UID: "9b7bf8e5-7877-46e0-88dd-a3503b7639c5") : secret "insights-runtime-extractor-tls" not found Oct 31 04:04:08.152456 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:04:08.152435 5019 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 04:04:08.152576 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:04:08.152506 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/197dee13-748f-4c6d-acc5-1793b5cee80a-samples-operator-tls podName:197dee13-748f-4c6d-acc5-1793b5cee80a nodeName:}" failed. No retries permitted until 2026-10-31 04:04:10.152488767 +0000 UTC m=+4.908826800 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/197dee13-748f-4c6d-acc5-1793b5cee80a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-c64cl" (UID: "197dee13-748f-4c6d-acc5-1793b5cee80a") : secret "samples-operator-tls" not found Oct 31 04:04:08.152641 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:04:08.152580 5019 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 04:04:08.152641 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:04:08.152617 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48309c9d-5f34-49bf-b55b-f74959984e0a-metrics-tls podName:48309c9d-5f34-49bf-b55b-f74959984e0a nodeName:}" failed. No retries permitted until 2026-10-31 04:04:10.152605977 +0000 UTC m=+4.908944010 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/48309c9d-5f34-49bf-b55b-f74959984e0a-metrics-tls") pod "dns-default-zjpdv" (UID: "48309c9d-5f34-49bf-b55b-f74959984e0a") : secret "dns-default-metrics-tls" not found Oct 31 04:04:08.154476 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:08.154449 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-registry-tls\") pod \"image-registry-6f69bcc59c-w96bl\" (UID: \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\") " pod="openshift-image-registry/image-registry-6f69bcc59c-w96bl" Oct 31 04:04:08.155206 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:08.155179 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a173790b-6c13-45a1-a361-3fe51844b4c2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-zqvkx\" (UID: \"a173790b-6c13-45a1-a361-3fe51844b4c2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zqvkx" Oct 31 04:04:08.155304 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:08.155254 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/93beb8b8-c9e7-4cce-8baf-8497b971e193-registry-tls\") pod \"image-registry-7f78cfdfd8-w98g7\" (UID: \"93beb8b8-c9e7-4cce-8baf-8497b971e193\") " pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" Oct 31 04:04:08.227149 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:08.227105 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zqvkx" Oct 31 04:04:08.227308 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:08.227241 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f69bcc59c-w96bl" Oct 31 04:04:08.234375 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:08.234314 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" Oct 31 04:04:08.481571 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:08.481467 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-pnwds" event={"ID":"183f4178-99c0-4945-b490-c71cbd2780f3","Type":"ContainerStarted","Data":"86a2718fc40c761886a478b375b44bd85f7114db8d086f54cf476b81f043b0c3"} Oct 31 04:04:08.483268 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:08.483244 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s88kp_3945ff65-e877-4372-9334-3968c68b7295/console-operator/2.log" Oct 31 04:04:08.485214 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:08.485188 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-clnlz" event={"ID":"97c312b1-4ca5-49c8-8c8d-d247c0fa14ac","Type":"ContainerStarted","Data":"852566088de53de45af002df4a7f711676339b2dd0d946c55060cd6c17b43a28"} Oct 31 04:04:08.485372 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:08.485237 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-clnlz" event={"ID":"97c312b1-4ca5-49c8-8c8d-d247c0fa14ac","Type":"ContainerStarted","Data":"c81a3f859b6bad154a4eb7b1ff5d39805f6398e1c7fc0a3beadbd3c607ed9ad2"} Oct 31 04:04:08.532599 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:08.532567 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-zqvkx"] Oct 31 04:04:08.536171 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:08.536140 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda173790b_6c13_45a1_a361_3fe51844b4c2.slice/crio-572f15c58df04bd48d109f7fd6c5bd3a24ba1a24a090e0a0152659303f6573be WatchSource:0}: Error finding container 572f15c58df04bd48d109f7fd6c5bd3a24ba1a24a090e0a0152659303f6573be: Status 404 returned error can't find the container with id 572f15c58df04bd48d109f7fd6c5bd3a24ba1a24a090e0a0152659303f6573be Oct 31 04:04:08.582589 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:08.582548 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f78cfdfd8-w98g7"] Oct 31 04:04:08.585712 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:08.585683 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93beb8b8_c9e7_4cce_8baf_8497b971e193.slice/crio-fb7daaf87051b201e3957163cef4274739d5fec13c059e4f0b7c3a60def772fc WatchSource:0}: Error finding container fb7daaf87051b201e3957163cef4274739d5fec13c059e4f0b7c3a60def772fc: Status 404 returned error can't find the container with id fb7daaf87051b201e3957163cef4274739d5fec13c059e4f0b7c3a60def772fc Oct 31 04:04:08.607479 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:08.607450 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f69bcc59c-w96bl"] Oct 31 04:04:08.620705 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:08.620670 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode72e00a8_bf39_4ae3_93e5_6f0d58fd9f22.slice/crio-47915f7d9337e24318fb5ca597c226f543a6f9cdb35cf3ca4dbff32de99539ad WatchSource:0}: Error finding container 47915f7d9337e24318fb5ca597c226f543a6f9cdb35cf3ca4dbff32de99539ad: Status 404 returned error can't find the container with id 47915f7d9337e24318fb5ca597c226f543a6f9cdb35cf3ca4dbff32de99539ad Oct 31 04:04:09.113868 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:09.113827 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4b7d5" Oct 31 04:04:09.492000 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:09.491967 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f69bcc59c-w96bl" event={"ID":"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22","Type":"ContainerStarted","Data":"f8f84bda9df0f5c26f554d2e8b6eb99408b7a44612ddabc8bbf1c7b9b6597531"} Oct 31 04:04:09.492425 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:09.492026 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f69bcc59c-w96bl" event={"ID":"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22","Type":"ContainerStarted","Data":"47915f7d9337e24318fb5ca597c226f543a6f9cdb35cf3ca4dbff32de99539ad"} Oct 31 04:04:09.493598 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:09.493566 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zqvkx" event={"ID":"a173790b-6c13-45a1-a361-3fe51844b4c2","Type":"ContainerStarted","Data":"572f15c58df04bd48d109f7fd6c5bd3a24ba1a24a090e0a0152659303f6573be"} Oct 31 04:04:09.495647 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:09.495621 5019 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 04:04:09.495647 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:09.495627 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" event={"ID":"93beb8b8-c9e7-4cce-8baf-8497b971e193","Type":"ContainerStarted","Data":"c06e84d78c5e52e8c80db792fd749ef6219ed414166d85168a53d83644de3f21"} Oct 31 04:04:09.495855 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:09.495670 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" event={"ID":"93beb8b8-c9e7-4cce-8baf-8497b971e193","Type":"ContainerStarted","Data":"fb7daaf87051b201e3957163cef4274739d5fec13c059e4f0b7c3a60def772fc"} Oct 31 04:04:10.171153 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:10.171110 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c6gcg\" (UID: \"9b7bf8e5-7877-46e0-88dd-a3503b7639c5\") " pod="openshift-insights/insights-runtime-extractor-c6gcg" Oct 31 04:04:10.171316 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:10.171160 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/197dee13-748f-4c6d-acc5-1793b5cee80a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-c64cl\" (UID: \"197dee13-748f-4c6d-acc5-1793b5cee80a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c64cl" Oct 31 04:04:10.171316 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:10.171222 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/48309c9d-5f34-49bf-b55b-f74959984e0a-metrics-tls\") pod \"dns-default-zjpdv\" (UID: \"48309c9d-5f34-49bf-b55b-f74959984e0a\") " pod="openshift-dns/dns-default-zjpdv" Oct 31 04:04:10.171316 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:04:10.171269 5019 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 04:04:10.171442 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:04:10.171353 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-insights-runtime-extractor-tls podName:9b7bf8e5-7877-46e0-88dd-a3503b7639c5 nodeName:}" failed. No retries permitted until 2026-10-31 04:04:14.171321049 +0000 UTC m=+8.927659089 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c6gcg" (UID: "9b7bf8e5-7877-46e0-88dd-a3503b7639c5") : secret "insights-runtime-extractor-tls" not found Oct 31 04:04:10.173722 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:10.173687 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/48309c9d-5f34-49bf-b55b-f74959984e0a-metrics-tls\") pod \"dns-default-zjpdv\" (UID: \"48309c9d-5f34-49bf-b55b-f74959984e0a\") " pod="openshift-dns/dns-default-zjpdv" Oct 31 04:04:10.173840 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:10.173732 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/197dee13-748f-4c6d-acc5-1793b5cee80a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-c64cl\" (UID: \"197dee13-748f-4c6d-acc5-1793b5cee80a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c64cl" Oct 31 04:04:10.285809 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:10.285775 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6f69bcc59c-w96bl" Oct 31 04:04:10.327053 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:10.327017 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c64cl" Oct 31 04:04:10.335370 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:10.335317 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zjpdv" Oct 31 04:04:10.414270 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:10.414240 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-s88kp" Oct 31 04:04:10.414659 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:10.414644 5019 scope.go:122] "RemoveContainer" containerID="ef40d036b07738b2d805b25bb5ec7917ab3f073f96ba04d020b998b639161f1d" Oct 31 04:04:10.414861 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:04:10.414842 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-s88kp_openshift-console-operator(3945ff65-e877-4372-9334-3968c68b7295)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-s88kp" podUID="3945ff65-e877-4372-9334-3968c68b7295" Oct 31 04:04:10.504195 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:10.504156 5019 generic.go:356] "Generic (PLEG): container finished" podID="c3c90e6f-df5e-4c17-bec4-554280a8e8e7" containerID="646dc8afa76d49bdf223f7226aadcaaa754fcf0a10e173603e622a072a919d98" exitCode=0 Oct 31 04:04:10.504667 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:10.504235 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bt9c9" event={"ID":"c3c90e6f-df5e-4c17-bec4-554280a8e8e7","Type":"ContainerDied","Data":"646dc8afa76d49bdf223f7226aadcaaa754fcf0a10e173603e622a072a919d98"} Oct 31 04:04:10.504707 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:10.504683 5019 scope.go:122] "RemoveContainer" containerID="646dc8afa76d49bdf223f7226aadcaaa754fcf0a10e173603e622a072a919d98" Oct 31 04:04:10.507080 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:10.507040 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zqvkx" event={"ID":"a173790b-6c13-45a1-a361-3fe51844b4c2","Type":"ContainerStarted","Data":"96897a71140bd3c593a0b28463938152d23c4014b4c4c684da1d9593b5c139e5"} Oct 31 04:04:10.509544 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:10.509521 5019 generic.go:356] "Generic (PLEG): container finished" podID="e667439c-7121-4c5f-84fe-b36f2391dc66" containerID="284e9cfc7e02f8f1e278322b51bbe397a68e25d935271bc660bfb0b55f60f5d2" exitCode=0 Oct 31 04:04:10.509657 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:10.509593 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-ld96l" event={"ID":"e667439c-7121-4c5f-84fe-b36f2391dc66","Type":"ContainerDied","Data":"284e9cfc7e02f8f1e278322b51bbe397a68e25d935271bc660bfb0b55f60f5d2"} Oct 31 04:04:10.509715 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:10.509601 5019 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 04:04:10.510493 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:10.510472 5019 scope.go:122] "RemoveContainer" containerID="284e9cfc7e02f8f1e278322b51bbe397a68e25d935271bc660bfb0b55f60f5d2" Oct 31 04:04:10.632316 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:10.632186 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c64cl"] Oct 31 04:04:10.646053 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:10.646025 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zjpdv"] Oct 31 04:04:10.651565 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:10.651537 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48309c9d_5f34_49bf_b55b_f74959984e0a.slice/crio-1168c5335212323c8e7b042572dba08ff1eeaf314f971679100a5326eb3250e1 WatchSource:0}: Error finding container 1168c5335212323c8e7b042572dba08ff1eeaf314f971679100a5326eb3250e1: Status 404 returned error can't find the container with id 1168c5335212323c8e7b042572dba08ff1eeaf314f971679100a5326eb3250e1 Oct 31 04:04:10.662813 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:10.662781 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:04:11.192753 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:11.192049 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hm5cr" Oct 31 04:04:11.519653 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:11.519618 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-ld96l" event={"ID":"e667439c-7121-4c5f-84fe-b36f2391dc66","Type":"ContainerStarted","Data":"910db4748dccf394050932467c4aa2e15423c5fd8e1502c91c73530555c9d9fe"} Oct 31 04:04:11.522020 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:11.521981 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zjpdv" event={"ID":"48309c9d-5f34-49bf-b55b-f74959984e0a","Type":"ContainerStarted","Data":"1168c5335212323c8e7b042572dba08ff1eeaf314f971679100a5326eb3250e1"} Oct 31 04:04:11.524971 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:11.524371 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bt9c9" event={"ID":"c3c90e6f-df5e-4c17-bec4-554280a8e8e7","Type":"ContainerStarted","Data":"a44d07c96e908226ab6faf0abd418375df14db1db9150c95a998ea9dcfbbea61"} Oct 31 04:04:11.527688 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:11.527617 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c64cl" event={"ID":"197dee13-748f-4c6d-acc5-1793b5cee80a","Type":"ContainerStarted","Data":"3fab61351571c514dce7a6695fa3a73f91670288112d45a5defcbcb0668f55fd"} Oct 31 04:04:12.533881 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:12.533842 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c64cl" event={"ID":"197dee13-748f-4c6d-acc5-1793b5cee80a","Type":"ContainerStarted","Data":"b5f0acdeb975710ce4afbe5f550844b98364fc4f35699021359efac06c3b50a7"} Oct 31 04:04:12.630861 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:12.629983 5019 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-s88kp" Oct 31 04:04:12.630861 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:12.630535 5019 scope.go:122] "RemoveContainer" containerID="ef40d036b07738b2d805b25bb5ec7917ab3f073f96ba04d020b998b639161f1d" Oct 31 04:04:12.630861 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:04:12.630786 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-s88kp_openshift-console-operator(3945ff65-e877-4372-9334-3968c68b7295)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-s88kp" podUID="3945ff65-e877-4372-9334-3968c68b7295" Oct 31 04:04:13.539644 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:13.539601 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zjpdv" event={"ID":"48309c9d-5f34-49bf-b55b-f74959984e0a","Type":"ContainerStarted","Data":"dfdbd998809109dd5b29bc65a044e5bb23855cf6efdcdfe6ecd155ec17422532"} Oct 31 04:04:13.540113 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:13.539663 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zjpdv" event={"ID":"48309c9d-5f34-49bf-b55b-f74959984e0a","Type":"ContainerStarted","Data":"8997fd2f913f516283bf35b506a5f37294a16f2c33851725a2056238493f9af5"} Oct 31 04:04:13.540113 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:13.539771 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zjpdv" Oct 31 04:04:13.541600 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:13.541573 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c64cl" event={"ID":"197dee13-748f-4c6d-acc5-1793b5cee80a","Type":"ContainerStarted","Data":"5289159c82ec2b4664248a3aa8bc66a7b490ab132d334e96b70ea86f368f4bd7"} Oct 31 04:04:14.211044 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:14.210997 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c6gcg\" (UID: \"9b7bf8e5-7877-46e0-88dd-a3503b7639c5\") " pod="openshift-insights/insights-runtime-extractor-c6gcg" Oct 31 04:04:14.213526 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:14.213493 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9b7bf8e5-7877-46e0-88dd-a3503b7639c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c6gcg\" (UID: \"9b7bf8e5-7877-46e0-88dd-a3503b7639c5\") " pod="openshift-insights/insights-runtime-extractor-c6gcg" Oct 31 04:04:14.238663 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:14.238612 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-c6gcg" Oct 31 04:04:14.535112 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:14.535007 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-c6gcg"] Oct 31 04:04:14.538093 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:14.538047 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b7bf8e5_7877_46e0_88dd_a3503b7639c5.slice/crio-5f58243232f75ef6ebdd2fde38b28c83e8d6b59b6fc3bca9fbeb9f5d3b7a21d9 WatchSource:0}: Error finding container 5f58243232f75ef6ebdd2fde38b28c83e8d6b59b6fc3bca9fbeb9f5d3b7a21d9: Status 404 returned error can't find the container with id 5f58243232f75ef6ebdd2fde38b28c83e8d6b59b6fc3bca9fbeb9f5d3b7a21d9 Oct 31 04:04:14.546837 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:14.546801 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c6gcg" event={"ID":"9b7bf8e5-7877-46e0-88dd-a3503b7639c5","Type":"ContainerStarted","Data":"5f58243232f75ef6ebdd2fde38b28c83e8d6b59b6fc3bca9fbeb9f5d3b7a21d9"} Oct 31 04:04:15.558445 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:15.558361 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c6gcg" event={"ID":"9b7bf8e5-7877-46e0-88dd-a3503b7639c5","Type":"ContainerStarted","Data":"ad09e482a5397140ee22bcf5e0d800390314426a0d61854a074cfff83d3b1773"} Oct 31 04:04:15.558862 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:15.558449 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c6gcg" event={"ID":"9b7bf8e5-7877-46e0-88dd-a3503b7639c5","Type":"ContainerStarted","Data":"b4c19ba8fa1135d9023e644a2dfefc27aa8203283f15ab78dca055469ef9e704"} Oct 31 04:04:15.688644 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:15.688616 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:04:15.688862 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:15.688847 5019 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 04:04:15.832750 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:15.832665 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" Oct 31 04:04:16.065402 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:16.065363 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-678d747c5f-bgj92" Oct 31 04:04:16.065594 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:16.065519 5019 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 04:04:16.066982 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:16.066953 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-678d747c5f-bgj92" Oct 31 04:04:16.565916 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:16.565880 5019 generic.go:356] "Generic (PLEG): container finished" podID="e667439c-7121-4c5f-84fe-b36f2391dc66" containerID="910db4748dccf394050932467c4aa2e15423c5fd8e1502c91c73530555c9d9fe" exitCode=0 Oct 31 04:04:16.566376 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:16.565953 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-ld96l" event={"ID":"e667439c-7121-4c5f-84fe-b36f2391dc66","Type":"ContainerDied","Data":"910db4748dccf394050932467c4aa2e15423c5fd8e1502c91c73530555c9d9fe"} Oct 31 04:04:16.566376 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:16.566017 5019 scope.go:122] "RemoveContainer" containerID="284e9cfc7e02f8f1e278322b51bbe397a68e25d935271bc660bfb0b55f60f5d2" Oct 31 04:04:16.566479 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:16.566396 5019 scope.go:122] "RemoveContainer" containerID="910db4748dccf394050932467c4aa2e15423c5fd8e1502c91c73530555c9d9fe" Oct 31 04:04:16.566622 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:04:16.566600 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"insights-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=insights-operator pod=insights-operator-7c4986d449-ld96l_openshift-insights(e667439c-7121-4c5f-84fe-b36f2391dc66)\"" pod="openshift-insights/insights-operator-7c4986d449-ld96l" podUID="e667439c-7121-4c5f-84fe-b36f2391dc66" Oct 31 04:04:17.572408 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:17.572368 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c6gcg" event={"ID":"9b7bf8e5-7877-46e0-88dd-a3503b7639c5","Type":"ContainerStarted","Data":"96fa5f9580a2df4d064b26496162a73be7fb0eb8897a96248e7f68201922c11e"} Oct 31 04:04:19.404901 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:19.404870 5019 scope.go:122] "RemoveContainer" containerID="ef40d036b07738b2d805b25bb5ec7917ab3f073f96ba04d020b998b639161f1d" Oct 31 04:04:19.583874 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:19.583843 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s88kp_3945ff65-e877-4372-9334-3968c68b7295/console-operator/2.log" Oct 31 04:04:19.584020 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:19.583940 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-s88kp" event={"ID":"3945ff65-e877-4372-9334-3968c68b7295","Type":"ContainerStarted","Data":"6e3cbbf6ddf8b38d36020bd85fe3dd7c7b3a8c62c10163be78b7dcac0f23e3ba"} Oct 31 04:04:19.584249 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:19.584219 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-s88kp" Oct 31 04:04:19.957871 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:19.957840 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-s88kp" Oct 31 04:04:21.932606 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:21.932572 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-hcrqf-p69z8"] Oct 31 04:04:21.936276 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:21.936251 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hcrqf-p69z8" Oct 31 04:04:21.970494 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:21.970455 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-hcrqf-p69z8"] Oct 31 04:04:21.973018 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:21.972988 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/908f1ac2-00d8-4fb6-8f92-daebf51f0ad8-service-ca-bundle\") pod \"periodic-gathering-hcrqf-p69z8\" (UID: \"908f1ac2-00d8-4fb6-8f92-daebf51f0ad8\") " pod="openshift-insights/periodic-gathering-hcrqf-p69z8" Oct 31 04:04:21.973018 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:21.973021 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/908f1ac2-00d8-4fb6-8f92-daebf51f0ad8-trusted-ca-bundle\") pod \"periodic-gathering-hcrqf-p69z8\" (UID: \"908f1ac2-00d8-4fb6-8f92-daebf51f0ad8\") " pod="openshift-insights/periodic-gathering-hcrqf-p69z8" Oct 31 04:04:21.973235 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:21.973088 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh5x7\" (UniqueName: \"kubernetes.io/projected/908f1ac2-00d8-4fb6-8f92-daebf51f0ad8-kube-api-access-mh5x7\") pod \"periodic-gathering-hcrqf-p69z8\" (UID: \"908f1ac2-00d8-4fb6-8f92-daebf51f0ad8\") " pod="openshift-insights/periodic-gathering-hcrqf-p69z8" Oct 31 04:04:21.973235 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:21.973162 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/908f1ac2-00d8-4fb6-8f92-daebf51f0ad8-archives-path\") pod \"periodic-gathering-hcrqf-p69z8\" (UID: \"908f1ac2-00d8-4fb6-8f92-daebf51f0ad8\") " pod="openshift-insights/periodic-gathering-hcrqf-p69z8" Oct 31 04:04:22.000174 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:22.000134 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6f69bcc59c-w96bl"] Oct 31 04:04:22.076852 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:22.074584 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/908f1ac2-00d8-4fb6-8f92-daebf51f0ad8-archives-path\") pod \"periodic-gathering-hcrqf-p69z8\" (UID: \"908f1ac2-00d8-4fb6-8f92-daebf51f0ad8\") " pod="openshift-insights/periodic-gathering-hcrqf-p69z8" Oct 31 04:04:22.076852 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:22.074695 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/908f1ac2-00d8-4fb6-8f92-daebf51f0ad8-service-ca-bundle\") pod \"periodic-gathering-hcrqf-p69z8\" (UID: \"908f1ac2-00d8-4fb6-8f92-daebf51f0ad8\") " pod="openshift-insights/periodic-gathering-hcrqf-p69z8" Oct 31 04:04:22.076852 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:22.074726 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/908f1ac2-00d8-4fb6-8f92-daebf51f0ad8-trusted-ca-bundle\") pod \"periodic-gathering-hcrqf-p69z8\" (UID: \"908f1ac2-00d8-4fb6-8f92-daebf51f0ad8\") " pod="openshift-insights/periodic-gathering-hcrqf-p69z8" Oct 31 04:04:22.076852 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:22.074770 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh5x7\" (UniqueName: \"kubernetes.io/projected/908f1ac2-00d8-4fb6-8f92-daebf51f0ad8-kube-api-access-mh5x7\") pod \"periodic-gathering-hcrqf-p69z8\" (UID: \"908f1ac2-00d8-4fb6-8f92-daebf51f0ad8\") " pod="openshift-insights/periodic-gathering-hcrqf-p69z8" Oct 31 04:04:22.076852 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:22.075313 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/908f1ac2-00d8-4fb6-8f92-daebf51f0ad8-archives-path\") pod \"periodic-gathering-hcrqf-p69z8\" (UID: \"908f1ac2-00d8-4fb6-8f92-daebf51f0ad8\") " pod="openshift-insights/periodic-gathering-hcrqf-p69z8" Oct 31 04:04:22.076852 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:22.076044 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/908f1ac2-00d8-4fb6-8f92-daebf51f0ad8-service-ca-bundle\") pod \"periodic-gathering-hcrqf-p69z8\" (UID: \"908f1ac2-00d8-4fb6-8f92-daebf51f0ad8\") " pod="openshift-insights/periodic-gathering-hcrqf-p69z8" Oct 31 04:04:22.076852 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:22.076777 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/908f1ac2-00d8-4fb6-8f92-daebf51f0ad8-trusted-ca-bundle\") pod \"periodic-gathering-hcrqf-p69z8\" (UID: \"908f1ac2-00d8-4fb6-8f92-daebf51f0ad8\") " pod="openshift-insights/periodic-gathering-hcrqf-p69z8" Oct 31 04:04:22.103773 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:22.103740 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh5x7\" (UniqueName: \"kubernetes.io/projected/908f1ac2-00d8-4fb6-8f92-daebf51f0ad8-kube-api-access-mh5x7\") pod \"periodic-gathering-hcrqf-p69z8\" (UID: \"908f1ac2-00d8-4fb6-8f92-daebf51f0ad8\") " pod="openshift-insights/periodic-gathering-hcrqf-p69z8" Oct 31 04:04:22.255802 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:22.255763 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hcrqf-p69z8" Oct 31 04:04:22.390054 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:04:22.390014 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:04:22.550271 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:22.550190 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-hcrqf-p69z8"] Oct 31 04:04:22.553526 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:22.553493 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod908f1ac2_00d8_4fb6_8f92_daebf51f0ad8.slice/crio-86b93a3a277c0f30f1af32cd2ba5332fbd3e6361c1697b427762fd11f2e0c550 WatchSource:0}: Error finding container 86b93a3a277c0f30f1af32cd2ba5332fbd3e6361c1697b427762fd11f2e0c550: Status 404 returned error can't find the container with id 86b93a3a277c0f30f1af32cd2ba5332fbd3e6361c1697b427762fd11f2e0c550 Oct 31 04:04:22.597954 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:22.597922 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hcrqf-p69z8" event={"ID":"908f1ac2-00d8-4fb6-8f92-daebf51f0ad8","Type":"ContainerStarted","Data":"86b93a3a277c0f30f1af32cd2ba5332fbd3e6361c1697b427762fd11f2e0c550"} Oct 31 04:04:23.549269 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:23.549241 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zjpdv" Oct 31 04:04:23.605290 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:23.605246 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hcrqf-p69z8" event={"ID":"908f1ac2-00d8-4fb6-8f92-daebf51f0ad8","Type":"ContainerStarted","Data":"0fab36732cbfe02fbbf445e40bffedb3a0940d76570a1f6aab06f3e4803612ee"} Oct 31 04:04:23.645274 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:23.645200 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-hcrqf-p69z8" podStartSLOduration=2.645179718 podStartE2EDuration="2.645179718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:04:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:04:23.644639162 +0000 UTC m=+18.400977202" watchObservedRunningTime="2026-10-31 04:04:23.645179718 +0000 UTC m=+18.401517756" Oct 31 04:04:25.837877 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:25.837828 5019 patch_prober.go:29] interesting pod/image-registry-7f78cfdfd8-w98g7 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 31 04:04:25.838371 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:25.837903 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" podUID="93beb8b8-c9e7-4cce-8baf-8497b971e193" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 04:04:26.996285 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:26.996253 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zjpdv_48309c9d-5f34-49bf-b55b-f74959984e0a/dns/0.log" Oct 31 04:04:27.196521 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:27.196490 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zjpdv_48309c9d-5f34-49bf-b55b-f74959984e0a/kube-rbac-proxy/0.log" Oct 31 04:04:27.596284 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:27.596254 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nbl9z_3a05c43f-e2db-44c0-b9bd-4bdabca92a98/dns-node-resolver/0.log" Oct 31 04:04:27.996562 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:27.996534 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6f69bcc59c-w96bl_e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22/registry/0.log" Oct 31 04:04:28.396513 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:28.396428 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7f78cfdfd8-w98g7_93beb8b8-c9e7-4cce-8baf-8497b971e193/registry/0.log" Oct 31 04:04:28.598781 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:28.598752 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-h9sww_31221235-f863-445b-a5ca-375aae905d38/node-ca/0.log" Oct 31 04:04:28.611326 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:28.611290 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tmt7k" Oct 31 04:04:28.846657 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:28.846612 5019 patch_prober.go:29] interesting pod/image-registry-7f78cfdfd8-w98g7 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 31 04:04:28.846838 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:28.846698 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" podUID="93beb8b8-c9e7-4cce-8baf-8497b971e193" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 04:04:29.197059 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:29.196970 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-678d747c5f-bgj92_d22707c7-21fa-4511-b0b1-869b41fb2351/router/0.log" Oct 31 04:04:29.399791 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:29.399761 5019 scope.go:122] "RemoveContainer" containerID="910db4748dccf394050932467c4aa2e15423c5fd8e1502c91c73530555c9d9fe" Oct 31 04:04:29.595758 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:29.595712 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vqsgm_a2ce92f3-5b05-4c89-9d53-a57bc85b17ef/serve-healthcheck-canary/0.log" Oct 31 04:04:29.633357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:29.633305 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-ld96l" event={"ID":"e667439c-7121-4c5f-84fe-b36f2391dc66","Type":"ContainerStarted","Data":"6bb1535aafcd331d56f89fcf3a21538b337440537a53e199d4a5d7f719696cef"} Oct 31 04:04:31.048391 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:31.048325 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-x7gql"] Oct 31 04:04:31.080460 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:31.080407 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-x7gql" Oct 31 04:04:31.127328 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:31.127291 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 04:04:31.127706 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:31.127666 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 04:04:31.127706 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:31.127677 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 04:04:31.138943 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:31.138909 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mfl64\"" Oct 31 04:04:31.139151 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:31.139132 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 04:04:31.151639 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:31.151603 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w562d\" (UniqueName: \"kubernetes.io/projected/39fa44fb-3af0-46c2-82a6-d3b21c3e5383-kube-api-access-w562d\") pod \"node-exporter-x7gql\" (UID: \"39fa44fb-3af0-46c2-82a6-d3b21c3e5383\") " pod="openshift-monitoring/node-exporter-x7gql" Oct 31 04:04:31.151809 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:31.151652 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/39fa44fb-3af0-46c2-82a6-d3b21c3e5383-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x7gql\" (UID: \"39fa44fb-3af0-46c2-82a6-d3b21c3e5383\") " pod="openshift-monitoring/node-exporter-x7gql" Oct 31 04:04:31.151809 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:31.151749 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/39fa44fb-3af0-46c2-82a6-d3b21c3e5383-node-exporter-textfile\") pod \"node-exporter-x7gql\" (UID: \"39fa44fb-3af0-46c2-82a6-d3b21c3e5383\") " pod="openshift-monitoring/node-exporter-x7gql" Oct 31 04:04:31.151809 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:31.151780 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/39fa44fb-3af0-46c2-82a6-d3b21c3e5383-metrics-client-ca\") pod \"node-exporter-x7gql\" (UID: \"39fa44fb-3af0-46c2-82a6-d3b21c3e5383\") " pod="openshift-monitoring/node-exporter-x7gql" Oct 31 04:04:31.151809 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:31.151801 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/39fa44fb-3af0-46c2-82a6-d3b21c3e5383-node-exporter-accelerators-collector-config\") pod \"node-exporter-x7gql\" (UID: \"39fa44fb-3af0-46c2-82a6-d3b21c3e5383\") " pod="openshift-monitoring/node-exporter-x7gql" Oct 31 04:04:31.152017 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:31.151825 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/39fa44fb-3af0-46c2-82a6-d3b21c3e5383-sys\") pod \"node-exporter-x7gql\" (UID: \"39fa44fb-3af0-46c2-82a6-d3b21c3e5383\") " pod="openshift-monitoring/node-exporter-x7gql" Oct 31 04:04:31.152017 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:31.151847 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/39fa44fb-3af0-46c2-82a6-d3b21c3e5383-node-exporter-tls\") pod \"node-exporter-x7gql\" (UID: \"39fa44fb-3af0-46c2-82a6-d3b21c3e5383\") " pod="openshift-monitoring/node-exporter-x7gql" Oct 31 04:04:31.152017 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:31.151981 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/39fa44fb-3af0-46c2-82a6-d3b21c3e5383-node-exporter-wtmp\") pod \"node-exporter-x7gql\" (UID: \"39fa44fb-3af0-46c2-82a6-d3b21c3e5383\") " pod="openshift-monitoring/node-exporter-x7gql" Oct 31 04:04:31.152133 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:31.152030 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/39fa44fb-3af0-46c2-82a6-d3b21c3e5383-root\") pod \"node-exporter-x7gql\" (UID: \"39fa44fb-3af0-46c2-82a6-d3b21c3e5383\") " pod="openshift-monitoring/node-exporter-x7gql" Oct 31 04:04:31.252976 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:31.252933 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/39fa44fb-3af0-46c2-82a6-d3b21c3e5383-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x7gql\" (UID: \"39fa44fb-3af0-46c2-82a6-d3b21c3e5383\") " pod="openshift-monitoring/node-exporter-x7gql" Oct 31 04:04:31.253191 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:31.252984 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/39fa44fb-3af0-46c2-82a6-d3b21c3e5383-node-exporter-textfile\") pod \"node-exporter-x7gql\" (UID: \"39fa44fb-3af0-46c2-82a6-d3b21c3e5383\") " pod="openshift-monitoring/node-exporter-x7gql" Oct 31 04:04:31.253191 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:31.253010 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/39fa44fb-3af0-46c2-82a6-d3b21c3e5383-metrics-client-ca\") pod \"node-exporter-x7gql\" (UID: \"39fa44fb-3af0-46c2-82a6-d3b21c3e5383\") " pod="openshift-monitoring/node-exporter-x7gql" Oct 31 04:04:31.253191 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:31.253051 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/39fa44fb-3af0-46c2-82a6-d3b21c3e5383-node-exporter-accelerators-collector-config\") pod \"node-exporter-x7gql\" (UID: \"39fa44fb-3af0-46c2-82a6-d3b21c3e5383\") " pod="openshift-monitoring/node-exporter-x7gql" Oct 31 04:04:31.253191 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:31.253087 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/39fa44fb-3af0-46c2-82a6-d3b21c3e5383-sys\") pod \"node-exporter-x7gql\" (UID: \"39fa44fb-3af0-46c2-82a6-d3b21c3e5383\") " pod="openshift-monitoring/node-exporter-x7gql" Oct 31 04:04:31.253191 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:31.253143 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/39fa44fb-3af0-46c2-82a6-d3b21c3e5383-sys\") pod \"node-exporter-x7gql\" (UID: \"39fa44fb-3af0-46c2-82a6-d3b21c3e5383\") " pod="openshift-monitoring/node-exporter-x7gql" Oct 31 04:04:31.253507 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:31.253211 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/39fa44fb-3af0-46c2-82a6-d3b21c3e5383-node-exporter-tls\") pod \"node-exporter-x7gql\" (UID: \"39fa44fb-3af0-46c2-82a6-d3b21c3e5383\") " pod="openshift-monitoring/node-exporter-x7gql" Oct 31 04:04:31.253507 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:31.253257 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/39fa44fb-3af0-46c2-82a6-d3b21c3e5383-node-exporter-wtmp\") pod \"node-exporter-x7gql\" (UID: \"39fa44fb-3af0-46c2-82a6-d3b21c3e5383\") " pod="openshift-monitoring/node-exporter-x7gql" Oct 31 04:04:31.253507 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:31.253290 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/39fa44fb-3af0-46c2-82a6-d3b21c3e5383-root\") pod \"node-exporter-x7gql\" (UID: \"39fa44fb-3af0-46c2-82a6-d3b21c3e5383\") " pod="openshift-monitoring/node-exporter-x7gql" Oct 31 04:04:31.253507 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:31.253390 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w562d\" (UniqueName: \"kubernetes.io/projected/39fa44fb-3af0-46c2-82a6-d3b21c3e5383-kube-api-access-w562d\") pod \"node-exporter-x7gql\" (UID: \"39fa44fb-3af0-46c2-82a6-d3b21c3e5383\") " pod="openshift-monitoring/node-exporter-x7gql" Oct 31 04:04:31.253507 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:31.253484 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/39fa44fb-3af0-46c2-82a6-d3b21c3e5383-node-exporter-wtmp\") pod \"node-exporter-x7gql\" (UID: \"39fa44fb-3af0-46c2-82a6-d3b21c3e5383\") " pod="openshift-monitoring/node-exporter-x7gql" Oct 31 04:04:31.253760 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:31.253484 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/39fa44fb-3af0-46c2-82a6-d3b21c3e5383-root\") pod \"node-exporter-x7gql\" (UID: \"39fa44fb-3af0-46c2-82a6-d3b21c3e5383\") " pod="openshift-monitoring/node-exporter-x7gql" Oct 31 04:04:31.253760 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:31.253609 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/39fa44fb-3af0-46c2-82a6-d3b21c3e5383-node-exporter-textfile\") pod \"node-exporter-x7gql\" (UID: \"39fa44fb-3af0-46c2-82a6-d3b21c3e5383\") " pod="openshift-monitoring/node-exporter-x7gql" Oct 31 04:04:31.253760 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:31.253729 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/39fa44fb-3af0-46c2-82a6-d3b21c3e5383-metrics-client-ca\") pod \"node-exporter-x7gql\" (UID: \"39fa44fb-3af0-46c2-82a6-d3b21c3e5383\") " pod="openshift-monitoring/node-exporter-x7gql" Oct 31 04:04:31.253859 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:31.253770 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/39fa44fb-3af0-46c2-82a6-d3b21c3e5383-node-exporter-accelerators-collector-config\") pod \"node-exporter-x7gql\" (UID: \"39fa44fb-3af0-46c2-82a6-d3b21c3e5383\") " pod="openshift-monitoring/node-exporter-x7gql" Oct 31 04:04:31.255500 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:31.255473 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/39fa44fb-3af0-46c2-82a6-d3b21c3e5383-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x7gql\" (UID: \"39fa44fb-3af0-46c2-82a6-d3b21c3e5383\") " pod="openshift-monitoring/node-exporter-x7gql" Oct 31 04:04:31.255621 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:31.255580 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/39fa44fb-3af0-46c2-82a6-d3b21c3e5383-node-exporter-tls\") pod \"node-exporter-x7gql\" (UID: \"39fa44fb-3af0-46c2-82a6-d3b21c3e5383\") " pod="openshift-monitoring/node-exporter-x7gql" Oct 31 04:04:31.276762 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:31.276720 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w562d\" (UniqueName: \"kubernetes.io/projected/39fa44fb-3af0-46c2-82a6-d3b21c3e5383-kube-api-access-w562d\") pod \"node-exporter-x7gql\" (UID: \"39fa44fb-3af0-46c2-82a6-d3b21c3e5383\") " pod="openshift-monitoring/node-exporter-x7gql" Oct 31 04:04:31.433021 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:31.432921 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-x7gql" Oct 31 04:04:31.469005 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:04:31.468968 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39fa44fb_3af0_46c2_82a6_d3b21c3e5383.slice/crio-2c01d756ed58d586530846e243807d9528f87a210d42e01000eea0aa944c630d WatchSource:0}: Error finding container 2c01d756ed58d586530846e243807d9528f87a210d42e01000eea0aa944c630d: Status 404 returned error can't find the container with id 2c01d756ed58d586530846e243807d9528f87a210d42e01000eea0aa944c630d Oct 31 04:04:31.645323 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:31.645283 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x7gql" event={"ID":"39fa44fb-3af0-46c2-82a6-d3b21c3e5383","Type":"ContainerStarted","Data":"2c01d756ed58d586530846e243807d9528f87a210d42e01000eea0aa944c630d"} Oct 31 04:04:32.006838 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:32.006804 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6f69bcc59c-w96bl" Oct 31 04:04:33.654996 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:33.654953 5019 generic.go:356] "Generic (PLEG): container finished" podID="39fa44fb-3af0-46c2-82a6-d3b21c3e5383" containerID="6c834870f404e0ccc1c5645934cb4e8d35d96ed44ffbb28b1928c78df4b6cdf1" exitCode=0 Oct 31 04:04:33.655501 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:33.655039 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x7gql" event={"ID":"39fa44fb-3af0-46c2-82a6-d3b21c3e5383","Type":"ContainerDied","Data":"6c834870f404e0ccc1c5645934cb4e8d35d96ed44ffbb28b1928c78df4b6cdf1"} Oct 31 04:04:34.661517 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:34.661464 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x7gql" event={"ID":"39fa44fb-3af0-46c2-82a6-d3b21c3e5383","Type":"ContainerStarted","Data":"775834d8e7464f5eb7fa35001e36889ac0ebd90fdd087cf824c629c23cf28ba0"} Oct 31 04:04:34.661924 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:34.661590 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x7gql" event={"ID":"39fa44fb-3af0-46c2-82a6-d3b21c3e5383","Type":"ContainerStarted","Data":"a440b56bea8ebbdfc98e75ddd0966419adbbadd7aae8d46928fbf6fd9dd8b531"} Oct 31 04:04:34.717202 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:34.717137 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-x7gql" podStartSLOduration=2.231430304 podStartE2EDuration="3.717121735s" totalImagesPullingTime="1.401000219s" totalInitContainerRuntime="84.691212ms" isStatefulPod=true podCreationTimestamp="2026-10-31 04:04:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:04:34.716301341 +0000 UTC m=+29.472639383" watchObservedRunningTime="2026-10-31 04:04:34.717121735 +0000 UTC m=+29.473459771" Oct 31 04:04:35.837995 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:35.837956 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" Oct 31 04:04:39.117007 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:39.116971 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-4b7d5" Oct 31 04:04:45.963617 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:45.963507 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f78cfdfd8-w98g7"] Oct 31 04:04:47.023166 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:47.023097 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6f69bcc59c-w96bl" podUID="e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22" containerName="registry" containerID="cri-o://f8f84bda9df0f5c26f554d2e8b6eb99408b7a44612ddabc8bbf1c7b9b6597531" gracePeriod=30 Oct 31 04:04:47.276804 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:47.276732 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f69bcc59c-w96bl" Oct 31 04:04:47.421447 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:47.421402 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-ca-trust-extracted\") pod \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\" (UID: \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\") " Oct 31 04:04:47.421637 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:47.421464 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-installation-pull-secrets\") pod \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\" (UID: \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\") " Oct 31 04:04:47.421637 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:47.421492 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-bound-sa-token\") pod \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\" (UID: \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\") " Oct 31 04:04:47.421637 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:47.421539 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-image-registry-private-configuration\") pod \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\" (UID: \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\") " Oct 31 04:04:47.421637 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:47.421572 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-kube-api-access-xc2vq\" (UniqueName: \"kubernetes.io/projected/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-kube-api-access-xc2vq\") pod \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\" (UID: \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\") " Oct 31 04:04:47.421829 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:47.421659 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-registry-certificates\") pod \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\" (UID: \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\") " Oct 31 04:04:47.421829 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:47.421690 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-trusted-ca\") pod \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\" (UID: \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\") " Oct 31 04:04:47.421897 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:47.421843 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-registry-tls\" (UniqueName: \"kubernetes.io/projected/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-registry-tls\") pod \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\" (UID: \"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22\") " Oct 31 04:04:47.422399 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:47.422363 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-registry-certificates" pod "e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22" (UID: "e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:04:47.422607 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:47.422385 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-trusted-ca" pod "e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22" (UID: "e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:04:47.424562 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:47.424511 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-installation-pull-secrets" pod "e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22" (UID: "e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:04:47.424562 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:47.424540 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-registry-tls" pod "e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22" (UID: "e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:04:47.424761 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:47.424646 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-image-registry-private-configuration" pod "e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22" (UID: "e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:04:47.424761 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:47.424670 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-bound-sa-token" pod "e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22" (UID: "e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:04:47.424831 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:47.424777 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-kube-api-access-xc2vq" pod "e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22" (UID: "e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22"). InnerVolumeSpecName "kube-api-access-xc2vq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:04:47.428947 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:47.428901 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-ca-trust-extracted" pod "e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22" (UID: "e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:04:47.522747 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:47.522706 5019 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-registry-certificates\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:04:47.522747 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:47.522738 5019 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-trusted-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:04:47.522747 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:47.522747 5019 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-registry-tls\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:04:47.522747 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:47.522755 5019 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-ca-trust-extracted\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:04:47.523010 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:47.522764 5019 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-installation-pull-secrets\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:04:47.523010 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:47.522773 5019 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-bound-sa-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:04:47.523010 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:47.522783 5019 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-image-registry-private-configuration\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:04:47.523010 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:47.522793 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xc2vq\" (UniqueName: \"kubernetes.io/projected/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22-kube-api-access-xc2vq\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:04:47.723841 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:47.723797 5019 generic.go:356] "Generic (PLEG): container finished" podID="e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22" containerID="f8f84bda9df0f5c26f554d2e8b6eb99408b7a44612ddabc8bbf1c7b9b6597531" exitCode=0 Oct 31 04:04:47.724043 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:47.723867 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f69bcc59c-w96bl" Oct 31 04:04:47.724043 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:47.723880 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f69bcc59c-w96bl" event={"ID":"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22","Type":"ContainerDied","Data":"f8f84bda9df0f5c26f554d2e8b6eb99408b7a44612ddabc8bbf1c7b9b6597531"} Oct 31 04:04:47.724043 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:47.723932 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f69bcc59c-w96bl" event={"ID":"e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22","Type":"ContainerDied","Data":"47915f7d9337e24318fb5ca597c226f543a6f9cdb35cf3ca4dbff32de99539ad"} Oct 31 04:04:47.724043 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:47.723951 5019 scope.go:122] "RemoveContainer" containerID="f8f84bda9df0f5c26f554d2e8b6eb99408b7a44612ddabc8bbf1c7b9b6597531" Oct 31 04:04:47.733368 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:47.733343 5019 scope.go:122] "RemoveContainer" containerID="f8f84bda9df0f5c26f554d2e8b6eb99408b7a44612ddabc8bbf1c7b9b6597531" Oct 31 04:04:47.733714 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:04:47.733696 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8f84bda9df0f5c26f554d2e8b6eb99408b7a44612ddabc8bbf1c7b9b6597531\": container with ID starting with f8f84bda9df0f5c26f554d2e8b6eb99408b7a44612ddabc8bbf1c7b9b6597531 not found: ID does not exist" containerID="f8f84bda9df0f5c26f554d2e8b6eb99408b7a44612ddabc8bbf1c7b9b6597531" Oct 31 04:04:47.733760 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:47.733724 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8f84bda9df0f5c26f554d2e8b6eb99408b7a44612ddabc8bbf1c7b9b6597531"} err="failed to get container status \"f8f84bda9df0f5c26f554d2e8b6eb99408b7a44612ddabc8bbf1c7b9b6597531\": rpc error: code = NotFound desc = could not find container \"f8f84bda9df0f5c26f554d2e8b6eb99408b7a44612ddabc8bbf1c7b9b6597531\": container with ID starting with f8f84bda9df0f5c26f554d2e8b6eb99408b7a44612ddabc8bbf1c7b9b6597531 not found: ID does not exist" Oct 31 04:04:47.773905 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:47.773870 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6f69bcc59c-w96bl"] Oct 31 04:04:47.785996 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:47.785955 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6f69bcc59c-w96bl"] Oct 31 04:04:49.404930 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:49.404892 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22" path="/var/lib/kubelet/pods/e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22/volumes" Oct 31 04:04:52.390196 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:04:52.390156 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:04:53.418498 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:53.418449 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-hcrqf-p69z8"] Oct 31 04:04:53.419079 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:53.418715 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-hcrqf-p69z8" podUID="908f1ac2-00d8-4fb6-8f92-daebf51f0ad8" containerName="insights-gathering" containerID="cri-o://0fab36732cbfe02fbbf445e40bffedb3a0940d76570a1f6aab06f3e4803612ee" gracePeriod=30 Oct 31 04:04:53.423607 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:53.423567 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-wh6x6-t7s8v"] Oct 31 04:04:53.423866 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:53.423819 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-wh6x6-t7s8v" podUID="07320d54-b911-4157-aaf5-2517187d745c" containerName="insights-gathering" containerID="cri-o://eab65427a89c6dce6446b410bfe76a21561a7d8a57d4d33db299596819ab954e" gracePeriod=30 Oct 31 04:04:53.705104 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:53.705070 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hcrqf-p69z8" Oct 31 04:04:53.708436 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:53.708407 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wh6x6-t7s8v" Oct 31 04:04:53.752599 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:53.752557 5019 generic.go:356] "Generic (PLEG): container finished" podID="908f1ac2-00d8-4fb6-8f92-daebf51f0ad8" containerID="0fab36732cbfe02fbbf445e40bffedb3a0940d76570a1f6aab06f3e4803612ee" exitCode=2 Oct 31 04:04:53.752805 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:53.752626 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hcrqf-p69z8" Oct 31 04:04:53.752805 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:53.752651 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hcrqf-p69z8" event={"ID":"908f1ac2-00d8-4fb6-8f92-daebf51f0ad8","Type":"ContainerDied","Data":"0fab36732cbfe02fbbf445e40bffedb3a0940d76570a1f6aab06f3e4803612ee"} Oct 31 04:04:53.752805 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:53.752711 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hcrqf-p69z8" event={"ID":"908f1ac2-00d8-4fb6-8f92-daebf51f0ad8","Type":"ContainerDied","Data":"86b93a3a277c0f30f1af32cd2ba5332fbd3e6361c1697b427762fd11f2e0c550"} Oct 31 04:04:53.752805 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:53.752739 5019 scope.go:122] "RemoveContainer" containerID="0fab36732cbfe02fbbf445e40bffedb3a0940d76570a1f6aab06f3e4803612ee" Oct 31 04:04:53.754236 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:53.754206 5019 generic.go:356] "Generic (PLEG): container finished" podID="07320d54-b911-4157-aaf5-2517187d745c" containerID="eab65427a89c6dce6446b410bfe76a21561a7d8a57d4d33db299596819ab954e" exitCode=2 Oct 31 04:04:53.754351 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:53.754268 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wh6x6-t7s8v" Oct 31 04:04:53.754351 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:53.754294 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wh6x6-t7s8v" event={"ID":"07320d54-b911-4157-aaf5-2517187d745c","Type":"ContainerDied","Data":"eab65427a89c6dce6446b410bfe76a21561a7d8a57d4d33db299596819ab954e"} Oct 31 04:04:53.754450 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:53.754379 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wh6x6-t7s8v" event={"ID":"07320d54-b911-4157-aaf5-2517187d745c","Type":"ContainerDied","Data":"a599fe68f82ea190e75d07f1f172d52fc495af94b489a72947ed0658320f0e77"} Oct 31 04:04:53.762923 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:53.762895 5019 scope.go:122] "RemoveContainer" containerID="0fab36732cbfe02fbbf445e40bffedb3a0940d76570a1f6aab06f3e4803612ee" Oct 31 04:04:53.763383 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:04:53.763362 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fab36732cbfe02fbbf445e40bffedb3a0940d76570a1f6aab06f3e4803612ee\": container with ID starting with 0fab36732cbfe02fbbf445e40bffedb3a0940d76570a1f6aab06f3e4803612ee not found: ID does not exist" containerID="0fab36732cbfe02fbbf445e40bffedb3a0940d76570a1f6aab06f3e4803612ee" Oct 31 04:04:53.763489 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:53.763397 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fab36732cbfe02fbbf445e40bffedb3a0940d76570a1f6aab06f3e4803612ee"} err="failed to get container status \"0fab36732cbfe02fbbf445e40bffedb3a0940d76570a1f6aab06f3e4803612ee\": rpc error: code = NotFound desc = could not find container \"0fab36732cbfe02fbbf445e40bffedb3a0940d76570a1f6aab06f3e4803612ee\": container with ID starting with 0fab36732cbfe02fbbf445e40bffedb3a0940d76570a1f6aab06f3e4803612ee not found: ID does not exist" Oct 31 04:04:53.763489 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:53.763423 5019 scope.go:122] "RemoveContainer" containerID="eab65427a89c6dce6446b410bfe76a21561a7d8a57d4d33db299596819ab954e" Oct 31 04:04:53.772409 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:53.772383 5019 scope.go:122] "RemoveContainer" containerID="eab65427a89c6dce6446b410bfe76a21561a7d8a57d4d33db299596819ab954e" Oct 31 04:04:53.772805 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:04:53.772783 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eab65427a89c6dce6446b410bfe76a21561a7d8a57d4d33db299596819ab954e\": container with ID starting with eab65427a89c6dce6446b410bfe76a21561a7d8a57d4d33db299596819ab954e not found: ID does not exist" containerID="eab65427a89c6dce6446b410bfe76a21561a7d8a57d4d33db299596819ab954e" Oct 31 04:04:53.772852 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:53.772816 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eab65427a89c6dce6446b410bfe76a21561a7d8a57d4d33db299596819ab954e"} err="failed to get container status \"eab65427a89c6dce6446b410bfe76a21561a7d8a57d4d33db299596819ab954e\": rpc error: code = NotFound desc = could not find container \"eab65427a89c6dce6446b410bfe76a21561a7d8a57d4d33db299596819ab954e\": container with ID starting with eab65427a89c6dce6446b410bfe76a21561a7d8a57d4d33db299596819ab954e not found: ID does not exist" Oct 31 04:04:53.879189 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:53.879145 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07320d54-b911-4157-aaf5-2517187d745c-kube-api-access-x4pc9\" (UniqueName: \"kubernetes.io/projected/07320d54-b911-4157-aaf5-2517187d745c-kube-api-access-x4pc9\") pod \"07320d54-b911-4157-aaf5-2517187d745c\" (UID: \"07320d54-b911-4157-aaf5-2517187d745c\") " Oct 31 04:04:53.879189 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:53.879196 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07320d54-b911-4157-aaf5-2517187d745c-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07320d54-b911-4157-aaf5-2517187d745c-service-ca-bundle\") pod \"07320d54-b911-4157-aaf5-2517187d745c\" (UID: \"07320d54-b911-4157-aaf5-2517187d745c\") " Oct 31 04:04:53.879482 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:53.879237 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/908f1ac2-00d8-4fb6-8f92-daebf51f0ad8-kube-api-access-mh5x7\" (UniqueName: \"kubernetes.io/projected/908f1ac2-00d8-4fb6-8f92-daebf51f0ad8-kube-api-access-mh5x7\") pod \"908f1ac2-00d8-4fb6-8f92-daebf51f0ad8\" (UID: \"908f1ac2-00d8-4fb6-8f92-daebf51f0ad8\") " Oct 31 04:04:53.879482 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:53.879305 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07320d54-b911-4157-aaf5-2517187d745c-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/07320d54-b911-4157-aaf5-2517187d745c-archives-path\") pod \"07320d54-b911-4157-aaf5-2517187d745c\" (UID: \"07320d54-b911-4157-aaf5-2517187d745c\") " Oct 31 04:04:53.879482 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:53.879363 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/908f1ac2-00d8-4fb6-8f92-daebf51f0ad8-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/908f1ac2-00d8-4fb6-8f92-daebf51f0ad8-service-ca-bundle\") pod \"908f1ac2-00d8-4fb6-8f92-daebf51f0ad8\" (UID: \"908f1ac2-00d8-4fb6-8f92-daebf51f0ad8\") " Oct 31 04:04:53.879482 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:53.879390 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/908f1ac2-00d8-4fb6-8f92-daebf51f0ad8-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/908f1ac2-00d8-4fb6-8f92-daebf51f0ad8-trusted-ca-bundle\") pod \"908f1ac2-00d8-4fb6-8f92-daebf51f0ad8\" (UID: \"908f1ac2-00d8-4fb6-8f92-daebf51f0ad8\") " Oct 31 04:04:53.879482 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:53.879426 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/908f1ac2-00d8-4fb6-8f92-daebf51f0ad8-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/908f1ac2-00d8-4fb6-8f92-daebf51f0ad8-archives-path\") pod \"908f1ac2-00d8-4fb6-8f92-daebf51f0ad8\" (UID: \"908f1ac2-00d8-4fb6-8f92-daebf51f0ad8\") " Oct 31 04:04:53.879482 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:53.879457 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07320d54-b911-4157-aaf5-2517187d745c-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07320d54-b911-4157-aaf5-2517187d745c-trusted-ca-bundle\") pod \"07320d54-b911-4157-aaf5-2517187d745c\" (UID: \"07320d54-b911-4157-aaf5-2517187d745c\") " Oct 31 04:04:53.879773 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:53.879717 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07320d54-b911-4157-aaf5-2517187d745c-service-ca-bundle" pod "07320d54-b911-4157-aaf5-2517187d745c" (UID: "07320d54-b911-4157-aaf5-2517187d745c"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:04:53.879773 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:53.879758 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/908f1ac2-00d8-4fb6-8f92-daebf51f0ad8-service-ca-bundle" pod "908f1ac2-00d8-4fb6-8f92-daebf51f0ad8" (UID: "908f1ac2-00d8-4fb6-8f92-daebf51f0ad8"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:04:53.880002 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:53.879807 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/908f1ac2-00d8-4fb6-8f92-daebf51f0ad8-archives-path" pod "908f1ac2-00d8-4fb6-8f92-daebf51f0ad8" (UID: "908f1ac2-00d8-4fb6-8f92-daebf51f0ad8"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:04:53.880002 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:53.879813 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/908f1ac2-00d8-4fb6-8f92-daebf51f0ad8-trusted-ca-bundle" pod "908f1ac2-00d8-4fb6-8f92-daebf51f0ad8" (UID: "908f1ac2-00d8-4fb6-8f92-daebf51f0ad8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:04:53.880112 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:53.879999 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07320d54-b911-4157-aaf5-2517187d745c-archives-path" pod "07320d54-b911-4157-aaf5-2517187d745c" (UID: "07320d54-b911-4157-aaf5-2517187d745c"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:04:53.880112 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:53.880094 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07320d54-b911-4157-aaf5-2517187d745c-trusted-ca-bundle" pod "07320d54-b911-4157-aaf5-2517187d745c" (UID: "07320d54-b911-4157-aaf5-2517187d745c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:04:53.881679 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:53.881652 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/908f1ac2-00d8-4fb6-8f92-daebf51f0ad8-kube-api-access-mh5x7" pod "908f1ac2-00d8-4fb6-8f92-daebf51f0ad8" (UID: "908f1ac2-00d8-4fb6-8f92-daebf51f0ad8"). InnerVolumeSpecName "kube-api-access-mh5x7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:04:53.881764 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:53.881711 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07320d54-b911-4157-aaf5-2517187d745c-kube-api-access-x4pc9" pod "07320d54-b911-4157-aaf5-2517187d745c" (UID: "07320d54-b911-4157-aaf5-2517187d745c"). InnerVolumeSpecName "kube-api-access-x4pc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:04:53.980609 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:53.980559 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mh5x7\" (UniqueName: \"kubernetes.io/projected/908f1ac2-00d8-4fb6-8f92-daebf51f0ad8-kube-api-access-mh5x7\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:04:53.980801 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:53.980638 5019 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/07320d54-b911-4157-aaf5-2517187d745c-archives-path\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:04:53.980801 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:53.980648 5019 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/908f1ac2-00d8-4fb6-8f92-daebf51f0ad8-service-ca-bundle\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:04:53.980801 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:53.980658 5019 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/908f1ac2-00d8-4fb6-8f92-daebf51f0ad8-trusted-ca-bundle\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:04:53.980801 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:53.980667 5019 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/908f1ac2-00d8-4fb6-8f92-daebf51f0ad8-archives-path\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:04:53.980801 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:53.980676 5019 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07320d54-b911-4157-aaf5-2517187d745c-trusted-ca-bundle\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:04:53.980801 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:53.980684 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4pc9\" (UniqueName: \"kubernetes.io/projected/07320d54-b911-4157-aaf5-2517187d745c-kube-api-access-x4pc9\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:04:53.980801 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:53.980694 5019 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07320d54-b911-4157-aaf5-2517187d745c-service-ca-bundle\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:04:54.099543 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:54.099505 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-wh6x6-t7s8v"] Oct 31 04:04:54.111780 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:54.111743 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-wh6x6-t7s8v"] Oct 31 04:04:54.137902 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:54.137865 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-hcrqf-p69z8"] Oct 31 04:04:54.142570 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:54.142537 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-hcrqf-p69z8"] Oct 31 04:04:55.405539 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:55.405505 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07320d54-b911-4157-aaf5-2517187d745c" path="/var/lib/kubelet/pods/07320d54-b911-4157-aaf5-2517187d745c/volumes" Oct 31 04:04:55.405929 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:04:55.405883 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="908f1ac2-00d8-4fb6-8f92-daebf51f0ad8" path="/var/lib/kubelet/pods/908f1ac2-00d8-4fb6-8f92-daebf51f0ad8/volumes" Oct 31 04:05:10.985449 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:05:10.985393 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" podUID="93beb8b8-c9e7-4cce-8baf-8497b971e193" containerName="registry" containerID="cri-o://c06e84d78c5e52e8c80db792fd749ef6219ed414166d85168a53d83644de3f21" gracePeriod=30 Oct 31 04:05:11.246520 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:05:11.246452 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" Oct 31 04:05:11.316676 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:05:11.316636 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93beb8b8-c9e7-4cce-8baf-8497b971e193-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/93beb8b8-c9e7-4cce-8baf-8497b971e193-installation-pull-secrets\") pod \"93beb8b8-c9e7-4cce-8baf-8497b971e193\" (UID: \"93beb8b8-c9e7-4cce-8baf-8497b971e193\") " Oct 31 04:05:11.316676 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:05:11.316689 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93beb8b8-c9e7-4cce-8baf-8497b971e193-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/93beb8b8-c9e7-4cce-8baf-8497b971e193-registry-certificates\") pod \"93beb8b8-c9e7-4cce-8baf-8497b971e193\" (UID: \"93beb8b8-c9e7-4cce-8baf-8497b971e193\") " Oct 31 04:05:11.316921 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:05:11.316724 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93beb8b8-c9e7-4cce-8baf-8497b971e193-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/93beb8b8-c9e7-4cce-8baf-8497b971e193-image-registry-private-configuration\") pod \"93beb8b8-c9e7-4cce-8baf-8497b971e193\" (UID: \"93beb8b8-c9e7-4cce-8baf-8497b971e193\") " Oct 31 04:05:11.316921 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:05:11.316838 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93beb8b8-c9e7-4cce-8baf-8497b971e193-kube-api-access-25x8r\" (UniqueName: \"kubernetes.io/projected/93beb8b8-c9e7-4cce-8baf-8497b971e193-kube-api-access-25x8r\") pod \"93beb8b8-c9e7-4cce-8baf-8497b971e193\" (UID: \"93beb8b8-c9e7-4cce-8baf-8497b971e193\") " Oct 31 04:05:11.316921 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:05:11.316880 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93beb8b8-c9e7-4cce-8baf-8497b971e193-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/93beb8b8-c9e7-4cce-8baf-8497b971e193-ca-trust-extracted\") pod \"93beb8b8-c9e7-4cce-8baf-8497b971e193\" (UID: \"93beb8b8-c9e7-4cce-8baf-8497b971e193\") " Oct 31 04:05:11.316921 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:05:11.316920 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93beb8b8-c9e7-4cce-8baf-8497b971e193-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/93beb8b8-c9e7-4cce-8baf-8497b971e193-bound-sa-token\") pod \"93beb8b8-c9e7-4cce-8baf-8497b971e193\" (UID: \"93beb8b8-c9e7-4cce-8baf-8497b971e193\") " Oct 31 04:05:11.317124 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:05:11.316942 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93beb8b8-c9e7-4cce-8baf-8497b971e193-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/93beb8b8-c9e7-4cce-8baf-8497b971e193-trusted-ca\") pod \"93beb8b8-c9e7-4cce-8baf-8497b971e193\" (UID: \"93beb8b8-c9e7-4cce-8baf-8497b971e193\") " Oct 31 04:05:11.317124 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:05:11.316979 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93beb8b8-c9e7-4cce-8baf-8497b971e193-registry-tls\" (UniqueName: \"kubernetes.io/projected/93beb8b8-c9e7-4cce-8baf-8497b971e193-registry-tls\") pod \"93beb8b8-c9e7-4cce-8baf-8497b971e193\" (UID: \"93beb8b8-c9e7-4cce-8baf-8497b971e193\") " Oct 31 04:05:11.317742 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:05:11.317685 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93beb8b8-c9e7-4cce-8baf-8497b971e193-registry-certificates" pod "93beb8b8-c9e7-4cce-8baf-8497b971e193" (UID: "93beb8b8-c9e7-4cce-8baf-8497b971e193"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:05:11.318684 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:05:11.318570 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93beb8b8-c9e7-4cce-8baf-8497b971e193-trusted-ca" pod "93beb8b8-c9e7-4cce-8baf-8497b971e193" (UID: "93beb8b8-c9e7-4cce-8baf-8497b971e193"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:05:11.322589 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:05:11.322543 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93beb8b8-c9e7-4cce-8baf-8497b971e193-bound-sa-token" pod "93beb8b8-c9e7-4cce-8baf-8497b971e193" (UID: "93beb8b8-c9e7-4cce-8baf-8497b971e193"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:05:11.322745 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:05:11.322544 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93beb8b8-c9e7-4cce-8baf-8497b971e193-image-registry-private-configuration" pod "93beb8b8-c9e7-4cce-8baf-8497b971e193" (UID: "93beb8b8-c9e7-4cce-8baf-8497b971e193"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:05:11.322745 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:05:11.322624 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93beb8b8-c9e7-4cce-8baf-8497b971e193-installation-pull-secrets" pod "93beb8b8-c9e7-4cce-8baf-8497b971e193" (UID: "93beb8b8-c9e7-4cce-8baf-8497b971e193"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:05:11.323065 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:05:11.323032 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93beb8b8-c9e7-4cce-8baf-8497b971e193-kube-api-access-25x8r" pod "93beb8b8-c9e7-4cce-8baf-8497b971e193" (UID: "93beb8b8-c9e7-4cce-8baf-8497b971e193"). InnerVolumeSpecName "kube-api-access-25x8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:05:11.325856 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:05:11.325634 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93beb8b8-c9e7-4cce-8baf-8497b971e193-registry-tls" pod "93beb8b8-c9e7-4cce-8baf-8497b971e193" (UID: "93beb8b8-c9e7-4cce-8baf-8497b971e193"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:05:11.327210 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:05:11.327182 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93beb8b8-c9e7-4cce-8baf-8497b971e193-ca-trust-extracted" pod "93beb8b8-c9e7-4cce-8baf-8497b971e193" (UID: "93beb8b8-c9e7-4cce-8baf-8497b971e193"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:05:11.418389 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:05:11.418351 5019 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/93beb8b8-c9e7-4cce-8baf-8497b971e193-bound-sa-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:05:11.418389 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:05:11.418386 5019 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/93beb8b8-c9e7-4cce-8baf-8497b971e193-trusted-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:05:11.418389 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:05:11.418396 5019 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/93beb8b8-c9e7-4cce-8baf-8497b971e193-registry-tls\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:05:11.418608 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:05:11.418405 5019 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/93beb8b8-c9e7-4cce-8baf-8497b971e193-installation-pull-secrets\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:05:11.418608 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:05:11.418416 5019 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/93beb8b8-c9e7-4cce-8baf-8497b971e193-registry-certificates\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:05:11.418608 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:05:11.418427 5019 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/93beb8b8-c9e7-4cce-8baf-8497b971e193-image-registry-private-configuration\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:05:11.418608 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:05:11.418437 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25x8r\" (UniqueName: \"kubernetes.io/projected/93beb8b8-c9e7-4cce-8baf-8497b971e193-kube-api-access-25x8r\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:05:11.418608 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:05:11.418445 5019 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/93beb8b8-c9e7-4cce-8baf-8497b971e193-ca-trust-extracted\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:05:11.833752 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:05:11.833712 5019 generic.go:356] "Generic (PLEG): container finished" podID="93beb8b8-c9e7-4cce-8baf-8497b971e193" containerID="c06e84d78c5e52e8c80db792fd749ef6219ed414166d85168a53d83644de3f21" exitCode=0 Oct 31 04:05:11.833934 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:05:11.833775 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" event={"ID":"93beb8b8-c9e7-4cce-8baf-8497b971e193","Type":"ContainerDied","Data":"c06e84d78c5e52e8c80db792fd749ef6219ed414166d85168a53d83644de3f21"} Oct 31 04:05:11.833934 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:05:11.833818 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" event={"ID":"93beb8b8-c9e7-4cce-8baf-8497b971e193","Type":"ContainerDied","Data":"fb7daaf87051b201e3957163cef4274739d5fec13c059e4f0b7c3a60def772fc"} Oct 31 04:05:11.833934 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:05:11.833838 5019 scope.go:122] "RemoveContainer" containerID="c06e84d78c5e52e8c80db792fd749ef6219ed414166d85168a53d83644de3f21" Oct 31 04:05:11.833934 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:05:11.833778 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f78cfdfd8-w98g7" Oct 31 04:05:11.842602 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:05:11.842555 5019 scope.go:122] "RemoveContainer" containerID="c06e84d78c5e52e8c80db792fd749ef6219ed414166d85168a53d83644de3f21" Oct 31 04:05:11.842945 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:05:11.842921 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c06e84d78c5e52e8c80db792fd749ef6219ed414166d85168a53d83644de3f21\": container with ID starting with c06e84d78c5e52e8c80db792fd749ef6219ed414166d85168a53d83644de3f21 not found: ID does not exist" containerID="c06e84d78c5e52e8c80db792fd749ef6219ed414166d85168a53d83644de3f21" Oct 31 04:05:11.843024 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:05:11.842960 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c06e84d78c5e52e8c80db792fd749ef6219ed414166d85168a53d83644de3f21"} err="failed to get container status \"c06e84d78c5e52e8c80db792fd749ef6219ed414166d85168a53d83644de3f21\": rpc error: code = NotFound desc = could not find container \"c06e84d78c5e52e8c80db792fd749ef6219ed414166d85168a53d83644de3f21\": container with ID starting with c06e84d78c5e52e8c80db792fd749ef6219ed414166d85168a53d83644de3f21 not found: ID does not exist" Oct 31 04:05:11.896788 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:05:11.896755 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f78cfdfd8-w98g7"] Oct 31 04:05:11.909208 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:05:11.909173 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7f78cfdfd8-w98g7"] Oct 31 04:05:13.405410 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:05:13.405369 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93beb8b8-c9e7-4cce-8baf-8497b971e193" path="/var/lib/kubelet/pods/93beb8b8-c9e7-4cce-8baf-8497b971e193/volumes" Oct 31 04:05:22.389647 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:05:22.389605 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:05:34.926829 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:05:34.926741 5019 generic.go:356] "Generic (PLEG): container finished" podID="944c9831-b2fa-4f58-bba2-93288793b125" containerID="11e7117bd4a15f93d0d609b64f88edc2a6d3d0cd2726c8e903546716dc0b63c2" exitCode=0 Oct 31 04:05:34.926829 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:05:34.926807 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-khdq4" event={"ID":"944c9831-b2fa-4f58-bba2-93288793b125","Type":"ContainerDied","Data":"11e7117bd4a15f93d0d609b64f88edc2a6d3d0cd2726c8e903546716dc0b63c2"} Oct 31 04:05:34.927286 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:05:34.927151 5019 scope.go:122] "RemoveContainer" containerID="11e7117bd4a15f93d0d609b64f88edc2a6d3d0cd2726c8e903546716dc0b63c2" Oct 31 04:05:35.932267 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:05:35.932231 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-khdq4" event={"ID":"944c9831-b2fa-4f58-bba2-93288793b125","Type":"ContainerStarted","Data":"4dda7fa4ab9cc0a2f075aa15925693114c04d55b6110f541a6e7bb2d860ee95e"} Oct 31 04:05:52.390305 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:05:52.390267 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:06:22.387797 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:06:22.387756 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:06:52.389123 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:06:52.389082 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:07:22.389661 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:07:22.389618 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:07:52.390198 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:07:52.390115 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:08:19.607942 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:19.607903 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pv8lb"] Oct 31 04:08:19.608417 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:19.608220 5019 cpu_manager.go:414] "containerMap: removing container" podUID="07320d54-b911-4157-aaf5-2517187d745c" containerName="insights-gathering" Oct 31 04:08:19.608417 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:19.608232 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07320d54-b911-4157-aaf5-2517187d745c" containerName="insights-gathering" Oct 31 04:08:19.608417 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:19.608246 5019 cpu_manager.go:414] "containerMap: removing container" podUID="908f1ac2-00d8-4fb6-8f92-daebf51f0ad8" containerName="insights-gathering" Oct 31 04:08:19.608417 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:19.608251 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="908f1ac2-00d8-4fb6-8f92-daebf51f0ad8" containerName="insights-gathering" Oct 31 04:08:19.608417 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:19.608259 5019 cpu_manager.go:414] "containerMap: removing container" podUID="93beb8b8-c9e7-4cce-8baf-8497b971e193" containerName="registry" Oct 31 04:08:19.608417 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:19.608264 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93beb8b8-c9e7-4cce-8baf-8497b971e193" containerName="registry" Oct 31 04:08:19.608417 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:19.608270 5019 cpu_manager.go:414] "containerMap: removing container" podUID="e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22" containerName="registry" Oct 31 04:08:19.608417 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:19.608275 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22" containerName="registry" Oct 31 04:08:19.608417 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:19.608323 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="07320d54-b911-4157-aaf5-2517187d745c" containerName="insights-gathering" Oct 31 04:08:19.608417 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:19.608344 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="908f1ac2-00d8-4fb6-8f92-daebf51f0ad8" containerName="insights-gathering" Oct 31 04:08:19.608417 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:19.608351 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="93beb8b8-c9e7-4cce-8baf-8497b971e193" containerName="registry" Oct 31 04:08:19.608417 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:19.608355 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e72e00a8-bf39-4ae3-93e5-6f0d58fd9f22" containerName="registry" Oct 31 04:08:19.611584 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:19.611563 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pv8lb" Oct 31 04:08:19.638461 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:19.638424 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 31 04:08:19.638657 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:19.638528 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-vg2xt\"" Oct 31 04:08:19.638657 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:19.638616 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 31 04:08:19.639624 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:19.639599 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pv8lb"] Oct 31 04:08:19.670975 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:19.670943 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5940a785-8b91-4dac-93a2-0f2cd52cb841-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pv8lb\" (UID: \"5940a785-8b91-4dac-93a2-0f2cd52cb841\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pv8lb" Oct 31 04:08:19.670975 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:19.670982 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5940a785-8b91-4dac-93a2-0f2cd52cb841-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pv8lb\" (UID: \"5940a785-8b91-4dac-93a2-0f2cd52cb841\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pv8lb" Oct 31 04:08:19.671190 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:19.671081 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp9gh\" (UniqueName: \"kubernetes.io/projected/5940a785-8b91-4dac-93a2-0f2cd52cb841-kube-api-access-jp9gh\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pv8lb\" (UID: \"5940a785-8b91-4dac-93a2-0f2cd52cb841\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pv8lb" Oct 31 04:08:19.771652 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:19.771613 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5940a785-8b91-4dac-93a2-0f2cd52cb841-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pv8lb\" (UID: \"5940a785-8b91-4dac-93a2-0f2cd52cb841\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pv8lb" Oct 31 04:08:19.771802 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:19.771684 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp9gh\" (UniqueName: \"kubernetes.io/projected/5940a785-8b91-4dac-93a2-0f2cd52cb841-kube-api-access-jp9gh\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pv8lb\" (UID: \"5940a785-8b91-4dac-93a2-0f2cd52cb841\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pv8lb" Oct 31 04:08:19.771802 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:19.771755 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5940a785-8b91-4dac-93a2-0f2cd52cb841-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pv8lb\" (UID: \"5940a785-8b91-4dac-93a2-0f2cd52cb841\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pv8lb" Oct 31 04:08:19.772105 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:19.772082 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5940a785-8b91-4dac-93a2-0f2cd52cb841-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pv8lb\" (UID: \"5940a785-8b91-4dac-93a2-0f2cd52cb841\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pv8lb" Oct 31 04:08:19.772140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:19.772125 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5940a785-8b91-4dac-93a2-0f2cd52cb841-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pv8lb\" (UID: \"5940a785-8b91-4dac-93a2-0f2cd52cb841\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pv8lb" Oct 31 04:08:19.795182 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:19.795149 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp9gh\" (UniqueName: \"kubernetes.io/projected/5940a785-8b91-4dac-93a2-0f2cd52cb841-kube-api-access-jp9gh\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pv8lb\" (UID: \"5940a785-8b91-4dac-93a2-0f2cd52cb841\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pv8lb" Oct 31 04:08:19.927427 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:19.927313 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pv8lb" Oct 31 04:08:20.219547 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:20.219517 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pv8lb"] Oct 31 04:08:20.222326 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:08:20.222291 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5940a785_8b91_4dac_93a2_0f2cd52cb841.slice/crio-8167a7b5c651c3c7042c3f97206582866fee12bef3809d2112f243d3bf8cdef7 WatchSource:0}: Error finding container 8167a7b5c651c3c7042c3f97206582866fee12bef3809d2112f243d3bf8cdef7: Status 404 returned error can't find the container with id 8167a7b5c651c3c7042c3f97206582866fee12bef3809d2112f243d3bf8cdef7 Oct 31 04:08:20.581057 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:20.580962 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pv8lb" event={"ID":"5940a785-8b91-4dac-93a2-0f2cd52cb841","Type":"ContainerStarted","Data":"8167a7b5c651c3c7042c3f97206582866fee12bef3809d2112f243d3bf8cdef7"} Oct 31 04:08:21.661245 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:21.661208 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwmdv"] Oct 31 04:08:21.665323 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:21.665294 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwmdv" Oct 31 04:08:21.694143 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:21.694081 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-mdg59\"" Oct 31 04:08:21.695612 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:21.695580 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwmdv"] Oct 31 04:08:21.792179 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:21.792137 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/201d9cd0-dea4-4313-912b-10b5e5ba3fae-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwmdv\" (UID: \"201d9cd0-dea4-4313-912b-10b5e5ba3fae\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwmdv" Oct 31 04:08:21.792386 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:21.792292 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pkw7\" (UniqueName: \"kubernetes.io/projected/201d9cd0-dea4-4313-912b-10b5e5ba3fae-kube-api-access-2pkw7\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwmdv\" (UID: \"201d9cd0-dea4-4313-912b-10b5e5ba3fae\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwmdv" Oct 31 04:08:21.792386 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:21.792368 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/201d9cd0-dea4-4313-912b-10b5e5ba3fae-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwmdv\" (UID: \"201d9cd0-dea4-4313-912b-10b5e5ba3fae\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwmdv" Oct 31 04:08:21.804959 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:21.804921 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67hxl7"] Oct 31 04:08:21.809176 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:21.809131 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67hxl7" Oct 31 04:08:21.828675 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:21.828639 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67hxl7"] Oct 31 04:08:21.829109 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:21.829085 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-bcxrv\"" Oct 31 04:08:21.867457 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:21.867411 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7gx8v"] Oct 31 04:08:21.871181 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:21.871147 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7gx8v" Oct 31 04:08:21.886682 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:21.886629 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-9xj5k\"" Oct 31 04:08:21.892009 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:21.891963 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7gx8v"] Oct 31 04:08:21.893604 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:21.893569 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/201d9cd0-dea4-4313-912b-10b5e5ba3fae-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwmdv\" (UID: \"201d9cd0-dea4-4313-912b-10b5e5ba3fae\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwmdv" Oct 31 04:08:21.893781 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:21.893629 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/201d9cd0-dea4-4313-912b-10b5e5ba3fae-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwmdv\" (UID: \"201d9cd0-dea4-4313-912b-10b5e5ba3fae\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwmdv" Oct 31 04:08:21.893781 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:21.893680 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d193edf4-187f-41c7-b7c9-2084a9dac2ef-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67hxl7\" (UID: \"d193edf4-187f-41c7-b7c9-2084a9dac2ef\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67hxl7" Oct 31 04:08:21.893781 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:21.893722 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjrpm\" (UniqueName: \"kubernetes.io/projected/d193edf4-187f-41c7-b7c9-2084a9dac2ef-kube-api-access-xjrpm\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67hxl7\" (UID: \"d193edf4-187f-41c7-b7c9-2084a9dac2ef\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67hxl7" Oct 31 04:08:21.893781 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:21.893756 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d193edf4-187f-41c7-b7c9-2084a9dac2ef-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67hxl7\" (UID: \"d193edf4-187f-41c7-b7c9-2084a9dac2ef\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67hxl7" Oct 31 04:08:21.893998 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:21.893861 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pkw7\" (UniqueName: \"kubernetes.io/projected/201d9cd0-dea4-4313-912b-10b5e5ba3fae-kube-api-access-2pkw7\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwmdv\" (UID: \"201d9cd0-dea4-4313-912b-10b5e5ba3fae\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwmdv" Oct 31 04:08:21.893998 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:21.893953 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/201d9cd0-dea4-4313-912b-10b5e5ba3fae-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwmdv\" (UID: \"201d9cd0-dea4-4313-912b-10b5e5ba3fae\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwmdv" Oct 31 04:08:21.894277 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:21.894253 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/201d9cd0-dea4-4313-912b-10b5e5ba3fae-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwmdv\" (UID: \"201d9cd0-dea4-4313-912b-10b5e5ba3fae\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwmdv" Oct 31 04:08:21.922197 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:21.922117 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pkw7\" (UniqueName: \"kubernetes.io/projected/201d9cd0-dea4-4313-912b-10b5e5ba3fae-kube-api-access-2pkw7\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwmdv\" (UID: \"201d9cd0-dea4-4313-912b-10b5e5ba3fae\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwmdv" Oct 31 04:08:21.981996 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:21.981963 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78lr2c"] Oct 31 04:08:21.983696 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:21.983660 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwmdv" Oct 31 04:08:21.987189 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:21.986989 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78lr2c" Oct 31 04:08:21.995504 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:21.995467 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjrpm\" (UniqueName: \"kubernetes.io/projected/d193edf4-187f-41c7-b7c9-2084a9dac2ef-kube-api-access-xjrpm\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67hxl7\" (UID: \"d193edf4-187f-41c7-b7c9-2084a9dac2ef\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67hxl7" Oct 31 04:08:21.995644 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:21.995540 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d256ddad-5170-4dee-963b-98098e1ba52c-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7gx8v\" (UID: \"d256ddad-5170-4dee-963b-98098e1ba52c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7gx8v" Oct 31 04:08:21.995644 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:21.995573 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d193edf4-187f-41c7-b7c9-2084a9dac2ef-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67hxl7\" (UID: \"d193edf4-187f-41c7-b7c9-2084a9dac2ef\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67hxl7" Oct 31 04:08:21.995729 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:21.995654 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d256ddad-5170-4dee-963b-98098e1ba52c-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7gx8v\" (UID: \"d256ddad-5170-4dee-963b-98098e1ba52c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7gx8v" Oct 31 04:08:21.995729 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:21.995681 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlxpc\" (UniqueName: \"kubernetes.io/projected/d256ddad-5170-4dee-963b-98098e1ba52c-kube-api-access-dlxpc\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7gx8v\" (UID: \"d256ddad-5170-4dee-963b-98098e1ba52c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7gx8v" Oct 31 04:08:21.995729 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:21.995721 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d193edf4-187f-41c7-b7c9-2084a9dac2ef-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67hxl7\" (UID: \"d193edf4-187f-41c7-b7c9-2084a9dac2ef\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67hxl7" Oct 31 04:08:21.996136 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:21.996116 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d193edf4-187f-41c7-b7c9-2084a9dac2ef-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67hxl7\" (UID: \"d193edf4-187f-41c7-b7c9-2084a9dac2ef\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67hxl7" Oct 31 04:08:21.996669 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:21.996623 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d193edf4-187f-41c7-b7c9-2084a9dac2ef-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67hxl7\" (UID: \"d193edf4-187f-41c7-b7c9-2084a9dac2ef\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67hxl7" Oct 31 04:08:22.006032 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.005995 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-vzj8n\"" Oct 31 04:08:22.007538 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.007506 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78lr2c"] Oct 31 04:08:22.022446 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.022413 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjrpm\" (UniqueName: \"kubernetes.io/projected/d193edf4-187f-41c7-b7c9-2084a9dac2ef-kube-api-access-xjrpm\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67hxl7\" (UID: \"d193edf4-187f-41c7-b7c9-2084a9dac2ef\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67hxl7" Oct 31 04:08:22.096235 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.096191 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6caaff37-4e5d-40f9-809d-fecd1c35cb12-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78lr2c\" (UID: \"6caaff37-4e5d-40f9-809d-fecd1c35cb12\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78lr2c" Oct 31 04:08:22.096429 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.096289 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6caaff37-4e5d-40f9-809d-fecd1c35cb12-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78lr2c\" (UID: \"6caaff37-4e5d-40f9-809d-fecd1c35cb12\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78lr2c" Oct 31 04:08:22.096429 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.096358 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d256ddad-5170-4dee-963b-98098e1ba52c-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7gx8v\" (UID: \"d256ddad-5170-4dee-963b-98098e1ba52c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7gx8v" Oct 31 04:08:22.096429 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.096380 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlxpc\" (UniqueName: \"kubernetes.io/projected/d256ddad-5170-4dee-963b-98098e1ba52c-kube-api-access-dlxpc\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7gx8v\" (UID: \"d256ddad-5170-4dee-963b-98098e1ba52c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7gx8v" Oct 31 04:08:22.096598 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.096479 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d256ddad-5170-4dee-963b-98098e1ba52c-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7gx8v\" (UID: \"d256ddad-5170-4dee-963b-98098e1ba52c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7gx8v" Oct 31 04:08:22.096598 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.096534 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds24g\" (UniqueName: \"kubernetes.io/projected/6caaff37-4e5d-40f9-809d-fecd1c35cb12-kube-api-access-ds24g\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78lr2c\" (UID: \"6caaff37-4e5d-40f9-809d-fecd1c35cb12\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78lr2c" Oct 31 04:08:22.096945 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.096926 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d256ddad-5170-4dee-963b-98098e1ba52c-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7gx8v\" (UID: \"d256ddad-5170-4dee-963b-98098e1ba52c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7gx8v" Oct 31 04:08:22.097003 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.096965 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d256ddad-5170-4dee-963b-98098e1ba52c-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7gx8v\" (UID: \"d256ddad-5170-4dee-963b-98098e1ba52c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7gx8v" Oct 31 04:08:22.120600 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.120566 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlxpc\" (UniqueName: \"kubernetes.io/projected/d256ddad-5170-4dee-963b-98098e1ba52c-kube-api-access-dlxpc\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7gx8v\" (UID: \"d256ddad-5170-4dee-963b-98098e1ba52c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7gx8v" Oct 31 04:08:22.129282 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.129239 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67hxl7" Oct 31 04:08:22.190045 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.189939 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7gx8v" Oct 31 04:08:22.197547 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.197507 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds24g\" (UniqueName: \"kubernetes.io/projected/6caaff37-4e5d-40f9-809d-fecd1c35cb12-kube-api-access-ds24g\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78lr2c\" (UID: \"6caaff37-4e5d-40f9-809d-fecd1c35cb12\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78lr2c" Oct 31 04:08:22.199292 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.198151 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6caaff37-4e5d-40f9-809d-fecd1c35cb12-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78lr2c\" (UID: \"6caaff37-4e5d-40f9-809d-fecd1c35cb12\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78lr2c" Oct 31 04:08:22.199292 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.198203 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6caaff37-4e5d-40f9-809d-fecd1c35cb12-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78lr2c\" (UID: \"6caaff37-4e5d-40f9-809d-fecd1c35cb12\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78lr2c" Oct 31 04:08:22.199292 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.198765 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6caaff37-4e5d-40f9-809d-fecd1c35cb12-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78lr2c\" (UID: \"6caaff37-4e5d-40f9-809d-fecd1c35cb12\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78lr2c" Oct 31 04:08:22.199292 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.199002 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6caaff37-4e5d-40f9-809d-fecd1c35cb12-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78lr2c\" (UID: \"6caaff37-4e5d-40f9-809d-fecd1c35cb12\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78lr2c" Oct 31 04:08:22.229732 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.229696 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds24g\" (UniqueName: \"kubernetes.io/projected/6caaff37-4e5d-40f9-809d-fecd1c35cb12-kube-api-access-ds24g\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78lr2c\" (UID: \"6caaff37-4e5d-40f9-809d-fecd1c35cb12\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78lr2c" Oct 31 04:08:22.291627 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.291591 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwmdv"] Oct 31 04:08:22.295591 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:08:22.295556 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod201d9cd0_dea4_4313_912b_10b5e5ba3fae.slice/crio-6b38752d7d65f5ecc624ebe9c9c0d2a2b09b12d4b44b72e66a9c0ee839b286c3 WatchSource:0}: Error finding container 6b38752d7d65f5ecc624ebe9c9c0d2a2b09b12d4b44b72e66a9c0ee839b286c3: Status 404 returned error can't find the container with id 6b38752d7d65f5ecc624ebe9c9c0d2a2b09b12d4b44b72e66a9c0ee839b286c3 Oct 31 04:08:22.334629 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.334593 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78lr2c" Oct 31 04:08:22.399881 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:08:22.399801 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:08:22.441843 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.441027 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67hxl7"] Oct 31 04:08:22.444850 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:08:22.444811 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd193edf4_187f_41c7_b7c9_2084a9dac2ef.slice/crio-c23264c617c6cafdf81b3eb7e105ec565e07c416ee1021b7ab7e0f9ba9b9152d WatchSource:0}: Error finding container c23264c617c6cafdf81b3eb7e105ec565e07c416ee1021b7ab7e0f9ba9b9152d: Status 404 returned error can't find the container with id c23264c617c6cafdf81b3eb7e105ec565e07c416ee1021b7ab7e0f9ba9b9152d Oct 31 04:08:22.509747 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.509706 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7gx8v"] Oct 31 04:08:22.513444 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:08:22.513405 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd256ddad_5170_4dee_963b_98098e1ba52c.slice/crio-4444b21b278727467f2336ed5971ec120599790e47132a4301423a0a7e9ee1ff WatchSource:0}: Error finding container 4444b21b278727467f2336ed5971ec120599790e47132a4301423a0a7e9ee1ff: Status 404 returned error can't find the container with id 4444b21b278727467f2336ed5971ec120599790e47132a4301423a0a7e9ee1ff Oct 31 04:08:22.592346 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.592306 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwmdv" event={"ID":"201d9cd0-dea4-4313-912b-10b5e5ba3fae","Type":"ContainerStarted","Data":"6b38752d7d65f5ecc624ebe9c9c0d2a2b09b12d4b44b72e66a9c0ee839b286c3"} Oct 31 04:08:22.593683 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.593653 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7gx8v" event={"ID":"d256ddad-5170-4dee-963b-98098e1ba52c","Type":"ContainerStarted","Data":"4444b21b278727467f2336ed5971ec120599790e47132a4301423a0a7e9ee1ff"} Oct 31 04:08:22.595097 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.595049 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67hxl7" event={"ID":"d193edf4-187f-41c7-b7c9-2084a9dac2ef","Type":"ContainerStarted","Data":"c23264c617c6cafdf81b3eb7e105ec565e07c416ee1021b7ab7e0f9ba9b9152d"} Oct 31 04:08:22.647351 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.647301 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78lr2c"] Oct 31 04:08:22.651295 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:08:22.651256 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6caaff37_4e5d_40f9_809d_fecd1c35cb12.slice/crio-e3ef97af91dbc01467c1aa961d514506d8cfa0b74a547ce7c7a87910676ca1de WatchSource:0}: Error finding container e3ef97af91dbc01467c1aa961d514506d8cfa0b74a547ce7c7a87910676ca1de: Status 404 returned error can't find the container with id e3ef97af91dbc01467c1aa961d514506d8cfa0b74a547ce7c7a87910676ca1de Oct 31 04:08:22.745286 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.743635 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7"] Oct 31 04:08:22.748711 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.748677 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7" Oct 31 04:08:22.771697 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.771658 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7"] Oct 31 04:08:22.771873 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.771843 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 31 04:08:22.810643 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.810401 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0ccc03e-0e5e-40b7-8a40-718f199dd747-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pghv7\" (UID: \"e0ccc03e-0e5e-40b7-8a40-718f199dd747\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7" Oct 31 04:08:22.810643 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.810534 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0ccc03e-0e5e-40b7-8a40-718f199dd747-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pghv7\" (UID: \"e0ccc03e-0e5e-40b7-8a40-718f199dd747\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7" Oct 31 04:08:22.810643 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.810596 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0ccc03e-0e5e-40b7-8a40-718f199dd747-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pghv7\" (UID: \"e0ccc03e-0e5e-40b7-8a40-718f199dd747\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7" Oct 31 04:08:22.810950 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.810692 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0ccc03e-0e5e-40b7-8a40-718f199dd747-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pghv7\" (UID: \"e0ccc03e-0e5e-40b7-8a40-718f199dd747\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7" Oct 31 04:08:22.810950 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.810740 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ccc03e-0e5e-40b7-8a40-718f199dd747-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pghv7\" (UID: \"e0ccc03e-0e5e-40b7-8a40-718f199dd747\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7" Oct 31 04:08:22.810950 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.810862 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0ccc03e-0e5e-40b7-8a40-718f199dd747-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pghv7\" (UID: \"e0ccc03e-0e5e-40b7-8a40-718f199dd747\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7" Oct 31 04:08:22.810950 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.810909 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0ccc03e-0e5e-40b7-8a40-718f199dd747-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pghv7\" (UID: \"e0ccc03e-0e5e-40b7-8a40-718f199dd747\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7" Oct 31 04:08:22.810950 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.810937 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-999n5\" (UniqueName: \"kubernetes.io/projected/e0ccc03e-0e5e-40b7-8a40-718f199dd747-kube-api-access-999n5\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pghv7\" (UID: \"e0ccc03e-0e5e-40b7-8a40-718f199dd747\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7" Oct 31 04:08:22.811295 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.810965 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ccc03e-0e5e-40b7-8a40-718f199dd747-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pghv7\" (UID: \"e0ccc03e-0e5e-40b7-8a40-718f199dd747\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7" Oct 31 04:08:22.812316 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.812272 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-sqqs6\"" Oct 31 04:08:22.886048 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.886009 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq"] Oct 31 04:08:22.906408 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.904592 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq" Oct 31 04:08:22.914813 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.914777 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0ccc03e-0e5e-40b7-8a40-718f199dd747-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pghv7\" (UID: \"e0ccc03e-0e5e-40b7-8a40-718f199dd747\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7" Oct 31 04:08:22.915512 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.915486 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0ccc03e-0e5e-40b7-8a40-718f199dd747-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pghv7\" (UID: \"e0ccc03e-0e5e-40b7-8a40-718f199dd747\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7" Oct 31 04:08:22.915650 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.915537 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0ccc03e-0e5e-40b7-8a40-718f199dd747-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pghv7\" (UID: \"e0ccc03e-0e5e-40b7-8a40-718f199dd747\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7" Oct 31 04:08:22.915650 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.915574 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ccc03e-0e5e-40b7-8a40-718f199dd747-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pghv7\" (UID: \"e0ccc03e-0e5e-40b7-8a40-718f199dd747\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7" Oct 31 04:08:22.915650 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.915635 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0ccc03e-0e5e-40b7-8a40-718f199dd747-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pghv7\" (UID: \"e0ccc03e-0e5e-40b7-8a40-718f199dd747\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7" Oct 31 04:08:22.915812 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.915667 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0ccc03e-0e5e-40b7-8a40-718f199dd747-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pghv7\" (UID: \"e0ccc03e-0e5e-40b7-8a40-718f199dd747\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7" Oct 31 04:08:22.915812 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.915692 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-999n5\" (UniqueName: \"kubernetes.io/projected/e0ccc03e-0e5e-40b7-8a40-718f199dd747-kube-api-access-999n5\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pghv7\" (UID: \"e0ccc03e-0e5e-40b7-8a40-718f199dd747\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7" Oct 31 04:08:22.915812 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.915718 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ccc03e-0e5e-40b7-8a40-718f199dd747-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pghv7\" (UID: \"e0ccc03e-0e5e-40b7-8a40-718f199dd747\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7" Oct 31 04:08:22.915812 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.915751 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0ccc03e-0e5e-40b7-8a40-718f199dd747-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pghv7\" (UID: \"e0ccc03e-0e5e-40b7-8a40-718f199dd747\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7" Oct 31 04:08:22.917006 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.916686 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0ccc03e-0e5e-40b7-8a40-718f199dd747-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pghv7\" (UID: \"e0ccc03e-0e5e-40b7-8a40-718f199dd747\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7" Oct 31 04:08:22.917006 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.916904 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ccc03e-0e5e-40b7-8a40-718f199dd747-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pghv7\" (UID: \"e0ccc03e-0e5e-40b7-8a40-718f199dd747\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7" Oct 31 04:08:22.917006 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.917000 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0ccc03e-0e5e-40b7-8a40-718f199dd747-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pghv7\" (UID: \"e0ccc03e-0e5e-40b7-8a40-718f199dd747\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7" Oct 31 04:08:22.917842 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.917783 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0ccc03e-0e5e-40b7-8a40-718f199dd747-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pghv7\" (UID: \"e0ccc03e-0e5e-40b7-8a40-718f199dd747\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7" Oct 31 04:08:22.918708 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.918679 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ccc03e-0e5e-40b7-8a40-718f199dd747-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pghv7\" (UID: \"e0ccc03e-0e5e-40b7-8a40-718f199dd747\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7" Oct 31 04:08:22.919112 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.919092 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq"] Oct 31 04:08:22.920982 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.920818 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0ccc03e-0e5e-40b7-8a40-718f199dd747-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pghv7\" (UID: \"e0ccc03e-0e5e-40b7-8a40-718f199dd747\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7" Oct 31 04:08:22.922911 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.922862 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0ccc03e-0e5e-40b7-8a40-718f199dd747-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pghv7\" (UID: \"e0ccc03e-0e5e-40b7-8a40-718f199dd747\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7" Oct 31 04:08:22.923411 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.923359 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-7rkmf\"" Oct 31 04:08:22.943681 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.943647 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-999n5\" (UniqueName: \"kubernetes.io/projected/e0ccc03e-0e5e-40b7-8a40-718f199dd747-kube-api-access-999n5\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pghv7\" (UID: \"e0ccc03e-0e5e-40b7-8a40-718f199dd747\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7" Oct 31 04:08:22.946747 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:22.946686 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0ccc03e-0e5e-40b7-8a40-718f199dd747-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pghv7\" (UID: \"e0ccc03e-0e5e-40b7-8a40-718f199dd747\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7" Oct 31 04:08:23.016717 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:23.016618 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17297059-70a0-480f-ad3c-4427585cf15c-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq\" (UID: \"17297059-70a0-480f-ad3c-4427585cf15c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq" Oct 31 04:08:23.016717 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:23.016670 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17297059-70a0-480f-ad3c-4427585cf15c-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq\" (UID: \"17297059-70a0-480f-ad3c-4427585cf15c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq" Oct 31 04:08:23.016961 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:23.016719 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17297059-70a0-480f-ad3c-4427585cf15c-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq\" (UID: \"17297059-70a0-480f-ad3c-4427585cf15c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq" Oct 31 04:08:23.016961 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:23.016770 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17297059-70a0-480f-ad3c-4427585cf15c-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq\" (UID: \"17297059-70a0-480f-ad3c-4427585cf15c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq" Oct 31 04:08:23.016961 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:23.016807 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8b5r\" (UniqueName: \"kubernetes.io/projected/17297059-70a0-480f-ad3c-4427585cf15c-kube-api-access-l8b5r\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq\" (UID: \"17297059-70a0-480f-ad3c-4427585cf15c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq" Oct 31 04:08:23.016961 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:23.016836 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17297059-70a0-480f-ad3c-4427585cf15c-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq\" (UID: \"17297059-70a0-480f-ad3c-4427585cf15c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq" Oct 31 04:08:23.016961 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:23.016876 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17297059-70a0-480f-ad3c-4427585cf15c-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq\" (UID: \"17297059-70a0-480f-ad3c-4427585cf15c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq" Oct 31 04:08:23.016961 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:23.016908 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17297059-70a0-480f-ad3c-4427585cf15c-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq\" (UID: \"17297059-70a0-480f-ad3c-4427585cf15c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq" Oct 31 04:08:23.016961 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:23.016943 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17297059-70a0-480f-ad3c-4427585cf15c-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq\" (UID: \"17297059-70a0-480f-ad3c-4427585cf15c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq" Oct 31 04:08:23.066581 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:23.066530 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7" Oct 31 04:08:23.119431 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:23.118443 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17297059-70a0-480f-ad3c-4427585cf15c-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq\" (UID: \"17297059-70a0-480f-ad3c-4427585cf15c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq" Oct 31 04:08:23.119431 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:23.118526 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17297059-70a0-480f-ad3c-4427585cf15c-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq\" (UID: \"17297059-70a0-480f-ad3c-4427585cf15c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq" Oct 31 04:08:23.119431 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:23.118569 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8b5r\" (UniqueName: \"kubernetes.io/projected/17297059-70a0-480f-ad3c-4427585cf15c-kube-api-access-l8b5r\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq\" (UID: \"17297059-70a0-480f-ad3c-4427585cf15c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq" Oct 31 04:08:23.119431 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:23.118599 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17297059-70a0-480f-ad3c-4427585cf15c-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq\" (UID: \"17297059-70a0-480f-ad3c-4427585cf15c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq" Oct 31 04:08:23.119431 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:23.118644 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17297059-70a0-480f-ad3c-4427585cf15c-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq\" (UID: \"17297059-70a0-480f-ad3c-4427585cf15c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq" Oct 31 04:08:23.119431 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:23.118672 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17297059-70a0-480f-ad3c-4427585cf15c-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq\" (UID: \"17297059-70a0-480f-ad3c-4427585cf15c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq" Oct 31 04:08:23.119431 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:23.118706 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17297059-70a0-480f-ad3c-4427585cf15c-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq\" (UID: \"17297059-70a0-480f-ad3c-4427585cf15c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq" Oct 31 04:08:23.119431 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:23.118770 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17297059-70a0-480f-ad3c-4427585cf15c-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq\" (UID: \"17297059-70a0-480f-ad3c-4427585cf15c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq" Oct 31 04:08:23.119431 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:23.118795 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17297059-70a0-480f-ad3c-4427585cf15c-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq\" (UID: \"17297059-70a0-480f-ad3c-4427585cf15c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq" Oct 31 04:08:23.120958 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:23.120370 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17297059-70a0-480f-ad3c-4427585cf15c-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq\" (UID: \"17297059-70a0-480f-ad3c-4427585cf15c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq" Oct 31 04:08:23.120958 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:23.120410 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17297059-70a0-480f-ad3c-4427585cf15c-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq\" (UID: \"17297059-70a0-480f-ad3c-4427585cf15c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq" Oct 31 04:08:23.120958 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:23.120467 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17297059-70a0-480f-ad3c-4427585cf15c-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq\" (UID: \"17297059-70a0-480f-ad3c-4427585cf15c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq" Oct 31 04:08:23.120958 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:23.120620 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17297059-70a0-480f-ad3c-4427585cf15c-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq\" (UID: \"17297059-70a0-480f-ad3c-4427585cf15c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq" Oct 31 04:08:23.120958 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:23.120891 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17297059-70a0-480f-ad3c-4427585cf15c-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq\" (UID: \"17297059-70a0-480f-ad3c-4427585cf15c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq" Oct 31 04:08:23.123480 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:23.123345 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17297059-70a0-480f-ad3c-4427585cf15c-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq\" (UID: \"17297059-70a0-480f-ad3c-4427585cf15c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq" Oct 31 04:08:23.123792 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:23.123768 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17297059-70a0-480f-ad3c-4427585cf15c-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq\" (UID: \"17297059-70a0-480f-ad3c-4427585cf15c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq" Oct 31 04:08:23.144247 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:23.144211 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17297059-70a0-480f-ad3c-4427585cf15c-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq\" (UID: \"17297059-70a0-480f-ad3c-4427585cf15c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq" Oct 31 04:08:23.148676 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:23.148048 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8b5r\" (UniqueName: \"kubernetes.io/projected/17297059-70a0-480f-ad3c-4427585cf15c-kube-api-access-l8b5r\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq\" (UID: \"17297059-70a0-480f-ad3c-4427585cf15c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq" Oct 31 04:08:23.268168 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:23.267755 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq" Oct 31 04:08:23.418841 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:23.418660 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7"] Oct 31 04:08:23.427014 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:08:23.426926 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0ccc03e_0e5e_40b7_8a40_718f199dd747.slice/crio-06048bc2a80e48fc0b36ab3e56a74ba9d07535cb47e67eab0f90a8115a701e93 WatchSource:0}: Error finding container 06048bc2a80e48fc0b36ab3e56a74ba9d07535cb47e67eab0f90a8115a701e93: Status 404 returned error can't find the container with id 06048bc2a80e48fc0b36ab3e56a74ba9d07535cb47e67eab0f90a8115a701e93 Oct 31 04:08:23.602553 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:23.602457 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7" event={"ID":"e0ccc03e-0e5e-40b7-8a40-718f199dd747","Type":"ContainerStarted","Data":"06048bc2a80e48fc0b36ab3e56a74ba9d07535cb47e67eab0f90a8115a701e93"} Oct 31 04:08:23.604509 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:23.604422 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78lr2c" event={"ID":"6caaff37-4e5d-40f9-809d-fecd1c35cb12","Type":"ContainerStarted","Data":"e3ef97af91dbc01467c1aa961d514506d8cfa0b74a547ce7c7a87910676ca1de"} Oct 31 04:08:23.616536 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:23.616440 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq"] Oct 31 04:08:23.623081 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:08:23.623042 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17297059_70a0_480f_ad3c_4427585cf15c.slice/crio-e5ae59fa6726c24e72b76f9b9a6ebc4a65e972c0b73ad00c4f588781620b4c7c WatchSource:0}: Error finding container e5ae59fa6726c24e72b76f9b9a6ebc4a65e972c0b73ad00c4f588781620b4c7c: Status 404 returned error can't find the container with id e5ae59fa6726c24e72b76f9b9a6ebc4a65e972c0b73ad00c4f588781620b4c7c Oct 31 04:08:24.618958 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:24.618878 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq" event={"ID":"17297059-70a0-480f-ad3c-4427585cf15c","Type":"ContainerStarted","Data":"e5ae59fa6726c24e72b76f9b9a6ebc4a65e972c0b73ad00c4f588781620b4c7c"} Oct 31 04:08:29.681231 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:29.680460 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:08:29.681231 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:29.680555 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:08:29.681231 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:29.680598 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:08:30.129421 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:30.128589 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:08:30.129421 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:30.128702 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:08:30.129421 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:30.128752 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:08:30.661708 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:30.660777 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7" event={"ID":"e0ccc03e-0e5e-40b7-8a40-718f199dd747","Type":"ContainerStarted","Data":"f0f1b0a9d04e4f7be16d6c8de71e04915a34ca21d3c7c2dd2a5caf4de378669e"} Oct 31 04:08:30.665377 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:30.665327 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq" event={"ID":"17297059-70a0-480f-ad3c-4427585cf15c","Type":"ContainerStarted","Data":"2d53df32d7e20b0bd4c12f316b366ab8bbf78e055330ba2a4429ff1ebf76c204"} Oct 31 04:08:30.733297 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:30.732541 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7" podStartSLOduration=2.48213294 podStartE2EDuration="8.73251926s" totalImagesPullingTime="6.25038632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:08:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:08:30.719875976 +0000 UTC m=+265.476214021" watchObservedRunningTime="2026-10-31 04:08:30.73251926 +0000 UTC m=+265.488857301" Oct 31 04:08:31.067370 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:31.067318 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7" Oct 31 04:08:31.074380 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:31.074296 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7" Oct 31 04:08:31.120906 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:31.120846 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq" podStartSLOduration=2.619187074 podStartE2EDuration="9.120824011s" totalImagesPullingTime="6.501636937s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:08:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:08:30.766678752 +0000 UTC m=+265.523016770" watchObservedRunningTime="2026-10-31 04:08:31.120824011 +0000 UTC m=+265.877162114" Oct 31 04:08:31.270031 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:31.269992 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq" Oct 31 04:08:31.273887 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:31.273678 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq" Oct 31 04:08:31.677399 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:31.673735 5019 generic.go:356] "Generic (PLEG): container finished" podID="5940a785-8b91-4dac-93a2-0f2cd52cb841" containerID="4da54463ebd936fc28619574759988326b38c6a9712daa8bb8c7fdb08b9978f5" exitCode=0 Oct 31 04:08:31.677399 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:31.675432 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pv8lb" event={"ID":"5940a785-8b91-4dac-93a2-0f2cd52cb841","Type":"ContainerDied","Data":"4da54463ebd936fc28619574759988326b38c6a9712daa8bb8c7fdb08b9978f5"} Oct 31 04:08:31.677399 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:31.675488 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7" Oct 31 04:08:31.677399 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:31.676476 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq" Oct 31 04:08:31.678272 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:31.678169 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq" Oct 31 04:08:31.678812 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:31.678717 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7" Oct 31 04:08:34.692747 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:34.692708 5019 generic.go:356] "Generic (PLEG): container finished" podID="201d9cd0-dea4-4313-912b-10b5e5ba3fae" containerID="561b21437f21c3318ba0077f0ee63fbfdb6801bdf3be93f5d778d6dc5e088cce" exitCode=0 Oct 31 04:08:34.693155 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:34.692797 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwmdv" event={"ID":"201d9cd0-dea4-4313-912b-10b5e5ba3fae","Type":"ContainerDied","Data":"561b21437f21c3318ba0077f0ee63fbfdb6801bdf3be93f5d778d6dc5e088cce"} Oct 31 04:08:34.694463 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:34.694433 5019 generic.go:356] "Generic (PLEG): container finished" podID="d256ddad-5170-4dee-963b-98098e1ba52c" containerID="26e97a5b0db9da8a450486b6144053a44e9d8aacd0ebb0fcae588df1ae416be1" exitCode=0 Oct 31 04:08:34.694602 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:34.694527 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7gx8v" event={"ID":"d256ddad-5170-4dee-963b-98098e1ba52c","Type":"ContainerDied","Data":"26e97a5b0db9da8a450486b6144053a44e9d8aacd0ebb0fcae588df1ae416be1"} Oct 31 04:08:34.696200 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:34.696176 5019 generic.go:356] "Generic (PLEG): container finished" podID="d193edf4-187f-41c7-b7c9-2084a9dac2ef" containerID="e0179aa501eceab9a91b8be91076e3e0ab35ff6ad0f93520edf26fee8100fdce" exitCode=0 Oct 31 04:08:34.696308 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:34.696243 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67hxl7" event={"ID":"d193edf4-187f-41c7-b7c9-2084a9dac2ef","Type":"ContainerDied","Data":"e0179aa501eceab9a91b8be91076e3e0ab35ff6ad0f93520edf26fee8100fdce"} Oct 31 04:08:34.697788 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:34.697769 5019 generic.go:356] "Generic (PLEG): container finished" podID="6caaff37-4e5d-40f9-809d-fecd1c35cb12" containerID="ed2dcb1e3565c2acdf37f6bb87502b87333f0e472e0ae2009b10e081b4af7281" exitCode=0 Oct 31 04:08:34.697873 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:34.697852 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78lr2c" event={"ID":"6caaff37-4e5d-40f9-809d-fecd1c35cb12","Type":"ContainerDied","Data":"ed2dcb1e3565c2acdf37f6bb87502b87333f0e472e0ae2009b10e081b4af7281"} Oct 31 04:08:35.706054 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:35.706021 5019 generic.go:356] "Generic (PLEG): container finished" podID="d256ddad-5170-4dee-963b-98098e1ba52c" containerID="1f52da1e295f76b8bdb89adaa619251dc2d7a288ccee528c07c4043af39e0eb4" exitCode=0 Oct 31 04:08:35.706549 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:35.706117 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7gx8v" event={"ID":"d256ddad-5170-4dee-963b-98098e1ba52c","Type":"ContainerDied","Data":"1f52da1e295f76b8bdb89adaa619251dc2d7a288ccee528c07c4043af39e0eb4"} Oct 31 04:08:35.708199 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:35.708174 5019 generic.go:356] "Generic (PLEG): container finished" podID="6caaff37-4e5d-40f9-809d-fecd1c35cb12" containerID="edd186edd82dde958334e22f063427335022b7de7b6f4bfa3c701655367e26ca" exitCode=0 Oct 31 04:08:35.708296 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:35.708226 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78lr2c" event={"ID":"6caaff37-4e5d-40f9-809d-fecd1c35cb12","Type":"ContainerDied","Data":"edd186edd82dde958334e22f063427335022b7de7b6f4bfa3c701655367e26ca"} Oct 31 04:08:36.717620 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:36.717569 5019 generic.go:356] "Generic (PLEG): container finished" podID="d193edf4-187f-41c7-b7c9-2084a9dac2ef" containerID="a8807f9b90f2eda60fde77618abbbe7ace0412627e1f1bbf60ae149fda521a74" exitCode=0 Oct 31 04:08:36.718222 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:36.717746 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67hxl7" event={"ID":"d193edf4-187f-41c7-b7c9-2084a9dac2ef","Type":"ContainerDied","Data":"a8807f9b90f2eda60fde77618abbbe7ace0412627e1f1bbf60ae149fda521a74"} Oct 31 04:08:36.722272 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:36.722240 5019 generic.go:356] "Generic (PLEG): container finished" podID="201d9cd0-dea4-4313-912b-10b5e5ba3fae" containerID="5e756125ec865f9b38bee73505951cfb30aee3e2c98fa035b9e1a95a2d5d8583" exitCode=0 Oct 31 04:08:36.722403 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:36.722301 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwmdv" event={"ID":"201d9cd0-dea4-4313-912b-10b5e5ba3fae","Type":"ContainerDied","Data":"5e756125ec865f9b38bee73505951cfb30aee3e2c98fa035b9e1a95a2d5d8583"} Oct 31 04:08:39.751830 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:39.751219 5019 generic.go:356] "Generic (PLEG): container finished" podID="5940a785-8b91-4dac-93a2-0f2cd52cb841" containerID="bf54958fac0677639499ea54dcae8ee395da4a4a81ad60e9c327c7e62b4c49fe" exitCode=0 Oct 31 04:08:39.751830 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:39.751376 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pv8lb" event={"ID":"5940a785-8b91-4dac-93a2-0f2cd52cb841","Type":"ContainerDied","Data":"bf54958fac0677639499ea54dcae8ee395da4a4a81ad60e9c327c7e62b4c49fe"} Oct 31 04:08:46.805547 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:46.805504 5019 generic.go:356] "Generic (PLEG): container finished" podID="d256ddad-5170-4dee-963b-98098e1ba52c" containerID="9df15baf591cb606475a57fc50535dd17a23cbd069f00c6d245cc80f41907350" exitCode=0 Oct 31 04:08:46.806003 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:46.805578 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7gx8v" event={"ID":"d256ddad-5170-4dee-963b-98098e1ba52c","Type":"ContainerDied","Data":"9df15baf591cb606475a57fc50535dd17a23cbd069f00c6d245cc80f41907350"} Oct 31 04:08:46.808609 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:46.808563 5019 generic.go:356] "Generic (PLEG): container finished" podID="6caaff37-4e5d-40f9-809d-fecd1c35cb12" containerID="9e1229fd1de81ec8ee95b94d851a9d2be85b97e5ba184e09fdc8cf3d77b26d24" exitCode=0 Oct 31 04:08:46.808758 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:46.808710 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78lr2c" event={"ID":"6caaff37-4e5d-40f9-809d-fecd1c35cb12","Type":"ContainerDied","Data":"9e1229fd1de81ec8ee95b94d851a9d2be85b97e5ba184e09fdc8cf3d77b26d24"} Oct 31 04:08:46.811517 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:46.811487 5019 generic.go:356] "Generic (PLEG): container finished" podID="5940a785-8b91-4dac-93a2-0f2cd52cb841" containerID="f26f9cd4cb9f6bf93ef537872aa545f32d719bf7ba2e94148b9a16dcca7c5e52" exitCode=0 Oct 31 04:08:46.811666 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:46.811544 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pv8lb" event={"ID":"5940a785-8b91-4dac-93a2-0f2cd52cb841","Type":"ContainerDied","Data":"f26f9cd4cb9f6bf93ef537872aa545f32d719bf7ba2e94148b9a16dcca7c5e52"} Oct 31 04:08:47.823063 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:47.823025 5019 generic.go:356] "Generic (PLEG): container finished" podID="d193edf4-187f-41c7-b7c9-2084a9dac2ef" containerID="515ac49454b21ceb892644fd8b8f8beb3997a42fce3d0f14fffd156aa4304974" exitCode=0 Oct 31 04:08:47.823578 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:47.823107 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67hxl7" event={"ID":"d193edf4-187f-41c7-b7c9-2084a9dac2ef","Type":"ContainerDied","Data":"515ac49454b21ceb892644fd8b8f8beb3997a42fce3d0f14fffd156aa4304974"} Oct 31 04:08:47.825252 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:47.825227 5019 generic.go:356] "Generic (PLEG): container finished" podID="201d9cd0-dea4-4313-912b-10b5e5ba3fae" containerID="0074ccb28f3a6af810ae3cdf51d22733bbabb69009af0ccb5b370be9e0645b3a" exitCode=0 Oct 31 04:08:47.825381 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:47.825319 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwmdv" event={"ID":"201d9cd0-dea4-4313-912b-10b5e5ba3fae","Type":"ContainerDied","Data":"0074ccb28f3a6af810ae3cdf51d22733bbabb69009af0ccb5b370be9e0645b3a"} Oct 31 04:08:47.962018 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:47.961990 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7gx8v" Oct 31 04:08:47.982074 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:47.982046 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78lr2c" Oct 31 04:08:48.012719 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:48.012690 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pv8lb" Oct 31 04:08:48.077895 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:48.077821 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d256ddad-5170-4dee-963b-98098e1ba52c-kube-api-access-dlxpc\" (UniqueName: \"kubernetes.io/projected/d256ddad-5170-4dee-963b-98098e1ba52c-kube-api-access-dlxpc\") pod \"d256ddad-5170-4dee-963b-98098e1ba52c\" (UID: \"d256ddad-5170-4dee-963b-98098e1ba52c\") " Oct 31 04:08:48.078041 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:48.077926 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d256ddad-5170-4dee-963b-98098e1ba52c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d256ddad-5170-4dee-963b-98098e1ba52c-bundle\") pod \"d256ddad-5170-4dee-963b-98098e1ba52c\" (UID: \"d256ddad-5170-4dee-963b-98098e1ba52c\") " Oct 31 04:08:48.078041 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:48.077964 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d256ddad-5170-4dee-963b-98098e1ba52c-util\" (UniqueName: \"kubernetes.io/empty-dir/d256ddad-5170-4dee-963b-98098e1ba52c-util\") pod \"d256ddad-5170-4dee-963b-98098e1ba52c\" (UID: \"d256ddad-5170-4dee-963b-98098e1ba52c\") " Oct 31 04:08:48.078806 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:48.078777 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d256ddad-5170-4dee-963b-98098e1ba52c-bundle" pod "d256ddad-5170-4dee-963b-98098e1ba52c" (UID: "d256ddad-5170-4dee-963b-98098e1ba52c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:08:48.079948 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:48.079923 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d256ddad-5170-4dee-963b-98098e1ba52c-kube-api-access-dlxpc" pod "d256ddad-5170-4dee-963b-98098e1ba52c" (UID: "d256ddad-5170-4dee-963b-98098e1ba52c"). InnerVolumeSpecName "kube-api-access-dlxpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:08:48.083784 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:48.083759 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d256ddad-5170-4dee-963b-98098e1ba52c-util" pod "d256ddad-5170-4dee-963b-98098e1ba52c" (UID: "d256ddad-5170-4dee-963b-98098e1ba52c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:08:48.179283 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:48.179250 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6caaff37-4e5d-40f9-809d-fecd1c35cb12-kube-api-access-ds24g\" (UniqueName: \"kubernetes.io/projected/6caaff37-4e5d-40f9-809d-fecd1c35cb12-kube-api-access-ds24g\") pod \"6caaff37-4e5d-40f9-809d-fecd1c35cb12\" (UID: \"6caaff37-4e5d-40f9-809d-fecd1c35cb12\") " Oct 31 04:08:48.179440 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:48.179323 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6caaff37-4e5d-40f9-809d-fecd1c35cb12-util\" (UniqueName: \"kubernetes.io/empty-dir/6caaff37-4e5d-40f9-809d-fecd1c35cb12-util\") pod \"6caaff37-4e5d-40f9-809d-fecd1c35cb12\" (UID: \"6caaff37-4e5d-40f9-809d-fecd1c35cb12\") " Oct 31 04:08:48.179440 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:48.179405 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5940a785-8b91-4dac-93a2-0f2cd52cb841-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5940a785-8b91-4dac-93a2-0f2cd52cb841-bundle\") pod \"5940a785-8b91-4dac-93a2-0f2cd52cb841\" (UID: \"5940a785-8b91-4dac-93a2-0f2cd52cb841\") " Oct 31 04:08:48.179550 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:48.179444 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5940a785-8b91-4dac-93a2-0f2cd52cb841-util\" (UniqueName: \"kubernetes.io/empty-dir/5940a785-8b91-4dac-93a2-0f2cd52cb841-util\") pod \"5940a785-8b91-4dac-93a2-0f2cd52cb841\" (UID: \"5940a785-8b91-4dac-93a2-0f2cd52cb841\") " Oct 31 04:08:48.179550 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:48.179493 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5940a785-8b91-4dac-93a2-0f2cd52cb841-kube-api-access-jp9gh\" (UniqueName: \"kubernetes.io/projected/5940a785-8b91-4dac-93a2-0f2cd52cb841-kube-api-access-jp9gh\") pod \"5940a785-8b91-4dac-93a2-0f2cd52cb841\" (UID: \"5940a785-8b91-4dac-93a2-0f2cd52cb841\") " Oct 31 04:08:48.179550 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:48.179527 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6caaff37-4e5d-40f9-809d-fecd1c35cb12-bundle\" (UniqueName: \"kubernetes.io/empty-dir/6caaff37-4e5d-40f9-809d-fecd1c35cb12-bundle\") pod \"6caaff37-4e5d-40f9-809d-fecd1c35cb12\" (UID: \"6caaff37-4e5d-40f9-809d-fecd1c35cb12\") " Oct 31 04:08:48.179835 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:48.179815 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlxpc\" (UniqueName: \"kubernetes.io/projected/d256ddad-5170-4dee-963b-98098e1ba52c-kube-api-access-dlxpc\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:08:48.179910 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:48.179842 5019 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d256ddad-5170-4dee-963b-98098e1ba52c-bundle\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:08:48.179910 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:48.179857 5019 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d256ddad-5170-4dee-963b-98098e1ba52c-util\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:08:48.179910 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:48.179812 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5940a785-8b91-4dac-93a2-0f2cd52cb841-bundle" pod "5940a785-8b91-4dac-93a2-0f2cd52cb841" (UID: "5940a785-8b91-4dac-93a2-0f2cd52cb841"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:08:48.180177 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:48.180151 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6caaff37-4e5d-40f9-809d-fecd1c35cb12-bundle" pod "6caaff37-4e5d-40f9-809d-fecd1c35cb12" (UID: "6caaff37-4e5d-40f9-809d-fecd1c35cb12"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:08:48.182002 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:48.181972 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5940a785-8b91-4dac-93a2-0f2cd52cb841-kube-api-access-jp9gh" pod "5940a785-8b91-4dac-93a2-0f2cd52cb841" (UID: "5940a785-8b91-4dac-93a2-0f2cd52cb841"). InnerVolumeSpecName "kube-api-access-jp9gh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:08:48.182097 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:48.181994 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6caaff37-4e5d-40f9-809d-fecd1c35cb12-kube-api-access-ds24g" pod "6caaff37-4e5d-40f9-809d-fecd1c35cb12" (UID: "6caaff37-4e5d-40f9-809d-fecd1c35cb12"). InnerVolumeSpecName "kube-api-access-ds24g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:08:48.186122 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:48.186096 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5940a785-8b91-4dac-93a2-0f2cd52cb841-util" pod "5940a785-8b91-4dac-93a2-0f2cd52cb841" (UID: "5940a785-8b91-4dac-93a2-0f2cd52cb841"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:08:48.186924 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:48.186906 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6caaff37-4e5d-40f9-809d-fecd1c35cb12-util" pod "6caaff37-4e5d-40f9-809d-fecd1c35cb12" (UID: "6caaff37-4e5d-40f9-809d-fecd1c35cb12"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:08:48.280637 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:48.280584 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ds24g\" (UniqueName: \"kubernetes.io/projected/6caaff37-4e5d-40f9-809d-fecd1c35cb12-kube-api-access-ds24g\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:08:48.280637 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:48.280630 5019 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6caaff37-4e5d-40f9-809d-fecd1c35cb12-util\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:08:48.280637 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:48.280640 5019 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5940a785-8b91-4dac-93a2-0f2cd52cb841-bundle\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:08:48.280637 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:48.280648 5019 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5940a785-8b91-4dac-93a2-0f2cd52cb841-util\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:08:48.280637 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:48.280656 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp9gh\" (UniqueName: \"kubernetes.io/projected/5940a785-8b91-4dac-93a2-0f2cd52cb841-kube-api-access-jp9gh\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:08:48.280939 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:48.280666 5019 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6caaff37-4e5d-40f9-809d-fecd1c35cb12-bundle\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:08:48.831043 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:48.831007 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pv8lb" Oct 31 04:08:48.831043 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:48.831017 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pv8lb" event={"ID":"5940a785-8b91-4dac-93a2-0f2cd52cb841","Type":"ContainerDied","Data":"8167a7b5c651c3c7042c3f97206582866fee12bef3809d2112f243d3bf8cdef7"} Oct 31 04:08:48.831572 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:48.831070 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8167a7b5c651c3c7042c3f97206582866fee12bef3809d2112f243d3bf8cdef7" Oct 31 04:08:48.832900 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:48.832878 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7gx8v" Oct 31 04:08:48.832900 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:48.832890 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7gx8v" event={"ID":"d256ddad-5170-4dee-963b-98098e1ba52c","Type":"ContainerDied","Data":"4444b21b278727467f2336ed5971ec120599790e47132a4301423a0a7e9ee1ff"} Oct 31 04:08:48.833080 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:48.832930 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4444b21b278727467f2336ed5971ec120599790e47132a4301423a0a7e9ee1ff" Oct 31 04:08:48.834857 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:48.834840 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78lr2c" Oct 31 04:08:48.834978 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:48.834870 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78lr2c" event={"ID":"6caaff37-4e5d-40f9-809d-fecd1c35cb12","Type":"ContainerDied","Data":"e3ef97af91dbc01467c1aa961d514506d8cfa0b74a547ce7c7a87910676ca1de"} Oct 31 04:08:48.834978 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:48.834908 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3ef97af91dbc01467c1aa961d514506d8cfa0b74a547ce7c7a87910676ca1de" Oct 31 04:08:48.974490 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:48.974464 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwmdv" Oct 31 04:08:48.990271 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:48.990247 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67hxl7" Oct 31 04:08:49.087944 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:49.087863 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/201d9cd0-dea4-4313-912b-10b5e5ba3fae-kube-api-access-2pkw7\" (UniqueName: \"kubernetes.io/projected/201d9cd0-dea4-4313-912b-10b5e5ba3fae-kube-api-access-2pkw7\") pod \"201d9cd0-dea4-4313-912b-10b5e5ba3fae\" (UID: \"201d9cd0-dea4-4313-912b-10b5e5ba3fae\") " Oct 31 04:08:49.087944 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:49.087907 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/201d9cd0-dea4-4313-912b-10b5e5ba3fae-bundle\" (UniqueName: \"kubernetes.io/empty-dir/201d9cd0-dea4-4313-912b-10b5e5ba3fae-bundle\") pod \"201d9cd0-dea4-4313-912b-10b5e5ba3fae\" (UID: \"201d9cd0-dea4-4313-912b-10b5e5ba3fae\") " Oct 31 04:08:49.088131 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:49.087997 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/201d9cd0-dea4-4313-912b-10b5e5ba3fae-util\" (UniqueName: \"kubernetes.io/empty-dir/201d9cd0-dea4-4313-912b-10b5e5ba3fae-util\") pod \"201d9cd0-dea4-4313-912b-10b5e5ba3fae\" (UID: \"201d9cd0-dea4-4313-912b-10b5e5ba3fae\") " Oct 31 04:08:49.088610 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:49.088571 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/201d9cd0-dea4-4313-912b-10b5e5ba3fae-bundle" pod "201d9cd0-dea4-4313-912b-10b5e5ba3fae" (UID: "201d9cd0-dea4-4313-912b-10b5e5ba3fae"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:08:49.090016 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:49.089994 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/201d9cd0-dea4-4313-912b-10b5e5ba3fae-kube-api-access-2pkw7" pod "201d9cd0-dea4-4313-912b-10b5e5ba3fae" (UID: "201d9cd0-dea4-4313-912b-10b5e5ba3fae"). InnerVolumeSpecName "kube-api-access-2pkw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:08:49.093799 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:49.093775 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/201d9cd0-dea4-4313-912b-10b5e5ba3fae-util" pod "201d9cd0-dea4-4313-912b-10b5e5ba3fae" (UID: "201d9cd0-dea4-4313-912b-10b5e5ba3fae"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:08:49.189193 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:49.189157 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d193edf4-187f-41c7-b7c9-2084a9dac2ef-util\" (UniqueName: \"kubernetes.io/empty-dir/d193edf4-187f-41c7-b7c9-2084a9dac2ef-util\") pod \"d193edf4-187f-41c7-b7c9-2084a9dac2ef\" (UID: \"d193edf4-187f-41c7-b7c9-2084a9dac2ef\") " Oct 31 04:08:49.189359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:49.189244 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d193edf4-187f-41c7-b7c9-2084a9dac2ef-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d193edf4-187f-41c7-b7c9-2084a9dac2ef-bundle\") pod \"d193edf4-187f-41c7-b7c9-2084a9dac2ef\" (UID: \"d193edf4-187f-41c7-b7c9-2084a9dac2ef\") " Oct 31 04:08:49.189359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:49.189266 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d193edf4-187f-41c7-b7c9-2084a9dac2ef-kube-api-access-xjrpm\" (UniqueName: \"kubernetes.io/projected/d193edf4-187f-41c7-b7c9-2084a9dac2ef-kube-api-access-xjrpm\") pod \"d193edf4-187f-41c7-b7c9-2084a9dac2ef\" (UID: \"d193edf4-187f-41c7-b7c9-2084a9dac2ef\") " Oct 31 04:08:49.189489 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:49.189472 5019 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/201d9cd0-dea4-4313-912b-10b5e5ba3fae-util\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:08:49.189522 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:49.189496 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pkw7\" (UniqueName: \"kubernetes.io/projected/201d9cd0-dea4-4313-912b-10b5e5ba3fae-kube-api-access-2pkw7\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:08:49.189522 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:49.189512 5019 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/201d9cd0-dea4-4313-912b-10b5e5ba3fae-bundle\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:08:49.189803 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:49.189781 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d193edf4-187f-41c7-b7c9-2084a9dac2ef-bundle" pod "d193edf4-187f-41c7-b7c9-2084a9dac2ef" (UID: "d193edf4-187f-41c7-b7c9-2084a9dac2ef"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:08:49.191450 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:49.191423 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d193edf4-187f-41c7-b7c9-2084a9dac2ef-kube-api-access-xjrpm" pod "d193edf4-187f-41c7-b7c9-2084a9dac2ef" (UID: "d193edf4-187f-41c7-b7c9-2084a9dac2ef"). InnerVolumeSpecName "kube-api-access-xjrpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:08:49.194895 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:49.194872 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d193edf4-187f-41c7-b7c9-2084a9dac2ef-util" pod "d193edf4-187f-41c7-b7c9-2084a9dac2ef" (UID: "d193edf4-187f-41c7-b7c9-2084a9dac2ef"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:08:49.290889 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:49.290837 5019 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d193edf4-187f-41c7-b7c9-2084a9dac2ef-util\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:08:49.290889 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:49.290883 5019 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d193edf4-187f-41c7-b7c9-2084a9dac2ef-bundle\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:08:49.290889 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:49.290893 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjrpm\" (UniqueName: \"kubernetes.io/projected/d193edf4-187f-41c7-b7c9-2084a9dac2ef-kube-api-access-xjrpm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:08:49.842174 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:49.842146 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwmdv" Oct 31 04:08:49.842639 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:49.842168 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvwmdv" event={"ID":"201d9cd0-dea4-4313-912b-10b5e5ba3fae","Type":"ContainerDied","Data":"6b38752d7d65f5ecc624ebe9c9c0d2a2b09b12d4b44b72e66a9c0ee839b286c3"} Oct 31 04:08:49.842639 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:49.842217 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b38752d7d65f5ecc624ebe9c9c0d2a2b09b12d4b44b72e66a9c0ee839b286c3" Oct 31 04:08:49.844086 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:49.844063 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67hxl7" event={"ID":"d193edf4-187f-41c7-b7c9-2084a9dac2ef","Type":"ContainerDied","Data":"c23264c617c6cafdf81b3eb7e105ec565e07c416ee1021b7ab7e0f9ba9b9152d"} Oct 31 04:08:49.844239 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:49.844091 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67hxl7" Oct 31 04:08:49.844239 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:49.844102 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c23264c617c6cafdf81b3eb7e105ec565e07c416ee1021b7ab7e0f9ba9b9152d" Oct 31 04:08:51.276047 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.276005 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-2lpkb"] Oct 31 04:08:51.276445 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.276349 5019 cpu_manager.go:414] "containerMap: removing container" podUID="6caaff37-4e5d-40f9-809d-fecd1c35cb12" containerName="util" Oct 31 04:08:51.276445 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.276360 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6caaff37-4e5d-40f9-809d-fecd1c35cb12" containerName="util" Oct 31 04:08:51.276445 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.276370 5019 cpu_manager.go:414] "containerMap: removing container" podUID="201d9cd0-dea4-4313-912b-10b5e5ba3fae" containerName="pull" Oct 31 04:08:51.276445 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.276375 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="201d9cd0-dea4-4313-912b-10b5e5ba3fae" containerName="pull" Oct 31 04:08:51.276445 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.276388 5019 cpu_manager.go:414] "containerMap: removing container" podUID="5940a785-8b91-4dac-93a2-0f2cd52cb841" containerName="util" Oct 31 04:08:51.276445 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.276394 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5940a785-8b91-4dac-93a2-0f2cd52cb841" containerName="util" Oct 31 04:08:51.276445 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.276402 5019 cpu_manager.go:414] "containerMap: removing container" podUID="d193edf4-187f-41c7-b7c9-2084a9dac2ef" containerName="pull" Oct 31 04:08:51.276445 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.276406 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d193edf4-187f-41c7-b7c9-2084a9dac2ef" containerName="pull" Oct 31 04:08:51.276445 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.276413 5019 cpu_manager.go:414] "containerMap: removing container" podUID="6caaff37-4e5d-40f9-809d-fecd1c35cb12" containerName="pull" Oct 31 04:08:51.276445 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.276417 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6caaff37-4e5d-40f9-809d-fecd1c35cb12" containerName="pull" Oct 31 04:08:51.276445 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.276423 5019 cpu_manager.go:414] "containerMap: removing container" podUID="201d9cd0-dea4-4313-912b-10b5e5ba3fae" containerName="extract" Oct 31 04:08:51.276445 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.276427 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="201d9cd0-dea4-4313-912b-10b5e5ba3fae" containerName="extract" Oct 31 04:08:51.276445 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.276435 5019 cpu_manager.go:414] "containerMap: removing container" podUID="d256ddad-5170-4dee-963b-98098e1ba52c" containerName="pull" Oct 31 04:08:51.276445 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.276439 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d256ddad-5170-4dee-963b-98098e1ba52c" containerName="pull" Oct 31 04:08:51.276445 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.276448 5019 cpu_manager.go:414] "containerMap: removing container" podUID="5940a785-8b91-4dac-93a2-0f2cd52cb841" containerName="extract" Oct 31 04:08:51.276884 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.276453 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5940a785-8b91-4dac-93a2-0f2cd52cb841" containerName="extract" Oct 31 04:08:51.276884 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.276461 5019 cpu_manager.go:414] "containerMap: removing container" podUID="201d9cd0-dea4-4313-912b-10b5e5ba3fae" containerName="util" Oct 31 04:08:51.276884 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.276466 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="201d9cd0-dea4-4313-912b-10b5e5ba3fae" containerName="util" Oct 31 04:08:51.276884 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.276473 5019 cpu_manager.go:414] "containerMap: removing container" podUID="d256ddad-5170-4dee-963b-98098e1ba52c" containerName="util" Oct 31 04:08:51.276884 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.276478 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d256ddad-5170-4dee-963b-98098e1ba52c" containerName="util" Oct 31 04:08:51.276884 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.276485 5019 cpu_manager.go:414] "containerMap: removing container" podUID="d193edf4-187f-41c7-b7c9-2084a9dac2ef" containerName="util" Oct 31 04:08:51.276884 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.276489 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d193edf4-187f-41c7-b7c9-2084a9dac2ef" containerName="util" Oct 31 04:08:51.276884 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.276495 5019 cpu_manager.go:414] "containerMap: removing container" podUID="5940a785-8b91-4dac-93a2-0f2cd52cb841" containerName="pull" Oct 31 04:08:51.276884 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.276499 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5940a785-8b91-4dac-93a2-0f2cd52cb841" containerName="pull" Oct 31 04:08:51.276884 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.276506 5019 cpu_manager.go:414] "containerMap: removing container" podUID="d256ddad-5170-4dee-963b-98098e1ba52c" containerName="extract" Oct 31 04:08:51.276884 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.276510 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d256ddad-5170-4dee-963b-98098e1ba52c" containerName="extract" Oct 31 04:08:51.276884 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.276519 5019 cpu_manager.go:414] "containerMap: removing container" podUID="6caaff37-4e5d-40f9-809d-fecd1c35cb12" containerName="extract" Oct 31 04:08:51.276884 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.276523 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6caaff37-4e5d-40f9-809d-fecd1c35cb12" containerName="extract" Oct 31 04:08:51.276884 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.276528 5019 cpu_manager.go:414] "containerMap: removing container" podUID="d193edf4-187f-41c7-b7c9-2084a9dac2ef" containerName="extract" Oct 31 04:08:51.276884 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.276532 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d193edf4-187f-41c7-b7c9-2084a9dac2ef" containerName="extract" Oct 31 04:08:51.276884 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.276586 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="6caaff37-4e5d-40f9-809d-fecd1c35cb12" containerName="extract" Oct 31 04:08:51.276884 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.276594 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="d193edf4-187f-41c7-b7c9-2084a9dac2ef" containerName="extract" Oct 31 04:08:51.276884 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.276602 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="201d9cd0-dea4-4313-912b-10b5e5ba3fae" containerName="extract" Oct 31 04:08:51.276884 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.276609 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="5940a785-8b91-4dac-93a2-0f2cd52cb841" containerName="extract" Oct 31 04:08:51.276884 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.276615 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="d256ddad-5170-4dee-963b-98098e1ba52c" containerName="extract" Oct 31 04:08:51.280935 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.280914 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-2lpkb" Oct 31 04:08:51.303857 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.303819 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vtwm\" (UniqueName: \"kubernetes.io/projected/0181e356-3068-47a2-abe8-96cef1740e6b-kube-api-access-5vtwm\") pod \"opendatahub-operator-controller-manager-79b6bb9d79-2lpkb\" (UID: \"0181e356-3068-47a2-abe8-96cef1740e6b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-2lpkb" Oct 31 04:08:51.303857 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.303862 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0181e356-3068-47a2-abe8-96cef1740e6b-metrics-cert\") pod \"opendatahub-operator-controller-manager-79b6bb9d79-2lpkb\" (UID: \"0181e356-3068-47a2-abe8-96cef1740e6b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-2lpkb" Oct 31 04:08:51.304079 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.303881 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0181e356-3068-47a2-abe8-96cef1740e6b-tmp\") pod \"opendatahub-operator-controller-manager-79b6bb9d79-2lpkb\" (UID: \"0181e356-3068-47a2-abe8-96cef1740e6b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-2lpkb" Oct 31 04:08:51.304079 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.303996 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0181e356-3068-47a2-abe8-96cef1740e6b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-79b6bb9d79-2lpkb\" (UID: \"0181e356-3068-47a2-abe8-96cef1740e6b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-2lpkb" Oct 31 04:08:51.304079 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.304060 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0181e356-3068-47a2-abe8-96cef1740e6b-webhook-cert\") pod \"opendatahub-operator-controller-manager-79b6bb9d79-2lpkb\" (UID: \"0181e356-3068-47a2-abe8-96cef1740e6b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-2lpkb" Oct 31 04:08:51.304196 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.304093 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0181e356-3068-47a2-abe8-96cef1740e6b-manifests\") pod \"opendatahub-operator-controller-manager-79b6bb9d79-2lpkb\" (UID: \"0181e356-3068-47a2-abe8-96cef1740e6b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-2lpkb" Oct 31 04:08:51.322468 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.322427 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 31 04:08:51.322661 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.322636 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 31 04:08:51.322661 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.322651 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 31 04:08:51.322780 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.322674 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 31 04:08:51.322780 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.322686 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-cbq7v\"" Oct 31 04:08:51.322869 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.322858 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 31 04:08:51.324615 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.324591 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-2lpkb"] Oct 31 04:08:51.404998 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.404965 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0181e356-3068-47a2-abe8-96cef1740e6b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-79b6bb9d79-2lpkb\" (UID: \"0181e356-3068-47a2-abe8-96cef1740e6b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-2lpkb" Oct 31 04:08:51.405177 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.405022 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0181e356-3068-47a2-abe8-96cef1740e6b-webhook-cert\") pod \"opendatahub-operator-controller-manager-79b6bb9d79-2lpkb\" (UID: \"0181e356-3068-47a2-abe8-96cef1740e6b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-2lpkb" Oct 31 04:08:51.405177 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.405059 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0181e356-3068-47a2-abe8-96cef1740e6b-manifests\") pod \"opendatahub-operator-controller-manager-79b6bb9d79-2lpkb\" (UID: \"0181e356-3068-47a2-abe8-96cef1740e6b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-2lpkb" Oct 31 04:08:51.405177 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.405118 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vtwm\" (UniqueName: \"kubernetes.io/projected/0181e356-3068-47a2-abe8-96cef1740e6b-kube-api-access-5vtwm\") pod \"opendatahub-operator-controller-manager-79b6bb9d79-2lpkb\" (UID: \"0181e356-3068-47a2-abe8-96cef1740e6b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-2lpkb" Oct 31 04:08:51.405177 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.405147 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0181e356-3068-47a2-abe8-96cef1740e6b-metrics-cert\") pod \"opendatahub-operator-controller-manager-79b6bb9d79-2lpkb\" (UID: \"0181e356-3068-47a2-abe8-96cef1740e6b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-2lpkb" Oct 31 04:08:51.405177 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.405172 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0181e356-3068-47a2-abe8-96cef1740e6b-tmp\") pod \"opendatahub-operator-controller-manager-79b6bb9d79-2lpkb\" (UID: \"0181e356-3068-47a2-abe8-96cef1740e6b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-2lpkb" Oct 31 04:08:51.405638 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.405603 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0181e356-3068-47a2-abe8-96cef1740e6b-manifests\") pod \"opendatahub-operator-controller-manager-79b6bb9d79-2lpkb\" (UID: \"0181e356-3068-47a2-abe8-96cef1740e6b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-2lpkb" Oct 31 04:08:51.407751 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.407721 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0181e356-3068-47a2-abe8-96cef1740e6b-webhook-cert\") pod \"opendatahub-operator-controller-manager-79b6bb9d79-2lpkb\" (UID: \"0181e356-3068-47a2-abe8-96cef1740e6b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-2lpkb" Oct 31 04:08:51.407854 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.407770 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0181e356-3068-47a2-abe8-96cef1740e6b-metrics-cert\") pod \"opendatahub-operator-controller-manager-79b6bb9d79-2lpkb\" (UID: \"0181e356-3068-47a2-abe8-96cef1740e6b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-2lpkb" Oct 31 04:08:51.407946 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.407924 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0181e356-3068-47a2-abe8-96cef1740e6b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-79b6bb9d79-2lpkb\" (UID: \"0181e356-3068-47a2-abe8-96cef1740e6b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-2lpkb" Oct 31 04:08:51.408547 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.408526 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0181e356-3068-47a2-abe8-96cef1740e6b-tmp\") pod \"opendatahub-operator-controller-manager-79b6bb9d79-2lpkb\" (UID: \"0181e356-3068-47a2-abe8-96cef1740e6b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-2lpkb" Oct 31 04:08:51.434343 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.434293 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vtwm\" (UniqueName: \"kubernetes.io/projected/0181e356-3068-47a2-abe8-96cef1740e6b-kube-api-access-5vtwm\") pod \"opendatahub-operator-controller-manager-79b6bb9d79-2lpkb\" (UID: \"0181e356-3068-47a2-abe8-96cef1740e6b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-2lpkb" Oct 31 04:08:51.603183 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.603084 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-2lpkb" Oct 31 04:08:51.958162 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:51.958077 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-2lpkb"] Oct 31 04:08:51.961315 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:08:51.961270 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0181e356_3068_47a2_abe8_96cef1740e6b.slice/crio-54485bfd334bc07786604dc16d1d0b128b563b26f34dd4d94aa3fac429a1921c WatchSource:0}: Error finding container 54485bfd334bc07786604dc16d1d0b128b563b26f34dd4d94aa3fac429a1921c: Status 404 returned error can't find the container with id 54485bfd334bc07786604dc16d1d0b128b563b26f34dd4d94aa3fac429a1921c Oct 31 04:08:52.391744 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:08:52.391704 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:08:52.862373 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:52.862318 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-2lpkb" event={"ID":"0181e356-3068-47a2-abe8-96cef1740e6b","Type":"ContainerStarted","Data":"54485bfd334bc07786604dc16d1d0b128b563b26f34dd4d94aa3fac429a1921c"} Oct 31 04:08:55.880876 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:55.880772 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-2lpkb" event={"ID":"0181e356-3068-47a2-abe8-96cef1740e6b","Type":"ContainerStarted","Data":"66a2e84f4e8ea02aef00defcc1d7352372cf9f647e385de4502dc265d8a18440"} Oct 31 04:08:56.887203 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:56.887166 5019 generic.go:356] "Generic (PLEG): container finished" podID="0181e356-3068-47a2-abe8-96cef1740e6b" containerID="66a2e84f4e8ea02aef00defcc1d7352372cf9f647e385de4502dc265d8a18440" exitCode=0 Oct 31 04:08:56.887673 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:56.887246 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-2lpkb" event={"ID":"0181e356-3068-47a2-abe8-96cef1740e6b","Type":"ContainerDied","Data":"66a2e84f4e8ea02aef00defcc1d7352372cf9f647e385de4502dc265d8a18440"} Oct 31 04:08:57.894478 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:57.894436 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-2lpkb" event={"ID":"0181e356-3068-47a2-abe8-96cef1740e6b","Type":"ContainerStarted","Data":"5e7568250f49fe7e1182395702bbbb1806bc48255551dfec713116af279d39e5"} Oct 31 04:08:57.894987 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:57.894619 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-2lpkb" Oct 31 04:08:57.996863 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:08:57.996801 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-2lpkb" podStartSLOduration=2.660392221 podStartE2EDuration="6.99678323s" totalImagesPullingTime="3.856888074s" totalInitContainerRuntime="479.502935ms" isStatefulPod=false podCreationTimestamp="2026-10-31 04:08:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:08:57.983074872 +0000 UTC m=+292.739412931" watchObservedRunningTime="2026-10-31 04:08:57.99678323 +0000 UTC m=+292.753121267" Oct 31 04:09:02.765840 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:02.765796 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-gsxbf"] Oct 31 04:09:02.774719 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:02.774680 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-gsxbf" Oct 31 04:09:02.790496 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:02.790461 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-gsxbf"] Oct 31 04:09:02.800483 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:02.800446 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 31 04:09:02.800691 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:02.800585 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 31 04:09:02.800816 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:02.800802 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-dgmtq\"" Oct 31 04:09:02.812303 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:02.812262 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t7vs\" (UniqueName: \"kubernetes.io/projected/176b5a4d-06b5-484e-b6ba-e5727c90afba-kube-api-access-5t7vs\") pod \"limitador-operator-controller-manager-5778bb4449-gsxbf\" (UID: \"176b5a4d-06b5-484e-b6ba-e5727c90afba\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-gsxbf" Oct 31 04:09:02.913705 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:02.913663 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t7vs\" (UniqueName: \"kubernetes.io/projected/176b5a4d-06b5-484e-b6ba-e5727c90afba-kube-api-access-5t7vs\") pod \"limitador-operator-controller-manager-5778bb4449-gsxbf\" (UID: \"176b5a4d-06b5-484e-b6ba-e5727c90afba\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-gsxbf" Oct 31 04:09:02.943208 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:02.943159 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t7vs\" (UniqueName: \"kubernetes.io/projected/176b5a4d-06b5-484e-b6ba-e5727c90afba-kube-api-access-5t7vs\") pod \"limitador-operator-controller-manager-5778bb4449-gsxbf\" (UID: \"176b5a4d-06b5-484e-b6ba-e5727c90afba\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-gsxbf" Oct 31 04:09:03.088021 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:03.087916 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-gsxbf" Oct 31 04:09:03.382845 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:03.382813 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-gsxbf"] Oct 31 04:09:03.385501 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:09:03.385450 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod176b5a4d_06b5_484e_b6ba_e5727c90afba.slice/crio-a49bbc8cf3f804fcdbf2dc41f72878ce4f30f236b835dae96c84d7b329a3fe71 WatchSource:0}: Error finding container a49bbc8cf3f804fcdbf2dc41f72878ce4f30f236b835dae96c84d7b329a3fe71: Status 404 returned error can't find the container with id a49bbc8cf3f804fcdbf2dc41f72878ce4f30f236b835dae96c84d7b329a3fe71 Oct 31 04:09:03.923514 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:03.923476 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-gsxbf" event={"ID":"176b5a4d-06b5-484e-b6ba-e5727c90afba","Type":"ContainerStarted","Data":"a49bbc8cf3f804fcdbf2dc41f72878ce4f30f236b835dae96c84d7b329a3fe71"} Oct 31 04:09:05.436098 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:05.436070 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s88kp_3945ff65-e877-4372-9334-3968c68b7295/console-operator/2.log" Oct 31 04:09:05.436597 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:05.436439 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s88kp_3945ff65-e877-4372-9334-3968c68b7295/console-operator/2.log" Oct 31 04:09:06.941257 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:06.941201 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-gsxbf" event={"ID":"176b5a4d-06b5-484e-b6ba-e5727c90afba","Type":"ContainerStarted","Data":"d3c9a099dccc715e44fa9c658e34dfdb6cd8b00cfeeaf4797034c221d89692c9"} Oct 31 04:09:06.941803 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:06.941291 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-gsxbf" Oct 31 04:09:06.987979 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:06.987925 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-gsxbf" podStartSLOduration=2.106510057 podStartE2EDuration="4.98790902s" totalImagesPullingTime="2.881398963s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:09:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:09:06.98724868 +0000 UTC m=+301.743586732" watchObservedRunningTime="2026-10-31 04:09:06.98790902 +0000 UTC m=+301.744247060" Oct 31 04:09:08.902596 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:08.902563 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-2lpkb" Oct 31 04:09:10.784812 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:10.784773 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-9qv62"] Oct 31 04:09:10.788279 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:10.788258 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-9qv62" Oct 31 04:09:10.809227 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:10.809192 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-9qv62"] Oct 31 04:09:10.819727 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:10.819690 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-5jq5s\"" Oct 31 04:09:10.878883 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:10.878834 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lccqb\" (UniqueName: \"kubernetes.io/projected/7c48eb82-caa2-464d-9c1b-fdca4fe4b38e-kube-api-access-lccqb\") pod \"kuadrant-operator-controller-manager-cf8469d7b-9qv62\" (UID: \"7c48eb82-caa2-464d-9c1b-fdca4fe4b38e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-9qv62" Oct 31 04:09:10.878883 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:10.878884 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7c48eb82-caa2-464d-9c1b-fdca4fe4b38e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-9qv62\" (UID: \"7c48eb82-caa2-464d-9c1b-fdca4fe4b38e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-9qv62" Oct 31 04:09:10.980268 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:10.980226 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lccqb\" (UniqueName: \"kubernetes.io/projected/7c48eb82-caa2-464d-9c1b-fdca4fe4b38e-kube-api-access-lccqb\") pod \"kuadrant-operator-controller-manager-cf8469d7b-9qv62\" (UID: \"7c48eb82-caa2-464d-9c1b-fdca4fe4b38e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-9qv62" Oct 31 04:09:10.980268 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:10.980271 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7c48eb82-caa2-464d-9c1b-fdca4fe4b38e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-9qv62\" (UID: \"7c48eb82-caa2-464d-9c1b-fdca4fe4b38e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-9qv62" Oct 31 04:09:10.980660 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:10.980643 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7c48eb82-caa2-464d-9c1b-fdca4fe4b38e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-9qv62\" (UID: \"7c48eb82-caa2-464d-9c1b-fdca4fe4b38e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-9qv62" Oct 31 04:09:11.006789 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:11.006754 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lccqb\" (UniqueName: \"kubernetes.io/projected/7c48eb82-caa2-464d-9c1b-fdca4fe4b38e-kube-api-access-lccqb\") pod \"kuadrant-operator-controller-manager-cf8469d7b-9qv62\" (UID: \"7c48eb82-caa2-464d-9c1b-fdca4fe4b38e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-9qv62" Oct 31 04:09:11.101942 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:11.101835 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-9qv62" Oct 31 04:09:11.420413 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:11.420385 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-9qv62"] Oct 31 04:09:11.422602 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:09:11.422571 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c48eb82_caa2_464d_9c1b_fdca4fe4b38e.slice/crio-364d53bdeb608a885918a6924b91e3a54399d75ba3c1e0df389a392a41f3c6f3 WatchSource:0}: Error finding container 364d53bdeb608a885918a6924b91e3a54399d75ba3c1e0df389a392a41f3c6f3: Status 404 returned error can't find the container with id 364d53bdeb608a885918a6924b91e3a54399d75ba3c1e0df389a392a41f3c6f3 Oct 31 04:09:11.425137 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:11.425117 5019 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 04:09:11.965413 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:11.965366 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-9qv62" event={"ID":"7c48eb82-caa2-464d-9c1b-fdca4fe4b38e","Type":"ContainerStarted","Data":"364d53bdeb608a885918a6924b91e3a54399d75ba3c1e0df389a392a41f3c6f3"} Oct 31 04:09:14.715380 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:14.715312 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-ggcnf"] Oct 31 04:09:14.719156 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:14.719126 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ggcnf" Oct 31 04:09:14.747848 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:14.747802 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-2bl9s\"" Oct 31 04:09:14.748041 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:14.747988 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 31 04:09:14.750098 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:14.750064 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-ggcnf"] Oct 31 04:09:14.813707 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:14.813664 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6mg4\" (UniqueName: \"kubernetes.io/projected/cd96ff5f-fb42-4491-aae0-bcdd6e4002aa-kube-api-access-b6mg4\") pod \"dns-operator-controller-manager-8557cbdb84-ggcnf\" (UID: \"cd96ff5f-fb42-4491-aae0-bcdd6e4002aa\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ggcnf" Oct 31 04:09:14.914924 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:14.914877 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6mg4\" (UniqueName: \"kubernetes.io/projected/cd96ff5f-fb42-4491-aae0-bcdd6e4002aa-kube-api-access-b6mg4\") pod \"dns-operator-controller-manager-8557cbdb84-ggcnf\" (UID: \"cd96ff5f-fb42-4491-aae0-bcdd6e4002aa\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ggcnf" Oct 31 04:09:14.940093 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:14.940055 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6mg4\" (UniqueName: \"kubernetes.io/projected/cd96ff5f-fb42-4491-aae0-bcdd6e4002aa-kube-api-access-b6mg4\") pod \"dns-operator-controller-manager-8557cbdb84-ggcnf\" (UID: \"cd96ff5f-fb42-4491-aae0-bcdd6e4002aa\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ggcnf" Oct 31 04:09:15.035141 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:15.035097 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ggcnf" Oct 31 04:09:15.342540 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:15.342505 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-ggcnf"] Oct 31 04:09:15.345235 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:09:15.345201 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd96ff5f_fb42_4491_aae0_bcdd6e4002aa.slice/crio-2be0f9f63fd8205a9a1f33173b916855ca15cffaa176320d81e10ea419b7c31d WatchSource:0}: Error finding container 2be0f9f63fd8205a9a1f33173b916855ca15cffaa176320d81e10ea419b7c31d: Status 404 returned error can't find the container with id 2be0f9f63fd8205a9a1f33173b916855ca15cffaa176320d81e10ea419b7c31d Oct 31 04:09:15.990326 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:15.990281 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ggcnf" event={"ID":"cd96ff5f-fb42-4491-aae0-bcdd6e4002aa","Type":"ContainerStarted","Data":"2be0f9f63fd8205a9a1f33173b916855ca15cffaa176320d81e10ea419b7c31d"} Oct 31 04:09:15.992233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:15.992200 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-9qv62" event={"ID":"7c48eb82-caa2-464d-9c1b-fdca4fe4b38e","Type":"ContainerStarted","Data":"a75f1d40247d790dc255cc6c7d10968925ea33ff0bf5b198235c168a9a882a8a"} Oct 31 04:09:15.992408 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:15.992284 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-9qv62" Oct 31 04:09:16.063507 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:16.063434 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-9qv62" podStartSLOduration=1.8298784609999998 podStartE2EDuration="6.063416846s" totalImagesPullingTime="4.233538385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:09:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:09:16.050656658 +0000 UTC m=+310.806994698" watchObservedRunningTime="2026-10-31 04:09:16.063416846 +0000 UTC m=+310.819754887" Oct 31 04:09:17.949829 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:17.948857 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-gsxbf" Oct 31 04:09:18.005899 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:18.005864 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ggcnf" event={"ID":"cd96ff5f-fb42-4491-aae0-bcdd6e4002aa","Type":"ContainerStarted","Data":"fa6229f2474cd75f8626c6494b9e7b6874b7918b2afa5dd630d8a2e40c7d359f"} Oct 31 04:09:18.006089 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:18.005983 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ggcnf" Oct 31 04:09:18.049289 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:18.049219 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ggcnf" podStartSLOduration=1.681389715 podStartE2EDuration="4.049195974s" totalImagesPullingTime="2.367806259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:09:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:09:18.046841623 +0000 UTC m=+312.803179684" watchObservedRunningTime="2026-10-31 04:09:18.049195974 +0000 UTC m=+312.805534016" Oct 31 04:09:18.676871 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:18.676832 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-r22dt"] Oct 31 04:09:18.680654 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:18.680623 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-r22dt" Oct 31 04:09:18.702267 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:18.702230 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-r22dt"] Oct 31 04:09:18.719726 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:18.719689 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-mhls6\"" Oct 31 04:09:18.749236 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:18.749195 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqg5n\" (UniqueName: \"kubernetes.io/projected/48e6e3f6-9d22-42a3-bc22-a27e516aeda1-kube-api-access-vqg5n\") pod \"authorino-operator-7b7bb4bc77-r22dt\" (UID: \"48e6e3f6-9d22-42a3-bc22-a27e516aeda1\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-r22dt" Oct 31 04:09:18.850803 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:18.850740 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqg5n\" (UniqueName: \"kubernetes.io/projected/48e6e3f6-9d22-42a3-bc22-a27e516aeda1-kube-api-access-vqg5n\") pod \"authorino-operator-7b7bb4bc77-r22dt\" (UID: \"48e6e3f6-9d22-42a3-bc22-a27e516aeda1\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-r22dt" Oct 31 04:09:18.879527 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:18.879495 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqg5n\" (UniqueName: \"kubernetes.io/projected/48e6e3f6-9d22-42a3-bc22-a27e516aeda1-kube-api-access-vqg5n\") pod \"authorino-operator-7b7bb4bc77-r22dt\" (UID: \"48e6e3f6-9d22-42a3-bc22-a27e516aeda1\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-r22dt" Oct 31 04:09:18.995199 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:18.995161 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-r22dt" Oct 31 04:09:19.299718 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:19.299679 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-r22dt"] Oct 31 04:09:19.302450 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:09:19.302415 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48e6e3f6_9d22_42a3_bc22_a27e516aeda1.slice/crio-0d4be91a83ea1b9edf3fb320c5c42f2a78fcd3daba15dccfdfd38fedca02cb79 WatchSource:0}: Error finding container 0d4be91a83ea1b9edf3fb320c5c42f2a78fcd3daba15dccfdfd38fedca02cb79: Status 404 returned error can't find the container with id 0d4be91a83ea1b9edf3fb320c5c42f2a78fcd3daba15dccfdfd38fedca02cb79 Oct 31 04:09:19.707251 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:19.707156 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898flnvv2"] Oct 31 04:09:19.712187 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:19.712151 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898flnvv2" Oct 31 04:09:19.737856 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:19.737770 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898flnvv2"] Oct 31 04:09:19.747253 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:19.747218 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-6l5r7\"" Oct 31 04:09:19.760635 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:19.760593 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e142c20-377f-4aa7-a507-780e8efc906d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898flnvv2\" (UID: \"1e142c20-377f-4aa7-a507-780e8efc906d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898flnvv2" Oct 31 04:09:19.760635 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:19.760639 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e142c20-377f-4aa7-a507-780e8efc906d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898flnvv2\" (UID: \"1e142c20-377f-4aa7-a507-780e8efc906d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898flnvv2" Oct 31 04:09:19.760858 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:19.760674 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e142c20-377f-4aa7-a507-780e8efc906d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898flnvv2\" (UID: \"1e142c20-377f-4aa7-a507-780e8efc906d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898flnvv2" Oct 31 04:09:19.760858 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:19.760821 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e142c20-377f-4aa7-a507-780e8efc906d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898flnvv2\" (UID: \"1e142c20-377f-4aa7-a507-780e8efc906d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898flnvv2" Oct 31 04:09:19.760943 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:19.760875 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krptz\" (UniqueName: \"kubernetes.io/projected/1e142c20-377f-4aa7-a507-780e8efc906d-kube-api-access-krptz\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898flnvv2\" (UID: \"1e142c20-377f-4aa7-a507-780e8efc906d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898flnvv2" Oct 31 04:09:19.760943 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:19.760911 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e142c20-377f-4aa7-a507-780e8efc906d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898flnvv2\" (UID: \"1e142c20-377f-4aa7-a507-780e8efc906d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898flnvv2" Oct 31 04:09:19.761046 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:19.760945 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e142c20-377f-4aa7-a507-780e8efc906d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898flnvv2\" (UID: \"1e142c20-377f-4aa7-a507-780e8efc906d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898flnvv2" Oct 31 04:09:19.761046 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:19.761018 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e142c20-377f-4aa7-a507-780e8efc906d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898flnvv2\" (UID: \"1e142c20-377f-4aa7-a507-780e8efc906d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898flnvv2" Oct 31 04:09:19.761223 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:19.761080 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e142c20-377f-4aa7-a507-780e8efc906d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898flnvv2\" (UID: \"1e142c20-377f-4aa7-a507-780e8efc906d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898flnvv2" Oct 31 04:09:19.861793 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:19.861744 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e142c20-377f-4aa7-a507-780e8efc906d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898flnvv2\" (UID: \"1e142c20-377f-4aa7-a507-780e8efc906d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898flnvv2" Oct 31 04:09:19.861793 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:19.861791 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krptz\" (UniqueName: \"kubernetes.io/projected/1e142c20-377f-4aa7-a507-780e8efc906d-kube-api-access-krptz\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898flnvv2\" (UID: \"1e142c20-377f-4aa7-a507-780e8efc906d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898flnvv2" Oct 31 04:09:19.862144 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:19.861811 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e142c20-377f-4aa7-a507-780e8efc906d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898flnvv2\" (UID: \"1e142c20-377f-4aa7-a507-780e8efc906d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898flnvv2" Oct 31 04:09:19.862144 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:19.861833 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e142c20-377f-4aa7-a507-780e8efc906d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898flnvv2\" (UID: \"1e142c20-377f-4aa7-a507-780e8efc906d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898flnvv2" Oct 31 04:09:19.862144 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:19.861852 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e142c20-377f-4aa7-a507-780e8efc906d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898flnvv2\" (UID: \"1e142c20-377f-4aa7-a507-780e8efc906d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898flnvv2" Oct 31 04:09:19.862144 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:19.861881 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e142c20-377f-4aa7-a507-780e8efc906d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898flnvv2\" (UID: \"1e142c20-377f-4aa7-a507-780e8efc906d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898flnvv2" Oct 31 04:09:19.862144 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:19.861930 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e142c20-377f-4aa7-a507-780e8efc906d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898flnvv2\" (UID: \"1e142c20-377f-4aa7-a507-780e8efc906d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898flnvv2" Oct 31 04:09:19.862144 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:19.861956 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e142c20-377f-4aa7-a507-780e8efc906d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898flnvv2\" (UID: \"1e142c20-377f-4aa7-a507-780e8efc906d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898flnvv2" Oct 31 04:09:19.862144 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:19.861989 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e142c20-377f-4aa7-a507-780e8efc906d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898flnvv2\" (UID: \"1e142c20-377f-4aa7-a507-780e8efc906d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898flnvv2" Oct 31 04:09:19.862482 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:19.862368 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e142c20-377f-4aa7-a507-780e8efc906d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898flnvv2\" (UID: \"1e142c20-377f-4aa7-a507-780e8efc906d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898flnvv2" Oct 31 04:09:19.862885 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:19.862548 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e142c20-377f-4aa7-a507-780e8efc906d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898flnvv2\" (UID: \"1e142c20-377f-4aa7-a507-780e8efc906d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898flnvv2" Oct 31 04:09:19.862885 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:19.862869 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e142c20-377f-4aa7-a507-780e8efc906d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898flnvv2\" (UID: \"1e142c20-377f-4aa7-a507-780e8efc906d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898flnvv2" Oct 31 04:09:19.863238 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:19.863209 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e142c20-377f-4aa7-a507-780e8efc906d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898flnvv2\" (UID: \"1e142c20-377f-4aa7-a507-780e8efc906d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898flnvv2" Oct 31 04:09:19.863690 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:19.863662 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e142c20-377f-4aa7-a507-780e8efc906d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898flnvv2\" (UID: \"1e142c20-377f-4aa7-a507-780e8efc906d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898flnvv2" Oct 31 04:09:19.871214 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:19.871179 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e142c20-377f-4aa7-a507-780e8efc906d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898flnvv2\" (UID: \"1e142c20-377f-4aa7-a507-780e8efc906d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898flnvv2" Oct 31 04:09:19.871786 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:19.871763 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e142c20-377f-4aa7-a507-780e8efc906d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898flnvv2\" (UID: \"1e142c20-377f-4aa7-a507-780e8efc906d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898flnvv2" Oct 31 04:09:19.889321 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:19.889285 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krptz\" (UniqueName: \"kubernetes.io/projected/1e142c20-377f-4aa7-a507-780e8efc906d-kube-api-access-krptz\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898flnvv2\" (UID: \"1e142c20-377f-4aa7-a507-780e8efc906d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898flnvv2" Oct 31 04:09:19.889695 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:19.889638 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e142c20-377f-4aa7-a507-780e8efc906d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898flnvv2\" (UID: \"1e142c20-377f-4aa7-a507-780e8efc906d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898flnvv2" Oct 31 04:09:20.021740 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:20.021696 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-r22dt" event={"ID":"48e6e3f6-9d22-42a3-bc22-a27e516aeda1","Type":"ContainerStarted","Data":"0d4be91a83ea1b9edf3fb320c5c42f2a78fcd3daba15dccfdfd38fedca02cb79"} Oct 31 04:09:20.028007 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:20.027970 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898flnvv2" Oct 31 04:09:20.357016 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:20.356988 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898flnvv2"] Oct 31 04:09:20.359182 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:09:20.359153 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e142c20_377f_4aa7_a507_780e8efc906d.slice/crio-e6d67423c987236327ecf7fde9019198c356e473b46489499f78b6c74ae04db7 WatchSource:0}: Error finding container e6d67423c987236327ecf7fde9019198c356e473b46489499f78b6c74ae04db7: Status 404 returned error can't find the container with id e6d67423c987236327ecf7fde9019198c356e473b46489499f78b6c74ae04db7 Oct 31 04:09:20.361461 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:20.361413 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:09:20.361528 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:20.361504 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:09:20.361563 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:20.361535 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:09:21.029863 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:21.029823 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898flnvv2" event={"ID":"1e142c20-377f-4aa7-a507-780e8efc906d","Type":"ContainerStarted","Data":"9031640811e8c38fc5e53a8c57dc4639d33e1f6258118d3baceb48c9075d6a99"} Oct 31 04:09:21.030263 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:21.029892 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898flnvv2" event={"ID":"1e142c20-377f-4aa7-a507-780e8efc906d","Type":"ContainerStarted","Data":"e6d67423c987236327ecf7fde9019198c356e473b46489499f78b6c74ae04db7"} Oct 31 04:09:21.077988 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:21.077924 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898flnvv2" podStartSLOduration=2.077908721 podStartE2EDuration="2.077908721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:09:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:09:21.076605063 +0000 UTC m=+315.832943093" watchObservedRunningTime="2026-10-31 04:09:21.077908721 +0000 UTC m=+315.834246760" Oct 31 04:09:22.029035 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:22.028986 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898flnvv2" Oct 31 04:09:22.034613 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:22.034582 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898flnvv2" Oct 31 04:09:22.036775 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:22.036741 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-r22dt" event={"ID":"48e6e3f6-9d22-42a3-bc22-a27e516aeda1","Type":"ContainerStarted","Data":"0dd701ae6284ab5e8ed673bf16f74a84db0ee4a76c88fc41934d32864b0a11d9"} Oct 31 04:09:22.036934 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:22.036838 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-r22dt" Oct 31 04:09:22.037179 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:22.037163 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898flnvv2" Oct 31 04:09:22.038186 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:22.038165 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898flnvv2" Oct 31 04:09:22.148054 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:22.148000 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-r22dt" podStartSLOduration=2.351256298 podStartE2EDuration="4.147984933s" totalImagesPullingTime="1.796728635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:09:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:09:22.146062113 +0000 UTC m=+316.902400158" watchObservedRunningTime="2026-10-31 04:09:22.147984933 +0000 UTC m=+316.904322973" Oct 31 04:09:22.391142 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:09:22.391052 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:09:27.001015 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:27.000977 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-9qv62" Oct 31 04:09:27.491708 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:27.491632 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vwcwp"] Oct 31 04:09:27.498995 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:27.498960 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vwcwp" Oct 31 04:09:27.532187 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:27.532142 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vwcwp"] Oct 31 04:09:27.634379 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:27.634325 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1f9f7baf-a850-453c-8e16-849942ba754a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-vwcwp\" (UID: \"1f9f7baf-a850-453c-8e16-849942ba754a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vwcwp" Oct 31 04:09:27.634571 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:27.634478 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wffw4\" (UniqueName: \"kubernetes.io/projected/1f9f7baf-a850-453c-8e16-849942ba754a-kube-api-access-wffw4\") pod \"kuadrant-operator-controller-manager-678d87766f-vwcwp\" (UID: \"1f9f7baf-a850-453c-8e16-849942ba754a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vwcwp" Oct 31 04:09:27.735100 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:27.735052 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wffw4\" (UniqueName: \"kubernetes.io/projected/1f9f7baf-a850-453c-8e16-849942ba754a-kube-api-access-wffw4\") pod \"kuadrant-operator-controller-manager-678d87766f-vwcwp\" (UID: \"1f9f7baf-a850-453c-8e16-849942ba754a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vwcwp" Oct 31 04:09:27.735292 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:27.735112 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1f9f7baf-a850-453c-8e16-849942ba754a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-vwcwp\" (UID: \"1f9f7baf-a850-453c-8e16-849942ba754a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vwcwp" Oct 31 04:09:27.735509 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:27.735490 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1f9f7baf-a850-453c-8e16-849942ba754a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-vwcwp\" (UID: \"1f9f7baf-a850-453c-8e16-849942ba754a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vwcwp" Oct 31 04:09:27.762781 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:27.762698 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wffw4\" (UniqueName: \"kubernetes.io/projected/1f9f7baf-a850-453c-8e16-849942ba754a-kube-api-access-wffw4\") pod \"kuadrant-operator-controller-manager-678d87766f-vwcwp\" (UID: \"1f9f7baf-a850-453c-8e16-849942ba754a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vwcwp" Oct 31 04:09:27.813154 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:27.813113 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vwcwp" Oct 31 04:09:28.161420 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:28.159718 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vwcwp"] Oct 31 04:09:28.164261 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:09:28.164225 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f9f7baf_a850_453c_8e16_849942ba754a.slice/crio-a67cae05638d07ae6cdd048cb13748c9a85745fd483e414ab659d354bfe2805d WatchSource:0}: Error finding container a67cae05638d07ae6cdd048cb13748c9a85745fd483e414ab659d354bfe2805d: Status 404 returned error can't find the container with id a67cae05638d07ae6cdd048cb13748c9a85745fd483e414ab659d354bfe2805d Oct 31 04:09:29.014674 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:29.014644 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ggcnf" Oct 31 04:09:29.075606 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:29.075557 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vwcwp" event={"ID":"1f9f7baf-a850-453c-8e16-849942ba754a","Type":"ContainerStarted","Data":"2a9928a702fcca2791e0857443fcbcff01dc994a018a93a1a2f29af6c1cdfeaf"} Oct 31 04:09:29.075801 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:29.075634 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vwcwp" event={"ID":"1f9f7baf-a850-453c-8e16-849942ba754a","Type":"ContainerStarted","Data":"a67cae05638d07ae6cdd048cb13748c9a85745fd483e414ab659d354bfe2805d"} Oct 31 04:09:29.075801 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:29.075689 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vwcwp" Oct 31 04:09:29.124303 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:29.124245 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vwcwp" podStartSLOduration=2.124229642 podStartE2EDuration="2.124229642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:09:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:09:29.12240908 +0000 UTC m=+323.878747121" watchObservedRunningTime="2026-10-31 04:09:29.124229642 +0000 UTC m=+323.880567682" Oct 31 04:09:33.043939 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:33.043901 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-r22dt" Oct 31 04:09:40.083260 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:40.083228 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vwcwp" Oct 31 04:09:40.174799 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:40.174750 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-9qv62"] Oct 31 04:09:40.175092 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:40.175029 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-9qv62" podUID="7c48eb82-caa2-464d-9c1b-fdca4fe4b38e" containerName="manager" containerID="cri-o://a75f1d40247d790dc255cc6c7d10968925ea33ff0bf5b198235c168a9a882a8a" gracePeriod=10 Oct 31 04:09:40.435559 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:40.435532 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-9qv62" Oct 31 04:09:40.554157 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:40.554121 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c48eb82-caa2-464d-9c1b-fdca4fe4b38e-kube-api-access-lccqb\" (UniqueName: \"kubernetes.io/projected/7c48eb82-caa2-464d-9c1b-fdca4fe4b38e-kube-api-access-lccqb\") pod \"7c48eb82-caa2-464d-9c1b-fdca4fe4b38e\" (UID: \"7c48eb82-caa2-464d-9c1b-fdca4fe4b38e\") " Oct 31 04:09:40.554393 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:40.554178 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c48eb82-caa2-464d-9c1b-fdca4fe4b38e-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7c48eb82-caa2-464d-9c1b-fdca4fe4b38e-extensions-socket-volume\") pod \"7c48eb82-caa2-464d-9c1b-fdca4fe4b38e\" (UID: \"7c48eb82-caa2-464d-9c1b-fdca4fe4b38e\") " Oct 31 04:09:40.554832 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:40.554790 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c48eb82-caa2-464d-9c1b-fdca4fe4b38e-extensions-socket-volume" pod "7c48eb82-caa2-464d-9c1b-fdca4fe4b38e" (UID: "7c48eb82-caa2-464d-9c1b-fdca4fe4b38e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:09:40.556498 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:40.556472 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c48eb82-caa2-464d-9c1b-fdca4fe4b38e-kube-api-access-lccqb" pod "7c48eb82-caa2-464d-9c1b-fdca4fe4b38e" (UID: "7c48eb82-caa2-464d-9c1b-fdca4fe4b38e"). InnerVolumeSpecName "kube-api-access-lccqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:09:40.655295 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:40.655199 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lccqb\" (UniqueName: \"kubernetes.io/projected/7c48eb82-caa2-464d-9c1b-fdca4fe4b38e-kube-api-access-lccqb\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:09:40.655295 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:40.655232 5019 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7c48eb82-caa2-464d-9c1b-fdca4fe4b38e-extensions-socket-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:09:41.134638 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:41.134603 5019 generic.go:356] "Generic (PLEG): container finished" podID="7c48eb82-caa2-464d-9c1b-fdca4fe4b38e" containerID="a75f1d40247d790dc255cc6c7d10968925ea33ff0bf5b198235c168a9a882a8a" exitCode=0 Oct 31 04:09:41.135032 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:41.134649 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-9qv62" Oct 31 04:09:41.135032 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:41.134694 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-9qv62" event={"ID":"7c48eb82-caa2-464d-9c1b-fdca4fe4b38e","Type":"ContainerDied","Data":"a75f1d40247d790dc255cc6c7d10968925ea33ff0bf5b198235c168a9a882a8a"} Oct 31 04:09:41.135032 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:41.134752 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-9qv62" event={"ID":"7c48eb82-caa2-464d-9c1b-fdca4fe4b38e","Type":"ContainerDied","Data":"364d53bdeb608a885918a6924b91e3a54399d75ba3c1e0df389a392a41f3c6f3"} Oct 31 04:09:41.135032 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:41.134774 5019 scope.go:122] "RemoveContainer" containerID="a75f1d40247d790dc255cc6c7d10968925ea33ff0bf5b198235c168a9a882a8a" Oct 31 04:09:41.144723 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:41.144703 5019 scope.go:122] "RemoveContainer" containerID="a75f1d40247d790dc255cc6c7d10968925ea33ff0bf5b198235c168a9a882a8a" Oct 31 04:09:41.145076 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:09:41.145056 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a75f1d40247d790dc255cc6c7d10968925ea33ff0bf5b198235c168a9a882a8a\": container with ID starting with a75f1d40247d790dc255cc6c7d10968925ea33ff0bf5b198235c168a9a882a8a not found: ID does not exist" containerID="a75f1d40247d790dc255cc6c7d10968925ea33ff0bf5b198235c168a9a882a8a" Oct 31 04:09:41.145148 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:41.145091 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a75f1d40247d790dc255cc6c7d10968925ea33ff0bf5b198235c168a9a882a8a"} err="failed to get container status \"a75f1d40247d790dc255cc6c7d10968925ea33ff0bf5b198235c168a9a882a8a\": rpc error: code = NotFound desc = could not find container \"a75f1d40247d790dc255cc6c7d10968925ea33ff0bf5b198235c168a9a882a8a\": container with ID starting with a75f1d40247d790dc255cc6c7d10968925ea33ff0bf5b198235c168a9a882a8a not found: ID does not exist" Oct 31 04:09:41.206923 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:41.206879 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-9qv62"] Oct 31 04:09:41.218934 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:41.218896 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-9qv62"] Oct 31 04:09:41.406076 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:41.405990 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c48eb82-caa2-464d-9c1b-fdca4fe4b38e" path="/var/lib/kubelet/pods/7c48eb82-caa2-464d-9c1b-fdca4fe4b38e/volumes" Oct 31 04:09:42.545769 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:42.545735 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-766d8d5547-gddts"] Oct 31 04:09:42.546204 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:42.546122 5019 cpu_manager.go:414] "containerMap: removing container" podUID="7c48eb82-caa2-464d-9c1b-fdca4fe4b38e" containerName="manager" Oct 31 04:09:42.546204 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:42.546139 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c48eb82-caa2-464d-9c1b-fdca4fe4b38e" containerName="manager" Oct 31 04:09:42.546873 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:42.546846 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c48eb82-caa2-464d-9c1b-fdca4fe4b38e" containerName="manager" Oct 31 04:09:42.553200 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:42.553171 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-766d8d5547-gddts" Oct 31 04:09:42.568786 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:42.568749 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Oct 31 04:09:42.571612 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:42.571576 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-766d8d5547-gddts"] Oct 31 04:09:42.581215 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:42.581168 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 31 04:09:42.581215 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:42.581187 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Oct 31 04:09:42.581215 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:42.581200 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 31 04:09:42.581215 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:42.581184 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Oct 31 04:09:42.668682 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:42.668651 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-zhl5z\"" Oct 31 04:09:42.674125 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:42.674086 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ef05bb6-1afb-4abb-858d-dc8104a696e4-metrics-certs\") pod \"jobset-controller-manager-766d8d5547-gddts\" (UID: \"1ef05bb6-1afb-4abb-858d-dc8104a696e4\") " pod="openshift-jobset-operator/jobset-controller-manager-766d8d5547-gddts" Oct 31 04:09:42.674125 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:42.674131 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1ef05bb6-1afb-4abb-858d-dc8104a696e4-manager-config\") pod \"jobset-controller-manager-766d8d5547-gddts\" (UID: \"1ef05bb6-1afb-4abb-858d-dc8104a696e4\") " pod="openshift-jobset-operator/jobset-controller-manager-766d8d5547-gddts" Oct 31 04:09:42.674397 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:42.674159 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1ef05bb6-1afb-4abb-858d-dc8104a696e4-cert\") pod \"jobset-controller-manager-766d8d5547-gddts\" (UID: \"1ef05bb6-1afb-4abb-858d-dc8104a696e4\") " pod="openshift-jobset-operator/jobset-controller-manager-766d8d5547-gddts" Oct 31 04:09:42.674397 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:42.674279 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnjcc\" (UniqueName: \"kubernetes.io/projected/1ef05bb6-1afb-4abb-858d-dc8104a696e4-kube-api-access-xnjcc\") pod \"jobset-controller-manager-766d8d5547-gddts\" (UID: \"1ef05bb6-1afb-4abb-858d-dc8104a696e4\") " pod="openshift-jobset-operator/jobset-controller-manager-766d8d5547-gddts" Oct 31 04:09:42.775358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:42.775290 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1ef05bb6-1afb-4abb-858d-dc8104a696e4-manager-config\") pod \"jobset-controller-manager-766d8d5547-gddts\" (UID: \"1ef05bb6-1afb-4abb-858d-dc8104a696e4\") " pod="openshift-jobset-operator/jobset-controller-manager-766d8d5547-gddts" Oct 31 04:09:42.775358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:42.775357 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1ef05bb6-1afb-4abb-858d-dc8104a696e4-cert\") pod \"jobset-controller-manager-766d8d5547-gddts\" (UID: \"1ef05bb6-1afb-4abb-858d-dc8104a696e4\") " pod="openshift-jobset-operator/jobset-controller-manager-766d8d5547-gddts" Oct 31 04:09:42.775605 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:42.775384 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnjcc\" (UniqueName: \"kubernetes.io/projected/1ef05bb6-1afb-4abb-858d-dc8104a696e4-kube-api-access-xnjcc\") pod \"jobset-controller-manager-766d8d5547-gddts\" (UID: \"1ef05bb6-1afb-4abb-858d-dc8104a696e4\") " pod="openshift-jobset-operator/jobset-controller-manager-766d8d5547-gddts" Oct 31 04:09:42.775605 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:42.775477 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ef05bb6-1afb-4abb-858d-dc8104a696e4-metrics-certs\") pod \"jobset-controller-manager-766d8d5547-gddts\" (UID: \"1ef05bb6-1afb-4abb-858d-dc8104a696e4\") " pod="openshift-jobset-operator/jobset-controller-manager-766d8d5547-gddts" Oct 31 04:09:42.776073 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:42.776043 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1ef05bb6-1afb-4abb-858d-dc8104a696e4-manager-config\") pod \"jobset-controller-manager-766d8d5547-gddts\" (UID: \"1ef05bb6-1afb-4abb-858d-dc8104a696e4\") " pod="openshift-jobset-operator/jobset-controller-manager-766d8d5547-gddts" Oct 31 04:09:42.778010 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:42.777979 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1ef05bb6-1afb-4abb-858d-dc8104a696e4-cert\") pod \"jobset-controller-manager-766d8d5547-gddts\" (UID: \"1ef05bb6-1afb-4abb-858d-dc8104a696e4\") " pod="openshift-jobset-operator/jobset-controller-manager-766d8d5547-gddts" Oct 31 04:09:42.778010 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:42.778000 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ef05bb6-1afb-4abb-858d-dc8104a696e4-metrics-certs\") pod \"jobset-controller-manager-766d8d5547-gddts\" (UID: \"1ef05bb6-1afb-4abb-858d-dc8104a696e4\") " pod="openshift-jobset-operator/jobset-controller-manager-766d8d5547-gddts" Oct 31 04:09:42.799629 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:42.799551 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnjcc\" (UniqueName: \"kubernetes.io/projected/1ef05bb6-1afb-4abb-858d-dc8104a696e4-kube-api-access-xnjcc\") pod \"jobset-controller-manager-766d8d5547-gddts\" (UID: \"1ef05bb6-1afb-4abb-858d-dc8104a696e4\") " pod="openshift-jobset-operator/jobset-controller-manager-766d8d5547-gddts" Oct 31 04:09:42.866432 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:42.866385 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-766d8d5547-gddts" Oct 31 04:09:43.164939 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:43.164907 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-766d8d5547-gddts"] Oct 31 04:09:43.167002 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:09:43.166971 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ef05bb6_1afb_4abb_858d_dc8104a696e4.slice/crio-761a84d6731f10349f29dc74491bfe631da5b5071bd469a3b5de7296290cad35 WatchSource:0}: Error finding container 761a84d6731f10349f29dc74491bfe631da5b5071bd469a3b5de7296290cad35: Status 404 returned error can't find the container with id 761a84d6731f10349f29dc74491bfe631da5b5071bd469a3b5de7296290cad35 Oct 31 04:09:44.153739 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:44.153694 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-766d8d5547-gddts" event={"ID":"1ef05bb6-1afb-4abb-858d-dc8104a696e4","Type":"ContainerStarted","Data":"761a84d6731f10349f29dc74491bfe631da5b5071bd469a3b5de7296290cad35"} Oct 31 04:09:46.170098 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:46.169999 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-766d8d5547-gddts" event={"ID":"1ef05bb6-1afb-4abb-858d-dc8104a696e4","Type":"ContainerStarted","Data":"0a77a2e80e7d359ccbba6b7aed6488f5bada2e5abb5f6c2dc636420e001fe751"} Oct 31 04:09:46.170482 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:46.170090 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-766d8d5547-gddts" Oct 31 04:09:46.213062 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:46.213002 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-766d8d5547-gddts" podStartSLOduration=1.521954099 podStartE2EDuration="4.212983484s" totalImagesPullingTime="2.691029385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:09:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:09:46.211095964 +0000 UTC m=+340.967434006" watchObservedRunningTime="2026-10-31 04:09:46.212983484 +0000 UTC m=+340.969321521" Oct 31 04:09:52.390342 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:09:52.390286 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:09:57.183344 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:09:57.183289 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-766d8d5547-gddts" Oct 31 04:10:22.401549 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:10:22.401507 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:10:52.390348 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:10:52.390250 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:11:22.390981 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:11:22.390937 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:11:35.312987 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:11:35.312948 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5698fc8df4-kqgsq"] Oct 31 04:11:35.316788 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:11:35.316767 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5698fc8df4-kqgsq" Oct 31 04:11:35.336643 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:11:35.336609 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5698fc8df4-kqgsq"] Oct 31 04:11:35.346226 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:11:35.346192 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 31 04:11:35.346226 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:11:35.346223 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-hc5c9\"" Oct 31 04:11:35.362834 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:11:35.362798 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ad2f7eeb-489b-48e2-a5c8-02e146a03cff-tls-cert\") pod \"authorino-5698fc8df4-kqgsq\" (UID: \"ad2f7eeb-489b-48e2-a5c8-02e146a03cff\") " pod="kuadrant-system/authorino-5698fc8df4-kqgsq" Oct 31 04:11:35.363050 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:11:35.362848 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh4jr\" (UniqueName: \"kubernetes.io/projected/ad2f7eeb-489b-48e2-a5c8-02e146a03cff-kube-api-access-fh4jr\") pod \"authorino-5698fc8df4-kqgsq\" (UID: \"ad2f7eeb-489b-48e2-a5c8-02e146a03cff\") " pod="kuadrant-system/authorino-5698fc8df4-kqgsq" Oct 31 04:11:35.463946 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:11:35.463902 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ad2f7eeb-489b-48e2-a5c8-02e146a03cff-tls-cert\") pod \"authorino-5698fc8df4-kqgsq\" (UID: \"ad2f7eeb-489b-48e2-a5c8-02e146a03cff\") " pod="kuadrant-system/authorino-5698fc8df4-kqgsq" Oct 31 04:11:35.464131 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:11:35.463955 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh4jr\" (UniqueName: \"kubernetes.io/projected/ad2f7eeb-489b-48e2-a5c8-02e146a03cff-kube-api-access-fh4jr\") pod \"authorino-5698fc8df4-kqgsq\" (UID: \"ad2f7eeb-489b-48e2-a5c8-02e146a03cff\") " pod="kuadrant-system/authorino-5698fc8df4-kqgsq" Oct 31 04:11:35.466526 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:11:35.466498 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ad2f7eeb-489b-48e2-a5c8-02e146a03cff-tls-cert\") pod \"authorino-5698fc8df4-kqgsq\" (UID: \"ad2f7eeb-489b-48e2-a5c8-02e146a03cff\") " pod="kuadrant-system/authorino-5698fc8df4-kqgsq" Oct 31 04:11:35.487840 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:11:35.487809 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh4jr\" (UniqueName: \"kubernetes.io/projected/ad2f7eeb-489b-48e2-a5c8-02e146a03cff-kube-api-access-fh4jr\") pod \"authorino-5698fc8df4-kqgsq\" (UID: \"ad2f7eeb-489b-48e2-a5c8-02e146a03cff\") " pod="kuadrant-system/authorino-5698fc8df4-kqgsq" Oct 31 04:11:35.642545 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:11:35.642461 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5698fc8df4-kqgsq" Oct 31 04:11:35.726898 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:11:35.726854 5019 generic.go:356] "Generic (PLEG): container finished" podID="e0ccc03e-0e5e-40b7-8a40-718f199dd747" containerID="f0f1b0a9d04e4f7be16d6c8de71e04915a34ca21d3c7c2dd2a5caf4de378669e" exitCode=0 Oct 31 04:11:35.727047 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:11:35.726915 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7" event={"ID":"e0ccc03e-0e5e-40b7-8a40-718f199dd747","Type":"ContainerDied","Data":"f0f1b0a9d04e4f7be16d6c8de71e04915a34ca21d3c7c2dd2a5caf4de378669e"} Oct 31 04:11:35.727520 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:11:35.727500 5019 scope.go:122] "RemoveContainer" containerID="f0f1b0a9d04e4f7be16d6c8de71e04915a34ca21d3c7c2dd2a5caf4de378669e" Oct 31 04:11:35.728790 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:11:35.728762 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:11:35.728940 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:11:35.728926 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:11:35.729026 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:11:35.729016 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:11:35.933880 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:11:35.933839 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5698fc8df4-kqgsq"] Oct 31 04:11:35.935479 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:11:35.935443 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podad2f7eeb_489b_48e2_a5c8_02e146a03cff.slice/crio-37dff8dd8b6247e6b9f8f4f7e0caa49f3c1c09775d0d5ba9cd3cd47f4bc3dcfa WatchSource:0}: Error finding container 37dff8dd8b6247e6b9f8f4f7e0caa49f3c1c09775d0d5ba9cd3cd47f4bc3dcfa: Status 404 returned error can't find the container with id 37dff8dd8b6247e6b9f8f4f7e0caa49f3c1c09775d0d5ba9cd3cd47f4bc3dcfa Oct 31 04:11:36.734104 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:11:36.734068 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5698fc8df4-kqgsq" event={"ID":"ad2f7eeb-489b-48e2-a5c8-02e146a03cff","Type":"ContainerStarted","Data":"37dff8dd8b6247e6b9f8f4f7e0caa49f3c1c09775d0d5ba9cd3cd47f4bc3dcfa"} Oct 31 04:11:36.736027 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:11:36.736003 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7" event={"ID":"e0ccc03e-0e5e-40b7-8a40-718f199dd747","Type":"ContainerStarted","Data":"2e4684a3042b06f9b8f590a39685056778deeec1cd39a10a0daf25c952f3b422"} Oct 31 04:11:37.067230 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:11:37.067150 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7" Oct 31 04:11:37.067496 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:11:37.067468 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-pghv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Oct 31 04:11:37.067587 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:11:37.067523 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7" podUID="e0ccc03e-0e5e-40b7-8a40-718f199dd747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Oct 31 04:11:38.067512 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:11:38.067468 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-pghv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Oct 31 04:11:38.068034 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:11:38.067541 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7" podUID="e0ccc03e-0e5e-40b7-8a40-718f199dd747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Oct 31 04:11:38.749183 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:11:38.749141 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5698fc8df4-kqgsq" event={"ID":"ad2f7eeb-489b-48e2-a5c8-02e146a03cff","Type":"ContainerStarted","Data":"ec648ac7bfd4c07b0d00297f71b7405e15a7f5795114608b31c6a2f857b2eac2"} Oct 31 04:11:39.067791 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:11:39.067686 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-pghv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Oct 31 04:11:39.067791 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:11:39.067760 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7" podUID="e0ccc03e-0e5e-40b7-8a40-718f199dd747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Oct 31 04:11:40.068098 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:11:40.068063 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-pghv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Oct 31 04:11:40.068544 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:11:40.068120 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7" podUID="e0ccc03e-0e5e-40b7-8a40-718f199dd747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Oct 31 04:11:41.068125 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:11:41.068083 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-pghv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Oct 31 04:11:41.068584 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:11:41.068157 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7" podUID="e0ccc03e-0e5e-40b7-8a40-718f199dd747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Oct 31 04:11:42.071197 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:11:42.071162 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7" Oct 31 04:11:42.071631 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:11:42.071522 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7" Oct 31 04:11:42.072429 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:11:42.072410 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7" Oct 31 04:11:42.130610 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:11:42.130547 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5698fc8df4-kqgsq" podStartSLOduration=4.8152314149999995 podStartE2EDuration="7.13052888s" totalImagesPullingTime="2.315297465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:11:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:11:38.790524936 +0000 UTC m=+453.546862979" watchObservedRunningTime="2026-10-31 04:11:42.13052888 +0000 UTC m=+456.886866919" Oct 31 04:11:52.388754 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:11:52.388715 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:12:22.391280 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:12:22.391235 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:12:27.580704 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:27.580669 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-6dc65d95b6-gpnbc"] Oct 31 04:12:27.584233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:27.584208 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6dc65d95b6-gpnbc" Oct 31 04:12:27.613493 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:27.613470 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-x8t52\"" Oct 31 04:12:27.613636 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:27.613523 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 31 04:12:27.613636 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:27.613552 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 31 04:12:27.613636 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:27.613565 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 31 04:12:27.613756 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:27.613641 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 31 04:12:27.615455 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:27.615437 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6dc65d95b6-gpnbc"] Oct 31 04:12:27.724463 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:27.724435 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d194e31f-b57e-490d-a5c8-9f7b12de4b2c-metrics-certs\") pod \"ai-gateway-controller-6dc65d95b6-gpnbc\" (UID: \"d194e31f-b57e-490d-a5c8-9f7b12de4b2c\") " pod="opendatahub/ai-gateway-controller-6dc65d95b6-gpnbc" Oct 31 04:12:27.724605 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:27.724545 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7plk\" (UniqueName: \"kubernetes.io/projected/d194e31f-b57e-490d-a5c8-9f7b12de4b2c-kube-api-access-x7plk\") pod \"ai-gateway-controller-6dc65d95b6-gpnbc\" (UID: \"d194e31f-b57e-490d-a5c8-9f7b12de4b2c\") " pod="opendatahub/ai-gateway-controller-6dc65d95b6-gpnbc" Oct 31 04:12:27.825213 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:27.825169 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7plk\" (UniqueName: \"kubernetes.io/projected/d194e31f-b57e-490d-a5c8-9f7b12de4b2c-kube-api-access-x7plk\") pod \"ai-gateway-controller-6dc65d95b6-gpnbc\" (UID: \"d194e31f-b57e-490d-a5c8-9f7b12de4b2c\") " pod="opendatahub/ai-gateway-controller-6dc65d95b6-gpnbc" Oct 31 04:12:27.825427 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:27.825258 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d194e31f-b57e-490d-a5c8-9f7b12de4b2c-metrics-certs\") pod \"ai-gateway-controller-6dc65d95b6-gpnbc\" (UID: \"d194e31f-b57e-490d-a5c8-9f7b12de4b2c\") " pod="opendatahub/ai-gateway-controller-6dc65d95b6-gpnbc" Oct 31 04:12:27.827742 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:27.827721 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d194e31f-b57e-490d-a5c8-9f7b12de4b2c-metrics-certs\") pod \"ai-gateway-controller-6dc65d95b6-gpnbc\" (UID: \"d194e31f-b57e-490d-a5c8-9f7b12de4b2c\") " pod="opendatahub/ai-gateway-controller-6dc65d95b6-gpnbc" Oct 31 04:12:27.848667 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:27.848608 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7plk\" (UniqueName: \"kubernetes.io/projected/d194e31f-b57e-490d-a5c8-9f7b12de4b2c-kube-api-access-x7plk\") pod \"ai-gateway-controller-6dc65d95b6-gpnbc\" (UID: \"d194e31f-b57e-490d-a5c8-9f7b12de4b2c\") " pod="opendatahub/ai-gateway-controller-6dc65d95b6-gpnbc" Oct 31 04:12:27.896598 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:27.896564 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6dc65d95b6-gpnbc" Oct 31 04:12:28.189621 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:28.189595 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6dc65d95b6-gpnbc"] Oct 31 04:12:28.191564 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:12:28.191539 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd194e31f_b57e_490d_a5c8_9f7b12de4b2c.slice/crio-acd07fccd748169803e54acbbbb58518b6e7c6502ecc88464527167a73426812 WatchSource:0}: Error finding container acd07fccd748169803e54acbbbb58518b6e7c6502ecc88464527167a73426812: Status 404 returned error can't find the container with id acd07fccd748169803e54acbbbb58518b6e7c6502ecc88464527167a73426812 Oct 31 04:12:28.985784 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:28.985741 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6dc65d95b6-gpnbc" event={"ID":"d194e31f-b57e-490d-a5c8-9f7b12de4b2c","Type":"ContainerStarted","Data":"acd07fccd748169803e54acbbbb58518b6e7c6502ecc88464527167a73426812"} Oct 31 04:12:30.997695 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:30.997659 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6dc65d95b6-gpnbc" event={"ID":"d194e31f-b57e-490d-a5c8-9f7b12de4b2c","Type":"ContainerStarted","Data":"eee21e4e4b45f16c50b909df20be8525c4070e7e3794629fdaa95474583c7ab2"} Oct 31 04:12:30.998107 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:30.997747 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-6dc65d95b6-gpnbc" Oct 31 04:12:31.038324 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:31.038272 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-6dc65d95b6-gpnbc" podStartSLOduration=1.9256069789999999 podStartE2EDuration="4.038259362s" totalImagesPullingTime="2.112652383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:12:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:12:31.038044355 +0000 UTC m=+505.794382394" watchObservedRunningTime="2026-10-31 04:12:31.038259362 +0000 UTC m=+505.794597402" Oct 31 04:12:42.005428 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:42.005398 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-6dc65d95b6-gpnbc" Oct 31 04:12:52.392190 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:12:52.392151 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:12:53.514915 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:53.514881 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-mprn9"] Oct 31 04:12:53.520972 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:53.520949 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mprn9" Oct 31 04:12:53.537013 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:53.536989 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 31 04:12:53.549107 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:53.549081 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-wcn6k\"" Oct 31 04:12:53.549107 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:53.549094 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 31 04:12:53.549291 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:53.549182 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 31 04:12:53.551763 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:53.551738 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-mprn9"] Oct 31 04:12:53.639587 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:53.639567 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffacd465-1055-444a-9f85-f46a57ac3724-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mprn9\" (UID: \"ffacd465-1055-444a-9f85-f46a57ac3724\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mprn9" Oct 31 04:12:53.639696 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:53.639611 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffacd465-1055-444a-9f85-f46a57ac3724-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mprn9\" (UID: \"ffacd465-1055-444a-9f85-f46a57ac3724\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mprn9" Oct 31 04:12:53.639696 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:53.639642 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffacd465-1055-444a-9f85-f46a57ac3724-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mprn9\" (UID: \"ffacd465-1055-444a-9f85-f46a57ac3724\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mprn9" Oct 31 04:12:53.639770 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:53.639718 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffacd465-1055-444a-9f85-f46a57ac3724-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mprn9\" (UID: \"ffacd465-1055-444a-9f85-f46a57ac3724\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mprn9" Oct 31 04:12:53.639770 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:53.639761 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffacd465-1055-444a-9f85-f46a57ac3724-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mprn9\" (UID: \"ffacd465-1055-444a-9f85-f46a57ac3724\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mprn9" Oct 31 04:12:53.639850 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:53.639832 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw5qh\" (UniqueName: \"kubernetes.io/projected/ffacd465-1055-444a-9f85-f46a57ac3724-kube-api-access-kw5qh\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mprn9\" (UID: \"ffacd465-1055-444a-9f85-f46a57ac3724\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mprn9" Oct 31 04:12:53.741089 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:53.741058 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffacd465-1055-444a-9f85-f46a57ac3724-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mprn9\" (UID: \"ffacd465-1055-444a-9f85-f46a57ac3724\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mprn9" Oct 31 04:12:53.741249 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:53.741096 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw5qh\" (UniqueName: \"kubernetes.io/projected/ffacd465-1055-444a-9f85-f46a57ac3724-kube-api-access-kw5qh\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mprn9\" (UID: \"ffacd465-1055-444a-9f85-f46a57ac3724\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mprn9" Oct 31 04:12:53.741249 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:53.741129 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffacd465-1055-444a-9f85-f46a57ac3724-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mprn9\" (UID: \"ffacd465-1055-444a-9f85-f46a57ac3724\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mprn9" Oct 31 04:12:53.741249 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:53.741159 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffacd465-1055-444a-9f85-f46a57ac3724-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mprn9\" (UID: \"ffacd465-1055-444a-9f85-f46a57ac3724\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mprn9" Oct 31 04:12:53.741249 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:53.741178 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffacd465-1055-444a-9f85-f46a57ac3724-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mprn9\" (UID: \"ffacd465-1055-444a-9f85-f46a57ac3724\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mprn9" Oct 31 04:12:53.741249 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:53.741201 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffacd465-1055-444a-9f85-f46a57ac3724-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mprn9\" (UID: \"ffacd465-1055-444a-9f85-f46a57ac3724\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mprn9" Oct 31 04:12:53.741524 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:53.741483 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffacd465-1055-444a-9f85-f46a57ac3724-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mprn9\" (UID: \"ffacd465-1055-444a-9f85-f46a57ac3724\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mprn9" Oct 31 04:12:53.741647 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:53.741629 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffacd465-1055-444a-9f85-f46a57ac3724-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mprn9\" (UID: \"ffacd465-1055-444a-9f85-f46a57ac3724\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mprn9" Oct 31 04:12:53.741711 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:53.741683 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffacd465-1055-444a-9f85-f46a57ac3724-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mprn9\" (UID: \"ffacd465-1055-444a-9f85-f46a57ac3724\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mprn9" Oct 31 04:12:53.743419 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:53.743402 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffacd465-1055-444a-9f85-f46a57ac3724-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mprn9\" (UID: \"ffacd465-1055-444a-9f85-f46a57ac3724\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mprn9" Oct 31 04:12:53.743696 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:53.743680 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffacd465-1055-444a-9f85-f46a57ac3724-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mprn9\" (UID: \"ffacd465-1055-444a-9f85-f46a57ac3724\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mprn9" Oct 31 04:12:53.764220 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:53.764200 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw5qh\" (UniqueName: \"kubernetes.io/projected/ffacd465-1055-444a-9f85-f46a57ac3724-kube-api-access-kw5qh\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mprn9\" (UID: \"ffacd465-1055-444a-9f85-f46a57ac3724\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mprn9" Oct 31 04:12:53.838323 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:53.838272 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mprn9" Oct 31 04:12:54.132524 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:54.132496 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-mprn9"] Oct 31 04:12:54.134211 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:12:54.134189 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffacd465_1055_444a_9f85_f46a57ac3724.slice/crio-49f718e69a2fa3e5487bfaf86638e71bb4e0538ef5ba82839586ae382c121966 WatchSource:0}: Error finding container 49f718e69a2fa3e5487bfaf86638e71bb4e0538ef5ba82839586ae382c121966: Status 404 returned error can't find the container with id 49f718e69a2fa3e5487bfaf86638e71bb4e0538ef5ba82839586ae382c121966 Oct 31 04:12:55.116207 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:55.116166 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mprn9" event={"ID":"ffacd465-1055-444a-9f85-f46a57ac3724","Type":"ContainerStarted","Data":"49f718e69a2fa3e5487bfaf86638e71bb4e0538ef5ba82839586ae382c121966"} Oct 31 04:12:55.318152 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:55.318109 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-9mgfd"] Oct 31 04:12:55.322137 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:55.322107 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9mgfd" Oct 31 04:12:55.354396 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:55.352802 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Oct 31 04:12:55.357214 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:55.357183 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/705ea68d-0f37-4d66-a90a-5f089d437b09-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9mgfd\" (UID: \"705ea68d-0f37-4d66-a90a-5f089d437b09\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9mgfd" Oct 31 04:12:55.357454 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:55.357435 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/705ea68d-0f37-4d66-a90a-5f089d437b09-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9mgfd\" (UID: \"705ea68d-0f37-4d66-a90a-5f089d437b09\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9mgfd" Oct 31 04:12:55.357602 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:55.357585 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/705ea68d-0f37-4d66-a90a-5f089d437b09-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9mgfd\" (UID: \"705ea68d-0f37-4d66-a90a-5f089d437b09\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9mgfd" Oct 31 04:12:55.357775 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:55.357758 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b6jx\" (UniqueName: \"kubernetes.io/projected/705ea68d-0f37-4d66-a90a-5f089d437b09-kube-api-access-9b6jx\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9mgfd\" (UID: \"705ea68d-0f37-4d66-a90a-5f089d437b09\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9mgfd" Oct 31 04:12:55.358961 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:55.358941 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/705ea68d-0f37-4d66-a90a-5f089d437b09-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9mgfd\" (UID: \"705ea68d-0f37-4d66-a90a-5f089d437b09\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9mgfd" Oct 31 04:12:55.359102 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:55.359085 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/705ea68d-0f37-4d66-a90a-5f089d437b09-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9mgfd\" (UID: \"705ea68d-0f37-4d66-a90a-5f089d437b09\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9mgfd" Oct 31 04:12:55.373899 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:55.373872 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-9mgfd"] Oct 31 04:12:55.460481 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:55.460448 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/705ea68d-0f37-4d66-a90a-5f089d437b09-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9mgfd\" (UID: \"705ea68d-0f37-4d66-a90a-5f089d437b09\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9mgfd" Oct 31 04:12:55.460610 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:55.460514 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/705ea68d-0f37-4d66-a90a-5f089d437b09-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9mgfd\" (UID: \"705ea68d-0f37-4d66-a90a-5f089d437b09\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9mgfd" Oct 31 04:12:55.460610 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:55.460551 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/705ea68d-0f37-4d66-a90a-5f089d437b09-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9mgfd\" (UID: \"705ea68d-0f37-4d66-a90a-5f089d437b09\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9mgfd" Oct 31 04:12:55.460945 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:55.460922 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b6jx\" (UniqueName: \"kubernetes.io/projected/705ea68d-0f37-4d66-a90a-5f089d437b09-kube-api-access-9b6jx\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9mgfd\" (UID: \"705ea68d-0f37-4d66-a90a-5f089d437b09\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9mgfd" Oct 31 04:12:55.461039 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:55.460942 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/705ea68d-0f37-4d66-a90a-5f089d437b09-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9mgfd\" (UID: \"705ea68d-0f37-4d66-a90a-5f089d437b09\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9mgfd" Oct 31 04:12:55.461039 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:55.460989 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/705ea68d-0f37-4d66-a90a-5f089d437b09-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9mgfd\" (UID: \"705ea68d-0f37-4d66-a90a-5f089d437b09\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9mgfd" Oct 31 04:12:55.461039 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:55.461014 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/705ea68d-0f37-4d66-a90a-5f089d437b09-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9mgfd\" (UID: \"705ea68d-0f37-4d66-a90a-5f089d437b09\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9mgfd" Oct 31 04:12:55.461195 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:55.461040 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/705ea68d-0f37-4d66-a90a-5f089d437b09-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9mgfd\" (UID: \"705ea68d-0f37-4d66-a90a-5f089d437b09\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9mgfd" Oct 31 04:12:55.461323 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:55.461304 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/705ea68d-0f37-4d66-a90a-5f089d437b09-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9mgfd\" (UID: \"705ea68d-0f37-4d66-a90a-5f089d437b09\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9mgfd" Oct 31 04:12:55.463103 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:55.463077 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/705ea68d-0f37-4d66-a90a-5f089d437b09-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9mgfd\" (UID: \"705ea68d-0f37-4d66-a90a-5f089d437b09\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9mgfd" Oct 31 04:12:55.463447 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:55.463429 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/705ea68d-0f37-4d66-a90a-5f089d437b09-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9mgfd\" (UID: \"705ea68d-0f37-4d66-a90a-5f089d437b09\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9mgfd" Oct 31 04:12:55.514581 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:55.502064 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b6jx\" (UniqueName: \"kubernetes.io/projected/705ea68d-0f37-4d66-a90a-5f089d437b09-kube-api-access-9b6jx\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9mgfd\" (UID: \"705ea68d-0f37-4d66-a90a-5f089d437b09\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9mgfd" Oct 31 04:12:55.642298 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:55.642211 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9mgfd" Oct 31 04:12:55.953394 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:55.953013 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-9mgfd"] Oct 31 04:12:55.955147 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:12:55.955109 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod705ea68d_0f37_4d66_a90a_5f089d437b09.slice/crio-fb1a2c5989ef0d03c916b23a264342680826187422235210238389204008471b WatchSource:0}: Error finding container fb1a2c5989ef0d03c916b23a264342680826187422235210238389204008471b: Status 404 returned error can't find the container with id fb1a2c5989ef0d03c916b23a264342680826187422235210238389204008471b Oct 31 04:12:56.122909 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:56.122864 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mprn9" event={"ID":"ffacd465-1055-444a-9f85-f46a57ac3724","Type":"ContainerStarted","Data":"d3f5e1d75a9898f9875b8e0fe4684701dbd6d74f39e54df1d6ee8b5c3cdf47be"} Oct 31 04:12:56.124144 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:56.124122 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9mgfd" event={"ID":"705ea68d-0f37-4d66-a90a-5f089d437b09","Type":"ContainerStarted","Data":"fb1a2c5989ef0d03c916b23a264342680826187422235210238389204008471b"} Oct 31 04:12:56.165050 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:56.165002 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mprn9" podStartSLOduration=1.923863487 podStartE2EDuration="3.164991333s" totalImagesPullingTime="1.241127846s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:12:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:12:56.163790891 +0000 UTC m=+530.920128931" watchObservedRunningTime="2026-10-31 04:12:56.164991333 +0000 UTC m=+530.921329373" Oct 31 04:12:56.838500 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:56.838465 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mprn9" Oct 31 04:12:57.131865 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:57.131779 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9mgfd" event={"ID":"705ea68d-0f37-4d66-a90a-5f089d437b09","Type":"ContainerStarted","Data":"7e9db91474afb3a27a15071a5f8cc83b08b8729173cb6f4d1fc01bd9f237c9b7"} Oct 31 04:12:57.173735 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:57.173662 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9mgfd" podStartSLOduration=1.9632045809999998 podStartE2EDuration="2.173648084s" totalImagesPullingTime="210.443503ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:12:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:12:57.172154916 +0000 UTC m=+531.928492957" watchObservedRunningTime="2026-10-31 04:12:57.173648084 +0000 UTC m=+531.929986512" Oct 31 04:12:57.643205 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:57.643173 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9mgfd" Oct 31 04:12:59.362714 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:59.362678 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c44pm"] Oct 31 04:12:59.367637 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:59.367600 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c44pm" Oct 31 04:12:59.382896 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:59.382863 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Oct 31 04:12:59.397717 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:59.397687 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c44pm"] Oct 31 04:12:59.499276 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:59.499240 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb29m\" (UniqueName: \"kubernetes.io/projected/0513559e-98f2-4889-b6cd-5b2fff981f3e-kube-api-access-qb29m\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c44pm\" (UID: \"0513559e-98f2-4889-b6cd-5b2fff981f3e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c44pm" Oct 31 04:12:59.499463 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:59.499361 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0513559e-98f2-4889-b6cd-5b2fff981f3e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c44pm\" (UID: \"0513559e-98f2-4889-b6cd-5b2fff981f3e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c44pm" Oct 31 04:12:59.499463 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:59.499450 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0513559e-98f2-4889-b6cd-5b2fff981f3e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c44pm\" (UID: \"0513559e-98f2-4889-b6cd-5b2fff981f3e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c44pm" Oct 31 04:12:59.499556 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:59.499492 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0513559e-98f2-4889-b6cd-5b2fff981f3e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c44pm\" (UID: \"0513559e-98f2-4889-b6cd-5b2fff981f3e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c44pm" Oct 31 04:12:59.499556 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:59.499522 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0513559e-98f2-4889-b6cd-5b2fff981f3e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c44pm\" (UID: \"0513559e-98f2-4889-b6cd-5b2fff981f3e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c44pm" Oct 31 04:12:59.499640 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:59.499572 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0513559e-98f2-4889-b6cd-5b2fff981f3e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c44pm\" (UID: \"0513559e-98f2-4889-b6cd-5b2fff981f3e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c44pm" Oct 31 04:12:59.600915 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:59.600883 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb29m\" (UniqueName: \"kubernetes.io/projected/0513559e-98f2-4889-b6cd-5b2fff981f3e-kube-api-access-qb29m\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c44pm\" (UID: \"0513559e-98f2-4889-b6cd-5b2fff981f3e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c44pm" Oct 31 04:12:59.601107 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:59.600939 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0513559e-98f2-4889-b6cd-5b2fff981f3e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c44pm\" (UID: \"0513559e-98f2-4889-b6cd-5b2fff981f3e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c44pm" Oct 31 04:12:59.601107 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:59.601002 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0513559e-98f2-4889-b6cd-5b2fff981f3e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c44pm\" (UID: \"0513559e-98f2-4889-b6cd-5b2fff981f3e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c44pm" Oct 31 04:12:59.601107 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:59.601050 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0513559e-98f2-4889-b6cd-5b2fff981f3e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c44pm\" (UID: \"0513559e-98f2-4889-b6cd-5b2fff981f3e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c44pm" Oct 31 04:12:59.601107 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:59.601088 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0513559e-98f2-4889-b6cd-5b2fff981f3e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c44pm\" (UID: \"0513559e-98f2-4889-b6cd-5b2fff981f3e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c44pm" Oct 31 04:12:59.601322 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:59.601111 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0513559e-98f2-4889-b6cd-5b2fff981f3e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c44pm\" (UID: \"0513559e-98f2-4889-b6cd-5b2fff981f3e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c44pm" Oct 31 04:12:59.601709 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:59.601678 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0513559e-98f2-4889-b6cd-5b2fff981f3e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c44pm\" (UID: \"0513559e-98f2-4889-b6cd-5b2fff981f3e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c44pm" Oct 31 04:12:59.601836 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:59.601678 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0513559e-98f2-4889-b6cd-5b2fff981f3e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c44pm\" (UID: \"0513559e-98f2-4889-b6cd-5b2fff981f3e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c44pm" Oct 31 04:12:59.601923 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:59.601899 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0513559e-98f2-4889-b6cd-5b2fff981f3e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c44pm\" (UID: \"0513559e-98f2-4889-b6cd-5b2fff981f3e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c44pm" Oct 31 04:12:59.603360 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:59.603319 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0513559e-98f2-4889-b6cd-5b2fff981f3e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c44pm\" (UID: \"0513559e-98f2-4889-b6cd-5b2fff981f3e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c44pm" Oct 31 04:12:59.603904 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:59.603881 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0513559e-98f2-4889-b6cd-5b2fff981f3e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c44pm\" (UID: \"0513559e-98f2-4889-b6cd-5b2fff981f3e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c44pm" Oct 31 04:12:59.625752 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:59.625692 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb29m\" (UniqueName: \"kubernetes.io/projected/0513559e-98f2-4889-b6cd-5b2fff981f3e-kube-api-access-qb29m\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c44pm\" (UID: \"0513559e-98f2-4889-b6cd-5b2fff981f3e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c44pm" Oct 31 04:12:59.681558 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:12:59.681449 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c44pm" Oct 31 04:13:00.006285 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:00.006182 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c44pm"] Oct 31 04:13:00.010396 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:13:00.010363 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0513559e_98f2_4889_b6cd_5b2fff981f3e.slice/crio-211d991895bafd75a37c9d533d0ecfb4debf199308f34737449ca9ba891c9e6b WatchSource:0}: Error finding container 211d991895bafd75a37c9d533d0ecfb4debf199308f34737449ca9ba891c9e6b: Status 404 returned error can't find the container with id 211d991895bafd75a37c9d533d0ecfb4debf199308f34737449ca9ba891c9e6b Oct 31 04:13:00.149097 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:00.149054 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c44pm" event={"ID":"0513559e-98f2-4889-b6cd-5b2fff981f3e","Type":"ContainerStarted","Data":"211d991895bafd75a37c9d533d0ecfb4debf199308f34737449ca9ba891c9e6b"} Oct 31 04:13:01.154838 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:01.154802 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c44pm" event={"ID":"0513559e-98f2-4889-b6cd-5b2fff981f3e","Type":"ContainerStarted","Data":"37bd4f10dfbf4427916e3a133f893ce03777025ff37b05b25761edf9fa311bf1"} Oct 31 04:13:01.198271 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:01.198191 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c44pm" podStartSLOduration=1.988285074 podStartE2EDuration="2.198174185s" totalImagesPullingTime="209.889111ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:12:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:13:01.196373101 +0000 UTC m=+535.952711142" watchObservedRunningTime="2026-10-31 04:13:01.198174185 +0000 UTC m=+535.954512225" Oct 31 04:13:01.682399 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:01.682361 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c44pm" Oct 31 04:13:01.692488 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:01.692455 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cv556"] Oct 31 04:13:01.696072 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:01.696048 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cv556" Oct 31 04:13:01.711180 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:01.711154 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 31 04:13:01.726782 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:01.726751 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cv556"] Oct 31 04:13:01.832702 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:01.832670 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63726a00-18c8-4056-b970-ba1809e2cfc4-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cv556\" (UID: \"63726a00-18c8-4056-b970-ba1809e2cfc4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cv556" Oct 31 04:13:01.832884 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:01.832748 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63726a00-18c8-4056-b970-ba1809e2cfc4-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cv556\" (UID: \"63726a00-18c8-4056-b970-ba1809e2cfc4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cv556" Oct 31 04:13:01.832884 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:01.832866 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63726a00-18c8-4056-b970-ba1809e2cfc4-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cv556\" (UID: \"63726a00-18c8-4056-b970-ba1809e2cfc4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cv556" Oct 31 04:13:01.832984 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:01.832895 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63726a00-18c8-4056-b970-ba1809e2cfc4-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cv556\" (UID: \"63726a00-18c8-4056-b970-ba1809e2cfc4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cv556" Oct 31 04:13:01.832984 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:01.832927 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74hvv\" (UniqueName: \"kubernetes.io/projected/63726a00-18c8-4056-b970-ba1809e2cfc4-kube-api-access-74hvv\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cv556\" (UID: \"63726a00-18c8-4056-b970-ba1809e2cfc4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cv556" Oct 31 04:13:01.833074 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:01.833047 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63726a00-18c8-4056-b970-ba1809e2cfc4-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cv556\" (UID: \"63726a00-18c8-4056-b970-ba1809e2cfc4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cv556" Oct 31 04:13:01.934343 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:01.934237 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63726a00-18c8-4056-b970-ba1809e2cfc4-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cv556\" (UID: \"63726a00-18c8-4056-b970-ba1809e2cfc4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cv556" Oct 31 04:13:01.934500 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:01.934382 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63726a00-18c8-4056-b970-ba1809e2cfc4-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cv556\" (UID: \"63726a00-18c8-4056-b970-ba1809e2cfc4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cv556" Oct 31 04:13:01.934500 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:01.934417 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63726a00-18c8-4056-b970-ba1809e2cfc4-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cv556\" (UID: \"63726a00-18c8-4056-b970-ba1809e2cfc4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cv556" Oct 31 04:13:01.934500 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:01.934459 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74hvv\" (UniqueName: \"kubernetes.io/projected/63726a00-18c8-4056-b970-ba1809e2cfc4-kube-api-access-74hvv\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cv556\" (UID: \"63726a00-18c8-4056-b970-ba1809e2cfc4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cv556" Oct 31 04:13:01.934500 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:01.934487 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63726a00-18c8-4056-b970-ba1809e2cfc4-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cv556\" (UID: \"63726a00-18c8-4056-b970-ba1809e2cfc4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cv556" Oct 31 04:13:01.934702 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:01.934516 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63726a00-18c8-4056-b970-ba1809e2cfc4-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cv556\" (UID: \"63726a00-18c8-4056-b970-ba1809e2cfc4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cv556" Oct 31 04:13:01.934794 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:01.934765 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63726a00-18c8-4056-b970-ba1809e2cfc4-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cv556\" (UID: \"63726a00-18c8-4056-b970-ba1809e2cfc4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cv556" Oct 31 04:13:01.934917 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:01.934772 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63726a00-18c8-4056-b970-ba1809e2cfc4-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cv556\" (UID: \"63726a00-18c8-4056-b970-ba1809e2cfc4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cv556" Oct 31 04:13:01.934917 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:01.934862 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63726a00-18c8-4056-b970-ba1809e2cfc4-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cv556\" (UID: \"63726a00-18c8-4056-b970-ba1809e2cfc4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cv556" Oct 31 04:13:01.936778 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:01.936753 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63726a00-18c8-4056-b970-ba1809e2cfc4-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cv556\" (UID: \"63726a00-18c8-4056-b970-ba1809e2cfc4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cv556" Oct 31 04:13:01.937188 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:01.937165 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63726a00-18c8-4056-b970-ba1809e2cfc4-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cv556\" (UID: \"63726a00-18c8-4056-b970-ba1809e2cfc4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cv556" Oct 31 04:13:01.963917 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:01.963886 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74hvv\" (UniqueName: \"kubernetes.io/projected/63726a00-18c8-4056-b970-ba1809e2cfc4-kube-api-access-74hvv\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cv556\" (UID: \"63726a00-18c8-4056-b970-ba1809e2cfc4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cv556" Oct 31 04:13:02.012735 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:02.012688 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cv556" Oct 31 04:13:02.317788 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:02.317750 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cv556"] Oct 31 04:13:02.319150 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:13:02.319121 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63726a00_18c8_4056_b970_ba1809e2cfc4.slice/crio-2d4b6e28067c45ea0abbb4fb62d43d91a68de21e5908c6163a9cff07a69c0ef7 WatchSource:0}: Error finding container 2d4b6e28067c45ea0abbb4fb62d43d91a68de21e5908c6163a9cff07a69c0ef7: Status 404 returned error can't find the container with id 2d4b6e28067c45ea0abbb4fb62d43d91a68de21e5908c6163a9cff07a69c0ef7 Oct 31 04:13:03.178778 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:03.178715 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cv556" event={"ID":"63726a00-18c8-4056-b970-ba1809e2cfc4","Type":"ContainerStarted","Data":"4548fb7981aa66ac9dce34cc68b5759223410787dfc66bade5c9e92b0ab4a741"} Oct 31 04:13:03.179029 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:03.178795 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cv556" event={"ID":"63726a00-18c8-4056-b970-ba1809e2cfc4","Type":"ContainerStarted","Data":"2d4b6e28067c45ea0abbb4fb62d43d91a68de21e5908c6163a9cff07a69c0ef7"} Oct 31 04:13:03.226501 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:03.226444 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cv556" podStartSLOduration=2.029940811 podStartE2EDuration="2.226428192s" totalImagesPullingTime="196.487381ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:13:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:13:03.221421248 +0000 UTC m=+537.977759318" watchObservedRunningTime="2026-10-31 04:13:03.226428192 +0000 UTC m=+537.982766231" Oct 31 04:13:03.839213 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:03.839184 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mprn9" Oct 31 04:13:03.852279 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:03.852255 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mprn9" Oct 31 04:13:04.013225 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:04.013188 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cv556" Oct 31 04:13:04.195825 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:04.195754 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mprn9" Oct 31 04:13:05.642793 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:05.642757 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9mgfd" Oct 31 04:13:05.658019 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:05.657998 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9mgfd" Oct 31 04:13:06.214595 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:06.214546 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9mgfd" Oct 31 04:13:09.682559 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:09.682526 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c44pm" Oct 31 04:13:09.694702 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:09.694676 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c44pm" Oct 31 04:13:10.233126 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:10.233099 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c44pm" Oct 31 04:13:12.013200 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:12.013157 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cv556" Oct 31 04:13:12.027344 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:12.027313 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cv556" Oct 31 04:13:12.244244 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:12.244217 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cv556" Oct 31 04:13:20.182041 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:20.182007 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-77ddb887cc-6csbj"] Oct 31 04:13:20.185617 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:20.185595 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77ddb887cc-6csbj" Oct 31 04:13:20.204737 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:20.204709 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-77ddb887cc-6csbj"] Oct 31 04:13:20.293214 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:20.293184 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/720f62c6-de2c-4269-bc48-b785891f8fec-tls-cert\") pod \"authorino-77ddb887cc-6csbj\" (UID: \"720f62c6-de2c-4269-bc48-b785891f8fec\") " pod="kuadrant-system/authorino-77ddb887cc-6csbj" Oct 31 04:13:20.293353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:20.293220 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckwd7\" (UniqueName: \"kubernetes.io/projected/720f62c6-de2c-4269-bc48-b785891f8fec-kube-api-access-ckwd7\") pod \"authorino-77ddb887cc-6csbj\" (UID: \"720f62c6-de2c-4269-bc48-b785891f8fec\") " pod="kuadrant-system/authorino-77ddb887cc-6csbj" Oct 31 04:13:20.394526 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:20.394499 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/720f62c6-de2c-4269-bc48-b785891f8fec-tls-cert\") pod \"authorino-77ddb887cc-6csbj\" (UID: \"720f62c6-de2c-4269-bc48-b785891f8fec\") " pod="kuadrant-system/authorino-77ddb887cc-6csbj" Oct 31 04:13:20.394666 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:20.394542 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckwd7\" (UniqueName: \"kubernetes.io/projected/720f62c6-de2c-4269-bc48-b785891f8fec-kube-api-access-ckwd7\") pod \"authorino-77ddb887cc-6csbj\" (UID: \"720f62c6-de2c-4269-bc48-b785891f8fec\") " pod="kuadrant-system/authorino-77ddb887cc-6csbj" Oct 31 04:13:20.396993 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:20.396968 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/720f62c6-de2c-4269-bc48-b785891f8fec-tls-cert\") pod \"authorino-77ddb887cc-6csbj\" (UID: \"720f62c6-de2c-4269-bc48-b785891f8fec\") " pod="kuadrant-system/authorino-77ddb887cc-6csbj" Oct 31 04:13:20.417613 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:20.417586 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckwd7\" (UniqueName: \"kubernetes.io/projected/720f62c6-de2c-4269-bc48-b785891f8fec-kube-api-access-ckwd7\") pod \"authorino-77ddb887cc-6csbj\" (UID: \"720f62c6-de2c-4269-bc48-b785891f8fec\") " pod="kuadrant-system/authorino-77ddb887cc-6csbj" Oct 31 04:13:20.498577 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:20.498555 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77ddb887cc-6csbj" Oct 31 04:13:20.794036 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:20.794005 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-77ddb887cc-6csbj"] Oct 31 04:13:20.795523 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:13:20.795497 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod720f62c6_de2c_4269_bc48_b785891f8fec.slice/crio-9094771f4729a02619e41fd0ba11ff1766b1b818dfdbafe2f34ebfdb7a17292a WatchSource:0}: Error finding container 9094771f4729a02619e41fd0ba11ff1766b1b818dfdbafe2f34ebfdb7a17292a: Status 404 returned error can't find the container with id 9094771f4729a02619e41fd0ba11ff1766b1b818dfdbafe2f34ebfdb7a17292a Oct 31 04:13:21.279559 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:21.279527 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77ddb887cc-6csbj" event={"ID":"720f62c6-de2c-4269-bc48-b785891f8fec","Type":"ContainerStarted","Data":"9094771f4729a02619e41fd0ba11ff1766b1b818dfdbafe2f34ebfdb7a17292a"} Oct 31 04:13:22.288357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:22.288288 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77ddb887cc-6csbj" event={"ID":"720f62c6-de2c-4269-bc48-b785891f8fec","Type":"ContainerStarted","Data":"49686b996cca596eece5e5ebfd31ff2b4fb19e0a507d9a3bb460974c28d1e167"} Oct 31 04:13:22.329654 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:22.329597 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-77ddb887cc-6csbj" podStartSLOduration=1.667158609 podStartE2EDuration="2.329579212s" totalImagesPullingTime="662.420603ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:13:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:13:22.328253211 +0000 UTC m=+557.084591255" watchObservedRunningTime="2026-10-31 04:13:22.329579212 +0000 UTC m=+557.085917351" Oct 31 04:13:22.357208 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:22.357167 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5698fc8df4-kqgsq"] Oct 31 04:13:22.357424 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:22.357398 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-5698fc8df4-kqgsq" podUID="ad2f7eeb-489b-48e2-a5c8-02e146a03cff" containerName="authorino" containerID="cri-o://ec648ac7bfd4c07b0d00297f71b7405e15a7f5795114608b31c6a2f857b2eac2" gracePeriod=30 Oct 31 04:13:22.417377 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:13:22.416558 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:13:22.621298 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:22.621269 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5698fc8df4-kqgsq" Oct 31 04:13:22.717998 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:22.717969 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad2f7eeb-489b-48e2-a5c8-02e146a03cff-tls-cert\" (UniqueName: \"kubernetes.io/secret/ad2f7eeb-489b-48e2-a5c8-02e146a03cff-tls-cert\") pod \"ad2f7eeb-489b-48e2-a5c8-02e146a03cff\" (UID: \"ad2f7eeb-489b-48e2-a5c8-02e146a03cff\") " Oct 31 04:13:22.718168 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:22.718021 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad2f7eeb-489b-48e2-a5c8-02e146a03cff-kube-api-access-fh4jr\" (UniqueName: \"kubernetes.io/projected/ad2f7eeb-489b-48e2-a5c8-02e146a03cff-kube-api-access-fh4jr\") pod \"ad2f7eeb-489b-48e2-a5c8-02e146a03cff\" (UID: \"ad2f7eeb-489b-48e2-a5c8-02e146a03cff\") " Oct 31 04:13:22.720073 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:22.720037 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad2f7eeb-489b-48e2-a5c8-02e146a03cff-kube-api-access-fh4jr" pod "ad2f7eeb-489b-48e2-a5c8-02e146a03cff" (UID: "ad2f7eeb-489b-48e2-a5c8-02e146a03cff"). InnerVolumeSpecName "kube-api-access-fh4jr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:13:22.728833 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:22.728793 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad2f7eeb-489b-48e2-a5c8-02e146a03cff-tls-cert" pod "ad2f7eeb-489b-48e2-a5c8-02e146a03cff" (UID: "ad2f7eeb-489b-48e2-a5c8-02e146a03cff"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:13:22.819461 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:22.819422 5019 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ad2f7eeb-489b-48e2-a5c8-02e146a03cff-tls-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:13:22.819461 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:22.819456 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh4jr\" (UniqueName: \"kubernetes.io/projected/ad2f7eeb-489b-48e2-a5c8-02e146a03cff-kube-api-access-fh4jr\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:13:23.296177 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:23.296142 5019 generic.go:356] "Generic (PLEG): container finished" podID="ad2f7eeb-489b-48e2-a5c8-02e146a03cff" containerID="ec648ac7bfd4c07b0d00297f71b7405e15a7f5795114608b31c6a2f857b2eac2" exitCode=0 Oct 31 04:13:23.296693 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:23.296204 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5698fc8df4-kqgsq" Oct 31 04:13:23.296693 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:23.296230 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5698fc8df4-kqgsq" event={"ID":"ad2f7eeb-489b-48e2-a5c8-02e146a03cff","Type":"ContainerDied","Data":"ec648ac7bfd4c07b0d00297f71b7405e15a7f5795114608b31c6a2f857b2eac2"} Oct 31 04:13:23.296693 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:23.296280 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5698fc8df4-kqgsq" event={"ID":"ad2f7eeb-489b-48e2-a5c8-02e146a03cff","Type":"ContainerDied","Data":"37dff8dd8b6247e6b9f8f4f7e0caa49f3c1c09775d0d5ba9cd3cd47f4bc3dcfa"} Oct 31 04:13:23.296693 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:23.296300 5019 scope.go:122] "RemoveContainer" containerID="ec648ac7bfd4c07b0d00297f71b7405e15a7f5795114608b31c6a2f857b2eac2" Oct 31 04:13:23.308060 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:23.308044 5019 scope.go:122] "RemoveContainer" containerID="ec648ac7bfd4c07b0d00297f71b7405e15a7f5795114608b31c6a2f857b2eac2" Oct 31 04:13:23.308364 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:13:23.308324 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec648ac7bfd4c07b0d00297f71b7405e15a7f5795114608b31c6a2f857b2eac2\": container with ID starting with ec648ac7bfd4c07b0d00297f71b7405e15a7f5795114608b31c6a2f857b2eac2 not found: ID does not exist" containerID="ec648ac7bfd4c07b0d00297f71b7405e15a7f5795114608b31c6a2f857b2eac2" Oct 31 04:13:23.308451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:23.308374 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec648ac7bfd4c07b0d00297f71b7405e15a7f5795114608b31c6a2f857b2eac2"} err="failed to get container status \"ec648ac7bfd4c07b0d00297f71b7405e15a7f5795114608b31c6a2f857b2eac2\": rpc error: code = NotFound desc = could not find container \"ec648ac7bfd4c07b0d00297f71b7405e15a7f5795114608b31c6a2f857b2eac2\": container with ID starting with ec648ac7bfd4c07b0d00297f71b7405e15a7f5795114608b31c6a2f857b2eac2 not found: ID does not exist" Oct 31 04:13:23.352965 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:23.352933 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5698fc8df4-kqgsq"] Oct 31 04:13:23.365135 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:23.365105 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5698fc8df4-kqgsq"] Oct 31 04:13:23.406948 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:23.406914 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad2f7eeb-489b-48e2-a5c8-02e146a03cff" path="/var/lib/kubelet/pods/ad2f7eeb-489b-48e2-a5c8-02e146a03cff/volumes" Oct 31 04:13:23.712959 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:23.712877 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-srnk2"] Oct 31 04:13:23.713618 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:23.713597 5019 cpu_manager.go:414] "containerMap: removing container" podUID="ad2f7eeb-489b-48e2-a5c8-02e146a03cff" containerName="authorino" Oct 31 04:13:23.713665 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:23.713624 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad2f7eeb-489b-48e2-a5c8-02e146a03cff" containerName="authorino" Oct 31 04:13:23.713750 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:23.713738 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad2f7eeb-489b-48e2-a5c8-02e146a03cff" containerName="authorino" Oct 31 04:13:23.718908 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:23.718880 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-srnk2" Oct 31 04:13:23.749785 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:23.749759 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-srnk2"] Oct 31 04:13:23.830030 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:23.829999 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19032517-0eb5-47e6-86f3-a2772a21b1b8-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-srnk2\" (UID: \"19032517-0eb5-47e6-86f3-a2772a21b1b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-srnk2" Oct 31 04:13:23.830199 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:23.830039 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19032517-0eb5-47e6-86f3-a2772a21b1b8-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-srnk2\" (UID: \"19032517-0eb5-47e6-86f3-a2772a21b1b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-srnk2" Oct 31 04:13:23.830199 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:23.830160 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19032517-0eb5-47e6-86f3-a2772a21b1b8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-srnk2\" (UID: \"19032517-0eb5-47e6-86f3-a2772a21b1b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-srnk2" Oct 31 04:13:23.830292 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:23.830249 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19032517-0eb5-47e6-86f3-a2772a21b1b8-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-srnk2\" (UID: \"19032517-0eb5-47e6-86f3-a2772a21b1b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-srnk2" Oct 31 04:13:23.830292 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:23.830288 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19032517-0eb5-47e6-86f3-a2772a21b1b8-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-srnk2\" (UID: \"19032517-0eb5-47e6-86f3-a2772a21b1b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-srnk2" Oct 31 04:13:23.830376 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:23.830350 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzt62\" (UniqueName: \"kubernetes.io/projected/19032517-0eb5-47e6-86f3-a2772a21b1b8-kube-api-access-gzt62\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-srnk2\" (UID: \"19032517-0eb5-47e6-86f3-a2772a21b1b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-srnk2" Oct 31 04:13:23.830611 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:23.830592 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19032517-0eb5-47e6-86f3-a2772a21b1b8-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-srnk2\" (UID: \"19032517-0eb5-47e6-86f3-a2772a21b1b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-srnk2" Oct 31 04:13:23.830647 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:23.830628 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19032517-0eb5-47e6-86f3-a2772a21b1b8-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-srnk2\" (UID: \"19032517-0eb5-47e6-86f3-a2772a21b1b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-srnk2" Oct 31 04:13:23.830707 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:23.830655 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19032517-0eb5-47e6-86f3-a2772a21b1b8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-srnk2\" (UID: \"19032517-0eb5-47e6-86f3-a2772a21b1b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-srnk2" Oct 31 04:13:23.931864 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:23.931820 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19032517-0eb5-47e6-86f3-a2772a21b1b8-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-srnk2\" (UID: \"19032517-0eb5-47e6-86f3-a2772a21b1b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-srnk2" Oct 31 04:13:23.932565 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:23.932543 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19032517-0eb5-47e6-86f3-a2772a21b1b8-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-srnk2\" (UID: \"19032517-0eb5-47e6-86f3-a2772a21b1b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-srnk2" Oct 31 04:13:23.933050 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:23.933032 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19032517-0eb5-47e6-86f3-a2772a21b1b8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-srnk2\" (UID: \"19032517-0eb5-47e6-86f3-a2772a21b1b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-srnk2" Oct 31 04:13:23.933954 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:23.933935 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19032517-0eb5-47e6-86f3-a2772a21b1b8-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-srnk2\" (UID: \"19032517-0eb5-47e6-86f3-a2772a21b1b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-srnk2" Oct 31 04:13:23.934385 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:23.934368 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19032517-0eb5-47e6-86f3-a2772a21b1b8-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-srnk2\" (UID: \"19032517-0eb5-47e6-86f3-a2772a21b1b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-srnk2" Oct 31 04:13:23.934547 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:23.934527 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19032517-0eb5-47e6-86f3-a2772a21b1b8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-srnk2\" (UID: \"19032517-0eb5-47e6-86f3-a2772a21b1b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-srnk2" Oct 31 04:13:23.934696 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:23.934570 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19032517-0eb5-47e6-86f3-a2772a21b1b8-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-srnk2\" (UID: \"19032517-0eb5-47e6-86f3-a2772a21b1b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-srnk2" Oct 31 04:13:23.934696 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:23.934623 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19032517-0eb5-47e6-86f3-a2772a21b1b8-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-srnk2\" (UID: \"19032517-0eb5-47e6-86f3-a2772a21b1b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-srnk2" Oct 31 04:13:23.934696 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:23.934650 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzt62\" (UniqueName: \"kubernetes.io/projected/19032517-0eb5-47e6-86f3-a2772a21b1b8-kube-api-access-gzt62\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-srnk2\" (UID: \"19032517-0eb5-47e6-86f3-a2772a21b1b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-srnk2" Oct 31 04:13:23.934993 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:23.932479 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19032517-0eb5-47e6-86f3-a2772a21b1b8-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-srnk2\" (UID: \"19032517-0eb5-47e6-86f3-a2772a21b1b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-srnk2" Oct 31 04:13:23.935210 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:23.933789 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19032517-0eb5-47e6-86f3-a2772a21b1b8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-srnk2\" (UID: \"19032517-0eb5-47e6-86f3-a2772a21b1b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-srnk2" Oct 31 04:13:23.935754 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:23.934300 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19032517-0eb5-47e6-86f3-a2772a21b1b8-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-srnk2\" (UID: \"19032517-0eb5-47e6-86f3-a2772a21b1b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-srnk2" Oct 31 04:13:23.935879 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:23.932983 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19032517-0eb5-47e6-86f3-a2772a21b1b8-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-srnk2\" (UID: \"19032517-0eb5-47e6-86f3-a2772a21b1b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-srnk2" Oct 31 04:13:23.935976 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:23.935933 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19032517-0eb5-47e6-86f3-a2772a21b1b8-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-srnk2\" (UID: \"19032517-0eb5-47e6-86f3-a2772a21b1b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-srnk2" Oct 31 04:13:23.937912 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:23.937889 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19032517-0eb5-47e6-86f3-a2772a21b1b8-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-srnk2\" (UID: \"19032517-0eb5-47e6-86f3-a2772a21b1b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-srnk2" Oct 31 04:13:23.938192 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:23.938173 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19032517-0eb5-47e6-86f3-a2772a21b1b8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-srnk2\" (UID: \"19032517-0eb5-47e6-86f3-a2772a21b1b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-srnk2" Oct 31 04:13:23.961324 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:23.961299 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19032517-0eb5-47e6-86f3-a2772a21b1b8-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-srnk2\" (UID: \"19032517-0eb5-47e6-86f3-a2772a21b1b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-srnk2" Oct 31 04:13:23.961442 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:23.961370 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzt62\" (UniqueName: \"kubernetes.io/projected/19032517-0eb5-47e6-86f3-a2772a21b1b8-kube-api-access-gzt62\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-srnk2\" (UID: \"19032517-0eb5-47e6-86f3-a2772a21b1b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-srnk2" Oct 31 04:13:24.035554 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:24.035517 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-srnk2" Oct 31 04:13:24.343374 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:24.343310 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-srnk2"] Oct 31 04:13:24.346385 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:13:24.346353 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19032517_0eb5_47e6_86f3_a2772a21b1b8.slice/crio-ca9e81705b63d42254e5674a4b7d2856000bcfc34a5e8e4a66394e0e2c7bc72e WatchSource:0}: Error finding container ca9e81705b63d42254e5674a4b7d2856000bcfc34a5e8e4a66394e0e2c7bc72e: Status 404 returned error can't find the container with id ca9e81705b63d42254e5674a4b7d2856000bcfc34a5e8e4a66394e0e2c7bc72e Oct 31 04:13:24.351059 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:24.351023 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:13:24.351177 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:24.351088 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:13:24.351177 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:24.351115 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:13:25.321146 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:25.320153 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-srnk2" event={"ID":"19032517-0eb5-47e6-86f3-a2772a21b1b8","Type":"ContainerStarted","Data":"37a02397ad9328319794fc667f46249918a42440b5fe7af0bcc2c6e2d2c98cfe"} Oct 31 04:13:25.321146 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:25.320255 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-srnk2" event={"ID":"19032517-0eb5-47e6-86f3-a2772a21b1b8","Type":"ContainerStarted","Data":"ca9e81705b63d42254e5674a4b7d2856000bcfc34a5e8e4a66394e0e2c7bc72e"} Oct 31 04:13:26.035886 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:26.035850 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-srnk2" Oct 31 04:13:26.037594 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:26.037543 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-srnk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 31 04:13:26.037594 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:26.037590 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-srnk2" podUID="19032517-0eb5-47e6-86f3-a2772a21b1b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 31 04:13:27.036456 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:27.036414 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-srnk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 31 04:13:27.037139 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:27.036487 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-srnk2" podUID="19032517-0eb5-47e6-86f3-a2772a21b1b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 31 04:13:28.036456 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:28.036421 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-srnk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 31 04:13:28.036982 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:28.036502 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-srnk2" podUID="19032517-0eb5-47e6-86f3-a2772a21b1b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 31 04:13:29.036504 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:29.036468 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-srnk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 31 04:13:29.036995 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:29.036520 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-srnk2" podUID="19032517-0eb5-47e6-86f3-a2772a21b1b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 31 04:13:30.036181 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:30.036142 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-srnk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 31 04:13:30.036392 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:30.036199 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-srnk2" podUID="19032517-0eb5-47e6-86f3-a2772a21b1b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 31 04:13:31.036459 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:31.036416 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-srnk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 31 04:13:31.036961 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:31.036479 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-srnk2" podUID="19032517-0eb5-47e6-86f3-a2772a21b1b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 31 04:13:32.740769 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:32.740740 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-srnk2" Oct 31 04:13:32.741175 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:32.741115 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-srnk2" Oct 31 04:13:32.742003 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:32.741975 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-srnk2" Oct 31 04:13:32.804839 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:32.804789 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-srnk2" podStartSLOduration=9.804777514 podStartE2EDuration="9.804777514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:13:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:13:25.420476768 +0000 UTC m=+560.176814809" watchObservedRunningTime="2026-10-31 04:13:32.804777514 +0000 UTC m=+567.561115590" Oct 31 04:13:38.760109 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:38.760076 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x"] Oct 31 04:13:38.766321 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:38.766294 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x" Oct 31 04:13:38.796965 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:38.796938 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x"] Oct 31 04:13:38.811303 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:38.811281 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-p6f5x\" (UID: \"9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x" Oct 31 04:13:38.811434 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:38.811313 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-p6f5x\" (UID: \"9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x" Oct 31 04:13:38.811478 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:38.811437 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-p6f5x\" (UID: \"9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x" Oct 31 04:13:38.811523 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:38.811492 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-p6f5x\" (UID: \"9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x" Oct 31 04:13:38.811569 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:38.811552 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mkg6\" (UniqueName: \"kubernetes.io/projected/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-kube-api-access-8mkg6\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-p6f5x\" (UID: \"9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x" Oct 31 04:13:38.811613 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:38.811601 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-p6f5x\" (UID: \"9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x" Oct 31 04:13:38.811647 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:38.811630 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-p6f5x\" (UID: \"9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x" Oct 31 04:13:38.811698 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:38.811681 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-p6f5x\" (UID: \"9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x" Oct 31 04:13:38.811732 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:38.811723 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-p6f5x\" (UID: \"9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x" Oct 31 04:13:38.912641 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:38.912610 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-p6f5x\" (UID: \"9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x" Oct 31 04:13:38.912734 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:38.912654 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-p6f5x\" (UID: \"9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x" Oct 31 04:13:38.912734 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:38.912681 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-p6f5x\" (UID: \"9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x" Oct 31 04:13:38.912734 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:38.912717 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-p6f5x\" (UID: \"9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x" Oct 31 04:13:38.912839 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:38.912762 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-p6f5x\" (UID: \"9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x" Oct 31 04:13:38.912928 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:38.912890 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mkg6\" (UniqueName: \"kubernetes.io/projected/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-kube-api-access-8mkg6\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-p6f5x\" (UID: \"9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x" Oct 31 04:13:38.913007 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:38.912988 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-p6f5x\" (UID: \"9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x" Oct 31 04:13:38.913064 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:38.913034 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-p6f5x\" (UID: \"9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x" Oct 31 04:13:38.913116 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:38.913062 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-p6f5x\" (UID: \"9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x" Oct 31 04:13:38.913116 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:38.913072 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-p6f5x\" (UID: \"9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x" Oct 31 04:13:38.913326 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:38.913291 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-p6f5x\" (UID: \"9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x" Oct 31 04:13:38.913470 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:38.913440 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-p6f5x\" (UID: \"9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x" Oct 31 04:13:38.913558 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:38.913535 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-p6f5x\" (UID: \"9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x" Oct 31 04:13:38.913607 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:38.913532 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-p6f5x\" (UID: \"9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x" Oct 31 04:13:38.915227 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:38.915195 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-p6f5x\" (UID: \"9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x" Oct 31 04:13:38.915397 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:38.915375 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-p6f5x\" (UID: \"9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x" Oct 31 04:13:38.935788 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:38.935769 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-p6f5x\" (UID: \"9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x" Oct 31 04:13:38.936019 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:38.936002 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mkg6\" (UniqueName: \"kubernetes.io/projected/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-kube-api-access-8mkg6\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-p6f5x\" (UID: \"9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x" Oct 31 04:13:39.082361 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:39.082283 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x" Oct 31 04:13:39.392551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:39.392524 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x"] Oct 31 04:13:39.394557 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:13:39.394530 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f67449a_4ed1_4c5e_bdad_decbb3b1b6cb.slice/crio-769a9d91d01bfc482c31e09e74101c1d71b1e079b8e0adbf9f2c83b3d33f6f06 WatchSource:0}: Error finding container 769a9d91d01bfc482c31e09e74101c1d71b1e079b8e0adbf9f2c83b3d33f6f06: Status 404 returned error can't find the container with id 769a9d91d01bfc482c31e09e74101c1d71b1e079b8e0adbf9f2c83b3d33f6f06 Oct 31 04:13:39.396799 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:39.396763 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:13:39.396876 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:39.396832 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:13:39.396876 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:39.396864 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:13:40.405566 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:40.405533 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x" event={"ID":"9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb","Type":"ContainerStarted","Data":"1f1b0319462ddc3cbb721c9c282e88d9d333816bf3ecf1b1049d48af5dc00cfa"} Oct 31 04:13:40.405972 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:40.405586 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x" event={"ID":"9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb","Type":"ContainerStarted","Data":"769a9d91d01bfc482c31e09e74101c1d71b1e079b8e0adbf9f2c83b3d33f6f06"} Oct 31 04:13:40.463499 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:40.463449 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x" podStartSLOduration=2.463436217 podStartE2EDuration="2.463436217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:13:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:13:40.462605546 +0000 UTC m=+575.218943586" watchObservedRunningTime="2026-10-31 04:13:40.463436217 +0000 UTC m=+575.219774258" Oct 31 04:13:41.092355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:41.085059 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x" Oct 31 04:13:41.092355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:41.085178 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 31 04:13:41.092355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:41.085223 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x" podUID="9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 31 04:13:42.093356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:42.087665 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 31 04:13:42.093356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:42.087726 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x" podUID="9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 31 04:13:43.091356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:43.090325 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 31 04:13:43.091356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:43.090409 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x" podUID="9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 31 04:13:44.090356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:44.088648 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 31 04:13:44.090356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:44.088709 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x" podUID="9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 31 04:13:45.088711 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:45.088674 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 31 04:13:45.088984 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:45.088954 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x" podUID="9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 31 04:13:46.083343 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:46.083301 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 31 04:13:46.083726 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:46.083367 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x" podUID="9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 31 04:13:47.554475 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:47.554440 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x" Oct 31 04:13:47.554953 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:47.554825 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x" Oct 31 04:13:47.555635 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:47.555616 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x" Oct 31 04:13:52.390479 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:13:52.390443 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:13:54.981682 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:54.981637 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb"] Oct 31 04:13:54.986646 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:54.986613 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb" Oct 31 04:13:55.012198 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.012158 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb"] Oct 31 04:13:55.067741 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.067698 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec1714c9-5e99-400a-b392-486c04926361-istiod-ca-cert\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb\" (UID: \"ec1714c9-5e99-400a-b392-486c04926361\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb" Oct 31 04:13:55.067741 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.067740 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec1714c9-5e99-400a-b392-486c04926361-workload-certs\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb\" (UID: \"ec1714c9-5e99-400a-b392-486c04926361\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb" Oct 31 04:13:55.067964 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.067850 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec1714c9-5e99-400a-b392-486c04926361-istio-envoy\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb\" (UID: \"ec1714c9-5e99-400a-b392-486c04926361\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb" Oct 31 04:13:55.067964 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.067879 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec1714c9-5e99-400a-b392-486c04926361-istio-data\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb\" (UID: \"ec1714c9-5e99-400a-b392-486c04926361\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb" Oct 31 04:13:55.067964 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.067927 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7nvl\" (UniqueName: \"kubernetes.io/projected/ec1714c9-5e99-400a-b392-486c04926361-kube-api-access-b7nvl\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb\" (UID: \"ec1714c9-5e99-400a-b392-486c04926361\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb" Oct 31 04:13:55.068064 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.068019 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1714c9-5e99-400a-b392-486c04926361-credential-socket\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb\" (UID: \"ec1714c9-5e99-400a-b392-486c04926361\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb" Oct 31 04:13:55.068098 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.068064 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec1714c9-5e99-400a-b392-486c04926361-istio-podinfo\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb\" (UID: \"ec1714c9-5e99-400a-b392-486c04926361\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb" Oct 31 04:13:55.068098 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.068094 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec1714c9-5e99-400a-b392-486c04926361-istio-token\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb\" (UID: \"ec1714c9-5e99-400a-b392-486c04926361\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb" Oct 31 04:13:55.068192 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.068172 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1714c9-5e99-400a-b392-486c04926361-workload-socket\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb\" (UID: \"ec1714c9-5e99-400a-b392-486c04926361\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb" Oct 31 04:13:55.073721 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.073693 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-44f032ad-openshift-default-dockercfg-qx5tv\"" Oct 31 04:13:55.169046 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.169008 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1714c9-5e99-400a-b392-486c04926361-workload-socket\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb\" (UID: \"ec1714c9-5e99-400a-b392-486c04926361\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb" Oct 31 04:13:55.169212 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.169067 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec1714c9-5e99-400a-b392-486c04926361-istiod-ca-cert\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb\" (UID: \"ec1714c9-5e99-400a-b392-486c04926361\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb" Oct 31 04:13:55.169212 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.169099 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec1714c9-5e99-400a-b392-486c04926361-workload-certs\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb\" (UID: \"ec1714c9-5e99-400a-b392-486c04926361\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb" Oct 31 04:13:55.169212 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.169138 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec1714c9-5e99-400a-b392-486c04926361-istio-envoy\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb\" (UID: \"ec1714c9-5e99-400a-b392-486c04926361\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb" Oct 31 04:13:55.169212 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.169161 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec1714c9-5e99-400a-b392-486c04926361-istio-data\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb\" (UID: \"ec1714c9-5e99-400a-b392-486c04926361\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb" Oct 31 04:13:55.169477 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.169214 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7nvl\" (UniqueName: \"kubernetes.io/projected/ec1714c9-5e99-400a-b392-486c04926361-kube-api-access-b7nvl\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb\" (UID: \"ec1714c9-5e99-400a-b392-486c04926361\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb" Oct 31 04:13:55.169477 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.169260 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1714c9-5e99-400a-b392-486c04926361-credential-socket\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb\" (UID: \"ec1714c9-5e99-400a-b392-486c04926361\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb" Oct 31 04:13:55.169477 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.169300 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec1714c9-5e99-400a-b392-486c04926361-istio-podinfo\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb\" (UID: \"ec1714c9-5e99-400a-b392-486c04926361\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb" Oct 31 04:13:55.169477 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.169380 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec1714c9-5e99-400a-b392-486c04926361-istio-token\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb\" (UID: \"ec1714c9-5e99-400a-b392-486c04926361\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb" Oct 31 04:13:55.169685 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.169590 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec1714c9-5e99-400a-b392-486c04926361-workload-certs\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb\" (UID: \"ec1714c9-5e99-400a-b392-486c04926361\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb" Oct 31 04:13:55.170151 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.169817 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1714c9-5e99-400a-b392-486c04926361-workload-socket\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb\" (UID: \"ec1714c9-5e99-400a-b392-486c04926361\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb" Oct 31 04:13:55.170151 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.169941 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec1714c9-5e99-400a-b392-486c04926361-istio-data\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb\" (UID: \"ec1714c9-5e99-400a-b392-486c04926361\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb" Oct 31 04:13:55.170151 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.170091 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1714c9-5e99-400a-b392-486c04926361-credential-socket\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb\" (UID: \"ec1714c9-5e99-400a-b392-486c04926361\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb" Oct 31 04:13:55.170151 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.170094 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec1714c9-5e99-400a-b392-486c04926361-istiod-ca-cert\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb\" (UID: \"ec1714c9-5e99-400a-b392-486c04926361\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb" Oct 31 04:13:55.171667 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.171644 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec1714c9-5e99-400a-b392-486c04926361-istio-envoy\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb\" (UID: \"ec1714c9-5e99-400a-b392-486c04926361\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb" Oct 31 04:13:55.172177 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.172147 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec1714c9-5e99-400a-b392-486c04926361-istio-podinfo\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb\" (UID: \"ec1714c9-5e99-400a-b392-486c04926361\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb" Oct 31 04:13:55.227064 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.226986 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec1714c9-5e99-400a-b392-486c04926361-istio-token\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb\" (UID: \"ec1714c9-5e99-400a-b392-486c04926361\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb" Oct 31 04:13:55.227980 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.227948 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7nvl\" (UniqueName: \"kubernetes.io/projected/ec1714c9-5e99-400a-b392-486c04926361-kube-api-access-b7nvl\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb\" (UID: \"ec1714c9-5e99-400a-b392-486c04926361\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb" Oct 31 04:13:55.250743 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.250655 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n"] Oct 31 04:13:55.257299 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.257273 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n" Oct 31 04:13:55.281347 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.281291 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n"] Oct 31 04:13:55.303294 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.303244 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb" Oct 31 04:13:55.313391 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.312403 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-a0d1a61d-gw-openshift-default-dockercfg-xkccn\"" Oct 31 04:13:55.372239 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.371546 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9qf5\" (UniqueName: \"kubernetes.io/projected/6b943d4d-c829-410c-9233-0b53cbc1c414-kube-api-access-f9qf5\") pod \"e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n\" (UID: \"6b943d4d-c829-410c-9233-0b53cbc1c414\") " pod="openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n" Oct 31 04:13:55.372239 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.371636 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b943d4d-c829-410c-9233-0b53cbc1c414-istio-envoy\") pod \"e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n\" (UID: \"6b943d4d-c829-410c-9233-0b53cbc1c414\") " pod="openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n" Oct 31 04:13:55.372239 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.371664 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b943d4d-c829-410c-9233-0b53cbc1c414-workload-socket\") pod \"e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n\" (UID: \"6b943d4d-c829-410c-9233-0b53cbc1c414\") " pod="openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n" Oct 31 04:13:55.372239 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.371680 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b943d4d-c829-410c-9233-0b53cbc1c414-workload-certs\") pod \"e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n\" (UID: \"6b943d4d-c829-410c-9233-0b53cbc1c414\") " pod="openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n" Oct 31 04:13:55.372239 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.371706 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b943d4d-c829-410c-9233-0b53cbc1c414-credential-socket\") pod \"e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n\" (UID: \"6b943d4d-c829-410c-9233-0b53cbc1c414\") " pod="openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n" Oct 31 04:13:55.372239 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.371730 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b943d4d-c829-410c-9233-0b53cbc1c414-istio-podinfo\") pod \"e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n\" (UID: \"6b943d4d-c829-410c-9233-0b53cbc1c414\") " pod="openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n" Oct 31 04:13:55.372239 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.371747 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b943d4d-c829-410c-9233-0b53cbc1c414-istio-token\") pod \"e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n\" (UID: \"6b943d4d-c829-410c-9233-0b53cbc1c414\") " pod="openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n" Oct 31 04:13:55.372239 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.371763 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b943d4d-c829-410c-9233-0b53cbc1c414-istiod-ca-cert\") pod \"e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n\" (UID: \"6b943d4d-c829-410c-9233-0b53cbc1c414\") " pod="openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n" Oct 31 04:13:55.372239 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.371792 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b943d4d-c829-410c-9233-0b53cbc1c414-istio-data\") pod \"e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n\" (UID: \"6b943d4d-c829-410c-9233-0b53cbc1c414\") " pod="openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n" Oct 31 04:13:55.475532 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.473059 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b943d4d-c829-410c-9233-0b53cbc1c414-workload-socket\") pod \"e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n\" (UID: \"6b943d4d-c829-410c-9233-0b53cbc1c414\") " pod="openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n" Oct 31 04:13:55.475532 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.473123 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b943d4d-c829-410c-9233-0b53cbc1c414-workload-certs\") pod \"e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n\" (UID: \"6b943d4d-c829-410c-9233-0b53cbc1c414\") " pod="openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n" Oct 31 04:13:55.475532 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.473179 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b943d4d-c829-410c-9233-0b53cbc1c414-credential-socket\") pod \"e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n\" (UID: \"6b943d4d-c829-410c-9233-0b53cbc1c414\") " pod="openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n" Oct 31 04:13:55.475532 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.473225 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b943d4d-c829-410c-9233-0b53cbc1c414-istio-podinfo\") pod \"e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n\" (UID: \"6b943d4d-c829-410c-9233-0b53cbc1c414\") " pod="openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n" Oct 31 04:13:55.475532 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.473256 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b943d4d-c829-410c-9233-0b53cbc1c414-istio-token\") pod \"e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n\" (UID: \"6b943d4d-c829-410c-9233-0b53cbc1c414\") " pod="openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n" Oct 31 04:13:55.475532 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.473293 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b943d4d-c829-410c-9233-0b53cbc1c414-istiod-ca-cert\") pod \"e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n\" (UID: \"6b943d4d-c829-410c-9233-0b53cbc1c414\") " pod="openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n" Oct 31 04:13:55.475532 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.473390 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b943d4d-c829-410c-9233-0b53cbc1c414-istio-data\") pod \"e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n\" (UID: \"6b943d4d-c829-410c-9233-0b53cbc1c414\") " pod="openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n" Oct 31 04:13:55.475532 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.473463 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9qf5\" (UniqueName: \"kubernetes.io/projected/6b943d4d-c829-410c-9233-0b53cbc1c414-kube-api-access-f9qf5\") pod \"e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n\" (UID: \"6b943d4d-c829-410c-9233-0b53cbc1c414\") " pod="openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n" Oct 31 04:13:55.475532 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.473585 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b943d4d-c829-410c-9233-0b53cbc1c414-istio-envoy\") pod \"e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n\" (UID: \"6b943d4d-c829-410c-9233-0b53cbc1c414\") " pod="openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n" Oct 31 04:13:55.475532 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.474526 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b943d4d-c829-410c-9233-0b53cbc1c414-workload-socket\") pod \"e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n\" (UID: \"6b943d4d-c829-410c-9233-0b53cbc1c414\") " pod="openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n" Oct 31 04:13:55.475532 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.474804 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b943d4d-c829-410c-9233-0b53cbc1c414-workload-certs\") pod \"e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n\" (UID: \"6b943d4d-c829-410c-9233-0b53cbc1c414\") " pod="openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n" Oct 31 04:13:55.475532 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.475055 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b943d4d-c829-410c-9233-0b53cbc1c414-credential-socket\") pod \"e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n\" (UID: \"6b943d4d-c829-410c-9233-0b53cbc1c414\") " pod="openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n" Oct 31 04:13:55.477678 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.477648 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b943d4d-c829-410c-9233-0b53cbc1c414-istiod-ca-cert\") pod \"e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n\" (UID: \"6b943d4d-c829-410c-9233-0b53cbc1c414\") " pod="openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n" Oct 31 04:13:55.478439 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.478412 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b943d4d-c829-410c-9233-0b53cbc1c414-istio-data\") pod \"e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n\" (UID: \"6b943d4d-c829-410c-9233-0b53cbc1c414\") " pod="openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n" Oct 31 04:13:55.485304 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.485266 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b943d4d-c829-410c-9233-0b53cbc1c414-istio-podinfo\") pod \"e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n\" (UID: \"6b943d4d-c829-410c-9233-0b53cbc1c414\") " pod="openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n" Oct 31 04:13:55.488241 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.488203 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b943d4d-c829-410c-9233-0b53cbc1c414-istio-envoy\") pod \"e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n\" (UID: \"6b943d4d-c829-410c-9233-0b53cbc1c414\") " pod="openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n" Oct 31 04:13:55.500422 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.500379 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b943d4d-c829-410c-9233-0b53cbc1c414-istio-token\") pod \"e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n\" (UID: \"6b943d4d-c829-410c-9233-0b53cbc1c414\") " pod="openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n" Oct 31 04:13:55.500608 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.500558 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9qf5\" (UniqueName: \"kubernetes.io/projected/6b943d4d-c829-410c-9233-0b53cbc1c414-kube-api-access-f9qf5\") pod \"e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n\" (UID: \"6b943d4d-c829-410c-9233-0b53cbc1c414\") " pod="openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n" Oct 31 04:13:55.575534 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.575432 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n" Oct 31 04:13:55.632633 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.632359 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb"] Oct 31 04:13:55.635026 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:13:55.634996 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec1714c9_5e99_400a_b392_486c04926361.slice/crio-b84833c23487a75dcab5f515d1139451c7b1317fa86678a9dc985e52d10df579 WatchSource:0}: Error finding container b84833c23487a75dcab5f515d1139451c7b1317fa86678a9dc985e52d10df579: Status 404 returned error can't find the container with id b84833c23487a75dcab5f515d1139451c7b1317fa86678a9dc985e52d10df579 Oct 31 04:13:55.638445 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.638394 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:13:55.638575 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.638532 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:13:55.638575 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.638568 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:13:55.905266 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.905231 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n"] Oct 31 04:13:55.908059 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:13:55.908019 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b943d4d_c829_410c_9233_0b53cbc1c414.slice/crio-606b24b8f7b3663d2cef4e9865185002f08a8393e293906bc98b159cd51b7177 WatchSource:0}: Error finding container 606b24b8f7b3663d2cef4e9865185002f08a8393e293906bc98b159cd51b7177: Status 404 returned error can't find the container with id 606b24b8f7b3663d2cef4e9865185002f08a8393e293906bc98b159cd51b7177 Oct 31 04:13:55.911145 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.911104 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:13:55.911253 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.911184 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:13:55.911253 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:55.911214 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:13:56.505397 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:56.505325 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n" event={"ID":"6b943d4d-c829-410c-9233-0b53cbc1c414","Type":"ContainerStarted","Data":"e648ace0ce8cd6ec13266002b01c48ae54052a86bbe33a3c08ce77fe41873aa0"} Oct 31 04:13:56.505941 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:56.505421 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n" event={"ID":"6b943d4d-c829-410c-9233-0b53cbc1c414","Type":"ContainerStarted","Data":"606b24b8f7b3663d2cef4e9865185002f08a8393e293906bc98b159cd51b7177"} Oct 31 04:13:56.507146 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:56.507115 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb" event={"ID":"ec1714c9-5e99-400a-b392-486c04926361","Type":"ContainerStarted","Data":"9753078996d2416ba0e209fc2cc4eba325b375a272c80f14733af5bdbbf2c45f"} Oct 31 04:13:56.507267 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:56.507166 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb" event={"ID":"ec1714c9-5e99-400a-b392-486c04926361","Type":"ContainerStarted","Data":"b84833c23487a75dcab5f515d1139451c7b1317fa86678a9dc985e52d10df579"} Oct 31 04:13:56.553559 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:56.553480 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n" podStartSLOduration=1.553463368 podStartE2EDuration="1.553463368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:13:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:13:56.549814021 +0000 UTC m=+591.306152086" watchObservedRunningTime="2026-10-31 04:13:56.553463368 +0000 UTC m=+591.309801479" Oct 31 04:13:56.575766 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:56.575729 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n" Oct 31 04:13:56.581606 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:56.581574 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n" Oct 31 04:13:56.598618 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:56.598546 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb" podStartSLOduration=2.598526483 podStartE2EDuration="2.598526483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:13:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:13:56.595838018 +0000 UTC m=+591.352176060" watchObservedRunningTime="2026-10-31 04:13:56.598526483 +0000 UTC m=+591.354864600" Oct 31 04:13:57.304112 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:57.304066 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb" Oct 31 04:13:57.309871 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:57.309833 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb" Oct 31 04:13:57.513364 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:57.513310 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n" Oct 31 04:13:57.513881 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:57.513538 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb" Oct 31 04:13:57.514694 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:57.514661 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n" Oct 31 04:13:57.515077 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:13:57.515061 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb" Oct 31 04:14:01.154716 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:01.154674 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a0d1a61d-6499787965-w979c"] Oct 31 04:14:01.160818 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:01.160734 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-a0d1a61d-6499787965-w979c" Oct 31 04:14:01.174378 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:01.174028 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a0d1a61d-6499787965-w979c"] Oct 31 04:14:01.182771 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:01.182737 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-a0d1a61d-dockercfg-2rwjr\"" Oct 31 04:14:01.182945 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:01.182783 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-a0d1a61d\"" Oct 31 04:14:01.243045 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:01.243001 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbsgp\" (UniqueName: \"kubernetes.io/projected/533c24d6-b721-4f7f-afea-6fea0bac4a51-kube-api-access-mbsgp\") pod \"payload-processing-e2e-migrate-a0d1a61d-6499787965-w979c\" (UID: \"533c24d6-b721-4f7f-afea-6fea0bac4a51\") " pod="openshift-ingress/payload-processing-e2e-migrate-a0d1a61d-6499787965-w979c" Oct 31 04:14:01.243246 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:01.243063 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/533c24d6-b721-4f7f-afea-6fea0bac4a51-plugins-config-volume\") pod \"payload-processing-e2e-migrate-a0d1a61d-6499787965-w979c\" (UID: \"533c24d6-b721-4f7f-afea-6fea0bac4a51\") " pod="openshift-ingress/payload-processing-e2e-migrate-a0d1a61d-6499787965-w979c" Oct 31 04:14:01.344824 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:01.344782 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbsgp\" (UniqueName: \"kubernetes.io/projected/533c24d6-b721-4f7f-afea-6fea0bac4a51-kube-api-access-mbsgp\") pod \"payload-processing-e2e-migrate-a0d1a61d-6499787965-w979c\" (UID: \"533c24d6-b721-4f7f-afea-6fea0bac4a51\") " pod="openshift-ingress/payload-processing-e2e-migrate-a0d1a61d-6499787965-w979c" Oct 31 04:14:01.344824 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:01.344839 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/533c24d6-b721-4f7f-afea-6fea0bac4a51-plugins-config-volume\") pod \"payload-processing-e2e-migrate-a0d1a61d-6499787965-w979c\" (UID: \"533c24d6-b721-4f7f-afea-6fea0bac4a51\") " pod="openshift-ingress/payload-processing-e2e-migrate-a0d1a61d-6499787965-w979c" Oct 31 04:14:01.345870 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:01.345837 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/533c24d6-b721-4f7f-afea-6fea0bac4a51-plugins-config-volume\") pod \"payload-processing-e2e-migrate-a0d1a61d-6499787965-w979c\" (UID: \"533c24d6-b721-4f7f-afea-6fea0bac4a51\") " pod="openshift-ingress/payload-processing-e2e-migrate-a0d1a61d-6499787965-w979c" Oct 31 04:14:01.369996 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:01.369954 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbsgp\" (UniqueName: \"kubernetes.io/projected/533c24d6-b721-4f7f-afea-6fea0bac4a51-kube-api-access-mbsgp\") pod \"payload-processing-e2e-migrate-a0d1a61d-6499787965-w979c\" (UID: \"533c24d6-b721-4f7f-afea-6fea0bac4a51\") " pod="openshift-ingress/payload-processing-e2e-migrate-a0d1a61d-6499787965-w979c" Oct 31 04:14:01.480492 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:01.480448 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-a0d1a61d-6499787965-w979c" Oct 31 04:14:01.793874 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:01.793840 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a0d1a61d-6499787965-w979c"] Oct 31 04:14:01.795100 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:14:01.795073 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod533c24d6_b721_4f7f_afea_6fea0bac4a51.slice/crio-de6183c95411a4b6075d37f59ab6ca6c78196e6010ce2b11e9f993255cdb163a WatchSource:0}: Error finding container de6183c95411a4b6075d37f59ab6ca6c78196e6010ce2b11e9f993255cdb163a: Status 404 returned error can't find the container with id de6183c95411a4b6075d37f59ab6ca6c78196e6010ce2b11e9f993255cdb163a Oct 31 04:14:02.569368 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:02.568871 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-a0d1a61d-6499787965-w979c" event={"ID":"533c24d6-b721-4f7f-afea-6fea0bac4a51","Type":"ContainerStarted","Data":"de6183c95411a4b6075d37f59ab6ca6c78196e6010ce2b11e9f993255cdb163a"} Oct 31 04:14:02.987731 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:02.987645 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr"] Oct 31 04:14:02.992422 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:02.992393 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr" Oct 31 04:14:03.012017 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:03.011930 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-f8a8088d-openshift-default-dockercfg-rzvwv\"" Oct 31 04:14:03.030175 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:03.030138 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr"] Oct 31 04:14:03.069746 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:03.069696 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4mhj\" (UniqueName: \"kubernetes.io/projected/3dd4051c-8d79-496e-b3cb-d90263e6f1be-kube-api-access-x4mhj\") pod \"e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr\" (UID: \"3dd4051c-8d79-496e-b3cb-d90263e6f1be\") " pod="openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr" Oct 31 04:14:03.069951 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:03.069827 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dd4051c-8d79-496e-b3cb-d90263e6f1be-istio-podinfo\") pod \"e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr\" (UID: \"3dd4051c-8d79-496e-b3cb-d90263e6f1be\") " pod="openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr" Oct 31 04:14:03.069951 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:03.069861 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dd4051c-8d79-496e-b3cb-d90263e6f1be-workload-socket\") pod \"e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr\" (UID: \"3dd4051c-8d79-496e-b3cb-d90263e6f1be\") " pod="openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr" Oct 31 04:14:03.069951 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:03.069894 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dd4051c-8d79-496e-b3cb-d90263e6f1be-istiod-ca-cert\") pod \"e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr\" (UID: \"3dd4051c-8d79-496e-b3cb-d90263e6f1be\") " pod="openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr" Oct 31 04:14:03.069951 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:03.069925 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dd4051c-8d79-496e-b3cb-d90263e6f1be-workload-certs\") pod \"e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr\" (UID: \"3dd4051c-8d79-496e-b3cb-d90263e6f1be\") " pod="openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr" Oct 31 04:14:03.070103 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:03.069986 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dd4051c-8d79-496e-b3cb-d90263e6f1be-credential-socket\") pod \"e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr\" (UID: \"3dd4051c-8d79-496e-b3cb-d90263e6f1be\") " pod="openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr" Oct 31 04:14:03.070103 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:03.070016 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dd4051c-8d79-496e-b3cb-d90263e6f1be-istio-envoy\") pod \"e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr\" (UID: \"3dd4051c-8d79-496e-b3cb-d90263e6f1be\") " pod="openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr" Oct 31 04:14:03.070103 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:03.070037 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dd4051c-8d79-496e-b3cb-d90263e6f1be-istio-data\") pod \"e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr\" (UID: \"3dd4051c-8d79-496e-b3cb-d90263e6f1be\") " pod="openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr" Oct 31 04:14:03.070103 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:03.070083 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dd4051c-8d79-496e-b3cb-d90263e6f1be-istio-token\") pod \"e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr\" (UID: \"3dd4051c-8d79-496e-b3cb-d90263e6f1be\") " pod="openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr" Oct 31 04:14:03.174916 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:03.174869 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dd4051c-8d79-496e-b3cb-d90263e6f1be-istiod-ca-cert\") pod \"e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr\" (UID: \"3dd4051c-8d79-496e-b3cb-d90263e6f1be\") " pod="openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr" Oct 31 04:14:03.175069 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:03.174937 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dd4051c-8d79-496e-b3cb-d90263e6f1be-workload-certs\") pod \"e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr\" (UID: \"3dd4051c-8d79-496e-b3cb-d90263e6f1be\") " pod="openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr" Oct 31 04:14:03.175069 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:03.174987 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dd4051c-8d79-496e-b3cb-d90263e6f1be-credential-socket\") pod \"e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr\" (UID: \"3dd4051c-8d79-496e-b3cb-d90263e6f1be\") " pod="openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr" Oct 31 04:14:03.175069 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:03.175013 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dd4051c-8d79-496e-b3cb-d90263e6f1be-istio-envoy\") pod \"e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr\" (UID: \"3dd4051c-8d79-496e-b3cb-d90263e6f1be\") " pod="openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr" Oct 31 04:14:03.175069 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:03.175035 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dd4051c-8d79-496e-b3cb-d90263e6f1be-istio-data\") pod \"e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr\" (UID: \"3dd4051c-8d79-496e-b3cb-d90263e6f1be\") " pod="openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr" Oct 31 04:14:03.175069 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:03.175066 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dd4051c-8d79-496e-b3cb-d90263e6f1be-istio-token\") pod \"e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr\" (UID: \"3dd4051c-8d79-496e-b3cb-d90263e6f1be\") " pod="openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr" Oct 31 04:14:03.175253 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:03.175087 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4mhj\" (UniqueName: \"kubernetes.io/projected/3dd4051c-8d79-496e-b3cb-d90263e6f1be-kube-api-access-x4mhj\") pod \"e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr\" (UID: \"3dd4051c-8d79-496e-b3cb-d90263e6f1be\") " pod="openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr" Oct 31 04:14:03.175253 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:03.175123 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dd4051c-8d79-496e-b3cb-d90263e6f1be-istio-podinfo\") pod \"e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr\" (UID: \"3dd4051c-8d79-496e-b3cb-d90263e6f1be\") " pod="openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr" Oct 31 04:14:03.175253 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:03.175142 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dd4051c-8d79-496e-b3cb-d90263e6f1be-workload-socket\") pod \"e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr\" (UID: \"3dd4051c-8d79-496e-b3cb-d90263e6f1be\") " pod="openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr" Oct 31 04:14:03.175559 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:03.175523 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dd4051c-8d79-496e-b3cb-d90263e6f1be-workload-socket\") pod \"e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr\" (UID: \"3dd4051c-8d79-496e-b3cb-d90263e6f1be\") " pod="openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr" Oct 31 04:14:03.175698 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:03.175683 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dd4051c-8d79-496e-b3cb-d90263e6f1be-istio-data\") pod \"e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr\" (UID: \"3dd4051c-8d79-496e-b3cb-d90263e6f1be\") " pod="openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr" Oct 31 04:14:03.175789 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:03.175740 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dd4051c-8d79-496e-b3cb-d90263e6f1be-credential-socket\") pod \"e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr\" (UID: \"3dd4051c-8d79-496e-b3cb-d90263e6f1be\") " pod="openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr" Oct 31 04:14:03.175789 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:03.175761 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dd4051c-8d79-496e-b3cb-d90263e6f1be-istiod-ca-cert\") pod \"e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr\" (UID: \"3dd4051c-8d79-496e-b3cb-d90263e6f1be\") " pod="openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr" Oct 31 04:14:03.176327 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:03.176283 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dd4051c-8d79-496e-b3cb-d90263e6f1be-workload-certs\") pod \"e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr\" (UID: \"3dd4051c-8d79-496e-b3cb-d90263e6f1be\") " pod="openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr" Oct 31 04:14:03.178604 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:03.178556 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dd4051c-8d79-496e-b3cb-d90263e6f1be-istio-envoy\") pod \"e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr\" (UID: \"3dd4051c-8d79-496e-b3cb-d90263e6f1be\") " pod="openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr" Oct 31 04:14:03.178732 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:03.178704 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dd4051c-8d79-496e-b3cb-d90263e6f1be-istio-podinfo\") pod \"e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr\" (UID: \"3dd4051c-8d79-496e-b3cb-d90263e6f1be\") " pod="openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr" Oct 31 04:14:03.201238 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:03.201200 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dd4051c-8d79-496e-b3cb-d90263e6f1be-istio-token\") pod \"e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr\" (UID: \"3dd4051c-8d79-496e-b3cb-d90263e6f1be\") " pod="openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr" Oct 31 04:14:03.201409 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:03.201357 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4mhj\" (UniqueName: \"kubernetes.io/projected/3dd4051c-8d79-496e-b3cb-d90263e6f1be-kube-api-access-x4mhj\") pod \"e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr\" (UID: \"3dd4051c-8d79-496e-b3cb-d90263e6f1be\") " pod="openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr" Oct 31 04:14:03.312723 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:03.312629 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr" Oct 31 04:14:03.663647 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:03.663604 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr"] Oct 31 04:14:03.664158 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:14:03.664126 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dd4051c_8d79_496e_b3cb_d90263e6f1be.slice/crio-d2ebaa84183620a46393bd71ce08162df1c46fead6701680f0d28f3c5cfadd75 WatchSource:0}: Error finding container d2ebaa84183620a46393bd71ce08162df1c46fead6701680f0d28f3c5cfadd75: Status 404 returned error can't find the container with id d2ebaa84183620a46393bd71ce08162df1c46fead6701680f0d28f3c5cfadd75 Oct 31 04:14:03.667073 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:03.667036 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:14:03.667187 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:03.667131 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:14:03.667187 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:03.667174 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:14:04.610939 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:04.610898 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr" event={"ID":"3dd4051c-8d79-496e-b3cb-d90263e6f1be","Type":"ContainerStarted","Data":"afce53d39f9ff4fcef00d8bc832bcecb6cf0d5396985dd3e4a85a7327ea06788"} Oct 31 04:14:04.611183 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:04.610998 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr" event={"ID":"3dd4051c-8d79-496e-b3cb-d90263e6f1be","Type":"ContainerStarted","Data":"d2ebaa84183620a46393bd71ce08162df1c46fead6701680f0d28f3c5cfadd75"} Oct 31 04:14:04.612753 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:04.612718 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-a0d1a61d-6499787965-w979c" event={"ID":"533c24d6-b721-4f7f-afea-6fea0bac4a51","Type":"ContainerStarted","Data":"082790aeda772b89b0519a5017a1e677bfc18ffd69065f169478570399d263db"} Oct 31 04:14:04.612899 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:04.612823 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-a0d1a61d-6499787965-w979c" Oct 31 04:14:04.658183 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:04.658055 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr" podStartSLOduration=2.6580383789999997 podStartE2EDuration="2.658038379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:14:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:14:04.655527685 +0000 UTC m=+599.411865727" watchObservedRunningTime="2026-10-31 04:14:04.658038379 +0000 UTC m=+599.414376419" Oct 31 04:14:04.697562 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:04.697507 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-a0d1a61d-6499787965-w979c" podStartSLOduration=1.108704913 podStartE2EDuration="3.697491309s" totalImagesPullingTime="2.588786396s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:14:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:14:04.694895746 +0000 UTC m=+599.451233785" watchObservedRunningTime="2026-10-31 04:14:04.697491309 +0000 UTC m=+599.453829348" Oct 31 04:14:05.313314 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:05.313276 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr" Oct 31 04:14:05.319583 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:05.319535 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr" Oct 31 04:14:05.511491 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:05.511462 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s88kp_3945ff65-e877-4372-9334-3968c68b7295/console-operator/2.log" Oct 31 04:14:05.513261 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:05.513229 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s88kp_3945ff65-e877-4372-9334-3968c68b7295/console-operator/2.log" Oct 31 04:14:05.619070 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:05.618983 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr" Oct 31 04:14:05.619946 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:05.619913 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr" Oct 31 04:14:08.648096 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:08.648058 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f8a8088d-858594b57f-d7j74"] Oct 31 04:14:08.653211 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:08.653188 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-f8a8088d-858594b57f-d7j74" Oct 31 04:14:08.683896 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:08.683863 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-f8a8088d-dockercfg-bfvjw\"" Oct 31 04:14:08.684250 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:08.684236 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-f8a8088d\"" Oct 31 04:14:08.705663 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:08.705619 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f8a8088d-858594b57f-d7j74"] Oct 31 04:14:08.744727 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:08.744689 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f8a8088d-7bb6787865-bmt6k"] Oct 31 04:14:08.749767 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:08.749736 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-f8a8088d-7bb6787865-bmt6k" Oct 31 04:14:08.750681 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:08.749905 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a5cbfbc-7654-4f7f-a036-c8ffd3c26819-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-f8a8088d-858594b57f-d7j74\" (UID: \"0a5cbfbc-7654-4f7f-a036-c8ffd3c26819\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f8a8088d-858594b57f-d7j74" Oct 31 04:14:08.751546 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:08.751513 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trx5n\" (UniqueName: \"kubernetes.io/projected/0a5cbfbc-7654-4f7f-a036-c8ffd3c26819-kube-api-access-trx5n\") pod \"payload-pre-processing-e2e-derive-f8a8088d-858594b57f-d7j74\" (UID: \"0a5cbfbc-7654-4f7f-a036-c8ffd3c26819\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f8a8088d-858594b57f-d7j74" Oct 31 04:14:08.807432 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:08.793798 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f8a8088d-7bb6787865-bmt6k"] Oct 31 04:14:08.852889 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:08.852843 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc918812-af5e-4f8e-9f3d-35f245dff2fd-plugins-config-volume\") pod \"payload-processing-e2e-derive-f8a8088d-7bb6787865-bmt6k\" (UID: \"dc918812-af5e-4f8e-9f3d-35f245dff2fd\") " pod="openshift-ingress/payload-processing-e2e-derive-f8a8088d-7bb6787865-bmt6k" Oct 31 04:14:08.853092 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:08.852985 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fghqr\" (UniqueName: \"kubernetes.io/projected/dc918812-af5e-4f8e-9f3d-35f245dff2fd-kube-api-access-fghqr\") pod \"payload-processing-e2e-derive-f8a8088d-7bb6787865-bmt6k\" (UID: \"dc918812-af5e-4f8e-9f3d-35f245dff2fd\") " pod="openshift-ingress/payload-processing-e2e-derive-f8a8088d-7bb6787865-bmt6k" Oct 31 04:14:08.853092 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:08.853052 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trx5n\" (UniqueName: \"kubernetes.io/projected/0a5cbfbc-7654-4f7f-a036-c8ffd3c26819-kube-api-access-trx5n\") pod \"payload-pre-processing-e2e-derive-f8a8088d-858594b57f-d7j74\" (UID: \"0a5cbfbc-7654-4f7f-a036-c8ffd3c26819\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f8a8088d-858594b57f-d7j74" Oct 31 04:14:08.853221 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:08.853131 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a5cbfbc-7654-4f7f-a036-c8ffd3c26819-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-f8a8088d-858594b57f-d7j74\" (UID: \"0a5cbfbc-7654-4f7f-a036-c8ffd3c26819\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f8a8088d-858594b57f-d7j74" Oct 31 04:14:08.853905 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:08.853871 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a5cbfbc-7654-4f7f-a036-c8ffd3c26819-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-f8a8088d-858594b57f-d7j74\" (UID: \"0a5cbfbc-7654-4f7f-a036-c8ffd3c26819\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f8a8088d-858594b57f-d7j74" Oct 31 04:14:08.898409 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:08.898109 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trx5n\" (UniqueName: \"kubernetes.io/projected/0a5cbfbc-7654-4f7f-a036-c8ffd3c26819-kube-api-access-trx5n\") pod \"payload-pre-processing-e2e-derive-f8a8088d-858594b57f-d7j74\" (UID: \"0a5cbfbc-7654-4f7f-a036-c8ffd3c26819\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f8a8088d-858594b57f-d7j74" Oct 31 04:14:08.954823 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:08.954781 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc918812-af5e-4f8e-9f3d-35f245dff2fd-plugins-config-volume\") pod \"payload-processing-e2e-derive-f8a8088d-7bb6787865-bmt6k\" (UID: \"dc918812-af5e-4f8e-9f3d-35f245dff2fd\") " pod="openshift-ingress/payload-processing-e2e-derive-f8a8088d-7bb6787865-bmt6k" Oct 31 04:14:08.955017 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:08.954880 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fghqr\" (UniqueName: \"kubernetes.io/projected/dc918812-af5e-4f8e-9f3d-35f245dff2fd-kube-api-access-fghqr\") pod \"payload-processing-e2e-derive-f8a8088d-7bb6787865-bmt6k\" (UID: \"dc918812-af5e-4f8e-9f3d-35f245dff2fd\") " pod="openshift-ingress/payload-processing-e2e-derive-f8a8088d-7bb6787865-bmt6k" Oct 31 04:14:08.955696 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:08.955671 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc918812-af5e-4f8e-9f3d-35f245dff2fd-plugins-config-volume\") pod \"payload-processing-e2e-derive-f8a8088d-7bb6787865-bmt6k\" (UID: \"dc918812-af5e-4f8e-9f3d-35f245dff2fd\") " pod="openshift-ingress/payload-processing-e2e-derive-f8a8088d-7bb6787865-bmt6k" Oct 31 04:14:08.968636 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:08.968596 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-f8a8088d-858594b57f-d7j74" Oct 31 04:14:08.996141 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:08.996043 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fghqr\" (UniqueName: \"kubernetes.io/projected/dc918812-af5e-4f8e-9f3d-35f245dff2fd-kube-api-access-fghqr\") pod \"payload-processing-e2e-derive-f8a8088d-7bb6787865-bmt6k\" (UID: \"dc918812-af5e-4f8e-9f3d-35f245dff2fd\") " pod="openshift-ingress/payload-processing-e2e-derive-f8a8088d-7bb6787865-bmt6k" Oct 31 04:14:09.111718 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:09.111666 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-f8a8088d-7bb6787865-bmt6k" Oct 31 04:14:09.306047 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:14:09.306010 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a5cbfbc_7654_4f7f_a036_c8ffd3c26819.slice/crio-028568a2d9a9f172a1a2bd912e49dfd7d32d8cbd230e250df2f7c8f191ff4e90 WatchSource:0}: Error finding container 028568a2d9a9f172a1a2bd912e49dfd7d32d8cbd230e250df2f7c8f191ff4e90: Status 404 returned error can't find the container with id 028568a2d9a9f172a1a2bd912e49dfd7d32d8cbd230e250df2f7c8f191ff4e90 Oct 31 04:14:09.306899 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:09.306831 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f8a8088d-858594b57f-d7j74"] Oct 31 04:14:09.427035 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:09.427005 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f8a8088d-7bb6787865-bmt6k"] Oct 31 04:14:09.652278 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:09.652233 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-f8a8088d-7bb6787865-bmt6k" event={"ID":"dc918812-af5e-4f8e-9f3d-35f245dff2fd","Type":"ContainerStarted","Data":"e226c0cb07356b1f620730b389be88d40a29b074281c821a5b090b1981fceca2"} Oct 31 04:14:09.652799 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:09.652309 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-f8a8088d-7bb6787865-bmt6k" event={"ID":"dc918812-af5e-4f8e-9f3d-35f245dff2fd","Type":"ContainerStarted","Data":"b732e6b14b0747323ea7a4a9b423f9d37f69da22c15013a7d368fdea153ed242"} Oct 31 04:14:09.652799 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:09.652493 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-f8a8088d-7bb6787865-bmt6k" Oct 31 04:14:09.654525 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:09.654485 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-f8a8088d-858594b57f-d7j74" event={"ID":"0a5cbfbc-7654-4f7f-a036-c8ffd3c26819","Type":"ContainerStarted","Data":"dc6c8c7598bc8a7de01825c8d63e0a17ae5d2d9ff8e41da058590440f9004bae"} Oct 31 04:14:09.654673 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:09.654543 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-f8a8088d-858594b57f-d7j74" event={"ID":"0a5cbfbc-7654-4f7f-a036-c8ffd3c26819","Type":"ContainerStarted","Data":"028568a2d9a9f172a1a2bd912e49dfd7d32d8cbd230e250df2f7c8f191ff4e90"} Oct 31 04:14:09.654673 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:09.654572 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-f8a8088d-858594b57f-d7j74" Oct 31 04:14:09.745616 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:09.745534 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-f8a8088d-858594b57f-d7j74" podStartSLOduration=1.745517092 podStartE2EDuration="1.745517092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:14:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:14:09.744294913 +0000 UTC m=+604.500632964" watchObservedRunningTime="2026-10-31 04:14:09.745517092 +0000 UTC m=+604.501855132" Oct 31 04:14:09.747884 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:09.747832 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-f8a8088d-7bb6787865-bmt6k" podStartSLOduration=1.7478186770000002 podStartE2EDuration="1.747818677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:14:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:14:09.704220764 +0000 UTC m=+604.460558810" watchObservedRunningTime="2026-10-31 04:14:09.747818677 +0000 UTC m=+604.504156786" Oct 31 04:14:13.487555 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:13.487502 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cda44a-fq5wl"] Oct 31 04:14:13.493717 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:13.493682 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cda44a-fq5wl" Oct 31 04:14:13.511777 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:13.511666 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cda44a-fq5wl"] Oct 31 04:14:13.519101 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:13.519065 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:14:13.519730 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:13.519709 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:14:13.519855 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:13.519781 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:14:13.520354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:13.520302 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:14:13.638057 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:13.638021 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e28a4486-3e22-45ad-bcd4-62bfd11d49dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-16cda44a-fq5wl\" (UID: \"e28a4486-3e22-45ad-bcd4-62bfd11d49dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cda44a-fq5wl" Oct 31 04:14:13.638241 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:13.638089 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd2sh\" (UniqueName: \"kubernetes.io/projected/e28a4486-3e22-45ad-bcd4-62bfd11d49dd-kube-api-access-hd2sh\") pod \"maas-api-revoke-keys-e2e-ait-16cda44a-fq5wl\" (UID: \"e28a4486-3e22-45ad-bcd4-62bfd11d49dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cda44a-fq5wl" Oct 31 04:14:13.638241 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:13.638220 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e28a4486-3e22-45ad-bcd4-62bfd11d49dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-16cda44a-fq5wl\" (UID: \"e28a4486-3e22-45ad-bcd4-62bfd11d49dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cda44a-fq5wl" Oct 31 04:14:13.739787 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:13.739742 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e28a4486-3e22-45ad-bcd4-62bfd11d49dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-16cda44a-fq5wl\" (UID: \"e28a4486-3e22-45ad-bcd4-62bfd11d49dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cda44a-fq5wl" Oct 31 04:14:13.740004 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:13.739862 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e28a4486-3e22-45ad-bcd4-62bfd11d49dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-16cda44a-fq5wl\" (UID: \"e28a4486-3e22-45ad-bcd4-62bfd11d49dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cda44a-fq5wl" Oct 31 04:14:13.740004 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:13.739953 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd2sh\" (UniqueName: \"kubernetes.io/projected/e28a4486-3e22-45ad-bcd4-62bfd11d49dd-kube-api-access-hd2sh\") pod \"maas-api-revoke-keys-e2e-ait-16cda44a-fq5wl\" (UID: \"e28a4486-3e22-45ad-bcd4-62bfd11d49dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cda44a-fq5wl" Oct 31 04:14:13.740627 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:13.740601 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e28a4486-3e22-45ad-bcd4-62bfd11d49dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-16cda44a-fq5wl\" (UID: \"e28a4486-3e22-45ad-bcd4-62bfd11d49dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cda44a-fq5wl" Oct 31 04:14:13.742367 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:13.742320 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e28a4486-3e22-45ad-bcd4-62bfd11d49dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-16cda44a-fq5wl\" (UID: \"e28a4486-3e22-45ad-bcd4-62bfd11d49dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cda44a-fq5wl" Oct 31 04:14:13.769601 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:13.769561 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd2sh\" (UniqueName: \"kubernetes.io/projected/e28a4486-3e22-45ad-bcd4-62bfd11d49dd-kube-api-access-hd2sh\") pod \"maas-api-revoke-keys-e2e-ait-16cda44a-fq5wl\" (UID: \"e28a4486-3e22-45ad-bcd4-62bfd11d49dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cda44a-fq5wl" Oct 31 04:14:13.818568 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:13.818520 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cda44a-fq5wl" Oct 31 04:14:14.140424 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:14.140313 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cda44a-fq5wl"] Oct 31 04:14:14.144842 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:14:14.144806 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode28a4486_3e22_45ad_bcd4_62bfd11d49dd.slice/crio-d0ab2f2f9f9406801f9304df0f96a2044db436a2cb19a0784eb094b8db9b6cfc WatchSource:0}: Error finding container d0ab2f2f9f9406801f9304df0f96a2044db436a2cb19a0784eb094b8db9b6cfc: Status 404 returned error can't find the container with id d0ab2f2f9f9406801f9304df0f96a2044db436a2cb19a0784eb094b8db9b6cfc Oct 31 04:14:14.147625 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:14.147596 5019 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 04:14:14.711387 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:14.710945 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cda44a-fq5wl" event={"ID":"e28a4486-3e22-45ad-bcd4-62bfd11d49dd","Type":"ContainerStarted","Data":"d0ab2f2f9f9406801f9304df0f96a2044db436a2cb19a0784eb094b8db9b6cfc"} Oct 31 04:14:15.623058 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:15.623027 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-a0d1a61d-6499787965-w979c" Oct 31 04:14:18.750486 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:18.750447 5019 generic.go:356] "Generic (PLEG): container finished" podID="6b943d4d-c829-410c-9233-0b53cbc1c414" containerID="e648ace0ce8cd6ec13266002b01c48ae54052a86bbe33a3c08ce77fe41873aa0" exitCode=0 Oct 31 04:14:18.750994 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:18.750525 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n" event={"ID":"6b943d4d-c829-410c-9233-0b53cbc1c414","Type":"ContainerDied","Data":"e648ace0ce8cd6ec13266002b01c48ae54052a86bbe33a3c08ce77fe41873aa0"} Oct 31 04:14:18.751061 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:18.751046 5019 scope.go:122] "RemoveContainer" containerID="e648ace0ce8cd6ec13266002b01c48ae54052a86bbe33a3c08ce77fe41873aa0" Oct 31 04:14:18.752406 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:18.752371 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cda44a-fq5wl" event={"ID":"e28a4486-3e22-45ad-bcd4-62bfd11d49dd","Type":"ContainerStarted","Data":"287c2e7c572c9f16f3f5788f29f23c65ea1941a2390f58627c45ebffbc424650"} Oct 31 04:14:18.752602 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:18.752570 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:14:18.752691 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:18.752675 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:14:18.752739 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:18.752724 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:14:18.857142 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:18.857079 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cda44a-fq5wl" podStartSLOduration=2.0702514929999998 podStartE2EDuration="5.857057176s" totalImagesPullingTime="3.786805683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:14:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:14:18.855586709 +0000 UTC m=+613.611924750" watchObservedRunningTime="2026-10-31 04:14:18.857057176 +0000 UTC m=+613.613395219" Oct 31 04:14:18.988356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:18.976776 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0d1a61d-6hsws"] Oct 31 04:14:19.009412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:19.007399 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0d1a61d-6hsws"] Oct 31 04:14:19.009412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:19.007526 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0d1a61d-6hsws" Oct 31 04:14:19.125256 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:19.125218 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpkvv\" (UniqueName: \"kubernetes.io/projected/a8266bc2-1ff3-40c1-8e82-e91d7f25e6df-kube-api-access-wpkvv\") pod \"maas-api-revoke-keys-e2e-migrate-a0d1a61d-6hsws\" (UID: \"a8266bc2-1ff3-40c1-8e82-e91d7f25e6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0d1a61d-6hsws" Oct 31 04:14:19.125444 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:19.125397 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8266bc2-1ff3-40c1-8e82-e91d7f25e6df-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a0d1a61d-6hsws\" (UID: \"a8266bc2-1ff3-40c1-8e82-e91d7f25e6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0d1a61d-6hsws" Oct 31 04:14:19.125444 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:19.125427 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8266bc2-1ff3-40c1-8e82-e91d7f25e6df-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a0d1a61d-6hsws\" (UID: \"a8266bc2-1ff3-40c1-8e82-e91d7f25e6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0d1a61d-6hsws" Oct 31 04:14:19.226394 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:19.226353 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpkvv\" (UniqueName: \"kubernetes.io/projected/a8266bc2-1ff3-40c1-8e82-e91d7f25e6df-kube-api-access-wpkvv\") pod \"maas-api-revoke-keys-e2e-migrate-a0d1a61d-6hsws\" (UID: \"a8266bc2-1ff3-40c1-8e82-e91d7f25e6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0d1a61d-6hsws" Oct 31 04:14:19.226610 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:19.226469 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8266bc2-1ff3-40c1-8e82-e91d7f25e6df-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a0d1a61d-6hsws\" (UID: \"a8266bc2-1ff3-40c1-8e82-e91d7f25e6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0d1a61d-6hsws" Oct 31 04:14:19.226610 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:19.226499 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8266bc2-1ff3-40c1-8e82-e91d7f25e6df-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a0d1a61d-6hsws\" (UID: \"a8266bc2-1ff3-40c1-8e82-e91d7f25e6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0d1a61d-6hsws" Oct 31 04:14:19.227555 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:19.227525 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8266bc2-1ff3-40c1-8e82-e91d7f25e6df-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a0d1a61d-6hsws\" (UID: \"a8266bc2-1ff3-40c1-8e82-e91d7f25e6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0d1a61d-6hsws" Oct 31 04:14:19.231780 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:19.231743 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8266bc2-1ff3-40c1-8e82-e91d7f25e6df-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a0d1a61d-6hsws\" (UID: \"a8266bc2-1ff3-40c1-8e82-e91d7f25e6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0d1a61d-6hsws" Oct 31 04:14:19.250138 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:19.250108 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpkvv\" (UniqueName: \"kubernetes.io/projected/a8266bc2-1ff3-40c1-8e82-e91d7f25e6df-kube-api-access-wpkvv\") pod \"maas-api-revoke-keys-e2e-migrate-a0d1a61d-6hsws\" (UID: \"a8266bc2-1ff3-40c1-8e82-e91d7f25e6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0d1a61d-6hsws" Oct 31 04:14:19.325700 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:19.325613 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0d1a61d-6hsws" Oct 31 04:14:19.660557 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:19.660519 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0d1a61d-6hsws"] Oct 31 04:14:19.662190 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:14:19.662158 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8266bc2_1ff3_40c1_8e82_e91d7f25e6df.slice/crio-b18b9e9c85f42b45a9b6b0f4783d0e66fd90e4df730379bce47e7c4f71440267 WatchSource:0}: Error finding container b18b9e9c85f42b45a9b6b0f4783d0e66fd90e4df730379bce47e7c4f71440267: Status 404 returned error can't find the container with id b18b9e9c85f42b45a9b6b0f4783d0e66fd90e4df730379bce47e7c4f71440267 Oct 31 04:14:19.763918 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:19.763870 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n" event={"ID":"6b943d4d-c829-410c-9233-0b53cbc1c414","Type":"ContainerStarted","Data":"adac2c933bec465a85952b2d64beeac94099d19d13c9bc5b080bad43f9376efa"} Oct 31 04:14:19.798119 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:19.797710 5019 generic.go:356] "Generic (PLEG): container finished" podID="e28a4486-3e22-45ad-bcd4-62bfd11d49dd" containerID="287c2e7c572c9f16f3f5788f29f23c65ea1941a2390f58627c45ebffbc424650" exitCode=0 Oct 31 04:14:19.798119 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:19.797895 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cda44a-fq5wl" event={"ID":"e28a4486-3e22-45ad-bcd4-62bfd11d49dd","Type":"ContainerDied","Data":"287c2e7c572c9f16f3f5788f29f23c65ea1941a2390f58627c45ebffbc424650"} Oct 31 04:14:19.834713 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:19.834675 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0d1a61d-6hsws" event={"ID":"a8266bc2-1ff3-40c1-8e82-e91d7f25e6df","Type":"ContainerStarted","Data":"b18b9e9c85f42b45a9b6b0f4783d0e66fd90e4df730379bce47e7c4f71440267"} Oct 31 04:14:20.600583 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:20.596120 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n" Oct 31 04:14:20.600583 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:20.597025 5019 patch_prober.go:29] interesting pod/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 31 04:14:20.600583 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:20.597082 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n" podUID="6b943d4d-c829-410c-9233-0b53cbc1c414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 31 04:14:20.665664 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:20.664940 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-f8a8088d-858594b57f-d7j74" Oct 31 04:14:20.665664 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:20.665372 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-f8a8088d-7bb6787865-bmt6k" Oct 31 04:14:20.955365 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:20.954626 5019 generic.go:356] "Generic (PLEG): container finished" podID="a8266bc2-1ff3-40c1-8e82-e91d7f25e6df" containerID="cd258e729b3fb3f56ee9b49096d9c29eadd41f2b6d8a774a75131628b8fd70c0" exitCode=0 Oct 31 04:14:20.955861 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:20.955379 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0d1a61d-6hsws" event={"ID":"a8266bc2-1ff3-40c1-8e82-e91d7f25e6df","Type":"ContainerDied","Data":"cd258e729b3fb3f56ee9b49096d9c29eadd41f2b6d8a774a75131628b8fd70c0"} Oct 31 04:14:21.321359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:21.320584 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cda44a-fq5wl" Oct 31 04:14:21.367361 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:21.366530 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e28a4486-3e22-45ad-bcd4-62bfd11d49dd-kube-api-access-hd2sh\" (UniqueName: \"kubernetes.io/projected/e28a4486-3e22-45ad-bcd4-62bfd11d49dd-kube-api-access-hd2sh\") pod \"e28a4486-3e22-45ad-bcd4-62bfd11d49dd\" (UID: \"e28a4486-3e22-45ad-bcd4-62bfd11d49dd\") " Oct 31 04:14:21.367361 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:21.367154 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e28a4486-3e22-45ad-bcd4-62bfd11d49dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e28a4486-3e22-45ad-bcd4-62bfd11d49dd-maas-api-xks-ca\") pod \"e28a4486-3e22-45ad-bcd4-62bfd11d49dd\" (UID: \"e28a4486-3e22-45ad-bcd4-62bfd11d49dd\") " Oct 31 04:14:21.368447 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:21.368421 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e28a4486-3e22-45ad-bcd4-62bfd11d49dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e28a4486-3e22-45ad-bcd4-62bfd11d49dd-maas-api-openshift-ca\") pod \"e28a4486-3e22-45ad-bcd4-62bfd11d49dd\" (UID: \"e28a4486-3e22-45ad-bcd4-62bfd11d49dd\") " Oct 31 04:14:21.368935 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:21.368789 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e28a4486-3e22-45ad-bcd4-62bfd11d49dd-maas-api-openshift-ca" pod "e28a4486-3e22-45ad-bcd4-62bfd11d49dd" (UID: "e28a4486-3e22-45ad-bcd4-62bfd11d49dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:14:21.369069 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:21.368951 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e28a4486-3e22-45ad-bcd4-62bfd11d49dd-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:14:21.391399 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:21.389522 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e28a4486-3e22-45ad-bcd4-62bfd11d49dd-kube-api-access-hd2sh" pod "e28a4486-3e22-45ad-bcd4-62bfd11d49dd" (UID: "e28a4486-3e22-45ad-bcd4-62bfd11d49dd"). InnerVolumeSpecName "kube-api-access-hd2sh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:14:21.441834 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:21.413977 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e28a4486-3e22-45ad-bcd4-62bfd11d49dd-maas-api-xks-ca" pod "e28a4486-3e22-45ad-bcd4-62bfd11d49dd" (UID: "e28a4486-3e22-45ad-bcd4-62bfd11d49dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:14:21.480423 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:21.479193 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hd2sh\" (UniqueName: \"kubernetes.io/projected/e28a4486-3e22-45ad-bcd4-62bfd11d49dd-kube-api-access-hd2sh\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:14:21.480423 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:21.479243 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e28a4486-3e22-45ad-bcd4-62bfd11d49dd-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:14:21.580358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:21.579827 5019 patch_prober.go:29] interesting pod/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 31 04:14:21.580358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:21.579904 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n" podUID="6b943d4d-c829-410c-9233-0b53cbc1c414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 31 04:14:21.963037 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:21.962946 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cda44a-fq5wl" event={"ID":"e28a4486-3e22-45ad-bcd4-62bfd11d49dd","Type":"ContainerDied","Data":"d0ab2f2f9f9406801f9304df0f96a2044db436a2cb19a0784eb094b8db9b6cfc"} Oct 31 04:14:21.963037 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:21.962972 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cda44a-fq5wl" Oct 31 04:14:21.963037 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:21.963009 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0ab2f2f9f9406801f9304df0f96a2044db436a2cb19a0784eb094b8db9b6cfc" Oct 31 04:14:22.325634 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:22.325585 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0d1a61d-6hsws" Oct 31 04:14:22.442695 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:22.442646 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8266bc2-1ff3-40c1-8e82-e91d7f25e6df-kube-api-access-wpkvv\" (UniqueName: \"kubernetes.io/projected/a8266bc2-1ff3-40c1-8e82-e91d7f25e6df-kube-api-access-wpkvv\") pod \"a8266bc2-1ff3-40c1-8e82-e91d7f25e6df\" (UID: \"a8266bc2-1ff3-40c1-8e82-e91d7f25e6df\") " Oct 31 04:14:22.442914 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:22.442717 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8266bc2-1ff3-40c1-8e82-e91d7f25e6df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8266bc2-1ff3-40c1-8e82-e91d7f25e6df-maas-api-xks-ca\") pod \"a8266bc2-1ff3-40c1-8e82-e91d7f25e6df\" (UID: \"a8266bc2-1ff3-40c1-8e82-e91d7f25e6df\") " Oct 31 04:14:22.442914 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:22.442760 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8266bc2-1ff3-40c1-8e82-e91d7f25e6df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8266bc2-1ff3-40c1-8e82-e91d7f25e6df-maas-api-openshift-ca\") pod \"a8266bc2-1ff3-40c1-8e82-e91d7f25e6df\" (UID: \"a8266bc2-1ff3-40c1-8e82-e91d7f25e6df\") " Oct 31 04:14:22.444203 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:22.444166 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8266bc2-1ff3-40c1-8e82-e91d7f25e6df-maas-api-openshift-ca" pod "a8266bc2-1ff3-40c1-8e82-e91d7f25e6df" (UID: "a8266bc2-1ff3-40c1-8e82-e91d7f25e6df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:14:22.462036 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:22.461989 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8266bc2-1ff3-40c1-8e82-e91d7f25e6df-kube-api-access-wpkvv" pod "a8266bc2-1ff3-40c1-8e82-e91d7f25e6df" (UID: "a8266bc2-1ff3-40c1-8e82-e91d7f25e6df"). InnerVolumeSpecName "kube-api-access-wpkvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:14:22.464005 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:22.463966 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8266bc2-1ff3-40c1-8e82-e91d7f25e6df-maas-api-xks-ca" pod "a8266bc2-1ff3-40c1-8e82-e91d7f25e6df" (UID: "a8266bc2-1ff3-40c1-8e82-e91d7f25e6df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:14:22.497412 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:14:22.481600 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:14:22.544947 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:22.544485 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpkvv\" (UniqueName: \"kubernetes.io/projected/a8266bc2-1ff3-40c1-8e82-e91d7f25e6df-kube-api-access-wpkvv\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:14:22.544947 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:22.544529 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8266bc2-1ff3-40c1-8e82-e91d7f25e6df-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:14:22.544947 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:22.544546 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8266bc2-1ff3-40c1-8e82-e91d7f25e6df-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:14:22.581371 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:22.577839 5019 patch_prober.go:29] interesting pod/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 31 04:14:22.581371 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:22.577916 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n" podUID="6b943d4d-c829-410c-9233-0b53cbc1c414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 31 04:14:22.993412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:22.987244 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0d1a61d-6hsws" event={"ID":"a8266bc2-1ff3-40c1-8e82-e91d7f25e6df","Type":"ContainerDied","Data":"b18b9e9c85f42b45a9b6b0f4783d0e66fd90e4df730379bce47e7c4f71440267"} Oct 31 04:14:22.993412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:22.987355 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b18b9e9c85f42b45a9b6b0f4783d0e66fd90e4df730379bce47e7c4f71440267" Oct 31 04:14:22.993412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:22.987610 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0d1a61d-6hsws" Oct 31 04:14:23.579362 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:23.578852 5019 patch_prober.go:29] interesting pod/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 31 04:14:23.579362 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:23.578918 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n" podUID="6b943d4d-c829-410c-9233-0b53cbc1c414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 31 04:14:24.580359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:24.577929 5019 patch_prober.go:29] interesting pod/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 31 04:14:24.580359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:24.578020 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n" podUID="6b943d4d-c829-410c-9233-0b53cbc1c414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 31 04:14:25.128738 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:25.128681 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc"] Oct 31 04:14:25.129199 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:25.129185 5019 cpu_manager.go:414] "containerMap: removing container" podUID="e28a4486-3e22-45ad-bcd4-62bfd11d49dd" containerName="revoke-keys" Oct 31 04:14:25.129246 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:25.129200 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e28a4486-3e22-45ad-bcd4-62bfd11d49dd" containerName="revoke-keys" Oct 31 04:14:25.129246 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:25.129229 5019 cpu_manager.go:414] "containerMap: removing container" podUID="a8266bc2-1ff3-40c1-8e82-e91d7f25e6df" containerName="revoke-keys" Oct 31 04:14:25.129246 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:25.129235 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8266bc2-1ff3-40c1-8e82-e91d7f25e6df" containerName="revoke-keys" Oct 31 04:14:25.129359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:25.129317 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8266bc2-1ff3-40c1-8e82-e91d7f25e6df" containerName="revoke-keys" Oct 31 04:14:25.129359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:25.129341 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e28a4486-3e22-45ad-bcd4-62bfd11d49dd" containerName="revoke-keys" Oct 31 04:14:25.133316 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:25.133291 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc" Oct 31 04:14:25.169111 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:25.169073 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc"] Oct 31 04:14:25.205356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:25.187802 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c65c601e-7e95-4e7d-844c-a4cadd680d1b-istio-data\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc\" (UID: \"c65c601e-7e95-4e7d-844c-a4cadd680d1b\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc" Oct 31 04:14:25.232533 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:25.232462 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c65c601e-7e95-4e7d-844c-a4cadd680d1b-workload-socket\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc\" (UID: \"c65c601e-7e95-4e7d-844c-a4cadd680d1b\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc" Oct 31 04:14:25.232756 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:25.232575 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s78l\" (UniqueName: \"kubernetes.io/projected/c65c601e-7e95-4e7d-844c-a4cadd680d1b-kube-api-access-7s78l\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc\" (UID: \"c65c601e-7e95-4e7d-844c-a4cadd680d1b\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc" Oct 31 04:14:25.232756 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:25.232611 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c65c601e-7e95-4e7d-844c-a4cadd680d1b-istio-token\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc\" (UID: \"c65c601e-7e95-4e7d-844c-a4cadd680d1b\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc" Oct 31 04:14:25.232756 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:25.232681 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c65c601e-7e95-4e7d-844c-a4cadd680d1b-istio-podinfo\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc\" (UID: \"c65c601e-7e95-4e7d-844c-a4cadd680d1b\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc" Oct 31 04:14:25.232756 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:25.232728 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c65c601e-7e95-4e7d-844c-a4cadd680d1b-workload-certs\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc\" (UID: \"c65c601e-7e95-4e7d-844c-a4cadd680d1b\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc" Oct 31 04:14:25.232993 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:25.232834 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c65c601e-7e95-4e7d-844c-a4cadd680d1b-istio-envoy\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc\" (UID: \"c65c601e-7e95-4e7d-844c-a4cadd680d1b\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc" Oct 31 04:14:25.232993 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:25.232873 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c65c601e-7e95-4e7d-844c-a4cadd680d1b-credential-socket\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc\" (UID: \"c65c601e-7e95-4e7d-844c-a4cadd680d1b\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc" Oct 31 04:14:25.232993 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:25.232931 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c65c601e-7e95-4e7d-844c-a4cadd680d1b-istiod-ca-cert\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc\" (UID: \"c65c601e-7e95-4e7d-844c-a4cadd680d1b\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc" Oct 31 04:14:25.334228 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:25.334173 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c65c601e-7e95-4e7d-844c-a4cadd680d1b-istio-envoy\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc\" (UID: \"c65c601e-7e95-4e7d-844c-a4cadd680d1b\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc" Oct 31 04:14:25.334228 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:25.334226 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c65c601e-7e95-4e7d-844c-a4cadd680d1b-credential-socket\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc\" (UID: \"c65c601e-7e95-4e7d-844c-a4cadd680d1b\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc" Oct 31 04:14:25.334537 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:25.334256 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c65c601e-7e95-4e7d-844c-a4cadd680d1b-istiod-ca-cert\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc\" (UID: \"c65c601e-7e95-4e7d-844c-a4cadd680d1b\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc" Oct 31 04:14:25.334537 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:25.334291 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c65c601e-7e95-4e7d-844c-a4cadd680d1b-istio-data\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc\" (UID: \"c65c601e-7e95-4e7d-844c-a4cadd680d1b\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc" Oct 31 04:14:25.334537 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:25.334312 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c65c601e-7e95-4e7d-844c-a4cadd680d1b-workload-socket\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc\" (UID: \"c65c601e-7e95-4e7d-844c-a4cadd680d1b\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc" Oct 31 04:14:25.334537 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:25.334361 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s78l\" (UniqueName: \"kubernetes.io/projected/c65c601e-7e95-4e7d-844c-a4cadd680d1b-kube-api-access-7s78l\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc\" (UID: \"c65c601e-7e95-4e7d-844c-a4cadd680d1b\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc" Oct 31 04:14:25.334537 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:25.334381 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c65c601e-7e95-4e7d-844c-a4cadd680d1b-istio-token\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc\" (UID: \"c65c601e-7e95-4e7d-844c-a4cadd680d1b\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc" Oct 31 04:14:25.334537 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:25.334414 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c65c601e-7e95-4e7d-844c-a4cadd680d1b-istio-podinfo\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc\" (UID: \"c65c601e-7e95-4e7d-844c-a4cadd680d1b\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc" Oct 31 04:14:25.334537 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:25.334436 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c65c601e-7e95-4e7d-844c-a4cadd680d1b-workload-certs\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc\" (UID: \"c65c601e-7e95-4e7d-844c-a4cadd680d1b\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc" Oct 31 04:14:25.334906 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:25.334789 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c65c601e-7e95-4e7d-844c-a4cadd680d1b-workload-certs\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc\" (UID: \"c65c601e-7e95-4e7d-844c-a4cadd680d1b\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc" Oct 31 04:14:25.335387 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:25.335362 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c65c601e-7e95-4e7d-844c-a4cadd680d1b-workload-socket\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc\" (UID: \"c65c601e-7e95-4e7d-844c-a4cadd680d1b\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc" Oct 31 04:14:25.335606 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:25.335588 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c65c601e-7e95-4e7d-844c-a4cadd680d1b-istio-data\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc\" (UID: \"c65c601e-7e95-4e7d-844c-a4cadd680d1b\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc" Oct 31 04:14:25.335903 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:25.335879 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c65c601e-7e95-4e7d-844c-a4cadd680d1b-credential-socket\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc\" (UID: \"c65c601e-7e95-4e7d-844c-a4cadd680d1b\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc" Oct 31 04:14:25.336539 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:25.336519 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c65c601e-7e95-4e7d-844c-a4cadd680d1b-istiod-ca-cert\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc\" (UID: \"c65c601e-7e95-4e7d-844c-a4cadd680d1b\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc" Oct 31 04:14:25.338001 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:25.337976 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c65c601e-7e95-4e7d-844c-a4cadd680d1b-istio-podinfo\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc\" (UID: \"c65c601e-7e95-4e7d-844c-a4cadd680d1b\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc" Oct 31 04:14:25.338649 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:25.338628 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c65c601e-7e95-4e7d-844c-a4cadd680d1b-istio-envoy\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc\" (UID: \"c65c601e-7e95-4e7d-844c-a4cadd680d1b\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc" Oct 31 04:14:25.360996 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:25.360956 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s78l\" (UniqueName: \"kubernetes.io/projected/c65c601e-7e95-4e7d-844c-a4cadd680d1b-kube-api-access-7s78l\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc\" (UID: \"c65c601e-7e95-4e7d-844c-a4cadd680d1b\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc" Oct 31 04:14:25.362841 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:25.362809 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c65c601e-7e95-4e7d-844c-a4cadd680d1b-istio-token\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc\" (UID: \"c65c601e-7e95-4e7d-844c-a4cadd680d1b\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc" Oct 31 04:14:25.449772 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:25.449649 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc" Oct 31 04:14:25.764968 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:25.764921 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n" Oct 31 04:14:25.765734 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:25.765705 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n" Oct 31 04:14:25.767071 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:25.767034 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n" Oct 31 04:14:25.841683 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:25.841510 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4805aacc-w7hhm"] Oct 31 04:14:25.847058 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:25.847015 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4805aacc-w7hhm" Oct 31 04:14:25.866256 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:25.863825 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4805aacc-w7hhm"] Oct 31 04:14:25.867783 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:25.867752 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:14:25.868028 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:25.868012 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:14:25.868251 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:25.868222 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:14:25.868402 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:25.868292 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:14:25.920488 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:14:25.920444 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc65c601e_7e95_4e7d_844c_a4cadd680d1b.slice/crio-12023b1b1cbd5f7c9ed5961cba4bd9a9c642909e30fc0bf75db2d440dc1524e6 WatchSource:0}: Error finding container 12023b1b1cbd5f7c9ed5961cba4bd9a9c642909e30fc0bf75db2d440dc1524e6: Status 404 returned error can't find the container with id 12023b1b1cbd5f7c9ed5961cba4bd9a9c642909e30fc0bf75db2d440dc1524e6 Oct 31 04:14:25.930592 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:25.930566 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc"] Oct 31 04:14:25.931860 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:25.931818 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:14:25.932073 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:25.932053 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:14:25.932200 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:25.932183 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:14:25.943175 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:25.943142 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f813717a-cc13-46f3-b010-fd9898d4052a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4805aacc-w7hhm\" (UID: \"f813717a-cc13-46f3-b010-fd9898d4052a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4805aacc-w7hhm" Oct 31 04:14:25.943435 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:25.943413 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xmmd\" (UniqueName: \"kubernetes.io/projected/f813717a-cc13-46f3-b010-fd9898d4052a-kube-api-access-8xmmd\") pod \"maas-api-revoke-keys-e2e-ait-4805aacc-w7hhm\" (UID: \"f813717a-cc13-46f3-b010-fd9898d4052a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4805aacc-w7hhm" Oct 31 04:14:25.943577 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:25.943564 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f813717a-cc13-46f3-b010-fd9898d4052a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4805aacc-w7hhm\" (UID: \"f813717a-cc13-46f3-b010-fd9898d4052a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4805aacc-w7hhm" Oct 31 04:14:26.028735 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:26.028650 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc" event={"ID":"c65c601e-7e95-4e7d-844c-a4cadd680d1b","Type":"ContainerStarted","Data":"12023b1b1cbd5f7c9ed5961cba4bd9a9c642909e30fc0bf75db2d440dc1524e6"} Oct 31 04:14:26.046030 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:26.045677 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f813717a-cc13-46f3-b010-fd9898d4052a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4805aacc-w7hhm\" (UID: \"f813717a-cc13-46f3-b010-fd9898d4052a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4805aacc-w7hhm" Oct 31 04:14:26.046030 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:26.045731 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xmmd\" (UniqueName: \"kubernetes.io/projected/f813717a-cc13-46f3-b010-fd9898d4052a-kube-api-access-8xmmd\") pod \"maas-api-revoke-keys-e2e-ait-4805aacc-w7hhm\" (UID: \"f813717a-cc13-46f3-b010-fd9898d4052a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4805aacc-w7hhm" Oct 31 04:14:26.046030 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:26.045784 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f813717a-cc13-46f3-b010-fd9898d4052a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4805aacc-w7hhm\" (UID: \"f813717a-cc13-46f3-b010-fd9898d4052a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4805aacc-w7hhm" Oct 31 04:14:26.047886 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:26.047740 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f813717a-cc13-46f3-b010-fd9898d4052a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4805aacc-w7hhm\" (UID: \"f813717a-cc13-46f3-b010-fd9898d4052a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4805aacc-w7hhm" Oct 31 04:14:26.049087 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:26.049057 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f813717a-cc13-46f3-b010-fd9898d4052a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4805aacc-w7hhm\" (UID: \"f813717a-cc13-46f3-b010-fd9898d4052a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4805aacc-w7hhm" Oct 31 04:14:26.079112 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:26.079069 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xmmd\" (UniqueName: \"kubernetes.io/projected/f813717a-cc13-46f3-b010-fd9898d4052a-kube-api-access-8xmmd\") pod \"maas-api-revoke-keys-e2e-ait-4805aacc-w7hhm\" (UID: \"f813717a-cc13-46f3-b010-fd9898d4052a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4805aacc-w7hhm" Oct 31 04:14:26.211293 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:26.210931 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4805aacc-w7hhm" Oct 31 04:14:26.554929 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:26.554897 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4805aacc-w7hhm"] Oct 31 04:14:26.556443 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:14:26.556409 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf813717a_cc13_46f3_b010_fd9898d4052a.slice/crio-758a7d7542394aa4bdee6dc68722b7a4bbb76915d3537850193ac720fec9a910 WatchSource:0}: Error finding container 758a7d7542394aa4bdee6dc68722b7a4bbb76915d3537850193ac720fec9a910: Status 404 returned error can't find the container with id 758a7d7542394aa4bdee6dc68722b7a4bbb76915d3537850193ac720fec9a910 Oct 31 04:14:27.043208 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:27.043169 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4805aacc-w7hhm" event={"ID":"f813717a-cc13-46f3-b010-fd9898d4052a","Type":"ContainerStarted","Data":"758a7d7542394aa4bdee6dc68722b7a4bbb76915d3537850193ac720fec9a910"} Oct 31 04:14:27.046447 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:27.046410 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc" event={"ID":"c65c601e-7e95-4e7d-844c-a4cadd680d1b","Type":"ContainerStarted","Data":"5b0032a70f02f03c928a2893d87d6d17bc50a01807463715fbb9b4b67058f484"} Oct 31 04:14:27.101261 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:27.101128 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc" podStartSLOduration=2.101109095 podStartE2EDuration="2.101109095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:14:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:14:27.091069888 +0000 UTC m=+621.847407929" watchObservedRunningTime="2026-10-31 04:14:27.101109095 +0000 UTC m=+621.857447134" Oct 31 04:14:27.450158 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:27.450058 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc" Oct 31 04:14:27.451871 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:27.451840 5019 patch_prober.go:29] interesting pod/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 31 04:14:27.452002 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:27.451908 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc" podUID="c65c601e-7e95-4e7d-844c-a4cadd680d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 31 04:14:27.734854 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:27.734687 5019 patch_prober.go:29] interesting pod/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": read tcp 10.132.0.2:51950->10.132.0.53:15021: read: connection reset by peer" start-of-body= Oct 31 04:14:27.734854 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:27.734761 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb" podUID="ec1714c9-5e99-400a-b392-486c04926361" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": read tcp 10.132.0.2:51950->10.132.0.53:15021: read: connection reset by peer" Oct 31 04:14:28.054883 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:28.054780 5019 generic.go:356] "Generic (PLEG): container finished" podID="ec1714c9-5e99-400a-b392-486c04926361" containerID="9753078996d2416ba0e209fc2cc4eba325b375a272c80f14733af5bdbbf2c45f" exitCode=0 Oct 31 04:14:28.054883 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:28.054855 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb" event={"ID":"ec1714c9-5e99-400a-b392-486c04926361","Type":"ContainerDied","Data":"9753078996d2416ba0e209fc2cc4eba325b375a272c80f14733af5bdbbf2c45f"} Oct 31 04:14:28.055426 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:28.055370 5019 scope.go:122] "RemoveContainer" containerID="9753078996d2416ba0e209fc2cc4eba325b375a272c80f14733af5bdbbf2c45f" Oct 31 04:14:28.056919 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:28.056883 5019 generic.go:356] "Generic (PLEG): container finished" podID="f813717a-cc13-46f3-b010-fd9898d4052a" containerID="28dd974596f9c8da9a638a382bc8fbde5f5a7af8f3c32fa56cf5712419b888f7" exitCode=0 Oct 31 04:14:28.057112 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:28.057078 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4805aacc-w7hhm" event={"ID":"f813717a-cc13-46f3-b010-fd9898d4052a","Type":"ContainerDied","Data":"28dd974596f9c8da9a638a382bc8fbde5f5a7af8f3c32fa56cf5712419b888f7"} Oct 31 04:14:28.057393 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:28.057076 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:14:28.057393 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:28.057221 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:14:28.057393 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:28.057277 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:14:28.451176 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:28.451083 5019 patch_prober.go:29] interesting pod/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 31 04:14:28.451176 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:28.451153 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc" podUID="c65c601e-7e95-4e7d-844c-a4cadd680d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 31 04:14:29.066063 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:29.066028 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb" event={"ID":"ec1714c9-5e99-400a-b392-486c04926361","Type":"ContainerStarted","Data":"00d09e779f30b22182e3ea6673996014cdf91ca88e1de182b6d8b965fbc777c2"} Oct 31 04:14:29.328976 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:29.305639 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb" Oct 31 04:14:29.330430 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:29.308484 5019 patch_prober.go:29] interesting pod/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 31 04:14:29.330430 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:29.330396 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb" podUID="ec1714c9-5e99-400a-b392-486c04926361" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 31 04:14:29.365284 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:29.365251 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4805aacc-w7hhm" Oct 31 04:14:29.450110 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:29.450074 5019 patch_prober.go:29] interesting pod/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 31 04:14:29.450295 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:29.450164 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc" podUID="c65c601e-7e95-4e7d-844c-a4cadd680d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 31 04:14:29.539644 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:29.539601 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a0d1a61d-6499787965-w979c"] Oct 31 04:14:29.542056 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:29.542026 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f813717a-cc13-46f3-b010-fd9898d4052a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f813717a-cc13-46f3-b010-fd9898d4052a-maas-api-xks-ca\") pod \"f813717a-cc13-46f3-b010-fd9898d4052a\" (UID: \"f813717a-cc13-46f3-b010-fd9898d4052a\") " Oct 31 04:14:29.542228 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:29.542116 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f813717a-cc13-46f3-b010-fd9898d4052a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f813717a-cc13-46f3-b010-fd9898d4052a-maas-api-openshift-ca\") pod \"f813717a-cc13-46f3-b010-fd9898d4052a\" (UID: \"f813717a-cc13-46f3-b010-fd9898d4052a\") " Oct 31 04:14:29.542228 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:29.542208 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f813717a-cc13-46f3-b010-fd9898d4052a-kube-api-access-8xmmd\" (UniqueName: \"kubernetes.io/projected/f813717a-cc13-46f3-b010-fd9898d4052a-kube-api-access-8xmmd\") pod \"f813717a-cc13-46f3-b010-fd9898d4052a\" (UID: \"f813717a-cc13-46f3-b010-fd9898d4052a\") " Oct 31 04:14:29.543187 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:29.543152 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f813717a-cc13-46f3-b010-fd9898d4052a-maas-api-openshift-ca" pod "f813717a-cc13-46f3-b010-fd9898d4052a" (UID: "f813717a-cc13-46f3-b010-fd9898d4052a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:14:29.544670 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:29.544625 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f813717a-cc13-46f3-b010-fd9898d4052a-maas-api-xks-ca" pod "f813717a-cc13-46f3-b010-fd9898d4052a" (UID: "f813717a-cc13-46f3-b010-fd9898d4052a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:14:29.545065 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:29.545035 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f813717a-cc13-46f3-b010-fd9898d4052a-kube-api-access-8xmmd" pod "f813717a-cc13-46f3-b010-fd9898d4052a" (UID: "f813717a-cc13-46f3-b010-fd9898d4052a"). InnerVolumeSpecName "kube-api-access-8xmmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:14:29.648024 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:29.647868 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f813717a-cc13-46f3-b010-fd9898d4052a-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:14:29.648024 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:29.647912 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f813717a-cc13-46f3-b010-fd9898d4052a-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:14:29.648024 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:29.647929 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xmmd\" (UniqueName: \"kubernetes.io/projected/f813717a-cc13-46f3-b010-fd9898d4052a-kube-api-access-8xmmd\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:14:30.120359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:30.119880 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4805aacc-w7hhm" Oct 31 04:14:30.125121 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:30.123454 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4805aacc-w7hhm" event={"ID":"f813717a-cc13-46f3-b010-fd9898d4052a","Type":"ContainerDied","Data":"758a7d7542394aa4bdee6dc68722b7a4bbb76915d3537850193ac720fec9a910"} Oct 31 04:14:30.125121 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:30.123815 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="758a7d7542394aa4bdee6dc68722b7a4bbb76915d3537850193ac720fec9a910" Oct 31 04:14:30.308361 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:30.307185 5019 patch_prober.go:29] interesting pod/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 31 04:14:30.308361 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:30.307267 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb" podUID="ec1714c9-5e99-400a-b392-486c04926361" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 31 04:14:30.389424 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:30.389097 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44f032ad-fzt68"] Oct 31 04:14:30.392583 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:30.389910 5019 cpu_manager.go:414] "containerMap: removing container" podUID="f813717a-cc13-46f3-b010-fd9898d4052a" containerName="revoke-keys" Oct 31 04:14:30.392583 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:30.389952 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f813717a-cc13-46f3-b010-fd9898d4052a" containerName="revoke-keys" Oct 31 04:14:30.392583 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:30.390133 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="f813717a-cc13-46f3-b010-fd9898d4052a" containerName="revoke-keys" Oct 31 04:14:30.394028 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:30.393995 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44f032ad-fzt68" Oct 31 04:14:30.412009 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:30.410553 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:14:30.412009 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:30.411288 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:14:30.416279 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:30.416245 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44f032ad-fzt68"] Oct 31 04:14:30.448360 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:30.445824 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:14:30.448360 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:30.445945 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:14:30.452364 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:30.450975 5019 patch_prober.go:29] interesting pod/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 31 04:14:30.452364 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:30.451044 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc" podUID="c65c601e-7e95-4e7d-844c-a4cadd680d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 31 04:14:30.461968 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:30.461919 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2jzb\" (UniqueName: \"kubernetes.io/projected/f6382469-cb51-401e-bccf-cbf15f15075c-kube-api-access-v2jzb\") pod \"maas-api-revoke-keys-e2e-ait-44f032ad-fzt68\" (UID: \"f6382469-cb51-401e-bccf-cbf15f15075c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44f032ad-fzt68" Oct 31 04:14:30.462155 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:30.461998 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6382469-cb51-401e-bccf-cbf15f15075c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-44f032ad-fzt68\" (UID: \"f6382469-cb51-401e-bccf-cbf15f15075c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44f032ad-fzt68" Oct 31 04:14:30.462239 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:30.462159 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6382469-cb51-401e-bccf-cbf15f15075c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-44f032ad-fzt68\" (UID: \"f6382469-cb51-401e-bccf-cbf15f15075c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44f032ad-fzt68" Oct 31 04:14:30.562903 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:30.562862 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2jzb\" (UniqueName: \"kubernetes.io/projected/f6382469-cb51-401e-bccf-cbf15f15075c-kube-api-access-v2jzb\") pod \"maas-api-revoke-keys-e2e-ait-44f032ad-fzt68\" (UID: \"f6382469-cb51-401e-bccf-cbf15f15075c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44f032ad-fzt68" Oct 31 04:14:30.563087 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:30.562919 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6382469-cb51-401e-bccf-cbf15f15075c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-44f032ad-fzt68\" (UID: \"f6382469-cb51-401e-bccf-cbf15f15075c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44f032ad-fzt68" Oct 31 04:14:30.563087 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:30.562964 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6382469-cb51-401e-bccf-cbf15f15075c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-44f032ad-fzt68\" (UID: \"f6382469-cb51-401e-bccf-cbf15f15075c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44f032ad-fzt68" Oct 31 04:14:30.563689 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:30.563669 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6382469-cb51-401e-bccf-cbf15f15075c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-44f032ad-fzt68\" (UID: \"f6382469-cb51-401e-bccf-cbf15f15075c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44f032ad-fzt68" Oct 31 04:14:30.566071 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:30.566050 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6382469-cb51-401e-bccf-cbf15f15075c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-44f032ad-fzt68\" (UID: \"f6382469-cb51-401e-bccf-cbf15f15075c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44f032ad-fzt68" Oct 31 04:14:30.627359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:30.627181 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2jzb\" (UniqueName: \"kubernetes.io/projected/f6382469-cb51-401e-bccf-cbf15f15075c-kube-api-access-v2jzb\") pod \"maas-api-revoke-keys-e2e-ait-44f032ad-fzt68\" (UID: \"f6382469-cb51-401e-bccf-cbf15f15075c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44f032ad-fzt68" Oct 31 04:14:30.757821 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:30.757783 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44f032ad-fzt68" Oct 31 04:14:31.224466 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:31.223008 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44f032ad-fzt68"] Oct 31 04:14:31.305322 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:31.304854 5019 patch_prober.go:29] interesting pod/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 31 04:14:31.305322 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:31.304919 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb" podUID="ec1714c9-5e99-400a-b392-486c04926361" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 31 04:14:31.450526 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:31.450489 5019 patch_prober.go:29] interesting pod/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 31 04:14:31.450701 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:31.450546 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc" podUID="c65c601e-7e95-4e7d-844c-a4cadd680d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 31 04:14:31.534680 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:31.534577 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f8a8088d-ft9l5"] Oct 31 04:14:31.539573 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:31.539545 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f8a8088d-ft9l5" Oct 31 04:14:31.557298 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:31.557260 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f8a8088d-ft9l5"] Oct 31 04:14:31.628617 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:31.628577 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2553577b-451f-4d98-803d-0f81dd263822-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-f8a8088d-ft9l5\" (UID: \"2553577b-451f-4d98-803d-0f81dd263822\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f8a8088d-ft9l5" Oct 31 04:14:31.628973 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:31.628951 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8wfv\" (UniqueName: \"kubernetes.io/projected/2553577b-451f-4d98-803d-0f81dd263822-kube-api-access-j8wfv\") pod \"maas-api-revoke-keys-e2e-derive-f8a8088d-ft9l5\" (UID: \"2553577b-451f-4d98-803d-0f81dd263822\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f8a8088d-ft9l5" Oct 31 04:14:31.629156 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:31.629141 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2553577b-451f-4d98-803d-0f81dd263822-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-f8a8088d-ft9l5\" (UID: \"2553577b-451f-4d98-803d-0f81dd263822\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f8a8088d-ft9l5" Oct 31 04:14:31.730206 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:31.730176 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2553577b-451f-4d98-803d-0f81dd263822-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-f8a8088d-ft9l5\" (UID: \"2553577b-451f-4d98-803d-0f81dd263822\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f8a8088d-ft9l5" Oct 31 04:14:31.730365 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:31.730325 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8wfv\" (UniqueName: \"kubernetes.io/projected/2553577b-451f-4d98-803d-0f81dd263822-kube-api-access-j8wfv\") pod \"maas-api-revoke-keys-e2e-derive-f8a8088d-ft9l5\" (UID: \"2553577b-451f-4d98-803d-0f81dd263822\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f8a8088d-ft9l5" Oct 31 04:14:31.730473 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:31.730457 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2553577b-451f-4d98-803d-0f81dd263822-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-f8a8088d-ft9l5\" (UID: \"2553577b-451f-4d98-803d-0f81dd263822\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f8a8088d-ft9l5" Oct 31 04:14:31.731290 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:31.731267 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2553577b-451f-4d98-803d-0f81dd263822-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-f8a8088d-ft9l5\" (UID: \"2553577b-451f-4d98-803d-0f81dd263822\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f8a8088d-ft9l5" Oct 31 04:14:31.733947 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:31.733928 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2553577b-451f-4d98-803d-0f81dd263822-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-f8a8088d-ft9l5\" (UID: \"2553577b-451f-4d98-803d-0f81dd263822\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f8a8088d-ft9l5" Oct 31 04:14:31.754456 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:31.754427 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8wfv\" (UniqueName: \"kubernetes.io/projected/2553577b-451f-4d98-803d-0f81dd263822-kube-api-access-j8wfv\") pod \"maas-api-revoke-keys-e2e-derive-f8a8088d-ft9l5\" (UID: \"2553577b-451f-4d98-803d-0f81dd263822\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f8a8088d-ft9l5" Oct 31 04:14:31.854363 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:31.854263 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f8a8088d-ft9l5" Oct 31 04:14:32.146268 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:32.146176 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44f032ad-fzt68" event={"ID":"f6382469-cb51-401e-bccf-cbf15f15075c","Type":"ContainerStarted","Data":"8eb9b51dc9e91baf59b100cbb21f72d6f39ac01d6efc2dbcec3e885b7dbb8d42"} Oct 31 04:14:32.146268 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:32.146242 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44f032ad-fzt68" event={"ID":"f6382469-cb51-401e-bccf-cbf15f15075c","Type":"ContainerStarted","Data":"d550219d3045e321d82cfcb9034a86ad23c151a1aff43cf4b5a2c8da74c0bee5"} Oct 31 04:14:32.227263 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:32.227158 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44f032ad-fzt68" podStartSLOduration=2.2271362359999998 podStartE2EDuration="2.227136236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:14:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:14:32.21785031 +0000 UTC m=+626.974188351" watchObservedRunningTime="2026-10-31 04:14:32.227136236 +0000 UTC m=+626.983474269" Oct 31 04:14:32.233074 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:32.233028 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f8a8088d-ft9l5"] Oct 31 04:14:32.306958 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:32.306479 5019 patch_prober.go:29] interesting pod/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 31 04:14:32.306958 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:32.306547 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb" podUID="ec1714c9-5e99-400a-b392-486c04926361" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 31 04:14:32.450572 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:32.450479 5019 patch_prober.go:29] interesting pod/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 31 04:14:32.450572 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:32.450537 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc" podUID="c65c601e-7e95-4e7d-844c-a4cadd680d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 31 04:14:33.217604 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:33.215672 5019 generic.go:356] "Generic (PLEG): container finished" podID="f6382469-cb51-401e-bccf-cbf15f15075c" containerID="8eb9b51dc9e91baf59b100cbb21f72d6f39ac01d6efc2dbcec3e885b7dbb8d42" exitCode=0 Oct 31 04:14:33.217604 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:33.215989 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44f032ad-fzt68" event={"ID":"f6382469-cb51-401e-bccf-cbf15f15075c","Type":"ContainerDied","Data":"8eb9b51dc9e91baf59b100cbb21f72d6f39ac01d6efc2dbcec3e885b7dbb8d42"} Oct 31 04:14:33.227074 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:33.226250 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f8a8088d-ft9l5" event={"ID":"2553577b-451f-4d98-803d-0f81dd263822","Type":"ContainerStarted","Data":"aa6275cbc5f84d8c407867f7f4563a135a0a1c2c9f1038af018d8a063bcf6504"} Oct 31 04:14:33.227074 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:33.226305 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f8a8088d-ft9l5" event={"ID":"2553577b-451f-4d98-803d-0f81dd263822","Type":"ContainerStarted","Data":"d92bbdd5016e5c12ce9ffc543c9fa0dc3e7f99adf888bb94d52cd6bbcbbbd174"} Oct 31 04:14:33.304246 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:33.304158 5019 patch_prober.go:29] interesting pod/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 31 04:14:33.304246 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:33.304219 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb" podUID="ec1714c9-5e99-400a-b392-486c04926361" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 31 04:14:33.314709 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:33.314656 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f8a8088d-ft9l5" podStartSLOduration=2.314638167 podStartE2EDuration="2.314638167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:14:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:14:33.311161484 +0000 UTC m=+628.067499525" watchObservedRunningTime="2026-10-31 04:14:33.314638167 +0000 UTC m=+628.070976207" Oct 31 04:14:33.455102 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:33.455012 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc" Oct 31 04:14:33.455386 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:33.455364 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc" Oct 31 04:14:33.456186 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:33.456165 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc" Oct 31 04:14:33.713615 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:33.713503 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cda44a-fq5wl"] Oct 31 04:14:33.716710 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:33.716657 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cda44a-fq5wl"] Oct 31 04:14:34.234562 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:34.234521 5019 generic.go:356] "Generic (PLEG): container finished" podID="2553577b-451f-4d98-803d-0f81dd263822" containerID="aa6275cbc5f84d8c407867f7f4563a135a0a1c2c9f1038af018d8a063bcf6504" exitCode=0 Oct 31 04:14:34.234792 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:34.234766 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f8a8088d-ft9l5" event={"ID":"2553577b-451f-4d98-803d-0f81dd263822","Type":"ContainerDied","Data":"aa6275cbc5f84d8c407867f7f4563a135a0a1c2c9f1038af018d8a063bcf6504"} Oct 31 04:14:34.236698 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:34.236669 5019 generic.go:356] "Generic (PLEG): container finished" podID="3dd4051c-8d79-496e-b3cb-d90263e6f1be" containerID="afce53d39f9ff4fcef00d8bc832bcecb6cf0d5396985dd3e4a85a7327ea06788" exitCode=0 Oct 31 04:14:34.236855 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:34.236750 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr" event={"ID":"3dd4051c-8d79-496e-b3cb-d90263e6f1be","Type":"ContainerDied","Data":"afce53d39f9ff4fcef00d8bc832bcecb6cf0d5396985dd3e4a85a7327ea06788"} Oct 31 04:14:34.237409 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:34.237384 5019 scope.go:122] "RemoveContainer" containerID="afce53d39f9ff4fcef00d8bc832bcecb6cf0d5396985dd3e4a85a7327ea06788" Oct 31 04:14:34.238735 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:34.238689 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:14:34.238840 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:34.238787 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:14:34.238840 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:34.238827 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:14:34.305009 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:34.304621 5019 patch_prober.go:29] interesting pod/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 31 04:14:34.305009 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:34.304675 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb" podUID="ec1714c9-5e99-400a-b392-486c04926361" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 31 04:14:34.497655 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:34.497567 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44f032ad-fzt68" Oct 31 04:14:34.539974 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:34.539892 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-a0d1a61d-6499787965-w979c" podUID="533c24d6-b721-4f7f-afea-6fea0bac4a51" containerName="payload-processing" containerID="cri-o://082790aeda772b89b0519a5017a1e677bfc18ffd69065f169478570399d263db" gracePeriod=25 Oct 31 04:14:34.568578 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:34.567617 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6382469-cb51-401e-bccf-cbf15f15075c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6382469-cb51-401e-bccf-cbf15f15075c-maas-api-xks-ca\") pod \"f6382469-cb51-401e-bccf-cbf15f15075c\" (UID: \"f6382469-cb51-401e-bccf-cbf15f15075c\") " Oct 31 04:14:34.568578 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:34.567747 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6382469-cb51-401e-bccf-cbf15f15075c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6382469-cb51-401e-bccf-cbf15f15075c-maas-api-openshift-ca\") pod \"f6382469-cb51-401e-bccf-cbf15f15075c\" (UID: \"f6382469-cb51-401e-bccf-cbf15f15075c\") " Oct 31 04:14:34.568578 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:34.567843 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6382469-cb51-401e-bccf-cbf15f15075c-kube-api-access-v2jzb\" (UniqueName: \"kubernetes.io/projected/f6382469-cb51-401e-bccf-cbf15f15075c-kube-api-access-v2jzb\") pod \"f6382469-cb51-401e-bccf-cbf15f15075c\" (UID: \"f6382469-cb51-401e-bccf-cbf15f15075c\") " Oct 31 04:14:34.571821 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:34.571559 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6382469-cb51-401e-bccf-cbf15f15075c-kube-api-access-v2jzb" pod "f6382469-cb51-401e-bccf-cbf15f15075c" (UID: "f6382469-cb51-401e-bccf-cbf15f15075c"). InnerVolumeSpecName "kube-api-access-v2jzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:14:34.572209 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:34.572149 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6382469-cb51-401e-bccf-cbf15f15075c-maas-api-openshift-ca" pod "f6382469-cb51-401e-bccf-cbf15f15075c" (UID: "f6382469-cb51-401e-bccf-cbf15f15075c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:14:34.574041 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:34.574004 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6382469-cb51-401e-bccf-cbf15f15075c-maas-api-xks-ca" pod "f6382469-cb51-401e-bccf-cbf15f15075c" (UID: "f6382469-cb51-401e-bccf-cbf15f15075c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:14:34.670262 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:34.670213 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6382469-cb51-401e-bccf-cbf15f15075c-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:14:34.670262 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:34.670264 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6382469-cb51-401e-bccf-cbf15f15075c-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:14:34.670540 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:34.670282 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2jzb\" (UniqueName: \"kubernetes.io/projected/f6382469-cb51-401e-bccf-cbf15f15075c-kube-api-access-v2jzb\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:14:34.847111 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:34.847071 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-a0d1a61d-6499787965-w979c" Oct 31 04:14:34.902367 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:34.885757 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/533c24d6-b721-4f7f-afea-6fea0bac4a51-kube-api-access-mbsgp\" (UniqueName: \"kubernetes.io/projected/533c24d6-b721-4f7f-afea-6fea0bac4a51-kube-api-access-mbsgp\") pod \"533c24d6-b721-4f7f-afea-6fea0bac4a51\" (UID: \"533c24d6-b721-4f7f-afea-6fea0bac4a51\") " Oct 31 04:14:34.902367 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:34.885859 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/533c24d6-b721-4f7f-afea-6fea0bac4a51-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/533c24d6-b721-4f7f-afea-6fea0bac4a51-plugins-config-volume\") pod \"533c24d6-b721-4f7f-afea-6fea0bac4a51\" (UID: \"533c24d6-b721-4f7f-afea-6fea0bac4a51\") " Oct 31 04:14:34.903392 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:34.903046 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/533c24d6-b721-4f7f-afea-6fea0bac4a51-plugins-config-volume" pod "533c24d6-b721-4f7f-afea-6fea0bac4a51" (UID: "533c24d6-b721-4f7f-afea-6fea0bac4a51"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:14:34.909857 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:34.909809 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/533c24d6-b721-4f7f-afea-6fea0bac4a51-kube-api-access-mbsgp" pod "533c24d6-b721-4f7f-afea-6fea0bac4a51" (UID: "533c24d6-b721-4f7f-afea-6fea0bac4a51"). InnerVolumeSpecName "kube-api-access-mbsgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:14:34.994387 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:34.993984 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbsgp\" (UniqueName: \"kubernetes.io/projected/533c24d6-b721-4f7f-afea-6fea0bac4a51-kube-api-access-mbsgp\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:14:34.994387 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:34.994025 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/533c24d6-b721-4f7f-afea-6fea0bac4a51-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:14:35.203421 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.203298 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx"] Oct 31 04:14:35.215354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.211240 5019 cpu_manager.go:414] "containerMap: removing container" podUID="f6382469-cb51-401e-bccf-cbf15f15075c" containerName="revoke-keys" Oct 31 04:14:35.215354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.211277 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6382469-cb51-401e-bccf-cbf15f15075c" containerName="revoke-keys" Oct 31 04:14:35.215354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.211309 5019 cpu_manager.go:414] "containerMap: removing container" podUID="533c24d6-b721-4f7f-afea-6fea0bac4a51" containerName="payload-processing" Oct 31 04:14:35.215354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.211317 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="533c24d6-b721-4f7f-afea-6fea0bac4a51" containerName="payload-processing" Oct 31 04:14:35.215354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.211562 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6382469-cb51-401e-bccf-cbf15f15075c" containerName="revoke-keys" Oct 31 04:14:35.215354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.211580 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="533c24d6-b721-4f7f-afea-6fea0bac4a51" containerName="payload-processing" Oct 31 04:14:35.219354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.217229 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx" Oct 31 04:14:35.229244 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.229057 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx"] Oct 31 04:14:35.238320 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.234817 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-09a20a-openshift-default-dockercfg-hf9qb\"" Oct 31 04:14:35.249886 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.249850 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr" event={"ID":"3dd4051c-8d79-496e-b3cb-d90263e6f1be","Type":"ContainerStarted","Data":"4740e12b96e508a28489b8c3f623a13d68cc2b35e989d52cb19f9fd569ca6157"} Oct 31 04:14:35.252175 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.252138 5019 generic.go:356] "Generic (PLEG): container finished" podID="533c24d6-b721-4f7f-afea-6fea0bac4a51" containerID="082790aeda772b89b0519a5017a1e677bfc18ffd69065f169478570399d263db" exitCode=0 Oct 31 04:14:35.252175 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.252182 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-a0d1a61d-6499787965-w979c" Oct 31 04:14:35.252415 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.252317 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-a0d1a61d-6499787965-w979c" event={"ID":"533c24d6-b721-4f7f-afea-6fea0bac4a51","Type":"ContainerDied","Data":"082790aeda772b89b0519a5017a1e677bfc18ffd69065f169478570399d263db"} Oct 31 04:14:35.252415 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.252374 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-a0d1a61d-6499787965-w979c" event={"ID":"533c24d6-b721-4f7f-afea-6fea0bac4a51","Type":"ContainerDied","Data":"de6183c95411a4b6075d37f59ab6ca6c78196e6010ce2b11e9f993255cdb163a"} Oct 31 04:14:35.252415 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.252401 5019 scope.go:122] "RemoveContainer" containerID="082790aeda772b89b0519a5017a1e677bfc18ffd69065f169478570399d263db" Oct 31 04:14:35.257001 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.255437 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44f032ad-fzt68" event={"ID":"f6382469-cb51-401e-bccf-cbf15f15075c","Type":"ContainerDied","Data":"d550219d3045e321d82cfcb9034a86ad23c151a1aff43cf4b5a2c8da74c0bee5"} Oct 31 04:14:35.257001 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.255498 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d550219d3045e321d82cfcb9034a86ad23c151a1aff43cf4b5a2c8da74c0bee5" Oct 31 04:14:35.257001 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.255760 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44f032ad-fzt68" Oct 31 04:14:35.309383 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.307522 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-istio-envoy\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx\" (UID: \"cc1adc72-6548-4daf-9dc1-2b96c853cdbf\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx" Oct 31 04:14:35.309383 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.307597 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-istiod-ca-cert\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx\" (UID: \"cc1adc72-6548-4daf-9dc1-2b96c853cdbf\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx" Oct 31 04:14:35.309383 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.307656 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-workload-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx\" (UID: \"cc1adc72-6548-4daf-9dc1-2b96c853cdbf\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx" Oct 31 04:14:35.309383 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.307795 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-istio-podinfo\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx\" (UID: \"cc1adc72-6548-4daf-9dc1-2b96c853cdbf\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx" Oct 31 04:14:35.309383 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.307885 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-workload-certs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx\" (UID: \"cc1adc72-6548-4daf-9dc1-2b96c853cdbf\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx" Oct 31 04:14:35.309383 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.307915 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-istio-data\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx\" (UID: \"cc1adc72-6548-4daf-9dc1-2b96c853cdbf\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx" Oct 31 04:14:35.309383 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.307938 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-istio-token\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx\" (UID: \"cc1adc72-6548-4daf-9dc1-2b96c853cdbf\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx" Oct 31 04:14:35.309383 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.308022 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlnj8\" (UniqueName: \"kubernetes.io/projected/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-kube-api-access-rlnj8\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx\" (UID: \"cc1adc72-6548-4daf-9dc1-2b96c853cdbf\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx" Oct 31 04:14:35.309383 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.308104 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-credential-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx\" (UID: \"cc1adc72-6548-4daf-9dc1-2b96c853cdbf\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx" Oct 31 04:14:35.330848 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.329702 5019 scope.go:122] "RemoveContainer" containerID="082790aeda772b89b0519a5017a1e677bfc18ffd69065f169478570399d263db" Oct 31 04:14:35.331225 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:14:35.331166 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"082790aeda772b89b0519a5017a1e677bfc18ffd69065f169478570399d263db\": container with ID starting with 082790aeda772b89b0519a5017a1e677bfc18ffd69065f169478570399d263db not found: ID does not exist" containerID="082790aeda772b89b0519a5017a1e677bfc18ffd69065f169478570399d263db" Oct 31 04:14:35.352542 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.352498 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"082790aeda772b89b0519a5017a1e677bfc18ffd69065f169478570399d263db"} err="failed to get container status \"082790aeda772b89b0519a5017a1e677bfc18ffd69065f169478570399d263db\": rpc error: code = NotFound desc = could not find container \"082790aeda772b89b0519a5017a1e677bfc18ffd69065f169478570399d263db\": container with ID starting with 082790aeda772b89b0519a5017a1e677bfc18ffd69065f169478570399d263db not found: ID does not exist" Oct 31 04:14:35.367438 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.367383 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr" Oct 31 04:14:35.426393 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.374579 5019 patch_prober.go:29] interesting pod/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 31 04:14:35.426393 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.374643 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr" podUID="3dd4051c-8d79-496e-b3cb-d90263e6f1be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 31 04:14:35.426393 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.384730 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb" Oct 31 04:14:35.426393 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.410483 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-istio-token\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx\" (UID: \"cc1adc72-6548-4daf-9dc1-2b96c853cdbf\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx" Oct 31 04:14:35.426393 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.410593 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlnj8\" (UniqueName: \"kubernetes.io/projected/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-kube-api-access-rlnj8\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx\" (UID: \"cc1adc72-6548-4daf-9dc1-2b96c853cdbf\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx" Oct 31 04:14:35.426393 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.410649 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-credential-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx\" (UID: \"cc1adc72-6548-4daf-9dc1-2b96c853cdbf\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx" Oct 31 04:14:35.426393 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.410704 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-istio-envoy\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx\" (UID: \"cc1adc72-6548-4daf-9dc1-2b96c853cdbf\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx" Oct 31 04:14:35.426393 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.410748 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-istiod-ca-cert\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx\" (UID: \"cc1adc72-6548-4daf-9dc1-2b96c853cdbf\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx" Oct 31 04:14:35.426393 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.410832 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-workload-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx\" (UID: \"cc1adc72-6548-4daf-9dc1-2b96c853cdbf\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx" Oct 31 04:14:35.426393 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.410964 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-istio-podinfo\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx\" (UID: \"cc1adc72-6548-4daf-9dc1-2b96c853cdbf\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx" Oct 31 04:14:35.426393 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.411052 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-workload-certs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx\" (UID: \"cc1adc72-6548-4daf-9dc1-2b96c853cdbf\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx" Oct 31 04:14:35.426393 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.411083 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-istio-data\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx\" (UID: \"cc1adc72-6548-4daf-9dc1-2b96c853cdbf\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx" Oct 31 04:14:35.426393 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.411605 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-istio-data\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx\" (UID: \"cc1adc72-6548-4daf-9dc1-2b96c853cdbf\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx" Oct 31 04:14:35.426393 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.419383 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-credential-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx\" (UID: \"cc1adc72-6548-4daf-9dc1-2b96c853cdbf\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx" Oct 31 04:14:35.426393 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.420401 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-istiod-ca-cert\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx\" (UID: \"cc1adc72-6548-4daf-9dc1-2b96c853cdbf\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx" Oct 31 04:14:35.426393 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.420989 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-workload-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx\" (UID: \"cc1adc72-6548-4daf-9dc1-2b96c853cdbf\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx" Oct 31 04:14:35.426393 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.423078 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb" Oct 31 04:14:35.426393 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.423161 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb" Oct 31 04:14:35.426393 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.423515 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-workload-certs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx\" (UID: \"cc1adc72-6548-4daf-9dc1-2b96c853cdbf\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx" Oct 31 04:14:35.430455 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.426607 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-istio-podinfo\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx\" (UID: \"cc1adc72-6548-4daf-9dc1-2b96c853cdbf\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx" Oct 31 04:14:35.435785 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.435519 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e28a4486-3e22-45ad-bcd4-62bfd11d49dd" path="/var/lib/kubelet/pods/e28a4486-3e22-45ad-bcd4-62bfd11d49dd/volumes" Oct 31 04:14:35.440700 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.440539 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-istio-envoy\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx\" (UID: \"cc1adc72-6548-4daf-9dc1-2b96c853cdbf\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx" Oct 31 04:14:35.441296 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.441231 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-istio-token\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx\" (UID: \"cc1adc72-6548-4daf-9dc1-2b96c853cdbf\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx" Oct 31 04:14:35.445500 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.445455 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlnj8\" (UniqueName: \"kubernetes.io/projected/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-kube-api-access-rlnj8\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx\" (UID: \"cc1adc72-6548-4daf-9dc1-2b96c853cdbf\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx" Oct 31 04:14:35.510010 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.509943 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a0d1a61d-6499787965-w979c"] Oct 31 04:14:35.514745 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.514701 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a0d1a61d-6499787965-w979c"] Oct 31 04:14:35.543692 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.543648 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx" Oct 31 04:14:35.608545 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.608512 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f8a8088d-ft9l5" Oct 31 04:14:35.619299 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.619263 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr" Oct 31 04:14:35.714370 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.714037 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2553577b-451f-4d98-803d-0f81dd263822-kube-api-access-j8wfv\" (UniqueName: \"kubernetes.io/projected/2553577b-451f-4d98-803d-0f81dd263822-kube-api-access-j8wfv\") pod \"2553577b-451f-4d98-803d-0f81dd263822\" (UID: \"2553577b-451f-4d98-803d-0f81dd263822\") " Oct 31 04:14:35.714370 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.714127 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2553577b-451f-4d98-803d-0f81dd263822-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2553577b-451f-4d98-803d-0f81dd263822-maas-api-openshift-ca\") pod \"2553577b-451f-4d98-803d-0f81dd263822\" (UID: \"2553577b-451f-4d98-803d-0f81dd263822\") " Oct 31 04:14:35.714370 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.714152 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2553577b-451f-4d98-803d-0f81dd263822-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2553577b-451f-4d98-803d-0f81dd263822-maas-api-xks-ca\") pod \"2553577b-451f-4d98-803d-0f81dd263822\" (UID: \"2553577b-451f-4d98-803d-0f81dd263822\") " Oct 31 04:14:35.715574 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.715517 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2553577b-451f-4d98-803d-0f81dd263822-maas-api-openshift-ca" pod "2553577b-451f-4d98-803d-0f81dd263822" (UID: "2553577b-451f-4d98-803d-0f81dd263822"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:14:35.719644 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.719416 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2553577b-451f-4d98-803d-0f81dd263822-maas-api-xks-ca" pod "2553577b-451f-4d98-803d-0f81dd263822" (UID: "2553577b-451f-4d98-803d-0f81dd263822"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:14:35.719951 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.719899 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2553577b-451f-4d98-803d-0f81dd263822-kube-api-access-j8wfv" pod "2553577b-451f-4d98-803d-0f81dd263822" (UID: "2553577b-451f-4d98-803d-0f81dd263822"). InnerVolumeSpecName "kube-api-access-j8wfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:14:35.817223 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.817119 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8wfv\" (UniqueName: \"kubernetes.io/projected/2553577b-451f-4d98-803d-0f81dd263822-kube-api-access-j8wfv\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:14:35.817223 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.817159 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2553577b-451f-4d98-803d-0f81dd263822-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:14:35.817223 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.817171 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2553577b-451f-4d98-803d-0f81dd263822-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:14:35.902100 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.902021 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx"] Oct 31 04:14:35.906829 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:14:35.906793 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc1adc72_6548_4daf_9dc1_2b96c853cdbf.slice/crio-abeba8990513c0e70f11cd581c43ed10c9259b29230b4729fdb9aa347937146d WatchSource:0}: Error finding container abeba8990513c0e70f11cd581c43ed10c9259b29230b4729fdb9aa347937146d: Status 404 returned error can't find the container with id abeba8990513c0e70f11cd581c43ed10c9259b29230b4729fdb9aa347937146d Oct 31 04:14:35.909766 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.909721 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:14:35.909899 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.909819 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:14:35.909899 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:35.909870 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:14:36.126697 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:36.126600 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0d1a61d-6hsws"] Oct 31 04:14:36.130879 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:36.130840 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0d1a61d-6hsws"] Oct 31 04:14:36.264688 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:36.264650 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f8a8088d-ft9l5" event={"ID":"2553577b-451f-4d98-803d-0f81dd263822","Type":"ContainerDied","Data":"d92bbdd5016e5c12ce9ffc543c9fa0dc3e7f99adf888bb94d52cd6bbcbbbd174"} Oct 31 04:14:36.264874 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:36.264707 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d92bbdd5016e5c12ce9ffc543c9fa0dc3e7f99adf888bb94d52cd6bbcbbbd174" Oct 31 04:14:36.264874 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:36.264794 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f8a8088d-ft9l5" Oct 31 04:14:36.308477 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:36.308434 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx" event={"ID":"cc1adc72-6548-4daf-9dc1-2b96c853cdbf","Type":"ContainerStarted","Data":"8b49042a4d5200c98a78e4465ba5fbf55837632fc4c8f40d09980add49bdd0fc"} Oct 31 04:14:36.308624 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:36.308525 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx" event={"ID":"cc1adc72-6548-4daf-9dc1-2b96c853cdbf","Type":"ContainerStarted","Data":"abeba8990513c0e70f11cd581c43ed10c9259b29230b4729fdb9aa347937146d"} Oct 31 04:14:36.309851 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:36.309823 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n"] Oct 31 04:14:36.310294 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:36.310029 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n" podUID="6b943d4d-c829-410c-9233-0b53cbc1c414" containerName="istio-proxy" containerID="cri-o://adac2c933bec465a85952b2d64beeac94099d19d13c9bc5b080bad43f9376efa" gracePeriod=30 Oct 31 04:14:36.315052 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:36.314910 5019 patch_prober.go:29] interesting pod/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 31 04:14:36.315052 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:36.314981 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr" podUID="3dd4051c-8d79-496e-b3cb-d90263e6f1be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 31 04:14:36.433409 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:36.427854 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx" podStartSLOduration=1.427830112 podStartE2EDuration="1.427830112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:14:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:14:36.4248235 +0000 UTC m=+631.181161579" watchObservedRunningTime="2026-10-31 04:14:36.427830112 +0000 UTC m=+631.184168154" Oct 31 04:14:36.550285 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:36.550254 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx" Oct 31 04:14:37.314178 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:37.314139 5019 patch_prober.go:29] interesting pod/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 31 04:14:37.314790 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:37.314756 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr" podUID="3dd4051c-8d79-496e-b3cb-d90263e6f1be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 31 04:14:37.408813 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:37.408775 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="533c24d6-b721-4f7f-afea-6fea0bac4a51" path="/var/lib/kubelet/pods/533c24d6-b721-4f7f-afea-6fea0bac4a51/volumes" Oct 31 04:14:37.409247 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:37.409231 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8266bc2-1ff3-40c1-8e82-e91d7f25e6df" path="/var/lib/kubelet/pods/a8266bc2-1ff3-40c1-8e82-e91d7f25e6df/volumes" Oct 31 04:14:37.550492 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:37.550458 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx" Oct 31 04:14:37.634604 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:37.634498 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v"] Oct 31 04:14:37.635147 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:37.635124 5019 cpu_manager.go:414] "containerMap: removing container" podUID="2553577b-451f-4d98-803d-0f81dd263822" containerName="revoke-keys" Oct 31 04:14:37.635147 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:37.635147 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2553577b-451f-4d98-803d-0f81dd263822" containerName="revoke-keys" Oct 31 04:14:37.635388 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:37.635280 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="2553577b-451f-4d98-803d-0f81dd263822" containerName="revoke-keys" Oct 31 04:14:37.642944 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:37.640760 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v" Oct 31 04:14:37.664849 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:37.663483 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v"] Oct 31 04:14:37.664849 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:37.663570 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e33d7c-168f-4001-8e39-22bea132574a-workload-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v\" (UID: \"c4e33d7c-168f-4001-8e39-22bea132574a\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v" Oct 31 04:14:37.664849 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:37.663614 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4e33d7c-168f-4001-8e39-22bea132574a-istiod-ca-cert\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v\" (UID: \"c4e33d7c-168f-4001-8e39-22bea132574a\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v" Oct 31 04:14:37.664849 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:37.663658 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4e33d7c-168f-4001-8e39-22bea132574a-istio-token\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v\" (UID: \"c4e33d7c-168f-4001-8e39-22bea132574a\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v" Oct 31 04:14:37.664849 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:37.663692 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4e33d7c-168f-4001-8e39-22bea132574a-workload-certs\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v\" (UID: \"c4e33d7c-168f-4001-8e39-22bea132574a\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v" Oct 31 04:14:37.664849 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:37.663801 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4e33d7c-168f-4001-8e39-22bea132574a-istio-data\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v\" (UID: \"c4e33d7c-168f-4001-8e39-22bea132574a\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v" Oct 31 04:14:37.664849 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:37.663856 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg428\" (UniqueName: \"kubernetes.io/projected/c4e33d7c-168f-4001-8e39-22bea132574a-kube-api-access-dg428\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v\" (UID: \"c4e33d7c-168f-4001-8e39-22bea132574a\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v" Oct 31 04:14:37.664849 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:37.663923 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e33d7c-168f-4001-8e39-22bea132574a-credential-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v\" (UID: \"c4e33d7c-168f-4001-8e39-22bea132574a\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v" Oct 31 04:14:37.664849 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:37.663946 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4e33d7c-168f-4001-8e39-22bea132574a-istio-podinfo\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v\" (UID: \"c4e33d7c-168f-4001-8e39-22bea132574a\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v" Oct 31 04:14:37.664849 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:37.663995 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4e33d7c-168f-4001-8e39-22bea132574a-istio-envoy\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v\" (UID: \"c4e33d7c-168f-4001-8e39-22bea132574a\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v" Oct 31 04:14:37.721717 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:37.721671 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-4c2387-openshift-default-dockercfg-x4j6w\"" Oct 31 04:14:37.789279 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:37.789236 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4e33d7c-168f-4001-8e39-22bea132574a-istio-data\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v\" (UID: \"c4e33d7c-168f-4001-8e39-22bea132574a\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v" Oct 31 04:14:37.789622 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:37.789592 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg428\" (UniqueName: \"kubernetes.io/projected/c4e33d7c-168f-4001-8e39-22bea132574a-kube-api-access-dg428\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v\" (UID: \"c4e33d7c-168f-4001-8e39-22bea132574a\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v" Oct 31 04:14:37.789846 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:37.789830 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e33d7c-168f-4001-8e39-22bea132574a-credential-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v\" (UID: \"c4e33d7c-168f-4001-8e39-22bea132574a\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v" Oct 31 04:14:37.789958 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:37.789945 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4e33d7c-168f-4001-8e39-22bea132574a-istio-podinfo\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v\" (UID: \"c4e33d7c-168f-4001-8e39-22bea132574a\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v" Oct 31 04:14:37.790090 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:37.790077 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4e33d7c-168f-4001-8e39-22bea132574a-istio-envoy\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v\" (UID: \"c4e33d7c-168f-4001-8e39-22bea132574a\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v" Oct 31 04:14:37.790218 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:37.790204 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e33d7c-168f-4001-8e39-22bea132574a-workload-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v\" (UID: \"c4e33d7c-168f-4001-8e39-22bea132574a\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v" Oct 31 04:14:37.790326 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:37.790314 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4e33d7c-168f-4001-8e39-22bea132574a-istiod-ca-cert\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v\" (UID: \"c4e33d7c-168f-4001-8e39-22bea132574a\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v" Oct 31 04:14:37.795422 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:37.790480 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4e33d7c-168f-4001-8e39-22bea132574a-istio-token\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v\" (UID: \"c4e33d7c-168f-4001-8e39-22bea132574a\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v" Oct 31 04:14:37.799360 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:37.795835 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4e33d7c-168f-4001-8e39-22bea132574a-workload-certs\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v\" (UID: \"c4e33d7c-168f-4001-8e39-22bea132574a\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v" Oct 31 04:14:37.823459 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:37.815094 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4e33d7c-168f-4001-8e39-22bea132574a-istio-data\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v\" (UID: \"c4e33d7c-168f-4001-8e39-22bea132574a\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v" Oct 31 04:14:37.823459 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:37.815754 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e33d7c-168f-4001-8e39-22bea132574a-credential-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v\" (UID: \"c4e33d7c-168f-4001-8e39-22bea132574a\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v" Oct 31 04:14:37.823459 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:37.816176 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e33d7c-168f-4001-8e39-22bea132574a-workload-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v\" (UID: \"c4e33d7c-168f-4001-8e39-22bea132574a\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v" Oct 31 04:14:37.841031 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:37.839070 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4e33d7c-168f-4001-8e39-22bea132574a-istio-envoy\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v\" (UID: \"c4e33d7c-168f-4001-8e39-22bea132574a\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v" Oct 31 04:14:37.841031 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:37.839754 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4e33d7c-168f-4001-8e39-22bea132574a-workload-certs\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v\" (UID: \"c4e33d7c-168f-4001-8e39-22bea132574a\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v" Oct 31 04:14:37.841031 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:37.839942 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4e33d7c-168f-4001-8e39-22bea132574a-istiod-ca-cert\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v\" (UID: \"c4e33d7c-168f-4001-8e39-22bea132574a\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v" Oct 31 04:14:37.846693 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:37.843696 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4e33d7c-168f-4001-8e39-22bea132574a-istio-podinfo\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v\" (UID: \"c4e33d7c-168f-4001-8e39-22bea132574a\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v" Oct 31 04:14:37.846693 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:37.844221 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg428\" (UniqueName: \"kubernetes.io/projected/c4e33d7c-168f-4001-8e39-22bea132574a-kube-api-access-dg428\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v\" (UID: \"c4e33d7c-168f-4001-8e39-22bea132574a\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v" Oct 31 04:14:37.863213 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:37.863171 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4e33d7c-168f-4001-8e39-22bea132574a-istio-token\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v\" (UID: \"c4e33d7c-168f-4001-8e39-22bea132574a\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v" Oct 31 04:14:37.967479 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:37.967450 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v" Oct 31 04:14:38.313209 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:38.313099 5019 patch_prober.go:29] interesting pod/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 31 04:14:38.313209 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:38.313179 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr" podUID="3dd4051c-8d79-496e-b3cb-d90263e6f1be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 31 04:14:38.319616 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:38.319571 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v"] Oct 31 04:14:38.320009 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:14:38.319754 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4e33d7c_168f_4001_8e39_22bea132574a.slice/crio-330833c6127d895bb14be86276cc63041d5ec617006184cc090b5564e51b408e WatchSource:0}: Error finding container 330833c6127d895bb14be86276cc63041d5ec617006184cc090b5564e51b408e: Status 404 returned error can't find the container with id 330833c6127d895bb14be86276cc63041d5ec617006184cc090b5564e51b408e Oct 31 04:14:38.322804 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:38.322767 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:14:38.322898 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:38.322843 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:14:38.322898 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:38.322876 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:14:38.363237 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:38.363204 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v" event={"ID":"c4e33d7c-168f-4001-8e39-22bea132574a","Type":"ContainerStarted","Data":"330833c6127d895bb14be86276cc63041d5ec617006184cc090b5564e51b408e"} Oct 31 04:14:38.363630 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:38.363610 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx" Oct 31 04:14:38.364730 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:38.364711 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx" Oct 31 04:14:39.313798 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:39.313763 5019 patch_prober.go:29] interesting pod/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 31 04:14:39.313980 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:39.313817 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr" podUID="3dd4051c-8d79-496e-b3cb-d90263e6f1be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 31 04:14:39.396499 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:39.396455 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v" event={"ID":"c4e33d7c-168f-4001-8e39-22bea132574a","Type":"ContainerStarted","Data":"13130f4cbef0ca565dad709c9689dde75da0cc0e67e923aa4f059cdd2af27a83"} Oct 31 04:14:39.441684 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:39.441624 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v" podStartSLOduration=2.441607979 podStartE2EDuration="2.441607979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:14:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:14:39.441301241 +0000 UTC m=+634.197639285" watchObservedRunningTime="2026-10-31 04:14:39.441607979 +0000 UTC m=+634.197946018" Oct 31 04:14:39.968068 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:39.968023 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v" Oct 31 04:14:39.998112 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:39.998077 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v" Oct 31 04:14:40.314485 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:40.313705 5019 patch_prober.go:29] interesting pod/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 31 04:14:40.314485 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:40.313803 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr" podUID="3dd4051c-8d79-496e-b3cb-d90263e6f1be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 31 04:14:40.408840 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:40.408802 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v" Oct 31 04:14:40.410071 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:40.410039 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v" Oct 31 04:14:40.768415 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:40.768368 5019 patch_prober.go:29] interesting pod/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 04:14:40.768598 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:40.768436 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n" podUID="6b943d4d-c829-410c-9233-0b53cbc1c414" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 04:14:40.808894 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:40.808851 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4805aacc-w7hhm"] Oct 31 04:14:40.813249 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:40.813205 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4805aacc-w7hhm"] Oct 31 04:14:41.319400 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:41.319367 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr" Oct 31 04:14:41.320691 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:41.320658 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr" Oct 31 04:14:41.408935 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:41.408896 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f813717a-cc13-46f3-b010-fd9898d4052a" path="/var/lib/kubelet/pods/f813717a-cc13-46f3-b010-fd9898d4052a/volumes" Oct 31 04:14:41.666351 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:41.666311 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n" Oct 31 04:14:41.768261 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:41.768218 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b943d4d-c829-410c-9233-0b53cbc1c414-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b943d4d-c829-410c-9233-0b53cbc1c414-istio-data\") pod \"6b943d4d-c829-410c-9233-0b53cbc1c414\" (UID: \"6b943d4d-c829-410c-9233-0b53cbc1c414\") " Oct 31 04:14:41.768507 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:41.768275 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b943d4d-c829-410c-9233-0b53cbc1c414-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b943d4d-c829-410c-9233-0b53cbc1c414-workload-socket\") pod \"6b943d4d-c829-410c-9233-0b53cbc1c414\" (UID: \"6b943d4d-c829-410c-9233-0b53cbc1c414\") " Oct 31 04:14:41.768507 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:41.768308 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b943d4d-c829-410c-9233-0b53cbc1c414-kube-api-access-f9qf5\" (UniqueName: \"kubernetes.io/projected/6b943d4d-c829-410c-9233-0b53cbc1c414-kube-api-access-f9qf5\") pod \"6b943d4d-c829-410c-9233-0b53cbc1c414\" (UID: \"6b943d4d-c829-410c-9233-0b53cbc1c414\") " Oct 31 04:14:41.768507 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:41.768363 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b943d4d-c829-410c-9233-0b53cbc1c414-istio-token\" (UniqueName: \"kubernetes.io/projected/6b943d4d-c829-410c-9233-0b53cbc1c414-istio-token\") pod \"6b943d4d-c829-410c-9233-0b53cbc1c414\" (UID: \"6b943d4d-c829-410c-9233-0b53cbc1c414\") " Oct 31 04:14:41.768507 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:41.768399 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b943d4d-c829-410c-9233-0b53cbc1c414-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b943d4d-c829-410c-9233-0b53cbc1c414-workload-certs\") pod \"6b943d4d-c829-410c-9233-0b53cbc1c414\" (UID: \"6b943d4d-c829-410c-9233-0b53cbc1c414\") " Oct 31 04:14:41.768728 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:41.768541 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b943d4d-c829-410c-9233-0b53cbc1c414-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b943d4d-c829-410c-9233-0b53cbc1c414-credential-socket\") pod \"6b943d4d-c829-410c-9233-0b53cbc1c414\" (UID: \"6b943d4d-c829-410c-9233-0b53cbc1c414\") " Oct 31 04:14:41.768728 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:41.768565 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b943d4d-c829-410c-9233-0b53cbc1c414-istio-data" pod "6b943d4d-c829-410c-9233-0b53cbc1c414" (UID: "6b943d4d-c829-410c-9233-0b53cbc1c414"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:14:41.768728 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:41.768611 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b943d4d-c829-410c-9233-0b53cbc1c414-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b943d4d-c829-410c-9233-0b53cbc1c414-istiod-ca-cert\") pod \"6b943d4d-c829-410c-9233-0b53cbc1c414\" (UID: \"6b943d4d-c829-410c-9233-0b53cbc1c414\") " Oct 31 04:14:41.768728 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:41.768663 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b943d4d-c829-410c-9233-0b53cbc1c414-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b943d4d-c829-410c-9233-0b53cbc1c414-istio-podinfo\") pod \"6b943d4d-c829-410c-9233-0b53cbc1c414\" (UID: \"6b943d4d-c829-410c-9233-0b53cbc1c414\") " Oct 31 04:14:41.768728 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:41.768692 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b943d4d-c829-410c-9233-0b53cbc1c414-workload-socket" pod "6b943d4d-c829-410c-9233-0b53cbc1c414" (UID: "6b943d4d-c829-410c-9233-0b53cbc1c414"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:14:41.768728 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:41.768708 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b943d4d-c829-410c-9233-0b53cbc1c414-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b943d4d-c829-410c-9233-0b53cbc1c414-istio-envoy\") pod \"6b943d4d-c829-410c-9233-0b53cbc1c414\" (UID: \"6b943d4d-c829-410c-9233-0b53cbc1c414\") " Oct 31 04:14:41.769034 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:41.768704 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b943d4d-c829-410c-9233-0b53cbc1c414-workload-certs" pod "6b943d4d-c829-410c-9233-0b53cbc1c414" (UID: "6b943d4d-c829-410c-9233-0b53cbc1c414"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:14:41.769034 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:41.768864 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b943d4d-c829-410c-9233-0b53cbc1c414-credential-socket" pod "6b943d4d-c829-410c-9233-0b53cbc1c414" (UID: "6b943d4d-c829-410c-9233-0b53cbc1c414"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:14:41.769151 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:41.769058 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b943d4d-c829-410c-9233-0b53cbc1c414-istiod-ca-cert" pod "6b943d4d-c829-410c-9233-0b53cbc1c414" (UID: "6b943d4d-c829-410c-9233-0b53cbc1c414"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:14:41.769256 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:41.769234 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b943d4d-c829-410c-9233-0b53cbc1c414-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:14:41.769430 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:41.769260 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b943d4d-c829-410c-9233-0b53cbc1c414-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:14:41.769430 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:41.769275 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b943d4d-c829-410c-9233-0b53cbc1c414-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:14:41.769430 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:41.769292 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b943d4d-c829-410c-9233-0b53cbc1c414-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:14:41.769430 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:41.769307 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b943d4d-c829-410c-9233-0b53cbc1c414-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:14:41.771003 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:41.770973 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b943d4d-c829-410c-9233-0b53cbc1c414-istio-token" pod "6b943d4d-c829-410c-9233-0b53cbc1c414" (UID: "6b943d4d-c829-410c-9233-0b53cbc1c414"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:14:41.771166 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:41.771147 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b943d4d-c829-410c-9233-0b53cbc1c414-istio-envoy" pod "6b943d4d-c829-410c-9233-0b53cbc1c414" (UID: "6b943d4d-c829-410c-9233-0b53cbc1c414"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:14:41.771223 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:41.771164 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b943d4d-c829-410c-9233-0b53cbc1c414-kube-api-access-f9qf5" pod "6b943d4d-c829-410c-9233-0b53cbc1c414" (UID: "6b943d4d-c829-410c-9233-0b53cbc1c414"). InnerVolumeSpecName "kube-api-access-f9qf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:14:41.771223 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:41.771169 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b943d4d-c829-410c-9233-0b53cbc1c414-istio-podinfo" pod "6b943d4d-c829-410c-9233-0b53cbc1c414" (UID: "6b943d4d-c829-410c-9233-0b53cbc1c414"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:14:41.808460 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:41.808422 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f8a8088d-7bb6787865-bmt6k"] Oct 31 04:14:41.838513 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:41.838474 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f8a8088d-858594b57f-d7j74"] Oct 31 04:14:41.869934 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:41.869895 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9qf5\" (UniqueName: \"kubernetes.io/projected/6b943d4d-c829-410c-9233-0b53cbc1c414-kube-api-access-f9qf5\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:14:41.869934 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:41.869928 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b943d4d-c829-410c-9233-0b53cbc1c414-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:14:41.869934 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:41.869939 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b943d4d-c829-410c-9233-0b53cbc1c414-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:14:41.870179 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:41.869948 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b943d4d-c829-410c-9233-0b53cbc1c414-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:14:42.427039 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:42.427000 5019 generic.go:356] "Generic (PLEG): container finished" podID="6b943d4d-c829-410c-9233-0b53cbc1c414" containerID="adac2c933bec465a85952b2d64beeac94099d19d13c9bc5b080bad43f9376efa" exitCode=0 Oct 31 04:14:42.427605 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:42.427056 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n" Oct 31 04:14:42.427605 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:42.427107 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n" event={"ID":"6b943d4d-c829-410c-9233-0b53cbc1c414","Type":"ContainerDied","Data":"adac2c933bec465a85952b2d64beeac94099d19d13c9bc5b080bad43f9376efa"} Oct 31 04:14:42.427605 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:42.427179 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n" event={"ID":"6b943d4d-c829-410c-9233-0b53cbc1c414","Type":"ContainerDied","Data":"606b24b8f7b3663d2cef4e9865185002f08a8393e293906bc98b159cd51b7177"} Oct 31 04:14:42.427605 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:42.427211 5019 scope.go:122] "RemoveContainer" containerID="adac2c933bec465a85952b2d64beeac94099d19d13c9bc5b080bad43f9376efa" Oct 31 04:14:42.440123 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:42.439969 5019 scope.go:122] "RemoveContainer" containerID="e648ace0ce8cd6ec13266002b01c48ae54052a86bbe33a3c08ce77fe41873aa0" Oct 31 04:14:42.449864 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:42.449841 5019 scope.go:122] "RemoveContainer" containerID="adac2c933bec465a85952b2d64beeac94099d19d13c9bc5b080bad43f9376efa" Oct 31 04:14:42.450271 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:14:42.450250 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adac2c933bec465a85952b2d64beeac94099d19d13c9bc5b080bad43f9376efa\": container with ID starting with adac2c933bec465a85952b2d64beeac94099d19d13c9bc5b080bad43f9376efa not found: ID does not exist" containerID="adac2c933bec465a85952b2d64beeac94099d19d13c9bc5b080bad43f9376efa" Oct 31 04:14:42.450318 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:42.450283 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adac2c933bec465a85952b2d64beeac94099d19d13c9bc5b080bad43f9376efa"} err="failed to get container status \"adac2c933bec465a85952b2d64beeac94099d19d13c9bc5b080bad43f9376efa\": rpc error: code = NotFound desc = could not find container \"adac2c933bec465a85952b2d64beeac94099d19d13c9bc5b080bad43f9376efa\": container with ID starting with adac2c933bec465a85952b2d64beeac94099d19d13c9bc5b080bad43f9376efa not found: ID does not exist" Oct 31 04:14:42.450318 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:42.450304 5019 scope.go:122] "RemoveContainer" containerID="e648ace0ce8cd6ec13266002b01c48ae54052a86bbe33a3c08ce77fe41873aa0" Oct 31 04:14:42.450581 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:14:42.450561 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e648ace0ce8cd6ec13266002b01c48ae54052a86bbe33a3c08ce77fe41873aa0\": container with ID starting with e648ace0ce8cd6ec13266002b01c48ae54052a86bbe33a3c08ce77fe41873aa0 not found: ID does not exist" containerID="e648ace0ce8cd6ec13266002b01c48ae54052a86bbe33a3c08ce77fe41873aa0" Oct 31 04:14:42.450649 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:42.450592 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e648ace0ce8cd6ec13266002b01c48ae54052a86bbe33a3c08ce77fe41873aa0"} err="failed to get container status \"e648ace0ce8cd6ec13266002b01c48ae54052a86bbe33a3c08ce77fe41873aa0\": rpc error: code = NotFound desc = could not find container \"e648ace0ce8cd6ec13266002b01c48ae54052a86bbe33a3c08ce77fe41873aa0\": container with ID starting with e648ace0ce8cd6ec13266002b01c48ae54052a86bbe33a3c08ce77fe41873aa0 not found: ID does not exist" Oct 31 04:14:42.481756 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:42.481720 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n"] Oct 31 04:14:42.493451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:42.493412 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-a0d1a61d-gw-openshift-default-7789c75ddf-4pt7n"] Oct 31 04:14:43.420357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:43.419946 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b943d4d-c829-410c-9233-0b53cbc1c414" path="/var/lib/kubelet/pods/6b943d4d-c829-410c-9233-0b53cbc1c414/volumes" Oct 31 04:14:44.189898 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:44.189853 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-4c2387-78c59569f5-pklf6"] Oct 31 04:14:44.190677 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:44.190646 5019 cpu_manager.go:414] "containerMap: removing container" podUID="6b943d4d-c829-410c-9233-0b53cbc1c414" containerName="istio-proxy" Oct 31 04:14:44.190677 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:44.190673 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b943d4d-c829-410c-9233-0b53cbc1c414" containerName="istio-proxy" Oct 31 04:14:44.190890 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:44.190818 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b943d4d-c829-410c-9233-0b53cbc1c414" containerName="istio-proxy" Oct 31 04:14:44.190890 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:44.190831 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b943d4d-c829-410c-9233-0b53cbc1c414" containerName="istio-proxy" Oct 31 04:14:44.195232 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:44.195206 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-4c2387-78c59569f5-pklf6" Oct 31 04:14:44.213935 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:44.213900 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-4c2387-78c59569f5-pklf6"] Oct 31 04:14:44.221254 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:44.221219 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-4c2387\"" Oct 31 04:14:44.221254 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:44.221244 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-4c2387-dockercfg-bbs8q\"" Oct 31 04:14:44.258320 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:44.258278 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-4c2387-78fdcfbd75-2twf4"] Oct 31 04:14:44.258849 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:44.258834 5019 cpu_manager.go:414] "containerMap: removing container" podUID="6b943d4d-c829-410c-9233-0b53cbc1c414" containerName="istio-proxy" Oct 31 04:14:44.258887 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:44.258851 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b943d4d-c829-410c-9233-0b53cbc1c414" containerName="istio-proxy" Oct 31 04:14:44.262372 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:44.262345 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4c2387-78fdcfbd75-2twf4" Oct 31 04:14:44.284288 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:44.284248 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-4c2387-78fdcfbd75-2twf4"] Oct 31 04:14:44.350149 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:44.350101 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af4b877c-fb8f-48be-917c-a3e99fc1eb7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-4c2387-78fdcfbd75-2twf4\" (UID: \"af4b877c-fb8f-48be-917c-a3e99fc1eb7f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4c2387-78fdcfbd75-2twf4" Oct 31 04:14:44.354593 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:44.354559 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rq5r\" (UniqueName: \"kubernetes.io/projected/c60c1b8f-b960-4a87-9382-1dfc846ca8b5-kube-api-access-9rq5r\") pod \"payload-processing-e2e-worker-w3-4c2387-78c59569f5-pklf6\" (UID: \"c60c1b8f-b960-4a87-9382-1dfc846ca8b5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-4c2387-78c59569f5-pklf6" Oct 31 04:14:44.368355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:44.354986 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2cw6\" (UniqueName: \"kubernetes.io/projected/af4b877c-fb8f-48be-917c-a3e99fc1eb7f-kube-api-access-v2cw6\") pod \"payload-pre-processing-e2e-worker-w3-4c2387-78fdcfbd75-2twf4\" (UID: \"af4b877c-fb8f-48be-917c-a3e99fc1eb7f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4c2387-78fdcfbd75-2twf4" Oct 31 04:14:44.369552 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:44.369512 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c60c1b8f-b960-4a87-9382-1dfc846ca8b5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-4c2387-78c59569f5-pklf6\" (UID: \"c60c1b8f-b960-4a87-9382-1dfc846ca8b5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-4c2387-78c59569f5-pklf6" Oct 31 04:14:44.471267 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:44.471222 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af4b877c-fb8f-48be-917c-a3e99fc1eb7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-4c2387-78fdcfbd75-2twf4\" (UID: \"af4b877c-fb8f-48be-917c-a3e99fc1eb7f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4c2387-78fdcfbd75-2twf4" Oct 31 04:14:44.471267 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:44.471272 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rq5r\" (UniqueName: \"kubernetes.io/projected/c60c1b8f-b960-4a87-9382-1dfc846ca8b5-kube-api-access-9rq5r\") pod \"payload-processing-e2e-worker-w3-4c2387-78c59569f5-pklf6\" (UID: \"c60c1b8f-b960-4a87-9382-1dfc846ca8b5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-4c2387-78c59569f5-pklf6" Oct 31 04:14:44.471559 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:44.471355 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2cw6\" (UniqueName: \"kubernetes.io/projected/af4b877c-fb8f-48be-917c-a3e99fc1eb7f-kube-api-access-v2cw6\") pod \"payload-pre-processing-e2e-worker-w3-4c2387-78fdcfbd75-2twf4\" (UID: \"af4b877c-fb8f-48be-917c-a3e99fc1eb7f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4c2387-78fdcfbd75-2twf4" Oct 31 04:14:44.471559 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:44.471397 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c60c1b8f-b960-4a87-9382-1dfc846ca8b5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-4c2387-78c59569f5-pklf6\" (UID: \"c60c1b8f-b960-4a87-9382-1dfc846ca8b5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-4c2387-78c59569f5-pklf6" Oct 31 04:14:44.472249 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:44.472218 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af4b877c-fb8f-48be-917c-a3e99fc1eb7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-4c2387-78fdcfbd75-2twf4\" (UID: \"af4b877c-fb8f-48be-917c-a3e99fc1eb7f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4c2387-78fdcfbd75-2twf4" Oct 31 04:14:44.472643 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:44.472622 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c60c1b8f-b960-4a87-9382-1dfc846ca8b5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-4c2387-78c59569f5-pklf6\" (UID: \"c60c1b8f-b960-4a87-9382-1dfc846ca8b5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-4c2387-78c59569f5-pklf6" Oct 31 04:14:44.495086 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:44.495046 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2cw6\" (UniqueName: \"kubernetes.io/projected/af4b877c-fb8f-48be-917c-a3e99fc1eb7f-kube-api-access-v2cw6\") pod \"payload-pre-processing-e2e-worker-w3-4c2387-78fdcfbd75-2twf4\" (UID: \"af4b877c-fb8f-48be-917c-a3e99fc1eb7f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4c2387-78fdcfbd75-2twf4" Oct 31 04:14:44.495264 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:44.495112 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rq5r\" (UniqueName: \"kubernetes.io/projected/c60c1b8f-b960-4a87-9382-1dfc846ca8b5-kube-api-access-9rq5r\") pod \"payload-processing-e2e-worker-w3-4c2387-78c59569f5-pklf6\" (UID: \"c60c1b8f-b960-4a87-9382-1dfc846ca8b5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-4c2387-78c59569f5-pklf6" Oct 31 04:14:44.514022 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:44.513987 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-4c2387-78c59569f5-pklf6" Oct 31 04:14:44.577721 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:44.577678 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4c2387-78fdcfbd75-2twf4" Oct 31 04:14:44.815135 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:44.815102 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-4c2387-78c59569f5-pklf6"] Oct 31 04:14:44.816190 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:14:44.816159 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc60c1b8f_b960_4a87_9382_1dfc846ca8b5.slice/crio-495f19a16703802d4d4f6c77c79732f727c2e250aa9222976cbaa012071c4df2 WatchSource:0}: Error finding container 495f19a16703802d4d4f6c77c79732f727c2e250aa9222976cbaa012071c4df2: Status 404 returned error can't find the container with id 495f19a16703802d4d4f6c77c79732f727c2e250aa9222976cbaa012071c4df2 Oct 31 04:14:44.905124 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:44.905086 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-4c2387-78fdcfbd75-2twf4"] Oct 31 04:14:44.917396 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:14:44.907601 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf4b877c_fb8f_48be_917c_a3e99fc1eb7f.slice/crio-ccf10d75895a16c011c24acd8be2bad35912c024f457e3b8ebbb028620f186cb WatchSource:0}: Error finding container ccf10d75895a16c011c24acd8be2bad35912c024f457e3b8ebbb028620f186cb: Status 404 returned error can't find the container with id ccf10d75895a16c011c24acd8be2bad35912c024f457e3b8ebbb028620f186cb Oct 31 04:14:45.466653 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:45.466608 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-4c2387-78c59569f5-pklf6" event={"ID":"c60c1b8f-b960-4a87-9382-1dfc846ca8b5","Type":"ContainerStarted","Data":"92ed61316fe2122ecfbe33607c347f05b32a297ce2df922340397c66b31e8700"} Oct 31 04:14:45.467148 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:45.466872 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-4c2387-78c59569f5-pklf6" Oct 31 04:14:45.467148 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:45.466935 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-4c2387-78c59569f5-pklf6" event={"ID":"c60c1b8f-b960-4a87-9382-1dfc846ca8b5","Type":"ContainerStarted","Data":"495f19a16703802d4d4f6c77c79732f727c2e250aa9222976cbaa012071c4df2"} Oct 31 04:14:45.468318 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:45.468293 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4c2387-78fdcfbd75-2twf4" event={"ID":"af4b877c-fb8f-48be-917c-a3e99fc1eb7f","Type":"ContainerStarted","Data":"db954a6e84eaaf447a3bdcbe78f773fa99e31994c552c3b0d7e257bc561d8c0e"} Oct 31 04:14:45.468412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:45.468357 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4c2387-78fdcfbd75-2twf4" event={"ID":"af4b877c-fb8f-48be-917c-a3e99fc1eb7f","Type":"ContainerStarted","Data":"ccf10d75895a16c011c24acd8be2bad35912c024f457e3b8ebbb028620f186cb"} Oct 31 04:14:45.468412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:45.468407 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4c2387-78fdcfbd75-2twf4" Oct 31 04:14:45.508625 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:45.508575 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-4c2387-78c59569f5-pklf6" podStartSLOduration=1.5085599269999999 podStartE2EDuration="1.508559927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:14:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:14:45.507242497 +0000 UTC m=+640.263580538" watchObservedRunningTime="2026-10-31 04:14:45.508559927 +0000 UTC m=+640.264898004" Oct 31 04:14:45.551142 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:45.551085 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4c2387-78fdcfbd75-2twf4" podStartSLOduration=1.551068934 podStartE2EDuration="1.551068934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:14:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:14:45.549715317 +0000 UTC m=+640.306053358" watchObservedRunningTime="2026-10-31 04:14:45.551068934 +0000 UTC m=+640.307406985" Oct 31 04:14:46.809442 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:46.809394 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-f8a8088d-7bb6787865-bmt6k" podUID="dc918812-af5e-4f8e-9f3d-35f245dff2fd" containerName="payload-processing" containerID="cri-o://e226c0cb07356b1f620730b389be88d40a29b074281c821a5b090b1981fceca2" gracePeriod=25 Oct 31 04:14:46.839553 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:46.839504 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-f8a8088d-858594b57f-d7j74" podUID="0a5cbfbc-7654-4f7f-a036-c8ffd3c26819" containerName="payload-pre-processing" containerID="cri-o://dc6c8c7598bc8a7de01825c8d63e0a17ae5d2d9ff8e41da058590440f9004bae" gracePeriod=25 Oct 31 04:14:47.176803 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:47.176763 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-f8a8088d-7bb6787865-bmt6k" Oct 31 04:14:47.205247 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:47.204730 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc918812-af5e-4f8e-9f3d-35f245dff2fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc918812-af5e-4f8e-9f3d-35f245dff2fd-plugins-config-volume\") pod \"dc918812-af5e-4f8e-9f3d-35f245dff2fd\" (UID: \"dc918812-af5e-4f8e-9f3d-35f245dff2fd\") " Oct 31 04:14:47.205247 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:47.204831 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc918812-af5e-4f8e-9f3d-35f245dff2fd-kube-api-access-fghqr\" (UniqueName: \"kubernetes.io/projected/dc918812-af5e-4f8e-9f3d-35f245dff2fd-kube-api-access-fghqr\") pod \"dc918812-af5e-4f8e-9f3d-35f245dff2fd\" (UID: \"dc918812-af5e-4f8e-9f3d-35f245dff2fd\") " Oct 31 04:14:47.205657 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:47.205606 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc918812-af5e-4f8e-9f3d-35f245dff2fd-plugins-config-volume" pod "dc918812-af5e-4f8e-9f3d-35f245dff2fd" (UID: "dc918812-af5e-4f8e-9f3d-35f245dff2fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:14:47.207560 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:47.207534 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc918812-af5e-4f8e-9f3d-35f245dff2fd-kube-api-access-fghqr" pod "dc918812-af5e-4f8e-9f3d-35f245dff2fd" (UID: "dc918812-af5e-4f8e-9f3d-35f245dff2fd"). InnerVolumeSpecName "kube-api-access-fghqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:14:47.228451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:47.227676 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-f8a8088d-858594b57f-d7j74" Oct 31 04:14:47.249847 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:47.244521 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f8a8088d-ft9l5"] Oct 31 04:14:47.249847 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:47.246123 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f8a8088d-ft9l5"] Oct 31 04:14:47.306193 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:47.306158 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a5cbfbc-7654-4f7f-a036-c8ffd3c26819-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a5cbfbc-7654-4f7f-a036-c8ffd3c26819-plugins-config-volume\") pod \"0a5cbfbc-7654-4f7f-a036-c8ffd3c26819\" (UID: \"0a5cbfbc-7654-4f7f-a036-c8ffd3c26819\") " Oct 31 04:14:47.306430 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:47.306384 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a5cbfbc-7654-4f7f-a036-c8ffd3c26819-kube-api-access-trx5n\" (UniqueName: \"kubernetes.io/projected/0a5cbfbc-7654-4f7f-a036-c8ffd3c26819-kube-api-access-trx5n\") pod \"0a5cbfbc-7654-4f7f-a036-c8ffd3c26819\" (UID: \"0a5cbfbc-7654-4f7f-a036-c8ffd3c26819\") " Oct 31 04:14:47.306694 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:47.306677 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc918812-af5e-4f8e-9f3d-35f245dff2fd-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:14:47.306771 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:47.306698 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fghqr\" (UniqueName: \"kubernetes.io/projected/dc918812-af5e-4f8e-9f3d-35f245dff2fd-kube-api-access-fghqr\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:14:47.306937 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:47.306910 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a5cbfbc-7654-4f7f-a036-c8ffd3c26819-plugins-config-volume" pod "0a5cbfbc-7654-4f7f-a036-c8ffd3c26819" (UID: "0a5cbfbc-7654-4f7f-a036-c8ffd3c26819"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:14:47.311128 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:47.311097 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a5cbfbc-7654-4f7f-a036-c8ffd3c26819-kube-api-access-trx5n" pod "0a5cbfbc-7654-4f7f-a036-c8ffd3c26819" (UID: "0a5cbfbc-7654-4f7f-a036-c8ffd3c26819"). InnerVolumeSpecName "kube-api-access-trx5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:14:47.379881 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:47.379773 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr"] Oct 31 04:14:47.380091 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:47.380060 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr" podUID="3dd4051c-8d79-496e-b3cb-d90263e6f1be" containerName="istio-proxy" containerID="cri-o://4740e12b96e508a28489b8c3f623a13d68cc2b35e989d52cb19f9fd569ca6157" gracePeriod=30 Oct 31 04:14:47.407554 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:47.407507 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2553577b-451f-4d98-803d-0f81dd263822" path="/var/lib/kubelet/pods/2553577b-451f-4d98-803d-0f81dd263822/volumes" Oct 31 04:14:47.407746 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:47.407617 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a5cbfbc-7654-4f7f-a036-c8ffd3c26819-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:14:47.407746 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:47.407640 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trx5n\" (UniqueName: \"kubernetes.io/projected/0a5cbfbc-7654-4f7f-a036-c8ffd3c26819-kube-api-access-trx5n\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:14:47.524697 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:47.524659 5019 generic.go:356] "Generic (PLEG): container finished" podID="0a5cbfbc-7654-4f7f-a036-c8ffd3c26819" containerID="dc6c8c7598bc8a7de01825c8d63e0a17ae5d2d9ff8e41da058590440f9004bae" exitCode=0 Oct 31 04:14:47.525254 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:47.525220 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-f8a8088d-858594b57f-d7j74" event={"ID":"0a5cbfbc-7654-4f7f-a036-c8ffd3c26819","Type":"ContainerDied","Data":"dc6c8c7598bc8a7de01825c8d63e0a17ae5d2d9ff8e41da058590440f9004bae"} Oct 31 04:14:47.525465 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:47.525445 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-f8a8088d-858594b57f-d7j74" event={"ID":"0a5cbfbc-7654-4f7f-a036-c8ffd3c26819","Type":"ContainerDied","Data":"028568a2d9a9f172a1a2bd912e49dfd7d32d8cbd230e250df2f7c8f191ff4e90"} Oct 31 04:14:47.525682 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:47.525663 5019 scope.go:122] "RemoveContainer" containerID="dc6c8c7598bc8a7de01825c8d63e0a17ae5d2d9ff8e41da058590440f9004bae" Oct 31 04:14:47.525989 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:47.525973 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-f8a8088d-858594b57f-d7j74" Oct 31 04:14:47.531174 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:47.531140 5019 generic.go:356] "Generic (PLEG): container finished" podID="dc918812-af5e-4f8e-9f3d-35f245dff2fd" containerID="e226c0cb07356b1f620730b389be88d40a29b074281c821a5b090b1981fceca2" exitCode=0 Oct 31 04:14:47.531427 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:47.531238 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-f8a8088d-7bb6787865-bmt6k" event={"ID":"dc918812-af5e-4f8e-9f3d-35f245dff2fd","Type":"ContainerDied","Data":"e226c0cb07356b1f620730b389be88d40a29b074281c821a5b090b1981fceca2"} Oct 31 04:14:47.531427 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:47.531284 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-f8a8088d-7bb6787865-bmt6k" event={"ID":"dc918812-af5e-4f8e-9f3d-35f245dff2fd","Type":"ContainerDied","Data":"b732e6b14b0747323ea7a4a9b423f9d37f69da22c15013a7d368fdea153ed242"} Oct 31 04:14:47.531427 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:47.531284 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-f8a8088d-7bb6787865-bmt6k" Oct 31 04:14:47.544035 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:47.543900 5019 scope.go:122] "RemoveContainer" containerID="dc6c8c7598bc8a7de01825c8d63e0a17ae5d2d9ff8e41da058590440f9004bae" Oct 31 04:14:47.545189 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:14:47.544962 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc6c8c7598bc8a7de01825c8d63e0a17ae5d2d9ff8e41da058590440f9004bae\": container with ID starting with dc6c8c7598bc8a7de01825c8d63e0a17ae5d2d9ff8e41da058590440f9004bae not found: ID does not exist" containerID="dc6c8c7598bc8a7de01825c8d63e0a17ae5d2d9ff8e41da058590440f9004bae" Oct 31 04:14:47.545189 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:47.545018 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc6c8c7598bc8a7de01825c8d63e0a17ae5d2d9ff8e41da058590440f9004bae"} err="failed to get container status \"dc6c8c7598bc8a7de01825c8d63e0a17ae5d2d9ff8e41da058590440f9004bae\": rpc error: code = NotFound desc = could not find container \"dc6c8c7598bc8a7de01825c8d63e0a17ae5d2d9ff8e41da058590440f9004bae\": container with ID starting with dc6c8c7598bc8a7de01825c8d63e0a17ae5d2d9ff8e41da058590440f9004bae not found: ID does not exist" Oct 31 04:14:47.545189 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:47.545044 5019 scope.go:122] "RemoveContainer" containerID="e226c0cb07356b1f620730b389be88d40a29b074281c821a5b090b1981fceca2" Oct 31 04:14:47.555668 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:47.555637 5019 scope.go:122] "RemoveContainer" containerID="e226c0cb07356b1f620730b389be88d40a29b074281c821a5b090b1981fceca2" Oct 31 04:14:47.556069 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:14:47.556038 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e226c0cb07356b1f620730b389be88d40a29b074281c821a5b090b1981fceca2\": container with ID starting with e226c0cb07356b1f620730b389be88d40a29b074281c821a5b090b1981fceca2 not found: ID does not exist" containerID="e226c0cb07356b1f620730b389be88d40a29b074281c821a5b090b1981fceca2" Oct 31 04:14:47.556174 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:47.556072 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e226c0cb07356b1f620730b389be88d40a29b074281c821a5b090b1981fceca2"} err="failed to get container status \"e226c0cb07356b1f620730b389be88d40a29b074281c821a5b090b1981fceca2\": rpc error: code = NotFound desc = could not find container \"e226c0cb07356b1f620730b389be88d40a29b074281c821a5b090b1981fceca2\": container with ID starting with e226c0cb07356b1f620730b389be88d40a29b074281c821a5b090b1981fceca2 not found: ID does not exist" Oct 31 04:14:47.579449 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:47.579399 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f8a8088d-858594b57f-d7j74"] Oct 31 04:14:47.582020 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:47.581992 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f8a8088d-858594b57f-d7j74"] Oct 31 04:14:47.614134 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:47.614091 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f8a8088d-7bb6787865-bmt6k"] Oct 31 04:14:47.617911 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:47.617874 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f8a8088d-7bb6787865-bmt6k"] Oct 31 04:14:49.408799 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:49.408761 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a5cbfbc-7654-4f7f-a036-c8ffd3c26819" path="/var/lib/kubelet/pods/0a5cbfbc-7654-4f7f-a036-c8ffd3c26819/volumes" Oct 31 04:14:49.409373 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:49.409350 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc918812-af5e-4f8e-9f3d-35f245dff2fd" path="/var/lib/kubelet/pods/dc918812-af5e-4f8e-9f3d-35f245dff2fd/volumes" Oct 31 04:14:49.885992 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:49.885950 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44f032ad-fzt68"] Oct 31 04:14:49.897875 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:49.897832 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44f032ad-fzt68"] Oct 31 04:14:51.407764 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:51.407730 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6382469-cb51-401e-bccf-cbf15f15075c" path="/var/lib/kubelet/pods/f6382469-cb51-401e-bccf-cbf15f15075c/volumes" Oct 31 04:14:51.451910 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:51.451874 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc"] Oct 31 04:14:51.452415 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:51.452375 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc" podUID="c65c601e-7e95-4e7d-844c-a4cadd680d1b" containerName="istio-proxy" containerID="cri-o://5b0032a70f02f03c928a2893d87d6d17bc50a01807463715fbb9b4b67058f484" gracePeriod=30 Oct 31 04:14:51.461068 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:51.461037 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb"] Oct 31 04:14:51.461279 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:51.461243 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb" podUID="ec1714c9-5e99-400a-b392-486c04926361" containerName="istio-proxy" containerID="cri-o://00d09e779f30b22182e3ea6673996014cdf91ca88e1de182b6d8b965fbc777c2" gracePeriod=30 Oct 31 04:14:52.420468 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:14:52.420431 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:14:52.629472 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:52.629430 5019 generic.go:356] "Generic (PLEG): container finished" podID="3dd4051c-8d79-496e-b3cb-d90263e6f1be" containerID="4740e12b96e508a28489b8c3f623a13d68cc2b35e989d52cb19f9fd569ca6157" exitCode=0 Oct 31 04:14:52.629646 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:52.629625 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr" event={"ID":"3dd4051c-8d79-496e-b3cb-d90263e6f1be","Type":"ContainerDied","Data":"4740e12b96e508a28489b8c3f623a13d68cc2b35e989d52cb19f9fd569ca6157"} Oct 31 04:14:52.629704 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:52.629689 5019 scope.go:122] "RemoveContainer" containerID="afce53d39f9ff4fcef00d8bc832bcecb6cf0d5396985dd3e4a85a7327ea06788" Oct 31 04:14:52.709962 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:52.709926 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr" Oct 31 04:14:52.825357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:52.825241 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3dd4051c-8d79-496e-b3cb-d90263e6f1be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dd4051c-8d79-496e-b3cb-d90263e6f1be-istio-podinfo\") pod \"3dd4051c-8d79-496e-b3cb-d90263e6f1be\" (UID: \"3dd4051c-8d79-496e-b3cb-d90263e6f1be\") " Oct 31 04:14:52.825357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:52.825322 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dd4051c-8d79-496e-b3cb-d90263e6f1be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dd4051c-8d79-496e-b3cb-d90263e6f1be-workload-socket\") pod \"3dd4051c-8d79-496e-b3cb-d90263e6f1be\" (UID: \"3dd4051c-8d79-496e-b3cb-d90263e6f1be\") " Oct 31 04:14:52.825627 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:52.825416 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dd4051c-8d79-496e-b3cb-d90263e6f1be-kube-api-access-x4mhj\" (UniqueName: \"kubernetes.io/projected/3dd4051c-8d79-496e-b3cb-d90263e6f1be-kube-api-access-x4mhj\") pod \"3dd4051c-8d79-496e-b3cb-d90263e6f1be\" (UID: \"3dd4051c-8d79-496e-b3cb-d90263e6f1be\") " Oct 31 04:14:52.825627 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:52.825452 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dd4051c-8d79-496e-b3cb-d90263e6f1be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dd4051c-8d79-496e-b3cb-d90263e6f1be-workload-certs\") pod \"3dd4051c-8d79-496e-b3cb-d90263e6f1be\" (UID: \"3dd4051c-8d79-496e-b3cb-d90263e6f1be\") " Oct 31 04:14:52.825627 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:52.825483 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dd4051c-8d79-496e-b3cb-d90263e6f1be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dd4051c-8d79-496e-b3cb-d90263e6f1be-istio-data\") pod \"3dd4051c-8d79-496e-b3cb-d90263e6f1be\" (UID: \"3dd4051c-8d79-496e-b3cb-d90263e6f1be\") " Oct 31 04:14:52.825627 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:52.825534 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dd4051c-8d79-496e-b3cb-d90263e6f1be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dd4051c-8d79-496e-b3cb-d90263e6f1be-credential-socket\") pod \"3dd4051c-8d79-496e-b3cb-d90263e6f1be\" (UID: \"3dd4051c-8d79-496e-b3cb-d90263e6f1be\") " Oct 31 04:14:52.825627 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:52.825571 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dd4051c-8d79-496e-b3cb-d90263e6f1be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dd4051c-8d79-496e-b3cb-d90263e6f1be-istio-envoy\") pod \"3dd4051c-8d79-496e-b3cb-d90263e6f1be\" (UID: \"3dd4051c-8d79-496e-b3cb-d90263e6f1be\") " Oct 31 04:14:52.825627 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:52.825622 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dd4051c-8d79-496e-b3cb-d90263e6f1be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dd4051c-8d79-496e-b3cb-d90263e6f1be-istiod-ca-cert\") pod \"3dd4051c-8d79-496e-b3cb-d90263e6f1be\" (UID: \"3dd4051c-8d79-496e-b3cb-d90263e6f1be\") " Oct 31 04:14:52.825974 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:52.825655 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dd4051c-8d79-496e-b3cb-d90263e6f1be-istio-token\" (UniqueName: \"kubernetes.io/projected/3dd4051c-8d79-496e-b3cb-d90263e6f1be-istio-token\") pod \"3dd4051c-8d79-496e-b3cb-d90263e6f1be\" (UID: \"3dd4051c-8d79-496e-b3cb-d90263e6f1be\") " Oct 31 04:14:52.831441 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:52.827783 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dd4051c-8d79-496e-b3cb-d90263e6f1be-credential-socket" pod "3dd4051c-8d79-496e-b3cb-d90263e6f1be" (UID: "3dd4051c-8d79-496e-b3cb-d90263e6f1be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:14:52.831441 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:52.828485 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dd4051c-8d79-496e-b3cb-d90263e6f1be-istio-data" pod "3dd4051c-8d79-496e-b3cb-d90263e6f1be" (UID: "3dd4051c-8d79-496e-b3cb-d90263e6f1be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:14:52.831441 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:52.828915 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dd4051c-8d79-496e-b3cb-d90263e6f1be-istiod-ca-cert" pod "3dd4051c-8d79-496e-b3cb-d90263e6f1be" (UID: "3dd4051c-8d79-496e-b3cb-d90263e6f1be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:14:52.831441 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:52.829289 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dd4051c-8d79-496e-b3cb-d90263e6f1be-workload-socket" pod "3dd4051c-8d79-496e-b3cb-d90263e6f1be" (UID: "3dd4051c-8d79-496e-b3cb-d90263e6f1be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:14:52.832444 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:52.832405 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dd4051c-8d79-496e-b3cb-d90263e6f1be-workload-certs" pod "3dd4051c-8d79-496e-b3cb-d90263e6f1be" (UID: "3dd4051c-8d79-496e-b3cb-d90263e6f1be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:14:52.853978 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:52.848912 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3dd4051c-8d79-496e-b3cb-d90263e6f1be-istio-podinfo" pod "3dd4051c-8d79-496e-b3cb-d90263e6f1be" (UID: "3dd4051c-8d79-496e-b3cb-d90263e6f1be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:14:52.857544 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:52.857242 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dd4051c-8d79-496e-b3cb-d90263e6f1be-istio-envoy" pod "3dd4051c-8d79-496e-b3cb-d90263e6f1be" (UID: "3dd4051c-8d79-496e-b3cb-d90263e6f1be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:14:52.857544 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:52.857429 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dd4051c-8d79-496e-b3cb-d90263e6f1be-kube-api-access-x4mhj" pod "3dd4051c-8d79-496e-b3cb-d90263e6f1be" (UID: "3dd4051c-8d79-496e-b3cb-d90263e6f1be"). InnerVolumeSpecName "kube-api-access-x4mhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:14:52.898359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:52.887529 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dd4051c-8d79-496e-b3cb-d90263e6f1be-istio-token" pod "3dd4051c-8d79-496e-b3cb-d90263e6f1be" (UID: "3dd4051c-8d79-496e-b3cb-d90263e6f1be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:14:52.927519 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:52.927424 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4mhj\" (UniqueName: \"kubernetes.io/projected/3dd4051c-8d79-496e-b3cb-d90263e6f1be-kube-api-access-x4mhj\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:14:52.927519 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:52.927466 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dd4051c-8d79-496e-b3cb-d90263e6f1be-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:14:52.927519 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:52.927476 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dd4051c-8d79-496e-b3cb-d90263e6f1be-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:14:52.927519 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:52.927487 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dd4051c-8d79-496e-b3cb-d90263e6f1be-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:14:52.927519 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:52.927495 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dd4051c-8d79-496e-b3cb-d90263e6f1be-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:14:52.927519 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:52.927505 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dd4051c-8d79-496e-b3cb-d90263e6f1be-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:14:52.927519 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:52.927513 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dd4051c-8d79-496e-b3cb-d90263e6f1be-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:14:52.927519 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:52.927521 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dd4051c-8d79-496e-b3cb-d90263e6f1be-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:14:52.927519 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:52.927529 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dd4051c-8d79-496e-b3cb-d90263e6f1be-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:14:53.637825 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:53.637793 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr" Oct 31 04:14:53.638267 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:53.637818 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr" event={"ID":"3dd4051c-8d79-496e-b3cb-d90263e6f1be","Type":"ContainerDied","Data":"d2ebaa84183620a46393bd71ce08162df1c46fead6701680f0d28f3c5cfadd75"} Oct 31 04:14:53.638267 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:53.637887 5019 scope.go:122] "RemoveContainer" containerID="4740e12b96e508a28489b8c3f623a13d68cc2b35e989d52cb19f9fd569ca6157" Oct 31 04:14:53.717759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:53.717687 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr"] Oct 31 04:14:53.722688 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:53.722638 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-f8a8088d-openshift-default-5b7849fcb5-bs6xr"] Oct 31 04:14:55.449626 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:55.449540 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dd4051c-8d79-496e-b3cb-d90263e6f1be" path="/var/lib/kubelet/pods/3dd4051c-8d79-496e-b3cb-d90263e6f1be/volumes" Oct 31 04:14:56.504779 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:56.504627 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-4c2387-78c59569f5-pklf6" Oct 31 04:14:56.505629 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:56.505424 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4c2387-78fdcfbd75-2twf4" Oct 31 04:14:56.669412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:56.667098 5019 generic.go:356] "Generic (PLEG): container finished" podID="c65c601e-7e95-4e7d-844c-a4cadd680d1b" containerID="5b0032a70f02f03c928a2893d87d6d17bc50a01807463715fbb9b4b67058f484" exitCode=0 Oct 31 04:14:56.669412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:56.667259 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc" event={"ID":"c65c601e-7e95-4e7d-844c-a4cadd680d1b","Type":"ContainerDied","Data":"5b0032a70f02f03c928a2893d87d6d17bc50a01807463715fbb9b4b67058f484"} Oct 31 04:14:56.669412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:56.669395 5019 generic.go:356] "Generic (PLEG): container finished" podID="ec1714c9-5e99-400a-b392-486c04926361" containerID="00d09e779f30b22182e3ea6673996014cdf91ca88e1de182b6d8b965fbc777c2" exitCode=0 Oct 31 04:14:56.669712 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:56.669524 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb" event={"ID":"ec1714c9-5e99-400a-b392-486c04926361","Type":"ContainerDied","Data":"00d09e779f30b22182e3ea6673996014cdf91ca88e1de182b6d8b965fbc777c2"} Oct 31 04:14:56.669712 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:56.669579 5019 scope.go:122] "RemoveContainer" containerID="9753078996d2416ba0e209fc2cc4eba325b375a272c80f14733af5bdbbf2c45f" Oct 31 04:14:56.726655 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:56.726612 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-439989-6b4f868455-v228h"] Oct 31 04:14:56.727378 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:56.727352 5019 cpu_manager.go:414] "containerMap: removing container" podUID="3dd4051c-8d79-496e-b3cb-d90263e6f1be" containerName="istio-proxy" Oct 31 04:14:56.727480 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:56.727379 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dd4051c-8d79-496e-b3cb-d90263e6f1be" containerName="istio-proxy" Oct 31 04:14:56.727480 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:56.727417 5019 cpu_manager.go:414] "containerMap: removing container" podUID="dc918812-af5e-4f8e-9f3d-35f245dff2fd" containerName="payload-processing" Oct 31 04:14:56.727480 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:56.727426 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc918812-af5e-4f8e-9f3d-35f245dff2fd" containerName="payload-processing" Oct 31 04:14:56.727480 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:56.727449 5019 cpu_manager.go:414] "containerMap: removing container" podUID="0a5cbfbc-7654-4f7f-a036-c8ffd3c26819" containerName="payload-pre-processing" Oct 31 04:14:56.727480 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:56.727457 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a5cbfbc-7654-4f7f-a036-c8ffd3c26819" containerName="payload-pre-processing" Oct 31 04:14:56.727956 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:56.727935 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc918812-af5e-4f8e-9f3d-35f245dff2fd" containerName="payload-processing" Oct 31 04:14:56.727998 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:56.727964 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a5cbfbc-7654-4f7f-a036-c8ffd3c26819" containerName="payload-pre-processing" Oct 31 04:14:56.727998 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:56.727985 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dd4051c-8d79-496e-b3cb-d90263e6f1be" containerName="istio-proxy" Oct 31 04:14:56.727998 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:56.727995 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dd4051c-8d79-496e-b3cb-d90263e6f1be" containerName="istio-proxy" Oct 31 04:14:56.733303 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:56.733260 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-439989-6b4f868455-v228h" Oct 31 04:14:56.748050 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:56.747999 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-439989-6b4f868455-v228h"] Oct 31 04:14:56.754502 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:56.754462 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-439989-dockercfg-hwjxl\"" Oct 31 04:14:56.754648 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:56.754564 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-439989\"" Oct 31 04:14:56.805605 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:56.805186 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnf59\" (UniqueName: \"kubernetes.io/projected/406761c9-51b9-4105-88ce-bd0fac558eb1-kube-api-access-qnf59\") pod \"payload-pre-processing-e2e-worker-w1-439989-6b4f868455-v228h\" (UID: \"406761c9-51b9-4105-88ce-bd0fac558eb1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-439989-6b4f868455-v228h" Oct 31 04:14:56.805605 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:56.805249 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/406761c9-51b9-4105-88ce-bd0fac558eb1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-439989-6b4f868455-v228h\" (UID: \"406761c9-51b9-4105-88ce-bd0fac558eb1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-439989-6b4f868455-v228h" Oct 31 04:14:56.913282 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:56.912856 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnf59\" (UniqueName: \"kubernetes.io/projected/406761c9-51b9-4105-88ce-bd0fac558eb1-kube-api-access-qnf59\") pod \"payload-pre-processing-e2e-worker-w1-439989-6b4f868455-v228h\" (UID: \"406761c9-51b9-4105-88ce-bd0fac558eb1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-439989-6b4f868455-v228h" Oct 31 04:14:56.913591 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:56.913567 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/406761c9-51b9-4105-88ce-bd0fac558eb1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-439989-6b4f868455-v228h\" (UID: \"406761c9-51b9-4105-88ce-bd0fac558eb1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-439989-6b4f868455-v228h" Oct 31 04:14:56.914717 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:56.914591 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/406761c9-51b9-4105-88ce-bd0fac558eb1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-439989-6b4f868455-v228h\" (UID: \"406761c9-51b9-4105-88ce-bd0fac558eb1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-439989-6b4f868455-v228h" Oct 31 04:14:56.943802 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:56.943730 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnf59\" (UniqueName: \"kubernetes.io/projected/406761c9-51b9-4105-88ce-bd0fac558eb1-kube-api-access-qnf59\") pod \"payload-pre-processing-e2e-worker-w1-439989-6b4f868455-v228h\" (UID: \"406761c9-51b9-4105-88ce-bd0fac558eb1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-439989-6b4f868455-v228h" Oct 31 04:14:56.957206 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:56.957074 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb" Oct 31 04:14:56.999568 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:56.999532 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc" Oct 31 04:14:57.016175 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.016135 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec1714c9-5e99-400a-b392-486c04926361-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec1714c9-5e99-400a-b392-486c04926361-istio-data\") pod \"ec1714c9-5e99-400a-b392-486c04926361\" (UID: \"ec1714c9-5e99-400a-b392-486c04926361\") " Oct 31 04:14:57.016369 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.016208 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec1714c9-5e99-400a-b392-486c04926361-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1714c9-5e99-400a-b392-486c04926361-workload-socket\") pod \"ec1714c9-5e99-400a-b392-486c04926361\" (UID: \"ec1714c9-5e99-400a-b392-486c04926361\") " Oct 31 04:14:57.016369 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.016249 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec1714c9-5e99-400a-b392-486c04926361-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec1714c9-5e99-400a-b392-486c04926361-workload-certs\") pod \"ec1714c9-5e99-400a-b392-486c04926361\" (UID: \"ec1714c9-5e99-400a-b392-486c04926361\") " Oct 31 04:14:57.016369 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.016305 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec1714c9-5e99-400a-b392-486c04926361-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec1714c9-5e99-400a-b392-486c04926361-istio-envoy\") pod \"ec1714c9-5e99-400a-b392-486c04926361\" (UID: \"ec1714c9-5e99-400a-b392-486c04926361\") " Oct 31 04:14:57.016542 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.016388 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec1714c9-5e99-400a-b392-486c04926361-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec1714c9-5e99-400a-b392-486c04926361-istio-podinfo\") pod \"ec1714c9-5e99-400a-b392-486c04926361\" (UID: \"ec1714c9-5e99-400a-b392-486c04926361\") " Oct 31 04:14:57.016542 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.016446 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec1714c9-5e99-400a-b392-486c04926361-kube-api-access-b7nvl\" (UniqueName: \"kubernetes.io/projected/ec1714c9-5e99-400a-b392-486c04926361-kube-api-access-b7nvl\") pod \"ec1714c9-5e99-400a-b392-486c04926361\" (UID: \"ec1714c9-5e99-400a-b392-486c04926361\") " Oct 31 04:14:57.016542 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.016507 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec1714c9-5e99-400a-b392-486c04926361-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1714c9-5e99-400a-b392-486c04926361-credential-socket\") pod \"ec1714c9-5e99-400a-b392-486c04926361\" (UID: \"ec1714c9-5e99-400a-b392-486c04926361\") " Oct 31 04:14:57.016690 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.016561 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec1714c9-5e99-400a-b392-486c04926361-istio-token\" (UniqueName: \"kubernetes.io/projected/ec1714c9-5e99-400a-b392-486c04926361-istio-token\") pod \"ec1714c9-5e99-400a-b392-486c04926361\" (UID: \"ec1714c9-5e99-400a-b392-486c04926361\") " Oct 31 04:14:57.016690 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.016652 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec1714c9-5e99-400a-b392-486c04926361-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec1714c9-5e99-400a-b392-486c04926361-istiod-ca-cert\") pod \"ec1714c9-5e99-400a-b392-486c04926361\" (UID: \"ec1714c9-5e99-400a-b392-486c04926361\") " Oct 31 04:14:57.017631 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.017598 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec1714c9-5e99-400a-b392-486c04926361-istiod-ca-cert" pod "ec1714c9-5e99-400a-b392-486c04926361" (UID: "ec1714c9-5e99-400a-b392-486c04926361"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:14:57.018227 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.017950 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec1714c9-5e99-400a-b392-486c04926361-credential-socket" pod "ec1714c9-5e99-400a-b392-486c04926361" (UID: "ec1714c9-5e99-400a-b392-486c04926361"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:14:57.021313 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.020644 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec1714c9-5e99-400a-b392-486c04926361-kube-api-access-b7nvl" pod "ec1714c9-5e99-400a-b392-486c04926361" (UID: "ec1714c9-5e99-400a-b392-486c04926361"). InnerVolumeSpecName "kube-api-access-b7nvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:14:57.021313 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.020891 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec1714c9-5e99-400a-b392-486c04926361-workload-socket" pod "ec1714c9-5e99-400a-b392-486c04926361" (UID: "ec1714c9-5e99-400a-b392-486c04926361"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:14:57.021313 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.021088 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec1714c9-5e99-400a-b392-486c04926361-istio-data" pod "ec1714c9-5e99-400a-b392-486c04926361" (UID: "ec1714c9-5e99-400a-b392-486c04926361"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:14:57.023943 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.023916 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec1714c9-5e99-400a-b392-486c04926361-istio-token" pod "ec1714c9-5e99-400a-b392-486c04926361" (UID: "ec1714c9-5e99-400a-b392-486c04926361"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:14:57.025085 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.025057 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec1714c9-5e99-400a-b392-486c04926361-workload-certs" pod "ec1714c9-5e99-400a-b392-486c04926361" (UID: "ec1714c9-5e99-400a-b392-486c04926361"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:14:57.027235 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.027209 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec1714c9-5e99-400a-b392-486c04926361-istio-envoy" pod "ec1714c9-5e99-400a-b392-486c04926361" (UID: "ec1714c9-5e99-400a-b392-486c04926361"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:14:57.039767 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.039668 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec1714c9-5e99-400a-b392-486c04926361-istio-podinfo" pod "ec1714c9-5e99-400a-b392-486c04926361" (UID: "ec1714c9-5e99-400a-b392-486c04926361"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:14:57.058533 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.058274 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-439989-6b4f868455-v228h" Oct 31 04:14:57.119355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.117946 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c65c601e-7e95-4e7d-844c-a4cadd680d1b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c65c601e-7e95-4e7d-844c-a4cadd680d1b-istiod-ca-cert\") pod \"c65c601e-7e95-4e7d-844c-a4cadd680d1b\" (UID: \"c65c601e-7e95-4e7d-844c-a4cadd680d1b\") " Oct 31 04:14:57.119355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.118020 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c65c601e-7e95-4e7d-844c-a4cadd680d1b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c65c601e-7e95-4e7d-844c-a4cadd680d1b-workload-socket\") pod \"c65c601e-7e95-4e7d-844c-a4cadd680d1b\" (UID: \"c65c601e-7e95-4e7d-844c-a4cadd680d1b\") " Oct 31 04:14:57.119355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.118116 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c65c601e-7e95-4e7d-844c-a4cadd680d1b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c65c601e-7e95-4e7d-844c-a4cadd680d1b-istio-podinfo\") pod \"c65c601e-7e95-4e7d-844c-a4cadd680d1b\" (UID: \"c65c601e-7e95-4e7d-844c-a4cadd680d1b\") " Oct 31 04:14:57.119355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.118153 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c65c601e-7e95-4e7d-844c-a4cadd680d1b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c65c601e-7e95-4e7d-844c-a4cadd680d1b-istio-data\") pod \"c65c601e-7e95-4e7d-844c-a4cadd680d1b\" (UID: \"c65c601e-7e95-4e7d-844c-a4cadd680d1b\") " Oct 31 04:14:57.119355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.118201 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c65c601e-7e95-4e7d-844c-a4cadd680d1b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c65c601e-7e95-4e7d-844c-a4cadd680d1b-workload-certs\") pod \"c65c601e-7e95-4e7d-844c-a4cadd680d1b\" (UID: \"c65c601e-7e95-4e7d-844c-a4cadd680d1b\") " Oct 31 04:14:57.119355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.118277 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c65c601e-7e95-4e7d-844c-a4cadd680d1b-kube-api-access-7s78l\" (UniqueName: \"kubernetes.io/projected/c65c601e-7e95-4e7d-844c-a4cadd680d1b-kube-api-access-7s78l\") pod \"c65c601e-7e95-4e7d-844c-a4cadd680d1b\" (UID: \"c65c601e-7e95-4e7d-844c-a4cadd680d1b\") " Oct 31 04:14:57.119355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.118319 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c65c601e-7e95-4e7d-844c-a4cadd680d1b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c65c601e-7e95-4e7d-844c-a4cadd680d1b-credential-socket\") pod \"c65c601e-7e95-4e7d-844c-a4cadd680d1b\" (UID: \"c65c601e-7e95-4e7d-844c-a4cadd680d1b\") " Oct 31 04:14:57.119355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.118407 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c65c601e-7e95-4e7d-844c-a4cadd680d1b-istio-token\" (UniqueName: \"kubernetes.io/projected/c65c601e-7e95-4e7d-844c-a4cadd680d1b-istio-token\") pod \"c65c601e-7e95-4e7d-844c-a4cadd680d1b\" (UID: \"c65c601e-7e95-4e7d-844c-a4cadd680d1b\") " Oct 31 04:14:57.119355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.118434 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c65c601e-7e95-4e7d-844c-a4cadd680d1b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c65c601e-7e95-4e7d-844c-a4cadd680d1b-istio-envoy\") pod \"c65c601e-7e95-4e7d-844c-a4cadd680d1b\" (UID: \"c65c601e-7e95-4e7d-844c-a4cadd680d1b\") " Oct 31 04:14:57.119355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.118844 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec1714c9-5e99-400a-b392-486c04926361-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:14:57.119355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.118862 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1714c9-5e99-400a-b392-486c04926361-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:14:57.119355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.118876 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec1714c9-5e99-400a-b392-486c04926361-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:14:57.119355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.118889 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec1714c9-5e99-400a-b392-486c04926361-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:14:57.119355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.118901 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec1714c9-5e99-400a-b392-486c04926361-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:14:57.119355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.118914 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7nvl\" (UniqueName: \"kubernetes.io/projected/ec1714c9-5e99-400a-b392-486c04926361-kube-api-access-b7nvl\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:14:57.119355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.118928 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1714c9-5e99-400a-b392-486c04926361-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:14:57.119355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.118941 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec1714c9-5e99-400a-b392-486c04926361-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:14:57.119355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.118955 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec1714c9-5e99-400a-b392-486c04926361-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:14:57.120827 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.120768 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c65c601e-7e95-4e7d-844c-a4cadd680d1b-istio-data" pod "c65c601e-7e95-4e7d-844c-a4cadd680d1b" (UID: "c65c601e-7e95-4e7d-844c-a4cadd680d1b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:14:57.121047 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.121020 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c65c601e-7e95-4e7d-844c-a4cadd680d1b-istio-envoy" pod "c65c601e-7e95-4e7d-844c-a4cadd680d1b" (UID: "c65c601e-7e95-4e7d-844c-a4cadd680d1b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:14:57.121122 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.120826 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c65c601e-7e95-4e7d-844c-a4cadd680d1b-workload-certs" pod "c65c601e-7e95-4e7d-844c-a4cadd680d1b" (UID: "c65c601e-7e95-4e7d-844c-a4cadd680d1b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:14:57.121190 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.120841 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c65c601e-7e95-4e7d-844c-a4cadd680d1b-workload-socket" pod "c65c601e-7e95-4e7d-844c-a4cadd680d1b" (UID: "c65c601e-7e95-4e7d-844c-a4cadd680d1b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:14:57.121240 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.121196 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c65c601e-7e95-4e7d-844c-a4cadd680d1b-credential-socket" pod "c65c601e-7e95-4e7d-844c-a4cadd680d1b" (UID: "c65c601e-7e95-4e7d-844c-a4cadd680d1b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:14:57.121751 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.121721 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c65c601e-7e95-4e7d-844c-a4cadd680d1b-istiod-ca-cert" pod "c65c601e-7e95-4e7d-844c-a4cadd680d1b" (UID: "c65c601e-7e95-4e7d-844c-a4cadd680d1b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:14:57.123471 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.123450 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c65c601e-7e95-4e7d-844c-a4cadd680d1b-kube-api-access-7s78l" pod "c65c601e-7e95-4e7d-844c-a4cadd680d1b" (UID: "c65c601e-7e95-4e7d-844c-a4cadd680d1b"). InnerVolumeSpecName "kube-api-access-7s78l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:14:57.123699 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.123676 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c65c601e-7e95-4e7d-844c-a4cadd680d1b-istio-podinfo" pod "c65c601e-7e95-4e7d-844c-a4cadd680d1b" (UID: "c65c601e-7e95-4e7d-844c-a4cadd680d1b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:14:57.124652 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.124612 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c65c601e-7e95-4e7d-844c-a4cadd680d1b-istio-token" pod "c65c601e-7e95-4e7d-844c-a4cadd680d1b" (UID: "c65c601e-7e95-4e7d-844c-a4cadd680d1b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:14:57.220512 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.219785 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c65c601e-7e95-4e7d-844c-a4cadd680d1b-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:14:57.220512 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.219847 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c65c601e-7e95-4e7d-844c-a4cadd680d1b-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:14:57.220512 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.219862 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c65c601e-7e95-4e7d-844c-a4cadd680d1b-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:14:57.220512 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.219878 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c65c601e-7e95-4e7d-844c-a4cadd680d1b-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:14:57.220512 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.219892 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c65c601e-7e95-4e7d-844c-a4cadd680d1b-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:14:57.220512 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.219906 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s78l\" (UniqueName: \"kubernetes.io/projected/c65c601e-7e95-4e7d-844c-a4cadd680d1b-kube-api-access-7s78l\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:14:57.220512 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.219921 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c65c601e-7e95-4e7d-844c-a4cadd680d1b-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:14:57.220512 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.219936 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c65c601e-7e95-4e7d-844c-a4cadd680d1b-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:14:57.220512 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.219951 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c65c601e-7e95-4e7d-844c-a4cadd680d1b-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:14:57.413033 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:14:57.412971 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod406761c9_51b9_4105_88ce_bd0fac558eb1.slice/crio-d0e6f4a5fbc8d8385469b4f1af642d66352a7889ff62dca62812e9df4667bd99 WatchSource:0}: Error finding container d0e6f4a5fbc8d8385469b4f1af642d66352a7889ff62dca62812e9df4667bd99: Status 404 returned error can't find the container with id d0e6f4a5fbc8d8385469b4f1af642d66352a7889ff62dca62812e9df4667bd99 Oct 31 04:14:57.417832 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.417553 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-439989-6b4f868455-v228h"] Oct 31 04:14:57.692407 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.691027 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc" event={"ID":"c65c601e-7e95-4e7d-844c-a4cadd680d1b","Type":"ContainerDied","Data":"12023b1b1cbd5f7c9ed5961cba4bd9a9c642909e30fc0bf75db2d440dc1524e6"} Oct 31 04:14:57.692407 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.691104 5019 scope.go:122] "RemoveContainer" containerID="5b0032a70f02f03c928a2893d87d6d17bc50a01807463715fbb9b4b67058f484" Oct 31 04:14:57.692407 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.691270 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc" Oct 31 04:14:57.703088 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.702240 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-439989-6b4f868455-v228h" event={"ID":"406761c9-51b9-4105-88ce-bd0fac558eb1","Type":"ContainerStarted","Data":"e3de073ce9e0303b08fd111494884cd51743cb376c8b21e0cb83879589726517"} Oct 31 04:14:57.703088 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.702307 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-439989-6b4f868455-v228h" event={"ID":"406761c9-51b9-4105-88ce-bd0fac558eb1","Type":"ContainerStarted","Data":"d0e6f4a5fbc8d8385469b4f1af642d66352a7889ff62dca62812e9df4667bd99"} Oct 31 04:14:57.703088 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.702835 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-439989-6b4f868455-v228h" Oct 31 04:14:57.705415 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.704639 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb" event={"ID":"ec1714c9-5e99-400a-b392-486c04926361","Type":"ContainerDied","Data":"b84833c23487a75dcab5f515d1139451c7b1317fa86678a9dc985e52d10df579"} Oct 31 04:14:57.705415 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.704759 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb" Oct 31 04:14:57.716259 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.716230 5019 scope.go:122] "RemoveContainer" containerID="00d09e779f30b22182e3ea6673996014cdf91ca88e1de182b6d8b965fbc777c2" Oct 31 04:14:57.755376 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.755326 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc"] Oct 31 04:14:57.765703 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.765667 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-m6bjc"] Oct 31 04:14:57.798148 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.798074 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-439989-6b4f868455-v228h" podStartSLOduration=1.7980498900000002 podStartE2EDuration="1.79804989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:14:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:14:57.79475727 +0000 UTC m=+652.551095311" watchObservedRunningTime="2026-10-31 04:14:57.79804989 +0000 UTC m=+652.554387932" Oct 31 04:14:57.833739 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.833700 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb"] Oct 31 04:14:57.839978 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:57.839929 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-hm4zb"] Oct 31 04:14:59.410679 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:59.409902 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c65c601e-7e95-4e7d-844c-a4cadd680d1b" path="/var/lib/kubelet/pods/c65c601e-7e95-4e7d-844c-a4cadd680d1b/volumes" Oct 31 04:14:59.410679 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:14:59.410377 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec1714c9-5e99-400a-b392-486c04926361" path="/var/lib/kubelet/pods/ec1714c9-5e99-400a-b392-486c04926361/volumes" Oct 31 04:15:00.003138 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.003092 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c4006d-548d7c6f4c-nj2nr"] Oct 31 04:15:00.003907 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.003880 5019 cpu_manager.go:414] "containerMap: removing container" podUID="ec1714c9-5e99-400a-b392-486c04926361" containerName="istio-proxy" Oct 31 04:15:00.004056 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.003909 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec1714c9-5e99-400a-b392-486c04926361" containerName="istio-proxy" Oct 31 04:15:00.004056 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.003945 5019 cpu_manager.go:414] "containerMap: removing container" podUID="c65c601e-7e95-4e7d-844c-a4cadd680d1b" containerName="istio-proxy" Oct 31 04:15:00.004056 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.003953 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c65c601e-7e95-4e7d-844c-a4cadd680d1b" containerName="istio-proxy" Oct 31 04:15:00.004056 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.003987 5019 cpu_manager.go:414] "containerMap: removing container" podUID="3dd4051c-8d79-496e-b3cb-d90263e6f1be" containerName="istio-proxy" Oct 31 04:15:00.004056 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.003995 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dd4051c-8d79-496e-b3cb-d90263e6f1be" containerName="istio-proxy" Oct 31 04:15:00.006436 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.006399 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec1714c9-5e99-400a-b392-486c04926361" containerName="istio-proxy" Oct 31 04:15:00.006554 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.006441 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec1714c9-5e99-400a-b392-486c04926361" containerName="istio-proxy" Oct 31 04:15:00.006554 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.006461 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="c65c601e-7e95-4e7d-844c-a4cadd680d1b" containerName="istio-proxy" Oct 31 04:15:00.011491 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.011467 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c4006d-548d7c6f4c-nj2nr" Oct 31 04:15:00.041214 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.040566 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-c4006d\"" Oct 31 04:15:00.050825 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.050783 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c4006d-548d7c6f4c-nj2nr"] Oct 31 04:15:00.051140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.051121 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c4006d-7bd575dbdf-h26mj"] Oct 31 04:15:00.052267 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.052240 5019 cpu_manager.go:414] "containerMap: removing container" podUID="ec1714c9-5e99-400a-b392-486c04926361" containerName="istio-proxy" Oct 31 04:15:00.052479 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.052463 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec1714c9-5e99-400a-b392-486c04926361" containerName="istio-proxy" Oct 31 04:15:00.057857 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.057829 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-c4006d-7bd575dbdf-h26mj" Oct 31 04:15:00.064064 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.063950 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab2c1b1c-1f3f-43fb-aa9d-248870c31da1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-c4006d-548d7c6f4c-nj2nr\" (UID: \"ab2c1b1c-1f3f-43fb-aa9d-248870c31da1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c4006d-548d7c6f4c-nj2nr" Oct 31 04:15:00.064064 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.064028 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khjbw\" (UniqueName: \"kubernetes.io/projected/ab2c1b1c-1f3f-43fb-aa9d-248870c31da1-kube-api-access-khjbw\") pod \"payload-pre-processing-e2e-worker-w4-c4006d-548d7c6f4c-nj2nr\" (UID: \"ab2c1b1c-1f3f-43fb-aa9d-248870c31da1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c4006d-548d7c6f4c-nj2nr" Oct 31 04:15:00.136510 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.119907 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c4006d-7bd575dbdf-h26mj"] Oct 31 04:15:00.136510 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.125918 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-c4006d-dockercfg-cspf2\"" Oct 31 04:15:00.162429 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.160932 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890335-l9tgn"] Oct 31 04:15:00.164989 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.164958 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890335-bpznn"] Oct 31 04:15:00.165121 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.165088 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890335-l9tgn" Oct 31 04:15:00.167942 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.167463 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31f4b14c-49f7-45f4-936e-e1be3123640c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-c4006d-7bd575dbdf-h26mj\" (UID: \"31f4b14c-49f7-45f4-936e-e1be3123640c\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c4006d-7bd575dbdf-h26mj" Oct 31 04:15:00.167942 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.167568 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brlqv\" (UniqueName: \"kubernetes.io/projected/31f4b14c-49f7-45f4-936e-e1be3123640c-kube-api-access-brlqv\") pod \"payload-processing-e2e-worker-w4-c4006d-7bd575dbdf-h26mj\" (UID: \"31f4b14c-49f7-45f4-936e-e1be3123640c\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c4006d-7bd575dbdf-h26mj" Oct 31 04:15:00.167942 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.167721 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab2c1b1c-1f3f-43fb-aa9d-248870c31da1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-c4006d-548d7c6f4c-nj2nr\" (UID: \"ab2c1b1c-1f3f-43fb-aa9d-248870c31da1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c4006d-548d7c6f4c-nj2nr" Oct 31 04:15:00.167942 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.167816 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khjbw\" (UniqueName: \"kubernetes.io/projected/ab2c1b1c-1f3f-43fb-aa9d-248870c31da1-kube-api-access-khjbw\") pod \"payload-pre-processing-e2e-worker-w4-c4006d-548d7c6f4c-nj2nr\" (UID: \"ab2c1b1c-1f3f-43fb-aa9d-248870c31da1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c4006d-548d7c6f4c-nj2nr" Oct 31 04:15:00.170360 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.170276 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab2c1b1c-1f3f-43fb-aa9d-248870c31da1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-c4006d-548d7c6f4c-nj2nr\" (UID: \"ab2c1b1c-1f3f-43fb-aa9d-248870c31da1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c4006d-548d7c6f4c-nj2nr" Oct 31 04:15:00.192065 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.191482 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:15:00.192065 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.191596 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:15:00.218353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.216953 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890335-9txq4"] Oct 31 04:15:00.218353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.218207 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-bpznn" Oct 31 04:15:00.237197 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.237158 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khjbw\" (UniqueName: \"kubernetes.io/projected/ab2c1b1c-1f3f-43fb-aa9d-248870c31da1-kube-api-access-khjbw\") pod \"payload-pre-processing-e2e-worker-w4-c4006d-548d7c6f4c-nj2nr\" (UID: \"ab2c1b1c-1f3f-43fb-aa9d-248870c31da1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c4006d-548d7c6f4c-nj2nr" Oct 31 04:15:00.254748 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.238962 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:15:00.254748 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.240072 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890335-l9tgn"] Oct 31 04:15:00.254748 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.240190 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890335-9txq4" Oct 31 04:15:00.254748 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.245748 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890335-bpznn"] Oct 31 04:15:00.254748 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.251838 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890335-9txq4"] Oct 31 04:15:00.269240 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.269197 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pg4b\" (UniqueName: \"kubernetes.io/projected/5706a564-0118-4cbf-805a-c5fcbcfee2d6-kube-api-access-7pg4b\") pod \"maas-api-key-cleanup-29890335-bpznn\" (UID: \"5706a564-0118-4cbf-805a-c5fcbcfee2d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-bpznn" Oct 31 04:15:00.275507 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.269273 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2db08bdd-9125-4b67-8434-10ab50702c61-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-4c2387-29890335-l9tgn\" (UID: \"2db08bdd-9125-4b67-8434-10ab50702c61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890335-l9tgn" Oct 31 04:15:00.275507 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.269387 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31f4b14c-49f7-45f4-936e-e1be3123640c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-c4006d-7bd575dbdf-h26mj\" (UID: \"31f4b14c-49f7-45f4-936e-e1be3123640c\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c4006d-7bd575dbdf-h26mj" Oct 31 04:15:00.275507 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.269425 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rff5\" (UniqueName: \"kubernetes.io/projected/2db08bdd-9125-4b67-8434-10ab50702c61-kube-api-access-2rff5\") pod \"maas-api-key-cleanup-e2e-worker-w3-4c2387-29890335-l9tgn\" (UID: \"2db08bdd-9125-4b67-8434-10ab50702c61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890335-l9tgn" Oct 31 04:15:00.275507 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.269460 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5706a564-0118-4cbf-805a-c5fcbcfee2d6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890335-bpznn\" (UID: \"5706a564-0118-4cbf-805a-c5fcbcfee2d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-bpznn" Oct 31 04:15:00.275507 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.269516 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brlqv\" (UniqueName: \"kubernetes.io/projected/31f4b14c-49f7-45f4-936e-e1be3123640c-kube-api-access-brlqv\") pod \"payload-processing-e2e-worker-w4-c4006d-7bd575dbdf-h26mj\" (UID: \"31f4b14c-49f7-45f4-936e-e1be3123640c\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c4006d-7bd575dbdf-h26mj" Oct 31 04:15:00.275507 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.270964 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31f4b14c-49f7-45f4-936e-e1be3123640c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-c4006d-7bd575dbdf-h26mj\" (UID: \"31f4b14c-49f7-45f4-936e-e1be3123640c\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c4006d-7bd575dbdf-h26mj" Oct 31 04:15:00.318960 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.318926 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brlqv\" (UniqueName: \"kubernetes.io/projected/31f4b14c-49f7-45f4-936e-e1be3123640c-kube-api-access-brlqv\") pod \"payload-processing-e2e-worker-w4-c4006d-7bd575dbdf-h26mj\" (UID: \"31f4b14c-49f7-45f4-936e-e1be3123640c\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c4006d-7bd575dbdf-h26mj" Oct 31 04:15:00.345445 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.345404 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c4006d-548d7c6f4c-nj2nr" Oct 31 04:15:00.393243 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.381945 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rff5\" (UniqueName: \"kubernetes.io/projected/2db08bdd-9125-4b67-8434-10ab50702c61-kube-api-access-2rff5\") pod \"maas-api-key-cleanup-e2e-worker-w3-4c2387-29890335-l9tgn\" (UID: \"2db08bdd-9125-4b67-8434-10ab50702c61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890335-l9tgn" Oct 31 04:15:00.393243 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.382017 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5706a564-0118-4cbf-805a-c5fcbcfee2d6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890335-bpznn\" (UID: \"5706a564-0118-4cbf-805a-c5fcbcfee2d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-bpznn" Oct 31 04:15:00.393243 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.382112 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cea1544c-25d6-4a37-9205-78fe8ece0a3a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-09a20a-29890335-9txq4\" (UID: \"cea1544c-25d6-4a37-9205-78fe8ece0a3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890335-9txq4" Oct 31 04:15:00.393243 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.382198 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pg4b\" (UniqueName: \"kubernetes.io/projected/5706a564-0118-4cbf-805a-c5fcbcfee2d6-kube-api-access-7pg4b\") pod \"maas-api-key-cleanup-29890335-bpznn\" (UID: \"5706a564-0118-4cbf-805a-c5fcbcfee2d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-bpznn" Oct 31 04:15:00.393243 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.382236 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqh89\" (UniqueName: \"kubernetes.io/projected/cea1544c-25d6-4a37-9205-78fe8ece0a3a-kube-api-access-wqh89\") pod \"maas-api-key-cleanup-e2e-worker-w2-09a20a-29890335-9txq4\" (UID: \"cea1544c-25d6-4a37-9205-78fe8ece0a3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890335-9txq4" Oct 31 04:15:00.393243 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.382268 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2db08bdd-9125-4b67-8434-10ab50702c61-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-4c2387-29890335-l9tgn\" (UID: \"2db08bdd-9125-4b67-8434-10ab50702c61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890335-l9tgn" Oct 31 04:15:00.393243 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.383561 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2db08bdd-9125-4b67-8434-10ab50702c61-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-4c2387-29890335-l9tgn\" (UID: \"2db08bdd-9125-4b67-8434-10ab50702c61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890335-l9tgn" Oct 31 04:15:00.393243 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.384234 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5706a564-0118-4cbf-805a-c5fcbcfee2d6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890335-bpznn\" (UID: \"5706a564-0118-4cbf-805a-c5fcbcfee2d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-bpznn" Oct 31 04:15:00.408050 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.408019 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pg4b\" (UniqueName: \"kubernetes.io/projected/5706a564-0118-4cbf-805a-c5fcbcfee2d6-kube-api-access-7pg4b\") pod \"maas-api-key-cleanup-29890335-bpznn\" (UID: \"5706a564-0118-4cbf-805a-c5fcbcfee2d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-bpznn" Oct 31 04:15:00.408783 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.408757 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rff5\" (UniqueName: \"kubernetes.io/projected/2db08bdd-9125-4b67-8434-10ab50702c61-kube-api-access-2rff5\") pod \"maas-api-key-cleanup-e2e-worker-w3-4c2387-29890335-l9tgn\" (UID: \"2db08bdd-9125-4b67-8434-10ab50702c61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890335-l9tgn" Oct 31 04:15:00.425855 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.425816 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-c4006d-7bd575dbdf-h26mj" Oct 31 04:15:00.514806 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.498961 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqh89\" (UniqueName: \"kubernetes.io/projected/cea1544c-25d6-4a37-9205-78fe8ece0a3a-kube-api-access-wqh89\") pod \"maas-api-key-cleanup-e2e-worker-w2-09a20a-29890335-9txq4\" (UID: \"cea1544c-25d6-4a37-9205-78fe8ece0a3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890335-9txq4" Oct 31 04:15:00.514806 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.499192 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cea1544c-25d6-4a37-9205-78fe8ece0a3a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-09a20a-29890335-9txq4\" (UID: \"cea1544c-25d6-4a37-9205-78fe8ece0a3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890335-9txq4" Oct 31 04:15:00.514806 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.500060 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cea1544c-25d6-4a37-9205-78fe8ece0a3a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-09a20a-29890335-9txq4\" (UID: \"cea1544c-25d6-4a37-9205-78fe8ece0a3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890335-9txq4" Oct 31 04:15:00.530038 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.528958 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqh89\" (UniqueName: \"kubernetes.io/projected/cea1544c-25d6-4a37-9205-78fe8ece0a3a-kube-api-access-wqh89\") pod \"maas-api-key-cleanup-e2e-worker-w2-09a20a-29890335-9txq4\" (UID: \"cea1544c-25d6-4a37-9205-78fe8ece0a3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890335-9txq4" Oct 31 04:15:00.625801 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.625766 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-bpznn" Oct 31 04:15:00.647270 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.647236 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890335-l9tgn" Oct 31 04:15:00.662352 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.662299 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890335-9txq4" Oct 31 04:15:00.701380 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.701271 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c4006d-548d7c6f4c-nj2nr"] Oct 31 04:15:00.737376 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.736960 5019 generic.go:356] "Generic (PLEG): container finished" podID="cc1adc72-6548-4daf-9dc1-2b96c853cdbf" containerID="8b49042a4d5200c98a78e4465ba5fbf55837632fc4c8f40d09980add49bdd0fc" exitCode=0 Oct 31 04:15:00.737376 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.737094 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx" event={"ID":"cc1adc72-6548-4daf-9dc1-2b96c853cdbf","Type":"ContainerDied","Data":"8b49042a4d5200c98a78e4465ba5fbf55837632fc4c8f40d09980add49bdd0fc"} Oct 31 04:15:00.737573 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.737501 5019 scope.go:122] "RemoveContainer" containerID="8b49042a4d5200c98a78e4465ba5fbf55837632fc4c8f40d09980add49bdd0fc" Oct 31 04:15:00.739311 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.739279 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:15:00.739465 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.739394 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:15:00.739532 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.739472 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:15:00.747488 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.744758 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c4006d-548d7c6f4c-nj2nr" event={"ID":"ab2c1b1c-1f3f-43fb-aa9d-248870c31da1","Type":"ContainerStarted","Data":"0f6d38deab920feb0b4b809ada7c36daa3eac73371a5be1222653aa9c77e8279"} Oct 31 04:15:00.808107 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:00.807636 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c4006d-7bd575dbdf-h26mj"] Oct 31 04:15:01.008720 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:01.008003 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890335-bpznn"] Oct 31 04:15:01.009913 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:01.009884 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890335-l9tgn"] Oct 31 04:15:01.015918 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:15:01.015867 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2db08bdd_9125_4b67_8434_10ab50702c61.slice/crio-a8efe7749164d4b058e609e61c95880d0fab6afe66dca7bdab5f7ae954c0d08d WatchSource:0}: Error finding container a8efe7749164d4b058e609e61c95880d0fab6afe66dca7bdab5f7ae954c0d08d: Status 404 returned error can't find the container with id a8efe7749164d4b058e609e61c95880d0fab6afe66dca7bdab5f7ae954c0d08d Oct 31 04:15:01.064642 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:01.064608 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890335-9txq4"] Oct 31 04:15:01.066288 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:15:01.066259 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcea1544c_25d6_4a37_9205_78fe8ece0a3a.slice/crio-cea44c4c84e5b184e70ca1e8e716c9ecc31b1ed77daf6c33427dcdb1a4b0f6d7 WatchSource:0}: Error finding container cea44c4c84e5b184e70ca1e8e716c9ecc31b1ed77daf6c33427dcdb1a4b0f6d7: Status 404 returned error can't find the container with id cea44c4c84e5b184e70ca1e8e716c9ecc31b1ed77daf6c33427dcdb1a4b0f6d7 Oct 31 04:15:01.752859 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:01.752813 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-c4006d-7bd575dbdf-h26mj" event={"ID":"31f4b14c-49f7-45f4-936e-e1be3123640c","Type":"ContainerStarted","Data":"8ec295a135d952b8d849c8118b1fd07cef6a9e32b87542d5b24b2dbc7205a7c8"} Oct 31 04:15:01.753308 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:01.752876 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-c4006d-7bd575dbdf-h26mj" event={"ID":"31f4b14c-49f7-45f4-936e-e1be3123640c","Type":"ContainerStarted","Data":"d91fd5b6ab45fb5517286117375299ed3bfe591d411c02db66818168cb8d24ed"} Oct 31 04:15:01.753308 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:01.753019 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-c4006d-7bd575dbdf-h26mj" Oct 31 04:15:01.755187 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:01.755156 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx" event={"ID":"cc1adc72-6548-4daf-9dc1-2b96c853cdbf","Type":"ContainerStarted","Data":"2289634591bd47dc3795be818dac69b25930c63f38c5395757dae4755423d8ca"} Oct 31 04:15:01.757242 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:01.757012 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890335-l9tgn" event={"ID":"2db08bdd-9125-4b67-8434-10ab50702c61","Type":"ContainerStarted","Data":"1e037554bf2a6d4469bd87611de76ba1ce3e95e9eb8cb7ad86b33933569cabad"} Oct 31 04:15:01.757242 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:01.757070 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890335-l9tgn" event={"ID":"2db08bdd-9125-4b67-8434-10ab50702c61","Type":"ContainerStarted","Data":"a8efe7749164d4b058e609e61c95880d0fab6afe66dca7bdab5f7ae954c0d08d"} Oct 31 04:15:01.758851 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:01.758582 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890335-9txq4" event={"ID":"cea1544c-25d6-4a37-9205-78fe8ece0a3a","Type":"ContainerStarted","Data":"db909bf92d9c65805ce5465d47b1d8694d374e99edf242ac9d73257abe1d5c66"} Oct 31 04:15:01.758851 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:01.758631 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890335-9txq4" event={"ID":"cea1544c-25d6-4a37-9205-78fe8ece0a3a","Type":"ContainerStarted","Data":"cea44c4c84e5b184e70ca1e8e716c9ecc31b1ed77daf6c33427dcdb1a4b0f6d7"} Oct 31 04:15:01.760391 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:01.760360 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c4006d-548d7c6f4c-nj2nr" event={"ID":"ab2c1b1c-1f3f-43fb-aa9d-248870c31da1","Type":"ContainerStarted","Data":"edddbd5cf556da8f6b319a00ce62442d6539e6ba1184d6a84df98513f0c1c64a"} Oct 31 04:15:01.760487 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:01.760424 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c4006d-548d7c6f4c-nj2nr" Oct 31 04:15:01.762128 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:01.762103 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-bpznn" event={"ID":"5706a564-0118-4cbf-805a-c5fcbcfee2d6","Type":"ContainerStarted","Data":"d2ff341bd6f63e2e578ab850ffb2f6377e7c657575a080456dd16d92912d5d3e"} Oct 31 04:15:01.762222 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:01.762149 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-bpznn" event={"ID":"5706a564-0118-4cbf-805a-c5fcbcfee2d6","Type":"ContainerStarted","Data":"47ed60ec2595dfa8ebf9bb067483d63593e987c84fd065a28c091dd1ef844b03"} Oct 31 04:15:01.804355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:01.797942 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-c4006d-7bd575dbdf-h26mj" podStartSLOduration=1.79792181 podStartE2EDuration="1.79792181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:15:01.794843265 +0000 UTC m=+656.551181306" watchObservedRunningTime="2026-10-31 04:15:01.79792181 +0000 UTC m=+656.554259855" Oct 31 04:15:01.838143 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:01.838079 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890335-l9tgn" podStartSLOduration=1.8380572 podStartE2EDuration="1.8380572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:15:01.836017663 +0000 UTC m=+656.592355706" watchObservedRunningTime="2026-10-31 04:15:01.8380572 +0000 UTC m=+656.594395244" Oct 31 04:15:01.876321 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:01.876192 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890335-9txq4" podStartSLOduration=1.876167592 podStartE2EDuration="1.876167592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:15:01.874942819 +0000 UTC m=+656.631280859" watchObservedRunningTime="2026-10-31 04:15:01.876167592 +0000 UTC m=+656.632505634" Oct 31 04:15:01.930538 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:01.930504 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z"] Oct 31 04:15:01.939686 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:01.939650 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" Oct 31 04:15:01.955654 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:01.955619 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-09a20a\"/\"kube-root-ca.crt\"" Oct 31 04:15:01.955859 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:01.955792 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-09a20a\"/\"openshift-service-ca.crt\"" Oct 31 04:15:01.955974 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:01.955875 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-09a20a\"/\"simulator-w2-09a20a-kserve-self-signed-certs\"" Oct 31 04:15:01.956067 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:01.955918 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-09a20a\"/\"default-dockercfg-qcqb8\"" Oct 31 04:15:01.956899 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:01.956874 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z"] Oct 31 04:15:02.010401 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:02.009623 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c4006d-548d7c6f4c-nj2nr" podStartSLOduration=3.00960331 podStartE2EDuration="3.00960331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:14:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:15:02.00931624 +0000 UTC m=+656.765654280" watchObservedRunningTime="2026-10-31 04:15:02.00960331 +0000 UTC m=+656.765941350" Oct 31 04:15:02.029638 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:02.029598 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dccd15a9-47f5-4bd1-b21b-9d5ca8b05845-home\") pod \"simulator-w2-09a20a-kserve-5dc88fc844-8jf6z\" (UID: \"dccd15a9-47f5-4bd1-b21b-9d5ca8b05845\") " pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" Oct 31 04:15:02.029839 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:02.029680 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dccd15a9-47f5-4bd1-b21b-9d5ca8b05845-model-cache\") pod \"simulator-w2-09a20a-kserve-5dc88fc844-8jf6z\" (UID: \"dccd15a9-47f5-4bd1-b21b-9d5ca8b05845\") " pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" Oct 31 04:15:02.029839 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:02.029703 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22z6n\" (UniqueName: \"kubernetes.io/projected/dccd15a9-47f5-4bd1-b21b-9d5ca8b05845-kube-api-access-22z6n\") pod \"simulator-w2-09a20a-kserve-5dc88fc844-8jf6z\" (UID: \"dccd15a9-47f5-4bd1-b21b-9d5ca8b05845\") " pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" Oct 31 04:15:02.029839 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:02.029744 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dccd15a9-47f5-4bd1-b21b-9d5ca8b05845-dshm\") pod \"simulator-w2-09a20a-kserve-5dc88fc844-8jf6z\" (UID: \"dccd15a9-47f5-4bd1-b21b-9d5ca8b05845\") " pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" Oct 31 04:15:02.029839 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:02.029821 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dccd15a9-47f5-4bd1-b21b-9d5ca8b05845-tmp-dir\") pod \"simulator-w2-09a20a-kserve-5dc88fc844-8jf6z\" (UID: \"dccd15a9-47f5-4bd1-b21b-9d5ca8b05845\") " pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" Oct 31 04:15:02.030057 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:02.029855 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dccd15a9-47f5-4bd1-b21b-9d5ca8b05845-tls-certs\") pod \"simulator-w2-09a20a-kserve-5dc88fc844-8jf6z\" (UID: \"dccd15a9-47f5-4bd1-b21b-9d5ca8b05845\") " pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" Oct 31 04:15:02.052301 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:02.052235 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-bpznn" podStartSLOduration=2.05221667 podStartE2EDuration="2.05221667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:15:02.049868989 +0000 UTC m=+656.806207030" watchObservedRunningTime="2026-10-31 04:15:02.05221667 +0000 UTC m=+656.808554710" Oct 31 04:15:02.131052 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:02.130940 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dccd15a9-47f5-4bd1-b21b-9d5ca8b05845-tmp-dir\") pod \"simulator-w2-09a20a-kserve-5dc88fc844-8jf6z\" (UID: \"dccd15a9-47f5-4bd1-b21b-9d5ca8b05845\") " pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" Oct 31 04:15:02.131363 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:02.131318 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dccd15a9-47f5-4bd1-b21b-9d5ca8b05845-tmp-dir\") pod \"simulator-w2-09a20a-kserve-5dc88fc844-8jf6z\" (UID: \"dccd15a9-47f5-4bd1-b21b-9d5ca8b05845\") " pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" Oct 31 04:15:02.131486 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:02.131324 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dccd15a9-47f5-4bd1-b21b-9d5ca8b05845-tls-certs\") pod \"simulator-w2-09a20a-kserve-5dc88fc844-8jf6z\" (UID: \"dccd15a9-47f5-4bd1-b21b-9d5ca8b05845\") " pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" Oct 31 04:15:02.131486 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:02.131425 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dccd15a9-47f5-4bd1-b21b-9d5ca8b05845-home\") pod \"simulator-w2-09a20a-kserve-5dc88fc844-8jf6z\" (UID: \"dccd15a9-47f5-4bd1-b21b-9d5ca8b05845\") " pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" Oct 31 04:15:02.131486 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:02.131482 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dccd15a9-47f5-4bd1-b21b-9d5ca8b05845-model-cache\") pod \"simulator-w2-09a20a-kserve-5dc88fc844-8jf6z\" (UID: \"dccd15a9-47f5-4bd1-b21b-9d5ca8b05845\") " pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" Oct 31 04:15:02.131643 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:02.131509 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22z6n\" (UniqueName: \"kubernetes.io/projected/dccd15a9-47f5-4bd1-b21b-9d5ca8b05845-kube-api-access-22z6n\") pod \"simulator-w2-09a20a-kserve-5dc88fc844-8jf6z\" (UID: \"dccd15a9-47f5-4bd1-b21b-9d5ca8b05845\") " pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" Oct 31 04:15:02.131643 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:02.131555 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dccd15a9-47f5-4bd1-b21b-9d5ca8b05845-dshm\") pod \"simulator-w2-09a20a-kserve-5dc88fc844-8jf6z\" (UID: \"dccd15a9-47f5-4bd1-b21b-9d5ca8b05845\") " pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" Oct 31 04:15:02.131765 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:02.131743 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dccd15a9-47f5-4bd1-b21b-9d5ca8b05845-home\") pod \"simulator-w2-09a20a-kserve-5dc88fc844-8jf6z\" (UID: \"dccd15a9-47f5-4bd1-b21b-9d5ca8b05845\") " pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" Oct 31 04:15:02.131919 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:02.131892 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dccd15a9-47f5-4bd1-b21b-9d5ca8b05845-model-cache\") pod \"simulator-w2-09a20a-kserve-5dc88fc844-8jf6z\" (UID: \"dccd15a9-47f5-4bd1-b21b-9d5ca8b05845\") " pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" Oct 31 04:15:02.134355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:02.133949 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dccd15a9-47f5-4bd1-b21b-9d5ca8b05845-dshm\") pod \"simulator-w2-09a20a-kserve-5dc88fc844-8jf6z\" (UID: \"dccd15a9-47f5-4bd1-b21b-9d5ca8b05845\") " pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" Oct 31 04:15:02.134570 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:02.134551 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dccd15a9-47f5-4bd1-b21b-9d5ca8b05845-tls-certs\") pod \"simulator-w2-09a20a-kserve-5dc88fc844-8jf6z\" (UID: \"dccd15a9-47f5-4bd1-b21b-9d5ca8b05845\") " pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" Oct 31 04:15:02.155076 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:02.155039 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22z6n\" (UniqueName: \"kubernetes.io/projected/dccd15a9-47f5-4bd1-b21b-9d5ca8b05845-kube-api-access-22z6n\") pod \"simulator-w2-09a20a-kserve-5dc88fc844-8jf6z\" (UID: \"dccd15a9-47f5-4bd1-b21b-9d5ca8b05845\") " pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" Oct 31 04:15:02.263439 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:02.263194 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" Oct 31 04:15:02.544604 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:02.544573 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx" Oct 31 04:15:02.545457 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:02.545422 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 31 04:15:02.545591 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:02.545486 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx" podUID="cc1adc72-6548-4daf-9dc1-2b96c853cdbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 31 04:15:02.676368 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:02.670067 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z"] Oct 31 04:15:02.802596 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:02.802552 5019 generic.go:356] "Generic (PLEG): container finished" podID="2db08bdd-9125-4b67-8434-10ab50702c61" containerID="1e037554bf2a6d4469bd87611de76ba1ce3e95e9eb8cb7ad86b33933569cabad" exitCode=60 Oct 31 04:15:02.803081 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:02.802720 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890335-l9tgn" event={"ID":"2db08bdd-9125-4b67-8434-10ab50702c61","Type":"ContainerDied","Data":"1e037554bf2a6d4469bd87611de76ba1ce3e95e9eb8cb7ad86b33933569cabad"} Oct 31 04:15:02.803081 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:02.802878 5019 scope.go:122] "RemoveContainer" containerID="1e037554bf2a6d4469bd87611de76ba1ce3e95e9eb8cb7ad86b33933569cabad" Oct 31 04:15:02.804973 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:02.804948 5019 generic.go:356] "Generic (PLEG): container finished" podID="cea1544c-25d6-4a37-9205-78fe8ece0a3a" containerID="db909bf92d9c65805ce5465d47b1d8694d374e99edf242ac9d73257abe1d5c66" exitCode=60 Oct 31 04:15:02.805099 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:02.805062 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890335-9txq4" event={"ID":"cea1544c-25d6-4a37-9205-78fe8ece0a3a","Type":"ContainerDied","Data":"db909bf92d9c65805ce5465d47b1d8694d374e99edf242ac9d73257abe1d5c66"} Oct 31 04:15:02.805788 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:02.805735 5019 scope.go:122] "RemoveContainer" containerID="db909bf92d9c65805ce5465d47b1d8694d374e99edf242ac9d73257abe1d5c66" Oct 31 04:15:02.807738 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:02.807648 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" event={"ID":"dccd15a9-47f5-4bd1-b21b-9d5ca8b05845","Type":"ContainerStarted","Data":"b995751e0675e5910d98110c0eaa2d9c11aff682da5a2617db7d80616503b78e"} Oct 31 04:15:02.810185 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:02.810149 5019 generic.go:356] "Generic (PLEG): container finished" podID="5706a564-0118-4cbf-805a-c5fcbcfee2d6" containerID="d2ff341bd6f63e2e578ab850ffb2f6377e7c657575a080456dd16d92912d5d3e" exitCode=60 Oct 31 04:15:02.810436 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:02.810409 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-bpznn" event={"ID":"5706a564-0118-4cbf-805a-c5fcbcfee2d6","Type":"ContainerDied","Data":"d2ff341bd6f63e2e578ab850ffb2f6377e7c657575a080456dd16d92912d5d3e"} Oct 31 04:15:02.810598 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:02.810577 5019 scope.go:122] "RemoveContainer" containerID="d2ff341bd6f63e2e578ab850ffb2f6377e7c657575a080456dd16d92912d5d3e" Oct 31 04:15:03.545380 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:03.545284 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 31 04:15:03.545380 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:03.545356 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx" podUID="cc1adc72-6548-4daf-9dc1-2b96c853cdbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 31 04:15:03.820605 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:03.820505 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" event={"ID":"dccd15a9-47f5-4bd1-b21b-9d5ca8b05845","Type":"ContainerStarted","Data":"3df64f167fe46cef28bd6b07f1ee9c922d82d33b5a7bed53ae1eae3350321d01"} Oct 31 04:15:03.823718 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:03.823682 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-bpznn" event={"ID":"5706a564-0118-4cbf-805a-c5fcbcfee2d6","Type":"ContainerStarted","Data":"6dcb41e26d4d05c7908708ae9b967ddbde04d8083d79cf49917ee53491561c4c"} Oct 31 04:15:03.826532 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:03.826500 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890335-l9tgn" event={"ID":"2db08bdd-9125-4b67-8434-10ab50702c61","Type":"ContainerStarted","Data":"7cd6f685f6940c94521635d933adc9263d530bbf449869d1a6a506e196f338b5"} Oct 31 04:15:03.828959 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:03.828928 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890335-9txq4" event={"ID":"cea1544c-25d6-4a37-9205-78fe8ece0a3a","Type":"ContainerStarted","Data":"0a39439f78d5810e4d70fd0b9b94543452c112dd24fe8e28a8dbd8cf34df6597"} Oct 31 04:15:03.866005 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:03.865513 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" podStartSLOduration=2.865494884 podStartE2EDuration="2.865494884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:15:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:15:03.863872405 +0000 UTC m=+658.620210448" watchObservedRunningTime="2026-10-31 04:15:03.865494884 +0000 UTC m=+658.621832924" Oct 31 04:15:04.264043 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:04.264002 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" Oct 31 04:15:04.545167 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:04.545118 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 31 04:15:04.545368 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:04.545194 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx" podUID="cc1adc72-6548-4daf-9dc1-2b96c853cdbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 31 04:15:04.839729 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:04.839633 5019 generic.go:356] "Generic (PLEG): container finished" podID="5706a564-0118-4cbf-805a-c5fcbcfee2d6" containerID="6dcb41e26d4d05c7908708ae9b967ddbde04d8083d79cf49917ee53491561c4c" exitCode=60 Oct 31 04:15:04.839729 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:04.839704 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-bpznn" event={"ID":"5706a564-0118-4cbf-805a-c5fcbcfee2d6","Type":"ContainerDied","Data":"6dcb41e26d4d05c7908708ae9b967ddbde04d8083d79cf49917ee53491561c4c"} Oct 31 04:15:04.840288 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:04.839778 5019 scope.go:122] "RemoveContainer" containerID="d2ff341bd6f63e2e578ab850ffb2f6377e7c657575a080456dd16d92912d5d3e" Oct 31 04:15:04.840288 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:04.840083 5019 scope.go:122] "RemoveContainer" containerID="6dcb41e26d4d05c7908708ae9b967ddbde04d8083d79cf49917ee53491561c4c" Oct 31 04:15:04.840427 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:15:04.840401 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890335-bpznn_odh-ai-gateway-infra(5706a564-0118-4cbf-805a-c5fcbcfee2d6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-bpznn" podUID="5706a564-0118-4cbf-805a-c5fcbcfee2d6" Oct 31 04:15:04.842772 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:04.842738 5019 generic.go:356] "Generic (PLEG): container finished" podID="2db08bdd-9125-4b67-8434-10ab50702c61" containerID="7cd6f685f6940c94521635d933adc9263d530bbf449869d1a6a506e196f338b5" exitCode=60 Oct 31 04:15:04.842886 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:04.842780 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890335-l9tgn" event={"ID":"2db08bdd-9125-4b67-8434-10ab50702c61","Type":"ContainerDied","Data":"7cd6f685f6940c94521635d933adc9263d530bbf449869d1a6a506e196f338b5"} Oct 31 04:15:04.843152 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:04.843133 5019 scope.go:122] "RemoveContainer" containerID="7cd6f685f6940c94521635d933adc9263d530bbf449869d1a6a506e196f338b5" Oct 31 04:15:04.843532 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:15:04.843508 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-4c2387-29890335-l9tgn_odh-ai-gateway-infra(2db08bdd-9125-4b67-8434-10ab50702c61)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890335-l9tgn" podUID="2db08bdd-9125-4b67-8434-10ab50702c61" Oct 31 04:15:04.845233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:04.845203 5019 generic.go:356] "Generic (PLEG): container finished" podID="cea1544c-25d6-4a37-9205-78fe8ece0a3a" containerID="0a39439f78d5810e4d70fd0b9b94543452c112dd24fe8e28a8dbd8cf34df6597" exitCode=60 Oct 31 04:15:04.846359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:04.846316 5019 scope.go:122] "RemoveContainer" containerID="0a39439f78d5810e4d70fd0b9b94543452c112dd24fe8e28a8dbd8cf34df6597" Oct 31 04:15:04.846624 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:15:04.846600 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-09a20a-29890335-9txq4_odh-ai-gateway-infra(cea1544c-25d6-4a37-9205-78fe8ece0a3a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890335-9txq4" podUID="cea1544c-25d6-4a37-9205-78fe8ece0a3a" Oct 31 04:15:04.846705 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:04.846656 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890335-9txq4" event={"ID":"cea1544c-25d6-4a37-9205-78fe8ece0a3a","Type":"ContainerDied","Data":"0a39439f78d5810e4d70fd0b9b94543452c112dd24fe8e28a8dbd8cf34df6597"} Oct 31 04:15:04.857954 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:04.857922 5019 scope.go:122] "RemoveContainer" containerID="1e037554bf2a6d4469bd87611de76ba1ce3e95e9eb8cb7ad86b33933569cabad" Oct 31 04:15:04.902392 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:04.893546 5019 scope.go:122] "RemoveContainer" containerID="db909bf92d9c65805ce5465d47b1d8694d374e99edf242ac9d73257abe1d5c66" Oct 31 04:15:05.545542 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:05.545513 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 31 04:15:05.545630 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:05.545581 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx" podUID="cc1adc72-6548-4daf-9dc1-2b96c853cdbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 31 04:15:05.855824 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:05.855727 5019 scope.go:122] "RemoveContainer" containerID="0a39439f78d5810e4d70fd0b9b94543452c112dd24fe8e28a8dbd8cf34df6597" Oct 31 04:15:05.856269 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:15:05.856088 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-09a20a-29890335-9txq4_odh-ai-gateway-infra(cea1544c-25d6-4a37-9205-78fe8ece0a3a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890335-9txq4" podUID="cea1544c-25d6-4a37-9205-78fe8ece0a3a" Oct 31 04:15:05.857776 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:05.857712 5019 scope.go:122] "RemoveContainer" containerID="6dcb41e26d4d05c7908708ae9b967ddbde04d8083d79cf49917ee53491561c4c" Oct 31 04:15:05.858017 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:15:05.857985 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890335-bpznn_odh-ai-gateway-infra(5706a564-0118-4cbf-805a-c5fcbcfee2d6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-bpznn" podUID="5706a564-0118-4cbf-805a-c5fcbcfee2d6" Oct 31 04:15:05.859614 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:05.859595 5019 scope.go:122] "RemoveContainer" containerID="7cd6f685f6940c94521635d933adc9263d530bbf449869d1a6a506e196f338b5" Oct 31 04:15:05.859834 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:15:05.859817 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-4c2387-29890335-l9tgn_odh-ai-gateway-infra(2db08bdd-9125-4b67-8434-10ab50702c61)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890335-l9tgn" podUID="2db08bdd-9125-4b67-8434-10ab50702c61" Oct 31 04:15:06.545705 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:06.545665 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 31 04:15:06.545938 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:06.545728 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx" podUID="cc1adc72-6548-4daf-9dc1-2b96c853cdbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 31 04:15:06.886388 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:06.886280 5019 generic.go:356] "Generic (PLEG): container finished" podID="c4e33d7c-168f-4001-8e39-22bea132574a" containerID="13130f4cbef0ca565dad709c9689dde75da0cc0e67e923aa4f059cdd2af27a83" exitCode=0 Oct 31 04:15:06.886987 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:06.886961 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v" event={"ID":"c4e33d7c-168f-4001-8e39-22bea132574a","Type":"ContainerDied","Data":"13130f4cbef0ca565dad709c9689dde75da0cc0e67e923aa4f059cdd2af27a83"} Oct 31 04:15:06.887598 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:06.887576 5019 scope.go:122] "RemoveContainer" containerID="13130f4cbef0ca565dad709c9689dde75da0cc0e67e923aa4f059cdd2af27a83" Oct 31 04:15:06.895903 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:06.895848 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:15:06.896161 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:06.896139 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:15:06.896297 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:06.896280 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:15:07.909791 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:07.909222 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v" event={"ID":"c4e33d7c-168f-4001-8e39-22bea132574a","Type":"ContainerStarted","Data":"1264bd0ec634bcf3505b1094dbd8ceec6d8fda0fdd152b41795dbc63e4c43b18"} Oct 31 04:15:07.935507 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:07.935468 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx" Oct 31 04:15:07.936213 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:07.936183 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx" Oct 31 04:15:07.937035 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:07.937016 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx" Oct 31 04:15:07.967877 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:07.967840 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v" Oct 31 04:15:08.718804 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:08.718770 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-439989-6b4f868455-v228h" Oct 31 04:15:08.969002 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:08.968915 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 31 04:15:08.969002 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:08.968970 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v" podUID="c4e33d7c-168f-4001-8e39-22bea132574a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 31 04:15:09.968229 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:09.968193 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 31 04:15:09.968620 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:09.968261 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v" podUID="c4e33d7c-168f-4001-8e39-22bea132574a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 31 04:15:10.408713 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:10.408629 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v" Oct 31 04:15:10.968766 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:10.968720 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 31 04:15:10.968997 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:10.968776 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v" podUID="c4e33d7c-168f-4001-8e39-22bea132574a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 31 04:15:11.989821 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:11.989772 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 31 04:15:11.990386 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:11.989843 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v" podUID="c4e33d7c-168f-4001-8e39-22bea132574a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 31 04:15:12.264080 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:12.263979 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" Oct 31 04:15:12.314403 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:12.314378 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" Oct 31 04:15:12.718918 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:12.718878 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc"] Oct 31 04:15:12.726766 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:12.726734 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" Oct 31 04:15:12.731503 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:12.731467 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f81681e-182b-4ed4-89bd-918370538f86-tls-certs\") pod \"simulator-w3-4c2387-kserve-64f4d58559-jjksc\" (UID: \"1f81681e-182b-4ed4-89bd-918370538f86\") " pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" Oct 31 04:15:12.731759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:12.731734 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f81681e-182b-4ed4-89bd-918370538f86-dshm\") pod \"simulator-w3-4c2387-kserve-64f4d58559-jjksc\" (UID: \"1f81681e-182b-4ed4-89bd-918370538f86\") " pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" Oct 31 04:15:12.731886 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:12.731788 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8swp\" (UniqueName: \"kubernetes.io/projected/1f81681e-182b-4ed4-89bd-918370538f86-kube-api-access-p8swp\") pod \"simulator-w3-4c2387-kserve-64f4d58559-jjksc\" (UID: \"1f81681e-182b-4ed4-89bd-918370538f86\") " pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" Oct 31 04:15:12.731886 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:12.731840 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f81681e-182b-4ed4-89bd-918370538f86-tmp-dir\") pod \"simulator-w3-4c2387-kserve-64f4d58559-jjksc\" (UID: \"1f81681e-182b-4ed4-89bd-918370538f86\") " pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" Oct 31 04:15:12.731999 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:12.731921 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f81681e-182b-4ed4-89bd-918370538f86-model-cache\") pod \"simulator-w3-4c2387-kserve-64f4d58559-jjksc\" (UID: \"1f81681e-182b-4ed4-89bd-918370538f86\") " pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" Oct 31 04:15:12.731999 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:12.731945 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f81681e-182b-4ed4-89bd-918370538f86-home\") pod \"simulator-w3-4c2387-kserve-64f4d58559-jjksc\" (UID: \"1f81681e-182b-4ed4-89bd-918370538f86\") " pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" Oct 31 04:15:12.741959 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:12.741922 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-4c2387\"/\"default-dockercfg-zfb5b\"" Oct 31 04:15:12.742140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:12.742118 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-4c2387\"/\"kube-root-ca.crt\"" Oct 31 04:15:12.742214 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:12.742200 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-4c2387\"/\"openshift-service-ca.crt\"" Oct 31 04:15:12.742305 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:12.742288 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-4c2387\"/\"simulator-w3-4c2387-kserve-self-signed-certs\"" Oct 31 04:15:12.743177 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:12.743148 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc"] Oct 31 04:15:12.814750 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:12.814717 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c4006d-548d7c6f4c-nj2nr" Oct 31 04:15:12.814929 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:12.814804 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-c4006d-7bd575dbdf-h26mj" Oct 31 04:15:12.833186 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:12.833148 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f81681e-182b-4ed4-89bd-918370538f86-model-cache\") pod \"simulator-w3-4c2387-kserve-64f4d58559-jjksc\" (UID: \"1f81681e-182b-4ed4-89bd-918370538f86\") " pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" Oct 31 04:15:12.833186 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:12.833183 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f81681e-182b-4ed4-89bd-918370538f86-home\") pod \"simulator-w3-4c2387-kserve-64f4d58559-jjksc\" (UID: \"1f81681e-182b-4ed4-89bd-918370538f86\") " pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" Oct 31 04:15:12.833469 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:12.833362 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f81681e-182b-4ed4-89bd-918370538f86-tls-certs\") pod \"simulator-w3-4c2387-kserve-64f4d58559-jjksc\" (UID: \"1f81681e-182b-4ed4-89bd-918370538f86\") " pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" Oct 31 04:15:12.833469 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:12.833432 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f81681e-182b-4ed4-89bd-918370538f86-dshm\") pod \"simulator-w3-4c2387-kserve-64f4d58559-jjksc\" (UID: \"1f81681e-182b-4ed4-89bd-918370538f86\") " pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" Oct 31 04:15:12.833586 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:12.833506 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8swp\" (UniqueName: \"kubernetes.io/projected/1f81681e-182b-4ed4-89bd-918370538f86-kube-api-access-p8swp\") pod \"simulator-w3-4c2387-kserve-64f4d58559-jjksc\" (UID: \"1f81681e-182b-4ed4-89bd-918370538f86\") " pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" Oct 31 04:15:12.833586 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:12.833562 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f81681e-182b-4ed4-89bd-918370538f86-tmp-dir\") pod \"simulator-w3-4c2387-kserve-64f4d58559-jjksc\" (UID: \"1f81681e-182b-4ed4-89bd-918370538f86\") " pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" Oct 31 04:15:12.833704 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:12.833658 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f81681e-182b-4ed4-89bd-918370538f86-model-cache\") pod \"simulator-w3-4c2387-kserve-64f4d58559-jjksc\" (UID: \"1f81681e-182b-4ed4-89bd-918370538f86\") " pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" Oct 31 04:15:12.833945 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:12.833915 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f81681e-182b-4ed4-89bd-918370538f86-home\") pod \"simulator-w3-4c2387-kserve-64f4d58559-jjksc\" (UID: \"1f81681e-182b-4ed4-89bd-918370538f86\") " pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" Oct 31 04:15:12.834148 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:12.834123 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f81681e-182b-4ed4-89bd-918370538f86-tmp-dir\") pod \"simulator-w3-4c2387-kserve-64f4d58559-jjksc\" (UID: \"1f81681e-182b-4ed4-89bd-918370538f86\") " pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" Oct 31 04:15:12.835843 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:12.835822 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f81681e-182b-4ed4-89bd-918370538f86-dshm\") pod \"simulator-w3-4c2387-kserve-64f4d58559-jjksc\" (UID: \"1f81681e-182b-4ed4-89bd-918370538f86\") " pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" Oct 31 04:15:12.836317 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:12.836292 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f81681e-182b-4ed4-89bd-918370538f86-tls-certs\") pod \"simulator-w3-4c2387-kserve-64f4d58559-jjksc\" (UID: \"1f81681e-182b-4ed4-89bd-918370538f86\") " pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" Oct 31 04:15:12.906356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:12.905433 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8swp\" (UniqueName: \"kubernetes.io/projected/1f81681e-182b-4ed4-89bd-918370538f86-kube-api-access-p8swp\") pod \"simulator-w3-4c2387-kserve-64f4d58559-jjksc\" (UID: \"1f81681e-182b-4ed4-89bd-918370538f86\") " pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" Oct 31 04:15:12.953168 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:12.953137 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" Oct 31 04:15:12.968123 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:12.968087 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 31 04:15:12.968302 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:12.968158 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v" podUID="c4e33d7c-168f-4001-8e39-22bea132574a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 31 04:15:13.044445 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:13.044372 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" Oct 31 04:15:13.357250 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:13.357211 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc"] Oct 31 04:15:13.358315 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:15:13.358284 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f81681e_182b_4ed4_89bd_918370538f86.slice/crio-0e770de1bad92768fa2a98f3eceb0a67762f75490c2e3244fc808963295a323c WatchSource:0}: Error finding container 0e770de1bad92768fa2a98f3eceb0a67762f75490c2e3244fc808963295a323c: Status 404 returned error can't find the container with id 0e770de1bad92768fa2a98f3eceb0a67762f75490c2e3244fc808963295a323c Oct 31 04:15:13.949674 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:13.949636 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" event={"ID":"1f81681e-182b-4ed4-89bd-918370538f86","Type":"ContainerStarted","Data":"e80ded11cb307253a5e723b5b6ec6c1b75e53ecd91dc9683b682c4b229d361b3"} Oct 31 04:15:13.949674 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:13.949682 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" event={"ID":"1f81681e-182b-4ed4-89bd-918370538f86","Type":"ContainerStarted","Data":"0e770de1bad92768fa2a98f3eceb0a67762f75490c2e3244fc808963295a323c"} Oct 31 04:15:13.973073 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:13.973044 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v" Oct 31 04:15:13.974205 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:13.974185 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v" Oct 31 04:15:13.992895 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:13.992841 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" podStartSLOduration=1.992825224 podStartE2EDuration="1.992825224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:15:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:15:13.989950256 +0000 UTC m=+668.746288318" watchObservedRunningTime="2026-10-31 04:15:13.992825224 +0000 UTC m=+668.749163263" Oct 31 04:15:14.044613 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:14.044570 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" Oct 31 04:15:16.620452 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:16.620370 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l"] Oct 31 04:15:16.624911 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:16.624882 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" Oct 31 04:15:16.644833 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:16.644791 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l"] Oct 31 04:15:16.651977 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:16.651941 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-965f61\"/\"kube-root-ca.crt\"" Oct 31 04:15:16.655472 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:16.655440 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-965f61\"/\"simulator-w0-965f61-kserve-self-signed-certs\"" Oct 31 04:15:16.655878 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:16.655489 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-965f61\"/\"openshift-service-ca.crt\"" Oct 31 04:15:16.655878 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:16.655506 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-965f61\"/\"default-dockercfg-gvlvp\"" Oct 31 04:15:16.680916 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:16.680878 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e2c7041-ad52-49f4-8b55-d16691f4938f-home\") pod \"simulator-w0-965f61-kserve-79d9bd9b-g7s8l\" (UID: \"8e2c7041-ad52-49f4-8b55-d16691f4938f\") " pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" Oct 31 04:15:16.681095 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:16.680940 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e2c7041-ad52-49f4-8b55-d16691f4938f-tmp-dir\") pod \"simulator-w0-965f61-kserve-79d9bd9b-g7s8l\" (UID: \"8e2c7041-ad52-49f4-8b55-d16691f4938f\") " pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" Oct 31 04:15:16.681095 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:16.681047 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e2c7041-ad52-49f4-8b55-d16691f4938f-model-cache\") pod \"simulator-w0-965f61-kserve-79d9bd9b-g7s8l\" (UID: \"8e2c7041-ad52-49f4-8b55-d16691f4938f\") " pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" Oct 31 04:15:16.681189 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:16.681119 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhqnr\" (UniqueName: \"kubernetes.io/projected/8e2c7041-ad52-49f4-8b55-d16691f4938f-kube-api-access-xhqnr\") pod \"simulator-w0-965f61-kserve-79d9bd9b-g7s8l\" (UID: \"8e2c7041-ad52-49f4-8b55-d16691f4938f\") " pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" Oct 31 04:15:16.681189 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:16.681157 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e2c7041-ad52-49f4-8b55-d16691f4938f-tls-certs\") pod \"simulator-w0-965f61-kserve-79d9bd9b-g7s8l\" (UID: \"8e2c7041-ad52-49f4-8b55-d16691f4938f\") " pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" Oct 31 04:15:16.681321 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:16.681304 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e2c7041-ad52-49f4-8b55-d16691f4938f-dshm\") pod \"simulator-w0-965f61-kserve-79d9bd9b-g7s8l\" (UID: \"8e2c7041-ad52-49f4-8b55-d16691f4938f\") " pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" Oct 31 04:15:16.781973 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:16.781940 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e2c7041-ad52-49f4-8b55-d16691f4938f-model-cache\") pod \"simulator-w0-965f61-kserve-79d9bd9b-g7s8l\" (UID: \"8e2c7041-ad52-49f4-8b55-d16691f4938f\") " pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" Oct 31 04:15:16.782154 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:16.781995 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhqnr\" (UniqueName: \"kubernetes.io/projected/8e2c7041-ad52-49f4-8b55-d16691f4938f-kube-api-access-xhqnr\") pod \"simulator-w0-965f61-kserve-79d9bd9b-g7s8l\" (UID: \"8e2c7041-ad52-49f4-8b55-d16691f4938f\") " pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" Oct 31 04:15:16.782154 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:16.782133 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e2c7041-ad52-49f4-8b55-d16691f4938f-tls-certs\") pod \"simulator-w0-965f61-kserve-79d9bd9b-g7s8l\" (UID: \"8e2c7041-ad52-49f4-8b55-d16691f4938f\") " pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" Oct 31 04:15:16.782282 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:16.782265 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e2c7041-ad52-49f4-8b55-d16691f4938f-dshm\") pod \"simulator-w0-965f61-kserve-79d9bd9b-g7s8l\" (UID: \"8e2c7041-ad52-49f4-8b55-d16691f4938f\") " pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" Oct 31 04:15:16.782405 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:16.782382 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e2c7041-ad52-49f4-8b55-d16691f4938f-home\") pod \"simulator-w0-965f61-kserve-79d9bd9b-g7s8l\" (UID: \"8e2c7041-ad52-49f4-8b55-d16691f4938f\") " pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" Oct 31 04:15:16.782481 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:16.782441 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e2c7041-ad52-49f4-8b55-d16691f4938f-tmp-dir\") pod \"simulator-w0-965f61-kserve-79d9bd9b-g7s8l\" (UID: \"8e2c7041-ad52-49f4-8b55-d16691f4938f\") " pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" Oct 31 04:15:16.782528 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:16.782469 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e2c7041-ad52-49f4-8b55-d16691f4938f-model-cache\") pod \"simulator-w0-965f61-kserve-79d9bd9b-g7s8l\" (UID: \"8e2c7041-ad52-49f4-8b55-d16691f4938f\") " pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" Oct 31 04:15:16.782787 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:16.782762 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e2c7041-ad52-49f4-8b55-d16691f4938f-tmp-dir\") pod \"simulator-w0-965f61-kserve-79d9bd9b-g7s8l\" (UID: \"8e2c7041-ad52-49f4-8b55-d16691f4938f\") " pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" Oct 31 04:15:16.782919 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:16.782867 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e2c7041-ad52-49f4-8b55-d16691f4938f-home\") pod \"simulator-w0-965f61-kserve-79d9bd9b-g7s8l\" (UID: \"8e2c7041-ad52-49f4-8b55-d16691f4938f\") " pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" Oct 31 04:15:16.784660 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:16.784630 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e2c7041-ad52-49f4-8b55-d16691f4938f-dshm\") pod \"simulator-w0-965f61-kserve-79d9bd9b-g7s8l\" (UID: \"8e2c7041-ad52-49f4-8b55-d16691f4938f\") " pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" Oct 31 04:15:16.784864 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:16.784846 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e2c7041-ad52-49f4-8b55-d16691f4938f-tls-certs\") pod \"simulator-w0-965f61-kserve-79d9bd9b-g7s8l\" (UID: \"8e2c7041-ad52-49f4-8b55-d16691f4938f\") " pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" Oct 31 04:15:16.805066 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:16.805034 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhqnr\" (UniqueName: \"kubernetes.io/projected/8e2c7041-ad52-49f4-8b55-d16691f4938f-kube-api-access-xhqnr\") pod \"simulator-w0-965f61-kserve-79d9bd9b-g7s8l\" (UID: \"8e2c7041-ad52-49f4-8b55-d16691f4938f\") " pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" Oct 31 04:15:16.940829 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:16.940734 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" Oct 31 04:15:17.253940 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:17.253908 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l"] Oct 31 04:15:17.256109 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:15:17.256077 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e2c7041_ad52_49f4_8b55_d16691f4938f.slice/crio-6dffef76b3aa82f52bccf874354bb0501f35a3b70770f13fc85aab92a3a2bd8a WatchSource:0}: Error finding container 6dffef76b3aa82f52bccf874354bb0501f35a3b70770f13fc85aab92a3a2bd8a: Status 404 returned error can't find the container with id 6dffef76b3aa82f52bccf874354bb0501f35a3b70770f13fc85aab92a3a2bd8a Oct 31 04:15:17.400568 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:17.400539 5019 scope.go:122] "RemoveContainer" containerID="6dcb41e26d4d05c7908708ae9b967ddbde04d8083d79cf49917ee53491561c4c" Oct 31 04:15:17.400765 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:17.400741 5019 scope.go:122] "RemoveContainer" containerID="0a39439f78d5810e4d70fd0b9b94543452c112dd24fe8e28a8dbd8cf34df6597" Oct 31 04:15:17.985963 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:17.985927 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-bpznn" event={"ID":"5706a564-0118-4cbf-805a-c5fcbcfee2d6","Type":"ContainerStarted","Data":"e58f83731bbac725a26f5b5fc12f804ffd82ac8fb19a631174b6dc0eeffc0ce4"} Oct 31 04:15:17.988077 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:17.988043 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" event={"ID":"8e2c7041-ad52-49f4-8b55-d16691f4938f","Type":"ContainerStarted","Data":"e19ff4563ad03d33b456af1368f7f44162cd7c33aada086c298c758c8f152283"} Oct 31 04:15:17.988228 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:17.988100 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" event={"ID":"8e2c7041-ad52-49f4-8b55-d16691f4938f","Type":"ContainerStarted","Data":"6dffef76b3aa82f52bccf874354bb0501f35a3b70770f13fc85aab92a3a2bd8a"} Oct 31 04:15:17.991201 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:17.991172 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890335-9txq4" event={"ID":"cea1544c-25d6-4a37-9205-78fe8ece0a3a","Type":"ContainerStarted","Data":"76e8aa68a6e14b9885287e65f4ae370e1ce70ef2e183fb060b330d58cca6ee73"} Oct 31 04:15:18.069502 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:18.069425 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" podStartSLOduration=2.069402174 podStartE2EDuration="2.069402174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:15:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:15:18.065115571 +0000 UTC m=+672.821453612" watchObservedRunningTime="2026-10-31 04:15:18.069402174 +0000 UTC m=+672.825740215" Oct 31 04:15:18.402729 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:18.401226 5019 scope.go:122] "RemoveContainer" containerID="7cd6f685f6940c94521635d933adc9263d530bbf449869d1a6a506e196f338b5" Oct 31 04:15:18.941021 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:18.940941 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" Oct 31 04:15:18.999853 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:18.999813 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890335-l9tgn" event={"ID":"2db08bdd-9125-4b67-8434-10ab50702c61","Type":"ContainerStarted","Data":"53343be4c378a425b591295c8602f8d6b7654eb941149dd29693a3d7729b60ca"} Oct 31 04:15:19.001916 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:19.001881 5019 generic.go:356] "Generic (PLEG): container finished" podID="cea1544c-25d6-4a37-9205-78fe8ece0a3a" containerID="76e8aa68a6e14b9885287e65f4ae370e1ce70ef2e183fb060b330d58cca6ee73" exitCode=60 Oct 31 04:15:19.002057 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:19.001940 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890335-9txq4" event={"ID":"cea1544c-25d6-4a37-9205-78fe8ece0a3a","Type":"ContainerDied","Data":"76e8aa68a6e14b9885287e65f4ae370e1ce70ef2e183fb060b330d58cca6ee73"} Oct 31 04:15:19.002057 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:19.001991 5019 scope.go:122] "RemoveContainer" containerID="0a39439f78d5810e4d70fd0b9b94543452c112dd24fe8e28a8dbd8cf34df6597" Oct 31 04:15:19.002388 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:19.002368 5019 scope.go:122] "RemoveContainer" containerID="76e8aa68a6e14b9885287e65f4ae370e1ce70ef2e183fb060b330d58cca6ee73" Oct 31 04:15:19.002665 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:15:19.002642 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-09a20a-29890335-9txq4_odh-ai-gateway-infra(cea1544c-25d6-4a37-9205-78fe8ece0a3a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890335-9txq4" podUID="cea1544c-25d6-4a37-9205-78fe8ece0a3a" Oct 31 04:15:19.004726 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:19.004706 5019 generic.go:356] "Generic (PLEG): container finished" podID="5706a564-0118-4cbf-805a-c5fcbcfee2d6" containerID="e58f83731bbac725a26f5b5fc12f804ffd82ac8fb19a631174b6dc0eeffc0ce4" exitCode=60 Oct 31 04:15:19.004829 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:19.004733 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-bpznn" event={"ID":"5706a564-0118-4cbf-805a-c5fcbcfee2d6","Type":"ContainerDied","Data":"e58f83731bbac725a26f5b5fc12f804ffd82ac8fb19a631174b6dc0eeffc0ce4"} Oct 31 04:15:19.005163 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:19.005145 5019 scope.go:122] "RemoveContainer" containerID="e58f83731bbac725a26f5b5fc12f804ffd82ac8fb19a631174b6dc0eeffc0ce4" Oct 31 04:15:19.005388 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:15:19.005365 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890335-bpznn_odh-ai-gateway-infra(5706a564-0118-4cbf-805a-c5fcbcfee2d6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-bpznn" podUID="5706a564-0118-4cbf-805a-c5fcbcfee2d6" Oct 31 04:15:19.014826 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:19.014805 5019 scope.go:122] "RemoveContainer" containerID="6dcb41e26d4d05c7908708ae9b967ddbde04d8083d79cf49917ee53491561c4c" Oct 31 04:15:19.038294 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:19.038271 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890335-bpznn"] Oct 31 04:15:19.120862 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:19.120832 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890335-9txq4"] Oct 31 04:15:20.012954 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:20.012913 5019 generic.go:356] "Generic (PLEG): container finished" podID="2db08bdd-9125-4b67-8434-10ab50702c61" containerID="53343be4c378a425b591295c8602f8d6b7654eb941149dd29693a3d7729b60ca" exitCode=60 Oct 31 04:15:20.012954 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:20.012948 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890335-l9tgn" event={"ID":"2db08bdd-9125-4b67-8434-10ab50702c61","Type":"ContainerDied","Data":"53343be4c378a425b591295c8602f8d6b7654eb941149dd29693a3d7729b60ca"} Oct 31 04:15:20.013531 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:20.013017 5019 scope.go:122] "RemoveContainer" containerID="7cd6f685f6940c94521635d933adc9263d530bbf449869d1a6a506e196f338b5" Oct 31 04:15:20.013531 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:20.013443 5019 scope.go:122] "RemoveContainer" containerID="53343be4c378a425b591295c8602f8d6b7654eb941149dd29693a3d7729b60ca" Oct 31 04:15:20.013786 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:15:20.013758 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-4c2387-29890335-l9tgn_odh-ai-gateway-infra(2db08bdd-9125-4b67-8434-10ab50702c61)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890335-l9tgn" podUID="2db08bdd-9125-4b67-8434-10ab50702c61" Oct 31 04:15:20.052074 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:20.052039 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890335-l9tgn"] Oct 31 04:15:20.202393 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:20.199646 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890335-9txq4" Oct 31 04:15:20.248309 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:20.248277 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cea1544c-25d6-4a37-9205-78fe8ece0a3a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cea1544c-25d6-4a37-9205-78fe8ece0a3a-maas-api-service-ca\") pod \"cea1544c-25d6-4a37-9205-78fe8ece0a3a\" (UID: \"cea1544c-25d6-4a37-9205-78fe8ece0a3a\") " Oct 31 04:15:20.251575 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:20.248458 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cea1544c-25d6-4a37-9205-78fe8ece0a3a-kube-api-access-wqh89\" (UniqueName: \"kubernetes.io/projected/cea1544c-25d6-4a37-9205-78fe8ece0a3a-kube-api-access-wqh89\") pod \"cea1544c-25d6-4a37-9205-78fe8ece0a3a\" (UID: \"cea1544c-25d6-4a37-9205-78fe8ece0a3a\") " Oct 31 04:15:20.251575 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:20.248601 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cea1544c-25d6-4a37-9205-78fe8ece0a3a-maas-api-service-ca" pod "cea1544c-25d6-4a37-9205-78fe8ece0a3a" (UID: "cea1544c-25d6-4a37-9205-78fe8ece0a3a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:15:20.251575 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:20.249027 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cea1544c-25d6-4a37-9205-78fe8ece0a3a-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:15:20.253744 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:20.253715 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cea1544c-25d6-4a37-9205-78fe8ece0a3a-kube-api-access-wqh89" pod "cea1544c-25d6-4a37-9205-78fe8ece0a3a" (UID: "cea1544c-25d6-4a37-9205-78fe8ece0a3a"). InnerVolumeSpecName "kube-api-access-wqh89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:15:20.254928 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:20.254909 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-bpznn" Oct 31 04:15:20.350430 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:20.350399 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5706a564-0118-4cbf-805a-c5fcbcfee2d6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5706a564-0118-4cbf-805a-c5fcbcfee2d6-maas-api-service-ca\") pod \"5706a564-0118-4cbf-805a-c5fcbcfee2d6\" (UID: \"5706a564-0118-4cbf-805a-c5fcbcfee2d6\") " Oct 31 04:15:20.350612 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:20.350470 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5706a564-0118-4cbf-805a-c5fcbcfee2d6-kube-api-access-7pg4b\" (UniqueName: \"kubernetes.io/projected/5706a564-0118-4cbf-805a-c5fcbcfee2d6-kube-api-access-7pg4b\") pod \"5706a564-0118-4cbf-805a-c5fcbcfee2d6\" (UID: \"5706a564-0118-4cbf-805a-c5fcbcfee2d6\") " Oct 31 04:15:20.350846 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:20.350818 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5706a564-0118-4cbf-805a-c5fcbcfee2d6-maas-api-service-ca" pod "5706a564-0118-4cbf-805a-c5fcbcfee2d6" (UID: "5706a564-0118-4cbf-805a-c5fcbcfee2d6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:15:20.350846 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:20.350840 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqh89\" (UniqueName: \"kubernetes.io/projected/cea1544c-25d6-4a37-9205-78fe8ece0a3a-kube-api-access-wqh89\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:15:20.352520 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:20.352496 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5706a564-0118-4cbf-805a-c5fcbcfee2d6-kube-api-access-7pg4b" pod "5706a564-0118-4cbf-805a-c5fcbcfee2d6" (UID: "5706a564-0118-4cbf-805a-c5fcbcfee2d6"). InnerVolumeSpecName "kube-api-access-7pg4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:15:20.452100 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:20.452056 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5706a564-0118-4cbf-805a-c5fcbcfee2d6-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:15:20.452100 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:20.452100 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pg4b\" (UniqueName: \"kubernetes.io/projected/5706a564-0118-4cbf-805a-c5fcbcfee2d6-kube-api-access-7pg4b\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:15:21.025089 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:21.025053 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-bpznn" Oct 31 04:15:21.025089 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:21.025081 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-bpznn" event={"ID":"5706a564-0118-4cbf-805a-c5fcbcfee2d6","Type":"ContainerDied","Data":"47ed60ec2595dfa8ebf9bb067483d63593e987c84fd065a28c091dd1ef844b03"} Oct 31 04:15:21.025661 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:21.025148 5019 scope.go:122] "RemoveContainer" containerID="e58f83731bbac725a26f5b5fc12f804ffd82ac8fb19a631174b6dc0eeffc0ce4" Oct 31 04:15:21.028722 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:21.028692 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890335-9txq4" event={"ID":"cea1544c-25d6-4a37-9205-78fe8ece0a3a","Type":"ContainerDied","Data":"cea44c4c84e5b184e70ca1e8e716c9ecc31b1ed77daf6c33427dcdb1a4b0f6d7"} Oct 31 04:15:21.028842 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:21.028706 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890335-9txq4" Oct 31 04:15:21.035630 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:21.035603 5019 scope.go:122] "RemoveContainer" containerID="76e8aa68a6e14b9885287e65f4ae370e1ce70ef2e183fb060b330d58cca6ee73" Oct 31 04:15:21.131643 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:21.131611 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890335-bpznn"] Oct 31 04:15:21.140401 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:21.140355 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890335-bpznn"] Oct 31 04:15:21.173265 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:21.173211 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890335-9txq4"] Oct 31 04:15:21.176627 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:21.176598 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890335-9txq4"] Oct 31 04:15:21.192469 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:21.192428 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890335-l9tgn" Oct 31 04:15:21.259980 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:21.259953 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2db08bdd-9125-4b67-8434-10ab50702c61-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2db08bdd-9125-4b67-8434-10ab50702c61-maas-api-service-ca\") pod \"2db08bdd-9125-4b67-8434-10ab50702c61\" (UID: \"2db08bdd-9125-4b67-8434-10ab50702c61\") " Oct 31 04:15:21.259980 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:21.259990 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2db08bdd-9125-4b67-8434-10ab50702c61-kube-api-access-2rff5\" (UniqueName: \"kubernetes.io/projected/2db08bdd-9125-4b67-8434-10ab50702c61-kube-api-access-2rff5\") pod \"2db08bdd-9125-4b67-8434-10ab50702c61\" (UID: \"2db08bdd-9125-4b67-8434-10ab50702c61\") " Oct 31 04:15:21.260467 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:21.260435 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2db08bdd-9125-4b67-8434-10ab50702c61-maas-api-service-ca" pod "2db08bdd-9125-4b67-8434-10ab50702c61" (UID: "2db08bdd-9125-4b67-8434-10ab50702c61"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:15:21.271532 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:21.271207 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2db08bdd-9125-4b67-8434-10ab50702c61-kube-api-access-2rff5" pod "2db08bdd-9125-4b67-8434-10ab50702c61" (UID: "2db08bdd-9125-4b67-8434-10ab50702c61"). InnerVolumeSpecName "kube-api-access-2rff5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:15:21.362187 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:21.362101 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2db08bdd-9125-4b67-8434-10ab50702c61-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:15:21.362187 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:21.362134 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rff5\" (UniqueName: \"kubernetes.io/projected/2db08bdd-9125-4b67-8434-10ab50702c61-kube-api-access-2rff5\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:15:21.408570 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:21.408535 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5706a564-0118-4cbf-805a-c5fcbcfee2d6" path="/var/lib/kubelet/pods/5706a564-0118-4cbf-805a-c5fcbcfee2d6/volumes" Oct 31 04:15:21.408959 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:21.408946 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cea1544c-25d6-4a37-9205-78fe8ece0a3a" path="/var/lib/kubelet/pods/cea1544c-25d6-4a37-9205-78fe8ece0a3a/volumes" Oct 31 04:15:22.037631 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:22.037588 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890335-l9tgn" Oct 31 04:15:22.038137 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:22.037598 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890335-l9tgn" event={"ID":"2db08bdd-9125-4b67-8434-10ab50702c61","Type":"ContainerDied","Data":"a8efe7749164d4b058e609e61c95880d0fab6afe66dca7bdab5f7ae954c0d08d"} Oct 31 04:15:22.038137 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:22.037758 5019 scope.go:122] "RemoveContainer" containerID="53343be4c378a425b591295c8602f8d6b7654eb941149dd29693a3d7729b60ca" Oct 31 04:15:22.089898 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:22.089853 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890335-l9tgn"] Oct 31 04:15:22.095923 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:22.095892 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890335-l9tgn"] Oct 31 04:15:22.392155 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:15:22.392069 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:15:23.044598 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:23.044569 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" Oct 31 04:15:23.062425 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:23.062394 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" Oct 31 04:15:23.408257 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:23.408171 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2db08bdd-9125-4b67-8434-10ab50702c61" path="/var/lib/kubelet/pods/2db08bdd-9125-4b67-8434-10ab50702c61/volumes" Oct 31 04:15:24.058424 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:24.058391 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" Oct 31 04:15:26.941597 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:26.941520 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" Oct 31 04:15:26.954734 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:26.954706 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" Oct 31 04:15:27.086849 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:27.086822 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" Oct 31 04:15:29.557704 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:29.557665 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz"] Oct 31 04:15:29.558208 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:29.558191 5019 cpu_manager.go:414] "containerMap: removing container" podUID="cea1544c-25d6-4a37-9205-78fe8ece0a3a" containerName="cleanup" Oct 31 04:15:29.558208 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:29.558205 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cea1544c-25d6-4a37-9205-78fe8ece0a3a" containerName="cleanup" Oct 31 04:15:29.558351 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:29.558217 5019 cpu_manager.go:414] "containerMap: removing container" podUID="5706a564-0118-4cbf-805a-c5fcbcfee2d6" containerName="cleanup" Oct 31 04:15:29.558351 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:29.558222 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5706a564-0118-4cbf-805a-c5fcbcfee2d6" containerName="cleanup" Oct 31 04:15:29.558351 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:29.558234 5019 cpu_manager.go:414] "containerMap: removing container" podUID="2db08bdd-9125-4b67-8434-10ab50702c61" containerName="cleanup" Oct 31 04:15:29.558351 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:29.558241 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2db08bdd-9125-4b67-8434-10ab50702c61" containerName="cleanup" Oct 31 04:15:29.558351 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:29.558248 5019 cpu_manager.go:414] "containerMap: removing container" podUID="2db08bdd-9125-4b67-8434-10ab50702c61" containerName="cleanup" Oct 31 04:15:29.558351 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:29.558253 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2db08bdd-9125-4b67-8434-10ab50702c61" containerName="cleanup" Oct 31 04:15:29.558351 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:29.558263 5019 cpu_manager.go:414] "containerMap: removing container" podUID="cea1544c-25d6-4a37-9205-78fe8ece0a3a" containerName="cleanup" Oct 31 04:15:29.558351 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:29.558267 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cea1544c-25d6-4a37-9205-78fe8ece0a3a" containerName="cleanup" Oct 31 04:15:29.558351 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:29.558275 5019 cpu_manager.go:414] "containerMap: removing container" podUID="2db08bdd-9125-4b67-8434-10ab50702c61" containerName="cleanup" Oct 31 04:15:29.558351 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:29.558279 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2db08bdd-9125-4b67-8434-10ab50702c61" containerName="cleanup" Oct 31 04:15:29.558351 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:29.558295 5019 cpu_manager.go:414] "containerMap: removing container" podUID="5706a564-0118-4cbf-805a-c5fcbcfee2d6" containerName="cleanup" Oct 31 04:15:29.558351 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:29.558300 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5706a564-0118-4cbf-805a-c5fcbcfee2d6" containerName="cleanup" Oct 31 04:15:29.558351 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:29.558310 5019 cpu_manager.go:414] "containerMap: removing container" podUID="cea1544c-25d6-4a37-9205-78fe8ece0a3a" containerName="cleanup" Oct 31 04:15:29.558351 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:29.558314 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cea1544c-25d6-4a37-9205-78fe8ece0a3a" containerName="cleanup" Oct 31 04:15:29.558351 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:29.558321 5019 cpu_manager.go:414] "containerMap: removing container" podUID="5706a564-0118-4cbf-805a-c5fcbcfee2d6" containerName="cleanup" Oct 31 04:15:29.558351 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:29.558343 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5706a564-0118-4cbf-805a-c5fcbcfee2d6" containerName="cleanup" Oct 31 04:15:29.559135 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:29.558429 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="2db08bdd-9125-4b67-8434-10ab50702c61" containerName="cleanup" Oct 31 04:15:29.559135 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:29.558441 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="5706a564-0118-4cbf-805a-c5fcbcfee2d6" containerName="cleanup" Oct 31 04:15:29.559135 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:29.558447 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="cea1544c-25d6-4a37-9205-78fe8ece0a3a" containerName="cleanup" Oct 31 04:15:29.559135 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:29.558454 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="5706a564-0118-4cbf-805a-c5fcbcfee2d6" containerName="cleanup" Oct 31 04:15:29.559135 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:29.558461 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="cea1544c-25d6-4a37-9205-78fe8ece0a3a" containerName="cleanup" Oct 31 04:15:29.559135 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:29.558466 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="5706a564-0118-4cbf-805a-c5fcbcfee2d6" containerName="cleanup" Oct 31 04:15:29.559135 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:29.558473 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="2db08bdd-9125-4b67-8434-10ab50702c61" containerName="cleanup" Oct 31 04:15:29.563385 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:29.563322 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" Oct 31 04:15:29.580877 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:29.580846 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-439989\"/\"openshift-service-ca.crt\"" Oct 31 04:15:29.581470 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:29.581449 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz"] Oct 31 04:15:29.590287 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:29.590260 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-439989\"/\"kube-root-ca.crt\"" Oct 31 04:15:29.593035 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:29.593009 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-439989\"/\"default-dockercfg-bxjc6\"" Oct 31 04:15:29.593172 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:29.593050 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-439989\"/\"simulator-w1-439989-kserve-self-signed-certs\"" Oct 31 04:15:29.649853 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:29.649810 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/379847f6-6418-43cd-ae5e-dad6fb408eb0-tls-certs\") pod \"simulator-w1-439989-kserve-75445c78dd-77sjz\" (UID: \"379847f6-6418-43cd-ae5e-dad6fb408eb0\") " pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" Oct 31 04:15:29.650064 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:29.649858 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/379847f6-6418-43cd-ae5e-dad6fb408eb0-tmp-dir\") pod \"simulator-w1-439989-kserve-75445c78dd-77sjz\" (UID: \"379847f6-6418-43cd-ae5e-dad6fb408eb0\") " pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" Oct 31 04:15:29.650064 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:29.649999 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/379847f6-6418-43cd-ae5e-dad6fb408eb0-home\") pod \"simulator-w1-439989-kserve-75445c78dd-77sjz\" (UID: \"379847f6-6418-43cd-ae5e-dad6fb408eb0\") " pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" Oct 31 04:15:29.650064 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:29.650039 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/379847f6-6418-43cd-ae5e-dad6fb408eb0-model-cache\") pod \"simulator-w1-439989-kserve-75445c78dd-77sjz\" (UID: \"379847f6-6418-43cd-ae5e-dad6fb408eb0\") " pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" Oct 31 04:15:29.650225 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:29.650145 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/379847f6-6418-43cd-ae5e-dad6fb408eb0-dshm\") pod \"simulator-w1-439989-kserve-75445c78dd-77sjz\" (UID: \"379847f6-6418-43cd-ae5e-dad6fb408eb0\") " pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" Oct 31 04:15:29.650258 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:29.650227 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fklmw\" (UniqueName: \"kubernetes.io/projected/379847f6-6418-43cd-ae5e-dad6fb408eb0-kube-api-access-fklmw\") pod \"simulator-w1-439989-kserve-75445c78dd-77sjz\" (UID: \"379847f6-6418-43cd-ae5e-dad6fb408eb0\") " pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" Oct 31 04:15:29.751735 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:29.751695 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/379847f6-6418-43cd-ae5e-dad6fb408eb0-tls-certs\") pod \"simulator-w1-439989-kserve-75445c78dd-77sjz\" (UID: \"379847f6-6418-43cd-ae5e-dad6fb408eb0\") " pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" Oct 31 04:15:29.751735 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:29.751733 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/379847f6-6418-43cd-ae5e-dad6fb408eb0-tmp-dir\") pod \"simulator-w1-439989-kserve-75445c78dd-77sjz\" (UID: \"379847f6-6418-43cd-ae5e-dad6fb408eb0\") " pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" Oct 31 04:15:29.751987 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:29.751879 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/379847f6-6418-43cd-ae5e-dad6fb408eb0-home\") pod \"simulator-w1-439989-kserve-75445c78dd-77sjz\" (UID: \"379847f6-6418-43cd-ae5e-dad6fb408eb0\") " pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" Oct 31 04:15:29.751987 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:29.751935 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/379847f6-6418-43cd-ae5e-dad6fb408eb0-model-cache\") pod \"simulator-w1-439989-kserve-75445c78dd-77sjz\" (UID: \"379847f6-6418-43cd-ae5e-dad6fb408eb0\") " pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" Oct 31 04:15:29.752099 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:29.751988 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/379847f6-6418-43cd-ae5e-dad6fb408eb0-dshm\") pod \"simulator-w1-439989-kserve-75445c78dd-77sjz\" (UID: \"379847f6-6418-43cd-ae5e-dad6fb408eb0\") " pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" Oct 31 04:15:29.752099 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:29.752060 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fklmw\" (UniqueName: \"kubernetes.io/projected/379847f6-6418-43cd-ae5e-dad6fb408eb0-kube-api-access-fklmw\") pod \"simulator-w1-439989-kserve-75445c78dd-77sjz\" (UID: \"379847f6-6418-43cd-ae5e-dad6fb408eb0\") " pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" Oct 31 04:15:29.752217 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:29.752196 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/379847f6-6418-43cd-ae5e-dad6fb408eb0-home\") pod \"simulator-w1-439989-kserve-75445c78dd-77sjz\" (UID: \"379847f6-6418-43cd-ae5e-dad6fb408eb0\") " pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" Oct 31 04:15:29.752307 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:29.752280 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/379847f6-6418-43cd-ae5e-dad6fb408eb0-tmp-dir\") pod \"simulator-w1-439989-kserve-75445c78dd-77sjz\" (UID: \"379847f6-6418-43cd-ae5e-dad6fb408eb0\") " pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" Oct 31 04:15:29.752380 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:29.752348 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/379847f6-6418-43cd-ae5e-dad6fb408eb0-model-cache\") pod \"simulator-w1-439989-kserve-75445c78dd-77sjz\" (UID: \"379847f6-6418-43cd-ae5e-dad6fb408eb0\") " pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" Oct 31 04:15:29.754181 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:29.754149 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/379847f6-6418-43cd-ae5e-dad6fb408eb0-dshm\") pod \"simulator-w1-439989-kserve-75445c78dd-77sjz\" (UID: \"379847f6-6418-43cd-ae5e-dad6fb408eb0\") " pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" Oct 31 04:15:29.754303 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:29.754285 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/379847f6-6418-43cd-ae5e-dad6fb408eb0-tls-certs\") pod \"simulator-w1-439989-kserve-75445c78dd-77sjz\" (UID: \"379847f6-6418-43cd-ae5e-dad6fb408eb0\") " pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" Oct 31 04:15:29.778528 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:29.778504 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fklmw\" (UniqueName: \"kubernetes.io/projected/379847f6-6418-43cd-ae5e-dad6fb408eb0-kube-api-access-fklmw\") pod \"simulator-w1-439989-kserve-75445c78dd-77sjz\" (UID: \"379847f6-6418-43cd-ae5e-dad6fb408eb0\") " pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" Oct 31 04:15:29.876376 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:29.876268 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" Oct 31 04:15:30.174542 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:30.174412 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz"] Oct 31 04:15:30.176759 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:15:30.176729 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod379847f6_6418_43cd_ae5e_dad6fb408eb0.slice/crio-7c1a9f3f22c988127940a3bbb7bacf0cf0519a4f1e951ceec968f462bfdb9d1c WatchSource:0}: Error finding container 7c1a9f3f22c988127940a3bbb7bacf0cf0519a4f1e951ceec968f462bfdb9d1c: Status 404 returned error can't find the container with id 7c1a9f3f22c988127940a3bbb7bacf0cf0519a4f1e951ceec968f462bfdb9d1c Oct 31 04:15:31.104689 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:31.104651 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" event={"ID":"379847f6-6418-43cd-ae5e-dad6fb408eb0","Type":"ContainerStarted","Data":"89ce61196420a55bb6c098348a2bab85a5be7f15a962f9020650e8734aac397f"} Oct 31 04:15:31.105191 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:31.104699 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" event={"ID":"379847f6-6418-43cd-ae5e-dad6fb408eb0","Type":"ContainerStarted","Data":"7c1a9f3f22c988127940a3bbb7bacf0cf0519a4f1e951ceec968f462bfdb9d1c"} Oct 31 04:15:31.146428 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:31.146373 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" podStartSLOduration=2.146358577 podStartE2EDuration="2.146358577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:15:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:15:31.144593001 +0000 UTC m=+685.900931042" watchObservedRunningTime="2026-10-31 04:15:31.146358577 +0000 UTC m=+685.902696614" Oct 31 04:15:31.876651 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:31.876619 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" Oct 31 04:15:39.876561 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:39.876519 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" Oct 31 04:15:39.890124 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:39.890088 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" Oct 31 04:15:40.180215 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:40.180116 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" Oct 31 04:15:46.496029 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:46.495988 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp"] Oct 31 04:15:46.497060 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:46.497036 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="2db08bdd-9125-4b67-8434-10ab50702c61" containerName="cleanup" Oct 31 04:15:46.497151 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:46.497074 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="cea1544c-25d6-4a37-9205-78fe8ece0a3a" containerName="cleanup" Oct 31 04:15:46.502404 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:46.502380 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" Oct 31 04:15:46.516744 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:46.516711 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp"] Oct 31 04:15:46.524943 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:46.524912 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-4c2387\"/\"premium-w3-4c2387-kserve-self-signed-certs\"" Oct 31 04:15:46.555114 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:46.555074 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/157a108f-36ab-44a7-bd7c-8131cb46c7e0-model-cache\") pod \"premium-w3-4c2387-kserve-5c59b67cbb-6trqp\" (UID: \"157a108f-36ab-44a7-bd7c-8131cb46c7e0\") " pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" Oct 31 04:15:46.555317 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:46.555130 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/157a108f-36ab-44a7-bd7c-8131cb46c7e0-tls-certs\") pod \"premium-w3-4c2387-kserve-5c59b67cbb-6trqp\" (UID: \"157a108f-36ab-44a7-bd7c-8131cb46c7e0\") " pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" Oct 31 04:15:46.555317 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:46.555212 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/157a108f-36ab-44a7-bd7c-8131cb46c7e0-home\") pod \"premium-w3-4c2387-kserve-5c59b67cbb-6trqp\" (UID: \"157a108f-36ab-44a7-bd7c-8131cb46c7e0\") " pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" Oct 31 04:15:46.555317 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:46.555296 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/157a108f-36ab-44a7-bd7c-8131cb46c7e0-tmp-dir\") pod \"premium-w3-4c2387-kserve-5c59b67cbb-6trqp\" (UID: \"157a108f-36ab-44a7-bd7c-8131cb46c7e0\") " pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" Oct 31 04:15:46.555483 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:46.555431 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/157a108f-36ab-44a7-bd7c-8131cb46c7e0-dshm\") pod \"premium-w3-4c2387-kserve-5c59b67cbb-6trqp\" (UID: \"157a108f-36ab-44a7-bd7c-8131cb46c7e0\") " pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" Oct 31 04:15:46.555544 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:46.555526 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb5r7\" (UniqueName: \"kubernetes.io/projected/157a108f-36ab-44a7-bd7c-8131cb46c7e0-kube-api-access-rb5r7\") pod \"premium-w3-4c2387-kserve-5c59b67cbb-6trqp\" (UID: \"157a108f-36ab-44a7-bd7c-8131cb46c7e0\") " pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" Oct 31 04:15:46.656869 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:46.656821 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/157a108f-36ab-44a7-bd7c-8131cb46c7e0-home\") pod \"premium-w3-4c2387-kserve-5c59b67cbb-6trqp\" (UID: \"157a108f-36ab-44a7-bd7c-8131cb46c7e0\") " pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" Oct 31 04:15:46.657072 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:46.656880 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/157a108f-36ab-44a7-bd7c-8131cb46c7e0-tmp-dir\") pod \"premium-w3-4c2387-kserve-5c59b67cbb-6trqp\" (UID: \"157a108f-36ab-44a7-bd7c-8131cb46c7e0\") " pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" Oct 31 04:15:46.657072 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:46.656927 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/157a108f-36ab-44a7-bd7c-8131cb46c7e0-dshm\") pod \"premium-w3-4c2387-kserve-5c59b67cbb-6trqp\" (UID: \"157a108f-36ab-44a7-bd7c-8131cb46c7e0\") " pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" Oct 31 04:15:46.657072 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:46.656971 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb5r7\" (UniqueName: \"kubernetes.io/projected/157a108f-36ab-44a7-bd7c-8131cb46c7e0-kube-api-access-rb5r7\") pod \"premium-w3-4c2387-kserve-5c59b67cbb-6trqp\" (UID: \"157a108f-36ab-44a7-bd7c-8131cb46c7e0\") " pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" Oct 31 04:15:46.657072 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:46.657016 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/157a108f-36ab-44a7-bd7c-8131cb46c7e0-model-cache\") pod \"premium-w3-4c2387-kserve-5c59b67cbb-6trqp\" (UID: \"157a108f-36ab-44a7-bd7c-8131cb46c7e0\") " pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" Oct 31 04:15:46.657072 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:46.657047 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/157a108f-36ab-44a7-bd7c-8131cb46c7e0-tls-certs\") pod \"premium-w3-4c2387-kserve-5c59b67cbb-6trqp\" (UID: \"157a108f-36ab-44a7-bd7c-8131cb46c7e0\") " pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" Oct 31 04:15:46.657362 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:46.657244 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/157a108f-36ab-44a7-bd7c-8131cb46c7e0-home\") pod \"premium-w3-4c2387-kserve-5c59b67cbb-6trqp\" (UID: \"157a108f-36ab-44a7-bd7c-8131cb46c7e0\") " pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" Oct 31 04:15:46.657362 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:46.657305 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/157a108f-36ab-44a7-bd7c-8131cb46c7e0-tmp-dir\") pod \"premium-w3-4c2387-kserve-5c59b67cbb-6trqp\" (UID: \"157a108f-36ab-44a7-bd7c-8131cb46c7e0\") " pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" Oct 31 04:15:46.657461 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:46.657419 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/157a108f-36ab-44a7-bd7c-8131cb46c7e0-model-cache\") pod \"premium-w3-4c2387-kserve-5c59b67cbb-6trqp\" (UID: \"157a108f-36ab-44a7-bd7c-8131cb46c7e0\") " pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" Oct 31 04:15:46.659353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:46.659304 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/157a108f-36ab-44a7-bd7c-8131cb46c7e0-dshm\") pod \"premium-w3-4c2387-kserve-5c59b67cbb-6trqp\" (UID: \"157a108f-36ab-44a7-bd7c-8131cb46c7e0\") " pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" Oct 31 04:15:46.659562 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:46.659546 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/157a108f-36ab-44a7-bd7c-8131cb46c7e0-tls-certs\") pod \"premium-w3-4c2387-kserve-5c59b67cbb-6trqp\" (UID: \"157a108f-36ab-44a7-bd7c-8131cb46c7e0\") " pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" Oct 31 04:15:46.684756 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:46.684684 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb5r7\" (UniqueName: \"kubernetes.io/projected/157a108f-36ab-44a7-bd7c-8131cb46c7e0-kube-api-access-rb5r7\") pod \"premium-w3-4c2387-kserve-5c59b67cbb-6trqp\" (UID: \"157a108f-36ab-44a7-bd7c-8131cb46c7e0\") " pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" Oct 31 04:15:46.817046 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:46.816943 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" Oct 31 04:15:47.122926 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:47.122891 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp"] Oct 31 04:15:47.124852 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:15:47.124820 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod157a108f_36ab_44a7_bd7c_8131cb46c7e0.slice/crio-93063657b647d768fdfe399c1413a4c6bb1e4542525d9cebfa1b5ff47f5edb2a WatchSource:0}: Error finding container 93063657b647d768fdfe399c1413a4c6bb1e4542525d9cebfa1b5ff47f5edb2a: Status 404 returned error can't find the container with id 93063657b647d768fdfe399c1413a4c6bb1e4542525d9cebfa1b5ff47f5edb2a Oct 31 04:15:47.217438 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:47.217396 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" event={"ID":"157a108f-36ab-44a7-bd7c-8131cb46c7e0","Type":"ContainerStarted","Data":"f0acbc3f06de2463824292c21ced776771ddc11152124a00db1b005a372a6eaf"} Oct 31 04:15:47.217642 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:47.217450 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" event={"ID":"157a108f-36ab-44a7-bd7c-8131cb46c7e0","Type":"ContainerStarted","Data":"93063657b647d768fdfe399c1413a4c6bb1e4542525d9cebfa1b5ff47f5edb2a"} Oct 31 04:15:47.262122 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:47.262063 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" podStartSLOduration=1.2620475469999999 podStartE2EDuration="1.262047547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:15:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:15:47.259568264 +0000 UTC m=+702.015906305" watchObservedRunningTime="2026-10-31 04:15:47.262047547 +0000 UTC m=+702.018385586" Oct 31 04:15:47.818006 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:47.817954 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" Oct 31 04:15:52.392910 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:15:52.392862 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:15:56.817298 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:56.817252 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" Oct 31 04:15:56.830482 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:56.830449 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" Oct 31 04:15:57.295785 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:15:57.295754 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" Oct 31 04:16:06.749766 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:06.749720 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b"] Oct 31 04:16:06.756446 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:06.756411 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" Oct 31 04:16:06.776380 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:06.776307 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b"] Oct 31 04:16:06.776834 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:06.776802 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c4006d\"/\"kube-root-ca.crt\"" Oct 31 04:16:06.777006 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:06.776978 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c4006d\"/\"openshift-service-ca.crt\"" Oct 31 04:16:06.783636 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:06.783602 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c4006d\"/\"premium-w4-c4006d-kserve-self-signed-certs\"" Oct 31 04:16:06.783636 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:06.783602 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c4006d\"/\"default-dockercfg-kj674\"" Oct 31 04:16:06.810254 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:06.810210 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/643433d8-418e-4102-8bc3-cd544932614a-dshm\") pod \"premium-w4-c4006d-kserve-59d5c64c77-5dv8b\" (UID: \"643433d8-418e-4102-8bc3-cd544932614a\") " pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" Oct 31 04:16:06.810481 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:06.810283 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/643433d8-418e-4102-8bc3-cd544932614a-tls-certs\") pod \"premium-w4-c4006d-kserve-59d5c64c77-5dv8b\" (UID: \"643433d8-418e-4102-8bc3-cd544932614a\") " pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" Oct 31 04:16:06.810551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:06.810492 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/643433d8-418e-4102-8bc3-cd544932614a-home\") pod \"premium-w4-c4006d-kserve-59d5c64c77-5dv8b\" (UID: \"643433d8-418e-4102-8bc3-cd544932614a\") " pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" Oct 31 04:16:06.810551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:06.810540 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8z8k\" (UniqueName: \"kubernetes.io/projected/643433d8-418e-4102-8bc3-cd544932614a-kube-api-access-j8z8k\") pod \"premium-w4-c4006d-kserve-59d5c64c77-5dv8b\" (UID: \"643433d8-418e-4102-8bc3-cd544932614a\") " pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" Oct 31 04:16:06.810661 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:06.810602 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/643433d8-418e-4102-8bc3-cd544932614a-tmp-dir\") pod \"premium-w4-c4006d-kserve-59d5c64c77-5dv8b\" (UID: \"643433d8-418e-4102-8bc3-cd544932614a\") " pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" Oct 31 04:16:06.810714 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:06.810662 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/643433d8-418e-4102-8bc3-cd544932614a-model-cache\") pod \"premium-w4-c4006d-kserve-59d5c64c77-5dv8b\" (UID: \"643433d8-418e-4102-8bc3-cd544932614a\") " pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" Oct 31 04:16:06.914120 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:06.914062 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/643433d8-418e-4102-8bc3-cd544932614a-tls-certs\") pod \"premium-w4-c4006d-kserve-59d5c64c77-5dv8b\" (UID: \"643433d8-418e-4102-8bc3-cd544932614a\") " pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" Oct 31 04:16:06.914491 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:06.914467 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/643433d8-418e-4102-8bc3-cd544932614a-home\") pod \"premium-w4-c4006d-kserve-59d5c64c77-5dv8b\" (UID: \"643433d8-418e-4102-8bc3-cd544932614a\") " pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" Oct 31 04:16:06.914621 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:06.914607 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8z8k\" (UniqueName: \"kubernetes.io/projected/643433d8-418e-4102-8bc3-cd544932614a-kube-api-access-j8z8k\") pod \"premium-w4-c4006d-kserve-59d5c64c77-5dv8b\" (UID: \"643433d8-418e-4102-8bc3-cd544932614a\") " pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" Oct 31 04:16:06.914759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:06.914746 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/643433d8-418e-4102-8bc3-cd544932614a-tmp-dir\") pod \"premium-w4-c4006d-kserve-59d5c64c77-5dv8b\" (UID: \"643433d8-418e-4102-8bc3-cd544932614a\") " pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" Oct 31 04:16:06.914893 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:06.914880 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/643433d8-418e-4102-8bc3-cd544932614a-model-cache\") pod \"premium-w4-c4006d-kserve-59d5c64c77-5dv8b\" (UID: \"643433d8-418e-4102-8bc3-cd544932614a\") " pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" Oct 31 04:16:06.915078 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:06.915065 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/643433d8-418e-4102-8bc3-cd544932614a-dshm\") pod \"premium-w4-c4006d-kserve-59d5c64c77-5dv8b\" (UID: \"643433d8-418e-4102-8bc3-cd544932614a\") " pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" Oct 31 04:16:06.919353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:06.915550 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/643433d8-418e-4102-8bc3-cd544932614a-home\") pod \"premium-w4-c4006d-kserve-59d5c64c77-5dv8b\" (UID: \"643433d8-418e-4102-8bc3-cd544932614a\") " pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" Oct 31 04:16:06.919353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:06.916297 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/643433d8-418e-4102-8bc3-cd544932614a-tmp-dir\") pod \"premium-w4-c4006d-kserve-59d5c64c77-5dv8b\" (UID: \"643433d8-418e-4102-8bc3-cd544932614a\") " pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" Oct 31 04:16:06.919353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:06.916568 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/643433d8-418e-4102-8bc3-cd544932614a-model-cache\") pod \"premium-w4-c4006d-kserve-59d5c64c77-5dv8b\" (UID: \"643433d8-418e-4102-8bc3-cd544932614a\") " pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" Oct 31 04:16:06.922937 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:06.922889 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/643433d8-418e-4102-8bc3-cd544932614a-dshm\") pod \"premium-w4-c4006d-kserve-59d5c64c77-5dv8b\" (UID: \"643433d8-418e-4102-8bc3-cd544932614a\") " pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" Oct 31 04:16:06.944518 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:06.944481 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8z8k\" (UniqueName: \"kubernetes.io/projected/643433d8-418e-4102-8bc3-cd544932614a-kube-api-access-j8z8k\") pod \"premium-w4-c4006d-kserve-59d5c64c77-5dv8b\" (UID: \"643433d8-418e-4102-8bc3-cd544932614a\") " pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" Oct 31 04:16:06.948920 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:06.948886 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/643433d8-418e-4102-8bc3-cd544932614a-tls-certs\") pod \"premium-w4-c4006d-kserve-59d5c64c77-5dv8b\" (UID: \"643433d8-418e-4102-8bc3-cd544932614a\") " pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" Oct 31 04:16:07.073773 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:07.073640 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" Oct 31 04:16:07.474675 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:07.474632 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b"] Oct 31 04:16:07.475301 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:16:07.475257 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod643433d8_418e_4102_8bc3_cd544932614a.slice/crio-8c82e0c993d14b9f29c658408067ee1379f9b217a5cb62374cda43eddfaef55d WatchSource:0}: Error finding container 8c82e0c993d14b9f29c658408067ee1379f9b217a5cb62374cda43eddfaef55d: Status 404 returned error can't find the container with id 8c82e0c993d14b9f29c658408067ee1379f9b217a5cb62374cda43eddfaef55d Oct 31 04:16:08.372032 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:08.370946 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" event={"ID":"643433d8-418e-4102-8bc3-cd544932614a","Type":"ContainerStarted","Data":"065bb5eff6b7e35473610efb70b6d9984ca6d35a9b639ceb8ea3ac7691259af1"} Oct 31 04:16:08.372032 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:08.371014 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" event={"ID":"643433d8-418e-4102-8bc3-cd544932614a","Type":"ContainerStarted","Data":"8c82e0c993d14b9f29c658408067ee1379f9b217a5cb62374cda43eddfaef55d"} Oct 31 04:16:08.415044 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:08.414981 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" podStartSLOduration=2.4149655389999998 podStartE2EDuration="2.414965539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:16:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:16:08.413409853 +0000 UTC m=+723.169747894" watchObservedRunningTime="2026-10-31 04:16:08.414965539 +0000 UTC m=+723.171303579" Oct 31 04:16:09.074889 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:09.074847 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" Oct 31 04:16:11.882076 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:11.882034 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b9ca02f-kk972"] Oct 31 04:16:11.886251 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:11.886222 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b9ca02f-kk972" Oct 31 04:16:11.901968 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:11.901931 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:16:11.902251 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:11.901969 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:16:11.902251 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:11.902163 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:16:11.914316 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:11.914273 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:16:11.917298 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:11.917262 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b9ca02f-kk972"] Oct 31 04:16:11.970041 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:11.969997 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpswh\" (UniqueName: \"kubernetes.io/projected/4c966c31-c0b5-4a53-83a0-2dafa8418e0f-kube-api-access-tpswh\") pod \"maas-api-revoke-sub-7b9ca02f-kk972\" (UID: \"4c966c31-c0b5-4a53-83a0-2dafa8418e0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b9ca02f-kk972" Oct 31 04:16:11.970041 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:11.970044 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c966c31-c0b5-4a53-83a0-2dafa8418e0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b9ca02f-kk972\" (UID: \"4c966c31-c0b5-4a53-83a0-2dafa8418e0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b9ca02f-kk972" Oct 31 04:16:11.970279 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:11.970163 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c966c31-c0b5-4a53-83a0-2dafa8418e0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b9ca02f-kk972\" (UID: \"4c966c31-c0b5-4a53-83a0-2dafa8418e0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b9ca02f-kk972" Oct 31 04:16:12.071799 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:12.071761 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpswh\" (UniqueName: \"kubernetes.io/projected/4c966c31-c0b5-4a53-83a0-2dafa8418e0f-kube-api-access-tpswh\") pod \"maas-api-revoke-sub-7b9ca02f-kk972\" (UID: \"4c966c31-c0b5-4a53-83a0-2dafa8418e0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b9ca02f-kk972" Oct 31 04:16:12.072022 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:12.071816 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c966c31-c0b5-4a53-83a0-2dafa8418e0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b9ca02f-kk972\" (UID: \"4c966c31-c0b5-4a53-83a0-2dafa8418e0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b9ca02f-kk972" Oct 31 04:16:12.072022 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:12.071903 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c966c31-c0b5-4a53-83a0-2dafa8418e0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b9ca02f-kk972\" (UID: \"4c966c31-c0b5-4a53-83a0-2dafa8418e0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b9ca02f-kk972" Oct 31 04:16:12.072953 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:12.072918 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c966c31-c0b5-4a53-83a0-2dafa8418e0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b9ca02f-kk972\" (UID: \"4c966c31-c0b5-4a53-83a0-2dafa8418e0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b9ca02f-kk972" Oct 31 04:16:12.075306 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:12.075276 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c966c31-c0b5-4a53-83a0-2dafa8418e0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b9ca02f-kk972\" (UID: \"4c966c31-c0b5-4a53-83a0-2dafa8418e0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b9ca02f-kk972" Oct 31 04:16:12.095629 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:12.095595 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpswh\" (UniqueName: \"kubernetes.io/projected/4c966c31-c0b5-4a53-83a0-2dafa8418e0f-kube-api-access-tpswh\") pod \"maas-api-revoke-sub-7b9ca02f-kk972\" (UID: \"4c966c31-c0b5-4a53-83a0-2dafa8418e0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b9ca02f-kk972" Oct 31 04:16:12.202243 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:12.202138 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b9ca02f-kk972" Oct 31 04:16:12.512357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:12.512311 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b9ca02f-kk972"] Oct 31 04:16:12.514322 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:16:12.514288 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c966c31_c0b5_4a53_83a0_2dafa8418e0f.slice/crio-022441349d8addcf175d04ddfc6ff95e76753482821c5573f07077e937dff24f WatchSource:0}: Error finding container 022441349d8addcf175d04ddfc6ff95e76753482821c5573f07077e937dff24f: Status 404 returned error can't find the container with id 022441349d8addcf175d04ddfc6ff95e76753482821c5573f07077e937dff24f Oct 31 04:16:13.411443 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:13.411400 5019 generic.go:356] "Generic (PLEG): container finished" podID="4c966c31-c0b5-4a53-83a0-2dafa8418e0f" containerID="082ddd2fe391efa043ef6abe7290c08ec836f283986d61b1d9ee0298936b2630" exitCode=0 Oct 31 04:16:13.411892 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:13.411487 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b9ca02f-kk972" event={"ID":"4c966c31-c0b5-4a53-83a0-2dafa8418e0f","Type":"ContainerDied","Data":"082ddd2fe391efa043ef6abe7290c08ec836f283986d61b1d9ee0298936b2630"} Oct 31 04:16:13.411892 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:13.411550 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b9ca02f-kk972" event={"ID":"4c966c31-c0b5-4a53-83a0-2dafa8418e0f","Type":"ContainerStarted","Data":"022441349d8addcf175d04ddfc6ff95e76753482821c5573f07077e937dff24f"} Oct 31 04:16:14.584789 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:14.584759 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b9ca02f-kk972" Oct 31 04:16:14.600311 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:14.600285 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c966c31-c0b5-4a53-83a0-2dafa8418e0f-kube-api-access-tpswh\" (UniqueName: \"kubernetes.io/projected/4c966c31-c0b5-4a53-83a0-2dafa8418e0f-kube-api-access-tpswh\") pod \"4c966c31-c0b5-4a53-83a0-2dafa8418e0f\" (UID: \"4c966c31-c0b5-4a53-83a0-2dafa8418e0f\") " Oct 31 04:16:14.600500 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:14.600382 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c966c31-c0b5-4a53-83a0-2dafa8418e0f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c966c31-c0b5-4a53-83a0-2dafa8418e0f-maas-api-xks-ca\") pod \"4c966c31-c0b5-4a53-83a0-2dafa8418e0f\" (UID: \"4c966c31-c0b5-4a53-83a0-2dafa8418e0f\") " Oct 31 04:16:14.600500 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:14.600473 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c966c31-c0b5-4a53-83a0-2dafa8418e0f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c966c31-c0b5-4a53-83a0-2dafa8418e0f-maas-api-openshift-ca\") pod \"4c966c31-c0b5-4a53-83a0-2dafa8418e0f\" (UID: \"4c966c31-c0b5-4a53-83a0-2dafa8418e0f\") " Oct 31 04:16:14.600861 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:14.600833 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c966c31-c0b5-4a53-83a0-2dafa8418e0f-maas-api-openshift-ca" pod "4c966c31-c0b5-4a53-83a0-2dafa8418e0f" (UID: "4c966c31-c0b5-4a53-83a0-2dafa8418e0f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:16:14.602603 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:14.602571 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c966c31-c0b5-4a53-83a0-2dafa8418e0f-maas-api-xks-ca" pod "4c966c31-c0b5-4a53-83a0-2dafa8418e0f" (UID: "4c966c31-c0b5-4a53-83a0-2dafa8418e0f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:16:14.602726 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:14.602612 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c966c31-c0b5-4a53-83a0-2dafa8418e0f-kube-api-access-tpswh" pod "4c966c31-c0b5-4a53-83a0-2dafa8418e0f" (UID: "4c966c31-c0b5-4a53-83a0-2dafa8418e0f"). InnerVolumeSpecName "kube-api-access-tpswh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:16:14.702416 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:14.702298 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpswh\" (UniqueName: \"kubernetes.io/projected/4c966c31-c0b5-4a53-83a0-2dafa8418e0f-kube-api-access-tpswh\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:16:14.702416 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:14.702355 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c966c31-c0b5-4a53-83a0-2dafa8418e0f-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:16:14.702416 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:14.702368 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c966c31-c0b5-4a53-83a0-2dafa8418e0f-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:16:15.427841 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:15.427808 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b9ca02f-kk972" Oct 31 04:16:15.427841 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:15.427827 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b9ca02f-kk972" event={"ID":"4c966c31-c0b5-4a53-83a0-2dafa8418e0f","Type":"ContainerDied","Data":"022441349d8addcf175d04ddfc6ff95e76753482821c5573f07077e937dff24f"} Oct 31 04:16:15.428061 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:15.427881 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="022441349d8addcf175d04ddfc6ff95e76753482821c5573f07077e937dff24f" Oct 31 04:16:17.076318 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:17.076283 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" Oct 31 04:16:17.091725 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:17.091688 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" Oct 31 04:16:17.455179 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:17.455091 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" Oct 31 04:16:22.392108 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:16:22.392067 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:16:27.184525 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:27.184460 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e700e24-547r4"] Oct 31 04:16:27.185104 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:27.185079 5019 cpu_manager.go:414] "containerMap: removing container" podUID="4c966c31-c0b5-4a53-83a0-2dafa8418e0f" containerName="revoke-keys" Oct 31 04:16:27.185104 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:27.185102 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c966c31-c0b5-4a53-83a0-2dafa8418e0f" containerName="revoke-keys" Oct 31 04:16:27.185313 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:27.185243 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c966c31-c0b5-4a53-83a0-2dafa8418e0f" containerName="revoke-keys" Oct 31 04:16:27.189197 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:27.189168 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e700e24-547r4" Oct 31 04:16:27.210123 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:27.210083 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e700e24-547r4"] Oct 31 04:16:27.216181 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:27.216146 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:16:27.216561 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:27.216543 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:16:27.216946 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:27.216921 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:16:27.217063 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:27.217049 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:16:27.283560 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:27.283509 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9ms5\" (UniqueName: \"kubernetes.io/projected/cdd06518-6e8d-4d39-af45-f70ace8e9ab7-kube-api-access-p9ms5\") pod \"maas-api-revoke-sub-6e700e24-547r4\" (UID: \"cdd06518-6e8d-4d39-af45-f70ace8e9ab7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e700e24-547r4" Oct 31 04:16:27.283774 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:27.283680 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdd06518-6e8d-4d39-af45-f70ace8e9ab7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e700e24-547r4\" (UID: \"cdd06518-6e8d-4d39-af45-f70ace8e9ab7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e700e24-547r4" Oct 31 04:16:27.283774 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:27.283734 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdd06518-6e8d-4d39-af45-f70ace8e9ab7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e700e24-547r4\" (UID: \"cdd06518-6e8d-4d39-af45-f70ace8e9ab7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e700e24-547r4" Oct 31 04:16:27.385619 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:27.385571 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9ms5\" (UniqueName: \"kubernetes.io/projected/cdd06518-6e8d-4d39-af45-f70ace8e9ab7-kube-api-access-p9ms5\") pod \"maas-api-revoke-sub-6e700e24-547r4\" (UID: \"cdd06518-6e8d-4d39-af45-f70ace8e9ab7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e700e24-547r4" Oct 31 04:16:27.385839 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:27.385697 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdd06518-6e8d-4d39-af45-f70ace8e9ab7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e700e24-547r4\" (UID: \"cdd06518-6e8d-4d39-af45-f70ace8e9ab7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e700e24-547r4" Oct 31 04:16:27.385839 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:27.385757 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdd06518-6e8d-4d39-af45-f70ace8e9ab7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e700e24-547r4\" (UID: \"cdd06518-6e8d-4d39-af45-f70ace8e9ab7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e700e24-547r4" Oct 31 04:16:27.386467 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:27.386426 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdd06518-6e8d-4d39-af45-f70ace8e9ab7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e700e24-547r4\" (UID: \"cdd06518-6e8d-4d39-af45-f70ace8e9ab7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e700e24-547r4" Oct 31 04:16:27.388528 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:27.388493 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdd06518-6e8d-4d39-af45-f70ace8e9ab7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e700e24-547r4\" (UID: \"cdd06518-6e8d-4d39-af45-f70ace8e9ab7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e700e24-547r4" Oct 31 04:16:27.409038 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:27.409004 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9ms5\" (UniqueName: \"kubernetes.io/projected/cdd06518-6e8d-4d39-af45-f70ace8e9ab7-kube-api-access-p9ms5\") pod \"maas-api-revoke-sub-6e700e24-547r4\" (UID: \"cdd06518-6e8d-4d39-af45-f70ace8e9ab7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e700e24-547r4" Oct 31 04:16:27.518874 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:27.518553 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e700e24-547r4" Oct 31 04:16:27.829343 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:27.829301 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e700e24-547r4"] Oct 31 04:16:27.831622 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:16:27.831572 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdd06518_6e8d_4d39_af45_f70ace8e9ab7.slice/crio-e8469a6ae3b734b3500744cd3142b5ade132239c50ad61bcc44f16b418237517 WatchSource:0}: Error finding container e8469a6ae3b734b3500744cd3142b5ade132239c50ad61bcc44f16b418237517: Status 404 returned error can't find the container with id e8469a6ae3b734b3500744cd3142b5ade132239c50ad61bcc44f16b418237517 Oct 31 04:16:28.136024 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:28.135974 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x"] Oct 31 04:16:28.140500 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:28.140472 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x" Oct 31 04:16:28.180489 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:28.180454 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-965f61-openshift-default-dockercfg-ggnnv\"" Oct 31 04:16:28.204996 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:28.204957 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x"] Oct 31 04:16:28.299356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:28.296038 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/661aa7e3-37aa-4131-aa1d-10c81372dd2d-istio-envoy\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-xln8x\" (UID: \"661aa7e3-37aa-4131-aa1d-10c81372dd2d\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x" Oct 31 04:16:28.299356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:28.296099 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/661aa7e3-37aa-4131-aa1d-10c81372dd2d-istio-podinfo\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-xln8x\" (UID: \"661aa7e3-37aa-4131-aa1d-10c81372dd2d\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x" Oct 31 04:16:28.299356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:28.296177 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/661aa7e3-37aa-4131-aa1d-10c81372dd2d-credential-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-xln8x\" (UID: \"661aa7e3-37aa-4131-aa1d-10c81372dd2d\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x" Oct 31 04:16:28.299356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:28.296210 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/661aa7e3-37aa-4131-aa1d-10c81372dd2d-istio-token\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-xln8x\" (UID: \"661aa7e3-37aa-4131-aa1d-10c81372dd2d\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x" Oct 31 04:16:28.299356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:28.296281 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/661aa7e3-37aa-4131-aa1d-10c81372dd2d-workload-certs\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-xln8x\" (UID: \"661aa7e3-37aa-4131-aa1d-10c81372dd2d\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x" Oct 31 04:16:28.299356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:28.296382 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/661aa7e3-37aa-4131-aa1d-10c81372dd2d-workload-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-xln8x\" (UID: \"661aa7e3-37aa-4131-aa1d-10c81372dd2d\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x" Oct 31 04:16:28.299356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:28.296436 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/661aa7e3-37aa-4131-aa1d-10c81372dd2d-istiod-ca-cert\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-xln8x\" (UID: \"661aa7e3-37aa-4131-aa1d-10c81372dd2d\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x" Oct 31 04:16:28.299356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:28.296476 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4dmw\" (UniqueName: \"kubernetes.io/projected/661aa7e3-37aa-4131-aa1d-10c81372dd2d-kube-api-access-g4dmw\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-xln8x\" (UID: \"661aa7e3-37aa-4131-aa1d-10c81372dd2d\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x" Oct 31 04:16:28.299356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:28.296508 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/661aa7e3-37aa-4131-aa1d-10c81372dd2d-istio-data\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-xln8x\" (UID: \"661aa7e3-37aa-4131-aa1d-10c81372dd2d\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x" Oct 31 04:16:28.400686 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:28.400575 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/661aa7e3-37aa-4131-aa1d-10c81372dd2d-credential-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-xln8x\" (UID: \"661aa7e3-37aa-4131-aa1d-10c81372dd2d\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x" Oct 31 04:16:28.400686 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:28.400630 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/661aa7e3-37aa-4131-aa1d-10c81372dd2d-istio-token\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-xln8x\" (UID: \"661aa7e3-37aa-4131-aa1d-10c81372dd2d\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x" Oct 31 04:16:28.400686 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:28.400682 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/661aa7e3-37aa-4131-aa1d-10c81372dd2d-workload-certs\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-xln8x\" (UID: \"661aa7e3-37aa-4131-aa1d-10c81372dd2d\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x" Oct 31 04:16:28.400986 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:28.400747 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/661aa7e3-37aa-4131-aa1d-10c81372dd2d-workload-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-xln8x\" (UID: \"661aa7e3-37aa-4131-aa1d-10c81372dd2d\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x" Oct 31 04:16:28.400986 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:28.400792 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/661aa7e3-37aa-4131-aa1d-10c81372dd2d-istiod-ca-cert\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-xln8x\" (UID: \"661aa7e3-37aa-4131-aa1d-10c81372dd2d\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x" Oct 31 04:16:28.400986 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:28.400824 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4dmw\" (UniqueName: \"kubernetes.io/projected/661aa7e3-37aa-4131-aa1d-10c81372dd2d-kube-api-access-g4dmw\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-xln8x\" (UID: \"661aa7e3-37aa-4131-aa1d-10c81372dd2d\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x" Oct 31 04:16:28.400986 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:28.400860 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/661aa7e3-37aa-4131-aa1d-10c81372dd2d-istio-data\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-xln8x\" (UID: \"661aa7e3-37aa-4131-aa1d-10c81372dd2d\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x" Oct 31 04:16:28.400986 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:28.400892 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/661aa7e3-37aa-4131-aa1d-10c81372dd2d-istio-envoy\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-xln8x\" (UID: \"661aa7e3-37aa-4131-aa1d-10c81372dd2d\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x" Oct 31 04:16:28.400986 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:28.400921 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/661aa7e3-37aa-4131-aa1d-10c81372dd2d-istio-podinfo\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-xln8x\" (UID: \"661aa7e3-37aa-4131-aa1d-10c81372dd2d\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x" Oct 31 04:16:28.401312 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:28.401070 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/661aa7e3-37aa-4131-aa1d-10c81372dd2d-credential-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-xln8x\" (UID: \"661aa7e3-37aa-4131-aa1d-10c81372dd2d\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x" Oct 31 04:16:28.402001 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:28.401787 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/661aa7e3-37aa-4131-aa1d-10c81372dd2d-istiod-ca-cert\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-xln8x\" (UID: \"661aa7e3-37aa-4131-aa1d-10c81372dd2d\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x" Oct 31 04:16:28.402369 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:28.402318 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/661aa7e3-37aa-4131-aa1d-10c81372dd2d-workload-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-xln8x\" (UID: \"661aa7e3-37aa-4131-aa1d-10c81372dd2d\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x" Oct 31 04:16:28.403015 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:28.402990 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/661aa7e3-37aa-4131-aa1d-10c81372dd2d-istio-data\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-xln8x\" (UID: \"661aa7e3-37aa-4131-aa1d-10c81372dd2d\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x" Oct 31 04:16:28.403168 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:28.403148 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/661aa7e3-37aa-4131-aa1d-10c81372dd2d-workload-certs\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-xln8x\" (UID: \"661aa7e3-37aa-4131-aa1d-10c81372dd2d\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x" Oct 31 04:16:28.404389 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:28.404366 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/661aa7e3-37aa-4131-aa1d-10c81372dd2d-istio-podinfo\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-xln8x\" (UID: \"661aa7e3-37aa-4131-aa1d-10c81372dd2d\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x" Oct 31 04:16:28.404662 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:28.404634 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/661aa7e3-37aa-4131-aa1d-10c81372dd2d-istio-envoy\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-xln8x\" (UID: \"661aa7e3-37aa-4131-aa1d-10c81372dd2d\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x" Oct 31 04:16:28.426029 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:28.425991 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/661aa7e3-37aa-4131-aa1d-10c81372dd2d-istio-token\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-xln8x\" (UID: \"661aa7e3-37aa-4131-aa1d-10c81372dd2d\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x" Oct 31 04:16:28.426200 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:28.426116 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4dmw\" (UniqueName: \"kubernetes.io/projected/661aa7e3-37aa-4131-aa1d-10c81372dd2d-kube-api-access-g4dmw\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-xln8x\" (UID: \"661aa7e3-37aa-4131-aa1d-10c81372dd2d\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x" Oct 31 04:16:28.456368 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:28.456307 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x" Oct 31 04:16:28.539451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:28.538994 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e700e24-547r4" event={"ID":"cdd06518-6e8d-4d39-af45-f70ace8e9ab7","Type":"ContainerStarted","Data":"42fba72c53c00791d286b6695aa10408498eb8b2cce6d101c52ec375f902c555"} Oct 31 04:16:28.539451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:28.539076 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e700e24-547r4" event={"ID":"cdd06518-6e8d-4d39-af45-f70ace8e9ab7","Type":"ContainerStarted","Data":"e8469a6ae3b734b3500744cd3142b5ade132239c50ad61bcc44f16b418237517"} Oct 31 04:16:28.583669 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:28.583594 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e700e24-547r4" podStartSLOduration=1.58357046 podStartE2EDuration="1.58357046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:16:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:16:28.581545467 +0000 UTC m=+743.337883529" watchObservedRunningTime="2026-10-31 04:16:28.58357046 +0000 UTC m=+743.339908502" Oct 31 04:16:28.800671 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:28.800627 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x"] Oct 31 04:16:28.805922 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:16:28.805880 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod661aa7e3_37aa_4131_aa1d_10c81372dd2d.slice/crio-a07527e0193ff21537650b5e13f6979c39accfe4b392ca32b7a9e09653fd7ac1 WatchSource:0}: Error finding container a07527e0193ff21537650b5e13f6979c39accfe4b392ca32b7a9e09653fd7ac1: Status 404 returned error can't find the container with id a07527e0193ff21537650b5e13f6979c39accfe4b392ca32b7a9e09653fd7ac1 Oct 31 04:16:28.812985 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:28.812924 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:16:28.813140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:28.813036 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:16:28.813140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:28.813081 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:16:29.555704 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:29.554639 5019 generic.go:356] "Generic (PLEG): container finished" podID="cdd06518-6e8d-4d39-af45-f70ace8e9ab7" containerID="42fba72c53c00791d286b6695aa10408498eb8b2cce6d101c52ec375f902c555" exitCode=0 Oct 31 04:16:29.555704 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:29.554815 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e700e24-547r4" event={"ID":"cdd06518-6e8d-4d39-af45-f70ace8e9ab7","Type":"ContainerDied","Data":"42fba72c53c00791d286b6695aa10408498eb8b2cce6d101c52ec375f902c555"} Oct 31 04:16:29.561135 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:29.561093 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x" event={"ID":"661aa7e3-37aa-4131-aa1d-10c81372dd2d","Type":"ContainerStarted","Data":"63ccf93e122b8400412e54e2ffc5e03f1e8f26c02a47748d031ac360d9d9d392"} Oct 31 04:16:29.561499 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:29.561471 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x" event={"ID":"661aa7e3-37aa-4131-aa1d-10c81372dd2d","Type":"ContainerStarted","Data":"a07527e0193ff21537650b5e13f6979c39accfe4b392ca32b7a9e09653fd7ac1"} Oct 31 04:16:29.650704 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:29.650631 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x" podStartSLOduration=1.650608988 podStartE2EDuration="1.650608988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:16:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:16:29.643433922 +0000 UTC m=+744.399771964" watchObservedRunningTime="2026-10-31 04:16:29.650608988 +0000 UTC m=+744.406947029" Oct 31 04:16:30.459358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:30.458781 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x" Oct 31 04:16:30.461719 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:30.461676 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 04:16:30.461922 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:30.461742 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x" podUID="661aa7e3-37aa-4131-aa1d-10c81372dd2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 04:16:31.063744 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:31.063716 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e700e24-547r4" Oct 31 04:16:31.158091 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:31.158050 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdd06518-6e8d-4d39-af45-f70ace8e9ab7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdd06518-6e8d-4d39-af45-f70ace8e9ab7-maas-api-xks-ca\") pod \"cdd06518-6e8d-4d39-af45-f70ace8e9ab7\" (UID: \"cdd06518-6e8d-4d39-af45-f70ace8e9ab7\") " Oct 31 04:16:31.158259 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:31.158169 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdd06518-6e8d-4d39-af45-f70ace8e9ab7-kube-api-access-p9ms5\" (UniqueName: \"kubernetes.io/projected/cdd06518-6e8d-4d39-af45-f70ace8e9ab7-kube-api-access-p9ms5\") pod \"cdd06518-6e8d-4d39-af45-f70ace8e9ab7\" (UID: \"cdd06518-6e8d-4d39-af45-f70ace8e9ab7\") " Oct 31 04:16:31.158595 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:31.158552 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdd06518-6e8d-4d39-af45-f70ace8e9ab7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdd06518-6e8d-4d39-af45-f70ace8e9ab7-maas-api-openshift-ca\") pod \"cdd06518-6e8d-4d39-af45-f70ace8e9ab7\" (UID: \"cdd06518-6e8d-4d39-af45-f70ace8e9ab7\") " Oct 31 04:16:31.159254 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:31.159122 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdd06518-6e8d-4d39-af45-f70ace8e9ab7-maas-api-openshift-ca" pod "cdd06518-6e8d-4d39-af45-f70ace8e9ab7" (UID: "cdd06518-6e8d-4d39-af45-f70ace8e9ab7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:16:31.160571 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:31.159377 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdd06518-6e8d-4d39-af45-f70ace8e9ab7-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:16:31.160916 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:31.160891 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdd06518-6e8d-4d39-af45-f70ace8e9ab7-maas-api-xks-ca" pod "cdd06518-6e8d-4d39-af45-f70ace8e9ab7" (UID: "cdd06518-6e8d-4d39-af45-f70ace8e9ab7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:16:31.160979 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:31.160922 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdd06518-6e8d-4d39-af45-f70ace8e9ab7-kube-api-access-p9ms5" pod "cdd06518-6e8d-4d39-af45-f70ace8e9ab7" (UID: "cdd06518-6e8d-4d39-af45-f70ace8e9ab7"). InnerVolumeSpecName "kube-api-access-p9ms5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:16:31.261357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:31.260453 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdd06518-6e8d-4d39-af45-f70ace8e9ab7-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:16:31.261357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:31.260486 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9ms5\" (UniqueName: \"kubernetes.io/projected/cdd06518-6e8d-4d39-af45-f70ace8e9ab7-kube-api-access-p9ms5\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:16:31.457600 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:31.457563 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 04:16:31.457768 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:31.457626 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x" podUID="661aa7e3-37aa-4131-aa1d-10c81372dd2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 04:16:31.600721 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:31.600474 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e700e24-547r4" Oct 31 04:16:31.600721 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:31.600605 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e700e24-547r4" event={"ID":"cdd06518-6e8d-4d39-af45-f70ace8e9ab7","Type":"ContainerDied","Data":"e8469a6ae3b734b3500744cd3142b5ade132239c50ad61bcc44f16b418237517"} Oct 31 04:16:31.600721 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:31.600674 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8469a6ae3b734b3500744cd3142b5ade132239c50ad61bcc44f16b418237517" Oct 31 04:16:32.457388 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:32.457326 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 04:16:32.457829 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:32.457407 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x" podUID="661aa7e3-37aa-4131-aa1d-10c81372dd2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 04:16:33.456670 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:33.456632 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 04:16:33.456869 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:33.456699 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x" podUID="661aa7e3-37aa-4131-aa1d-10c81372dd2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 04:16:34.457629 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:34.457586 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 04:16:34.458061 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:34.457643 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x" podUID="661aa7e3-37aa-4131-aa1d-10c81372dd2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 04:16:34.823539 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:34.823498 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2f19c96-4ncv7"] Oct 31 04:16:34.824252 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:34.824231 5019 cpu_manager.go:414] "containerMap: removing container" podUID="cdd06518-6e8d-4d39-af45-f70ace8e9ab7" containerName="revoke-keys" Oct 31 04:16:34.824252 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:34.824252 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdd06518-6e8d-4d39-af45-f70ace8e9ab7" containerName="revoke-keys" Oct 31 04:16:34.824491 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:34.824382 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdd06518-6e8d-4d39-af45-f70ace8e9ab7" containerName="revoke-keys" Oct 31 04:16:34.828241 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:34.828213 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2f19c96-4ncv7" Oct 31 04:16:34.847081 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:34.847046 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2f19c96-4ncv7"] Oct 31 04:16:34.855841 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:34.855320 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:16:34.855841 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:34.855633 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:16:34.855841 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:34.855638 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:16:34.855841 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:34.855700 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:16:34.934803 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:34.934755 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/877cc431-548f-4df1-bcbe-a4d63c5f5595-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2f19c96-4ncv7\" (UID: \"877cc431-548f-4df1-bcbe-a4d63c5f5595\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2f19c96-4ncv7" Oct 31 04:16:34.934803 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:34.934799 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2ttn\" (UniqueName: \"kubernetes.io/projected/877cc431-548f-4df1-bcbe-a4d63c5f5595-kube-api-access-w2ttn\") pod \"maas-api-revoke-sub-e2f19c96-4ncv7\" (UID: \"877cc431-548f-4df1-bcbe-a4d63c5f5595\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2f19c96-4ncv7" Oct 31 04:16:34.935076 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:34.934972 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/877cc431-548f-4df1-bcbe-a4d63c5f5595-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2f19c96-4ncv7\" (UID: \"877cc431-548f-4df1-bcbe-a4d63c5f5595\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2f19c96-4ncv7" Oct 31 04:16:35.036260 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:35.036217 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/877cc431-548f-4df1-bcbe-a4d63c5f5595-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2f19c96-4ncv7\" (UID: \"877cc431-548f-4df1-bcbe-a4d63c5f5595\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2f19c96-4ncv7" Oct 31 04:16:35.036518 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:35.036417 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/877cc431-548f-4df1-bcbe-a4d63c5f5595-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2f19c96-4ncv7\" (UID: \"877cc431-548f-4df1-bcbe-a4d63c5f5595\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2f19c96-4ncv7" Oct 31 04:16:35.036518 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:35.036456 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2ttn\" (UniqueName: \"kubernetes.io/projected/877cc431-548f-4df1-bcbe-a4d63c5f5595-kube-api-access-w2ttn\") pod \"maas-api-revoke-sub-e2f19c96-4ncv7\" (UID: \"877cc431-548f-4df1-bcbe-a4d63c5f5595\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2f19c96-4ncv7" Oct 31 04:16:35.037066 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:35.037039 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/877cc431-548f-4df1-bcbe-a4d63c5f5595-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2f19c96-4ncv7\" (UID: \"877cc431-548f-4df1-bcbe-a4d63c5f5595\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2f19c96-4ncv7" Oct 31 04:16:35.038818 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:35.038792 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/877cc431-548f-4df1-bcbe-a4d63c5f5595-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2f19c96-4ncv7\" (UID: \"877cc431-548f-4df1-bcbe-a4d63c5f5595\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2f19c96-4ncv7" Oct 31 04:16:35.060910 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:35.060872 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2ttn\" (UniqueName: \"kubernetes.io/projected/877cc431-548f-4df1-bcbe-a4d63c5f5595-kube-api-access-w2ttn\") pod \"maas-api-revoke-sub-e2f19c96-4ncv7\" (UID: \"877cc431-548f-4df1-bcbe-a4d63c5f5595\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2f19c96-4ncv7" Oct 31 04:16:35.143795 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:35.143693 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2f19c96-4ncv7" Oct 31 04:16:35.456819 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:35.456731 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 04:16:35.456819 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:35.456795 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x" podUID="661aa7e3-37aa-4131-aa1d-10c81372dd2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 04:16:35.461504 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:35.461464 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2f19c96-4ncv7"] Oct 31 04:16:35.462573 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:16:35.462535 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod877cc431_548f_4df1_bcbe_a4d63c5f5595.slice/crio-493c570204ca4ce69587e99543df09bc3262146c007ba230b1508060e10b9094 WatchSource:0}: Error finding container 493c570204ca4ce69587e99543df09bc3262146c007ba230b1508060e10b9094: Status 404 returned error can't find the container with id 493c570204ca4ce69587e99543df09bc3262146c007ba230b1508060e10b9094 Oct 31 04:16:35.643803 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:35.643680 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2f19c96-4ncv7" event={"ID":"877cc431-548f-4df1-bcbe-a4d63c5f5595","Type":"ContainerStarted","Data":"493c570204ca4ce69587e99543df09bc3262146c007ba230b1508060e10b9094"} Oct 31 04:16:36.559279 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:36.559244 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x" Oct 31 04:16:36.561033 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:36.561002 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x" Oct 31 04:16:36.562370 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:36.561912 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x" Oct 31 04:16:36.652559 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:36.652527 5019 generic.go:356] "Generic (PLEG): container finished" podID="877cc431-548f-4df1-bcbe-a4d63c5f5595" containerID="da034093389171276ad29506b7c5af074290ac60adce8f310b3bbb0dfe1aeb17" exitCode=0 Oct 31 04:16:36.652740 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:36.652618 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2f19c96-4ncv7" event={"ID":"877cc431-548f-4df1-bcbe-a4d63c5f5595","Type":"ContainerDied","Data":"da034093389171276ad29506b7c5af074290ac60adce8f310b3bbb0dfe1aeb17"} Oct 31 04:16:37.825372 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:37.825315 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2f19c96-4ncv7" Oct 31 04:16:37.880954 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:37.880904 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/877cc431-548f-4df1-bcbe-a4d63c5f5595-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/877cc431-548f-4df1-bcbe-a4d63c5f5595-maas-api-openshift-ca\") pod \"877cc431-548f-4df1-bcbe-a4d63c5f5595\" (UID: \"877cc431-548f-4df1-bcbe-a4d63c5f5595\") " Oct 31 04:16:37.881175 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:37.881023 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/877cc431-548f-4df1-bcbe-a4d63c5f5595-kube-api-access-w2ttn\" (UniqueName: \"kubernetes.io/projected/877cc431-548f-4df1-bcbe-a4d63c5f5595-kube-api-access-w2ttn\") pod \"877cc431-548f-4df1-bcbe-a4d63c5f5595\" (UID: \"877cc431-548f-4df1-bcbe-a4d63c5f5595\") " Oct 31 04:16:37.881175 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:37.881101 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/877cc431-548f-4df1-bcbe-a4d63c5f5595-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/877cc431-548f-4df1-bcbe-a4d63c5f5595-maas-api-xks-ca\") pod \"877cc431-548f-4df1-bcbe-a4d63c5f5595\" (UID: \"877cc431-548f-4df1-bcbe-a4d63c5f5595\") " Oct 31 04:16:37.881473 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:37.881441 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/877cc431-548f-4df1-bcbe-a4d63c5f5595-maas-api-openshift-ca" pod "877cc431-548f-4df1-bcbe-a4d63c5f5595" (UID: "877cc431-548f-4df1-bcbe-a4d63c5f5595"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:16:37.881609 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:37.881589 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/877cc431-548f-4df1-bcbe-a4d63c5f5595-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:16:37.884044 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:37.884000 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/877cc431-548f-4df1-bcbe-a4d63c5f5595-maas-api-xks-ca" pod "877cc431-548f-4df1-bcbe-a4d63c5f5595" (UID: "877cc431-548f-4df1-bcbe-a4d63c5f5595"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:16:37.884044 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:37.884008 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/877cc431-548f-4df1-bcbe-a4d63c5f5595-kube-api-access-w2ttn" pod "877cc431-548f-4df1-bcbe-a4d63c5f5595" (UID: "877cc431-548f-4df1-bcbe-a4d63c5f5595"). InnerVolumeSpecName "kube-api-access-w2ttn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:16:37.982288 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:37.982252 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2ttn\" (UniqueName: \"kubernetes.io/projected/877cc431-548f-4df1-bcbe-a4d63c5f5595-kube-api-access-w2ttn\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:16:37.982288 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:37.982286 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/877cc431-548f-4df1-bcbe-a4d63c5f5595-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:16:38.668011 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:38.667969 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2f19c96-4ncv7" event={"ID":"877cc431-548f-4df1-bcbe-a4d63c5f5595","Type":"ContainerDied","Data":"493c570204ca4ce69587e99543df09bc3262146c007ba230b1508060e10b9094"} Oct 31 04:16:38.668011 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:38.668007 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2f19c96-4ncv7" Oct 31 04:16:38.668234 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:38.668028 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="493c570204ca4ce69587e99543df09bc3262146c007ba230b1508060e10b9094" Oct 31 04:16:40.063440 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:40.063402 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba3c9c5d-x862m"] Oct 31 04:16:40.063994 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:40.063978 5019 cpu_manager.go:414] "containerMap: removing container" podUID="877cc431-548f-4df1-bcbe-a4d63c5f5595" containerName="revoke-keys" Oct 31 04:16:40.064055 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:40.063994 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="877cc431-548f-4df1-bcbe-a4d63c5f5595" containerName="revoke-keys" Oct 31 04:16:40.064118 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:40.064108 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="877cc431-548f-4df1-bcbe-a4d63c5f5595" containerName="revoke-keys" Oct 31 04:16:40.067769 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:40.067736 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3c9c5d-x862m" Oct 31 04:16:40.096183 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:40.096148 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:16:40.096453 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:40.096162 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:16:40.096453 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:40.096182 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:16:40.096582 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:40.096304 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:16:40.101369 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:40.101306 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba3c9c5d-x862m"] Oct 31 04:16:40.208242 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:40.208202 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ab5359a-8e15-4809-a42d-d6855696376d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba3c9c5d-x862m\" (UID: \"0ab5359a-8e15-4809-a42d-d6855696376d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3c9c5d-x862m" Oct 31 04:16:40.208532 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:40.208270 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ab5359a-8e15-4809-a42d-d6855696376d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba3c9c5d-x862m\" (UID: \"0ab5359a-8e15-4809-a42d-d6855696376d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3c9c5d-x862m" Oct 31 04:16:40.208532 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:40.208465 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qff5g\" (UniqueName: \"kubernetes.io/projected/0ab5359a-8e15-4809-a42d-d6855696376d-kube-api-access-qff5g\") pod \"maas-api-revoke-sub-ba3c9c5d-x862m\" (UID: \"0ab5359a-8e15-4809-a42d-d6855696376d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3c9c5d-x862m" Oct 31 04:16:40.310343 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:40.310285 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qff5g\" (UniqueName: \"kubernetes.io/projected/0ab5359a-8e15-4809-a42d-d6855696376d-kube-api-access-qff5g\") pod \"maas-api-revoke-sub-ba3c9c5d-x862m\" (UID: \"0ab5359a-8e15-4809-a42d-d6855696376d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3c9c5d-x862m" Oct 31 04:16:40.310537 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:40.310480 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ab5359a-8e15-4809-a42d-d6855696376d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba3c9c5d-x862m\" (UID: \"0ab5359a-8e15-4809-a42d-d6855696376d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3c9c5d-x862m" Oct 31 04:16:40.310596 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:40.310537 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ab5359a-8e15-4809-a42d-d6855696376d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba3c9c5d-x862m\" (UID: \"0ab5359a-8e15-4809-a42d-d6855696376d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3c9c5d-x862m" Oct 31 04:16:40.311189 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:40.311159 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ab5359a-8e15-4809-a42d-d6855696376d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba3c9c5d-x862m\" (UID: \"0ab5359a-8e15-4809-a42d-d6855696376d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3c9c5d-x862m" Oct 31 04:16:40.313008 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:40.312974 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ab5359a-8e15-4809-a42d-d6855696376d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba3c9c5d-x862m\" (UID: \"0ab5359a-8e15-4809-a42d-d6855696376d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3c9c5d-x862m" Oct 31 04:16:40.334496 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:40.334417 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qff5g\" (UniqueName: \"kubernetes.io/projected/0ab5359a-8e15-4809-a42d-d6855696376d-kube-api-access-qff5g\") pod \"maas-api-revoke-sub-ba3c9c5d-x862m\" (UID: \"0ab5359a-8e15-4809-a42d-d6855696376d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3c9c5d-x862m" Oct 31 04:16:40.386174 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:40.386127 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3c9c5d-x862m" Oct 31 04:16:40.687655 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:40.687219 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba3c9c5d-x862m"] Oct 31 04:16:40.689871 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:16:40.689818 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ab5359a_8e15_4809_a42d_d6855696376d.slice/crio-bb03c31c8651595e793803e319b1f18cc60f3546bad8be12c85d727496a70e60 WatchSource:0}: Error finding container bb03c31c8651595e793803e319b1f18cc60f3546bad8be12c85d727496a70e60: Status 404 returned error can't find the container with id bb03c31c8651595e793803e319b1f18cc60f3546bad8be12c85d727496a70e60 Oct 31 04:16:41.692504 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:41.692468 5019 generic.go:356] "Generic (PLEG): container finished" podID="0ab5359a-8e15-4809-a42d-d6855696376d" containerID="563745c7ff3aba3843df63a930c0efc87fc18dc998fa4243a859348ac665d8f0" exitCode=0 Oct 31 04:16:41.692906 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:41.692560 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3c9c5d-x862m" event={"ID":"0ab5359a-8e15-4809-a42d-d6855696376d","Type":"ContainerDied","Data":"563745c7ff3aba3843df63a930c0efc87fc18dc998fa4243a859348ac665d8f0"} Oct 31 04:16:41.692906 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:41.692617 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3c9c5d-x862m" event={"ID":"0ab5359a-8e15-4809-a42d-d6855696376d","Type":"ContainerStarted","Data":"bb03c31c8651595e793803e319b1f18cc60f3546bad8be12c85d727496a70e60"} Oct 31 04:16:42.862357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:42.862306 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3c9c5d-x862m" Oct 31 04:16:42.944443 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:42.944399 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ab5359a-8e15-4809-a42d-d6855696376d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ab5359a-8e15-4809-a42d-d6855696376d-maas-api-xks-ca\") pod \"0ab5359a-8e15-4809-a42d-d6855696376d\" (UID: \"0ab5359a-8e15-4809-a42d-d6855696376d\") " Oct 31 04:16:42.944595 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:42.944511 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ab5359a-8e15-4809-a42d-d6855696376d-kube-api-access-qff5g\" (UniqueName: \"kubernetes.io/projected/0ab5359a-8e15-4809-a42d-d6855696376d-kube-api-access-qff5g\") pod \"0ab5359a-8e15-4809-a42d-d6855696376d\" (UID: \"0ab5359a-8e15-4809-a42d-d6855696376d\") " Oct 31 04:16:42.944689 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:42.944672 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ab5359a-8e15-4809-a42d-d6855696376d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ab5359a-8e15-4809-a42d-d6855696376d-maas-api-openshift-ca\") pod \"0ab5359a-8e15-4809-a42d-d6855696376d\" (UID: \"0ab5359a-8e15-4809-a42d-d6855696376d\") " Oct 31 04:16:42.945085 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:42.945049 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ab5359a-8e15-4809-a42d-d6855696376d-maas-api-openshift-ca" pod "0ab5359a-8e15-4809-a42d-d6855696376d" (UID: "0ab5359a-8e15-4809-a42d-d6855696376d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:16:42.945252 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:42.945170 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ab5359a-8e15-4809-a42d-d6855696376d-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:16:42.946781 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:42.946742 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ab5359a-8e15-4809-a42d-d6855696376d-maas-api-xks-ca" pod "0ab5359a-8e15-4809-a42d-d6855696376d" (UID: "0ab5359a-8e15-4809-a42d-d6855696376d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:16:42.947038 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:42.947016 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ab5359a-8e15-4809-a42d-d6855696376d-kube-api-access-qff5g" pod "0ab5359a-8e15-4809-a42d-d6855696376d" (UID: "0ab5359a-8e15-4809-a42d-d6855696376d"). InnerVolumeSpecName "kube-api-access-qff5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:16:43.046020 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:43.045974 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ab5359a-8e15-4809-a42d-d6855696376d-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:16:43.046020 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:43.046010 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qff5g\" (UniqueName: \"kubernetes.io/projected/0ab5359a-8e15-4809-a42d-d6855696376d-kube-api-access-qff5g\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:16:43.712916 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:43.712828 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3c9c5d-x862m" Oct 31 04:16:43.712916 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:43.712852 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3c9c5d-x862m" event={"ID":"0ab5359a-8e15-4809-a42d-d6855696376d","Type":"ContainerDied","Data":"bb03c31c8651595e793803e319b1f18cc60f3546bad8be12c85d727496a70e60"} Oct 31 04:16:43.712916 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:43.712907 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb03c31c8651595e793803e319b1f18cc60f3546bad8be12c85d727496a70e60" Oct 31 04:16:49.356076 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:49.355990 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49e61f07-xwlzd"] Oct 31 04:16:49.357625 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:49.357591 5019 cpu_manager.go:414] "containerMap: removing container" podUID="0ab5359a-8e15-4809-a42d-d6855696376d" containerName="revoke-keys" Oct 31 04:16:49.357852 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:49.357815 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ab5359a-8e15-4809-a42d-d6855696376d" containerName="revoke-keys" Oct 31 04:16:49.358024 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:49.358008 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ab5359a-8e15-4809-a42d-d6855696376d" containerName="revoke-keys" Oct 31 04:16:49.362257 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:49.362198 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e61f07-xwlzd" Oct 31 04:16:49.383942 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:49.383905 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49e61f07-xwlzd"] Oct 31 04:16:49.390861 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:49.390825 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:16:49.391111 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:49.390890 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:16:49.391111 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:49.390930 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:16:49.391304 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:49.390975 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:16:49.526113 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:49.526077 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f1cb632-2629-49a0-99fe-5170142214e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49e61f07-xwlzd\" (UID: \"5f1cb632-2629-49a0-99fe-5170142214e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e61f07-xwlzd" Oct 31 04:16:49.526310 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:49.526138 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f1cb632-2629-49a0-99fe-5170142214e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49e61f07-xwlzd\" (UID: \"5f1cb632-2629-49a0-99fe-5170142214e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e61f07-xwlzd" Oct 31 04:16:49.526310 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:49.526247 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxbx8\" (UniqueName: \"kubernetes.io/projected/5f1cb632-2629-49a0-99fe-5170142214e4-kube-api-access-pxbx8\") pod \"maas-api-revoke-sub-49e61f07-xwlzd\" (UID: \"5f1cb632-2629-49a0-99fe-5170142214e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e61f07-xwlzd" Oct 31 04:16:49.627607 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:49.627501 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxbx8\" (UniqueName: \"kubernetes.io/projected/5f1cb632-2629-49a0-99fe-5170142214e4-kube-api-access-pxbx8\") pod \"maas-api-revoke-sub-49e61f07-xwlzd\" (UID: \"5f1cb632-2629-49a0-99fe-5170142214e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e61f07-xwlzd" Oct 31 04:16:49.627789 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:49.627647 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f1cb632-2629-49a0-99fe-5170142214e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49e61f07-xwlzd\" (UID: \"5f1cb632-2629-49a0-99fe-5170142214e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e61f07-xwlzd" Oct 31 04:16:49.627789 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:49.627691 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f1cb632-2629-49a0-99fe-5170142214e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49e61f07-xwlzd\" (UID: \"5f1cb632-2629-49a0-99fe-5170142214e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e61f07-xwlzd" Oct 31 04:16:49.628435 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:49.628406 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f1cb632-2629-49a0-99fe-5170142214e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49e61f07-xwlzd\" (UID: \"5f1cb632-2629-49a0-99fe-5170142214e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e61f07-xwlzd" Oct 31 04:16:49.630229 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:49.630200 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f1cb632-2629-49a0-99fe-5170142214e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49e61f07-xwlzd\" (UID: \"5f1cb632-2629-49a0-99fe-5170142214e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e61f07-xwlzd" Oct 31 04:16:49.655950 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:49.655911 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxbx8\" (UniqueName: \"kubernetes.io/projected/5f1cb632-2629-49a0-99fe-5170142214e4-kube-api-access-pxbx8\") pod \"maas-api-revoke-sub-49e61f07-xwlzd\" (UID: \"5f1cb632-2629-49a0-99fe-5170142214e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e61f07-xwlzd" Oct 31 04:16:49.692292 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:49.692242 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e61f07-xwlzd" Oct 31 04:16:49.995087 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:49.995051 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49e61f07-xwlzd"] Oct 31 04:16:49.998509 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:16:49.998471 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f1cb632_2629_49a0_99fe_5170142214e4.slice/crio-9a671bd521197731cffe9856beabeda4a2d4f1d2b94f8dc7fce2881a17247d45 WatchSource:0}: Error finding container 9a671bd521197731cffe9856beabeda4a2d4f1d2b94f8dc7fce2881a17247d45: Status 404 returned error can't find the container with id 9a671bd521197731cffe9856beabeda4a2d4f1d2b94f8dc7fce2881a17247d45 Oct 31 04:16:50.768916 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:50.768878 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e61f07-xwlzd" event={"ID":"5f1cb632-2629-49a0-99fe-5170142214e4","Type":"ContainerStarted","Data":"1d579311d0b7a1fb59ffea97fdf927f8b061573faa60c8fc446be82226540472"} Oct 31 04:16:50.769353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:50.768935 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e61f07-xwlzd" event={"ID":"5f1cb632-2629-49a0-99fe-5170142214e4","Type":"ContainerStarted","Data":"9a671bd521197731cffe9856beabeda4a2d4f1d2b94f8dc7fce2881a17247d45"} Oct 31 04:16:50.811324 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:50.811262 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e61f07-xwlzd" podStartSLOduration=1.811245591 podStartE2EDuration="1.811245591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:16:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:16:50.808278307 +0000 UTC m=+765.564616348" watchObservedRunningTime="2026-10-31 04:16:50.811245591 +0000 UTC m=+765.567583659" Oct 31 04:16:51.781895 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:51.781854 5019 generic.go:356] "Generic (PLEG): container finished" podID="5f1cb632-2629-49a0-99fe-5170142214e4" containerID="1d579311d0b7a1fb59ffea97fdf927f8b061573faa60c8fc446be82226540472" exitCode=0 Oct 31 04:16:51.782475 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:51.782019 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e61f07-xwlzd" event={"ID":"5f1cb632-2629-49a0-99fe-5170142214e4","Type":"ContainerDied","Data":"1d579311d0b7a1fb59ffea97fdf927f8b061573faa60c8fc446be82226540472"} Oct 31 04:16:52.098413 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:52.098285 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-160cd22e-dh4ff"] Oct 31 04:16:52.103004 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:52.102957 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-160cd22e-dh4ff" Oct 31 04:16:52.123402 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:52.123364 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-160cd22e-dh4ff"] Oct 31 04:16:52.164819 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:52.164771 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8bhg\" (UniqueName: \"kubernetes.io/projected/558750ea-02ff-4ff2-b126-4ebb050f9092-kube-api-access-d8bhg\") pod \"maas-api-revoke-sub-160cd22e-dh4ff\" (UID: \"558750ea-02ff-4ff2-b126-4ebb050f9092\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-160cd22e-dh4ff" Oct 31 04:16:52.165022 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:52.164891 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/558750ea-02ff-4ff2-b126-4ebb050f9092-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-160cd22e-dh4ff\" (UID: \"558750ea-02ff-4ff2-b126-4ebb050f9092\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-160cd22e-dh4ff" Oct 31 04:16:52.165022 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:52.164971 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/558750ea-02ff-4ff2-b126-4ebb050f9092-maas-api-xks-ca\") pod \"maas-api-revoke-sub-160cd22e-dh4ff\" (UID: \"558750ea-02ff-4ff2-b126-4ebb050f9092\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-160cd22e-dh4ff" Oct 31 04:16:52.266110 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:52.266061 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8bhg\" (UniqueName: \"kubernetes.io/projected/558750ea-02ff-4ff2-b126-4ebb050f9092-kube-api-access-d8bhg\") pod \"maas-api-revoke-sub-160cd22e-dh4ff\" (UID: \"558750ea-02ff-4ff2-b126-4ebb050f9092\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-160cd22e-dh4ff" Oct 31 04:16:52.266354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:52.266145 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/558750ea-02ff-4ff2-b126-4ebb050f9092-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-160cd22e-dh4ff\" (UID: \"558750ea-02ff-4ff2-b126-4ebb050f9092\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-160cd22e-dh4ff" Oct 31 04:16:52.266354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:52.266188 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/558750ea-02ff-4ff2-b126-4ebb050f9092-maas-api-xks-ca\") pod \"maas-api-revoke-sub-160cd22e-dh4ff\" (UID: \"558750ea-02ff-4ff2-b126-4ebb050f9092\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-160cd22e-dh4ff" Oct 31 04:16:52.267031 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:52.267002 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/558750ea-02ff-4ff2-b126-4ebb050f9092-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-160cd22e-dh4ff\" (UID: \"558750ea-02ff-4ff2-b126-4ebb050f9092\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-160cd22e-dh4ff" Oct 31 04:16:52.268936 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:52.268904 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/558750ea-02ff-4ff2-b126-4ebb050f9092-maas-api-xks-ca\") pod \"maas-api-revoke-sub-160cd22e-dh4ff\" (UID: \"558750ea-02ff-4ff2-b126-4ebb050f9092\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-160cd22e-dh4ff" Oct 31 04:16:52.289779 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:52.289745 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8bhg\" (UniqueName: \"kubernetes.io/projected/558750ea-02ff-4ff2-b126-4ebb050f9092-kube-api-access-d8bhg\") pod \"maas-api-revoke-sub-160cd22e-dh4ff\" (UID: \"558750ea-02ff-4ff2-b126-4ebb050f9092\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-160cd22e-dh4ff" Oct 31 04:16:52.391649 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:16:52.391541 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:16:52.425663 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:52.425619 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-160cd22e-dh4ff" Oct 31 04:16:52.732768 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:52.732734 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-160cd22e-dh4ff"] Oct 31 04:16:52.734255 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:16:52.734224 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod558750ea_02ff_4ff2_b126_4ebb050f9092.slice/crio-7da0f067fe266775ab71f01fababca68ac83d901928470d70f0bd48d6a6a83a4 WatchSource:0}: Error finding container 7da0f067fe266775ab71f01fababca68ac83d901928470d70f0bd48d6a6a83a4: Status 404 returned error can't find the container with id 7da0f067fe266775ab71f01fababca68ac83d901928470d70f0bd48d6a6a83a4 Oct 31 04:16:52.790309 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:52.790272 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-160cd22e-dh4ff" event={"ID":"558750ea-02ff-4ff2-b126-4ebb050f9092","Type":"ContainerStarted","Data":"7da0f067fe266775ab71f01fababca68ac83d901928470d70f0bd48d6a6a83a4"} Oct 31 04:16:52.956013 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:52.955979 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e61f07-xwlzd" Oct 31 04:16:52.975263 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:52.975235 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f1cb632-2629-49a0-99fe-5170142214e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f1cb632-2629-49a0-99fe-5170142214e4-maas-api-openshift-ca\") pod \"5f1cb632-2629-49a0-99fe-5170142214e4\" (UID: \"5f1cb632-2629-49a0-99fe-5170142214e4\") " Oct 31 04:16:52.975470 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:52.975293 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f1cb632-2629-49a0-99fe-5170142214e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f1cb632-2629-49a0-99fe-5170142214e4-maas-api-xks-ca\") pod \"5f1cb632-2629-49a0-99fe-5170142214e4\" (UID: \"5f1cb632-2629-49a0-99fe-5170142214e4\") " Oct 31 04:16:52.975470 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:52.975355 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f1cb632-2629-49a0-99fe-5170142214e4-kube-api-access-pxbx8\" (UniqueName: \"kubernetes.io/projected/5f1cb632-2629-49a0-99fe-5170142214e4-kube-api-access-pxbx8\") pod \"5f1cb632-2629-49a0-99fe-5170142214e4\" (UID: \"5f1cb632-2629-49a0-99fe-5170142214e4\") " Oct 31 04:16:52.975706 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:52.975682 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f1cb632-2629-49a0-99fe-5170142214e4-maas-api-openshift-ca" pod "5f1cb632-2629-49a0-99fe-5170142214e4" (UID: "5f1cb632-2629-49a0-99fe-5170142214e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:16:52.975816 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:52.975800 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f1cb632-2629-49a0-99fe-5170142214e4-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:16:52.977519 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:52.977488 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f1cb632-2629-49a0-99fe-5170142214e4-maas-api-xks-ca" pod "5f1cb632-2629-49a0-99fe-5170142214e4" (UID: "5f1cb632-2629-49a0-99fe-5170142214e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:16:52.977719 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:52.977698 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f1cb632-2629-49a0-99fe-5170142214e4-kube-api-access-pxbx8" pod "5f1cb632-2629-49a0-99fe-5170142214e4" (UID: "5f1cb632-2629-49a0-99fe-5170142214e4"). InnerVolumeSpecName "kube-api-access-pxbx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:16:53.062674 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:53.062583 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr"] Oct 31 04:16:53.063157 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:53.063141 5019 cpu_manager.go:414] "containerMap: removing container" podUID="5f1cb632-2629-49a0-99fe-5170142214e4" containerName="revoke-keys" Oct 31 04:16:53.063217 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:53.063158 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f1cb632-2629-49a0-99fe-5170142214e4" containerName="revoke-keys" Oct 31 04:16:53.063317 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:53.063304 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f1cb632-2629-49a0-99fe-5170142214e4" containerName="revoke-keys" Oct 31 04:16:53.067588 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:53.067556 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr" Oct 31 04:16:53.076817 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:53.076786 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f1cb632-2629-49a0-99fe-5170142214e4-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:16:53.076980 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:53.076820 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxbx8\" (UniqueName: \"kubernetes.io/projected/5f1cb632-2629-49a0-99fe-5170142214e4-kube-api-access-pxbx8\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:16:53.082079 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:53.082050 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-c4006d-openshift-default-dockercfg-h2s7k\"" Oct 31 04:16:53.090278 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:53.090237 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr"] Oct 31 04:16:53.178346 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:53.178286 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fksdw\" (UniqueName: \"kubernetes.io/projected/2df6e942-8414-427c-83df-928e51465964-kube-api-access-fksdw\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr\" (UID: \"2df6e942-8414-427c-83df-928e51465964\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr" Oct 31 04:16:53.178666 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:53.178356 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2df6e942-8414-427c-83df-928e51465964-istio-data\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr\" (UID: \"2df6e942-8414-427c-83df-928e51465964\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr" Oct 31 04:16:53.178666 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:53.178541 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2df6e942-8414-427c-83df-928e51465964-istio-podinfo\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr\" (UID: \"2df6e942-8414-427c-83df-928e51465964\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr" Oct 31 04:16:53.178666 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:53.178597 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2df6e942-8414-427c-83df-928e51465964-workload-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr\" (UID: \"2df6e942-8414-427c-83df-928e51465964\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr" Oct 31 04:16:53.178820 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:53.178678 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2df6e942-8414-427c-83df-928e51465964-credential-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr\" (UID: \"2df6e942-8414-427c-83df-928e51465964\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr" Oct 31 04:16:53.178820 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:53.178744 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2df6e942-8414-427c-83df-928e51465964-istio-token\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr\" (UID: \"2df6e942-8414-427c-83df-928e51465964\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr" Oct 31 04:16:53.178820 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:53.178772 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2df6e942-8414-427c-83df-928e51465964-istio-envoy\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr\" (UID: \"2df6e942-8414-427c-83df-928e51465964\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr" Oct 31 04:16:53.178820 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:53.178802 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2df6e942-8414-427c-83df-928e51465964-workload-certs\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr\" (UID: \"2df6e942-8414-427c-83df-928e51465964\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr" Oct 31 04:16:53.178984 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:53.178903 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2df6e942-8414-427c-83df-928e51465964-istiod-ca-cert\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr\" (UID: \"2df6e942-8414-427c-83df-928e51465964\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr" Oct 31 04:16:53.280264 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:53.280216 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fksdw\" (UniqueName: \"kubernetes.io/projected/2df6e942-8414-427c-83df-928e51465964-kube-api-access-fksdw\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr\" (UID: \"2df6e942-8414-427c-83df-928e51465964\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr" Oct 31 04:16:53.280502 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:53.280294 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2df6e942-8414-427c-83df-928e51465964-istio-data\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr\" (UID: \"2df6e942-8414-427c-83df-928e51465964\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr" Oct 31 04:16:53.280502 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:53.280422 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2df6e942-8414-427c-83df-928e51465964-istio-podinfo\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr\" (UID: \"2df6e942-8414-427c-83df-928e51465964\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr" Oct 31 04:16:53.280502 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:53.280459 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2df6e942-8414-427c-83df-928e51465964-workload-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr\" (UID: \"2df6e942-8414-427c-83df-928e51465964\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr" Oct 31 04:16:53.280694 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:53.280512 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2df6e942-8414-427c-83df-928e51465964-credential-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr\" (UID: \"2df6e942-8414-427c-83df-928e51465964\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr" Oct 31 04:16:53.280694 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:53.280562 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2df6e942-8414-427c-83df-928e51465964-istio-token\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr\" (UID: \"2df6e942-8414-427c-83df-928e51465964\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr" Oct 31 04:16:53.280694 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:53.280591 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2df6e942-8414-427c-83df-928e51465964-istio-envoy\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr\" (UID: \"2df6e942-8414-427c-83df-928e51465964\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr" Oct 31 04:16:53.280694 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:53.280634 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2df6e942-8414-427c-83df-928e51465964-workload-certs\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr\" (UID: \"2df6e942-8414-427c-83df-928e51465964\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr" Oct 31 04:16:53.280694 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:53.280674 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2df6e942-8414-427c-83df-928e51465964-istiod-ca-cert\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr\" (UID: \"2df6e942-8414-427c-83df-928e51465964\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr" Oct 31 04:16:53.281105 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:53.281078 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2df6e942-8414-427c-83df-928e51465964-istio-data\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr\" (UID: \"2df6e942-8414-427c-83df-928e51465964\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr" Oct 31 04:16:53.281255 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:53.281102 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2df6e942-8414-427c-83df-928e51465964-workload-certs\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr\" (UID: \"2df6e942-8414-427c-83df-928e51465964\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr" Oct 31 04:16:53.281379 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:53.281130 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2df6e942-8414-427c-83df-928e51465964-credential-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr\" (UID: \"2df6e942-8414-427c-83df-928e51465964\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr" Oct 31 04:16:53.281579 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:53.281551 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2df6e942-8414-427c-83df-928e51465964-istiod-ca-cert\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr\" (UID: \"2df6e942-8414-427c-83df-928e51465964\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr" Oct 31 04:16:53.281656 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:53.281595 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2df6e942-8414-427c-83df-928e51465964-workload-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr\" (UID: \"2df6e942-8414-427c-83df-928e51465964\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr" Oct 31 04:16:53.283219 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:53.283201 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2df6e942-8414-427c-83df-928e51465964-istio-envoy\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr\" (UID: \"2df6e942-8414-427c-83df-928e51465964\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr" Oct 31 04:16:53.283711 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:53.283665 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2df6e942-8414-427c-83df-928e51465964-istio-podinfo\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr\" (UID: \"2df6e942-8414-427c-83df-928e51465964\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr" Oct 31 04:16:53.303733 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:53.303696 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2df6e942-8414-427c-83df-928e51465964-istio-token\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr\" (UID: \"2df6e942-8414-427c-83df-928e51465964\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr" Oct 31 04:16:53.304302 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:53.304275 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fksdw\" (UniqueName: \"kubernetes.io/projected/2df6e942-8414-427c-83df-928e51465964-kube-api-access-fksdw\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr\" (UID: \"2df6e942-8414-427c-83df-928e51465964\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr" Oct 31 04:16:53.384573 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:53.384469 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr" Oct 31 04:16:53.716627 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:53.716593 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr"] Oct 31 04:16:53.744929 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:16:53.744885 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2df6e942_8414_427c_83df_928e51465964.slice/crio-0f1a20fe4e1b53dc5097c8775cbb9e935a4465e8e6af55c8f7debc5f4ee2656b WatchSource:0}: Error finding container 0f1a20fe4e1b53dc5097c8775cbb9e935a4465e8e6af55c8f7debc5f4ee2656b: Status 404 returned error can't find the container with id 0f1a20fe4e1b53dc5097c8775cbb9e935a4465e8e6af55c8f7debc5f4ee2656b Oct 31 04:16:53.747688 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:53.747626 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:16:53.747793 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:53.747730 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:16:53.747793 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:53.747762 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:16:53.799241 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:53.799203 5019 generic.go:356] "Generic (PLEG): container finished" podID="558750ea-02ff-4ff2-b126-4ebb050f9092" containerID="8f2833fd1cf37f6602e62153528a3cf256608b58cdb428203a35d1899988e445" exitCode=0 Oct 31 04:16:53.799676 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:53.799284 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-160cd22e-dh4ff" event={"ID":"558750ea-02ff-4ff2-b126-4ebb050f9092","Type":"ContainerDied","Data":"8f2833fd1cf37f6602e62153528a3cf256608b58cdb428203a35d1899988e445"} Oct 31 04:16:53.800750 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:53.800729 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e61f07-xwlzd" Oct 31 04:16:53.800891 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:53.800750 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e61f07-xwlzd" event={"ID":"5f1cb632-2629-49a0-99fe-5170142214e4","Type":"ContainerDied","Data":"9a671bd521197731cffe9856beabeda4a2d4f1d2b94f8dc7fce2881a17247d45"} Oct 31 04:16:53.800891 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:53.800801 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a671bd521197731cffe9856beabeda4a2d4f1d2b94f8dc7fce2881a17247d45" Oct 31 04:16:53.802387 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:53.802358 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr" event={"ID":"2df6e942-8414-427c-83df-928e51465964","Type":"ContainerStarted","Data":"0f1a20fe4e1b53dc5097c8775cbb9e935a4465e8e6af55c8f7debc5f4ee2656b"} Oct 31 04:16:54.812626 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:54.812589 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr" event={"ID":"2df6e942-8414-427c-83df-928e51465964","Type":"ContainerStarted","Data":"6f5f8311d71344629e668869636e95d440616d983fd51f0d949b4e823edeba0e"} Oct 31 04:16:54.862255 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:54.861629 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr" podStartSLOduration=1.861609741 podStartE2EDuration="1.861609741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:16:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:16:54.857778725 +0000 UTC m=+769.614116777" watchObservedRunningTime="2026-10-31 04:16:54.861609741 +0000 UTC m=+769.617947780" Oct 31 04:16:55.013310 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:55.013272 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-160cd22e-dh4ff" Oct 31 04:16:55.107414 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:55.107297 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/558750ea-02ff-4ff2-b126-4ebb050f9092-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/558750ea-02ff-4ff2-b126-4ebb050f9092-maas-api-openshift-ca\") pod \"558750ea-02ff-4ff2-b126-4ebb050f9092\" (UID: \"558750ea-02ff-4ff2-b126-4ebb050f9092\") " Oct 31 04:16:55.107414 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:55.107384 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/558750ea-02ff-4ff2-b126-4ebb050f9092-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/558750ea-02ff-4ff2-b126-4ebb050f9092-maas-api-xks-ca\") pod \"558750ea-02ff-4ff2-b126-4ebb050f9092\" (UID: \"558750ea-02ff-4ff2-b126-4ebb050f9092\") " Oct 31 04:16:55.107688 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:55.107510 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/558750ea-02ff-4ff2-b126-4ebb050f9092-kube-api-access-d8bhg\" (UniqueName: \"kubernetes.io/projected/558750ea-02ff-4ff2-b126-4ebb050f9092-kube-api-access-d8bhg\") pod \"558750ea-02ff-4ff2-b126-4ebb050f9092\" (UID: \"558750ea-02ff-4ff2-b126-4ebb050f9092\") " Oct 31 04:16:55.107822 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:55.107789 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/558750ea-02ff-4ff2-b126-4ebb050f9092-maas-api-openshift-ca" pod "558750ea-02ff-4ff2-b126-4ebb050f9092" (UID: "558750ea-02ff-4ff2-b126-4ebb050f9092"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:16:55.108395 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:55.108370 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/558750ea-02ff-4ff2-b126-4ebb050f9092-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:16:55.109825 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:55.109794 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/558750ea-02ff-4ff2-b126-4ebb050f9092-maas-api-xks-ca" pod "558750ea-02ff-4ff2-b126-4ebb050f9092" (UID: "558750ea-02ff-4ff2-b126-4ebb050f9092"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:16:55.110188 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:55.110154 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/558750ea-02ff-4ff2-b126-4ebb050f9092-kube-api-access-d8bhg" pod "558750ea-02ff-4ff2-b126-4ebb050f9092" (UID: "558750ea-02ff-4ff2-b126-4ebb050f9092"). InnerVolumeSpecName "kube-api-access-d8bhg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:16:55.211733 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:55.209873 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/558750ea-02ff-4ff2-b126-4ebb050f9092-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:16:55.211733 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:55.209915 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8bhg\" (UniqueName: \"kubernetes.io/projected/558750ea-02ff-4ff2-b126-4ebb050f9092-kube-api-access-d8bhg\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:16:55.391423 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:55.388409 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr" Oct 31 04:16:55.394357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:55.393479 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 31 04:16:55.394357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:55.393546 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr" podUID="2df6e942-8414-427c-83df-928e51465964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 31 04:16:55.823231 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:55.823204 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-160cd22e-dh4ff" Oct 31 04:16:55.823766 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:55.823211 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-160cd22e-dh4ff" event={"ID":"558750ea-02ff-4ff2-b126-4ebb050f9092","Type":"ContainerDied","Data":"7da0f067fe266775ab71f01fababca68ac83d901928470d70f0bd48d6a6a83a4"} Oct 31 04:16:55.823766 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:55.823482 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7da0f067fe266775ab71f01fababca68ac83d901928470d70f0bd48d6a6a83a4" Oct 31 04:16:56.390357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:56.389822 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 31 04:16:56.390357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:56.389914 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr" podUID="2df6e942-8414-427c-83df-928e51465964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 31 04:16:57.388358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:57.387803 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 31 04:16:57.388358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:57.387892 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr" podUID="2df6e942-8414-427c-83df-928e51465964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 31 04:16:57.910594 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:57.910552 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea662224-dshjp"] Oct 31 04:16:57.911197 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:57.911167 5019 cpu_manager.go:414] "containerMap: removing container" podUID="558750ea-02ff-4ff2-b126-4ebb050f9092" containerName="revoke-keys" Oct 31 04:16:57.911197 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:57.911185 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="558750ea-02ff-4ff2-b126-4ebb050f9092" containerName="revoke-keys" Oct 31 04:16:57.911424 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:57.911262 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="558750ea-02ff-4ff2-b126-4ebb050f9092" containerName="revoke-keys" Oct 31 04:16:57.915828 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:57.915804 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea662224-dshjp" Oct 31 04:16:57.934056 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:57.934018 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea662224-dshjp"] Oct 31 04:16:57.942295 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:57.942264 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:16:57.942295 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:57.942289 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:16:57.942551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:57.942347 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:16:57.942551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:57.942432 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:16:57.944157 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:57.944128 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e646830-d7f6-4e36-845d-f910078d4864-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea662224-dshjp\" (UID: \"1e646830-d7f6-4e36-845d-f910078d4864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea662224-dshjp" Oct 31 04:16:57.944317 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:57.944186 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e646830-d7f6-4e36-845d-f910078d4864-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea662224-dshjp\" (UID: \"1e646830-d7f6-4e36-845d-f910078d4864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea662224-dshjp" Oct 31 04:16:57.944410 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:57.944359 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkfcx\" (UniqueName: \"kubernetes.io/projected/1e646830-d7f6-4e36-845d-f910078d4864-kube-api-access-wkfcx\") pod \"maas-api-revoke-sub-ea662224-dshjp\" (UID: \"1e646830-d7f6-4e36-845d-f910078d4864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea662224-dshjp" Oct 31 04:16:58.045428 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:58.045393 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkfcx\" (UniqueName: \"kubernetes.io/projected/1e646830-d7f6-4e36-845d-f910078d4864-kube-api-access-wkfcx\") pod \"maas-api-revoke-sub-ea662224-dshjp\" (UID: \"1e646830-d7f6-4e36-845d-f910078d4864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea662224-dshjp" Oct 31 04:16:58.045793 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:58.045770 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e646830-d7f6-4e36-845d-f910078d4864-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea662224-dshjp\" (UID: \"1e646830-d7f6-4e36-845d-f910078d4864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea662224-dshjp" Oct 31 04:16:58.045984 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:58.045967 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e646830-d7f6-4e36-845d-f910078d4864-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea662224-dshjp\" (UID: \"1e646830-d7f6-4e36-845d-f910078d4864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea662224-dshjp" Oct 31 04:16:58.046691 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:58.046670 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e646830-d7f6-4e36-845d-f910078d4864-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea662224-dshjp\" (UID: \"1e646830-d7f6-4e36-845d-f910078d4864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea662224-dshjp" Oct 31 04:16:58.053388 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:58.048394 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e646830-d7f6-4e36-845d-f910078d4864-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea662224-dshjp\" (UID: \"1e646830-d7f6-4e36-845d-f910078d4864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea662224-dshjp" Oct 31 04:16:58.108355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:58.105967 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkfcx\" (UniqueName: \"kubernetes.io/projected/1e646830-d7f6-4e36-845d-f910078d4864-kube-api-access-wkfcx\") pod \"maas-api-revoke-sub-ea662224-dshjp\" (UID: \"1e646830-d7f6-4e36-845d-f910078d4864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea662224-dshjp" Oct 31 04:16:58.229954 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:58.229914 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea662224-dshjp" Oct 31 04:16:58.394358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:58.388771 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 31 04:16:58.394358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:58.388886 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr" podUID="2df6e942-8414-427c-83df-928e51465964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 31 04:16:58.587359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:58.584668 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea662224-dshjp"] Oct 31 04:16:58.602734 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:16:58.594653 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e646830_d7f6_4e36_845d_f910078d4864.slice/crio-10170557524c2a1a2eb9b23728f1d1798fed6f61047ebaef2a1f32e174b3709d WatchSource:0}: Error finding container 10170557524c2a1a2eb9b23728f1d1798fed6f61047ebaef2a1f32e174b3709d: Status 404 returned error can't find the container with id 10170557524c2a1a2eb9b23728f1d1798fed6f61047ebaef2a1f32e174b3709d Oct 31 04:16:58.845986 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:58.845891 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea662224-dshjp" event={"ID":"1e646830-d7f6-4e36-845d-f910078d4864","Type":"ContainerStarted","Data":"10170557524c2a1a2eb9b23728f1d1798fed6f61047ebaef2a1f32e174b3709d"} Oct 31 04:16:59.392360 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:59.389837 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 31 04:16:59.392360 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:59.389922 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr" podUID="2df6e942-8414-427c-83df-928e51465964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 31 04:16:59.854554 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:59.854513 5019 generic.go:356] "Generic (PLEG): container finished" podID="1e646830-d7f6-4e36-845d-f910078d4864" containerID="47a9b26998e5b46c6a8e0fb1f0f1b4ade97e8a1978216ddd9b911a4e81520bb7" exitCode=0 Oct 31 04:16:59.854974 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:16:59.854608 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea662224-dshjp" event={"ID":"1e646830-d7f6-4e36-845d-f910078d4864","Type":"ContainerDied","Data":"47a9b26998e5b46c6a8e0fb1f0f1b4ade97e8a1978216ddd9b911a4e81520bb7"} Oct 31 04:17:00.639319 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:00.639254 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr" Oct 31 04:17:00.639733 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:00.639712 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr" Oct 31 04:17:00.641127 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:00.641105 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr" Oct 31 04:17:01.088510 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:01.088472 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea662224-dshjp" Oct 31 04:17:01.191391 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:01.191348 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e646830-d7f6-4e36-845d-f910078d4864-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e646830-d7f6-4e36-845d-f910078d4864-maas-api-xks-ca\") pod \"1e646830-d7f6-4e36-845d-f910078d4864\" (UID: \"1e646830-d7f6-4e36-845d-f910078d4864\") " Oct 31 04:17:01.191611 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:01.191405 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e646830-d7f6-4e36-845d-f910078d4864-kube-api-access-wkfcx\" (UniqueName: \"kubernetes.io/projected/1e646830-d7f6-4e36-845d-f910078d4864-kube-api-access-wkfcx\") pod \"1e646830-d7f6-4e36-845d-f910078d4864\" (UID: \"1e646830-d7f6-4e36-845d-f910078d4864\") " Oct 31 04:17:01.191611 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:01.191603 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e646830-d7f6-4e36-845d-f910078d4864-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e646830-d7f6-4e36-845d-f910078d4864-maas-api-openshift-ca\") pod \"1e646830-d7f6-4e36-845d-f910078d4864\" (UID: \"1e646830-d7f6-4e36-845d-f910078d4864\") " Oct 31 04:17:01.192022 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:01.191993 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e646830-d7f6-4e36-845d-f910078d4864-maas-api-openshift-ca" pod "1e646830-d7f6-4e36-845d-f910078d4864" (UID: "1e646830-d7f6-4e36-845d-f910078d4864"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:17:01.192179 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:01.192154 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e646830-d7f6-4e36-845d-f910078d4864-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:17:01.193755 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:01.193719 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e646830-d7f6-4e36-845d-f910078d4864-maas-api-xks-ca" pod "1e646830-d7f6-4e36-845d-f910078d4864" (UID: "1e646830-d7f6-4e36-845d-f910078d4864"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:17:01.193891 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:01.193830 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e646830-d7f6-4e36-845d-f910078d4864-kube-api-access-wkfcx" pod "1e646830-d7f6-4e36-845d-f910078d4864" (UID: "1e646830-d7f6-4e36-845d-f910078d4864"). InnerVolumeSpecName "kube-api-access-wkfcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:17:01.294157 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:01.294043 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e646830-d7f6-4e36-845d-f910078d4864-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:17:01.294157 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:01.294092 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkfcx\" (UniqueName: \"kubernetes.io/projected/1e646830-d7f6-4e36-845d-f910078d4864-kube-api-access-wkfcx\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:17:01.869407 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:01.869372 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea662224-dshjp" Oct 31 04:17:01.869622 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:01.869369 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea662224-dshjp" event={"ID":"1e646830-d7f6-4e36-845d-f910078d4864","Type":"ContainerDied","Data":"10170557524c2a1a2eb9b23728f1d1798fed6f61047ebaef2a1f32e174b3709d"} Oct 31 04:17:01.869622 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:01.869559 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10170557524c2a1a2eb9b23728f1d1798fed6f61047ebaef2a1f32e174b3709d" Oct 31 04:17:08.281378 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:08.281326 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cefbe68-t88hv"] Oct 31 04:17:08.281900 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:08.281885 5019 cpu_manager.go:414] "containerMap: removing container" podUID="1e646830-d7f6-4e36-845d-f910078d4864" containerName="revoke-keys" Oct 31 04:17:08.281940 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:08.281901 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e646830-d7f6-4e36-845d-f910078d4864" containerName="revoke-keys" Oct 31 04:17:08.281995 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:08.281986 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e646830-d7f6-4e36-845d-f910078d4864" containerName="revoke-keys" Oct 31 04:17:08.286859 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:08.286829 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cefbe68-t88hv" Oct 31 04:17:08.305495 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:08.305462 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cefbe68-t88hv"] Oct 31 04:17:08.313682 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:08.313649 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:17:08.313926 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:08.313660 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:17:08.313926 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:08.313671 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:17:08.314111 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:08.313727 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:17:08.399033 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:08.398985 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9608436-efc1-4cef-b9a0-55ba08255e6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4cefbe68-t88hv\" (UID: \"c9608436-efc1-4cef-b9a0-55ba08255e6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cefbe68-t88hv" Oct 31 04:17:08.399033 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:08.399044 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bw5w\" (UniqueName: \"kubernetes.io/projected/c9608436-efc1-4cef-b9a0-55ba08255e6b-kube-api-access-6bw5w\") pod \"maas-api-revoke-sub-4cefbe68-t88hv\" (UID: \"c9608436-efc1-4cef-b9a0-55ba08255e6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cefbe68-t88hv" Oct 31 04:17:08.399310 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:08.399160 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9608436-efc1-4cef-b9a0-55ba08255e6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4cefbe68-t88hv\" (UID: \"c9608436-efc1-4cef-b9a0-55ba08255e6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cefbe68-t88hv" Oct 31 04:17:08.500964 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:08.500917 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9608436-efc1-4cef-b9a0-55ba08255e6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4cefbe68-t88hv\" (UID: \"c9608436-efc1-4cef-b9a0-55ba08255e6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cefbe68-t88hv" Oct 31 04:17:08.501187 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:08.500990 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bw5w\" (UniqueName: \"kubernetes.io/projected/c9608436-efc1-4cef-b9a0-55ba08255e6b-kube-api-access-6bw5w\") pod \"maas-api-revoke-sub-4cefbe68-t88hv\" (UID: \"c9608436-efc1-4cef-b9a0-55ba08255e6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cefbe68-t88hv" Oct 31 04:17:08.501187 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:08.501058 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9608436-efc1-4cef-b9a0-55ba08255e6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4cefbe68-t88hv\" (UID: \"c9608436-efc1-4cef-b9a0-55ba08255e6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cefbe68-t88hv" Oct 31 04:17:08.501853 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:08.501829 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9608436-efc1-4cef-b9a0-55ba08255e6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4cefbe68-t88hv\" (UID: \"c9608436-efc1-4cef-b9a0-55ba08255e6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cefbe68-t88hv" Oct 31 04:17:08.503621 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:08.503593 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9608436-efc1-4cef-b9a0-55ba08255e6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4cefbe68-t88hv\" (UID: \"c9608436-efc1-4cef-b9a0-55ba08255e6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cefbe68-t88hv" Oct 31 04:17:08.524458 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:08.524429 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bw5w\" (UniqueName: \"kubernetes.io/projected/c9608436-efc1-4cef-b9a0-55ba08255e6b-kube-api-access-6bw5w\") pod \"maas-api-revoke-sub-4cefbe68-t88hv\" (UID: \"c9608436-efc1-4cef-b9a0-55ba08255e6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cefbe68-t88hv" Oct 31 04:17:08.601984 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:08.601883 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cefbe68-t88hv" Oct 31 04:17:08.911759 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:17:08.911718 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9608436_efc1_4cef_b9a0_55ba08255e6b.slice/crio-6bdeb2d04cbbc485c9c920cdf4b9f1827b3543bde72ee7e11d7522b33030192d WatchSource:0}: Error finding container 6bdeb2d04cbbc485c9c920cdf4b9f1827b3543bde72ee7e11d7522b33030192d: Status 404 returned error can't find the container with id 6bdeb2d04cbbc485c9c920cdf4b9f1827b3543bde72ee7e11d7522b33030192d Oct 31 04:17:08.911936 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:08.911855 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cefbe68-t88hv"] Oct 31 04:17:08.945381 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:08.945312 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cefbe68-t88hv" event={"ID":"c9608436-efc1-4cef-b9a0-55ba08255e6b","Type":"ContainerStarted","Data":"6bdeb2d04cbbc485c9c920cdf4b9f1827b3543bde72ee7e11d7522b33030192d"} Oct 31 04:17:09.955707 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:09.955612 5019 generic.go:356] "Generic (PLEG): container finished" podID="c9608436-efc1-4cef-b9a0-55ba08255e6b" containerID="58b3e4ed32ff3e64514c6cca9cb584fddb7962af9d41624bb383aa8a0859855f" exitCode=0 Oct 31 04:17:09.955707 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:09.955693 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cefbe68-t88hv" event={"ID":"c9608436-efc1-4cef-b9a0-55ba08255e6b","Type":"ContainerDied","Data":"58b3e4ed32ff3e64514c6cca9cb584fddb7962af9d41624bb383aa8a0859855f"} Oct 31 04:17:11.188400 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:11.182560 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cefbe68-t88hv" Oct 31 04:17:11.263168 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:11.263123 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9608436-efc1-4cef-b9a0-55ba08255e6b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9608436-efc1-4cef-b9a0-55ba08255e6b-maas-api-openshift-ca\") pod \"c9608436-efc1-4cef-b9a0-55ba08255e6b\" (UID: \"c9608436-efc1-4cef-b9a0-55ba08255e6b\") " Oct 31 04:17:11.267119 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:11.267078 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9608436-efc1-4cef-b9a0-55ba08255e6b-kube-api-access-6bw5w\" (UniqueName: \"kubernetes.io/projected/c9608436-efc1-4cef-b9a0-55ba08255e6b-kube-api-access-6bw5w\") pod \"c9608436-efc1-4cef-b9a0-55ba08255e6b\" (UID: \"c9608436-efc1-4cef-b9a0-55ba08255e6b\") " Oct 31 04:17:11.269314 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:11.267713 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9608436-efc1-4cef-b9a0-55ba08255e6b-maas-api-openshift-ca" pod "c9608436-efc1-4cef-b9a0-55ba08255e6b" (UID: "c9608436-efc1-4cef-b9a0-55ba08255e6b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:17:11.271324 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:11.269431 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9608436-efc1-4cef-b9a0-55ba08255e6b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9608436-efc1-4cef-b9a0-55ba08255e6b-maas-api-xks-ca\") pod \"c9608436-efc1-4cef-b9a0-55ba08255e6b\" (UID: \"c9608436-efc1-4cef-b9a0-55ba08255e6b\") " Oct 31 04:17:11.277978 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:11.277916 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9608436-efc1-4cef-b9a0-55ba08255e6b-maas-api-xks-ca" pod "c9608436-efc1-4cef-b9a0-55ba08255e6b" (UID: "c9608436-efc1-4cef-b9a0-55ba08255e6b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:17:11.278302 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:11.278266 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9608436-efc1-4cef-b9a0-55ba08255e6b-kube-api-access-6bw5w" pod "c9608436-efc1-4cef-b9a0-55ba08255e6b" (UID: "c9608436-efc1-4cef-b9a0-55ba08255e6b"). InnerVolumeSpecName "kube-api-access-6bw5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:17:11.373248 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:11.373192 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9608436-efc1-4cef-b9a0-55ba08255e6b-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:17:11.373248 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:11.373241 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bw5w\" (UniqueName: \"kubernetes.io/projected/c9608436-efc1-4cef-b9a0-55ba08255e6b-kube-api-access-6bw5w\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:17:11.373594 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:11.373260 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9608436-efc1-4cef-b9a0-55ba08255e6b-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:17:11.973709 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:11.973668 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cefbe68-t88hv" Oct 31 04:17:11.973709 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:11.973684 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cefbe68-t88hv" event={"ID":"c9608436-efc1-4cef-b9a0-55ba08255e6b","Type":"ContainerDied","Data":"6bdeb2d04cbbc485c9c920cdf4b9f1827b3543bde72ee7e11d7522b33030192d"} Oct 31 04:17:11.973995 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:11.973741 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bdeb2d04cbbc485c9c920cdf4b9f1827b3543bde72ee7e11d7522b33030192d" Oct 31 04:17:22.467068 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:17:22.467016 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:17:24.375586 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:24.375532 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd"] Oct 31 04:17:24.376185 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:24.376170 5019 cpu_manager.go:414] "containerMap: removing container" podUID="c9608436-efc1-4cef-b9a0-55ba08255e6b" containerName="revoke-keys" Oct 31 04:17:24.376235 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:24.376187 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9608436-efc1-4cef-b9a0-55ba08255e6b" containerName="revoke-keys" Oct 31 04:17:24.376303 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:24.376294 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9608436-efc1-4cef-b9a0-55ba08255e6b" containerName="revoke-keys" Oct 31 04:17:24.379935 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:24.379905 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd" Oct 31 04:17:24.416238 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:24.416197 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd"] Oct 31 04:17:24.461745 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:24.461691 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-b4nmd\" (UID: \"d4062db4-f8d8-4cbb-8f2d-428c06d30db7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd" Oct 31 04:17:24.461745 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:24.461749 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-b4nmd\" (UID: \"d4062db4-f8d8-4cbb-8f2d-428c06d30db7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd" Oct 31 04:17:24.461991 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:24.461912 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-b4nmd\" (UID: \"d4062db4-f8d8-4cbb-8f2d-428c06d30db7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd" Oct 31 04:17:24.461991 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:24.461950 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-b4nmd\" (UID: \"d4062db4-f8d8-4cbb-8f2d-428c06d30db7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd" Oct 31 04:17:24.461991 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:24.461966 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-b4nmd\" (UID: \"d4062db4-f8d8-4cbb-8f2d-428c06d30db7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd" Oct 31 04:17:24.462170 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:24.462144 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qrr6\" (UniqueName: \"kubernetes.io/projected/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-kube-api-access-8qrr6\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-b4nmd\" (UID: \"d4062db4-f8d8-4cbb-8f2d-428c06d30db7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd" Oct 31 04:17:24.462230 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:24.462192 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-b4nmd\" (UID: \"d4062db4-f8d8-4cbb-8f2d-428c06d30db7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd" Oct 31 04:17:24.462303 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:24.462282 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-b4nmd\" (UID: \"d4062db4-f8d8-4cbb-8f2d-428c06d30db7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd" Oct 31 04:17:24.462402 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:24.462384 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-b4nmd\" (UID: \"d4062db4-f8d8-4cbb-8f2d-428c06d30db7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd" Oct 31 04:17:24.546916 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:24.546878 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-57567b5c87-8r7nx"] Oct 31 04:17:24.551219 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:24.551191 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-57567b5c87-8r7nx" Oct 31 04:17:24.563732 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:24.563698 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-b4nmd\" (UID: \"d4062db4-f8d8-4cbb-8f2d-428c06d30db7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd" Oct 31 04:17:24.563912 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:24.563739 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/185e60a3-6578-4f3d-a45a-cca447bd5071-provider-credentials\") pod \"payload-processing-external-model-57567b5c87-8r7nx\" (UID: \"185e60a3-6578-4f3d-a45a-cca447bd5071\") " pod="models-as-a-service/payload-processing-external-model-57567b5c87-8r7nx" Oct 31 04:17:24.563912 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:24.563836 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/185e60a3-6578-4f3d-a45a-cca447bd5071-routing-overlay-volume\") pod \"payload-processing-external-model-57567b5c87-8r7nx\" (UID: \"185e60a3-6578-4f3d-a45a-cca447bd5071\") " pod="models-as-a-service/payload-processing-external-model-57567b5c87-8r7nx" Oct 31 04:17:24.564095 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:24.563921 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-b4nmd\" (UID: \"d4062db4-f8d8-4cbb-8f2d-428c06d30db7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd" Oct 31 04:17:24.564095 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:24.563961 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzhk4\" (UniqueName: \"kubernetes.io/projected/185e60a3-6578-4f3d-a45a-cca447bd5071-kube-api-access-tzhk4\") pod \"payload-processing-external-model-57567b5c87-8r7nx\" (UID: \"185e60a3-6578-4f3d-a45a-cca447bd5071\") " pod="models-as-a-service/payload-processing-external-model-57567b5c87-8r7nx" Oct 31 04:17:24.564095 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:24.564001 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-b4nmd\" (UID: \"d4062db4-f8d8-4cbb-8f2d-428c06d30db7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd" Oct 31 04:17:24.564095 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:24.564028 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-b4nmd\" (UID: \"d4062db4-f8d8-4cbb-8f2d-428c06d30db7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd" Oct 31 04:17:24.564095 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:24.564079 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/185e60a3-6578-4f3d-a45a-cca447bd5071-plugins-config-volume\") pod \"payload-processing-external-model-57567b5c87-8r7nx\" (UID: \"185e60a3-6578-4f3d-a45a-cca447bd5071\") " pod="models-as-a-service/payload-processing-external-model-57567b5c87-8r7nx" Oct 31 04:17:24.564251 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:24.564132 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-b4nmd\" (UID: \"d4062db4-f8d8-4cbb-8f2d-428c06d30db7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd" Oct 31 04:17:24.564251 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:24.564153 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-b4nmd\" (UID: \"d4062db4-f8d8-4cbb-8f2d-428c06d30db7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd" Oct 31 04:17:24.564251 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:24.564167 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-b4nmd\" (UID: \"d4062db4-f8d8-4cbb-8f2d-428c06d30db7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd" Oct 31 04:17:24.564251 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:24.564228 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qrr6\" (UniqueName: \"kubernetes.io/projected/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-kube-api-access-8qrr6\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-b4nmd\" (UID: \"d4062db4-f8d8-4cbb-8f2d-428c06d30db7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd" Oct 31 04:17:24.564251 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:24.564246 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-b4nmd\" (UID: \"d4062db4-f8d8-4cbb-8f2d-428c06d30db7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd" Oct 31 04:17:24.565287 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:24.564682 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-b4nmd\" (UID: \"d4062db4-f8d8-4cbb-8f2d-428c06d30db7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd" Oct 31 04:17:24.565287 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:24.564892 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-b4nmd\" (UID: \"d4062db4-f8d8-4cbb-8f2d-428c06d30db7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd" Oct 31 04:17:24.565287 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:24.564980 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-b4nmd\" (UID: \"d4062db4-f8d8-4cbb-8f2d-428c06d30db7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd" Oct 31 04:17:24.565287 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:24.565113 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-b4nmd\" (UID: \"d4062db4-f8d8-4cbb-8f2d-428c06d30db7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd" Oct 31 04:17:24.565287 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:24.565139 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-b4nmd\" (UID: \"d4062db4-f8d8-4cbb-8f2d-428c06d30db7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd" Oct 31 04:17:24.567169 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:24.567141 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-b4nmd\" (UID: \"d4062db4-f8d8-4cbb-8f2d-428c06d30db7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd" Oct 31 04:17:24.567324 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:24.567273 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 31 04:17:24.568078 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:24.567995 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-b4nmd\" (UID: \"d4062db4-f8d8-4cbb-8f2d-428c06d30db7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd" Oct 31 04:17:24.572349 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:24.572308 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-57567b5c87-8r7nx"] Oct 31 04:17:24.572664 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:24.572641 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-4f55842b7e8e\"" Oct 31 04:17:24.572829 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:24.572641 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 31 04:17:24.572944 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:24.572657 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 31 04:17:24.572944 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:24.572719 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-5qlm5\"" Oct 31 04:17:24.587962 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:24.587926 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-b4nmd\" (UID: \"d4062db4-f8d8-4cbb-8f2d-428c06d30db7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd" Oct 31 04:17:24.588119 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:24.588058 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qrr6\" (UniqueName: \"kubernetes.io/projected/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-kube-api-access-8qrr6\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-b4nmd\" (UID: \"d4062db4-f8d8-4cbb-8f2d-428c06d30db7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd" Oct 31 04:17:24.665277 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:24.665194 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 31 04:17:24.665504 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:24.665482 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/185e60a3-6578-4f3d-a45a-cca447bd5071-provider-credentials\") pod \"payload-processing-external-model-57567b5c87-8r7nx\" (UID: \"185e60a3-6578-4f3d-a45a-cca447bd5071\") " pod="models-as-a-service/payload-processing-external-model-57567b5c87-8r7nx" Oct 31 04:17:24.665554 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:24.665521 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/185e60a3-6578-4f3d-a45a-cca447bd5071-routing-overlay-volume\") pod \"payload-processing-external-model-57567b5c87-8r7nx\" (UID: \"185e60a3-6578-4f3d-a45a-cca447bd5071\") " pod="models-as-a-service/payload-processing-external-model-57567b5c87-8r7nx" Oct 31 04:17:24.665594 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:24.665564 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzhk4\" (UniqueName: \"kubernetes.io/projected/185e60a3-6578-4f3d-a45a-cca447bd5071-kube-api-access-tzhk4\") pod \"payload-processing-external-model-57567b5c87-8r7nx\" (UID: \"185e60a3-6578-4f3d-a45a-cca447bd5071\") " pod="models-as-a-service/payload-processing-external-model-57567b5c87-8r7nx" Oct 31 04:17:24.665665 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:24.665647 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/185e60a3-6578-4f3d-a45a-cca447bd5071-plugins-config-volume\") pod \"payload-processing-external-model-57567b5c87-8r7nx\" (UID: \"185e60a3-6578-4f3d-a45a-cca447bd5071\") " pod="models-as-a-service/payload-processing-external-model-57567b5c87-8r7nx" Oct 31 04:17:24.666219 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:24.666198 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/185e60a3-6578-4f3d-a45a-cca447bd5071-routing-overlay-volume\") pod \"payload-processing-external-model-57567b5c87-8r7nx\" (UID: \"185e60a3-6578-4f3d-a45a-cca447bd5071\") " pod="models-as-a-service/payload-processing-external-model-57567b5c87-8r7nx" Oct 31 04:17:24.666459 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:24.666436 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/185e60a3-6578-4f3d-a45a-cca447bd5071-plugins-config-volume\") pod \"payload-processing-external-model-57567b5c87-8r7nx\" (UID: \"185e60a3-6578-4f3d-a45a-cca447bd5071\") " pod="models-as-a-service/payload-processing-external-model-57567b5c87-8r7nx" Oct 31 04:17:24.668404 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:24.668376 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/185e60a3-6578-4f3d-a45a-cca447bd5071-provider-credentials\") pod \"payload-processing-external-model-57567b5c87-8r7nx\" (UID: \"185e60a3-6578-4f3d-a45a-cca447bd5071\") " pod="models-as-a-service/payload-processing-external-model-57567b5c87-8r7nx" Oct 31 04:17:24.688609 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:24.688576 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzhk4\" (UniqueName: \"kubernetes.io/projected/185e60a3-6578-4f3d-a45a-cca447bd5071-kube-api-access-tzhk4\") pod \"payload-processing-external-model-57567b5c87-8r7nx\" (UID: \"185e60a3-6578-4f3d-a45a-cca447bd5071\") " pod="models-as-a-service/payload-processing-external-model-57567b5c87-8r7nx" Oct 31 04:17:24.698551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:24.698514 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd" Oct 31 04:17:24.866272 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:24.866227 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-57567b5c87-8r7nx" Oct 31 04:17:25.026838 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:25.026802 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd"] Oct 31 04:17:25.027925 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:17:25.027891 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4062db4_f8d8_4cbb_8f2d_428c06d30db7.slice/crio-4a1ca431a8623371f11770acb01da89be78c911740acd76355df09a172bf8232 WatchSource:0}: Error finding container 4a1ca431a8623371f11770acb01da89be78c911740acd76355df09a172bf8232: Status 404 returned error can't find the container with id 4a1ca431a8623371f11770acb01da89be78c911740acd76355df09a172bf8232 Oct 31 04:17:25.030390 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:25.030353 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:17:25.030493 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:25.030426 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:17:25.030493 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:25.030457 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:17:25.125229 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:25.125195 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd" event={"ID":"d4062db4-f8d8-4cbb-8f2d-428c06d30db7","Type":"ContainerStarted","Data":"4a1ca431a8623371f11770acb01da89be78c911740acd76355df09a172bf8232"} Oct 31 04:17:25.181463 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:25.181392 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-57567b5c87-8r7nx"] Oct 31 04:17:25.182897 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:17:25.182866 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod185e60a3_6578_4f3d_a45a_cca447bd5071.slice/crio-ad0c3667fc41b5a6d265804464cba3fb1bd4cbdf3b89b9e53163d0df2064f0c0 WatchSource:0}: Error finding container ad0c3667fc41b5a6d265804464cba3fb1bd4cbdf3b89b9e53163d0df2064f0c0: Status 404 returned error can't find the container with id ad0c3667fc41b5a6d265804464cba3fb1bd4cbdf3b89b9e53163d0df2064f0c0 Oct 31 04:17:26.134913 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:26.134870 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd" event={"ID":"d4062db4-f8d8-4cbb-8f2d-428c06d30db7","Type":"ContainerStarted","Data":"16def1431858b071d0663d1233f8aa1864e13c85002f304693ecb8a306b6ad09"} Oct 31 04:17:26.136940 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:26.136912 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-57567b5c87-8r7nx" event={"ID":"185e60a3-6578-4f3d-a45a-cca447bd5071","Type":"ContainerStarted","Data":"e13d3dee48a2ac2dcc6734bbd4def7883d208ee8b68ace5624c0be3133340840"} Oct 31 04:17:26.137072 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:26.136961 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-57567b5c87-8r7nx" event={"ID":"185e60a3-6578-4f3d-a45a-cca447bd5071","Type":"ContainerStarted","Data":"ad0c3667fc41b5a6d265804464cba3fb1bd4cbdf3b89b9e53163d0df2064f0c0"} Oct 31 04:17:26.137072 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:26.136982 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-57567b5c87-8r7nx" Oct 31 04:17:26.204160 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:26.204095 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd" podStartSLOduration=2.204078656 podStartE2EDuration="2.204078656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:17:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:17:26.202596274 +0000 UTC m=+800.958934317" watchObservedRunningTime="2026-10-31 04:17:26.204078656 +0000 UTC m=+800.960416696" Oct 31 04:17:26.246439 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:26.246380 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-57567b5c87-8r7nx" podStartSLOduration=2.246360016 podStartE2EDuration="2.246360016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:17:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:17:26.242461203 +0000 UTC m=+800.998799242" watchObservedRunningTime="2026-10-31 04:17:26.246360016 +0000 UTC m=+801.002698054" Oct 31 04:17:26.699511 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:26.699458 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd" Oct 31 04:17:26.701349 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:26.701299 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 31 04:17:26.701522 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:26.701376 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd" podUID="d4062db4-f8d8-4cbb-8f2d-428c06d30db7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 31 04:17:27.700358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:27.699474 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 31 04:17:27.700358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:27.699555 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd" podUID="d4062db4-f8d8-4cbb-8f2d-428c06d30db7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 31 04:17:28.699663 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:28.699609 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 31 04:17:28.699866 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:28.699699 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd" podUID="d4062db4-f8d8-4cbb-8f2d-428c06d30db7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 31 04:17:29.701356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:29.699805 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 31 04:17:29.701356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:29.699878 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd" podUID="d4062db4-f8d8-4cbb-8f2d-428c06d30db7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 31 04:17:30.700357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:30.699771 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 31 04:17:30.700357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:30.699838 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd" podUID="d4062db4-f8d8-4cbb-8f2d-428c06d30db7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 31 04:17:31.699175 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:31.699130 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 31 04:17:31.699612 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:31.699202 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd" podUID="d4062db4-f8d8-4cbb-8f2d-428c06d30db7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 31 04:17:32.700358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:32.699855 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 31 04:17:32.700358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:32.699917 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd" podUID="d4062db4-f8d8-4cbb-8f2d-428c06d30db7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 31 04:17:34.266356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:34.265845 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj"] Oct 31 04:17:34.319360 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:34.312765 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj" Oct 31 04:17:34.353727 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:34.353681 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-439989-openshift-default-dockercfg-9247r\"" Oct 31 04:17:34.418490 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:34.416880 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj"] Oct 31 04:17:34.467646 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:34.467589 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1453873b-b35e-40f3-8c37-ad757223b110-istio-podinfo\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj\" (UID: \"1453873b-b35e-40f3-8c37-ad757223b110\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj" Oct 31 04:17:34.467969 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:34.467929 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1453873b-b35e-40f3-8c37-ad757223b110-istio-data\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj\" (UID: \"1453873b-b35e-40f3-8c37-ad757223b110\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj" Oct 31 04:17:34.468201 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:34.468181 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1453873b-b35e-40f3-8c37-ad757223b110-istiod-ca-cert\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj\" (UID: \"1453873b-b35e-40f3-8c37-ad757223b110\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj" Oct 31 04:17:34.468453 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:34.468433 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1453873b-b35e-40f3-8c37-ad757223b110-istio-token\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj\" (UID: \"1453873b-b35e-40f3-8c37-ad757223b110\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj" Oct 31 04:17:34.468643 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:34.468593 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1453873b-b35e-40f3-8c37-ad757223b110-workload-certs\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj\" (UID: \"1453873b-b35e-40f3-8c37-ad757223b110\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj" Oct 31 04:17:34.468818 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:34.468775 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1453873b-b35e-40f3-8c37-ad757223b110-workload-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj\" (UID: \"1453873b-b35e-40f3-8c37-ad757223b110\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj" Oct 31 04:17:34.469002 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:34.468972 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1453873b-b35e-40f3-8c37-ad757223b110-istio-envoy\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj\" (UID: \"1453873b-b35e-40f3-8c37-ad757223b110\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj" Oct 31 04:17:34.469267 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:34.469162 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cccg5\" (UniqueName: \"kubernetes.io/projected/1453873b-b35e-40f3-8c37-ad757223b110-kube-api-access-cccg5\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj\" (UID: \"1453873b-b35e-40f3-8c37-ad757223b110\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj" Oct 31 04:17:34.476466 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:34.469426 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1453873b-b35e-40f3-8c37-ad757223b110-credential-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj\" (UID: \"1453873b-b35e-40f3-8c37-ad757223b110\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj" Oct 31 04:17:34.586020 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:34.585921 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1453873b-b35e-40f3-8c37-ad757223b110-istio-token\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj\" (UID: \"1453873b-b35e-40f3-8c37-ad757223b110\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj" Oct 31 04:17:34.586749 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:34.586264 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" podUID="643433d8-418e-4102-8bc3-cd544932614a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:17:34.588700 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:34.588672 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1453873b-b35e-40f3-8c37-ad757223b110-workload-certs\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj\" (UID: \"1453873b-b35e-40f3-8c37-ad757223b110\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj" Oct 31 04:17:34.588909 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:34.586712 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1453873b-b35e-40f3-8c37-ad757223b110-workload-certs\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj\" (UID: \"1453873b-b35e-40f3-8c37-ad757223b110\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj" Oct 31 04:17:34.589069 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:34.589052 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1453873b-b35e-40f3-8c37-ad757223b110-workload-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj\" (UID: \"1453873b-b35e-40f3-8c37-ad757223b110\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj" Oct 31 04:17:34.589551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:34.589533 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1453873b-b35e-40f3-8c37-ad757223b110-istio-envoy\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj\" (UID: \"1453873b-b35e-40f3-8c37-ad757223b110\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj" Oct 31 04:17:34.599776 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:34.599742 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cccg5\" (UniqueName: \"kubernetes.io/projected/1453873b-b35e-40f3-8c37-ad757223b110-kube-api-access-cccg5\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj\" (UID: \"1453873b-b35e-40f3-8c37-ad757223b110\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj" Oct 31 04:17:34.600411 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:34.599622 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1453873b-b35e-40f3-8c37-ad757223b110-istio-envoy\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj\" (UID: \"1453873b-b35e-40f3-8c37-ad757223b110\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj" Oct 31 04:17:34.600612 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:34.589441 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1453873b-b35e-40f3-8c37-ad757223b110-workload-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj\" (UID: \"1453873b-b35e-40f3-8c37-ad757223b110\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj" Oct 31 04:17:34.600736 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:34.600709 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1453873b-b35e-40f3-8c37-ad757223b110-credential-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj\" (UID: \"1453873b-b35e-40f3-8c37-ad757223b110\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj" Oct 31 04:17:34.600902 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:34.600400 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1453873b-b35e-40f3-8c37-ad757223b110-credential-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj\" (UID: \"1453873b-b35e-40f3-8c37-ad757223b110\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj" Oct 31 04:17:34.601136 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:34.601117 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1453873b-b35e-40f3-8c37-ad757223b110-istio-podinfo\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj\" (UID: \"1453873b-b35e-40f3-8c37-ad757223b110\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj" Oct 31 04:17:34.601760 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:34.601741 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1453873b-b35e-40f3-8c37-ad757223b110-istio-data\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj\" (UID: \"1453873b-b35e-40f3-8c37-ad757223b110\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj" Oct 31 04:17:34.602257 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:34.602137 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1453873b-b35e-40f3-8c37-ad757223b110-istio-data\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj\" (UID: \"1453873b-b35e-40f3-8c37-ad757223b110\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj" Oct 31 04:17:34.602456 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:34.602435 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1453873b-b35e-40f3-8c37-ad757223b110-istiod-ca-cert\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj\" (UID: \"1453873b-b35e-40f3-8c37-ad757223b110\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj" Oct 31 04:17:34.607375 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:34.603202 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1453873b-b35e-40f3-8c37-ad757223b110-istiod-ca-cert\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj\" (UID: \"1453873b-b35e-40f3-8c37-ad757223b110\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj" Oct 31 04:17:34.623997 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:34.623960 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1453873b-b35e-40f3-8c37-ad757223b110-istio-podinfo\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj\" (UID: \"1453873b-b35e-40f3-8c37-ad757223b110\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj" Oct 31 04:17:34.648911 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:34.641960 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1453873b-b35e-40f3-8c37-ad757223b110-istio-token\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj\" (UID: \"1453873b-b35e-40f3-8c37-ad757223b110\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj" Oct 31 04:17:34.650193 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:34.650118 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cccg5\" (UniqueName: \"kubernetes.io/projected/1453873b-b35e-40f3-8c37-ad757223b110-kube-api-access-cccg5\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj\" (UID: \"1453873b-b35e-40f3-8c37-ad757223b110\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj" Oct 31 04:17:34.752374 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:34.751514 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:17:34.752374 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:34.751619 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd" podUID="d4062db4-f8d8-4cbb-8f2d-428c06d30db7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:17:34.771369 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:34.770971 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj" Oct 31 04:17:35.063700 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:35.063667 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd" Oct 31 04:17:35.079359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:35.065447 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd" Oct 31 04:17:35.093727 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:35.093637 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd" Oct 31 04:17:35.181557 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:35.167211 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:17:35.299367 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:35.298543 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:17:35.675401 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:35.590963 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" podUID="643433d8-418e-4102-8bc3-cd544932614a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:17:35.920481 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:35.920448 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" Oct 31 04:17:36.159080 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:36.159033 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj"] Oct 31 04:17:36.159306 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:36.159175 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" Oct 31 04:17:36.160637 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:35.992407 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:17:36.160637 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:36.159711 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:17:36.160637 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:36.159823 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:17:36.179705 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:36.067458 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:17:36.305418 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:36.304445 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:17:36.305418 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:36.305013 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" Oct 31 04:17:36.315118 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:36.314619 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" podUID="8e2c7041-ad52-49f4-8b55-d16691f4938f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 04:17:36.468318 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:36.465875 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" Oct 31 04:17:36.584651 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:36.584547 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj" event={"ID":"1453873b-b35e-40f3-8c37-ad757223b110","Type":"ContainerStarted","Data":"73ff8e29c39a3ad1a4d270761712381e4fe379f5450ac2ffb2ac51f2a01169a5"} Oct 31 04:17:36.584651 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:36.584621 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj" event={"ID":"1453873b-b35e-40f3-8c37-ad757223b110","Type":"ContainerStarted","Data":"2303578281a36de4f30b512f610f67effbdac550e719b1ebe96f3182fdb21c05"} Oct 31 04:17:36.639485 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:36.639142 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj" podStartSLOduration=2.639121948 podStartE2EDuration="2.639121948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:17:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:17:36.631909511 +0000 UTC m=+811.388247552" watchObservedRunningTime="2026-10-31 04:17:36.639121948 +0000 UTC m=+811.395459989" Oct 31 04:17:36.772429 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:36.772380 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj" Oct 31 04:17:37.146657 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:37.146620 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-57567b5c87-8r7nx" Oct 31 04:17:37.805357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:37.804482 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 04:17:37.805357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:37.804555 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj" podUID="1453873b-b35e-40f3-8c37-ad757223b110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 04:17:38.804357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:38.803855 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 04:17:38.804357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:38.803920 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj" podUID="1453873b-b35e-40f3-8c37-ad757223b110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 04:17:39.780896 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:39.780847 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 04:17:39.781536 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:39.781498 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj" podUID="1453873b-b35e-40f3-8c37-ad757223b110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 04:17:40.780591 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:40.780527 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 04:17:40.780934 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:40.780885 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj" podUID="1453873b-b35e-40f3-8c37-ad757223b110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 04:17:41.776800 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:41.776755 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 04:17:41.777101 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:41.777070 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj" podUID="1453873b-b35e-40f3-8c37-ad757223b110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 04:17:42.772473 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:42.772430 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 04:17:42.772907 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:42.772487 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj" podUID="1453873b-b35e-40f3-8c37-ad757223b110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 04:17:43.777479 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:43.777432 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj" Oct 31 04:17:43.777913 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:43.777740 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj" Oct 31 04:17:43.778685 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:43.778668 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj" Oct 31 04:17:43.982580 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:43.982542 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6c6d95cc8-5ww4s"] Oct 31 04:17:43.990627 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:43.990591 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6c6d95cc8-5ww4s" Oct 31 04:17:44.005191 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:44.005155 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-cdd9c78cff58\"" Oct 31 04:17:44.006273 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:44.006245 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c6d95cc8-5ww4s"] Oct 31 04:17:44.046217 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:44.046116 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2-provider-credentials\") pod \"payload-processing-external-model-6c6d95cc8-5ww4s\" (UID: \"6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2\") " pod="models-as-a-service/payload-processing-external-model-6c6d95cc8-5ww4s" Oct 31 04:17:44.046217 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:44.046173 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z564f\" (UniqueName: \"kubernetes.io/projected/6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2-kube-api-access-z564f\") pod \"payload-processing-external-model-6c6d95cc8-5ww4s\" (UID: \"6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2\") " pod="models-as-a-service/payload-processing-external-model-6c6d95cc8-5ww4s" Oct 31 04:17:44.046626 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:44.046608 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2-routing-overlay-volume\") pod \"payload-processing-external-model-6c6d95cc8-5ww4s\" (UID: \"6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2\") " pod="models-as-a-service/payload-processing-external-model-6c6d95cc8-5ww4s" Oct 31 04:17:44.046772 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:44.046758 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2-plugins-config-volume\") pod \"payload-processing-external-model-6c6d95cc8-5ww4s\" (UID: \"6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2\") " pod="models-as-a-service/payload-processing-external-model-6c6d95cc8-5ww4s" Oct 31 04:17:44.147417 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:44.147380 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2-provider-credentials\") pod \"payload-processing-external-model-6c6d95cc8-5ww4s\" (UID: \"6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2\") " pod="models-as-a-service/payload-processing-external-model-6c6d95cc8-5ww4s" Oct 31 04:17:44.147417 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:44.147424 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z564f\" (UniqueName: \"kubernetes.io/projected/6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2-kube-api-access-z564f\") pod \"payload-processing-external-model-6c6d95cc8-5ww4s\" (UID: \"6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2\") " pod="models-as-a-service/payload-processing-external-model-6c6d95cc8-5ww4s" Oct 31 04:17:44.147695 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:44.147515 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2-routing-overlay-volume\") pod \"payload-processing-external-model-6c6d95cc8-5ww4s\" (UID: \"6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2\") " pod="models-as-a-service/payload-processing-external-model-6c6d95cc8-5ww4s" Oct 31 04:17:44.147695 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:44.147674 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2-plugins-config-volume\") pod \"payload-processing-external-model-6c6d95cc8-5ww4s\" (UID: \"6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2\") " pod="models-as-a-service/payload-processing-external-model-6c6d95cc8-5ww4s" Oct 31 04:17:44.148243 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:44.148214 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2-routing-overlay-volume\") pod \"payload-processing-external-model-6c6d95cc8-5ww4s\" (UID: \"6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2\") " pod="models-as-a-service/payload-processing-external-model-6c6d95cc8-5ww4s" Oct 31 04:17:44.148453 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:44.148431 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2-plugins-config-volume\") pod \"payload-processing-external-model-6c6d95cc8-5ww4s\" (UID: \"6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2\") " pod="models-as-a-service/payload-processing-external-model-6c6d95cc8-5ww4s" Oct 31 04:17:44.149921 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:44.149898 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2-provider-credentials\") pod \"payload-processing-external-model-6c6d95cc8-5ww4s\" (UID: \"6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2\") " pod="models-as-a-service/payload-processing-external-model-6c6d95cc8-5ww4s" Oct 31 04:17:44.171226 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:44.171185 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z564f\" (UniqueName: \"kubernetes.io/projected/6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2-kube-api-access-z564f\") pod \"payload-processing-external-model-6c6d95cc8-5ww4s\" (UID: \"6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2\") " pod="models-as-a-service/payload-processing-external-model-6c6d95cc8-5ww4s" Oct 31 04:17:44.308565 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:44.308458 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6c6d95cc8-5ww4s" Oct 31 04:17:44.616550 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:44.616504 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c6d95cc8-5ww4s"] Oct 31 04:17:44.677526 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:44.677486 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6c6d95cc8-5ww4s" event={"ID":"6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2","Type":"ContainerStarted","Data":"1ac9114f44378c881c0e9e1cb3131d8d23e9f2be6e60e4dc58a14d9205fd7af7"} Oct 31 04:17:45.685848 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:45.685806 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6c6d95cc8-5ww4s" event={"ID":"6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2","Type":"ContainerStarted","Data":"0399da119b3c173d1f454572795ed9e119ca4fb5507de3fea6a434c262be3a3b"} Oct 31 04:17:45.686322 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:45.686016 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6c6d95cc8-5ww4s" Oct 31 04:17:45.726875 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:45.726819 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6c6d95cc8-5ww4s" podStartSLOduration=2.726803614 podStartE2EDuration="2.726803614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:17:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:17:45.726228911 +0000 UTC m=+820.482566949" watchObservedRunningTime="2026-10-31 04:17:45.726803614 +0000 UTC m=+820.483141710" Oct 31 04:17:50.801353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:50.800955 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c"] Oct 31 04:17:50.958357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:50.952599 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c"] Oct 31 04:17:50.958357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:50.952730 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c" Oct 31 04:17:51.174491 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:51.173715 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwqzb\" (UniqueName: \"kubernetes.io/projected/43e793da-0b3f-4247-bc2e-6d19de53d75c-kube-api-access-xwqzb\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c\" (UID: \"43e793da-0b3f-4247-bc2e-6d19de53d75c\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c" Oct 31 04:17:51.174491 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:51.173769 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43e793da-0b3f-4247-bc2e-6d19de53d75c-istio-data\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c\" (UID: \"43e793da-0b3f-4247-bc2e-6d19de53d75c\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c" Oct 31 04:17:51.174491 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:51.173806 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e793da-0b3f-4247-bc2e-6d19de53d75c-workload-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c\" (UID: \"43e793da-0b3f-4247-bc2e-6d19de53d75c\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c" Oct 31 04:17:51.174491 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:51.173844 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e793da-0b3f-4247-bc2e-6d19de53d75c-credential-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c\" (UID: \"43e793da-0b3f-4247-bc2e-6d19de53d75c\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c" Oct 31 04:17:51.174491 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:51.173886 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43e793da-0b3f-4247-bc2e-6d19de53d75c-istiod-ca-cert\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c\" (UID: \"43e793da-0b3f-4247-bc2e-6d19de53d75c\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c" Oct 31 04:17:51.174491 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:51.173963 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43e793da-0b3f-4247-bc2e-6d19de53d75c-istio-token\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c\" (UID: \"43e793da-0b3f-4247-bc2e-6d19de53d75c\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c" Oct 31 04:17:51.174491 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:51.174000 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43e793da-0b3f-4247-bc2e-6d19de53d75c-istio-envoy\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c\" (UID: \"43e793da-0b3f-4247-bc2e-6d19de53d75c\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c" Oct 31 04:17:51.174491 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:51.174059 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43e793da-0b3f-4247-bc2e-6d19de53d75c-istio-podinfo\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c\" (UID: \"43e793da-0b3f-4247-bc2e-6d19de53d75c\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c" Oct 31 04:17:51.174491 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:51.174117 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43e793da-0b3f-4247-bc2e-6d19de53d75c-workload-certs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c\" (UID: \"43e793da-0b3f-4247-bc2e-6d19de53d75c\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c" Oct 31 04:17:51.307372 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:51.303032 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43e793da-0b3f-4247-bc2e-6d19de53d75c-istio-token\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c\" (UID: \"43e793da-0b3f-4247-bc2e-6d19de53d75c\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c" Oct 31 04:17:51.307372 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:51.303096 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43e793da-0b3f-4247-bc2e-6d19de53d75c-istio-envoy\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c\" (UID: \"43e793da-0b3f-4247-bc2e-6d19de53d75c\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c" Oct 31 04:17:51.307372 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:51.303155 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43e793da-0b3f-4247-bc2e-6d19de53d75c-istio-podinfo\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c\" (UID: \"43e793da-0b3f-4247-bc2e-6d19de53d75c\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c" Oct 31 04:17:51.307372 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:51.303211 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43e793da-0b3f-4247-bc2e-6d19de53d75c-workload-certs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c\" (UID: \"43e793da-0b3f-4247-bc2e-6d19de53d75c\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c" Oct 31 04:17:51.307372 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:51.303310 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwqzb\" (UniqueName: \"kubernetes.io/projected/43e793da-0b3f-4247-bc2e-6d19de53d75c-kube-api-access-xwqzb\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c\" (UID: \"43e793da-0b3f-4247-bc2e-6d19de53d75c\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c" Oct 31 04:17:51.307372 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:51.303365 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43e793da-0b3f-4247-bc2e-6d19de53d75c-istio-data\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c\" (UID: \"43e793da-0b3f-4247-bc2e-6d19de53d75c\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c" Oct 31 04:17:51.307372 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:51.303396 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e793da-0b3f-4247-bc2e-6d19de53d75c-workload-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c\" (UID: \"43e793da-0b3f-4247-bc2e-6d19de53d75c\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c" Oct 31 04:17:51.307372 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:51.303432 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e793da-0b3f-4247-bc2e-6d19de53d75c-credential-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c\" (UID: \"43e793da-0b3f-4247-bc2e-6d19de53d75c\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c" Oct 31 04:17:51.307372 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:51.303468 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43e793da-0b3f-4247-bc2e-6d19de53d75c-istiod-ca-cert\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c\" (UID: \"43e793da-0b3f-4247-bc2e-6d19de53d75c\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c" Oct 31 04:17:51.307372 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:51.304388 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43e793da-0b3f-4247-bc2e-6d19de53d75c-istiod-ca-cert\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c\" (UID: \"43e793da-0b3f-4247-bc2e-6d19de53d75c\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c" Oct 31 04:17:51.318853 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:51.318810 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43e793da-0b3f-4247-bc2e-6d19de53d75c-workload-certs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c\" (UID: \"43e793da-0b3f-4247-bc2e-6d19de53d75c\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c" Oct 31 04:17:51.319420 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:51.319390 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e793da-0b3f-4247-bc2e-6d19de53d75c-workload-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c\" (UID: \"43e793da-0b3f-4247-bc2e-6d19de53d75c\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c" Oct 31 04:17:51.319976 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:51.319950 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43e793da-0b3f-4247-bc2e-6d19de53d75c-istio-data\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c\" (UID: \"43e793da-0b3f-4247-bc2e-6d19de53d75c\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c" Oct 31 04:17:51.320410 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:51.320387 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e793da-0b3f-4247-bc2e-6d19de53d75c-credential-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c\" (UID: \"43e793da-0b3f-4247-bc2e-6d19de53d75c\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c" Oct 31 04:17:51.337359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:51.333932 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43e793da-0b3f-4247-bc2e-6d19de53d75c-istio-podinfo\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c\" (UID: \"43e793da-0b3f-4247-bc2e-6d19de53d75c\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c" Oct 31 04:17:51.337359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:51.335871 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43e793da-0b3f-4247-bc2e-6d19de53d75c-istio-token\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c\" (UID: \"43e793da-0b3f-4247-bc2e-6d19de53d75c\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c" Oct 31 04:17:51.357359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:51.344810 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43e793da-0b3f-4247-bc2e-6d19de53d75c-istio-envoy\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c\" (UID: \"43e793da-0b3f-4247-bc2e-6d19de53d75c\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c" Oct 31 04:17:51.375409 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:51.375373 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwqzb\" (UniqueName: \"kubernetes.io/projected/43e793da-0b3f-4247-bc2e-6d19de53d75c-kube-api-access-xwqzb\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c\" (UID: \"43e793da-0b3f-4247-bc2e-6d19de53d75c\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c" Oct 31 04:17:51.521619 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:51.521575 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c" Oct 31 04:17:52.091605 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:52.091448 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:17:52.208384 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:52.208167 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:17:52.345159 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:52.345058 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" podUID="8e2c7041-ad52-49f4-8b55-d16691f4938f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 04:17:52.522154 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:52.522116 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c"] Oct 31 04:17:52.564966 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:52.564910 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:17:52.565236 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:52.565210 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:17:52.565451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:52.565410 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:17:52.613669 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:17:52.612650 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:17:53.329393 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:53.329357 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c" event={"ID":"43e793da-0b3f-4247-bc2e-6d19de53d75c","Type":"ContainerStarted","Data":"9e2b12eee106d23f52b7e6ff3ca751e293adacbcf8bd8e8439395a11aa7349fc"} Oct 31 04:17:53.329768 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:53.329407 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c" event={"ID":"43e793da-0b3f-4247-bc2e-6d19de53d75c","Type":"ContainerStarted","Data":"9638d6f6e7372b62d39afc81d0a40c0c95fc9d1e17ee2e43eeeb9c3f0f636576"} Oct 31 04:17:53.522155 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:53.522116 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c" Oct 31 04:17:53.523419 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:53.523383 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 31 04:17:53.523599 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:53.523440 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c" podUID="43e793da-0b3f-4247-bc2e-6d19de53d75c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 31 04:17:54.523326 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:54.523289 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 31 04:17:54.523812 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:54.523358 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c" podUID="43e793da-0b3f-4247-bc2e-6d19de53d75c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 31 04:17:55.522189 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:55.522149 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 31 04:17:55.522420 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:55.522204 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c" podUID="43e793da-0b3f-4247-bc2e-6d19de53d75c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 31 04:17:56.522666 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:56.522619 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 31 04:17:56.523127 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:56.522691 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c" podUID="43e793da-0b3f-4247-bc2e-6d19de53d75c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 31 04:17:56.703910 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:56.703817 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-6c6d95cc8-5ww4s" Oct 31 04:17:56.748941 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:56.748882 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c" podStartSLOduration=6.748864723 podStartE2EDuration="6.748864723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:17:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:17:53.392692463 +0000 UTC m=+828.149030504" watchObservedRunningTime="2026-10-31 04:17:56.748864723 +0000 UTC m=+831.505202761" Oct 31 04:17:56.812064 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:56.812026 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-57567b5c87-8r7nx"] Oct 31 04:17:57.522987 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:57.522947 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 31 04:17:57.523421 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:57.523015 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c" podUID="43e793da-0b3f-4247-bc2e-6d19de53d75c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 31 04:17:58.522540 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:58.522497 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 31 04:17:58.522721 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:58.522561 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c" podUID="43e793da-0b3f-4247-bc2e-6d19de53d75c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 31 04:17:59.754657 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:59.754621 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c" Oct 31 04:17:59.755664 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:59.755637 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c" Oct 31 04:17:59.756603 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:17:59.756578 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c" Oct 31 04:18:01.813710 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:01.813655 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-57567b5c87-8r7nx" podUID="185e60a3-6578-4f3d-a45a-cca447bd5071" containerName="payload-processing" containerID="cri-o://e13d3dee48a2ac2dcc6734bbd4def7883d208ee8b68ace5624c0be3133340840" gracePeriod=25 Oct 31 04:18:02.101432 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:02.101403 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-57567b5c87-8r7nx" Oct 31 04:18:02.181618 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:02.179721 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/185e60a3-6578-4f3d-a45a-cca447bd5071-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/185e60a3-6578-4f3d-a45a-cca447bd5071-routing-overlay-volume\") pod \"185e60a3-6578-4f3d-a45a-cca447bd5071\" (UID: \"185e60a3-6578-4f3d-a45a-cca447bd5071\") " Oct 31 04:18:02.181618 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:02.179777 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/185e60a3-6578-4f3d-a45a-cca447bd5071-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/185e60a3-6578-4f3d-a45a-cca447bd5071-plugins-config-volume\") pod \"185e60a3-6578-4f3d-a45a-cca447bd5071\" (UID: \"185e60a3-6578-4f3d-a45a-cca447bd5071\") " Oct 31 04:18:02.181618 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:02.179837 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/185e60a3-6578-4f3d-a45a-cca447bd5071-provider-credentials\" (UniqueName: \"kubernetes.io/projected/185e60a3-6578-4f3d-a45a-cca447bd5071-provider-credentials\") pod \"185e60a3-6578-4f3d-a45a-cca447bd5071\" (UID: \"185e60a3-6578-4f3d-a45a-cca447bd5071\") " Oct 31 04:18:02.181618 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:02.180060 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/185e60a3-6578-4f3d-a45a-cca447bd5071-kube-api-access-tzhk4\" (UniqueName: \"kubernetes.io/projected/185e60a3-6578-4f3d-a45a-cca447bd5071-kube-api-access-tzhk4\") pod \"185e60a3-6578-4f3d-a45a-cca447bd5071\" (UID: \"185e60a3-6578-4f3d-a45a-cca447bd5071\") " Oct 31 04:18:02.181618 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:02.181173 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/185e60a3-6578-4f3d-a45a-cca447bd5071-plugins-config-volume" pod "185e60a3-6578-4f3d-a45a-cca447bd5071" (UID: "185e60a3-6578-4f3d-a45a-cca447bd5071"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:18:02.181618 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:02.181606 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/185e60a3-6578-4f3d-a45a-cca447bd5071-routing-overlay-volume" pod "185e60a3-6578-4f3d-a45a-cca447bd5071" (UID: "185e60a3-6578-4f3d-a45a-cca447bd5071"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:18:02.184533 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:02.183273 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/185e60a3-6578-4f3d-a45a-cca447bd5071-kube-api-access-tzhk4" pod "185e60a3-6578-4f3d-a45a-cca447bd5071" (UID: "185e60a3-6578-4f3d-a45a-cca447bd5071"). InnerVolumeSpecName "kube-api-access-tzhk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:18:02.184533 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:02.183875 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/185e60a3-6578-4f3d-a45a-cca447bd5071-provider-credentials" pod "185e60a3-6578-4f3d-a45a-cca447bd5071" (UID: "185e60a3-6578-4f3d-a45a-cca447bd5071"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:18:02.281831 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:02.281791 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzhk4\" (UniqueName: \"kubernetes.io/projected/185e60a3-6578-4f3d-a45a-cca447bd5071-kube-api-access-tzhk4\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:18:02.281831 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:02.281829 5019 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/185e60a3-6578-4f3d-a45a-cca447bd5071-routing-overlay-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:18:02.282043 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:02.281843 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/185e60a3-6578-4f3d-a45a-cca447bd5071-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:18:02.282043 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:02.281857 5019 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/185e60a3-6578-4f3d-a45a-cca447bd5071-provider-credentials\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:18:02.430272 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:02.430183 5019 generic.go:356] "Generic (PLEG): container finished" podID="185e60a3-6578-4f3d-a45a-cca447bd5071" containerID="e13d3dee48a2ac2dcc6734bbd4def7883d208ee8b68ace5624c0be3133340840" exitCode=0 Oct 31 04:18:02.430272 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:02.430231 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-57567b5c87-8r7nx" Oct 31 04:18:02.430497 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:02.430281 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-57567b5c87-8r7nx" event={"ID":"185e60a3-6578-4f3d-a45a-cca447bd5071","Type":"ContainerDied","Data":"e13d3dee48a2ac2dcc6734bbd4def7883d208ee8b68ace5624c0be3133340840"} Oct 31 04:18:02.430497 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:02.430351 5019 scope.go:122] "RemoveContainer" containerID="e13d3dee48a2ac2dcc6734bbd4def7883d208ee8b68ace5624c0be3133340840" Oct 31 04:18:02.430497 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:02.430345 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-57567b5c87-8r7nx" event={"ID":"185e60a3-6578-4f3d-a45a-cca447bd5071","Type":"ContainerDied","Data":"ad0c3667fc41b5a6d265804464cba3fb1bd4cbdf3b89b9e53163d0df2064f0c0"} Oct 31 04:18:02.444181 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:02.444154 5019 scope.go:122] "RemoveContainer" containerID="e13d3dee48a2ac2dcc6734bbd4def7883d208ee8b68ace5624c0be3133340840" Oct 31 04:18:02.445217 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:18:02.445177 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e13d3dee48a2ac2dcc6734bbd4def7883d208ee8b68ace5624c0be3133340840\": container with ID starting with e13d3dee48a2ac2dcc6734bbd4def7883d208ee8b68ace5624c0be3133340840 not found: ID does not exist" containerID="e13d3dee48a2ac2dcc6734bbd4def7883d208ee8b68ace5624c0be3133340840" Oct 31 04:18:02.445379 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:02.445228 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e13d3dee48a2ac2dcc6734bbd4def7883d208ee8b68ace5624c0be3133340840"} err="failed to get container status \"e13d3dee48a2ac2dcc6734bbd4def7883d208ee8b68ace5624c0be3133340840\": rpc error: code = NotFound desc = could not find container \"e13d3dee48a2ac2dcc6734bbd4def7883d208ee8b68ace5624c0be3133340840\": container with ID starting with e13d3dee48a2ac2dcc6734bbd4def7883d208ee8b68ace5624c0be3133340840 not found: ID does not exist" Oct 31 04:18:02.483878 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:02.483807 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-57567b5c87-8r7nx"] Oct 31 04:18:02.493790 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:02.493756 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-57567b5c87-8r7nx"] Oct 31 04:18:03.408810 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:03.408765 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="185e60a3-6578-4f3d-a45a-cca447bd5071" path="/var/lib/kubelet/pods/185e60a3-6578-4f3d-a45a-cca447bd5071/volumes" Oct 31 04:18:13.786273 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:13.786213 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f"] Oct 31 04:18:13.788894 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:13.788863 5019 cpu_manager.go:414] "containerMap: removing container" podUID="185e60a3-6578-4f3d-a45a-cca447bd5071" containerName="payload-processing" Oct 31 04:18:13.789222 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:13.789204 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="185e60a3-6578-4f3d-a45a-cca447bd5071" containerName="payload-processing" Oct 31 04:18:13.789568 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:13.789551 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="185e60a3-6578-4f3d-a45a-cca447bd5071" containerName="payload-processing" Oct 31 04:18:13.799035 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:13.799001 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" Oct 31 04:18:13.802470 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:13.802429 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f"] Oct 31 04:18:13.864356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:13.861007 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c4006d\"/\"distinct-w4-c4006d-kserve-self-signed-certs\"" Oct 31 04:18:13.941460 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:13.941420 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4ced3ab-41e4-4c54-93eb-95eddaae992d-home\") pod \"distinct-w4-c4006d-kserve-569c4d8784-r6s2f\" (UID: \"b4ced3ab-41e4-4c54-93eb-95eddaae992d\") " pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" Oct 31 04:18:13.941842 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:13.941821 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4ced3ab-41e4-4c54-93eb-95eddaae992d-dshm\") pod \"distinct-w4-c4006d-kserve-569c4d8784-r6s2f\" (UID: \"b4ced3ab-41e4-4c54-93eb-95eddaae992d\") " pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" Oct 31 04:18:13.942028 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:13.942011 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4ced3ab-41e4-4c54-93eb-95eddaae992d-tmp-dir\") pod \"distinct-w4-c4006d-kserve-569c4d8784-r6s2f\" (UID: \"b4ced3ab-41e4-4c54-93eb-95eddaae992d\") " pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" Oct 31 04:18:13.942138 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:13.942125 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4ced3ab-41e4-4c54-93eb-95eddaae992d-tls-certs\") pod \"distinct-w4-c4006d-kserve-569c4d8784-r6s2f\" (UID: \"b4ced3ab-41e4-4c54-93eb-95eddaae992d\") " pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" Oct 31 04:18:13.942241 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:13.942226 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl24p\" (UniqueName: \"kubernetes.io/projected/b4ced3ab-41e4-4c54-93eb-95eddaae992d-kube-api-access-gl24p\") pod \"distinct-w4-c4006d-kserve-569c4d8784-r6s2f\" (UID: \"b4ced3ab-41e4-4c54-93eb-95eddaae992d\") " pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" Oct 31 04:18:13.942369 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:13.942351 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4ced3ab-41e4-4c54-93eb-95eddaae992d-model-cache\") pod \"distinct-w4-c4006d-kserve-569c4d8784-r6s2f\" (UID: \"b4ced3ab-41e4-4c54-93eb-95eddaae992d\") " pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" Oct 31 04:18:14.051525 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:14.051169 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4ced3ab-41e4-4c54-93eb-95eddaae992d-dshm\") pod \"distinct-w4-c4006d-kserve-569c4d8784-r6s2f\" (UID: \"b4ced3ab-41e4-4c54-93eb-95eddaae992d\") " pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" Oct 31 04:18:14.052382 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:14.052352 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4ced3ab-41e4-4c54-93eb-95eddaae992d-tmp-dir\") pod \"distinct-w4-c4006d-kserve-569c4d8784-r6s2f\" (UID: \"b4ced3ab-41e4-4c54-93eb-95eddaae992d\") " pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" Oct 31 04:18:14.053027 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:14.053005 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4ced3ab-41e4-4c54-93eb-95eddaae992d-tls-certs\") pod \"distinct-w4-c4006d-kserve-569c4d8784-r6s2f\" (UID: \"b4ced3ab-41e4-4c54-93eb-95eddaae992d\") " pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" Oct 31 04:18:14.054686 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:14.054663 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl24p\" (UniqueName: \"kubernetes.io/projected/b4ced3ab-41e4-4c54-93eb-95eddaae992d-kube-api-access-gl24p\") pod \"distinct-w4-c4006d-kserve-569c4d8784-r6s2f\" (UID: \"b4ced3ab-41e4-4c54-93eb-95eddaae992d\") " pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" Oct 31 04:18:14.055208 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:14.055188 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4ced3ab-41e4-4c54-93eb-95eddaae992d-model-cache\") pod \"distinct-w4-c4006d-kserve-569c4d8784-r6s2f\" (UID: \"b4ced3ab-41e4-4c54-93eb-95eddaae992d\") " pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" Oct 31 04:18:14.055894 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:14.055872 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4ced3ab-41e4-4c54-93eb-95eddaae992d-home\") pod \"distinct-w4-c4006d-kserve-569c4d8784-r6s2f\" (UID: \"b4ced3ab-41e4-4c54-93eb-95eddaae992d\") " pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" Oct 31 04:18:14.056517 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:14.052954 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4ced3ab-41e4-4c54-93eb-95eddaae992d-tmp-dir\") pod \"distinct-w4-c4006d-kserve-569c4d8784-r6s2f\" (UID: \"b4ced3ab-41e4-4c54-93eb-95eddaae992d\") " pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" Oct 31 04:18:14.056655 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:14.056285 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4ced3ab-41e4-4c54-93eb-95eddaae992d-home\") pod \"distinct-w4-c4006d-kserve-569c4d8784-r6s2f\" (UID: \"b4ced3ab-41e4-4c54-93eb-95eddaae992d\") " pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" Oct 31 04:18:14.056981 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:14.055734 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4ced3ab-41e4-4c54-93eb-95eddaae992d-model-cache\") pod \"distinct-w4-c4006d-kserve-569c4d8784-r6s2f\" (UID: \"b4ced3ab-41e4-4c54-93eb-95eddaae992d\") " pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" Oct 31 04:18:14.075806 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:14.075767 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4ced3ab-41e4-4c54-93eb-95eddaae992d-dshm\") pod \"distinct-w4-c4006d-kserve-569c4d8784-r6s2f\" (UID: \"b4ced3ab-41e4-4c54-93eb-95eddaae992d\") " pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" Oct 31 04:18:14.109358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:14.085299 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4ced3ab-41e4-4c54-93eb-95eddaae992d-tls-certs\") pod \"distinct-w4-c4006d-kserve-569c4d8784-r6s2f\" (UID: \"b4ced3ab-41e4-4c54-93eb-95eddaae992d\") " pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" Oct 31 04:18:14.110272 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:14.097527 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl24p\" (UniqueName: \"kubernetes.io/projected/b4ced3ab-41e4-4c54-93eb-95eddaae992d-kube-api-access-gl24p\") pod \"distinct-w4-c4006d-kserve-569c4d8784-r6s2f\" (UID: \"b4ced3ab-41e4-4c54-93eb-95eddaae992d\") " pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" Oct 31 04:18:14.335407 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:14.334645 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" Oct 31 04:18:14.890052 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:14.890014 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f"] Oct 31 04:18:15.749437 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:15.737544 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" event={"ID":"b4ced3ab-41e4-4c54-93eb-95eddaae992d","Type":"ContainerStarted","Data":"1572354f776386eebc19afe3eb66490a6ba9b9d9704c82e72a5a37fab4eb377a"} Oct 31 04:18:15.749437 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:15.737632 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" event={"ID":"b4ced3ab-41e4-4c54-93eb-95eddaae992d","Type":"ContainerStarted","Data":"0dfe9a5548d56a8e23a6875ee933f84c47011085f7527e1060348b4bb72eeffe"} Oct 31 04:18:15.842690 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:15.842612 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" podStartSLOduration=2.842586351 podStartE2EDuration="2.842586351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:18:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:18:15.815500681 +0000 UTC m=+850.571838720" watchObservedRunningTime="2026-10-31 04:18:15.842586351 +0000 UTC m=+850.598924391" Oct 31 04:18:16.347967 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:16.336441 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" Oct 31 04:18:16.501355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:16.500923 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" podUID="643433d8-418e-4102-8bc3-cd544932614a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:18:16.981357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:16.978424 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" podUID="dccd15a9-47f5-4bd1-b21b-9d5ca8b05845" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:18:17.067547 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:17.067483 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:18:17.090578 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:17.090521 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" podUID="8e2c7041-ad52-49f4-8b55-d16691f4938f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:18:17.169538 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:17.169488 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:18:17.484008 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:17.483961 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:18:19.717164 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:19.716950 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c6d95cc8-5ww4s"] Oct 31 04:18:22.392429 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:18:22.392377 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:18:24.335211 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:24.335176 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" Oct 31 04:18:24.347479 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:24.347457 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" Oct 31 04:18:24.717527 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:24.717485 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6c6d95cc8-5ww4s" podUID="6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2" containerName="payload-processing" containerID="cri-o://0399da119b3c173d1f454572795ed9e119ca4fb5507de3fea6a434c262be3a3b" gracePeriod=25 Oct 31 04:18:24.924094 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:24.924056 5019 generic.go:356] "Generic (PLEG): container finished" podID="6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2" containerID="0399da119b3c173d1f454572795ed9e119ca4fb5507de3fea6a434c262be3a3b" exitCode=0 Oct 31 04:18:24.924261 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:24.924121 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6c6d95cc8-5ww4s" event={"ID":"6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2","Type":"ContainerDied","Data":"0399da119b3c173d1f454572795ed9e119ca4fb5507de3fea6a434c262be3a3b"} Oct 31 04:18:24.938964 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:24.938927 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" Oct 31 04:18:24.998648 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:24.998603 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6c6d95cc8-5ww4s" Oct 31 04:18:25.117040 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:25.116997 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2-plugins-config-volume\") pod \"6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2\" (UID: \"6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2\") " Oct 31 04:18:25.117224 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:25.117114 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2-provider-credentials\") pod \"6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2\" (UID: \"6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2\") " Oct 31 04:18:25.117224 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:25.117180 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2-kube-api-access-z564f\" (UniqueName: \"kubernetes.io/projected/6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2-kube-api-access-z564f\") pod \"6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2\" (UID: \"6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2\") " Oct 31 04:18:25.117224 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:25.117216 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2-routing-overlay-volume\") pod \"6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2\" (UID: \"6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2\") " Oct 31 04:18:25.117881 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:25.117453 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2-plugins-config-volume" pod "6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2" (UID: "6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:18:25.117881 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:25.117675 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2-routing-overlay-volume" pod "6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2" (UID: "6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:18:25.117881 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:25.117793 5019 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2-routing-overlay-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:18:25.117881 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:25.117826 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:18:25.119423 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:25.119313 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2-provider-credentials" pod "6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2" (UID: "6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:18:25.119726 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:25.119710 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2-kube-api-access-z564f" pod "6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2" (UID: "6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2"). InnerVolumeSpecName "kube-api-access-z564f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:18:25.218737 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:25.218703 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z564f\" (UniqueName: \"kubernetes.io/projected/6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2-kube-api-access-z564f\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:18:25.218737 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:25.218737 5019 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2-provider-credentials\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:18:25.932116 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:25.932088 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6c6d95cc8-5ww4s" Oct 31 04:18:25.932574 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:25.932124 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6c6d95cc8-5ww4s" event={"ID":"6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2","Type":"ContainerDied","Data":"1ac9114f44378c881c0e9e1cb3131d8d23e9f2be6e60e4dc58a14d9205fd7af7"} Oct 31 04:18:25.932574 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:25.932188 5019 scope.go:122] "RemoveContainer" containerID="0399da119b3c173d1f454572795ed9e119ca4fb5507de3fea6a434c262be3a3b" Oct 31 04:18:25.985725 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:25.985690 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c6d95cc8-5ww4s"] Oct 31 04:18:25.997989 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:25.997952 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c6d95cc8-5ww4s"] Oct 31 04:18:27.410303 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:27.410264 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2" path="/var/lib/kubelet/pods/6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2/volumes" Oct 31 04:18:29.925540 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:29.925496 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" podUID="b4ced3ab-41e4-4c54-93eb-95eddaae992d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Oct 31 04:18:29.975359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:29.970903 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" podUID="dccd15a9-47f5-4bd1-b21b-9d5ca8b05845" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:18:31.961503 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:31.961427 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" podUID="dccd15a9-47f5-4bd1-b21b-9d5ca8b05845" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:18:32.071442 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:32.071398 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:18:32.093449 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:32.093405 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" podUID="8e2c7041-ad52-49f4-8b55-d16691f4938f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:18:32.957498 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:32.957467 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-718ebc99-j2jrs"] Oct 31 04:18:32.958063 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:32.958048 5019 cpu_manager.go:414] "containerMap: removing container" podUID="6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2" containerName="payload-processing" Oct 31 04:18:32.958121 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:32.958064 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2" containerName="payload-processing" Oct 31 04:18:32.958160 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:32.958145 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d546f34-d1ee-45ac-ac0f-dc45ae2d61b2" containerName="payload-processing" Oct 31 04:18:32.961862 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:32.961845 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-718ebc99-j2jrs" Oct 31 04:18:32.980227 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:32.980191 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:18:32.992448 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:32.992426 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:18:32.992616 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:32.992563 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:18:32.992727 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:32.992638 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:18:32.996506 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:32.996477 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-718ebc99-j2jrs"] Oct 31 04:18:33.065244 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:33.065213 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/619a5998-8522-4f92-939a-32f500de6d6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-718ebc99-j2jrs\" (UID: \"619a5998-8522-4f92-939a-32f500de6d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-718ebc99-j2jrs" Oct 31 04:18:33.065244 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:33.065252 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxjlw\" (UniqueName: \"kubernetes.io/projected/619a5998-8522-4f92-939a-32f500de6d6c-kube-api-access-kxjlw\") pod \"maas-api-revoke-sub-718ebc99-j2jrs\" (UID: \"619a5998-8522-4f92-939a-32f500de6d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-718ebc99-j2jrs" Oct 31 04:18:33.065496 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:33.065477 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/619a5998-8522-4f92-939a-32f500de6d6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-718ebc99-j2jrs\" (UID: \"619a5998-8522-4f92-939a-32f500de6d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-718ebc99-j2jrs" Oct 31 04:18:33.166174 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:33.166139 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/619a5998-8522-4f92-939a-32f500de6d6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-718ebc99-j2jrs\" (UID: \"619a5998-8522-4f92-939a-32f500de6d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-718ebc99-j2jrs" Oct 31 04:18:33.166377 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:33.166219 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/619a5998-8522-4f92-939a-32f500de6d6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-718ebc99-j2jrs\" (UID: \"619a5998-8522-4f92-939a-32f500de6d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-718ebc99-j2jrs" Oct 31 04:18:33.166377 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:33.166245 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxjlw\" (UniqueName: \"kubernetes.io/projected/619a5998-8522-4f92-939a-32f500de6d6c-kube-api-access-kxjlw\") pod \"maas-api-revoke-sub-718ebc99-j2jrs\" (UID: \"619a5998-8522-4f92-939a-32f500de6d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-718ebc99-j2jrs" Oct 31 04:18:33.166941 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:33.166914 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/619a5998-8522-4f92-939a-32f500de6d6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-718ebc99-j2jrs\" (UID: \"619a5998-8522-4f92-939a-32f500de6d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-718ebc99-j2jrs" Oct 31 04:18:33.168722 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:33.168697 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/619a5998-8522-4f92-939a-32f500de6d6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-718ebc99-j2jrs\" (UID: \"619a5998-8522-4f92-939a-32f500de6d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-718ebc99-j2jrs" Oct 31 04:18:33.189282 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:33.189257 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxjlw\" (UniqueName: \"kubernetes.io/projected/619a5998-8522-4f92-939a-32f500de6d6c-kube-api-access-kxjlw\") pod \"maas-api-revoke-sub-718ebc99-j2jrs\" (UID: \"619a5998-8522-4f92-939a-32f500de6d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-718ebc99-j2jrs" Oct 31 04:18:33.275449 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:33.275419 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-718ebc99-j2jrs" Oct 31 04:18:33.580865 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:18:33.580813 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod619a5998_8522_4f92_939a_32f500de6d6c.slice/crio-256e66cc6bbb39eab27be2b7bd648ce87157dd73d96c144d178c9aab68a29dcd WatchSource:0}: Error finding container 256e66cc6bbb39eab27be2b7bd648ce87157dd73d96c144d178c9aab68a29dcd: Status 404 returned error can't find the container with id 256e66cc6bbb39eab27be2b7bd648ce87157dd73d96c144d178c9aab68a29dcd Oct 31 04:18:33.584092 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:33.584063 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-718ebc99-j2jrs"] Oct 31 04:18:34.306147 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:34.306059 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-718ebc99-j2jrs" event={"ID":"619a5998-8522-4f92-939a-32f500de6d6c","Type":"ContainerStarted","Data":"04cbfe43942620973d21e7cdd1c0e0bba095cf3b8df2c9caaa6d6a98f63cc71a"} Oct 31 04:18:34.306147 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:34.306115 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-718ebc99-j2jrs" event={"ID":"619a5998-8522-4f92-939a-32f500de6d6c","Type":"ContainerStarted","Data":"256e66cc6bbb39eab27be2b7bd648ce87157dd73d96c144d178c9aab68a29dcd"} Oct 31 04:18:34.345972 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:34.345923 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-718ebc99-j2jrs" podStartSLOduration=2.345910757 podStartE2EDuration="2.345910757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:18:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:18:34.34550138 +0000 UTC m=+869.101839413" watchObservedRunningTime="2026-10-31 04:18:34.345910757 +0000 UTC m=+869.102248795" Oct 31 04:18:35.315205 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:35.315165 5019 generic.go:356] "Generic (PLEG): container finished" podID="619a5998-8522-4f92-939a-32f500de6d6c" containerID="04cbfe43942620973d21e7cdd1c0e0bba095cf3b8df2c9caaa6d6a98f63cc71a" exitCode=0 Oct 31 04:18:35.315688 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:35.315448 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-718ebc99-j2jrs" event={"ID":"619a5998-8522-4f92-939a-32f500de6d6c","Type":"ContainerDied","Data":"04cbfe43942620973d21e7cdd1c0e0bba095cf3b8df2c9caaa6d6a98f63cc71a"} Oct 31 04:18:36.479703 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:36.479675 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-718ebc99-j2jrs" Oct 31 04:18:36.607876 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:36.607797 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/619a5998-8522-4f92-939a-32f500de6d6c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/619a5998-8522-4f92-939a-32f500de6d6c-maas-api-openshift-ca\") pod \"619a5998-8522-4f92-939a-32f500de6d6c\" (UID: \"619a5998-8522-4f92-939a-32f500de6d6c\") " Oct 31 04:18:36.608036 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:36.607886 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/619a5998-8522-4f92-939a-32f500de6d6c-kube-api-access-kxjlw\" (UniqueName: \"kubernetes.io/projected/619a5998-8522-4f92-939a-32f500de6d6c-kube-api-access-kxjlw\") pod \"619a5998-8522-4f92-939a-32f500de6d6c\" (UID: \"619a5998-8522-4f92-939a-32f500de6d6c\") " Oct 31 04:18:36.608036 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:36.607930 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/619a5998-8522-4f92-939a-32f500de6d6c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/619a5998-8522-4f92-939a-32f500de6d6c-maas-api-xks-ca\") pod \"619a5998-8522-4f92-939a-32f500de6d6c\" (UID: \"619a5998-8522-4f92-939a-32f500de6d6c\") " Oct 31 04:18:36.608202 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:36.608174 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/619a5998-8522-4f92-939a-32f500de6d6c-maas-api-openshift-ca" pod "619a5998-8522-4f92-939a-32f500de6d6c" (UID: "619a5998-8522-4f92-939a-32f500de6d6c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:18:36.609971 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:36.609949 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/619a5998-8522-4f92-939a-32f500de6d6c-maas-api-xks-ca" pod "619a5998-8522-4f92-939a-32f500de6d6c" (UID: "619a5998-8522-4f92-939a-32f500de6d6c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:18:36.610076 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:36.609982 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/619a5998-8522-4f92-939a-32f500de6d6c-kube-api-access-kxjlw" pod "619a5998-8522-4f92-939a-32f500de6d6c" (UID: "619a5998-8522-4f92-939a-32f500de6d6c"). InnerVolumeSpecName "kube-api-access-kxjlw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:18:36.709285 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:36.709259 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/619a5998-8522-4f92-939a-32f500de6d6c-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:18:36.709285 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:36.709283 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/619a5998-8522-4f92-939a-32f500de6d6c-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:18:36.709441 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:36.709294 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxjlw\" (UniqueName: \"kubernetes.io/projected/619a5998-8522-4f92-939a-32f500de6d6c-kube-api-access-kxjlw\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:18:37.330195 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:37.330168 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-718ebc99-j2jrs" Oct 31 04:18:37.330421 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:37.330159 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-718ebc99-j2jrs" event={"ID":"619a5998-8522-4f92-939a-32f500de6d6c","Type":"ContainerDied","Data":"256e66cc6bbb39eab27be2b7bd648ce87157dd73d96c144d178c9aab68a29dcd"} Oct 31 04:18:37.330421 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:37.330323 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="256e66cc6bbb39eab27be2b7bd648ce87157dd73d96c144d178c9aab68a29dcd" Oct 31 04:18:41.639253 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:41.639220 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q"] Oct 31 04:18:41.640098 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:41.640077 5019 cpu_manager.go:414] "containerMap: removing container" podUID="619a5998-8522-4f92-939a-32f500de6d6c" containerName="revoke-keys" Oct 31 04:18:41.640166 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:41.640102 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="619a5998-8522-4f92-939a-32f500de6d6c" containerName="revoke-keys" Oct 31 04:18:41.640272 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:41.640258 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="619a5998-8522-4f92-939a-32f500de6d6c" containerName="revoke-keys" Oct 31 04:18:41.644462 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:41.644441 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" Oct 31 04:18:41.661181 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:41.661156 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q"] Oct 31 04:18:41.669032 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:41.669013 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-965f61\"/\"distinct-2-w0-965f61-kserve-self-signed-certs\"" Oct 31 04:18:41.766119 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:41.766096 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0f4b8c2-14f4-4052-bf54-dd65bf59e413-tmp-dir\") pod \"distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q\" (UID: \"e0f4b8c2-14f4-4052-bf54-dd65bf59e413\") " pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" Oct 31 04:18:41.766222 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:41.766209 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0f4b8c2-14f4-4052-bf54-dd65bf59e413-model-cache\") pod \"distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q\" (UID: \"e0f4b8c2-14f4-4052-bf54-dd65bf59e413\") " pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" Oct 31 04:18:41.766310 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:41.766296 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stspp\" (UniqueName: \"kubernetes.io/projected/e0f4b8c2-14f4-4052-bf54-dd65bf59e413-kube-api-access-stspp\") pod \"distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q\" (UID: \"e0f4b8c2-14f4-4052-bf54-dd65bf59e413\") " pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" Oct 31 04:18:41.766387 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:41.766326 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0f4b8c2-14f4-4052-bf54-dd65bf59e413-dshm\") pod \"distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q\" (UID: \"e0f4b8c2-14f4-4052-bf54-dd65bf59e413\") " pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" Oct 31 04:18:41.766387 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:41.766372 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0f4b8c2-14f4-4052-bf54-dd65bf59e413-tls-certs\") pod \"distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q\" (UID: \"e0f4b8c2-14f4-4052-bf54-dd65bf59e413\") " pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" Oct 31 04:18:41.766501 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:41.766458 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0f4b8c2-14f4-4052-bf54-dd65bf59e413-home\") pod \"distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q\" (UID: \"e0f4b8c2-14f4-4052-bf54-dd65bf59e413\") " pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" Oct 31 04:18:41.868221 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:41.868182 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0f4b8c2-14f4-4052-bf54-dd65bf59e413-tmp-dir\") pod \"distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q\" (UID: \"e0f4b8c2-14f4-4052-bf54-dd65bf59e413\") " pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" Oct 31 04:18:41.868442 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:41.868293 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0f4b8c2-14f4-4052-bf54-dd65bf59e413-model-cache\") pod \"distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q\" (UID: \"e0f4b8c2-14f4-4052-bf54-dd65bf59e413\") " pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" Oct 31 04:18:41.868442 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:41.868416 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stspp\" (UniqueName: \"kubernetes.io/projected/e0f4b8c2-14f4-4052-bf54-dd65bf59e413-kube-api-access-stspp\") pod \"distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q\" (UID: \"e0f4b8c2-14f4-4052-bf54-dd65bf59e413\") " pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" Oct 31 04:18:41.868534 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:41.868446 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0f4b8c2-14f4-4052-bf54-dd65bf59e413-dshm\") pod \"distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q\" (UID: \"e0f4b8c2-14f4-4052-bf54-dd65bf59e413\") " pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" Oct 31 04:18:41.868534 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:41.868462 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0f4b8c2-14f4-4052-bf54-dd65bf59e413-tls-certs\") pod \"distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q\" (UID: \"e0f4b8c2-14f4-4052-bf54-dd65bf59e413\") " pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" Oct 31 04:18:41.868534 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:41.868493 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0f4b8c2-14f4-4052-bf54-dd65bf59e413-home\") pod \"distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q\" (UID: \"e0f4b8c2-14f4-4052-bf54-dd65bf59e413\") " pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" Oct 31 04:18:41.868871 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:41.868853 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0f4b8c2-14f4-4052-bf54-dd65bf59e413-home\") pod \"distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q\" (UID: \"e0f4b8c2-14f4-4052-bf54-dd65bf59e413\") " pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" Oct 31 04:18:41.869066 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:41.869053 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0f4b8c2-14f4-4052-bf54-dd65bf59e413-tmp-dir\") pod \"distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q\" (UID: \"e0f4b8c2-14f4-4052-bf54-dd65bf59e413\") " pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" Oct 31 04:18:41.869240 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:41.869226 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0f4b8c2-14f4-4052-bf54-dd65bf59e413-model-cache\") pod \"distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q\" (UID: \"e0f4b8c2-14f4-4052-bf54-dd65bf59e413\") " pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" Oct 31 04:18:41.872566 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:41.872544 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0f4b8c2-14f4-4052-bf54-dd65bf59e413-dshm\") pod \"distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q\" (UID: \"e0f4b8c2-14f4-4052-bf54-dd65bf59e413\") " pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" Oct 31 04:18:41.872956 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:41.872934 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0f4b8c2-14f4-4052-bf54-dd65bf59e413-tls-certs\") pod \"distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q\" (UID: \"e0f4b8c2-14f4-4052-bf54-dd65bf59e413\") " pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" Oct 31 04:18:41.892045 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:41.891992 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stspp\" (UniqueName: \"kubernetes.io/projected/e0f4b8c2-14f4-4052-bf54-dd65bf59e413-kube-api-access-stspp\") pod \"distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q\" (UID: \"e0f4b8c2-14f4-4052-bf54-dd65bf59e413\") " pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" Oct 31 04:18:41.958724 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:41.958698 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" Oct 31 04:18:42.264901 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:42.264873 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q"] Oct 31 04:18:42.265359 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:18:42.265319 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode0f4b8c2_14f4_4052_bf54_dd65bf59e413.slice/crio-6bb6907ed3dda1faa26f2766c6e3bff81200e1b23848c2d09862b0f66af4a63d WatchSource:0}: Error finding container 6bb6907ed3dda1faa26f2766c6e3bff81200e1b23848c2d09862b0f66af4a63d: Status 404 returned error can't find the container with id 6bb6907ed3dda1faa26f2766c6e3bff81200e1b23848c2d09862b0f66af4a63d Oct 31 04:18:42.364234 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:42.364200 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" event={"ID":"e0f4b8c2-14f4-4052-bf54-dd65bf59e413","Type":"ContainerStarted","Data":"3e11da66d7d3a56da9a97a480f10b9e1b4754852bb727e952312c442bd6f2588"} Oct 31 04:18:42.364369 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:42.364250 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" event={"ID":"e0f4b8c2-14f4-4052-bf54-dd65bf59e413","Type":"ContainerStarted","Data":"6bb6907ed3dda1faa26f2766c6e3bff81200e1b23848c2d09862b0f66af4a63d"} Oct 31 04:18:42.407875 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:42.407821 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" podStartSLOduration=1.407804391 podStartE2EDuration="1.407804391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:18:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:18:42.405759917 +0000 UTC m=+877.162097956" watchObservedRunningTime="2026-10-31 04:18:42.407804391 +0000 UTC m=+877.164142431" Oct 31 04:18:42.959158 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:42.959129 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" Oct 31 04:18:45.450869 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:45.450830 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a9eebb8-cvcfp"] Oct 31 04:18:45.454907 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:45.454879 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a9eebb8-cvcfp" Oct 31 04:18:45.483242 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:45.483209 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:18:45.483559 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:45.483321 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:18:45.483696 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:45.483459 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:18:45.483776 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:45.483476 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:18:45.486664 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:45.486633 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a9eebb8-cvcfp"] Oct 31 04:18:45.521667 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:45.521619 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn7qb\" (UniqueName: \"kubernetes.io/projected/3024e5fc-2bb5-4018-b86c-a02d939c0d15-kube-api-access-gn7qb\") pod \"maas-api-revoke-sub-6a9eebb8-cvcfp\" (UID: \"3024e5fc-2bb5-4018-b86c-a02d939c0d15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a9eebb8-cvcfp" Oct 31 04:18:45.521838 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:45.521795 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3024e5fc-2bb5-4018-b86c-a02d939c0d15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a9eebb8-cvcfp\" (UID: \"3024e5fc-2bb5-4018-b86c-a02d939c0d15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a9eebb8-cvcfp" Oct 31 04:18:45.521966 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:45.521944 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3024e5fc-2bb5-4018-b86c-a02d939c0d15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a9eebb8-cvcfp\" (UID: \"3024e5fc-2bb5-4018-b86c-a02d939c0d15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a9eebb8-cvcfp" Oct 31 04:18:45.624026 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:45.623980 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn7qb\" (UniqueName: \"kubernetes.io/projected/3024e5fc-2bb5-4018-b86c-a02d939c0d15-kube-api-access-gn7qb\") pod \"maas-api-revoke-sub-6a9eebb8-cvcfp\" (UID: \"3024e5fc-2bb5-4018-b86c-a02d939c0d15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a9eebb8-cvcfp" Oct 31 04:18:45.624222 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:45.624088 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3024e5fc-2bb5-4018-b86c-a02d939c0d15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a9eebb8-cvcfp\" (UID: \"3024e5fc-2bb5-4018-b86c-a02d939c0d15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a9eebb8-cvcfp" Oct 31 04:18:45.624222 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:45.624187 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3024e5fc-2bb5-4018-b86c-a02d939c0d15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a9eebb8-cvcfp\" (UID: \"3024e5fc-2bb5-4018-b86c-a02d939c0d15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a9eebb8-cvcfp" Oct 31 04:18:45.624957 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:45.624929 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3024e5fc-2bb5-4018-b86c-a02d939c0d15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a9eebb8-cvcfp\" (UID: \"3024e5fc-2bb5-4018-b86c-a02d939c0d15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a9eebb8-cvcfp" Oct 31 04:18:45.626755 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:45.626728 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3024e5fc-2bb5-4018-b86c-a02d939c0d15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a9eebb8-cvcfp\" (UID: \"3024e5fc-2bb5-4018-b86c-a02d939c0d15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a9eebb8-cvcfp" Oct 31 04:18:45.647379 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:45.647326 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn7qb\" (UniqueName: \"kubernetes.io/projected/3024e5fc-2bb5-4018-b86c-a02d939c0d15-kube-api-access-gn7qb\") pod \"maas-api-revoke-sub-6a9eebb8-cvcfp\" (UID: \"3024e5fc-2bb5-4018-b86c-a02d939c0d15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a9eebb8-cvcfp" Oct 31 04:18:45.771652 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:45.771619 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a9eebb8-cvcfp" Oct 31 04:18:46.071872 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:18:46.071836 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3024e5fc_2bb5_4018_b86c_a02d939c0d15.slice/crio-7b1ab539db7a05c535f39f7ffc94cf18bcb74d849ed42ba834fc95b2208bfb3b WatchSource:0}: Error finding container 7b1ab539db7a05c535f39f7ffc94cf18bcb74d849ed42ba834fc95b2208bfb3b: Status 404 returned error can't find the container with id 7b1ab539db7a05c535f39f7ffc94cf18bcb74d849ed42ba834fc95b2208bfb3b Oct 31 04:18:46.072044 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:46.072027 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a9eebb8-cvcfp"] Oct 31 04:18:46.396207 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:46.396156 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a9eebb8-cvcfp" event={"ID":"3024e5fc-2bb5-4018-b86c-a02d939c0d15","Type":"ContainerStarted","Data":"c2b15521cc87fd13e20ec688b4151a7b944e0acce2494bf97e1f1680959d9046"} Oct 31 04:18:46.396367 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:46.396224 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a9eebb8-cvcfp" event={"ID":"3024e5fc-2bb5-4018-b86c-a02d939c0d15","Type":"ContainerStarted","Data":"7b1ab539db7a05c535f39f7ffc94cf18bcb74d849ed42ba834fc95b2208bfb3b"} Oct 31 04:18:46.435763 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:46.435700 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a9eebb8-cvcfp" podStartSLOduration=1.435680987 podStartE2EDuration="1.435680987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:18:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:18:46.435194917 +0000 UTC m=+881.191532955" watchObservedRunningTime="2026-10-31 04:18:46.435680987 +0000 UTC m=+881.192019030" Oct 31 04:18:47.405554 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:47.405453 5019 generic.go:356] "Generic (PLEG): container finished" podID="3024e5fc-2bb5-4018-b86c-a02d939c0d15" containerID="c2b15521cc87fd13e20ec688b4151a7b944e0acce2494bf97e1f1680959d9046" exitCode=0 Oct 31 04:18:47.409416 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:47.409385 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a9eebb8-cvcfp" event={"ID":"3024e5fc-2bb5-4018-b86c-a02d939c0d15","Type":"ContainerDied","Data":"c2b15521cc87fd13e20ec688b4151a7b944e0acce2494bf97e1f1680959d9046"} Oct 31 04:18:48.592147 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:48.590668 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a9eebb8-cvcfp" Oct 31 04:18:48.665549 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:48.665469 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3024e5fc-2bb5-4018-b86c-a02d939c0d15-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3024e5fc-2bb5-4018-b86c-a02d939c0d15-maas-api-xks-ca\") pod \"3024e5fc-2bb5-4018-b86c-a02d939c0d15\" (UID: \"3024e5fc-2bb5-4018-b86c-a02d939c0d15\") " Oct 31 04:18:48.665712 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:48.665614 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3024e5fc-2bb5-4018-b86c-a02d939c0d15-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3024e5fc-2bb5-4018-b86c-a02d939c0d15-maas-api-openshift-ca\") pod \"3024e5fc-2bb5-4018-b86c-a02d939c0d15\" (UID: \"3024e5fc-2bb5-4018-b86c-a02d939c0d15\") " Oct 31 04:18:48.665782 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:48.665724 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3024e5fc-2bb5-4018-b86c-a02d939c0d15-kube-api-access-gn7qb\" (UniqueName: \"kubernetes.io/projected/3024e5fc-2bb5-4018-b86c-a02d939c0d15-kube-api-access-gn7qb\") pod \"3024e5fc-2bb5-4018-b86c-a02d939c0d15\" (UID: \"3024e5fc-2bb5-4018-b86c-a02d939c0d15\") " Oct 31 04:18:48.666064 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:48.666031 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3024e5fc-2bb5-4018-b86c-a02d939c0d15-maas-api-openshift-ca" pod "3024e5fc-2bb5-4018-b86c-a02d939c0d15" (UID: "3024e5fc-2bb5-4018-b86c-a02d939c0d15"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:18:48.666307 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:48.666285 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3024e5fc-2bb5-4018-b86c-a02d939c0d15-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:18:48.667911 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:48.667877 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3024e5fc-2bb5-4018-b86c-a02d939c0d15-maas-api-xks-ca" pod "3024e5fc-2bb5-4018-b86c-a02d939c0d15" (UID: "3024e5fc-2bb5-4018-b86c-a02d939c0d15"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:18:48.673426 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:48.673320 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3024e5fc-2bb5-4018-b86c-a02d939c0d15-kube-api-access-gn7qb" pod "3024e5fc-2bb5-4018-b86c-a02d939c0d15" (UID: "3024e5fc-2bb5-4018-b86c-a02d939c0d15"). InnerVolumeSpecName "kube-api-access-gn7qb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:18:48.767578 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:48.767535 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn7qb\" (UniqueName: \"kubernetes.io/projected/3024e5fc-2bb5-4018-b86c-a02d939c0d15-kube-api-access-gn7qb\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:18:48.767578 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:48.767573 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3024e5fc-2bb5-4018-b86c-a02d939c0d15-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:18:49.409960 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:49.409932 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r"] Oct 31 04:18:49.410512 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:49.410496 5019 cpu_manager.go:414] "containerMap: removing container" podUID="3024e5fc-2bb5-4018-b86c-a02d939c0d15" containerName="revoke-keys" Oct 31 04:18:49.410575 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:49.410514 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3024e5fc-2bb5-4018-b86c-a02d939c0d15" containerName="revoke-keys" Oct 31 04:18:49.410607 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:49.410595 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="3024e5fc-2bb5-4018-b86c-a02d939c0d15" containerName="revoke-keys" Oct 31 04:18:49.414320 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:49.414293 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r" Oct 31 04:18:49.427658 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:49.425682 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a9eebb8-cvcfp" Oct 31 04:18:49.427658 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:49.425722 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a9eebb8-cvcfp" event={"ID":"3024e5fc-2bb5-4018-b86c-a02d939c0d15","Type":"ContainerDied","Data":"7b1ab539db7a05c535f39f7ffc94cf18bcb74d849ed42ba834fc95b2208bfb3b"} Oct 31 04:18:49.427658 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:49.426913 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b1ab539db7a05c535f39f7ffc94cf18bcb74d849ed42ba834fc95b2208bfb3b" Oct 31 04:18:49.434925 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:49.434899 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r"] Oct 31 04:18:49.576649 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:49.576614 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fffb7c60-8cb2-40f0-903c-9b98c62a1741-istio-data\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r\" (UID: \"fffb7c60-8cb2-40f0-903c-9b98c62a1741\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r" Oct 31 04:18:49.576813 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:49.576677 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fffb7c60-8cb2-40f0-903c-9b98c62a1741-workload-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r\" (UID: \"fffb7c60-8cb2-40f0-903c-9b98c62a1741\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r" Oct 31 04:18:49.576813 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:49.576740 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fffb7c60-8cb2-40f0-903c-9b98c62a1741-workload-certs\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r\" (UID: \"fffb7c60-8cb2-40f0-903c-9b98c62a1741\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r" Oct 31 04:18:49.576909 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:49.576811 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fffb7c60-8cb2-40f0-903c-9b98c62a1741-istio-envoy\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r\" (UID: \"fffb7c60-8cb2-40f0-903c-9b98c62a1741\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r" Oct 31 04:18:49.576944 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:49.576907 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fffb7c60-8cb2-40f0-903c-9b98c62a1741-istio-token\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r\" (UID: \"fffb7c60-8cb2-40f0-903c-9b98c62a1741\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r" Oct 31 04:18:49.576980 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:49.576951 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fffb7c60-8cb2-40f0-903c-9b98c62a1741-credential-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r\" (UID: \"fffb7c60-8cb2-40f0-903c-9b98c62a1741\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r" Oct 31 04:18:49.576980 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:49.576971 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbq88\" (UniqueName: \"kubernetes.io/projected/fffb7c60-8cb2-40f0-903c-9b98c62a1741-kube-api-access-lbq88\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r\" (UID: \"fffb7c60-8cb2-40f0-903c-9b98c62a1741\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r" Oct 31 04:18:49.577114 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:49.577091 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fffb7c60-8cb2-40f0-903c-9b98c62a1741-istio-podinfo\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r\" (UID: \"fffb7c60-8cb2-40f0-903c-9b98c62a1741\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r" Oct 31 04:18:49.577158 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:49.577143 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fffb7c60-8cb2-40f0-903c-9b98c62a1741-istiod-ca-cert\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r\" (UID: \"fffb7c60-8cb2-40f0-903c-9b98c62a1741\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r" Oct 31 04:18:49.678836 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:49.678740 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fffb7c60-8cb2-40f0-903c-9b98c62a1741-istio-data\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r\" (UID: \"fffb7c60-8cb2-40f0-903c-9b98c62a1741\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r" Oct 31 04:18:49.678836 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:49.678803 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fffb7c60-8cb2-40f0-903c-9b98c62a1741-workload-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r\" (UID: \"fffb7c60-8cb2-40f0-903c-9b98c62a1741\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r" Oct 31 04:18:49.678836 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:49.678823 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fffb7c60-8cb2-40f0-903c-9b98c62a1741-workload-certs\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r\" (UID: \"fffb7c60-8cb2-40f0-903c-9b98c62a1741\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r" Oct 31 04:18:49.679374 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:49.678878 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fffb7c60-8cb2-40f0-903c-9b98c62a1741-istio-envoy\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r\" (UID: \"fffb7c60-8cb2-40f0-903c-9b98c62a1741\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r" Oct 31 04:18:49.679374 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:49.678918 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fffb7c60-8cb2-40f0-903c-9b98c62a1741-istio-token\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r\" (UID: \"fffb7c60-8cb2-40f0-903c-9b98c62a1741\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r" Oct 31 04:18:49.679374 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:49.678967 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fffb7c60-8cb2-40f0-903c-9b98c62a1741-credential-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r\" (UID: \"fffb7c60-8cb2-40f0-903c-9b98c62a1741\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r" Oct 31 04:18:49.679374 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:49.678988 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbq88\" (UniqueName: \"kubernetes.io/projected/fffb7c60-8cb2-40f0-903c-9b98c62a1741-kube-api-access-lbq88\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r\" (UID: \"fffb7c60-8cb2-40f0-903c-9b98c62a1741\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r" Oct 31 04:18:49.679374 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:49.679033 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fffb7c60-8cb2-40f0-903c-9b98c62a1741-istio-podinfo\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r\" (UID: \"fffb7c60-8cb2-40f0-903c-9b98c62a1741\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r" Oct 31 04:18:49.679374 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:49.679070 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fffb7c60-8cb2-40f0-903c-9b98c62a1741-istiod-ca-cert\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r\" (UID: \"fffb7c60-8cb2-40f0-903c-9b98c62a1741\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r" Oct 31 04:18:49.679374 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:49.679216 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fffb7c60-8cb2-40f0-903c-9b98c62a1741-istio-data\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r\" (UID: \"fffb7c60-8cb2-40f0-903c-9b98c62a1741\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r" Oct 31 04:18:49.679374 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:49.679270 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fffb7c60-8cb2-40f0-903c-9b98c62a1741-workload-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r\" (UID: \"fffb7c60-8cb2-40f0-903c-9b98c62a1741\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r" Oct 31 04:18:49.679374 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:49.679306 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fffb7c60-8cb2-40f0-903c-9b98c62a1741-credential-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r\" (UID: \"fffb7c60-8cb2-40f0-903c-9b98c62a1741\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r" Oct 31 04:18:49.679816 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:49.679462 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fffb7c60-8cb2-40f0-903c-9b98c62a1741-workload-certs\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r\" (UID: \"fffb7c60-8cb2-40f0-903c-9b98c62a1741\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r" Oct 31 04:18:49.679854 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:49.679823 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fffb7c60-8cb2-40f0-903c-9b98c62a1741-istiod-ca-cert\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r\" (UID: \"fffb7c60-8cb2-40f0-903c-9b98c62a1741\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r" Oct 31 04:18:49.681288 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:49.681265 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fffb7c60-8cb2-40f0-903c-9b98c62a1741-istio-envoy\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r\" (UID: \"fffb7c60-8cb2-40f0-903c-9b98c62a1741\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r" Oct 31 04:18:49.681662 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:49.681640 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fffb7c60-8cb2-40f0-903c-9b98c62a1741-istio-podinfo\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r\" (UID: \"fffb7c60-8cb2-40f0-903c-9b98c62a1741\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r" Oct 31 04:18:49.694936 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:49.694911 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-952b35ab-dqbsg"] Oct 31 04:18:49.699553 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:49.699532 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-952b35ab-dqbsg" Oct 31 04:18:49.703984 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:49.703956 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fffb7c60-8cb2-40f0-903c-9b98c62a1741-istio-token\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r\" (UID: \"fffb7c60-8cb2-40f0-903c-9b98c62a1741\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r" Oct 31 04:18:49.704841 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:49.704823 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbq88\" (UniqueName: \"kubernetes.io/projected/fffb7c60-8cb2-40f0-903c-9b98c62a1741-kube-api-access-lbq88\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r\" (UID: \"fffb7c60-8cb2-40f0-903c-9b98c62a1741\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r" Oct 31 04:18:49.715207 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:49.714591 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:18:49.715207 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:49.714684 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:18:49.715207 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:49.714724 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:18:49.715207 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:49.714771 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:18:49.719224 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:49.719203 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-952b35ab-dqbsg"] Oct 31 04:18:49.755100 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:49.755069 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r" Oct 31 04:18:49.780885 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:49.780858 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/305f3b1f-387e-483d-a472-473fb4e33b3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-952b35ab-dqbsg\" (UID: \"305f3b1f-387e-483d-a472-473fb4e33b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-952b35ab-dqbsg" Oct 31 04:18:49.781060 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:49.780900 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9vgm\" (UniqueName: \"kubernetes.io/projected/305f3b1f-387e-483d-a472-473fb4e33b3e-kube-api-access-g9vgm\") pod \"maas-api-revoke-sub-952b35ab-dqbsg\" (UID: \"305f3b1f-387e-483d-a472-473fb4e33b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-952b35ab-dqbsg" Oct 31 04:18:49.781131 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:49.781055 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/305f3b1f-387e-483d-a472-473fb4e33b3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-952b35ab-dqbsg\" (UID: \"305f3b1f-387e-483d-a472-473fb4e33b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-952b35ab-dqbsg" Oct 31 04:18:49.882315 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:49.882270 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/305f3b1f-387e-483d-a472-473fb4e33b3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-952b35ab-dqbsg\" (UID: \"305f3b1f-387e-483d-a472-473fb4e33b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-952b35ab-dqbsg" Oct 31 04:18:49.882486 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:49.882364 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9vgm\" (UniqueName: \"kubernetes.io/projected/305f3b1f-387e-483d-a472-473fb4e33b3e-kube-api-access-g9vgm\") pod \"maas-api-revoke-sub-952b35ab-dqbsg\" (UID: \"305f3b1f-387e-483d-a472-473fb4e33b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-952b35ab-dqbsg" Oct 31 04:18:49.882486 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:49.882405 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/305f3b1f-387e-483d-a472-473fb4e33b3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-952b35ab-dqbsg\" (UID: \"305f3b1f-387e-483d-a472-473fb4e33b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-952b35ab-dqbsg" Oct 31 04:18:49.882953 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:49.882923 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/305f3b1f-387e-483d-a472-473fb4e33b3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-952b35ab-dqbsg\" (UID: \"305f3b1f-387e-483d-a472-473fb4e33b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-952b35ab-dqbsg" Oct 31 04:18:49.885315 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:49.885292 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/305f3b1f-387e-483d-a472-473fb4e33b3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-952b35ab-dqbsg\" (UID: \"305f3b1f-387e-483d-a472-473fb4e33b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-952b35ab-dqbsg" Oct 31 04:18:49.906272 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:49.906249 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9vgm\" (UniqueName: \"kubernetes.io/projected/305f3b1f-387e-483d-a472-473fb4e33b3e-kube-api-access-g9vgm\") pod \"maas-api-revoke-sub-952b35ab-dqbsg\" (UID: \"305f3b1f-387e-483d-a472-473fb4e33b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-952b35ab-dqbsg" Oct 31 04:18:50.040552 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:50.040522 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-952b35ab-dqbsg" Oct 31 04:18:50.074060 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:50.073423 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r"] Oct 31 04:18:50.076199 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:18:50.076163 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfffb7c60_8cb2_40f0_903c_9b98c62a1741.slice/crio-86e8307da6c3ff5799de89d143615d2f97529769b6b70660bacf8dd77f009d8d WatchSource:0}: Error finding container 86e8307da6c3ff5799de89d143615d2f97529769b6b70660bacf8dd77f009d8d: Status 404 returned error can't find the container with id 86e8307da6c3ff5799de89d143615d2f97529769b6b70660bacf8dd77f009d8d Oct 31 04:18:50.078972 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:50.078942 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:18:50.079044 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:50.079007 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:18:50.079044 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:50.079034 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:18:50.347836 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:50.347808 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-952b35ab-dqbsg"] Oct 31 04:18:50.350756 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:18:50.350723 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod305f3b1f_387e_483d_a472_473fb4e33b3e.slice/crio-bb93909ce33d57376f14f46c01b0b44affb4813e699328711ef3bf3b2762c443 WatchSource:0}: Error finding container bb93909ce33d57376f14f46c01b0b44affb4813e699328711ef3bf3b2762c443: Status 404 returned error can't find the container with id bb93909ce33d57376f14f46c01b0b44affb4813e699328711ef3bf3b2762c443 Oct 31 04:18:50.435753 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:50.435720 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r" event={"ID":"fffb7c60-8cb2-40f0-903c-9b98c62a1741","Type":"ContainerStarted","Data":"d13c99ec9cfcc52db329243fc9f3e286ec2b59763c5037648baa5571038be0f6"} Oct 31 04:18:50.435929 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:50.435775 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r" event={"ID":"fffb7c60-8cb2-40f0-903c-9b98c62a1741","Type":"ContainerStarted","Data":"86e8307da6c3ff5799de89d143615d2f97529769b6b70660bacf8dd77f009d8d"} Oct 31 04:18:50.437504 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:50.437461 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-952b35ab-dqbsg" event={"ID":"305f3b1f-387e-483d-a472-473fb4e33b3e","Type":"ContainerStarted","Data":"bb93909ce33d57376f14f46c01b0b44affb4813e699328711ef3bf3b2762c443"} Oct 31 04:18:50.487540 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:50.487473 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r" podStartSLOduration=1.487456025 podStartE2EDuration="1.487456025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:18:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:18:50.485270164 +0000 UTC m=+885.241608240" watchObservedRunningTime="2026-10-31 04:18:50.487456025 +0000 UTC m=+885.243794083" Oct 31 04:18:50.756028 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:50.755987 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r" Oct 31 04:18:51.447367 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:51.447320 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-952b35ab-dqbsg" event={"ID":"305f3b1f-387e-483d-a472-473fb4e33b3e","Type":"ContainerStarted","Data":"e4d9a7262157688bd59f57b898ce3b513977aba993de2d5906ad6c1ed35070af"} Oct 31 04:18:51.505710 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:51.505649 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-952b35ab-dqbsg" podStartSLOduration=2.505628098 podStartE2EDuration="2.505628098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:18:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:18:51.49394999 +0000 UTC m=+886.250288031" watchObservedRunningTime="2026-10-31 04:18:51.505628098 +0000 UTC m=+886.261966138" Oct 31 04:18:51.757457 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:51.757421 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 31 04:18:51.757904 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:51.757478 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r" podUID="fffb7c60-8cb2-40f0-903c-9b98c62a1741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 31 04:18:51.959880 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:51.959846 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" Oct 31 04:18:52.001501 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:52.001468 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" Oct 31 04:18:52.421362 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:18:52.420954 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:18:52.455519 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:52.455486 5019 generic.go:356] "Generic (PLEG): container finished" podID="305f3b1f-387e-483d-a472-473fb4e33b3e" containerID="e4d9a7262157688bd59f57b898ce3b513977aba993de2d5906ad6c1ed35070af" exitCode=0 Oct 31 04:18:52.455750 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:52.455597 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-952b35ab-dqbsg" event={"ID":"305f3b1f-387e-483d-a472-473fb4e33b3e","Type":"ContainerDied","Data":"e4d9a7262157688bd59f57b898ce3b513977aba993de2d5906ad6c1ed35070af"} Oct 31 04:18:52.466709 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:52.466684 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" Oct 31 04:18:52.755691 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:52.755658 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 31 04:18:52.755838 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:52.755712 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r" podUID="fffb7c60-8cb2-40f0-903c-9b98c62a1741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 31 04:18:53.715874 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:53.715795 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-952b35ab-dqbsg" Oct 31 04:18:53.756040 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:53.755994 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 31 04:18:53.756634 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:53.756071 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r" podUID="fffb7c60-8cb2-40f0-903c-9b98c62a1741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 31 04:18:53.838594 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:53.838560 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/305f3b1f-387e-483d-a472-473fb4e33b3e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/305f3b1f-387e-483d-a472-473fb4e33b3e-maas-api-openshift-ca\") pod \"305f3b1f-387e-483d-a472-473fb4e33b3e\" (UID: \"305f3b1f-387e-483d-a472-473fb4e33b3e\") " Oct 31 04:18:53.838915 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:53.838891 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/305f3b1f-387e-483d-a472-473fb4e33b3e-kube-api-access-g9vgm\" (UniqueName: \"kubernetes.io/projected/305f3b1f-387e-483d-a472-473fb4e33b3e-kube-api-access-g9vgm\") pod \"305f3b1f-387e-483d-a472-473fb4e33b3e\" (UID: \"305f3b1f-387e-483d-a472-473fb4e33b3e\") " Oct 31 04:18:53.839027 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:53.838996 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/305f3b1f-387e-483d-a472-473fb4e33b3e-maas-api-openshift-ca" pod "305f3b1f-387e-483d-a472-473fb4e33b3e" (UID: "305f3b1f-387e-483d-a472-473fb4e33b3e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:18:53.839382 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:53.839359 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/305f3b1f-387e-483d-a472-473fb4e33b3e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/305f3b1f-387e-483d-a472-473fb4e33b3e-maas-api-xks-ca\") pod \"305f3b1f-387e-483d-a472-473fb4e33b3e\" (UID: \"305f3b1f-387e-483d-a472-473fb4e33b3e\") " Oct 31 04:18:53.840170 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:53.840151 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/305f3b1f-387e-483d-a472-473fb4e33b3e-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:18:53.844870 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:53.842582 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/305f3b1f-387e-483d-a472-473fb4e33b3e-kube-api-access-g9vgm" pod "305f3b1f-387e-483d-a472-473fb4e33b3e" (UID: "305f3b1f-387e-483d-a472-473fb4e33b3e"). InnerVolumeSpecName "kube-api-access-g9vgm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:18:53.845419 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:53.845398 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/305f3b1f-387e-483d-a472-473fb4e33b3e-maas-api-xks-ca" pod "305f3b1f-387e-483d-a472-473fb4e33b3e" (UID: "305f3b1f-387e-483d-a472-473fb4e33b3e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:18:53.941191 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:53.941153 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/305f3b1f-387e-483d-a472-473fb4e33b3e-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:18:53.941191 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:53.941185 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9vgm\" (UniqueName: \"kubernetes.io/projected/305f3b1f-387e-483d-a472-473fb4e33b3e-kube-api-access-g9vgm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:18:54.494531 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:54.494488 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-952b35ab-dqbsg" event={"ID":"305f3b1f-387e-483d-a472-473fb4e33b3e","Type":"ContainerDied","Data":"bb93909ce33d57376f14f46c01b0b44affb4813e699328711ef3bf3b2762c443"} Oct 31 04:18:54.494705 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:54.494556 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb93909ce33d57376f14f46c01b0b44affb4813e699328711ef3bf3b2762c443" Oct 31 04:18:54.494705 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:54.494621 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-952b35ab-dqbsg" Oct 31 04:18:54.759415 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:54.758703 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 31 04:18:54.759415 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:54.758764 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r" podUID="fffb7c60-8cb2-40f0-903c-9b98c62a1741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 31 04:18:55.755515 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:55.755481 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 31 04:18:55.755848 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:55.755816 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r" podUID="fffb7c60-8cb2-40f0-903c-9b98c62a1741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 31 04:18:56.755849 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:56.755810 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 31 04:18:56.756450 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:56.756414 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r" podUID="fffb7c60-8cb2-40f0-903c-9b98c62a1741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 31 04:18:58.797357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:58.797300 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:18:58.798071 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:58.798035 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r" podUID="fffb7c60-8cb2-40f0-903c-9b98c62a1741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:18:59.840537 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:59.840490 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:18:59.841062 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:59.840603 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r" podUID="fffb7c60-8cb2-40f0-903c-9b98c62a1741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:18:59.898020 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:59.897983 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r" Oct 31 04:18:59.898521 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:59.898493 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r" Oct 31 04:18:59.899178 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:18:59.899160 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r" Oct 31 04:19:04.886571 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:04.886530 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb113745-5x7gx"] Oct 31 04:19:04.887364 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:04.887346 5019 cpu_manager.go:414] "containerMap: removing container" podUID="305f3b1f-387e-483d-a472-473fb4e33b3e" containerName="revoke-keys" Oct 31 04:19:04.887431 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:04.887366 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="305f3b1f-387e-483d-a472-473fb4e33b3e" containerName="revoke-keys" Oct 31 04:19:04.887597 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:04.887585 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="305f3b1f-387e-483d-a472-473fb4e33b3e" containerName="revoke-keys" Oct 31 04:19:04.891947 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:04.891926 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb113745-5x7gx" Oct 31 04:19:04.913804 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:04.913763 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb113745-5x7gx"] Oct 31 04:19:04.924238 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:04.924204 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:19:04.924465 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:04.924248 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:19:04.924465 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:04.924325 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:19:04.924465 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:04.924374 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:19:04.962897 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:04.962855 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqvmd\" (UniqueName: \"kubernetes.io/projected/3bb40fb5-2be2-4273-93f4-fe4d4ca8ddef-kube-api-access-fqvmd\") pod \"maas-api-revoke-sub-fb113745-5x7gx\" (UID: \"3bb40fb5-2be2-4273-93f4-fe4d4ca8ddef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb113745-5x7gx" Oct 31 04:19:04.963106 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:04.962902 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bb40fb5-2be2-4273-93f4-fe4d4ca8ddef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb113745-5x7gx\" (UID: \"3bb40fb5-2be2-4273-93f4-fe4d4ca8ddef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb113745-5x7gx" Oct 31 04:19:04.963184 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:04.963138 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bb40fb5-2be2-4273-93f4-fe4d4ca8ddef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb113745-5x7gx\" (UID: \"3bb40fb5-2be2-4273-93f4-fe4d4ca8ddef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb113745-5x7gx" Oct 31 04:19:05.064174 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:05.064131 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bb40fb5-2be2-4273-93f4-fe4d4ca8ddef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb113745-5x7gx\" (UID: \"3bb40fb5-2be2-4273-93f4-fe4d4ca8ddef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb113745-5x7gx" Oct 31 04:19:05.064380 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:05.064251 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqvmd\" (UniqueName: \"kubernetes.io/projected/3bb40fb5-2be2-4273-93f4-fe4d4ca8ddef-kube-api-access-fqvmd\") pod \"maas-api-revoke-sub-fb113745-5x7gx\" (UID: \"3bb40fb5-2be2-4273-93f4-fe4d4ca8ddef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb113745-5x7gx" Oct 31 04:19:05.064380 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:05.064277 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bb40fb5-2be2-4273-93f4-fe4d4ca8ddef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb113745-5x7gx\" (UID: \"3bb40fb5-2be2-4273-93f4-fe4d4ca8ddef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb113745-5x7gx" Oct 31 04:19:05.065143 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:05.065040 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bb40fb5-2be2-4273-93f4-fe4d4ca8ddef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb113745-5x7gx\" (UID: \"3bb40fb5-2be2-4273-93f4-fe4d4ca8ddef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb113745-5x7gx" Oct 31 04:19:05.067055 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:05.067022 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bb40fb5-2be2-4273-93f4-fe4d4ca8ddef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb113745-5x7gx\" (UID: \"3bb40fb5-2be2-4273-93f4-fe4d4ca8ddef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb113745-5x7gx" Oct 31 04:19:05.087624 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:05.087588 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqvmd\" (UniqueName: \"kubernetes.io/projected/3bb40fb5-2be2-4273-93f4-fe4d4ca8ddef-kube-api-access-fqvmd\") pod \"maas-api-revoke-sub-fb113745-5x7gx\" (UID: \"3bb40fb5-2be2-4273-93f4-fe4d4ca8ddef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb113745-5x7gx" Oct 31 04:19:05.206070 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:05.205982 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb113745-5x7gx" Oct 31 04:19:05.518537 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:05.518505 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb113745-5x7gx"] Oct 31 04:19:05.519843 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:19:05.519795 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bb40fb5_2be2_4273_93f4_fe4d4ca8ddef.slice/crio-83e2a2c85e6ff8badce9dee04b91b8d0467466ed63512de31f12478f787feaad WatchSource:0}: Error finding container 83e2a2c85e6ff8badce9dee04b91b8d0467466ed63512de31f12478f787feaad: Status 404 returned error can't find the container with id 83e2a2c85e6ff8badce9dee04b91b8d0467466ed63512de31f12478f787feaad Oct 31 04:19:05.578959 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:05.578933 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s88kp_3945ff65-e877-4372-9334-3968c68b7295/console-operator/2.log" Oct 31 04:19:05.580351 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:05.580313 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s88kp_3945ff65-e877-4372-9334-3968c68b7295/console-operator/2.log" Oct 31 04:19:05.971084 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:05.971040 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb113745-5x7gx" event={"ID":"3bb40fb5-2be2-4273-93f4-fe4d4ca8ddef","Type":"ContainerStarted","Data":"c348f48234dc85b6a574036dfed847fc43be2fadb935ae94fa9c6a97b4344f81"} Oct 31 04:19:05.971600 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:05.971104 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb113745-5x7gx" event={"ID":"3bb40fb5-2be2-4273-93f4-fe4d4ca8ddef","Type":"ContainerStarted","Data":"83e2a2c85e6ff8badce9dee04b91b8d0467466ed63512de31f12478f787feaad"} Oct 31 04:19:06.014729 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:06.014675 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb113745-5x7gx" podStartSLOduration=2.014657537 podStartE2EDuration="2.014657537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:19:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:19:06.010433248 +0000 UTC m=+900.766771291" watchObservedRunningTime="2026-10-31 04:19:06.014657537 +0000 UTC m=+900.770995578" Oct 31 04:19:06.984121 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:06.984086 5019 generic.go:356] "Generic (PLEG): container finished" podID="3bb40fb5-2be2-4273-93f4-fe4d4ca8ddef" containerID="c348f48234dc85b6a574036dfed847fc43be2fadb935ae94fa9c6a97b4344f81" exitCode=0 Oct 31 04:19:06.984653 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:06.984176 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb113745-5x7gx" event={"ID":"3bb40fb5-2be2-4273-93f4-fe4d4ca8ddef","Type":"ContainerDied","Data":"c348f48234dc85b6a574036dfed847fc43be2fadb935ae94fa9c6a97b4344f81"} Oct 31 04:19:08.163687 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:08.163663 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb113745-5x7gx" Oct 31 04:19:08.306964 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:08.306876 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bb40fb5-2be2-4273-93f4-fe4d4ca8ddef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bb40fb5-2be2-4273-93f4-fe4d4ca8ddef-maas-api-xks-ca\") pod \"3bb40fb5-2be2-4273-93f4-fe4d4ca8ddef\" (UID: \"3bb40fb5-2be2-4273-93f4-fe4d4ca8ddef\") " Oct 31 04:19:08.307115 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:08.307035 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bb40fb5-2be2-4273-93f4-fe4d4ca8ddef-kube-api-access-fqvmd\" (UniqueName: \"kubernetes.io/projected/3bb40fb5-2be2-4273-93f4-fe4d4ca8ddef-kube-api-access-fqvmd\") pod \"3bb40fb5-2be2-4273-93f4-fe4d4ca8ddef\" (UID: \"3bb40fb5-2be2-4273-93f4-fe4d4ca8ddef\") " Oct 31 04:19:08.307115 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:08.307096 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bb40fb5-2be2-4273-93f4-fe4d4ca8ddef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bb40fb5-2be2-4273-93f4-fe4d4ca8ddef-maas-api-openshift-ca\") pod \"3bb40fb5-2be2-4273-93f4-fe4d4ca8ddef\" (UID: \"3bb40fb5-2be2-4273-93f4-fe4d4ca8ddef\") " Oct 31 04:19:08.307574 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:08.307539 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bb40fb5-2be2-4273-93f4-fe4d4ca8ddef-maas-api-openshift-ca" pod "3bb40fb5-2be2-4273-93f4-fe4d4ca8ddef" (UID: "3bb40fb5-2be2-4273-93f4-fe4d4ca8ddef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:19:08.309190 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:08.309163 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bb40fb5-2be2-4273-93f4-fe4d4ca8ddef-maas-api-xks-ca" pod "3bb40fb5-2be2-4273-93f4-fe4d4ca8ddef" (UID: "3bb40fb5-2be2-4273-93f4-fe4d4ca8ddef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:19:08.309387 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:08.309353 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bb40fb5-2be2-4273-93f4-fe4d4ca8ddef-kube-api-access-fqvmd" pod "3bb40fb5-2be2-4273-93f4-fe4d4ca8ddef" (UID: "3bb40fb5-2be2-4273-93f4-fe4d4ca8ddef"). InnerVolumeSpecName "kube-api-access-fqvmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:19:08.408564 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:08.408529 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bb40fb5-2be2-4273-93f4-fe4d4ca8ddef-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:19:08.408564 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:08.408562 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqvmd\" (UniqueName: \"kubernetes.io/projected/3bb40fb5-2be2-4273-93f4-fe4d4ca8ddef-kube-api-access-fqvmd\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:19:08.408761 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:08.408579 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bb40fb5-2be2-4273-93f4-fe4d4ca8ddef-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:19:09.005933 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:09.005878 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb113745-5x7gx" event={"ID":"3bb40fb5-2be2-4273-93f4-fe4d4ca8ddef","Type":"ContainerDied","Data":"83e2a2c85e6ff8badce9dee04b91b8d0467466ed63512de31f12478f787feaad"} Oct 31 04:19:09.006113 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:09.006015 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83e2a2c85e6ff8badce9dee04b91b8d0467466ed63512de31f12478f787feaad" Oct 31 04:19:09.006113 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:09.006110 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb113745-5x7gx" Oct 31 04:19:13.494362 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:13.490456 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:13.945921 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:13.945750 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" podUID="b4ced3ab-41e4-4c54-93eb-95eddaae992d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:14.070576 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:14.070516 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:14.165527 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:14.165414 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" podUID="dccd15a9-47f5-4bd1-b21b-9d5ca8b05845" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 31 04:19:14.205280 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:14.205163 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:14.501382 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:14.496487 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 31 04:19:14.501382 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:14.497300 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" Oct 31 04:19:14.859608 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:14.859473 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:19:14.859908 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:14.859866 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj" podUID="1453873b-b35e-40f3-8c37-ad757223b110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:14.984359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:14.968426 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" podUID="b4ced3ab-41e4-4c54-93eb-95eddaae992d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:14.984359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:14.977139 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" Oct 31 04:19:15.097963 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:15.097916 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:15.212864 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:15.122448 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" podUID="8e2c7041-ad52-49f4-8b55-d16691f4938f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:15.252087 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:15.213595 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" Oct 31 04:19:15.276387 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:15.214164 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" podUID="dccd15a9-47f5-4bd1-b21b-9d5ca8b05845" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:15.342365 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:15.214211 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:15.342365 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:15.341633 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" Oct 31 04:19:15.342365 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:15.341700 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" Oct 31 04:19:15.342682 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:15.340702 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" podUID="b4ced3ab-41e4-4c54-93eb-95eddaae992d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:15.457355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:15.454218 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" Oct 31 04:19:15.472448 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:15.467495 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" podUID="e0f4b8c2-14f4-4052-bf54-dd65bf59e413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:15.472448 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:15.468430 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" podUID="643433d8-418e-4102-8bc3-cd544932614a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 04:19:15.552396 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:15.532989 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 31 04:19:16.022409 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:16.015731 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" podUID="b4ced3ab-41e4-4c54-93eb-95eddaae992d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:16.254192 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:16.254144 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" podUID="8e2c7041-ad52-49f4-8b55-d16691f4938f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 31 04:19:16.341362 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:16.341170 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" podUID="dccd15a9-47f5-4bd1-b21b-9d5ca8b05845" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:16.341914 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:16.341747 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:16.379363 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:16.375730 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" Oct 31 04:19:16.404359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:16.379818 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" Oct 31 04:19:16.475005 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:16.474961 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" podUID="643433d8-418e-4102-8bc3-cd544932614a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:16.476316 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:16.476290 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" Oct 31 04:19:16.495410 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:16.479464 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 31 04:19:16.713531 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:16.713406 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" Oct 31 04:19:16.769749 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:16.769719 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" Oct 31 04:19:16.776789 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:16.776758 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" Oct 31 04:19:16.814025 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:16.813995 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" Oct 31 04:19:16.894130 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:16.894097 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" Oct 31 04:19:17.225701 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:17.225652 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cf46490-9gq8n"] Oct 31 04:19:17.226676 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:17.226647 5019 cpu_manager.go:414] "containerMap: removing container" podUID="3bb40fb5-2be2-4273-93f4-fe4d4ca8ddef" containerName="revoke-keys" Oct 31 04:19:17.226676 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:17.226675 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bb40fb5-2be2-4273-93f4-fe4d4ca8ddef" containerName="revoke-keys" Oct 31 04:19:17.226905 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:17.226810 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bb40fb5-2be2-4273-93f4-fe4d4ca8ddef" containerName="revoke-keys" Oct 31 04:19:17.231577 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:17.231542 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf46490-9gq8n" Oct 31 04:19:17.255453 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:17.254111 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:19:17.255453 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:17.254283 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:19:17.255453 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:17.254913 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:19:17.259788 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:17.259740 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79da2fc6-818e-4f6b-a99c-53ab652e796f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cf46490-9gq8n\" (UID: \"79da2fc6-818e-4f6b-a99c-53ab652e796f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf46490-9gq8n" Oct 31 04:19:17.260008 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:17.259988 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbdwk\" (UniqueName: \"kubernetes.io/projected/79da2fc6-818e-4f6b-a99c-53ab652e796f-kube-api-access-nbdwk\") pod \"maas-api-revoke-sub-0cf46490-9gq8n\" (UID: \"79da2fc6-818e-4f6b-a99c-53ab652e796f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf46490-9gq8n" Oct 31 04:19:17.260096 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:17.260022 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79da2fc6-818e-4f6b-a99c-53ab652e796f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cf46490-9gq8n\" (UID: \"79da2fc6-818e-4f6b-a99c-53ab652e796f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf46490-9gq8n" Oct 31 04:19:17.261787 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:17.261758 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:19:17.262821 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:17.262798 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cf46490-9gq8n"] Oct 31 04:19:17.361167 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:17.361123 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79da2fc6-818e-4f6b-a99c-53ab652e796f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cf46490-9gq8n\" (UID: \"79da2fc6-818e-4f6b-a99c-53ab652e796f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf46490-9gq8n" Oct 31 04:19:17.361411 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:17.361391 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbdwk\" (UniqueName: \"kubernetes.io/projected/79da2fc6-818e-4f6b-a99c-53ab652e796f-kube-api-access-nbdwk\") pod \"maas-api-revoke-sub-0cf46490-9gq8n\" (UID: \"79da2fc6-818e-4f6b-a99c-53ab652e796f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf46490-9gq8n" Oct 31 04:19:17.361462 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:17.361427 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79da2fc6-818e-4f6b-a99c-53ab652e796f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cf46490-9gq8n\" (UID: \"79da2fc6-818e-4f6b-a99c-53ab652e796f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf46490-9gq8n" Oct 31 04:19:17.362270 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:17.362246 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79da2fc6-818e-4f6b-a99c-53ab652e796f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cf46490-9gq8n\" (UID: \"79da2fc6-818e-4f6b-a99c-53ab652e796f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf46490-9gq8n" Oct 31 04:19:17.364094 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:17.364068 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79da2fc6-818e-4f6b-a99c-53ab652e796f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cf46490-9gq8n\" (UID: \"79da2fc6-818e-4f6b-a99c-53ab652e796f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf46490-9gq8n" Oct 31 04:19:17.385487 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:17.385449 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbdwk\" (UniqueName: \"kubernetes.io/projected/79da2fc6-818e-4f6b-a99c-53ab652e796f-kube-api-access-nbdwk\") pod \"maas-api-revoke-sub-0cf46490-9gq8n\" (UID: \"79da2fc6-818e-4f6b-a99c-53ab652e796f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf46490-9gq8n" Oct 31 04:19:17.571044 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:17.570943 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf46490-9gq8n" Oct 31 04:19:17.878296 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:17.878210 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cf46490-9gq8n"] Oct 31 04:19:17.884529 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:19:17.884490 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79da2fc6_818e_4f6b_a99c_53ab652e796f.slice/crio-09c84c81bc1fbff0508b9c1da89d862ffaf9d01a1e798a2998c21266b5996e2e WatchSource:0}: Error finding container 09c84c81bc1fbff0508b9c1da89d862ffaf9d01a1e798a2998c21266b5996e2e: Status 404 returned error can't find the container with id 09c84c81bc1fbff0508b9c1da89d862ffaf9d01a1e798a2998c21266b5996e2e Oct 31 04:19:18.713799 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:18.713706 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c009b7e-4rsg4"] Oct 31 04:19:18.719030 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:18.719003 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c009b7e-4rsg4" Oct 31 04:19:18.735428 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:18.735397 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c009b7e-4rsg4"] Oct 31 04:19:18.781400 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:18.781341 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brqdd\" (UniqueName: \"kubernetes.io/projected/42c8c3da-6bd9-40b8-9fed-e34b8e1114c7-kube-api-access-brqdd\") pod \"maas-api-revoke-sub-2c009b7e-4rsg4\" (UID: \"42c8c3da-6bd9-40b8-9fed-e34b8e1114c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c009b7e-4rsg4" Oct 31 04:19:18.781608 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:18.781424 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42c8c3da-6bd9-40b8-9fed-e34b8e1114c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c009b7e-4rsg4\" (UID: \"42c8c3da-6bd9-40b8-9fed-e34b8e1114c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c009b7e-4rsg4" Oct 31 04:19:18.781608 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:18.781514 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42c8c3da-6bd9-40b8-9fed-e34b8e1114c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c009b7e-4rsg4\" (UID: \"42c8c3da-6bd9-40b8-9fed-e34b8e1114c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c009b7e-4rsg4" Oct 31 04:19:18.871646 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:18.871614 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf46490-9gq8n" event={"ID":"79da2fc6-818e-4f6b-a99c-53ab652e796f","Type":"ContainerStarted","Data":"08ef458afe0b08196e861aa53f685033bf2f564bfeec4ad23358d2a241ec1648"} Oct 31 04:19:18.871797 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:18.871682 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf46490-9gq8n" event={"ID":"79da2fc6-818e-4f6b-a99c-53ab652e796f","Type":"ContainerStarted","Data":"09c84c81bc1fbff0508b9c1da89d862ffaf9d01a1e798a2998c21266b5996e2e"} Oct 31 04:19:18.882999 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:18.882971 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42c8c3da-6bd9-40b8-9fed-e34b8e1114c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c009b7e-4rsg4\" (UID: \"42c8c3da-6bd9-40b8-9fed-e34b8e1114c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c009b7e-4rsg4" Oct 31 04:19:18.883496 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:18.883069 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42c8c3da-6bd9-40b8-9fed-e34b8e1114c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c009b7e-4rsg4\" (UID: \"42c8c3da-6bd9-40b8-9fed-e34b8e1114c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c009b7e-4rsg4" Oct 31 04:19:18.883496 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:18.883256 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brqdd\" (UniqueName: \"kubernetes.io/projected/42c8c3da-6bd9-40b8-9fed-e34b8e1114c7-kube-api-access-brqdd\") pod \"maas-api-revoke-sub-2c009b7e-4rsg4\" (UID: \"42c8c3da-6bd9-40b8-9fed-e34b8e1114c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c009b7e-4rsg4" Oct 31 04:19:18.883935 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:18.883903 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42c8c3da-6bd9-40b8-9fed-e34b8e1114c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c009b7e-4rsg4\" (UID: \"42c8c3da-6bd9-40b8-9fed-e34b8e1114c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c009b7e-4rsg4" Oct 31 04:19:18.885363 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:18.885323 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42c8c3da-6bd9-40b8-9fed-e34b8e1114c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c009b7e-4rsg4\" (UID: \"42c8c3da-6bd9-40b8-9fed-e34b8e1114c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c009b7e-4rsg4" Oct 31 04:19:18.906286 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:18.906256 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brqdd\" (UniqueName: \"kubernetes.io/projected/42c8c3da-6bd9-40b8-9fed-e34b8e1114c7-kube-api-access-brqdd\") pod \"maas-api-revoke-sub-2c009b7e-4rsg4\" (UID: \"42c8c3da-6bd9-40b8-9fed-e34b8e1114c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c009b7e-4rsg4" Oct 31 04:19:18.911863 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:18.911805 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf46490-9gq8n" podStartSLOduration=1.911790616 podStartE2EDuration="1.911790616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:19:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:19:18.910284414 +0000 UTC m=+913.666622454" watchObservedRunningTime="2026-10-31 04:19:18.911790616 +0000 UTC m=+913.668128656" Oct 31 04:19:19.035859 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:19.035816 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c009b7e-4rsg4" Oct 31 04:19:19.345683 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:19.345651 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c009b7e-4rsg4"] Oct 31 04:19:19.346032 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:19:19.346005 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42c8c3da_6bd9_40b8_9fed_e34b8e1114c7.slice/crio-a96b792d467d559376e1651d23314e92d336399b718b5cade7450db1b0f21144 WatchSource:0}: Error finding container a96b792d467d559376e1651d23314e92d336399b718b5cade7450db1b0f21144: Status 404 returned error can't find the container with id a96b792d467d559376e1651d23314e92d336399b718b5cade7450db1b0f21144 Oct 31 04:19:19.508234 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:19.508110 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45"] Oct 31 04:19:19.513775 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:19.513739 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45" Oct 31 04:19:19.542475 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:19.541294 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45"] Oct 31 04:19:19.594093 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:19.594000 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eed10564-909d-42d8-b5c6-e6a97da5ceff-istiod-ca-cert\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45\" (UID: \"eed10564-909d-42d8-b5c6-e6a97da5ceff\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45" Oct 31 04:19:19.594093 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:19.594049 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eed10564-909d-42d8-b5c6-e6a97da5ceff-istio-envoy\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45\" (UID: \"eed10564-909d-42d8-b5c6-e6a97da5ceff\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45" Oct 31 04:19:19.594321 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:19.594114 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eed10564-909d-42d8-b5c6-e6a97da5ceff-workload-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45\" (UID: \"eed10564-909d-42d8-b5c6-e6a97da5ceff\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45" Oct 31 04:19:19.594321 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:19.594148 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eed10564-909d-42d8-b5c6-e6a97da5ceff-istio-data\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45\" (UID: \"eed10564-909d-42d8-b5c6-e6a97da5ceff\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45" Oct 31 04:19:19.594321 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:19.594174 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eed10564-909d-42d8-b5c6-e6a97da5ceff-workload-certs\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45\" (UID: \"eed10564-909d-42d8-b5c6-e6a97da5ceff\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45" Oct 31 04:19:19.594481 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:19.594375 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eed10564-909d-42d8-b5c6-e6a97da5ceff-istio-token\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45\" (UID: \"eed10564-909d-42d8-b5c6-e6a97da5ceff\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45" Oct 31 04:19:19.594481 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:19.594430 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz2nf\" (UniqueName: \"kubernetes.io/projected/eed10564-909d-42d8-b5c6-e6a97da5ceff-kube-api-access-gz2nf\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45\" (UID: \"eed10564-909d-42d8-b5c6-e6a97da5ceff\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45" Oct 31 04:19:19.594481 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:19.594472 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eed10564-909d-42d8-b5c6-e6a97da5ceff-istio-podinfo\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45\" (UID: \"eed10564-909d-42d8-b5c6-e6a97da5ceff\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45" Oct 31 04:19:19.594625 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:19.594507 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eed10564-909d-42d8-b5c6-e6a97da5ceff-credential-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45\" (UID: \"eed10564-909d-42d8-b5c6-e6a97da5ceff\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45" Oct 31 04:19:19.695645 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:19.695602 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz2nf\" (UniqueName: \"kubernetes.io/projected/eed10564-909d-42d8-b5c6-e6a97da5ceff-kube-api-access-gz2nf\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45\" (UID: \"eed10564-909d-42d8-b5c6-e6a97da5ceff\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45" Oct 31 04:19:19.695645 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:19.695643 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eed10564-909d-42d8-b5c6-e6a97da5ceff-istio-podinfo\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45\" (UID: \"eed10564-909d-42d8-b5c6-e6a97da5ceff\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45" Oct 31 04:19:19.695915 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:19.695681 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eed10564-909d-42d8-b5c6-e6a97da5ceff-credential-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45\" (UID: \"eed10564-909d-42d8-b5c6-e6a97da5ceff\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45" Oct 31 04:19:19.695915 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:19.695750 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eed10564-909d-42d8-b5c6-e6a97da5ceff-istiod-ca-cert\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45\" (UID: \"eed10564-909d-42d8-b5c6-e6a97da5ceff\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45" Oct 31 04:19:19.695915 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:19.695782 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eed10564-909d-42d8-b5c6-e6a97da5ceff-istio-envoy\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45\" (UID: \"eed10564-909d-42d8-b5c6-e6a97da5ceff\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45" Oct 31 04:19:19.695915 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:19.695822 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eed10564-909d-42d8-b5c6-e6a97da5ceff-workload-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45\" (UID: \"eed10564-909d-42d8-b5c6-e6a97da5ceff\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45" Oct 31 04:19:19.695915 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:19.695883 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eed10564-909d-42d8-b5c6-e6a97da5ceff-istio-data\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45\" (UID: \"eed10564-909d-42d8-b5c6-e6a97da5ceff\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45" Oct 31 04:19:19.695915 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:19.695929 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eed10564-909d-42d8-b5c6-e6a97da5ceff-workload-certs\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45\" (UID: \"eed10564-909d-42d8-b5c6-e6a97da5ceff\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45" Oct 31 04:19:19.696239 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:19.696003 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eed10564-909d-42d8-b5c6-e6a97da5ceff-istio-token\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45\" (UID: \"eed10564-909d-42d8-b5c6-e6a97da5ceff\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45" Oct 31 04:19:19.696472 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:19.696445 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eed10564-909d-42d8-b5c6-e6a97da5ceff-workload-certs\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45\" (UID: \"eed10564-909d-42d8-b5c6-e6a97da5ceff\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45" Oct 31 04:19:19.696566 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:19.696495 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eed10564-909d-42d8-b5c6-e6a97da5ceff-istio-data\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45\" (UID: \"eed10564-909d-42d8-b5c6-e6a97da5ceff\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45" Oct 31 04:19:19.696628 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:19.696561 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eed10564-909d-42d8-b5c6-e6a97da5ceff-credential-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45\" (UID: \"eed10564-909d-42d8-b5c6-e6a97da5ceff\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45" Oct 31 04:19:19.696628 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:19.696610 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eed10564-909d-42d8-b5c6-e6a97da5ceff-workload-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45\" (UID: \"eed10564-909d-42d8-b5c6-e6a97da5ceff\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45" Oct 31 04:19:19.696984 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:19.696962 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eed10564-909d-42d8-b5c6-e6a97da5ceff-istiod-ca-cert\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45\" (UID: \"eed10564-909d-42d8-b5c6-e6a97da5ceff\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45" Oct 31 04:19:19.698580 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:19.698560 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eed10564-909d-42d8-b5c6-e6a97da5ceff-istio-envoy\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45\" (UID: \"eed10564-909d-42d8-b5c6-e6a97da5ceff\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45" Oct 31 04:19:19.698834 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:19.698816 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eed10564-909d-42d8-b5c6-e6a97da5ceff-istio-podinfo\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45\" (UID: \"eed10564-909d-42d8-b5c6-e6a97da5ceff\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45" Oct 31 04:19:19.729896 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:19.729857 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eed10564-909d-42d8-b5c6-e6a97da5ceff-istio-token\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45\" (UID: \"eed10564-909d-42d8-b5c6-e6a97da5ceff\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45" Oct 31 04:19:19.730192 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:19.730165 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz2nf\" (UniqueName: \"kubernetes.io/projected/eed10564-909d-42d8-b5c6-e6a97da5ceff-kube-api-access-gz2nf\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45\" (UID: \"eed10564-909d-42d8-b5c6-e6a97da5ceff\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45" Oct 31 04:19:19.830846 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:19.830812 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45" Oct 31 04:19:19.882537 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:19.882498 5019 generic.go:356] "Generic (PLEG): container finished" podID="79da2fc6-818e-4f6b-a99c-53ab652e796f" containerID="08ef458afe0b08196e861aa53f685033bf2f564bfeec4ad23358d2a241ec1648" exitCode=0 Oct 31 04:19:19.882740 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:19.882650 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf46490-9gq8n" event={"ID":"79da2fc6-818e-4f6b-a99c-53ab652e796f","Type":"ContainerDied","Data":"08ef458afe0b08196e861aa53f685033bf2f564bfeec4ad23358d2a241ec1648"} Oct 31 04:19:19.884828 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:19.884793 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c009b7e-4rsg4" event={"ID":"42c8c3da-6bd9-40b8-9fed-e34b8e1114c7","Type":"ContainerStarted","Data":"ddd7cc5f2d0e908074ee241acf9cd19e8ada248fa2ddac1bdb51d63b6f224795"} Oct 31 04:19:19.885203 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:19.884852 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c009b7e-4rsg4" event={"ID":"42c8c3da-6bd9-40b8-9fed-e34b8e1114c7","Type":"ContainerStarted","Data":"a96b792d467d559376e1651d23314e92d336399b718b5cade7450db1b0f21144"} Oct 31 04:19:19.962734 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:19.962680 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c009b7e-4rsg4" podStartSLOduration=1.962660487 podStartE2EDuration="1.962660487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:19:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:19:19.960729224 +0000 UTC m=+914.717067291" watchObservedRunningTime="2026-10-31 04:19:19.962660487 +0000 UTC m=+914.718998530" Oct 31 04:19:20.152726 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:20.152696 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45"] Oct 31 04:19:20.153004 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:19:20.152979 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeed10564_909d_42d8_b5c6_e6a97da5ceff.slice/crio-e75c941a7ca3ac38e628e17af52c0ff6195fb31e038c1b387307b7636ee88c60 WatchSource:0}: Error finding container e75c941a7ca3ac38e628e17af52c0ff6195fb31e038c1b387307b7636ee88c60: Status 404 returned error can't find the container with id e75c941a7ca3ac38e628e17af52c0ff6195fb31e038c1b387307b7636ee88c60 Oct 31 04:19:20.155515 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:20.155477 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:19:20.155619 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:20.155560 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:19:20.155619 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:20.155591 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:19:20.906201 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:20.906103 5019 generic.go:356] "Generic (PLEG): container finished" podID="42c8c3da-6bd9-40b8-9fed-e34b8e1114c7" containerID="ddd7cc5f2d0e908074ee241acf9cd19e8ada248fa2ddac1bdb51d63b6f224795" exitCode=0 Oct 31 04:19:20.906644 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:20.906188 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c009b7e-4rsg4" event={"ID":"42c8c3da-6bd9-40b8-9fed-e34b8e1114c7","Type":"ContainerDied","Data":"ddd7cc5f2d0e908074ee241acf9cd19e8ada248fa2ddac1bdb51d63b6f224795"} Oct 31 04:19:20.908142 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:20.908117 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45" event={"ID":"eed10564-909d-42d8-b5c6-e6a97da5ceff","Type":"ContainerStarted","Data":"8ad9ce25a3e2fbdce74bddd57b2cd734928aa7a12ab8143c09f2f92478baa058"} Oct 31 04:19:20.908264 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:20.908164 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45" event={"ID":"eed10564-909d-42d8-b5c6-e6a97da5ceff","Type":"ContainerStarted","Data":"e75c941a7ca3ac38e628e17af52c0ff6195fb31e038c1b387307b7636ee88c60"} Oct 31 04:19:21.006804 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:21.004323 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45" podStartSLOduration=2.004300277 podStartE2EDuration="2.004300277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:19:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:19:21.000288594 +0000 UTC m=+915.756626662" watchObservedRunningTime="2026-10-31 04:19:21.004300277 +0000 UTC m=+915.760638319" Oct 31 04:19:21.144034 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:21.144003 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf46490-9gq8n" Oct 31 04:19:21.215396 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:21.215283 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79da2fc6-818e-4f6b-a99c-53ab652e796f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79da2fc6-818e-4f6b-a99c-53ab652e796f-maas-api-xks-ca\") pod \"79da2fc6-818e-4f6b-a99c-53ab652e796f\" (UID: \"79da2fc6-818e-4f6b-a99c-53ab652e796f\") " Oct 31 04:19:21.215695 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:21.215674 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79da2fc6-818e-4f6b-a99c-53ab652e796f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79da2fc6-818e-4f6b-a99c-53ab652e796f-maas-api-openshift-ca\") pod \"79da2fc6-818e-4f6b-a99c-53ab652e796f\" (UID: \"79da2fc6-818e-4f6b-a99c-53ab652e796f\") " Oct 31 04:19:21.216171 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:21.216068 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79da2fc6-818e-4f6b-a99c-53ab652e796f-kube-api-access-nbdwk\" (UniqueName: \"kubernetes.io/projected/79da2fc6-818e-4f6b-a99c-53ab652e796f-kube-api-access-nbdwk\") pod \"79da2fc6-818e-4f6b-a99c-53ab652e796f\" (UID: \"79da2fc6-818e-4f6b-a99c-53ab652e796f\") " Oct 31 04:19:21.219626 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:21.219403 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79da2fc6-818e-4f6b-a99c-53ab652e796f-maas-api-xks-ca" pod "79da2fc6-818e-4f6b-a99c-53ab652e796f" (UID: "79da2fc6-818e-4f6b-a99c-53ab652e796f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:19:21.219626 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:21.219587 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79da2fc6-818e-4f6b-a99c-53ab652e796f-maas-api-openshift-ca" pod "79da2fc6-818e-4f6b-a99c-53ab652e796f" (UID: "79da2fc6-818e-4f6b-a99c-53ab652e796f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:19:21.219842 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:21.219693 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79da2fc6-818e-4f6b-a99c-53ab652e796f-kube-api-access-nbdwk" pod "79da2fc6-818e-4f6b-a99c-53ab652e796f" (UID: "79da2fc6-818e-4f6b-a99c-53ab652e796f"). InnerVolumeSpecName "kube-api-access-nbdwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:19:21.318312 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:21.318270 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbdwk\" (UniqueName: \"kubernetes.io/projected/79da2fc6-818e-4f6b-a99c-53ab652e796f-kube-api-access-nbdwk\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:19:21.318312 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:21.318311 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79da2fc6-818e-4f6b-a99c-53ab652e796f-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:19:21.318312 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:21.318326 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79da2fc6-818e-4f6b-a99c-53ab652e796f-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:19:21.831920 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:21.831868 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45" Oct 31 04:19:21.833542 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:21.833510 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 31 04:19:21.833699 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:21.833573 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45" podUID="eed10564-909d-42d8-b5c6-e6a97da5ceff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 31 04:19:21.930982 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:21.930950 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf46490-9gq8n" Oct 31 04:19:21.930982 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:21.930965 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf46490-9gq8n" event={"ID":"79da2fc6-818e-4f6b-a99c-53ab652e796f","Type":"ContainerDied","Data":"09c84c81bc1fbff0508b9c1da89d862ffaf9d01a1e798a2998c21266b5996e2e"} Oct 31 04:19:21.931509 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:21.931056 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09c84c81bc1fbff0508b9c1da89d862ffaf9d01a1e798a2998c21266b5996e2e" Oct 31 04:19:22.143785 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:22.143753 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c009b7e-4rsg4" Oct 31 04:19:22.232588 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:22.232547 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42c8c3da-6bd9-40b8-9fed-e34b8e1114c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42c8c3da-6bd9-40b8-9fed-e34b8e1114c7-maas-api-xks-ca\") pod \"42c8c3da-6bd9-40b8-9fed-e34b8e1114c7\" (UID: \"42c8c3da-6bd9-40b8-9fed-e34b8e1114c7\") " Oct 31 04:19:22.232794 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:22.232781 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42c8c3da-6bd9-40b8-9fed-e34b8e1114c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42c8c3da-6bd9-40b8-9fed-e34b8e1114c7-maas-api-openshift-ca\") pod \"42c8c3da-6bd9-40b8-9fed-e34b8e1114c7\" (UID: \"42c8c3da-6bd9-40b8-9fed-e34b8e1114c7\") " Oct 31 04:19:22.232865 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:22.232820 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42c8c3da-6bd9-40b8-9fed-e34b8e1114c7-kube-api-access-brqdd\" (UniqueName: \"kubernetes.io/projected/42c8c3da-6bd9-40b8-9fed-e34b8e1114c7-kube-api-access-brqdd\") pod \"42c8c3da-6bd9-40b8-9fed-e34b8e1114c7\" (UID: \"42c8c3da-6bd9-40b8-9fed-e34b8e1114c7\") " Oct 31 04:19:22.233376 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:22.233312 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42c8c3da-6bd9-40b8-9fed-e34b8e1114c7-maas-api-openshift-ca" pod "42c8c3da-6bd9-40b8-9fed-e34b8e1114c7" (UID: "42c8c3da-6bd9-40b8-9fed-e34b8e1114c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:19:22.235772 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:22.235744 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42c8c3da-6bd9-40b8-9fed-e34b8e1114c7-maas-api-xks-ca" pod "42c8c3da-6bd9-40b8-9fed-e34b8e1114c7" (UID: "42c8c3da-6bd9-40b8-9fed-e34b8e1114c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:19:22.236866 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:22.236833 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42c8c3da-6bd9-40b8-9fed-e34b8e1114c7-kube-api-access-brqdd" pod "42c8c3da-6bd9-40b8-9fed-e34b8e1114c7" (UID: "42c8c3da-6bd9-40b8-9fed-e34b8e1114c7"). InnerVolumeSpecName "kube-api-access-brqdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:19:22.334422 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:22.334381 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42c8c3da-6bd9-40b8-9fed-e34b8e1114c7-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:19:22.334422 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:22.334421 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brqdd\" (UniqueName: \"kubernetes.io/projected/42c8c3da-6bd9-40b8-9fed-e34b8e1114c7-kube-api-access-brqdd\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:19:22.334642 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:22.334436 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42c8c3da-6bd9-40b8-9fed-e34b8e1114c7-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:19:22.415781 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:19:22.415678 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:19:22.831761 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:22.831728 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 31 04:19:22.831949 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:22.831795 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45" podUID="eed10564-909d-42d8-b5c6-e6a97da5ceff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 31 04:19:22.940472 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:22.940441 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c009b7e-4rsg4" Oct 31 04:19:22.940888 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:22.940478 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c009b7e-4rsg4" event={"ID":"42c8c3da-6bd9-40b8-9fed-e34b8e1114c7","Type":"ContainerDied","Data":"a96b792d467d559376e1651d23314e92d336399b718b5cade7450db1b0f21144"} Oct 31 04:19:22.940888 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:22.940533 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a96b792d467d559376e1651d23314e92d336399b718b5cade7450db1b0f21144" Oct 31 04:19:23.526511 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:23.526465 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c"] Oct 31 04:19:23.527412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:23.527388 5019 cpu_manager.go:414] "containerMap: removing container" podUID="79da2fc6-818e-4f6b-a99c-53ab652e796f" containerName="revoke-keys" Oct 31 04:19:23.527412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:23.527412 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79da2fc6-818e-4f6b-a99c-53ab652e796f" containerName="revoke-keys" Oct 31 04:19:23.527584 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:23.527438 5019 cpu_manager.go:414] "containerMap: removing container" podUID="42c8c3da-6bd9-40b8-9fed-e34b8e1114c7" containerName="revoke-keys" Oct 31 04:19:23.527584 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:23.527446 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42c8c3da-6bd9-40b8-9fed-e34b8e1114c7" containerName="revoke-keys" Oct 31 04:19:23.527641 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:23.527624 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="79da2fc6-818e-4f6b-a99c-53ab652e796f" containerName="revoke-keys" Oct 31 04:19:23.527671 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:23.527640 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="42c8c3da-6bd9-40b8-9fed-e34b8e1114c7" containerName="revoke-keys" Oct 31 04:19:23.534116 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:23.534087 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c" Oct 31 04:19:23.545478 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:23.543703 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95027377-8zmvk"] Oct 31 04:19:23.548186 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:23.548152 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-istio-data\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c\" (UID: \"dda0567c-8a01-47e8-95f9-c0a4ad08a1c6\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c" Oct 31 04:19:23.548309 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:23.548241 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-istio-token\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c\" (UID: \"dda0567c-8a01-47e8-95f9-c0a4ad08a1c6\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c" Oct 31 04:19:23.548309 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:23.548285 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-istio-podinfo\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c\" (UID: \"dda0567c-8a01-47e8-95f9-c0a4ad08a1c6\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c" Oct 31 04:19:23.548510 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:23.548345 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-istio-envoy\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c\" (UID: \"dda0567c-8a01-47e8-95f9-c0a4ad08a1c6\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c" Oct 31 04:19:23.548510 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:23.548407 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqwh6\" (UniqueName: \"kubernetes.io/projected/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-kube-api-access-tqwh6\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c\" (UID: \"dda0567c-8a01-47e8-95f9-c0a4ad08a1c6\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c" Oct 31 04:19:23.548510 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:23.548446 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-credential-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c\" (UID: \"dda0567c-8a01-47e8-95f9-c0a4ad08a1c6\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c" Oct 31 04:19:23.548510 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:23.548482 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-istiod-ca-cert\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c\" (UID: \"dda0567c-8a01-47e8-95f9-c0a4ad08a1c6\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c" Oct 31 04:19:23.548724 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:23.548516 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-workload-certs\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c\" (UID: \"dda0567c-8a01-47e8-95f9-c0a4ad08a1c6\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c" Oct 31 04:19:23.548724 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:23.548546 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-workload-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c\" (UID: \"dda0567c-8a01-47e8-95f9-c0a4ad08a1c6\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c" Oct 31 04:19:23.555687 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:23.555657 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c"] Oct 31 04:19:23.555824 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:23.555790 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95027377-8zmvk" Oct 31 04:19:23.569253 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:23.568988 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95027377-8zmvk"] Oct 31 04:19:23.588360 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:23.586739 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:19:23.588360 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:23.586845 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:19:23.588360 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:23.586970 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:19:23.588360 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:23.587076 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:19:23.650445 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:23.650391 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a85e6a3c-fc5f-44a6-8616-8b7fc45cd35c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95027377-8zmvk\" (UID: \"a85e6a3c-fc5f-44a6-8616-8b7fc45cd35c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95027377-8zmvk" Oct 31 04:19:23.650662 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:23.650530 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-istio-data\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c\" (UID: \"dda0567c-8a01-47e8-95f9-c0a4ad08a1c6\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c" Oct 31 04:19:23.650662 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:23.650586 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9kz9\" (UniqueName: \"kubernetes.io/projected/a85e6a3c-fc5f-44a6-8616-8b7fc45cd35c-kube-api-access-z9kz9\") pod \"maas-api-revoke-sub-95027377-8zmvk\" (UID: \"a85e6a3c-fc5f-44a6-8616-8b7fc45cd35c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95027377-8zmvk" Oct 31 04:19:23.650662 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:23.650638 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-istio-token\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c\" (UID: \"dda0567c-8a01-47e8-95f9-c0a4ad08a1c6\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c" Oct 31 04:19:23.650838 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:23.650679 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-istio-podinfo\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c\" (UID: \"dda0567c-8a01-47e8-95f9-c0a4ad08a1c6\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c" Oct 31 04:19:23.650838 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:23.650725 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-istio-envoy\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c\" (UID: \"dda0567c-8a01-47e8-95f9-c0a4ad08a1c6\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c" Oct 31 04:19:23.650838 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:23.650759 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a85e6a3c-fc5f-44a6-8616-8b7fc45cd35c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95027377-8zmvk\" (UID: \"a85e6a3c-fc5f-44a6-8616-8b7fc45cd35c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95027377-8zmvk" Oct 31 04:19:23.650838 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:23.650814 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqwh6\" (UniqueName: \"kubernetes.io/projected/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-kube-api-access-tqwh6\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c\" (UID: \"dda0567c-8a01-47e8-95f9-c0a4ad08a1c6\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c" Oct 31 04:19:23.651055 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:23.650854 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-credential-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c\" (UID: \"dda0567c-8a01-47e8-95f9-c0a4ad08a1c6\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c" Oct 31 04:19:23.651055 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:23.650922 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-istiod-ca-cert\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c\" (UID: \"dda0567c-8a01-47e8-95f9-c0a4ad08a1c6\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c" Oct 31 04:19:23.651055 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:23.650964 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-workload-certs\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c\" (UID: \"dda0567c-8a01-47e8-95f9-c0a4ad08a1c6\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c" Oct 31 04:19:23.651055 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:23.650997 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-workload-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c\" (UID: \"dda0567c-8a01-47e8-95f9-c0a4ad08a1c6\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c" Oct 31 04:19:23.651553 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:23.651458 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-workload-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c\" (UID: \"dda0567c-8a01-47e8-95f9-c0a4ad08a1c6\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c" Oct 31 04:19:23.651995 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:23.651962 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-istio-data\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c\" (UID: \"dda0567c-8a01-47e8-95f9-c0a4ad08a1c6\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c" Oct 31 04:19:23.652765 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:23.652071 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-credential-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c\" (UID: \"dda0567c-8a01-47e8-95f9-c0a4ad08a1c6\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c" Oct 31 04:19:23.652888 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:23.652427 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-workload-certs\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c\" (UID: \"dda0567c-8a01-47e8-95f9-c0a4ad08a1c6\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c" Oct 31 04:19:23.653771 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:23.653733 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-istiod-ca-cert\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c\" (UID: \"dda0567c-8a01-47e8-95f9-c0a4ad08a1c6\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c" Oct 31 04:19:23.654261 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:23.654162 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-istio-envoy\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c\" (UID: \"dda0567c-8a01-47e8-95f9-c0a4ad08a1c6\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c" Oct 31 04:19:23.656007 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:23.655989 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-istio-podinfo\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c\" (UID: \"dda0567c-8a01-47e8-95f9-c0a4ad08a1c6\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c" Oct 31 04:19:23.714301 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:23.714231 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqwh6\" (UniqueName: \"kubernetes.io/projected/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-kube-api-access-tqwh6\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c\" (UID: \"dda0567c-8a01-47e8-95f9-c0a4ad08a1c6\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c" Oct 31 04:19:23.716093 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:23.716022 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-istio-token\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c\" (UID: \"dda0567c-8a01-47e8-95f9-c0a4ad08a1c6\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c" Oct 31 04:19:23.752636 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:23.752600 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9kz9\" (UniqueName: \"kubernetes.io/projected/a85e6a3c-fc5f-44a6-8616-8b7fc45cd35c-kube-api-access-z9kz9\") pod \"maas-api-revoke-sub-95027377-8zmvk\" (UID: \"a85e6a3c-fc5f-44a6-8616-8b7fc45cd35c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95027377-8zmvk" Oct 31 04:19:23.752893 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:23.752881 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a85e6a3c-fc5f-44a6-8616-8b7fc45cd35c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95027377-8zmvk\" (UID: \"a85e6a3c-fc5f-44a6-8616-8b7fc45cd35c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95027377-8zmvk" Oct 31 04:19:23.753040 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:23.753029 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a85e6a3c-fc5f-44a6-8616-8b7fc45cd35c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95027377-8zmvk\" (UID: \"a85e6a3c-fc5f-44a6-8616-8b7fc45cd35c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95027377-8zmvk" Oct 31 04:19:23.753597 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:23.753571 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a85e6a3c-fc5f-44a6-8616-8b7fc45cd35c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95027377-8zmvk\" (UID: \"a85e6a3c-fc5f-44a6-8616-8b7fc45cd35c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95027377-8zmvk" Oct 31 04:19:23.755518 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:23.755499 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a85e6a3c-fc5f-44a6-8616-8b7fc45cd35c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95027377-8zmvk\" (UID: \"a85e6a3c-fc5f-44a6-8616-8b7fc45cd35c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95027377-8zmvk" Oct 31 04:19:23.803185 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:23.803031 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9kz9\" (UniqueName: \"kubernetes.io/projected/a85e6a3c-fc5f-44a6-8616-8b7fc45cd35c-kube-api-access-z9kz9\") pod \"maas-api-revoke-sub-95027377-8zmvk\" (UID: \"a85e6a3c-fc5f-44a6-8616-8b7fc45cd35c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95027377-8zmvk" Oct 31 04:19:23.831283 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:23.831245 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 31 04:19:23.831466 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:23.831299 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45" podUID="eed10564-909d-42d8-b5c6-e6a97da5ceff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 31 04:19:23.855713 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:23.855676 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c" Oct 31 04:19:23.920377 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:23.919769 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95027377-8zmvk" Oct 31 04:19:24.210292 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:24.210166 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c"] Oct 31 04:19:24.219138 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:24.219101 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:19:24.219263 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:24.219193 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:19:24.219263 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:24.219238 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:19:24.307668 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:24.307624 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95027377-8zmvk"] Oct 31 04:19:24.314248 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:19:24.314216 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda85e6a3c_fc5f_44a6_8616_8b7fc45cd35c.slice/crio-1ede34a31ea2f9748625265f1e2f1ef954177fedfb8e69fc7081b29044903cec WatchSource:0}: Error finding container 1ede34a31ea2f9748625265f1e2f1ef954177fedfb8e69fc7081b29044903cec: Status 404 returned error can't find the container with id 1ede34a31ea2f9748625265f1e2f1ef954177fedfb8e69fc7081b29044903cec Oct 31 04:19:24.831892 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:24.831859 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 31 04:19:24.832076 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:24.831912 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45" podUID="eed10564-909d-42d8-b5c6-e6a97da5ceff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 31 04:19:24.989355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:24.985028 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95027377-8zmvk" event={"ID":"a85e6a3c-fc5f-44a6-8616-8b7fc45cd35c","Type":"ContainerStarted","Data":"5209d794ee83cd75e772052ddc68dff4c0670716ce6d210c7d80b221bf66cd51"} Oct 31 04:19:24.989355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:24.985141 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95027377-8zmvk" event={"ID":"a85e6a3c-fc5f-44a6-8616-8b7fc45cd35c","Type":"ContainerStarted","Data":"1ede34a31ea2f9748625265f1e2f1ef954177fedfb8e69fc7081b29044903cec"} Oct 31 04:19:25.008837 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:25.008791 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c" event={"ID":"dda0567c-8a01-47e8-95f9-c0a4ad08a1c6","Type":"ContainerStarted","Data":"41d29e3aedb2beba64a8b693cc2881be01db94eeecc9604fa8af1854e39bc023"} Oct 31 04:19:25.009104 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:25.009079 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c" event={"ID":"dda0567c-8a01-47e8-95f9-c0a4ad08a1c6","Type":"ContainerStarted","Data":"de159798d731af5201f266074bbf41e5e3898d063b98bc7fae3efd7851a70e19"} Oct 31 04:19:25.052327 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:25.052248 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95027377-8zmvk" podStartSLOduration=2.052228167 podStartE2EDuration="2.052228167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:19:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:19:25.049798718 +0000 UTC m=+919.806136758" watchObservedRunningTime="2026-10-31 04:19:25.052228167 +0000 UTC m=+919.808566208" Oct 31 04:19:25.105356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:25.104841 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c" podStartSLOduration=2.104826424 podStartE2EDuration="2.104826424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:19:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:19:25.104822288 +0000 UTC m=+919.861160329" watchObservedRunningTime="2026-10-31 04:19:25.104826424 +0000 UTC m=+919.861164473" Oct 31 04:19:25.835457 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:25.834912 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 31 04:19:25.835457 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:25.835048 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45" podUID="eed10564-909d-42d8-b5c6-e6a97da5ceff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 31 04:19:25.856508 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:25.856471 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c" Oct 31 04:19:25.857708 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:25.857686 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 31 04:19:25.857800 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:25.857731 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c" podUID="dda0567c-8a01-47e8-95f9-c0a4ad08a1c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 31 04:19:26.038611 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:26.038578 5019 generic.go:356] "Generic (PLEG): container finished" podID="a85e6a3c-fc5f-44a6-8616-8b7fc45cd35c" containerID="5209d794ee83cd75e772052ddc68dff4c0670716ce6d210c7d80b221bf66cd51" exitCode=0 Oct 31 04:19:26.039225 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:26.039198 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95027377-8zmvk" event={"ID":"a85e6a3c-fc5f-44a6-8616-8b7fc45cd35c","Type":"ContainerDied","Data":"5209d794ee83cd75e772052ddc68dff4c0670716ce6d210c7d80b221bf66cd51"} Oct 31 04:19:26.831972 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:26.831381 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 31 04:19:26.831972 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:26.831461 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45" podUID="eed10564-909d-42d8-b5c6-e6a97da5ceff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 31 04:19:26.856425 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:26.856388 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 31 04:19:26.856817 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:26.856471 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c" podUID="dda0567c-8a01-47e8-95f9-c0a4ad08a1c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 31 04:19:27.368004 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:27.367689 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95027377-8zmvk" Oct 31 04:19:27.434879 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:27.434770 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a85e6a3c-fc5f-44a6-8616-8b7fc45cd35c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a85e6a3c-fc5f-44a6-8616-8b7fc45cd35c-maas-api-xks-ca\") pod \"a85e6a3c-fc5f-44a6-8616-8b7fc45cd35c\" (UID: \"a85e6a3c-fc5f-44a6-8616-8b7fc45cd35c\") " Oct 31 04:19:27.435581 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:27.435550 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a85e6a3c-fc5f-44a6-8616-8b7fc45cd35c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a85e6a3c-fc5f-44a6-8616-8b7fc45cd35c-maas-api-openshift-ca\") pod \"a85e6a3c-fc5f-44a6-8616-8b7fc45cd35c\" (UID: \"a85e6a3c-fc5f-44a6-8616-8b7fc45cd35c\") " Oct 31 04:19:27.436017 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:27.435916 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a85e6a3c-fc5f-44a6-8616-8b7fc45cd35c-maas-api-openshift-ca" pod "a85e6a3c-fc5f-44a6-8616-8b7fc45cd35c" (UID: "a85e6a3c-fc5f-44a6-8616-8b7fc45cd35c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:19:27.436095 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:27.435988 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a85e6a3c-fc5f-44a6-8616-8b7fc45cd35c-kube-api-access-z9kz9\" (UniqueName: \"kubernetes.io/projected/a85e6a3c-fc5f-44a6-8616-8b7fc45cd35c-kube-api-access-z9kz9\") pod \"a85e6a3c-fc5f-44a6-8616-8b7fc45cd35c\" (UID: \"a85e6a3c-fc5f-44a6-8616-8b7fc45cd35c\") " Oct 31 04:19:27.462382 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:27.439015 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a85e6a3c-fc5f-44a6-8616-8b7fc45cd35c-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:19:27.462382 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:27.460506 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a85e6a3c-fc5f-44a6-8616-8b7fc45cd35c-kube-api-access-z9kz9" pod "a85e6a3c-fc5f-44a6-8616-8b7fc45cd35c" (UID: "a85e6a3c-fc5f-44a6-8616-8b7fc45cd35c"). InnerVolumeSpecName "kube-api-access-z9kz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:19:27.463613 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:27.463579 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a85e6a3c-fc5f-44a6-8616-8b7fc45cd35c-maas-api-xks-ca" pod "a85e6a3c-fc5f-44a6-8616-8b7fc45cd35c" (UID: "a85e6a3c-fc5f-44a6-8616-8b7fc45cd35c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:19:27.559397 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:27.542417 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a85e6a3c-fc5f-44a6-8616-8b7fc45cd35c-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:19:27.559397 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:27.542464 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9kz9\" (UniqueName: \"kubernetes.io/projected/a85e6a3c-fc5f-44a6-8616-8b7fc45cd35c-kube-api-access-z9kz9\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:19:27.860986 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:27.860940 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 31 04:19:27.861636 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:27.861584 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c" podUID="dda0567c-8a01-47e8-95f9-c0a4ad08a1c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 31 04:19:28.021277 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:28.021241 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a88e2fc-hlqdw"] Oct 31 04:19:28.022587 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:28.022552 5019 cpu_manager.go:414] "containerMap: removing container" podUID="a85e6a3c-fc5f-44a6-8616-8b7fc45cd35c" containerName="revoke-keys" Oct 31 04:19:28.022800 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:28.022783 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a85e6a3c-fc5f-44a6-8616-8b7fc45cd35c" containerName="revoke-keys" Oct 31 04:19:28.023128 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:28.023111 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="a85e6a3c-fc5f-44a6-8616-8b7fc45cd35c" containerName="revoke-keys" Oct 31 04:19:28.040838 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:28.040747 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a88e2fc-hlqdw" Oct 31 04:19:28.123390 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:28.123012 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct7l8\" (UniqueName: \"kubernetes.io/projected/4989f9aa-dd46-46da-9213-4055a2a3500e-kube-api-access-ct7l8\") pod \"maas-api-revoke-sub-8a88e2fc-hlqdw\" (UID: \"4989f9aa-dd46-46da-9213-4055a2a3500e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a88e2fc-hlqdw" Oct 31 04:19:28.123390 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:28.123145 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4989f9aa-dd46-46da-9213-4055a2a3500e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a88e2fc-hlqdw\" (UID: \"4989f9aa-dd46-46da-9213-4055a2a3500e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a88e2fc-hlqdw" Oct 31 04:19:28.123390 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:28.123216 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4989f9aa-dd46-46da-9213-4055a2a3500e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a88e2fc-hlqdw\" (UID: \"4989f9aa-dd46-46da-9213-4055a2a3500e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a88e2fc-hlqdw" Oct 31 04:19:28.164478 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:28.163537 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95027377-8zmvk" event={"ID":"a85e6a3c-fc5f-44a6-8616-8b7fc45cd35c","Type":"ContainerDied","Data":"1ede34a31ea2f9748625265f1e2f1ef954177fedfb8e69fc7081b29044903cec"} Oct 31 04:19:28.164478 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:28.163609 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ede34a31ea2f9748625265f1e2f1ef954177fedfb8e69fc7081b29044903cec" Oct 31 04:19:28.164478 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:28.163688 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95027377-8zmvk" Oct 31 04:19:28.216446 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:28.216400 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a88e2fc-hlqdw"] Oct 31 04:19:28.246171 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:28.245362 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct7l8\" (UniqueName: \"kubernetes.io/projected/4989f9aa-dd46-46da-9213-4055a2a3500e-kube-api-access-ct7l8\") pod \"maas-api-revoke-sub-8a88e2fc-hlqdw\" (UID: \"4989f9aa-dd46-46da-9213-4055a2a3500e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a88e2fc-hlqdw" Oct 31 04:19:28.246171 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:28.245491 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4989f9aa-dd46-46da-9213-4055a2a3500e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a88e2fc-hlqdw\" (UID: \"4989f9aa-dd46-46da-9213-4055a2a3500e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a88e2fc-hlqdw" Oct 31 04:19:28.246171 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:28.245560 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4989f9aa-dd46-46da-9213-4055a2a3500e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a88e2fc-hlqdw\" (UID: \"4989f9aa-dd46-46da-9213-4055a2a3500e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a88e2fc-hlqdw" Oct 31 04:19:28.257374 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:28.249127 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4989f9aa-dd46-46da-9213-4055a2a3500e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a88e2fc-hlqdw\" (UID: \"4989f9aa-dd46-46da-9213-4055a2a3500e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a88e2fc-hlqdw" Oct 31 04:19:28.309426 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:28.297862 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4989f9aa-dd46-46da-9213-4055a2a3500e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a88e2fc-hlqdw\" (UID: \"4989f9aa-dd46-46da-9213-4055a2a3500e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a88e2fc-hlqdw" Oct 31 04:19:28.309426 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:28.298201 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct7l8\" (UniqueName: \"kubernetes.io/projected/4989f9aa-dd46-46da-9213-4055a2a3500e-kube-api-access-ct7l8\") pod \"maas-api-revoke-sub-8a88e2fc-hlqdw\" (UID: \"4989f9aa-dd46-46da-9213-4055a2a3500e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a88e2fc-hlqdw" Oct 31 04:19:28.519399 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:28.487419 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a88e2fc-hlqdw" Oct 31 04:19:29.055429 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:29.049754 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" podUID="643433d8-418e-4102-8bc3-cd544932614a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 04:19:29.055429 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:29.050134 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 31 04:19:29.129724 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:29.103221 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 31 04:19:29.129724 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:29.103304 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c" podUID="dda0567c-8a01-47e8-95f9-c0a4ad08a1c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 31 04:19:29.168359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:29.161209 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:19:29.184523 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:29.172422 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45" podUID="eed10564-909d-42d8-b5c6-e6a97da5ceff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:29.210356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:29.194247 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx"] Oct 31 04:19:29.389669 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:29.389557 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 31 04:19:29.585476 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:29.519601 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" podUID="e0f4b8c2-14f4-4052-bf54-dd65bf59e413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 04:19:29.655221 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:29.651497 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx" Oct 31 04:19:29.663466 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:29.663421 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:29.781379 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:29.771770 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9256bfe-8c85-477e-bcc9-9938f4c689ab-workload-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-rxktx\" (UID: \"a9256bfe-8c85-477e-bcc9-9938f4c689ab\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx" Oct 31 04:19:29.781379 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:29.771849 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9256bfe-8c85-477e-bcc9-9938f4c689ab-istio-podinfo\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-rxktx\" (UID: \"a9256bfe-8c85-477e-bcc9-9938f4c689ab\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx" Oct 31 04:19:29.781379 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:29.771876 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9256bfe-8c85-477e-bcc9-9938f4c689ab-istiod-ca-cert\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-rxktx\" (UID: \"a9256bfe-8c85-477e-bcc9-9938f4c689ab\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx" Oct 31 04:19:29.781379 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:29.771940 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9256bfe-8c85-477e-bcc9-9938f4c689ab-istio-token\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-rxktx\" (UID: \"a9256bfe-8c85-477e-bcc9-9938f4c689ab\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx" Oct 31 04:19:29.781379 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:29.772034 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9256bfe-8c85-477e-bcc9-9938f4c689ab-istio-envoy\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-rxktx\" (UID: \"a9256bfe-8c85-477e-bcc9-9938f4c689ab\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx" Oct 31 04:19:29.781379 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:29.772084 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9256bfe-8c85-477e-bcc9-9938f4c689ab-istio-data\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-rxktx\" (UID: \"a9256bfe-8c85-477e-bcc9-9938f4c689ab\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx" Oct 31 04:19:29.781379 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:29.772138 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txvzn\" (UniqueName: \"kubernetes.io/projected/a9256bfe-8c85-477e-bcc9-9938f4c689ab-kube-api-access-txvzn\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-rxktx\" (UID: \"a9256bfe-8c85-477e-bcc9-9938f4c689ab\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx" Oct 31 04:19:29.781379 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:29.772199 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9256bfe-8c85-477e-bcc9-9938f4c689ab-workload-certs\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-rxktx\" (UID: \"a9256bfe-8c85-477e-bcc9-9938f4c689ab\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx" Oct 31 04:19:29.781379 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:29.772293 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9256bfe-8c85-477e-bcc9-9938f4c689ab-credential-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-rxktx\" (UID: \"a9256bfe-8c85-477e-bcc9-9938f4c689ab\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx" Oct 31 04:19:29.913514 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:29.913116 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9256bfe-8c85-477e-bcc9-9938f4c689ab-istio-envoy\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-rxktx\" (UID: \"a9256bfe-8c85-477e-bcc9-9938f4c689ab\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx" Oct 31 04:19:29.913514 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:29.913164 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9256bfe-8c85-477e-bcc9-9938f4c689ab-istio-data\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-rxktx\" (UID: \"a9256bfe-8c85-477e-bcc9-9938f4c689ab\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx" Oct 31 04:19:29.913514 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:29.913193 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txvzn\" (UniqueName: \"kubernetes.io/projected/a9256bfe-8c85-477e-bcc9-9938f4c689ab-kube-api-access-txvzn\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-rxktx\" (UID: \"a9256bfe-8c85-477e-bcc9-9938f4c689ab\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx" Oct 31 04:19:29.913514 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:29.913232 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9256bfe-8c85-477e-bcc9-9938f4c689ab-workload-certs\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-rxktx\" (UID: \"a9256bfe-8c85-477e-bcc9-9938f4c689ab\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx" Oct 31 04:19:29.913514 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:29.913288 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9256bfe-8c85-477e-bcc9-9938f4c689ab-credential-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-rxktx\" (UID: \"a9256bfe-8c85-477e-bcc9-9938f4c689ab\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx" Oct 31 04:19:29.913514 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:29.913310 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9256bfe-8c85-477e-bcc9-9938f4c689ab-workload-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-rxktx\" (UID: \"a9256bfe-8c85-477e-bcc9-9938f4c689ab\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx" Oct 31 04:19:29.914932 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:29.914071 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9256bfe-8c85-477e-bcc9-9938f4c689ab-istio-podinfo\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-rxktx\" (UID: \"a9256bfe-8c85-477e-bcc9-9938f4c689ab\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx" Oct 31 04:19:29.914932 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:29.914110 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9256bfe-8c85-477e-bcc9-9938f4c689ab-istiod-ca-cert\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-rxktx\" (UID: \"a9256bfe-8c85-477e-bcc9-9938f4c689ab\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx" Oct 31 04:19:29.914932 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:29.914174 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9256bfe-8c85-477e-bcc9-9938f4c689ab-istio-token\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-rxktx\" (UID: \"a9256bfe-8c85-477e-bcc9-9938f4c689ab\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx" Oct 31 04:19:29.917518 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:29.915984 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9256bfe-8c85-477e-bcc9-9938f4c689ab-istio-data\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-rxktx\" (UID: \"a9256bfe-8c85-477e-bcc9-9938f4c689ab\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx" Oct 31 04:19:29.917518 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:29.916532 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9256bfe-8c85-477e-bcc9-9938f4c689ab-workload-certs\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-rxktx\" (UID: \"a9256bfe-8c85-477e-bcc9-9938f4c689ab\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx" Oct 31 04:19:29.917518 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:29.916982 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9256bfe-8c85-477e-bcc9-9938f4c689ab-credential-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-rxktx\" (UID: \"a9256bfe-8c85-477e-bcc9-9938f4c689ab\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx" Oct 31 04:19:29.917518 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:29.917209 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9256bfe-8c85-477e-bcc9-9938f4c689ab-workload-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-rxktx\" (UID: \"a9256bfe-8c85-477e-bcc9-9938f4c689ab\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx" Oct 31 04:19:29.927474 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:29.927441 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 31 04:19:29.927713 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:29.927684 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c" podUID="dda0567c-8a01-47e8-95f9-c0a4ad08a1c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 31 04:19:29.929084 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:29.929056 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9256bfe-8c85-477e-bcc9-9938f4c689ab-istiod-ca-cert\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-rxktx\" (UID: \"a9256bfe-8c85-477e-bcc9-9938f4c689ab\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx" Oct 31 04:19:29.963668 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:29.953465 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9256bfe-8c85-477e-bcc9-9938f4c689ab-istio-envoy\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-rxktx\" (UID: \"a9256bfe-8c85-477e-bcc9-9938f4c689ab\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx" Oct 31 04:19:29.988874 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:29.988720 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx"] Oct 31 04:19:29.988874 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:29.988807 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a714eaa5-w46mr"] Oct 31 04:19:30.008979 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:30.008651 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txvzn\" (UniqueName: \"kubernetes.io/projected/a9256bfe-8c85-477e-bcc9-9938f4c689ab-kube-api-access-txvzn\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-rxktx\" (UID: \"a9256bfe-8c85-477e-bcc9-9938f4c689ab\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx" Oct 31 04:19:30.008979 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:30.008921 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9256bfe-8c85-477e-bcc9-9938f4c689ab-istio-podinfo\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-rxktx\" (UID: \"a9256bfe-8c85-477e-bcc9-9938f4c689ab\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx" Oct 31 04:19:30.010702 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:30.010680 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a714eaa5-w46mr" Oct 31 04:19:30.018430 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:30.017817 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9256bfe-8c85-477e-bcc9-9938f4c689ab-istio-token\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-rxktx\" (UID: \"a9256bfe-8c85-477e-bcc9-9938f4c689ab\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx" Oct 31 04:19:30.019354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:30.019096 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fde3d56b-03cd-45b0-a5b6-6f0377f79a84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a714eaa5-w46mr\" (UID: \"fde3d56b-03cd-45b0-a5b6-6f0377f79a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a714eaa5-w46mr" Oct 31 04:19:30.019354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:30.019224 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fde3d56b-03cd-45b0-a5b6-6f0377f79a84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a714eaa5-w46mr\" (UID: \"fde3d56b-03cd-45b0-a5b6-6f0377f79a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a714eaa5-w46mr" Oct 31 04:19:30.019354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:30.019263 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkjjf\" (UniqueName: \"kubernetes.io/projected/fde3d56b-03cd-45b0-a5b6-6f0377f79a84-kube-api-access-gkjjf\") pod \"maas-api-revoke-sub-a714eaa5-w46mr\" (UID: \"fde3d56b-03cd-45b0-a5b6-6f0377f79a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a714eaa5-w46mr" Oct 31 04:19:30.063663 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:30.063598 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" podUID="643433d8-418e-4102-8bc3-cd544932614a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:30.064630 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:30.064372 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" podUID="8e2c7041-ad52-49f4-8b55-d16691f4938f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:30.070132 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:30.070087 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" Oct 31 04:19:30.074975 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:30.072885 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" podUID="b4ced3ab-41e4-4c54-93eb-95eddaae992d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:30.079294 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:30.079227 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a714eaa5-w46mr"] Oct 31 04:19:30.107364 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:30.105591 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx" Oct 31 04:19:30.112371 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:30.110527 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:30.112371 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:30.111066 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" Oct 31 04:19:30.226455 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:30.170736 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fde3d56b-03cd-45b0-a5b6-6f0377f79a84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a714eaa5-w46mr\" (UID: \"fde3d56b-03cd-45b0-a5b6-6f0377f79a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a714eaa5-w46mr" Oct 31 04:19:30.226455 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:30.170798 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkjjf\" (UniqueName: \"kubernetes.io/projected/fde3d56b-03cd-45b0-a5b6-6f0377f79a84-kube-api-access-gkjjf\") pod \"maas-api-revoke-sub-a714eaa5-w46mr\" (UID: \"fde3d56b-03cd-45b0-a5b6-6f0377f79a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a714eaa5-w46mr" Oct 31 04:19:30.226455 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:30.171074 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fde3d56b-03cd-45b0-a5b6-6f0377f79a84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a714eaa5-w46mr\" (UID: \"fde3d56b-03cd-45b0-a5b6-6f0377f79a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a714eaa5-w46mr" Oct 31 04:19:30.226455 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:30.172383 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fde3d56b-03cd-45b0-a5b6-6f0377f79a84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a714eaa5-w46mr\" (UID: \"fde3d56b-03cd-45b0-a5b6-6f0377f79a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a714eaa5-w46mr" Oct 31 04:19:30.226455 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:30.199542 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:19:30.226455 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:30.199604 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45" podUID="eed10564-909d-42d8-b5c6-e6a97da5ceff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:30.662888 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:30.662837 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a88e2fc-hlqdw"] Oct 31 04:19:30.706166 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:30.697476 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:30.706166 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:30.698240 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" Oct 31 04:19:30.743471 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:30.733590 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" podUID="dccd15a9-47f5-4bd1-b21b-9d5ca8b05845" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:30.893418 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:30.889366 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:30.911505 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:30.898030 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkjjf\" (UniqueName: \"kubernetes.io/projected/fde3d56b-03cd-45b0-a5b6-6f0377f79a84-kube-api-access-gkjjf\") pod \"maas-api-revoke-sub-a714eaa5-w46mr\" (UID: \"fde3d56b-03cd-45b0-a5b6-6f0377f79a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a714eaa5-w46mr" Oct 31 04:19:30.912167 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:30.912133 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:19:30.916427 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:30.916374 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r" podUID="fffb7c60-8cb2-40f0-903c-9b98c62a1741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:30.987424 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:30.986688 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fde3d56b-03cd-45b0-a5b6-6f0377f79a84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a714eaa5-w46mr\" (UID: \"fde3d56b-03cd-45b0-a5b6-6f0377f79a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a714eaa5-w46mr" Oct 31 04:19:30.998375 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:30.989483 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" Oct 31 04:19:30.999966 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:30.999535 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a714eaa5-w46mr" Oct 31 04:19:31.020729 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:31.020694 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45" Oct 31 04:19:31.028781 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:31.024965 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45" Oct 31 04:19:31.040842 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:31.040811 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" Oct 31 04:19:31.041024 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:31.040901 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45" Oct 31 04:19:31.052085 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:31.052055 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 31 04:19:31.052375 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:31.052322 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c" podUID="dda0567c-8a01-47e8-95f9-c0a4ad08a1c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 31 04:19:31.056506 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:31.056475 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a88e2fc-hlqdw" event={"ID":"4989f9aa-dd46-46da-9213-4055a2a3500e","Type":"ContainerStarted","Data":"b96270d8823e9d3bd484e8519022f9c82c59c449528cdbeae2cc70f0b4ab651e"} Oct 31 04:19:31.058185 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:31.058141 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" Oct 31 04:19:31.419462 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:19:31.419420 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9256bfe_8c85_477e_bcc9_9938f4c689ab.slice/crio-62872a8668c593cca5a3547af0b3c7fa4c891dfc555442cd74100f0af39e3b52 WatchSource:0}: Error finding container 62872a8668c593cca5a3547af0b3c7fa4c891dfc555442cd74100f0af39e3b52: Status 404 returned error can't find the container with id 62872a8668c593cca5a3547af0b3c7fa4c891dfc555442cd74100f0af39e3b52 Oct 31 04:19:31.423782 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:31.423734 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:19:31.423917 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:31.423851 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:19:31.423987 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:31.423915 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:19:31.439410 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:31.439380 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx"] Oct 31 04:19:31.467002 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:31.466971 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a714eaa5-w46mr"] Oct 31 04:19:31.856740 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:31.856700 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 31 04:19:31.856952 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:31.856783 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c" podUID="dda0567c-8a01-47e8-95f9-c0a4ad08a1c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 31 04:19:32.065285 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:32.065252 5019 generic.go:356] "Generic (PLEG): container finished" podID="4989f9aa-dd46-46da-9213-4055a2a3500e" containerID="97784aa525a4c157390b525855e2d43e0c21a8ddc608ca7b7b2149b53c20a6a1" exitCode=0 Oct 31 04:19:32.065511 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:32.065379 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a88e2fc-hlqdw" event={"ID":"4989f9aa-dd46-46da-9213-4055a2a3500e","Type":"ContainerDied","Data":"97784aa525a4c157390b525855e2d43e0c21a8ddc608ca7b7b2149b53c20a6a1"} Oct 31 04:19:32.067022 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:32.066993 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a714eaa5-w46mr" event={"ID":"fde3d56b-03cd-45b0-a5b6-6f0377f79a84","Type":"ContainerStarted","Data":"16bfe4bf35bf7238f6a5558bb61bd3594c7fce3e9013a4ec6adb4f130a98cfc5"} Oct 31 04:19:32.067153 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:32.067052 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a714eaa5-w46mr" event={"ID":"fde3d56b-03cd-45b0-a5b6-6f0377f79a84","Type":"ContainerStarted","Data":"f7ef8b281a6eb7fa3af6e710abc9c1af9862ecac96cc4b2336a784525b092a3e"} Oct 31 04:19:32.068918 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:32.068894 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx" event={"ID":"a9256bfe-8c85-477e-bcc9-9938f4c689ab","Type":"ContainerStarted","Data":"84c03351a7346208e097d915cb61ad8f2ad39cf138cbc607b6e18e154308f2bb"} Oct 31 04:19:32.069052 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:32.068940 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx" event={"ID":"a9256bfe-8c85-477e-bcc9-9938f4c689ab","Type":"ContainerStarted","Data":"62872a8668c593cca5a3547af0b3c7fa4c891dfc555442cd74100f0af39e3b52"} Oct 31 04:19:32.106169 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:32.106043 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx" Oct 31 04:19:32.108288 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:32.108241 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 31 04:19:32.108472 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:32.108306 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx" podUID="a9256bfe-8c85-477e-bcc9-9938f4c689ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 31 04:19:32.150411 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:32.150345 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a714eaa5-w46mr" podStartSLOduration=3.150312246 podStartE2EDuration="3.150312246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:19:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:19:32.145622668 +0000 UTC m=+926.901960707" watchObservedRunningTime="2026-10-31 04:19:32.150312246 +0000 UTC m=+926.906650291" Oct 31 04:19:32.193371 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:32.193270 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx" podStartSLOduration=4.193250032 podStartE2EDuration="4.193250032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:19:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:19:32.190259776 +0000 UTC m=+926.946597816" watchObservedRunningTime="2026-10-31 04:19:32.193250032 +0000 UTC m=+926.949588074" Oct 31 04:19:33.118067 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:33.117504 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 31 04:19:33.118067 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:33.117570 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx" podUID="a9256bfe-8c85-477e-bcc9-9938f4c689ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 31 04:19:33.122603 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:33.122414 5019 generic.go:356] "Generic (PLEG): container finished" podID="fde3d56b-03cd-45b0-a5b6-6f0377f79a84" containerID="16bfe4bf35bf7238f6a5558bb61bd3594c7fce3e9013a4ec6adb4f130a98cfc5" exitCode=0 Oct 31 04:19:33.122603 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:33.122503 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a714eaa5-w46mr" event={"ID":"fde3d56b-03cd-45b0-a5b6-6f0377f79a84","Type":"ContainerDied","Data":"16bfe4bf35bf7238f6a5558bb61bd3594c7fce3e9013a4ec6adb4f130a98cfc5"} Oct 31 04:19:33.521955 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:33.521925 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a88e2fc-hlqdw" Oct 31 04:19:33.698361 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:33.694573 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4989f9aa-dd46-46da-9213-4055a2a3500e-maas-api-openshift-ca" pod "4989f9aa-dd46-46da-9213-4055a2a3500e" (UID: "4989f9aa-dd46-46da-9213-4055a2a3500e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:19:33.733354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:33.691006 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4989f9aa-dd46-46da-9213-4055a2a3500e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4989f9aa-dd46-46da-9213-4055a2a3500e-maas-api-openshift-ca\") pod \"4989f9aa-dd46-46da-9213-4055a2a3500e\" (UID: \"4989f9aa-dd46-46da-9213-4055a2a3500e\") " Oct 31 04:19:33.733354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:33.720822 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4989f9aa-dd46-46da-9213-4055a2a3500e-kube-api-access-ct7l8\" (UniqueName: \"kubernetes.io/projected/4989f9aa-dd46-46da-9213-4055a2a3500e-kube-api-access-ct7l8\") pod \"4989f9aa-dd46-46da-9213-4055a2a3500e\" (UID: \"4989f9aa-dd46-46da-9213-4055a2a3500e\") " Oct 31 04:19:33.733354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:33.720864 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4989f9aa-dd46-46da-9213-4055a2a3500e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4989f9aa-dd46-46da-9213-4055a2a3500e-maas-api-xks-ca\") pod \"4989f9aa-dd46-46da-9213-4055a2a3500e\" (UID: \"4989f9aa-dd46-46da-9213-4055a2a3500e\") " Oct 31 04:19:33.733354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:33.721420 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4989f9aa-dd46-46da-9213-4055a2a3500e-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:19:33.733354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:33.729762 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4989f9aa-dd46-46da-9213-4055a2a3500e-kube-api-access-ct7l8" pod "4989f9aa-dd46-46da-9213-4055a2a3500e" (UID: "4989f9aa-dd46-46da-9213-4055a2a3500e"). InnerVolumeSpecName "kube-api-access-ct7l8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:19:33.736123 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:33.736076 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4989f9aa-dd46-46da-9213-4055a2a3500e-maas-api-xks-ca" pod "4989f9aa-dd46-46da-9213-4055a2a3500e" (UID: "4989f9aa-dd46-46da-9213-4055a2a3500e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:19:33.822578 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:33.822539 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ct7l8\" (UniqueName: \"kubernetes.io/projected/4989f9aa-dd46-46da-9213-4055a2a3500e-kube-api-access-ct7l8\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:19:33.822578 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:33.822573 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4989f9aa-dd46-46da-9213-4055a2a3500e-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:19:33.857239 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:33.857193 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:19:33.857454 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:33.857258 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c" podUID="dda0567c-8a01-47e8-95f9-c0a4ad08a1c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:34.107227 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:34.107134 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 31 04:19:34.107227 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:34.107208 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx" podUID="a9256bfe-8c85-477e-bcc9-9938f4c689ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 31 04:19:34.133473 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:34.133386 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a88e2fc-hlqdw" event={"ID":"4989f9aa-dd46-46da-9213-4055a2a3500e","Type":"ContainerDied","Data":"b96270d8823e9d3bd484e8519022f9c82c59c449528cdbeae2cc70f0b4ab651e"} Oct 31 04:19:34.133473 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:34.133454 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b96270d8823e9d3bd484e8519022f9c82c59c449528cdbeae2cc70f0b4ab651e" Oct 31 04:19:34.134630 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:34.134407 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a88e2fc-hlqdw" Oct 31 04:19:34.456971 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:34.456941 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a714eaa5-w46mr" Oct 31 04:19:34.645356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:34.644499 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fde3d56b-03cd-45b0-a5b6-6f0377f79a84-kube-api-access-gkjjf\" (UniqueName: \"kubernetes.io/projected/fde3d56b-03cd-45b0-a5b6-6f0377f79a84-kube-api-access-gkjjf\") pod \"fde3d56b-03cd-45b0-a5b6-6f0377f79a84\" (UID: \"fde3d56b-03cd-45b0-a5b6-6f0377f79a84\") " Oct 31 04:19:34.645356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:34.644565 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fde3d56b-03cd-45b0-a5b6-6f0377f79a84-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fde3d56b-03cd-45b0-a5b6-6f0377f79a84-maas-api-xks-ca\") pod \"fde3d56b-03cd-45b0-a5b6-6f0377f79a84\" (UID: \"fde3d56b-03cd-45b0-a5b6-6f0377f79a84\") " Oct 31 04:19:34.645356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:34.644671 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fde3d56b-03cd-45b0-a5b6-6f0377f79a84-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fde3d56b-03cd-45b0-a5b6-6f0377f79a84-maas-api-openshift-ca\") pod \"fde3d56b-03cd-45b0-a5b6-6f0377f79a84\" (UID: \"fde3d56b-03cd-45b0-a5b6-6f0377f79a84\") " Oct 31 04:19:34.645684 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:34.645540 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fde3d56b-03cd-45b0-a5b6-6f0377f79a84-maas-api-openshift-ca" pod "fde3d56b-03cd-45b0-a5b6-6f0377f79a84" (UID: "fde3d56b-03cd-45b0-a5b6-6f0377f79a84"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:19:34.652095 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:34.647424 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fde3d56b-03cd-45b0-a5b6-6f0377f79a84-kube-api-access-gkjjf" pod "fde3d56b-03cd-45b0-a5b6-6f0377f79a84" (UID: "fde3d56b-03cd-45b0-a5b6-6f0377f79a84"). InnerVolumeSpecName "kube-api-access-gkjjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:19:34.652095 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:34.648097 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fde3d56b-03cd-45b0-a5b6-6f0377f79a84-maas-api-xks-ca" pod "fde3d56b-03cd-45b0-a5b6-6f0377f79a84" (UID: "fde3d56b-03cd-45b0-a5b6-6f0377f79a84"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:19:34.747169 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:34.747127 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkjjf\" (UniqueName: \"kubernetes.io/projected/fde3d56b-03cd-45b0-a5b6-6f0377f79a84-kube-api-access-gkjjf\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:19:34.747447 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:34.747427 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fde3d56b-03cd-45b0-a5b6-6f0377f79a84-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:19:34.747553 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:34.747540 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fde3d56b-03cd-45b0-a5b6-6f0377f79a84-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:19:34.899624 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:34.899582 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:19:34.899944 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:34.899899 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c" podUID="dda0567c-8a01-47e8-95f9-c0a4ad08a1c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:35.035248 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:35.035149 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c" Oct 31 04:19:35.035618 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:35.035588 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c" Oct 31 04:19:35.036633 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:35.036609 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c" Oct 31 04:19:35.106368 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:35.106302 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 31 04:19:35.106578 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:35.106406 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx" podUID="a9256bfe-8c85-477e-bcc9-9938f4c689ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 31 04:19:35.147986 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:35.147954 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a714eaa5-w46mr" Oct 31 04:19:35.149422 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:35.149391 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a714eaa5-w46mr" event={"ID":"fde3d56b-03cd-45b0-a5b6-6f0377f79a84","Type":"ContainerDied","Data":"f7ef8b281a6eb7fa3af6e710abc9c1af9862ecac96cc4b2336a784525b092a3e"} Oct 31 04:19:35.149549 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:35.149454 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7ef8b281a6eb7fa3af6e710abc9c1af9862ecac96cc4b2336a784525b092a3e" Oct 31 04:19:36.106411 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:36.106372 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 31 04:19:36.106610 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:36.106441 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx" podUID="a9256bfe-8c85-477e-bcc9-9938f4c689ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 31 04:19:37.106988 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:37.106943 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 31 04:19:37.107417 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:37.107010 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx" podUID="a9256bfe-8c85-477e-bcc9-9938f4c689ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 31 04:19:37.107417 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:37.107221 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9ee9cc0-gjdnb"] Oct 31 04:19:37.108117 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:37.108096 5019 cpu_manager.go:414] "containerMap: removing container" podUID="4989f9aa-dd46-46da-9213-4055a2a3500e" containerName="revoke-keys" Oct 31 04:19:37.108162 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:37.108120 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4989f9aa-dd46-46da-9213-4055a2a3500e" containerName="revoke-keys" Oct 31 04:19:37.108162 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:37.108144 5019 cpu_manager.go:414] "containerMap: removing container" podUID="fde3d56b-03cd-45b0-a5b6-6f0377f79a84" containerName="revoke-keys" Oct 31 04:19:37.108162 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:37.108152 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fde3d56b-03cd-45b0-a5b6-6f0377f79a84" containerName="revoke-keys" Oct 31 04:19:37.108361 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:37.108349 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="4989f9aa-dd46-46da-9213-4055a2a3500e" containerName="revoke-keys" Oct 31 04:19:37.108401 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:37.108369 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="fde3d56b-03cd-45b0-a5b6-6f0377f79a84" containerName="revoke-keys" Oct 31 04:19:37.112769 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:37.112743 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9ee9cc0-gjdnb" Oct 31 04:19:37.130205 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:37.130163 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9ee9cc0-gjdnb"] Oct 31 04:19:37.136259 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:37.136223 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:19:37.136458 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:37.136320 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:19:37.136458 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:37.136424 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:19:37.150378 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:37.142600 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:19:37.208107 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:37.208068 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf6kj\" (UniqueName: \"kubernetes.io/projected/c76b84e1-d1e0-4a8d-bcbe-9fe1f6792097-kube-api-access-qf6kj\") pod \"maas-api-revoke-sub-f9ee9cc0-gjdnb\" (UID: \"c76b84e1-d1e0-4a8d-bcbe-9fe1f6792097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9ee9cc0-gjdnb" Oct 31 04:19:37.208451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:37.208427 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c76b84e1-d1e0-4a8d-bcbe-9fe1f6792097-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9ee9cc0-gjdnb\" (UID: \"c76b84e1-d1e0-4a8d-bcbe-9fe1f6792097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9ee9cc0-gjdnb" Oct 31 04:19:37.208632 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:37.208613 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c76b84e1-d1e0-4a8d-bcbe-9fe1f6792097-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9ee9cc0-gjdnb\" (UID: \"c76b84e1-d1e0-4a8d-bcbe-9fe1f6792097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9ee9cc0-gjdnb" Oct 31 04:19:37.310161 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:37.310121 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf6kj\" (UniqueName: \"kubernetes.io/projected/c76b84e1-d1e0-4a8d-bcbe-9fe1f6792097-kube-api-access-qf6kj\") pod \"maas-api-revoke-sub-f9ee9cc0-gjdnb\" (UID: \"c76b84e1-d1e0-4a8d-bcbe-9fe1f6792097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9ee9cc0-gjdnb" Oct 31 04:19:37.310362 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:37.310177 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c76b84e1-d1e0-4a8d-bcbe-9fe1f6792097-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9ee9cc0-gjdnb\" (UID: \"c76b84e1-d1e0-4a8d-bcbe-9fe1f6792097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9ee9cc0-gjdnb" Oct 31 04:19:37.310362 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:37.310309 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c76b84e1-d1e0-4a8d-bcbe-9fe1f6792097-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9ee9cc0-gjdnb\" (UID: \"c76b84e1-d1e0-4a8d-bcbe-9fe1f6792097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9ee9cc0-gjdnb" Oct 31 04:19:37.310973 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:37.310943 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c76b84e1-d1e0-4a8d-bcbe-9fe1f6792097-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9ee9cc0-gjdnb\" (UID: \"c76b84e1-d1e0-4a8d-bcbe-9fe1f6792097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9ee9cc0-gjdnb" Oct 31 04:19:37.312921 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:37.312891 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c76b84e1-d1e0-4a8d-bcbe-9fe1f6792097-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9ee9cc0-gjdnb\" (UID: \"c76b84e1-d1e0-4a8d-bcbe-9fe1f6792097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9ee9cc0-gjdnb" Oct 31 04:19:37.333518 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:37.333485 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf6kj\" (UniqueName: \"kubernetes.io/projected/c76b84e1-d1e0-4a8d-bcbe-9fe1f6792097-kube-api-access-qf6kj\") pod \"maas-api-revoke-sub-f9ee9cc0-gjdnb\" (UID: \"c76b84e1-d1e0-4a8d-bcbe-9fe1f6792097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9ee9cc0-gjdnb" Oct 31 04:19:37.432257 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:37.432147 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9ee9cc0-gjdnb" Oct 31 04:19:37.798941 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:37.767751 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9ee9cc0-gjdnb"] Oct 31 04:19:37.798941 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:19:37.768068 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc76b84e1_d1e0_4a8d_bcbe_9fe1f6792097.slice/crio-28ec5392a6e6d4d3b6344d1d07054089df85dbee2e1b9e122445d434206139e2 WatchSource:0}: Error finding container 28ec5392a6e6d4d3b6344d1d07054089df85dbee2e1b9e122445d434206139e2: Status 404 returned error can't find the container with id 28ec5392a6e6d4d3b6344d1d07054089df85dbee2e1b9e122445d434206139e2 Oct 31 04:19:38.106807 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:38.106682 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 31 04:19:38.106807 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:38.106751 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx" podUID="a9256bfe-8c85-477e-bcc9-9938f4c689ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 31 04:19:38.217730 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:38.217689 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9ee9cc0-gjdnb" event={"ID":"c76b84e1-d1e0-4a8d-bcbe-9fe1f6792097","Type":"ContainerStarted","Data":"819300b37635043a8938891567191ac2ec3998701be3318bfeca62ed0e5cb7cb"} Oct 31 04:19:38.218229 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:38.217757 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9ee9cc0-gjdnb" event={"ID":"c76b84e1-d1e0-4a8d-bcbe-9fe1f6792097","Type":"ContainerStarted","Data":"28ec5392a6e6d4d3b6344d1d07054089df85dbee2e1b9e122445d434206139e2"} Oct 31 04:19:38.259044 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:38.258986 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9ee9cc0-gjdnb" podStartSLOduration=1.258972204 podStartE2EDuration="1.258972204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:19:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:19:38.257036857 +0000 UTC m=+933.013374908" watchObservedRunningTime="2026-10-31 04:19:38.258972204 +0000 UTC m=+933.015310281" Oct 31 04:19:39.230006 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:39.229966 5019 generic.go:356] "Generic (PLEG): container finished" podID="c76b84e1-d1e0-4a8d-bcbe-9fe1f6792097" containerID="819300b37635043a8938891567191ac2ec3998701be3318bfeca62ed0e5cb7cb" exitCode=0 Oct 31 04:19:39.230618 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:39.230072 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9ee9cc0-gjdnb" event={"ID":"c76b84e1-d1e0-4a8d-bcbe-9fe1f6792097","Type":"ContainerDied","Data":"819300b37635043a8938891567191ac2ec3998701be3318bfeca62ed0e5cb7cb"} Oct 31 04:19:39.422720 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:39.422685 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx" Oct 31 04:19:39.423708 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:39.423541 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx" Oct 31 04:19:39.424807 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:39.424782 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx" Oct 31 04:19:40.429801 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:40.429762 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9ee9cc0-gjdnb" Oct 31 04:19:40.453994 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:40.453959 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c76b84e1-d1e0-4a8d-bcbe-9fe1f6792097-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c76b84e1-d1e0-4a8d-bcbe-9fe1f6792097-maas-api-openshift-ca\") pod \"c76b84e1-d1e0-4a8d-bcbe-9fe1f6792097\" (UID: \"c76b84e1-d1e0-4a8d-bcbe-9fe1f6792097\") " Oct 31 04:19:40.454178 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:40.454020 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c76b84e1-d1e0-4a8d-bcbe-9fe1f6792097-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c76b84e1-d1e0-4a8d-bcbe-9fe1f6792097-maas-api-xks-ca\") pod \"c76b84e1-d1e0-4a8d-bcbe-9fe1f6792097\" (UID: \"c76b84e1-d1e0-4a8d-bcbe-9fe1f6792097\") " Oct 31 04:19:40.454178 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:40.454113 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c76b84e1-d1e0-4a8d-bcbe-9fe1f6792097-kube-api-access-qf6kj\" (UniqueName: \"kubernetes.io/projected/c76b84e1-d1e0-4a8d-bcbe-9fe1f6792097-kube-api-access-qf6kj\") pod \"c76b84e1-d1e0-4a8d-bcbe-9fe1f6792097\" (UID: \"c76b84e1-d1e0-4a8d-bcbe-9fe1f6792097\") " Oct 31 04:19:40.454476 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:40.454434 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c76b84e1-d1e0-4a8d-bcbe-9fe1f6792097-maas-api-openshift-ca" pod "c76b84e1-d1e0-4a8d-bcbe-9fe1f6792097" (UID: "c76b84e1-d1e0-4a8d-bcbe-9fe1f6792097"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:19:40.454801 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:40.454768 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c76b84e1-d1e0-4a8d-bcbe-9fe1f6792097-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:19:40.456228 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:40.456202 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c76b84e1-d1e0-4a8d-bcbe-9fe1f6792097-maas-api-xks-ca" pod "c76b84e1-d1e0-4a8d-bcbe-9fe1f6792097" (UID: "c76b84e1-d1e0-4a8d-bcbe-9fe1f6792097"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:19:40.456483 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:40.456458 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c76b84e1-d1e0-4a8d-bcbe-9fe1f6792097-kube-api-access-qf6kj" pod "c76b84e1-d1e0-4a8d-bcbe-9fe1f6792097" (UID: "c76b84e1-d1e0-4a8d-bcbe-9fe1f6792097"). InnerVolumeSpecName "kube-api-access-qf6kj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:19:40.556589 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:40.556501 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf6kj\" (UniqueName: \"kubernetes.io/projected/c76b84e1-d1e0-4a8d-bcbe-9fe1f6792097-kube-api-access-qf6kj\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:19:40.556589 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:40.556537 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c76b84e1-d1e0-4a8d-bcbe-9fe1f6792097-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:19:41.252311 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:41.252283 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9ee9cc0-gjdnb" Oct 31 04:19:41.252522 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:41.252304 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9ee9cc0-gjdnb" event={"ID":"c76b84e1-d1e0-4a8d-bcbe-9fe1f6792097","Type":"ContainerDied","Data":"28ec5392a6e6d4d3b6344d1d07054089df85dbee2e1b9e122445d434206139e2"} Oct 31 04:19:41.252522 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:41.252418 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28ec5392a6e6d4d3b6344d1d07054089df85dbee2e1b9e122445d434206139e2" Oct 31 04:19:41.665602 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:41.665519 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bdf0224-qlllb"] Oct 31 04:19:41.666122 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:41.666108 5019 cpu_manager.go:414] "containerMap: removing container" podUID="c76b84e1-d1e0-4a8d-bcbe-9fe1f6792097" containerName="revoke-keys" Oct 31 04:19:41.666165 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:41.666124 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c76b84e1-d1e0-4a8d-bcbe-9fe1f6792097" containerName="revoke-keys" Oct 31 04:19:41.666225 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:41.666217 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="c76b84e1-d1e0-4a8d-bcbe-9fe1f6792097" containerName="revoke-keys" Oct 31 04:19:41.670012 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:41.669985 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bdf0224-qlllb" Oct 31 04:19:41.685763 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:41.685733 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:19:41.689974 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:41.689940 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bdf0224-qlllb"] Oct 31 04:19:41.695915 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:41.695886 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:19:41.697661 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:41.697624 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:19:41.697661 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:41.697639 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:19:41.776885 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:41.776848 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38a952e7-44a3-4c01-85b6-ea60d9db1984-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bdf0224-qlllb\" (UID: \"38a952e7-44a3-4c01-85b6-ea60d9db1984\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bdf0224-qlllb" Oct 31 04:19:41.777070 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:41.777017 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8pv8\" (UniqueName: \"kubernetes.io/projected/38a952e7-44a3-4c01-85b6-ea60d9db1984-kube-api-access-k8pv8\") pod \"maas-api-revoke-sub-9bdf0224-qlllb\" (UID: \"38a952e7-44a3-4c01-85b6-ea60d9db1984\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bdf0224-qlllb" Oct 31 04:19:41.777070 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:41.777046 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38a952e7-44a3-4c01-85b6-ea60d9db1984-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bdf0224-qlllb\" (UID: \"38a952e7-44a3-4c01-85b6-ea60d9db1984\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bdf0224-qlllb" Oct 31 04:19:41.878538 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:41.878499 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38a952e7-44a3-4c01-85b6-ea60d9db1984-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bdf0224-qlllb\" (UID: \"38a952e7-44a3-4c01-85b6-ea60d9db1984\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bdf0224-qlllb" Oct 31 04:19:41.878696 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:41.878630 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8pv8\" (UniqueName: \"kubernetes.io/projected/38a952e7-44a3-4c01-85b6-ea60d9db1984-kube-api-access-k8pv8\") pod \"maas-api-revoke-sub-9bdf0224-qlllb\" (UID: \"38a952e7-44a3-4c01-85b6-ea60d9db1984\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bdf0224-qlllb" Oct 31 04:19:41.878696 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:41.878657 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38a952e7-44a3-4c01-85b6-ea60d9db1984-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bdf0224-qlllb\" (UID: \"38a952e7-44a3-4c01-85b6-ea60d9db1984\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bdf0224-qlllb" Oct 31 04:19:41.879188 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:41.879161 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38a952e7-44a3-4c01-85b6-ea60d9db1984-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bdf0224-qlllb\" (UID: \"38a952e7-44a3-4c01-85b6-ea60d9db1984\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bdf0224-qlllb" Oct 31 04:19:41.881148 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:41.881119 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38a952e7-44a3-4c01-85b6-ea60d9db1984-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bdf0224-qlllb\" (UID: \"38a952e7-44a3-4c01-85b6-ea60d9db1984\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bdf0224-qlllb" Oct 31 04:19:41.902174 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:41.902144 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8pv8\" (UniqueName: \"kubernetes.io/projected/38a952e7-44a3-4c01-85b6-ea60d9db1984-kube-api-access-k8pv8\") pod \"maas-api-revoke-sub-9bdf0224-qlllb\" (UID: \"38a952e7-44a3-4c01-85b6-ea60d9db1984\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bdf0224-qlllb" Oct 31 04:19:41.984615 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:41.984562 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bdf0224-qlllb" Oct 31 04:19:42.290548 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:42.290513 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bdf0224-qlllb"] Oct 31 04:19:42.292125 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:19:42.292095 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38a952e7_44a3_4c01_85b6_ea60d9db1984.slice/crio-bc543a940fe26afdce78835a4c716295c71f25881e7996052b8335a9c152673a WatchSource:0}: Error finding container bc543a940fe26afdce78835a4c716295c71f25881e7996052b8335a9c152673a: Status 404 returned error can't find the container with id bc543a940fe26afdce78835a4c716295c71f25881e7996052b8335a9c152673a Oct 31 04:19:43.268158 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:43.268125 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8809a9e-jsfw5"] Oct 31 04:19:43.272957 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:43.272921 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bdf0224-qlllb" event={"ID":"38a952e7-44a3-4c01-85b6-ea60d9db1984","Type":"ContainerStarted","Data":"b33c291d16db72d110593a6fe200bb4f12be8b1a840b8a452ae9355d972d94dc"} Oct 31 04:19:43.273114 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:43.272969 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bdf0224-qlllb" event={"ID":"38a952e7-44a3-4c01-85b6-ea60d9db1984","Type":"ContainerStarted","Data":"bc543a940fe26afdce78835a4c716295c71f25881e7996052b8335a9c152673a"} Oct 31 04:19:43.273114 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:43.273036 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8809a9e-jsfw5" Oct 31 04:19:43.294161 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:43.294128 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8809a9e-jsfw5"] Oct 31 04:19:43.319245 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:43.319174 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bdf0224-qlllb" podStartSLOduration=2.319150739 podStartE2EDuration="2.319150739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:19:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:19:43.312122813 +0000 UTC m=+938.068460853" watchObservedRunningTime="2026-10-31 04:19:43.319150739 +0000 UTC m=+938.075488780" Oct 31 04:19:43.400883 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:43.400840 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/939530ef-60ca-4099-b12e-f5c922c564d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8809a9e-jsfw5\" (UID: \"939530ef-60ca-4099-b12e-f5c922c564d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8809a9e-jsfw5" Oct 31 04:19:43.400883 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:43.400894 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/939530ef-60ca-4099-b12e-f5c922c564d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8809a9e-jsfw5\" (UID: \"939530ef-60ca-4099-b12e-f5c922c564d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8809a9e-jsfw5" Oct 31 04:19:43.401243 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:43.401210 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gxr2\" (UniqueName: \"kubernetes.io/projected/939530ef-60ca-4099-b12e-f5c922c564d7-kube-api-access-4gxr2\") pod \"maas-api-revoke-sub-e8809a9e-jsfw5\" (UID: \"939530ef-60ca-4099-b12e-f5c922c564d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8809a9e-jsfw5" Oct 31 04:19:43.502485 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:43.502445 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gxr2\" (UniqueName: \"kubernetes.io/projected/939530ef-60ca-4099-b12e-f5c922c564d7-kube-api-access-4gxr2\") pod \"maas-api-revoke-sub-e8809a9e-jsfw5\" (UID: \"939530ef-60ca-4099-b12e-f5c922c564d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8809a9e-jsfw5" Oct 31 04:19:43.502672 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:43.502570 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/939530ef-60ca-4099-b12e-f5c922c564d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8809a9e-jsfw5\" (UID: \"939530ef-60ca-4099-b12e-f5c922c564d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8809a9e-jsfw5" Oct 31 04:19:43.502672 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:43.502595 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/939530ef-60ca-4099-b12e-f5c922c564d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8809a9e-jsfw5\" (UID: \"939530ef-60ca-4099-b12e-f5c922c564d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8809a9e-jsfw5" Oct 31 04:19:43.503536 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:43.503503 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/939530ef-60ca-4099-b12e-f5c922c564d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8809a9e-jsfw5\" (UID: \"939530ef-60ca-4099-b12e-f5c922c564d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8809a9e-jsfw5" Oct 31 04:19:43.514354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:43.514294 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/939530ef-60ca-4099-b12e-f5c922c564d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8809a9e-jsfw5\" (UID: \"939530ef-60ca-4099-b12e-f5c922c564d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8809a9e-jsfw5" Oct 31 04:19:43.526111 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:43.526036 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gxr2\" (UniqueName: \"kubernetes.io/projected/939530ef-60ca-4099-b12e-f5c922c564d7-kube-api-access-4gxr2\") pod \"maas-api-revoke-sub-e8809a9e-jsfw5\" (UID: \"939530ef-60ca-4099-b12e-f5c922c564d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8809a9e-jsfw5" Oct 31 04:19:43.676386 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:43.676342 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8809a9e-jsfw5" Oct 31 04:19:43.979083 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:19:43.979039 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod939530ef_60ca_4099_b12e_f5c922c564d7.slice/crio-c64c2dc609f9d9683619ef208193ef3a87a8d9cd7fb4a5118e3719ab0fcc3578 WatchSource:0}: Error finding container c64c2dc609f9d9683619ef208193ef3a87a8d9cd7fb4a5118e3719ab0fcc3578: Status 404 returned error can't find the container with id c64c2dc609f9d9683619ef208193ef3a87a8d9cd7fb4a5118e3719ab0fcc3578 Oct 31 04:19:43.980622 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:43.980589 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8809a9e-jsfw5"] Oct 31 04:19:44.282575 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:44.282534 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8809a9e-jsfw5" event={"ID":"939530ef-60ca-4099-b12e-f5c922c564d7","Type":"ContainerStarted","Data":"c64c2dc609f9d9683619ef208193ef3a87a8d9cd7fb4a5118e3719ab0fcc3578"} Oct 31 04:19:44.284444 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:44.284406 5019 generic.go:356] "Generic (PLEG): container finished" podID="38a952e7-44a3-4c01-85b6-ea60d9db1984" containerID="b33c291d16db72d110593a6fe200bb4f12be8b1a840b8a452ae9355d972d94dc" exitCode=0 Oct 31 04:19:44.284551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:44.284469 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bdf0224-qlllb" event={"ID":"38a952e7-44a3-4c01-85b6-ea60d9db1984","Type":"ContainerDied","Data":"b33c291d16db72d110593a6fe200bb4f12be8b1a840b8a452ae9355d972d94dc"} Oct 31 04:19:45.295385 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:45.295274 5019 generic.go:356] "Generic (PLEG): container finished" podID="939530ef-60ca-4099-b12e-f5c922c564d7" containerID="c5a979510bd229a6c9219856e2b2aef14d35654aeb94673f056c8f7b2f7813b9" exitCode=0 Oct 31 04:19:45.295385 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:45.295368 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8809a9e-jsfw5" event={"ID":"939530ef-60ca-4099-b12e-f5c922c564d7","Type":"ContainerDied","Data":"c5a979510bd229a6c9219856e2b2aef14d35654aeb94673f056c8f7b2f7813b9"} Oct 31 04:19:45.479824 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:45.479779 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bdf0224-qlllb" Oct 31 04:19:45.530877 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:45.530835 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38a952e7-44a3-4c01-85b6-ea60d9db1984-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38a952e7-44a3-4c01-85b6-ea60d9db1984-maas-api-xks-ca\") pod \"38a952e7-44a3-4c01-85b6-ea60d9db1984\" (UID: \"38a952e7-44a3-4c01-85b6-ea60d9db1984\") " Oct 31 04:19:45.531077 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:45.530896 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38a952e7-44a3-4c01-85b6-ea60d9db1984-kube-api-access-k8pv8\" (UniqueName: \"kubernetes.io/projected/38a952e7-44a3-4c01-85b6-ea60d9db1984-kube-api-access-k8pv8\") pod \"38a952e7-44a3-4c01-85b6-ea60d9db1984\" (UID: \"38a952e7-44a3-4c01-85b6-ea60d9db1984\") " Oct 31 04:19:45.531077 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:45.531040 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38a952e7-44a3-4c01-85b6-ea60d9db1984-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38a952e7-44a3-4c01-85b6-ea60d9db1984-maas-api-openshift-ca\") pod \"38a952e7-44a3-4c01-85b6-ea60d9db1984\" (UID: \"38a952e7-44a3-4c01-85b6-ea60d9db1984\") " Oct 31 04:19:45.531487 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:45.531460 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38a952e7-44a3-4c01-85b6-ea60d9db1984-maas-api-openshift-ca" pod "38a952e7-44a3-4c01-85b6-ea60d9db1984" (UID: "38a952e7-44a3-4c01-85b6-ea60d9db1984"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:19:45.531824 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:45.531800 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38a952e7-44a3-4c01-85b6-ea60d9db1984-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:19:45.533715 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:45.533685 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38a952e7-44a3-4c01-85b6-ea60d9db1984-maas-api-xks-ca" pod "38a952e7-44a3-4c01-85b6-ea60d9db1984" (UID: "38a952e7-44a3-4c01-85b6-ea60d9db1984"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:19:45.533818 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:45.533777 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38a952e7-44a3-4c01-85b6-ea60d9db1984-kube-api-access-k8pv8" pod "38a952e7-44a3-4c01-85b6-ea60d9db1984" (UID: "38a952e7-44a3-4c01-85b6-ea60d9db1984"). InnerVolumeSpecName "kube-api-access-k8pv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:19:45.633014 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:45.632972 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38a952e7-44a3-4c01-85b6-ea60d9db1984-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:19:45.633014 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:45.633010 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8pv8\" (UniqueName: \"kubernetes.io/projected/38a952e7-44a3-4c01-85b6-ea60d9db1984-kube-api-access-k8pv8\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:19:46.305943 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:46.305910 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bdf0224-qlllb" Oct 31 04:19:46.305943 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:46.305937 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bdf0224-qlllb" event={"ID":"38a952e7-44a3-4c01-85b6-ea60d9db1984","Type":"ContainerDied","Data":"bc543a940fe26afdce78835a4c716295c71f25881e7996052b8335a9c152673a"} Oct 31 04:19:46.306456 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:46.305994 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc543a940fe26afdce78835a4c716295c71f25881e7996052b8335a9c152673a" Oct 31 04:19:46.495564 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:46.495520 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8809a9e-jsfw5" Oct 31 04:19:46.551811 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:46.551768 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/939530ef-60ca-4099-b12e-f5c922c564d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/939530ef-60ca-4099-b12e-f5c922c564d7-maas-api-xks-ca\") pod \"939530ef-60ca-4099-b12e-f5c922c564d7\" (UID: \"939530ef-60ca-4099-b12e-f5c922c564d7\") " Oct 31 04:19:46.551997 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:46.551906 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/939530ef-60ca-4099-b12e-f5c922c564d7-kube-api-access-4gxr2\" (UniqueName: \"kubernetes.io/projected/939530ef-60ca-4099-b12e-f5c922c564d7-kube-api-access-4gxr2\") pod \"939530ef-60ca-4099-b12e-f5c922c564d7\" (UID: \"939530ef-60ca-4099-b12e-f5c922c564d7\") " Oct 31 04:19:46.552136 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:46.552120 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/939530ef-60ca-4099-b12e-f5c922c564d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/939530ef-60ca-4099-b12e-f5c922c564d7-maas-api-openshift-ca\") pod \"939530ef-60ca-4099-b12e-f5c922c564d7\" (UID: \"939530ef-60ca-4099-b12e-f5c922c564d7\") " Oct 31 04:19:46.552614 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:46.552579 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/939530ef-60ca-4099-b12e-f5c922c564d7-maas-api-openshift-ca" pod "939530ef-60ca-4099-b12e-f5c922c564d7" (UID: "939530ef-60ca-4099-b12e-f5c922c564d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:19:46.553277 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:46.553252 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/939530ef-60ca-4099-b12e-f5c922c564d7-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:19:46.555922 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:46.555890 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/939530ef-60ca-4099-b12e-f5c922c564d7-maas-api-xks-ca" pod "939530ef-60ca-4099-b12e-f5c922c564d7" (UID: "939530ef-60ca-4099-b12e-f5c922c564d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:19:46.556088 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:46.556024 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/939530ef-60ca-4099-b12e-f5c922c564d7-kube-api-access-4gxr2" pod "939530ef-60ca-4099-b12e-f5c922c564d7" (UID: "939530ef-60ca-4099-b12e-f5c922c564d7"). InnerVolumeSpecName "kube-api-access-4gxr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:19:46.654748 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:46.654705 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/939530ef-60ca-4099-b12e-f5c922c564d7-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:19:46.654748 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:46.654742 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gxr2\" (UniqueName: \"kubernetes.io/projected/939530ef-60ca-4099-b12e-f5c922c564d7-kube-api-access-4gxr2\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:19:47.130512 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:47.130480 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20fdab3e-2nmbn"] Oct 31 04:19:47.131207 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:47.131175 5019 cpu_manager.go:414] "containerMap: removing container" podUID="38a952e7-44a3-4c01-85b6-ea60d9db1984" containerName="revoke-keys" Oct 31 04:19:47.131207 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:47.131198 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38a952e7-44a3-4c01-85b6-ea60d9db1984" containerName="revoke-keys" Oct 31 04:19:47.131441 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:47.131236 5019 cpu_manager.go:414] "containerMap: removing container" podUID="939530ef-60ca-4099-b12e-f5c922c564d7" containerName="revoke-keys" Oct 31 04:19:47.131441 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:47.131243 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="939530ef-60ca-4099-b12e-f5c922c564d7" containerName="revoke-keys" Oct 31 04:19:47.131441 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:47.131416 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="939530ef-60ca-4099-b12e-f5c922c564d7" containerName="revoke-keys" Oct 31 04:19:47.131441 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:47.131442 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="38a952e7-44a3-4c01-85b6-ea60d9db1984" containerName="revoke-keys" Oct 31 04:19:47.135218 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:47.135198 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20fdab3e-2nmbn" Oct 31 04:19:47.193745 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:47.193702 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20fdab3e-2nmbn"] Oct 31 04:19:47.266056 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:47.266022 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dfba3a7-1ce8-41a3-9367-534e5916b92b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20fdab3e-2nmbn\" (UID: \"2dfba3a7-1ce8-41a3-9367-534e5916b92b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20fdab3e-2nmbn" Oct 31 04:19:47.266220 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:47.266069 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr8hj\" (UniqueName: \"kubernetes.io/projected/2dfba3a7-1ce8-41a3-9367-534e5916b92b-kube-api-access-rr8hj\") pod \"maas-api-revoke-sub-20fdab3e-2nmbn\" (UID: \"2dfba3a7-1ce8-41a3-9367-534e5916b92b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20fdab3e-2nmbn" Oct 31 04:19:47.266288 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:47.266263 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dfba3a7-1ce8-41a3-9367-534e5916b92b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20fdab3e-2nmbn\" (UID: \"2dfba3a7-1ce8-41a3-9367-534e5916b92b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20fdab3e-2nmbn" Oct 31 04:19:47.315892 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:47.315854 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8809a9e-jsfw5" event={"ID":"939530ef-60ca-4099-b12e-f5c922c564d7","Type":"ContainerDied","Data":"c64c2dc609f9d9683619ef208193ef3a87a8d9cd7fb4a5118e3719ab0fcc3578"} Oct 31 04:19:47.316312 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:47.315928 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c64c2dc609f9d9683619ef208193ef3a87a8d9cd7fb4a5118e3719ab0fcc3578" Oct 31 04:19:47.316312 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:47.315865 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8809a9e-jsfw5" Oct 31 04:19:47.367662 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:47.367624 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dfba3a7-1ce8-41a3-9367-534e5916b92b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20fdab3e-2nmbn\" (UID: \"2dfba3a7-1ce8-41a3-9367-534e5916b92b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20fdab3e-2nmbn" Oct 31 04:19:47.367841 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:47.367777 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dfba3a7-1ce8-41a3-9367-534e5916b92b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20fdab3e-2nmbn\" (UID: \"2dfba3a7-1ce8-41a3-9367-534e5916b92b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20fdab3e-2nmbn" Oct 31 04:19:47.367841 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:47.367823 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr8hj\" (UniqueName: \"kubernetes.io/projected/2dfba3a7-1ce8-41a3-9367-534e5916b92b-kube-api-access-rr8hj\") pod \"maas-api-revoke-sub-20fdab3e-2nmbn\" (UID: \"2dfba3a7-1ce8-41a3-9367-534e5916b92b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20fdab3e-2nmbn" Oct 31 04:19:47.368414 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:47.368385 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dfba3a7-1ce8-41a3-9367-534e5916b92b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20fdab3e-2nmbn\" (UID: \"2dfba3a7-1ce8-41a3-9367-534e5916b92b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20fdab3e-2nmbn" Oct 31 04:19:47.370920 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:47.370891 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dfba3a7-1ce8-41a3-9367-534e5916b92b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20fdab3e-2nmbn\" (UID: \"2dfba3a7-1ce8-41a3-9367-534e5916b92b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20fdab3e-2nmbn" Oct 31 04:19:47.394425 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:47.394315 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr8hj\" (UniqueName: \"kubernetes.io/projected/2dfba3a7-1ce8-41a3-9367-534e5916b92b-kube-api-access-rr8hj\") pod \"maas-api-revoke-sub-20fdab3e-2nmbn\" (UID: \"2dfba3a7-1ce8-41a3-9367-534e5916b92b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20fdab3e-2nmbn" Oct 31 04:19:47.459353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:47.458896 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20fdab3e-2nmbn" Oct 31 04:19:47.594494 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:47.594440 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eda0d787-wc2p8"] Oct 31 04:19:47.604758 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:47.603914 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda0d787-wc2p8" Oct 31 04:19:47.628081 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:47.627250 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2d5db3b-8540-4881-acf7-6316cccfada4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eda0d787-wc2p8\" (UID: \"e2d5db3b-8540-4881-acf7-6316cccfada4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda0d787-wc2p8" Oct 31 04:19:47.628081 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:47.627455 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2d5db3b-8540-4881-acf7-6316cccfada4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eda0d787-wc2p8\" (UID: \"e2d5db3b-8540-4881-acf7-6316cccfada4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda0d787-wc2p8" Oct 31 04:19:47.628081 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:47.627516 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjvnh\" (UniqueName: \"kubernetes.io/projected/e2d5db3b-8540-4881-acf7-6316cccfada4-kube-api-access-qjvnh\") pod \"maas-api-revoke-sub-eda0d787-wc2p8\" (UID: \"e2d5db3b-8540-4881-acf7-6316cccfada4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda0d787-wc2p8" Oct 31 04:19:47.628441 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:47.628288 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eda0d787-wc2p8"] Oct 31 04:19:47.757087 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:47.753194 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2d5db3b-8540-4881-acf7-6316cccfada4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eda0d787-wc2p8\" (UID: \"e2d5db3b-8540-4881-acf7-6316cccfada4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda0d787-wc2p8" Oct 31 04:19:47.757087 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:47.753284 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjvnh\" (UniqueName: \"kubernetes.io/projected/e2d5db3b-8540-4881-acf7-6316cccfada4-kube-api-access-qjvnh\") pod \"maas-api-revoke-sub-eda0d787-wc2p8\" (UID: \"e2d5db3b-8540-4881-acf7-6316cccfada4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda0d787-wc2p8" Oct 31 04:19:47.757087 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:47.753452 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2d5db3b-8540-4881-acf7-6316cccfada4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eda0d787-wc2p8\" (UID: \"e2d5db3b-8540-4881-acf7-6316cccfada4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda0d787-wc2p8" Oct 31 04:19:47.760494 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:47.757415 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2d5db3b-8540-4881-acf7-6316cccfada4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eda0d787-wc2p8\" (UID: \"e2d5db3b-8540-4881-acf7-6316cccfada4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda0d787-wc2p8" Oct 31 04:19:47.856785 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:47.854070 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2d5db3b-8540-4881-acf7-6316cccfada4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eda0d787-wc2p8\" (UID: \"e2d5db3b-8540-4881-acf7-6316cccfada4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda0d787-wc2p8" Oct 31 04:19:47.861560 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:47.861524 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjvnh\" (UniqueName: \"kubernetes.io/projected/e2d5db3b-8540-4881-acf7-6316cccfada4-kube-api-access-qjvnh\") pod \"maas-api-revoke-sub-eda0d787-wc2p8\" (UID: \"e2d5db3b-8540-4881-acf7-6316cccfada4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda0d787-wc2p8" Oct 31 04:19:47.950415 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:47.941575 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda0d787-wc2p8" Oct 31 04:19:48.240975 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:19:48.240928 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dfba3a7_1ce8_41a3_9367_534e5916b92b.slice/crio-4c7bcdad9ac338b39b84c210c8e532489861d0dd6e93e1f9709c28cb16cff3cd WatchSource:0}: Error finding container 4c7bcdad9ac338b39b84c210c8e532489861d0dd6e93e1f9709c28cb16cff3cd: Status 404 returned error can't find the container with id 4c7bcdad9ac338b39b84c210c8e532489861d0dd6e93e1f9709c28cb16cff3cd Oct 31 04:19:48.351596 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:48.351558 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20fdab3e-2nmbn"] Oct 31 04:19:48.414617 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:48.414520 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20fdab3e-2nmbn" event={"ID":"2dfba3a7-1ce8-41a3-9367-534e5916b92b","Type":"ContainerStarted","Data":"4c7bcdad9ac338b39b84c210c8e532489861d0dd6e93e1f9709c28cb16cff3cd"} Oct 31 04:19:49.044632 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:49.040766 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 31 04:19:49.312358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:49.306501 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:49.575359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:49.571229 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eda0d787-wc2p8"] Oct 31 04:19:49.729722 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:49.727607 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" podUID="dccd15a9-47f5-4bd1-b21b-9d5ca8b05845" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:49.819129 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:49.816830 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" podUID="8e2c7041-ad52-49f4-8b55-d16691f4938f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:50.099420 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:50.056237 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" podUID="643433d8-418e-4102-8bc3-cd544932614a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:50.099420 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:50.060009 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" podUID="b4ced3ab-41e4-4c54-93eb-95eddaae992d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:50.162357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:50.159474 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:50.162357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:50.161163 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" Oct 31 04:19:50.384650 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:50.374778 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:50.429464 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:50.379519 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:50.429464 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:50.415944 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" Oct 31 04:19:50.537139 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:50.537088 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20fdab3e-2nmbn" event={"ID":"2dfba3a7-1ce8-41a3-9367-534e5916b92b","Type":"ContainerStarted","Data":"6fbf4904b9dc01642a9c2697f046fa7b0e0c5dd72fbc01b5c66d557df0408561"} Oct 31 04:19:50.554386 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:50.546593 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" podUID="e0f4b8c2-14f4-4052-bf54-dd65bf59e413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:50.792356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:50.790572 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" podUID="dccd15a9-47f5-4bd1-b21b-9d5ca8b05845" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:50.830358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:50.804767 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" Oct 31 04:19:50.869149 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:50.835056 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" podUID="8e2c7041-ad52-49f4-8b55-d16691f4938f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:50.869149 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:50.858844 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" Oct 31 04:19:50.897839 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:50.892371 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20fdab3e-2nmbn" podStartSLOduration=3.892350575 podStartE2EDuration="3.892350575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:19:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:19:50.669590662 +0000 UTC m=+945.425928721" watchObservedRunningTime="2026-10-31 04:19:50.892350575 +0000 UTC m=+945.648688617" Oct 31 04:19:50.915872 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:50.915754 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:51.060605 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:51.058078 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" podUID="643433d8-418e-4102-8bc3-cd544932614a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 04:19:51.079384 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:51.067479 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" Oct 31 04:19:51.107538 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:51.107493 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:19:51.107873 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:51.107840 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c" podUID="dda0567c-8a01-47e8-95f9-c0a4ad08a1c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:51.121976 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:51.121926 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" podUID="b4ced3ab-41e4-4c54-93eb-95eddaae992d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:51.123028 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:51.123001 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" Oct 31 04:19:51.179488 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:51.179432 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:51.313472 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:51.245149 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" Oct 31 04:19:51.340358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:51.320813 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" Oct 31 04:19:51.415580 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:51.415541 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 31 04:19:51.437593 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:51.437546 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" Oct 31 04:19:51.571555 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:51.551097 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" podUID="e0f4b8c2-14f4-4052-bf54-dd65bf59e413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:51.599395 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:51.593762 5019 generic.go:356] "Generic (PLEG): container finished" podID="2dfba3a7-1ce8-41a3-9367-534e5916b92b" containerID="6fbf4904b9dc01642a9c2697f046fa7b0e0c5dd72fbc01b5c66d557df0408561" exitCode=0 Oct 31 04:19:51.735639 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:51.735593 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20fdab3e-2nmbn" event={"ID":"2dfba3a7-1ce8-41a3-9367-534e5916b92b","Type":"ContainerDied","Data":"6fbf4904b9dc01642a9c2697f046fa7b0e0c5dd72fbc01b5c66d557df0408561"} Oct 31 04:19:51.735935 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:51.735895 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" Oct 31 04:19:51.736026 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:51.735945 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda0d787-wc2p8" event={"ID":"e2d5db3b-8540-4881-acf7-6316cccfada4","Type":"ContainerStarted","Data":"2a68b36aa4da6ed9290358fda840c7c494ea38957655382498b732d4ab0bca89"} Oct 31 04:19:51.761980 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:51.761951 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" Oct 31 04:19:51.849414 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:51.804465 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" podUID="dccd15a9-47f5-4bd1-b21b-9d5ca8b05845" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 31 04:19:51.889504 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:51.852636 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" podUID="8e2c7041-ad52-49f4-8b55-d16691f4938f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:52.160432 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:52.132758 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" podUID="b4ced3ab-41e4-4c54-93eb-95eddaae992d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:52.192376 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:52.190847 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:52.431623 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:52.431512 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:19:52.460355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:52.439742 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" Oct 31 04:19:52.460355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:52.444415 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" Oct 31 04:19:52.460355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:52.449168 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" Oct 31 04:19:52.460355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:52.452170 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" Oct 31 04:19:52.465583 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:52.465554 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" Oct 31 04:19:52.513556 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:19:52.513175 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:19:52.560615 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:52.560580 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2b7955f-jz569"] Oct 31 04:19:52.565143 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:52.565113 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b7955f-jz569" Oct 31 04:19:52.588196 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:52.588159 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2b7955f-jz569"] Oct 31 04:19:52.614388 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:52.614299 5019 generic.go:356] "Generic (PLEG): container finished" podID="e2d5db3b-8540-4881-acf7-6316cccfada4" containerID="d4dee8a184f7fe3d398e015dc8f3b0e079f81c25c0802cb6b25c594fd546ba91" exitCode=0 Oct 31 04:19:52.614584 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:52.614417 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda0d787-wc2p8" event={"ID":"e2d5db3b-8540-4881-acf7-6316cccfada4","Type":"ContainerDied","Data":"d4dee8a184f7fe3d398e015dc8f3b0e079f81c25c0802cb6b25c594fd546ba91"} Oct 31 04:19:52.639311 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:52.639245 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1791edb-3151-4dde-9263-1d628e06b247-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2b7955f-jz569\" (UID: \"d1791edb-3151-4dde-9263-1d628e06b247\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b7955f-jz569" Oct 31 04:19:52.639591 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:52.639412 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1791edb-3151-4dde-9263-1d628e06b247-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2b7955f-jz569\" (UID: \"d1791edb-3151-4dde-9263-1d628e06b247\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b7955f-jz569" Oct 31 04:19:52.640375 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:52.639666 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ssfg\" (UniqueName: \"kubernetes.io/projected/d1791edb-3151-4dde-9263-1d628e06b247-kube-api-access-5ssfg\") pod \"maas-api-revoke-sub-f2b7955f-jz569\" (UID: \"d1791edb-3151-4dde-9263-1d628e06b247\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b7955f-jz569" Oct 31 04:19:52.740884 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:52.740571 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1791edb-3151-4dde-9263-1d628e06b247-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2b7955f-jz569\" (UID: \"d1791edb-3151-4dde-9263-1d628e06b247\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b7955f-jz569" Oct 31 04:19:52.740884 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:52.740648 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1791edb-3151-4dde-9263-1d628e06b247-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2b7955f-jz569\" (UID: \"d1791edb-3151-4dde-9263-1d628e06b247\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b7955f-jz569" Oct 31 04:19:52.740884 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:52.740802 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ssfg\" (UniqueName: \"kubernetes.io/projected/d1791edb-3151-4dde-9263-1d628e06b247-kube-api-access-5ssfg\") pod \"maas-api-revoke-sub-f2b7955f-jz569\" (UID: \"d1791edb-3151-4dde-9263-1d628e06b247\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b7955f-jz569" Oct 31 04:19:52.741493 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:52.741435 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1791edb-3151-4dde-9263-1d628e06b247-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2b7955f-jz569\" (UID: \"d1791edb-3151-4dde-9263-1d628e06b247\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b7955f-jz569" Oct 31 04:19:52.747134 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:52.747093 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1791edb-3151-4dde-9263-1d628e06b247-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2b7955f-jz569\" (UID: \"d1791edb-3151-4dde-9263-1d628e06b247\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b7955f-jz569" Oct 31 04:19:52.765044 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:52.765009 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ssfg\" (UniqueName: \"kubernetes.io/projected/d1791edb-3151-4dde-9263-1d628e06b247-kube-api-access-5ssfg\") pod \"maas-api-revoke-sub-f2b7955f-jz569\" (UID: \"d1791edb-3151-4dde-9263-1d628e06b247\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b7955f-jz569" Oct 31 04:19:52.864207 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:52.864175 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20fdab3e-2nmbn" Oct 31 04:19:52.889883 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:52.889842 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b7955f-jz569" Oct 31 04:19:52.943518 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:52.943467 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2dfba3a7-1ce8-41a3-9367-534e5916b92b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dfba3a7-1ce8-41a3-9367-534e5916b92b-maas-api-xks-ca\") pod \"2dfba3a7-1ce8-41a3-9367-534e5916b92b\" (UID: \"2dfba3a7-1ce8-41a3-9367-534e5916b92b\") " Oct 31 04:19:52.943746 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:52.943728 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dfba3a7-1ce8-41a3-9367-534e5916b92b-kube-api-access-rr8hj\" (UniqueName: \"kubernetes.io/projected/2dfba3a7-1ce8-41a3-9367-534e5916b92b-kube-api-access-rr8hj\") pod \"2dfba3a7-1ce8-41a3-9367-534e5916b92b\" (UID: \"2dfba3a7-1ce8-41a3-9367-534e5916b92b\") " Oct 31 04:19:52.943826 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:52.943757 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dfba3a7-1ce8-41a3-9367-534e5916b92b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dfba3a7-1ce8-41a3-9367-534e5916b92b-maas-api-openshift-ca\") pod \"2dfba3a7-1ce8-41a3-9367-534e5916b92b\" (UID: \"2dfba3a7-1ce8-41a3-9367-534e5916b92b\") " Oct 31 04:19:52.944395 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:52.944178 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dfba3a7-1ce8-41a3-9367-534e5916b92b-maas-api-openshift-ca" pod "2dfba3a7-1ce8-41a3-9367-534e5916b92b" (UID: "2dfba3a7-1ce8-41a3-9367-534e5916b92b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:19:52.944395 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:52.944363 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dfba3a7-1ce8-41a3-9367-534e5916b92b-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:19:52.945919 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:52.945891 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dfba3a7-1ce8-41a3-9367-534e5916b92b-kube-api-access-rr8hj" pod "2dfba3a7-1ce8-41a3-9367-534e5916b92b" (UID: "2dfba3a7-1ce8-41a3-9367-534e5916b92b"). InnerVolumeSpecName "kube-api-access-rr8hj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:19:52.946055 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:52.945891 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dfba3a7-1ce8-41a3-9367-534e5916b92b-maas-api-xks-ca" pod "2dfba3a7-1ce8-41a3-9367-534e5916b92b" (UID: "2dfba3a7-1ce8-41a3-9367-534e5916b92b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:19:53.045464 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:53.045424 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dfba3a7-1ce8-41a3-9367-534e5916b92b-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:19:53.045464 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:53.045461 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rr8hj\" (UniqueName: \"kubernetes.io/projected/2dfba3a7-1ce8-41a3-9367-534e5916b92b-kube-api-access-rr8hj\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:19:53.222858 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:19:53.222817 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1791edb_3151_4dde_9263_1d628e06b247.slice/crio-679cb3c1b19cbaf0d00d134c19304ccde382c02b0a3aebc01f11e0125db48a22 WatchSource:0}: Error finding container 679cb3c1b19cbaf0d00d134c19304ccde382c02b0a3aebc01f11e0125db48a22: Status 404 returned error can't find the container with id 679cb3c1b19cbaf0d00d134c19304ccde382c02b0a3aebc01f11e0125db48a22 Oct 31 04:19:53.223168 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:53.223141 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2b7955f-jz569"] Oct 31 04:19:53.624839 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:53.624805 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20fdab3e-2nmbn" Oct 31 04:19:53.624839 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:53.624828 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20fdab3e-2nmbn" event={"ID":"2dfba3a7-1ce8-41a3-9367-534e5916b92b","Type":"ContainerDied","Data":"4c7bcdad9ac338b39b84c210c8e532489861d0dd6e93e1f9709c28cb16cff3cd"} Oct 31 04:19:53.625066 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:53.624880 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c7bcdad9ac338b39b84c210c8e532489861d0dd6e93e1f9709c28cb16cff3cd" Oct 31 04:19:53.627747 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:53.627067 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b7955f-jz569" event={"ID":"d1791edb-3151-4dde-9263-1d628e06b247","Type":"ContainerStarted","Data":"0def4fcf709a929ff3ee22d0efeea73da37529e90303d0af9a98790591c0d0a6"} Oct 31 04:19:53.627747 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:53.627129 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b7955f-jz569" event={"ID":"d1791edb-3151-4dde-9263-1d628e06b247","Type":"ContainerStarted","Data":"679cb3c1b19cbaf0d00d134c19304ccde382c02b0a3aebc01f11e0125db48a22"} Oct 31 04:19:53.671910 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:53.671853 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b7955f-jz569" podStartSLOduration=1.67183358 podStartE2EDuration="1.67183358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:19:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:19:53.666612614 +0000 UTC m=+948.422950656" watchObservedRunningTime="2026-10-31 04:19:53.67183358 +0000 UTC m=+948.428171620" Oct 31 04:19:53.819017 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:53.818977 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda0d787-wc2p8" Oct 31 04:19:53.855895 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:53.855849 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2d5db3b-8540-4881-acf7-6316cccfada4-kube-api-access-qjvnh\" (UniqueName: \"kubernetes.io/projected/e2d5db3b-8540-4881-acf7-6316cccfada4-kube-api-access-qjvnh\") pod \"e2d5db3b-8540-4881-acf7-6316cccfada4\" (UID: \"e2d5db3b-8540-4881-acf7-6316cccfada4\") " Oct 31 04:19:53.856107 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:53.856046 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2d5db3b-8540-4881-acf7-6316cccfada4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2d5db3b-8540-4881-acf7-6316cccfada4-maas-api-xks-ca\") pod \"e2d5db3b-8540-4881-acf7-6316cccfada4\" (UID: \"e2d5db3b-8540-4881-acf7-6316cccfada4\") " Oct 31 04:19:53.856224 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:53.856203 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2d5db3b-8540-4881-acf7-6316cccfada4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2d5db3b-8540-4881-acf7-6316cccfada4-maas-api-openshift-ca\") pod \"e2d5db3b-8540-4881-acf7-6316cccfada4\" (UID: \"e2d5db3b-8540-4881-acf7-6316cccfada4\") " Oct 31 04:19:53.864550 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:53.864499 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2d5db3b-8540-4881-acf7-6316cccfada4-kube-api-access-qjvnh" pod "e2d5db3b-8540-4881-acf7-6316cccfada4" (UID: "e2d5db3b-8540-4881-acf7-6316cccfada4"). InnerVolumeSpecName "kube-api-access-qjvnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:19:53.865064 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:53.864614 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2d5db3b-8540-4881-acf7-6316cccfada4-maas-api-openshift-ca" pod "e2d5db3b-8540-4881-acf7-6316cccfada4" (UID: "e2d5db3b-8540-4881-acf7-6316cccfada4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:19:53.866375 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:53.866291 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2d5db3b-8540-4881-acf7-6316cccfada4-maas-api-xks-ca" pod "e2d5db3b-8540-4881-acf7-6316cccfada4" (UID: "e2d5db3b-8540-4881-acf7-6316cccfada4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:19:53.958030 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:53.957980 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2d5db3b-8540-4881-acf7-6316cccfada4-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:19:53.958030 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:53.958018 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2d5db3b-8540-4881-acf7-6316cccfada4-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:19:53.958030 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:53.958029 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjvnh\" (UniqueName: \"kubernetes.io/projected/e2d5db3b-8540-4881-acf7-6316cccfada4-kube-api-access-qjvnh\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:19:54.636878 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:54.636836 5019 generic.go:356] "Generic (PLEG): container finished" podID="d1791edb-3151-4dde-9263-1d628e06b247" containerID="0def4fcf709a929ff3ee22d0efeea73da37529e90303d0af9a98790591c0d0a6" exitCode=0 Oct 31 04:19:54.637036 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:54.636929 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b7955f-jz569" event={"ID":"d1791edb-3151-4dde-9263-1d628e06b247","Type":"ContainerDied","Data":"0def4fcf709a929ff3ee22d0efeea73da37529e90303d0af9a98790591c0d0a6"} Oct 31 04:19:54.638571 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:54.638540 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda0d787-wc2p8" event={"ID":"e2d5db3b-8540-4881-acf7-6316cccfada4","Type":"ContainerDied","Data":"2a68b36aa4da6ed9290358fda840c7c494ea38957655382498b732d4ab0bca89"} Oct 31 04:19:54.638694 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:54.638589 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a68b36aa4da6ed9290358fda840c7c494ea38957655382498b732d4ab0bca89" Oct 31 04:19:54.638694 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:54.638548 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda0d787-wc2p8" Oct 31 04:19:55.835969 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:55.835933 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b7955f-jz569" Oct 31 04:19:55.884427 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:55.884380 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1791edb-3151-4dde-9263-1d628e06b247-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1791edb-3151-4dde-9263-1d628e06b247-maas-api-xks-ca\") pod \"d1791edb-3151-4dde-9263-1d628e06b247\" (UID: \"d1791edb-3151-4dde-9263-1d628e06b247\") " Oct 31 04:19:55.884604 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:55.884494 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1791edb-3151-4dde-9263-1d628e06b247-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1791edb-3151-4dde-9263-1d628e06b247-maas-api-openshift-ca\") pod \"d1791edb-3151-4dde-9263-1d628e06b247\" (UID: \"d1791edb-3151-4dde-9263-1d628e06b247\") " Oct 31 04:19:55.884681 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:55.884615 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1791edb-3151-4dde-9263-1d628e06b247-kube-api-access-5ssfg\" (UniqueName: \"kubernetes.io/projected/d1791edb-3151-4dde-9263-1d628e06b247-kube-api-access-5ssfg\") pod \"d1791edb-3151-4dde-9263-1d628e06b247\" (UID: \"d1791edb-3151-4dde-9263-1d628e06b247\") " Oct 31 04:19:55.885520 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:55.885479 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1791edb-3151-4dde-9263-1d628e06b247-maas-api-openshift-ca" pod "d1791edb-3151-4dde-9263-1d628e06b247" (UID: "d1791edb-3151-4dde-9263-1d628e06b247"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:19:55.887036 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:55.886998 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1791edb-3151-4dde-9263-1d628e06b247-kube-api-access-5ssfg" pod "d1791edb-3151-4dde-9263-1d628e06b247" (UID: "d1791edb-3151-4dde-9263-1d628e06b247"). InnerVolumeSpecName "kube-api-access-5ssfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:19:55.887498 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:55.887472 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1791edb-3151-4dde-9263-1d628e06b247-maas-api-xks-ca" pod "d1791edb-3151-4dde-9263-1d628e06b247" (UID: "d1791edb-3151-4dde-9263-1d628e06b247"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:19:55.986214 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:55.986171 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1791edb-3151-4dde-9263-1d628e06b247-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:19:55.986214 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:55.986211 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1791edb-3151-4dde-9263-1d628e06b247-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:19:55.986433 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:55.986226 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ssfg\" (UniqueName: \"kubernetes.io/projected/d1791edb-3151-4dde-9263-1d628e06b247-kube-api-access-5ssfg\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:19:56.013487 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:56.013437 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85ff0b4b-rmrth"] Oct 31 04:19:56.014139 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:56.014122 5019 cpu_manager.go:414] "containerMap: removing container" podUID="d1791edb-3151-4dde-9263-1d628e06b247" containerName="revoke-keys" Oct 31 04:19:56.014139 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:56.014139 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1791edb-3151-4dde-9263-1d628e06b247" containerName="revoke-keys" Oct 31 04:19:56.014281 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:56.014198 5019 cpu_manager.go:414] "containerMap: removing container" podUID="e2d5db3b-8540-4881-acf7-6316cccfada4" containerName="revoke-keys" Oct 31 04:19:56.014281 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:56.014204 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2d5db3b-8540-4881-acf7-6316cccfada4" containerName="revoke-keys" Oct 31 04:19:56.014281 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:56.014226 5019 cpu_manager.go:414] "containerMap: removing container" podUID="2dfba3a7-1ce8-41a3-9367-534e5916b92b" containerName="revoke-keys" Oct 31 04:19:56.014281 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:56.014231 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dfba3a7-1ce8-41a3-9367-534e5916b92b" containerName="revoke-keys" Oct 31 04:19:56.014461 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:56.014315 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dfba3a7-1ce8-41a3-9367-534e5916b92b" containerName="revoke-keys" Oct 31 04:19:56.014461 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:56.014348 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2d5db3b-8540-4881-acf7-6316cccfada4" containerName="revoke-keys" Oct 31 04:19:56.014461 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:56.014360 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1791edb-3151-4dde-9263-1d628e06b247" containerName="revoke-keys" Oct 31 04:19:56.018392 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:56.018369 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85ff0b4b-rmrth" Oct 31 04:19:56.038006 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:56.037973 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85ff0b4b-rmrth"] Oct 31 04:19:56.087706 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:56.087662 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c81aa0f9-82a9-41db-b185-b6fc44b774a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85ff0b4b-rmrth\" (UID: \"c81aa0f9-82a9-41db-b185-b6fc44b774a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85ff0b4b-rmrth" Oct 31 04:19:56.087886 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:56.087844 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxw49\" (UniqueName: \"kubernetes.io/projected/c81aa0f9-82a9-41db-b185-b6fc44b774a4-kube-api-access-gxw49\") pod \"maas-api-revoke-sub-85ff0b4b-rmrth\" (UID: \"c81aa0f9-82a9-41db-b185-b6fc44b774a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85ff0b4b-rmrth" Oct 31 04:19:56.087997 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:56.087974 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c81aa0f9-82a9-41db-b185-b6fc44b774a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85ff0b4b-rmrth\" (UID: \"c81aa0f9-82a9-41db-b185-b6fc44b774a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85ff0b4b-rmrth" Oct 31 04:19:56.188861 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:56.188823 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c81aa0f9-82a9-41db-b185-b6fc44b774a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85ff0b4b-rmrth\" (UID: \"c81aa0f9-82a9-41db-b185-b6fc44b774a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85ff0b4b-rmrth" Oct 31 04:19:56.189045 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:56.188942 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxw49\" (UniqueName: \"kubernetes.io/projected/c81aa0f9-82a9-41db-b185-b6fc44b774a4-kube-api-access-gxw49\") pod \"maas-api-revoke-sub-85ff0b4b-rmrth\" (UID: \"c81aa0f9-82a9-41db-b185-b6fc44b774a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85ff0b4b-rmrth" Oct 31 04:19:56.189045 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:56.188999 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c81aa0f9-82a9-41db-b185-b6fc44b774a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85ff0b4b-rmrth\" (UID: \"c81aa0f9-82a9-41db-b185-b6fc44b774a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85ff0b4b-rmrth" Oct 31 04:19:56.189622 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:56.189593 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c81aa0f9-82a9-41db-b185-b6fc44b774a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85ff0b4b-rmrth\" (UID: \"c81aa0f9-82a9-41db-b185-b6fc44b774a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85ff0b4b-rmrth" Oct 31 04:19:56.191617 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:56.191588 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c81aa0f9-82a9-41db-b185-b6fc44b774a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85ff0b4b-rmrth\" (UID: \"c81aa0f9-82a9-41db-b185-b6fc44b774a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85ff0b4b-rmrth" Oct 31 04:19:56.212799 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:56.212769 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxw49\" (UniqueName: \"kubernetes.io/projected/c81aa0f9-82a9-41db-b185-b6fc44b774a4-kube-api-access-gxw49\") pod \"maas-api-revoke-sub-85ff0b4b-rmrth\" (UID: \"c81aa0f9-82a9-41db-b185-b6fc44b774a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85ff0b4b-rmrth" Oct 31 04:19:56.340769 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:56.340678 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85ff0b4b-rmrth" Oct 31 04:19:56.654149 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:19:56.654116 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc81aa0f9_82a9_41db_b185_b6fc44b774a4.slice/crio-524621b8ecfcad5250fada7255de516967b8f132b99e84fecb3b3ae43c9f7baa WatchSource:0}: Error finding container 524621b8ecfcad5250fada7255de516967b8f132b99e84fecb3b3ae43c9f7baa: Status 404 returned error can't find the container with id 524621b8ecfcad5250fada7255de516967b8f132b99e84fecb3b3ae43c9f7baa Oct 31 04:19:56.655170 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:56.655137 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85ff0b4b-rmrth"] Oct 31 04:19:56.662777 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:56.662752 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b7955f-jz569" Oct 31 04:19:56.662875 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:56.662781 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b7955f-jz569" event={"ID":"d1791edb-3151-4dde-9263-1d628e06b247","Type":"ContainerDied","Data":"679cb3c1b19cbaf0d00d134c19304ccde382c02b0a3aebc01f11e0125db48a22"} Oct 31 04:19:56.662875 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:56.662834 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="679cb3c1b19cbaf0d00d134c19304ccde382c02b0a3aebc01f11e0125db48a22" Oct 31 04:19:57.673474 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:57.673437 5019 generic.go:356] "Generic (PLEG): container finished" podID="c81aa0f9-82a9-41db-b185-b6fc44b774a4" containerID="037a7cfbbcf3a5dc261e05440707690234569c71d53d314fa4907fc72a786840" exitCode=0 Oct 31 04:19:57.673856 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:57.673513 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85ff0b4b-rmrth" event={"ID":"c81aa0f9-82a9-41db-b185-b6fc44b774a4","Type":"ContainerDied","Data":"037a7cfbbcf3a5dc261e05440707690234569c71d53d314fa4907fc72a786840"} Oct 31 04:19:57.673856 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:57.673567 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85ff0b4b-rmrth" event={"ID":"c81aa0f9-82a9-41db-b185-b6fc44b774a4","Type":"ContainerStarted","Data":"524621b8ecfcad5250fada7255de516967b8f132b99e84fecb3b3ae43c9f7baa"} Oct 31 04:19:58.846875 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:58.846843 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85ff0b4b-rmrth" Oct 31 04:19:58.929411 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:58.929361 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c81aa0f9-82a9-41db-b185-b6fc44b774a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c81aa0f9-82a9-41db-b185-b6fc44b774a4-maas-api-xks-ca\") pod \"c81aa0f9-82a9-41db-b185-b6fc44b774a4\" (UID: \"c81aa0f9-82a9-41db-b185-b6fc44b774a4\") " Oct 31 04:19:58.929613 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:58.929509 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c81aa0f9-82a9-41db-b185-b6fc44b774a4-kube-api-access-gxw49\" (UniqueName: \"kubernetes.io/projected/c81aa0f9-82a9-41db-b185-b6fc44b774a4-kube-api-access-gxw49\") pod \"c81aa0f9-82a9-41db-b185-b6fc44b774a4\" (UID: \"c81aa0f9-82a9-41db-b185-b6fc44b774a4\") " Oct 31 04:19:58.929687 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:58.929667 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c81aa0f9-82a9-41db-b185-b6fc44b774a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c81aa0f9-82a9-41db-b185-b6fc44b774a4-maas-api-openshift-ca\") pod \"c81aa0f9-82a9-41db-b185-b6fc44b774a4\" (UID: \"c81aa0f9-82a9-41db-b185-b6fc44b774a4\") " Oct 31 04:19:58.930117 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:58.930084 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c81aa0f9-82a9-41db-b185-b6fc44b774a4-maas-api-openshift-ca" pod "c81aa0f9-82a9-41db-b185-b6fc44b774a4" (UID: "c81aa0f9-82a9-41db-b185-b6fc44b774a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:19:58.930349 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:58.930313 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c81aa0f9-82a9-41db-b185-b6fc44b774a4-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:19:58.931812 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:58.931774 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c81aa0f9-82a9-41db-b185-b6fc44b774a4-kube-api-access-gxw49" pod "c81aa0f9-82a9-41db-b185-b6fc44b774a4" (UID: "c81aa0f9-82a9-41db-b185-b6fc44b774a4"). InnerVolumeSpecName "kube-api-access-gxw49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:19:58.932167 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:58.932132 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c81aa0f9-82a9-41db-b185-b6fc44b774a4-maas-api-xks-ca" pod "c81aa0f9-82a9-41db-b185-b6fc44b774a4" (UID: "c81aa0f9-82a9-41db-b185-b6fc44b774a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:19:59.031940 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:59.031901 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c81aa0f9-82a9-41db-b185-b6fc44b774a4-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:19:59.031940 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:59.031941 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxw49\" (UniqueName: \"kubernetes.io/projected/c81aa0f9-82a9-41db-b185-b6fc44b774a4-kube-api-access-gxw49\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:19:59.546405 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:59.546371 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr"] Oct 31 04:19:59.547073 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:59.547053 5019 cpu_manager.go:414] "containerMap: removing container" podUID="c81aa0f9-82a9-41db-b185-b6fc44b774a4" containerName="revoke-keys" Oct 31 04:19:59.547205 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:59.547189 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c81aa0f9-82a9-41db-b185-b6fc44b774a4" containerName="revoke-keys" Oct 31 04:19:59.547402 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:59.547389 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="c81aa0f9-82a9-41db-b185-b6fc44b774a4" containerName="revoke-keys" Oct 31 04:19:59.551431 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:59.551408 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr" Oct 31 04:19:59.567230 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:59.567194 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-146320d5-openshift-default-dockercfg-hgf5f\"" Oct 31 04:19:59.588894 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:59.588857 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr"] Oct 31 04:19:59.640405 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:59.640363 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59824c73-334b-446b-8d63-bde791f5b216-istiod-ca-cert\") pod \"e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr\" (UID: \"59824c73-334b-446b-8d63-bde791f5b216\") " pod="openshift-ingress/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr" Oct 31 04:19:59.640740 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:59.640712 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59824c73-334b-446b-8d63-bde791f5b216-workload-socket\") pod \"e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr\" (UID: \"59824c73-334b-446b-8d63-bde791f5b216\") " pod="openshift-ingress/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr" Oct 31 04:19:59.640908 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:59.640752 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59824c73-334b-446b-8d63-bde791f5b216-istio-podinfo\") pod \"e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr\" (UID: \"59824c73-334b-446b-8d63-bde791f5b216\") " pod="openshift-ingress/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr" Oct 31 04:19:59.640908 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:59.640803 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59824c73-334b-446b-8d63-bde791f5b216-istio-data\") pod \"e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr\" (UID: \"59824c73-334b-446b-8d63-bde791f5b216\") " pod="openshift-ingress/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr" Oct 31 04:19:59.640908 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:59.640841 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59824c73-334b-446b-8d63-bde791f5b216-credential-socket\") pod \"e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr\" (UID: \"59824c73-334b-446b-8d63-bde791f5b216\") " pod="openshift-ingress/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr" Oct 31 04:19:59.640908 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:59.640881 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59824c73-334b-446b-8d63-bde791f5b216-workload-certs\") pod \"e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr\" (UID: \"59824c73-334b-446b-8d63-bde791f5b216\") " pod="openshift-ingress/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr" Oct 31 04:19:59.640908 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:59.640902 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59824c73-334b-446b-8d63-bde791f5b216-istio-token\") pod \"e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr\" (UID: \"59824c73-334b-446b-8d63-bde791f5b216\") " pod="openshift-ingress/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr" Oct 31 04:19:59.641216 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:59.640983 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59824c73-334b-446b-8d63-bde791f5b216-istio-envoy\") pod \"e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr\" (UID: \"59824c73-334b-446b-8d63-bde791f5b216\") " pod="openshift-ingress/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr" Oct 31 04:19:59.641216 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:59.641034 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjpzm\" (UniqueName: \"kubernetes.io/projected/59824c73-334b-446b-8d63-bde791f5b216-kube-api-access-jjpzm\") pod \"e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr\" (UID: \"59824c73-334b-446b-8d63-bde791f5b216\") " pod="openshift-ingress/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr" Oct 31 04:19:59.740754 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:59.740712 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85ff0b4b-rmrth" event={"ID":"c81aa0f9-82a9-41db-b185-b6fc44b774a4","Type":"ContainerDied","Data":"524621b8ecfcad5250fada7255de516967b8f132b99e84fecb3b3ae43c9f7baa"} Oct 31 04:19:59.741175 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:59.741150 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="524621b8ecfcad5250fada7255de516967b8f132b99e84fecb3b3ae43c9f7baa" Oct 31 04:19:59.741428 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:59.741412 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85ff0b4b-rmrth" Oct 31 04:19:59.745482 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:59.744856 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59824c73-334b-446b-8d63-bde791f5b216-istiod-ca-cert\") pod \"e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr\" (UID: \"59824c73-334b-446b-8d63-bde791f5b216\") " pod="openshift-ingress/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr" Oct 31 04:19:59.746764 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:59.746733 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59824c73-334b-446b-8d63-bde791f5b216-workload-socket\") pod \"e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr\" (UID: \"59824c73-334b-446b-8d63-bde791f5b216\") " pod="openshift-ingress/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr" Oct 31 04:19:59.746948 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:59.746930 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59824c73-334b-446b-8d63-bde791f5b216-istio-podinfo\") pod \"e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr\" (UID: \"59824c73-334b-446b-8d63-bde791f5b216\") " pod="openshift-ingress/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr" Oct 31 04:19:59.747135 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:59.747115 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59824c73-334b-446b-8d63-bde791f5b216-istio-data\") pod \"e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr\" (UID: \"59824c73-334b-446b-8d63-bde791f5b216\") " pod="openshift-ingress/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr" Oct 31 04:19:59.747277 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:59.747261 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59824c73-334b-446b-8d63-bde791f5b216-credential-socket\") pod \"e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr\" (UID: \"59824c73-334b-446b-8d63-bde791f5b216\") " pod="openshift-ingress/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr" Oct 31 04:19:59.747739 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:59.745724 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59824c73-334b-446b-8d63-bde791f5b216-istiod-ca-cert\") pod \"e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr\" (UID: \"59824c73-334b-446b-8d63-bde791f5b216\") " pod="openshift-ingress/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr" Oct 31 04:19:59.747944 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:59.747929 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59824c73-334b-446b-8d63-bde791f5b216-workload-certs\") pod \"e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr\" (UID: \"59824c73-334b-446b-8d63-bde791f5b216\") " pod="openshift-ingress/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr" Oct 31 04:19:59.748074 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:59.748056 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59824c73-334b-446b-8d63-bde791f5b216-istio-token\") pod \"e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr\" (UID: \"59824c73-334b-446b-8d63-bde791f5b216\") " pod="openshift-ingress/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr" Oct 31 04:19:59.748649 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:59.748601 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59824c73-334b-446b-8d63-bde791f5b216-istio-envoy\") pod \"e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr\" (UID: \"59824c73-334b-446b-8d63-bde791f5b216\") " pod="openshift-ingress/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr" Oct 31 04:19:59.748870 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:59.748849 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjpzm\" (UniqueName: \"kubernetes.io/projected/59824c73-334b-446b-8d63-bde791f5b216-kube-api-access-jjpzm\") pod \"e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr\" (UID: \"59824c73-334b-446b-8d63-bde791f5b216\") " pod="openshift-ingress/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr" Oct 31 04:19:59.751429 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:59.750770 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59824c73-334b-446b-8d63-bde791f5b216-workload-socket\") pod \"e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr\" (UID: \"59824c73-334b-446b-8d63-bde791f5b216\") " pod="openshift-ingress/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr" Oct 31 04:19:59.751429 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:59.750842 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59824c73-334b-446b-8d63-bde791f5b216-istio-data\") pod \"e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr\" (UID: \"59824c73-334b-446b-8d63-bde791f5b216\") " pod="openshift-ingress/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr" Oct 31 04:19:59.752372 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:59.752183 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59824c73-334b-446b-8d63-bde791f5b216-workload-certs\") pod \"e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr\" (UID: \"59824c73-334b-446b-8d63-bde791f5b216\") " pod="openshift-ingress/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr" Oct 31 04:19:59.752977 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:59.752952 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59824c73-334b-446b-8d63-bde791f5b216-credential-socket\") pod \"e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr\" (UID: \"59824c73-334b-446b-8d63-bde791f5b216\") " pod="openshift-ingress/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr" Oct 31 04:19:59.761942 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:59.760447 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59824c73-334b-446b-8d63-bde791f5b216-istio-envoy\") pod \"e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr\" (UID: \"59824c73-334b-446b-8d63-bde791f5b216\") " pod="openshift-ingress/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr" Oct 31 04:19:59.763758 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:59.763726 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59824c73-334b-446b-8d63-bde791f5b216-istio-podinfo\") pod \"e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr\" (UID: \"59824c73-334b-446b-8d63-bde791f5b216\") " pod="openshift-ingress/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr" Oct 31 04:19:59.788508 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:59.788161 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59824c73-334b-446b-8d63-bde791f5b216-istio-token\") pod \"e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr\" (UID: \"59824c73-334b-446b-8d63-bde791f5b216\") " pod="openshift-ingress/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr" Oct 31 04:19:59.791496 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:59.791466 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjpzm\" (UniqueName: \"kubernetes.io/projected/59824c73-334b-446b-8d63-bde791f5b216-kube-api-access-jjpzm\") pod \"e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr\" (UID: \"59824c73-334b-446b-8d63-bde791f5b216\") " pod="openshift-ingress/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr" Oct 31 04:19:59.866201 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:19:59.866161 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr" Oct 31 04:20:00.214654 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:00.214620 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr"] Oct 31 04:20:00.216720 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:20:00.216681 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59824c73_334b_446b_8d63_bde791f5b216.slice/crio-df60cd55452824403a1f64d32fca59a0696b7cbf1ac856a4a8473218b613bbae WatchSource:0}: Error finding container df60cd55452824403a1f64d32fca59a0696b7cbf1ac856a4a8473218b613bbae: Status 404 returned error can't find the container with id df60cd55452824403a1f64d32fca59a0696b7cbf1ac856a4a8473218b613bbae Oct 31 04:20:00.219718 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:00.219679 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:20:00.219828 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:00.219758 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:20:00.219828 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:00.219793 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:20:00.756711 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:00.756669 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr" event={"ID":"59824c73-334b-446b-8d63-bde791f5b216","Type":"ContainerStarted","Data":"e6f0da0c29aaf18aa4bb1379e308be5f3d0f10dceedf00cd8dd7c5a4d767dba4"} Oct 31 04:20:00.757027 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:00.756728 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr" event={"ID":"59824c73-334b-446b-8d63-bde791f5b216","Type":"ContainerStarted","Data":"df60cd55452824403a1f64d32fca59a0696b7cbf1ac856a4a8473218b613bbae"} Oct 31 04:20:00.813858 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:00.813793 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr" podStartSLOduration=1.813771847 podStartE2EDuration="1.813771847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:19:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:20:00.812093632 +0000 UTC m=+955.568431677" watchObservedRunningTime="2026-10-31 04:20:00.813771847 +0000 UTC m=+955.570109888" Oct 31 04:20:00.867345 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:00.867293 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr" Oct 31 04:20:01.872435 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:01.872402 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr" Oct 31 04:20:01.912763 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:01.912728 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e173eac-7j8xh"] Oct 31 04:20:01.918565 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:01.918536 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e173eac-7j8xh" Oct 31 04:20:01.944495 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:01.944461 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:20:01.950266 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:01.950224 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:20:01.950472 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:01.950308 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:20:01.950472 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:01.950461 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:20:01.951743 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:01.951716 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e173eac-7j8xh"] Oct 31 04:20:01.986367 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:01.981870 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4fcefcb-a397-4a74-800d-efd33730069d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e173eac-7j8xh\" (UID: \"e4fcefcb-a397-4a74-800d-efd33730069d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e173eac-7j8xh" Oct 31 04:20:01.986367 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:01.981937 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnclk\" (UniqueName: \"kubernetes.io/projected/e4fcefcb-a397-4a74-800d-efd33730069d-kube-api-access-nnclk\") pod \"maas-api-revoke-sub-0e173eac-7j8xh\" (UID: \"e4fcefcb-a397-4a74-800d-efd33730069d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e173eac-7j8xh" Oct 31 04:20:01.986367 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:01.982113 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4fcefcb-a397-4a74-800d-efd33730069d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e173eac-7j8xh\" (UID: \"e4fcefcb-a397-4a74-800d-efd33730069d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e173eac-7j8xh" Oct 31 04:20:02.083683 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:02.083629 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4fcefcb-a397-4a74-800d-efd33730069d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e173eac-7j8xh\" (UID: \"e4fcefcb-a397-4a74-800d-efd33730069d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e173eac-7j8xh" Oct 31 04:20:02.083962 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:02.083835 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4fcefcb-a397-4a74-800d-efd33730069d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e173eac-7j8xh\" (UID: \"e4fcefcb-a397-4a74-800d-efd33730069d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e173eac-7j8xh" Oct 31 04:20:02.083962 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:02.083877 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnclk\" (UniqueName: \"kubernetes.io/projected/e4fcefcb-a397-4a74-800d-efd33730069d-kube-api-access-nnclk\") pod \"maas-api-revoke-sub-0e173eac-7j8xh\" (UID: \"e4fcefcb-a397-4a74-800d-efd33730069d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e173eac-7j8xh" Oct 31 04:20:02.084582 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:02.084555 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4fcefcb-a397-4a74-800d-efd33730069d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e173eac-7j8xh\" (UID: \"e4fcefcb-a397-4a74-800d-efd33730069d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e173eac-7j8xh" Oct 31 04:20:02.086716 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:02.086688 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4fcefcb-a397-4a74-800d-efd33730069d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e173eac-7j8xh\" (UID: \"e4fcefcb-a397-4a74-800d-efd33730069d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e173eac-7j8xh" Oct 31 04:20:02.107134 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:02.107093 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnclk\" (UniqueName: \"kubernetes.io/projected/e4fcefcb-a397-4a74-800d-efd33730069d-kube-api-access-nnclk\") pod \"maas-api-revoke-sub-0e173eac-7j8xh\" (UID: \"e4fcefcb-a397-4a74-800d-efd33730069d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e173eac-7j8xh" Oct 31 04:20:02.233809 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:02.233769 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e173eac-7j8xh" Oct 31 04:20:02.545966 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:20:02.545928 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4fcefcb_a397_4a74_800d_efd33730069d.slice/crio-525ac7cfdb47c86b983cab6469040c80d1e7b6962721e2f4edd3afd1db904276 WatchSource:0}: Error finding container 525ac7cfdb47c86b983cab6469040c80d1e7b6962721e2f4edd3afd1db904276: Status 404 returned error can't find the container with id 525ac7cfdb47c86b983cab6469040c80d1e7b6962721e2f4edd3afd1db904276 Oct 31 04:20:02.546741 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:02.546713 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e173eac-7j8xh"] Oct 31 04:20:02.779853 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:02.779813 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e173eac-7j8xh" event={"ID":"e4fcefcb-a397-4a74-800d-efd33730069d","Type":"ContainerStarted","Data":"525ac7cfdb47c86b983cab6469040c80d1e7b6962721e2f4edd3afd1db904276"} Oct 31 04:20:02.780202 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:02.780166 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr" Oct 31 04:20:02.781297 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:02.781276 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr" Oct 31 04:20:03.433810 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:03.433769 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30a74489-ncq47"] Oct 31 04:20:03.438996 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:03.438963 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30a74489-ncq47" Oct 31 04:20:03.459079 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:03.459033 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30a74489-ncq47"] Oct 31 04:20:03.503586 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:03.503536 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83f22018-3412-4765-bce4-f273cdc41aa4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30a74489-ncq47\" (UID: \"83f22018-3412-4765-bce4-f273cdc41aa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30a74489-ncq47" Oct 31 04:20:03.503870 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:03.503841 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmdps\" (UniqueName: \"kubernetes.io/projected/83f22018-3412-4765-bce4-f273cdc41aa4-kube-api-access-vmdps\") pod \"maas-api-revoke-sub-30a74489-ncq47\" (UID: \"83f22018-3412-4765-bce4-f273cdc41aa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30a74489-ncq47" Oct 31 04:20:03.504308 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:03.504280 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83f22018-3412-4765-bce4-f273cdc41aa4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30a74489-ncq47\" (UID: \"83f22018-3412-4765-bce4-f273cdc41aa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30a74489-ncq47" Oct 31 04:20:03.605991 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:03.605949 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmdps\" (UniqueName: \"kubernetes.io/projected/83f22018-3412-4765-bce4-f273cdc41aa4-kube-api-access-vmdps\") pod \"maas-api-revoke-sub-30a74489-ncq47\" (UID: \"83f22018-3412-4765-bce4-f273cdc41aa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30a74489-ncq47" Oct 31 04:20:03.606228 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:03.606003 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83f22018-3412-4765-bce4-f273cdc41aa4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30a74489-ncq47\" (UID: \"83f22018-3412-4765-bce4-f273cdc41aa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30a74489-ncq47" Oct 31 04:20:03.606228 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:03.606099 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83f22018-3412-4765-bce4-f273cdc41aa4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30a74489-ncq47\" (UID: \"83f22018-3412-4765-bce4-f273cdc41aa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30a74489-ncq47" Oct 31 04:20:03.606888 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:03.606859 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83f22018-3412-4765-bce4-f273cdc41aa4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30a74489-ncq47\" (UID: \"83f22018-3412-4765-bce4-f273cdc41aa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30a74489-ncq47" Oct 31 04:20:03.608861 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:03.608828 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83f22018-3412-4765-bce4-f273cdc41aa4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30a74489-ncq47\" (UID: \"83f22018-3412-4765-bce4-f273cdc41aa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30a74489-ncq47" Oct 31 04:20:03.629485 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:03.629447 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmdps\" (UniqueName: \"kubernetes.io/projected/83f22018-3412-4765-bce4-f273cdc41aa4-kube-api-access-vmdps\") pod \"maas-api-revoke-sub-30a74489-ncq47\" (UID: \"83f22018-3412-4765-bce4-f273cdc41aa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30a74489-ncq47" Oct 31 04:20:03.757502 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:03.757460 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30a74489-ncq47" Oct 31 04:20:03.793186 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:03.793147 5019 generic.go:356] "Generic (PLEG): container finished" podID="e4fcefcb-a397-4a74-800d-efd33730069d" containerID="312fb96716141c036e0c4ff584983c5153445a44b3c6605da344f3cd03a1ce58" exitCode=0 Oct 31 04:20:03.793432 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:03.793247 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e173eac-7j8xh" event={"ID":"e4fcefcb-a397-4a74-800d-efd33730069d","Type":"ContainerDied","Data":"312fb96716141c036e0c4ff584983c5153445a44b3c6605da344f3cd03a1ce58"} Oct 31 04:20:04.069185 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:20:04.069151 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83f22018_3412_4765_bce4_f273cdc41aa4.slice/crio-3bc88fc4889ed6ea457bfc720649bd9ca2637ef6b1e45c791db52b45ddf76985 WatchSource:0}: Error finding container 3bc88fc4889ed6ea457bfc720649bd9ca2637ef6b1e45c791db52b45ddf76985: Status 404 returned error can't find the container with id 3bc88fc4889ed6ea457bfc720649bd9ca2637ef6b1e45c791db52b45ddf76985 Oct 31 04:20:04.069382 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:04.069248 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30a74489-ncq47"] Oct 31 04:20:04.808185 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:04.808078 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30a74489-ncq47" event={"ID":"83f22018-3412-4765-bce4-f273cdc41aa4","Type":"ContainerStarted","Data":"65c7bcabdcde60229ec981ab973e38769855967635769cf3bf0fd547656c0eff"} Oct 31 04:20:04.808185 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:04.808152 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30a74489-ncq47" event={"ID":"83f22018-3412-4765-bce4-f273cdc41aa4","Type":"ContainerStarted","Data":"3bc88fc4889ed6ea457bfc720649bd9ca2637ef6b1e45c791db52b45ddf76985"} Oct 31 04:20:04.848257 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:04.848202 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30a74489-ncq47" podStartSLOduration=1.848187642 podStartE2EDuration="1.848187642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:20:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:20:04.847441282 +0000 UTC m=+959.603779323" watchObservedRunningTime="2026-10-31 04:20:04.848187642 +0000 UTC m=+959.604525682" Oct 31 04:20:05.006627 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:05.006593 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e173eac-7j8xh" Oct 31 04:20:05.121372 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:05.121286 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-146320d5-586748b797-4lhzm"] Oct 31 04:20:05.122002 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:05.121984 5019 cpu_manager.go:414] "containerMap: removing container" podUID="e4fcefcb-a397-4a74-800d-efd33730069d" containerName="revoke-keys" Oct 31 04:20:05.122053 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:05.122004 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4fcefcb-a397-4a74-800d-efd33730069d" containerName="revoke-keys" Oct 31 04:20:05.122127 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:05.122117 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4fcefcb-a397-4a74-800d-efd33730069d" containerName="revoke-keys" Oct 31 04:20:05.125808 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:05.125778 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-146320d5-586748b797-4lhzm" Oct 31 04:20:05.126515 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:05.126488 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4fcefcb-a397-4a74-800d-efd33730069d-kube-api-access-nnclk\" (UniqueName: \"kubernetes.io/projected/e4fcefcb-a397-4a74-800d-efd33730069d-kube-api-access-nnclk\") pod \"e4fcefcb-a397-4a74-800d-efd33730069d\" (UID: \"e4fcefcb-a397-4a74-800d-efd33730069d\") " Oct 31 04:20:05.126649 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:05.126631 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4fcefcb-a397-4a74-800d-efd33730069d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4fcefcb-a397-4a74-800d-efd33730069d-maas-api-openshift-ca\") pod \"e4fcefcb-a397-4a74-800d-efd33730069d\" (UID: \"e4fcefcb-a397-4a74-800d-efd33730069d\") " Oct 31 04:20:05.126693 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:05.126662 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4fcefcb-a397-4a74-800d-efd33730069d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4fcefcb-a397-4a74-800d-efd33730069d-maas-api-xks-ca\") pod \"e4fcefcb-a397-4a74-800d-efd33730069d\" (UID: \"e4fcefcb-a397-4a74-800d-efd33730069d\") " Oct 31 04:20:05.127019 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:05.126994 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4fcefcb-a397-4a74-800d-efd33730069d-maas-api-openshift-ca" pod "e4fcefcb-a397-4a74-800d-efd33730069d" (UID: "e4fcefcb-a397-4a74-800d-efd33730069d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:20:05.128875 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:05.128841 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4fcefcb-a397-4a74-800d-efd33730069d-maas-api-xks-ca" pod "e4fcefcb-a397-4a74-800d-efd33730069d" (UID: "e4fcefcb-a397-4a74-800d-efd33730069d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:20:05.128875 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:05.128865 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4fcefcb-a397-4a74-800d-efd33730069d-kube-api-access-nnclk" pod "e4fcefcb-a397-4a74-800d-efd33730069d" (UID: "e4fcefcb-a397-4a74-800d-efd33730069d"). InnerVolumeSpecName "kube-api-access-nnclk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:20:05.140516 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:05.140317 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-146320d5\"" Oct 31 04:20:05.140735 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:05.140677 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-146320d5-dockercfg-hst7z\"" Oct 31 04:20:05.146060 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:05.146023 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-146320d5-586748b797-4lhzm"] Oct 31 04:20:05.210938 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:05.210895 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-146320d5-65b76f4b67-fnz6w"] Oct 31 04:20:05.223770 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:05.223733 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-146320d5-65b76f4b67-fnz6w"] Oct 31 04:20:05.223982 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:05.223852 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-146320d5-65b76f4b67-fnz6w" Oct 31 04:20:05.231612 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:05.230296 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8a637de-5932-47ad-b5e7-8d32ec65957f-plugins-config-volume\") pod \"payload-processing-e2e-mt-146320d5-586748b797-4lhzm\" (UID: \"d8a637de-5932-47ad-b5e7-8d32ec65957f\") " pod="openshift-ingress/payload-processing-e2e-mt-146320d5-586748b797-4lhzm" Oct 31 04:20:05.231612 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:05.230555 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jclx2\" (UniqueName: \"kubernetes.io/projected/d8a637de-5932-47ad-b5e7-8d32ec65957f-kube-api-access-jclx2\") pod \"payload-processing-e2e-mt-146320d5-586748b797-4lhzm\" (UID: \"d8a637de-5932-47ad-b5e7-8d32ec65957f\") " pod="openshift-ingress/payload-processing-e2e-mt-146320d5-586748b797-4lhzm" Oct 31 04:20:05.231612 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:05.230675 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4fcefcb-a397-4a74-800d-efd33730069d-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:20:05.231612 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:05.230693 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnclk\" (UniqueName: \"kubernetes.io/projected/e4fcefcb-a397-4a74-800d-efd33730069d-kube-api-access-nnclk\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:20:05.231612 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:05.230708 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4fcefcb-a397-4a74-800d-efd33730069d-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:20:05.343875 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:05.343770 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h67k5\" (UniqueName: \"kubernetes.io/projected/5cc83f70-27bb-42ee-9f5b-4d12dcb55df8-kube-api-access-h67k5\") pod \"payload-pre-processing-e2e-mt-146320d5-65b76f4b67-fnz6w\" (UID: \"5cc83f70-27bb-42ee-9f5b-4d12dcb55df8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-146320d5-65b76f4b67-fnz6w" Oct 31 04:20:05.344217 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:05.344187 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cc83f70-27bb-42ee-9f5b-4d12dcb55df8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-146320d5-65b76f4b67-fnz6w\" (UID: \"5cc83f70-27bb-42ee-9f5b-4d12dcb55df8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-146320d5-65b76f4b67-fnz6w" Oct 31 04:20:05.344917 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:05.344608 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8a637de-5932-47ad-b5e7-8d32ec65957f-plugins-config-volume\") pod \"payload-processing-e2e-mt-146320d5-586748b797-4lhzm\" (UID: \"d8a637de-5932-47ad-b5e7-8d32ec65957f\") " pod="openshift-ingress/payload-processing-e2e-mt-146320d5-586748b797-4lhzm" Oct 31 04:20:05.345230 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:05.345206 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jclx2\" (UniqueName: \"kubernetes.io/projected/d8a637de-5932-47ad-b5e7-8d32ec65957f-kube-api-access-jclx2\") pod \"payload-processing-e2e-mt-146320d5-586748b797-4lhzm\" (UID: \"d8a637de-5932-47ad-b5e7-8d32ec65957f\") " pod="openshift-ingress/payload-processing-e2e-mt-146320d5-586748b797-4lhzm" Oct 31 04:20:05.346120 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:05.346035 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8a637de-5932-47ad-b5e7-8d32ec65957f-plugins-config-volume\") pod \"payload-processing-e2e-mt-146320d5-586748b797-4lhzm\" (UID: \"d8a637de-5932-47ad-b5e7-8d32ec65957f\") " pod="openshift-ingress/payload-processing-e2e-mt-146320d5-586748b797-4lhzm" Oct 31 04:20:05.422428 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:05.420234 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jclx2\" (UniqueName: \"kubernetes.io/projected/d8a637de-5932-47ad-b5e7-8d32ec65957f-kube-api-access-jclx2\") pod \"payload-processing-e2e-mt-146320d5-586748b797-4lhzm\" (UID: \"d8a637de-5932-47ad-b5e7-8d32ec65957f\") " pod="openshift-ingress/payload-processing-e2e-mt-146320d5-586748b797-4lhzm" Oct 31 04:20:05.446925 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:05.446877 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h67k5\" (UniqueName: \"kubernetes.io/projected/5cc83f70-27bb-42ee-9f5b-4d12dcb55df8-kube-api-access-h67k5\") pod \"payload-pre-processing-e2e-mt-146320d5-65b76f4b67-fnz6w\" (UID: \"5cc83f70-27bb-42ee-9f5b-4d12dcb55df8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-146320d5-65b76f4b67-fnz6w" Oct 31 04:20:05.447204 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:05.447182 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cc83f70-27bb-42ee-9f5b-4d12dcb55df8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-146320d5-65b76f4b67-fnz6w\" (UID: \"5cc83f70-27bb-42ee-9f5b-4d12dcb55df8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-146320d5-65b76f4b67-fnz6w" Oct 31 04:20:05.448033 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:05.448011 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cc83f70-27bb-42ee-9f5b-4d12dcb55df8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-146320d5-65b76f4b67-fnz6w\" (UID: \"5cc83f70-27bb-42ee-9f5b-4d12dcb55df8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-146320d5-65b76f4b67-fnz6w" Oct 31 04:20:05.470499 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:05.470460 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h67k5\" (UniqueName: \"kubernetes.io/projected/5cc83f70-27bb-42ee-9f5b-4d12dcb55df8-kube-api-access-h67k5\") pod \"payload-pre-processing-e2e-mt-146320d5-65b76f4b67-fnz6w\" (UID: \"5cc83f70-27bb-42ee-9f5b-4d12dcb55df8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-146320d5-65b76f4b67-fnz6w" Oct 31 04:20:05.542899 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:05.542859 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-146320d5-dockercfg-hst7z\"" Oct 31 04:20:05.549731 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:05.549698 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-146320d5-586748b797-4lhzm" Oct 31 04:20:05.562285 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:05.562252 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-146320d5-65b76f4b67-fnz6w" Oct 31 04:20:05.821442 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:05.821401 5019 generic.go:356] "Generic (PLEG): container finished" podID="83f22018-3412-4765-bce4-f273cdc41aa4" containerID="65c7bcabdcde60229ec981ab973e38769855967635769cf3bf0fd547656c0eff" exitCode=0 Oct 31 04:20:05.821883 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:05.821469 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30a74489-ncq47" event={"ID":"83f22018-3412-4765-bce4-f273cdc41aa4","Type":"ContainerDied","Data":"65c7bcabdcde60229ec981ab973e38769855967635769cf3bf0fd547656c0eff"} Oct 31 04:20:05.823296 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:05.823267 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e173eac-7j8xh" Oct 31 04:20:05.823296 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:05.823279 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e173eac-7j8xh" event={"ID":"e4fcefcb-a397-4a74-800d-efd33730069d","Type":"ContainerDied","Data":"525ac7cfdb47c86b983cab6469040c80d1e7b6962721e2f4edd3afd1db904276"} Oct 31 04:20:05.823514 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:05.823319 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="525ac7cfdb47c86b983cab6469040c80d1e7b6962721e2f4edd3afd1db904276" Oct 31 04:20:05.871858 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:20:05.871822 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8a637de_5932_47ad_b5e7_8d32ec65957f.slice/crio-c1a9674759d635243a755d4b4219e886b160a410e3ac9dd34d8db5a4ca83459d WatchSource:0}: Error finding container c1a9674759d635243a755d4b4219e886b160a410e3ac9dd34d8db5a4ca83459d: Status 404 returned error can't find the container with id c1a9674759d635243a755d4b4219e886b160a410e3ac9dd34d8db5a4ca83459d Oct 31 04:20:05.874142 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:05.874064 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-146320d5-586748b797-4lhzm"] Oct 31 04:20:05.901470 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:05.901433 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-146320d5-65b76f4b67-fnz6w"] Oct 31 04:20:06.833834 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:06.833795 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-146320d5-586748b797-4lhzm" event={"ID":"d8a637de-5932-47ad-b5e7-8d32ec65957f","Type":"ContainerStarted","Data":"14fa50afb7e9d61a9ac9990ae030ca72d9e7dbe698f4564630d64f1e392311f5"} Oct 31 04:20:06.836610 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:06.833869 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-146320d5-586748b797-4lhzm" Oct 31 04:20:06.836610 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:06.833897 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-146320d5-586748b797-4lhzm" event={"ID":"d8a637de-5932-47ad-b5e7-8d32ec65957f","Type":"ContainerStarted","Data":"c1a9674759d635243a755d4b4219e886b160a410e3ac9dd34d8db5a4ca83459d"} Oct 31 04:20:06.836610 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:06.835648 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-146320d5-65b76f4b67-fnz6w" event={"ID":"5cc83f70-27bb-42ee-9f5b-4d12dcb55df8","Type":"ContainerStarted","Data":"e652de40d2105aa57f76d99d8e338c91c518b69d0cb9da7e9a82dbec1bc90228"} Oct 31 04:20:06.836610 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:06.835690 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-146320d5-65b76f4b67-fnz6w" event={"ID":"5cc83f70-27bb-42ee-9f5b-4d12dcb55df8","Type":"ContainerStarted","Data":"0f947275ed3ed33e4b1e53b10a6ff7bce4dd221c575d4938a402f9d3491d083e"} Oct 31 04:20:06.878906 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:06.878829 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-146320d5-586748b797-4lhzm" podStartSLOduration=1.8787943440000001 podStartE2EDuration="1.878794344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:20:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:20:06.873424895 +0000 UTC m=+961.629762934" watchObservedRunningTime="2026-10-31 04:20:06.878794344 +0000 UTC m=+961.635132385" Oct 31 04:20:06.992908 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:06.992845 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-146320d5-65b76f4b67-fnz6w" podStartSLOduration=1.9928184249999998 podStartE2EDuration="1.992818425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:20:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:20:06.923885132 +0000 UTC m=+961.680223173" watchObservedRunningTime="2026-10-31 04:20:06.992818425 +0000 UTC m=+961.749156466" Oct 31 04:20:07.005372 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:07.001239 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42765d06-6w9tv"] Oct 31 04:20:07.015890 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:07.015855 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42765d06-6w9tv" Oct 31 04:20:07.041994 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:07.041433 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42765d06-6w9tv"] Oct 31 04:20:07.097882 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:07.093003 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvlpc\" (UniqueName: \"kubernetes.io/projected/2f871c0f-0aa9-488e-b247-2555e9c7854e-kube-api-access-wvlpc\") pod \"maas-api-revoke-sub-42765d06-6w9tv\" (UID: \"2f871c0f-0aa9-488e-b247-2555e9c7854e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42765d06-6w9tv" Oct 31 04:20:07.097882 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:07.093232 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f871c0f-0aa9-488e-b247-2555e9c7854e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42765d06-6w9tv\" (UID: \"2f871c0f-0aa9-488e-b247-2555e9c7854e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42765d06-6w9tv" Oct 31 04:20:07.097882 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:07.095562 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f871c0f-0aa9-488e-b247-2555e9c7854e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42765d06-6w9tv\" (UID: \"2f871c0f-0aa9-488e-b247-2555e9c7854e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42765d06-6w9tv" Oct 31 04:20:07.110746 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:07.110689 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30a74489-ncq47" Oct 31 04:20:07.201356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:07.196956 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83f22018-3412-4765-bce4-f273cdc41aa4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83f22018-3412-4765-bce4-f273cdc41aa4-maas-api-xks-ca\") pod \"83f22018-3412-4765-bce4-f273cdc41aa4\" (UID: \"83f22018-3412-4765-bce4-f273cdc41aa4\") " Oct 31 04:20:07.201356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:07.197042 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83f22018-3412-4765-bce4-f273cdc41aa4-kube-api-access-vmdps\" (UniqueName: \"kubernetes.io/projected/83f22018-3412-4765-bce4-f273cdc41aa4-kube-api-access-vmdps\") pod \"83f22018-3412-4765-bce4-f273cdc41aa4\" (UID: \"83f22018-3412-4765-bce4-f273cdc41aa4\") " Oct 31 04:20:07.201356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:07.197219 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83f22018-3412-4765-bce4-f273cdc41aa4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83f22018-3412-4765-bce4-f273cdc41aa4-maas-api-openshift-ca\") pod \"83f22018-3412-4765-bce4-f273cdc41aa4\" (UID: \"83f22018-3412-4765-bce4-f273cdc41aa4\") " Oct 31 04:20:07.201356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:07.197601 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f871c0f-0aa9-488e-b247-2555e9c7854e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42765d06-6w9tv\" (UID: \"2f871c0f-0aa9-488e-b247-2555e9c7854e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42765d06-6w9tv" Oct 31 04:20:07.201356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:07.197736 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f871c0f-0aa9-488e-b247-2555e9c7854e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42765d06-6w9tv\" (UID: \"2f871c0f-0aa9-488e-b247-2555e9c7854e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42765d06-6w9tv" Oct 31 04:20:07.201356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:07.197820 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvlpc\" (UniqueName: \"kubernetes.io/projected/2f871c0f-0aa9-488e-b247-2555e9c7854e-kube-api-access-wvlpc\") pod \"maas-api-revoke-sub-42765d06-6w9tv\" (UID: \"2f871c0f-0aa9-488e-b247-2555e9c7854e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42765d06-6w9tv" Oct 31 04:20:07.201356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:07.200905 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83f22018-3412-4765-bce4-f273cdc41aa4-maas-api-openshift-ca" pod "83f22018-3412-4765-bce4-f273cdc41aa4" (UID: "83f22018-3412-4765-bce4-f273cdc41aa4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:20:07.203725 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:07.203667 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83f22018-3412-4765-bce4-f273cdc41aa4-maas-api-xks-ca" pod "83f22018-3412-4765-bce4-f273cdc41aa4" (UID: "83f22018-3412-4765-bce4-f273cdc41aa4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:20:07.204310 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:07.204187 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f871c0f-0aa9-488e-b247-2555e9c7854e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42765d06-6w9tv\" (UID: \"2f871c0f-0aa9-488e-b247-2555e9c7854e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42765d06-6w9tv" Oct 31 04:20:07.216388 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:07.206321 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f871c0f-0aa9-488e-b247-2555e9c7854e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42765d06-6w9tv\" (UID: \"2f871c0f-0aa9-488e-b247-2555e9c7854e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42765d06-6w9tv" Oct 31 04:20:07.216388 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:07.209513 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83f22018-3412-4765-bce4-f273cdc41aa4-kube-api-access-vmdps" pod "83f22018-3412-4765-bce4-f273cdc41aa4" (UID: "83f22018-3412-4765-bce4-f273cdc41aa4"). InnerVolumeSpecName "kube-api-access-vmdps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:20:07.221416 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:07.221347 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvlpc\" (UniqueName: \"kubernetes.io/projected/2f871c0f-0aa9-488e-b247-2555e9c7854e-kube-api-access-wvlpc\") pod \"maas-api-revoke-sub-42765d06-6w9tv\" (UID: \"2f871c0f-0aa9-488e-b247-2555e9c7854e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42765d06-6w9tv" Oct 31 04:20:07.299807 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:07.299766 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmdps\" (UniqueName: \"kubernetes.io/projected/83f22018-3412-4765-bce4-f273cdc41aa4-kube-api-access-vmdps\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:20:07.299807 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:07.299799 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83f22018-3412-4765-bce4-f273cdc41aa4-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:20:07.299807 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:07.299810 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83f22018-3412-4765-bce4-f273cdc41aa4-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:20:07.398698 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:07.398585 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42765d06-6w9tv" Oct 31 04:20:07.710188 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:20:07.710145 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f871c0f_0aa9_488e_b247_2555e9c7854e.slice/crio-acfde3c7036bd4b30fd6a6e1335e044e0c17c6cf0e7a68e319a310187bfa6ef1 WatchSource:0}: Error finding container acfde3c7036bd4b30fd6a6e1335e044e0c17c6cf0e7a68e319a310187bfa6ef1: Status 404 returned error can't find the container with id acfde3c7036bd4b30fd6a6e1335e044e0c17c6cf0e7a68e319a310187bfa6ef1 Oct 31 04:20:07.711123 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:07.711094 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42765d06-6w9tv"] Oct 31 04:20:07.844957 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:07.844920 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42765d06-6w9tv" event={"ID":"2f871c0f-0aa9-488e-b247-2555e9c7854e","Type":"ContainerStarted","Data":"acfde3c7036bd4b30fd6a6e1335e044e0c17c6cf0e7a68e319a310187bfa6ef1"} Oct 31 04:20:07.846474 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:07.846450 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30a74489-ncq47" Oct 31 04:20:07.846605 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:07.846483 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30a74489-ncq47" event={"ID":"83f22018-3412-4765-bce4-f273cdc41aa4","Type":"ContainerDied","Data":"3bc88fc4889ed6ea457bfc720649bd9ca2637ef6b1e45c791db52b45ddf76985"} Oct 31 04:20:07.846605 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:07.846543 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3bc88fc4889ed6ea457bfc720649bd9ca2637ef6b1e45c791db52b45ddf76985" Oct 31 04:20:07.846947 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:07.846930 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-146320d5-65b76f4b67-fnz6w" Oct 31 04:20:08.857997 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:08.857951 5019 generic.go:356] "Generic (PLEG): container finished" podID="2f871c0f-0aa9-488e-b247-2555e9c7854e" containerID="44e75fb5f36a7f4b9daa78339daebf3146e8ccb04c496eaff46c881a72a4bb67" exitCode=0 Oct 31 04:20:08.858501 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:08.858071 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42765d06-6w9tv" event={"ID":"2f871c0f-0aa9-488e-b247-2555e9c7854e","Type":"ContainerDied","Data":"44e75fb5f36a7f4b9daa78339daebf3146e8ccb04c496eaff46c881a72a4bb67"} Oct 31 04:20:09.334245 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:09.333843 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4de11517-qf5pb"] Oct 31 04:20:09.334887 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:09.334855 5019 cpu_manager.go:414] "containerMap: removing container" podUID="83f22018-3412-4765-bce4-f273cdc41aa4" containerName="revoke-keys" Oct 31 04:20:09.334887 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:09.334888 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83f22018-3412-4765-bce4-f273cdc41aa4" containerName="revoke-keys" Oct 31 04:20:09.335375 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:09.335098 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="83f22018-3412-4765-bce4-f273cdc41aa4" containerName="revoke-keys" Oct 31 04:20:09.346838 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:09.346789 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de11517-qf5pb" Oct 31 04:20:09.356221 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:09.356183 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4de11517-qf5pb"] Oct 31 04:20:09.446388 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:09.439347 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0a65aa5-25f8-4835-bcbd-88236c70410e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4de11517-qf5pb\" (UID: \"a0a65aa5-25f8-4835-bcbd-88236c70410e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de11517-qf5pb" Oct 31 04:20:09.446388 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:09.439455 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0a65aa5-25f8-4835-bcbd-88236c70410e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4de11517-qf5pb\" (UID: \"a0a65aa5-25f8-4835-bcbd-88236c70410e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de11517-qf5pb" Oct 31 04:20:09.446388 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:09.439516 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csvx9\" (UniqueName: \"kubernetes.io/projected/a0a65aa5-25f8-4835-bcbd-88236c70410e-kube-api-access-csvx9\") pod \"maas-api-revoke-sub-4de11517-qf5pb\" (UID: \"a0a65aa5-25f8-4835-bcbd-88236c70410e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de11517-qf5pb" Oct 31 04:20:09.555367 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:09.554397 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0a65aa5-25f8-4835-bcbd-88236c70410e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4de11517-qf5pb\" (UID: \"a0a65aa5-25f8-4835-bcbd-88236c70410e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de11517-qf5pb" Oct 31 04:20:09.555367 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:09.554506 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0a65aa5-25f8-4835-bcbd-88236c70410e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4de11517-qf5pb\" (UID: \"a0a65aa5-25f8-4835-bcbd-88236c70410e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de11517-qf5pb" Oct 31 04:20:09.555367 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:09.554560 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csvx9\" (UniqueName: \"kubernetes.io/projected/a0a65aa5-25f8-4835-bcbd-88236c70410e-kube-api-access-csvx9\") pod \"maas-api-revoke-sub-4de11517-qf5pb\" (UID: \"a0a65aa5-25f8-4835-bcbd-88236c70410e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de11517-qf5pb" Oct 31 04:20:09.603442 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:09.582544 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0a65aa5-25f8-4835-bcbd-88236c70410e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4de11517-qf5pb\" (UID: \"a0a65aa5-25f8-4835-bcbd-88236c70410e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de11517-qf5pb" Oct 31 04:20:09.608383 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:09.604723 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0a65aa5-25f8-4835-bcbd-88236c70410e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4de11517-qf5pb\" (UID: \"a0a65aa5-25f8-4835-bcbd-88236c70410e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de11517-qf5pb" Oct 31 04:20:09.627009 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:09.622633 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csvx9\" (UniqueName: \"kubernetes.io/projected/a0a65aa5-25f8-4835-bcbd-88236c70410e-kube-api-access-csvx9\") pod \"maas-api-revoke-sub-4de11517-qf5pb\" (UID: \"a0a65aa5-25f8-4835-bcbd-88236c70410e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de11517-qf5pb" Oct 31 04:20:09.699017 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:09.697637 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de11517-qf5pb" Oct 31 04:20:11.445689 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:11.445635 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 31 04:20:11.511693 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:11.474117 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" podUID="643433d8-418e-4102-8bc3-cd544932614a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:11.511693 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:11.474528 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" podUID="b4ced3ab-41e4-4c54-93eb-95eddaae992d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:11.511693 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:11.477081 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 31 04:20:11.641362 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:11.641227 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" podUID="8e2c7041-ad52-49f4-8b55-d16691f4938f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 31 04:20:11.938541 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:11.926849 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" podUID="dccd15a9-47f5-4bd1-b21b-9d5ca8b05845" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:12.220671 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:12.199831 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4de11517-qf5pb"] Oct 31 04:20:12.259381 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:12.231603 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42765d06-6w9tv" Oct 31 04:20:12.462865 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:12.462764 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f871c0f-0aa9-488e-b247-2555e9c7854e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f871c0f-0aa9-488e-b247-2555e9c7854e-maas-api-openshift-ca\") pod \"2f871c0f-0aa9-488e-b247-2555e9c7854e\" (UID: \"2f871c0f-0aa9-488e-b247-2555e9c7854e\") " Oct 31 04:20:12.462865 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:12.462844 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f871c0f-0aa9-488e-b247-2555e9c7854e-kube-api-access-wvlpc\" (UniqueName: \"kubernetes.io/projected/2f871c0f-0aa9-488e-b247-2555e9c7854e-kube-api-access-wvlpc\") pod \"2f871c0f-0aa9-488e-b247-2555e9c7854e\" (UID: \"2f871c0f-0aa9-488e-b247-2555e9c7854e\") " Oct 31 04:20:12.463445 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:12.462887 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f871c0f-0aa9-488e-b247-2555e9c7854e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f871c0f-0aa9-488e-b247-2555e9c7854e-maas-api-xks-ca\") pod \"2f871c0f-0aa9-488e-b247-2555e9c7854e\" (UID: \"2f871c0f-0aa9-488e-b247-2555e9c7854e\") " Oct 31 04:20:12.498497 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:12.481623 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 31 04:20:12.498497 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:12.483392 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:12.498497 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:12.483778 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 31 04:20:12.498497 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:12.484023 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" podUID="643433d8-418e-4102-8bc3-cd544932614a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 04:20:12.498497 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:12.496575 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f871c0f-0aa9-488e-b247-2555e9c7854e-maas-api-openshift-ca" pod "2f871c0f-0aa9-488e-b247-2555e9c7854e" (UID: "2f871c0f-0aa9-488e-b247-2555e9c7854e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:20:12.624408 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:12.555054 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f871c0f-0aa9-488e-b247-2555e9c7854e-maas-api-xks-ca" pod "2f871c0f-0aa9-488e-b247-2555e9c7854e" (UID: "2f871c0f-0aa9-488e-b247-2555e9c7854e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:20:12.624408 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:12.555217 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f871c0f-0aa9-488e-b247-2555e9c7854e-kube-api-access-wvlpc" pod "2f871c0f-0aa9-488e-b247-2555e9c7854e" (UID: "2f871c0f-0aa9-488e-b247-2555e9c7854e"). InnerVolumeSpecName "kube-api-access-wvlpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:20:12.624408 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:12.562235 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" Oct 31 04:20:12.624408 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:12.575084 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" Oct 31 04:20:12.624408 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:12.575166 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" Oct 31 04:20:12.637735 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:12.637630 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f871c0f-0aa9-488e-b247-2555e9c7854e-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:20:12.637735 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:12.637676 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvlpc\" (UniqueName: \"kubernetes.io/projected/2f871c0f-0aa9-488e-b247-2555e9c7854e-kube-api-access-wvlpc\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:20:12.637735 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:12.637695 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f871c0f-0aa9-488e-b247-2555e9c7854e-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:20:12.718128 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:12.643461 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" podUID="8e2c7041-ad52-49f4-8b55-d16691f4938f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 31 04:20:12.718128 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:12.643960 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" Oct 31 04:20:12.718128 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:12.645783 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" podUID="b4ced3ab-41e4-4c54-93eb-95eddaae992d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:12.718128 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:12.646157 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" Oct 31 04:20:12.790488 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:12.790323 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" podUID="e0f4b8c2-14f4-4052-bf54-dd65bf59e413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 04:20:13.108438 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:12.972717 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" podUID="dccd15a9-47f5-4bd1-b21b-9d5ca8b05845" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 31 04:20:13.128865 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:13.099812 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" podUID="e0f4b8c2-14f4-4052-bf54-dd65bf59e413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:13.129999 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:13.129853 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" Oct 31 04:20:13.372379 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:13.319427 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" podUID="dccd15a9-47f5-4bd1-b21b-9d5ca8b05845" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:13.463466 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:13.453608 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42765d06-6w9tv" Oct 31 04:20:13.482317 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:13.482270 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:13.501764 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:13.483838 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:13.501764 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:13.485360 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" podUID="643433d8-418e-4102-8bc3-cd544932614a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:13.501764 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:13.498445 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 31 04:20:13.611934 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:13.589979 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" Oct 31 04:20:13.611934 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:13.590027 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42765d06-6w9tv" event={"ID":"2f871c0f-0aa9-488e-b247-2555e9c7854e","Type":"ContainerDied","Data":"acfde3c7036bd4b30fd6a6e1335e044e0c17c6cf0e7a68e319a310187bfa6ef1"} Oct 31 04:20:13.611934 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:13.590067 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acfde3c7036bd4b30fd6a6e1335e044e0c17c6cf0e7a68e319a310187bfa6ef1" Oct 31 04:20:13.611934 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:13.590079 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de11517-qf5pb" event={"ID":"a0a65aa5-25f8-4835-bcbd-88236c70410e","Type":"ContainerStarted","Data":"12c6b392a40d085977e3d4afadb9644c798cdfcd62cc44d33530e8d41f39841d"} Oct 31 04:20:13.650685 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:13.647607 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" podUID="8e2c7041-ad52-49f4-8b55-d16691f4938f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 31 04:20:13.650970 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:13.650929 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" podUID="b4ced3ab-41e4-4c54-93eb-95eddaae992d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:13.803499 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:13.803460 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" podUID="e0f4b8c2-14f4-4052-bf54-dd65bf59e413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 04:20:13.804185 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:13.804162 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" Oct 31 04:20:14.263471 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:14.257719 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:14.263471 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:14.258299 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" podUID="dccd15a9-47f5-4bd1-b21b-9d5ca8b05845" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 31 04:20:14.519388 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:14.518154 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:14.519388 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:14.518570 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:14.519388 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:14.518849 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 31 04:20:14.519388 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:14.519119 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" podUID="643433d8-418e-4102-8bc3-cd544932614a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 04:20:14.535360 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:14.531262 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-802ac966-5tzmm"] Oct 31 04:20:14.556455 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:14.556423 5019 cpu_manager.go:414] "containerMap: removing container" podUID="2f871c0f-0aa9-488e-b247-2555e9c7854e" containerName="revoke-keys" Oct 31 04:20:14.556722 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:14.556705 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f871c0f-0aa9-488e-b247-2555e9c7854e" containerName="revoke-keys" Oct 31 04:20:14.557138 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:14.557124 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f871c0f-0aa9-488e-b247-2555e9c7854e" containerName="revoke-keys" Oct 31 04:20:14.596484 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:14.596454 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-802ac966-5tzmm" Oct 31 04:20:14.669859 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:14.669814 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" podUID="8e2c7041-ad52-49f4-8b55-d16691f4938f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:14.686280 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:14.684828 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" podUID="b4ced3ab-41e4-4c54-93eb-95eddaae992d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:15.026412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:15.026352 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" podUID="e0f4b8c2-14f4-4052-bf54-dd65bf59e413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 04:20:15.063407 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:15.037961 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/587549d7-b0e2-4f3d-a956-934ca87749f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-802ac966-5tzmm\" (UID: \"587549d7-b0e2-4f3d-a956-934ca87749f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-802ac966-5tzmm" Oct 31 04:20:15.066443 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:15.046525 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:20:15.066833 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:15.066785 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v" podUID="c4e33d7c-168f-4001-8e39-22bea132574a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:15.069639 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:15.069608 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jsp6\" (UniqueName: \"kubernetes.io/projected/587549d7-b0e2-4f3d-a956-934ca87749f1-kube-api-access-8jsp6\") pod \"maas-api-revoke-sub-802ac966-5tzmm\" (UID: \"587549d7-b0e2-4f3d-a956-934ca87749f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-802ac966-5tzmm" Oct 31 04:20:15.069935 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:15.069911 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/587549d7-b0e2-4f3d-a956-934ca87749f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-802ac966-5tzmm\" (UID: \"587549d7-b0e2-4f3d-a956-934ca87749f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-802ac966-5tzmm" Oct 31 04:20:15.222186 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:15.222084 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/587549d7-b0e2-4f3d-a956-934ca87749f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-802ac966-5tzmm\" (UID: \"587549d7-b0e2-4f3d-a956-934ca87749f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-802ac966-5tzmm" Oct 31 04:20:15.222504 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:15.222477 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jsp6\" (UniqueName: \"kubernetes.io/projected/587549d7-b0e2-4f3d-a956-934ca87749f1-kube-api-access-8jsp6\") pod \"maas-api-revoke-sub-802ac966-5tzmm\" (UID: \"587549d7-b0e2-4f3d-a956-934ca87749f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-802ac966-5tzmm" Oct 31 04:20:15.222668 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:15.222653 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/587549d7-b0e2-4f3d-a956-934ca87749f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-802ac966-5tzmm\" (UID: \"587549d7-b0e2-4f3d-a956-934ca87749f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-802ac966-5tzmm" Oct 31 04:20:15.229362 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:15.224065 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/587549d7-b0e2-4f3d-a956-934ca87749f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-802ac966-5tzmm\" (UID: \"587549d7-b0e2-4f3d-a956-934ca87749f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-802ac966-5tzmm" Oct 31 04:20:15.419360 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:15.279470 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" podUID="dccd15a9-47f5-4bd1-b21b-9d5ca8b05845" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 31 04:20:15.467363 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:15.373105 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" podUID="b4ced3ab-41e4-4c54-93eb-95eddaae992d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/health\": context deadline exceeded" Oct 31 04:20:15.577497 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:15.369984 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de11517-qf5pb" event={"ID":"a0a65aa5-25f8-4835-bcbd-88236c70410e","Type":"ContainerStarted","Data":"7d54dc032880acf3f7ffc4d6e6b69b0ff345071b5f1e86b1c7622fedb2c901ed"} Oct 31 04:20:15.578148 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:15.533599 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 31 04:20:15.603355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:15.533653 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 31 04:20:15.637008 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:15.533676 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 31 04:20:15.700439 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:15.548427 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" podUID="643433d8-418e-4102-8bc3-cd544932614a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:15.727837 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:15.717595 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" podUID="b4ced3ab-41e4-4c54-93eb-95eddaae992d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:15.753407 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:15.733580 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" podUID="8e2c7041-ad52-49f4-8b55-d16691f4938f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:15.829410 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:15.824518 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jsp6\" (UniqueName: \"kubernetes.io/projected/587549d7-b0e2-4f3d-a956-934ca87749f1-kube-api-access-8jsp6\") pod \"maas-api-revoke-sub-802ac966-5tzmm\" (UID: \"587549d7-b0e2-4f3d-a956-934ca87749f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-802ac966-5tzmm" Oct 31 04:20:15.851463 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:15.839874 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/587549d7-b0e2-4f3d-a956-934ca87749f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-802ac966-5tzmm\" (UID: \"587549d7-b0e2-4f3d-a956-934ca87749f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-802ac966-5tzmm" Oct 31 04:20:15.913358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:15.897557 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-802ac966-5tzmm" Oct 31 04:20:16.056358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:16.034419 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" podUID="e0f4b8c2-14f4-4052-bf54-dd65bf59e413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 04:20:16.156676 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:16.077694 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-802ac966-5tzmm"] Oct 31 04:20:16.429558 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:16.428497 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" podUID="dccd15a9-47f5-4bd1-b21b-9d5ca8b05845" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:16.447245 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:16.446695 5019 generic.go:356] "Generic (PLEG): container finished" podID="a0a65aa5-25f8-4835-bcbd-88236c70410e" containerID="7d54dc032880acf3f7ffc4d6e6b69b0ff345071b5f1e86b1c7622fedb2c901ed" exitCode=0 Oct 31 04:20:16.447245 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:16.446792 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de11517-qf5pb" event={"ID":"a0a65aa5-25f8-4835-bcbd-88236c70410e","Type":"ContainerDied","Data":"7d54dc032880acf3f7ffc4d6e6b69b0ff345071b5f1e86b1c7622fedb2c901ed"} Oct 31 04:20:16.533360 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:16.511661 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp"] Oct 31 04:20:16.587354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:16.585592 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 31 04:20:16.626359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:16.619975 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp" Oct 31 04:20:16.626359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:16.621099 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r"] Oct 31 04:20:16.656351 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:16.651321 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 31 04:20:16.971133 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:16.654078 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db10a00c-b537-4c62-b9aa-b98dc5577720-istio-token\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp\" (UID: \"db10a00c-b537-4c62-b9aa-b98dc5577720\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp" Oct 31 04:20:16.971644 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:16.971613 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44xck\" (UniqueName: \"kubernetes.io/projected/db10a00c-b537-4c62-b9aa-b98dc5577720-kube-api-access-44xck\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp\" (UID: \"db10a00c-b537-4c62-b9aa-b98dc5577720\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp" Oct 31 04:20:16.972024 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:16.972006 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db10a00c-b537-4c62-b9aa-b98dc5577720-credential-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp\" (UID: \"db10a00c-b537-4c62-b9aa-b98dc5577720\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp" Oct 31 04:20:16.972196 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:16.972181 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db10a00c-b537-4c62-b9aa-b98dc5577720-workload-certs\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp\" (UID: \"db10a00c-b537-4c62-b9aa-b98dc5577720\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp" Oct 31 04:20:16.972382 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:16.972365 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db10a00c-b537-4c62-b9aa-b98dc5577720-istiod-ca-cert\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp\" (UID: \"db10a00c-b537-4c62-b9aa-b98dc5577720\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp" Oct 31 04:20:16.972576 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:16.972562 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db10a00c-b537-4c62-b9aa-b98dc5577720-workload-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp\" (UID: \"db10a00c-b537-4c62-b9aa-b98dc5577720\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp" Oct 31 04:20:16.972731 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:16.972718 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db10a00c-b537-4c62-b9aa-b98dc5577720-istio-data\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp\" (UID: \"db10a00c-b537-4c62-b9aa-b98dc5577720\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp" Oct 31 04:20:16.976690 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:16.737442 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" podUID="643433d8-418e-4102-8bc3-cd544932614a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:16.976690 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:16.738369 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" Oct 31 04:20:16.979381 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:16.976984 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp"] Oct 31 04:20:16.979381 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:16.735706 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r" Oct 31 04:20:16.980173 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:16.654163 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 31 04:20:16.983505 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:16.983468 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db10a00c-b537-4c62-b9aa-b98dc5577720-istio-envoy\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp\" (UID: \"db10a00c-b537-4c62-b9aa-b98dc5577720\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp" Oct 31 04:20:16.983791 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:16.983773 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db10a00c-b537-4c62-b9aa-b98dc5577720-istio-podinfo\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp\" (UID: \"db10a00c-b537-4c62-b9aa-b98dc5577720\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp" Oct 31 04:20:17.051030 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.050990 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" Oct 31 04:20:17.074354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.065061 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r"] Oct 31 04:20:17.095354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.093929 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" podUID="e0f4b8c2-14f4-4052-bf54-dd65bf59e413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 04:20:17.107579 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.099123 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8d33677-060d-418a-8806-84eeb0c17ecb-workload-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r\" (UID: \"c8d33677-060d-418a-8806-84eeb0c17ecb\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r" Oct 31 04:20:17.107579 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.099180 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8d33677-060d-418a-8806-84eeb0c17ecb-istio-data\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r\" (UID: \"c8d33677-060d-418a-8806-84eeb0c17ecb\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r" Oct 31 04:20:17.107579 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.099247 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db10a00c-b537-4c62-b9aa-b98dc5577720-credential-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp\" (UID: \"db10a00c-b537-4c62-b9aa-b98dc5577720\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp" Oct 31 04:20:17.107579 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.099274 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8d33677-060d-418a-8806-84eeb0c17ecb-istiod-ca-cert\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r\" (UID: \"c8d33677-060d-418a-8806-84eeb0c17ecb\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r" Oct 31 04:20:17.107579 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.099317 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db10a00c-b537-4c62-b9aa-b98dc5577720-workload-certs\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp\" (UID: \"db10a00c-b537-4c62-b9aa-b98dc5577720\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp" Oct 31 04:20:17.107579 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.099388 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db10a00c-b537-4c62-b9aa-b98dc5577720-istiod-ca-cert\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp\" (UID: \"db10a00c-b537-4c62-b9aa-b98dc5577720\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp" Oct 31 04:20:17.107579 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.099440 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db10a00c-b537-4c62-b9aa-b98dc5577720-workload-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp\" (UID: \"db10a00c-b537-4c62-b9aa-b98dc5577720\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp" Oct 31 04:20:17.107579 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.099479 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db10a00c-b537-4c62-b9aa-b98dc5577720-istio-data\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp\" (UID: \"db10a00c-b537-4c62-b9aa-b98dc5577720\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp" Oct 31 04:20:17.107579 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.099573 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db10a00c-b537-4c62-b9aa-b98dc5577720-istio-envoy\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp\" (UID: \"db10a00c-b537-4c62-b9aa-b98dc5577720\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp" Oct 31 04:20:17.107579 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.099617 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8d33677-060d-418a-8806-84eeb0c17ecb-istio-envoy\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r\" (UID: \"c8d33677-060d-418a-8806-84eeb0c17ecb\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r" Oct 31 04:20:17.107579 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.099669 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db10a00c-b537-4c62-b9aa-b98dc5577720-istio-podinfo\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp\" (UID: \"db10a00c-b537-4c62-b9aa-b98dc5577720\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp" Oct 31 04:20:17.107579 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.099694 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8d33677-060d-418a-8806-84eeb0c17ecb-istio-token\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r\" (UID: \"c8d33677-060d-418a-8806-84eeb0c17ecb\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r" Oct 31 04:20:17.107579 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.099773 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db10a00c-b537-4c62-b9aa-b98dc5577720-istio-token\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp\" (UID: \"db10a00c-b537-4c62-b9aa-b98dc5577720\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp" Oct 31 04:20:17.107579 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.099876 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8d33677-060d-418a-8806-84eeb0c17ecb-credential-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r\" (UID: \"c8d33677-060d-418a-8806-84eeb0c17ecb\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r" Oct 31 04:20:17.107579 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.099903 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8d33677-060d-418a-8806-84eeb0c17ecb-workload-certs\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r\" (UID: \"c8d33677-060d-418a-8806-84eeb0c17ecb\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r" Oct 31 04:20:17.107579 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.099925 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8d33677-060d-418a-8806-84eeb0c17ecb-istio-podinfo\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r\" (UID: \"c8d33677-060d-418a-8806-84eeb0c17ecb\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r" Oct 31 04:20:17.107579 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.099957 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxx9f\" (UniqueName: \"kubernetes.io/projected/c8d33677-060d-418a-8806-84eeb0c17ecb-kube-api-access-bxx9f\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r\" (UID: \"c8d33677-060d-418a-8806-84eeb0c17ecb\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r" Oct 31 04:20:17.107579 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.100060 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44xck\" (UniqueName: \"kubernetes.io/projected/db10a00c-b537-4c62-b9aa-b98dc5577720-kube-api-access-44xck\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp\" (UID: \"db10a00c-b537-4c62-b9aa-b98dc5577720\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp" Oct 31 04:20:17.107579 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.100646 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db10a00c-b537-4c62-b9aa-b98dc5577720-workload-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp\" (UID: \"db10a00c-b537-4c62-b9aa-b98dc5577720\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp" Oct 31 04:20:17.107579 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.100814 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db10a00c-b537-4c62-b9aa-b98dc5577720-istio-data\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp\" (UID: \"db10a00c-b537-4c62-b9aa-b98dc5577720\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp" Oct 31 04:20:17.107579 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.101651 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" Oct 31 04:20:17.107579 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.102077 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db10a00c-b537-4c62-b9aa-b98dc5577720-credential-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp\" (UID: \"db10a00c-b537-4c62-b9aa-b98dc5577720\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp" Oct 31 04:20:17.107579 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.102384 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db10a00c-b537-4c62-b9aa-b98dc5577720-workload-certs\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp\" (UID: \"db10a00c-b537-4c62-b9aa-b98dc5577720\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp" Oct 31 04:20:17.107579 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.103047 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db10a00c-b537-4c62-b9aa-b98dc5577720-istiod-ca-cert\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp\" (UID: \"db10a00c-b537-4c62-b9aa-b98dc5577720\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp" Oct 31 04:20:17.113930 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.110573 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db10a00c-b537-4c62-b9aa-b98dc5577720-istio-envoy\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp\" (UID: \"db10a00c-b537-4c62-b9aa-b98dc5577720\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp" Oct 31 04:20:17.121635 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.118902 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de11517-qf5pb" podStartSLOduration=8.118876359 podStartE2EDuration="8.118876359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:20:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:20:15.674582298 +0000 UTC m=+970.430920340" watchObservedRunningTime="2026-10-31 04:20:17.118876359 +0000 UTC m=+971.875214399" Oct 31 04:20:17.123982 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.122041 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db10a00c-b537-4c62-b9aa-b98dc5577720-istio-podinfo\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp\" (UID: \"db10a00c-b537-4c62-b9aa-b98dc5577720\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp" Oct 31 04:20:17.136597 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.136556 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db10a00c-b537-4c62-b9aa-b98dc5577720-istio-token\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp\" (UID: \"db10a00c-b537-4c62-b9aa-b98dc5577720\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp" Oct 31 04:20:17.181608 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.166114 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" Oct 31 04:20:17.181608 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:20:17.167454 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod587549d7_b0e2_4f3d_a956_934ca87749f1.slice/crio-1c7685ba8802d99c3692fadac4224501def05e2a4dd71389d4ce9b74693b2811 WatchSource:0}: Error finding container 1c7685ba8802d99c3692fadac4224501def05e2a4dd71389d4ce9b74693b2811: Status 404 returned error can't find the container with id 1c7685ba8802d99c3692fadac4224501def05e2a4dd71389d4ce9b74693b2811 Oct 31 04:20:17.181608 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.175271 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44xck\" (UniqueName: \"kubernetes.io/projected/db10a00c-b537-4c62-b9aa-b98dc5577720-kube-api-access-44xck\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp\" (UID: \"db10a00c-b537-4c62-b9aa-b98dc5577720\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp" Oct 31 04:20:17.181608 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.175890 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" Oct 31 04:20:17.235740 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.205885 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8d33677-060d-418a-8806-84eeb0c17ecb-credential-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r\" (UID: \"c8d33677-060d-418a-8806-84eeb0c17ecb\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r" Oct 31 04:20:17.235740 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.205935 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8d33677-060d-418a-8806-84eeb0c17ecb-workload-certs\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r\" (UID: \"c8d33677-060d-418a-8806-84eeb0c17ecb\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r" Oct 31 04:20:17.235740 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.205959 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8d33677-060d-418a-8806-84eeb0c17ecb-istio-podinfo\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r\" (UID: \"c8d33677-060d-418a-8806-84eeb0c17ecb\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r" Oct 31 04:20:17.235740 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.205989 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxx9f\" (UniqueName: \"kubernetes.io/projected/c8d33677-060d-418a-8806-84eeb0c17ecb-kube-api-access-bxx9f\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r\" (UID: \"c8d33677-060d-418a-8806-84eeb0c17ecb\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r" Oct 31 04:20:17.235740 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.206167 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8d33677-060d-418a-8806-84eeb0c17ecb-workload-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r\" (UID: \"c8d33677-060d-418a-8806-84eeb0c17ecb\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r" Oct 31 04:20:17.235740 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.206708 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8d33677-060d-418a-8806-84eeb0c17ecb-workload-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r\" (UID: \"c8d33677-060d-418a-8806-84eeb0c17ecb\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r" Oct 31 04:20:17.235740 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.207598 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8d33677-060d-418a-8806-84eeb0c17ecb-credential-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r\" (UID: \"c8d33677-060d-418a-8806-84eeb0c17ecb\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r" Oct 31 04:20:17.235740 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.207834 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8d33677-060d-418a-8806-84eeb0c17ecb-workload-certs\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r\" (UID: \"c8d33677-060d-418a-8806-84eeb0c17ecb\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r" Oct 31 04:20:17.282354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.282272 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8d33677-060d-418a-8806-84eeb0c17ecb-istio-data\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r\" (UID: \"c8d33677-060d-418a-8806-84eeb0c17ecb\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r" Oct 31 04:20:17.282586 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.282422 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8d33677-060d-418a-8806-84eeb0c17ecb-istiod-ca-cert\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r\" (UID: \"c8d33677-060d-418a-8806-84eeb0c17ecb\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r" Oct 31 04:20:17.282668 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.282648 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8d33677-060d-418a-8806-84eeb0c17ecb-istio-envoy\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r\" (UID: \"c8d33677-060d-418a-8806-84eeb0c17ecb\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r" Oct 31 04:20:17.282737 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.282703 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8d33677-060d-418a-8806-84eeb0c17ecb-istio-token\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r\" (UID: \"c8d33677-060d-418a-8806-84eeb0c17ecb\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r" Oct 31 04:20:17.285471 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.285426 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8d33677-060d-418a-8806-84eeb0c17ecb-istio-podinfo\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r\" (UID: \"c8d33677-060d-418a-8806-84eeb0c17ecb\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r" Oct 31 04:20:17.342842 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.342790 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8d33677-060d-418a-8806-84eeb0c17ecb-istio-data\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r\" (UID: \"c8d33677-060d-418a-8806-84eeb0c17ecb\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r" Oct 31 04:20:17.357983 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.343758 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8d33677-060d-418a-8806-84eeb0c17ecb-istiod-ca-cert\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r\" (UID: \"c8d33677-060d-418a-8806-84eeb0c17ecb\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r" Oct 31 04:20:17.361785 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.361755 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" Oct 31 04:20:17.402842 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.371728 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-802ac966-5tzmm"] Oct 31 04:20:17.402842 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.372725 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" Oct 31 04:20:17.402842 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.374288 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp" Oct 31 04:20:17.416887 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.405778 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxx9f\" (UniqueName: \"kubernetes.io/projected/c8d33677-060d-418a-8806-84eeb0c17ecb-kube-api-access-bxx9f\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r\" (UID: \"c8d33677-060d-418a-8806-84eeb0c17ecb\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r" Oct 31 04:20:17.416887 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.407296 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" Oct 31 04:20:17.416887 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.407790 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8d33677-060d-418a-8806-84eeb0c17ecb-istio-token\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r\" (UID: \"c8d33677-060d-418a-8806-84eeb0c17ecb\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r" Oct 31 04:20:17.427844 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.427794 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8d33677-060d-418a-8806-84eeb0c17ecb-istio-envoy\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r\" (UID: \"c8d33677-060d-418a-8806-84eeb0c17ecb\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r" Oct 31 04:20:17.483324 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.483280 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-802ac966-5tzmm" event={"ID":"587549d7-b0e2-4f3d-a956-934ca87749f1","Type":"ContainerStarted","Data":"1c7685ba8802d99c3692fadac4224501def05e2a4dd71389d4ce9b74693b2811"} Oct 31 04:20:17.683376 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.683324 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r" Oct 31 04:20:17.781076 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.781038 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de11517-qf5pb" Oct 31 04:20:17.849642 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.849555 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-146320d5-586748b797-4lhzm" Oct 31 04:20:17.891672 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:20:17.891640 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb10a00c_b537_4c62_b9aa_b98dc5577720.slice/crio-a5b60e8f8c732799366f395e340d2d0ffbc1c8675b273c5c12680489650a3dbb WatchSource:0}: Error finding container a5b60e8f8c732799366f395e340d2d0ffbc1c8675b273c5c12680489650a3dbb: Status 404 returned error can't find the container with id a5b60e8f8c732799366f395e340d2d0ffbc1c8675b273c5c12680489650a3dbb Oct 31 04:20:17.892766 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.892745 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp"] Oct 31 04:20:17.895044 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.895003 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:20:17.895161 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.895095 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:20:17.895220 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.895157 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:20:17.901382 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.901353 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0a65aa5-25f8-4835-bcbd-88236c70410e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0a65aa5-25f8-4835-bcbd-88236c70410e-maas-api-openshift-ca\") pod \"a0a65aa5-25f8-4835-bcbd-88236c70410e\" (UID: \"a0a65aa5-25f8-4835-bcbd-88236c70410e\") " Oct 31 04:20:17.901540 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.901410 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0a65aa5-25f8-4835-bcbd-88236c70410e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0a65aa5-25f8-4835-bcbd-88236c70410e-maas-api-xks-ca\") pod \"a0a65aa5-25f8-4835-bcbd-88236c70410e\" (UID: \"a0a65aa5-25f8-4835-bcbd-88236c70410e\") " Oct 31 04:20:17.901540 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.901525 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0a65aa5-25f8-4835-bcbd-88236c70410e-kube-api-access-csvx9\" (UniqueName: \"kubernetes.io/projected/a0a65aa5-25f8-4835-bcbd-88236c70410e-kube-api-access-csvx9\") pod \"a0a65aa5-25f8-4835-bcbd-88236c70410e\" (UID: \"a0a65aa5-25f8-4835-bcbd-88236c70410e\") " Oct 31 04:20:17.902167 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.902002 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0a65aa5-25f8-4835-bcbd-88236c70410e-maas-api-openshift-ca" pod "a0a65aa5-25f8-4835-bcbd-88236c70410e" (UID: "a0a65aa5-25f8-4835-bcbd-88236c70410e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:20:17.902167 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.902138 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0a65aa5-25f8-4835-bcbd-88236c70410e-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:20:17.904544 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.904513 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0a65aa5-25f8-4835-bcbd-88236c70410e-maas-api-xks-ca" pod "a0a65aa5-25f8-4835-bcbd-88236c70410e" (UID: "a0a65aa5-25f8-4835-bcbd-88236c70410e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:20:17.904690 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:17.904617 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0a65aa5-25f8-4835-bcbd-88236c70410e-kube-api-access-csvx9" pod "a0a65aa5-25f8-4835-bcbd-88236c70410e" (UID: "a0a65aa5-25f8-4835-bcbd-88236c70410e"). InnerVolumeSpecName "kube-api-access-csvx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:20:18.013591 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:18.013552 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r"] Oct 31 04:20:18.014916 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:18.014887 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csvx9\" (UniqueName: \"kubernetes.io/projected/a0a65aa5-25f8-4835-bcbd-88236c70410e-kube-api-access-csvx9\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:20:18.015114 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:18.015095 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0a65aa5-25f8-4835-bcbd-88236c70410e-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:20:18.021973 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:18.021929 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:20:18.022105 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:18.022022 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:20:18.022105 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:18.022066 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:20:18.510023 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:18.509986 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de11517-qf5pb" event={"ID":"a0a65aa5-25f8-4835-bcbd-88236c70410e","Type":"ContainerDied","Data":"12c6b392a40d085977e3d4afadb9644c798cdfcd62cc44d33530e8d41f39841d"} Oct 31 04:20:18.510262 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:18.510246 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12c6b392a40d085977e3d4afadb9644c798cdfcd62cc44d33530e8d41f39841d" Oct 31 04:20:18.510550 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:18.510536 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de11517-qf5pb" Oct 31 04:20:18.538069 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:18.529558 5019 generic.go:356] "Generic (PLEG): container finished" podID="587549d7-b0e2-4f3d-a956-934ca87749f1" containerID="ec20c5e6ea49c5521970d81c3bada96aa3d3137a5234c404467fde719c920a36" exitCode=0 Oct 31 04:20:18.538069 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:18.537518 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-802ac966-5tzmm" event={"ID":"587549d7-b0e2-4f3d-a956-934ca87749f1","Type":"ContainerDied","Data":"ec20c5e6ea49c5521970d81c3bada96aa3d3137a5234c404467fde719c920a36"} Oct 31 04:20:18.560859 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:18.560379 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp" event={"ID":"db10a00c-b537-4c62-b9aa-b98dc5577720","Type":"ContainerStarted","Data":"c3787789dee95da24e43f92f2211b942ec68369808ab5c10673a61e7c2dfa06b"} Oct 31 04:20:18.561125 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:18.561101 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp" event={"ID":"db10a00c-b537-4c62-b9aa-b98dc5577720","Type":"ContainerStarted","Data":"a5b60e8f8c732799366f395e340d2d0ffbc1c8675b273c5c12680489650a3dbb"} Oct 31 04:20:18.563272 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:18.563243 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r" event={"ID":"c8d33677-060d-418a-8806-84eeb0c17ecb","Type":"ContainerStarted","Data":"73465b304bdfbaa1a725a5f42d4c53f10db63b82c7a62b8cf5b914da40a10521"} Oct 31 04:20:18.563447 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:18.563289 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r" event={"ID":"c8d33677-060d-418a-8806-84eeb0c17ecb","Type":"ContainerStarted","Data":"b6f8e5d498789a581fc87bb44136e2428377bf070635ef30eb614f66322f91f9"} Oct 31 04:20:18.633644 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:18.633572 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp" podStartSLOduration=5.633548763 podStartE2EDuration="5.633548763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:20:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:20:18.630666751 +0000 UTC m=+973.387004805" watchObservedRunningTime="2026-10-31 04:20:18.633548763 +0000 UTC m=+973.389886804" Oct 31 04:20:18.701418 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:18.698522 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r" Oct 31 04:20:18.850010 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:18.849947 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r" podStartSLOduration=5.849930714 podStartE2EDuration="5.849930714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:20:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:20:18.720056664 +0000 UTC m=+973.476394706" watchObservedRunningTime="2026-10-31 04:20:18.849930714 +0000 UTC m=+973.606268754" Oct 31 04:20:18.862306 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:18.862274 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-146320d5-65b76f4b67-fnz6w" Oct 31 04:20:19.384357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:19.378150 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 31 04:20:19.384357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:19.378217 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp" podUID="db10a00c-b537-4c62-b9aa-b98dc5577720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 31 04:20:19.384357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:19.378671 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp" Oct 31 04:20:19.717003 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:19.716968 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 31 04:20:19.717733 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:19.717694 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r" podUID="c8d33677-060d-418a-8806-84eeb0c17ecb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 31 04:20:20.068023 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:20.067996 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-802ac966-5tzmm" Oct 31 04:20:20.224213 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:20.224170 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/587549d7-b0e2-4f3d-a956-934ca87749f1-kube-api-access-8jsp6\" (UniqueName: \"kubernetes.io/projected/587549d7-b0e2-4f3d-a956-934ca87749f1-kube-api-access-8jsp6\") pod \"587549d7-b0e2-4f3d-a956-934ca87749f1\" (UID: \"587549d7-b0e2-4f3d-a956-934ca87749f1\") " Oct 31 04:20:20.228671 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:20.228636 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/587549d7-b0e2-4f3d-a956-934ca87749f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/587549d7-b0e2-4f3d-a956-934ca87749f1-maas-api-xks-ca\") pod \"587549d7-b0e2-4f3d-a956-934ca87749f1\" (UID: \"587549d7-b0e2-4f3d-a956-934ca87749f1\") " Oct 31 04:20:20.228905 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:20.228884 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/587549d7-b0e2-4f3d-a956-934ca87749f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/587549d7-b0e2-4f3d-a956-934ca87749f1-maas-api-openshift-ca\") pod \"587549d7-b0e2-4f3d-a956-934ca87749f1\" (UID: \"587549d7-b0e2-4f3d-a956-934ca87749f1\") " Oct 31 04:20:20.253417 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:20.253375 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/587549d7-b0e2-4f3d-a956-934ca87749f1-maas-api-openshift-ca" pod "587549d7-b0e2-4f3d-a956-934ca87749f1" (UID: "587549d7-b0e2-4f3d-a956-934ca87749f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:20:20.270409 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:20.266635 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/587549d7-b0e2-4f3d-a956-934ca87749f1-kube-api-access-8jsp6" pod "587549d7-b0e2-4f3d-a956-934ca87749f1" (UID: "587549d7-b0e2-4f3d-a956-934ca87749f1"). InnerVolumeSpecName "kube-api-access-8jsp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:20:20.353954 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:20.353910 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/587549d7-b0e2-4f3d-a956-934ca87749f1-maas-api-xks-ca" pod "587549d7-b0e2-4f3d-a956-934ca87749f1" (UID: "587549d7-b0e2-4f3d-a956-934ca87749f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:20:20.355621 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:20.355588 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/587549d7-b0e2-4f3d-a956-934ca87749f1-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:20:20.355787 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:20.355629 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/587549d7-b0e2-4f3d-a956-934ca87749f1-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:20:20.355787 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:20.355646 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jsp6\" (UniqueName: \"kubernetes.io/projected/587549d7-b0e2-4f3d-a956-934ca87749f1-kube-api-access-8jsp6\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:20:20.389371 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:20.379062 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 31 04:20:20.389371 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:20.379132 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp" podUID="db10a00c-b537-4c62-b9aa-b98dc5577720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 31 04:20:20.665475 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:20.665355 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-802ac966-5tzmm" Oct 31 04:20:20.667238 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:20.667195 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-802ac966-5tzmm" event={"ID":"587549d7-b0e2-4f3d-a956-934ca87749f1","Type":"ContainerDied","Data":"1c7685ba8802d99c3692fadac4224501def05e2a4dd71389d4ce9b74693b2811"} Oct 31 04:20:20.667453 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:20.667264 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c7685ba8802d99c3692fadac4224501def05e2a4dd71389d4ce9b74693b2811" Oct 31 04:20:20.706354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:20.700834 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 31 04:20:20.706354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:20.700902 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r" podUID="c8d33677-060d-418a-8806-84eeb0c17ecb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 31 04:20:21.237942 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:21.237905 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64"] Oct 31 04:20:21.239499 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:21.239469 5019 cpu_manager.go:414] "containerMap: removing container" podUID="a0a65aa5-25f8-4835-bcbd-88236c70410e" containerName="revoke-keys" Oct 31 04:20:21.239679 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:21.239665 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0a65aa5-25f8-4835-bcbd-88236c70410e" containerName="revoke-keys" Oct 31 04:20:21.239810 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:21.239800 5019 cpu_manager.go:414] "containerMap: removing container" podUID="587549d7-b0e2-4f3d-a956-934ca87749f1" containerName="revoke-keys" Oct 31 04:20:21.239877 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:21.239868 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="587549d7-b0e2-4f3d-a956-934ca87749f1" containerName="revoke-keys" Oct 31 04:20:21.240136 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:21.240125 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="587549d7-b0e2-4f3d-a956-934ca87749f1" containerName="revoke-keys" Oct 31 04:20:21.240218 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:21.240209 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0a65aa5-25f8-4835-bcbd-88236c70410e" containerName="revoke-keys" Oct 31 04:20:21.245058 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:21.245028 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64" Oct 31 04:20:21.253368 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:21.253087 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64"] Oct 31 04:20:21.393414 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:21.386909 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 31 04:20:21.393414 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:21.392456 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp" podUID="db10a00c-b537-4c62-b9aa-b98dc5577720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 31 04:20:21.430395 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:21.429794 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/735acd81-e3db-44fa-aada-9632da8c3e68-istio-envoy\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64\" (UID: \"735acd81-e3db-44fa-aada-9632da8c3e68\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64" Oct 31 04:20:21.430395 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:21.429896 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/735acd81-e3db-44fa-aada-9632da8c3e68-istiod-ca-cert\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64\" (UID: \"735acd81-e3db-44fa-aada-9632da8c3e68\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64" Oct 31 04:20:21.430395 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:21.429961 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/735acd81-e3db-44fa-aada-9632da8c3e68-workload-certs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64\" (UID: \"735acd81-e3db-44fa-aada-9632da8c3e68\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64" Oct 31 04:20:21.430395 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:21.430044 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/735acd81-e3db-44fa-aada-9632da8c3e68-credential-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64\" (UID: \"735acd81-e3db-44fa-aada-9632da8c3e68\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64" Oct 31 04:20:21.430395 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:21.430136 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/735acd81-e3db-44fa-aada-9632da8c3e68-istio-token\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64\" (UID: \"735acd81-e3db-44fa-aada-9632da8c3e68\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64" Oct 31 04:20:21.430395 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:21.430162 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cznsb\" (UniqueName: \"kubernetes.io/projected/735acd81-e3db-44fa-aada-9632da8c3e68-kube-api-access-cznsb\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64\" (UID: \"735acd81-e3db-44fa-aada-9632da8c3e68\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64" Oct 31 04:20:21.430395 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:21.430275 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/735acd81-e3db-44fa-aada-9632da8c3e68-workload-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64\" (UID: \"735acd81-e3db-44fa-aada-9632da8c3e68\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64" Oct 31 04:20:21.430395 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:21.430302 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/735acd81-e3db-44fa-aada-9632da8c3e68-istio-data\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64\" (UID: \"735acd81-e3db-44fa-aada-9632da8c3e68\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64" Oct 31 04:20:21.430395 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:21.430387 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/735acd81-e3db-44fa-aada-9632da8c3e68-istio-podinfo\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64\" (UID: \"735acd81-e3db-44fa-aada-9632da8c3e68\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64" Oct 31 04:20:21.531793 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:21.531677 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/735acd81-e3db-44fa-aada-9632da8c3e68-istiod-ca-cert\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64\" (UID: \"735acd81-e3db-44fa-aada-9632da8c3e68\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64" Oct 31 04:20:21.531793 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:21.531769 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/735acd81-e3db-44fa-aada-9632da8c3e68-workload-certs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64\" (UID: \"735acd81-e3db-44fa-aada-9632da8c3e68\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64" Oct 31 04:20:21.532020 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:21.531859 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/735acd81-e3db-44fa-aada-9632da8c3e68-credential-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64\" (UID: \"735acd81-e3db-44fa-aada-9632da8c3e68\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64" Oct 31 04:20:21.532298 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:21.532271 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/735acd81-e3db-44fa-aada-9632da8c3e68-credential-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64\" (UID: \"735acd81-e3db-44fa-aada-9632da8c3e68\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64" Oct 31 04:20:21.532366 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:21.532342 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/735acd81-e3db-44fa-aada-9632da8c3e68-istio-token\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64\" (UID: \"735acd81-e3db-44fa-aada-9632da8c3e68\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64" Oct 31 04:20:21.532406 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:21.532379 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cznsb\" (UniqueName: \"kubernetes.io/projected/735acd81-e3db-44fa-aada-9632da8c3e68-kube-api-access-cznsb\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64\" (UID: \"735acd81-e3db-44fa-aada-9632da8c3e68\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64" Oct 31 04:20:21.532534 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:21.532492 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/735acd81-e3db-44fa-aada-9632da8c3e68-workload-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64\" (UID: \"735acd81-e3db-44fa-aada-9632da8c3e68\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64" Oct 31 04:20:21.532534 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:21.532522 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/735acd81-e3db-44fa-aada-9632da8c3e68-istio-data\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64\" (UID: \"735acd81-e3db-44fa-aada-9632da8c3e68\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64" Oct 31 04:20:21.532612 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:21.532588 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/735acd81-e3db-44fa-aada-9632da8c3e68-istio-podinfo\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64\" (UID: \"735acd81-e3db-44fa-aada-9632da8c3e68\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64" Oct 31 04:20:21.532672 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:21.532657 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/735acd81-e3db-44fa-aada-9632da8c3e68-istio-envoy\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64\" (UID: \"735acd81-e3db-44fa-aada-9632da8c3e68\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64" Oct 31 04:20:21.532709 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:21.532683 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/735acd81-e3db-44fa-aada-9632da8c3e68-istiod-ca-cert\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64\" (UID: \"735acd81-e3db-44fa-aada-9632da8c3e68\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64" Oct 31 04:20:21.533140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:21.533115 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/735acd81-e3db-44fa-aada-9632da8c3e68-istio-data\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64\" (UID: \"735acd81-e3db-44fa-aada-9632da8c3e68\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64" Oct 31 04:20:21.533253 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:21.533179 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/735acd81-e3db-44fa-aada-9632da8c3e68-workload-certs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64\" (UID: \"735acd81-e3db-44fa-aada-9632da8c3e68\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64" Oct 31 04:20:21.533611 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:21.533513 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/735acd81-e3db-44fa-aada-9632da8c3e68-workload-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64\" (UID: \"735acd81-e3db-44fa-aada-9632da8c3e68\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64" Oct 31 04:20:21.535450 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:21.535418 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/735acd81-e3db-44fa-aada-9632da8c3e68-istio-envoy\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64\" (UID: \"735acd81-e3db-44fa-aada-9632da8c3e68\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64" Oct 31 04:20:21.535569 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:21.535558 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/735acd81-e3db-44fa-aada-9632da8c3e68-istio-podinfo\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64\" (UID: \"735acd81-e3db-44fa-aada-9632da8c3e68\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64" Oct 31 04:20:21.556553 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:21.556511 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/735acd81-e3db-44fa-aada-9632da8c3e68-istio-token\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64\" (UID: \"735acd81-e3db-44fa-aada-9632da8c3e68\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64" Oct 31 04:20:21.556726 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:21.556698 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cznsb\" (UniqueName: \"kubernetes.io/projected/735acd81-e3db-44fa-aada-9632da8c3e68-kube-api-access-cznsb\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64\" (UID: \"735acd81-e3db-44fa-aada-9632da8c3e68\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64" Oct 31 04:20:21.568438 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:21.568141 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64" Oct 31 04:20:21.714359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:21.701868 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 31 04:20:21.714359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:21.701946 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r" podUID="c8d33677-060d-418a-8806-84eeb0c17ecb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 31 04:20:22.042877 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:22.042820 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64"] Oct 31 04:20:22.047817 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:20:22.047745 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod735acd81_e3db_44fa_aada_9632da8c3e68.slice/crio-6b7ed979b83ca83615f7ab66f8db6585fd2141340c472a246aeef45519fc861a WatchSource:0}: Error finding container 6b7ed979b83ca83615f7ab66f8db6585fd2141340c472a246aeef45519fc861a: Status 404 returned error can't find the container with id 6b7ed979b83ca83615f7ab66f8db6585fd2141340c472a246aeef45519fc861a Oct 31 04:20:22.050685 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:22.050642 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:20:22.050813 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:22.050721 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:20:22.050813 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:22.050755 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:20:22.413306 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:22.413205 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 31 04:20:22.422917 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:22.422859 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp" podUID="db10a00c-b537-4c62-b9aa-b98dc5577720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 31 04:20:22.757365 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:22.747958 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 31 04:20:22.757365 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:22.748024 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r" podUID="c8d33677-060d-418a-8806-84eeb0c17ecb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 31 04:20:22.864379 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:22.849865 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64" event={"ID":"735acd81-e3db-44fa-aada-9632da8c3e68","Type":"ContainerStarted","Data":"32bc2aebe88b04bc28ea6d7269774a0b5354f0f72da4bc4d9fe5892be2191eff"} Oct 31 04:20:22.864379 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:22.849949 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64" event={"ID":"735acd81-e3db-44fa-aada-9632da8c3e68","Type":"ContainerStarted","Data":"6b7ed979b83ca83615f7ab66f8db6585fd2141340c472a246aeef45519fc861a"} Oct 31 04:20:22.864379 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:20:22.857430 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:20:22.918371 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:22.916484 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64" podStartSLOduration=1.916458202 podStartE2EDuration="1.916458202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:20:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:20:22.899877925 +0000 UTC m=+977.656215966" watchObservedRunningTime="2026-10-31 04:20:22.916458202 +0000 UTC m=+977.672796246" Oct 31 04:20:23.385359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:23.379801 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 31 04:20:23.385359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:23.379872 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp" podUID="db10a00c-b537-4c62-b9aa-b98dc5577720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 31 04:20:23.572353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:23.568833 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64" Oct 31 04:20:23.572353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:23.570390 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 31 04:20:23.572353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:23.570450 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64" podUID="735acd81-e3db-44fa-aada-9632da8c3e68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 31 04:20:23.690416 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:23.689887 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 31 04:20:23.690416 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:23.689956 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r" podUID="c8d33677-060d-418a-8806-84eeb0c17ecb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 31 04:20:23.753926 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:23.753890 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt"] Oct 31 04:20:23.764387 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:23.763298 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt" Oct 31 04:20:23.770370 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:23.767740 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc"] Oct 31 04:20:23.791063 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:23.791021 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-istio-envoy\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt\" (UID: \"56aad6d5-8f29-4414-a5e9-f7cb565f8e57\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt" Oct 31 04:20:23.791317 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:23.791287 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-istio-data\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt\" (UID: \"56aad6d5-8f29-4414-a5e9-f7cb565f8e57\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt" Oct 31 04:20:23.791705 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:23.791684 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-istio-token\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt\" (UID: \"56aad6d5-8f29-4414-a5e9-f7cb565f8e57\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt" Oct 31 04:20:23.791874 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:23.791857 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-credential-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt\" (UID: \"56aad6d5-8f29-4414-a5e9-f7cb565f8e57\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt" Oct 31 04:20:23.792010 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:23.791996 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-istio-podinfo\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt\" (UID: \"56aad6d5-8f29-4414-a5e9-f7cb565f8e57\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt" Oct 31 04:20:23.792105 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:23.792091 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-istiod-ca-cert\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt\" (UID: \"56aad6d5-8f29-4414-a5e9-f7cb565f8e57\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt" Oct 31 04:20:23.805839 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:23.792423 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc" Oct 31 04:20:23.806132 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:23.806091 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8gd9\" (UniqueName: \"kubernetes.io/projected/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-kube-api-access-j8gd9\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt\" (UID: \"56aad6d5-8f29-4414-a5e9-f7cb565f8e57\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt" Oct 31 04:20:23.806318 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:23.806302 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-workload-certs\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt\" (UID: \"56aad6d5-8f29-4414-a5e9-f7cb565f8e57\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt" Oct 31 04:20:23.806786 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:23.806761 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-workload-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt\" (UID: \"56aad6d5-8f29-4414-a5e9-f7cb565f8e57\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt" Oct 31 04:20:23.860390 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:23.858254 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt"] Oct 31 04:20:23.963374 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:23.963237 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-workload-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt\" (UID: \"56aad6d5-8f29-4414-a5e9-f7cb565f8e57\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt" Oct 31 04:20:23.963709 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:23.963680 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efc828fc-91bd-43f3-943d-144c3d9dc1b1-istio-envoy\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc\" (UID: \"efc828fc-91bd-43f3-943d-144c3d9dc1b1\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc" Oct 31 04:20:23.963856 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:23.963839 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efc828fc-91bd-43f3-943d-144c3d9dc1b1-istio-podinfo\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc\" (UID: \"efc828fc-91bd-43f3-943d-144c3d9dc1b1\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc" Oct 31 04:20:23.964063 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:23.964042 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-istio-envoy\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt\" (UID: \"56aad6d5-8f29-4414-a5e9-f7cb565f8e57\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt" Oct 31 04:20:23.964173 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:23.964157 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-istio-data\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt\" (UID: \"56aad6d5-8f29-4414-a5e9-f7cb565f8e57\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt" Oct 31 04:20:23.964292 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:23.964278 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-istio-token\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt\" (UID: \"56aad6d5-8f29-4414-a5e9-f7cb565f8e57\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt" Oct 31 04:20:23.964479 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:23.964461 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efc828fc-91bd-43f3-943d-144c3d9dc1b1-istio-token\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc\" (UID: \"efc828fc-91bd-43f3-943d-144c3d9dc1b1\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc" Oct 31 04:20:23.964622 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:23.964603 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-credential-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt\" (UID: \"56aad6d5-8f29-4414-a5e9-f7cb565f8e57\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt" Oct 31 04:20:23.964738 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:23.964722 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbxb2\" (UniqueName: \"kubernetes.io/projected/efc828fc-91bd-43f3-943d-144c3d9dc1b1-kube-api-access-tbxb2\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc\" (UID: \"efc828fc-91bd-43f3-943d-144c3d9dc1b1\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc" Oct 31 04:20:23.964858 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:23.964840 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-istio-podinfo\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt\" (UID: \"56aad6d5-8f29-4414-a5e9-f7cb565f8e57\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt" Oct 31 04:20:23.964962 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:23.964948 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-istiod-ca-cert\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt\" (UID: \"56aad6d5-8f29-4414-a5e9-f7cb565f8e57\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt" Oct 31 04:20:23.965151 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:23.965071 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc828fc-91bd-43f3-943d-144c3d9dc1b1-credential-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc\" (UID: \"efc828fc-91bd-43f3-943d-144c3d9dc1b1\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc" Oct 31 04:20:23.965301 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:23.965282 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8gd9\" (UniqueName: \"kubernetes.io/projected/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-kube-api-access-j8gd9\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt\" (UID: \"56aad6d5-8f29-4414-a5e9-f7cb565f8e57\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt" Oct 31 04:20:23.965443 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:23.965422 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-workload-certs\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt\" (UID: \"56aad6d5-8f29-4414-a5e9-f7cb565f8e57\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt" Oct 31 04:20:23.965554 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:23.965538 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efc828fc-91bd-43f3-943d-144c3d9dc1b1-istiod-ca-cert\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc\" (UID: \"efc828fc-91bd-43f3-943d-144c3d9dc1b1\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc" Oct 31 04:20:23.965730 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:23.965677 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc828fc-91bd-43f3-943d-144c3d9dc1b1-workload-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc\" (UID: \"efc828fc-91bd-43f3-943d-144c3d9dc1b1\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc" Oct 31 04:20:23.965946 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:23.965926 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efc828fc-91bd-43f3-943d-144c3d9dc1b1-istio-data\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc\" (UID: \"efc828fc-91bd-43f3-943d-144c3d9dc1b1\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc" Oct 31 04:20:23.966093 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:23.966078 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efc828fc-91bd-43f3-943d-144c3d9dc1b1-workload-certs\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc\" (UID: \"efc828fc-91bd-43f3-943d-144c3d9dc1b1\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc" Oct 31 04:20:23.967016 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:23.966989 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-workload-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt\" (UID: \"56aad6d5-8f29-4414-a5e9-f7cb565f8e57\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt" Oct 31 04:20:23.970898 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:23.969396 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-istiod-ca-cert\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt\" (UID: \"56aad6d5-8f29-4414-a5e9-f7cb565f8e57\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt" Oct 31 04:20:23.970898 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:23.969693 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-istio-data\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt\" (UID: \"56aad6d5-8f29-4414-a5e9-f7cb565f8e57\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt" Oct 31 04:20:23.970898 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:23.970367 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-credential-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt\" (UID: \"56aad6d5-8f29-4414-a5e9-f7cb565f8e57\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt" Oct 31 04:20:24.035462 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:23.971592 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-workload-certs\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt\" (UID: \"56aad6d5-8f29-4414-a5e9-f7cb565f8e57\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt" Oct 31 04:20:24.140371 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:24.054459 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc"] Oct 31 04:20:24.140371 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:24.066566 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8gd9\" (UniqueName: \"kubernetes.io/projected/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-kube-api-access-j8gd9\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt\" (UID: \"56aad6d5-8f29-4414-a5e9-f7cb565f8e57\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt" Oct 31 04:20:24.140371 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:24.067677 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc828fc-91bd-43f3-943d-144c3d9dc1b1-credential-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc\" (UID: \"efc828fc-91bd-43f3-943d-144c3d9dc1b1\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc" Oct 31 04:20:24.140371 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:24.067726 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efc828fc-91bd-43f3-943d-144c3d9dc1b1-istiod-ca-cert\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc\" (UID: \"efc828fc-91bd-43f3-943d-144c3d9dc1b1\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc" Oct 31 04:20:24.140371 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:24.067759 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc828fc-91bd-43f3-943d-144c3d9dc1b1-workload-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc\" (UID: \"efc828fc-91bd-43f3-943d-144c3d9dc1b1\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc" Oct 31 04:20:24.140371 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:24.067838 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efc828fc-91bd-43f3-943d-144c3d9dc1b1-istio-data\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc\" (UID: \"efc828fc-91bd-43f3-943d-144c3d9dc1b1\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc" Oct 31 04:20:24.140371 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:24.067892 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efc828fc-91bd-43f3-943d-144c3d9dc1b1-workload-certs\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc\" (UID: \"efc828fc-91bd-43f3-943d-144c3d9dc1b1\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc" Oct 31 04:20:24.140371 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:24.068068 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efc828fc-91bd-43f3-943d-144c3d9dc1b1-istio-envoy\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc\" (UID: \"efc828fc-91bd-43f3-943d-144c3d9dc1b1\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc" Oct 31 04:20:24.140371 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:24.068095 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efc828fc-91bd-43f3-943d-144c3d9dc1b1-istio-podinfo\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc\" (UID: \"efc828fc-91bd-43f3-943d-144c3d9dc1b1\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc" Oct 31 04:20:24.140371 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:24.068245 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efc828fc-91bd-43f3-943d-144c3d9dc1b1-istio-token\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc\" (UID: \"efc828fc-91bd-43f3-943d-144c3d9dc1b1\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc" Oct 31 04:20:24.140371 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:24.068293 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbxb2\" (UniqueName: \"kubernetes.io/projected/efc828fc-91bd-43f3-943d-144c3d9dc1b1-kube-api-access-tbxb2\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc\" (UID: \"efc828fc-91bd-43f3-943d-144c3d9dc1b1\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc" Oct 31 04:20:24.140371 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:24.069252 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc828fc-91bd-43f3-943d-144c3d9dc1b1-credential-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc\" (UID: \"efc828fc-91bd-43f3-943d-144c3d9dc1b1\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc" Oct 31 04:20:24.140371 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:24.073165 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efc828fc-91bd-43f3-943d-144c3d9dc1b1-istiod-ca-cert\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc\" (UID: \"efc828fc-91bd-43f3-943d-144c3d9dc1b1\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc" Oct 31 04:20:24.140371 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:24.073515 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc828fc-91bd-43f3-943d-144c3d9dc1b1-workload-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc\" (UID: \"efc828fc-91bd-43f3-943d-144c3d9dc1b1\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc" Oct 31 04:20:24.140371 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:24.073785 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efc828fc-91bd-43f3-943d-144c3d9dc1b1-istio-data\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc\" (UID: \"efc828fc-91bd-43f3-943d-144c3d9dc1b1\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc" Oct 31 04:20:24.140371 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:24.074025 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efc828fc-91bd-43f3-943d-144c3d9dc1b1-workload-certs\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc\" (UID: \"efc828fc-91bd-43f3-943d-144c3d9dc1b1\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc" Oct 31 04:20:24.140371 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:24.119428 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbxb2\" (UniqueName: \"kubernetes.io/projected/efc828fc-91bd-43f3-943d-144c3d9dc1b1-kube-api-access-tbxb2\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc\" (UID: \"efc828fc-91bd-43f3-943d-144c3d9dc1b1\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc" Oct 31 04:20:24.199401 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:24.173989 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efc828fc-91bd-43f3-943d-144c3d9dc1b1-istio-podinfo\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc\" (UID: \"efc828fc-91bd-43f3-943d-144c3d9dc1b1\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc" Oct 31 04:20:24.262792 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:24.260141 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-istio-token\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt\" (UID: \"56aad6d5-8f29-4414-a5e9-f7cb565f8e57\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt" Oct 31 04:20:24.266963 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:24.266874 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efc828fc-91bd-43f3-943d-144c3d9dc1b1-istio-token\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc\" (UID: \"efc828fc-91bd-43f3-943d-144c3d9dc1b1\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc" Oct 31 04:20:24.320234 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:24.268956 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-istio-podinfo\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt\" (UID: \"56aad6d5-8f29-4414-a5e9-f7cb565f8e57\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt" Oct 31 04:20:24.358384 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:24.349500 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-istio-envoy\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt\" (UID: \"56aad6d5-8f29-4414-a5e9-f7cb565f8e57\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt" Oct 31 04:20:24.358384 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:24.349585 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efc828fc-91bd-43f3-943d-144c3d9dc1b1-istio-envoy\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc\" (UID: \"efc828fc-91bd-43f3-943d-144c3d9dc1b1\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc" Oct 31 04:20:24.415359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:24.411015 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 31 04:20:24.415359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:24.411082 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp" podUID="db10a00c-b537-4c62-b9aa-b98dc5577720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 31 04:20:24.445360 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:24.443731 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt" Oct 31 04:20:24.633299 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:24.633264 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 31 04:20:24.633905 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:24.633874 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64" podUID="735acd81-e3db-44fa-aada-9632da8c3e68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 31 04:20:24.634237 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:24.634009 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc" Oct 31 04:20:24.685863 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:24.685739 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 31 04:20:24.685863 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:24.685800 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r" podUID="c8d33677-060d-418a-8806-84eeb0c17ecb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 31 04:20:24.918938 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:20:24.918905 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56aad6d5_8f29_4414_a5e9_f7cb565f8e57.slice/crio-01ceb2f396e1ab3d16e40b8b9afca53323d7676513d1f9d60e8a444ea37ba994 WatchSource:0}: Error finding container 01ceb2f396e1ab3d16e40b8b9afca53323d7676513d1f9d60e8a444ea37ba994: Status 404 returned error can't find the container with id 01ceb2f396e1ab3d16e40b8b9afca53323d7676513d1f9d60e8a444ea37ba994 Oct 31 04:20:24.923361 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:24.922674 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:20:24.923361 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:24.922768 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:20:24.923361 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:24.922811 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:20:24.928648 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:24.928087 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt"] Oct 31 04:20:25.123397 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:25.119023 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc"] Oct 31 04:20:25.149205 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:25.149156 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:20:25.149315 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:25.149262 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:20:25.149374 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:25.149309 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:20:25.298068 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:25.297964 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc" event={"ID":"efc828fc-91bd-43f3-943d-144c3d9dc1b1","Type":"ContainerStarted","Data":"2e2bf00e73f26f882b6c6e2496e31d79ea2cac8f93f0a3289df69e5f12776f33"} Oct 31 04:20:25.301577 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:25.301538 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt" event={"ID":"56aad6d5-8f29-4414-a5e9-f7cb565f8e57","Type":"ContainerStarted","Data":"5329525c6f8da3655e4eb5c265a21d8c999ecc1bf6582145b10216c9d042f097"} Oct 31 04:20:25.301835 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:25.301796 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt" event={"ID":"56aad6d5-8f29-4414-a5e9-f7cb565f8e57","Type":"ContainerStarted","Data":"01ceb2f396e1ab3d16e40b8b9afca53323d7676513d1f9d60e8a444ea37ba994"} Oct 31 04:20:25.358326 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:25.357058 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt" podStartSLOduration=2.357035881 podStartE2EDuration="2.357035881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:20:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:20:25.35349298 +0000 UTC m=+980.109831017" watchObservedRunningTime="2026-10-31 04:20:25.357035881 +0000 UTC m=+980.113373918" Oct 31 04:20:25.380793 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:25.380742 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 31 04:20:25.380987 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:25.380815 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp" podUID="db10a00c-b537-4c62-b9aa-b98dc5577720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 31 04:20:25.467069 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:25.467033 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt" Oct 31 04:20:25.569355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:25.569234 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 31 04:20:25.569524 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:25.569313 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64" podUID="735acd81-e3db-44fa-aada-9632da8c3e68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 31 04:20:25.686853 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:25.686808 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 31 04:20:25.687530 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:25.687492 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r" podUID="c8d33677-060d-418a-8806-84eeb0c17ecb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 31 04:20:26.471105 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:26.471066 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 31 04:20:26.471384 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:26.471359 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt" podUID="56aad6d5-8f29-4414-a5e9-f7cb565f8e57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 31 04:20:26.499506 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:26.499464 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc" event={"ID":"efc828fc-91bd-43f3-943d-144c3d9dc1b1","Type":"ContainerStarted","Data":"a1ae8aa1b3d05374777fa0e7524e4d32d8947ef2baf5192d5ed132974ed2fa1d"} Oct 31 04:20:26.575989 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:26.575944 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 31 04:20:26.576259 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:26.576234 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64" podUID="735acd81-e3db-44fa-aada-9632da8c3e68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 31 04:20:26.602489 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:26.602414 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc" podStartSLOduration=3.602389889 podStartE2EDuration="3.602389889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:20:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:20:26.588862728 +0000 UTC m=+981.345200769" watchObservedRunningTime="2026-10-31 04:20:26.602389889 +0000 UTC m=+981.358727931" Oct 31 04:20:26.634698 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:26.634662 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc" Oct 31 04:20:26.693359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:26.692838 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 31 04:20:26.693359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:26.692920 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc" podUID="efc828fc-91bd-43f3-943d-144c3d9dc1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 31 04:20:27.050357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:27.048468 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" podUID="e0f4b8c2-14f4-4052-bf54-dd65bf59e413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:27.220858 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:27.220810 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp" Oct 31 04:20:27.223055 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:27.223015 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp" Oct 31 04:20:27.236087 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:27.236056 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp" Oct 31 04:20:27.352610 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:27.352505 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" podUID="dccd15a9-47f5-4bd1-b21b-9d5ca8b05845" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:27.364765 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:27.362734 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r" Oct 31 04:20:27.364765 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:27.364569 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r" Oct 31 04:20:27.373910 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:27.373828 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r" Oct 31 04:20:27.495545 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:27.495471 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" podUID="643433d8-418e-4102-8bc3-cd544932614a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:27.656417 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:27.656137 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" podUID="8e2c7041-ad52-49f4-8b55-d16691f4938f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:27.711499 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:27.705136 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 31 04:20:27.711499 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:27.705204 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt" podUID="56aad6d5-8f29-4414-a5e9-f7cb565f8e57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 31 04:20:27.735011 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:27.734953 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 31 04:20:27.738189 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:27.737930 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 31 04:20:27.738189 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:27.738006 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64" podUID="735acd81-e3db-44fa-aada-9632da8c3e68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 31 04:20:27.738189 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:27.738118 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 31 04:20:27.738189 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:27.738142 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc" podUID="efc828fc-91bd-43f3-943d-144c3d9dc1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 31 04:20:27.753002 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:27.752542 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:27.753988 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:27.753940 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:28.040368 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:27.974222 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": context deadline exceeded" Oct 31 04:20:28.085358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:28.081497 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" podUID="643433d8-418e-4102-8bc3-cd544932614a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:28.085358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:28.081662 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" podUID="e0f4b8c2-14f4-4052-bf54-dd65bf59e413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:28.085358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:28.082193 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" Oct 31 04:20:28.462174 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:28.462057 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 31 04:20:28.462174 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:28.462124 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt" podUID="56aad6d5-8f29-4414-a5e9-f7cb565f8e57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 31 04:20:28.565355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:28.556568 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" Oct 31 04:20:28.593387 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:28.578506 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 31 04:20:28.593387 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:28.578579 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64" podUID="735acd81-e3db-44fa-aada-9632da8c3e68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 31 04:20:28.655201 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:28.655161 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 31 04:20:28.655516 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:28.655487 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc" podUID="efc828fc-91bd-43f3-943d-144c3d9dc1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 31 04:20:29.524017 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:29.523972 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 31 04:20:29.524672 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:29.524634 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt" podUID="56aad6d5-8f29-4414-a5e9-f7cb565f8e57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 31 04:20:29.587783 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:29.587741 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 31 04:20:29.591028 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:29.590984 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64" podUID="735acd81-e3db-44fa-aada-9632da8c3e68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 31 04:20:29.638744 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:29.638704 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 31 04:20:29.639030 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:29.639003 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc" podUID="efc828fc-91bd-43f3-943d-144c3d9dc1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 31 04:20:30.455233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:30.455193 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 31 04:20:30.455594 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:30.455554 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt" podUID="56aad6d5-8f29-4414-a5e9-f7cb565f8e57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 31 04:20:30.569250 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:30.569210 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 31 04:20:30.569836 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:30.569390 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64" podUID="735acd81-e3db-44fa-aada-9632da8c3e68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 31 04:20:30.660956 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:30.660176 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 31 04:20:30.660956 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:30.660251 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc" podUID="efc828fc-91bd-43f3-943d-144c3d9dc1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 31 04:20:31.452351 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:31.452280 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 31 04:20:31.452536 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:31.452390 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt" podUID="56aad6d5-8f29-4414-a5e9-f7cb565f8e57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 31 04:20:31.635017 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:31.634931 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 31 04:20:31.635841 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:31.635798 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc" podUID="efc828fc-91bd-43f3-943d-144c3d9dc1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 31 04:20:32.276359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:32.275428 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8c0086-78b8c6c95-vwbxc"] Oct 31 04:20:32.392066 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:32.392033 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8c0086-78b8c6c95-vwbxc" Oct 31 04:20:32.548420 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:32.437015 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzzv5\" (UniqueName: \"kubernetes.io/projected/43f3a427-7289-484b-bd5d-f304eaf5a247-kube-api-access-zzzv5\") pod \"payload-pre-processing-e2e-api-a-8c0086-78b8c6c95-vwbxc\" (UID: \"43f3a427-7289-484b-bd5d-f304eaf5a247\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8c0086-78b8c6c95-vwbxc" Oct 31 04:20:32.548902 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:32.548852 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43f3a427-7289-484b-bd5d-f304eaf5a247-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-8c0086-78b8c6c95-vwbxc\" (UID: \"43f3a427-7289-484b-bd5d-f304eaf5a247\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8c0086-78b8c6c95-vwbxc" Oct 31 04:20:32.561357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:32.469934 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-8c0086\"" Oct 31 04:20:32.561357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:32.474984 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-8c0086-dockercfg-hd9jv\"" Oct 31 04:20:32.587061 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:32.584158 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 31 04:20:32.587061 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:32.584317 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt" podUID="56aad6d5-8f29-4414-a5e9-f7cb565f8e57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 31 04:20:32.637518 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:32.637463 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:20:32.643662 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:32.643610 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64" podUID="735acd81-e3db-44fa-aada-9632da8c3e68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:32.655790 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:32.655143 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzzv5\" (UniqueName: \"kubernetes.io/projected/43f3a427-7289-484b-bd5d-f304eaf5a247-kube-api-access-zzzv5\") pod \"payload-pre-processing-e2e-api-a-8c0086-78b8c6c95-vwbxc\" (UID: \"43f3a427-7289-484b-bd5d-f304eaf5a247\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8c0086-78b8c6c95-vwbxc" Oct 31 04:20:32.666440 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:32.656213 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43f3a427-7289-484b-bd5d-f304eaf5a247-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-8c0086-78b8c6c95-vwbxc\" (UID: \"43f3a427-7289-484b-bd5d-f304eaf5a247\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8c0086-78b8c6c95-vwbxc" Oct 31 04:20:32.667450 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:32.660911 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 31 04:20:32.667853 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:32.667801 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc" podUID="efc828fc-91bd-43f3-943d-144c3d9dc1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 31 04:20:32.678236 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:32.678187 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43f3a427-7289-484b-bd5d-f304eaf5a247-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-8c0086-78b8c6c95-vwbxc\" (UID: \"43f3a427-7289-484b-bd5d-f304eaf5a247\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8c0086-78b8c6c95-vwbxc" Oct 31 04:20:32.709122 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:32.709083 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8c0086-78b8c6c95-vwbxc"] Oct 31 04:20:33.024838 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:33.024127 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzzv5\" (UniqueName: \"kubernetes.io/projected/43f3a427-7289-484b-bd5d-f304eaf5a247-kube-api-access-zzzv5\") pod \"payload-pre-processing-e2e-api-a-8c0086-78b8c6c95-vwbxc\" (UID: \"43f3a427-7289-484b-bd5d-f304eaf5a247\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8c0086-78b8c6c95-vwbxc" Oct 31 04:20:33.156060 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:33.156018 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64" Oct 31 04:20:33.157821 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:33.157792 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64" Oct 31 04:20:33.163447 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:33.163414 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64" Oct 31 04:20:33.234915 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:33.233942 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8c0086-78b8c6c95-vwbxc" Oct 31 04:20:33.452674 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:33.452613 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 31 04:20:33.453308 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:33.453256 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt" podUID="56aad6d5-8f29-4414-a5e9-f7cb565f8e57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 31 04:20:33.610558 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:33.609081 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8c0086-78b8c6c95-vwbxc"] Oct 31 04:20:33.635849 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:33.635817 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 31 04:20:33.636020 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:33.635882 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc" podUID="efc828fc-91bd-43f3-943d-144c3d9dc1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 31 04:20:33.968621 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:33.968570 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8c0086-78b8c6c95-vwbxc" event={"ID":"43f3a427-7289-484b-bd5d-f304eaf5a247","Type":"ContainerStarted","Data":"9f8bfb886c6c4c292cefd0a63bff3f27eb9c22e0e55054a19949ae2fe0b91821"} Oct 31 04:20:33.969061 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:33.968656 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8c0086-78b8c6c95-vwbxc" Oct 31 04:20:33.969061 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:33.968687 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8c0086-78b8c6c95-vwbxc" event={"ID":"43f3a427-7289-484b-bd5d-f304eaf5a247","Type":"ContainerStarted","Data":"aaf88ef724eb34aaee8772071d3097941ce43ff4eea34cd430d3ec5be7046df1"} Oct 31 04:20:34.010355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:34.009526 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8c0086-78b8c6c95-vwbxc" podStartSLOduration=2.009502793 podStartE2EDuration="2.009502793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:20:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:20:34.008744047 +0000 UTC m=+988.765082111" watchObservedRunningTime="2026-10-31 04:20:34.009502793 +0000 UTC m=+988.765840834" Oct 31 04:20:34.864481 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:34.864442 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc" Oct 31 04:20:34.865434 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:34.865405 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc" Oct 31 04:20:34.866243 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:34.866217 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc" Oct 31 04:20:34.958771 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:34.958731 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt" Oct 31 04:20:34.959102 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:34.959074 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt" Oct 31 04:20:34.960019 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:34.960000 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt" Oct 31 04:20:37.346370 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:37.346307 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9883e4-6b849c5655-vlf6g"] Oct 31 04:20:37.352463 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:37.352429 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9883e4-6b849c5655-vlf6g" Oct 31 04:20:37.373264 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:37.373227 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9883e4-6b849c5655-vlf6g"] Oct 31 04:20:37.376554 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:37.376475 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-9883e4\"" Oct 31 04:20:37.376789 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:37.376766 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-9883e4-dockercfg-79md2\"" Oct 31 04:20:37.529356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:37.522532 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d45c27c-8c7d-4973-bc2f-46beb3b8fdb1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9883e4-6b849c5655-vlf6g\" (UID: \"9d45c27c-8c7d-4973-bc2f-46beb3b8fdb1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9883e4-6b849c5655-vlf6g" Oct 31 04:20:37.529356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:37.522694 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5crm\" (UniqueName: \"kubernetes.io/projected/9d45c27c-8c7d-4973-bc2f-46beb3b8fdb1-kube-api-access-x5crm\") pod \"payload-pre-processing-e2e-api-a-9883e4-6b849c5655-vlf6g\" (UID: \"9d45c27c-8c7d-4973-bc2f-46beb3b8fdb1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9883e4-6b849c5655-vlf6g" Oct 31 04:20:37.648766 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:37.641366 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d45c27c-8c7d-4973-bc2f-46beb3b8fdb1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9883e4-6b849c5655-vlf6g\" (UID: \"9d45c27c-8c7d-4973-bc2f-46beb3b8fdb1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9883e4-6b849c5655-vlf6g" Oct 31 04:20:37.648766 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:37.641444 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5crm\" (UniqueName: \"kubernetes.io/projected/9d45c27c-8c7d-4973-bc2f-46beb3b8fdb1-kube-api-access-x5crm\") pod \"payload-pre-processing-e2e-api-a-9883e4-6b849c5655-vlf6g\" (UID: \"9d45c27c-8c7d-4973-bc2f-46beb3b8fdb1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9883e4-6b849c5655-vlf6g" Oct 31 04:20:37.651030 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:37.650993 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d45c27c-8c7d-4973-bc2f-46beb3b8fdb1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9883e4-6b849c5655-vlf6g\" (UID: \"9d45c27c-8c7d-4973-bc2f-46beb3b8fdb1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9883e4-6b849c5655-vlf6g" Oct 31 04:20:37.666486 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:37.666428 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9883e4-6f4c589fd5-cmtkl"] Oct 31 04:20:37.675504 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:37.675474 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9883e4-6f4c589fd5-cmtkl" Oct 31 04:20:37.688917 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:37.687938 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5crm\" (UniqueName: \"kubernetes.io/projected/9d45c27c-8c7d-4973-bc2f-46beb3b8fdb1-kube-api-access-x5crm\") pod \"payload-pre-processing-e2e-api-a-9883e4-6b849c5655-vlf6g\" (UID: \"9d45c27c-8c7d-4973-bc2f-46beb3b8fdb1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9883e4-6b849c5655-vlf6g" Oct 31 04:20:37.697219 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:37.697182 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9883e4-6f4c589fd5-cmtkl"] Oct 31 04:20:37.742657 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:37.742611 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f4pb\" (UniqueName: \"kubernetes.io/projected/837a88ff-dd11-4ccc-85dc-f8a83c41687b-kube-api-access-9f4pb\") pod \"payload-processing-e2e-api-a-9883e4-6f4c589fd5-cmtkl\" (UID: \"837a88ff-dd11-4ccc-85dc-f8a83c41687b\") " pod="openshift-ingress/payload-processing-e2e-api-a-9883e4-6f4c589fd5-cmtkl" Oct 31 04:20:37.742834 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:37.742741 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/837a88ff-dd11-4ccc-85dc-f8a83c41687b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9883e4-6f4c589fd5-cmtkl\" (UID: \"837a88ff-dd11-4ccc-85dc-f8a83c41687b\") " pod="openshift-ingress/payload-processing-e2e-api-a-9883e4-6f4c589fd5-cmtkl" Oct 31 04:20:37.844144 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:37.844101 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f4pb\" (UniqueName: \"kubernetes.io/projected/837a88ff-dd11-4ccc-85dc-f8a83c41687b-kube-api-access-9f4pb\") pod \"payload-processing-e2e-api-a-9883e4-6f4c589fd5-cmtkl\" (UID: \"837a88ff-dd11-4ccc-85dc-f8a83c41687b\") " pod="openshift-ingress/payload-processing-e2e-api-a-9883e4-6f4c589fd5-cmtkl" Oct 31 04:20:37.844380 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:37.844194 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/837a88ff-dd11-4ccc-85dc-f8a83c41687b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9883e4-6f4c589fd5-cmtkl\" (UID: \"837a88ff-dd11-4ccc-85dc-f8a83c41687b\") " pod="openshift-ingress/payload-processing-e2e-api-a-9883e4-6f4c589fd5-cmtkl" Oct 31 04:20:37.845028 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:37.845006 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/837a88ff-dd11-4ccc-85dc-f8a83c41687b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9883e4-6f4c589fd5-cmtkl\" (UID: \"837a88ff-dd11-4ccc-85dc-f8a83c41687b\") " pod="openshift-ingress/payload-processing-e2e-api-a-9883e4-6f4c589fd5-cmtkl" Oct 31 04:20:37.867840 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:37.867803 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f4pb\" (UniqueName: \"kubernetes.io/projected/837a88ff-dd11-4ccc-85dc-f8a83c41687b-kube-api-access-9f4pb\") pod \"payload-processing-e2e-api-a-9883e4-6f4c589fd5-cmtkl\" (UID: \"837a88ff-dd11-4ccc-85dc-f8a83c41687b\") " pod="openshift-ingress/payload-processing-e2e-api-a-9883e4-6f4c589fd5-cmtkl" Oct 31 04:20:37.978224 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:37.978177 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9883e4-6b849c5655-vlf6g" Oct 31 04:20:38.022371 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:38.021972 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9883e4-6f4c589fd5-cmtkl" Oct 31 04:20:38.341043 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:20:38.340994 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d45c27c_8c7d_4973_bc2f_46beb3b8fdb1.slice/crio-55d4f114e9be5066b37fcc83617cb5804b2d887eb1b51c8457cfe2a32f94c4e4 WatchSource:0}: Error finding container 55d4f114e9be5066b37fcc83617cb5804b2d887eb1b51c8457cfe2a32f94c4e4: Status 404 returned error can't find the container with id 55d4f114e9be5066b37fcc83617cb5804b2d887eb1b51c8457cfe2a32f94c4e4 Oct 31 04:20:38.341375 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:38.341315 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9883e4-6b849c5655-vlf6g"] Oct 31 04:20:38.372446 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:38.372285 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9883e4-6f4c589fd5-cmtkl"] Oct 31 04:20:39.069048 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:39.069010 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9883e4-6b849c5655-vlf6g" event={"ID":"9d45c27c-8c7d-4973-bc2f-46beb3b8fdb1","Type":"ContainerStarted","Data":"d42b537e9017f6a3ae63ed440020b716d68c3f27e152b6f9fac0fabd3bc28b7e"} Oct 31 04:20:39.069229 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:39.069075 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9883e4-6b849c5655-vlf6g" Oct 31 04:20:39.069229 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:39.069096 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9883e4-6b849c5655-vlf6g" event={"ID":"9d45c27c-8c7d-4973-bc2f-46beb3b8fdb1","Type":"ContainerStarted","Data":"55d4f114e9be5066b37fcc83617cb5804b2d887eb1b51c8457cfe2a32f94c4e4"} Oct 31 04:20:39.071405 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:39.071371 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9883e4-6f4c589fd5-cmtkl" event={"ID":"837a88ff-dd11-4ccc-85dc-f8a83c41687b","Type":"ContainerStarted","Data":"8d02e5f1639fe301e34ec975aedf6072492b0d4ac6940b5832398c4d93856e60"} Oct 31 04:20:39.071600 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:39.071426 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9883e4-6f4c589fd5-cmtkl" event={"ID":"837a88ff-dd11-4ccc-85dc-f8a83c41687b","Type":"ContainerStarted","Data":"bbd8d56a6a1ebea792cc7bf1a3c91a4af6c9f8e5afd2ba815b3ddd0e502cb560"} Oct 31 04:20:39.071600 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:39.071455 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-9883e4-6f4c589fd5-cmtkl" Oct 31 04:20:39.113595 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:39.113533 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9883e4-6b849c5655-vlf6g" podStartSLOduration=2.113514169 podStartE2EDuration="2.113514169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:20:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:20:39.109661687 +0000 UTC m=+993.865999727" watchObservedRunningTime="2026-10-31 04:20:39.113514169 +0000 UTC m=+993.869852218" Oct 31 04:20:39.152258 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:39.152196 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-9883e4-6f4c589fd5-cmtkl" podStartSLOduration=2.152177748 podStartE2EDuration="2.152177748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:20:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:20:39.149520763 +0000 UTC m=+993.905858805" watchObservedRunningTime="2026-10-31 04:20:39.152177748 +0000 UTC m=+993.908515835" Oct 31 04:20:41.280366 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:41.273290 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" podUID="dccd15a9-47f5-4bd1-b21b-9d5ca8b05845" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:41.413628 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:41.413576 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:41.497849 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:41.496647 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:41.515534 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:41.514438 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" podUID="643433d8-418e-4102-8bc3-cd544932614a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:41.581586 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:41.581471 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" podUID="e0f4b8c2-14f4-4052-bf54-dd65bf59e413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:41.603622 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:41.603560 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:41.672501 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:41.672438 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" podUID="8e2c7041-ad52-49f4-8b55-d16691f4938f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:41.713534 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:41.673667 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" podUID="b4ced3ab-41e4-4c54-93eb-95eddaae992d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Oct 31 04:20:41.738465 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:41.738414 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-146320d5/e2e-lifecycle-model-146320d5-kserve-649fc69f97-k5vz4"] Oct 31 04:20:42.025417 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:41.971798 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-146320d5/e2e-lifecycle-model-146320d5-kserve-649fc69f97-k5vz4" Oct 31 04:20:42.052915 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:42.052632 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/def655ab-7eaf-4943-80a4-354b5f13640e-tmp-dir\") pod \"e2e-lifecycle-model-146320d5-kserve-649fc69f97-k5vz4\" (UID: \"def655ab-7eaf-4943-80a4-354b5f13640e\") " pod="ai-tenant-e2e-mt-146320d5/e2e-lifecycle-model-146320d5-kserve-649fc69f97-k5vz4" Oct 31 04:20:42.091514 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:42.091463 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/def655ab-7eaf-4943-80a4-354b5f13640e-model-cache\") pod \"e2e-lifecycle-model-146320d5-kserve-649fc69f97-k5vz4\" (UID: \"def655ab-7eaf-4943-80a4-354b5f13640e\") " pod="ai-tenant-e2e-mt-146320d5/e2e-lifecycle-model-146320d5-kserve-649fc69f97-k5vz4" Oct 31 04:20:42.091864 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:42.091836 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/def655ab-7eaf-4943-80a4-354b5f13640e-tls-certs\") pod \"e2e-lifecycle-model-146320d5-kserve-649fc69f97-k5vz4\" (UID: \"def655ab-7eaf-4943-80a4-354b5f13640e\") " pod="ai-tenant-e2e-mt-146320d5/e2e-lifecycle-model-146320d5-kserve-649fc69f97-k5vz4" Oct 31 04:20:42.095487 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:42.095443 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm4p9\" (UniqueName: \"kubernetes.io/projected/def655ab-7eaf-4943-80a4-354b5f13640e-kube-api-access-mm4p9\") pod \"e2e-lifecycle-model-146320d5-kserve-649fc69f97-k5vz4\" (UID: \"def655ab-7eaf-4943-80a4-354b5f13640e\") " pod="ai-tenant-e2e-mt-146320d5/e2e-lifecycle-model-146320d5-kserve-649fc69f97-k5vz4" Oct 31 04:20:42.120555 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:42.120505 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/def655ab-7eaf-4943-80a4-354b5f13640e-dshm\") pod \"e2e-lifecycle-model-146320d5-kserve-649fc69f97-k5vz4\" (UID: \"def655ab-7eaf-4943-80a4-354b5f13640e\") " pod="ai-tenant-e2e-mt-146320d5/e2e-lifecycle-model-146320d5-kserve-649fc69f97-k5vz4" Oct 31 04:20:42.120872 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:42.120850 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/def655ab-7eaf-4943-80a4-354b5f13640e-home\") pod \"e2e-lifecycle-model-146320d5-kserve-649fc69f97-k5vz4\" (UID: \"def655ab-7eaf-4943-80a4-354b5f13640e\") " pod="ai-tenant-e2e-mt-146320d5/e2e-lifecycle-model-146320d5-kserve-649fc69f97-k5vz4" Oct 31 04:20:42.121403 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:42.098922 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-146320d5/e2e-lifecycle-model-146320d5-kserve-649fc69f97-k5vz4"] Oct 31 04:20:42.150096 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:42.150062 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-146320d5\"/\"default-dockercfg-k4tfk\"" Oct 31 04:20:42.151817 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:42.150450 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-146320d5\"/\"openshift-service-ca.crt\"" Oct 31 04:20:42.152024 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:42.151002 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-146320d5\"/\"e2e-lifecycle-model-146320d5-kserve-self-signed-certs\"" Oct 31 04:20:42.152126 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:42.151153 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-146320d5\"/\"kube-root-ca.crt\"" Oct 31 04:20:42.242359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:42.239997 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/def655ab-7eaf-4943-80a4-354b5f13640e-tmp-dir\") pod \"e2e-lifecycle-model-146320d5-kserve-649fc69f97-k5vz4\" (UID: \"def655ab-7eaf-4943-80a4-354b5f13640e\") " pod="ai-tenant-e2e-mt-146320d5/e2e-lifecycle-model-146320d5-kserve-649fc69f97-k5vz4" Oct 31 04:20:42.242359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:42.240078 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/def655ab-7eaf-4943-80a4-354b5f13640e-model-cache\") pod \"e2e-lifecycle-model-146320d5-kserve-649fc69f97-k5vz4\" (UID: \"def655ab-7eaf-4943-80a4-354b5f13640e\") " pod="ai-tenant-e2e-mt-146320d5/e2e-lifecycle-model-146320d5-kserve-649fc69f97-k5vz4" Oct 31 04:20:42.242359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:42.240110 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/def655ab-7eaf-4943-80a4-354b5f13640e-tls-certs\") pod \"e2e-lifecycle-model-146320d5-kserve-649fc69f97-k5vz4\" (UID: \"def655ab-7eaf-4943-80a4-354b5f13640e\") " pod="ai-tenant-e2e-mt-146320d5/e2e-lifecycle-model-146320d5-kserve-649fc69f97-k5vz4" Oct 31 04:20:42.242359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:42.240157 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm4p9\" (UniqueName: \"kubernetes.io/projected/def655ab-7eaf-4943-80a4-354b5f13640e-kube-api-access-mm4p9\") pod \"e2e-lifecycle-model-146320d5-kserve-649fc69f97-k5vz4\" (UID: \"def655ab-7eaf-4943-80a4-354b5f13640e\") " pod="ai-tenant-e2e-mt-146320d5/e2e-lifecycle-model-146320d5-kserve-649fc69f97-k5vz4" Oct 31 04:20:42.242359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:42.240245 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/def655ab-7eaf-4943-80a4-354b5f13640e-dshm\") pod \"e2e-lifecycle-model-146320d5-kserve-649fc69f97-k5vz4\" (UID: \"def655ab-7eaf-4943-80a4-354b5f13640e\") " pod="ai-tenant-e2e-mt-146320d5/e2e-lifecycle-model-146320d5-kserve-649fc69f97-k5vz4" Oct 31 04:20:42.242359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:42.240298 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/def655ab-7eaf-4943-80a4-354b5f13640e-home\") pod \"e2e-lifecycle-model-146320d5-kserve-649fc69f97-k5vz4\" (UID: \"def655ab-7eaf-4943-80a4-354b5f13640e\") " pod="ai-tenant-e2e-mt-146320d5/e2e-lifecycle-model-146320d5-kserve-649fc69f97-k5vz4" Oct 31 04:20:42.242359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:42.241086 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/def655ab-7eaf-4943-80a4-354b5f13640e-home\") pod \"e2e-lifecycle-model-146320d5-kserve-649fc69f97-k5vz4\" (UID: \"def655ab-7eaf-4943-80a4-354b5f13640e\") " pod="ai-tenant-e2e-mt-146320d5/e2e-lifecycle-model-146320d5-kserve-649fc69f97-k5vz4" Oct 31 04:20:42.242359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:42.241400 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/def655ab-7eaf-4943-80a4-354b5f13640e-tmp-dir\") pod \"e2e-lifecycle-model-146320d5-kserve-649fc69f97-k5vz4\" (UID: \"def655ab-7eaf-4943-80a4-354b5f13640e\") " pod="ai-tenant-e2e-mt-146320d5/e2e-lifecycle-model-146320d5-kserve-649fc69f97-k5vz4" Oct 31 04:20:42.242359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:42.241686 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/def655ab-7eaf-4943-80a4-354b5f13640e-model-cache\") pod \"e2e-lifecycle-model-146320d5-kserve-649fc69f97-k5vz4\" (UID: \"def655ab-7eaf-4943-80a4-354b5f13640e\") " pod="ai-tenant-e2e-mt-146320d5/e2e-lifecycle-model-146320d5-kserve-649fc69f97-k5vz4" Oct 31 04:20:42.304471 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:42.299512 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" podUID="dccd15a9-47f5-4bd1-b21b-9d5ca8b05845" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 31 04:20:42.304471 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:42.302354 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/def655ab-7eaf-4943-80a4-354b5f13640e-dshm\") pod \"e2e-lifecycle-model-146320d5-kserve-649fc69f97-k5vz4\" (UID: \"def655ab-7eaf-4943-80a4-354b5f13640e\") " pod="ai-tenant-e2e-mt-146320d5/e2e-lifecycle-model-146320d5-kserve-649fc69f97-k5vz4" Oct 31 04:20:42.304471 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:42.303829 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" Oct 31 04:20:42.334754 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:42.330663 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm4p9\" (UniqueName: \"kubernetes.io/projected/def655ab-7eaf-4943-80a4-354b5f13640e-kube-api-access-mm4p9\") pod \"e2e-lifecycle-model-146320d5-kserve-649fc69f97-k5vz4\" (UID: \"def655ab-7eaf-4943-80a4-354b5f13640e\") " pod="ai-tenant-e2e-mt-146320d5/e2e-lifecycle-model-146320d5-kserve-649fc69f97-k5vz4" Oct 31 04:20:42.356601 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:42.354548 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/def655ab-7eaf-4943-80a4-354b5f13640e-tls-certs\") pod \"e2e-lifecycle-model-146320d5-kserve-649fc69f97-k5vz4\" (UID: \"def655ab-7eaf-4943-80a4-354b5f13640e\") " pod="ai-tenant-e2e-mt-146320d5/e2e-lifecycle-model-146320d5-kserve-649fc69f97-k5vz4" Oct 31 04:20:42.500325 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:42.489033 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:42.500325 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:42.489832 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" Oct 31 04:20:42.513716 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:42.501115 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:42.552417 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:42.522529 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" podUID="643433d8-418e-4102-8bc3-cd544932614a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:42.632751 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:42.606633 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:42.632751 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:42.615438 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" podUID="e0f4b8c2-14f4-4052-bf54-dd65bf59e413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:42.632751 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:42.620972 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" Oct 31 04:20:42.632751 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:42.621535 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" Oct 31 04:20:42.686476 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:42.646965 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" Oct 31 04:20:42.686476 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:42.647081 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" Oct 31 04:20:42.686476 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:42.685444 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" podUID="8e2c7041-ad52-49f4-8b55-d16691f4938f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:42.738503 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:42.738454 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" Oct 31 04:20:42.752376 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:42.740068 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" podUID="b4ced3ab-41e4-4c54-93eb-95eddaae992d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Oct 31 04:20:42.852622 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:42.766151 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-146320d5/e2e-lifecycle-model-146320d5-kserve-649fc69f97-k5vz4" Oct 31 04:20:42.915356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:42.891816 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" Oct 31 04:20:43.052380 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:43.021474 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" podUID="e0f4b8c2-14f4-4052-bf54-dd65bf59e413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": context deadline exceeded" Oct 31 04:20:43.164610 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:43.159444 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-pghv7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:20:43.164610 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:43.159517 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7" podUID="e0ccc03e-0e5e-40b7-8a40-718f199dd747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:43.315905 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:43.315853 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" podUID="dccd15a9-47f5-4bd1-b21b-9d5ca8b05845" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:43.395608 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:43.395557 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" podUID="dccd15a9-47f5-4bd1-b21b-9d5ca8b05845" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:43.502560 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:43.502505 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:43.524601 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:43.524549 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" podUID="643433d8-418e-4102-8bc3-cd544932614a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 04:20:43.560536 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:43.560477 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:43.624503 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:43.624439 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" podUID="e0f4b8c2-14f4-4052-bf54-dd65bf59e413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:43.626314 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:43.626269 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:43.772425 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:43.722143 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" podUID="8e2c7041-ad52-49f4-8b55-d16691f4938f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 31 04:20:43.910836 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:43.910759 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" podUID="b4ced3ab-41e4-4c54-93eb-95eddaae992d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:44.202471 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:44.180465 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:44.401360 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:44.395639 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" podUID="dccd15a9-47f5-4bd1-b21b-9d5ca8b05845" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 31 04:20:44.544256 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:44.544198 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 31 04:20:44.546495 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:44.546441 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" podUID="643433d8-418e-4102-8bc3-cd544932614a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:44.630447 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:44.622708 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 31 04:20:44.630984 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:44.630940 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" podUID="e0f4b8c2-14f4-4052-bf54-dd65bf59e413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 04:20:44.632866 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:44.631971 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:44.911456 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:44.887925 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" podUID="8e2c7041-ad52-49f4-8b55-d16691f4938f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 31 04:20:44.915391 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:44.911812 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" podUID="b4ced3ab-41e4-4c54-93eb-95eddaae992d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:45.037420 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:45.031589 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8c0086-78b8c6c95-vwbxc" Oct 31 04:20:45.358679 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:45.358605 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" podUID="b4ced3ab-41e4-4c54-93eb-95eddaae992d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/health\": context deadline exceeded" Oct 31 04:20:45.632767 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:45.632645 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:45.884427 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:45.634666 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" podUID="e0f4b8c2-14f4-4052-bf54-dd65bf59e413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 04:20:45.921358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:45.676322 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" podUID="dccd15a9-47f5-4bd1-b21b-9d5ca8b05845" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 31 04:20:45.954600 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:45.688493 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 31 04:20:45.967982 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:45.688533 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" podUID="643433d8-418e-4102-8bc3-cd544932614a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:45.975181 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:45.695258 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 31 04:20:45.996064 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:45.962845 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" podUID="b4ced3ab-41e4-4c54-93eb-95eddaae992d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:46.064361 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:46.062492 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" podUID="8e2c7041-ad52-49f4-8b55-d16691f4938f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:46.888836 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:46.888774 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" podUID="e0f4b8c2-14f4-4052-bf54-dd65bf59e413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 04:20:46.916401 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:46.902530 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" podUID="dccd15a9-47f5-4bd1-b21b-9d5ca8b05845" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:46.939429 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:46.932073 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" Oct 31 04:20:46.941137 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:46.941103 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" Oct 31 04:20:46.941469 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:46.941434 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-146320d5/e2e-lifecycle-model-146320d5-kserve-649fc69f97-k5vz4"] Oct 31 04:20:46.989404 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:46.983871 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:47.013359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:47.000698 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" podUID="b4ced3ab-41e4-4c54-93eb-95eddaae992d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:47.013359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:47.001051 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 31 04:20:47.088523 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:47.087477 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" podUID="8e2c7041-ad52-49f4-8b55-d16691f4938f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:20:47.349373 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:47.343633 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" Oct 31 04:20:47.368937 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:47.366137 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" Oct 31 04:20:47.372397 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:47.372365 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" Oct 31 04:20:47.479357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:47.469254 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" Oct 31 04:20:47.759910 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:47.759874 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" Oct 31 04:20:47.760417 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:47.759977 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" Oct 31 04:20:47.830534 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:47.830361 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-146320d5/e2e-lifecycle-model-146320d5-kserve-649fc69f97-k5vz4" event={"ID":"def655ab-7eaf-4943-80a4-354b5f13640e","Type":"ContainerStarted","Data":"0a8cdae8b227028ae44b12b250c928587e4540398224fecc39fe462882258b1f"} Oct 31 04:20:48.843075 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:48.843031 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-146320d5/e2e-lifecycle-model-146320d5-kserve-649fc69f97-k5vz4" event={"ID":"def655ab-7eaf-4943-80a4-354b5f13640e","Type":"ContainerStarted","Data":"67a949150b647a19e8130fd904d665fa02e6575696d2e26a8e0f720e2b50f425"} Oct 31 04:20:48.904632 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:48.904593 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-146320d5/e2e-lifecycle-model-146320d5-kserve-649fc69f97-k5vz4" Oct 31 04:20:48.908846 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:48.908776 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-146320d5/e2e-lifecycle-model-146320d5-kserve-649fc69f97-k5vz4" podStartSLOduration=7.908752724 podStartE2EDuration="7.908752724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:20:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:20:48.885868512 +0000 UTC m=+1003.642206552" watchObservedRunningTime="2026-10-31 04:20:48.908752724 +0000 UTC m=+1003.665090766" Oct 31 04:20:50.244618 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:50.244587 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9883e4-6b849c5655-vlf6g" Oct 31 04:20:50.248536 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:50.248510 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-9883e4-6f4c589fd5-cmtkl" Oct 31 04:20:52.405542 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:20:52.405497 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:20:52.899718 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:52.898004 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-146320d5/e2e-lifecycle-model-146320d5-kserve-649fc69f97-k5vz4" Oct 31 04:20:52.917873 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:52.917833 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-146320d5/e2e-lifecycle-model-146320d5-kserve-649fc69f97-k5vz4" Oct 31 04:20:53.072946 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:53.072912 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-146320d5/e2e-lifecycle-model-146320d5-kserve-649fc69f97-k5vz4" Oct 31 04:20:55.094124 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:55.094090 5019 generic.go:356] "Generic (PLEG): container finished" podID="59824c73-334b-446b-8d63-bde791f5b216" containerID="e6f0da0c29aaf18aa4bb1379e308be5f3d0f10dceedf00cd8dd7c5a4d767dba4" exitCode=0 Oct 31 04:20:55.094614 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:55.094176 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr" event={"ID":"59824c73-334b-446b-8d63-bde791f5b216","Type":"ContainerDied","Data":"e6f0da0c29aaf18aa4bb1379e308be5f3d0f10dceedf00cd8dd7c5a4d767dba4"} Oct 31 04:20:55.094774 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:55.094757 5019 scope.go:122] "RemoveContainer" containerID="e6f0da0c29aaf18aa4bb1379e308be5f3d0f10dceedf00cd8dd7c5a4d767dba4" Oct 31 04:20:55.096496 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:55.096464 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:20:55.096603 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:55.096536 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:20:55.096603 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:55.096568 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:20:56.113564 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:56.113519 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr" event={"ID":"59824c73-334b-446b-8d63-bde791f5b216","Type":"ContainerStarted","Data":"b73d89ec659705cb66106ccecfac22ce040657b0a88b64f2f101b8b04d0d32d9"} Oct 31 04:20:56.867444 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:56.867408 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr" Oct 31 04:20:56.867700 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:56.867678 5019 patch_prober.go:29] interesting pod/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 31 04:20:56.867755 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:56.867727 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr" podUID="59824c73-334b-446b-8d63-bde791f5b216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 31 04:20:57.867020 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:57.866983 5019 patch_prober.go:29] interesting pod/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 31 04:20:57.867496 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:57.867464 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr" podUID="59824c73-334b-446b-8d63-bde791f5b216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 31 04:20:58.866872 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:58.866828 5019 patch_prober.go:29] interesting pod/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 31 04:20:58.867068 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:58.866899 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr" podUID="59824c73-334b-446b-8d63-bde791f5b216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 31 04:20:59.867286 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:59.867241 5019 patch_prober.go:29] interesting pod/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 31 04:20:59.867755 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:20:59.867308 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr" podUID="59824c73-334b-446b-8d63-bde791f5b216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 31 04:21:00.867175 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:00.867138 5019 patch_prober.go:29] interesting pod/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 31 04:21:00.867377 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:00.867195 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr" podUID="59824c73-334b-446b-8d63-bde791f5b216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 31 04:21:01.871670 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:01.871632 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr" Oct 31 04:21:01.872213 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:01.872190 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr" Oct 31 04:21:01.873267 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:01.873241 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr" Oct 31 04:21:05.677498 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:05.677456 5019 scope.go:122] "RemoveContainer" containerID="28dd974596f9c8da9a638a382bc8fbde5f5a7af8f3c32fa56cf5712419b888f7" Oct 31 04:21:05.688962 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:05.688935 5019 scope.go:122] "RemoveContainer" containerID="287c2e7c572c9f16f3f5788f29f23c65ea1941a2390f58627c45ebffbc424650" Oct 31 04:21:05.703361 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:05.703300 5019 scope.go:122] "RemoveContainer" containerID="aa6275cbc5f84d8c407867f7f4563a135a0a1c2c9f1038af018d8a063bcf6504" Oct 31 04:21:05.715209 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:05.715179 5019 scope.go:122] "RemoveContainer" containerID="8eb9b51dc9e91baf59b100cbb21f72d6f39ac01d6efc2dbcec3e885b7dbb8d42" Oct 31 04:21:05.726871 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:05.726848 5019 scope.go:122] "RemoveContainer" containerID="cd258e729b3fb3f56ee9b49096d9c29eadd41f2b6d8a774a75131628b8fd70c0" Oct 31 04:21:08.865926 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:08.865870 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm"] Oct 31 04:21:08.894138 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:08.894045 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm" Oct 31 04:21:08.949837 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:08.949797 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm"] Oct 31 04:21:09.084753 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:09.084675 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/441c6ad5-5141-4b8e-b9db-9497f594f479-credential-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm\" (UID: \"441c6ad5-5141-4b8e-b9db-9497f594f479\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm" Oct 31 04:21:09.085180 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:09.085147 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/441c6ad5-5141-4b8e-b9db-9497f594f479-istio-envoy\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm\" (UID: \"441c6ad5-5141-4b8e-b9db-9497f594f479\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm" Oct 31 04:21:09.085477 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:09.085455 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/441c6ad5-5141-4b8e-b9db-9497f594f479-workload-certs\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm\" (UID: \"441c6ad5-5141-4b8e-b9db-9497f594f479\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm" Oct 31 04:21:09.085706 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:09.085688 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/441c6ad5-5141-4b8e-b9db-9497f594f479-istio-data\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm\" (UID: \"441c6ad5-5141-4b8e-b9db-9497f594f479\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm" Oct 31 04:21:09.094923 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:09.094883 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/441c6ad5-5141-4b8e-b9db-9497f594f479-workload-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm\" (UID: \"441c6ad5-5141-4b8e-b9db-9497f594f479\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm" Oct 31 04:21:09.095285 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:09.095257 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/441c6ad5-5141-4b8e-b9db-9497f594f479-istio-token\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm\" (UID: \"441c6ad5-5141-4b8e-b9db-9497f594f479\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm" Oct 31 04:21:09.095461 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:09.095440 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/441c6ad5-5141-4b8e-b9db-9497f594f479-istiod-ca-cert\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm\" (UID: \"441c6ad5-5141-4b8e-b9db-9497f594f479\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm" Oct 31 04:21:09.095750 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:09.095728 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67j6w\" (UniqueName: \"kubernetes.io/projected/441c6ad5-5141-4b8e-b9db-9497f594f479-kube-api-access-67j6w\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm\" (UID: \"441c6ad5-5141-4b8e-b9db-9497f594f479\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm" Oct 31 04:21:09.095937 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:09.095903 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/441c6ad5-5141-4b8e-b9db-9497f594f479-istio-podinfo\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm\" (UID: \"441c6ad5-5141-4b8e-b9db-9497f594f479\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm" Oct 31 04:21:09.201470 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:09.201359 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/441c6ad5-5141-4b8e-b9db-9497f594f479-workload-certs\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm\" (UID: \"441c6ad5-5141-4b8e-b9db-9497f594f479\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm" Oct 31 04:21:09.202259 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:09.202239 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/441c6ad5-5141-4b8e-b9db-9497f594f479-istio-data\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm\" (UID: \"441c6ad5-5141-4b8e-b9db-9497f594f479\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm" Oct 31 04:21:09.202740 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:09.202723 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/441c6ad5-5141-4b8e-b9db-9497f594f479-workload-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm\" (UID: \"441c6ad5-5141-4b8e-b9db-9497f594f479\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm" Oct 31 04:21:09.203191 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:09.203175 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/441c6ad5-5141-4b8e-b9db-9497f594f479-istio-token\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm\" (UID: \"441c6ad5-5141-4b8e-b9db-9497f594f479\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm" Oct 31 04:21:09.203687 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:09.203666 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/441c6ad5-5141-4b8e-b9db-9497f594f479-istiod-ca-cert\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm\" (UID: \"441c6ad5-5141-4b8e-b9db-9497f594f479\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm" Oct 31 04:21:09.209530 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:09.209489 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67j6w\" (UniqueName: \"kubernetes.io/projected/441c6ad5-5141-4b8e-b9db-9497f594f479-kube-api-access-67j6w\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm\" (UID: \"441c6ad5-5141-4b8e-b9db-9497f594f479\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm" Oct 31 04:21:09.210359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:09.210319 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/441c6ad5-5141-4b8e-b9db-9497f594f479-istio-podinfo\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm\" (UID: \"441c6ad5-5141-4b8e-b9db-9497f594f479\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm" Oct 31 04:21:09.210573 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:09.210558 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/441c6ad5-5141-4b8e-b9db-9497f594f479-credential-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm\" (UID: \"441c6ad5-5141-4b8e-b9db-9497f594f479\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm" Oct 31 04:21:09.211485 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:09.211464 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/441c6ad5-5141-4b8e-b9db-9497f594f479-istio-envoy\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm\" (UID: \"441c6ad5-5141-4b8e-b9db-9497f594f479\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm" Oct 31 04:21:09.217778 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:09.210258 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/441c6ad5-5141-4b8e-b9db-9497f594f479-istiod-ca-cert\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm\" (UID: \"441c6ad5-5141-4b8e-b9db-9497f594f479\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm" Oct 31 04:21:09.218043 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:09.202094 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/441c6ad5-5141-4b8e-b9db-9497f594f479-workload-certs\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm\" (UID: \"441c6ad5-5141-4b8e-b9db-9497f594f479\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm" Oct 31 04:21:09.218142 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:09.202645 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/441c6ad5-5141-4b8e-b9db-9497f594f479-istio-data\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm\" (UID: \"441c6ad5-5141-4b8e-b9db-9497f594f479\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm" Oct 31 04:21:09.218222 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:09.211364 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/441c6ad5-5141-4b8e-b9db-9497f594f479-credential-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm\" (UID: \"441c6ad5-5141-4b8e-b9db-9497f594f479\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm" Oct 31 04:21:09.218319 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:09.203049 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/441c6ad5-5141-4b8e-b9db-9497f594f479-workload-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm\" (UID: \"441c6ad5-5141-4b8e-b9db-9497f594f479\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm" Oct 31 04:21:09.241162 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:09.241125 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/441c6ad5-5141-4b8e-b9db-9497f594f479-istio-envoy\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm\" (UID: \"441c6ad5-5141-4b8e-b9db-9497f594f479\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm" Oct 31 04:21:09.263363 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:09.258021 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/441c6ad5-5141-4b8e-b9db-9497f594f479-istio-token\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm\" (UID: \"441c6ad5-5141-4b8e-b9db-9497f594f479\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm" Oct 31 04:21:09.270084 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:09.270009 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/441c6ad5-5141-4b8e-b9db-9497f594f479-istio-podinfo\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm\" (UID: \"441c6ad5-5141-4b8e-b9db-9497f594f479\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm" Oct 31 04:21:09.270476 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:09.270421 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67j6w\" (UniqueName: \"kubernetes.io/projected/441c6ad5-5141-4b8e-b9db-9497f594f479-kube-api-access-67j6w\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm\" (UID: \"441c6ad5-5141-4b8e-b9db-9497f594f479\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm" Oct 31 04:21:09.349155 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:09.349110 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm" Oct 31 04:21:09.716426 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:21:09.716375 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod441c6ad5_5141_4b8e_b9db_9497f594f479.slice/crio-8abb84d49d7a0e6158e4b84cc653845f22bd7caa98ecfa650a96aeba5efa7fee WatchSource:0}: Error finding container 8abb84d49d7a0e6158e4b84cc653845f22bd7caa98ecfa650a96aeba5efa7fee: Status 404 returned error can't find the container with id 8abb84d49d7a0e6158e4b84cc653845f22bd7caa98ecfa650a96aeba5efa7fee Oct 31 04:21:09.717033 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:09.717006 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm"] Oct 31 04:21:09.721803 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:09.721758 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:21:09.721954 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:09.721866 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:21:09.721954 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:09.721916 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:21:10.322486 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:10.322436 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm" event={"ID":"441c6ad5-5141-4b8e-b9db-9497f594f479","Type":"ContainerStarted","Data":"7a3961cea39f27cda42d781e29232cfe4c3ae6f605bd35ae90df10b6d3d39bc3"} Oct 31 04:21:10.323014 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:10.322516 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm" event={"ID":"441c6ad5-5141-4b8e-b9db-9497f594f479","Type":"ContainerStarted","Data":"8abb84d49d7a0e6158e4b84cc653845f22bd7caa98ecfa650a96aeba5efa7fee"} Oct 31 04:21:10.351391 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:10.350650 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm" Oct 31 04:21:10.352775 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:10.352740 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Oct 31 04:21:10.352930 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:10.352799 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm" podUID="441c6ad5-5141-4b8e-b9db-9497f594f479" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Oct 31 04:21:10.416356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:10.407367 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm" podStartSLOduration=2.40734657 podStartE2EDuration="2.40734657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:21:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:21:10.383976834 +0000 UTC m=+1025.140314886" watchObservedRunningTime="2026-10-31 04:21:10.40734657 +0000 UTC m=+1025.163684603" Oct 31 04:21:11.350326 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:11.350290 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Oct 31 04:21:11.350789 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:11.350361 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm" podUID="441c6ad5-5141-4b8e-b9db-9497f594f479" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Oct 31 04:21:11.525353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:11.524864 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv"] Oct 31 04:21:11.531410 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:11.530921 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv" Oct 31 04:21:11.566897 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:11.565132 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv"] Oct 31 04:21:11.566987 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:11.566906 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-0fc77f-openshift-default-dockercfg-7znkz\"" Oct 31 04:21:11.668226 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:11.668106 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffd1c004-3338-4cf6-b6f7-afebf7d01574-istio-envoy\") pod \"e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv\" (UID: \"ffd1c004-3338-4cf6-b6f7-afebf7d01574\") " pod="openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv" Oct 31 04:21:11.668454 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:11.668251 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffd1c004-3338-4cf6-b6f7-afebf7d01574-istio-podinfo\") pod \"e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv\" (UID: \"ffd1c004-3338-4cf6-b6f7-afebf7d01574\") " pod="openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv" Oct 31 04:21:11.668454 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:11.668342 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd1c004-3338-4cf6-b6f7-afebf7d01574-workload-socket\") pod \"e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv\" (UID: \"ffd1c004-3338-4cf6-b6f7-afebf7d01574\") " pod="openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv" Oct 31 04:21:11.668454 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:11.668434 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffd1c004-3338-4cf6-b6f7-afebf7d01574-istio-token\") pod \"e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv\" (UID: \"ffd1c004-3338-4cf6-b6f7-afebf7d01574\") " pod="openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv" Oct 31 04:21:11.668615 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:11.668474 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffd1c004-3338-4cf6-b6f7-afebf7d01574-istiod-ca-cert\") pod \"e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv\" (UID: \"ffd1c004-3338-4cf6-b6f7-afebf7d01574\") " pod="openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv" Oct 31 04:21:11.668615 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:11.668537 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffd1c004-3338-4cf6-b6f7-afebf7d01574-istio-data\") pod \"e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv\" (UID: \"ffd1c004-3338-4cf6-b6f7-afebf7d01574\") " pod="openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv" Oct 31 04:21:11.668615 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:11.668606 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlfxn\" (UniqueName: \"kubernetes.io/projected/ffd1c004-3338-4cf6-b6f7-afebf7d01574-kube-api-access-mlfxn\") pod \"e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv\" (UID: \"ffd1c004-3338-4cf6-b6f7-afebf7d01574\") " pod="openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv" Oct 31 04:21:11.668766 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:11.668688 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffd1c004-3338-4cf6-b6f7-afebf7d01574-workload-certs\") pod \"e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv\" (UID: \"ffd1c004-3338-4cf6-b6f7-afebf7d01574\") " pod="openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv" Oct 31 04:21:11.668822 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:11.668773 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd1c004-3338-4cf6-b6f7-afebf7d01574-credential-socket\") pod \"e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv\" (UID: \"ffd1c004-3338-4cf6-b6f7-afebf7d01574\") " pod="openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv" Oct 31 04:21:11.784360 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:11.770462 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffd1c004-3338-4cf6-b6f7-afebf7d01574-workload-certs\") pod \"e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv\" (UID: \"ffd1c004-3338-4cf6-b6f7-afebf7d01574\") " pod="openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv" Oct 31 04:21:11.785461 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:11.785430 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd1c004-3338-4cf6-b6f7-afebf7d01574-credential-socket\") pod \"e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv\" (UID: \"ffd1c004-3338-4cf6-b6f7-afebf7d01574\") " pod="openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv" Oct 31 04:21:11.786152 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:11.786131 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffd1c004-3338-4cf6-b6f7-afebf7d01574-istio-envoy\") pod \"e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv\" (UID: \"ffd1c004-3338-4cf6-b6f7-afebf7d01574\") " pod="openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv" Oct 31 04:21:11.786785 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:11.786758 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffd1c004-3338-4cf6-b6f7-afebf7d01574-istio-podinfo\") pod \"e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv\" (UID: \"ffd1c004-3338-4cf6-b6f7-afebf7d01574\") " pod="openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv" Oct 31 04:21:11.787441 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:11.787421 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd1c004-3338-4cf6-b6f7-afebf7d01574-workload-socket\") pod \"e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv\" (UID: \"ffd1c004-3338-4cf6-b6f7-afebf7d01574\") " pod="openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv" Oct 31 04:21:11.788037 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:11.788019 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffd1c004-3338-4cf6-b6f7-afebf7d01574-istio-token\") pod \"e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv\" (UID: \"ffd1c004-3338-4cf6-b6f7-afebf7d01574\") " pod="openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv" Oct 31 04:21:11.788576 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:11.788554 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffd1c004-3338-4cf6-b6f7-afebf7d01574-istiod-ca-cert\") pod \"e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv\" (UID: \"ffd1c004-3338-4cf6-b6f7-afebf7d01574\") " pod="openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv" Oct 31 04:21:11.789547 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:11.789525 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffd1c004-3338-4cf6-b6f7-afebf7d01574-istio-data\") pod \"e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv\" (UID: \"ffd1c004-3338-4cf6-b6f7-afebf7d01574\") " pod="openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv" Oct 31 04:21:11.803084 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:11.803041 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlfxn\" (UniqueName: \"kubernetes.io/projected/ffd1c004-3338-4cf6-b6f7-afebf7d01574-kube-api-access-mlfxn\") pod \"e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv\" (UID: \"ffd1c004-3338-4cf6-b6f7-afebf7d01574\") " pod="openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv" Oct 31 04:21:11.804138 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:11.787876 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd1c004-3338-4cf6-b6f7-afebf7d01574-workload-socket\") pod \"e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv\" (UID: \"ffd1c004-3338-4cf6-b6f7-afebf7d01574\") " pod="openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv" Oct 31 04:21:11.804401 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:11.789419 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffd1c004-3338-4cf6-b6f7-afebf7d01574-istiod-ca-cert\") pod \"e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv\" (UID: \"ffd1c004-3338-4cf6-b6f7-afebf7d01574\") " pod="openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv" Oct 31 04:21:11.804535 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:11.785228 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffd1c004-3338-4cf6-b6f7-afebf7d01574-workload-certs\") pod \"e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv\" (UID: \"ffd1c004-3338-4cf6-b6f7-afebf7d01574\") " pod="openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv" Oct 31 04:21:11.804635 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:11.789966 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffd1c004-3338-4cf6-b6f7-afebf7d01574-istio-data\") pod \"e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv\" (UID: \"ffd1c004-3338-4cf6-b6f7-afebf7d01574\") " pod="openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv" Oct 31 04:21:11.804735 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:11.785934 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd1c004-3338-4cf6-b6f7-afebf7d01574-credential-socket\") pod \"e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv\" (UID: \"ffd1c004-3338-4cf6-b6f7-afebf7d01574\") " pod="openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv" Oct 31 04:21:11.833514 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:11.833477 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffd1c004-3338-4cf6-b6f7-afebf7d01574-istio-podinfo\") pod \"e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv\" (UID: \"ffd1c004-3338-4cf6-b6f7-afebf7d01574\") " pod="openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv" Oct 31 04:21:11.834205 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:11.834169 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffd1c004-3338-4cf6-b6f7-afebf7d01574-istio-envoy\") pod \"e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv\" (UID: \"ffd1c004-3338-4cf6-b6f7-afebf7d01574\") " pod="openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv" Oct 31 04:21:11.856591 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:11.854887 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffd1c004-3338-4cf6-b6f7-afebf7d01574-istio-token\") pod \"e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv\" (UID: \"ffd1c004-3338-4cf6-b6f7-afebf7d01574\") " pod="openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv" Oct 31 04:21:11.867958 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:11.867927 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlfxn\" (UniqueName: \"kubernetes.io/projected/ffd1c004-3338-4cf6-b6f7-afebf7d01574-kube-api-access-mlfxn\") pod \"e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv\" (UID: \"ffd1c004-3338-4cf6-b6f7-afebf7d01574\") " pod="openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv" Oct 31 04:21:11.900720 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:11.900683 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv" Oct 31 04:21:12.350864 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:12.350774 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Oct 31 04:21:12.350864 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:12.350845 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm" podUID="441c6ad5-5141-4b8e-b9db-9497f594f479" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Oct 31 04:21:12.399709 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:12.399646 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv"] Oct 31 04:21:12.407787 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:21:12.407750 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffd1c004_3338_4cf6_b6f7_afebf7d01574.slice/crio-78e12522820d66b1edd1baa27b98b00466f4cd92e3654ebbf8300dae88db419f WatchSource:0}: Error finding container 78e12522820d66b1edd1baa27b98b00466f4cd92e3654ebbf8300dae88db419f: Status 404 returned error can't find the container with id 78e12522820d66b1edd1baa27b98b00466f4cd92e3654ebbf8300dae88db419f Oct 31 04:21:12.411371 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:12.411283 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:21:12.411490 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:12.411437 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:21:12.411531 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:12.411485 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:21:13.350967 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:13.350929 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Oct 31 04:21:13.351427 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:13.350999 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm" podUID="441c6ad5-5141-4b8e-b9db-9497f594f479" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Oct 31 04:21:13.358606 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:13.358567 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv" event={"ID":"ffd1c004-3338-4cf6-b6f7-afebf7d01574","Type":"ContainerStarted","Data":"dece7da60db378cea7ec0d7fef6b187d2624546e21795d2c15856442df3500e9"} Oct 31 04:21:13.358814 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:13.358625 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv" event={"ID":"ffd1c004-3338-4cf6-b6f7-afebf7d01574","Type":"ContainerStarted","Data":"78e12522820d66b1edd1baa27b98b00466f4cd92e3654ebbf8300dae88db419f"} Oct 31 04:21:13.902479 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:13.902442 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv" Oct 31 04:21:13.908976 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:13.908945 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv" Oct 31 04:21:13.956829 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:13.956725 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv" podStartSLOduration=2.956691493 podStartE2EDuration="2.956691493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:21:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:21:13.439322284 +0000 UTC m=+1028.195660324" watchObservedRunningTime="2026-10-31 04:21:13.956691493 +0000 UTC m=+1028.713029534" Oct 31 04:21:14.350790 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:14.350682 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Oct 31 04:21:14.350790 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:14.350758 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm" podUID="441c6ad5-5141-4b8e-b9db-9497f594f479" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Oct 31 04:21:14.393359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:14.389405 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv" Oct 31 04:21:14.393359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:14.389553 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv" Oct 31 04:21:14.604231 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:14.602854 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35fb49a3-tthwf"] Oct 31 04:21:14.612195 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:14.612155 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35fb49a3-tthwf"] Oct 31 04:21:14.612399 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:14.612299 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35fb49a3-tthwf" Oct 31 04:21:14.630779 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:14.628202 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:21:14.630779 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:14.628540 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:21:14.630779 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:14.629691 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:21:14.630779 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:14.630599 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:21:14.711580 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:14.711528 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03b83c43-5a5c-48c1-a804-c92f53fa02a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35fb49a3-tthwf\" (UID: \"03b83c43-5a5c-48c1-a804-c92f53fa02a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35fb49a3-tthwf" Oct 31 04:21:14.711960 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:14.711930 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf9g7\" (UniqueName: \"kubernetes.io/projected/03b83c43-5a5c-48c1-a804-c92f53fa02a2-kube-api-access-kf9g7\") pod \"maas-api-revoke-sub-35fb49a3-tthwf\" (UID: \"03b83c43-5a5c-48c1-a804-c92f53fa02a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35fb49a3-tthwf" Oct 31 04:21:14.712191 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:14.712176 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03b83c43-5a5c-48c1-a804-c92f53fa02a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35fb49a3-tthwf\" (UID: \"03b83c43-5a5c-48c1-a804-c92f53fa02a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35fb49a3-tthwf" Oct 31 04:21:14.813964 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:14.813921 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03b83c43-5a5c-48c1-a804-c92f53fa02a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35fb49a3-tthwf\" (UID: \"03b83c43-5a5c-48c1-a804-c92f53fa02a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35fb49a3-tthwf" Oct 31 04:21:14.814259 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:14.814232 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03b83c43-5a5c-48c1-a804-c92f53fa02a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35fb49a3-tthwf\" (UID: \"03b83c43-5a5c-48c1-a804-c92f53fa02a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35fb49a3-tthwf" Oct 31 04:21:14.814368 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:14.814297 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf9g7\" (UniqueName: \"kubernetes.io/projected/03b83c43-5a5c-48c1-a804-c92f53fa02a2-kube-api-access-kf9g7\") pod \"maas-api-revoke-sub-35fb49a3-tthwf\" (UID: \"03b83c43-5a5c-48c1-a804-c92f53fa02a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35fb49a3-tthwf" Oct 31 04:21:14.815018 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:14.814997 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03b83c43-5a5c-48c1-a804-c92f53fa02a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35fb49a3-tthwf\" (UID: \"03b83c43-5a5c-48c1-a804-c92f53fa02a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35fb49a3-tthwf" Oct 31 04:21:14.818611 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:14.818444 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03b83c43-5a5c-48c1-a804-c92f53fa02a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35fb49a3-tthwf\" (UID: \"03b83c43-5a5c-48c1-a804-c92f53fa02a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35fb49a3-tthwf" Oct 31 04:21:14.838349 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:14.838304 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf9g7\" (UniqueName: \"kubernetes.io/projected/03b83c43-5a5c-48c1-a804-c92f53fa02a2-kube-api-access-kf9g7\") pod \"maas-api-revoke-sub-35fb49a3-tthwf\" (UID: \"03b83c43-5a5c-48c1-a804-c92f53fa02a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35fb49a3-tthwf" Oct 31 04:21:14.937327 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:14.937226 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35fb49a3-tthwf" Oct 31 04:21:15.050445 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:15.050397 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b9ca02f-kk972"] Oct 31 04:21:15.061378 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:15.061102 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b9ca02f-kk972"] Oct 31 04:21:15.353638 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:15.351830 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Oct 31 04:21:15.353638 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:15.351899 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm" podUID="441c6ad5-5141-4b8e-b9db-9497f594f479" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Oct 31 04:21:15.353638 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:21:15.353189 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03b83c43_5a5c_48c1_a804_c92f53fa02a2.slice/crio-b219089eff2b7902b9875741af2e037995257a2fc2e6f01e74f0fa4b1d697761 WatchSource:0}: Error finding container b219089eff2b7902b9875741af2e037995257a2fc2e6f01e74f0fa4b1d697761: Status 404 returned error can't find the container with id b219089eff2b7902b9875741af2e037995257a2fc2e6f01e74f0fa4b1d697761 Oct 31 04:21:15.356299 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:15.356248 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35fb49a3-tthwf"] Oct 31 04:21:15.414118 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:15.414081 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c966c31-c0b5-4a53-83a0-2dafa8418e0f" path="/var/lib/kubelet/pods/4c966c31-c0b5-4a53-83a0-2dafa8418e0f/volumes" Oct 31 04:21:15.414565 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:15.414541 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35fb49a3-tthwf" event={"ID":"03b83c43-5a5c-48c1-a804-c92f53fa02a2","Type":"ContainerStarted","Data":"b219089eff2b7902b9875741af2e037995257a2fc2e6f01e74f0fa4b1d697761"} Oct 31 04:21:16.350562 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:16.350527 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Oct 31 04:21:16.350751 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:16.350585 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm" podUID="441c6ad5-5141-4b8e-b9db-9497f594f479" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Oct 31 04:21:16.425412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:16.425366 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35fb49a3-tthwf" event={"ID":"03b83c43-5a5c-48c1-a804-c92f53fa02a2","Type":"ContainerStarted","Data":"a780e1cde60cb486f89fa80c87c5048d248bb903a6c0abeb02934c5b3fa71545"} Oct 31 04:21:16.481423 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:16.481347 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35fb49a3-tthwf" podStartSLOduration=2.4813078490000002 podStartE2EDuration="2.481307849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:21:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:21:16.477173717 +0000 UTC m=+1031.233511773" watchObservedRunningTime="2026-10-31 04:21:16.481307849 +0000 UTC m=+1031.237645890" Oct 31 04:21:17.441185 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:17.441149 5019 generic.go:356] "Generic (PLEG): container finished" podID="03b83c43-5a5c-48c1-a804-c92f53fa02a2" containerID="a780e1cde60cb486f89fa80c87c5048d248bb903a6c0abeb02934c5b3fa71545" exitCode=0 Oct 31 04:21:17.441681 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:17.441245 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35fb49a3-tthwf" event={"ID":"03b83c43-5a5c-48c1-a804-c92f53fa02a2","Type":"ContainerDied","Data":"a780e1cde60cb486f89fa80c87c5048d248bb903a6c0abeb02934c5b3fa71545"} Oct 31 04:21:17.556560 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:17.556507 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm" Oct 31 04:21:17.558246 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:17.558219 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm" Oct 31 04:21:17.558792 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:17.558768 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm" Oct 31 04:21:18.168784 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:18.168746 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0fc77f-774488f5b7-jpmqw"] Oct 31 04:21:18.173934 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:18.173908 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0fc77f-774488f5b7-jpmqw" Oct 31 04:21:18.193378 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:18.193308 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0fc77f-774488f5b7-jpmqw"] Oct 31 04:21:18.193670 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:18.193649 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-0fc77f\"" Oct 31 04:21:18.193846 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:18.193831 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-0fc77f-dockercfg-2rq6q\"" Oct 31 04:21:18.303360 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:18.298543 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwhqx\" (UniqueName: \"kubernetes.io/projected/a136b866-abc0-4c60-b977-ce6f05b0bdbf-kube-api-access-gwhqx\") pod \"payload-pre-processing-e2e-api-b-0fc77f-774488f5b7-jpmqw\" (UID: \"a136b866-abc0-4c60-b977-ce6f05b0bdbf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0fc77f-774488f5b7-jpmqw" Oct 31 04:21:18.303360 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:18.298599 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a136b866-abc0-4c60-b977-ce6f05b0bdbf-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0fc77f-774488f5b7-jpmqw\" (UID: \"a136b866-abc0-4c60-b977-ce6f05b0bdbf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0fc77f-774488f5b7-jpmqw" Oct 31 04:21:18.345361 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:18.344723 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0fc77f-6d886fb5cf-tht76"] Oct 31 04:21:18.364364 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:18.352125 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0fc77f-6d886fb5cf-tht76" Oct 31 04:21:18.364364 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:18.362011 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0fc77f-6d886fb5cf-tht76"] Oct 31 04:21:18.485267 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:18.485209 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwhqx\" (UniqueName: \"kubernetes.io/projected/a136b866-abc0-4c60-b977-ce6f05b0bdbf-kube-api-access-gwhqx\") pod \"payload-pre-processing-e2e-api-b-0fc77f-774488f5b7-jpmqw\" (UID: \"a136b866-abc0-4c60-b977-ce6f05b0bdbf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0fc77f-774488f5b7-jpmqw" Oct 31 04:21:18.488972 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:18.485298 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a136b866-abc0-4c60-b977-ce6f05b0bdbf-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0fc77f-774488f5b7-jpmqw\" (UID: \"a136b866-abc0-4c60-b977-ce6f05b0bdbf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0fc77f-774488f5b7-jpmqw" Oct 31 04:21:18.509192 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:18.507474 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a136b866-abc0-4c60-b977-ce6f05b0bdbf-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0fc77f-774488f5b7-jpmqw\" (UID: \"a136b866-abc0-4c60-b977-ce6f05b0bdbf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0fc77f-774488f5b7-jpmqw" Oct 31 04:21:18.514225 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:18.514198 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwhqx\" (UniqueName: \"kubernetes.io/projected/a136b866-abc0-4c60-b977-ce6f05b0bdbf-kube-api-access-gwhqx\") pod \"payload-pre-processing-e2e-api-b-0fc77f-774488f5b7-jpmqw\" (UID: \"a136b866-abc0-4c60-b977-ce6f05b0bdbf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0fc77f-774488f5b7-jpmqw" Oct 31 04:21:18.610918 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:18.609783 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5053ea34-585c-4b1d-89d9-6eab5205471a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0fc77f-6d886fb5cf-tht76\" (UID: \"5053ea34-585c-4b1d-89d9-6eab5205471a\") " pod="openshift-ingress/payload-processing-e2e-api-b-0fc77f-6d886fb5cf-tht76" Oct 31 04:21:18.610918 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:18.610434 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjvjg\" (UniqueName: \"kubernetes.io/projected/5053ea34-585c-4b1d-89d9-6eab5205471a-kube-api-access-zjvjg\") pod \"payload-processing-e2e-api-b-0fc77f-6d886fb5cf-tht76\" (UID: \"5053ea34-585c-4b1d-89d9-6eab5205471a\") " pod="openshift-ingress/payload-processing-e2e-api-b-0fc77f-6d886fb5cf-tht76" Oct 31 04:21:18.713216 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:18.713174 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5053ea34-585c-4b1d-89d9-6eab5205471a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0fc77f-6d886fb5cf-tht76\" (UID: \"5053ea34-585c-4b1d-89d9-6eab5205471a\") " pod="openshift-ingress/payload-processing-e2e-api-b-0fc77f-6d886fb5cf-tht76" Oct 31 04:21:18.713393 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:18.713300 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjvjg\" (UniqueName: \"kubernetes.io/projected/5053ea34-585c-4b1d-89d9-6eab5205471a-kube-api-access-zjvjg\") pod \"payload-processing-e2e-api-b-0fc77f-6d886fb5cf-tht76\" (UID: \"5053ea34-585c-4b1d-89d9-6eab5205471a\") " pod="openshift-ingress/payload-processing-e2e-api-b-0fc77f-6d886fb5cf-tht76" Oct 31 04:21:18.714110 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:18.714085 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5053ea34-585c-4b1d-89d9-6eab5205471a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0fc77f-6d886fb5cf-tht76\" (UID: \"5053ea34-585c-4b1d-89d9-6eab5205471a\") " pod="openshift-ingress/payload-processing-e2e-api-b-0fc77f-6d886fb5cf-tht76" Oct 31 04:21:18.742266 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:18.742180 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjvjg\" (UniqueName: \"kubernetes.io/projected/5053ea34-585c-4b1d-89d9-6eab5205471a-kube-api-access-zjvjg\") pod \"payload-processing-e2e-api-b-0fc77f-6d886fb5cf-tht76\" (UID: \"5053ea34-585c-4b1d-89d9-6eab5205471a\") " pod="openshift-ingress/payload-processing-e2e-api-b-0fc77f-6d886fb5cf-tht76" Oct 31 04:21:18.748142 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:18.747750 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35fb49a3-tthwf" Oct 31 04:21:18.806872 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:18.806830 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0fc77f-774488f5b7-jpmqw" Oct 31 04:21:18.814788 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:18.814755 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0fc77f-6d886fb5cf-tht76" Oct 31 04:21:18.815509 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:18.815477 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03b83c43-5a5c-48c1-a804-c92f53fa02a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03b83c43-5a5c-48c1-a804-c92f53fa02a2-maas-api-xks-ca\") pod \"03b83c43-5a5c-48c1-a804-c92f53fa02a2\" (UID: \"03b83c43-5a5c-48c1-a804-c92f53fa02a2\") " Oct 31 04:21:18.815669 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:18.815549 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03b83c43-5a5c-48c1-a804-c92f53fa02a2-kube-api-access-kf9g7\" (UniqueName: \"kubernetes.io/projected/03b83c43-5a5c-48c1-a804-c92f53fa02a2-kube-api-access-kf9g7\") pod \"03b83c43-5a5c-48c1-a804-c92f53fa02a2\" (UID: \"03b83c43-5a5c-48c1-a804-c92f53fa02a2\") " Oct 31 04:21:18.815734 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:18.815663 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03b83c43-5a5c-48c1-a804-c92f53fa02a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03b83c43-5a5c-48c1-a804-c92f53fa02a2-maas-api-openshift-ca\") pod \"03b83c43-5a5c-48c1-a804-c92f53fa02a2\" (UID: \"03b83c43-5a5c-48c1-a804-c92f53fa02a2\") " Oct 31 04:21:18.817385 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:18.817084 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03b83c43-5a5c-48c1-a804-c92f53fa02a2-maas-api-openshift-ca" pod "03b83c43-5a5c-48c1-a804-c92f53fa02a2" (UID: "03b83c43-5a5c-48c1-a804-c92f53fa02a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:21:18.821104 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:18.819764 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03b83c43-5a5c-48c1-a804-c92f53fa02a2-kube-api-access-kf9g7" pod "03b83c43-5a5c-48c1-a804-c92f53fa02a2" (UID: "03b83c43-5a5c-48c1-a804-c92f53fa02a2"). InnerVolumeSpecName "kube-api-access-kf9g7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:21:18.827865 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:18.827827 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03b83c43-5a5c-48c1-a804-c92f53fa02a2-maas-api-xks-ca" pod "03b83c43-5a5c-48c1-a804-c92f53fa02a2" (UID: "03b83c43-5a5c-48c1-a804-c92f53fa02a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:21:18.917997 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:18.917960 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf9g7\" (UniqueName: \"kubernetes.io/projected/03b83c43-5a5c-48c1-a804-c92f53fa02a2-kube-api-access-kf9g7\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:21:18.917997 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:18.917998 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03b83c43-5a5c-48c1-a804-c92f53fa02a2-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:21:18.918180 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:18.918013 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03b83c43-5a5c-48c1-a804-c92f53fa02a2-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:21:19.176655 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:21:19.176611 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda136b866_abc0_4c60_b977_ce6f05b0bdbf.slice/crio-545e5b0f7c1ba53777ab51655dd188ce4a79fda81e09afd521dea503bb8cc7bb WatchSource:0}: Error finding container 545e5b0f7c1ba53777ab51655dd188ce4a79fda81e09afd521dea503bb8cc7bb: Status 404 returned error can't find the container with id 545e5b0f7c1ba53777ab51655dd188ce4a79fda81e09afd521dea503bb8cc7bb Oct 31 04:21:19.177709 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:19.177580 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0fc77f-774488f5b7-jpmqw"] Oct 31 04:21:19.184002 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:19.183972 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0fc77f-6d886fb5cf-tht76"] Oct 31 04:21:19.186033 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:21:19.185993 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5053ea34_585c_4b1d_89d9_6eab5205471a.slice/crio-9e7eff6586dc24def34b93c53ff9e21c179ebd78cf51d94cdc563b190d5ab126 WatchSource:0}: Error finding container 9e7eff6586dc24def34b93c53ff9e21c179ebd78cf51d94cdc563b190d5ab126: Status 404 returned error can't find the container with id 9e7eff6586dc24def34b93c53ff9e21c179ebd78cf51d94cdc563b190d5ab126 Oct 31 04:21:19.519393 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:19.519296 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0fc77f-6d886fb5cf-tht76" event={"ID":"5053ea34-585c-4b1d-89d9-6eab5205471a","Type":"ContainerStarted","Data":"091f198ea7a6d09aafc5f7c16954cc9bc58b79684321b3bcd17a8699a38d32bb"} Oct 31 04:21:19.519393 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:19.519377 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0fc77f-6d886fb5cf-tht76" event={"ID":"5053ea34-585c-4b1d-89d9-6eab5205471a","Type":"ContainerStarted","Data":"9e7eff6586dc24def34b93c53ff9e21c179ebd78cf51d94cdc563b190d5ab126"} Oct 31 04:21:19.519900 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:19.519410 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-0fc77f-6d886fb5cf-tht76" Oct 31 04:21:19.521301 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:19.521155 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35fb49a3-tthwf" Oct 31 04:21:19.521301 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:19.521191 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35fb49a3-tthwf" event={"ID":"03b83c43-5a5c-48c1-a804-c92f53fa02a2","Type":"ContainerDied","Data":"b219089eff2b7902b9875741af2e037995257a2fc2e6f01e74f0fa4b1d697761"} Oct 31 04:21:19.521301 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:19.521245 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b219089eff2b7902b9875741af2e037995257a2fc2e6f01e74f0fa4b1d697761" Oct 31 04:21:19.523219 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:19.523188 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0fc77f-774488f5b7-jpmqw" event={"ID":"a136b866-abc0-4c60-b977-ce6f05b0bdbf","Type":"ContainerStarted","Data":"4ca1ac6b9878f8a426eef15f82591b32c92c075db278cc84c49dc223f41ec0b0"} Oct 31 04:21:19.523375 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:19.523243 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0fc77f-774488f5b7-jpmqw" event={"ID":"a136b866-abc0-4c60-b977-ce6f05b0bdbf","Type":"ContainerStarted","Data":"545e5b0f7c1ba53777ab51655dd188ce4a79fda81e09afd521dea503bb8cc7bb"} Oct 31 04:21:19.523375 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:19.523271 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0fc77f-774488f5b7-jpmqw" Oct 31 04:21:19.562244 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:19.562178 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-0fc77f-6d886fb5cf-tht76" podStartSLOduration=1.562152778 podStartE2EDuration="1.562152778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:21:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:21:19.559113283 +0000 UTC m=+1034.315451322" watchObservedRunningTime="2026-10-31 04:21:19.562152778 +0000 UTC m=+1034.318490820" Oct 31 04:21:19.615737 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:19.615666 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0fc77f-774488f5b7-jpmqw" podStartSLOduration=1.6156476309999999 podStartE2EDuration="1.615647631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:21:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:21:19.613570698 +0000 UTC m=+1034.369908739" watchObservedRunningTime="2026-10-31 04:21:19.615647631 +0000 UTC m=+1034.371985672" Oct 31 04:21:22.400371 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:21:22.400314 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:21:22.755355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:22.752434 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a43a70-579b95c6df-7m45l"] Oct 31 04:21:22.755355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:22.753688 5019 cpu_manager.go:414] "containerMap: removing container" podUID="03b83c43-5a5c-48c1-a804-c92f53fa02a2" containerName="revoke-keys" Oct 31 04:21:22.755355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:22.753755 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03b83c43-5a5c-48c1-a804-c92f53fa02a2" containerName="revoke-keys" Oct 31 04:21:22.755355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:22.753943 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="03b83c43-5a5c-48c1-a804-c92f53fa02a2" containerName="revoke-keys" Oct 31 04:21:22.861258 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:22.860581 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a43a70-579b95c6df-7m45l"] Oct 31 04:21:22.861258 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:22.860728 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a43a70-579b95c6df-7m45l" Oct 31 04:21:22.885376 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:22.878189 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-a43a70-dockercfg-8mx9p\"" Oct 31 04:21:22.893355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:22.889799 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-a43a70\"" Oct 31 04:21:22.996397 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:22.996343 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9a72222-d749-4677-9621-99f1306e4ffd-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a43a70-579b95c6df-7m45l\" (UID: \"a9a72222-d749-4677-9621-99f1306e4ffd\") " pod="openshift-ingress/payload-processing-e2e-api-b-a43a70-579b95c6df-7m45l" Oct 31 04:21:22.996610 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:22.996455 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcsml\" (UniqueName: \"kubernetes.io/projected/a9a72222-d749-4677-9621-99f1306e4ffd-kube-api-access-lcsml\") pod \"payload-processing-e2e-api-b-a43a70-579b95c6df-7m45l\" (UID: \"a9a72222-d749-4677-9621-99f1306e4ffd\") " pod="openshift-ingress/payload-processing-e2e-api-b-a43a70-579b95c6df-7m45l" Oct 31 04:21:23.098308 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:23.098206 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9a72222-d749-4677-9621-99f1306e4ffd-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a43a70-579b95c6df-7m45l\" (UID: \"a9a72222-d749-4677-9621-99f1306e4ffd\") " pod="openshift-ingress/payload-processing-e2e-api-b-a43a70-579b95c6df-7m45l" Oct 31 04:21:23.098493 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:23.098315 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcsml\" (UniqueName: \"kubernetes.io/projected/a9a72222-d749-4677-9621-99f1306e4ffd-kube-api-access-lcsml\") pod \"payload-processing-e2e-api-b-a43a70-579b95c6df-7m45l\" (UID: \"a9a72222-d749-4677-9621-99f1306e4ffd\") " pod="openshift-ingress/payload-processing-e2e-api-b-a43a70-579b95c6df-7m45l" Oct 31 04:21:23.099208 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:23.099182 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9a72222-d749-4677-9621-99f1306e4ffd-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a43a70-579b95c6df-7m45l\" (UID: \"a9a72222-d749-4677-9621-99f1306e4ffd\") " pod="openshift-ingress/payload-processing-e2e-api-b-a43a70-579b95c6df-7m45l" Oct 31 04:21:23.122167 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:23.122128 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcsml\" (UniqueName: \"kubernetes.io/projected/a9a72222-d749-4677-9621-99f1306e4ffd-kube-api-access-lcsml\") pod \"payload-processing-e2e-api-b-a43a70-579b95c6df-7m45l\" (UID: \"a9a72222-d749-4677-9621-99f1306e4ffd\") " pod="openshift-ingress/payload-processing-e2e-api-b-a43a70-579b95c6df-7m45l" Oct 31 04:21:23.205030 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:23.204980 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a43a70-579b95c6df-7m45l" Oct 31 04:21:23.517419 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:21:23.517361 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9a72222_d749_4677_9621_99f1306e4ffd.slice/crio-8a5abe34a5fae78050c5950edd74a53b34638dc5e346f04816d11d122ed88524 WatchSource:0}: Error finding container 8a5abe34a5fae78050c5950edd74a53b34638dc5e346f04816d11d122ed88524: Status 404 returned error can't find the container with id 8a5abe34a5fae78050c5950edd74a53b34638dc5e346f04816d11d122ed88524 Oct 31 04:21:23.519384 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:23.519317 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a43a70-579b95c6df-7m45l"] Oct 31 04:21:23.566253 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:23.566210 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a43a70-579b95c6df-7m45l" event={"ID":"a9a72222-d749-4677-9621-99f1306e4ffd","Type":"ContainerStarted","Data":"8a5abe34a5fae78050c5950edd74a53b34638dc5e346f04816d11d122ed88524"} Oct 31 04:21:24.591376 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:24.587953 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a43a70-579b95c6df-7m45l" event={"ID":"a9a72222-d749-4677-9621-99f1306e4ffd","Type":"ContainerStarted","Data":"5f10c724d1a5eb3d491667c020774062dacc30652fae3f63c62000df580151b6"} Oct 31 04:21:24.591376 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:24.588067 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-a43a70-579b95c6df-7m45l" Oct 31 04:21:24.634705 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:24.634633 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-a43a70-579b95c6df-7m45l" podStartSLOduration=2.634611374 podStartE2EDuration="2.634611374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:21:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:21:24.629101105 +0000 UTC m=+1039.385439163" watchObservedRunningTime="2026-10-31 04:21:24.634611374 +0000 UTC m=+1039.390949414" Oct 31 04:21:30.537630 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:30.537598 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-0fc77f-6d886fb5cf-tht76" Oct 31 04:21:30.538154 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:30.537671 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0fc77f-774488f5b7-jpmqw" Oct 31 04:21:31.034180 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:31.034138 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e700e24-547r4"] Oct 31 04:21:31.040135 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:31.040092 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e700e24-547r4"] Oct 31 04:21:31.411193 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:31.411097 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdd06518-6e8d-4d39-af45-f70ace8e9ab7" path="/var/lib/kubelet/pods/cdd06518-6e8d-4d39-af45-f70ace8e9ab7/volumes" Oct 31 04:21:35.606385 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:35.606354 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-a43a70-579b95c6df-7m45l" Oct 31 04:21:38.030404 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:38.030359 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2f19c96-4ncv7"] Oct 31 04:21:38.042061 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:38.042027 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2f19c96-4ncv7"] Oct 31 04:21:38.976671 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:38.976627 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4"] Oct 31 04:21:38.984227 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:38.984190 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4" Oct 31 04:21:39.024809 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:39.024753 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4"] Oct 31 04:21:39.085709 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:39.085653 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnfzt\" (UniqueName: \"kubernetes.io/projected/b35297c9-af92-4356-9951-2a90b09fb3cd-kube-api-access-qnfzt\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4\" (UID: \"b35297c9-af92-4356-9951-2a90b09fb3cd\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4" Oct 31 04:21:39.086209 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:39.085812 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b35297c9-af92-4356-9951-2a90b09fb3cd-istio-envoy\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4\" (UID: \"b35297c9-af92-4356-9951-2a90b09fb3cd\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4" Oct 31 04:21:39.086209 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:39.085972 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b35297c9-af92-4356-9951-2a90b09fb3cd-credential-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4\" (UID: \"b35297c9-af92-4356-9951-2a90b09fb3cd\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4" Oct 31 04:21:39.086209 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:39.086021 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b35297c9-af92-4356-9951-2a90b09fb3cd-workload-certs\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4\" (UID: \"b35297c9-af92-4356-9951-2a90b09fb3cd\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4" Oct 31 04:21:39.086209 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:39.086083 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b35297c9-af92-4356-9951-2a90b09fb3cd-istio-data\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4\" (UID: \"b35297c9-af92-4356-9951-2a90b09fb3cd\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4" Oct 31 04:21:39.086209 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:39.086138 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b35297c9-af92-4356-9951-2a90b09fb3cd-workload-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4\" (UID: \"b35297c9-af92-4356-9951-2a90b09fb3cd\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4" Oct 31 04:21:39.086209 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:39.086163 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b35297c9-af92-4356-9951-2a90b09fb3cd-istio-token\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4\" (UID: \"b35297c9-af92-4356-9951-2a90b09fb3cd\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4" Oct 31 04:21:39.086537 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:39.086293 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b35297c9-af92-4356-9951-2a90b09fb3cd-istio-podinfo\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4\" (UID: \"b35297c9-af92-4356-9951-2a90b09fb3cd\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4" Oct 31 04:21:39.086537 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:39.086403 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b35297c9-af92-4356-9951-2a90b09fb3cd-istiod-ca-cert\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4\" (UID: \"b35297c9-af92-4356-9951-2a90b09fb3cd\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4" Oct 31 04:21:39.188050 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:39.188004 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b35297c9-af92-4356-9951-2a90b09fb3cd-istio-envoy\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4\" (UID: \"b35297c9-af92-4356-9951-2a90b09fb3cd\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4" Oct 31 04:21:39.188262 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:39.188161 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b35297c9-af92-4356-9951-2a90b09fb3cd-credential-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4\" (UID: \"b35297c9-af92-4356-9951-2a90b09fb3cd\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4" Oct 31 04:21:39.188262 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:39.188192 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b35297c9-af92-4356-9951-2a90b09fb3cd-workload-certs\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4\" (UID: \"b35297c9-af92-4356-9951-2a90b09fb3cd\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4" Oct 31 04:21:39.188262 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:39.188219 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b35297c9-af92-4356-9951-2a90b09fb3cd-istio-data\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4\" (UID: \"b35297c9-af92-4356-9951-2a90b09fb3cd\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4" Oct 31 04:21:39.188461 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:39.188270 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b35297c9-af92-4356-9951-2a90b09fb3cd-workload-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4\" (UID: \"b35297c9-af92-4356-9951-2a90b09fb3cd\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4" Oct 31 04:21:39.188461 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:39.188298 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b35297c9-af92-4356-9951-2a90b09fb3cd-istio-token\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4\" (UID: \"b35297c9-af92-4356-9951-2a90b09fb3cd\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4" Oct 31 04:21:39.188461 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:39.188431 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b35297c9-af92-4356-9951-2a90b09fb3cd-istio-podinfo\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4\" (UID: \"b35297c9-af92-4356-9951-2a90b09fb3cd\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4" Oct 31 04:21:39.188625 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:39.188488 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b35297c9-af92-4356-9951-2a90b09fb3cd-istiod-ca-cert\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4\" (UID: \"b35297c9-af92-4356-9951-2a90b09fb3cd\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4" Oct 31 04:21:39.188625 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:39.188560 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnfzt\" (UniqueName: \"kubernetes.io/projected/b35297c9-af92-4356-9951-2a90b09fb3cd-kube-api-access-qnfzt\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4\" (UID: \"b35297c9-af92-4356-9951-2a90b09fb3cd\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4" Oct 31 04:21:39.189103 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:39.188685 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b35297c9-af92-4356-9951-2a90b09fb3cd-credential-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4\" (UID: \"b35297c9-af92-4356-9951-2a90b09fb3cd\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4" Oct 31 04:21:39.189103 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:39.188751 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b35297c9-af92-4356-9951-2a90b09fb3cd-workload-certs\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4\" (UID: \"b35297c9-af92-4356-9951-2a90b09fb3cd\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4" Oct 31 04:21:39.189103 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:39.189002 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b35297c9-af92-4356-9951-2a90b09fb3cd-workload-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4\" (UID: \"b35297c9-af92-4356-9951-2a90b09fb3cd\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4" Oct 31 04:21:39.189248 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:39.189158 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b35297c9-af92-4356-9951-2a90b09fb3cd-istio-data\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4\" (UID: \"b35297c9-af92-4356-9951-2a90b09fb3cd\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4" Oct 31 04:21:39.189643 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:39.189621 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b35297c9-af92-4356-9951-2a90b09fb3cd-istiod-ca-cert\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4\" (UID: \"b35297c9-af92-4356-9951-2a90b09fb3cd\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4" Oct 31 04:21:39.191388 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:39.191315 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b35297c9-af92-4356-9951-2a90b09fb3cd-istio-podinfo\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4\" (UID: \"b35297c9-af92-4356-9951-2a90b09fb3cd\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4" Oct 31 04:21:39.191488 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:39.191366 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b35297c9-af92-4356-9951-2a90b09fb3cd-istio-envoy\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4\" (UID: \"b35297c9-af92-4356-9951-2a90b09fb3cd\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4" Oct 31 04:21:39.212202 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:39.212157 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b35297c9-af92-4356-9951-2a90b09fb3cd-istio-token\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4\" (UID: \"b35297c9-af92-4356-9951-2a90b09fb3cd\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4" Oct 31 04:21:39.212543 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:39.212517 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnfzt\" (UniqueName: \"kubernetes.io/projected/b35297c9-af92-4356-9951-2a90b09fb3cd-kube-api-access-qnfzt\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4\" (UID: \"b35297c9-af92-4356-9951-2a90b09fb3cd\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4" Oct 31 04:21:39.335136 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:39.335030 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4" Oct 31 04:21:39.413882 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:39.413848 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="877cc431-548f-4df1-bcbe-a4d63c5f5595" path="/var/lib/kubelet/pods/877cc431-548f-4df1-bcbe-a4d63c5f5595/volumes" Oct 31 04:21:39.668272 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:21:39.668235 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb35297c9_af92_4356_9951_2a90b09fb3cd.slice/crio-8c27acbf208c35d6b3a120e7a5aa367dae0cd8bce64effde3a7740eb8267656b WatchSource:0}: Error finding container 8c27acbf208c35d6b3a120e7a5aa367dae0cd8bce64effde3a7740eb8267656b: Status 404 returned error can't find the container with id 8c27acbf208c35d6b3a120e7a5aa367dae0cd8bce64effde3a7740eb8267656b Oct 31 04:21:39.668964 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:39.668934 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4"] Oct 31 04:21:39.671295 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:39.671251 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:21:39.671410 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:39.671368 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:21:39.671455 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:39.671414 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:21:39.798516 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:39.798476 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4" event={"ID":"b35297c9-af92-4356-9951-2a90b09fb3cd","Type":"ContainerStarted","Data":"eb9ef275fc5a81b1ef005c5b04d7a927d89d420a8dda42bb37679e3d305ba699"} Oct 31 04:21:39.798737 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:39.798544 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4" event={"ID":"b35297c9-af92-4356-9951-2a90b09fb3cd","Type":"ContainerStarted","Data":"8c27acbf208c35d6b3a120e7a5aa367dae0cd8bce64effde3a7740eb8267656b"} Oct 31 04:21:39.849588 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:39.849519 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4" podStartSLOduration=1.849497211 podStartE2EDuration="1.849497211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:21:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:21:39.843452315 +0000 UTC m=+1054.599790355" watchObservedRunningTime="2026-10-31 04:21:39.849497211 +0000 UTC m=+1054.605835318" Oct 31 04:21:40.335608 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:40.335570 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4" Oct 31 04:21:40.337146 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:40.337119 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 31 04:21:40.337294 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:40.337195 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4" podUID="b35297c9-af92-4356-9951-2a90b09fb3cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 31 04:21:41.336106 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:41.336069 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 31 04:21:41.336609 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:41.336135 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4" podUID="b35297c9-af92-4356-9951-2a90b09fb3cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 31 04:21:42.336024 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:42.335982 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 31 04:21:42.336225 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:42.336046 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4" podUID="b35297c9-af92-4356-9951-2a90b09fb3cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 31 04:21:43.034017 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:43.033972 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba3c9c5d-x862m"] Oct 31 04:21:43.045397 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:43.045359 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba3c9c5d-x862m"] Oct 31 04:21:43.336643 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:43.336539 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 31 04:21:43.336643 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:43.336617 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4" podUID="b35297c9-af92-4356-9951-2a90b09fb3cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 31 04:21:43.445172 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:43.445129 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ab5359a-8e15-4809-a42d-d6855696376d" path="/var/lib/kubelet/pods/0ab5359a-8e15-4809-a42d-d6855696376d/volumes" Oct 31 04:21:44.336021 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:44.335982 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 31 04:21:44.336218 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:44.336050 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4" podUID="b35297c9-af92-4356-9951-2a90b09fb3cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 31 04:21:45.336399 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:45.336315 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 31 04:21:45.336788 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:45.336448 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4" podUID="b35297c9-af92-4356-9951-2a90b09fb3cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 31 04:21:46.336213 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:46.336173 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 31 04:21:46.336513 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:46.336248 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4" podUID="b35297c9-af92-4356-9951-2a90b09fb3cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 31 04:21:47.340719 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:47.340686 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4" Oct 31 04:21:47.341163 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:47.341148 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4" Oct 31 04:21:47.341961 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:47.341930 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4" Oct 31 04:21:49.027126 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:49.027067 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2a7c5b1-dj9cz"] Oct 31 04:21:49.038187 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:49.036399 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a7c5b1-dj9cz" Oct 31 04:21:49.050837 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:49.050647 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2a7c5b1-dj9cz"] Oct 31 04:21:49.066658 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:49.066623 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:21:49.066864 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:49.066741 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:21:49.066864 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:49.066772 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:21:49.066864 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:49.066848 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:21:49.116069 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:49.115980 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j4sb\" (UniqueName: \"kubernetes.io/projected/5a5bf674-97e5-4f45-be51-190d9fc24a71-kube-api-access-5j4sb\") pod \"maas-api-revoke-sub-e2a7c5b1-dj9cz\" (UID: \"5a5bf674-97e5-4f45-be51-190d9fc24a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a7c5b1-dj9cz" Oct 31 04:21:49.116974 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:49.116944 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a5bf674-97e5-4f45-be51-190d9fc24a71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2a7c5b1-dj9cz\" (UID: \"5a5bf674-97e5-4f45-be51-190d9fc24a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a7c5b1-dj9cz" Oct 31 04:21:49.117422 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:49.117399 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a5bf674-97e5-4f45-be51-190d9fc24a71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2a7c5b1-dj9cz\" (UID: \"5a5bf674-97e5-4f45-be51-190d9fc24a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a7c5b1-dj9cz" Oct 31 04:21:49.228525 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:49.228476 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a5bf674-97e5-4f45-be51-190d9fc24a71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2a7c5b1-dj9cz\" (UID: \"5a5bf674-97e5-4f45-be51-190d9fc24a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a7c5b1-dj9cz" Oct 31 04:21:49.229017 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:49.228988 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a5bf674-97e5-4f45-be51-190d9fc24a71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2a7c5b1-dj9cz\" (UID: \"5a5bf674-97e5-4f45-be51-190d9fc24a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a7c5b1-dj9cz" Oct 31 04:21:49.229289 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:49.229268 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j4sb\" (UniqueName: \"kubernetes.io/projected/5a5bf674-97e5-4f45-be51-190d9fc24a71-kube-api-access-5j4sb\") pod \"maas-api-revoke-sub-e2a7c5b1-dj9cz\" (UID: \"5a5bf674-97e5-4f45-be51-190d9fc24a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a7c5b1-dj9cz" Oct 31 04:21:49.248805 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:49.248765 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a5bf674-97e5-4f45-be51-190d9fc24a71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2a7c5b1-dj9cz\" (UID: \"5a5bf674-97e5-4f45-be51-190d9fc24a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a7c5b1-dj9cz" Oct 31 04:21:49.314313 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:49.314185 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j4sb\" (UniqueName: \"kubernetes.io/projected/5a5bf674-97e5-4f45-be51-190d9fc24a71-kube-api-access-5j4sb\") pod \"maas-api-revoke-sub-e2a7c5b1-dj9cz\" (UID: \"5a5bf674-97e5-4f45-be51-190d9fc24a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a7c5b1-dj9cz" Oct 31 04:21:49.315057 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:49.315034 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a5bf674-97e5-4f45-be51-190d9fc24a71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2a7c5b1-dj9cz\" (UID: \"5a5bf674-97e5-4f45-be51-190d9fc24a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a7c5b1-dj9cz" Oct 31 04:21:49.385418 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:49.385308 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a7c5b1-dj9cz" Oct 31 04:21:49.749247 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:49.749206 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2a7c5b1-dj9cz"] Oct 31 04:21:50.030418 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:50.029341 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a7c5b1-dj9cz" event={"ID":"5a5bf674-97e5-4f45-be51-190d9fc24a71","Type":"ContainerStarted","Data":"5c58dbf38d22ac6677c7ec65971799d631615ffcc3f7407aca1c52ac77317925"} Oct 31 04:21:50.615742 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:50.615689 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b56d736e-7s7nz"] Oct 31 04:21:50.622196 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:50.622161 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b56d736e-7s7nz" Oct 31 04:21:50.624825 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:50.624783 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b56d736e-7s7nz"] Oct 31 04:21:50.729078 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:50.729026 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-mjq8n-nfjq4"] Oct 31 04:21:50.734598 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:50.734566 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-mjq8n-nfjq4" Oct 31 04:21:50.745211 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:50.745178 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-mjq8n-nfjq4"] Oct 31 04:21:50.809366 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:50.808450 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce-trusted-ca-bundle\") pod \"periodic-gathering-mjq8n-nfjq4\" (UID: \"f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce\") " pod="openshift-insights/periodic-gathering-mjq8n-nfjq4" Oct 31 04:21:50.809366 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:50.808526 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce-service-ca-bundle\") pod \"periodic-gathering-mjq8n-nfjq4\" (UID: \"f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce\") " pod="openshift-insights/periodic-gathering-mjq8n-nfjq4" Oct 31 04:21:50.809802 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:50.809776 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltlkj\" (UniqueName: \"kubernetes.io/projected/ee3bb425-3793-4d03-b0b9-d3a0025e3b90-kube-api-access-ltlkj\") pod \"maas-api-revoke-sub-b56d736e-7s7nz\" (UID: \"ee3bb425-3793-4d03-b0b9-d3a0025e3b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b56d736e-7s7nz" Oct 31 04:21:50.810010 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:50.809982 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce-archives-path\") pod \"periodic-gathering-mjq8n-nfjq4\" (UID: \"f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce\") " pod="openshift-insights/periodic-gathering-mjq8n-nfjq4" Oct 31 04:21:50.810107 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:50.810087 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee3bb425-3793-4d03-b0b9-d3a0025e3b90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b56d736e-7s7nz\" (UID: \"ee3bb425-3793-4d03-b0b9-d3a0025e3b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b56d736e-7s7nz" Oct 31 04:21:50.810192 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:50.810177 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee3bb425-3793-4d03-b0b9-d3a0025e3b90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b56d736e-7s7nz\" (UID: \"ee3bb425-3793-4d03-b0b9-d3a0025e3b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b56d736e-7s7nz" Oct 31 04:21:50.810244 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:50.810231 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2286g\" (UniqueName: \"kubernetes.io/projected/f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce-kube-api-access-2286g\") pod \"periodic-gathering-mjq8n-nfjq4\" (UID: \"f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce\") " pod="openshift-insights/periodic-gathering-mjq8n-nfjq4" Oct 31 04:21:50.911534 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:50.911491 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce-archives-path\") pod \"periodic-gathering-mjq8n-nfjq4\" (UID: \"f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce\") " pod="openshift-insights/periodic-gathering-mjq8n-nfjq4" Oct 31 04:21:50.911806 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:50.911575 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee3bb425-3793-4d03-b0b9-d3a0025e3b90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b56d736e-7s7nz\" (UID: \"ee3bb425-3793-4d03-b0b9-d3a0025e3b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b56d736e-7s7nz" Oct 31 04:21:50.911806 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:50.911653 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee3bb425-3793-4d03-b0b9-d3a0025e3b90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b56d736e-7s7nz\" (UID: \"ee3bb425-3793-4d03-b0b9-d3a0025e3b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b56d736e-7s7nz" Oct 31 04:21:50.911806 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:50.911676 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2286g\" (UniqueName: \"kubernetes.io/projected/f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce-kube-api-access-2286g\") pod \"periodic-gathering-mjq8n-nfjq4\" (UID: \"f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce\") " pod="openshift-insights/periodic-gathering-mjq8n-nfjq4" Oct 31 04:21:50.911806 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:50.911739 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce-trusted-ca-bundle\") pod \"periodic-gathering-mjq8n-nfjq4\" (UID: \"f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce\") " pod="openshift-insights/periodic-gathering-mjq8n-nfjq4" Oct 31 04:21:50.911806 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:50.911759 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce-service-ca-bundle\") pod \"periodic-gathering-mjq8n-nfjq4\" (UID: \"f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce\") " pod="openshift-insights/periodic-gathering-mjq8n-nfjq4" Oct 31 04:21:50.912101 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:50.911835 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltlkj\" (UniqueName: \"kubernetes.io/projected/ee3bb425-3793-4d03-b0b9-d3a0025e3b90-kube-api-access-ltlkj\") pod \"maas-api-revoke-sub-b56d736e-7s7nz\" (UID: \"ee3bb425-3793-4d03-b0b9-d3a0025e3b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b56d736e-7s7nz" Oct 31 04:21:50.912101 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:50.912019 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce-archives-path\") pod \"periodic-gathering-mjq8n-nfjq4\" (UID: \"f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce\") " pod="openshift-insights/periodic-gathering-mjq8n-nfjq4" Oct 31 04:21:50.913159 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:50.913126 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce-service-ca-bundle\") pod \"periodic-gathering-mjq8n-nfjq4\" (UID: \"f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce\") " pod="openshift-insights/periodic-gathering-mjq8n-nfjq4" Oct 31 04:21:50.913357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:50.913148 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce-trusted-ca-bundle\") pod \"periodic-gathering-mjq8n-nfjq4\" (UID: \"f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce\") " pod="openshift-insights/periodic-gathering-mjq8n-nfjq4" Oct 31 04:21:50.913357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:50.913156 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee3bb425-3793-4d03-b0b9-d3a0025e3b90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b56d736e-7s7nz\" (UID: \"ee3bb425-3793-4d03-b0b9-d3a0025e3b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b56d736e-7s7nz" Oct 31 04:21:50.915418 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:50.915395 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee3bb425-3793-4d03-b0b9-d3a0025e3b90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b56d736e-7s7nz\" (UID: \"ee3bb425-3793-4d03-b0b9-d3a0025e3b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b56d736e-7s7nz" Oct 31 04:21:50.935346 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:50.935295 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltlkj\" (UniqueName: \"kubernetes.io/projected/ee3bb425-3793-4d03-b0b9-d3a0025e3b90-kube-api-access-ltlkj\") pod \"maas-api-revoke-sub-b56d736e-7s7nz\" (UID: \"ee3bb425-3793-4d03-b0b9-d3a0025e3b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b56d736e-7s7nz" Oct 31 04:21:50.936113 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:50.935352 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2286g\" (UniqueName: \"kubernetes.io/projected/f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce-kube-api-access-2286g\") pod \"periodic-gathering-mjq8n-nfjq4\" (UID: \"f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce\") " pod="openshift-insights/periodic-gathering-mjq8n-nfjq4" Oct 31 04:21:50.943241 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:50.943208 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b56d736e-7s7nz" Oct 31 04:21:51.044652 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:51.044556 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a7c5b1-dj9cz" event={"ID":"5a5bf674-97e5-4f45-be51-190d9fc24a71","Type":"ContainerStarted","Data":"9bdcd4d3dd00e3db2f82875cfe06b7454b451411ad8e7c4827a8c8f3bcae22cb"} Oct 31 04:21:51.054393 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:51.054100 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-mjq8n-nfjq4" Oct 31 04:21:51.130808 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:51.130739 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a7c5b1-dj9cz" podStartSLOduration=3.130718635 podStartE2EDuration="3.130718635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:21:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:21:51.111963648 +0000 UTC m=+1065.868301691" watchObservedRunningTime="2026-10-31 04:21:51.130718635 +0000 UTC m=+1065.887056676" Oct 31 04:21:51.342496 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:51.342256 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b56d736e-7s7nz"] Oct 31 04:21:51.545717 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:21:51.545674 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1a31334_e2b6_4e0b_93d4_ee9a003dd2ce.slice/crio-3b6e8d378ba8add6bb7620fa35fb8a3c90a2a4ad10041efca0fabb5440cb7bc3 WatchSource:0}: Error finding container 3b6e8d378ba8add6bb7620fa35fb8a3c90a2a4ad10041efca0fabb5440cb7bc3: Status 404 returned error can't find the container with id 3b6e8d378ba8add6bb7620fa35fb8a3c90a2a4ad10041efca0fabb5440cb7bc3 Oct 31 04:21:51.546583 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:51.546555 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-mjq8n-nfjq4"] Oct 31 04:21:52.056423 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:52.056309 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b56d736e-7s7nz" event={"ID":"ee3bb425-3793-4d03-b0b9-d3a0025e3b90","Type":"ContainerStarted","Data":"3733ebda239dbfa78f06d0e05cef44efd34de07205294fe1b927681cb03c0fd3"} Oct 31 04:21:52.056872 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:52.056458 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b56d736e-7s7nz" event={"ID":"ee3bb425-3793-4d03-b0b9-d3a0025e3b90","Type":"ContainerStarted","Data":"d070c883d3ccdbf1084a486f030e685f482790a8917e5f3c6f2267498c4bd39a"} Oct 31 04:21:52.058782 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:52.058748 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-mjq8n-nfjq4" event={"ID":"f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce","Type":"ContainerStarted","Data":"16d60cd88841ca545a8c86b48b42ed682c09a0954529ba62ee2762b9eb064aaa"} Oct 31 04:21:52.058958 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:52.058802 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-mjq8n-nfjq4" event={"ID":"f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce","Type":"ContainerStarted","Data":"3b6e8d378ba8add6bb7620fa35fb8a3c90a2a4ad10041efca0fabb5440cb7bc3"} Oct 31 04:21:52.060849 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:52.060817 5019 generic.go:356] "Generic (PLEG): container finished" podID="5a5bf674-97e5-4f45-be51-190d9fc24a71" containerID="9bdcd4d3dd00e3db2f82875cfe06b7454b451411ad8e7c4827a8c8f3bcae22cb" exitCode=0 Oct 31 04:21:52.060993 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:52.060895 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a7c5b1-dj9cz" event={"ID":"5a5bf674-97e5-4f45-be51-190d9fc24a71","Type":"ContainerDied","Data":"9bdcd4d3dd00e3db2f82875cfe06b7454b451411ad8e7c4827a8c8f3bcae22cb"} Oct 31 04:21:52.120134 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:52.120062 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b56d736e-7s7nz" podStartSLOduration=2.120043129 podStartE2EDuration="2.120043129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:21:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:21:52.119190152 +0000 UTC m=+1066.875528192" watchObservedRunningTime="2026-10-31 04:21:52.120043129 +0000 UTC m=+1066.876381170" Oct 31 04:21:52.429781 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:21:52.429690 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:21:53.038321 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:53.038249 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-mjq8n-nfjq4" podStartSLOduration=3.038228452 podStartE2EDuration="3.038228452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:21:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:21:52.2230491 +0000 UTC m=+1066.979387139" watchObservedRunningTime="2026-10-31 04:21:53.038228452 +0000 UTC m=+1067.794566491" Oct 31 04:21:53.040988 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:53.040953 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49e61f07-xwlzd"] Oct 31 04:21:53.057410 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:53.054556 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49e61f07-xwlzd"] Oct 31 04:21:53.108356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:53.107756 5019 generic.go:356] "Generic (PLEG): container finished" podID="ee3bb425-3793-4d03-b0b9-d3a0025e3b90" containerID="3733ebda239dbfa78f06d0e05cef44efd34de07205294fe1b927681cb03c0fd3" exitCode=0 Oct 31 04:21:53.109364 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:53.109292 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b56d736e-7s7nz" event={"ID":"ee3bb425-3793-4d03-b0b9-d3a0025e3b90","Type":"ContainerDied","Data":"3733ebda239dbfa78f06d0e05cef44efd34de07205294fe1b927681cb03c0fd3"} Oct 31 04:21:53.418174 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:53.418132 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f1cb632-2629-49a0-99fe-5170142214e4" path="/var/lib/kubelet/pods/5f1cb632-2629-49a0-99fe-5170142214e4/volumes" Oct 31 04:21:53.451587 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:53.451546 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a7c5b1-dj9cz" Oct 31 04:21:53.516046 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:53.516002 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a5bf674-97e5-4f45-be51-190d9fc24a71-kube-api-access-5j4sb\" (UniqueName: \"kubernetes.io/projected/5a5bf674-97e5-4f45-be51-190d9fc24a71-kube-api-access-5j4sb\") pod \"5a5bf674-97e5-4f45-be51-190d9fc24a71\" (UID: \"5a5bf674-97e5-4f45-be51-190d9fc24a71\") " Oct 31 04:21:53.516255 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:53.516195 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a5bf674-97e5-4f45-be51-190d9fc24a71-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a5bf674-97e5-4f45-be51-190d9fc24a71-maas-api-openshift-ca\") pod \"5a5bf674-97e5-4f45-be51-190d9fc24a71\" (UID: \"5a5bf674-97e5-4f45-be51-190d9fc24a71\") " Oct 31 04:21:53.516366 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:53.516283 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a5bf674-97e5-4f45-be51-190d9fc24a71-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a5bf674-97e5-4f45-be51-190d9fc24a71-maas-api-xks-ca\") pod \"5a5bf674-97e5-4f45-be51-190d9fc24a71\" (UID: \"5a5bf674-97e5-4f45-be51-190d9fc24a71\") " Oct 31 04:21:53.517510 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:53.517483 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a5bf674-97e5-4f45-be51-190d9fc24a71-maas-api-openshift-ca" pod "5a5bf674-97e5-4f45-be51-190d9fc24a71" (UID: "5a5bf674-97e5-4f45-be51-190d9fc24a71"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:21:53.525470 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:53.525428 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a5bf674-97e5-4f45-be51-190d9fc24a71-kube-api-access-5j4sb" pod "5a5bf674-97e5-4f45-be51-190d9fc24a71" (UID: "5a5bf674-97e5-4f45-be51-190d9fc24a71"). InnerVolumeSpecName "kube-api-access-5j4sb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:21:53.525965 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:53.525938 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a5bf674-97e5-4f45-be51-190d9fc24a71-maas-api-xks-ca" pod "5a5bf674-97e5-4f45-be51-190d9fc24a71" (UID: "5a5bf674-97e5-4f45-be51-190d9fc24a71"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:21:53.619475 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:53.619430 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5j4sb\" (UniqueName: \"kubernetes.io/projected/5a5bf674-97e5-4f45-be51-190d9fc24a71-kube-api-access-5j4sb\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:21:53.619475 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:53.619476 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a5bf674-97e5-4f45-be51-190d9fc24a71-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:21:53.619700 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:53.619490 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a5bf674-97e5-4f45-be51-190d9fc24a71-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:21:54.119473 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:54.119438 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a7c5b1-dj9cz" Oct 31 04:21:54.120822 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:54.119466 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a7c5b1-dj9cz" event={"ID":"5a5bf674-97e5-4f45-be51-190d9fc24a71","Type":"ContainerDied","Data":"5c58dbf38d22ac6677c7ec65971799d631615ffcc3f7407aca1c52ac77317925"} Oct 31 04:21:54.120924 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:54.120902 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c58dbf38d22ac6677c7ec65971799d631615ffcc3f7407aca1c52ac77317925" Oct 31 04:21:54.371439 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:54.369823 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b56d736e-7s7nz" Oct 31 04:21:54.440007 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:54.439971 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee3bb425-3793-4d03-b0b9-d3a0025e3b90-kube-api-access-ltlkj\" (UniqueName: \"kubernetes.io/projected/ee3bb425-3793-4d03-b0b9-d3a0025e3b90-kube-api-access-ltlkj\") pod \"ee3bb425-3793-4d03-b0b9-d3a0025e3b90\" (UID: \"ee3bb425-3793-4d03-b0b9-d3a0025e3b90\") " Oct 31 04:21:54.440432 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:54.440404 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee3bb425-3793-4d03-b0b9-d3a0025e3b90-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee3bb425-3793-4d03-b0b9-d3a0025e3b90-maas-api-xks-ca\") pod \"ee3bb425-3793-4d03-b0b9-d3a0025e3b90\" (UID: \"ee3bb425-3793-4d03-b0b9-d3a0025e3b90\") " Oct 31 04:21:54.440641 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:54.440622 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee3bb425-3793-4d03-b0b9-d3a0025e3b90-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee3bb425-3793-4d03-b0b9-d3a0025e3b90-maas-api-openshift-ca\") pod \"ee3bb425-3793-4d03-b0b9-d3a0025e3b90\" (UID: \"ee3bb425-3793-4d03-b0b9-d3a0025e3b90\") " Oct 31 04:21:54.441751 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:54.441726 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee3bb425-3793-4d03-b0b9-d3a0025e3b90-maas-api-openshift-ca" pod "ee3bb425-3793-4d03-b0b9-d3a0025e3b90" (UID: "ee3bb425-3793-4d03-b0b9-d3a0025e3b90"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:21:54.442463 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:54.442442 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee3bb425-3793-4d03-b0b9-d3a0025e3b90-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:21:54.444776 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:54.444749 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee3bb425-3793-4d03-b0b9-d3a0025e3b90-maas-api-xks-ca" pod "ee3bb425-3793-4d03-b0b9-d3a0025e3b90" (UID: "ee3bb425-3793-4d03-b0b9-d3a0025e3b90"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:21:54.444898 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:54.444882 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee3bb425-3793-4d03-b0b9-d3a0025e3b90-kube-api-access-ltlkj" pod "ee3bb425-3793-4d03-b0b9-d3a0025e3b90" (UID: "ee3bb425-3793-4d03-b0b9-d3a0025e3b90"). InnerVolumeSpecName "kube-api-access-ltlkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:21:54.544884 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:54.544853 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee3bb425-3793-4d03-b0b9-d3a0025e3b90-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:21:54.545070 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:54.544893 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltlkj\" (UniqueName: \"kubernetes.io/projected/ee3bb425-3793-4d03-b0b9-d3a0025e3b90-kube-api-access-ltlkj\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:21:55.131813 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:55.131780 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b56d736e-7s7nz" Oct 31 04:21:55.132257 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:55.131813 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b56d736e-7s7nz" event={"ID":"ee3bb425-3793-4d03-b0b9-d3a0025e3b90","Type":"ContainerDied","Data":"d070c883d3ccdbf1084a486f030e685f482790a8917e5f3c6f2267498c4bd39a"} Oct 31 04:21:55.132257 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:55.131868 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d070c883d3ccdbf1084a486f030e685f482790a8917e5f3c6f2267498c4bd39a" Oct 31 04:21:56.031611 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:56.031557 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-160cd22e-dh4ff"] Oct 31 04:21:56.049376 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:56.049323 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-160cd22e-dh4ff"] Oct 31 04:21:57.411066 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:21:57.411025 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="558750ea-02ff-4ff2-b126-4ebb050f9092" path="/var/lib/kubelet/pods/558750ea-02ff-4ff2-b126-4ebb050f9092/volumes" Oct 31 04:22:00.413551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:00.413499 5019 patch_prober.go:29] interesting pod/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:22:00.414250 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:00.413684 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv" podUID="ffd1c004-3338-4cf6-b6f7-afebf7d01574" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:22:01.200808 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:01.200706 5019 generic.go:356] "Generic (PLEG): container finished" podID="ffd1c004-3338-4cf6-b6f7-afebf7d01574" containerID="dece7da60db378cea7ec0d7fef6b187d2624546e21795d2c15856442df3500e9" exitCode=0 Oct 31 04:22:01.200808 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:01.200784 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv" event={"ID":"ffd1c004-3338-4cf6-b6f7-afebf7d01574","Type":"ContainerDied","Data":"dece7da60db378cea7ec0d7fef6b187d2624546e21795d2c15856442df3500e9"} Oct 31 04:22:01.201438 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:01.201410 5019 scope.go:122] "RemoveContainer" containerID="dece7da60db378cea7ec0d7fef6b187d2624546e21795d2c15856442df3500e9" Oct 31 04:22:01.203465 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:01.203429 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:22:01.203546 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:01.203525 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:22:01.203594 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:01.203577 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:22:01.821344 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:01.821295 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7102-606b8f"] Oct 31 04:22:01.822480 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:01.822457 5019 cpu_manager.go:414] "containerMap: removing container" podUID="ee3bb425-3793-4d03-b0b9-d3a0025e3b90" containerName="revoke-keys" Oct 31 04:22:01.822480 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:01.822481 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee3bb425-3793-4d03-b0b9-d3a0025e3b90" containerName="revoke-keys" Oct 31 04:22:01.822637 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:01.822517 5019 cpu_manager.go:414] "containerMap: removing container" podUID="5a5bf674-97e5-4f45-be51-190d9fc24a71" containerName="revoke-keys" Oct 31 04:22:01.822637 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:01.822522 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a5bf674-97e5-4f45-be51-190d9fc24a71" containerName="revoke-keys" Oct 31 04:22:01.822637 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:01.822635 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee3bb425-3793-4d03-b0b9-d3a0025e3b90" containerName="revoke-keys" Oct 31 04:22:01.822739 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:01.822647 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a5bf674-97e5-4f45-be51-190d9fc24a71" containerName="revoke-keys" Oct 31 04:22:01.826781 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:01.826759 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7102-606b8f" Oct 31 04:22:01.838829 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:01.838800 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7102-606b8f"] Oct 31 04:22:01.847728 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:01.847696 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-prswc\"" Oct 31 04:22:02.015946 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:02.015892 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd999\" (UniqueName: \"kubernetes.io/projected/4c6c4031-4494-4e9f-8c0d-9ddedc77a5e5-kube-api-access-dd999\") pod \"test-curl-7102-606b8f\" (UID: \"4c6c4031-4494-4e9f-8c0d-9ddedc77a5e5\") " pod="opendatahub/test-curl-7102-606b8f" Oct 31 04:22:02.032943 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:02.032896 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea662224-dshjp"] Oct 31 04:22:02.041241 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:02.041196 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea662224-dshjp"] Oct 31 04:22:02.118120 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:02.118015 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd999\" (UniqueName: \"kubernetes.io/projected/4c6c4031-4494-4e9f-8c0d-9ddedc77a5e5-kube-api-access-dd999\") pod \"test-curl-7102-606b8f\" (UID: \"4c6c4031-4494-4e9f-8c0d-9ddedc77a5e5\") " pod="opendatahub/test-curl-7102-606b8f" Oct 31 04:22:02.141219 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:02.141186 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd999\" (UniqueName: \"kubernetes.io/projected/4c6c4031-4494-4e9f-8c0d-9ddedc77a5e5-kube-api-access-dd999\") pod \"test-curl-7102-606b8f\" (UID: \"4c6c4031-4494-4e9f-8c0d-9ddedc77a5e5\") " pod="opendatahub/test-curl-7102-606b8f" Oct 31 04:22:02.143119 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:02.143091 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7102-606b8f" Oct 31 04:22:02.226353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:02.224107 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv" event={"ID":"ffd1c004-3338-4cf6-b6f7-afebf7d01574","Type":"ContainerStarted","Data":"a172f2dc436d9dbaeb890dc0a1b84e153bde2a68507062cd17aac0926fe6a981"} Oct 31 04:22:02.512744 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:02.512572 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7102-606b8f"] Oct 31 04:22:02.521791 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:22:02.521565 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c6c4031_4494_4e9f_8c0d_9ddedc77a5e5.slice/crio-fbadbc92654239df244df1fddd57e373d8e41e09b1dd86156592712f3c86082e WatchSource:0}: Error finding container fbadbc92654239df244df1fddd57e373d8e41e09b1dd86156592712f3c86082e: Status 404 returned error can't find the container with id fbadbc92654239df244df1fddd57e373d8e41e09b1dd86156592712f3c86082e Oct 31 04:22:02.524594 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:02.524563 5019 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 04:22:02.901865 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:02.901756 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv" Oct 31 04:22:02.902402 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:02.902120 5019 patch_prober.go:29] interesting pod/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 31 04:22:02.902402 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:02.902190 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv" podUID="ffd1c004-3338-4cf6-b6f7-afebf7d01574" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 31 04:22:03.238098 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:03.238056 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7102-606b8f" event={"ID":"4c6c4031-4494-4e9f-8c0d-9ddedc77a5e5","Type":"ContainerStarted","Data":"fbadbc92654239df244df1fddd57e373d8e41e09b1dd86156592712f3c86082e"} Oct 31 04:22:03.415515 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:03.415476 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e646830-d7f6-4e36-845d-f910078d4864" path="/var/lib/kubelet/pods/1e646830-d7f6-4e36-845d-f910078d4864/volumes" Oct 31 04:22:03.902033 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:03.901978 5019 patch_prober.go:29] interesting pod/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 31 04:22:03.902622 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:03.902049 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv" podUID="ffd1c004-3338-4cf6-b6f7-afebf7d01574" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 31 04:22:04.901850 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:04.901796 5019 patch_prober.go:29] interesting pod/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 31 04:22:04.902044 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:04.901872 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv" podUID="ffd1c004-3338-4cf6-b6f7-afebf7d01574" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 31 04:22:05.911751 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:05.911704 5019 patch_prober.go:29] interesting pod/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 31 04:22:05.912346 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:05.911775 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv" podUID="ffd1c004-3338-4cf6-b6f7-afebf7d01574" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 31 04:22:06.316807 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:06.316762 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7102-606b8f" event={"ID":"4c6c4031-4494-4e9f-8c0d-9ddedc77a5e5","Type":"ContainerStarted","Data":"20e7d5b39503b887822978aa1c3b8fd1751fcafbd19ca549bba91d8dfbce34a5"} Oct 31 04:22:06.354763 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:06.354701 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7102-606b8f" podStartSLOduration=1.9655414009999999 podStartE2EDuration="5.354687992s" totalImagesPullingTime="3.389146591s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:22:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:22:06.354163087 +0000 UTC m=+1081.110501127" watchObservedRunningTime="2026-10-31 04:22:06.354687992 +0000 UTC m=+1081.111026033" Oct 31 04:22:06.902013 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:06.901973 5019 patch_prober.go:29] interesting pod/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 31 04:22:06.902251 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:06.902054 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv" podUID="ffd1c004-3338-4cf6-b6f7-afebf7d01574" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 31 04:22:07.313676 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:07.313624 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7102-606b8f"] Oct 31 04:22:07.320865 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:07.320630 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7102-606b8f"] Oct 31 04:22:07.908574 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:07.908536 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv" Oct 31 04:22:07.909161 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:07.909138 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv" Oct 31 04:22:07.910306 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:07.910277 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv" Oct 31 04:22:07.993180 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:07.993134 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-146320d5-gcpp9"] Oct 31 04:22:07.994039 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:07.994014 5019 cpu_manager.go:414] "containerMap: removing container" podUID="4c6c4031-4494-4e9f-8c0d-9ddedc77a5e5" containerName="test-curl-7102-606b8f" Oct 31 04:22:07.994039 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:07.994041 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c6c4031-4494-4e9f-8c0d-9ddedc77a5e5" containerName="test-curl-7102-606b8f" Oct 31 04:22:07.994306 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:07.994290 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c6c4031-4494-4e9f-8c0d-9ddedc77a5e5" containerName="test-curl-7102-606b8f" Oct 31 04:22:07.998557 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:07.998528 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-146320d5-gcpp9" Oct 31 04:22:08.020777 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:08.018588 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:22:08.020777 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:08.018693 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:22:08.020777 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:08.018802 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:22:08.020777 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:08.018867 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:22:08.027696 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:08.025656 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-146320d5-gcpp9"] Oct 31 04:22:08.084109 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:08.084061 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drrrj\" (UniqueName: \"kubernetes.io/projected/ae53393e-c7c4-46fd-a12d-c49e2124f068-kube-api-access-drrrj\") pod \"maas-api-revoke-keys-e2e-mt-146320d5-gcpp9\" (UID: \"ae53393e-c7c4-46fd-a12d-c49e2124f068\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-146320d5-gcpp9" Oct 31 04:22:08.084538 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:08.084510 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae53393e-c7c4-46fd-a12d-c49e2124f068-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-146320d5-gcpp9\" (UID: \"ae53393e-c7c4-46fd-a12d-c49e2124f068\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-146320d5-gcpp9" Oct 31 04:22:08.084659 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:08.084606 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae53393e-c7c4-46fd-a12d-c49e2124f068-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-146320d5-gcpp9\" (UID: \"ae53393e-c7c4-46fd-a12d-c49e2124f068\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-146320d5-gcpp9" Oct 31 04:22:08.189585 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:08.189479 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drrrj\" (UniqueName: \"kubernetes.io/projected/ae53393e-c7c4-46fd-a12d-c49e2124f068-kube-api-access-drrrj\") pod \"maas-api-revoke-keys-e2e-mt-146320d5-gcpp9\" (UID: \"ae53393e-c7c4-46fd-a12d-c49e2124f068\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-146320d5-gcpp9" Oct 31 04:22:08.194559 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:08.193720 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae53393e-c7c4-46fd-a12d-c49e2124f068-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-146320d5-gcpp9\" (UID: \"ae53393e-c7c4-46fd-a12d-c49e2124f068\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-146320d5-gcpp9" Oct 31 04:22:08.194972 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:08.194942 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae53393e-c7c4-46fd-a12d-c49e2124f068-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-146320d5-gcpp9\" (UID: \"ae53393e-c7c4-46fd-a12d-c49e2124f068\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-146320d5-gcpp9" Oct 31 04:22:08.197682 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:08.195442 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae53393e-c7c4-46fd-a12d-c49e2124f068-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-146320d5-gcpp9\" (UID: \"ae53393e-c7c4-46fd-a12d-c49e2124f068\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-146320d5-gcpp9" Oct 31 04:22:08.212419 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:08.205627 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae53393e-c7c4-46fd-a12d-c49e2124f068-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-146320d5-gcpp9\" (UID: \"ae53393e-c7c4-46fd-a12d-c49e2124f068\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-146320d5-gcpp9" Oct 31 04:22:08.224719 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:08.223529 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drrrj\" (UniqueName: \"kubernetes.io/projected/ae53393e-c7c4-46fd-a12d-c49e2124f068-kube-api-access-drrrj\") pod \"maas-api-revoke-keys-e2e-mt-146320d5-gcpp9\" (UID: \"ae53393e-c7c4-46fd-a12d-c49e2124f068\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-146320d5-gcpp9" Oct 31 04:22:08.335126 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:08.335084 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-146320d5-gcpp9" Oct 31 04:22:08.347394 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:08.347290 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7102-606b8f" podUID="4c6c4031-4494-4e9f-8c0d-9ddedc77a5e5" containerName="test-curl-7102-606b8f" containerID="cri-o://20e7d5b39503b887822978aa1c3b8fd1751fcafbd19ca549bba91d8dfbce34a5" gracePeriod=2 Oct 31 04:22:08.647667 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:22:08.647630 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae53393e_c7c4_46fd_a12d_c49e2124f068.slice/crio-7e212945a3e01bcb7303c635e2a86699f649039cc6452e5643adc0c7c446d0a1 WatchSource:0}: Error finding container 7e212945a3e01bcb7303c635e2a86699f649039cc6452e5643adc0c7c446d0a1: Status 404 returned error can't find the container with id 7e212945a3e01bcb7303c635e2a86699f649039cc6452e5643adc0c7c446d0a1 Oct 31 04:22:08.649122 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:08.649096 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-146320d5-gcpp9"] Oct 31 04:22:09.358828 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:09.358737 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-146320d5-gcpp9" event={"ID":"ae53393e-c7c4-46fd-a12d-c49e2124f068","Type":"ContainerStarted","Data":"fe497322f646aa422a46e12333f90173e11208df8911866d50631f9169b9adaf"} Oct 31 04:22:09.358828 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:09.358796 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-146320d5-gcpp9" event={"ID":"ae53393e-c7c4-46fd-a12d-c49e2124f068","Type":"ContainerStarted","Data":"7e212945a3e01bcb7303c635e2a86699f649039cc6452e5643adc0c7c446d0a1"} Oct 31 04:22:09.401923 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:09.401854 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-146320d5-gcpp9" podStartSLOduration=2.40183315 podStartE2EDuration="2.40183315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:22:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:22:09.398425392 +0000 UTC m=+1084.154763433" watchObservedRunningTime="2026-10-31 04:22:09.40183315 +0000 UTC m=+1084.158171195" Oct 31 04:22:10.220709 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:10.220665 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8c0086-78b8c6c95-vwbxc"] Oct 31 04:22:10.374860 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:10.374824 5019 generic.go:356] "Generic (PLEG): container finished" podID="ae53393e-c7c4-46fd-a12d-c49e2124f068" containerID="fe497322f646aa422a46e12333f90173e11208df8911866d50631f9169b9adaf" exitCode=0 Oct 31 04:22:10.375261 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:10.374914 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-146320d5-gcpp9" event={"ID":"ae53393e-c7c4-46fd-a12d-c49e2124f068","Type":"ContainerDied","Data":"fe497322f646aa422a46e12333f90173e11208df8911866d50631f9169b9adaf"} Oct 31 04:22:10.693159 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:10.693132 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7102-606b8f" Oct 31 04:22:10.764531 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:10.764488 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c6c4031-4494-4e9f-8c0d-9ddedc77a5e5-kube-api-access-dd999\" (UniqueName: \"kubernetes.io/projected/4c6c4031-4494-4e9f-8c0d-9ddedc77a5e5-kube-api-access-dd999\") pod \"4c6c4031-4494-4e9f-8c0d-9ddedc77a5e5\" (UID: \"4c6c4031-4494-4e9f-8c0d-9ddedc77a5e5\") " Oct 31 04:22:10.767397 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:10.767317 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c6c4031-4494-4e9f-8c0d-9ddedc77a5e5-kube-api-access-dd999" pod "4c6c4031-4494-4e9f-8c0d-9ddedc77a5e5" (UID: "4c6c4031-4494-4e9f-8c0d-9ddedc77a5e5"). InnerVolumeSpecName "kube-api-access-dd999". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:22:10.878521 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:10.876222 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd999\" (UniqueName: \"kubernetes.io/projected/4c6c4031-4494-4e9f-8c0d-9ddedc77a5e5-kube-api-access-dd999\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:22:11.389429 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:11.389377 5019 generic.go:356] "Generic (PLEG): container finished" podID="4c6c4031-4494-4e9f-8c0d-9ddedc77a5e5" containerID="20e7d5b39503b887822978aa1c3b8fd1751fcafbd19ca549bba91d8dfbce34a5" exitCode=137 Oct 31 04:22:11.389960 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:11.389454 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7102-606b8f" Oct 31 04:22:11.389960 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:11.389498 5019 scope.go:122] "RemoveContainer" containerID="20e7d5b39503b887822978aa1c3b8fd1751fcafbd19ca549bba91d8dfbce34a5" Oct 31 04:22:11.404294 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:11.404267 5019 scope.go:122] "RemoveContainer" containerID="20e7d5b39503b887822978aa1c3b8fd1751fcafbd19ca549bba91d8dfbce34a5" Oct 31 04:22:11.404992 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:22:11.404953 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20e7d5b39503b887822978aa1c3b8fd1751fcafbd19ca549bba91d8dfbce34a5\": container with ID starting with 20e7d5b39503b887822978aa1c3b8fd1751fcafbd19ca549bba91d8dfbce34a5 not found: ID does not exist" containerID="20e7d5b39503b887822978aa1c3b8fd1751fcafbd19ca549bba91d8dfbce34a5" Oct 31 04:22:11.405105 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:11.405005 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20e7d5b39503b887822978aa1c3b8fd1751fcafbd19ca549bba91d8dfbce34a5"} err="failed to get container status \"20e7d5b39503b887822978aa1c3b8fd1751fcafbd19ca549bba91d8dfbce34a5\": rpc error: code = NotFound desc = could not find container \"20e7d5b39503b887822978aa1c3b8fd1751fcafbd19ca549bba91d8dfbce34a5\": container with ID starting with 20e7d5b39503b887822978aa1c3b8fd1751fcafbd19ca549bba91d8dfbce34a5 not found: ID does not exist" Oct 31 04:22:11.416734 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:11.416693 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c6c4031-4494-4e9f-8c0d-9ddedc77a5e5" path="/var/lib/kubelet/pods/4c6c4031-4494-4e9f-8c0d-9ddedc77a5e5/volumes" Oct 31 04:22:11.418177 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:22:11.418136 5019 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7102-606b8f\" is forbidden: User \"system:node:ip-10-0-130-57.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-57.ec2.internal' and this object" podUID="4c6c4031-4494-4e9f-8c0d-9ddedc77a5e5" pod="opendatahub/test-curl-7102-606b8f" Oct 31 04:22:11.599780 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:11.599749 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-146320d5-gcpp9" Oct 31 04:22:11.702407 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:11.702283 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae53393e-c7c4-46fd-a12d-c49e2124f068-kube-api-access-drrrj\" (UniqueName: \"kubernetes.io/projected/ae53393e-c7c4-46fd-a12d-c49e2124f068-kube-api-access-drrrj\") pod \"ae53393e-c7c4-46fd-a12d-c49e2124f068\" (UID: \"ae53393e-c7c4-46fd-a12d-c49e2124f068\") " Oct 31 04:22:11.702407 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:11.702391 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae53393e-c7c4-46fd-a12d-c49e2124f068-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae53393e-c7c4-46fd-a12d-c49e2124f068-maas-api-xks-ca\") pod \"ae53393e-c7c4-46fd-a12d-c49e2124f068\" (UID: \"ae53393e-c7c4-46fd-a12d-c49e2124f068\") " Oct 31 04:22:11.702880 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:11.702854 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae53393e-c7c4-46fd-a12d-c49e2124f068-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae53393e-c7c4-46fd-a12d-c49e2124f068-maas-api-openshift-ca\") pod \"ae53393e-c7c4-46fd-a12d-c49e2124f068\" (UID: \"ae53393e-c7c4-46fd-a12d-c49e2124f068\") " Oct 31 04:22:11.703444 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:11.703371 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae53393e-c7c4-46fd-a12d-c49e2124f068-maas-api-openshift-ca" pod "ae53393e-c7c4-46fd-a12d-c49e2124f068" (UID: "ae53393e-c7c4-46fd-a12d-c49e2124f068"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:22:11.703742 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:11.703720 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae53393e-c7c4-46fd-a12d-c49e2124f068-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:22:11.706250 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:11.706207 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae53393e-c7c4-46fd-a12d-c49e2124f068-maas-api-xks-ca" pod "ae53393e-c7c4-46fd-a12d-c49e2124f068" (UID: "ae53393e-c7c4-46fd-a12d-c49e2124f068"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:22:11.708242 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:11.708211 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae53393e-c7c4-46fd-a12d-c49e2124f068-kube-api-access-drrrj" pod "ae53393e-c7c4-46fd-a12d-c49e2124f068" (UID: "ae53393e-c7c4-46fd-a12d-c49e2124f068"). InnerVolumeSpecName "kube-api-access-drrrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:22:11.805555 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:11.805510 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drrrj\" (UniqueName: \"kubernetes.io/projected/ae53393e-c7c4-46fd-a12d-c49e2124f068-kube-api-access-drrrj\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:22:11.805555 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:11.805548 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae53393e-c7c4-46fd-a12d-c49e2124f068-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:22:12.032608 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:12.032566 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cefbe68-t88hv"] Oct 31 04:22:12.038440 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:12.038403 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cefbe68-t88hv"] Oct 31 04:22:12.404751 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:12.404660 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-146320d5-gcpp9" Oct 31 04:22:12.405201 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:12.404655 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-146320d5-gcpp9" event={"ID":"ae53393e-c7c4-46fd-a12d-c49e2124f068","Type":"ContainerDied","Data":"7e212945a3e01bcb7303c635e2a86699f649039cc6452e5643adc0c7c446d0a1"} Oct 31 04:22:12.405201 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:12.404795 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e212945a3e01bcb7303c635e2a86699f649039cc6452e5643adc0c7c446d0a1" Oct 31 04:22:13.413870 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:13.413826 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9608436-efc1-4cef-b9a0-55ba08255e6b" path="/var/lib/kubelet/pods/c9608436-efc1-4cef-b9a0-55ba08255e6b/volumes" Oct 31 04:22:14.250328 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:14.250280 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9883e4-6f4c589fd5-cmtkl"] Oct 31 04:22:14.285171 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:14.285123 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9883e4-6b849c5655-vlf6g"] Oct 31 04:22:15.221884 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:15.221832 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8c0086-78b8c6c95-vwbxc" podUID="43f3a427-7289-484b-bd5d-f304eaf5a247" containerName="payload-pre-processing" containerID="cri-o://9f8bfb886c6c4c292cefd0a63bff3f27eb9c22e0e55054a19949ae2fe0b91821" gracePeriod=25 Oct 31 04:22:15.450372 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:15.449081 5019 generic.go:356] "Generic (PLEG): container finished" podID="43f3a427-7289-484b-bd5d-f304eaf5a247" containerID="9f8bfb886c6c4c292cefd0a63bff3f27eb9c22e0e55054a19949ae2fe0b91821" exitCode=0 Oct 31 04:22:15.450372 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:15.449319 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8c0086-78b8c6c95-vwbxc" event={"ID":"43f3a427-7289-484b-bd5d-f304eaf5a247","Type":"ContainerDied","Data":"9f8bfb886c6c4c292cefd0a63bff3f27eb9c22e0e55054a19949ae2fe0b91821"} Oct 31 04:22:15.926210 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:15.924932 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8c0086-78b8c6c95-vwbxc" Oct 31 04:22:15.926629 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:15.926599 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43f3a427-7289-484b-bd5d-f304eaf5a247-kube-api-access-zzzv5\" (UniqueName: \"kubernetes.io/projected/43f3a427-7289-484b-bd5d-f304eaf5a247-kube-api-access-zzzv5\") pod \"43f3a427-7289-484b-bd5d-f304eaf5a247\" (UID: \"43f3a427-7289-484b-bd5d-f304eaf5a247\") " Oct 31 04:22:15.926970 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:15.926947 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43f3a427-7289-484b-bd5d-f304eaf5a247-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43f3a427-7289-484b-bd5d-f304eaf5a247-plugins-config-volume\") pod \"43f3a427-7289-484b-bd5d-f304eaf5a247\" (UID: \"43f3a427-7289-484b-bd5d-f304eaf5a247\") " Oct 31 04:22:15.941897 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:15.941861 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43f3a427-7289-484b-bd5d-f304eaf5a247-plugins-config-volume" pod "43f3a427-7289-484b-bd5d-f304eaf5a247" (UID: "43f3a427-7289-484b-bd5d-f304eaf5a247"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:22:15.995652 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:15.989892 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43f3a427-7289-484b-bd5d-f304eaf5a247-kube-api-access-zzzv5" pod "43f3a427-7289-484b-bd5d-f304eaf5a247" (UID: "43f3a427-7289-484b-bd5d-f304eaf5a247"). InnerVolumeSpecName "kube-api-access-zzzv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:22:16.079852 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:16.079749 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43f3a427-7289-484b-bd5d-f304eaf5a247-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:22:16.193167 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:16.193125 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzzv5\" (UniqueName: \"kubernetes.io/projected/43f3a427-7289-484b-bd5d-f304eaf5a247-kube-api-access-zzzv5\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:22:16.540096 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:16.540054 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8c0086-78b8c6c95-vwbxc" event={"ID":"43f3a427-7289-484b-bd5d-f304eaf5a247","Type":"ContainerDied","Data":"aaf88ef724eb34aaee8772071d3097941ce43ff4eea34cd430d3ec5be7046df1"} Oct 31 04:22:16.540812 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:16.540776 5019 scope.go:122] "RemoveContainer" containerID="9f8bfb886c6c4c292cefd0a63bff3f27eb9c22e0e55054a19949ae2fe0b91821" Oct 31 04:22:16.541083 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:16.541063 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8c0086-78b8c6c95-vwbxc" Oct 31 04:22:16.709479 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:16.709401 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8c0086-78b8c6c95-vwbxc"] Oct 31 04:22:16.797425 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:16.794856 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8c0086-78b8c6c95-vwbxc"] Oct 31 04:22:17.501357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:17.500379 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43f3a427-7289-484b-bd5d-f304eaf5a247" path="/var/lib/kubelet/pods/43f3a427-7289-484b-bd5d-f304eaf5a247/volumes" Oct 31 04:22:17.826427 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:17.823441 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:22:18.756358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:18.747894 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-146320d5-586748b797-4lhzm"] Oct 31 04:22:18.786083 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:18.786038 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-146320d5-65b76f4b67-fnz6w"] Oct 31 04:22:19.251321 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:19.251260 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-9883e4-6f4c589fd5-cmtkl" podUID="837a88ff-dd11-4ccc-85dc-f8a83c41687b" containerName="payload-processing" containerID="cri-o://8d02e5f1639fe301e34ec975aedf6072492b0d4ac6940b5832398c4d93856e60" gracePeriod=25 Oct 31 04:22:19.285924 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:19.285856 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9883e4-6b849c5655-vlf6g" podUID="9d45c27c-8c7d-4973-bc2f-46beb3b8fdb1" containerName="payload-pre-processing" containerID="cri-o://d42b537e9017f6a3ae63ed440020b716d68c3f27e152b6f9fac0fabd3bc28b7e" gracePeriod=25 Oct 31 04:22:19.769833 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:19.769736 5019 generic.go:356] "Generic (PLEG): container finished" podID="9d45c27c-8c7d-4973-bc2f-46beb3b8fdb1" containerID="d42b537e9017f6a3ae63ed440020b716d68c3f27e152b6f9fac0fabd3bc28b7e" exitCode=0 Oct 31 04:22:19.770194 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:19.770169 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9883e4-6b849c5655-vlf6g" event={"ID":"9d45c27c-8c7d-4973-bc2f-46beb3b8fdb1","Type":"ContainerDied","Data":"d42b537e9017f6a3ae63ed440020b716d68c3f27e152b6f9fac0fabd3bc28b7e"} Oct 31 04:22:19.772890 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:19.772858 5019 generic.go:356] "Generic (PLEG): container finished" podID="837a88ff-dd11-4ccc-85dc-f8a83c41687b" containerID="8d02e5f1639fe301e34ec975aedf6072492b0d4ac6940b5832398c4d93856e60" exitCode=0 Oct 31 04:22:19.773238 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:19.773216 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9883e4-6f4c589fd5-cmtkl" event={"ID":"837a88ff-dd11-4ccc-85dc-f8a83c41687b","Type":"ContainerDied","Data":"8d02e5f1639fe301e34ec975aedf6072492b0d4ac6940b5832398c4d93856e60"} Oct 31 04:22:19.925644 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:19.925585 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9883e4-6b849c5655-vlf6g" Oct 31 04:22:19.958158 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:19.958130 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9883e4-6f4c589fd5-cmtkl" Oct 31 04:22:20.016047 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:20.016001 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d45c27c-8c7d-4973-bc2f-46beb3b8fdb1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d45c27c-8c7d-4973-bc2f-46beb3b8fdb1-plugins-config-volume\") pod \"9d45c27c-8c7d-4973-bc2f-46beb3b8fdb1\" (UID: \"9d45c27c-8c7d-4973-bc2f-46beb3b8fdb1\") " Oct 31 04:22:20.016254 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:20.016086 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d45c27c-8c7d-4973-bc2f-46beb3b8fdb1-kube-api-access-x5crm\" (UniqueName: \"kubernetes.io/projected/9d45c27c-8c7d-4973-bc2f-46beb3b8fdb1-kube-api-access-x5crm\") pod \"9d45c27c-8c7d-4973-bc2f-46beb3b8fdb1\" (UID: \"9d45c27c-8c7d-4973-bc2f-46beb3b8fdb1\") " Oct 31 04:22:20.016327 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:20.016288 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/837a88ff-dd11-4ccc-85dc-f8a83c41687b-kube-api-access-9f4pb\" (UniqueName: \"kubernetes.io/projected/837a88ff-dd11-4ccc-85dc-f8a83c41687b-kube-api-access-9f4pb\") pod \"837a88ff-dd11-4ccc-85dc-f8a83c41687b\" (UID: \"837a88ff-dd11-4ccc-85dc-f8a83c41687b\") " Oct 31 04:22:20.016506 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:20.016479 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/837a88ff-dd11-4ccc-85dc-f8a83c41687b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/837a88ff-dd11-4ccc-85dc-f8a83c41687b-plugins-config-volume\") pod \"837a88ff-dd11-4ccc-85dc-f8a83c41687b\" (UID: \"837a88ff-dd11-4ccc-85dc-f8a83c41687b\") " Oct 31 04:22:20.017509 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:20.017479 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d45c27c-8c7d-4973-bc2f-46beb3b8fdb1-plugins-config-volume" pod "9d45c27c-8c7d-4973-bc2f-46beb3b8fdb1" (UID: "9d45c27c-8c7d-4973-bc2f-46beb3b8fdb1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:22:20.017985 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:20.017960 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/837a88ff-dd11-4ccc-85dc-f8a83c41687b-plugins-config-volume" pod "837a88ff-dd11-4ccc-85dc-f8a83c41687b" (UID: "837a88ff-dd11-4ccc-85dc-f8a83c41687b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:22:20.019397 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:20.018870 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d45c27c-8c7d-4973-bc2f-46beb3b8fdb1-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:22:20.019397 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:20.018913 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/837a88ff-dd11-4ccc-85dc-f8a83c41687b-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:22:20.019397 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:20.019229 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d45c27c-8c7d-4973-bc2f-46beb3b8fdb1-kube-api-access-x5crm" pod "9d45c27c-8c7d-4973-bc2f-46beb3b8fdb1" (UID: "9d45c27c-8c7d-4973-bc2f-46beb3b8fdb1"). InnerVolumeSpecName "kube-api-access-x5crm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:22:20.042738 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:20.041606 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/837a88ff-dd11-4ccc-85dc-f8a83c41687b-kube-api-access-9f4pb" pod "837a88ff-dd11-4ccc-85dc-f8a83c41687b" (UID: "837a88ff-dd11-4ccc-85dc-f8a83c41687b"). InnerVolumeSpecName "kube-api-access-9f4pb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:22:20.121097 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:20.121056 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5crm\" (UniqueName: \"kubernetes.io/projected/9d45c27c-8c7d-4973-bc2f-46beb3b8fdb1-kube-api-access-x5crm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:22:20.121097 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:20.121098 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f4pb\" (UniqueName: \"kubernetes.io/projected/837a88ff-dd11-4ccc-85dc-f8a83c41687b-kube-api-access-9f4pb\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:22:20.788174 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:20.788141 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9883e4-6b849c5655-vlf6g" Oct 31 04:22:20.788674 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:20.788162 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9883e4-6b849c5655-vlf6g" event={"ID":"9d45c27c-8c7d-4973-bc2f-46beb3b8fdb1","Type":"ContainerDied","Data":"55d4f114e9be5066b37fcc83617cb5804b2d887eb1b51c8457cfe2a32f94c4e4"} Oct 31 04:22:20.788674 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:20.788236 5019 scope.go:122] "RemoveContainer" containerID="d42b537e9017f6a3ae63ed440020b716d68c3f27e152b6f9fac0fabd3bc28b7e" Oct 31 04:22:20.791144 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:20.791110 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9883e4-6f4c589fd5-cmtkl" event={"ID":"837a88ff-dd11-4ccc-85dc-f8a83c41687b","Type":"ContainerDied","Data":"bbd8d56a6a1ebea792cc7bf1a3c91a4af6c9f8e5afd2ba815b3ddd0e502cb560"} Oct 31 04:22:20.791587 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:20.791543 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9883e4-6f4c589fd5-cmtkl" Oct 31 04:22:20.807621 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:20.807457 5019 scope.go:122] "RemoveContainer" containerID="8d02e5f1639fe301e34ec975aedf6072492b0d4ac6940b5832398c4d93856e60" Oct 31 04:22:20.850144 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:20.850049 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9883e4-6b849c5655-vlf6g"] Oct 31 04:22:20.862716 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:20.862675 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9883e4-6b849c5655-vlf6g"] Oct 31 04:22:20.917450 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:20.917410 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9883e4-6f4c589fd5-cmtkl"] Oct 31 04:22:20.923383 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:20.923322 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9883e4-6f4c589fd5-cmtkl"] Oct 31 04:22:21.453518 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:21.453468 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="837a88ff-dd11-4ccc-85dc-f8a83c41687b" path="/var/lib/kubelet/pods/837a88ff-dd11-4ccc-85dc-f8a83c41687b/volumes" Oct 31 04:22:21.454413 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:21.454386 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d45c27c-8c7d-4973-bc2f-46beb3b8fdb1" path="/var/lib/kubelet/pods/9d45c27c-8c7d-4973-bc2f-46beb3b8fdb1/volumes" Oct 31 04:22:22.408555 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:22:22.408498 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:22:23.749205 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:23.749147 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-146320d5-586748b797-4lhzm" podUID="d8a637de-5932-47ad-b5e7-8d32ec65957f" containerName="payload-processing" containerID="cri-o://14fa50afb7e9d61a9ac9990ae030ca72d9e7dbe698f4564630d64f1e392311f5" gracePeriod=25 Oct 31 04:22:23.787497 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:23.787449 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-146320d5-65b76f4b67-fnz6w" podUID="5cc83f70-27bb-42ee-9f5b-4d12dcb55df8" containerName="payload-pre-processing" containerID="cri-o://e652de40d2105aa57f76d99d8e338c91c518b69d0cb9da7e9a82dbec1bc90228" gracePeriod=25 Oct 31 04:22:27.848511 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:27.848467 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-146320d5-586748b797-4lhzm" podUID="d8a637de-5932-47ad-b5e7-8d32ec65957f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 04:22:28.860350 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:28.860299 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-146320d5-65b76f4b67-fnz6w" podUID="5cc83f70-27bb-42ee-9f5b-4d12dcb55df8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 04:22:35.976623 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:35.976582 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-093363ca-hl9xf"] Oct 31 04:22:35.977381 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:35.977364 5019 cpu_manager.go:414] "containerMap: removing container" podUID="43f3a427-7289-484b-bd5d-f304eaf5a247" containerName="payload-pre-processing" Oct 31 04:22:35.977381 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:35.977381 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43f3a427-7289-484b-bd5d-f304eaf5a247" containerName="payload-pre-processing" Oct 31 04:22:35.977494 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:35.977402 5019 cpu_manager.go:414] "containerMap: removing container" podUID="9d45c27c-8c7d-4973-bc2f-46beb3b8fdb1" containerName="payload-pre-processing" Oct 31 04:22:35.977494 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:35.977407 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d45c27c-8c7d-4973-bc2f-46beb3b8fdb1" containerName="payload-pre-processing" Oct 31 04:22:35.977494 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:35.977430 5019 cpu_manager.go:414] "containerMap: removing container" podUID="837a88ff-dd11-4ccc-85dc-f8a83c41687b" containerName="payload-processing" Oct 31 04:22:35.977494 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:35.977435 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="837a88ff-dd11-4ccc-85dc-f8a83c41687b" containerName="payload-processing" Oct 31 04:22:35.977494 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:35.977446 5019 cpu_manager.go:414] "containerMap: removing container" podUID="ae53393e-c7c4-46fd-a12d-c49e2124f068" containerName="revoke-keys" Oct 31 04:22:35.977494 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:35.977450 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae53393e-c7c4-46fd-a12d-c49e2124f068" containerName="revoke-keys" Oct 31 04:22:35.977675 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:35.977558 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="837a88ff-dd11-4ccc-85dc-f8a83c41687b" containerName="payload-processing" Oct 31 04:22:35.977675 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:35.977565 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae53393e-c7c4-46fd-a12d-c49e2124f068" containerName="revoke-keys" Oct 31 04:22:35.977675 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:35.977576 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="43f3a427-7289-484b-bd5d-f304eaf5a247" containerName="payload-pre-processing" Oct 31 04:22:35.977675 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:35.977584 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d45c27c-8c7d-4973-bc2f-46beb3b8fdb1" containerName="payload-pre-processing" Oct 31 04:22:35.986819 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:35.986791 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-093363ca-hl9xf" Oct 31 04:22:36.005131 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:36.005096 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-093363ca-hl9xf"] Oct 31 04:22:36.006014 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:36.005980 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:22:36.006014 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:36.005991 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:22:36.014553 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:36.014518 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:22:36.014786 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:36.014646 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:22:36.083246 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:36.083203 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2zx6\" (UniqueName: \"kubernetes.io/projected/679c7d7b-2edf-4359-be21-d5b5010459cb-kube-api-access-n2zx6\") pod \"maas-api-revoke-sub-093363ca-hl9xf\" (UID: \"679c7d7b-2edf-4359-be21-d5b5010459cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-093363ca-hl9xf" Oct 31 04:22:36.083502 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:36.083462 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/679c7d7b-2edf-4359-be21-d5b5010459cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-093363ca-hl9xf\" (UID: \"679c7d7b-2edf-4359-be21-d5b5010459cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-093363ca-hl9xf" Oct 31 04:22:36.083682 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:36.083570 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/679c7d7b-2edf-4359-be21-d5b5010459cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-093363ca-hl9xf\" (UID: \"679c7d7b-2edf-4359-be21-d5b5010459cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-093363ca-hl9xf" Oct 31 04:22:36.185289 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:36.185237 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/679c7d7b-2edf-4359-be21-d5b5010459cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-093363ca-hl9xf\" (UID: \"679c7d7b-2edf-4359-be21-d5b5010459cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-093363ca-hl9xf" Oct 31 04:22:36.185515 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:36.185317 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/679c7d7b-2edf-4359-be21-d5b5010459cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-093363ca-hl9xf\" (UID: \"679c7d7b-2edf-4359-be21-d5b5010459cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-093363ca-hl9xf" Oct 31 04:22:36.185593 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:36.185573 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2zx6\" (UniqueName: \"kubernetes.io/projected/679c7d7b-2edf-4359-be21-d5b5010459cb-kube-api-access-n2zx6\") pod \"maas-api-revoke-sub-093363ca-hl9xf\" (UID: \"679c7d7b-2edf-4359-be21-d5b5010459cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-093363ca-hl9xf" Oct 31 04:22:36.186189 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:36.186156 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/679c7d7b-2edf-4359-be21-d5b5010459cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-093363ca-hl9xf\" (UID: \"679c7d7b-2edf-4359-be21-d5b5010459cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-093363ca-hl9xf" Oct 31 04:22:36.188068 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:36.188038 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/679c7d7b-2edf-4359-be21-d5b5010459cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-093363ca-hl9xf\" (UID: \"679c7d7b-2edf-4359-be21-d5b5010459cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-093363ca-hl9xf" Oct 31 04:22:36.209295 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:36.209258 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2zx6\" (UniqueName: \"kubernetes.io/projected/679c7d7b-2edf-4359-be21-d5b5010459cb-kube-api-access-n2zx6\") pod \"maas-api-revoke-sub-093363ca-hl9xf\" (UID: \"679c7d7b-2edf-4359-be21-d5b5010459cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-093363ca-hl9xf" Oct 31 04:22:36.309067 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:36.308966 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-093363ca-hl9xf" Oct 31 04:22:36.617117 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:22:36.617072 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod679c7d7b_2edf_4359_be21_d5b5010459cb.slice/crio-3e137b48f192595de9fab11df22978408c48321344460d82c682e53169eeb36a WatchSource:0}: Error finding container 3e137b48f192595de9fab11df22978408c48321344460d82c682e53169eeb36a: Status 404 returned error can't find the container with id 3e137b48f192595de9fab11df22978408c48321344460d82c682e53169eeb36a Oct 31 04:22:36.618440 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:36.618413 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-093363ca-hl9xf"] Oct 31 04:22:37.009245 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:37.009201 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-093363ca-hl9xf" event={"ID":"679c7d7b-2edf-4359-be21-d5b5010459cb","Type":"ContainerStarted","Data":"8a69b339b42f22e7c3da86e0aabb4ee1bd92a777fe72201f5208c3b4eb31cab8"} Oct 31 04:22:37.009666 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:37.009268 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-093363ca-hl9xf" event={"ID":"679c7d7b-2edf-4359-be21-d5b5010459cb","Type":"ContainerStarted","Data":"3e137b48f192595de9fab11df22978408c48321344460d82c682e53169eeb36a"} Oct 31 04:22:37.054045 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:37.053986 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-093363ca-hl9xf" podStartSLOduration=2.053971461 podStartE2EDuration="2.053971461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:22:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:22:37.048636412 +0000 UTC m=+1111.804974454" watchObservedRunningTime="2026-10-31 04:22:37.053971461 +0000 UTC m=+1111.810309500" Oct 31 04:22:37.848505 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:37.848387 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-146320d5-586748b797-4lhzm" podUID="d8a637de-5932-47ad-b5e7-8d32ec65957f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 04:22:38.020939 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:38.020903 5019 generic.go:356] "Generic (PLEG): container finished" podID="679c7d7b-2edf-4359-be21-d5b5010459cb" containerID="8a69b339b42f22e7c3da86e0aabb4ee1bd92a777fe72201f5208c3b4eb31cab8" exitCode=0 Oct 31 04:22:38.021431 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:38.020989 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-093363ca-hl9xf" event={"ID":"679c7d7b-2edf-4359-be21-d5b5010459cb","Type":"ContainerDied","Data":"8a69b339b42f22e7c3da86e0aabb4ee1bd92a777fe72201f5208c3b4eb31cab8"} Oct 31 04:22:38.859640 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:38.859578 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-146320d5-65b76f4b67-fnz6w" podUID="5cc83f70-27bb-42ee-9f5b-4d12dcb55df8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 04:22:39.220055 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:39.220019 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-093363ca-hl9xf" Oct 31 04:22:39.237123 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:39.237089 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/679c7d7b-2edf-4359-be21-d5b5010459cb-kube-api-access-n2zx6\" (UniqueName: \"kubernetes.io/projected/679c7d7b-2edf-4359-be21-d5b5010459cb-kube-api-access-n2zx6\") pod \"679c7d7b-2edf-4359-be21-d5b5010459cb\" (UID: \"679c7d7b-2edf-4359-be21-d5b5010459cb\") " Oct 31 04:22:39.237304 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:39.237151 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/679c7d7b-2edf-4359-be21-d5b5010459cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/679c7d7b-2edf-4359-be21-d5b5010459cb-maas-api-xks-ca\") pod \"679c7d7b-2edf-4359-be21-d5b5010459cb\" (UID: \"679c7d7b-2edf-4359-be21-d5b5010459cb\") " Oct 31 04:22:39.237377 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:39.237298 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/679c7d7b-2edf-4359-be21-d5b5010459cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/679c7d7b-2edf-4359-be21-d5b5010459cb-maas-api-openshift-ca\") pod \"679c7d7b-2edf-4359-be21-d5b5010459cb\" (UID: \"679c7d7b-2edf-4359-be21-d5b5010459cb\") " Oct 31 04:22:39.237670 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:39.237643 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/679c7d7b-2edf-4359-be21-d5b5010459cb-maas-api-openshift-ca" pod "679c7d7b-2edf-4359-be21-d5b5010459cb" (UID: "679c7d7b-2edf-4359-be21-d5b5010459cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:22:39.238216 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:39.238186 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/679c7d7b-2edf-4359-be21-d5b5010459cb-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:22:39.239487 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:39.239465 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/679c7d7b-2edf-4359-be21-d5b5010459cb-maas-api-xks-ca" pod "679c7d7b-2edf-4359-be21-d5b5010459cb" (UID: "679c7d7b-2edf-4359-be21-d5b5010459cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:22:39.239684 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:39.239571 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/679c7d7b-2edf-4359-be21-d5b5010459cb-kube-api-access-n2zx6" pod "679c7d7b-2edf-4359-be21-d5b5010459cb" (UID: "679c7d7b-2edf-4359-be21-d5b5010459cb"). InnerVolumeSpecName "kube-api-access-n2zx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:22:39.339386 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:39.339308 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2zx6\" (UniqueName: \"kubernetes.io/projected/679c7d7b-2edf-4359-be21-d5b5010459cb-kube-api-access-n2zx6\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:22:39.339386 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:39.339380 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/679c7d7b-2edf-4359-be21-d5b5010459cb-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:22:40.052560 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:40.052529 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-093363ca-hl9xf" Oct 31 04:22:40.052749 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:40.052552 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-093363ca-hl9xf" event={"ID":"679c7d7b-2edf-4359-be21-d5b5010459cb","Type":"ContainerDied","Data":"3e137b48f192595de9fab11df22978408c48321344460d82c682e53169eeb36a"} Oct 31 04:22:40.052749 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:40.052620 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e137b48f192595de9fab11df22978408c48321344460d82c682e53169eeb36a" Oct 31 04:22:40.302757 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:40.302654 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0fc77f-6d886fb5cf-tht76"] Oct 31 04:22:40.362706 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:40.362666 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0fc77f-774488f5b7-jpmqw"] Oct 31 04:22:44.089351 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:44.089312 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-146320d5-65b76f4b67-fnz6w" Oct 31 04:22:44.120409 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:44.120309 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-146320d5-586748b797-4lhzm" Oct 31 04:22:44.132961 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:44.131689 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cc83f70-27bb-42ee-9f5b-4d12dcb55df8-kube-api-access-h67k5\" (UniqueName: \"kubernetes.io/projected/5cc83f70-27bb-42ee-9f5b-4d12dcb55df8-kube-api-access-h67k5\") pod \"5cc83f70-27bb-42ee-9f5b-4d12dcb55df8\" (UID: \"5cc83f70-27bb-42ee-9f5b-4d12dcb55df8\") " Oct 31 04:22:44.132961 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:44.131971 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cc83f70-27bb-42ee-9f5b-4d12dcb55df8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cc83f70-27bb-42ee-9f5b-4d12dcb55df8-plugins-config-volume\") pod \"5cc83f70-27bb-42ee-9f5b-4d12dcb55df8\" (UID: \"5cc83f70-27bb-42ee-9f5b-4d12dcb55df8\") " Oct 31 04:22:44.132961 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:44.132741 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cc83f70-27bb-42ee-9f5b-4d12dcb55df8-plugins-config-volume" pod "5cc83f70-27bb-42ee-9f5b-4d12dcb55df8" (UID: "5cc83f70-27bb-42ee-9f5b-4d12dcb55df8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:22:44.135709 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:44.135682 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cc83f70-27bb-42ee-9f5b-4d12dcb55df8-kube-api-access-h67k5" pod "5cc83f70-27bb-42ee-9f5b-4d12dcb55df8" (UID: "5cc83f70-27bb-42ee-9f5b-4d12dcb55df8"). InnerVolumeSpecName "kube-api-access-h67k5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:22:44.190162 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:44.190123 5019 generic.go:356] "Generic (PLEG): container finished" podID="d8a637de-5932-47ad-b5e7-8d32ec65957f" containerID="14fa50afb7e9d61a9ac9990ae030ca72d9e7dbe698f4564630d64f1e392311f5" exitCode=0 Oct 31 04:22:44.190427 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:44.190158 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-146320d5-586748b797-4lhzm" event={"ID":"d8a637de-5932-47ad-b5e7-8d32ec65957f","Type":"ContainerDied","Data":"14fa50afb7e9d61a9ac9990ae030ca72d9e7dbe698f4564630d64f1e392311f5"} Oct 31 04:22:44.190547 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:44.190453 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-146320d5-586748b797-4lhzm" event={"ID":"d8a637de-5932-47ad-b5e7-8d32ec65957f","Type":"ContainerDied","Data":"c1a9674759d635243a755d4b4219e886b160a410e3ac9dd34d8db5a4ca83459d"} Oct 31 04:22:44.190547 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:44.190178 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-146320d5-586748b797-4lhzm" Oct 31 04:22:44.190652 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:44.190477 5019 scope.go:122] "RemoveContainer" containerID="14fa50afb7e9d61a9ac9990ae030ca72d9e7dbe698f4564630d64f1e392311f5" Oct 31 04:22:44.192314 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:44.192286 5019 generic.go:356] "Generic (PLEG): container finished" podID="5cc83f70-27bb-42ee-9f5b-4d12dcb55df8" containerID="e652de40d2105aa57f76d99d8e338c91c518b69d0cb9da7e9a82dbec1bc90228" exitCode=0 Oct 31 04:22:44.192489 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:44.192422 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-146320d5-65b76f4b67-fnz6w" Oct 31 04:22:44.192489 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:44.192468 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-146320d5-65b76f4b67-fnz6w" event={"ID":"5cc83f70-27bb-42ee-9f5b-4d12dcb55df8","Type":"ContainerDied","Data":"e652de40d2105aa57f76d99d8e338c91c518b69d0cb9da7e9a82dbec1bc90228"} Oct 31 04:22:44.192572 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:44.192509 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-146320d5-65b76f4b67-fnz6w" event={"ID":"5cc83f70-27bb-42ee-9f5b-4d12dcb55df8","Type":"ContainerDied","Data":"0f947275ed3ed33e4b1e53b10a6ff7bce4dd221c575d4938a402f9d3491d083e"} Oct 31 04:22:44.205958 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:44.205933 5019 scope.go:122] "RemoveContainer" containerID="14fa50afb7e9d61a9ac9990ae030ca72d9e7dbe698f4564630d64f1e392311f5" Oct 31 04:22:44.206279 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:22:44.206259 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14fa50afb7e9d61a9ac9990ae030ca72d9e7dbe698f4564630d64f1e392311f5\": container with ID starting with 14fa50afb7e9d61a9ac9990ae030ca72d9e7dbe698f4564630d64f1e392311f5 not found: ID does not exist" containerID="14fa50afb7e9d61a9ac9990ae030ca72d9e7dbe698f4564630d64f1e392311f5" Oct 31 04:22:44.206382 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:44.206291 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14fa50afb7e9d61a9ac9990ae030ca72d9e7dbe698f4564630d64f1e392311f5"} err="failed to get container status \"14fa50afb7e9d61a9ac9990ae030ca72d9e7dbe698f4564630d64f1e392311f5\": rpc error: code = NotFound desc = could not find container \"14fa50afb7e9d61a9ac9990ae030ca72d9e7dbe698f4564630d64f1e392311f5\": container with ID starting with 14fa50afb7e9d61a9ac9990ae030ca72d9e7dbe698f4564630d64f1e392311f5 not found: ID does not exist" Oct 31 04:22:44.206382 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:44.206315 5019 scope.go:122] "RemoveContainer" containerID="e652de40d2105aa57f76d99d8e338c91c518b69d0cb9da7e9a82dbec1bc90228" Oct 31 04:22:44.218158 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:44.218062 5019 scope.go:122] "RemoveContainer" containerID="e652de40d2105aa57f76d99d8e338c91c518b69d0cb9da7e9a82dbec1bc90228" Oct 31 04:22:44.218420 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:22:44.218400 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e652de40d2105aa57f76d99d8e338c91c518b69d0cb9da7e9a82dbec1bc90228\": container with ID starting with e652de40d2105aa57f76d99d8e338c91c518b69d0cb9da7e9a82dbec1bc90228 not found: ID does not exist" containerID="e652de40d2105aa57f76d99d8e338c91c518b69d0cb9da7e9a82dbec1bc90228" Oct 31 04:22:44.218499 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:44.218431 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e652de40d2105aa57f76d99d8e338c91c518b69d0cb9da7e9a82dbec1bc90228"} err="failed to get container status \"e652de40d2105aa57f76d99d8e338c91c518b69d0cb9da7e9a82dbec1bc90228\": rpc error: code = NotFound desc = could not find container \"e652de40d2105aa57f76d99d8e338c91c518b69d0cb9da7e9a82dbec1bc90228\": container with ID starting with e652de40d2105aa57f76d99d8e338c91c518b69d0cb9da7e9a82dbec1bc90228 not found: ID does not exist" Oct 31 04:22:44.233316 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:44.233288 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8a637de-5932-47ad-b5e7-8d32ec65957f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8a637de-5932-47ad-b5e7-8d32ec65957f-plugins-config-volume\") pod \"d8a637de-5932-47ad-b5e7-8d32ec65957f\" (UID: \"d8a637de-5932-47ad-b5e7-8d32ec65957f\") " Oct 31 04:22:44.233466 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:44.233349 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8a637de-5932-47ad-b5e7-8d32ec65957f-kube-api-access-jclx2\" (UniqueName: \"kubernetes.io/projected/d8a637de-5932-47ad-b5e7-8d32ec65957f-kube-api-access-jclx2\") pod \"d8a637de-5932-47ad-b5e7-8d32ec65957f\" (UID: \"d8a637de-5932-47ad-b5e7-8d32ec65957f\") " Oct 31 04:22:44.233705 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:44.233679 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8a637de-5932-47ad-b5e7-8d32ec65957f-plugins-config-volume" pod "d8a637de-5932-47ad-b5e7-8d32ec65957f" (UID: "d8a637de-5932-47ad-b5e7-8d32ec65957f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:22:44.234318 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:44.234296 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cc83f70-27bb-42ee-9f5b-4d12dcb55df8-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:22:44.234431 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:44.234322 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h67k5\" (UniqueName: \"kubernetes.io/projected/5cc83f70-27bb-42ee-9f5b-4d12dcb55df8-kube-api-access-h67k5\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:22:44.234431 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:44.234349 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8a637de-5932-47ad-b5e7-8d32ec65957f-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:22:44.236614 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:44.236588 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8a637de-5932-47ad-b5e7-8d32ec65957f-kube-api-access-jclx2" pod "d8a637de-5932-47ad-b5e7-8d32ec65957f" (UID: "d8a637de-5932-47ad-b5e7-8d32ec65957f"). InnerVolumeSpecName "kube-api-access-jclx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:22:44.245391 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:44.245322 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-146320d5-65b76f4b67-fnz6w"] Oct 31 04:22:44.250606 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:44.250583 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-146320d5-65b76f4b67-fnz6w"] Oct 31 04:22:44.336246 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:44.336147 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jclx2\" (UniqueName: \"kubernetes.io/projected/d8a637de-5932-47ad-b5e7-8d32ec65957f-kube-api-access-jclx2\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:22:44.539723 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:44.539682 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-146320d5-586748b797-4lhzm"] Oct 31 04:22:44.544991 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:44.544953 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-146320d5-586748b797-4lhzm"] Oct 31 04:22:45.303713 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:45.303653 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-0fc77f-6d886fb5cf-tht76" podUID="5053ea34-585c-4b1d-89d9-6eab5205471a" containerName="payload-processing" containerID="cri-o://091f198ea7a6d09aafc5f7c16954cc9bc58b79684321b3bcd17a8699a38d32bb" gracePeriod=25 Oct 31 04:22:45.363795 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:45.363694 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0fc77f-774488f5b7-jpmqw" podUID="a136b866-abc0-4c60-b977-ce6f05b0bdbf" containerName="payload-pre-processing" containerID="cri-o://4ca1ac6b9878f8a426eef15f82591b32c92c075db278cc84c49dc223f41ec0b0" gracePeriod=25 Oct 31 04:22:45.452114 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:45.452078 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cc83f70-27bb-42ee-9f5b-4d12dcb55df8" path="/var/lib/kubelet/pods/5cc83f70-27bb-42ee-9f5b-4d12dcb55df8/volumes" Oct 31 04:22:45.452621 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:45.452598 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8a637de-5932-47ad-b5e7-8d32ec65957f" path="/var/lib/kubelet/pods/d8a637de-5932-47ad-b5e7-8d32ec65957f/volumes" Oct 31 04:22:45.622163 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:45.622132 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0fc77f-6d886fb5cf-tht76" Oct 31 04:22:45.660439 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:45.660406 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5053ea34-585c-4b1d-89d9-6eab5205471a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5053ea34-585c-4b1d-89d9-6eab5205471a-plugins-config-volume\") pod \"5053ea34-585c-4b1d-89d9-6eab5205471a\" (UID: \"5053ea34-585c-4b1d-89d9-6eab5205471a\") " Oct 31 04:22:45.660702 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:45.660678 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5053ea34-585c-4b1d-89d9-6eab5205471a-kube-api-access-zjvjg\" (UniqueName: \"kubernetes.io/projected/5053ea34-585c-4b1d-89d9-6eab5205471a-kube-api-access-zjvjg\") pod \"5053ea34-585c-4b1d-89d9-6eab5205471a\" (UID: \"5053ea34-585c-4b1d-89d9-6eab5205471a\") " Oct 31 04:22:45.661744 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:45.661718 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5053ea34-585c-4b1d-89d9-6eab5205471a-plugins-config-volume" pod "5053ea34-585c-4b1d-89d9-6eab5205471a" (UID: "5053ea34-585c-4b1d-89d9-6eab5205471a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:22:45.671110 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:45.670254 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5053ea34-585c-4b1d-89d9-6eab5205471a-kube-api-access-zjvjg" pod "5053ea34-585c-4b1d-89d9-6eab5205471a" (UID: "5053ea34-585c-4b1d-89d9-6eab5205471a"). InnerVolumeSpecName "kube-api-access-zjvjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:22:45.708227 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:45.708190 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0fc77f-774488f5b7-jpmqw" Oct 31 04:22:45.762354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:45.761985 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a136b866-abc0-4c60-b977-ce6f05b0bdbf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a136b866-abc0-4c60-b977-ce6f05b0bdbf-plugins-config-volume\") pod \"a136b866-abc0-4c60-b977-ce6f05b0bdbf\" (UID: \"a136b866-abc0-4c60-b977-ce6f05b0bdbf\") " Oct 31 04:22:45.762354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:45.762204 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a136b866-abc0-4c60-b977-ce6f05b0bdbf-kube-api-access-gwhqx\" (UniqueName: \"kubernetes.io/projected/a136b866-abc0-4c60-b977-ce6f05b0bdbf-kube-api-access-gwhqx\") pod \"a136b866-abc0-4c60-b977-ce6f05b0bdbf\" (UID: \"a136b866-abc0-4c60-b977-ce6f05b0bdbf\") " Oct 31 04:22:45.762608 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:45.762474 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a136b866-abc0-4c60-b977-ce6f05b0bdbf-plugins-config-volume" pod "a136b866-abc0-4c60-b977-ce6f05b0bdbf" (UID: "a136b866-abc0-4c60-b977-ce6f05b0bdbf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:22:45.767310 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:45.763377 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5053ea34-585c-4b1d-89d9-6eab5205471a-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:22:45.767310 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:45.763410 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a136b866-abc0-4c60-b977-ce6f05b0bdbf-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:22:45.767310 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:45.763422 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjvjg\" (UniqueName: \"kubernetes.io/projected/5053ea34-585c-4b1d-89d9-6eab5205471a-kube-api-access-zjvjg\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:22:45.767310 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:45.766145 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a136b866-abc0-4c60-b977-ce6f05b0bdbf-kube-api-access-gwhqx" pod "a136b866-abc0-4c60-b977-ce6f05b0bdbf" (UID: "a136b866-abc0-4c60-b977-ce6f05b0bdbf"). InnerVolumeSpecName "kube-api-access-gwhqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:22:45.864694 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:45.864597 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwhqx\" (UniqueName: \"kubernetes.io/projected/a136b866-abc0-4c60-b977-ce6f05b0bdbf-kube-api-access-gwhqx\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:22:46.222771 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:46.222732 5019 generic.go:356] "Generic (PLEG): container finished" podID="a136b866-abc0-4c60-b977-ce6f05b0bdbf" containerID="4ca1ac6b9878f8a426eef15f82591b32c92c075db278cc84c49dc223f41ec0b0" exitCode=0 Oct 31 04:22:46.223067 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:46.223044 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0fc77f-774488f5b7-jpmqw" event={"ID":"a136b866-abc0-4c60-b977-ce6f05b0bdbf","Type":"ContainerDied","Data":"4ca1ac6b9878f8a426eef15f82591b32c92c075db278cc84c49dc223f41ec0b0"} Oct 31 04:22:46.223141 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:46.223106 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0fc77f-774488f5b7-jpmqw" event={"ID":"a136b866-abc0-4c60-b977-ce6f05b0bdbf","Type":"ContainerDied","Data":"545e5b0f7c1ba53777ab51655dd188ce4a79fda81e09afd521dea503bb8cc7bb"} Oct 31 04:22:46.223201 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:46.223138 5019 scope.go:122] "RemoveContainer" containerID="4ca1ac6b9878f8a426eef15f82591b32c92c075db278cc84c49dc223f41ec0b0" Oct 31 04:22:46.223286 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:46.223274 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0fc77f-774488f5b7-jpmqw" Oct 31 04:22:46.233565 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:46.229713 5019 generic.go:356] "Generic (PLEG): container finished" podID="5053ea34-585c-4b1d-89d9-6eab5205471a" containerID="091f198ea7a6d09aafc5f7c16954cc9bc58b79684321b3bcd17a8699a38d32bb" exitCode=0 Oct 31 04:22:46.233565 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:46.229756 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0fc77f-6d886fb5cf-tht76" event={"ID":"5053ea34-585c-4b1d-89d9-6eab5205471a","Type":"ContainerDied","Data":"091f198ea7a6d09aafc5f7c16954cc9bc58b79684321b3bcd17a8699a38d32bb"} Oct 31 04:22:46.233565 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:46.229804 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0fc77f-6d886fb5cf-tht76" event={"ID":"5053ea34-585c-4b1d-89d9-6eab5205471a","Type":"ContainerDied","Data":"9e7eff6586dc24def34b93c53ff9e21c179ebd78cf51d94cdc563b190d5ab126"} Oct 31 04:22:46.233565 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:46.229881 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0fc77f-6d886fb5cf-tht76" Oct 31 04:22:46.246851 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:46.246817 5019 scope.go:122] "RemoveContainer" containerID="4ca1ac6b9878f8a426eef15f82591b32c92c075db278cc84c49dc223f41ec0b0" Oct 31 04:22:46.247224 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:22:46.247204 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ca1ac6b9878f8a426eef15f82591b32c92c075db278cc84c49dc223f41ec0b0\": container with ID starting with 4ca1ac6b9878f8a426eef15f82591b32c92c075db278cc84c49dc223f41ec0b0 not found: ID does not exist" containerID="4ca1ac6b9878f8a426eef15f82591b32c92c075db278cc84c49dc223f41ec0b0" Oct 31 04:22:46.247847 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:46.247233 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ca1ac6b9878f8a426eef15f82591b32c92c075db278cc84c49dc223f41ec0b0"} err="failed to get container status \"4ca1ac6b9878f8a426eef15f82591b32c92c075db278cc84c49dc223f41ec0b0\": rpc error: code = NotFound desc = could not find container \"4ca1ac6b9878f8a426eef15f82591b32c92c075db278cc84c49dc223f41ec0b0\": container with ID starting with 4ca1ac6b9878f8a426eef15f82591b32c92c075db278cc84c49dc223f41ec0b0 not found: ID does not exist" Oct 31 04:22:46.247847 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:46.247254 5019 scope.go:122] "RemoveContainer" containerID="091f198ea7a6d09aafc5f7c16954cc9bc58b79684321b3bcd17a8699a38d32bb" Oct 31 04:22:46.265404 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:46.264224 5019 scope.go:122] "RemoveContainer" containerID="091f198ea7a6d09aafc5f7c16954cc9bc58b79684321b3bcd17a8699a38d32bb" Oct 31 04:22:46.265404 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:22:46.264711 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"091f198ea7a6d09aafc5f7c16954cc9bc58b79684321b3bcd17a8699a38d32bb\": container with ID starting with 091f198ea7a6d09aafc5f7c16954cc9bc58b79684321b3bcd17a8699a38d32bb not found: ID does not exist" containerID="091f198ea7a6d09aafc5f7c16954cc9bc58b79684321b3bcd17a8699a38d32bb" Oct 31 04:22:46.265404 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:46.264748 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"091f198ea7a6d09aafc5f7c16954cc9bc58b79684321b3bcd17a8699a38d32bb"} err="failed to get container status \"091f198ea7a6d09aafc5f7c16954cc9bc58b79684321b3bcd17a8699a38d32bb\": rpc error: code = NotFound desc = could not find container \"091f198ea7a6d09aafc5f7c16954cc9bc58b79684321b3bcd17a8699a38d32bb\": container with ID starting with 091f198ea7a6d09aafc5f7c16954cc9bc58b79684321b3bcd17a8699a38d32bb not found: ID does not exist" Oct 31 04:22:46.294764 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:46.294715 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0fc77f-6d886fb5cf-tht76"] Oct 31 04:22:46.302586 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:46.302551 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0fc77f-6d886fb5cf-tht76"] Oct 31 04:22:46.334590 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:46.334557 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0fc77f-774488f5b7-jpmqw"] Oct 31 04:22:46.339505 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:46.339472 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0fc77f-774488f5b7-jpmqw"] Oct 31 04:22:47.410052 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:47.410016 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5053ea34-585c-4b1d-89d9-6eab5205471a" path="/var/lib/kubelet/pods/5053ea34-585c-4b1d-89d9-6eab5205471a/volumes" Oct 31 04:22:47.410685 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:47.410668 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a136b866-abc0-4c60-b977-ce6f05b0bdbf" path="/var/lib/kubelet/pods/a136b866-abc0-4c60-b977-ce6f05b0bdbf/volumes" Oct 31 04:22:52.328632 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:52.328588 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-146320d5-gcpp9"] Oct 31 04:22:52.340680 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:52.340629 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-146320d5-gcpp9"] Oct 31 04:22:52.392268 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:22:52.392222 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:22:53.415024 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:53.414991 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae53393e-c7c4-46fd-a12d-c49e2124f068" path="/var/lib/kubelet/pods/ae53393e-c7c4-46fd-a12d-c49e2124f068/volumes" Oct 31 04:22:53.913780 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:53.913743 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a43a70-579b95c6df-7m45l"] Oct 31 04:22:56.032866 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:56.032825 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv"] Oct 31 04:22:56.033365 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:56.033082 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv" podUID="ffd1c004-3338-4cf6-b6f7-afebf7d01574" containerName="istio-proxy" containerID="cri-o://a172f2dc436d9dbaeb890dc0a1b84e153bde2a68507062cd17aac0926fe6a981" gracePeriod=30 Oct 31 04:22:58.914903 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:58.914846 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-a43a70-579b95c6df-7m45l" podUID="a9a72222-d749-4677-9621-99f1306e4ffd" containerName="payload-processing" containerID="cri-o://5f10c724d1a5eb3d491667c020774062dacc30652fae3f63c62000df580151b6" gracePeriod=25 Oct 31 04:22:59.266995 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:59.266964 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a43a70-579b95c6df-7m45l" Oct 31 04:22:59.501468 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:59.490200 5019 generic.go:356] "Generic (PLEG): container finished" podID="a9a72222-d749-4677-9621-99f1306e4ffd" containerID="5f10c724d1a5eb3d491667c020774062dacc30652fae3f63c62000df580151b6" exitCode=0 Oct 31 04:22:59.506111 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:59.496724 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a43a70-579b95c6df-7m45l" Oct 31 04:22:59.629274 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:59.629107 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9a72222-d749-4677-9621-99f1306e4ffd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9a72222-d749-4677-9621-99f1306e4ffd-plugins-config-volume\") pod \"a9a72222-d749-4677-9621-99f1306e4ffd\" (UID: \"a9a72222-d749-4677-9621-99f1306e4ffd\") " Oct 31 04:22:59.629274 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:59.629203 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9a72222-d749-4677-9621-99f1306e4ffd-kube-api-access-lcsml\" (UniqueName: \"kubernetes.io/projected/a9a72222-d749-4677-9621-99f1306e4ffd-kube-api-access-lcsml\") pod \"a9a72222-d749-4677-9621-99f1306e4ffd\" (UID: \"a9a72222-d749-4677-9621-99f1306e4ffd\") " Oct 31 04:22:59.632250 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:59.632218 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9a72222-d749-4677-9621-99f1306e4ffd-plugins-config-volume" pod "a9a72222-d749-4677-9621-99f1306e4ffd" (UID: "a9a72222-d749-4677-9621-99f1306e4ffd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:22:59.637322 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:59.637272 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9a72222-d749-4677-9621-99f1306e4ffd-kube-api-access-lcsml" pod "a9a72222-d749-4677-9621-99f1306e4ffd" (UID: "a9a72222-d749-4677-9621-99f1306e4ffd"). InnerVolumeSpecName "kube-api-access-lcsml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:22:59.638180 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:59.638151 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a43a70-579b95c6df-7m45l" event={"ID":"a9a72222-d749-4677-9621-99f1306e4ffd","Type":"ContainerDied","Data":"5f10c724d1a5eb3d491667c020774062dacc30652fae3f63c62000df580151b6"} Oct 31 04:22:59.638287 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:59.638227 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a43a70-579b95c6df-7m45l" event={"ID":"a9a72222-d749-4677-9621-99f1306e4ffd","Type":"ContainerDied","Data":"8a5abe34a5fae78050c5950edd74a53b34638dc5e346f04816d11d122ed88524"} Oct 31 04:22:59.639448 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:59.638603 5019 scope.go:122] "RemoveContainer" containerID="5f10c724d1a5eb3d491667c020774062dacc30652fae3f63c62000df580151b6" Oct 31 04:22:59.674093 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:59.659904 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9a72222-d749-4677-9621-99f1306e4ffd-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:22:59.674093 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:59.665418 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcsml\" (UniqueName: \"kubernetes.io/projected/a9a72222-d749-4677-9621-99f1306e4ffd-kube-api-access-lcsml\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:22:59.702188 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:59.695124 5019 scope.go:122] "RemoveContainer" containerID="5f10c724d1a5eb3d491667c020774062dacc30652fae3f63c62000df580151b6" Oct 31 04:22:59.702188 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:22:59.695617 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f10c724d1a5eb3d491667c020774062dacc30652fae3f63c62000df580151b6\": container with ID starting with 5f10c724d1a5eb3d491667c020774062dacc30652fae3f63c62000df580151b6 not found: ID does not exist" containerID="5f10c724d1a5eb3d491667c020774062dacc30652fae3f63c62000df580151b6" Oct 31 04:22:59.702188 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:59.695656 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f10c724d1a5eb3d491667c020774062dacc30652fae3f63c62000df580151b6"} err="failed to get container status \"5f10c724d1a5eb3d491667c020774062dacc30652fae3f63c62000df580151b6\": rpc error: code = NotFound desc = could not find container \"5f10c724d1a5eb3d491667c020774062dacc30652fae3f63c62000df580151b6\": container with ID starting with 5f10c724d1a5eb3d491667c020774062dacc30652fae3f63c62000df580151b6 not found: ID does not exist" Oct 31 04:22:59.866342 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:59.866233 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a43a70-579b95c6df-7m45l"] Oct 31 04:22:59.881921 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:22:59.881886 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a43a70-579b95c6df-7m45l"] Oct 31 04:23:01.340774 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.340736 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv" Oct 31 04:23:01.410926 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.410889 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffd1c004-3338-4cf6-b6f7-afebf7d01574-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffd1c004-3338-4cf6-b6f7-afebf7d01574-istio-envoy\") pod \"ffd1c004-3338-4cf6-b6f7-afebf7d01574\" (UID: \"ffd1c004-3338-4cf6-b6f7-afebf7d01574\") " Oct 31 04:23:01.411118 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.410964 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffd1c004-3338-4cf6-b6f7-afebf7d01574-kube-api-access-mlfxn\" (UniqueName: \"kubernetes.io/projected/ffd1c004-3338-4cf6-b6f7-afebf7d01574-kube-api-access-mlfxn\") pod \"ffd1c004-3338-4cf6-b6f7-afebf7d01574\" (UID: \"ffd1c004-3338-4cf6-b6f7-afebf7d01574\") " Oct 31 04:23:01.411735 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.411215 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ffd1c004-3338-4cf6-b6f7-afebf7d01574-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffd1c004-3338-4cf6-b6f7-afebf7d01574-istio-podinfo\") pod \"ffd1c004-3338-4cf6-b6f7-afebf7d01574\" (UID: \"ffd1c004-3338-4cf6-b6f7-afebf7d01574\") " Oct 31 04:23:01.411735 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.411280 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffd1c004-3338-4cf6-b6f7-afebf7d01574-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd1c004-3338-4cf6-b6f7-afebf7d01574-credential-socket\") pod \"ffd1c004-3338-4cf6-b6f7-afebf7d01574\" (UID: \"ffd1c004-3338-4cf6-b6f7-afebf7d01574\") " Oct 31 04:23:01.411735 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.411388 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffd1c004-3338-4cf6-b6f7-afebf7d01574-istio-token\" (UniqueName: \"kubernetes.io/projected/ffd1c004-3338-4cf6-b6f7-afebf7d01574-istio-token\") pod \"ffd1c004-3338-4cf6-b6f7-afebf7d01574\" (UID: \"ffd1c004-3338-4cf6-b6f7-afebf7d01574\") " Oct 31 04:23:01.411735 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.411430 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffd1c004-3338-4cf6-b6f7-afebf7d01574-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd1c004-3338-4cf6-b6f7-afebf7d01574-workload-socket\") pod \"ffd1c004-3338-4cf6-b6f7-afebf7d01574\" (UID: \"ffd1c004-3338-4cf6-b6f7-afebf7d01574\") " Oct 31 04:23:01.411735 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.411495 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffd1c004-3338-4cf6-b6f7-afebf7d01574-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffd1c004-3338-4cf6-b6f7-afebf7d01574-istio-data\") pod \"ffd1c004-3338-4cf6-b6f7-afebf7d01574\" (UID: \"ffd1c004-3338-4cf6-b6f7-afebf7d01574\") " Oct 31 04:23:01.411735 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.411526 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffd1c004-3338-4cf6-b6f7-afebf7d01574-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffd1c004-3338-4cf6-b6f7-afebf7d01574-istiod-ca-cert\") pod \"ffd1c004-3338-4cf6-b6f7-afebf7d01574\" (UID: \"ffd1c004-3338-4cf6-b6f7-afebf7d01574\") " Oct 31 04:23:01.411735 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.411572 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffd1c004-3338-4cf6-b6f7-afebf7d01574-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffd1c004-3338-4cf6-b6f7-afebf7d01574-workload-certs\") pod \"ffd1c004-3338-4cf6-b6f7-afebf7d01574\" (UID: \"ffd1c004-3338-4cf6-b6f7-afebf7d01574\") " Oct 31 04:23:01.412642 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.412616 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffd1c004-3338-4cf6-b6f7-afebf7d01574-workload-certs" pod "ffd1c004-3338-4cf6-b6f7-afebf7d01574" (UID: "ffd1c004-3338-4cf6-b6f7-afebf7d01574"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:23:01.412798 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.412735 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffd1c004-3338-4cf6-b6f7-afebf7d01574-istio-data" pod "ffd1c004-3338-4cf6-b6f7-afebf7d01574" (UID: "ffd1c004-3338-4cf6-b6f7-afebf7d01574"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:23:01.412892 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.412802 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffd1c004-3338-4cf6-b6f7-afebf7d01574-istiod-ca-cert" pod "ffd1c004-3338-4cf6-b6f7-afebf7d01574" (UID: "ffd1c004-3338-4cf6-b6f7-afebf7d01574"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:23:01.413037 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.413011 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffd1c004-3338-4cf6-b6f7-afebf7d01574-credential-socket" pod "ffd1c004-3338-4cf6-b6f7-afebf7d01574" (UID: "ffd1c004-3338-4cf6-b6f7-afebf7d01574"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:23:01.413133 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.413113 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffd1c004-3338-4cf6-b6f7-afebf7d01574-workload-socket" pod "ffd1c004-3338-4cf6-b6f7-afebf7d01574" (UID: "ffd1c004-3338-4cf6-b6f7-afebf7d01574"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:23:01.415460 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.415415 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffd1c004-3338-4cf6-b6f7-afebf7d01574-istio-envoy" pod "ffd1c004-3338-4cf6-b6f7-afebf7d01574" (UID: "ffd1c004-3338-4cf6-b6f7-afebf7d01574"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:23:01.415931 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.415899 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ffd1c004-3338-4cf6-b6f7-afebf7d01574-istio-podinfo" pod "ffd1c004-3338-4cf6-b6f7-afebf7d01574" (UID: "ffd1c004-3338-4cf6-b6f7-afebf7d01574"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:23:01.419382 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.419293 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffd1c004-3338-4cf6-b6f7-afebf7d01574-istio-token" pod "ffd1c004-3338-4cf6-b6f7-afebf7d01574" (UID: "ffd1c004-3338-4cf6-b6f7-afebf7d01574"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:23:01.419976 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.419949 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffd1c004-3338-4cf6-b6f7-afebf7d01574-kube-api-access-mlfxn" pod "ffd1c004-3338-4cf6-b6f7-afebf7d01574" (UID: "ffd1c004-3338-4cf6-b6f7-afebf7d01574"). InnerVolumeSpecName "kube-api-access-mlfxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:23:01.465700 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.463803 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9a72222-d749-4677-9621-99f1306e4ffd" path="/var/lib/kubelet/pods/a9a72222-d749-4677-9621-99f1306e4ffd/volumes" Oct 31 04:23:01.514681 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.514647 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffd1c004-3338-4cf6-b6f7-afebf7d01574-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:23:01.514681 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.514680 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd1c004-3338-4cf6-b6f7-afebf7d01574-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:23:01.514891 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.514694 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffd1c004-3338-4cf6-b6f7-afebf7d01574-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:23:01.514891 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.514707 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd1c004-3338-4cf6-b6f7-afebf7d01574-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:23:01.514891 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.514718 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffd1c004-3338-4cf6-b6f7-afebf7d01574-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:23:01.514891 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.514732 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffd1c004-3338-4cf6-b6f7-afebf7d01574-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:23:01.514891 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.514744 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffd1c004-3338-4cf6-b6f7-afebf7d01574-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:23:01.514891 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.514756 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffd1c004-3338-4cf6-b6f7-afebf7d01574-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:23:01.514891 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.514769 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlfxn\" (UniqueName: \"kubernetes.io/projected/ffd1c004-3338-4cf6-b6f7-afebf7d01574-kube-api-access-mlfxn\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:23:01.535041 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.535003 5019 generic.go:356] "Generic (PLEG): container finished" podID="ffd1c004-3338-4cf6-b6f7-afebf7d01574" containerID="a172f2dc436d9dbaeb890dc0a1b84e153bde2a68507062cd17aac0926fe6a981" exitCode=0 Oct 31 04:23:01.535258 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.535061 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv" Oct 31 04:23:01.535391 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.535278 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv" event={"ID":"ffd1c004-3338-4cf6-b6f7-afebf7d01574","Type":"ContainerDied","Data":"a172f2dc436d9dbaeb890dc0a1b84e153bde2a68507062cd17aac0926fe6a981"} Oct 31 04:23:01.535391 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.535320 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv" event={"ID":"ffd1c004-3338-4cf6-b6f7-afebf7d01574","Type":"ContainerDied","Data":"78e12522820d66b1edd1baa27b98b00466f4cd92e3654ebbf8300dae88db419f"} Oct 31 04:23:01.535391 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.535389 5019 scope.go:122] "RemoveContainer" containerID="a172f2dc436d9dbaeb890dc0a1b84e153bde2a68507062cd17aac0926fe6a981" Oct 31 04:23:01.549968 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.549913 5019 scope.go:122] "RemoveContainer" containerID="dece7da60db378cea7ec0d7fef6b187d2624546e21795d2c15856442df3500e9" Oct 31 04:23:01.570787 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.570765 5019 scope.go:122] "RemoveContainer" containerID="a172f2dc436d9dbaeb890dc0a1b84e153bde2a68507062cd17aac0926fe6a981" Oct 31 04:23:01.571308 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:23:01.571273 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a172f2dc436d9dbaeb890dc0a1b84e153bde2a68507062cd17aac0926fe6a981\": container with ID starting with a172f2dc436d9dbaeb890dc0a1b84e153bde2a68507062cd17aac0926fe6a981 not found: ID does not exist" containerID="a172f2dc436d9dbaeb890dc0a1b84e153bde2a68507062cd17aac0926fe6a981" Oct 31 04:23:01.571961 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.571316 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a172f2dc436d9dbaeb890dc0a1b84e153bde2a68507062cd17aac0926fe6a981"} err="failed to get container status \"a172f2dc436d9dbaeb890dc0a1b84e153bde2a68507062cd17aac0926fe6a981\": rpc error: code = NotFound desc = could not find container \"a172f2dc436d9dbaeb890dc0a1b84e153bde2a68507062cd17aac0926fe6a981\": container with ID starting with a172f2dc436d9dbaeb890dc0a1b84e153bde2a68507062cd17aac0926fe6a981 not found: ID does not exist" Oct 31 04:23:01.571961 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.571357 5019 scope.go:122] "RemoveContainer" containerID="dece7da60db378cea7ec0d7fef6b187d2624546e21795d2c15856442df3500e9" Oct 31 04:23:01.571961 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:23:01.571813 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dece7da60db378cea7ec0d7fef6b187d2624546e21795d2c15856442df3500e9\": container with ID starting with dece7da60db378cea7ec0d7fef6b187d2624546e21795d2c15856442df3500e9 not found: ID does not exist" containerID="dece7da60db378cea7ec0d7fef6b187d2624546e21795d2c15856442df3500e9" Oct 31 04:23:01.571961 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.571881 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dece7da60db378cea7ec0d7fef6b187d2624546e21795d2c15856442df3500e9"} err="failed to get container status \"dece7da60db378cea7ec0d7fef6b187d2624546e21795d2c15856442df3500e9\": rpc error: code = NotFound desc = could not find container \"dece7da60db378cea7ec0d7fef6b187d2624546e21795d2c15856442df3500e9\": container with ID starting with dece7da60db378cea7ec0d7fef6b187d2624546e21795d2c15856442df3500e9 not found: ID does not exist" Oct 31 04:23:01.596235 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.596192 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv"] Oct 31 04:23:01.604453 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.604408 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-0fc77f-openshift-default-58dcf57b5-md2jv"] Oct 31 04:23:01.725460 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.725419 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-146320d5-65b76f4b67-m9qg8"] Oct 31 04:23:01.726082 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.726068 5019 cpu_manager.go:414] "containerMap: removing container" podUID="5053ea34-585c-4b1d-89d9-6eab5205471a" containerName="payload-processing" Oct 31 04:23:01.726133 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.726083 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5053ea34-585c-4b1d-89d9-6eab5205471a" containerName="payload-processing" Oct 31 04:23:01.726133 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.726102 5019 cpu_manager.go:414] "containerMap: removing container" podUID="a136b866-abc0-4c60-b977-ce6f05b0bdbf" containerName="payload-pre-processing" Oct 31 04:23:01.726133 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.726107 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a136b866-abc0-4c60-b977-ce6f05b0bdbf" containerName="payload-pre-processing" Oct 31 04:23:01.726133 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.726121 5019 cpu_manager.go:414] "containerMap: removing container" podUID="d8a637de-5932-47ad-b5e7-8d32ec65957f" containerName="payload-processing" Oct 31 04:23:01.726133 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.726126 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8a637de-5932-47ad-b5e7-8d32ec65957f" containerName="payload-processing" Oct 31 04:23:01.726287 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.726141 5019 cpu_manager.go:414] "containerMap: removing container" podUID="a9a72222-d749-4677-9621-99f1306e4ffd" containerName="payload-processing" Oct 31 04:23:01.726287 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.726145 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9a72222-d749-4677-9621-99f1306e4ffd" containerName="payload-processing" Oct 31 04:23:01.726287 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.726158 5019 cpu_manager.go:414] "containerMap: removing container" podUID="ffd1c004-3338-4cf6-b6f7-afebf7d01574" containerName="istio-proxy" Oct 31 04:23:01.726287 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.726162 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffd1c004-3338-4cf6-b6f7-afebf7d01574" containerName="istio-proxy" Oct 31 04:23:01.726287 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.726174 5019 cpu_manager.go:414] "containerMap: removing container" podUID="5cc83f70-27bb-42ee-9f5b-4d12dcb55df8" containerName="payload-pre-processing" Oct 31 04:23:01.726287 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.726179 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cc83f70-27bb-42ee-9f5b-4d12dcb55df8" containerName="payload-pre-processing" Oct 31 04:23:01.726287 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.726188 5019 cpu_manager.go:414] "containerMap: removing container" podUID="ffd1c004-3338-4cf6-b6f7-afebf7d01574" containerName="istio-proxy" Oct 31 04:23:01.726287 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.726193 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffd1c004-3338-4cf6-b6f7-afebf7d01574" containerName="istio-proxy" Oct 31 04:23:01.726287 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.726209 5019 cpu_manager.go:414] "containerMap: removing container" podUID="679c7d7b-2edf-4359-be21-d5b5010459cb" containerName="revoke-keys" Oct 31 04:23:01.726287 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.726213 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="679c7d7b-2edf-4359-be21-d5b5010459cb" containerName="revoke-keys" Oct 31 04:23:01.726585 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.726304 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="679c7d7b-2edf-4359-be21-d5b5010459cb" containerName="revoke-keys" Oct 31 04:23:01.726585 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.726315 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="5053ea34-585c-4b1d-89d9-6eab5205471a" containerName="payload-processing" Oct 31 04:23:01.726585 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.726323 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="a136b866-abc0-4c60-b977-ce6f05b0bdbf" containerName="payload-pre-processing" Oct 31 04:23:01.726585 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.726347 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8a637de-5932-47ad-b5e7-8d32ec65957f" containerName="payload-processing" Oct 31 04:23:01.726585 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.726358 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9a72222-d749-4677-9621-99f1306e4ffd" containerName="payload-processing" Oct 31 04:23:01.726585 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.726369 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffd1c004-3338-4cf6-b6f7-afebf7d01574" containerName="istio-proxy" Oct 31 04:23:01.726585 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.726376 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cc83f70-27bb-42ee-9f5b-4d12dcb55df8" containerName="payload-pre-processing" Oct 31 04:23:01.731875 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.731850 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-146320d5-65b76f4b67-m9qg8" Oct 31 04:23:01.746779 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.746726 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-146320d5\"" Oct 31 04:23:01.746925 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.746792 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-146320d5-dockercfg-hxzch\"" Oct 31 04:23:01.748687 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.748660 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-146320d5-65b76f4b67-m9qg8"] Oct 31 04:23:01.822299 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.822184 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfdb2\" (UniqueName: \"kubernetes.io/projected/154ee9f9-aa9e-443a-a882-a6856788f797-kube-api-access-xfdb2\") pod \"payload-pre-processing-e2e-mt-146320d5-65b76f4b67-m9qg8\" (UID: \"154ee9f9-aa9e-443a-a882-a6856788f797\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-146320d5-65b76f4b67-m9qg8" Oct 31 04:23:01.822299 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.822283 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/154ee9f9-aa9e-443a-a882-a6856788f797-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-146320d5-65b76f4b67-m9qg8\" (UID: \"154ee9f9-aa9e-443a-a882-a6856788f797\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-146320d5-65b76f4b67-m9qg8" Oct 31 04:23:01.843287 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.843245 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-146320d5-586748b797-k74kx"] Oct 31 04:23:01.844851 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.844816 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffd1c004-3338-4cf6-b6f7-afebf7d01574" containerName="istio-proxy" Oct 31 04:23:01.877716 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.877677 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-146320d5-586748b797-k74kx" Oct 31 04:23:01.925859 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.923816 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfdb2\" (UniqueName: \"kubernetes.io/projected/154ee9f9-aa9e-443a-a882-a6856788f797-kube-api-access-xfdb2\") pod \"payload-pre-processing-e2e-mt-146320d5-65b76f4b67-m9qg8\" (UID: \"154ee9f9-aa9e-443a-a882-a6856788f797\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-146320d5-65b76f4b67-m9qg8" Oct 31 04:23:01.925859 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.923918 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/154ee9f9-aa9e-443a-a882-a6856788f797-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-146320d5-65b76f4b67-m9qg8\" (UID: \"154ee9f9-aa9e-443a-a882-a6856788f797\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-146320d5-65b76f4b67-m9qg8" Oct 31 04:23:01.925859 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.924026 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0d84d00-8487-4d34-b7ac-ab2a2aed0883-plugins-config-volume\") pod \"payload-processing-e2e-mt-146320d5-586748b797-k74kx\" (UID: \"d0d84d00-8487-4d34-b7ac-ab2a2aed0883\") " pod="openshift-ingress/payload-processing-e2e-mt-146320d5-586748b797-k74kx" Oct 31 04:23:01.925859 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.924152 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hxt9\" (UniqueName: \"kubernetes.io/projected/d0d84d00-8487-4d34-b7ac-ab2a2aed0883-kube-api-access-8hxt9\") pod \"payload-processing-e2e-mt-146320d5-586748b797-k74kx\" (UID: \"d0d84d00-8487-4d34-b7ac-ab2a2aed0883\") " pod="openshift-ingress/payload-processing-e2e-mt-146320d5-586748b797-k74kx" Oct 31 04:23:01.925859 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.925748 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/154ee9f9-aa9e-443a-a882-a6856788f797-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-146320d5-65b76f4b67-m9qg8\" (UID: \"154ee9f9-aa9e-443a-a882-a6856788f797\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-146320d5-65b76f4b67-m9qg8" Oct 31 04:23:01.996357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:01.992014 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfdb2\" (UniqueName: \"kubernetes.io/projected/154ee9f9-aa9e-443a-a882-a6856788f797-kube-api-access-xfdb2\") pod \"payload-pre-processing-e2e-mt-146320d5-65b76f4b67-m9qg8\" (UID: \"154ee9f9-aa9e-443a-a882-a6856788f797\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-146320d5-65b76f4b67-m9qg8" Oct 31 04:23:02.021507 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:02.021465 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-146320d5-586748b797-k74kx"] Oct 31 04:23:02.030870 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:02.030831 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hxt9\" (UniqueName: \"kubernetes.io/projected/d0d84d00-8487-4d34-b7ac-ab2a2aed0883-kube-api-access-8hxt9\") pod \"payload-processing-e2e-mt-146320d5-586748b797-k74kx\" (UID: \"d0d84d00-8487-4d34-b7ac-ab2a2aed0883\") " pod="openshift-ingress/payload-processing-e2e-mt-146320d5-586748b797-k74kx" Oct 31 04:23:02.031563 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:02.031514 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0d84d00-8487-4d34-b7ac-ab2a2aed0883-plugins-config-volume\") pod \"payload-processing-e2e-mt-146320d5-586748b797-k74kx\" (UID: \"d0d84d00-8487-4d34-b7ac-ab2a2aed0883\") " pod="openshift-ingress/payload-processing-e2e-mt-146320d5-586748b797-k74kx" Oct 31 04:23:02.034905 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:02.034874 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0d84d00-8487-4d34-b7ac-ab2a2aed0883-plugins-config-volume\") pod \"payload-processing-e2e-mt-146320d5-586748b797-k74kx\" (UID: \"d0d84d00-8487-4d34-b7ac-ab2a2aed0883\") " pod="openshift-ingress/payload-processing-e2e-mt-146320d5-586748b797-k74kx" Oct 31 04:23:02.047966 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:02.047924 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-146320d5-65b76f4b67-m9qg8" Oct 31 04:23:02.064916 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:02.064873 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hxt9\" (UniqueName: \"kubernetes.io/projected/d0d84d00-8487-4d34-b7ac-ab2a2aed0883-kube-api-access-8hxt9\") pod \"payload-processing-e2e-mt-146320d5-586748b797-k74kx\" (UID: \"d0d84d00-8487-4d34-b7ac-ab2a2aed0883\") " pod="openshift-ingress/payload-processing-e2e-mt-146320d5-586748b797-k74kx" Oct 31 04:23:02.207297 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:02.206809 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-146320d5-586748b797-k74kx" Oct 31 04:23:02.410167 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:02.410132 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-146320d5-65b76f4b67-m9qg8"] Oct 31 04:23:02.410578 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:23:02.410274 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod154ee9f9_aa9e_443a_a882_a6856788f797.slice/crio-6ae2608e9119a824f37ad252335f2b4c00657db8558d2148639a502bfc27e0c2 WatchSource:0}: Error finding container 6ae2608e9119a824f37ad252335f2b4c00657db8558d2148639a502bfc27e0c2: Status 404 returned error can't find the container with id 6ae2608e9119a824f37ad252335f2b4c00657db8558d2148639a502bfc27e0c2 Oct 31 04:23:02.544441 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:02.544403 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-146320d5-586748b797-k74kx"] Oct 31 04:23:02.555685 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:23:02.555439 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0d84d00_8487_4d34_b7ac_ab2a2aed0883.slice/crio-e7c8f58404db3dc7d660c63e9c6db1740b0cca35a5e858ce5f8693e6879fae8a WatchSource:0}: Error finding container e7c8f58404db3dc7d660c63e9c6db1740b0cca35a5e858ce5f8693e6879fae8a: Status 404 returned error can't find the container with id e7c8f58404db3dc7d660c63e9c6db1740b0cca35a5e858ce5f8693e6879fae8a Oct 31 04:23:02.560973 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:02.560947 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-146320d5-65b76f4b67-m9qg8" event={"ID":"154ee9f9-aa9e-443a-a882-a6856788f797","Type":"ContainerStarted","Data":"cc8a1bc0b8af7b52069db8fce578b91a465f44e8cacdbef381307206257b970d"} Oct 31 04:23:02.561167 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:02.561140 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-146320d5-65b76f4b67-m9qg8" event={"ID":"154ee9f9-aa9e-443a-a882-a6856788f797","Type":"ContainerStarted","Data":"6ae2608e9119a824f37ad252335f2b4c00657db8558d2148639a502bfc27e0c2"} Oct 31 04:23:02.561272 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:02.561183 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-146320d5-65b76f4b67-m9qg8" Oct 31 04:23:02.609145 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:02.609083 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-146320d5-65b76f4b67-m9qg8" podStartSLOduration=1.609066304 podStartE2EDuration="1.609066304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:23:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:23:02.603319599 +0000 UTC m=+1137.359657640" watchObservedRunningTime="2026-10-31 04:23:02.609066304 +0000 UTC m=+1137.365404355" Oct 31 04:23:03.419254 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:03.419216 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffd1c004-3338-4cf6-b6f7-afebf7d01574" path="/var/lib/kubelet/pods/ffd1c004-3338-4cf6-b6f7-afebf7d01574/volumes" Oct 31 04:23:03.574406 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:03.574353 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-146320d5-586748b797-k74kx" event={"ID":"d0d84d00-8487-4d34-b7ac-ab2a2aed0883","Type":"ContainerStarted","Data":"a1b6b4cba9684bdb83b0a4835f5cdd6d82c407cebb413fb0c7719c5c9ba6b8b2"} Oct 31 04:23:03.574584 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:03.574428 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-146320d5-586748b797-k74kx" Oct 31 04:23:03.574584 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:03.574450 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-146320d5-586748b797-k74kx" event={"ID":"d0d84d00-8487-4d34-b7ac-ab2a2aed0883","Type":"ContainerStarted","Data":"e7c8f58404db3dc7d660c63e9c6db1740b0cca35a5e858ce5f8693e6879fae8a"} Oct 31 04:23:03.619546 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:03.619478 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-146320d5-586748b797-k74kx" podStartSLOduration=2.619457671 podStartE2EDuration="2.619457671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:23:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:23:03.616421823 +0000 UTC m=+1138.372759890" watchObservedRunningTime="2026-10-31 04:23:03.619457671 +0000 UTC m=+1138.375795712" Oct 31 04:23:03.975238 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:03.975194 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3430f6-664c86db9f-xxx7v"] Oct 31 04:23:03.979909 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:03.979882 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3430f6-664c86db9f-xxx7v" Oct 31 04:23:03.997574 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:03.997265 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-3430f6\"" Oct 31 04:23:03.997574 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:03.997361 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-3430f6-dockercfg-zbzfw\"" Oct 31 04:23:04.012520 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:04.012476 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3430f6-664c86db9f-xxx7v"] Oct 31 04:23:04.120356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:04.109362 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0acff3b9-9a6e-4e4a-8b39-188266bb7a80-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3430f6-664c86db9f-xxx7v\" (UID: \"0acff3b9-9a6e-4e4a-8b39-188266bb7a80\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3430f6-664c86db9f-xxx7v" Oct 31 04:23:04.124540 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:04.124500 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk2k8\" (UniqueName: \"kubernetes.io/projected/0acff3b9-9a6e-4e4a-8b39-188266bb7a80-kube-api-access-hk2k8\") pod \"payload-pre-processing-e2e-ipp-a-3430f6-664c86db9f-xxx7v\" (UID: \"0acff3b9-9a6e-4e4a-8b39-188266bb7a80\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3430f6-664c86db9f-xxx7v" Oct 31 04:23:04.185371 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:04.180466 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3430f6-7b4cd58d57-jm4mv"] Oct 31 04:23:04.217906 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:04.206637 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3430f6-7b4cd58d57-jm4mv" Oct 31 04:23:04.217906 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:04.216912 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3430f6-7b4cd58d57-jm4mv"] Oct 31 04:23:04.226903 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:04.226795 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk2k8\" (UniqueName: \"kubernetes.io/projected/0acff3b9-9a6e-4e4a-8b39-188266bb7a80-kube-api-access-hk2k8\") pod \"payload-pre-processing-e2e-ipp-a-3430f6-664c86db9f-xxx7v\" (UID: \"0acff3b9-9a6e-4e4a-8b39-188266bb7a80\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3430f6-664c86db9f-xxx7v" Oct 31 04:23:04.227190 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:04.227162 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0acff3b9-9a6e-4e4a-8b39-188266bb7a80-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3430f6-664c86db9f-xxx7v\" (UID: \"0acff3b9-9a6e-4e4a-8b39-188266bb7a80\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3430f6-664c86db9f-xxx7v" Oct 31 04:23:04.228324 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:04.228291 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0acff3b9-9a6e-4e4a-8b39-188266bb7a80-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3430f6-664c86db9f-xxx7v\" (UID: \"0acff3b9-9a6e-4e4a-8b39-188266bb7a80\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3430f6-664c86db9f-xxx7v" Oct 31 04:23:04.252374 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:04.251654 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk2k8\" (UniqueName: \"kubernetes.io/projected/0acff3b9-9a6e-4e4a-8b39-188266bb7a80-kube-api-access-hk2k8\") pod \"payload-pre-processing-e2e-ipp-a-3430f6-664c86db9f-xxx7v\" (UID: \"0acff3b9-9a6e-4e4a-8b39-188266bb7a80\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3430f6-664c86db9f-xxx7v" Oct 31 04:23:04.299789 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:04.299750 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3430f6-664c86db9f-xxx7v" Oct 31 04:23:04.329203 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:04.329164 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h7cb\" (UniqueName: \"kubernetes.io/projected/00ad44ab-ef7b-4a5c-9552-efea287113a3-kube-api-access-6h7cb\") pod \"payload-processing-e2e-ipp-a-3430f6-7b4cd58d57-jm4mv\" (UID: \"00ad44ab-ef7b-4a5c-9552-efea287113a3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3430f6-7b4cd58d57-jm4mv" Oct 31 04:23:04.329450 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:04.329310 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00ad44ab-ef7b-4a5c-9552-efea287113a3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3430f6-7b4cd58d57-jm4mv\" (UID: \"00ad44ab-ef7b-4a5c-9552-efea287113a3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3430f6-7b4cd58d57-jm4mv" Oct 31 04:23:04.431411 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:04.430687 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00ad44ab-ef7b-4a5c-9552-efea287113a3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3430f6-7b4cd58d57-jm4mv\" (UID: \"00ad44ab-ef7b-4a5c-9552-efea287113a3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3430f6-7b4cd58d57-jm4mv" Oct 31 04:23:04.431411 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:04.430884 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h7cb\" (UniqueName: \"kubernetes.io/projected/00ad44ab-ef7b-4a5c-9552-efea287113a3-kube-api-access-6h7cb\") pod \"payload-processing-e2e-ipp-a-3430f6-7b4cd58d57-jm4mv\" (UID: \"00ad44ab-ef7b-4a5c-9552-efea287113a3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3430f6-7b4cd58d57-jm4mv" Oct 31 04:23:04.431938 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:04.431720 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00ad44ab-ef7b-4a5c-9552-efea287113a3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3430f6-7b4cd58d57-jm4mv\" (UID: \"00ad44ab-ef7b-4a5c-9552-efea287113a3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3430f6-7b4cd58d57-jm4mv" Oct 31 04:23:04.454315 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:04.454287 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h7cb\" (UniqueName: \"kubernetes.io/projected/00ad44ab-ef7b-4a5c-9552-efea287113a3-kube-api-access-6h7cb\") pod \"payload-processing-e2e-ipp-a-3430f6-7b4cd58d57-jm4mv\" (UID: \"00ad44ab-ef7b-4a5c-9552-efea287113a3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3430f6-7b4cd58d57-jm4mv" Oct 31 04:23:04.561615 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:04.561507 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3430f6-7b4cd58d57-jm4mv" Oct 31 04:23:04.642084 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:23:04.642042 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0acff3b9_9a6e_4e4a_8b39_188266bb7a80.slice/crio-1d1308b893d71de02013d897c6a1aa0763402b4204d83d54c8add5588e534ce8 WatchSource:0}: Error finding container 1d1308b893d71de02013d897c6a1aa0763402b4204d83d54c8add5588e534ce8: Status 404 returned error can't find the container with id 1d1308b893d71de02013d897c6a1aa0763402b4204d83d54c8add5588e534ce8 Oct 31 04:23:04.648411 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:04.646086 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3430f6-664c86db9f-xxx7v"] Oct 31 04:23:04.745207 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:04.744726 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942"] Oct 31 04:23:04.754386 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:04.754305 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942" Oct 31 04:23:04.760653 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:04.760063 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942"] Oct 31 04:23:04.774404 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:04.771986 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-ceba51-openshift-default-dockercfg-f49dc\"" Oct 31 04:23:04.843090 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:04.843004 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce06671-540a-43bb-8f84-a86082d76ca9-credential-socket\") pod \"e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942\" (UID: \"2ce06671-540a-43bb-8f84-a86082d76ca9\") " pod="openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942" Oct 31 04:23:04.843090 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:04.843055 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ce06671-540a-43bb-8f84-a86082d76ca9-istio-envoy\") pod \"e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942\" (UID: \"2ce06671-540a-43bb-8f84-a86082d76ca9\") " pod="openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942" Oct 31 04:23:04.843277 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:04.843131 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57szj\" (UniqueName: \"kubernetes.io/projected/2ce06671-540a-43bb-8f84-a86082d76ca9-kube-api-access-57szj\") pod \"e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942\" (UID: \"2ce06671-540a-43bb-8f84-a86082d76ca9\") " pod="openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942" Oct 31 04:23:04.843384 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:04.843320 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce06671-540a-43bb-8f84-a86082d76ca9-workload-socket\") pod \"e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942\" (UID: \"2ce06671-540a-43bb-8f84-a86082d76ca9\") " pod="openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942" Oct 31 04:23:04.843384 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:04.843362 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ce06671-540a-43bb-8f84-a86082d76ca9-istio-token\") pod \"e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942\" (UID: \"2ce06671-540a-43bb-8f84-a86082d76ca9\") " pod="openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942" Oct 31 04:23:04.843462 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:04.843442 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ce06671-540a-43bb-8f84-a86082d76ca9-istio-data\") pod \"e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942\" (UID: \"2ce06671-540a-43bb-8f84-a86082d76ca9\") " pod="openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942" Oct 31 04:23:04.843500 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:04.843487 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ce06671-540a-43bb-8f84-a86082d76ca9-istio-podinfo\") pod \"e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942\" (UID: \"2ce06671-540a-43bb-8f84-a86082d76ca9\") " pod="openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942" Oct 31 04:23:04.843586 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:04.843573 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ce06671-540a-43bb-8f84-a86082d76ca9-istiod-ca-cert\") pod \"e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942\" (UID: \"2ce06671-540a-43bb-8f84-a86082d76ca9\") " pod="openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942" Oct 31 04:23:04.843635 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:04.843614 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ce06671-540a-43bb-8f84-a86082d76ca9-workload-certs\") pod \"e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942\" (UID: \"2ce06671-540a-43bb-8f84-a86082d76ca9\") " pod="openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942" Oct 31 04:23:05.025932 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:05.025887 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce06671-540a-43bb-8f84-a86082d76ca9-credential-socket\") pod \"e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942\" (UID: \"2ce06671-540a-43bb-8f84-a86082d76ca9\") " pod="openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942" Oct 31 04:23:05.026220 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:05.026199 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ce06671-540a-43bb-8f84-a86082d76ca9-istio-envoy\") pod \"e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942\" (UID: \"2ce06671-540a-43bb-8f84-a86082d76ca9\") " pod="openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942" Oct 31 04:23:05.026428 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:05.026406 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57szj\" (UniqueName: \"kubernetes.io/projected/2ce06671-540a-43bb-8f84-a86082d76ca9-kube-api-access-57szj\") pod \"e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942\" (UID: \"2ce06671-540a-43bb-8f84-a86082d76ca9\") " pod="openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942" Oct 31 04:23:05.026868 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:05.026844 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce06671-540a-43bb-8f84-a86082d76ca9-workload-socket\") pod \"e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942\" (UID: \"2ce06671-540a-43bb-8f84-a86082d76ca9\") " pod="openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942" Oct 31 04:23:05.027033 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:05.027012 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ce06671-540a-43bb-8f84-a86082d76ca9-istio-token\") pod \"e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942\" (UID: \"2ce06671-540a-43bb-8f84-a86082d76ca9\") " pod="openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942" Oct 31 04:23:05.027207 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:05.027186 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ce06671-540a-43bb-8f84-a86082d76ca9-istio-data\") pod \"e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942\" (UID: \"2ce06671-540a-43bb-8f84-a86082d76ca9\") " pod="openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942" Oct 31 04:23:05.027488 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:05.027433 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ce06671-540a-43bb-8f84-a86082d76ca9-istio-podinfo\") pod \"e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942\" (UID: \"2ce06671-540a-43bb-8f84-a86082d76ca9\") " pod="openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942" Oct 31 04:23:05.027896 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:05.027877 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ce06671-540a-43bb-8f84-a86082d76ca9-istiod-ca-cert\") pod \"e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942\" (UID: \"2ce06671-540a-43bb-8f84-a86082d76ca9\") " pod="openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942" Oct 31 04:23:05.028125 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:05.028100 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ce06671-540a-43bb-8f84-a86082d76ca9-workload-certs\") pod \"e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942\" (UID: \"2ce06671-540a-43bb-8f84-a86082d76ca9\") " pod="openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942" Oct 31 04:23:05.042391 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:05.036232 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce06671-540a-43bb-8f84-a86082d76ca9-workload-socket\") pod \"e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942\" (UID: \"2ce06671-540a-43bb-8f84-a86082d76ca9\") " pod="openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942" Oct 31 04:23:05.042890 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:05.042858 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ce06671-540a-43bb-8f84-a86082d76ca9-istiod-ca-cert\") pod \"e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942\" (UID: \"2ce06671-540a-43bb-8f84-a86082d76ca9\") " pod="openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942" Oct 31 04:23:05.055511 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:05.051125 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ce06671-540a-43bb-8f84-a86082d76ca9-workload-certs\") pod \"e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942\" (UID: \"2ce06671-540a-43bb-8f84-a86082d76ca9\") " pod="openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942" Oct 31 04:23:05.055899 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:05.034657 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce06671-540a-43bb-8f84-a86082d76ca9-credential-socket\") pod \"e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942\" (UID: \"2ce06671-540a-43bb-8f84-a86082d76ca9\") " pod="openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942" Oct 31 04:23:05.056124 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:05.054894 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3430f6-7b4cd58d57-jm4mv"] Oct 31 04:23:05.059516 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:05.055263 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ce06671-540a-43bb-8f84-a86082d76ca9-istio-data\") pod \"e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942\" (UID: \"2ce06671-540a-43bb-8f84-a86082d76ca9\") " pod="openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942" Oct 31 04:23:05.081366 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:05.078490 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ce06671-540a-43bb-8f84-a86082d76ca9-istio-envoy\") pod \"e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942\" (UID: \"2ce06671-540a-43bb-8f84-a86082d76ca9\") " pod="openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942" Oct 31 04:23:05.088413 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:05.085952 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57szj\" (UniqueName: \"kubernetes.io/projected/2ce06671-540a-43bb-8f84-a86082d76ca9-kube-api-access-57szj\") pod \"e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942\" (UID: \"2ce06671-540a-43bb-8f84-a86082d76ca9\") " pod="openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942" Oct 31 04:23:05.090271 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:05.090239 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ce06671-540a-43bb-8f84-a86082d76ca9-istio-token\") pod \"e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942\" (UID: \"2ce06671-540a-43bb-8f84-a86082d76ca9\") " pod="openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942" Oct 31 04:23:05.118376 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:05.117382 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ce06671-540a-43bb-8f84-a86082d76ca9-istio-podinfo\") pod \"e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942\" (UID: \"2ce06671-540a-43bb-8f84-a86082d76ca9\") " pod="openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942" Oct 31 04:23:05.397468 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:05.397382 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-ceba51-openshift-default-dockercfg-f49dc\"" Oct 31 04:23:05.404051 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:05.404015 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942" Oct 31 04:23:05.601483 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:05.600258 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3430f6-664c86db9f-xxx7v" event={"ID":"0acff3b9-9a6e-4e4a-8b39-188266bb7a80","Type":"ContainerStarted","Data":"f5b3f666a991e92a2249e6b8cbeaf178db5f23e2b5834b4c0c2d40526d008313"} Oct 31 04:23:05.601483 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:05.600327 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3430f6-664c86db9f-xxx7v" event={"ID":"0acff3b9-9a6e-4e4a-8b39-188266bb7a80","Type":"ContainerStarted","Data":"1d1308b893d71de02013d897c6a1aa0763402b4204d83d54c8add5588e534ce8"} Oct 31 04:23:05.601483 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:05.601422 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3430f6-664c86db9f-xxx7v" Oct 31 04:23:05.603733 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:05.603703 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3430f6-7b4cd58d57-jm4mv" event={"ID":"00ad44ab-ef7b-4a5c-9552-efea287113a3","Type":"ContainerStarted","Data":"dbc04665c3c364a7dd91f216719aa101845f665d3a980021f7694023eda304ff"} Oct 31 04:23:05.603894 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:05.603753 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3430f6-7b4cd58d57-jm4mv" event={"ID":"00ad44ab-ef7b-4a5c-9552-efea287113a3","Type":"ContainerStarted","Data":"ed73969659e5c8f19304fd2c11218329f100113771e69b82173d5a0ba739dcea"} Oct 31 04:23:05.604512 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:05.604494 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3430f6-7b4cd58d57-jm4mv" Oct 31 04:23:05.647449 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:05.647385 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3430f6-664c86db9f-xxx7v" podStartSLOduration=2.647366154 podStartE2EDuration="2.647366154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:23:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:23:05.642830137 +0000 UTC m=+1140.399168174" watchObservedRunningTime="2026-10-31 04:23:05.647366154 +0000 UTC m=+1140.403704195" Oct 31 04:23:05.688836 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:05.688700 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-3430f6-7b4cd58d57-jm4mv" podStartSLOduration=1.6886798889999999 podStartE2EDuration="1.688679889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:23:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:23:05.684938567 +0000 UTC m=+1140.441276606" watchObservedRunningTime="2026-10-31 04:23:05.688679889 +0000 UTC m=+1140.445017929" Oct 31 04:23:05.725672 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:23:05.725642 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ce06671_540a_43bb_8f84_a86082d76ca9.slice/crio-3fd486fdeb401496802b9fecb1ab43869ae6cf0f262b0ff0b871ac15bd51fde2 WatchSource:0}: Error finding container 3fd486fdeb401496802b9fecb1ab43869ae6cf0f262b0ff0b871ac15bd51fde2: Status 404 returned error can't find the container with id 3fd486fdeb401496802b9fecb1ab43869ae6cf0f262b0ff0b871ac15bd51fde2 Oct 31 04:23:05.726868 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:05.726842 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942"] Oct 31 04:23:05.728492 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:05.728449 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:23:05.728572 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:05.728522 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:23:05.728572 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:05.728556 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:23:05.907498 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:05.887093 5019 scope.go:122] "RemoveContainer" containerID="563745c7ff3aba3843df63a930c0efc87fc18dc998fa4243a859348ac665d8f0" Oct 31 04:23:06.116908 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:06.116878 5019 scope.go:122] "RemoveContainer" containerID="42fba72c53c00791d286b6695aa10408498eb8b2cce6d101c52ec375f902c555" Oct 31 04:23:06.136892 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:06.136620 5019 scope.go:122] "RemoveContainer" containerID="1d579311d0b7a1fb59ffea97fdf927f8b061573faa60c8fc446be82226540472" Oct 31 04:23:06.159688 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:06.159659 5019 scope.go:122] "RemoveContainer" containerID="da034093389171276ad29506b7c5af074290ac60adce8f310b3bbb0dfe1aeb17" Oct 31 04:23:06.174278 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:06.174248 5019 scope.go:122] "RemoveContainer" containerID="082ddd2fe391efa043ef6abe7290c08ec836f283986d61b1d9ee0298936b2630" Oct 31 04:23:06.188176 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:06.188151 5019 scope.go:122] "RemoveContainer" containerID="8f2833fd1cf37f6602e62153528a3cf256608b58cdb428203a35d1899988e445" Oct 31 04:23:06.203171 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:06.203115 5019 scope.go:122] "RemoveContainer" containerID="47a9b26998e5b46c6a8e0fb1f0f1b4ade97e8a1978216ddd9b911a4e81520bb7" Oct 31 04:23:06.619677 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:06.619581 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942" event={"ID":"2ce06671-540a-43bb-8f84-a86082d76ca9","Type":"ContainerStarted","Data":"3093e7549aa7ca9d878bb0fe4e0c4e3d8d6fce4acc7e1dc854ada043b30318b8"} Oct 31 04:23:06.619677 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:06.619646 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942" event={"ID":"2ce06671-540a-43bb-8f84-a86082d76ca9","Type":"ContainerStarted","Data":"3fd486fdeb401496802b9fecb1ab43869ae6cf0f262b0ff0b871ac15bd51fde2"} Oct 31 04:23:06.670109 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:06.670052 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942" podStartSLOduration=2.670036891 podStartE2EDuration="2.670036891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:23:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:23:06.664654712 +0000 UTC m=+1141.420992750" watchObservedRunningTime="2026-10-31 04:23:06.670036891 +0000 UTC m=+1141.426374929" Oct 31 04:23:07.412759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:07.412724 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942" Oct 31 04:23:07.414991 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:07.414953 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942" Oct 31 04:23:07.629618 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:07.629572 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942" Oct 31 04:23:07.630714 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:07.630691 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942" Oct 31 04:23:13.578289 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:13.578257 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-146320d5-65b76f4b67-m9qg8" Oct 31 04:23:14.590306 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:14.590272 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-146320d5-586748b797-k74kx" Oct 31 04:23:17.632799 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:17.632758 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3430f6-7b4cd58d57-jm4mv" Oct 31 04:23:17.633287 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:17.632860 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3430f6-664c86db9f-xxx7v" Oct 31 04:23:19.895359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:19.895308 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnhnx"] Oct 31 04:23:19.903903 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:19.903871 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnhnx" Oct 31 04:23:19.922205 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:19.922167 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnhnx"] Oct 31 04:23:19.930298 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:19.926686 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 31 04:23:19.930298 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:19.926738 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 31 04:23:19.930298 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:19.926685 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 31 04:23:19.930298 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:19.926820 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-m4kc9\"" Oct 31 04:23:20.054957 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:20.054920 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/156f03ac-9365-42af-b18a-30f81ab01b7e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fnhnx\" (UID: \"156f03ac-9365-42af-b18a-30f81ab01b7e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnhnx" Oct 31 04:23:20.055137 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:20.054975 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/156f03ac-9365-42af-b18a-30f81ab01b7e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fnhnx\" (UID: \"156f03ac-9365-42af-b18a-30f81ab01b7e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnhnx" Oct 31 04:23:20.055182 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:20.055168 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/156f03ac-9365-42af-b18a-30f81ab01b7e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fnhnx\" (UID: \"156f03ac-9365-42af-b18a-30f81ab01b7e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnhnx" Oct 31 04:23:20.055222 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:20.055205 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/156f03ac-9365-42af-b18a-30f81ab01b7e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fnhnx\" (UID: \"156f03ac-9365-42af-b18a-30f81ab01b7e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnhnx" Oct 31 04:23:20.055256 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:20.055228 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/156f03ac-9365-42af-b18a-30f81ab01b7e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fnhnx\" (UID: \"156f03ac-9365-42af-b18a-30f81ab01b7e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnhnx" Oct 31 04:23:20.055344 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:20.055310 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtgfc\" (UniqueName: \"kubernetes.io/projected/156f03ac-9365-42af-b18a-30f81ab01b7e-kube-api-access-dtgfc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fnhnx\" (UID: \"156f03ac-9365-42af-b18a-30f81ab01b7e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnhnx" Oct 31 04:23:20.055465 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:20.055446 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/156f03ac-9365-42af-b18a-30f81ab01b7e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fnhnx\" (UID: \"156f03ac-9365-42af-b18a-30f81ab01b7e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnhnx" Oct 31 04:23:20.157140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:20.157054 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/156f03ac-9365-42af-b18a-30f81ab01b7e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fnhnx\" (UID: \"156f03ac-9365-42af-b18a-30f81ab01b7e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnhnx" Oct 31 04:23:20.157140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:20.157107 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/156f03ac-9365-42af-b18a-30f81ab01b7e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fnhnx\" (UID: \"156f03ac-9365-42af-b18a-30f81ab01b7e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnhnx" Oct 31 04:23:20.157407 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:20.157224 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/156f03ac-9365-42af-b18a-30f81ab01b7e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fnhnx\" (UID: \"156f03ac-9365-42af-b18a-30f81ab01b7e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnhnx" Oct 31 04:23:20.157407 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:20.157248 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/156f03ac-9365-42af-b18a-30f81ab01b7e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fnhnx\" (UID: \"156f03ac-9365-42af-b18a-30f81ab01b7e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnhnx" Oct 31 04:23:20.157407 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:20.157267 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/156f03ac-9365-42af-b18a-30f81ab01b7e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fnhnx\" (UID: \"156f03ac-9365-42af-b18a-30f81ab01b7e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnhnx" Oct 31 04:23:20.157407 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:20.157286 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtgfc\" (UniqueName: \"kubernetes.io/projected/156f03ac-9365-42af-b18a-30f81ab01b7e-kube-api-access-dtgfc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fnhnx\" (UID: \"156f03ac-9365-42af-b18a-30f81ab01b7e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnhnx" Oct 31 04:23:20.157669 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:20.157408 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/156f03ac-9365-42af-b18a-30f81ab01b7e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fnhnx\" (UID: \"156f03ac-9365-42af-b18a-30f81ab01b7e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnhnx" Oct 31 04:23:20.158028 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:20.158002 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/156f03ac-9365-42af-b18a-30f81ab01b7e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fnhnx\" (UID: \"156f03ac-9365-42af-b18a-30f81ab01b7e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnhnx" Oct 31 04:23:20.159972 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:20.159937 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/156f03ac-9365-42af-b18a-30f81ab01b7e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fnhnx\" (UID: \"156f03ac-9365-42af-b18a-30f81ab01b7e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnhnx" Oct 31 04:23:20.160099 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:20.160071 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/156f03ac-9365-42af-b18a-30f81ab01b7e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fnhnx\" (UID: \"156f03ac-9365-42af-b18a-30f81ab01b7e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnhnx" Oct 31 04:23:20.160301 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:20.160279 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/156f03ac-9365-42af-b18a-30f81ab01b7e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fnhnx\" (UID: \"156f03ac-9365-42af-b18a-30f81ab01b7e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnhnx" Oct 31 04:23:20.160389 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:20.160295 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/156f03ac-9365-42af-b18a-30f81ab01b7e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fnhnx\" (UID: \"156f03ac-9365-42af-b18a-30f81ab01b7e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnhnx" Oct 31 04:23:20.180767 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:20.180732 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/156f03ac-9365-42af-b18a-30f81ab01b7e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fnhnx\" (UID: \"156f03ac-9365-42af-b18a-30f81ab01b7e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnhnx" Oct 31 04:23:20.181037 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:20.181018 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtgfc\" (UniqueName: \"kubernetes.io/projected/156f03ac-9365-42af-b18a-30f81ab01b7e-kube-api-access-dtgfc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fnhnx\" (UID: \"156f03ac-9365-42af-b18a-30f81ab01b7e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnhnx" Oct 31 04:23:20.220060 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:20.220021 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnhnx" Oct 31 04:23:20.545313 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:23:20.545283 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod156f03ac_9365_42af_b18a_30f81ab01b7e.slice/crio-5f77de6d47e2a9bbcccc915f18a69f7f556b9250345b947268d2d73e501cf89b WatchSource:0}: Error finding container 5f77de6d47e2a9bbcccc915f18a69f7f556b9250345b947268d2d73e501cf89b: Status 404 returned error can't find the container with id 5f77de6d47e2a9bbcccc915f18a69f7f556b9250345b947268d2d73e501cf89b Oct 31 04:23:20.546317 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:20.546293 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnhnx"] Oct 31 04:23:20.804542 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:20.800189 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnhnx" event={"ID":"156f03ac-9365-42af-b18a-30f81ab01b7e","Type":"ContainerStarted","Data":"5f77de6d47e2a9bbcccc915f18a69f7f556b9250345b947268d2d73e501cf89b"} Oct 31 04:23:22.401186 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:23:22.401138 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:23:23.306043 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:23.305999 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:23:23.306152 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:23.306116 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:23:23.837810 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:23.837771 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnhnx" event={"ID":"156f03ac-9365-42af-b18a-30f81ab01b7e","Type":"ContainerStarted","Data":"2fbd4d02f2286b41d7f24edafdacf39cefdf3c9dc97da394d3cdd53eb5214e4a"} Oct 31 04:23:23.838246 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:23.837860 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnhnx" Oct 31 04:23:23.900863 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:23.900538 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnhnx" podStartSLOduration=2.143572313 podStartE2EDuration="4.900522819s" totalImagesPullingTime="2.756950506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:23:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:23:23.896771647 +0000 UTC m=+1158.653109687" watchObservedRunningTime="2026-10-31 04:23:23.900522819 +0000 UTC m=+1158.656860858" Oct 31 04:23:24.848711 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:24.848674 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnhnx" Oct 31 04:23:28.883929 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:28.883887 5019 generic.go:356] "Generic (PLEG): container finished" podID="f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce" containerID="16d60cd88841ca545a8c86b48b42ed682c09a0954529ba62ee2762b9eb064aaa" exitCode=0 Oct 31 04:23:28.884439 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:28.883963 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-mjq8n-nfjq4" event={"ID":"f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce","Type":"ContainerDied","Data":"16d60cd88841ca545a8c86b48b42ed682c09a0954529ba62ee2762b9eb064aaa"} Oct 31 04:23:30.080376 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:30.080350 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-mjq8n-nfjq4" Oct 31 04:23:30.174057 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:30.173968 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce-service-ca-bundle\") pod \"f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce\" (UID: \"f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce\") " Oct 31 04:23:30.174207 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:30.174093 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce-trusted-ca-bundle\") pod \"f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce\" (UID: \"f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce\") " Oct 31 04:23:30.174207 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:30.174172 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce-archives-path\") pod \"f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce\" (UID: \"f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce\") " Oct 31 04:23:30.174396 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:30.174275 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce-kube-api-access-2286g\" (UniqueName: \"kubernetes.io/projected/f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce-kube-api-access-2286g\") pod \"f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce\" (UID: \"f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce\") " Oct 31 04:23:30.174472 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:30.174429 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce-service-ca-bundle" pod "f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce" (UID: "f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:23:30.174602 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:30.174570 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce-trusted-ca-bundle" pod "f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce" (UID: "f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:23:30.174823 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:30.174797 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce-archives-path" pod "f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce" (UID: "f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:23:30.175039 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:30.175020 5019 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce-service-ca-bundle\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:23:30.175104 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:30.175045 5019 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce-trusted-ca-bundle\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:23:30.175104 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:30.175058 5019 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce-archives-path\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:23:30.177062 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:30.177035 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce-kube-api-access-2286g" pod "f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce" (UID: "f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce"). InnerVolumeSpecName "kube-api-access-2286g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:23:30.277137 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:30.277094 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2286g\" (UniqueName: \"kubernetes.io/projected/f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce-kube-api-access-2286g\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:23:30.907515 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:30.907483 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-mjq8n-nfjq4" Oct 31 04:23:30.907815 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:30.907524 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-mjq8n-nfjq4" event={"ID":"f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce","Type":"ContainerDied","Data":"3b6e8d378ba8add6bb7620fa35fb8a3c90a2a4ad10041efca0fabb5440cb7bc3"} Oct 31 04:23:30.907815 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:30.907571 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b6e8d378ba8add6bb7620fa35fb8a3c90a2a4ad10041efca0fabb5440cb7bc3" Oct 31 04:23:37.033790 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:37.033749 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-718ebc99-j2jrs"] Oct 31 04:23:37.053260 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:37.053230 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-718ebc99-j2jrs"] Oct 31 04:23:37.409589 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:37.409509 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="619a5998-8522-4f92-939a-32f500de6d6c" path="/var/lib/kubelet/pods/619a5998-8522-4f92-939a-32f500de6d6c/volumes" Oct 31 04:23:42.129922 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:42.129883 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3430f6-7b4cd58d57-jm4mv"] Oct 31 04:23:42.164715 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:42.164673 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3430f6-664c86db9f-xxx7v"] Oct 31 04:23:44.308487 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:44.308446 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-146320d5/e2e-lifecycle-model-146320d5-kserve-649fc69f97-k5vz4"] Oct 31 04:23:47.130472 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:47.130392 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-3430f6-7b4cd58d57-jm4mv" podUID="00ad44ab-ef7b-4a5c-9552-efea287113a3" containerName="payload-processing" containerID="cri-o://dbc04665c3c364a7dd91f216719aa101845f665d3a980021f7694023eda304ff" gracePeriod=25 Oct 31 04:23:47.165303 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:47.165249 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3430f6-664c86db9f-xxx7v" podUID="0acff3b9-9a6e-4e4a-8b39-188266bb7a80" containerName="payload-pre-processing" containerID="cri-o://f5b3f666a991e92a2249e6b8cbeaf178db5f23e2b5834b4c0c2d40526d008313" gracePeriod=25 Oct 31 04:23:47.343569 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:47.343470 5019 generic.go:356] "Generic (PLEG): container finished" podID="0acff3b9-9a6e-4e4a-8b39-188266bb7a80" containerID="f5b3f666a991e92a2249e6b8cbeaf178db5f23e2b5834b4c0c2d40526d008313" exitCode=0 Oct 31 04:23:47.343886 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:47.343860 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3430f6-664c86db9f-xxx7v" event={"ID":"0acff3b9-9a6e-4e4a-8b39-188266bb7a80","Type":"ContainerDied","Data":"f5b3f666a991e92a2249e6b8cbeaf178db5f23e2b5834b4c0c2d40526d008313"} Oct 31 04:23:47.349092 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:47.349056 5019 generic.go:356] "Generic (PLEG): container finished" podID="00ad44ab-ef7b-4a5c-9552-efea287113a3" containerID="dbc04665c3c364a7dd91f216719aa101845f665d3a980021f7694023eda304ff" exitCode=0 Oct 31 04:23:47.350303 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:47.350259 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3430f6-7b4cd58d57-jm4mv" event={"ID":"00ad44ab-ef7b-4a5c-9552-efea287113a3","Type":"ContainerDied","Data":"dbc04665c3c364a7dd91f216719aa101845f665d3a980021f7694023eda304ff"} Oct 31 04:23:47.663577 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:47.663548 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3430f6-664c86db9f-xxx7v" Oct 31 04:23:47.725193 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:47.725150 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3430f6-7b4cd58d57-jm4mv" Oct 31 04:23:47.825807 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:47.825770 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0acff3b9-9a6e-4e4a-8b39-188266bb7a80-kube-api-access-hk2k8\" (UniqueName: \"kubernetes.io/projected/0acff3b9-9a6e-4e4a-8b39-188266bb7a80-kube-api-access-hk2k8\") pod \"0acff3b9-9a6e-4e4a-8b39-188266bb7a80\" (UID: \"0acff3b9-9a6e-4e4a-8b39-188266bb7a80\") " Oct 31 04:23:47.826075 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:47.826059 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00ad44ab-ef7b-4a5c-9552-efea287113a3-kube-api-access-6h7cb\" (UniqueName: \"kubernetes.io/projected/00ad44ab-ef7b-4a5c-9552-efea287113a3-kube-api-access-6h7cb\") pod \"00ad44ab-ef7b-4a5c-9552-efea287113a3\" (UID: \"00ad44ab-ef7b-4a5c-9552-efea287113a3\") " Oct 31 04:23:47.826402 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:47.826384 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00ad44ab-ef7b-4a5c-9552-efea287113a3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00ad44ab-ef7b-4a5c-9552-efea287113a3-plugins-config-volume\") pod \"00ad44ab-ef7b-4a5c-9552-efea287113a3\" (UID: \"00ad44ab-ef7b-4a5c-9552-efea287113a3\") " Oct 31 04:23:47.826734 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:47.826716 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0acff3b9-9a6e-4e4a-8b39-188266bb7a80-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0acff3b9-9a6e-4e4a-8b39-188266bb7a80-plugins-config-volume\") pod \"0acff3b9-9a6e-4e4a-8b39-188266bb7a80\" (UID: \"0acff3b9-9a6e-4e4a-8b39-188266bb7a80\") " Oct 31 04:23:47.827279 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:47.826935 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00ad44ab-ef7b-4a5c-9552-efea287113a3-plugins-config-volume" pod "00ad44ab-ef7b-4a5c-9552-efea287113a3" (UID: "00ad44ab-ef7b-4a5c-9552-efea287113a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:23:47.827279 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:47.827152 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0acff3b9-9a6e-4e4a-8b39-188266bb7a80-plugins-config-volume" pod "0acff3b9-9a6e-4e4a-8b39-188266bb7a80" (UID: "0acff3b9-9a6e-4e4a-8b39-188266bb7a80"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:23:47.827933 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:47.827919 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00ad44ab-ef7b-4a5c-9552-efea287113a3-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:23:47.828017 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:47.828008 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0acff3b9-9a6e-4e4a-8b39-188266bb7a80-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:23:47.828945 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:47.828911 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00ad44ab-ef7b-4a5c-9552-efea287113a3-kube-api-access-6h7cb" pod "00ad44ab-ef7b-4a5c-9552-efea287113a3" (UID: "00ad44ab-ef7b-4a5c-9552-efea287113a3"). InnerVolumeSpecName "kube-api-access-6h7cb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:23:47.830138 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:47.830102 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0acff3b9-9a6e-4e4a-8b39-188266bb7a80-kube-api-access-hk2k8" pod "0acff3b9-9a6e-4e4a-8b39-188266bb7a80" (UID: "0acff3b9-9a6e-4e4a-8b39-188266bb7a80"). InnerVolumeSpecName "kube-api-access-hk2k8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:23:47.929774 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:47.929738 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk2k8\" (UniqueName: \"kubernetes.io/projected/0acff3b9-9a6e-4e4a-8b39-188266bb7a80-kube-api-access-hk2k8\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:23:47.930026 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:47.930006 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h7cb\" (UniqueName: \"kubernetes.io/projected/00ad44ab-ef7b-4a5c-9552-efea287113a3-kube-api-access-6h7cb\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:23:48.379378 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:48.375847 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3430f6-7b4cd58d57-jm4mv" event={"ID":"00ad44ab-ef7b-4a5c-9552-efea287113a3","Type":"ContainerDied","Data":"ed73969659e5c8f19304fd2c11218329f100113771e69b82173d5a0ba739dcea"} Oct 31 04:23:48.379378 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:48.375922 5019 scope.go:122] "RemoveContainer" containerID="dbc04665c3c364a7dd91f216719aa101845f665d3a980021f7694023eda304ff" Oct 31 04:23:48.379378 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:48.376083 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3430f6-7b4cd58d57-jm4mv" Oct 31 04:23:48.439784 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:48.434858 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3430f6-664c86db9f-xxx7v" event={"ID":"0acff3b9-9a6e-4e4a-8b39-188266bb7a80","Type":"ContainerDied","Data":"1d1308b893d71de02013d897c6a1aa0763402b4204d83d54c8add5588e534ce8"} Oct 31 04:23:48.439784 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:48.434974 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3430f6-664c86db9f-xxx7v" Oct 31 04:23:48.449142 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:48.449116 5019 scope.go:122] "RemoveContainer" containerID="f5b3f666a991e92a2249e6b8cbeaf178db5f23e2b5834b4c0c2d40526d008313" Oct 31 04:23:48.494721 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:48.471845 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3430f6-7b4cd58d57-jm4mv"] Oct 31 04:23:48.521376 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:48.516555 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3430f6-7b4cd58d57-jm4mv"] Oct 31 04:23:48.551265 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:48.548801 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3430f6-664c86db9f-xxx7v"] Oct 31 04:23:48.554001 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:48.553968 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3430f6-664c86db9f-xxx7v"] Oct 31 04:23:48.641609 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:48.641570 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3430f6-664c86db9f-xxx7v" podUID="0acff3b9-9a6e-4e4a-8b39-188266bb7a80" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.150:50052\" within 1s: context deadline exceeded" Oct 31 04:23:48.641909 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:23:48.641893 5019 logging.go:55] [core] [Channel #1147 SubChannel #1148]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.150:50052", ServerName: "10.132.0.150:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.150:50052: operation was canceled" Oct 31 04:23:48.642012 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:48.641982 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-a-3430f6-7b4cd58d57-jm4mv" podUID="00ad44ab-ef7b-4a5c-9552-efea287113a3" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.151:50052\" within 1s: context deadline exceeded" Oct 31 04:23:48.642099 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:23:48.642081 5019 logging.go:55] [core] [Channel #1145 SubChannel #1146]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.151:50052", ServerName: "10.132.0.151:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.151:50052: operation was canceled" Oct 31 04:23:49.050683 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:49.050634 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a9eebb8-cvcfp"] Oct 31 04:23:49.062389 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:49.059173 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a9eebb8-cvcfp"] Oct 31 04:23:49.417400 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:49.416873 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00ad44ab-ef7b-4a5c-9552-efea287113a3" path="/var/lib/kubelet/pods/00ad44ab-ef7b-4a5c-9552-efea287113a3/volumes" Oct 31 04:23:49.417400 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:49.417363 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0acff3b9-9a6e-4e4a-8b39-188266bb7a80" path="/var/lib/kubelet/pods/0acff3b9-9a6e-4e4a-8b39-188266bb7a80/volumes" Oct 31 04:23:49.417814 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:49.417783 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3024e5fc-2bb5-4018-b86c-a02d939c0d15" path="/var/lib/kubelet/pods/3024e5fc-2bb5-4018-b86c-a02d939c0d15/volumes" Oct 31 04:23:52.401550 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:23:52.401504 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:23:53.670538 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:53.670492 5019 patch_prober.go:29] interesting pod/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:23:53.670960 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:53.670571 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942" podUID="2ce06671-540a-43bb-8f84-a86082d76ca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:23:54.035448 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:54.035408 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-952b35ab-dqbsg"] Oct 31 04:23:54.046289 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:54.046248 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-952b35ab-dqbsg"] Oct 31 04:23:55.415634 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:55.415593 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="305f3b1f-387e-483d-a472-473fb4e33b3e" path="/var/lib/kubelet/pods/305f3b1f-387e-483d-a472-473fb4e33b3e/volumes" Oct 31 04:23:55.511885 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:55.511840 5019 generic.go:356] "Generic (PLEG): container finished" podID="2ce06671-540a-43bb-8f84-a86082d76ca9" containerID="3093e7549aa7ca9d878bb0fe4e0c4e3d8d6fce4acc7e1dc854ada043b30318b8" exitCode=0 Oct 31 04:23:55.512070 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:55.511910 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942" event={"ID":"2ce06671-540a-43bb-8f84-a86082d76ca9","Type":"ContainerDied","Data":"3093e7549aa7ca9d878bb0fe4e0c4e3d8d6fce4acc7e1dc854ada043b30318b8"} Oct 31 04:23:55.512499 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:55.512481 5019 scope.go:122] "RemoveContainer" containerID="3093e7549aa7ca9d878bb0fe4e0c4e3d8d6fce4acc7e1dc854ada043b30318b8" Oct 31 04:23:55.514247 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:55.514208 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:23:55.514364 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:55.514301 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:23:55.514443 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:55.514395 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:23:56.523982 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:56.523945 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942" event={"ID":"2ce06671-540a-43bb-8f84-a86082d76ca9","Type":"ContainerStarted","Data":"1bef70deeb305caf75822933c41b35551728f0b965372ea9f99fab4a563b8ad9"} Oct 31 04:23:57.427393 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:57.424894 5019 patch_prober.go:29] interesting pod/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 31 04:23:57.427393 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:57.424979 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942" podUID="2ce06671-540a-43bb-8f84-a86082d76ca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 31 04:23:57.456123 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:57.456087 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942" Oct 31 04:23:58.404851 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:58.404802 5019 patch_prober.go:29] interesting pod/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 31 04:23:58.405255 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:58.404883 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942" podUID="2ce06671-540a-43bb-8f84-a86082d76ca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 31 04:23:59.404355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:59.398628 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-146320d5/e2e-lifecycle-model-146320d5-kserve-649fc69f97-k5vz4" podUID="def655ab-7eaf-4943-80a4-354b5f13640e" containerName="main" containerID="cri-o://67a949150b647a19e8130fd904d665fa02e6575696d2e26a8e0f720e2b50f425" gracePeriod=45 Oct 31 04:23:59.407859 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:59.407823 5019 patch_prober.go:29] interesting pod/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 31 04:23:59.408435 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:59.408396 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942" podUID="2ce06671-540a-43bb-8f84-a86082d76ca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 31 04:23:59.624236 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:59.624037 5019 generic.go:356] "Generic (PLEG): container finished" podID="def655ab-7eaf-4943-80a4-354b5f13640e" containerID="67a949150b647a19e8130fd904d665fa02e6575696d2e26a8e0f720e2b50f425" exitCode=0 Oct 31 04:23:59.624236 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:59.624083 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-146320d5/e2e-lifecycle-model-146320d5-kserve-649fc69f97-k5vz4" event={"ID":"def655ab-7eaf-4943-80a4-354b5f13640e","Type":"ContainerDied","Data":"67a949150b647a19e8130fd904d665fa02e6575696d2e26a8e0f720e2b50f425"} Oct 31 04:23:59.823069 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:59.823043 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-146320d5/e2e-lifecycle-model-146320d5-kserve-649fc69f97-k5vz4" Oct 31 04:23:59.831791 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:59.831683 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/def655ab-7eaf-4943-80a4-354b5f13640e-tls-certs\" (UniqueName: \"kubernetes.io/secret/def655ab-7eaf-4943-80a4-354b5f13640e-tls-certs\") pod \"def655ab-7eaf-4943-80a4-354b5f13640e\" (UID: \"def655ab-7eaf-4943-80a4-354b5f13640e\") " Oct 31 04:23:59.831791 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:59.831776 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/def655ab-7eaf-4943-80a4-354b5f13640e-kube-api-access-mm4p9\" (UniqueName: \"kubernetes.io/projected/def655ab-7eaf-4943-80a4-354b5f13640e-kube-api-access-mm4p9\") pod \"def655ab-7eaf-4943-80a4-354b5f13640e\" (UID: \"def655ab-7eaf-4943-80a4-354b5f13640e\") " Oct 31 04:23:59.833721 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:59.831915 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/def655ab-7eaf-4943-80a4-354b5f13640e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/def655ab-7eaf-4943-80a4-354b5f13640e-tmp-dir\") pod \"def655ab-7eaf-4943-80a4-354b5f13640e\" (UID: \"def655ab-7eaf-4943-80a4-354b5f13640e\") " Oct 31 04:23:59.833721 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:59.832056 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/def655ab-7eaf-4943-80a4-354b5f13640e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/def655ab-7eaf-4943-80a4-354b5f13640e-dshm\") pod \"def655ab-7eaf-4943-80a4-354b5f13640e\" (UID: \"def655ab-7eaf-4943-80a4-354b5f13640e\") " Oct 31 04:23:59.833721 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:59.832165 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def655ab-7eaf-4943-80a4-354b5f13640e-tmp-dir" pod "def655ab-7eaf-4943-80a4-354b5f13640e" (UID: "def655ab-7eaf-4943-80a4-354b5f13640e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:23:59.833721 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:59.832189 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/def655ab-7eaf-4943-80a4-354b5f13640e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/def655ab-7eaf-4943-80a4-354b5f13640e-model-cache\") pod \"def655ab-7eaf-4943-80a4-354b5f13640e\" (UID: \"def655ab-7eaf-4943-80a4-354b5f13640e\") " Oct 31 04:23:59.833721 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:59.832264 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/def655ab-7eaf-4943-80a4-354b5f13640e-home\" (UniqueName: \"kubernetes.io/empty-dir/def655ab-7eaf-4943-80a4-354b5f13640e-home\") pod \"def655ab-7eaf-4943-80a4-354b5f13640e\" (UID: \"def655ab-7eaf-4943-80a4-354b5f13640e\") " Oct 31 04:23:59.833721 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:59.833054 5019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/def655ab-7eaf-4943-80a4-354b5f13640e-tmp-dir\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:23:59.833721 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:59.833566 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def655ab-7eaf-4943-80a4-354b5f13640e-home" pod "def655ab-7eaf-4943-80a4-354b5f13640e" (UID: "def655ab-7eaf-4943-80a4-354b5f13640e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:23:59.834214 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:59.833906 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def655ab-7eaf-4943-80a4-354b5f13640e-model-cache" pod "def655ab-7eaf-4943-80a4-354b5f13640e" (UID: "def655ab-7eaf-4943-80a4-354b5f13640e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:23:59.834689 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:59.834668 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def655ab-7eaf-4943-80a4-354b5f13640e-dshm" pod "def655ab-7eaf-4943-80a4-354b5f13640e" (UID: "def655ab-7eaf-4943-80a4-354b5f13640e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:23:59.835483 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:59.835464 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/def655ab-7eaf-4943-80a4-354b5f13640e-tls-certs" pod "def655ab-7eaf-4943-80a4-354b5f13640e" (UID: "def655ab-7eaf-4943-80a4-354b5f13640e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:23:59.839398 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:59.838870 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/def655ab-7eaf-4943-80a4-354b5f13640e-kube-api-access-mm4p9" pod "def655ab-7eaf-4943-80a4-354b5f13640e" (UID: "def655ab-7eaf-4943-80a4-354b5f13640e"). InnerVolumeSpecName "kube-api-access-mm4p9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:23:59.938832 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:59.938793 5019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/def655ab-7eaf-4943-80a4-354b5f13640e-tls-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:23:59.939069 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:59.939052 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm4p9\" (UniqueName: \"kubernetes.io/projected/def655ab-7eaf-4943-80a4-354b5f13640e-kube-api-access-mm4p9\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:23:59.939254 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:59.939234 5019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/def655ab-7eaf-4943-80a4-354b5f13640e-dshm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:23:59.939376 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:59.939365 5019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/def655ab-7eaf-4943-80a4-354b5f13640e-model-cache\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:23:59.939462 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:23:59.939453 5019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/def655ab-7eaf-4943-80a4-354b5f13640e-home\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:24:00.405254 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:00.405215 5019 patch_prober.go:29] interesting pod/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 31 04:24:00.405467 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:00.405279 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942" podUID="2ce06671-540a-43bb-8f84-a86082d76ca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 31 04:24:00.645721 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:00.645686 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-146320d5/e2e-lifecycle-model-146320d5-kserve-649fc69f97-k5vz4" Oct 31 04:24:00.646155 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:00.645685 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-146320d5/e2e-lifecycle-model-146320d5-kserve-649fc69f97-k5vz4" event={"ID":"def655ab-7eaf-4943-80a4-354b5f13640e","Type":"ContainerDied","Data":"0a8cdae8b227028ae44b12b250c928587e4540398224fecc39fe462882258b1f"} Oct 31 04:24:00.646155 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:00.645790 5019 scope.go:122] "RemoveContainer" containerID="67a949150b647a19e8130fd904d665fa02e6575696d2e26a8e0f720e2b50f425" Oct 31 04:24:00.717924 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:00.717888 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-146320d5/e2e-lifecycle-model-146320d5-kserve-649fc69f97-k5vz4"] Oct 31 04:24:00.729577 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:00.729541 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-146320d5/e2e-lifecycle-model-146320d5-kserve-649fc69f97-k5vz4"] Oct 31 04:24:01.404815 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:01.404771 5019 patch_prober.go:29] interesting pod/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 31 04:24:01.405039 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:01.404849 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942" podUID="2ce06671-540a-43bb-8f84-a86082d76ca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 31 04:24:01.418586 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:01.418547 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="def655ab-7eaf-4943-80a4-354b5f13640e" path="/var/lib/kubelet/pods/def655ab-7eaf-4943-80a4-354b5f13640e/volumes" Oct 31 04:24:02.408654 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:02.408616 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942" Oct 31 04:24:02.409218 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:02.409132 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942" Oct 31 04:24:02.409794 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:02.409773 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942" Oct 31 04:24:05.671606 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:05.671562 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s88kp_3945ff65-e877-4372-9334-3968c68b7295/console-operator/2.log" Oct 31 04:24:05.671606 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:05.671596 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s88kp_3945ff65-e877-4372-9334-3968c68b7295/console-operator/2.log" Oct 31 04:24:06.382359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:06.382297 5019 scope.go:122] "RemoveContainer" containerID="58b3e4ed32ff3e64514c6cca9cb584fddb7962af9d41624bb383aa8a0859855f" Oct 31 04:24:06.927096 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:06.927062 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f91cd1-7f59958f95-c2qrv"] Oct 31 04:24:06.927778 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:06.927763 5019 cpu_manager.go:414] "containerMap: removing container" podUID="def655ab-7eaf-4943-80a4-354b5f13640e" containerName="main" Oct 31 04:24:06.927827 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:06.927779 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="def655ab-7eaf-4943-80a4-354b5f13640e" containerName="main" Oct 31 04:24:06.927827 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:06.927796 5019 cpu_manager.go:414] "containerMap: removing container" podUID="0acff3b9-9a6e-4e4a-8b39-188266bb7a80" containerName="payload-pre-processing" Oct 31 04:24:06.927827 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:06.927802 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0acff3b9-9a6e-4e4a-8b39-188266bb7a80" containerName="payload-pre-processing" Oct 31 04:24:06.927827 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:06.927811 5019 cpu_manager.go:414] "containerMap: removing container" podUID="f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce" containerName="insights-gathering" Oct 31 04:24:06.927827 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:06.927816 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce" containerName="insights-gathering" Oct 31 04:24:06.927985 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:06.927853 5019 cpu_manager.go:414] "containerMap: removing container" podUID="00ad44ab-ef7b-4a5c-9552-efea287113a3" containerName="payload-processing" Oct 31 04:24:06.927985 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:06.927858 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00ad44ab-ef7b-4a5c-9552-efea287113a3" containerName="payload-processing" Oct 31 04:24:06.927985 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:06.927968 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="def655ab-7eaf-4943-80a4-354b5f13640e" containerName="main" Oct 31 04:24:06.927985 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:06.927980 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="0acff3b9-9a6e-4e4a-8b39-188266bb7a80" containerName="payload-pre-processing" Oct 31 04:24:06.927985 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:06.927985 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1a31334-e2b6-4e0b-93d4-ee9a003dd2ce" containerName="insights-gathering" Oct 31 04:24:06.928133 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:06.927999 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="00ad44ab-ef7b-4a5c-9552-efea287113a3" containerName="payload-processing" Oct 31 04:24:06.933034 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:06.933010 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91cd1-7f59958f95-c2qrv" Oct 31 04:24:06.951661 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:06.951632 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f91cd1-7f59958f95-c2qrv"] Oct 31 04:24:06.956733 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:06.956696 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f91cd1\"" Oct 31 04:24:06.956918 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:06.956806 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f91cd1-dockercfg-lzxfr\"" Oct 31 04:24:07.073043 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:07.073001 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g6vl\" (UniqueName: \"kubernetes.io/projected/5c95c936-5fcf-4bb0-8701-09f544af2fe0-kube-api-access-7g6vl\") pod \"payload-pre-processing-e2e-ipp-a-f91cd1-7f59958f95-c2qrv\" (UID: \"5c95c936-5fcf-4bb0-8701-09f544af2fe0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91cd1-7f59958f95-c2qrv" Oct 31 04:24:07.073238 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:07.073078 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c95c936-5fcf-4bb0-8701-09f544af2fe0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f91cd1-7f59958f95-c2qrv\" (UID: \"5c95c936-5fcf-4bb0-8701-09f544af2fe0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91cd1-7f59958f95-c2qrv" Oct 31 04:24:07.174540 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:07.174486 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g6vl\" (UniqueName: \"kubernetes.io/projected/5c95c936-5fcf-4bb0-8701-09f544af2fe0-kube-api-access-7g6vl\") pod \"payload-pre-processing-e2e-ipp-a-f91cd1-7f59958f95-c2qrv\" (UID: \"5c95c936-5fcf-4bb0-8701-09f544af2fe0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91cd1-7f59958f95-c2qrv" Oct 31 04:24:07.174734 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:07.174577 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c95c936-5fcf-4bb0-8701-09f544af2fe0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f91cd1-7f59958f95-c2qrv\" (UID: \"5c95c936-5fcf-4bb0-8701-09f544af2fe0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91cd1-7f59958f95-c2qrv" Oct 31 04:24:07.175562 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:07.175533 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c95c936-5fcf-4bb0-8701-09f544af2fe0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f91cd1-7f59958f95-c2qrv\" (UID: \"5c95c936-5fcf-4bb0-8701-09f544af2fe0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91cd1-7f59958f95-c2qrv" Oct 31 04:24:07.211450 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:07.211367 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g6vl\" (UniqueName: \"kubernetes.io/projected/5c95c936-5fcf-4bb0-8701-09f544af2fe0-kube-api-access-7g6vl\") pod \"payload-pre-processing-e2e-ipp-a-f91cd1-7f59958f95-c2qrv\" (UID: \"5c95c936-5fcf-4bb0-8701-09f544af2fe0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91cd1-7f59958f95-c2qrv" Oct 31 04:24:07.248890 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:07.248850 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91cd1-7f59958f95-c2qrv" Oct 31 04:24:07.560370 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:24:07.560307 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c95c936_5fcf_4bb0_8701_09f544af2fe0.slice/crio-4862742438d9e9d75fa32f77caf40596ee228c9d203c715a80adc0f10cb5472c WatchSource:0}: Error finding container 4862742438d9e9d75fa32f77caf40596ee228c9d203c715a80adc0f10cb5472c: Status 404 returned error can't find the container with id 4862742438d9e9d75fa32f77caf40596ee228c9d203c715a80adc0f10cb5472c Oct 31 04:24:07.562012 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:07.561979 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f91cd1-7f59958f95-c2qrv"] Oct 31 04:24:07.825108 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:07.825020 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91cd1-7f59958f95-c2qrv" event={"ID":"5c95c936-5fcf-4bb0-8701-09f544af2fe0","Type":"ContainerStarted","Data":"4862742438d9e9d75fa32f77caf40596ee228c9d203c715a80adc0f10cb5472c"} Oct 31 04:24:09.033674 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:09.033630 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb113745-5x7gx"] Oct 31 04:24:09.045902 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:09.045861 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb113745-5x7gx"] Oct 31 04:24:09.413472 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:09.413316 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bb40fb5-2be2-4273-93f4-fe4d4ca8ddef" path="/var/lib/kubelet/pods/3bb40fb5-2be2-4273-93f4-fe4d4ca8ddef/volumes" Oct 31 04:24:09.851808 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:09.851760 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91cd1-7f59958f95-c2qrv" event={"ID":"5c95c936-5fcf-4bb0-8701-09f544af2fe0","Type":"ContainerStarted","Data":"8ba7f92a6cfc96cd7b27986b3603109da5ee467c6ec21fdfb8e261a4529e02a9"} Oct 31 04:24:09.852006 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:09.851876 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91cd1-7f59958f95-c2qrv" Oct 31 04:24:09.893543 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:09.893407 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91cd1-7f59958f95-c2qrv" podStartSLOduration=2.294279294 podStartE2EDuration="3.893387054s" totalImagesPullingTime="1.59910776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:24:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:24:09.892435762 +0000 UTC m=+1204.648773801" watchObservedRunningTime="2026-10-31 04:24:09.893387054 +0000 UTC m=+1204.649725094" Oct 31 04:24:13.636945 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:13.636906 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ceba51-9776c4ffb-vddkl"] Oct 31 04:24:13.643370 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:13.643130 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ceba51-9776c4ffb-vddkl" Oct 31 04:24:13.661010 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:13.660972 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ceba51-9776c4ffb-vddkl"] Oct 31 04:24:13.690218 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:13.665981 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-ceba51\"" Oct 31 04:24:13.690218 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:13.666139 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-ceba51-dockercfg-f9w4l\"" Oct 31 04:24:13.751440 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:13.751399 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bda0a48-7e59-4b49-a555-ba3d7c2d6fe7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ceba51-9776c4ffb-vddkl\" (UID: \"9bda0a48-7e59-4b49-a555-ba3d7c2d6fe7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ceba51-9776c4ffb-vddkl" Oct 31 04:24:13.751946 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:13.751917 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2l6q\" (UniqueName: \"kubernetes.io/projected/9bda0a48-7e59-4b49-a555-ba3d7c2d6fe7-kube-api-access-m2l6q\") pod \"payload-processing-e2e-ipp-a-ceba51-9776c4ffb-vddkl\" (UID: \"9bda0a48-7e59-4b49-a555-ba3d7c2d6fe7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ceba51-9776c4ffb-vddkl" Oct 31 04:24:13.856686 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:13.855701 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bda0a48-7e59-4b49-a555-ba3d7c2d6fe7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ceba51-9776c4ffb-vddkl\" (UID: \"9bda0a48-7e59-4b49-a555-ba3d7c2d6fe7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ceba51-9776c4ffb-vddkl" Oct 31 04:24:13.856686 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:13.855898 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2l6q\" (UniqueName: \"kubernetes.io/projected/9bda0a48-7e59-4b49-a555-ba3d7c2d6fe7-kube-api-access-m2l6q\") pod \"payload-processing-e2e-ipp-a-ceba51-9776c4ffb-vddkl\" (UID: \"9bda0a48-7e59-4b49-a555-ba3d7c2d6fe7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ceba51-9776c4ffb-vddkl" Oct 31 04:24:13.858066 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:13.858015 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bda0a48-7e59-4b49-a555-ba3d7c2d6fe7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ceba51-9776c4ffb-vddkl\" (UID: \"9bda0a48-7e59-4b49-a555-ba3d7c2d6fe7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ceba51-9776c4ffb-vddkl" Oct 31 04:24:14.010215 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:14.010182 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2l6q\" (UniqueName: \"kubernetes.io/projected/9bda0a48-7e59-4b49-a555-ba3d7c2d6fe7-kube-api-access-m2l6q\") pod \"payload-processing-e2e-ipp-a-ceba51-9776c4ffb-vddkl\" (UID: \"9bda0a48-7e59-4b49-a555-ba3d7c2d6fe7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ceba51-9776c4ffb-vddkl" Oct 31 04:24:14.055959 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:14.055916 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ceba51-9776c4ffb-vddkl" Oct 31 04:24:14.370937 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:24:14.370901 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bda0a48_7e59_4b49_a555_ba3d7c2d6fe7.slice/crio-670157b238931767fcd2309e2d5191c26a0f8135c1aa1f30ade276bca1759358 WatchSource:0}: Error finding container 670157b238931767fcd2309e2d5191c26a0f8135c1aa1f30ade276bca1759358: Status 404 returned error can't find the container with id 670157b238931767fcd2309e2d5191c26a0f8135c1aa1f30ade276bca1759358 Oct 31 04:24:14.371577 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:14.371546 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ceba51-9776c4ffb-vddkl"] Oct 31 04:24:15.029580 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:15.029543 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ceba51-9776c4ffb-vddkl" event={"ID":"9bda0a48-7e59-4b49-a555-ba3d7c2d6fe7","Type":"ContainerStarted","Data":"b845341f9712881c3562b23304768f72f641e1ee11865158a230e336d08ea91a"} Oct 31 04:24:15.030010 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:15.029601 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ceba51-9776c4ffb-vddkl" event={"ID":"9bda0a48-7e59-4b49-a555-ba3d7c2d6fe7","Type":"ContainerStarted","Data":"670157b238931767fcd2309e2d5191c26a0f8135c1aa1f30ade276bca1759358"} Oct 31 04:24:15.030010 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:15.029763 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ceba51-9776c4ffb-vddkl" Oct 31 04:24:15.076097 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:15.076027 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-ceba51-9776c4ffb-vddkl" podStartSLOduration=2.076005102 podStartE2EDuration="2.076005102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:24:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:24:15.07125745 +0000 UTC m=+1209.827595488" watchObservedRunningTime="2026-10-31 04:24:15.076005102 +0000 UTC m=+1209.832343141" Oct 31 04:24:19.639202 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:19.639163 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp"] Oct 31 04:24:19.644863 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:19.644831 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp" Oct 31 04:24:19.675988 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:19.675913 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-7b4185-openshift-default-dockercfg-zmmmj\"" Oct 31 04:24:19.694407 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:19.694370 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp"] Oct 31 04:24:19.696691 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:19.696658 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0712577c-fb53-4025-9d1e-6c62e6bf38cb-workload-socket\") pod \"e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp\" (UID: \"0712577c-fb53-4025-9d1e-6c62e6bf38cb\") " pod="openshift-ingress/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp" Oct 31 04:24:19.696945 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:19.696925 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27hbs\" (UniqueName: \"kubernetes.io/projected/0712577c-fb53-4025-9d1e-6c62e6bf38cb-kube-api-access-27hbs\") pod \"e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp\" (UID: \"0712577c-fb53-4025-9d1e-6c62e6bf38cb\") " pod="openshift-ingress/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp" Oct 31 04:24:19.697082 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:19.696982 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0712577c-fb53-4025-9d1e-6c62e6bf38cb-workload-certs\") pod \"e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp\" (UID: \"0712577c-fb53-4025-9d1e-6c62e6bf38cb\") " pod="openshift-ingress/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp" Oct 31 04:24:19.697197 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:19.697182 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0712577c-fb53-4025-9d1e-6c62e6bf38cb-istio-podinfo\") pod \"e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp\" (UID: \"0712577c-fb53-4025-9d1e-6c62e6bf38cb\") " pod="openshift-ingress/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp" Oct 31 04:24:19.697303 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:19.697224 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0712577c-fb53-4025-9d1e-6c62e6bf38cb-istio-envoy\") pod \"e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp\" (UID: \"0712577c-fb53-4025-9d1e-6c62e6bf38cb\") " pod="openshift-ingress/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp" Oct 31 04:24:19.697303 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:19.697257 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0712577c-fb53-4025-9d1e-6c62e6bf38cb-credential-socket\") pod \"e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp\" (UID: \"0712577c-fb53-4025-9d1e-6c62e6bf38cb\") " pod="openshift-ingress/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp" Oct 31 04:24:19.697303 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:19.697321 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0712577c-fb53-4025-9d1e-6c62e6bf38cb-istio-data\") pod \"e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp\" (UID: \"0712577c-fb53-4025-9d1e-6c62e6bf38cb\") " pod="openshift-ingress/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp" Oct 31 04:24:19.697600 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:19.697383 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0712577c-fb53-4025-9d1e-6c62e6bf38cb-istio-token\") pod \"e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp\" (UID: \"0712577c-fb53-4025-9d1e-6c62e6bf38cb\") " pod="openshift-ingress/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp" Oct 31 04:24:19.697600 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:19.697473 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0712577c-fb53-4025-9d1e-6c62e6bf38cb-istiod-ca-cert\") pod \"e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp\" (UID: \"0712577c-fb53-4025-9d1e-6c62e6bf38cb\") " pod="openshift-ingress/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp" Oct 31 04:24:19.813363 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:19.804375 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0712577c-fb53-4025-9d1e-6c62e6bf38cb-istiod-ca-cert\") pod \"e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp\" (UID: \"0712577c-fb53-4025-9d1e-6c62e6bf38cb\") " pod="openshift-ingress/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp" Oct 31 04:24:19.846677 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:19.814452 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0712577c-fb53-4025-9d1e-6c62e6bf38cb-istiod-ca-cert\") pod \"e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp\" (UID: \"0712577c-fb53-4025-9d1e-6c62e6bf38cb\") " pod="openshift-ingress/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp" Oct 31 04:24:19.847790 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:19.847175 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0712577c-fb53-4025-9d1e-6c62e6bf38cb-workload-socket\") pod \"e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp\" (UID: \"0712577c-fb53-4025-9d1e-6c62e6bf38cb\") " pod="openshift-ingress/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp" Oct 31 04:24:19.857651 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:19.847750 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0712577c-fb53-4025-9d1e-6c62e6bf38cb-workload-socket\") pod \"e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp\" (UID: \"0712577c-fb53-4025-9d1e-6c62e6bf38cb\") " pod="openshift-ingress/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp" Oct 31 04:24:19.858619 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:19.858407 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27hbs\" (UniqueName: \"kubernetes.io/projected/0712577c-fb53-4025-9d1e-6c62e6bf38cb-kube-api-access-27hbs\") pod \"e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp\" (UID: \"0712577c-fb53-4025-9d1e-6c62e6bf38cb\") " pod="openshift-ingress/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp" Oct 31 04:24:19.859015 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:19.858996 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0712577c-fb53-4025-9d1e-6c62e6bf38cb-workload-certs\") pod \"e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp\" (UID: \"0712577c-fb53-4025-9d1e-6c62e6bf38cb\") " pod="openshift-ingress/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp" Oct 31 04:24:19.863979 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:19.863713 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0712577c-fb53-4025-9d1e-6c62e6bf38cb-workload-certs\") pod \"e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp\" (UID: \"0712577c-fb53-4025-9d1e-6c62e6bf38cb\") " pod="openshift-ingress/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp" Oct 31 04:24:19.878941 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:19.878906 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0712577c-fb53-4025-9d1e-6c62e6bf38cb-istio-podinfo\") pod \"e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp\" (UID: \"0712577c-fb53-4025-9d1e-6c62e6bf38cb\") " pod="openshift-ingress/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp" Oct 31 04:24:19.879433 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:19.879402 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0712577c-fb53-4025-9d1e-6c62e6bf38cb-istio-envoy\") pod \"e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp\" (UID: \"0712577c-fb53-4025-9d1e-6c62e6bf38cb\") " pod="openshift-ingress/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp" Oct 31 04:24:19.880184 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:19.880155 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0712577c-fb53-4025-9d1e-6c62e6bf38cb-credential-socket\") pod \"e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp\" (UID: \"0712577c-fb53-4025-9d1e-6c62e6bf38cb\") " pod="openshift-ingress/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp" Oct 31 04:24:19.902930 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:19.880606 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0712577c-fb53-4025-9d1e-6c62e6bf38cb-credential-socket\") pod \"e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp\" (UID: \"0712577c-fb53-4025-9d1e-6c62e6bf38cb\") " pod="openshift-ingress/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp" Oct 31 04:24:19.903250 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:19.902253 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0712577c-fb53-4025-9d1e-6c62e6bf38cb-istio-envoy\") pod \"e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp\" (UID: \"0712577c-fb53-4025-9d1e-6c62e6bf38cb\") " pod="openshift-ingress/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp" Oct 31 04:24:19.903422 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:19.902996 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0712577c-fb53-4025-9d1e-6c62e6bf38cb-istio-data\") pod \"e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp\" (UID: \"0712577c-fb53-4025-9d1e-6c62e6bf38cb\") " pod="openshift-ingress/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp" Oct 31 04:24:19.903892 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:19.903285 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0712577c-fb53-4025-9d1e-6c62e6bf38cb-istio-data\") pod \"e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp\" (UID: \"0712577c-fb53-4025-9d1e-6c62e6bf38cb\") " pod="openshift-ingress/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp" Oct 31 04:24:19.904057 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:19.904039 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0712577c-fb53-4025-9d1e-6c62e6bf38cb-istio-token\") pod \"e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp\" (UID: \"0712577c-fb53-4025-9d1e-6c62e6bf38cb\") " pod="openshift-ingress/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp" Oct 31 04:24:19.924358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:19.923479 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27hbs\" (UniqueName: \"kubernetes.io/projected/0712577c-fb53-4025-9d1e-6c62e6bf38cb-kube-api-access-27hbs\") pod \"e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp\" (UID: \"0712577c-fb53-4025-9d1e-6c62e6bf38cb\") " pod="openshift-ingress/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp" Oct 31 04:24:19.928354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:19.928177 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0712577c-fb53-4025-9d1e-6c62e6bf38cb-istio-podinfo\") pod \"e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp\" (UID: \"0712577c-fb53-4025-9d1e-6c62e6bf38cb\") " pod="openshift-ingress/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp" Oct 31 04:24:19.986470 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:19.984966 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0712577c-fb53-4025-9d1e-6c62e6bf38cb-istio-token\") pod \"e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp\" (UID: \"0712577c-fb53-4025-9d1e-6c62e6bf38cb\") " pod="openshift-ingress/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp" Oct 31 04:24:19.990357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:19.990077 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp" Oct 31 04:24:20.258783 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:20.258741 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-146320d5-586748b797-k74kx"] Oct 31 04:24:20.293707 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:20.293185 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-146320d5-65b76f4b67-m9qg8"] Oct 31 04:24:20.378875 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:20.378842 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp"] Oct 31 04:24:20.379523 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:24:20.379484 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0712577c_fb53_4025_9d1e_6c62e6bf38cb.slice/crio-bd1ed829dbfa7de173662cad93e1379fca8e2b1593971647f6ceea2cf182fb92 WatchSource:0}: Error finding container bd1ed829dbfa7de173662cad93e1379fca8e2b1593971647f6ceea2cf182fb92: Status 404 returned error can't find the container with id bd1ed829dbfa7de173662cad93e1379fca8e2b1593971647f6ceea2cf182fb92 Oct 31 04:24:20.382758 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:20.382717 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:24:20.382888 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:20.382844 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:24:20.382950 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:20.382890 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:24:20.901492 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:20.901462 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91cd1-7f59958f95-c2qrv" Oct 31 04:24:21.034836 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:21.034791 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cf46490-9gq8n"] Oct 31 04:24:21.042063 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:21.042022 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cf46490-9gq8n"] Oct 31 04:24:21.259404 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:21.245212 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp" event={"ID":"0712577c-fb53-4025-9d1e-6c62e6bf38cb","Type":"ContainerStarted","Data":"c345d8d327628e6c8dde79bc244b066cd375bdec1301460167a7775f5ebf5822"} Oct 31 04:24:21.259404 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:21.245282 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp" event={"ID":"0712577c-fb53-4025-9d1e-6c62e6bf38cb","Type":"ContainerStarted","Data":"bd1ed829dbfa7de173662cad93e1379fca8e2b1593971647f6ceea2cf182fb92"} Oct 31 04:24:21.313355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:21.312577 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp" podStartSLOduration=2.312555068 podStartE2EDuration="2.312555068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:24:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:24:21.310581704 +0000 UTC m=+1216.066919742" watchObservedRunningTime="2026-10-31 04:24:21.312555068 +0000 UTC m=+1216.068893108" Oct 31 04:24:21.410916 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:21.410877 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79da2fc6-818e-4f6b-a99c-53ab652e796f" path="/var/lib/kubelet/pods/79da2fc6-818e-4f6b-a99c-53ab652e796f/volumes" Oct 31 04:24:22.007411 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:22.007323 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp" Oct 31 04:24:22.013842 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:22.013814 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp" Oct 31 04:24:22.253656 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:22.253616 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp" Oct 31 04:24:22.254771 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:22.254751 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp" Oct 31 04:24:22.394763 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:24:22.394669 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:24:23.030212 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:23.030175 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c009b7e-4rsg4"] Oct 31 04:24:23.042266 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:23.042213 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c009b7e-4rsg4"] Oct 31 04:24:23.410058 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:23.409970 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42c8c3da-6bd9-40b8-9fed-e34b8e1114c7" path="/var/lib/kubelet/pods/42c8c3da-6bd9-40b8-9fed-e34b8e1114c7/volumes" Oct 31 04:24:25.259447 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:25.259397 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-146320d5-586748b797-k74kx" podUID="d0d84d00-8487-4d34-b7ac-ab2a2aed0883" containerName="payload-processing" containerID="cri-o://a1b6b4cba9684bdb83b0a4835f5cdd6d82c407cebb413fb0c7719c5c9ba6b8b2" gracePeriod=25 Oct 31 04:24:25.293936 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:25.293865 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-146320d5-65b76f4b67-m9qg8" podUID="154ee9f9-aa9e-443a-a882-a6856788f797" containerName="payload-pre-processing" containerID="cri-o://cc8a1bc0b8af7b52069db8fce578b91a465f44e8cacdbef381307206257b970d" gracePeriod=25 Oct 31 04:24:25.876399 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:25.862948 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-146320d5-586748b797-k74kx" Oct 31 04:24:25.923419 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:25.921422 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-146320d5-65b76f4b67-m9qg8" Oct 31 04:24:25.996653 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:25.996618 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/154ee9f9-aa9e-443a-a882-a6856788f797-kube-api-access-xfdb2\" (UniqueName: \"kubernetes.io/projected/154ee9f9-aa9e-443a-a882-a6856788f797-kube-api-access-xfdb2\") pod \"154ee9f9-aa9e-443a-a882-a6856788f797\" (UID: \"154ee9f9-aa9e-443a-a882-a6856788f797\") " Oct 31 04:24:25.996871 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:25.996716 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0d84d00-8487-4d34-b7ac-ab2a2aed0883-kube-api-access-8hxt9\" (UniqueName: \"kubernetes.io/projected/d0d84d00-8487-4d34-b7ac-ab2a2aed0883-kube-api-access-8hxt9\") pod \"d0d84d00-8487-4d34-b7ac-ab2a2aed0883\" (UID: \"d0d84d00-8487-4d34-b7ac-ab2a2aed0883\") " Oct 31 04:24:25.997156 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:25.997135 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0d84d00-8487-4d34-b7ac-ab2a2aed0883-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0d84d00-8487-4d34-b7ac-ab2a2aed0883-plugins-config-volume\") pod \"d0d84d00-8487-4d34-b7ac-ab2a2aed0883\" (UID: \"d0d84d00-8487-4d34-b7ac-ab2a2aed0883\") " Oct 31 04:24:25.997236 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:25.997210 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/154ee9f9-aa9e-443a-a882-a6856788f797-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/154ee9f9-aa9e-443a-a882-a6856788f797-plugins-config-volume\") pod \"154ee9f9-aa9e-443a-a882-a6856788f797\" (UID: \"154ee9f9-aa9e-443a-a882-a6856788f797\") " Oct 31 04:24:25.998278 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:25.998247 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/154ee9f9-aa9e-443a-a882-a6856788f797-plugins-config-volume" pod "154ee9f9-aa9e-443a-a882-a6856788f797" (UID: "154ee9f9-aa9e-443a-a882-a6856788f797"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:24:25.998660 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:25.998638 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0d84d00-8487-4d34-b7ac-ab2a2aed0883-plugins-config-volume" pod "d0d84d00-8487-4d34-b7ac-ab2a2aed0883" (UID: "d0d84d00-8487-4d34-b7ac-ab2a2aed0883"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:24:26.005003 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:26.004965 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/154ee9f9-aa9e-443a-a882-a6856788f797-kube-api-access-xfdb2" pod "154ee9f9-aa9e-443a-a882-a6856788f797" (UID: "154ee9f9-aa9e-443a-a882-a6856788f797"). InnerVolumeSpecName "kube-api-access-xfdb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:24:26.006775 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:26.006741 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0d84d00-8487-4d34-b7ac-ab2a2aed0883-kube-api-access-8hxt9" pod "d0d84d00-8487-4d34-b7ac-ab2a2aed0883" (UID: "d0d84d00-8487-4d34-b7ac-ab2a2aed0883"). InnerVolumeSpecName "kube-api-access-8hxt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:24:26.041162 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:26.041126 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ceba51-9776c4ffb-vddkl" Oct 31 04:24:26.100562 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:26.100505 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/154ee9f9-aa9e-443a-a882-a6856788f797-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:24:26.100826 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:26.100804 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfdb2\" (UniqueName: \"kubernetes.io/projected/154ee9f9-aa9e-443a-a882-a6856788f797-kube-api-access-xfdb2\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:24:26.101012 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:26.100995 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hxt9\" (UniqueName: \"kubernetes.io/projected/d0d84d00-8487-4d34-b7ac-ab2a2aed0883-kube-api-access-8hxt9\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:24:26.101147 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:26.101132 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0d84d00-8487-4d34-b7ac-ab2a2aed0883-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:24:26.292684 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:26.292650 5019 generic.go:356] "Generic (PLEG): container finished" podID="d0d84d00-8487-4d34-b7ac-ab2a2aed0883" containerID="a1b6b4cba9684bdb83b0a4835f5cdd6d82c407cebb413fb0c7719c5c9ba6b8b2" exitCode=0 Oct 31 04:24:26.293187 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:26.292718 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-146320d5-586748b797-k74kx" Oct 31 04:24:26.293187 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:26.292733 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-146320d5-586748b797-k74kx" event={"ID":"d0d84d00-8487-4d34-b7ac-ab2a2aed0883","Type":"ContainerDied","Data":"a1b6b4cba9684bdb83b0a4835f5cdd6d82c407cebb413fb0c7719c5c9ba6b8b2"} Oct 31 04:24:26.293187 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:26.292804 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-146320d5-586748b797-k74kx" event={"ID":"d0d84d00-8487-4d34-b7ac-ab2a2aed0883","Type":"ContainerDied","Data":"e7c8f58404db3dc7d660c63e9c6db1740b0cca35a5e858ce5f8693e6879fae8a"} Oct 31 04:24:26.293187 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:26.292844 5019 scope.go:122] "RemoveContainer" containerID="a1b6b4cba9684bdb83b0a4835f5cdd6d82c407cebb413fb0c7719c5c9ba6b8b2" Oct 31 04:24:26.294905 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:26.294876 5019 generic.go:356] "Generic (PLEG): container finished" podID="154ee9f9-aa9e-443a-a882-a6856788f797" containerID="cc8a1bc0b8af7b52069db8fce578b91a465f44e8cacdbef381307206257b970d" exitCode=0 Oct 31 04:24:26.295032 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:26.294943 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-146320d5-65b76f4b67-m9qg8" Oct 31 04:24:26.295032 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:26.294953 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-146320d5-65b76f4b67-m9qg8" event={"ID":"154ee9f9-aa9e-443a-a882-a6856788f797","Type":"ContainerDied","Data":"cc8a1bc0b8af7b52069db8fce578b91a465f44e8cacdbef381307206257b970d"} Oct 31 04:24:26.295032 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:26.295003 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-146320d5-65b76f4b67-m9qg8" event={"ID":"154ee9f9-aa9e-443a-a882-a6856788f797","Type":"ContainerDied","Data":"6ae2608e9119a824f37ad252335f2b4c00657db8558d2148639a502bfc27e0c2"} Oct 31 04:24:26.306047 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:26.306021 5019 scope.go:122] "RemoveContainer" containerID="a1b6b4cba9684bdb83b0a4835f5cdd6d82c407cebb413fb0c7719c5c9ba6b8b2" Oct 31 04:24:26.306431 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:24:26.306402 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1b6b4cba9684bdb83b0a4835f5cdd6d82c407cebb413fb0c7719c5c9ba6b8b2\": container with ID starting with a1b6b4cba9684bdb83b0a4835f5cdd6d82c407cebb413fb0c7719c5c9ba6b8b2 not found: ID does not exist" containerID="a1b6b4cba9684bdb83b0a4835f5cdd6d82c407cebb413fb0c7719c5c9ba6b8b2" Oct 31 04:24:26.306526 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:26.306440 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1b6b4cba9684bdb83b0a4835f5cdd6d82c407cebb413fb0c7719c5c9ba6b8b2"} err="failed to get container status \"a1b6b4cba9684bdb83b0a4835f5cdd6d82c407cebb413fb0c7719c5c9ba6b8b2\": rpc error: code = NotFound desc = could not find container \"a1b6b4cba9684bdb83b0a4835f5cdd6d82c407cebb413fb0c7719c5c9ba6b8b2\": container with ID starting with a1b6b4cba9684bdb83b0a4835f5cdd6d82c407cebb413fb0c7719c5c9ba6b8b2 not found: ID does not exist" Oct 31 04:24:26.306526 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:26.306461 5019 scope.go:122] "RemoveContainer" containerID="cc8a1bc0b8af7b52069db8fce578b91a465f44e8cacdbef381307206257b970d" Oct 31 04:24:26.318827 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:26.318802 5019 scope.go:122] "RemoveContainer" containerID="cc8a1bc0b8af7b52069db8fce578b91a465f44e8cacdbef381307206257b970d" Oct 31 04:24:26.319203 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:24:26.319177 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc8a1bc0b8af7b52069db8fce578b91a465f44e8cacdbef381307206257b970d\": container with ID starting with cc8a1bc0b8af7b52069db8fce578b91a465f44e8cacdbef381307206257b970d not found: ID does not exist" containerID="cc8a1bc0b8af7b52069db8fce578b91a465f44e8cacdbef381307206257b970d" Oct 31 04:24:26.319290 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:26.319215 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc8a1bc0b8af7b52069db8fce578b91a465f44e8cacdbef381307206257b970d"} err="failed to get container status \"cc8a1bc0b8af7b52069db8fce578b91a465f44e8cacdbef381307206257b970d\": rpc error: code = NotFound desc = could not find container \"cc8a1bc0b8af7b52069db8fce578b91a465f44e8cacdbef381307206257b970d\": container with ID starting with cc8a1bc0b8af7b52069db8fce578b91a465f44e8cacdbef381307206257b970d not found: ID does not exist" Oct 31 04:24:26.394384 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:26.390606 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-146320d5-586748b797-k74kx"] Oct 31 04:24:26.417653 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:26.417602 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-146320d5-586748b797-k74kx"] Oct 31 04:24:26.469906 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:26.469868 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-146320d5-65b76f4b67-m9qg8"] Oct 31 04:24:26.478203 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:26.478166 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-146320d5-65b76f4b67-m9qg8"] Oct 31 04:24:27.432357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:27.432024 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="154ee9f9-aa9e-443a-a882-a6856788f797" path="/var/lib/kubelet/pods/154ee9f9-aa9e-443a-a882-a6856788f797/volumes" Oct 31 04:24:27.432860 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:27.432822 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0d84d00-8487-4d34-b7ac-ab2a2aed0883" path="/var/lib/kubelet/pods/d0d84d00-8487-4d34-b7ac-ab2a2aed0883/volumes" Oct 31 04:24:28.036788 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:28.036737 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95027377-8zmvk"] Oct 31 04:24:28.048745 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:28.048703 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95027377-8zmvk"] Oct 31 04:24:29.410984 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:29.410937 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a85e6a3c-fc5f-44a6-8616-8b7fc45cd35c" path="/var/lib/kubelet/pods/a85e6a3c-fc5f-44a6-8616-8b7fc45cd35c/volumes" Oct 31 04:24:32.665177 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:32.665138 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f"] Oct 31 04:24:32.665909 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:32.665888 5019 cpu_manager.go:414] "containerMap: removing container" podUID="154ee9f9-aa9e-443a-a882-a6856788f797" containerName="payload-pre-processing" Oct 31 04:24:32.665909 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:32.665908 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="154ee9f9-aa9e-443a-a882-a6856788f797" containerName="payload-pre-processing" Oct 31 04:24:32.666088 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:32.665926 5019 cpu_manager.go:414] "containerMap: removing container" podUID="d0d84d00-8487-4d34-b7ac-ab2a2aed0883" containerName="payload-processing" Oct 31 04:24:32.666088 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:32.665931 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0d84d00-8487-4d34-b7ac-ab2a2aed0883" containerName="payload-processing" Oct 31 04:24:32.666088 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:32.666044 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="154ee9f9-aa9e-443a-a882-a6856788f797" containerName="payload-pre-processing" Oct 31 04:24:32.666088 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:32.666052 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0d84d00-8487-4d34-b7ac-ab2a2aed0883" containerName="payload-processing" Oct 31 04:24:32.672793 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:32.672763 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f" Oct 31 04:24:32.695039 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:32.695003 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-a9b573-openshift-default-dockercfg-j6gbz\"" Oct 31 04:24:32.720432 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:32.715355 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f"] Oct 31 04:24:32.736292 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:32.736242 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1233843-bebe-44fc-82f5-6120e6302e4e-istio-token\") pod \"e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f\" (UID: \"e1233843-bebe-44fc-82f5-6120e6302e4e\") " pod="openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f" Oct 31 04:24:32.736292 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:32.736289 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1233843-bebe-44fc-82f5-6120e6302e4e-istiod-ca-cert\") pod \"e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f\" (UID: \"e1233843-bebe-44fc-82f5-6120e6302e4e\") " pod="openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f" Oct 31 04:24:32.736617 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:32.736440 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1233843-bebe-44fc-82f5-6120e6302e4e-istio-data\") pod \"e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f\" (UID: \"e1233843-bebe-44fc-82f5-6120e6302e4e\") " pod="openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f" Oct 31 04:24:32.736617 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:32.736485 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1233843-bebe-44fc-82f5-6120e6302e4e-credential-socket\") pod \"e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f\" (UID: \"e1233843-bebe-44fc-82f5-6120e6302e4e\") " pod="openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f" Oct 31 04:24:32.736617 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:32.736538 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1233843-bebe-44fc-82f5-6120e6302e4e-istio-podinfo\") pod \"e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f\" (UID: \"e1233843-bebe-44fc-82f5-6120e6302e4e\") " pod="openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f" Oct 31 04:24:32.736617 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:32.736570 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1233843-bebe-44fc-82f5-6120e6302e4e-workload-certs\") pod \"e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f\" (UID: \"e1233843-bebe-44fc-82f5-6120e6302e4e\") " pod="openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f" Oct 31 04:24:32.736848 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:32.736654 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1233843-bebe-44fc-82f5-6120e6302e4e-istio-envoy\") pod \"e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f\" (UID: \"e1233843-bebe-44fc-82f5-6120e6302e4e\") " pod="openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f" Oct 31 04:24:32.736848 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:32.736692 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp49s\" (UniqueName: \"kubernetes.io/projected/e1233843-bebe-44fc-82f5-6120e6302e4e-kube-api-access-zp49s\") pod \"e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f\" (UID: \"e1233843-bebe-44fc-82f5-6120e6302e4e\") " pod="openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f" Oct 31 04:24:32.736848 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:32.736776 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1233843-bebe-44fc-82f5-6120e6302e4e-workload-socket\") pod \"e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f\" (UID: \"e1233843-bebe-44fc-82f5-6120e6302e4e\") " pod="openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f" Oct 31 04:24:32.838347 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:32.838291 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1233843-bebe-44fc-82f5-6120e6302e4e-workload-socket\") pod \"e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f\" (UID: \"e1233843-bebe-44fc-82f5-6120e6302e4e\") " pod="openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f" Oct 31 04:24:32.838553 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:32.838531 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1233843-bebe-44fc-82f5-6120e6302e4e-istio-token\") pod \"e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f\" (UID: \"e1233843-bebe-44fc-82f5-6120e6302e4e\") " pod="openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f" Oct 31 04:24:32.838628 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:32.838569 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1233843-bebe-44fc-82f5-6120e6302e4e-istiod-ca-cert\") pod \"e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f\" (UID: \"e1233843-bebe-44fc-82f5-6120e6302e4e\") " pod="openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f" Oct 31 04:24:32.838628 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:32.838611 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1233843-bebe-44fc-82f5-6120e6302e4e-istio-data\") pod \"e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f\" (UID: \"e1233843-bebe-44fc-82f5-6120e6302e4e\") " pod="openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f" Oct 31 04:24:32.838733 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:32.838630 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1233843-bebe-44fc-82f5-6120e6302e4e-credential-socket\") pod \"e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f\" (UID: \"e1233843-bebe-44fc-82f5-6120e6302e4e\") " pod="openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f" Oct 31 04:24:32.838733 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:32.838666 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1233843-bebe-44fc-82f5-6120e6302e4e-istio-podinfo\") pod \"e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f\" (UID: \"e1233843-bebe-44fc-82f5-6120e6302e4e\") " pod="openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f" Oct 31 04:24:32.838733 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:32.838705 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1233843-bebe-44fc-82f5-6120e6302e4e-workload-certs\") pod \"e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f\" (UID: \"e1233843-bebe-44fc-82f5-6120e6302e4e\") " pod="openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f" Oct 31 04:24:32.838879 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:32.838804 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1233843-bebe-44fc-82f5-6120e6302e4e-istio-envoy\") pod \"e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f\" (UID: \"e1233843-bebe-44fc-82f5-6120e6302e4e\") " pod="openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f" Oct 31 04:24:32.838879 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:32.838847 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp49s\" (UniqueName: \"kubernetes.io/projected/e1233843-bebe-44fc-82f5-6120e6302e4e-kube-api-access-zp49s\") pod \"e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f\" (UID: \"e1233843-bebe-44fc-82f5-6120e6302e4e\") " pod="openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f" Oct 31 04:24:32.838985 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:32.838964 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1233843-bebe-44fc-82f5-6120e6302e4e-workload-socket\") pod \"e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f\" (UID: \"e1233843-bebe-44fc-82f5-6120e6302e4e\") " pod="openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f" Oct 31 04:24:32.839271 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:32.839234 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1233843-bebe-44fc-82f5-6120e6302e4e-credential-socket\") pod \"e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f\" (UID: \"e1233843-bebe-44fc-82f5-6120e6302e4e\") " pod="openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f" Oct 31 04:24:32.839466 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:32.839321 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1233843-bebe-44fc-82f5-6120e6302e4e-workload-certs\") pod \"e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f\" (UID: \"e1233843-bebe-44fc-82f5-6120e6302e4e\") " pod="openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f" Oct 31 04:24:32.839625 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:32.839601 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1233843-bebe-44fc-82f5-6120e6302e4e-istio-data\") pod \"e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f\" (UID: \"e1233843-bebe-44fc-82f5-6120e6302e4e\") " pod="openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f" Oct 31 04:24:32.840415 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:32.840390 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1233843-bebe-44fc-82f5-6120e6302e4e-istiod-ca-cert\") pod \"e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f\" (UID: \"e1233843-bebe-44fc-82f5-6120e6302e4e\") " pod="openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f" Oct 31 04:24:32.841595 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:32.841573 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1233843-bebe-44fc-82f5-6120e6302e4e-istio-envoy\") pod \"e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f\" (UID: \"e1233843-bebe-44fc-82f5-6120e6302e4e\") " pod="openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f" Oct 31 04:24:32.841762 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:32.841737 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1233843-bebe-44fc-82f5-6120e6302e4e-istio-podinfo\") pod \"e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f\" (UID: \"e1233843-bebe-44fc-82f5-6120e6302e4e\") " pod="openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f" Oct 31 04:24:32.905944 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:32.905870 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp49s\" (UniqueName: \"kubernetes.io/projected/e1233843-bebe-44fc-82f5-6120e6302e4e-kube-api-access-zp49s\") pod \"e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f\" (UID: \"e1233843-bebe-44fc-82f5-6120e6302e4e\") " pod="openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f" Oct 31 04:24:32.913417 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:32.912966 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1233843-bebe-44fc-82f5-6120e6302e4e-istio-token\") pod \"e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f\" (UID: \"e1233843-bebe-44fc-82f5-6120e6302e4e\") " pod="openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f" Oct 31 04:24:33.001138 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:33.000714 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f" Oct 31 04:24:33.391276 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:24:33.391236 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1233843_bebe_44fc_82f5_6120e6302e4e.slice/crio-46722d1f1ce73fd37c06390b0b97806ad251c7ce2245bd6717a88a49c93ffaaf WatchSource:0}: Error finding container 46722d1f1ce73fd37c06390b0b97806ad251c7ce2245bd6717a88a49c93ffaaf: Status 404 returned error can't find the container with id 46722d1f1ce73fd37c06390b0b97806ad251c7ce2245bd6717a88a49c93ffaaf Oct 31 04:24:33.392429 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:33.392397 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f"] Oct 31 04:24:33.394398 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:33.394308 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:24:33.394523 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:33.394413 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:24:33.394523 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:33.394459 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:24:34.043427 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:34.043374 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a88e2fc-hlqdw"] Oct 31 04:24:34.055177 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:34.055125 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a88e2fc-hlqdw"] Oct 31 04:24:34.404936 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:34.404804 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f" event={"ID":"e1233843-bebe-44fc-82f5-6120e6302e4e","Type":"ContainerStarted","Data":"c84058468ab86da9b8eb32217d97cde841ed75135638e3572c2ace3947c8b862"} Oct 31 04:24:34.404936 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:34.404917 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f" event={"ID":"e1233843-bebe-44fc-82f5-6120e6302e4e","Type":"ContainerStarted","Data":"46722d1f1ce73fd37c06390b0b97806ad251c7ce2245bd6717a88a49c93ffaaf"} Oct 31 04:24:34.450675 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:34.450613 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f" podStartSLOduration=2.450597628 podStartE2EDuration="2.450597628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:24:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:24:34.449686424 +0000 UTC m=+1229.206024483" watchObservedRunningTime="2026-10-31 04:24:34.450597628 +0000 UTC m=+1229.206935666" Oct 31 04:24:35.009923 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:35.001736 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f" Oct 31 04:24:35.073355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:35.062589 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a714eaa5-w46mr"] Oct 31 04:24:35.108357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:35.107357 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f" Oct 31 04:24:35.111257 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:35.109111 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a714eaa5-w46mr"] Oct 31 04:24:35.417037 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:35.416937 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4989f9aa-dd46-46da-9213-4055a2a3500e" path="/var/lib/kubelet/pods/4989f9aa-dd46-46da-9213-4055a2a3500e/volumes" Oct 31 04:24:35.417789 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:35.417759 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fde3d56b-03cd-45b0-a5b6-6f0377f79a84" path="/var/lib/kubelet/pods/fde3d56b-03cd-45b0-a5b6-6f0377f79a84/volumes" Oct 31 04:24:35.418303 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:35.418282 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f" Oct 31 04:24:35.419151 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:35.419132 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f" Oct 31 04:24:38.795604 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:38.795565 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a9b573-6c9676dddc-r5kx6"] Oct 31 04:24:38.815764 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:38.815702 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9b573-6c9676dddc-r5kx6" Oct 31 04:24:38.823684 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:38.823652 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a9b573-6c9676dddc-r5kx6"] Oct 31 04:24:38.844790 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:38.844746 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a9b573\"" Oct 31 04:24:38.844790 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:38.844758 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a9b573-dockercfg-7mvfl\"" Oct 31 04:24:38.977360 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:38.951483 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81895138-f13d-4457-894b-88439c752982-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a9b573-6c9676dddc-r5kx6\" (UID: \"81895138-f13d-4457-894b-88439c752982\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9b573-6c9676dddc-r5kx6" Oct 31 04:24:38.977360 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:38.951560 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95l8j\" (UniqueName: \"kubernetes.io/projected/81895138-f13d-4457-894b-88439c752982-kube-api-access-95l8j\") pod \"payload-pre-processing-e2e-ipp-b-a9b573-6c9676dddc-r5kx6\" (UID: \"81895138-f13d-4457-894b-88439c752982\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9b573-6c9676dddc-r5kx6" Oct 31 04:24:39.061019 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:39.060904 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81895138-f13d-4457-894b-88439c752982-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a9b573-6c9676dddc-r5kx6\" (UID: \"81895138-f13d-4457-894b-88439c752982\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9b573-6c9676dddc-r5kx6" Oct 31 04:24:39.061311 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:39.061281 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95l8j\" (UniqueName: \"kubernetes.io/projected/81895138-f13d-4457-894b-88439c752982-kube-api-access-95l8j\") pod \"payload-pre-processing-e2e-ipp-b-a9b573-6c9676dddc-r5kx6\" (UID: \"81895138-f13d-4457-894b-88439c752982\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9b573-6c9676dddc-r5kx6" Oct 31 04:24:39.063232 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:39.063201 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81895138-f13d-4457-894b-88439c752982-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a9b573-6c9676dddc-r5kx6\" (UID: \"81895138-f13d-4457-894b-88439c752982\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9b573-6c9676dddc-r5kx6" Oct 31 04:24:39.073055 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:39.073000 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a9b573-85d444bfdf-zzqlp"] Oct 31 04:24:39.115098 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:39.114925 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a9b573-85d444bfdf-zzqlp" Oct 31 04:24:39.121391 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:39.121317 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a9b573-85d444bfdf-zzqlp"] Oct 31 04:24:39.141934 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:39.138094 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95l8j\" (UniqueName: \"kubernetes.io/projected/81895138-f13d-4457-894b-88439c752982-kube-api-access-95l8j\") pod \"payload-pre-processing-e2e-ipp-b-a9b573-6c9676dddc-r5kx6\" (UID: \"81895138-f13d-4457-894b-88439c752982\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9b573-6c9676dddc-r5kx6" Oct 31 04:24:39.143927 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:39.143886 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9b573-6c9676dddc-r5kx6" Oct 31 04:24:39.266148 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:39.265578 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6ce1804-04ce-4380-af18-fcf81d528500-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a9b573-85d444bfdf-zzqlp\" (UID: \"c6ce1804-04ce-4380-af18-fcf81d528500\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a9b573-85d444bfdf-zzqlp" Oct 31 04:24:39.266148 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:39.265951 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bhqh\" (UniqueName: \"kubernetes.io/projected/c6ce1804-04ce-4380-af18-fcf81d528500-kube-api-access-9bhqh\") pod \"payload-processing-e2e-ipp-b-a9b573-85d444bfdf-zzqlp\" (UID: \"c6ce1804-04ce-4380-af18-fcf81d528500\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a9b573-85d444bfdf-zzqlp" Oct 31 04:24:39.367840 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:39.367732 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bhqh\" (UniqueName: \"kubernetes.io/projected/c6ce1804-04ce-4380-af18-fcf81d528500-kube-api-access-9bhqh\") pod \"payload-processing-e2e-ipp-b-a9b573-85d444bfdf-zzqlp\" (UID: \"c6ce1804-04ce-4380-af18-fcf81d528500\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a9b573-85d444bfdf-zzqlp" Oct 31 04:24:39.368033 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:39.367864 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6ce1804-04ce-4380-af18-fcf81d528500-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a9b573-85d444bfdf-zzqlp\" (UID: \"c6ce1804-04ce-4380-af18-fcf81d528500\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a9b573-85d444bfdf-zzqlp" Oct 31 04:24:39.368748 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:39.368716 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6ce1804-04ce-4380-af18-fcf81d528500-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a9b573-85d444bfdf-zzqlp\" (UID: \"c6ce1804-04ce-4380-af18-fcf81d528500\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a9b573-85d444bfdf-zzqlp" Oct 31 04:24:39.392528 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:39.392495 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bhqh\" (UniqueName: \"kubernetes.io/projected/c6ce1804-04ce-4380-af18-fcf81d528500-kube-api-access-9bhqh\") pod \"payload-processing-e2e-ipp-b-a9b573-85d444bfdf-zzqlp\" (UID: \"c6ce1804-04ce-4380-af18-fcf81d528500\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a9b573-85d444bfdf-zzqlp" Oct 31 04:24:39.476738 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:39.476701 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a9b573-85d444bfdf-zzqlp" Oct 31 04:24:39.507111 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:39.507079 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a9b573-6c9676dddc-r5kx6"] Oct 31 04:24:39.510269 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:24:39.510220 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81895138_f13d_4457_894b_88439c752982.slice/crio-25bdc897f68b5dde62526307e74fa2147bd3d03f2e5d32a52b856146f7fe0c7a WatchSource:0}: Error finding container 25bdc897f68b5dde62526307e74fa2147bd3d03f2e5d32a52b856146f7fe0c7a: Status 404 returned error can't find the container with id 25bdc897f68b5dde62526307e74fa2147bd3d03f2e5d32a52b856146f7fe0c7a Oct 31 04:24:39.818897 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:24:39.818855 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6ce1804_04ce_4380_af18_fcf81d528500.slice/crio-521cfca8f00b79c3a815690f50fc40c6528891499cb0f18e80ed6c90485592ad WatchSource:0}: Error finding container 521cfca8f00b79c3a815690f50fc40c6528891499cb0f18e80ed6c90485592ad: Status 404 returned error can't find the container with id 521cfca8f00b79c3a815690f50fc40c6528891499cb0f18e80ed6c90485592ad Oct 31 04:24:39.819987 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:39.819957 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a9b573-85d444bfdf-zzqlp"] Oct 31 04:24:40.470812 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:40.470762 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9b573-6c9676dddc-r5kx6" event={"ID":"81895138-f13d-4457-894b-88439c752982","Type":"ContainerStarted","Data":"b6661bf0c0be6b80307048a472fa8201913d230eebe2d841615f52eb56a1711b"} Oct 31 04:24:40.471043 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:40.470839 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9b573-6c9676dddc-r5kx6" event={"ID":"81895138-f13d-4457-894b-88439c752982","Type":"ContainerStarted","Data":"25bdc897f68b5dde62526307e74fa2147bd3d03f2e5d32a52b856146f7fe0c7a"} Oct 31 04:24:40.471043 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:40.470868 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9b573-6c9676dddc-r5kx6" Oct 31 04:24:40.473792 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:40.473763 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a9b573-85d444bfdf-zzqlp" event={"ID":"c6ce1804-04ce-4380-af18-fcf81d528500","Type":"ContainerStarted","Data":"4533f660cc24b2d16da871d49b50f0bb0e8adf0383b9b3bea8d41321755cd8db"} Oct 31 04:24:40.473967 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:40.473807 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a9b573-85d444bfdf-zzqlp" event={"ID":"c6ce1804-04ce-4380-af18-fcf81d528500","Type":"ContainerStarted","Data":"521cfca8f00b79c3a815690f50fc40c6528891499cb0f18e80ed6c90485592ad"} Oct 31 04:24:40.473967 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:40.473902 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a9b573-85d444bfdf-zzqlp" Oct 31 04:24:40.514305 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:40.514245 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9b573-6c9676dddc-r5kx6" podStartSLOduration=2.514228245 podStartE2EDuration="2.514228245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:24:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:24:40.51143857 +0000 UTC m=+1235.267776606" watchObservedRunningTime="2026-10-31 04:24:40.514228245 +0000 UTC m=+1235.270566284" Oct 31 04:24:40.546633 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:40.546594 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9154a6-6cb7bfddbf-mcqng"] Oct 31 04:24:40.551611 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:40.551581 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9154a6-6cb7bfddbf-mcqng" Oct 31 04:24:40.557813 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:40.557762 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a9b573-85d444bfdf-zzqlp" podStartSLOduration=2.5577460739999998 podStartE2EDuration="2.557746074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:24:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:24:40.551290423 +0000 UTC m=+1235.307628463" watchObservedRunningTime="2026-10-31 04:24:40.557746074 +0000 UTC m=+1235.314084113" Oct 31 04:24:40.570214 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:40.570173 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9154a6-6cb7bfddbf-mcqng"] Oct 31 04:24:40.570507 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:40.570432 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-9154a6\"" Oct 31 04:24:40.704357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:40.703748 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8476795a-d573-42e9-a42f-19e307af2592-plugins-config-volume\") pod \"payload-processing-e2e-api-b-9154a6-6cb7bfddbf-mcqng\" (UID: \"8476795a-d573-42e9-a42f-19e307af2592\") " pod="openshift-ingress/payload-processing-e2e-api-b-9154a6-6cb7bfddbf-mcqng" Oct 31 04:24:40.704357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:40.703806 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lssqf\" (UniqueName: \"kubernetes.io/projected/8476795a-d573-42e9-a42f-19e307af2592-kube-api-access-lssqf\") pod \"payload-processing-e2e-api-b-9154a6-6cb7bfddbf-mcqng\" (UID: \"8476795a-d573-42e9-a42f-19e307af2592\") " pod="openshift-ingress/payload-processing-e2e-api-b-9154a6-6cb7bfddbf-mcqng" Oct 31 04:24:40.807406 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:40.806378 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8476795a-d573-42e9-a42f-19e307af2592-plugins-config-volume\") pod \"payload-processing-e2e-api-b-9154a6-6cb7bfddbf-mcqng\" (UID: \"8476795a-d573-42e9-a42f-19e307af2592\") " pod="openshift-ingress/payload-processing-e2e-api-b-9154a6-6cb7bfddbf-mcqng" Oct 31 04:24:40.807406 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:40.806436 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lssqf\" (UniqueName: \"kubernetes.io/projected/8476795a-d573-42e9-a42f-19e307af2592-kube-api-access-lssqf\") pod \"payload-processing-e2e-api-b-9154a6-6cb7bfddbf-mcqng\" (UID: \"8476795a-d573-42e9-a42f-19e307af2592\") " pod="openshift-ingress/payload-processing-e2e-api-b-9154a6-6cb7bfddbf-mcqng" Oct 31 04:24:40.811360 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:40.808033 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8476795a-d573-42e9-a42f-19e307af2592-plugins-config-volume\") pod \"payload-processing-e2e-api-b-9154a6-6cb7bfddbf-mcqng\" (UID: \"8476795a-d573-42e9-a42f-19e307af2592\") " pod="openshift-ingress/payload-processing-e2e-api-b-9154a6-6cb7bfddbf-mcqng" Oct 31 04:24:40.851239 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:40.851149 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lssqf\" (UniqueName: \"kubernetes.io/projected/8476795a-d573-42e9-a42f-19e307af2592-kube-api-access-lssqf\") pod \"payload-processing-e2e-api-b-9154a6-6cb7bfddbf-mcqng\" (UID: \"8476795a-d573-42e9-a42f-19e307af2592\") " pod="openshift-ingress/payload-processing-e2e-api-b-9154a6-6cb7bfddbf-mcqng" Oct 31 04:24:40.875415 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:40.875107 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9154a6-6cb7bfddbf-mcqng" Oct 31 04:24:41.056367 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:41.056304 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9ee9cc0-gjdnb"] Oct 31 04:24:41.063464 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:41.063079 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9ee9cc0-gjdnb"] Oct 31 04:24:41.211902 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:24:41.211867 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8476795a_d573_42e9_a42f_19e307af2592.slice/crio-b55aaf434730518209d4da5897436696d09a77d206e2765dcefe05424421d361 WatchSource:0}: Error finding container b55aaf434730518209d4da5897436696d09a77d206e2765dcefe05424421d361: Status 404 returned error can't find the container with id b55aaf434730518209d4da5897436696d09a77d206e2765dcefe05424421d361 Oct 31 04:24:41.213934 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:41.213896 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9154a6-6cb7bfddbf-mcqng"] Oct 31 04:24:41.411484 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:41.411396 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c76b84e1-d1e0-4a8d-bcbe-9fe1f6792097" path="/var/lib/kubelet/pods/c76b84e1-d1e0-4a8d-bcbe-9fe1f6792097/volumes" Oct 31 04:24:41.485192 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:41.485145 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9154a6-6cb7bfddbf-mcqng" event={"ID":"8476795a-d573-42e9-a42f-19e307af2592","Type":"ContainerStarted","Data":"fed92357375992113c03cd6ca3ebb0ca93afb35afe7ca07a4ef6ef78553ea595"} Oct 31 04:24:41.485910 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:41.485223 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-9154a6-6cb7bfddbf-mcqng" Oct 31 04:24:41.485910 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:41.485247 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9154a6-6cb7bfddbf-mcqng" event={"ID":"8476795a-d573-42e9-a42f-19e307af2592","Type":"ContainerStarted","Data":"b55aaf434730518209d4da5897436696d09a77d206e2765dcefe05424421d361"} Oct 31 04:24:41.526859 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:41.526732 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-9154a6-6cb7bfddbf-mcqng" podStartSLOduration=1.526717016 podStartE2EDuration="1.526717016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:24:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:24:41.525547517 +0000 UTC m=+1236.281885558" watchObservedRunningTime="2026-10-31 04:24:41.526717016 +0000 UTC m=+1236.283055091" Oct 31 04:24:46.031247 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:46.031204 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bdf0224-qlllb"] Oct 31 04:24:46.042491 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:46.042455 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bdf0224-qlllb"] Oct 31 04:24:47.031570 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:47.031525 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8809a9e-jsfw5"] Oct 31 04:24:47.043449 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:47.043413 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8809a9e-jsfw5"] Oct 31 04:24:47.412045 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:47.411954 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38a952e7-44a3-4c01-85b6-ea60d9db1984" path="/var/lib/kubelet/pods/38a952e7-44a3-4c01-85b6-ea60d9db1984/volumes" Oct 31 04:24:47.412685 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:47.412659 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="939530ef-60ca-4099-b12e-f5c922c564d7" path="/var/lib/kubelet/pods/939530ef-60ca-4099-b12e-f5c922c564d7/volumes" Oct 31 04:24:51.489660 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:51.489629 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a9b573-85d444bfdf-zzqlp" Oct 31 04:24:51.490116 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:51.489700 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9b573-6c9676dddc-r5kx6" Oct 31 04:24:52.394256 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:24:52.394217 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:24:52.499646 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:52.499616 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-9154a6-6cb7bfddbf-mcqng" Oct 31 04:24:53.034533 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:53.034478 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20fdab3e-2nmbn"] Oct 31 04:24:53.042002 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:53.041955 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20fdab3e-2nmbn"] Oct 31 04:24:53.409712 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:53.409619 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dfba3a7-1ce8-41a3-9367-534e5916b92b" path="/var/lib/kubelet/pods/2dfba3a7-1ce8-41a3-9367-534e5916b92b/volumes" Oct 31 04:24:54.034481 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:54.034433 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eda0d787-wc2p8"] Oct 31 04:24:54.044848 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:54.044805 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eda0d787-wc2p8"] Oct 31 04:24:54.290033 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:54.289943 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr"] Oct 31 04:24:54.290271 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:54.290239 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr" podUID="59824c73-334b-446b-8d63-bde791f5b216" containerName="istio-proxy" containerID="cri-o://b73d89ec659705cb66106ccecfac22ce040657b0a88b64f2f101b8b04d0d32d9" gracePeriod=30 Oct 31 04:24:55.411166 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:55.411128 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2d5db3b-8540-4881-acf7-6316cccfada4" path="/var/lib/kubelet/pods/e2d5db3b-8540-4881-acf7-6316cccfada4/volumes" Oct 31 04:24:56.033765 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:56.033719 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2b7955f-jz569"] Oct 31 04:24:56.042405 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:56.042366 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2b7955f-jz569"] Oct 31 04:24:57.411245 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:57.411206 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1791edb-3151-4dde-9263-1d628e06b247" path="/var/lib/kubelet/pods/d1791edb-3151-4dde-9263-1d628e06b247/volumes" Oct 31 04:24:59.032950 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:59.032902 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85ff0b4b-rmrth"] Oct 31 04:24:59.045062 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:59.045022 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85ff0b4b-rmrth"] Oct 31 04:24:59.411682 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:59.411657 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c81aa0f9-82a9-41db-b185-b6fc44b774a4" path="/var/lib/kubelet/pods/c81aa0f9-82a9-41db-b185-b6fc44b774a4/volumes" Oct 31 04:24:59.606453 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:59.606416 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr" Oct 31 04:24:59.636579 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:59.636546 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59824c73-334b-446b-8d63-bde791f5b216-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59824c73-334b-446b-8d63-bde791f5b216-workload-certs\") pod \"59824c73-334b-446b-8d63-bde791f5b216\" (UID: \"59824c73-334b-446b-8d63-bde791f5b216\") " Oct 31 04:24:59.636772 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:59.636632 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59824c73-334b-446b-8d63-bde791f5b216-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59824c73-334b-446b-8d63-bde791f5b216-credential-socket\") pod \"59824c73-334b-446b-8d63-bde791f5b216\" (UID: \"59824c73-334b-446b-8d63-bde791f5b216\") " Oct 31 04:24:59.636772 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:59.636668 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59824c73-334b-446b-8d63-bde791f5b216-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59824c73-334b-446b-8d63-bde791f5b216-istio-data\") pod \"59824c73-334b-446b-8d63-bde791f5b216\" (UID: \"59824c73-334b-446b-8d63-bde791f5b216\") " Oct 31 04:24:59.636772 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:59.636727 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59824c73-334b-446b-8d63-bde791f5b216-kube-api-access-jjpzm\" (UniqueName: \"kubernetes.io/projected/59824c73-334b-446b-8d63-bde791f5b216-kube-api-access-jjpzm\") pod \"59824c73-334b-446b-8d63-bde791f5b216\" (UID: \"59824c73-334b-446b-8d63-bde791f5b216\") " Oct 31 04:24:59.636942 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:59.636802 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59824c73-334b-446b-8d63-bde791f5b216-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59824c73-334b-446b-8d63-bde791f5b216-istio-podinfo\") pod \"59824c73-334b-446b-8d63-bde791f5b216\" (UID: \"59824c73-334b-446b-8d63-bde791f5b216\") " Oct 31 04:24:59.636942 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:59.636827 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59824c73-334b-446b-8d63-bde791f5b216-istio-token\" (UniqueName: \"kubernetes.io/projected/59824c73-334b-446b-8d63-bde791f5b216-istio-token\") pod \"59824c73-334b-446b-8d63-bde791f5b216\" (UID: \"59824c73-334b-446b-8d63-bde791f5b216\") " Oct 31 04:24:59.636942 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:59.636833 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59824c73-334b-446b-8d63-bde791f5b216-workload-certs" pod "59824c73-334b-446b-8d63-bde791f5b216" (UID: "59824c73-334b-446b-8d63-bde791f5b216"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:24:59.637118 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:59.636931 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59824c73-334b-446b-8d63-bde791f5b216-credential-socket" pod "59824c73-334b-446b-8d63-bde791f5b216" (UID: "59824c73-334b-446b-8d63-bde791f5b216"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:24:59.637118 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:59.636998 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59824c73-334b-446b-8d63-bde791f5b216-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59824c73-334b-446b-8d63-bde791f5b216-istio-envoy\") pod \"59824c73-334b-446b-8d63-bde791f5b216\" (UID: \"59824c73-334b-446b-8d63-bde791f5b216\") " Oct 31 04:24:59.637118 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:59.637034 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59824c73-334b-446b-8d63-bde791f5b216-istio-data" pod "59824c73-334b-446b-8d63-bde791f5b216" (UID: "59824c73-334b-446b-8d63-bde791f5b216"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:24:59.637118 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:59.637111 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59824c73-334b-446b-8d63-bde791f5b216-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59824c73-334b-446b-8d63-bde791f5b216-istiod-ca-cert\") pod \"59824c73-334b-446b-8d63-bde791f5b216\" (UID: \"59824c73-334b-446b-8d63-bde791f5b216\") " Oct 31 04:24:59.637313 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:59.637245 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59824c73-334b-446b-8d63-bde791f5b216-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59824c73-334b-446b-8d63-bde791f5b216-workload-socket\") pod \"59824c73-334b-446b-8d63-bde791f5b216\" (UID: \"59824c73-334b-446b-8d63-bde791f5b216\") " Oct 31 04:24:59.637578 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:59.637554 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59824c73-334b-446b-8d63-bde791f5b216-workload-socket" pod "59824c73-334b-446b-8d63-bde791f5b216" (UID: "59824c73-334b-446b-8d63-bde791f5b216"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:24:59.637663 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:59.637603 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59824c73-334b-446b-8d63-bde791f5b216-istiod-ca-cert" pod "59824c73-334b-446b-8d63-bde791f5b216" (UID: "59824c73-334b-446b-8d63-bde791f5b216"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:24:59.638064 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:59.638045 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59824c73-334b-446b-8d63-bde791f5b216-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:24:59.638164 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:59.638069 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59824c73-334b-446b-8d63-bde791f5b216-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:24:59.638164 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:59.638083 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59824c73-334b-446b-8d63-bde791f5b216-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:24:59.638164 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:59.638096 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59824c73-334b-446b-8d63-bde791f5b216-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:24:59.638164 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:59.638108 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59824c73-334b-446b-8d63-bde791f5b216-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:24:59.639407 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:59.639325 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59824c73-334b-446b-8d63-bde791f5b216-kube-api-access-jjpzm" pod "59824c73-334b-446b-8d63-bde791f5b216" (UID: "59824c73-334b-446b-8d63-bde791f5b216"). InnerVolumeSpecName "kube-api-access-jjpzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:24:59.639583 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:59.639518 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59824c73-334b-446b-8d63-bde791f5b216-istio-podinfo" pod "59824c73-334b-446b-8d63-bde791f5b216" (UID: "59824c73-334b-446b-8d63-bde791f5b216"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:24:59.639697 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:59.639642 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59824c73-334b-446b-8d63-bde791f5b216-istio-envoy" pod "59824c73-334b-446b-8d63-bde791f5b216" (UID: "59824c73-334b-446b-8d63-bde791f5b216"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:24:59.640003 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:59.639981 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59824c73-334b-446b-8d63-bde791f5b216-istio-token" pod "59824c73-334b-446b-8d63-bde791f5b216" (UID: "59824c73-334b-446b-8d63-bde791f5b216"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:24:59.739663 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:59.739623 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59824c73-334b-446b-8d63-bde791f5b216-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:24:59.739663 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:59.739662 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59824c73-334b-446b-8d63-bde791f5b216-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:24:59.739663 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:59.739672 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59824c73-334b-446b-8d63-bde791f5b216-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:24:59.739918 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:59.739680 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjpzm\" (UniqueName: \"kubernetes.io/projected/59824c73-334b-446b-8d63-bde791f5b216-kube-api-access-jjpzm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:24:59.851349 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:59.851247 5019 generic.go:356] "Generic (PLEG): container finished" podID="59824c73-334b-446b-8d63-bde791f5b216" containerID="b73d89ec659705cb66106ccecfac22ce040657b0a88b64f2f101b8b04d0d32d9" exitCode=0 Oct 31 04:24:59.851349 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:59.851305 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr" Oct 31 04:24:59.851349 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:59.851323 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr" event={"ID":"59824c73-334b-446b-8d63-bde791f5b216","Type":"ContainerDied","Data":"b73d89ec659705cb66106ccecfac22ce040657b0a88b64f2f101b8b04d0d32d9"} Oct 31 04:24:59.851632 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:59.851379 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr" event={"ID":"59824c73-334b-446b-8d63-bde791f5b216","Type":"ContainerDied","Data":"df60cd55452824403a1f64d32fca59a0696b7cbf1ac856a4a8473218b613bbae"} Oct 31 04:24:59.851632 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:59.851403 5019 scope.go:122] "RemoveContainer" containerID="b73d89ec659705cb66106ccecfac22ce040657b0a88b64f2f101b8b04d0d32d9" Oct 31 04:24:59.864842 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:59.864816 5019 scope.go:122] "RemoveContainer" containerID="e6f0da0c29aaf18aa4bb1379e308be5f3d0f10dceedf00cd8dd7c5a4d767dba4" Oct 31 04:24:59.879855 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:59.879825 5019 scope.go:122] "RemoveContainer" containerID="b73d89ec659705cb66106ccecfac22ce040657b0a88b64f2f101b8b04d0d32d9" Oct 31 04:24:59.880282 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:24:59.880256 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b73d89ec659705cb66106ccecfac22ce040657b0a88b64f2f101b8b04d0d32d9\": container with ID starting with b73d89ec659705cb66106ccecfac22ce040657b0a88b64f2f101b8b04d0d32d9 not found: ID does not exist" containerID="b73d89ec659705cb66106ccecfac22ce040657b0a88b64f2f101b8b04d0d32d9" Oct 31 04:24:59.880416 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:59.880296 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b73d89ec659705cb66106ccecfac22ce040657b0a88b64f2f101b8b04d0d32d9"} err="failed to get container status \"b73d89ec659705cb66106ccecfac22ce040657b0a88b64f2f101b8b04d0d32d9\": rpc error: code = NotFound desc = could not find container \"b73d89ec659705cb66106ccecfac22ce040657b0a88b64f2f101b8b04d0d32d9\": container with ID starting with b73d89ec659705cb66106ccecfac22ce040657b0a88b64f2f101b8b04d0d32d9 not found: ID does not exist" Oct 31 04:24:59.880416 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:59.880315 5019 scope.go:122] "RemoveContainer" containerID="e6f0da0c29aaf18aa4bb1379e308be5f3d0f10dceedf00cd8dd7c5a4d767dba4" Oct 31 04:24:59.880645 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:24:59.880621 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6f0da0c29aaf18aa4bb1379e308be5f3d0f10dceedf00cd8dd7c5a4d767dba4\": container with ID starting with e6f0da0c29aaf18aa4bb1379e308be5f3d0f10dceedf00cd8dd7c5a4d767dba4 not found: ID does not exist" containerID="e6f0da0c29aaf18aa4bb1379e308be5f3d0f10dceedf00cd8dd7c5a4d767dba4" Oct 31 04:24:59.880725 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:59.880655 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6f0da0c29aaf18aa4bb1379e308be5f3d0f10dceedf00cd8dd7c5a4d767dba4"} err="failed to get container status \"e6f0da0c29aaf18aa4bb1379e308be5f3d0f10dceedf00cd8dd7c5a4d767dba4\": rpc error: code = NotFound desc = could not find container \"e6f0da0c29aaf18aa4bb1379e308be5f3d0f10dceedf00cd8dd7c5a4d767dba4\": container with ID starting with e6f0da0c29aaf18aa4bb1379e308be5f3d0f10dceedf00cd8dd7c5a4d767dba4 not found: ID does not exist" Oct 31 04:24:59.909787 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:59.909747 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr"] Oct 31 04:24:59.918122 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:24:59.918080 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-146320d5-openshift-default-75cbf47c85-zhzlr"] Oct 31 04:25:01.409165 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:01.409125 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59824c73-334b-446b-8d63-bde791f5b216" path="/var/lib/kubelet/pods/59824c73-334b-446b-8d63-bde791f5b216/volumes" Oct 31 04:25:06.038800 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:06.038761 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e173eac-7j8xh"] Oct 31 04:25:06.052239 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:06.052193 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e173eac-7j8xh"] Oct 31 04:25:06.504971 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:06.504931 5019 scope.go:122] "RemoveContainer" containerID="04cbfe43942620973d21e7cdd1c0e0bba095cf3b8df2c9caaa6d6a98f63cc71a" Oct 31 04:25:06.561046 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:06.560998 5019 scope.go:122] "RemoveContainer" containerID="c348f48234dc85b6a574036dfed847fc43be2fadb935ae94fa9c6a97b4344f81" Oct 31 04:25:06.592001 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:06.591972 5019 scope.go:122] "RemoveContainer" containerID="c2b15521cc87fd13e20ec688b4151a7b944e0acce2494bf97e1f1680959d9046" Oct 31 04:25:06.613275 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:06.612325 5019 scope.go:122] "RemoveContainer" containerID="e4d9a7262157688bd59f57b898ce3b513977aba993de2d5906ad6c1ed35070af" Oct 31 04:25:07.034315 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:07.034214 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30a74489-ncq47"] Oct 31 04:25:07.046685 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:07.046639 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30a74489-ncq47"] Oct 31 04:25:07.408996 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:07.408907 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83f22018-3412-4765-bce4-f273cdc41aa4" path="/var/lib/kubelet/pods/83f22018-3412-4765-bce4-f273cdc41aa4/volumes" Oct 31 04:25:07.409530 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:07.409507 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4fcefcb-a397-4a74-800d-efd33730069d" path="/var/lib/kubelet/pods/e4fcefcb-a397-4a74-800d-efd33730069d/volumes" Oct 31 04:25:13.031652 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:13.031613 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42765d06-6w9tv"] Oct 31 04:25:13.044861 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:13.042717 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42765d06-6w9tv"] Oct 31 04:25:13.413063 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:13.412963 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f871c0f-0aa9-488e-b247-2555e9c7854e" path="/var/lib/kubelet/pods/2f871c0f-0aa9-488e-b247-2555e9c7854e/volumes" Oct 31 04:25:15.903191 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:15.899476 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b8bb1e-6d9649764-p2b84"] Oct 31 04:25:15.903191 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:15.900579 5019 cpu_manager.go:414] "containerMap: removing container" podUID="59824c73-334b-446b-8d63-bde791f5b216" containerName="istio-proxy" Oct 31 04:25:15.903191 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:15.900603 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59824c73-334b-446b-8d63-bde791f5b216" containerName="istio-proxy" Oct 31 04:25:15.903191 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:15.900665 5019 cpu_manager.go:414] "containerMap: removing container" podUID="59824c73-334b-446b-8d63-bde791f5b216" containerName="istio-proxy" Oct 31 04:25:15.903191 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:15.900675 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59824c73-334b-446b-8d63-bde791f5b216" containerName="istio-proxy" Oct 31 04:25:15.903191 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:15.900865 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="59824c73-334b-446b-8d63-bde791f5b216" containerName="istio-proxy" Oct 31 04:25:15.903191 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:15.900886 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="59824c73-334b-446b-8d63-bde791f5b216" containerName="istio-proxy" Oct 31 04:25:15.964447 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:15.964203 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b8bb1e-6d9649764-p2b84" Oct 31 04:25:15.965988 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:15.965965 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b8bb1e-6d9649764-p2b84"] Oct 31 04:25:15.991702 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:15.991508 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b8bb1e\"" Oct 31 04:25:15.991702 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:15.991650 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b8bb1e-dockercfg-kzdg5\"" Oct 31 04:25:16.096670 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:16.096618 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mctqf\" (UniqueName: \"kubernetes.io/projected/abc129a9-efa9-45f0-960d-27a21a68f326-kube-api-access-mctqf\") pod \"payload-processing-e2e-ipp-b-b8bb1e-6d9649764-p2b84\" (UID: \"abc129a9-efa9-45f0-960d-27a21a68f326\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b8bb1e-6d9649764-p2b84" Oct 31 04:25:16.096841 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:16.096715 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abc129a9-efa9-45f0-960d-27a21a68f326-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b8bb1e-6d9649764-p2b84\" (UID: \"abc129a9-efa9-45f0-960d-27a21a68f326\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b8bb1e-6d9649764-p2b84" Oct 31 04:25:16.199229 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:16.199129 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mctqf\" (UniqueName: \"kubernetes.io/projected/abc129a9-efa9-45f0-960d-27a21a68f326-kube-api-access-mctqf\") pod \"payload-processing-e2e-ipp-b-b8bb1e-6d9649764-p2b84\" (UID: \"abc129a9-efa9-45f0-960d-27a21a68f326\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b8bb1e-6d9649764-p2b84" Oct 31 04:25:16.199229 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:16.199221 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abc129a9-efa9-45f0-960d-27a21a68f326-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b8bb1e-6d9649764-p2b84\" (UID: \"abc129a9-efa9-45f0-960d-27a21a68f326\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b8bb1e-6d9649764-p2b84" Oct 31 04:25:16.200273 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:16.200240 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abc129a9-efa9-45f0-960d-27a21a68f326-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b8bb1e-6d9649764-p2b84\" (UID: \"abc129a9-efa9-45f0-960d-27a21a68f326\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b8bb1e-6d9649764-p2b84" Oct 31 04:25:16.231826 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:16.231785 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mctqf\" (UniqueName: \"kubernetes.io/projected/abc129a9-efa9-45f0-960d-27a21a68f326-kube-api-access-mctqf\") pod \"payload-processing-e2e-ipp-b-b8bb1e-6d9649764-p2b84\" (UID: \"abc129a9-efa9-45f0-960d-27a21a68f326\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b8bb1e-6d9649764-p2b84" Oct 31 04:25:16.310380 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:16.310323 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b8bb1e-6d9649764-p2b84" Oct 31 04:25:16.639742 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:25:16.639706 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabc129a9_efa9_45f0_960d_27a21a68f326.slice/crio-dd8279091bb12e41a07660294c18f968b642ceaf22d111c598ba545b78f18e9b WatchSource:0}: Error finding container dd8279091bb12e41a07660294c18f968b642ceaf22d111c598ba545b78f18e9b: Status 404 returned error can't find the container with id dd8279091bb12e41a07660294c18f968b642ceaf22d111c598ba545b78f18e9b Oct 31 04:25:16.640728 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:16.640698 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b8bb1e-6d9649764-p2b84"] Oct 31 04:25:17.138127 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:17.138085 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b8bb1e-6d9649764-p2b84" event={"ID":"abc129a9-efa9-45f0-960d-27a21a68f326","Type":"ContainerStarted","Data":"9d647a62296dd426f25935cd623c8864f548804248b8e094cebbc4b80d0c8521"} Oct 31 04:25:17.140592 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:17.138154 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b8bb1e-6d9649764-p2b84" event={"ID":"abc129a9-efa9-45f0-960d-27a21a68f326","Type":"ContainerStarted","Data":"dd8279091bb12e41a07660294c18f968b642ceaf22d111c598ba545b78f18e9b"} Oct 31 04:25:17.140592 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:17.138323 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b8bb1e-6d9649764-p2b84" Oct 31 04:25:17.184050 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:17.183979 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-b8bb1e-6d9649764-p2b84" podStartSLOduration=2.183958489 podStartE2EDuration="2.183958489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:25:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:25:17.182434866 +0000 UTC m=+1271.938772905" watchObservedRunningTime="2026-10-31 04:25:17.183958489 +0000 UTC m=+1271.940296529" Oct 31 04:25:18.050805 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:18.049631 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4de11517-qf5pb"] Oct 31 04:25:18.056126 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:18.056082 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4de11517-qf5pb"] Oct 31 04:25:18.148735 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:18.148700 5019 generic.go:356] "Generic (PLEG): container finished" podID="0712577c-fb53-4025-9d1e-6c62e6bf38cb" containerID="c345d8d327628e6c8dde79bc244b066cd375bdec1301460167a7775f5ebf5822" exitCode=0 Oct 31 04:25:18.149144 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:18.148775 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp" event={"ID":"0712577c-fb53-4025-9d1e-6c62e6bf38cb","Type":"ContainerDied","Data":"c345d8d327628e6c8dde79bc244b066cd375bdec1301460167a7775f5ebf5822"} Oct 31 04:25:18.149372 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:18.149313 5019 scope.go:122] "RemoveContainer" containerID="c345d8d327628e6c8dde79bc244b066cd375bdec1301460167a7775f5ebf5822" Oct 31 04:25:18.150957 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:18.150927 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:25:18.151025 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:18.150995 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:25:18.151065 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:18.151030 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:25:19.160818 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:19.160780 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp" event={"ID":"0712577c-fb53-4025-9d1e-6c62e6bf38cb","Type":"ContainerStarted","Data":"0d2c1812809a4e09a55120bd6a6b28287556cb8153fa32ac151bdd9a9cb739d1"} Oct 31 04:25:19.410225 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:19.410188 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0a65aa5-25f8-4835-bcbd-88236c70410e" path="/var/lib/kubelet/pods/a0a65aa5-25f8-4835-bcbd-88236c70410e/volumes" Oct 31 04:25:20.006746 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:20.006703 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp" Oct 31 04:25:20.008466 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:20.008435 5019 patch_prober.go:29] interesting pod/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 31 04:25:20.008618 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:20.008484 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp" podUID="0712577c-fb53-4025-9d1e-6c62e6bf38cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 31 04:25:21.007394 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:21.007302 5019 patch_prober.go:29] interesting pod/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 31 04:25:21.007826 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:21.007405 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp" podUID="0712577c-fb53-4025-9d1e-6c62e6bf38cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 31 04:25:21.032153 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:21.032114 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-802ac966-5tzmm"] Oct 31 04:25:21.044004 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:21.043969 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-802ac966-5tzmm"] Oct 31 04:25:21.409751 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:21.409667 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="587549d7-b0e2-4f3d-a956-934ca87749f1" path="/var/lib/kubelet/pods/587549d7-b0e2-4f3d-a956-934ca87749f1/volumes" Oct 31 04:25:22.007652 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:22.007599 5019 patch_prober.go:29] interesting pod/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 31 04:25:22.008118 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:22.007670 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp" podUID="0712577c-fb53-4025-9d1e-6c62e6bf38cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 31 04:25:22.257064 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:22.253725 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp" Oct 31 04:25:22.432292 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:25:22.432199 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:25:23.007607 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:23.007568 5019 patch_prober.go:29] interesting pod/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 31 04:25:23.007825 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:23.007629 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp" podUID="0712577c-fb53-4025-9d1e-6c62e6bf38cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 31 04:25:24.007498 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:24.007457 5019 patch_prober.go:29] interesting pod/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 31 04:25:24.007710 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:24.007516 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp" podUID="0712577c-fb53-4025-9d1e-6c62e6bf38cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 31 04:25:25.012312 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:25.012279 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp" Oct 31 04:25:25.013700 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:25.013678 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp" Oct 31 04:25:28.151633 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:28.151592 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b8bb1e-6d9649764-p2b84" Oct 31 04:25:29.660196 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:29.660153 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp"] Oct 31 04:25:29.660712 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:29.660444 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp" podUID="db10a00c-b537-4c62-b9aa-b98dc5577720" containerName="istio-proxy" containerID="cri-o://c3787789dee95da24e43f92f2211b942ec68369808ab5c10673a61e7c2dfa06b" gracePeriod=30 Oct 31 04:25:29.673943 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:29.673893 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r"] Oct 31 04:25:29.674192 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:29.674160 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r" podUID="c8d33677-060d-418a-8806-84eeb0c17ecb" containerName="istio-proxy" containerID="cri-o://73465b304bdfbaa1a725a5f42d4c53f10db63b82c7a62b8cf5b914da40a10521" gracePeriod=30 Oct 31 04:25:29.678576 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:29.678547 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx"] Oct 31 04:25:29.678776 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:29.678749 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx" podUID="a9256bfe-8c85-477e-bcc9-9938f4c689ab" containerName="istio-proxy" containerID="cri-o://84c03351a7346208e097d915cb61ad8f2ad39cf138cbc607b6e18e154308f2bb" gracePeriod=30 Oct 31 04:25:29.686271 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:29.686214 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x"] Oct 31 04:25:29.686587 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:29.686549 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x" podUID="661aa7e3-37aa-4131-aa1d-10c81372dd2d" containerName="istio-proxy" containerID="cri-o://63ccf93e122b8400412e54e2ffc5e03f1e8f26c02a47748d031ac360d9d9d392" gracePeriod=30 Oct 31 04:25:35.074313 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.074099 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx" Oct 31 04:25:35.143445 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.139568 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp" Oct 31 04:25:35.180895 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.180863 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db10a00c-b537-4c62-b9aa-b98dc5577720-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db10a00c-b537-4c62-b9aa-b98dc5577720-workload-socket\") pod \"db10a00c-b537-4c62-b9aa-b98dc5577720\" (UID: \"db10a00c-b537-4c62-b9aa-b98dc5577720\") " Oct 31 04:25:35.180895 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.180907 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db10a00c-b537-4c62-b9aa-b98dc5577720-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db10a00c-b537-4c62-b9aa-b98dc5577720-credential-socket\") pod \"db10a00c-b537-4c62-b9aa-b98dc5577720\" (UID: \"db10a00c-b537-4c62-b9aa-b98dc5577720\") " Oct 31 04:25:35.181124 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.180952 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9256bfe-8c85-477e-bcc9-9938f4c689ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9256bfe-8c85-477e-bcc9-9938f4c689ab-workload-socket\") pod \"a9256bfe-8c85-477e-bcc9-9938f4c689ab\" (UID: \"a9256bfe-8c85-477e-bcc9-9938f4c689ab\") " Oct 31 04:25:35.181124 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.180979 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9256bfe-8c85-477e-bcc9-9938f4c689ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9256bfe-8c85-477e-bcc9-9938f4c689ab-istio-envoy\") pod \"a9256bfe-8c85-477e-bcc9-9938f4c689ab\" (UID: \"a9256bfe-8c85-477e-bcc9-9938f4c689ab\") " Oct 31 04:25:35.181124 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.181035 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db10a00c-b537-4c62-b9aa-b98dc5577720-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db10a00c-b537-4c62-b9aa-b98dc5577720-istio-data\") pod \"db10a00c-b537-4c62-b9aa-b98dc5577720\" (UID: \"db10a00c-b537-4c62-b9aa-b98dc5577720\") " Oct 31 04:25:35.181124 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.181072 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db10a00c-b537-4c62-b9aa-b98dc5577720-kube-api-access-44xck\" (UniqueName: \"kubernetes.io/projected/db10a00c-b537-4c62-b9aa-b98dc5577720-kube-api-access-44xck\") pod \"db10a00c-b537-4c62-b9aa-b98dc5577720\" (UID: \"db10a00c-b537-4c62-b9aa-b98dc5577720\") " Oct 31 04:25:35.181124 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.181089 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9256bfe-8c85-477e-bcc9-9938f4c689ab-istio-token\" (UniqueName: \"kubernetes.io/projected/a9256bfe-8c85-477e-bcc9-9938f4c689ab-istio-token\") pod \"a9256bfe-8c85-477e-bcc9-9938f4c689ab\" (UID: \"a9256bfe-8c85-477e-bcc9-9938f4c689ab\") " Oct 31 04:25:35.181411 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.181133 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9256bfe-8c85-477e-bcc9-9938f4c689ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9256bfe-8c85-477e-bcc9-9938f4c689ab-istio-podinfo\") pod \"a9256bfe-8c85-477e-bcc9-9938f4c689ab\" (UID: \"a9256bfe-8c85-477e-bcc9-9938f4c689ab\") " Oct 31 04:25:35.181411 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.181210 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db10a00c-b537-4c62-b9aa-b98dc5577720-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db10a00c-b537-4c62-b9aa-b98dc5577720-istio-podinfo\") pod \"db10a00c-b537-4c62-b9aa-b98dc5577720\" (UID: \"db10a00c-b537-4c62-b9aa-b98dc5577720\") " Oct 31 04:25:35.181411 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.181247 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db10a00c-b537-4c62-b9aa-b98dc5577720-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db10a00c-b537-4c62-b9aa-b98dc5577720-istio-envoy\") pod \"db10a00c-b537-4c62-b9aa-b98dc5577720\" (UID: \"db10a00c-b537-4c62-b9aa-b98dc5577720\") " Oct 31 04:25:35.181411 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.181282 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db10a00c-b537-4c62-b9aa-b98dc5577720-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db10a00c-b537-4c62-b9aa-b98dc5577720-workload-certs\") pod \"db10a00c-b537-4c62-b9aa-b98dc5577720\" (UID: \"db10a00c-b537-4c62-b9aa-b98dc5577720\") " Oct 31 04:25:35.181411 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.181316 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db10a00c-b537-4c62-b9aa-b98dc5577720-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db10a00c-b537-4c62-b9aa-b98dc5577720-istiod-ca-cert\") pod \"db10a00c-b537-4c62-b9aa-b98dc5577720\" (UID: \"db10a00c-b537-4c62-b9aa-b98dc5577720\") " Oct 31 04:25:35.181411 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.181356 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9256bfe-8c85-477e-bcc9-9938f4c689ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9256bfe-8c85-477e-bcc9-9938f4c689ab-istiod-ca-cert\") pod \"a9256bfe-8c85-477e-bcc9-9938f4c689ab\" (UID: \"a9256bfe-8c85-477e-bcc9-9938f4c689ab\") " Oct 31 04:25:35.181715 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.181421 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db10a00c-b537-4c62-b9aa-b98dc5577720-istio-token\" (UniqueName: \"kubernetes.io/projected/db10a00c-b537-4c62-b9aa-b98dc5577720-istio-token\") pod \"db10a00c-b537-4c62-b9aa-b98dc5577720\" (UID: \"db10a00c-b537-4c62-b9aa-b98dc5577720\") " Oct 31 04:25:35.181715 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.181475 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9256bfe-8c85-477e-bcc9-9938f4c689ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9256bfe-8c85-477e-bcc9-9938f4c689ab-istio-data\") pod \"a9256bfe-8c85-477e-bcc9-9938f4c689ab\" (UID: \"a9256bfe-8c85-477e-bcc9-9938f4c689ab\") " Oct 31 04:25:35.181715 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.181564 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9256bfe-8c85-477e-bcc9-9938f4c689ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9256bfe-8c85-477e-bcc9-9938f4c689ab-workload-certs\") pod \"a9256bfe-8c85-477e-bcc9-9938f4c689ab\" (UID: \"a9256bfe-8c85-477e-bcc9-9938f4c689ab\") " Oct 31 04:25:35.181715 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.181682 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9256bfe-8c85-477e-bcc9-9938f4c689ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9256bfe-8c85-477e-bcc9-9938f4c689ab-credential-socket\") pod \"a9256bfe-8c85-477e-bcc9-9938f4c689ab\" (UID: \"a9256bfe-8c85-477e-bcc9-9938f4c689ab\") " Oct 31 04:25:35.181999 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.181728 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9256bfe-8c85-477e-bcc9-9938f4c689ab-kube-api-access-txvzn\" (UniqueName: \"kubernetes.io/projected/a9256bfe-8c85-477e-bcc9-9938f4c689ab-kube-api-access-txvzn\") pod \"a9256bfe-8c85-477e-bcc9-9938f4c689ab\" (UID: \"a9256bfe-8c85-477e-bcc9-9938f4c689ab\") " Oct 31 04:25:35.185733 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.183325 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db10a00c-b537-4c62-b9aa-b98dc5577720-istio-data" pod "db10a00c-b537-4c62-b9aa-b98dc5577720" (UID: "db10a00c-b537-4c62-b9aa-b98dc5577720"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:25:35.185733 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.183657 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db10a00c-b537-4c62-b9aa-b98dc5577720-workload-certs" pod "db10a00c-b537-4c62-b9aa-b98dc5577720" (UID: "db10a00c-b537-4c62-b9aa-b98dc5577720"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:25:35.185733 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.183890 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9256bfe-8c85-477e-bcc9-9938f4c689ab-workload-socket" pod "a9256bfe-8c85-477e-bcc9-9938f4c689ab" (UID: "a9256bfe-8c85-477e-bcc9-9938f4c689ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:25:35.185733 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.184071 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9256bfe-8c85-477e-bcc9-9938f4c689ab-istio-token" pod "a9256bfe-8c85-477e-bcc9-9938f4c689ab" (UID: "a9256bfe-8c85-477e-bcc9-9938f4c689ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:25:35.185733 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.184191 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db10a00c-b537-4c62-b9aa-b98dc5577720-workload-socket" pod "db10a00c-b537-4c62-b9aa-b98dc5577720" (UID: "db10a00c-b537-4c62-b9aa-b98dc5577720"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:25:35.185733 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.184413 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db10a00c-b537-4c62-b9aa-b98dc5577720-credential-socket" pod "db10a00c-b537-4c62-b9aa-b98dc5577720" (UID: "db10a00c-b537-4c62-b9aa-b98dc5577720"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:25:35.185733 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.184730 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9256bfe-8c85-477e-bcc9-9938f4c689ab-istio-data" pod "a9256bfe-8c85-477e-bcc9-9938f4c689ab" (UID: "a9256bfe-8c85-477e-bcc9-9938f4c689ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:25:35.185733 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.185156 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9256bfe-8c85-477e-bcc9-9938f4c689ab-istiod-ca-cert" pod "a9256bfe-8c85-477e-bcc9-9938f4c689ab" (UID: "a9256bfe-8c85-477e-bcc9-9938f4c689ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:25:35.185733 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.185369 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9256bfe-8c85-477e-bcc9-9938f4c689ab-workload-certs" pod "a9256bfe-8c85-477e-bcc9-9938f4c689ab" (UID: "a9256bfe-8c85-477e-bcc9-9938f4c689ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:25:35.185733 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.185577 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9256bfe-8c85-477e-bcc9-9938f4c689ab-credential-socket" pod "a9256bfe-8c85-477e-bcc9-9938f4c689ab" (UID: "a9256bfe-8c85-477e-bcc9-9938f4c689ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:25:35.188729 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.188701 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db10a00c-b537-4c62-b9aa-b98dc5577720-istiod-ca-cert" pod "db10a00c-b537-4c62-b9aa-b98dc5577720" (UID: "db10a00c-b537-4c62-b9aa-b98dc5577720"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:25:35.190849 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.190809 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db10a00c-b537-4c62-b9aa-b98dc5577720-istio-token" pod "db10a00c-b537-4c62-b9aa-b98dc5577720" (UID: "db10a00c-b537-4c62-b9aa-b98dc5577720"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:25:35.204514 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.204458 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r" Oct 31 04:25:35.207187 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.207147 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db10a00c-b537-4c62-b9aa-b98dc5577720-istio-podinfo" pod "db10a00c-b537-4c62-b9aa-b98dc5577720" (UID: "db10a00c-b537-4c62-b9aa-b98dc5577720"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:25:35.208039 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.207988 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db10a00c-b537-4c62-b9aa-b98dc5577720-istio-envoy" pod "db10a00c-b537-4c62-b9aa-b98dc5577720" (UID: "db10a00c-b537-4c62-b9aa-b98dc5577720"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:25:35.208039 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.207997 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9256bfe-8c85-477e-bcc9-9938f4c689ab-istio-envoy" pod "a9256bfe-8c85-477e-bcc9-9938f4c689ab" (UID: "a9256bfe-8c85-477e-bcc9-9938f4c689ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:25:35.208221 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.208048 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9256bfe-8c85-477e-bcc9-9938f4c689ab-kube-api-access-txvzn" pod "a9256bfe-8c85-477e-bcc9-9938f4c689ab" (UID: "a9256bfe-8c85-477e-bcc9-9938f4c689ab"). InnerVolumeSpecName "kube-api-access-txvzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:25:35.208405 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.208303 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db10a00c-b537-4c62-b9aa-b98dc5577720-kube-api-access-44xck" pod "db10a00c-b537-4c62-b9aa-b98dc5577720" (UID: "db10a00c-b537-4c62-b9aa-b98dc5577720"). InnerVolumeSpecName "kube-api-access-44xck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:25:35.208745 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.208722 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9256bfe-8c85-477e-bcc9-9938f4c689ab-istio-podinfo" pod "a9256bfe-8c85-477e-bcc9-9938f4c689ab" (UID: "a9256bfe-8c85-477e-bcc9-9938f4c689ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:25:35.222248 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.222211 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x" Oct 31 04:25:35.284371 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.284309 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8d33677-060d-418a-8806-84eeb0c17ecb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8d33677-060d-418a-8806-84eeb0c17ecb-istiod-ca-cert\") pod \"c8d33677-060d-418a-8806-84eeb0c17ecb\" (UID: \"c8d33677-060d-418a-8806-84eeb0c17ecb\") " Oct 31 04:25:35.284580 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.284380 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/661aa7e3-37aa-4131-aa1d-10c81372dd2d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/661aa7e3-37aa-4131-aa1d-10c81372dd2d-istio-data\") pod \"661aa7e3-37aa-4131-aa1d-10c81372dd2d\" (UID: \"661aa7e3-37aa-4131-aa1d-10c81372dd2d\") " Oct 31 04:25:35.284580 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.284423 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8d33677-060d-418a-8806-84eeb0c17ecb-istio-token\" (UniqueName: \"kubernetes.io/projected/c8d33677-060d-418a-8806-84eeb0c17ecb-istio-token\") pod \"c8d33677-060d-418a-8806-84eeb0c17ecb\" (UID: \"c8d33677-060d-418a-8806-84eeb0c17ecb\") " Oct 31 04:25:35.284580 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.284472 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/661aa7e3-37aa-4131-aa1d-10c81372dd2d-kube-api-access-g4dmw\" (UniqueName: \"kubernetes.io/projected/661aa7e3-37aa-4131-aa1d-10c81372dd2d-kube-api-access-g4dmw\") pod \"661aa7e3-37aa-4131-aa1d-10c81372dd2d\" (UID: \"661aa7e3-37aa-4131-aa1d-10c81372dd2d\") " Oct 31 04:25:35.284580 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.284553 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/661aa7e3-37aa-4131-aa1d-10c81372dd2d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/661aa7e3-37aa-4131-aa1d-10c81372dd2d-credential-socket\") pod \"661aa7e3-37aa-4131-aa1d-10c81372dd2d\" (UID: \"661aa7e3-37aa-4131-aa1d-10c81372dd2d\") " Oct 31 04:25:35.284807 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.284632 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8d33677-060d-418a-8806-84eeb0c17ecb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8d33677-060d-418a-8806-84eeb0c17ecb-istio-data\") pod \"c8d33677-060d-418a-8806-84eeb0c17ecb\" (UID: \"c8d33677-060d-418a-8806-84eeb0c17ecb\") " Oct 31 04:25:35.284807 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.284670 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/661aa7e3-37aa-4131-aa1d-10c81372dd2d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/661aa7e3-37aa-4131-aa1d-10c81372dd2d-istio-podinfo\") pod \"661aa7e3-37aa-4131-aa1d-10c81372dd2d\" (UID: \"661aa7e3-37aa-4131-aa1d-10c81372dd2d\") " Oct 31 04:25:35.284807 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.284698 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/661aa7e3-37aa-4131-aa1d-10c81372dd2d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/661aa7e3-37aa-4131-aa1d-10c81372dd2d-istiod-ca-cert\") pod \"661aa7e3-37aa-4131-aa1d-10c81372dd2d\" (UID: \"661aa7e3-37aa-4131-aa1d-10c81372dd2d\") " Oct 31 04:25:35.284807 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.284729 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8d33677-060d-418a-8806-84eeb0c17ecb-kube-api-access-bxx9f\" (UniqueName: \"kubernetes.io/projected/c8d33677-060d-418a-8806-84eeb0c17ecb-kube-api-access-bxx9f\") pod \"c8d33677-060d-418a-8806-84eeb0c17ecb\" (UID: \"c8d33677-060d-418a-8806-84eeb0c17ecb\") " Oct 31 04:25:35.284807 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.284765 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/661aa7e3-37aa-4131-aa1d-10c81372dd2d-istio-data" pod "661aa7e3-37aa-4131-aa1d-10c81372dd2d" (UID: "661aa7e3-37aa-4131-aa1d-10c81372dd2d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:25:35.285064 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.284808 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8d33677-060d-418a-8806-84eeb0c17ecb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8d33677-060d-418a-8806-84eeb0c17ecb-istio-envoy\") pod \"c8d33677-060d-418a-8806-84eeb0c17ecb\" (UID: \"c8d33677-060d-418a-8806-84eeb0c17ecb\") " Oct 31 04:25:35.285064 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.284871 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8d33677-060d-418a-8806-84eeb0c17ecb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8d33677-060d-418a-8806-84eeb0c17ecb-workload-certs\") pod \"c8d33677-060d-418a-8806-84eeb0c17ecb\" (UID: \"c8d33677-060d-418a-8806-84eeb0c17ecb\") " Oct 31 04:25:35.285064 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.284903 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8d33677-060d-418a-8806-84eeb0c17ecb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8d33677-060d-418a-8806-84eeb0c17ecb-credential-socket\") pod \"c8d33677-060d-418a-8806-84eeb0c17ecb\" (UID: \"c8d33677-060d-418a-8806-84eeb0c17ecb\") " Oct 31 04:25:35.285064 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.284955 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/661aa7e3-37aa-4131-aa1d-10c81372dd2d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/661aa7e3-37aa-4131-aa1d-10c81372dd2d-istio-envoy\") pod \"661aa7e3-37aa-4131-aa1d-10c81372dd2d\" (UID: \"661aa7e3-37aa-4131-aa1d-10c81372dd2d\") " Oct 31 04:25:35.285064 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.285013 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/661aa7e3-37aa-4131-aa1d-10c81372dd2d-credential-socket" pod "661aa7e3-37aa-4131-aa1d-10c81372dd2d" (UID: "661aa7e3-37aa-4131-aa1d-10c81372dd2d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:25:35.285064 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.285052 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8d33677-060d-418a-8806-84eeb0c17ecb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8d33677-060d-418a-8806-84eeb0c17ecb-workload-socket\") pod \"c8d33677-060d-418a-8806-84eeb0c17ecb\" (UID: \"c8d33677-060d-418a-8806-84eeb0c17ecb\") " Oct 31 04:25:35.285376 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.285085 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/661aa7e3-37aa-4131-aa1d-10c81372dd2d-istio-token\" (UniqueName: \"kubernetes.io/projected/661aa7e3-37aa-4131-aa1d-10c81372dd2d-istio-token\") pod \"661aa7e3-37aa-4131-aa1d-10c81372dd2d\" (UID: \"661aa7e3-37aa-4131-aa1d-10c81372dd2d\") " Oct 31 04:25:35.285376 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.285137 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/661aa7e3-37aa-4131-aa1d-10c81372dd2d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/661aa7e3-37aa-4131-aa1d-10c81372dd2d-workload-certs\") pod \"661aa7e3-37aa-4131-aa1d-10c81372dd2d\" (UID: \"661aa7e3-37aa-4131-aa1d-10c81372dd2d\") " Oct 31 04:25:35.285376 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.285187 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c8d33677-060d-418a-8806-84eeb0c17ecb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8d33677-060d-418a-8806-84eeb0c17ecb-istio-podinfo\") pod \"c8d33677-060d-418a-8806-84eeb0c17ecb\" (UID: \"c8d33677-060d-418a-8806-84eeb0c17ecb\") " Oct 31 04:25:35.285376 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.285246 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/661aa7e3-37aa-4131-aa1d-10c81372dd2d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/661aa7e3-37aa-4131-aa1d-10c81372dd2d-workload-socket\") pod \"661aa7e3-37aa-4131-aa1d-10c81372dd2d\" (UID: \"661aa7e3-37aa-4131-aa1d-10c81372dd2d\") " Oct 31 04:25:35.285978 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.285950 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8d33677-060d-418a-8806-84eeb0c17ecb-istiod-ca-cert" pod "c8d33677-060d-418a-8806-84eeb0c17ecb" (UID: "c8d33677-060d-418a-8806-84eeb0c17ecb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:25:35.286115 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.286096 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db10a00c-b537-4c62-b9aa-b98dc5577720-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:25:35.286198 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.286181 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/661aa7e3-37aa-4131-aa1d-10c81372dd2d-workload-socket" pod "661aa7e3-37aa-4131-aa1d-10c81372dd2d" (UID: "661aa7e3-37aa-4131-aa1d-10c81372dd2d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:25:35.286258 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.286207 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9256bfe-8c85-477e-bcc9-9938f4c689ab-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:25:35.286258 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.286225 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9256bfe-8c85-477e-bcc9-9938f4c689ab-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:25:35.286258 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.286240 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9256bfe-8c85-477e-bcc9-9938f4c689ab-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:25:35.286435 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.286259 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txvzn\" (UniqueName: \"kubernetes.io/projected/a9256bfe-8c85-477e-bcc9-9938f4c689ab-kube-api-access-txvzn\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:25:35.286435 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.286274 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db10a00c-b537-4c62-b9aa-b98dc5577720-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:25:35.286435 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.286287 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db10a00c-b537-4c62-b9aa-b98dc5577720-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:25:35.286435 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.286302 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9256bfe-8c85-477e-bcc9-9938f4c689ab-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:25:35.286435 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.286315 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9256bfe-8c85-477e-bcc9-9938f4c689ab-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:25:35.286435 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.286327 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db10a00c-b537-4c62-b9aa-b98dc5577720-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:25:35.286435 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.286356 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44xck\" (UniqueName: \"kubernetes.io/projected/db10a00c-b537-4c62-b9aa-b98dc5577720-kube-api-access-44xck\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:25:35.286435 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.286369 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9256bfe-8c85-477e-bcc9-9938f4c689ab-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:25:35.286435 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.286376 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8d33677-060d-418a-8806-84eeb0c17ecb-istio-data" pod "c8d33677-060d-418a-8806-84eeb0c17ecb" (UID: "c8d33677-060d-418a-8806-84eeb0c17ecb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:25:35.286435 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.286411 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8d33677-060d-418a-8806-84eeb0c17ecb-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:25:35.286435 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.286426 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/661aa7e3-37aa-4131-aa1d-10c81372dd2d-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:25:35.286435 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.286440 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9256bfe-8c85-477e-bcc9-9938f4c689ab-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:25:35.287014 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.286453 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db10a00c-b537-4c62-b9aa-b98dc5577720-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:25:35.287014 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.286468 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/661aa7e3-37aa-4131-aa1d-10c81372dd2d-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:25:35.287014 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.286481 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db10a00c-b537-4c62-b9aa-b98dc5577720-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:25:35.287014 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.286493 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db10a00c-b537-4c62-b9aa-b98dc5577720-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:25:35.287014 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.286506 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db10a00c-b537-4c62-b9aa-b98dc5577720-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:25:35.287014 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.286518 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9256bfe-8c85-477e-bcc9-9938f4c689ab-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:25:35.287014 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.286954 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8d33677-060d-418a-8806-84eeb0c17ecb-credential-socket" pod "c8d33677-060d-418a-8806-84eeb0c17ecb" (UID: "c8d33677-060d-418a-8806-84eeb0c17ecb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:25:35.287363 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.287245 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/661aa7e3-37aa-4131-aa1d-10c81372dd2d-istiod-ca-cert" pod "661aa7e3-37aa-4131-aa1d-10c81372dd2d" (UID: "661aa7e3-37aa-4131-aa1d-10c81372dd2d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:25:35.288969 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.288115 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8d33677-060d-418a-8806-84eeb0c17ecb-istio-token" pod "c8d33677-060d-418a-8806-84eeb0c17ecb" (UID: "c8d33677-060d-418a-8806-84eeb0c17ecb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:25:35.288969 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.288294 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/661aa7e3-37aa-4131-aa1d-10c81372dd2d-workload-certs" pod "661aa7e3-37aa-4131-aa1d-10c81372dd2d" (UID: "661aa7e3-37aa-4131-aa1d-10c81372dd2d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:25:35.288969 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.288641 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8d33677-060d-418a-8806-84eeb0c17ecb-workload-socket" pod "c8d33677-060d-418a-8806-84eeb0c17ecb" (UID: "c8d33677-060d-418a-8806-84eeb0c17ecb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:25:35.288969 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.288861 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8d33677-060d-418a-8806-84eeb0c17ecb-workload-certs" pod "c8d33677-060d-418a-8806-84eeb0c17ecb" (UID: "c8d33677-060d-418a-8806-84eeb0c17ecb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:25:35.290635 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.290609 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8d33677-060d-418a-8806-84eeb0c17ecb-istio-podinfo" pod "c8d33677-060d-418a-8806-84eeb0c17ecb" (UID: "c8d33677-060d-418a-8806-84eeb0c17ecb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:25:35.291003 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.290971 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8d33677-060d-418a-8806-84eeb0c17ecb-istio-envoy" pod "c8d33677-060d-418a-8806-84eeb0c17ecb" (UID: "c8d33677-060d-418a-8806-84eeb0c17ecb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:25:35.291576 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.291550 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/661aa7e3-37aa-4131-aa1d-10c81372dd2d-istio-token" pod "661aa7e3-37aa-4131-aa1d-10c81372dd2d" (UID: "661aa7e3-37aa-4131-aa1d-10c81372dd2d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:25:35.292772 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.292750 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/661aa7e3-37aa-4131-aa1d-10c81372dd2d-istio-envoy" pod "661aa7e3-37aa-4131-aa1d-10c81372dd2d" (UID: "661aa7e3-37aa-4131-aa1d-10c81372dd2d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:25:35.296835 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.296801 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/661aa7e3-37aa-4131-aa1d-10c81372dd2d-istio-podinfo" pod "661aa7e3-37aa-4131-aa1d-10c81372dd2d" (UID: "661aa7e3-37aa-4131-aa1d-10c81372dd2d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:25:35.296928 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.296829 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/661aa7e3-37aa-4131-aa1d-10c81372dd2d-kube-api-access-g4dmw" pod "661aa7e3-37aa-4131-aa1d-10c81372dd2d" (UID: "661aa7e3-37aa-4131-aa1d-10c81372dd2d"). InnerVolumeSpecName "kube-api-access-g4dmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:25:35.298653 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.298629 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8d33677-060d-418a-8806-84eeb0c17ecb-kube-api-access-bxx9f" pod "c8d33677-060d-418a-8806-84eeb0c17ecb" (UID: "c8d33677-060d-418a-8806-84eeb0c17ecb"). InnerVolumeSpecName "kube-api-access-bxx9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:25:35.387925 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.387879 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8d33677-060d-418a-8806-84eeb0c17ecb-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:25:35.387925 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.387921 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8d33677-060d-418a-8806-84eeb0c17ecb-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:25:35.387925 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.387933 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8d33677-060d-418a-8806-84eeb0c17ecb-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:25:35.388215 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.387945 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/661aa7e3-37aa-4131-aa1d-10c81372dd2d-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:25:35.388215 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.387956 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8d33677-060d-418a-8806-84eeb0c17ecb-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:25:35.388215 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.387966 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/661aa7e3-37aa-4131-aa1d-10c81372dd2d-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:25:35.388215 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.387980 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/661aa7e3-37aa-4131-aa1d-10c81372dd2d-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:25:35.388215 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.387990 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8d33677-060d-418a-8806-84eeb0c17ecb-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:25:35.388215 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.388000 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/661aa7e3-37aa-4131-aa1d-10c81372dd2d-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:25:35.388215 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.388010 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8d33677-060d-418a-8806-84eeb0c17ecb-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:25:35.388215 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.388023 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4dmw\" (UniqueName: \"kubernetes.io/projected/661aa7e3-37aa-4131-aa1d-10c81372dd2d-kube-api-access-g4dmw\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:25:35.388215 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.388034 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8d33677-060d-418a-8806-84eeb0c17ecb-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:25:35.388215 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.388046 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/661aa7e3-37aa-4131-aa1d-10c81372dd2d-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:25:35.388215 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.388056 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/661aa7e3-37aa-4131-aa1d-10c81372dd2d-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:25:35.388215 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.388068 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxx9f\" (UniqueName: \"kubernetes.io/projected/c8d33677-060d-418a-8806-84eeb0c17ecb-kube-api-access-bxx9f\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:25:35.445574 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.445535 5019 generic.go:356] "Generic (PLEG): container finished" podID="c8d33677-060d-418a-8806-84eeb0c17ecb" containerID="73465b304bdfbaa1a725a5f42d4c53f10db63b82c7a62b8cf5b914da40a10521" exitCode=0 Oct 31 04:25:35.445757 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.445596 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r" Oct 31 04:25:35.445757 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.445626 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r" event={"ID":"c8d33677-060d-418a-8806-84eeb0c17ecb","Type":"ContainerDied","Data":"73465b304bdfbaa1a725a5f42d4c53f10db63b82c7a62b8cf5b914da40a10521"} Oct 31 04:25:35.445757 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.445692 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r" event={"ID":"c8d33677-060d-418a-8806-84eeb0c17ecb","Type":"ContainerDied","Data":"b6f8e5d498789a581fc87bb44136e2428377bf070635ef30eb614f66322f91f9"} Oct 31 04:25:35.445757 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.445729 5019 scope.go:122] "RemoveContainer" containerID="73465b304bdfbaa1a725a5f42d4c53f10db63b82c7a62b8cf5b914da40a10521" Oct 31 04:25:35.448486 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.448392 5019 generic.go:356] "Generic (PLEG): container finished" podID="a9256bfe-8c85-477e-bcc9-9938f4c689ab" containerID="84c03351a7346208e097d915cb61ad8f2ad39cf138cbc607b6e18e154308f2bb" exitCode=0 Oct 31 04:25:35.448654 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.448636 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx" Oct 31 04:25:35.448747 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.448722 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx" event={"ID":"a9256bfe-8c85-477e-bcc9-9938f4c689ab","Type":"ContainerDied","Data":"84c03351a7346208e097d915cb61ad8f2ad39cf138cbc607b6e18e154308f2bb"} Oct 31 04:25:35.448820 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.448782 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx" event={"ID":"a9256bfe-8c85-477e-bcc9-9938f4c689ab","Type":"ContainerDied","Data":"62872a8668c593cca5a3547af0b3c7fa4c891dfc555442cd74100f0af39e3b52"} Oct 31 04:25:35.450567 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.450530 5019 generic.go:356] "Generic (PLEG): container finished" podID="db10a00c-b537-4c62-b9aa-b98dc5577720" containerID="c3787789dee95da24e43f92f2211b942ec68369808ab5c10673a61e7c2dfa06b" exitCode=0 Oct 31 04:25:35.450697 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.450596 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp" Oct 31 04:25:35.450697 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.450611 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp" event={"ID":"db10a00c-b537-4c62-b9aa-b98dc5577720","Type":"ContainerDied","Data":"c3787789dee95da24e43f92f2211b942ec68369808ab5c10673a61e7c2dfa06b"} Oct 31 04:25:35.450697 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.450667 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp" event={"ID":"db10a00c-b537-4c62-b9aa-b98dc5577720","Type":"ContainerDied","Data":"a5b60e8f8c732799366f395e340d2d0ffbc1c8675b273c5c12680489650a3dbb"} Oct 31 04:25:35.452894 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.452868 5019 generic.go:356] "Generic (PLEG): container finished" podID="661aa7e3-37aa-4131-aa1d-10c81372dd2d" containerID="63ccf93e122b8400412e54e2ffc5e03f1e8f26c02a47748d031ac360d9d9d392" exitCode=0 Oct 31 04:25:35.452991 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.452914 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x" Oct 31 04:25:35.452991 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.452957 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x" event={"ID":"661aa7e3-37aa-4131-aa1d-10c81372dd2d","Type":"ContainerDied","Data":"63ccf93e122b8400412e54e2ffc5e03f1e8f26c02a47748d031ac360d9d9d392"} Oct 31 04:25:35.453124 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.453010 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x" event={"ID":"661aa7e3-37aa-4131-aa1d-10c81372dd2d","Type":"ContainerDied","Data":"a07527e0193ff21537650b5e13f6979c39accfe4b392ca32b7a9e09653fd7ac1"} Oct 31 04:25:35.462489 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.462465 5019 scope.go:122] "RemoveContainer" containerID="73465b304bdfbaa1a725a5f42d4c53f10db63b82c7a62b8cf5b914da40a10521" Oct 31 04:25:35.462905 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:25:35.462871 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73465b304bdfbaa1a725a5f42d4c53f10db63b82c7a62b8cf5b914da40a10521\": container with ID starting with 73465b304bdfbaa1a725a5f42d4c53f10db63b82c7a62b8cf5b914da40a10521 not found: ID does not exist" containerID="73465b304bdfbaa1a725a5f42d4c53f10db63b82c7a62b8cf5b914da40a10521" Oct 31 04:25:35.463011 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.462917 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73465b304bdfbaa1a725a5f42d4c53f10db63b82c7a62b8cf5b914da40a10521"} err="failed to get container status \"73465b304bdfbaa1a725a5f42d4c53f10db63b82c7a62b8cf5b914da40a10521\": rpc error: code = NotFound desc = could not find container \"73465b304bdfbaa1a725a5f42d4c53f10db63b82c7a62b8cf5b914da40a10521\": container with ID starting with 73465b304bdfbaa1a725a5f42d4c53f10db63b82c7a62b8cf5b914da40a10521 not found: ID does not exist" Oct 31 04:25:35.463011 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.462952 5019 scope.go:122] "RemoveContainer" containerID="84c03351a7346208e097d915cb61ad8f2ad39cf138cbc607b6e18e154308f2bb" Oct 31 04:25:35.475946 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.475919 5019 scope.go:122] "RemoveContainer" containerID="84c03351a7346208e097d915cb61ad8f2ad39cf138cbc607b6e18e154308f2bb" Oct 31 04:25:35.476287 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:25:35.476270 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84c03351a7346208e097d915cb61ad8f2ad39cf138cbc607b6e18e154308f2bb\": container with ID starting with 84c03351a7346208e097d915cb61ad8f2ad39cf138cbc607b6e18e154308f2bb not found: ID does not exist" containerID="84c03351a7346208e097d915cb61ad8f2ad39cf138cbc607b6e18e154308f2bb" Oct 31 04:25:35.476355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.476295 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84c03351a7346208e097d915cb61ad8f2ad39cf138cbc607b6e18e154308f2bb"} err="failed to get container status \"84c03351a7346208e097d915cb61ad8f2ad39cf138cbc607b6e18e154308f2bb\": rpc error: code = NotFound desc = could not find container \"84c03351a7346208e097d915cb61ad8f2ad39cf138cbc607b6e18e154308f2bb\": container with ID starting with 84c03351a7346208e097d915cb61ad8f2ad39cf138cbc607b6e18e154308f2bb not found: ID does not exist" Oct 31 04:25:35.476355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.476313 5019 scope.go:122] "RemoveContainer" containerID="c3787789dee95da24e43f92f2211b942ec68369808ab5c10673a61e7c2dfa06b" Oct 31 04:25:35.489384 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.489358 5019 scope.go:122] "RemoveContainer" containerID="c3787789dee95da24e43f92f2211b942ec68369808ab5c10673a61e7c2dfa06b" Oct 31 04:25:35.489858 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:25:35.489823 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3787789dee95da24e43f92f2211b942ec68369808ab5c10673a61e7c2dfa06b\": container with ID starting with c3787789dee95da24e43f92f2211b942ec68369808ab5c10673a61e7c2dfa06b not found: ID does not exist" containerID="c3787789dee95da24e43f92f2211b942ec68369808ab5c10673a61e7c2dfa06b" Oct 31 04:25:35.489969 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.489868 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3787789dee95da24e43f92f2211b942ec68369808ab5c10673a61e7c2dfa06b"} err="failed to get container status \"c3787789dee95da24e43f92f2211b942ec68369808ab5c10673a61e7c2dfa06b\": rpc error: code = NotFound desc = could not find container \"c3787789dee95da24e43f92f2211b942ec68369808ab5c10673a61e7c2dfa06b\": container with ID starting with c3787789dee95da24e43f92f2211b942ec68369808ab5c10673a61e7c2dfa06b not found: ID does not exist" Oct 31 04:25:35.489969 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.489894 5019 scope.go:122] "RemoveContainer" containerID="63ccf93e122b8400412e54e2ffc5e03f1e8f26c02a47748d031ac360d9d9d392" Oct 31 04:25:35.503300 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.503261 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x"] Oct 31 04:25:35.505063 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.505037 5019 scope.go:122] "RemoveContainer" containerID="63ccf93e122b8400412e54e2ffc5e03f1e8f26c02a47748d031ac360d9d9d392" Oct 31 04:25:35.505611 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:25:35.505579 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63ccf93e122b8400412e54e2ffc5e03f1e8f26c02a47748d031ac360d9d9d392\": container with ID starting with 63ccf93e122b8400412e54e2ffc5e03f1e8f26c02a47748d031ac360d9d9d392 not found: ID does not exist" containerID="63ccf93e122b8400412e54e2ffc5e03f1e8f26c02a47748d031ac360d9d9d392" Oct 31 04:25:35.505733 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.505620 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63ccf93e122b8400412e54e2ffc5e03f1e8f26c02a47748d031ac360d9d9d392"} err="failed to get container status \"63ccf93e122b8400412e54e2ffc5e03f1e8f26c02a47748d031ac360d9d9d392\": rpc error: code = NotFound desc = could not find container \"63ccf93e122b8400412e54e2ffc5e03f1e8f26c02a47748d031ac360d9d9d392\": container with ID starting with 63ccf93e122b8400412e54e2ffc5e03f1e8f26c02a47748d031ac360d9d9d392 not found: ID does not exist" Oct 31 04:25:35.510491 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.510458 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-xln8x"] Oct 31 04:25:35.575788 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.575757 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp"] Oct 31 04:25:35.613605 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.613566 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-p5dsp"] Oct 31 04:25:35.701701 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.701629 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r"] Oct 31 04:25:35.723155 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.723108 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-l9h6r"] Oct 31 04:25:35.757859 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.757820 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx"] Oct 31 04:25:35.763046 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:35.763017 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-rxktx"] Oct 31 04:25:37.410753 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:37.410715 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="661aa7e3-37aa-4131-aa1d-10c81372dd2d" path="/var/lib/kubelet/pods/661aa7e3-37aa-4131-aa1d-10c81372dd2d/volumes" Oct 31 04:25:37.411302 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:37.411288 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9256bfe-8c85-477e-bcc9-9938f4c689ab" path="/var/lib/kubelet/pods/a9256bfe-8c85-477e-bcc9-9938f4c689ab/volumes" Oct 31 04:25:37.411892 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:37.411875 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8d33677-060d-418a-8806-84eeb0c17ecb" path="/var/lib/kubelet/pods/c8d33677-060d-418a-8806-84eeb0c17ecb/volumes" Oct 31 04:25:37.412720 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:37.412703 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db10a00c-b537-4c62-b9aa-b98dc5577720" path="/var/lib/kubelet/pods/db10a00c-b537-4c62-b9aa-b98dc5577720/volumes" Oct 31 04:25:38.308308 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:38.308268 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a9b573-85d444bfdf-zzqlp"] Oct 31 04:25:38.338412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:38.338377 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a9b573-6c9676dddc-r5kx6"] Oct 31 04:25:43.309165 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:43.309112 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a9b573-85d444bfdf-zzqlp" podUID="c6ce1804-04ce-4380-af18-fcf81d528500" containerName="payload-processing" containerID="cri-o://4533f660cc24b2d16da871d49b50f0bb0e8adf0383b9b3bea8d41321755cd8db" gracePeriod=25 Oct 31 04:25:43.339065 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:43.339028 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9b573-6c9676dddc-r5kx6" podUID="81895138-f13d-4457-894b-88439c752982" containerName="payload-pre-processing" containerID="cri-o://b6661bf0c0be6b80307048a472fa8201913d230eebe2d841615f52eb56a1711b" gracePeriod=25 Oct 31 04:25:43.584407 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:43.582595 5019 generic.go:356] "Generic (PLEG): container finished" podID="c6ce1804-04ce-4380-af18-fcf81d528500" containerID="4533f660cc24b2d16da871d49b50f0bb0e8adf0383b9b3bea8d41321755cd8db" exitCode=0 Oct 31 04:25:43.584407 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:43.582853 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a9b573-85d444bfdf-zzqlp" event={"ID":"c6ce1804-04ce-4380-af18-fcf81d528500","Type":"ContainerDied","Data":"4533f660cc24b2d16da871d49b50f0bb0e8adf0383b9b3bea8d41321755cd8db"} Oct 31 04:25:43.586664 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:43.585240 5019 generic.go:356] "Generic (PLEG): container finished" podID="81895138-f13d-4457-894b-88439c752982" containerID="b6661bf0c0be6b80307048a472fa8201913d230eebe2d841615f52eb56a1711b" exitCode=0 Oct 31 04:25:43.586664 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:43.585320 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9b573-6c9676dddc-r5kx6" event={"ID":"81895138-f13d-4457-894b-88439c752982","Type":"ContainerDied","Data":"b6661bf0c0be6b80307048a472fa8201913d230eebe2d841615f52eb56a1711b"} Oct 31 04:25:43.677406 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:43.677367 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a9b573-85d444bfdf-zzqlp" Oct 31 04:25:43.766393 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:43.766312 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6ce1804-04ce-4380-af18-fcf81d528500-kube-api-access-9bhqh\" (UniqueName: \"kubernetes.io/projected/c6ce1804-04ce-4380-af18-fcf81d528500-kube-api-access-9bhqh\") pod \"c6ce1804-04ce-4380-af18-fcf81d528500\" (UID: \"c6ce1804-04ce-4380-af18-fcf81d528500\") " Oct 31 04:25:43.766568 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:43.766554 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6ce1804-04ce-4380-af18-fcf81d528500-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6ce1804-04ce-4380-af18-fcf81d528500-plugins-config-volume\") pod \"c6ce1804-04ce-4380-af18-fcf81d528500\" (UID: \"c6ce1804-04ce-4380-af18-fcf81d528500\") " Oct 31 04:25:43.767549 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:43.767508 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6ce1804-04ce-4380-af18-fcf81d528500-plugins-config-volume" pod "c6ce1804-04ce-4380-af18-fcf81d528500" (UID: "c6ce1804-04ce-4380-af18-fcf81d528500"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:25:43.770583 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:43.770543 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6ce1804-04ce-4380-af18-fcf81d528500-kube-api-access-9bhqh" pod "c6ce1804-04ce-4380-af18-fcf81d528500" (UID: "c6ce1804-04ce-4380-af18-fcf81d528500"). InnerVolumeSpecName "kube-api-access-9bhqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:25:43.797893 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:43.797866 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9b573-6c9676dddc-r5kx6" Oct 31 04:25:43.869079 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:43.868983 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6ce1804-04ce-4380-af18-fcf81d528500-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:25:43.869079 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:43.869024 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bhqh\" (UniqueName: \"kubernetes.io/projected/c6ce1804-04ce-4380-af18-fcf81d528500-kube-api-access-9bhqh\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:25:43.970459 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:43.970425 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81895138-f13d-4457-894b-88439c752982-kube-api-access-95l8j\" (UniqueName: \"kubernetes.io/projected/81895138-f13d-4457-894b-88439c752982-kube-api-access-95l8j\") pod \"81895138-f13d-4457-894b-88439c752982\" (UID: \"81895138-f13d-4457-894b-88439c752982\") " Oct 31 04:25:43.970644 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:43.970543 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81895138-f13d-4457-894b-88439c752982-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81895138-f13d-4457-894b-88439c752982-plugins-config-volume\") pod \"81895138-f13d-4457-894b-88439c752982\" (UID: \"81895138-f13d-4457-894b-88439c752982\") " Oct 31 04:25:43.971377 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:43.971019 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81895138-f13d-4457-894b-88439c752982-plugins-config-volume" pod "81895138-f13d-4457-894b-88439c752982" (UID: "81895138-f13d-4457-894b-88439c752982"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:25:43.971377 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:43.971312 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81895138-f13d-4457-894b-88439c752982-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:25:43.972735 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:43.972711 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81895138-f13d-4457-894b-88439c752982-kube-api-access-95l8j" pod "81895138-f13d-4457-894b-88439c752982" (UID: "81895138-f13d-4457-894b-88439c752982"). InnerVolumeSpecName "kube-api-access-95l8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:25:44.073129 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:44.073096 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95l8j\" (UniqueName: \"kubernetes.io/projected/81895138-f13d-4457-894b-88439c752982-kube-api-access-95l8j\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:25:44.603869 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:44.603781 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a9b573-85d444bfdf-zzqlp" Oct 31 04:25:44.604351 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:44.603787 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a9b573-85d444bfdf-zzqlp" event={"ID":"c6ce1804-04ce-4380-af18-fcf81d528500","Type":"ContainerDied","Data":"521cfca8f00b79c3a815690f50fc40c6528891499cb0f18e80ed6c90485592ad"} Oct 31 04:25:44.604351 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:44.603965 5019 scope.go:122] "RemoveContainer" containerID="4533f660cc24b2d16da871d49b50f0bb0e8adf0383b9b3bea8d41321755cd8db" Oct 31 04:25:44.605688 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:44.605667 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9b573-6c9676dddc-r5kx6" Oct 31 04:25:44.605917 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:44.605678 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9b573-6c9676dddc-r5kx6" event={"ID":"81895138-f13d-4457-894b-88439c752982","Type":"ContainerDied","Data":"25bdc897f68b5dde62526307e74fa2147bd3d03f2e5d32a52b856146f7fe0c7a"} Oct 31 04:25:44.627037 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:44.627009 5019 scope.go:122] "RemoveContainer" containerID="b6661bf0c0be6b80307048a472fa8201913d230eebe2d841615f52eb56a1711b" Oct 31 04:25:44.662786 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:44.662750 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a9b573-6c9676dddc-r5kx6"] Oct 31 04:25:44.675777 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:44.675735 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a9b573-6c9676dddc-r5kx6"] Oct 31 04:25:44.701237 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:44.701089 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a9b573-85d444bfdf-zzqlp"] Oct 31 04:25:44.705545 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:44.705509 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a9b573-85d444bfdf-zzqlp"] Oct 31 04:25:45.413116 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:45.413030 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81895138-f13d-4457-894b-88439c752982" path="/var/lib/kubelet/pods/81895138-f13d-4457-894b-88439c752982/volumes" Oct 31 04:25:45.413798 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:45.413776 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6ce1804-04ce-4380-af18-fcf81d528500" path="/var/lib/kubelet/pods/c6ce1804-04ce-4380-af18-fcf81d528500/volumes" Oct 31 04:25:46.631369 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:46.631299 5019 generic.go:356] "Generic (PLEG): container finished" podID="e1233843-bebe-44fc-82f5-6120e6302e4e" containerID="c84058468ab86da9b8eb32217d97cde841ed75135638e3572c2ace3947c8b862" exitCode=0 Oct 31 04:25:46.631854 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:46.631388 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f" event={"ID":"e1233843-bebe-44fc-82f5-6120e6302e4e","Type":"ContainerDied","Data":"c84058468ab86da9b8eb32217d97cde841ed75135638e3572c2ace3947c8b862"} Oct 31 04:25:46.632039 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:46.632011 5019 scope.go:122] "RemoveContainer" containerID="c84058468ab86da9b8eb32217d97cde841ed75135638e3572c2ace3947c8b862" Oct 31 04:25:46.634044 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:46.634013 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:25:46.634164 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:46.634107 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:25:46.634164 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:46.634158 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:25:47.652004 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:47.651966 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f" event={"ID":"e1233843-bebe-44fc-82f5-6120e6302e4e","Type":"ContainerStarted","Data":"b5e0abae72d16354a761904815875d5117551d3c1daf2a01d129c93062cac34c"} Oct 31 04:25:48.001351 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:48.001280 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f" Oct 31 04:25:48.002717 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:48.002690 5019 patch_prober.go:29] interesting pod/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 31 04:25:48.002860 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:48.002743 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f" podUID="e1233843-bebe-44fc-82f5-6120e6302e4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 31 04:25:49.001975 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:49.001928 5019 patch_prober.go:29] interesting pod/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 31 04:25:49.002539 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:49.001995 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f" podUID="e1233843-bebe-44fc-82f5-6120e6302e4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 31 04:25:50.001821 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:50.001776 5019 patch_prober.go:29] interesting pod/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 31 04:25:50.002132 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:50.002103 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f" podUID="e1233843-bebe-44fc-82f5-6120e6302e4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 31 04:25:50.418476 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:50.417605 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f" Oct 31 04:25:50.710399 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:50.710257 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj"] Oct 31 04:25:50.711325 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:50.711286 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj" podUID="1453873b-b35e-40f3-8c37-ad757223b110" containerName="istio-proxy" containerID="cri-o://73ff8e29c39a3ad1a4d270761712381e4fe379f5450ac2ffb2ac51f2a01169a5" gracePeriod=30 Oct 31 04:25:51.014357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:51.003823 5019 patch_prober.go:29] interesting pod/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 31 04:25:51.014357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:51.003901 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f" podUID="e1233843-bebe-44fc-82f5-6120e6302e4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 31 04:25:52.001950 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:52.001715 5019 patch_prober.go:29] interesting pod/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 31 04:25:52.001950 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:52.001776 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f" podUID="e1233843-bebe-44fc-82f5-6120e6302e4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 31 04:25:52.468054 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:25:52.468008 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:25:53.007860 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:53.007824 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f" Oct 31 04:25:53.009615 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:53.009588 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f" Oct 31 04:25:56.062245 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:56.062217 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj" Oct 31 04:25:56.172252 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:56.172215 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1453873b-b35e-40f3-8c37-ad757223b110-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1453873b-b35e-40f3-8c37-ad757223b110-istiod-ca-cert\") pod \"1453873b-b35e-40f3-8c37-ad757223b110\" (UID: \"1453873b-b35e-40f3-8c37-ad757223b110\") " Oct 31 04:25:56.172469 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:56.172295 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1453873b-b35e-40f3-8c37-ad757223b110-istio-token\" (UniqueName: \"kubernetes.io/projected/1453873b-b35e-40f3-8c37-ad757223b110-istio-token\") pod \"1453873b-b35e-40f3-8c37-ad757223b110\" (UID: \"1453873b-b35e-40f3-8c37-ad757223b110\") " Oct 31 04:25:56.172469 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:56.172404 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1453873b-b35e-40f3-8c37-ad757223b110-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1453873b-b35e-40f3-8c37-ad757223b110-istio-data\") pod \"1453873b-b35e-40f3-8c37-ad757223b110\" (UID: \"1453873b-b35e-40f3-8c37-ad757223b110\") " Oct 31 04:25:56.172598 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:56.172490 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1453873b-b35e-40f3-8c37-ad757223b110-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1453873b-b35e-40f3-8c37-ad757223b110-istio-podinfo\") pod \"1453873b-b35e-40f3-8c37-ad757223b110\" (UID: \"1453873b-b35e-40f3-8c37-ad757223b110\") " Oct 31 04:25:56.172598 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:56.172592 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1453873b-b35e-40f3-8c37-ad757223b110-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1453873b-b35e-40f3-8c37-ad757223b110-istio-envoy\") pod \"1453873b-b35e-40f3-8c37-ad757223b110\" (UID: \"1453873b-b35e-40f3-8c37-ad757223b110\") " Oct 31 04:25:56.172700 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:56.172614 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1453873b-b35e-40f3-8c37-ad757223b110-istiod-ca-cert" pod "1453873b-b35e-40f3-8c37-ad757223b110" (UID: "1453873b-b35e-40f3-8c37-ad757223b110"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:25:56.172757 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:56.172715 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1453873b-b35e-40f3-8c37-ad757223b110-istio-data" pod "1453873b-b35e-40f3-8c37-ad757223b110" (UID: "1453873b-b35e-40f3-8c37-ad757223b110"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:25:56.172858 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:56.172628 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1453873b-b35e-40f3-8c37-ad757223b110-kube-api-access-cccg5\" (UniqueName: \"kubernetes.io/projected/1453873b-b35e-40f3-8c37-ad757223b110-kube-api-access-cccg5\") pod \"1453873b-b35e-40f3-8c37-ad757223b110\" (UID: \"1453873b-b35e-40f3-8c37-ad757223b110\") " Oct 31 04:25:56.172972 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:56.172945 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1453873b-b35e-40f3-8c37-ad757223b110-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1453873b-b35e-40f3-8c37-ad757223b110-credential-socket\") pod \"1453873b-b35e-40f3-8c37-ad757223b110\" (UID: \"1453873b-b35e-40f3-8c37-ad757223b110\") " Oct 31 04:25:56.173078 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:56.172993 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1453873b-b35e-40f3-8c37-ad757223b110-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1453873b-b35e-40f3-8c37-ad757223b110-workload-certs\") pod \"1453873b-b35e-40f3-8c37-ad757223b110\" (UID: \"1453873b-b35e-40f3-8c37-ad757223b110\") " Oct 31 04:25:56.173145 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:56.173093 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1453873b-b35e-40f3-8c37-ad757223b110-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1453873b-b35e-40f3-8c37-ad757223b110-workload-socket\") pod \"1453873b-b35e-40f3-8c37-ad757223b110\" (UID: \"1453873b-b35e-40f3-8c37-ad757223b110\") " Oct 31 04:25:56.173456 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:56.173427 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1453873b-b35e-40f3-8c37-ad757223b110-credential-socket" pod "1453873b-b35e-40f3-8c37-ad757223b110" (UID: "1453873b-b35e-40f3-8c37-ad757223b110"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:25:56.173552 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:56.173436 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1453873b-b35e-40f3-8c37-ad757223b110-workload-certs" pod "1453873b-b35e-40f3-8c37-ad757223b110" (UID: "1453873b-b35e-40f3-8c37-ad757223b110"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:25:56.173668 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:56.173646 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1453873b-b35e-40f3-8c37-ad757223b110-workload-socket" pod "1453873b-b35e-40f3-8c37-ad757223b110" (UID: "1453873b-b35e-40f3-8c37-ad757223b110"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:25:56.173834 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:56.173816 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1453873b-b35e-40f3-8c37-ad757223b110-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:25:56.173919 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:56.173839 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1453873b-b35e-40f3-8c37-ad757223b110-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:25:56.173919 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:56.173849 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1453873b-b35e-40f3-8c37-ad757223b110-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:25:56.173919 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:56.173857 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1453873b-b35e-40f3-8c37-ad757223b110-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:25:56.173919 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:56.173867 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1453873b-b35e-40f3-8c37-ad757223b110-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:25:56.175044 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:56.175013 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1453873b-b35e-40f3-8c37-ad757223b110-istio-envoy" pod "1453873b-b35e-40f3-8c37-ad757223b110" (UID: "1453873b-b35e-40f3-8c37-ad757223b110"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:25:56.175138 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:56.175098 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1453873b-b35e-40f3-8c37-ad757223b110-istio-podinfo" pod "1453873b-b35e-40f3-8c37-ad757223b110" (UID: "1453873b-b35e-40f3-8c37-ad757223b110"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:25:56.175252 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:56.175233 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1453873b-b35e-40f3-8c37-ad757223b110-kube-api-access-cccg5" pod "1453873b-b35e-40f3-8c37-ad757223b110" (UID: "1453873b-b35e-40f3-8c37-ad757223b110"). InnerVolumeSpecName "kube-api-access-cccg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:25:56.176043 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:56.176025 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1453873b-b35e-40f3-8c37-ad757223b110-istio-token" pod "1453873b-b35e-40f3-8c37-ad757223b110" (UID: "1453873b-b35e-40f3-8c37-ad757223b110"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:25:56.283423 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:56.282413 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1453873b-b35e-40f3-8c37-ad757223b110-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:25:56.283423 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:56.282457 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1453873b-b35e-40f3-8c37-ad757223b110-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:25:56.283423 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:56.282472 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cccg5\" (UniqueName: \"kubernetes.io/projected/1453873b-b35e-40f3-8c37-ad757223b110-kube-api-access-cccg5\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:25:56.283423 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:56.282487 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1453873b-b35e-40f3-8c37-ad757223b110-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:25:56.789478 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:56.789443 5019 generic.go:356] "Generic (PLEG): container finished" podID="1453873b-b35e-40f3-8c37-ad757223b110" containerID="73ff8e29c39a3ad1a4d270761712381e4fe379f5450ac2ffb2ac51f2a01169a5" exitCode=0 Oct 31 04:25:56.789672 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:56.789516 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj" Oct 31 04:25:56.789672 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:56.789540 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj" event={"ID":"1453873b-b35e-40f3-8c37-ad757223b110","Type":"ContainerDied","Data":"73ff8e29c39a3ad1a4d270761712381e4fe379f5450ac2ffb2ac51f2a01169a5"} Oct 31 04:25:56.789672 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:56.789617 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj" event={"ID":"1453873b-b35e-40f3-8c37-ad757223b110","Type":"ContainerDied","Data":"2303578281a36de4f30b512f610f67effbdac550e719b1ebe96f3182fdb21c05"} Oct 31 04:25:56.789852 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:56.789653 5019 scope.go:122] "RemoveContainer" containerID="73ff8e29c39a3ad1a4d270761712381e4fe379f5450ac2ffb2ac51f2a01169a5" Oct 31 04:25:56.804406 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:56.804386 5019 scope.go:122] "RemoveContainer" containerID="73ff8e29c39a3ad1a4d270761712381e4fe379f5450ac2ffb2ac51f2a01169a5" Oct 31 04:25:56.804743 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:25:56.804716 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73ff8e29c39a3ad1a4d270761712381e4fe379f5450ac2ffb2ac51f2a01169a5\": container with ID starting with 73ff8e29c39a3ad1a4d270761712381e4fe379f5450ac2ffb2ac51f2a01169a5 not found: ID does not exist" containerID="73ff8e29c39a3ad1a4d270761712381e4fe379f5450ac2ffb2ac51f2a01169a5" Oct 31 04:25:56.804803 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:56.804755 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73ff8e29c39a3ad1a4d270761712381e4fe379f5450ac2ffb2ac51f2a01169a5"} err="failed to get container status \"73ff8e29c39a3ad1a4d270761712381e4fe379f5450ac2ffb2ac51f2a01169a5\": rpc error: code = NotFound desc = could not find container \"73ff8e29c39a3ad1a4d270761712381e4fe379f5450ac2ffb2ac51f2a01169a5\": container with ID starting with 73ff8e29c39a3ad1a4d270761712381e4fe379f5450ac2ffb2ac51f2a01169a5 not found: ID does not exist" Oct 31 04:25:56.843370 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:56.843319 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj"] Oct 31 04:25:56.855283 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:56.855250 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-b4cdj"] Oct 31 04:25:57.409358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:25:57.409314 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1453873b-b35e-40f3-8c37-ad757223b110" path="/var/lib/kubelet/pods/1453873b-b35e-40f3-8c37-ad757223b110/volumes" Oct 31 04:26:06.130047 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:06.130008 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9154a6-6cb7bfddbf-mcqng"] Oct 31 04:26:06.868175 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:06.867217 5019 scope.go:122] "RemoveContainer" containerID="b33c291d16db72d110593a6fe200bb4f12be8b1a840b8a452ae9355d972d94dc" Oct 31 04:26:06.905052 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:06.901555 5019 scope.go:122] "RemoveContainer" containerID="97784aa525a4c157390b525855e2d43e0c21a8ddc608ca7b7b2149b53c20a6a1" Oct 31 04:26:06.965969 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:06.965920 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c"] Oct 31 04:26:06.969563 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:06.969300 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c" podUID="43e793da-0b3f-4247-bc2e-6d19de53d75c" containerName="istio-proxy" containerID="cri-o://9e2b12eee106d23f52b7e6ff3ca751e293adacbcf8bd8e8439395a11aa7349fc" gracePeriod=30 Oct 31 04:26:06.975174 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:06.975128 5019 scope.go:122] "RemoveContainer" containerID="c5a979510bd229a6c9219856e2b2aef14d35654aeb94673f056c8f7b2f7813b9" Oct 31 04:26:07.052989 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:07.052614 5019 scope.go:122] "RemoveContainer" containerID="ddd7cc5f2d0e908074ee241acf9cd19e8ada248fa2ddac1bdb51d63b6f224795" Oct 31 04:26:07.098420 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:07.094391 5019 scope.go:122] "RemoveContainer" containerID="6fbf4904b9dc01642a9c2697f046fa7b0e0c5dd72fbc01b5c66d557df0408561" Oct 31 04:26:07.117927 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:07.117905 5019 scope.go:122] "RemoveContainer" containerID="65c7bcabdcde60229ec981ab973e38769855967635769cf3bf0fd547656c0eff" Oct 31 04:26:07.130534 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:07.130505 5019 scope.go:122] "RemoveContainer" containerID="037a7cfbbcf3a5dc261e05440707690234569c71d53d314fa4907fc72a786840" Oct 31 04:26:07.179553 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:07.178937 5019 scope.go:122] "RemoveContainer" containerID="08ef458afe0b08196e861aa53f685033bf2f564bfeec4ad23358d2a241ec1648" Oct 31 04:26:07.197537 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:07.197178 5019 scope.go:122] "RemoveContainer" containerID="16bfe4bf35bf7238f6a5558bb61bd3594c7fce3e9013a4ec6adb4f130a98cfc5" Oct 31 04:26:07.254536 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:07.253441 5019 scope.go:122] "RemoveContainer" containerID="312fb96716141c036e0c4ff584983c5153445a44b3c6605da344f3cd03a1ce58" Oct 31 04:26:07.273321 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:07.272533 5019 scope.go:122] "RemoveContainer" containerID="d4dee8a184f7fe3d398e015dc8f3b0e079f81c25c0802cb6b25c594fd546ba91" Oct 31 04:26:07.290003 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:07.289846 5019 scope.go:122] "RemoveContainer" containerID="819300b37635043a8938891567191ac2ec3998701be3318bfeca62ed0e5cb7cb" Oct 31 04:26:07.335924 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:07.335734 5019 scope.go:122] "RemoveContainer" containerID="5209d794ee83cd75e772052ddc68dff4c0670716ce6d210c7d80b221bf66cd51" Oct 31 04:26:07.348496 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:07.348352 5019 scope.go:122] "RemoveContainer" containerID="0def4fcf709a929ff3ee22d0efeea73da37529e90303d0af9a98790591c0d0a6" Oct 31 04:26:07.595481 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:07.591140 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a9b573-75678d44f9-l4cmb"] Oct 31 04:26:07.595481 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:07.592108 5019 cpu_manager.go:414] "containerMap: removing container" podUID="c6ce1804-04ce-4380-af18-fcf81d528500" containerName="payload-processing" Oct 31 04:26:07.595481 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:07.592131 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6ce1804-04ce-4380-af18-fcf81d528500" containerName="payload-processing" Oct 31 04:26:07.595481 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:07.592155 5019 cpu_manager.go:414] "containerMap: removing container" podUID="81895138-f13d-4457-894b-88439c752982" containerName="payload-pre-processing" Oct 31 04:26:07.595481 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:07.592163 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81895138-f13d-4457-894b-88439c752982" containerName="payload-pre-processing" Oct 31 04:26:07.595481 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:07.592187 5019 cpu_manager.go:414] "containerMap: removing container" podUID="661aa7e3-37aa-4131-aa1d-10c81372dd2d" containerName="istio-proxy" Oct 31 04:26:07.595481 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:07.592194 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="661aa7e3-37aa-4131-aa1d-10c81372dd2d" containerName="istio-proxy" Oct 31 04:26:07.595481 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:07.592238 5019 cpu_manager.go:414] "containerMap: removing container" podUID="1453873b-b35e-40f3-8c37-ad757223b110" containerName="istio-proxy" Oct 31 04:26:07.595481 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:07.592246 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1453873b-b35e-40f3-8c37-ad757223b110" containerName="istio-proxy" Oct 31 04:26:07.595481 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:07.592256 5019 cpu_manager.go:414] "containerMap: removing container" podUID="c8d33677-060d-418a-8806-84eeb0c17ecb" containerName="istio-proxy" Oct 31 04:26:07.595481 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:07.592262 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8d33677-060d-418a-8806-84eeb0c17ecb" containerName="istio-proxy" Oct 31 04:26:07.595481 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:07.592271 5019 cpu_manager.go:414] "containerMap: removing container" podUID="a9256bfe-8c85-477e-bcc9-9938f4c689ab" containerName="istio-proxy" Oct 31 04:26:07.595481 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:07.592279 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9256bfe-8c85-477e-bcc9-9938f4c689ab" containerName="istio-proxy" Oct 31 04:26:07.595481 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:07.592288 5019 cpu_manager.go:414] "containerMap: removing container" podUID="db10a00c-b537-4c62-b9aa-b98dc5577720" containerName="istio-proxy" Oct 31 04:26:07.595481 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:07.592294 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db10a00c-b537-4c62-b9aa-b98dc5577720" containerName="istio-proxy" Oct 31 04:26:07.595481 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:07.592478 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="1453873b-b35e-40f3-8c37-ad757223b110" containerName="istio-proxy" Oct 31 04:26:07.595481 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:07.592492 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8d33677-060d-418a-8806-84eeb0c17ecb" containerName="istio-proxy" Oct 31 04:26:07.595481 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:07.592500 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9256bfe-8c85-477e-bcc9-9938f4c689ab" containerName="istio-proxy" Oct 31 04:26:07.595481 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:07.592510 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="db10a00c-b537-4c62-b9aa-b98dc5577720" containerName="istio-proxy" Oct 31 04:26:07.595481 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:07.592527 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6ce1804-04ce-4380-af18-fcf81d528500" containerName="payload-processing" Oct 31 04:26:07.595481 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:07.592538 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="81895138-f13d-4457-894b-88439c752982" containerName="payload-pre-processing" Oct 31 04:26:07.595481 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:07.592547 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="661aa7e3-37aa-4131-aa1d-10c81372dd2d" containerName="istio-proxy" Oct 31 04:26:07.614654 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:07.613659 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9b573-75678d44f9-l4cmb" Oct 31 04:26:07.614654 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:07.613564 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a9b573-75678d44f9-l4cmb"] Oct 31 04:26:07.631635 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:07.630158 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a9b573\"" Oct 31 04:26:07.693568 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:07.693531 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7227\" (UniqueName: \"kubernetes.io/projected/7bfeaaa0-ebe7-49cb-9868-d02bed5e1240-kube-api-access-l7227\") pod \"payload-pre-processing-e2e-ipp-b-a9b573-75678d44f9-l4cmb\" (UID: \"7bfeaaa0-ebe7-49cb-9868-d02bed5e1240\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9b573-75678d44f9-l4cmb" Oct 31 04:26:07.694145 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:07.694125 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bfeaaa0-ebe7-49cb-9868-d02bed5e1240-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a9b573-75678d44f9-l4cmb\" (UID: \"7bfeaaa0-ebe7-49cb-9868-d02bed5e1240\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9b573-75678d44f9-l4cmb" Oct 31 04:26:07.796021 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:07.795980 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bfeaaa0-ebe7-49cb-9868-d02bed5e1240-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a9b573-75678d44f9-l4cmb\" (UID: \"7bfeaaa0-ebe7-49cb-9868-d02bed5e1240\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9b573-75678d44f9-l4cmb" Oct 31 04:26:07.796215 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:07.796170 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7227\" (UniqueName: \"kubernetes.io/projected/7bfeaaa0-ebe7-49cb-9868-d02bed5e1240-kube-api-access-l7227\") pod \"payload-pre-processing-e2e-ipp-b-a9b573-75678d44f9-l4cmb\" (UID: \"7bfeaaa0-ebe7-49cb-9868-d02bed5e1240\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9b573-75678d44f9-l4cmb" Oct 31 04:26:07.796780 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:07.796757 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bfeaaa0-ebe7-49cb-9868-d02bed5e1240-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a9b573-75678d44f9-l4cmb\" (UID: \"7bfeaaa0-ebe7-49cb-9868-d02bed5e1240\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9b573-75678d44f9-l4cmb" Oct 31 04:26:07.857411 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:07.857382 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7227\" (UniqueName: \"kubernetes.io/projected/7bfeaaa0-ebe7-49cb-9868-d02bed5e1240-kube-api-access-l7227\") pod \"payload-pre-processing-e2e-ipp-b-a9b573-75678d44f9-l4cmb\" (UID: \"7bfeaaa0-ebe7-49cb-9868-d02bed5e1240\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9b573-75678d44f9-l4cmb" Oct 31 04:26:07.941054 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:07.940970 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9b573-75678d44f9-l4cmb" Oct 31 04:26:08.246052 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:08.245976 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a9b573-75678d44f9-l4cmb"] Oct 31 04:26:08.907077 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:08.907035 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9b573-75678d44f9-l4cmb" event={"ID":"7bfeaaa0-ebe7-49cb-9868-d02bed5e1240","Type":"ContainerStarted","Data":"392310e6a7cee5a5bd369045329ba36b11b36d48d50e47a1b8b8ad45af1af542"} Oct 31 04:26:08.907296 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:08.907132 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9b573-75678d44f9-l4cmb" event={"ID":"7bfeaaa0-ebe7-49cb-9868-d02bed5e1240","Type":"ContainerStarted","Data":"acab2575e7349170b3e15e911ec1d7ffba5dec1542f241c479be664410c4ead7"} Oct 31 04:26:08.907296 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:08.907283 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9b573-75678d44f9-l4cmb" Oct 31 04:26:08.948808 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:08.948751 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9b573-75678d44f9-l4cmb" podStartSLOduration=1.948737429 podStartE2EDuration="1.948737429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:26:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:26:08.94772935 +0000 UTC m=+1323.704067389" watchObservedRunningTime="2026-10-31 04:26:08.948737429 +0000 UTC m=+1323.705075468" Oct 31 04:26:11.130895 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:11.130826 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-9154a6-6cb7bfddbf-mcqng" podUID="8476795a-d573-42e9-a42f-19e307af2592" containerName="payload-processing" containerID="cri-o://fed92357375992113c03cd6ca3ebb0ca93afb35afe7ca07a4ef6ef78553ea595" gracePeriod=25 Oct 31 04:26:11.428595 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:11.428570 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9154a6-6cb7bfddbf-mcqng" Oct 31 04:26:11.553955 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:11.553926 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8476795a-d573-42e9-a42f-19e307af2592-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8476795a-d573-42e9-a42f-19e307af2592-plugins-config-volume\") pod \"8476795a-d573-42e9-a42f-19e307af2592\" (UID: \"8476795a-d573-42e9-a42f-19e307af2592\") " Oct 31 04:26:11.554165 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:11.554109 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8476795a-d573-42e9-a42f-19e307af2592-kube-api-access-lssqf\" (UniqueName: \"kubernetes.io/projected/8476795a-d573-42e9-a42f-19e307af2592-kube-api-access-lssqf\") pod \"8476795a-d573-42e9-a42f-19e307af2592\" (UID: \"8476795a-d573-42e9-a42f-19e307af2592\") " Oct 31 04:26:11.554372 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:11.554348 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8476795a-d573-42e9-a42f-19e307af2592-plugins-config-volume" pod "8476795a-d573-42e9-a42f-19e307af2592" (UID: "8476795a-d573-42e9-a42f-19e307af2592"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:26:11.554692 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:11.554676 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8476795a-d573-42e9-a42f-19e307af2592-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:11.556255 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:11.556234 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8476795a-d573-42e9-a42f-19e307af2592-kube-api-access-lssqf" pod "8476795a-d573-42e9-a42f-19e307af2592" (UID: "8476795a-d573-42e9-a42f-19e307af2592"). InnerVolumeSpecName "kube-api-access-lssqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:26:11.655956 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:11.655858 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lssqf\" (UniqueName: \"kubernetes.io/projected/8476795a-d573-42e9-a42f-19e307af2592-kube-api-access-lssqf\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:11.933515 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:11.933422 5019 generic.go:356] "Generic (PLEG): container finished" podID="8476795a-d573-42e9-a42f-19e307af2592" containerID="fed92357375992113c03cd6ca3ebb0ca93afb35afe7ca07a4ef6ef78553ea595" exitCode=0 Oct 31 04:26:11.933515 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:11.933486 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9154a6-6cb7bfddbf-mcqng" Oct 31 04:26:11.933515 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:11.933504 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9154a6-6cb7bfddbf-mcqng" event={"ID":"8476795a-d573-42e9-a42f-19e307af2592","Type":"ContainerDied","Data":"fed92357375992113c03cd6ca3ebb0ca93afb35afe7ca07a4ef6ef78553ea595"} Oct 31 04:26:11.933783 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:11.933586 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9154a6-6cb7bfddbf-mcqng" event={"ID":"8476795a-d573-42e9-a42f-19e307af2592","Type":"ContainerDied","Data":"b55aaf434730518209d4da5897436696d09a77d206e2765dcefe05424421d361"} Oct 31 04:26:11.933783 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:11.933608 5019 scope.go:122] "RemoveContainer" containerID="fed92357375992113c03cd6ca3ebb0ca93afb35afe7ca07a4ef6ef78553ea595" Oct 31 04:26:11.944313 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:11.944289 5019 scope.go:122] "RemoveContainer" containerID="fed92357375992113c03cd6ca3ebb0ca93afb35afe7ca07a4ef6ef78553ea595" Oct 31 04:26:11.944616 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:26:11.944599 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fed92357375992113c03cd6ca3ebb0ca93afb35afe7ca07a4ef6ef78553ea595\": container with ID starting with fed92357375992113c03cd6ca3ebb0ca93afb35afe7ca07a4ef6ef78553ea595 not found: ID does not exist" containerID="fed92357375992113c03cd6ca3ebb0ca93afb35afe7ca07a4ef6ef78553ea595" Oct 31 04:26:11.944670 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:11.944627 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fed92357375992113c03cd6ca3ebb0ca93afb35afe7ca07a4ef6ef78553ea595"} err="failed to get container status \"fed92357375992113c03cd6ca3ebb0ca93afb35afe7ca07a4ef6ef78553ea595\": rpc error: code = NotFound desc = could not find container \"fed92357375992113c03cd6ca3ebb0ca93afb35afe7ca07a4ef6ef78553ea595\": container with ID starting with fed92357375992113c03cd6ca3ebb0ca93afb35afe7ca07a4ef6ef78553ea595 not found: ID does not exist" Oct 31 04:26:11.987209 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:11.987174 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9154a6-6cb7bfddbf-mcqng"] Oct 31 04:26:11.998556 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:11.998527 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9154a6-6cb7bfddbf-mcqng"] Oct 31 04:26:12.313001 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:12.312976 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c" Oct 31 04:26:12.465670 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:12.465585 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43e793da-0b3f-4247-bc2e-6d19de53d75c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43e793da-0b3f-4247-bc2e-6d19de53d75c-istiod-ca-cert\") pod \"43e793da-0b3f-4247-bc2e-6d19de53d75c\" (UID: \"43e793da-0b3f-4247-bc2e-6d19de53d75c\") " Oct 31 04:26:12.465670 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:12.465623 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43e793da-0b3f-4247-bc2e-6d19de53d75c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43e793da-0b3f-4247-bc2e-6d19de53d75c-istio-envoy\") pod \"43e793da-0b3f-4247-bc2e-6d19de53d75c\" (UID: \"43e793da-0b3f-4247-bc2e-6d19de53d75c\") " Oct 31 04:26:12.465670 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:12.465671 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43e793da-0b3f-4247-bc2e-6d19de53d75c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43e793da-0b3f-4247-bc2e-6d19de53d75c-workload-certs\") pod \"43e793da-0b3f-4247-bc2e-6d19de53d75c\" (UID: \"43e793da-0b3f-4247-bc2e-6d19de53d75c\") " Oct 31 04:26:12.465888 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:12.465725 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43e793da-0b3f-4247-bc2e-6d19de53d75c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e793da-0b3f-4247-bc2e-6d19de53d75c-workload-socket\") pod \"43e793da-0b3f-4247-bc2e-6d19de53d75c\" (UID: \"43e793da-0b3f-4247-bc2e-6d19de53d75c\") " Oct 31 04:26:12.465941 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:12.465894 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43e793da-0b3f-4247-bc2e-6d19de53d75c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43e793da-0b3f-4247-bc2e-6d19de53d75c-istio-data\") pod \"43e793da-0b3f-4247-bc2e-6d19de53d75c\" (UID: \"43e793da-0b3f-4247-bc2e-6d19de53d75c\") " Oct 31 04:26:12.466014 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:12.465989 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43e793da-0b3f-4247-bc2e-6d19de53d75c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e793da-0b3f-4247-bc2e-6d19de53d75c-credential-socket\") pod \"43e793da-0b3f-4247-bc2e-6d19de53d75c\" (UID: \"43e793da-0b3f-4247-bc2e-6d19de53d75c\") " Oct 31 04:26:12.466068 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:12.466020 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43e793da-0b3f-4247-bc2e-6d19de53d75c-workload-certs" pod "43e793da-0b3f-4247-bc2e-6d19de53d75c" (UID: "43e793da-0b3f-4247-bc2e-6d19de53d75c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:26:12.466124 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:12.466104 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43e793da-0b3f-4247-bc2e-6d19de53d75c-istiod-ca-cert" pod "43e793da-0b3f-4247-bc2e-6d19de53d75c" (UID: "43e793da-0b3f-4247-bc2e-6d19de53d75c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:26:12.466171 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:12.466122 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43e793da-0b3f-4247-bc2e-6d19de53d75c-workload-socket" pod "43e793da-0b3f-4247-bc2e-6d19de53d75c" (UID: "43e793da-0b3f-4247-bc2e-6d19de53d75c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:26:12.466222 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:12.466024 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43e793da-0b3f-4247-bc2e-6d19de53d75c-istio-token\" (UniqueName: \"kubernetes.io/projected/43e793da-0b3f-4247-bc2e-6d19de53d75c-istio-token\") pod \"43e793da-0b3f-4247-bc2e-6d19de53d75c\" (UID: \"43e793da-0b3f-4247-bc2e-6d19de53d75c\") " Oct 31 04:26:12.466275 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:12.466213 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43e793da-0b3f-4247-bc2e-6d19de53d75c-credential-socket" pod "43e793da-0b3f-4247-bc2e-6d19de53d75c" (UID: "43e793da-0b3f-4247-bc2e-6d19de53d75c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:26:12.466326 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:12.466280 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43e793da-0b3f-4247-bc2e-6d19de53d75c-kube-api-access-xwqzb\" (UniqueName: \"kubernetes.io/projected/43e793da-0b3f-4247-bc2e-6d19de53d75c-kube-api-access-xwqzb\") pod \"43e793da-0b3f-4247-bc2e-6d19de53d75c\" (UID: \"43e793da-0b3f-4247-bc2e-6d19de53d75c\") " Oct 31 04:26:12.466326 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:12.466307 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43e793da-0b3f-4247-bc2e-6d19de53d75c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43e793da-0b3f-4247-bc2e-6d19de53d75c-istio-podinfo\") pod \"43e793da-0b3f-4247-bc2e-6d19de53d75c\" (UID: \"43e793da-0b3f-4247-bc2e-6d19de53d75c\") " Oct 31 04:26:12.466688 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:12.466641 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43e793da-0b3f-4247-bc2e-6d19de53d75c-istio-data" pod "43e793da-0b3f-4247-bc2e-6d19de53d75c" (UID: "43e793da-0b3f-4247-bc2e-6d19de53d75c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:26:12.467191 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:12.467167 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43e793da-0b3f-4247-bc2e-6d19de53d75c-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:12.467191 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:12.467191 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43e793da-0b3f-4247-bc2e-6d19de53d75c-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:12.467408 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:12.467204 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e793da-0b3f-4247-bc2e-6d19de53d75c-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:12.467408 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:12.467216 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43e793da-0b3f-4247-bc2e-6d19de53d75c-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:12.467408 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:12.467229 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e793da-0b3f-4247-bc2e-6d19de53d75c-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:12.471208 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:12.471178 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43e793da-0b3f-4247-bc2e-6d19de53d75c-istio-envoy" pod "43e793da-0b3f-4247-bc2e-6d19de53d75c" (UID: "43e793da-0b3f-4247-bc2e-6d19de53d75c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:26:12.471325 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:12.471280 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43e793da-0b3f-4247-bc2e-6d19de53d75c-istio-podinfo" pod "43e793da-0b3f-4247-bc2e-6d19de53d75c" (UID: "43e793da-0b3f-4247-bc2e-6d19de53d75c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:26:12.471325 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:12.471303 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43e793da-0b3f-4247-bc2e-6d19de53d75c-kube-api-access-xwqzb" pod "43e793da-0b3f-4247-bc2e-6d19de53d75c" (UID: "43e793da-0b3f-4247-bc2e-6d19de53d75c"). InnerVolumeSpecName "kube-api-access-xwqzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:26:12.471625 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:12.471603 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43e793da-0b3f-4247-bc2e-6d19de53d75c-istio-token" pod "43e793da-0b3f-4247-bc2e-6d19de53d75c" (UID: "43e793da-0b3f-4247-bc2e-6d19de53d75c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:26:12.568135 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:12.568102 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwqzb\" (UniqueName: \"kubernetes.io/projected/43e793da-0b3f-4247-bc2e-6d19de53d75c-kube-api-access-xwqzb\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:12.568135 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:12.568133 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43e793da-0b3f-4247-bc2e-6d19de53d75c-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:12.568348 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:12.568146 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43e793da-0b3f-4247-bc2e-6d19de53d75c-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:12.568348 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:12.568158 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43e793da-0b3f-4247-bc2e-6d19de53d75c-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:12.941689 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:12.941657 5019 generic.go:356] "Generic (PLEG): container finished" podID="43e793da-0b3f-4247-bc2e-6d19de53d75c" containerID="9e2b12eee106d23f52b7e6ff3ca751e293adacbcf8bd8e8439395a11aa7349fc" exitCode=0 Oct 31 04:26:12.941878 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:12.941705 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c" Oct 31 04:26:12.941878 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:12.941737 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c" event={"ID":"43e793da-0b3f-4247-bc2e-6d19de53d75c","Type":"ContainerDied","Data":"9e2b12eee106d23f52b7e6ff3ca751e293adacbcf8bd8e8439395a11aa7349fc"} Oct 31 04:26:12.941878 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:12.941788 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c" event={"ID":"43e793da-0b3f-4247-bc2e-6d19de53d75c","Type":"ContainerDied","Data":"9638d6f6e7372b62d39afc81d0a40c0c95fc9d1e17ee2e43eeeb9c3f0f636576"} Oct 31 04:26:12.941878 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:12.941817 5019 scope.go:122] "RemoveContainer" containerID="9e2b12eee106d23f52b7e6ff3ca751e293adacbcf8bd8e8439395a11aa7349fc" Oct 31 04:26:12.952880 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:12.952863 5019 scope.go:122] "RemoveContainer" containerID="9e2b12eee106d23f52b7e6ff3ca751e293adacbcf8bd8e8439395a11aa7349fc" Oct 31 04:26:12.953146 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:26:12.953128 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e2b12eee106d23f52b7e6ff3ca751e293adacbcf8bd8e8439395a11aa7349fc\": container with ID starting with 9e2b12eee106d23f52b7e6ff3ca751e293adacbcf8bd8e8439395a11aa7349fc not found: ID does not exist" containerID="9e2b12eee106d23f52b7e6ff3ca751e293adacbcf8bd8e8439395a11aa7349fc" Oct 31 04:26:12.953189 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:12.953154 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e2b12eee106d23f52b7e6ff3ca751e293adacbcf8bd8e8439395a11aa7349fc"} err="failed to get container status \"9e2b12eee106d23f52b7e6ff3ca751e293adacbcf8bd8e8439395a11aa7349fc\": rpc error: code = NotFound desc = could not find container \"9e2b12eee106d23f52b7e6ff3ca751e293adacbcf8bd8e8439395a11aa7349fc\": container with ID starting with 9e2b12eee106d23f52b7e6ff3ca751e293adacbcf8bd8e8439395a11aa7349fc not found: ID does not exist" Oct 31 04:26:12.996055 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:12.996025 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c"] Oct 31 04:26:13.010179 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:13.010151 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-db98c"] Oct 31 04:26:13.419196 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:13.419162 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43e793da-0b3f-4247-bc2e-6d19de53d75c" path="/var/lib/kubelet/pods/43e793da-0b3f-4247-bc2e-6d19de53d75c/volumes" Oct 31 04:26:13.419948 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:13.419921 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8476795a-d573-42e9-a42f-19e307af2592" path="/var/lib/kubelet/pods/8476795a-d573-42e9-a42f-19e307af2592/volumes" Oct 31 04:26:15.089870 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:15.089831 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd"] Oct 31 04:26:15.090908 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:15.090887 5019 cpu_manager.go:414] "containerMap: removing container" podUID="43e793da-0b3f-4247-bc2e-6d19de53d75c" containerName="istio-proxy" Oct 31 04:26:15.090991 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:15.090910 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43e793da-0b3f-4247-bc2e-6d19de53d75c" containerName="istio-proxy" Oct 31 04:26:15.090991 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:15.090953 5019 cpu_manager.go:414] "containerMap: removing container" podUID="8476795a-d573-42e9-a42f-19e307af2592" containerName="payload-processing" Oct 31 04:26:15.090991 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:15.090963 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8476795a-d573-42e9-a42f-19e307af2592" containerName="payload-processing" Oct 31 04:26:15.091126 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:15.091113 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="8476795a-d573-42e9-a42f-19e307af2592" containerName="payload-processing" Oct 31 04:26:15.091170 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:15.091139 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="43e793da-0b3f-4247-bc2e-6d19de53d75c" containerName="istio-proxy" Oct 31 04:26:15.098712 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:15.098686 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd" Oct 31 04:26:15.131197 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:15.131169 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd"] Oct 31 04:26:15.137295 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:15.137243 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-ff43e3-openshift-default-dockercfg-zjn7r\"" Oct 31 04:26:15.229353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:15.224805 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-istio-envoy\") pod \"e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd\" (UID: \"a473bd3b-e50d-4a3d-ad54-fd49ee49f144\") " pod="openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd" Oct 31 04:26:15.229353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:15.224887 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-istiod-ca-cert\") pod \"e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd\" (UID: \"a473bd3b-e50d-4a3d-ad54-fd49ee49f144\") " pod="openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd" Oct 31 04:26:15.229353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:15.224921 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-istio-data\") pod \"e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd\" (UID: \"a473bd3b-e50d-4a3d-ad54-fd49ee49f144\") " pod="openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd" Oct 31 04:26:15.229353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:15.224979 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-istio-podinfo\") pod \"e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd\" (UID: \"a473bd3b-e50d-4a3d-ad54-fd49ee49f144\") " pod="openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd" Oct 31 04:26:15.229353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:15.225026 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-credential-socket\") pod \"e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd\" (UID: \"a473bd3b-e50d-4a3d-ad54-fd49ee49f144\") " pod="openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd" Oct 31 04:26:15.229353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:15.225075 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-istio-token\") pod \"e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd\" (UID: \"a473bd3b-e50d-4a3d-ad54-fd49ee49f144\") " pod="openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd" Oct 31 04:26:15.229353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:15.225187 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-workload-socket\") pod \"e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd\" (UID: \"a473bd3b-e50d-4a3d-ad54-fd49ee49f144\") " pod="openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd" Oct 31 04:26:15.229353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:15.225277 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8bk2\" (UniqueName: \"kubernetes.io/projected/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-kube-api-access-k8bk2\") pod \"e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd\" (UID: \"a473bd3b-e50d-4a3d-ad54-fd49ee49f144\") " pod="openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd" Oct 31 04:26:15.229353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:15.225320 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-workload-certs\") pod \"e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd\" (UID: \"a473bd3b-e50d-4a3d-ad54-fd49ee49f144\") " pod="openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd" Oct 31 04:26:15.328356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:15.327705 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-istio-token\") pod \"e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd\" (UID: \"a473bd3b-e50d-4a3d-ad54-fd49ee49f144\") " pod="openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd" Oct 31 04:26:15.328569 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:15.328446 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-workload-socket\") pod \"e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd\" (UID: \"a473bd3b-e50d-4a3d-ad54-fd49ee49f144\") " pod="openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd" Oct 31 04:26:15.328569 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:15.328551 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8bk2\" (UniqueName: \"kubernetes.io/projected/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-kube-api-access-k8bk2\") pod \"e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd\" (UID: \"a473bd3b-e50d-4a3d-ad54-fd49ee49f144\") " pod="openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd" Oct 31 04:26:15.328689 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:15.328599 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-workload-certs\") pod \"e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd\" (UID: \"a473bd3b-e50d-4a3d-ad54-fd49ee49f144\") " pod="openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd" Oct 31 04:26:15.329202 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:15.328642 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-istio-envoy\") pod \"e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd\" (UID: \"a473bd3b-e50d-4a3d-ad54-fd49ee49f144\") " pod="openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd" Oct 31 04:26:15.329202 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:15.328911 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-istiod-ca-cert\") pod \"e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd\" (UID: \"a473bd3b-e50d-4a3d-ad54-fd49ee49f144\") " pod="openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd" Oct 31 04:26:15.329202 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:15.328949 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-istio-data\") pod \"e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd\" (UID: \"a473bd3b-e50d-4a3d-ad54-fd49ee49f144\") " pod="openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd" Oct 31 04:26:15.329202 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:15.329014 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-istio-podinfo\") pod \"e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd\" (UID: \"a473bd3b-e50d-4a3d-ad54-fd49ee49f144\") " pod="openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd" Oct 31 04:26:15.329510 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:15.329067 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-credential-socket\") pod \"e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd\" (UID: \"a473bd3b-e50d-4a3d-ad54-fd49ee49f144\") " pod="openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd" Oct 31 04:26:15.336361 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:15.335876 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-workload-certs\") pod \"e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd\" (UID: \"a473bd3b-e50d-4a3d-ad54-fd49ee49f144\") " pod="openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd" Oct 31 04:26:15.348433 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:15.346787 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-istio-data\") pod \"e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd\" (UID: \"a473bd3b-e50d-4a3d-ad54-fd49ee49f144\") " pod="openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd" Oct 31 04:26:15.348433 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:15.348272 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-istiod-ca-cert\") pod \"e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd\" (UID: \"a473bd3b-e50d-4a3d-ad54-fd49ee49f144\") " pod="openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd" Oct 31 04:26:15.348651 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:15.348580 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-workload-socket\") pod \"e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd\" (UID: \"a473bd3b-e50d-4a3d-ad54-fd49ee49f144\") " pod="openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd" Oct 31 04:26:15.374673 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:15.348888 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-credential-socket\") pod \"e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd\" (UID: \"a473bd3b-e50d-4a3d-ad54-fd49ee49f144\") " pod="openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd" Oct 31 04:26:15.374673 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:15.370367 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8bk2\" (UniqueName: \"kubernetes.io/projected/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-kube-api-access-k8bk2\") pod \"e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd\" (UID: \"a473bd3b-e50d-4a3d-ad54-fd49ee49f144\") " pod="openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd" Oct 31 04:26:15.396382 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:15.395504 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-istio-envoy\") pod \"e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd\" (UID: \"a473bd3b-e50d-4a3d-ad54-fd49ee49f144\") " pod="openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd" Oct 31 04:26:15.426356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:15.421280 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-istio-podinfo\") pod \"e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd\" (UID: \"a473bd3b-e50d-4a3d-ad54-fd49ee49f144\") " pod="openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd" Oct 31 04:26:15.434619 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:15.433853 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-istio-token\") pod \"e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd\" (UID: \"a473bd3b-e50d-4a3d-ad54-fd49ee49f144\") " pod="openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd" Oct 31 04:26:15.734222 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:15.734182 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd" Oct 31 04:26:16.060208 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:16.060173 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd"] Oct 31 04:26:16.060722 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:26:16.060696 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda473bd3b_e50d_4a3d_ad54_fd49ee49f144.slice/crio-b803dc928a469e94e5e1ef458f1eea779d5f91fd93d8a96bef719f44eda633bb WatchSource:0}: Error finding container b803dc928a469e94e5e1ef458f1eea779d5f91fd93d8a96bef719f44eda633bb: Status 404 returned error can't find the container with id b803dc928a469e94e5e1ef458f1eea779d5f91fd93d8a96bef719f44eda633bb Oct 31 04:26:16.063542 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:16.063494 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:26:16.063634 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:16.063574 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:26:16.063634 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:16.063608 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:26:16.990530 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:16.990476 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd" event={"ID":"a473bd3b-e50d-4a3d-ad54-fd49ee49f144","Type":"ContainerStarted","Data":"ae4735ff03902e2365dcc8f480457ab57d0e20248b7405414b828b8d3667dca5"} Oct 31 04:26:16.990976 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:16.990549 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd" event={"ID":"a473bd3b-e50d-4a3d-ad54-fd49ee49f144","Type":"ContainerStarted","Data":"b803dc928a469e94e5e1ef458f1eea779d5f91fd93d8a96bef719f44eda633bb"} Oct 31 04:26:17.050165 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:17.050114 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd" podStartSLOduration=2.050101261 podStartE2EDuration="2.050101261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:26:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:26:17.046043518 +0000 UTC m=+1331.802381558" watchObservedRunningTime="2026-10-31 04:26:17.050101261 +0000 UTC m=+1331.806439300" Oct 31 04:26:17.734507 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:17.734471 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd" Oct 31 04:26:17.739941 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:17.739915 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd" Oct 31 04:26:17.997744 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:17.997651 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd" Oct 31 04:26:17.998840 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:17.998814 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd" Oct 31 04:26:19.031714 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:19.031680 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35fb49a3-tthwf"] Oct 31 04:26:19.044980 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:19.044952 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35fb49a3-tthwf"] Oct 31 04:26:19.409065 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:19.408987 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03b83c43-5a5c-48c1-a804-c92f53fa02a2" path="/var/lib/kubelet/pods/03b83c43-5a5c-48c1-a804-c92f53fa02a2/volumes" Oct 31 04:26:19.917024 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:19.916994 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9b573-75678d44f9-l4cmb" Oct 31 04:26:21.495374 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:21.495320 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ff43e3-5d974c979f-gz8bs"] Oct 31 04:26:21.501374 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:21.501314 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff43e3-5d974c979f-gz8bs" Oct 31 04:26:21.520004 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:21.519935 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ff43e3-5d974c979f-gz8bs"] Oct 31 04:26:21.561235 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:21.526682 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-ff43e3\"" Oct 31 04:26:21.561235 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:21.527399 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-ff43e3-dockercfg-jhhdh\"" Oct 31 04:26:21.643712 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:21.643660 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef78a46-e363-4411-90b8-df50a676c9e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ff43e3-5d974c979f-gz8bs\" (UID: \"3ef78a46-e363-4411-90b8-df50a676c9e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff43e3-5d974c979f-gz8bs" Oct 31 04:26:21.643939 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:21.643860 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldfpr\" (UniqueName: \"kubernetes.io/projected/3ef78a46-e363-4411-90b8-df50a676c9e5-kube-api-access-ldfpr\") pod \"payload-pre-processing-e2e-ipp-a-ff43e3-5d974c979f-gz8bs\" (UID: \"3ef78a46-e363-4411-90b8-df50a676c9e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff43e3-5d974c979f-gz8bs" Oct 31 04:26:21.746628 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:21.745490 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef78a46-e363-4411-90b8-df50a676c9e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ff43e3-5d974c979f-gz8bs\" (UID: \"3ef78a46-e363-4411-90b8-df50a676c9e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff43e3-5d974c979f-gz8bs" Oct 31 04:26:21.746628 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:21.745672 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldfpr\" (UniqueName: \"kubernetes.io/projected/3ef78a46-e363-4411-90b8-df50a676c9e5-kube-api-access-ldfpr\") pod \"payload-pre-processing-e2e-ipp-a-ff43e3-5d974c979f-gz8bs\" (UID: \"3ef78a46-e363-4411-90b8-df50a676c9e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff43e3-5d974c979f-gz8bs" Oct 31 04:26:21.746628 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:21.746545 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef78a46-e363-4411-90b8-df50a676c9e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ff43e3-5d974c979f-gz8bs\" (UID: \"3ef78a46-e363-4411-90b8-df50a676c9e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff43e3-5d974c979f-gz8bs" Oct 31 04:26:21.777828 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:21.777794 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldfpr\" (UniqueName: \"kubernetes.io/projected/3ef78a46-e363-4411-90b8-df50a676c9e5-kube-api-access-ldfpr\") pod \"payload-pre-processing-e2e-ipp-a-ff43e3-5d974c979f-gz8bs\" (UID: \"3ef78a46-e363-4411-90b8-df50a676c9e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff43e3-5d974c979f-gz8bs" Oct 31 04:26:21.820062 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:21.820023 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff43e3-5d974c979f-gz8bs" Oct 31 04:26:22.130777 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:22.130746 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ff43e3-5d974c979f-gz8bs"] Oct 31 04:26:22.131454 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:26:22.131426 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ef78a46_e363_4411_90b8_df50a676c9e5.slice/crio-06b7a60a985153cc4ddde9858de6741023382639205ce95d9757a760ee8df265 WatchSource:0}: Error finding container 06b7a60a985153cc4ddde9858de6741023382639205ce95d9757a760ee8df265: Status 404 returned error can't find the container with id 06b7a60a985153cc4ddde9858de6741023382639205ce95d9757a760ee8df265 Oct 31 04:26:22.392047 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:26:22.391950 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:26:23.045519 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:23.045475 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff43e3-5d974c979f-gz8bs" event={"ID":"3ef78a46-e363-4411-90b8-df50a676c9e5","Type":"ContainerStarted","Data":"66de9a18bedd6709bdbbf2bff81f9b590b839400af8e3bee6f3d3ff48fd3b44a"} Oct 31 04:26:23.046055 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:23.045689 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff43e3-5d974c979f-gz8bs" Oct 31 04:26:23.046055 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:23.045746 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff43e3-5d974c979f-gz8bs" event={"ID":"3ef78a46-e363-4411-90b8-df50a676c9e5","Type":"ContainerStarted","Data":"06b7a60a985153cc4ddde9858de6741023382639205ce95d9757a760ee8df265"} Oct 31 04:26:23.088723 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:23.088668 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff43e3-5d974c979f-gz8bs" podStartSLOduration=2.088651305 podStartE2EDuration="2.088651305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:26:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:26:23.085842257 +0000 UTC m=+1337.842180296" watchObservedRunningTime="2026-10-31 04:26:23.088651305 +0000 UTC m=+1337.844989384" Oct 31 04:26:24.694014 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:24.693972 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c"] Oct 31 04:26:24.694440 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:24.694228 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c" podUID="dda0567c-8a01-47e8-95f9-c0a4ad08a1c6" containerName="istio-proxy" containerID="cri-o://41d29e3aedb2beba64a8b693cc2881be01db94eeecc9604fa8af1854e39bc023" gracePeriod=30 Oct 31 04:26:24.706409 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:24.706371 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt"] Oct 31 04:26:24.706665 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:24.706640 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt" podUID="56aad6d5-8f29-4414-a5e9-f7cb565f8e57" containerName="istio-proxy" containerID="cri-o://5329525c6f8da3655e4eb5c265a21d8c999ecc1bf6582145b10216c9d042f097" gracePeriod=30 Oct 31 04:26:24.713255 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:24.712420 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc"] Oct 31 04:26:24.713255 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:24.712637 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc" podUID="efc828fc-91bd-43f3-943d-144c3d9dc1b1" containerName="istio-proxy" containerID="cri-o://a1ae8aa1b3d05374777fa0e7524e4d32d8947ef2baf5192d5ed132974ed2fa1d" gracePeriod=30 Oct 31 04:26:24.718639 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:24.718607 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr"] Oct 31 04:26:24.718894 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:24.718868 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr" podUID="2df6e942-8414-427c-83df-928e51465964" containerName="istio-proxy" containerID="cri-o://6f5f8311d71344629e668869636e95d440616d983fd51f0d949b4e823edeba0e" gracePeriod=30 Oct 31 04:26:27.892941 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:27.892241 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-b8bb1e-6d9649764-p2b84_abc129a9-efa9-45f0-960d-27a21a68f326/payload-processing/0.log" Oct 31 04:26:28.080051 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:28.080015 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-b8bb1e-6d9649764-p2b84_abc129a9-efa9-45f0-960d-27a21a68f326/payload-processing/0.log" Oct 31 04:26:30.128729 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.128643 5019 generic.go:356] "Generic (PLEG): container finished" podID="dda0567c-8a01-47e8-95f9-c0a4ad08a1c6" containerID="41d29e3aedb2beba64a8b693cc2881be01db94eeecc9604fa8af1854e39bc023" exitCode=0 Oct 31 04:26:30.129170 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.128804 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c" event={"ID":"dda0567c-8a01-47e8-95f9-c0a4ad08a1c6","Type":"ContainerDied","Data":"41d29e3aedb2beba64a8b693cc2881be01db94eeecc9604fa8af1854e39bc023"} Oct 31 04:26:30.130989 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.130934 5019 generic.go:356] "Generic (PLEG): container finished" podID="2df6e942-8414-427c-83df-928e51465964" containerID="6f5f8311d71344629e668869636e95d440616d983fd51f0d949b4e823edeba0e" exitCode=0 Oct 31 04:26:30.131130 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.131047 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr" event={"ID":"2df6e942-8414-427c-83df-928e51465964","Type":"ContainerDied","Data":"6f5f8311d71344629e668869636e95d440616d983fd51f0d949b4e823edeba0e"} Oct 31 04:26:30.133021 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.132995 5019 generic.go:356] "Generic (PLEG): container finished" podID="efc828fc-91bd-43f3-943d-144c3d9dc1b1" containerID="a1ae8aa1b3d05374777fa0e7524e4d32d8947ef2baf5192d5ed132974ed2fa1d" exitCode=0 Oct 31 04:26:30.133136 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.133069 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc" event={"ID":"efc828fc-91bd-43f3-943d-144c3d9dc1b1","Type":"ContainerDied","Data":"a1ae8aa1b3d05374777fa0e7524e4d32d8947ef2baf5192d5ed132974ed2fa1d"} Oct 31 04:26:30.135017 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.134992 5019 generic.go:356] "Generic (PLEG): container finished" podID="56aad6d5-8f29-4414-a5e9-f7cb565f8e57" containerID="5329525c6f8da3655e4eb5c265a21d8c999ecc1bf6582145b10216c9d042f097" exitCode=0 Oct 31 04:26:30.135241 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.135021 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt" event={"ID":"56aad6d5-8f29-4414-a5e9-f7cb565f8e57","Type":"ContainerDied","Data":"5329525c6f8da3655e4eb5c265a21d8c999ecc1bf6582145b10216c9d042f097"} Oct 31 04:26:30.178024 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.169845 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c" Oct 31 04:26:30.255723 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.255695 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt" Oct 31 04:26:30.285476 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.283321 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc" Oct 31 04:26:30.286790 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.286767 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-kube-api-access-tqwh6\" (UniqueName: \"kubernetes.io/projected/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-kube-api-access-tqwh6\") pod \"dda0567c-8a01-47e8-95f9-c0a4ad08a1c6\" (UID: \"dda0567c-8a01-47e8-95f9-c0a4ad08a1c6\") " Oct 31 04:26:30.286926 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.286824 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-istio-envoy\") pod \"dda0567c-8a01-47e8-95f9-c0a4ad08a1c6\" (UID: \"dda0567c-8a01-47e8-95f9-c0a4ad08a1c6\") " Oct 31 04:26:30.286926 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.286900 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-istio-data\") pod \"dda0567c-8a01-47e8-95f9-c0a4ad08a1c6\" (UID: \"dda0567c-8a01-47e8-95f9-c0a4ad08a1c6\") " Oct 31 04:26:30.287048 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.287031 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-istiod-ca-cert\") pod \"dda0567c-8a01-47e8-95f9-c0a4ad08a1c6\" (UID: \"dda0567c-8a01-47e8-95f9-c0a4ad08a1c6\") " Oct 31 04:26:30.287119 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.287104 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-workload-certs\") pod \"dda0567c-8a01-47e8-95f9-c0a4ad08a1c6\" (UID: \"dda0567c-8a01-47e8-95f9-c0a4ad08a1c6\") " Oct 31 04:26:30.287176 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.287156 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-istio-token\" (UniqueName: \"kubernetes.io/projected/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-istio-token\") pod \"dda0567c-8a01-47e8-95f9-c0a4ad08a1c6\" (UID: \"dda0567c-8a01-47e8-95f9-c0a4ad08a1c6\") " Oct 31 04:26:30.287241 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.287201 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-workload-socket\") pod \"dda0567c-8a01-47e8-95f9-c0a4ad08a1c6\" (UID: \"dda0567c-8a01-47e8-95f9-c0a4ad08a1c6\") " Oct 31 04:26:30.287314 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.287299 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-istio-podinfo\") pod \"dda0567c-8a01-47e8-95f9-c0a4ad08a1c6\" (UID: \"dda0567c-8a01-47e8-95f9-c0a4ad08a1c6\") " Oct 31 04:26:30.287409 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.287380 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-credential-socket\") pod \"dda0567c-8a01-47e8-95f9-c0a4ad08a1c6\" (UID: \"dda0567c-8a01-47e8-95f9-c0a4ad08a1c6\") " Oct 31 04:26:30.287565 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.287532 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-istio-data" pod "dda0567c-8a01-47e8-95f9-c0a4ad08a1c6" (UID: "dda0567c-8a01-47e8-95f9-c0a4ad08a1c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:26:30.287687 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.287663 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-istiod-ca-cert" pod "dda0567c-8a01-47e8-95f9-c0a4ad08a1c6" (UID: "dda0567c-8a01-47e8-95f9-c0a4ad08a1c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:26:30.287742 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.287724 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr" Oct 31 04:26:30.287797 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.287746 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-workload-certs" pod "dda0567c-8a01-47e8-95f9-c0a4ad08a1c6" (UID: "dda0567c-8a01-47e8-95f9-c0a4ad08a1c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:26:30.287892 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.287872 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-credential-socket" pod "dda0567c-8a01-47e8-95f9-c0a4ad08a1c6" (UID: "dda0567c-8a01-47e8-95f9-c0a4ad08a1c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:26:30.287991 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.287970 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-workload-socket" pod "dda0567c-8a01-47e8-95f9-c0a4ad08a1c6" (UID: "dda0567c-8a01-47e8-95f9-c0a4ad08a1c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:26:30.288101 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.288081 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:30.288153 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.288108 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:30.288153 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.288123 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:30.288153 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.288136 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:30.290488 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.290462 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-istio-podinfo" pod "dda0567c-8a01-47e8-95f9-c0a4ad08a1c6" (UID: "dda0567c-8a01-47e8-95f9-c0a4ad08a1c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:26:30.291770 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.291737 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-kube-api-access-tqwh6" pod "dda0567c-8a01-47e8-95f9-c0a4ad08a1c6" (UID: "dda0567c-8a01-47e8-95f9-c0a4ad08a1c6"). InnerVolumeSpecName "kube-api-access-tqwh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:26:30.292419 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.292395 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-istio-envoy" pod "dda0567c-8a01-47e8-95f9-c0a4ad08a1c6" (UID: "dda0567c-8a01-47e8-95f9-c0a4ad08a1c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:26:30.293103 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.293077 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-istio-token" pod "dda0567c-8a01-47e8-95f9-c0a4ad08a1c6" (UID: "dda0567c-8a01-47e8-95f9-c0a4ad08a1c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:26:30.389132 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.389093 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efc828fc-91bd-43f3-943d-144c3d9dc1b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efc828fc-91bd-43f3-943d-144c3d9dc1b1-istio-data\") pod \"efc828fc-91bd-43f3-943d-144c3d9dc1b1\" (UID: \"efc828fc-91bd-43f3-943d-144c3d9dc1b1\") " Oct 31 04:26:30.389132 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.389128 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2df6e942-8414-427c-83df-928e51465964-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2df6e942-8414-427c-83df-928e51465964-credential-socket\") pod \"2df6e942-8414-427c-83df-928e51465964\" (UID: \"2df6e942-8414-427c-83df-928e51465964\") " Oct 31 04:26:30.389467 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.389148 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2df6e942-8414-427c-83df-928e51465964-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2df6e942-8414-427c-83df-928e51465964-workload-socket\") pod \"2df6e942-8414-427c-83df-928e51465964\" (UID: \"2df6e942-8414-427c-83df-928e51465964\") " Oct 31 04:26:30.389467 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.389175 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-istio-data\") pod \"56aad6d5-8f29-4414-a5e9-f7cb565f8e57\" (UID: \"56aad6d5-8f29-4414-a5e9-f7cb565f8e57\") " Oct 31 04:26:30.389467 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.389212 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-workload-socket\") pod \"56aad6d5-8f29-4414-a5e9-f7cb565f8e57\" (UID: \"56aad6d5-8f29-4414-a5e9-f7cb565f8e57\") " Oct 31 04:26:30.389467 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.389239 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/efc828fc-91bd-43f3-943d-144c3d9dc1b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efc828fc-91bd-43f3-943d-144c3d9dc1b1-istio-podinfo\") pod \"efc828fc-91bd-43f3-943d-144c3d9dc1b1\" (UID: \"efc828fc-91bd-43f3-943d-144c3d9dc1b1\") " Oct 31 04:26:30.389467 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.389266 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efc828fc-91bd-43f3-943d-144c3d9dc1b1-istio-token\" (UniqueName: \"kubernetes.io/projected/efc828fc-91bd-43f3-943d-144c3d9dc1b1-istio-token\") pod \"efc828fc-91bd-43f3-943d-144c3d9dc1b1\" (UID: \"efc828fc-91bd-43f3-943d-144c3d9dc1b1\") " Oct 31 04:26:30.389467 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.389306 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efc828fc-91bd-43f3-943d-144c3d9dc1b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efc828fc-91bd-43f3-943d-144c3d9dc1b1-istiod-ca-cert\") pod \"efc828fc-91bd-43f3-943d-144c3d9dc1b1\" (UID: \"efc828fc-91bd-43f3-943d-144c3d9dc1b1\") " Oct 31 04:26:30.389467 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.389436 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2df6e942-8414-427c-83df-928e51465964-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2df6e942-8414-427c-83df-928e51465964-istio-podinfo\") pod \"2df6e942-8414-427c-83df-928e51465964\" (UID: \"2df6e942-8414-427c-83df-928e51465964\") " Oct 31 04:26:30.389834 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.389470 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2df6e942-8414-427c-83df-928e51465964-workload-socket" pod "2df6e942-8414-427c-83df-928e51465964" (UID: "2df6e942-8414-427c-83df-928e51465964"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:26:30.389834 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.389479 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-istio-podinfo\") pod \"56aad6d5-8f29-4414-a5e9-f7cb565f8e57\" (UID: \"56aad6d5-8f29-4414-a5e9-f7cb565f8e57\") " Oct 31 04:26:30.389834 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.389481 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2df6e942-8414-427c-83df-928e51465964-credential-socket" pod "2df6e942-8414-427c-83df-928e51465964" (UID: "2df6e942-8414-427c-83df-928e51465964"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:26:30.389834 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.389478 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efc828fc-91bd-43f3-943d-144c3d9dc1b1-istio-data" pod "efc828fc-91bd-43f3-943d-144c3d9dc1b1" (UID: "efc828fc-91bd-43f3-943d-144c3d9dc1b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:26:30.389834 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.389513 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-istiod-ca-cert\") pod \"56aad6d5-8f29-4414-a5e9-f7cb565f8e57\" (UID: \"56aad6d5-8f29-4414-a5e9-f7cb565f8e57\") " Oct 31 04:26:30.389834 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.389631 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-credential-socket\") pod \"56aad6d5-8f29-4414-a5e9-f7cb565f8e57\" (UID: \"56aad6d5-8f29-4414-a5e9-f7cb565f8e57\") " Oct 31 04:26:30.389834 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.389783 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efc828fc-91bd-43f3-943d-144c3d9dc1b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efc828fc-91bd-43f3-943d-144c3d9dc1b1-workload-certs\") pod \"efc828fc-91bd-43f3-943d-144c3d9dc1b1\" (UID: \"efc828fc-91bd-43f3-943d-144c3d9dc1b1\") " Oct 31 04:26:30.389834 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.389804 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-istiod-ca-cert" pod "56aad6d5-8f29-4414-a5e9-f7cb565f8e57" (UID: "56aad6d5-8f29-4414-a5e9-f7cb565f8e57"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:26:30.389834 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.389825 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2df6e942-8414-427c-83df-928e51465964-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2df6e942-8414-427c-83df-928e51465964-workload-certs\") pod \"2df6e942-8414-427c-83df-928e51465964\" (UID: \"2df6e942-8414-427c-83df-928e51465964\") " Oct 31 04:26:30.390276 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.389855 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2df6e942-8414-427c-83df-928e51465964-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2df6e942-8414-427c-83df-928e51465964-istio-data\") pod \"2df6e942-8414-427c-83df-928e51465964\" (UID: \"2df6e942-8414-427c-83df-928e51465964\") " Oct 31 04:26:30.390276 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.389892 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-kube-api-access-j8gd9\" (UniqueName: \"kubernetes.io/projected/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-kube-api-access-j8gd9\") pod \"56aad6d5-8f29-4414-a5e9-f7cb565f8e57\" (UID: \"56aad6d5-8f29-4414-a5e9-f7cb565f8e57\") " Oct 31 04:26:30.390276 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.389976 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-workload-certs\") pod \"56aad6d5-8f29-4414-a5e9-f7cb565f8e57\" (UID: \"56aad6d5-8f29-4414-a5e9-f7cb565f8e57\") " Oct 31 04:26:30.390276 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.390002 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efc828fc-91bd-43f3-943d-144c3d9dc1b1-kube-api-access-tbxb2\" (UniqueName: \"kubernetes.io/projected/efc828fc-91bd-43f3-943d-144c3d9dc1b1-kube-api-access-tbxb2\") pod \"efc828fc-91bd-43f3-943d-144c3d9dc1b1\" (UID: \"efc828fc-91bd-43f3-943d-144c3d9dc1b1\") " Oct 31 04:26:30.390276 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.390048 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efc828fc-91bd-43f3-943d-144c3d9dc1b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc828fc-91bd-43f3-943d-144c3d9dc1b1-workload-socket\") pod \"efc828fc-91bd-43f3-943d-144c3d9dc1b1\" (UID: \"efc828fc-91bd-43f3-943d-144c3d9dc1b1\") " Oct 31 04:26:30.390276 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.390110 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efc828fc-91bd-43f3-943d-144c3d9dc1b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc828fc-91bd-43f3-943d-144c3d9dc1b1-credential-socket\") pod \"efc828fc-91bd-43f3-943d-144c3d9dc1b1\" (UID: \"efc828fc-91bd-43f3-943d-144c3d9dc1b1\") " Oct 31 04:26:30.390276 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.390148 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efc828fc-91bd-43f3-943d-144c3d9dc1b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efc828fc-91bd-43f3-943d-144c3d9dc1b1-istio-envoy\") pod \"efc828fc-91bd-43f3-943d-144c3d9dc1b1\" (UID: \"efc828fc-91bd-43f3-943d-144c3d9dc1b1\") " Oct 31 04:26:30.390276 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.390182 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2df6e942-8414-427c-83df-928e51465964-kube-api-access-fksdw\" (UniqueName: \"kubernetes.io/projected/2df6e942-8414-427c-83df-928e51465964-kube-api-access-fksdw\") pod \"2df6e942-8414-427c-83df-928e51465964\" (UID: \"2df6e942-8414-427c-83df-928e51465964\") " Oct 31 04:26:30.390276 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.390208 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2df6e942-8414-427c-83df-928e51465964-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2df6e942-8414-427c-83df-928e51465964-istiod-ca-cert\") pod \"2df6e942-8414-427c-83df-928e51465964\" (UID: \"2df6e942-8414-427c-83df-928e51465964\") " Oct 31 04:26:30.390276 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.390254 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-istio-envoy\") pod \"56aad6d5-8f29-4414-a5e9-f7cb565f8e57\" (UID: \"56aad6d5-8f29-4414-a5e9-f7cb565f8e57\") " Oct 31 04:26:30.390276 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.390280 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-istio-token\" (UniqueName: \"kubernetes.io/projected/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-istio-token\") pod \"56aad6d5-8f29-4414-a5e9-f7cb565f8e57\" (UID: \"56aad6d5-8f29-4414-a5e9-f7cb565f8e57\") " Oct 31 04:26:30.391079 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.390309 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2df6e942-8414-427c-83df-928e51465964-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2df6e942-8414-427c-83df-928e51465964-istio-envoy\") pod \"2df6e942-8414-427c-83df-928e51465964\" (UID: \"2df6e942-8414-427c-83df-928e51465964\") " Oct 31 04:26:30.391079 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.390366 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-workload-socket" pod "56aad6d5-8f29-4414-a5e9-f7cb565f8e57" (UID: "56aad6d5-8f29-4414-a5e9-f7cb565f8e57"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:26:30.391079 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.390383 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2df6e942-8414-427c-83df-928e51465964-istio-token\" (UniqueName: \"kubernetes.io/projected/2df6e942-8414-427c-83df-928e51465964-istio-token\") pod \"2df6e942-8414-427c-83df-928e51465964\" (UID: \"2df6e942-8414-427c-83df-928e51465964\") " Oct 31 04:26:30.391079 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.390550 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efc828fc-91bd-43f3-943d-144c3d9dc1b1-workload-certs" pod "efc828fc-91bd-43f3-943d-144c3d9dc1b1" (UID: "efc828fc-91bd-43f3-943d-144c3d9dc1b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:26:30.391079 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.390725 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-credential-socket" pod "56aad6d5-8f29-4414-a5e9-f7cb565f8e57" (UID: "56aad6d5-8f29-4414-a5e9-f7cb565f8e57"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:26:30.391325 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.391144 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2df6e942-8414-427c-83df-928e51465964-istio-data" pod "2df6e942-8414-427c-83df-928e51465964" (UID: "2df6e942-8414-427c-83df-928e51465964"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:26:30.391325 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.391173 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqwh6\" (UniqueName: \"kubernetes.io/projected/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-kube-api-access-tqwh6\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:30.391325 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.391198 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:30.391325 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.391213 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:30.391325 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.391226 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:30.391325 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.391242 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efc828fc-91bd-43f3-943d-144c3d9dc1b1-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:30.391325 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.391255 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:30.391325 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.391270 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:30.391325 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.391283 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efc828fc-91bd-43f3-943d-144c3d9dc1b1-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:30.391325 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.391296 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2df6e942-8414-427c-83df-928e51465964-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:30.391325 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.391310 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2df6e942-8414-427c-83df-928e51465964-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:30.391325 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.391312 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2df6e942-8414-427c-83df-928e51465964-workload-certs" pod "2df6e942-8414-427c-83df-928e51465964" (UID: "2df6e942-8414-427c-83df-928e51465964"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:26:30.391325 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.391323 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:30.391975 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.391355 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:30.393585 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.393554 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-istio-data" pod "56aad6d5-8f29-4414-a5e9-f7cb565f8e57" (UID: "56aad6d5-8f29-4414-a5e9-f7cb565f8e57"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:26:30.394084 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.394060 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efc828fc-91bd-43f3-943d-144c3d9dc1b1-kube-api-access-tbxb2" pod "efc828fc-91bd-43f3-943d-144c3d9dc1b1" (UID: "efc828fc-91bd-43f3-943d-144c3d9dc1b1"). InnerVolumeSpecName "kube-api-access-tbxb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:26:30.394632 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.394611 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2df6e942-8414-427c-83df-928e51465964-kube-api-access-fksdw" pod "2df6e942-8414-427c-83df-928e51465964" (UID: "2df6e942-8414-427c-83df-928e51465964"). InnerVolumeSpecName "kube-api-access-fksdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:26:30.395056 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.395032 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2df6e942-8414-427c-83df-928e51465964-istiod-ca-cert" pod "2df6e942-8414-427c-83df-928e51465964" (UID: "2df6e942-8414-427c-83df-928e51465964"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:26:30.396555 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.396513 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efc828fc-91bd-43f3-943d-144c3d9dc1b1-istio-envoy" pod "efc828fc-91bd-43f3-943d-144c3d9dc1b1" (UID: "efc828fc-91bd-43f3-943d-144c3d9dc1b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:26:30.396755 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.396734 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efc828fc-91bd-43f3-943d-144c3d9dc1b1-workload-socket" pod "efc828fc-91bd-43f3-943d-144c3d9dc1b1" (UID: "efc828fc-91bd-43f3-943d-144c3d9dc1b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:26:30.396947 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.396926 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efc828fc-91bd-43f3-943d-144c3d9dc1b1-credential-socket" pod "efc828fc-91bd-43f3-943d-144c3d9dc1b1" (UID: "efc828fc-91bd-43f3-943d-144c3d9dc1b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:26:30.403725 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.397296 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-istio-envoy" pod "56aad6d5-8f29-4414-a5e9-f7cb565f8e57" (UID: "56aad6d5-8f29-4414-a5e9-f7cb565f8e57"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:26:30.403725 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.398270 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2df6e942-8414-427c-83df-928e51465964-istio-podinfo" pod "2df6e942-8414-427c-83df-928e51465964" (UID: "2df6e942-8414-427c-83df-928e51465964"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:26:30.403905 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.399661 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-kube-api-access-j8gd9" pod "56aad6d5-8f29-4414-a5e9-f7cb565f8e57" (UID: "56aad6d5-8f29-4414-a5e9-f7cb565f8e57"). InnerVolumeSpecName "kube-api-access-j8gd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:26:30.403967 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.399824 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-workload-certs" pod "56aad6d5-8f29-4414-a5e9-f7cb565f8e57" (UID: "56aad6d5-8f29-4414-a5e9-f7cb565f8e57"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:26:30.403967 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.400873 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-istio-token" pod "56aad6d5-8f29-4414-a5e9-f7cb565f8e57" (UID: "56aad6d5-8f29-4414-a5e9-f7cb565f8e57"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:26:30.403967 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.401144 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efc828fc-91bd-43f3-943d-144c3d9dc1b1-istiod-ca-cert" pod "efc828fc-91bd-43f3-943d-144c3d9dc1b1" (UID: "efc828fc-91bd-43f3-943d-144c3d9dc1b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:26:30.403967 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.403285 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-istio-podinfo" pod "56aad6d5-8f29-4414-a5e9-f7cb565f8e57" (UID: "56aad6d5-8f29-4414-a5e9-f7cb565f8e57"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:26:30.404702 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.404678 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/efc828fc-91bd-43f3-943d-144c3d9dc1b1-istio-podinfo" pod "efc828fc-91bd-43f3-943d-144c3d9dc1b1" (UID: "efc828fc-91bd-43f3-943d-144c3d9dc1b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:26:30.405080 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.405046 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efc828fc-91bd-43f3-943d-144c3d9dc1b1-istio-token" pod "efc828fc-91bd-43f3-943d-144c3d9dc1b1" (UID: "efc828fc-91bd-43f3-943d-144c3d9dc1b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:26:30.405925 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.405899 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2df6e942-8414-427c-83df-928e51465964-istio-token" pod "2df6e942-8414-427c-83df-928e51465964" (UID: "2df6e942-8414-427c-83df-928e51465964"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:26:30.406206 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.406185 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2df6e942-8414-427c-83df-928e51465964-istio-envoy" pod "2df6e942-8414-427c-83df-928e51465964" (UID: "2df6e942-8414-427c-83df-928e51465964"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:26:30.492374 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.492258 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2df6e942-8414-427c-83df-928e51465964-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:30.492374 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.492289 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2df6e942-8414-427c-83df-928e51465964-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:30.492374 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.492298 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8gd9\" (UniqueName: \"kubernetes.io/projected/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-kube-api-access-j8gd9\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:30.492374 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.492308 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:30.492374 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.492317 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbxb2\" (UniqueName: \"kubernetes.io/projected/efc828fc-91bd-43f3-943d-144c3d9dc1b1-kube-api-access-tbxb2\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:30.492374 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.492326 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc828fc-91bd-43f3-943d-144c3d9dc1b1-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:30.492374 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.492380 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc828fc-91bd-43f3-943d-144c3d9dc1b1-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:30.492702 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.492393 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efc828fc-91bd-43f3-943d-144c3d9dc1b1-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:30.492702 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.492406 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fksdw\" (UniqueName: \"kubernetes.io/projected/2df6e942-8414-427c-83df-928e51465964-kube-api-access-fksdw\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:30.492702 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.492416 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2df6e942-8414-427c-83df-928e51465964-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:30.492702 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.492424 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:30.492702 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.492432 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:30.492702 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.492442 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2df6e942-8414-427c-83df-928e51465964-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:30.492702 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.492449 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2df6e942-8414-427c-83df-928e51465964-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:30.492702 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.492456 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:30.492702 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.492464 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efc828fc-91bd-43f3-943d-144c3d9dc1b1-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:30.492702 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.492471 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efc828fc-91bd-43f3-943d-144c3d9dc1b1-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:30.492702 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.492478 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efc828fc-91bd-43f3-943d-144c3d9dc1b1-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:30.492702 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.492486 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2df6e942-8414-427c-83df-928e51465964-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:30.492702 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:30.492494 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56aad6d5-8f29-4414-a5e9-f7cb565f8e57-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:31.143509 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:31.143471 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c" event={"ID":"dda0567c-8a01-47e8-95f9-c0a4ad08a1c6","Type":"ContainerDied","Data":"de159798d731af5201f266074bbf41e5e3898d063b98bc7fae3efd7851a70e19"} Oct 31 04:26:31.143965 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:31.143483 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c" Oct 31 04:26:31.143965 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:31.143543 5019 scope.go:122] "RemoveContainer" containerID="41d29e3aedb2beba64a8b693cc2881be01db94eeecc9604fa8af1854e39bc023" Oct 31 04:26:31.145189 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:31.145168 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr" Oct 31 04:26:31.145277 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:31.145184 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr" event={"ID":"2df6e942-8414-427c-83df-928e51465964","Type":"ContainerDied","Data":"0f1a20fe4e1b53dc5097c8775cbb9e935a4465e8e6af55c8f7debc5f4ee2656b"} Oct 31 04:26:31.147003 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:31.146981 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc" Oct 31 04:26:31.147142 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:31.147116 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc" event={"ID":"efc828fc-91bd-43f3-943d-144c3d9dc1b1","Type":"ContainerDied","Data":"2e2bf00e73f26f882b6c6e2496e31d79ea2cac8f93f0a3289df69e5f12776f33"} Oct 31 04:26:31.148830 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:31.148804 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt" Oct 31 04:26:31.148923 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:31.148831 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt" event={"ID":"56aad6d5-8f29-4414-a5e9-f7cb565f8e57","Type":"ContainerDied","Data":"01ceb2f396e1ab3d16e40b8b9afca53323d7676513d1f9d60e8a444ea37ba994"} Oct 31 04:26:31.156401 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:31.156274 5019 scope.go:122] "RemoveContainer" containerID="6f5f8311d71344629e668869636e95d440616d983fd51f0d949b4e823edeba0e" Oct 31 04:26:31.167565 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:31.167545 5019 scope.go:122] "RemoveContainer" containerID="a1ae8aa1b3d05374777fa0e7524e4d32d8947ef2baf5192d5ed132974ed2fa1d" Oct 31 04:26:31.176697 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:31.176679 5019 scope.go:122] "RemoveContainer" containerID="5329525c6f8da3655e4eb5c265a21d8c999ecc1bf6582145b10216c9d042f097" Oct 31 04:26:31.260395 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:31.260353 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt"] Oct 31 04:26:31.266164 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:31.266132 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rgfnt"] Oct 31 04:26:31.305526 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:31.305487 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc"] Oct 31 04:26:31.309766 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:31.309739 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-5xqkc"] Oct 31 04:26:31.349385 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:31.349351 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr"] Oct 31 04:26:31.354863 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:31.354841 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-gd8rr"] Oct 31 04:26:31.389969 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:31.389939 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c"] Oct 31 04:26:31.396593 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:31.396525 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-ghc4c"] Oct 31 04:26:31.408597 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:31.408566 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2df6e942-8414-427c-83df-928e51465964" path="/var/lib/kubelet/pods/2df6e942-8414-427c-83df-928e51465964/volumes" Oct 31 04:26:31.409079 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:31.409063 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56aad6d5-8f29-4414-a5e9-f7cb565f8e57" path="/var/lib/kubelet/pods/56aad6d5-8f29-4414-a5e9-f7cb565f8e57/volumes" Oct 31 04:26:31.409545 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:31.409529 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dda0567c-8a01-47e8-95f9-c0a4ad08a1c6" path="/var/lib/kubelet/pods/dda0567c-8a01-47e8-95f9-c0a4ad08a1c6/volumes" Oct 31 04:26:31.410286 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:31.410272 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efc828fc-91bd-43f3-943d-144c3d9dc1b1" path="/var/lib/kubelet/pods/efc828fc-91bd-43f3-943d-144c3d9dc1b1/volumes" Oct 31 04:26:34.055564 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:34.055530 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff43e3-5d974c979f-gz8bs" Oct 31 04:26:36.823595 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:36.823562 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ceba51-9776c4ffb-vddkl"] Oct 31 04:26:36.824047 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:36.823765 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-ceba51-9776c4ffb-vddkl" podUID="9bda0a48-7e59-4b49-a555-ba3d7c2d6fe7" containerName="payload-processing" containerID="cri-o://b845341f9712881c3562b23304768f72f641e1ee11865158a230e336d08ea91a" gracePeriod=30 Oct 31 04:26:37.287028 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:37.285786 5019 generic.go:356] "Generic (PLEG): container finished" podID="9bda0a48-7e59-4b49-a555-ba3d7c2d6fe7" containerID="b845341f9712881c3562b23304768f72f641e1ee11865158a230e336d08ea91a" exitCode=0 Oct 31 04:26:37.287028 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:37.285999 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ceba51-9776c4ffb-vddkl" event={"ID":"9bda0a48-7e59-4b49-a555-ba3d7c2d6fe7","Type":"ContainerDied","Data":"b845341f9712881c3562b23304768f72f641e1ee11865158a230e336d08ea91a"} Oct 31 04:26:37.287028 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:37.286054 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ceba51-9776c4ffb-vddkl" event={"ID":"9bda0a48-7e59-4b49-a555-ba3d7c2d6fe7","Type":"ContainerDied","Data":"670157b238931767fcd2309e2d5191c26a0f8135c1aa1f30ade276bca1759358"} Oct 31 04:26:37.287028 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:37.286082 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="670157b238931767fcd2309e2d5191c26a0f8135c1aa1f30ade276bca1759358" Oct 31 04:26:37.287419 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:37.287167 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ceba51-9776c4ffb-vddkl" Oct 31 04:26:37.413032 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:37.412925 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bda0a48-7e59-4b49-a555-ba3d7c2d6fe7-kube-api-access-m2l6q\" (UniqueName: \"kubernetes.io/projected/9bda0a48-7e59-4b49-a555-ba3d7c2d6fe7-kube-api-access-m2l6q\") pod \"9bda0a48-7e59-4b49-a555-ba3d7c2d6fe7\" (UID: \"9bda0a48-7e59-4b49-a555-ba3d7c2d6fe7\") " Oct 31 04:26:37.413222 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:37.413176 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bda0a48-7e59-4b49-a555-ba3d7c2d6fe7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bda0a48-7e59-4b49-a555-ba3d7c2d6fe7-plugins-config-volume\") pod \"9bda0a48-7e59-4b49-a555-ba3d7c2d6fe7\" (UID: \"9bda0a48-7e59-4b49-a555-ba3d7c2d6fe7\") " Oct 31 04:26:37.413831 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:37.413804 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bda0a48-7e59-4b49-a555-ba3d7c2d6fe7-plugins-config-volume" pod "9bda0a48-7e59-4b49-a555-ba3d7c2d6fe7" (UID: "9bda0a48-7e59-4b49-a555-ba3d7c2d6fe7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:26:37.415478 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:37.415451 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bda0a48-7e59-4b49-a555-ba3d7c2d6fe7-kube-api-access-m2l6q" pod "9bda0a48-7e59-4b49-a555-ba3d7c2d6fe7" (UID: "9bda0a48-7e59-4b49-a555-ba3d7c2d6fe7"). InnerVolumeSpecName "kube-api-access-m2l6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:26:37.515311 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:37.515270 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2l6q\" (UniqueName: \"kubernetes.io/projected/9bda0a48-7e59-4b49-a555-ba3d7c2d6fe7-kube-api-access-m2l6q\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:37.515311 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:37.515312 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bda0a48-7e59-4b49-a555-ba3d7c2d6fe7-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:38.295901 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:38.295863 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ceba51-9776c4ffb-vddkl" Oct 31 04:26:38.360544 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:38.360500 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ceba51-9776c4ffb-vddkl"] Oct 31 04:26:38.372269 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:38.372234 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ceba51-9776c4ffb-vddkl"] Oct 31 04:26:39.410928 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:39.410891 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bda0a48-7e59-4b49-a555-ba3d7c2d6fe7" path="/var/lib/kubelet/pods/9bda0a48-7e59-4b49-a555-ba3d7c2d6fe7/volumes" Oct 31 04:26:40.114370 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:40.114325 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942"] Oct 31 04:26:40.114604 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:40.114582 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942" podUID="2ce06671-540a-43bb-8f84-a86082d76ca9" containerName="istio-proxy" containerID="cri-o://1bef70deeb305caf75822933c41b35551728f0b965372ea9f99fab4a563b8ad9" gracePeriod=30 Oct 31 04:26:45.385434 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:45.385286 5019 generic.go:356] "Generic (PLEG): container finished" podID="2ce06671-540a-43bb-8f84-a86082d76ca9" containerID="1bef70deeb305caf75822933c41b35551728f0b965372ea9f99fab4a563b8ad9" exitCode=0 Oct 31 04:26:45.385895 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:45.385616 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942" event={"ID":"2ce06671-540a-43bb-8f84-a86082d76ca9","Type":"ContainerDied","Data":"1bef70deeb305caf75822933c41b35551728f0b965372ea9f99fab4a563b8ad9"} Oct 31 04:26:45.385895 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:45.385674 5019 scope.go:122] "RemoveContainer" containerID="3093e7549aa7ca9d878bb0fe4e0c4e3d8d6fce4acc7e1dc854ada043b30318b8" Oct 31 04:26:45.447171 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:45.447145 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942" Oct 31 04:26:45.536859 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:45.536823 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ce06671-540a-43bb-8f84-a86082d76ca9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ce06671-540a-43bb-8f84-a86082d76ca9-workload-certs\") pod \"2ce06671-540a-43bb-8f84-a86082d76ca9\" (UID: \"2ce06671-540a-43bb-8f84-a86082d76ca9\") " Oct 31 04:26:45.537023 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:45.536891 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ce06671-540a-43bb-8f84-a86082d76ca9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce06671-540a-43bb-8f84-a86082d76ca9-workload-socket\") pod \"2ce06671-540a-43bb-8f84-a86082d76ca9\" (UID: \"2ce06671-540a-43bb-8f84-a86082d76ca9\") " Oct 31 04:26:45.537023 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:45.536955 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ce06671-540a-43bb-8f84-a86082d76ca9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ce06671-540a-43bb-8f84-a86082d76ca9-istio-envoy\") pod \"2ce06671-540a-43bb-8f84-a86082d76ca9\" (UID: \"2ce06671-540a-43bb-8f84-a86082d76ca9\") " Oct 31 04:26:45.537146 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:45.537034 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ce06671-540a-43bb-8f84-a86082d76ca9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ce06671-540a-43bb-8f84-a86082d76ca9-istio-podinfo\") pod \"2ce06671-540a-43bb-8f84-a86082d76ca9\" (UID: \"2ce06671-540a-43bb-8f84-a86082d76ca9\") " Oct 31 04:26:45.537146 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:45.537069 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ce06671-540a-43bb-8f84-a86082d76ca9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce06671-540a-43bb-8f84-a86082d76ca9-credential-socket\") pod \"2ce06671-540a-43bb-8f84-a86082d76ca9\" (UID: \"2ce06671-540a-43bb-8f84-a86082d76ca9\") " Oct 31 04:26:45.537254 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:45.537148 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ce06671-540a-43bb-8f84-a86082d76ca9-workload-certs" pod "2ce06671-540a-43bb-8f84-a86082d76ca9" (UID: "2ce06671-540a-43bb-8f84-a86082d76ca9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:26:45.537254 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:45.537194 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ce06671-540a-43bb-8f84-a86082d76ca9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ce06671-540a-43bb-8f84-a86082d76ca9-istio-data\") pod \"2ce06671-540a-43bb-8f84-a86082d76ca9\" (UID: \"2ce06671-540a-43bb-8f84-a86082d76ca9\") " Oct 31 04:26:45.537254 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:45.537215 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ce06671-540a-43bb-8f84-a86082d76ca9-workload-socket" pod "2ce06671-540a-43bb-8f84-a86082d76ca9" (UID: "2ce06671-540a-43bb-8f84-a86082d76ca9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:26:45.537254 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:45.537228 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ce06671-540a-43bb-8f84-a86082d76ca9-kube-api-access-57szj\" (UniqueName: \"kubernetes.io/projected/2ce06671-540a-43bb-8f84-a86082d76ca9-kube-api-access-57szj\") pod \"2ce06671-540a-43bb-8f84-a86082d76ca9\" (UID: \"2ce06671-540a-43bb-8f84-a86082d76ca9\") " Oct 31 04:26:45.537451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:45.537258 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ce06671-540a-43bb-8f84-a86082d76ca9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ce06671-540a-43bb-8f84-a86082d76ca9-istiod-ca-cert\") pod \"2ce06671-540a-43bb-8f84-a86082d76ca9\" (UID: \"2ce06671-540a-43bb-8f84-a86082d76ca9\") " Oct 31 04:26:45.537451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:45.537368 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ce06671-540a-43bb-8f84-a86082d76ca9-istio-token\" (UniqueName: \"kubernetes.io/projected/2ce06671-540a-43bb-8f84-a86082d76ca9-istio-token\") pod \"2ce06671-540a-43bb-8f84-a86082d76ca9\" (UID: \"2ce06671-540a-43bb-8f84-a86082d76ca9\") " Oct 31 04:26:45.537451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:45.537379 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ce06671-540a-43bb-8f84-a86082d76ca9-istio-data" pod "2ce06671-540a-43bb-8f84-a86082d76ca9" (UID: "2ce06671-540a-43bb-8f84-a86082d76ca9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:26:45.537711 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:45.537690 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ce06671-540a-43bb-8f84-a86082d76ca9-istiod-ca-cert" pod "2ce06671-540a-43bb-8f84-a86082d76ca9" (UID: "2ce06671-540a-43bb-8f84-a86082d76ca9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:26:45.537807 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:45.537767 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ce06671-540a-43bb-8f84-a86082d76ca9-credential-socket" pod "2ce06671-540a-43bb-8f84-a86082d76ca9" (UID: "2ce06671-540a-43bb-8f84-a86082d76ca9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:26:45.538073 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:45.538049 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce06671-540a-43bb-8f84-a86082d76ca9-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:45.538192 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:45.538075 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce06671-540a-43bb-8f84-a86082d76ca9-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:45.538192 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:45.538092 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ce06671-540a-43bb-8f84-a86082d76ca9-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:45.538192 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:45.538107 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ce06671-540a-43bb-8f84-a86082d76ca9-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:45.538192 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:45.538122 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ce06671-540a-43bb-8f84-a86082d76ca9-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:45.540022 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:45.539996 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ce06671-540a-43bb-8f84-a86082d76ca9-kube-api-access-57szj" pod "2ce06671-540a-43bb-8f84-a86082d76ca9" (UID: "2ce06671-540a-43bb-8f84-a86082d76ca9"). InnerVolumeSpecName "kube-api-access-57szj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:26:45.540442 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:45.540415 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ce06671-540a-43bb-8f84-a86082d76ca9-istio-token" pod "2ce06671-540a-43bb-8f84-a86082d76ca9" (UID: "2ce06671-540a-43bb-8f84-a86082d76ca9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:26:45.540577 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:45.540494 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ce06671-540a-43bb-8f84-a86082d76ca9-istio-envoy" pod "2ce06671-540a-43bb-8f84-a86082d76ca9" (UID: "2ce06671-540a-43bb-8f84-a86082d76ca9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:26:45.540853 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:45.540833 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ce06671-540a-43bb-8f84-a86082d76ca9-istio-podinfo" pod "2ce06671-540a-43bb-8f84-a86082d76ca9" (UID: "2ce06671-540a-43bb-8f84-a86082d76ca9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:26:45.639233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:45.639144 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ce06671-540a-43bb-8f84-a86082d76ca9-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:45.639233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:45.639175 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ce06671-540a-43bb-8f84-a86082d76ca9-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:45.639233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:45.639189 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57szj\" (UniqueName: \"kubernetes.io/projected/2ce06671-540a-43bb-8f84-a86082d76ca9-kube-api-access-57szj\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:45.639233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:45.639199 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ce06671-540a-43bb-8f84-a86082d76ca9-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:46.421718 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:46.421675 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942" event={"ID":"2ce06671-540a-43bb-8f84-a86082d76ca9","Type":"ContainerDied","Data":"3fd486fdeb401496802b9fecb1ab43869ae6cf0f262b0ff0b871ac15bd51fde2"} Oct 31 04:26:46.422321 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:46.421744 5019 scope.go:122] "RemoveContainer" containerID="1bef70deeb305caf75822933c41b35551728f0b965372ea9f99fab4a563b8ad9" Oct 31 04:26:46.422321 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:46.421890 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942" Oct 31 04:26:46.500632 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:46.500604 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942"] Oct 31 04:26:46.514406 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:46.514377 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-ceba51-openshift-default-d46fffd85-jw942"] Oct 31 04:26:47.408561 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:47.408525 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ce06671-540a-43bb-8f84-a86082d76ca9" path="/var/lib/kubelet/pods/2ce06671-540a-43bb-8f84-a86082d76ca9/volumes" Oct 31 04:26:52.394154 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:26:52.394110 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:26:54.028464 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:54.028427 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2a7c5b1-dj9cz"] Oct 31 04:26:54.040460 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:54.040428 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2a7c5b1-dj9cz"] Oct 31 04:26:54.680252 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:54.680209 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a9b573-75678d44f9-l4cmb"] Oct 31 04:26:54.680549 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:54.680513 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9b573-75678d44f9-l4cmb" podUID="7bfeaaa0-ebe7-49cb-9868-d02bed5e1240" containerName="payload-pre-processing" containerID="cri-o://392310e6a7cee5a5bd369045329ba36b11b36d48d50e47a1b8b8ad45af1af542" gracePeriod=30 Oct 31 04:26:55.048427 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:55.048385 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b56d736e-7s7nz"] Oct 31 04:26:55.054381 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:55.054279 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b56d736e-7s7nz"] Oct 31 04:26:55.085385 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:55.085190 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9b573-75678d44f9-l4cmb" Oct 31 04:26:55.196205 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:55.196174 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bfeaaa0-ebe7-49cb-9868-d02bed5e1240-kube-api-access-l7227\" (UniqueName: \"kubernetes.io/projected/7bfeaaa0-ebe7-49cb-9868-d02bed5e1240-kube-api-access-l7227\") pod \"7bfeaaa0-ebe7-49cb-9868-d02bed5e1240\" (UID: \"7bfeaaa0-ebe7-49cb-9868-d02bed5e1240\") " Oct 31 04:26:55.196478 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:55.196420 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bfeaaa0-ebe7-49cb-9868-d02bed5e1240-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bfeaaa0-ebe7-49cb-9868-d02bed5e1240-plugins-config-volume\") pod \"7bfeaaa0-ebe7-49cb-9868-d02bed5e1240\" (UID: \"7bfeaaa0-ebe7-49cb-9868-d02bed5e1240\") " Oct 31 04:26:55.196983 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:55.196892 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bfeaaa0-ebe7-49cb-9868-d02bed5e1240-plugins-config-volume" pod "7bfeaaa0-ebe7-49cb-9868-d02bed5e1240" (UID: "7bfeaaa0-ebe7-49cb-9868-d02bed5e1240"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:26:55.198855 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:55.198755 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bfeaaa0-ebe7-49cb-9868-d02bed5e1240-kube-api-access-l7227" pod "7bfeaaa0-ebe7-49cb-9868-d02bed5e1240" (UID: "7bfeaaa0-ebe7-49cb-9868-d02bed5e1240"). InnerVolumeSpecName "kube-api-access-l7227". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:26:55.298019 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:55.297929 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7227\" (UniqueName: \"kubernetes.io/projected/7bfeaaa0-ebe7-49cb-9868-d02bed5e1240-kube-api-access-l7227\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:55.298019 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:55.297965 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bfeaaa0-ebe7-49cb-9868-d02bed5e1240-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:26:55.410091 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:55.410055 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a5bf674-97e5-4f45-be51-190d9fc24a71" path="/var/lib/kubelet/pods/5a5bf674-97e5-4f45-be51-190d9fc24a71/volumes" Oct 31 04:26:55.410537 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:55.410522 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee3bb425-3793-4d03-b0b9-d3a0025e3b90" path="/var/lib/kubelet/pods/ee3bb425-3793-4d03-b0b9-d3a0025e3b90/volumes" Oct 31 04:26:55.494495 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:55.494417 5019 generic.go:356] "Generic (PLEG): container finished" podID="7bfeaaa0-ebe7-49cb-9868-d02bed5e1240" containerID="392310e6a7cee5a5bd369045329ba36b11b36d48d50e47a1b8b8ad45af1af542" exitCode=0 Oct 31 04:26:55.494495 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:55.494471 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9b573-75678d44f9-l4cmb" Oct 31 04:26:55.494495 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:55.494471 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9b573-75678d44f9-l4cmb" event={"ID":"7bfeaaa0-ebe7-49cb-9868-d02bed5e1240","Type":"ContainerDied","Data":"392310e6a7cee5a5bd369045329ba36b11b36d48d50e47a1b8b8ad45af1af542"} Oct 31 04:26:55.495270 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:55.494627 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9b573-75678d44f9-l4cmb" event={"ID":"7bfeaaa0-ebe7-49cb-9868-d02bed5e1240","Type":"ContainerDied","Data":"acab2575e7349170b3e15e911ec1d7ffba5dec1542f241c479be664410c4ead7"} Oct 31 04:26:55.495270 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:55.494668 5019 scope.go:122] "RemoveContainer" containerID="392310e6a7cee5a5bd369045329ba36b11b36d48d50e47a1b8b8ad45af1af542" Oct 31 04:26:55.509554 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:55.509453 5019 scope.go:122] "RemoveContainer" containerID="392310e6a7cee5a5bd369045329ba36b11b36d48d50e47a1b8b8ad45af1af542" Oct 31 04:26:55.510594 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:26:55.510562 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"392310e6a7cee5a5bd369045329ba36b11b36d48d50e47a1b8b8ad45af1af542\": container with ID starting with 392310e6a7cee5a5bd369045329ba36b11b36d48d50e47a1b8b8ad45af1af542 not found: ID does not exist" containerID="392310e6a7cee5a5bd369045329ba36b11b36d48d50e47a1b8b8ad45af1af542" Oct 31 04:26:55.510712 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:55.510600 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"392310e6a7cee5a5bd369045329ba36b11b36d48d50e47a1b8b8ad45af1af542"} err="failed to get container status \"392310e6a7cee5a5bd369045329ba36b11b36d48d50e47a1b8b8ad45af1af542\": rpc error: code = NotFound desc = could not find container \"392310e6a7cee5a5bd369045329ba36b11b36d48d50e47a1b8b8ad45af1af542\": container with ID starting with 392310e6a7cee5a5bd369045329ba36b11b36d48d50e47a1b8b8ad45af1af542 not found: ID does not exist" Oct 31 04:26:55.548062 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:55.547997 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a9b573-75678d44f9-l4cmb"] Oct 31 04:26:55.554753 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:55.554723 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a9b573-75678d44f9-l4cmb"] Oct 31 04:26:56.249206 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:56.249168 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp"] Oct 31 04:26:56.249674 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:56.249469 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp" podUID="0712577c-fb53-4025-9d1e-6c62e6bf38cb" containerName="istio-proxy" containerID="cri-o://0d2c1812809a4e09a55120bd6a6b28287556cb8153fa32ac151bdd9a9cb739d1" gracePeriod=30 Oct 31 04:26:57.409388 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:57.409351 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bfeaaa0-ebe7-49cb-9868-d02bed5e1240" path="/var/lib/kubelet/pods/7bfeaaa0-ebe7-49cb-9868-d02bed5e1240/volumes" Oct 31 04:26:58.166378 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.165651 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g"] Oct 31 04:26:58.166656 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.166630 5019 cpu_manager.go:414] "containerMap: removing container" podUID="2df6e942-8414-427c-83df-928e51465964" containerName="istio-proxy" Oct 31 04:26:58.166740 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.166660 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2df6e942-8414-427c-83df-928e51465964" containerName="istio-proxy" Oct 31 04:26:58.166740 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.166688 5019 cpu_manager.go:414] "containerMap: removing container" podUID="dda0567c-8a01-47e8-95f9-c0a4ad08a1c6" containerName="istio-proxy" Oct 31 04:26:58.166740 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.166696 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dda0567c-8a01-47e8-95f9-c0a4ad08a1c6" containerName="istio-proxy" Oct 31 04:26:58.166740 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.166706 5019 cpu_manager.go:414] "containerMap: removing container" podUID="9bda0a48-7e59-4b49-a555-ba3d7c2d6fe7" containerName="payload-processing" Oct 31 04:26:58.166740 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.166715 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bda0a48-7e59-4b49-a555-ba3d7c2d6fe7" containerName="payload-processing" Oct 31 04:26:58.166980 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.166751 5019 cpu_manager.go:414] "containerMap: removing container" podUID="56aad6d5-8f29-4414-a5e9-f7cb565f8e57" containerName="istio-proxy" Oct 31 04:26:58.166980 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.166759 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56aad6d5-8f29-4414-a5e9-f7cb565f8e57" containerName="istio-proxy" Oct 31 04:26:58.166980 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.166771 5019 cpu_manager.go:414] "containerMap: removing container" podUID="7bfeaaa0-ebe7-49cb-9868-d02bed5e1240" containerName="payload-pre-processing" Oct 31 04:26:58.166980 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.166780 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bfeaaa0-ebe7-49cb-9868-d02bed5e1240" containerName="payload-pre-processing" Oct 31 04:26:58.166980 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.166800 5019 cpu_manager.go:414] "containerMap: removing container" podUID="2ce06671-540a-43bb-8f84-a86082d76ca9" containerName="istio-proxy" Oct 31 04:26:58.166980 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.166806 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ce06671-540a-43bb-8f84-a86082d76ca9" containerName="istio-proxy" Oct 31 04:26:58.166980 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.166818 5019 cpu_manager.go:414] "containerMap: removing container" podUID="2ce06671-540a-43bb-8f84-a86082d76ca9" containerName="istio-proxy" Oct 31 04:26:58.166980 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.166825 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ce06671-540a-43bb-8f84-a86082d76ca9" containerName="istio-proxy" Oct 31 04:26:58.166980 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.166844 5019 cpu_manager.go:414] "containerMap: removing container" podUID="efc828fc-91bd-43f3-943d-144c3d9dc1b1" containerName="istio-proxy" Oct 31 04:26:58.166980 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.166850 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efc828fc-91bd-43f3-943d-144c3d9dc1b1" containerName="istio-proxy" Oct 31 04:26:58.167452 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.167014 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="2df6e942-8414-427c-83df-928e51465964" containerName="istio-proxy" Oct 31 04:26:58.167452 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.167031 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="dda0567c-8a01-47e8-95f9-c0a4ad08a1c6" containerName="istio-proxy" Oct 31 04:26:58.167452 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.167039 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bda0a48-7e59-4b49-a555-ba3d7c2d6fe7" containerName="payload-processing" Oct 31 04:26:58.167452 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.167052 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="56aad6d5-8f29-4414-a5e9-f7cb565f8e57" containerName="istio-proxy" Oct 31 04:26:58.167452 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.167060 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bfeaaa0-ebe7-49cb-9868-d02bed5e1240" containerName="payload-pre-processing" Oct 31 04:26:58.167452 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.167070 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ce06671-540a-43bb-8f84-a86082d76ca9" containerName="istio-proxy" Oct 31 04:26:58.167452 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.167080 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ce06671-540a-43bb-8f84-a86082d76ca9" containerName="istio-proxy" Oct 31 04:26:58.167452 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.167089 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="efc828fc-91bd-43f3-943d-144c3d9dc1b1" containerName="istio-proxy" Oct 31 04:26:58.183274 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.183243 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g" Oct 31 04:26:58.211509 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.211470 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-9ee727-openshift-default-dockercfg-hpcjw\"" Oct 31 04:26:58.218187 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.218153 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g"] Oct 31 04:26:58.236517 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.236462 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0165c2c6-76a2-42fc-a3f4-50674c99bc88-workload-socket\") pod \"e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g\" (UID: \"0165c2c6-76a2-42fc-a3f4-50674c99bc88\") " pod="openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g" Oct 31 04:26:58.236517 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.236519 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0165c2c6-76a2-42fc-a3f4-50674c99bc88-workload-certs\") pod \"e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g\" (UID: \"0165c2c6-76a2-42fc-a3f4-50674c99bc88\") " pod="openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g" Oct 31 04:26:58.236830 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.236616 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4c6x\" (UniqueName: \"kubernetes.io/projected/0165c2c6-76a2-42fc-a3f4-50674c99bc88-kube-api-access-b4c6x\") pod \"e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g\" (UID: \"0165c2c6-76a2-42fc-a3f4-50674c99bc88\") " pod="openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g" Oct 31 04:26:58.236830 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.236738 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0165c2c6-76a2-42fc-a3f4-50674c99bc88-istio-data\") pod \"e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g\" (UID: \"0165c2c6-76a2-42fc-a3f4-50674c99bc88\") " pod="openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g" Oct 31 04:26:58.236830 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.236791 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0165c2c6-76a2-42fc-a3f4-50674c99bc88-istio-token\") pod \"e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g\" (UID: \"0165c2c6-76a2-42fc-a3f4-50674c99bc88\") " pod="openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g" Oct 31 04:26:58.236830 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.236827 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0165c2c6-76a2-42fc-a3f4-50674c99bc88-istiod-ca-cert\") pod \"e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g\" (UID: \"0165c2c6-76a2-42fc-a3f4-50674c99bc88\") " pod="openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g" Oct 31 04:26:58.237075 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.236855 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0165c2c6-76a2-42fc-a3f4-50674c99bc88-istio-envoy\") pod \"e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g\" (UID: \"0165c2c6-76a2-42fc-a3f4-50674c99bc88\") " pod="openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g" Oct 31 04:26:58.237075 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.236908 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0165c2c6-76a2-42fc-a3f4-50674c99bc88-istio-podinfo\") pod \"e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g\" (UID: \"0165c2c6-76a2-42fc-a3f4-50674c99bc88\") " pod="openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g" Oct 31 04:26:58.237075 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.236954 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0165c2c6-76a2-42fc-a3f4-50674c99bc88-credential-socket\") pod \"e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g\" (UID: \"0165c2c6-76a2-42fc-a3f4-50674c99bc88\") " pod="openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g" Oct 31 04:26:58.343002 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.342491 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0165c2c6-76a2-42fc-a3f4-50674c99bc88-istio-data\") pod \"e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g\" (UID: \"0165c2c6-76a2-42fc-a3f4-50674c99bc88\") " pod="openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g" Oct 31 04:26:58.343002 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.342566 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0165c2c6-76a2-42fc-a3f4-50674c99bc88-istio-token\") pod \"e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g\" (UID: \"0165c2c6-76a2-42fc-a3f4-50674c99bc88\") " pod="openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g" Oct 31 04:26:58.343002 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.342605 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0165c2c6-76a2-42fc-a3f4-50674c99bc88-istiod-ca-cert\") pod \"e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g\" (UID: \"0165c2c6-76a2-42fc-a3f4-50674c99bc88\") " pod="openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g" Oct 31 04:26:58.343002 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.342634 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0165c2c6-76a2-42fc-a3f4-50674c99bc88-istio-envoy\") pod \"e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g\" (UID: \"0165c2c6-76a2-42fc-a3f4-50674c99bc88\") " pod="openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g" Oct 31 04:26:58.343002 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.342694 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0165c2c6-76a2-42fc-a3f4-50674c99bc88-istio-podinfo\") pod \"e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g\" (UID: \"0165c2c6-76a2-42fc-a3f4-50674c99bc88\") " pod="openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g" Oct 31 04:26:58.343002 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.342743 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0165c2c6-76a2-42fc-a3f4-50674c99bc88-credential-socket\") pod \"e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g\" (UID: \"0165c2c6-76a2-42fc-a3f4-50674c99bc88\") " pod="openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g" Oct 31 04:26:58.343002 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.342883 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0165c2c6-76a2-42fc-a3f4-50674c99bc88-workload-socket\") pod \"e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g\" (UID: \"0165c2c6-76a2-42fc-a3f4-50674c99bc88\") " pod="openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g" Oct 31 04:26:58.343002 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.342929 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0165c2c6-76a2-42fc-a3f4-50674c99bc88-workload-certs\") pod \"e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g\" (UID: \"0165c2c6-76a2-42fc-a3f4-50674c99bc88\") " pod="openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g" Oct 31 04:26:58.343002 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.342961 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4c6x\" (UniqueName: \"kubernetes.io/projected/0165c2c6-76a2-42fc-a3f4-50674c99bc88-kube-api-access-b4c6x\") pod \"e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g\" (UID: \"0165c2c6-76a2-42fc-a3f4-50674c99bc88\") " pod="openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g" Oct 31 04:26:58.367358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.356731 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0165c2c6-76a2-42fc-a3f4-50674c99bc88-workload-socket\") pod \"e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g\" (UID: \"0165c2c6-76a2-42fc-a3f4-50674c99bc88\") " pod="openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g" Oct 31 04:26:58.367358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.357032 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0165c2c6-76a2-42fc-a3f4-50674c99bc88-credential-socket\") pod \"e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g\" (UID: \"0165c2c6-76a2-42fc-a3f4-50674c99bc88\") " pod="openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g" Oct 31 04:26:58.367358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.362214 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0165c2c6-76a2-42fc-a3f4-50674c99bc88-istio-data\") pod \"e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g\" (UID: \"0165c2c6-76a2-42fc-a3f4-50674c99bc88\") " pod="openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g" Oct 31 04:26:58.367358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.364662 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0165c2c6-76a2-42fc-a3f4-50674c99bc88-workload-certs\") pod \"e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g\" (UID: \"0165c2c6-76a2-42fc-a3f4-50674c99bc88\") " pod="openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g" Oct 31 04:26:58.386351 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.378803 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0165c2c6-76a2-42fc-a3f4-50674c99bc88-istiod-ca-cert\") pod \"e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g\" (UID: \"0165c2c6-76a2-42fc-a3f4-50674c99bc88\") " pod="openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g" Oct 31 04:26:58.405240 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.404978 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0165c2c6-76a2-42fc-a3f4-50674c99bc88-istio-token\") pod \"e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g\" (UID: \"0165c2c6-76a2-42fc-a3f4-50674c99bc88\") " pod="openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g" Oct 31 04:26:58.406238 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.406203 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4c6x\" (UniqueName: \"kubernetes.io/projected/0165c2c6-76a2-42fc-a3f4-50674c99bc88-kube-api-access-b4c6x\") pod \"e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g\" (UID: \"0165c2c6-76a2-42fc-a3f4-50674c99bc88\") " pod="openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g" Oct 31 04:26:58.413437 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.413402 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0165c2c6-76a2-42fc-a3f4-50674c99bc88-istio-envoy\") pod \"e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g\" (UID: \"0165c2c6-76a2-42fc-a3f4-50674c99bc88\") " pod="openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g" Oct 31 04:26:58.414725 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.414682 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0165c2c6-76a2-42fc-a3f4-50674c99bc88-istio-podinfo\") pod \"e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g\" (UID: \"0165c2c6-76a2-42fc-a3f4-50674c99bc88\") " pod="openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g" Oct 31 04:26:58.537367 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.537315 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g" Oct 31 04:26:58.899945 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:26:58.899911 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0165c2c6_76a2_42fc_a3f4_50674c99bc88.slice/crio-1e60ff7bb346cac0bcdc83a22cc74a9af3f09d146509c5a14c017a59dc5f3145 WatchSource:0}: Error finding container 1e60ff7bb346cac0bcdc83a22cc74a9af3f09d146509c5a14c017a59dc5f3145: Status 404 returned error can't find the container with id 1e60ff7bb346cac0bcdc83a22cc74a9af3f09d146509c5a14c017a59dc5f3145 Oct 31 04:26:58.900811 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.900787 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g"] Oct 31 04:26:58.902681 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.902643 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:26:58.902779 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.902724 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:26:58.902779 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:58.902769 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:26:59.006397 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:59.006361 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8"] Oct 31 04:26:59.012347 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:59.012302 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8" Oct 31 04:26:59.086469 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:59.059672 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-c6859a-openshift-default-dockercfg-685d9\"" Oct 31 04:26:59.111817 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:59.111766 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8"] Oct 31 04:26:59.186402 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:59.181420 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92398da3-abea-427b-ad63-1d9fbd4385e5-istio-token\") pod \"e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8\" (UID: \"92398da3-abea-427b-ad63-1d9fbd4385e5\") " pod="openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8" Oct 31 04:26:59.186402 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:59.181828 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92398da3-abea-427b-ad63-1d9fbd4385e5-istiod-ca-cert\") pod \"e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8\" (UID: \"92398da3-abea-427b-ad63-1d9fbd4385e5\") " pod="openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8" Oct 31 04:26:59.186402 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:59.181860 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92398da3-abea-427b-ad63-1d9fbd4385e5-istio-podinfo\") pod \"e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8\" (UID: \"92398da3-abea-427b-ad63-1d9fbd4385e5\") " pod="openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8" Oct 31 04:26:59.186402 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:59.181900 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92398da3-abea-427b-ad63-1d9fbd4385e5-istio-data\") pod \"e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8\" (UID: \"92398da3-abea-427b-ad63-1d9fbd4385e5\") " pod="openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8" Oct 31 04:26:59.186402 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:59.181971 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqrtn\" (UniqueName: \"kubernetes.io/projected/92398da3-abea-427b-ad63-1d9fbd4385e5-kube-api-access-gqrtn\") pod \"e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8\" (UID: \"92398da3-abea-427b-ad63-1d9fbd4385e5\") " pod="openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8" Oct 31 04:26:59.186402 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:59.182023 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92398da3-abea-427b-ad63-1d9fbd4385e5-istio-envoy\") pod \"e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8\" (UID: \"92398da3-abea-427b-ad63-1d9fbd4385e5\") " pod="openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8" Oct 31 04:26:59.186402 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:59.182099 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92398da3-abea-427b-ad63-1d9fbd4385e5-workload-certs\") pod \"e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8\" (UID: \"92398da3-abea-427b-ad63-1d9fbd4385e5\") " pod="openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8" Oct 31 04:26:59.186402 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:59.182170 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92398da3-abea-427b-ad63-1d9fbd4385e5-credential-socket\") pod \"e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8\" (UID: \"92398da3-abea-427b-ad63-1d9fbd4385e5\") " pod="openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8" Oct 31 04:26:59.186402 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:59.182205 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92398da3-abea-427b-ad63-1d9fbd4385e5-workload-socket\") pod \"e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8\" (UID: \"92398da3-abea-427b-ad63-1d9fbd4385e5\") " pod="openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8" Oct 31 04:26:59.311354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:59.310718 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92398da3-abea-427b-ad63-1d9fbd4385e5-istio-envoy\") pod \"e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8\" (UID: \"92398da3-abea-427b-ad63-1d9fbd4385e5\") " pod="openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8" Oct 31 04:26:59.311354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:59.310819 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92398da3-abea-427b-ad63-1d9fbd4385e5-workload-certs\") pod \"e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8\" (UID: \"92398da3-abea-427b-ad63-1d9fbd4385e5\") " pod="openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8" Oct 31 04:26:59.311354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:59.310890 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92398da3-abea-427b-ad63-1d9fbd4385e5-credential-socket\") pod \"e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8\" (UID: \"92398da3-abea-427b-ad63-1d9fbd4385e5\") " pod="openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8" Oct 31 04:26:59.311354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:59.310928 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92398da3-abea-427b-ad63-1d9fbd4385e5-workload-socket\") pod \"e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8\" (UID: \"92398da3-abea-427b-ad63-1d9fbd4385e5\") " pod="openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8" Oct 31 04:26:59.311354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:59.310981 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92398da3-abea-427b-ad63-1d9fbd4385e5-istio-token\") pod \"e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8\" (UID: \"92398da3-abea-427b-ad63-1d9fbd4385e5\") " pod="openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8" Oct 31 04:26:59.311354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:59.311063 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92398da3-abea-427b-ad63-1d9fbd4385e5-istiod-ca-cert\") pod \"e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8\" (UID: \"92398da3-abea-427b-ad63-1d9fbd4385e5\") " pod="openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8" Oct 31 04:26:59.311354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:59.311092 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92398da3-abea-427b-ad63-1d9fbd4385e5-istio-podinfo\") pod \"e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8\" (UID: \"92398da3-abea-427b-ad63-1d9fbd4385e5\") " pod="openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8" Oct 31 04:26:59.311354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:59.311135 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92398da3-abea-427b-ad63-1d9fbd4385e5-istio-data\") pod \"e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8\" (UID: \"92398da3-abea-427b-ad63-1d9fbd4385e5\") " pod="openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8" Oct 31 04:26:59.311354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:59.311210 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqrtn\" (UniqueName: \"kubernetes.io/projected/92398da3-abea-427b-ad63-1d9fbd4385e5-kube-api-access-gqrtn\") pod \"e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8\" (UID: \"92398da3-abea-427b-ad63-1d9fbd4385e5\") " pod="openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8" Oct 31 04:26:59.329759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:59.312320 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92398da3-abea-427b-ad63-1d9fbd4385e5-istiod-ca-cert\") pod \"e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8\" (UID: \"92398da3-abea-427b-ad63-1d9fbd4385e5\") " pod="openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8" Oct 31 04:26:59.329759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:59.313486 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92398da3-abea-427b-ad63-1d9fbd4385e5-workload-certs\") pod \"e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8\" (UID: \"92398da3-abea-427b-ad63-1d9fbd4385e5\") " pod="openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8" Oct 31 04:26:59.329759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:59.314219 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92398da3-abea-427b-ad63-1d9fbd4385e5-workload-socket\") pod \"e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8\" (UID: \"92398da3-abea-427b-ad63-1d9fbd4385e5\") " pod="openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8" Oct 31 04:26:59.329759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:59.314689 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92398da3-abea-427b-ad63-1d9fbd4385e5-credential-socket\") pod \"e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8\" (UID: \"92398da3-abea-427b-ad63-1d9fbd4385e5\") " pod="openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8" Oct 31 04:26:59.329759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:59.314938 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92398da3-abea-427b-ad63-1d9fbd4385e5-istio-data\") pod \"e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8\" (UID: \"92398da3-abea-427b-ad63-1d9fbd4385e5\") " pod="openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8" Oct 31 04:26:59.329759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:59.327746 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92398da3-abea-427b-ad63-1d9fbd4385e5-istio-envoy\") pod \"e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8\" (UID: \"92398da3-abea-427b-ad63-1d9fbd4385e5\") " pod="openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8" Oct 31 04:26:59.329759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:59.329489 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92398da3-abea-427b-ad63-1d9fbd4385e5-istio-podinfo\") pod \"e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8\" (UID: \"92398da3-abea-427b-ad63-1d9fbd4385e5\") " pod="openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8" Oct 31 04:26:59.336626 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:59.336588 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92398da3-abea-427b-ad63-1d9fbd4385e5-istio-token\") pod \"e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8\" (UID: \"92398da3-abea-427b-ad63-1d9fbd4385e5\") " pod="openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8" Oct 31 04:26:59.337890 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:59.337860 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqrtn\" (UniqueName: \"kubernetes.io/projected/92398da3-abea-427b-ad63-1d9fbd4385e5-kube-api-access-gqrtn\") pod \"e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8\" (UID: \"92398da3-abea-427b-ad63-1d9fbd4385e5\") " pod="openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8" Oct 31 04:26:59.454067 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:59.453979 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8" Oct 31 04:26:59.563375 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:59.563299 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g" event={"ID":"0165c2c6-76a2-42fc-a3f4-50674c99bc88","Type":"ContainerStarted","Data":"f9f0f9d9da35d1e56014f64fe194e7b84639347ad128aaac04c83308e683cb00"} Oct 31 04:26:59.563580 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:59.563404 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g" event={"ID":"0165c2c6-76a2-42fc-a3f4-50674c99bc88","Type":"ContainerStarted","Data":"1e60ff7bb346cac0bcdc83a22cc74a9af3f09d146509c5a14c017a59dc5f3145"} Oct 31 04:26:59.609901 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:59.609828 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g" podStartSLOduration=1.6098096530000001 podStartE2EDuration="1.609809653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:26:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:26:59.608001139 +0000 UTC m=+1374.364339178" watchObservedRunningTime="2026-10-31 04:26:59.609809653 +0000 UTC m=+1374.366147694" Oct 31 04:26:59.780514 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:26:59.780478 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92398da3_abea_427b_ad63_1d9fbd4385e5.slice/crio-27d0ece5079b77213cc93f2d5db609e5ca4c9c42f170c6fbbd8c44593eaab633 WatchSource:0}: Error finding container 27d0ece5079b77213cc93f2d5db609e5ca4c9c42f170c6fbbd8c44593eaab633: Status 404 returned error can't find the container with id 27d0ece5079b77213cc93f2d5db609e5ca4c9c42f170c6fbbd8c44593eaab633 Oct 31 04:26:59.781001 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:59.780977 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8"] Oct 31 04:26:59.783616 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:59.783576 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:26:59.783716 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:59.783653 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:26:59.783716 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:26:59.783695 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:27:00.539677 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:00.539647 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g" Oct 31 04:27:00.545576 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:00.545552 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g" Oct 31 04:27:00.572686 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:00.572646 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8" event={"ID":"92398da3-abea-427b-ad63-1d9fbd4385e5","Type":"ContainerStarted","Data":"33fec37f6afc9a22df2004dd4083eb96dbf8e4d306eaf1eeffbfa190e52571e0"} Oct 31 04:27:00.572838 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:00.572706 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8" event={"ID":"92398da3-abea-427b-ad63-1d9fbd4385e5","Type":"ContainerStarted","Data":"27d0ece5079b77213cc93f2d5db609e5ca4c9c42f170c6fbbd8c44593eaab633"} Oct 31 04:27:00.573022 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:00.573002 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g" Oct 31 04:27:00.574031 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:00.574012 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g" Oct 31 04:27:00.654000 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:00.653955 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8" podStartSLOduration=2.653940573 podStartE2EDuration="2.653940573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:26:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:27:00.65071815 +0000 UTC m=+1375.407056190" watchObservedRunningTime="2026-10-31 04:27:00.653940573 +0000 UTC m=+1375.410278678" Oct 31 04:27:01.455166 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:01.455095 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8" Oct 31 04:27:01.461567 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:01.461533 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8" Oct 31 04:27:01.550767 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:01.550739 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp" Oct 31 04:27:01.590836 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:01.589042 5019 generic.go:356] "Generic (PLEG): container finished" podID="0712577c-fb53-4025-9d1e-6c62e6bf38cb" containerID="0d2c1812809a4e09a55120bd6a6b28287556cb8153fa32ac151bdd9a9cb739d1" exitCode=0 Oct 31 04:27:01.590836 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:01.590471 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp" Oct 31 04:27:01.590836 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:01.590675 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp" event={"ID":"0712577c-fb53-4025-9d1e-6c62e6bf38cb","Type":"ContainerDied","Data":"0d2c1812809a4e09a55120bd6a6b28287556cb8153fa32ac151bdd9a9cb739d1"} Oct 31 04:27:01.590836 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:01.590749 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp" event={"ID":"0712577c-fb53-4025-9d1e-6c62e6bf38cb","Type":"ContainerDied","Data":"bd1ed829dbfa7de173662cad93e1379fca8e2b1593971647f6ceea2cf182fb92"} Oct 31 04:27:01.590836 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:01.590789 5019 scope.go:122] "RemoveContainer" containerID="0d2c1812809a4e09a55120bd6a6b28287556cb8153fa32ac151bdd9a9cb739d1" Oct 31 04:27:01.591811 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:01.591359 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8" Oct 31 04:27:01.593508 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:01.593140 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8" Oct 31 04:27:01.610236 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:01.610062 5019 scope.go:122] "RemoveContainer" containerID="c345d8d327628e6c8dde79bc244b066cd375bdec1301460167a7775f5ebf5822" Oct 31 04:27:01.622963 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:01.622929 5019 scope.go:122] "RemoveContainer" containerID="0d2c1812809a4e09a55120bd6a6b28287556cb8153fa32ac151bdd9a9cb739d1" Oct 31 04:27:01.623495 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:27:01.623462 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d2c1812809a4e09a55120bd6a6b28287556cb8153fa32ac151bdd9a9cb739d1\": container with ID starting with 0d2c1812809a4e09a55120bd6a6b28287556cb8153fa32ac151bdd9a9cb739d1 not found: ID does not exist" containerID="0d2c1812809a4e09a55120bd6a6b28287556cb8153fa32ac151bdd9a9cb739d1" Oct 31 04:27:01.623604 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:01.623507 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d2c1812809a4e09a55120bd6a6b28287556cb8153fa32ac151bdd9a9cb739d1"} err="failed to get container status \"0d2c1812809a4e09a55120bd6a6b28287556cb8153fa32ac151bdd9a9cb739d1\": rpc error: code = NotFound desc = could not find container \"0d2c1812809a4e09a55120bd6a6b28287556cb8153fa32ac151bdd9a9cb739d1\": container with ID starting with 0d2c1812809a4e09a55120bd6a6b28287556cb8153fa32ac151bdd9a9cb739d1 not found: ID does not exist" Oct 31 04:27:01.623604 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:01.623530 5019 scope.go:122] "RemoveContainer" containerID="c345d8d327628e6c8dde79bc244b066cd375bdec1301460167a7775f5ebf5822" Oct 31 04:27:01.624210 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:27:01.624190 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c345d8d327628e6c8dde79bc244b066cd375bdec1301460167a7775f5ebf5822\": container with ID starting with c345d8d327628e6c8dde79bc244b066cd375bdec1301460167a7775f5ebf5822 not found: ID does not exist" containerID="c345d8d327628e6c8dde79bc244b066cd375bdec1301460167a7775f5ebf5822" Oct 31 04:27:01.624284 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:01.624222 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c345d8d327628e6c8dde79bc244b066cd375bdec1301460167a7775f5ebf5822"} err="failed to get container status \"c345d8d327628e6c8dde79bc244b066cd375bdec1301460167a7775f5ebf5822\": rpc error: code = NotFound desc = could not find container \"c345d8d327628e6c8dde79bc244b066cd375bdec1301460167a7775f5ebf5822\": container with ID starting with c345d8d327628e6c8dde79bc244b066cd375bdec1301460167a7775f5ebf5822 not found: ID does not exist" Oct 31 04:27:01.645746 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:01.645702 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0712577c-fb53-4025-9d1e-6c62e6bf38cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0712577c-fb53-4025-9d1e-6c62e6bf38cb-istio-podinfo\") pod \"0712577c-fb53-4025-9d1e-6c62e6bf38cb\" (UID: \"0712577c-fb53-4025-9d1e-6c62e6bf38cb\") " Oct 31 04:27:01.645872 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:01.645762 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0712577c-fb53-4025-9d1e-6c62e6bf38cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0712577c-fb53-4025-9d1e-6c62e6bf38cb-workload-socket\") pod \"0712577c-fb53-4025-9d1e-6c62e6bf38cb\" (UID: \"0712577c-fb53-4025-9d1e-6c62e6bf38cb\") " Oct 31 04:27:01.645923 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:01.645883 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0712577c-fb53-4025-9d1e-6c62e6bf38cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0712577c-fb53-4025-9d1e-6c62e6bf38cb-istio-data\") pod \"0712577c-fb53-4025-9d1e-6c62e6bf38cb\" (UID: \"0712577c-fb53-4025-9d1e-6c62e6bf38cb\") " Oct 31 04:27:01.645968 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:01.645938 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0712577c-fb53-4025-9d1e-6c62e6bf38cb-kube-api-access-27hbs\" (UniqueName: \"kubernetes.io/projected/0712577c-fb53-4025-9d1e-6c62e6bf38cb-kube-api-access-27hbs\") pod \"0712577c-fb53-4025-9d1e-6c62e6bf38cb\" (UID: \"0712577c-fb53-4025-9d1e-6c62e6bf38cb\") " Oct 31 04:27:01.646592 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:01.646561 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0712577c-fb53-4025-9d1e-6c62e6bf38cb-workload-socket" pod "0712577c-fb53-4025-9d1e-6c62e6bf38cb" (UID: "0712577c-fb53-4025-9d1e-6c62e6bf38cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:27:01.646691 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:01.646666 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0712577c-fb53-4025-9d1e-6c62e6bf38cb-istio-data" pod "0712577c-fb53-4025-9d1e-6c62e6bf38cb" (UID: "0712577c-fb53-4025-9d1e-6c62e6bf38cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:27:01.651916 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:01.651879 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0712577c-fb53-4025-9d1e-6c62e6bf38cb-kube-api-access-27hbs" pod "0712577c-fb53-4025-9d1e-6c62e6bf38cb" (UID: "0712577c-fb53-4025-9d1e-6c62e6bf38cb"). InnerVolumeSpecName "kube-api-access-27hbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:27:01.652858 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:01.652835 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0712577c-fb53-4025-9d1e-6c62e6bf38cb-istio-podinfo" pod "0712577c-fb53-4025-9d1e-6c62e6bf38cb" (UID: "0712577c-fb53-4025-9d1e-6c62e6bf38cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:27:01.653452 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:01.653428 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0712577c-fb53-4025-9d1e-6c62e6bf38cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0712577c-fb53-4025-9d1e-6c62e6bf38cb-credential-socket\") pod \"0712577c-fb53-4025-9d1e-6c62e6bf38cb\" (UID: \"0712577c-fb53-4025-9d1e-6c62e6bf38cb\") " Oct 31 04:27:01.653529 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:01.653490 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0712577c-fb53-4025-9d1e-6c62e6bf38cb-istio-token\" (UniqueName: \"kubernetes.io/projected/0712577c-fb53-4025-9d1e-6c62e6bf38cb-istio-token\") pod \"0712577c-fb53-4025-9d1e-6c62e6bf38cb\" (UID: \"0712577c-fb53-4025-9d1e-6c62e6bf38cb\") " Oct 31 04:27:01.653614 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:01.653599 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0712577c-fb53-4025-9d1e-6c62e6bf38cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0712577c-fb53-4025-9d1e-6c62e6bf38cb-istiod-ca-cert\") pod \"0712577c-fb53-4025-9d1e-6c62e6bf38cb\" (UID: \"0712577c-fb53-4025-9d1e-6c62e6bf38cb\") " Oct 31 04:27:01.653743 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:01.653728 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0712577c-fb53-4025-9d1e-6c62e6bf38cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0712577c-fb53-4025-9d1e-6c62e6bf38cb-workload-certs\") pod \"0712577c-fb53-4025-9d1e-6c62e6bf38cb\" (UID: \"0712577c-fb53-4025-9d1e-6c62e6bf38cb\") " Oct 31 04:27:01.653802 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:01.653784 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0712577c-fb53-4025-9d1e-6c62e6bf38cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0712577c-fb53-4025-9d1e-6c62e6bf38cb-istio-envoy\") pod \"0712577c-fb53-4025-9d1e-6c62e6bf38cb\" (UID: \"0712577c-fb53-4025-9d1e-6c62e6bf38cb\") " Oct 31 04:27:01.654781 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:01.654604 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0712577c-fb53-4025-9d1e-6c62e6bf38cb-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:27:01.654781 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:01.654630 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0712577c-fb53-4025-9d1e-6c62e6bf38cb-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:27:01.654781 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:01.654644 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0712577c-fb53-4025-9d1e-6c62e6bf38cb-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:27:01.654781 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:01.654659 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27hbs\" (UniqueName: \"kubernetes.io/projected/0712577c-fb53-4025-9d1e-6c62e6bf38cb-kube-api-access-27hbs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:27:01.656220 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:01.655057 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0712577c-fb53-4025-9d1e-6c62e6bf38cb-credential-socket" pod "0712577c-fb53-4025-9d1e-6c62e6bf38cb" (UID: "0712577c-fb53-4025-9d1e-6c62e6bf38cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:27:01.656220 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:01.655716 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0712577c-fb53-4025-9d1e-6c62e6bf38cb-istiod-ca-cert" pod "0712577c-fb53-4025-9d1e-6c62e6bf38cb" (UID: "0712577c-fb53-4025-9d1e-6c62e6bf38cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:27:01.656220 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:01.655973 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0712577c-fb53-4025-9d1e-6c62e6bf38cb-workload-certs" pod "0712577c-fb53-4025-9d1e-6c62e6bf38cb" (UID: "0712577c-fb53-4025-9d1e-6c62e6bf38cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:27:01.658525 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:01.658495 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0712577c-fb53-4025-9d1e-6c62e6bf38cb-istio-envoy" pod "0712577c-fb53-4025-9d1e-6c62e6bf38cb" (UID: "0712577c-fb53-4025-9d1e-6c62e6bf38cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:27:01.659658 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:01.659466 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0712577c-fb53-4025-9d1e-6c62e6bf38cb-istio-token" pod "0712577c-fb53-4025-9d1e-6c62e6bf38cb" (UID: "0712577c-fb53-4025-9d1e-6c62e6bf38cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:27:01.756075 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:01.756035 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0712577c-fb53-4025-9d1e-6c62e6bf38cb-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:27:01.756504 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:01.756463 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0712577c-fb53-4025-9d1e-6c62e6bf38cb-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:27:01.756613 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:01.756597 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0712577c-fb53-4025-9d1e-6c62e6bf38cb-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:27:01.756687 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:01.756620 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0712577c-fb53-4025-9d1e-6c62e6bf38cb-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:27:01.756687 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:01.756633 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0712577c-fb53-4025-9d1e-6c62e6bf38cb-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:27:01.938006 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:01.937972 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp"] Oct 31 04:27:01.949935 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:01.949907 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-7b4185-openshift-default-5cc48cbdb7-bczhp"] Oct 31 04:27:03.410866 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:03.410823 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0712577c-fb53-4025-9d1e-6c62e6bf38cb" path="/var/lib/kubelet/pods/0712577c-fb53-4025-9d1e-6c62e6bf38cb/volumes" Oct 31 04:27:03.880984 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:03.880943 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-9ee727-7f9ccf9d5-r5j4l"] Oct 31 04:27:03.881690 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:03.881669 5019 cpu_manager.go:414] "containerMap: removing container" podUID="0712577c-fb53-4025-9d1e-6c62e6bf38cb" containerName="istio-proxy" Oct 31 04:27:03.881690 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:03.881690 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0712577c-fb53-4025-9d1e-6c62e6bf38cb" containerName="istio-proxy" Oct 31 04:27:03.881874 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:03.881734 5019 cpu_manager.go:414] "containerMap: removing container" podUID="0712577c-fb53-4025-9d1e-6c62e6bf38cb" containerName="istio-proxy" Oct 31 04:27:03.881874 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:03.881742 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0712577c-fb53-4025-9d1e-6c62e6bf38cb" containerName="istio-proxy" Oct 31 04:27:03.881975 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:03.881902 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="0712577c-fb53-4025-9d1e-6c62e6bf38cb" containerName="istio-proxy" Oct 31 04:27:03.887070 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:03.887042 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-9ee727-7f9ccf9d5-r5j4l" Oct 31 04:27:03.904325 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:03.904298 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-9ee727-7f9ccf9d5-r5j4l"] Oct 31 04:27:03.917581 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:03.917535 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-9ee727\"" Oct 31 04:27:03.917727 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:03.917645 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-9ee727-dockercfg-dwmwl\"" Oct 31 04:27:03.939634 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:03.939602 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-9ee727-d7d75f767-qq7nn"] Oct 31 04:27:03.940743 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:03.940720 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="0712577c-fb53-4025-9d1e-6c62e6bf38cb" containerName="istio-proxy" Oct 31 04:27:03.944989 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:03.944967 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9ee727-d7d75f767-qq7nn" Oct 31 04:27:03.954785 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:03.954763 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-9ee727-d7d75f767-qq7nn"] Oct 31 04:27:04.014356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:04.000604 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cbg4\" (UniqueName: \"kubernetes.io/projected/4cc24bed-f9fb-43a0-80f4-411f33eb3f97-kube-api-access-5cbg4\") pod \"payload-pre-processing-e2e-praxis-a-9ee727-d7d75f767-qq7nn\" (UID: \"4cc24bed-f9fb-43a0-80f4-411f33eb3f97\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9ee727-d7d75f767-qq7nn" Oct 31 04:27:04.014356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:04.000690 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba7c4e82-982a-435c-9cc5-36042d5c88d5-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-9ee727-7f9ccf9d5-r5j4l\" (UID: \"ba7c4e82-982a-435c-9cc5-36042d5c88d5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-9ee727-7f9ccf9d5-r5j4l" Oct 31 04:27:04.014356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:04.000741 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cc24bed-f9fb-43a0-80f4-411f33eb3f97-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-9ee727-d7d75f767-qq7nn\" (UID: \"4cc24bed-f9fb-43a0-80f4-411f33eb3f97\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9ee727-d7d75f767-qq7nn" Oct 31 04:27:04.014356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:04.000798 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htqzv\" (UniqueName: \"kubernetes.io/projected/ba7c4e82-982a-435c-9cc5-36042d5c88d5-kube-api-access-htqzv\") pod \"payload-processing-e2e-praxis-a-9ee727-7f9ccf9d5-r5j4l\" (UID: \"ba7c4e82-982a-435c-9cc5-36042d5c88d5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-9ee727-7f9ccf9d5-r5j4l" Oct 31 04:27:04.108855 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:04.107179 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cbg4\" (UniqueName: \"kubernetes.io/projected/4cc24bed-f9fb-43a0-80f4-411f33eb3f97-kube-api-access-5cbg4\") pod \"payload-pre-processing-e2e-praxis-a-9ee727-d7d75f767-qq7nn\" (UID: \"4cc24bed-f9fb-43a0-80f4-411f33eb3f97\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9ee727-d7d75f767-qq7nn" Oct 31 04:27:04.108855 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:04.107257 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba7c4e82-982a-435c-9cc5-36042d5c88d5-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-9ee727-7f9ccf9d5-r5j4l\" (UID: \"ba7c4e82-982a-435c-9cc5-36042d5c88d5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-9ee727-7f9ccf9d5-r5j4l" Oct 31 04:27:04.108855 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:04.107287 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cc24bed-f9fb-43a0-80f4-411f33eb3f97-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-9ee727-d7d75f767-qq7nn\" (UID: \"4cc24bed-f9fb-43a0-80f4-411f33eb3f97\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9ee727-d7d75f767-qq7nn" Oct 31 04:27:04.108855 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:04.107366 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htqzv\" (UniqueName: \"kubernetes.io/projected/ba7c4e82-982a-435c-9cc5-36042d5c88d5-kube-api-access-htqzv\") pod \"payload-processing-e2e-praxis-a-9ee727-7f9ccf9d5-r5j4l\" (UID: \"ba7c4e82-982a-435c-9cc5-36042d5c88d5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-9ee727-7f9ccf9d5-r5j4l" Oct 31 04:27:04.109213 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:04.109102 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba7c4e82-982a-435c-9cc5-36042d5c88d5-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-9ee727-7f9ccf9d5-r5j4l\" (UID: \"ba7c4e82-982a-435c-9cc5-36042d5c88d5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-9ee727-7f9ccf9d5-r5j4l" Oct 31 04:27:04.124352 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:04.110013 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cc24bed-f9fb-43a0-80f4-411f33eb3f97-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-9ee727-d7d75f767-qq7nn\" (UID: \"4cc24bed-f9fb-43a0-80f4-411f33eb3f97\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9ee727-d7d75f767-qq7nn" Oct 31 04:27:04.177401 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:04.170365 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cbg4\" (UniqueName: \"kubernetes.io/projected/4cc24bed-f9fb-43a0-80f4-411f33eb3f97-kube-api-access-5cbg4\") pod \"payload-pre-processing-e2e-praxis-a-9ee727-d7d75f767-qq7nn\" (UID: \"4cc24bed-f9fb-43a0-80f4-411f33eb3f97\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9ee727-d7d75f767-qq7nn" Oct 31 04:27:04.214347 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:04.205021 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htqzv\" (UniqueName: \"kubernetes.io/projected/ba7c4e82-982a-435c-9cc5-36042d5c88d5-kube-api-access-htqzv\") pod \"payload-processing-e2e-praxis-a-9ee727-7f9ccf9d5-r5j4l\" (UID: \"ba7c4e82-982a-435c-9cc5-36042d5c88d5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-9ee727-7f9ccf9d5-r5j4l" Oct 31 04:27:04.260810 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:04.260781 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9ee727-d7d75f767-qq7nn" Oct 31 04:27:04.414770 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:04.414730 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f"] Oct 31 04:27:04.415319 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:04.414959 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f" podUID="e1233843-bebe-44fc-82f5-6120e6302e4e" containerName="istio-proxy" containerID="cri-o://b5e0abae72d16354a761904815875d5117551d3c1daf2a01d129c93062cac34c" gracePeriod=30 Oct 31 04:27:04.510441 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:04.510046 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-9ee727-7f9ccf9d5-r5j4l" Oct 31 04:27:04.639836 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:04.631499 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-9ee727-d7d75f767-qq7nn"] Oct 31 04:27:04.664120 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:27:04.664087 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cc24bed_f9fb_43a0_80f4_411f33eb3f97.slice/crio-68a85e335d8b7b451a29c269755a6bc813acc6715972660dbc31d0539d88b279 WatchSource:0}: Error finding container 68a85e335d8b7b451a29c269755a6bc813acc6715972660dbc31d0539d88b279: Status 404 returned error can't find the container with id 68a85e335d8b7b451a29c269755a6bc813acc6715972660dbc31d0539d88b279 Oct 31 04:27:04.971915 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:04.971883 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-9ee727-7f9ccf9d5-r5j4l"] Oct 31 04:27:04.972566 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:27:04.972530 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba7c4e82_982a_435c_9cc5_36042d5c88d5.slice/crio-fdfcd096487696500da8260413a5a5730f68b5231318c6bb0ba0e497b017d38c WatchSource:0}: Error finding container fdfcd096487696500da8260413a5a5730f68b5231318c6bb0ba0e497b017d38c: Status 404 returned error can't find the container with id fdfcd096487696500da8260413a5a5730f68b5231318c6bb0ba0e497b017d38c Oct 31 04:27:05.674624 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:05.674588 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9ee727-d7d75f767-qq7nn" event={"ID":"4cc24bed-f9fb-43a0-80f4-411f33eb3f97","Type":"ContainerStarted","Data":"efd16cbe95f4c7efc9e7bbde4711284635b88503ed33a0abec5981b8f7d74eab"} Oct 31 04:27:05.675096 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:05.674654 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9ee727-d7d75f767-qq7nn" event={"ID":"4cc24bed-f9fb-43a0-80f4-411f33eb3f97","Type":"ContainerStarted","Data":"68a85e335d8b7b451a29c269755a6bc813acc6715972660dbc31d0539d88b279"} Oct 31 04:27:05.675096 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:05.674681 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9ee727-d7d75f767-qq7nn" Oct 31 04:27:05.676558 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:05.676532 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-9ee727-7f9ccf9d5-r5j4l" event={"ID":"ba7c4e82-982a-435c-9cc5-36042d5c88d5","Type":"ContainerStarted","Data":"49daf298ba117bafd3029084a99ad703ceaec49f2f1ebc5ace390c1de370f03c"} Oct 31 04:27:05.676680 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:05.676576 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-9ee727-7f9ccf9d5-r5j4l" event={"ID":"ba7c4e82-982a-435c-9cc5-36042d5c88d5","Type":"ContainerStarted","Data":"fdfcd096487696500da8260413a5a5730f68b5231318c6bb0ba0e497b017d38c"} Oct 31 04:27:05.676680 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:05.676634 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-9ee727-7f9ccf9d5-r5j4l" Oct 31 04:27:05.718408 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:05.718301 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9ee727-d7d75f767-qq7nn" podStartSLOduration=2.718282439 podStartE2EDuration="2.718282439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:27:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:27:05.716921257 +0000 UTC m=+1380.473259299" watchObservedRunningTime="2026-10-31 04:27:05.718282439 +0000 UTC m=+1380.474620479" Oct 31 04:27:05.732198 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:05.732171 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ff43e3-5d974c979f-gz8bs"] Oct 31 04:27:05.759604 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:05.759555 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-9ee727-7f9ccf9d5-r5j4l" podStartSLOduration=2.7595437069999997 podStartE2EDuration="2.759543707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:27:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:27:05.755846004 +0000 UTC m=+1380.512184042" watchObservedRunningTime="2026-10-31 04:27:05.759543707 +0000 UTC m=+1380.515881740" Oct 31 04:27:07.181578 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:07.181538 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c6859a-5f796cd5cf-gmshg"] Oct 31 04:27:07.187858 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:07.187828 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c6859a-5f796cd5cf-gmshg" Oct 31 04:27:07.207885 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:07.207851 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c6859a-5f796cd5cf-gmshg"] Oct 31 04:27:07.214719 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:07.214658 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-c6859a-dockercfg-nbrk8\"" Oct 31 04:27:07.215057 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:07.215030 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-c6859a\"" Oct 31 04:27:07.295428 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:07.295391 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrqn5\" (UniqueName: \"kubernetes.io/projected/e4a67a18-4432-4411-9c7f-09b505667a44-kube-api-access-nrqn5\") pod \"payload-processing-e2e-praxis-a-c6859a-5f796cd5cf-gmshg\" (UID: \"e4a67a18-4432-4411-9c7f-09b505667a44\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c6859a-5f796cd5cf-gmshg" Oct 31 04:27:07.295621 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:07.295513 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4a67a18-4432-4411-9c7f-09b505667a44-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-c6859a-5f796cd5cf-gmshg\" (UID: \"e4a67a18-4432-4411-9c7f-09b505667a44\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c6859a-5f796cd5cf-gmshg" Oct 31 04:27:07.397728 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:07.397681 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrqn5\" (UniqueName: \"kubernetes.io/projected/e4a67a18-4432-4411-9c7f-09b505667a44-kube-api-access-nrqn5\") pod \"payload-processing-e2e-praxis-a-c6859a-5f796cd5cf-gmshg\" (UID: \"e4a67a18-4432-4411-9c7f-09b505667a44\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c6859a-5f796cd5cf-gmshg" Oct 31 04:27:07.397953 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:07.397791 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4a67a18-4432-4411-9c7f-09b505667a44-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-c6859a-5f796cd5cf-gmshg\" (UID: \"e4a67a18-4432-4411-9c7f-09b505667a44\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c6859a-5f796cd5cf-gmshg" Oct 31 04:27:07.399060 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:07.399029 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4a67a18-4432-4411-9c7f-09b505667a44-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-c6859a-5f796cd5cf-gmshg\" (UID: \"e4a67a18-4432-4411-9c7f-09b505667a44\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c6859a-5f796cd5cf-gmshg" Oct 31 04:27:07.460471 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:07.446087 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrqn5\" (UniqueName: \"kubernetes.io/projected/e4a67a18-4432-4411-9c7f-09b505667a44-kube-api-access-nrqn5\") pod \"payload-processing-e2e-praxis-a-c6859a-5f796cd5cf-gmshg\" (UID: \"e4a67a18-4432-4411-9c7f-09b505667a44\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c6859a-5f796cd5cf-gmshg" Oct 31 04:27:07.519221 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:07.518940 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c6859a-5f796cd5cf-gmshg" Oct 31 04:27:07.683457 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:07.683371 5019 scope.go:122] "RemoveContainer" containerID="ec20c5e6ea49c5521970d81c3bada96aa3d3137a5234c404467fde719c920a36" Oct 31 04:27:07.700944 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:07.699767 5019 scope.go:122] "RemoveContainer" containerID="7d54dc032880acf3f7ffc4d6e6b69b0ff345071b5f1e86b1c7622fedb2c901ed" Oct 31 04:27:07.732496 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:07.731460 5019 scope.go:122] "RemoveContainer" containerID="44e75fb5f36a7f4b9daa78339daebf3146e8ccb04c496eaff46c881a72a4bb67" Oct 31 04:27:08.032826 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:08.032750 5019 patch_prober.go:29] interesting pod/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 04:27:08.032826 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:08.032821 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f" podUID="e1233843-bebe-44fc-82f5-6120e6302e4e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 04:27:08.196357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:08.195711 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c6859a-5f796cd5cf-gmshg"] Oct 31 04:27:08.725657 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:08.725621 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c6859a-5f796cd5cf-gmshg" event={"ID":"e4a67a18-4432-4411-9c7f-09b505667a44","Type":"ContainerStarted","Data":"b5dfcc18b00d87e4f0aaf36a80c5b9a0f9ed2969b3e1ea11b7126b0a962e56b7"} Oct 31 04:27:08.725852 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:08.725674 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c6859a-5f796cd5cf-gmshg" event={"ID":"e4a67a18-4432-4411-9c7f-09b505667a44","Type":"ContainerStarted","Data":"0ffc78f8b6743dc89de31ebff6e7c61a3e2fce2c1a1b353287e23b758a5cb377"} Oct 31 04:27:08.725852 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:08.725743 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-c6859a-5f796cd5cf-gmshg" Oct 31 04:27:08.765904 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:08.765842 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-c6859a-5f796cd5cf-gmshg" podStartSLOduration=1.76582292 podStartE2EDuration="1.76582292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:27:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:27:08.765451612 +0000 UTC m=+1383.521789654" watchObservedRunningTime="2026-10-31 04:27:08.76582292 +0000 UTC m=+1383.522160959" Oct 31 04:27:09.756083 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:09.756038 5019 generic.go:356] "Generic (PLEG): container finished" podID="e1233843-bebe-44fc-82f5-6120e6302e4e" containerID="b5e0abae72d16354a761904815875d5117551d3c1daf2a01d129c93062cac34c" exitCode=0 Oct 31 04:27:09.756545 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:09.756104 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f" event={"ID":"e1233843-bebe-44fc-82f5-6120e6302e4e","Type":"ContainerDied","Data":"b5e0abae72d16354a761904815875d5117551d3c1daf2a01d129c93062cac34c"} Oct 31 04:27:09.756545 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:09.756167 5019 scope.go:122] "RemoveContainer" containerID="c84058468ab86da9b8eb32217d97cde841ed75135638e3572c2ace3947c8b862" Oct 31 04:27:09.813903 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:09.813879 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f" Oct 31 04:27:09.850195 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:09.850160 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4"] Oct 31 04:27:09.850409 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:09.850387 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4" podUID="b35297c9-af92-4356-9951-2a90b09fb3cd" containerName="istio-proxy" containerID="cri-o://eb9ef275fc5a81b1ef005c5b04d7a927d89d420a8dda42bb37679e3d305ba699" gracePeriod=30 Oct 31 04:27:09.936923 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:09.936892 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1233843-bebe-44fc-82f5-6120e6302e4e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1233843-bebe-44fc-82f5-6120e6302e4e-istiod-ca-cert\") pod \"e1233843-bebe-44fc-82f5-6120e6302e4e\" (UID: \"e1233843-bebe-44fc-82f5-6120e6302e4e\") " Oct 31 04:27:09.937077 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:09.936936 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1233843-bebe-44fc-82f5-6120e6302e4e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1233843-bebe-44fc-82f5-6120e6302e4e-workload-socket\") pod \"e1233843-bebe-44fc-82f5-6120e6302e4e\" (UID: \"e1233843-bebe-44fc-82f5-6120e6302e4e\") " Oct 31 04:27:09.937077 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:09.936998 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e1233843-bebe-44fc-82f5-6120e6302e4e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1233843-bebe-44fc-82f5-6120e6302e4e-istio-podinfo\") pod \"e1233843-bebe-44fc-82f5-6120e6302e4e\" (UID: \"e1233843-bebe-44fc-82f5-6120e6302e4e\") " Oct 31 04:27:09.937077 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:09.937042 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1233843-bebe-44fc-82f5-6120e6302e4e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1233843-bebe-44fc-82f5-6120e6302e4e-istio-data\") pod \"e1233843-bebe-44fc-82f5-6120e6302e4e\" (UID: \"e1233843-bebe-44fc-82f5-6120e6302e4e\") " Oct 31 04:27:09.937252 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:09.937082 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1233843-bebe-44fc-82f5-6120e6302e4e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1233843-bebe-44fc-82f5-6120e6302e4e-workload-certs\") pod \"e1233843-bebe-44fc-82f5-6120e6302e4e\" (UID: \"e1233843-bebe-44fc-82f5-6120e6302e4e\") " Oct 31 04:27:09.937252 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:09.937181 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1233843-bebe-44fc-82f5-6120e6302e4e-istio-token\" (UniqueName: \"kubernetes.io/projected/e1233843-bebe-44fc-82f5-6120e6302e4e-istio-token\") pod \"e1233843-bebe-44fc-82f5-6120e6302e4e\" (UID: \"e1233843-bebe-44fc-82f5-6120e6302e4e\") " Oct 31 04:27:09.937252 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:09.937237 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1233843-bebe-44fc-82f5-6120e6302e4e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1233843-bebe-44fc-82f5-6120e6302e4e-credential-socket\") pod \"e1233843-bebe-44fc-82f5-6120e6302e4e\" (UID: \"e1233843-bebe-44fc-82f5-6120e6302e4e\") " Oct 31 04:27:09.937375 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:09.937287 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1233843-bebe-44fc-82f5-6120e6302e4e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1233843-bebe-44fc-82f5-6120e6302e4e-istio-envoy\") pod \"e1233843-bebe-44fc-82f5-6120e6302e4e\" (UID: \"e1233843-bebe-44fc-82f5-6120e6302e4e\") " Oct 31 04:27:09.937530 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:09.937457 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1233843-bebe-44fc-82f5-6120e6302e4e-kube-api-access-zp49s\" (UniqueName: \"kubernetes.io/projected/e1233843-bebe-44fc-82f5-6120e6302e4e-kube-api-access-zp49s\") pod \"e1233843-bebe-44fc-82f5-6120e6302e4e\" (UID: \"e1233843-bebe-44fc-82f5-6120e6302e4e\") " Oct 31 04:27:09.937530 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:09.937489 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1233843-bebe-44fc-82f5-6120e6302e4e-istiod-ca-cert" pod "e1233843-bebe-44fc-82f5-6120e6302e4e" (UID: "e1233843-bebe-44fc-82f5-6120e6302e4e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:27:09.937654 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:09.937607 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1233843-bebe-44fc-82f5-6120e6302e4e-workload-socket" pod "e1233843-bebe-44fc-82f5-6120e6302e4e" (UID: "e1233843-bebe-44fc-82f5-6120e6302e4e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:27:09.937654 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:09.937627 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1233843-bebe-44fc-82f5-6120e6302e4e-workload-certs" pod "e1233843-bebe-44fc-82f5-6120e6302e4e" (UID: "e1233843-bebe-44fc-82f5-6120e6302e4e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:27:09.938015 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:09.937962 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1233843-bebe-44fc-82f5-6120e6302e4e-istio-data" pod "e1233843-bebe-44fc-82f5-6120e6302e4e" (UID: "e1233843-bebe-44fc-82f5-6120e6302e4e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:27:09.938180 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:09.938017 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1233843-bebe-44fc-82f5-6120e6302e4e-credential-socket" pod "e1233843-bebe-44fc-82f5-6120e6302e4e" (UID: "e1233843-bebe-44fc-82f5-6120e6302e4e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:27:09.938284 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:09.938234 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1233843-bebe-44fc-82f5-6120e6302e4e-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:27:09.938284 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:09.938256 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1233843-bebe-44fc-82f5-6120e6302e4e-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:27:09.938284 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:09.938269 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1233843-bebe-44fc-82f5-6120e6302e4e-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:27:09.939046 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:09.938283 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1233843-bebe-44fc-82f5-6120e6302e4e-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:27:09.939046 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:09.938297 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1233843-bebe-44fc-82f5-6120e6302e4e-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:27:09.941214 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:09.940195 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1233843-bebe-44fc-82f5-6120e6302e4e-istio-podinfo" pod "e1233843-bebe-44fc-82f5-6120e6302e4e" (UID: "e1233843-bebe-44fc-82f5-6120e6302e4e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:27:09.941214 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:09.941179 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1233843-bebe-44fc-82f5-6120e6302e4e-istio-envoy" pod "e1233843-bebe-44fc-82f5-6120e6302e4e" (UID: "e1233843-bebe-44fc-82f5-6120e6302e4e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:27:09.941977 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:09.941956 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1233843-bebe-44fc-82f5-6120e6302e4e-kube-api-access-zp49s" pod "e1233843-bebe-44fc-82f5-6120e6302e4e" (UID: "e1233843-bebe-44fc-82f5-6120e6302e4e"). InnerVolumeSpecName "kube-api-access-zp49s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:27:09.943946 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:09.943926 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1233843-bebe-44fc-82f5-6120e6302e4e-istio-token" pod "e1233843-bebe-44fc-82f5-6120e6302e4e" (UID: "e1233843-bebe-44fc-82f5-6120e6302e4e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:27:10.040242 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:10.040149 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp49s\" (UniqueName: \"kubernetes.io/projected/e1233843-bebe-44fc-82f5-6120e6302e4e-kube-api-access-zp49s\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:27:10.040242 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:10.040185 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1233843-bebe-44fc-82f5-6120e6302e4e-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:27:10.040242 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:10.040194 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1233843-bebe-44fc-82f5-6120e6302e4e-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:27:10.040242 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:10.040202 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1233843-bebe-44fc-82f5-6120e6302e4e-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:27:10.608381 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:10.608319 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c95c1e20-gvw7v"] Oct 31 04:27:10.609023 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:10.609003 5019 cpu_manager.go:414] "containerMap: removing container" podUID="e1233843-bebe-44fc-82f5-6120e6302e4e" containerName="istio-proxy" Oct 31 04:27:10.609023 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:10.609022 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1233843-bebe-44fc-82f5-6120e6302e4e" containerName="istio-proxy" Oct 31 04:27:10.609165 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:10.609036 5019 cpu_manager.go:414] "containerMap: removing container" podUID="e1233843-bebe-44fc-82f5-6120e6302e4e" containerName="istio-proxy" Oct 31 04:27:10.609165 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:10.609042 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1233843-bebe-44fc-82f5-6120e6302e4e" containerName="istio-proxy" Oct 31 04:27:10.609165 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:10.609164 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1233843-bebe-44fc-82f5-6120e6302e4e" containerName="istio-proxy" Oct 31 04:27:10.614399 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:10.614374 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c95c1e20-gvw7v" Oct 31 04:27:10.635357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:10.635002 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c95c1e20-gvw7v"] Oct 31 04:27:10.635698 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:10.635681 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:27:10.635826 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:10.635681 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:27:10.635923 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:10.635703 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:27:10.641850 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:10.641823 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:27:10.654016 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:10.653984 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50b851f6-c51d-4482-a1f5-94cfdfbc1741-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c95c1e20-gvw7v\" (UID: \"50b851f6-c51d-4482-a1f5-94cfdfbc1741\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c95c1e20-gvw7v" Oct 31 04:27:10.654192 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:10.654127 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50b851f6-c51d-4482-a1f5-94cfdfbc1741-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c95c1e20-gvw7v\" (UID: \"50b851f6-c51d-4482-a1f5-94cfdfbc1741\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c95c1e20-gvw7v" Oct 31 04:27:10.654313 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:10.654289 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24gv5\" (UniqueName: \"kubernetes.io/projected/50b851f6-c51d-4482-a1f5-94cfdfbc1741-kube-api-access-24gv5\") pod \"maas-api-revoke-sub-c95c1e20-gvw7v\" (UID: \"50b851f6-c51d-4482-a1f5-94cfdfbc1741\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c95c1e20-gvw7v" Oct 31 04:27:10.733121 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:10.733056 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff43e3-5d974c979f-gz8bs" podUID="3ef78a46-e363-4411-90b8-df50a676c9e5" containerName="payload-pre-processing" containerID="cri-o://66de9a18bedd6709bdbbf2bff81f9b590b839400af8e3bee6f3d3ff48fd3b44a" gracePeriod=25 Oct 31 04:27:10.755358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:10.755300 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50b851f6-c51d-4482-a1f5-94cfdfbc1741-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c95c1e20-gvw7v\" (UID: \"50b851f6-c51d-4482-a1f5-94cfdfbc1741\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c95c1e20-gvw7v" Oct 31 04:27:10.755467 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:10.755449 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24gv5\" (UniqueName: \"kubernetes.io/projected/50b851f6-c51d-4482-a1f5-94cfdfbc1741-kube-api-access-24gv5\") pod \"maas-api-revoke-sub-c95c1e20-gvw7v\" (UID: \"50b851f6-c51d-4482-a1f5-94cfdfbc1741\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c95c1e20-gvw7v" Oct 31 04:27:10.755645 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:10.755630 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50b851f6-c51d-4482-a1f5-94cfdfbc1741-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c95c1e20-gvw7v\" (UID: \"50b851f6-c51d-4482-a1f5-94cfdfbc1741\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c95c1e20-gvw7v" Oct 31 04:27:10.756297 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:10.756272 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50b851f6-c51d-4482-a1f5-94cfdfbc1741-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c95c1e20-gvw7v\" (UID: \"50b851f6-c51d-4482-a1f5-94cfdfbc1741\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c95c1e20-gvw7v" Oct 31 04:27:10.757585 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:10.757566 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50b851f6-c51d-4482-a1f5-94cfdfbc1741-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c95c1e20-gvw7v\" (UID: \"50b851f6-c51d-4482-a1f5-94cfdfbc1741\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c95c1e20-gvw7v" Oct 31 04:27:10.768283 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:10.768248 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f" event={"ID":"e1233843-bebe-44fc-82f5-6120e6302e4e","Type":"ContainerDied","Data":"46722d1f1ce73fd37c06390b0b97806ad251c7ce2245bd6717a88a49c93ffaaf"} Oct 31 04:27:10.768428 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:10.768375 5019 scope.go:122] "RemoveContainer" containerID="b5e0abae72d16354a761904815875d5117551d3c1daf2a01d129c93062cac34c" Oct 31 04:27:10.768801 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:10.768561 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f" Oct 31 04:27:10.805535 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:10.805494 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24gv5\" (UniqueName: \"kubernetes.io/projected/50b851f6-c51d-4482-a1f5-94cfdfbc1741-kube-api-access-24gv5\") pod \"maas-api-revoke-sub-c95c1e20-gvw7v\" (UID: \"50b851f6-c51d-4482-a1f5-94cfdfbc1741\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c95c1e20-gvw7v" Oct 31 04:27:10.910809 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:10.910640 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f"] Oct 31 04:27:10.910809 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:10.910720 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-a9b573-openshift-default-7c6bb5767f-tps4f"] Oct 31 04:27:10.931745 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:10.931701 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c95c1e20-gvw7v" Oct 31 04:27:11.319358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:11.314767 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff43e3-5d974c979f-gz8bs" Oct 31 04:27:11.412355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:11.408520 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ef78a46-e363-4411-90b8-df50a676c9e5-kube-api-access-ldfpr\" (UniqueName: \"kubernetes.io/projected/3ef78a46-e363-4411-90b8-df50a676c9e5-kube-api-access-ldfpr\") pod \"3ef78a46-e363-4411-90b8-df50a676c9e5\" (UID: \"3ef78a46-e363-4411-90b8-df50a676c9e5\") " Oct 31 04:27:11.412355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:11.408796 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ef78a46-e363-4411-90b8-df50a676c9e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef78a46-e363-4411-90b8-df50a676c9e5-plugins-config-volume\") pod \"3ef78a46-e363-4411-90b8-df50a676c9e5\" (UID: \"3ef78a46-e363-4411-90b8-df50a676c9e5\") " Oct 31 04:27:11.426359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:11.417558 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ef78a46-e363-4411-90b8-df50a676c9e5-plugins-config-volume" pod "3ef78a46-e363-4411-90b8-df50a676c9e5" (UID: "3ef78a46-e363-4411-90b8-df50a676c9e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:27:11.426359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:11.425041 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ef78a46-e363-4411-90b8-df50a676c9e5-kube-api-access-ldfpr" pod "3ef78a46-e363-4411-90b8-df50a676c9e5" (UID: "3ef78a46-e363-4411-90b8-df50a676c9e5"). InnerVolumeSpecName "kube-api-access-ldfpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:27:11.440600 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:11.440562 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1233843-bebe-44fc-82f5-6120e6302e4e" path="/var/lib/kubelet/pods/e1233843-bebe-44fc-82f5-6120e6302e4e/volumes" Oct 31 04:27:11.506877 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:11.506844 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c95c1e20-gvw7v"] Oct 31 04:27:11.510734 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:11.510702 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef78a46-e363-4411-90b8-df50a676c9e5-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:27:11.510848 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:11.510738 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldfpr\" (UniqueName: \"kubernetes.io/projected/3ef78a46-e363-4411-90b8-df50a676c9e5-kube-api-access-ldfpr\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:27:11.794354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:11.793444 5019 generic.go:356] "Generic (PLEG): container finished" podID="3ef78a46-e363-4411-90b8-df50a676c9e5" containerID="66de9a18bedd6709bdbbf2bff81f9b590b839400af8e3bee6f3d3ff48fd3b44a" exitCode=0 Oct 31 04:27:11.794354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:11.793573 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff43e3-5d974c979f-gz8bs" event={"ID":"3ef78a46-e363-4411-90b8-df50a676c9e5","Type":"ContainerDied","Data":"66de9a18bedd6709bdbbf2bff81f9b590b839400af8e3bee6f3d3ff48fd3b44a"} Oct 31 04:27:11.794354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:11.793624 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff43e3-5d974c979f-gz8bs" event={"ID":"3ef78a46-e363-4411-90b8-df50a676c9e5","Type":"ContainerDied","Data":"06b7a60a985153cc4ddde9858de6741023382639205ce95d9757a760ee8df265"} Oct 31 04:27:11.794354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:11.793651 5019 scope.go:122] "RemoveContainer" containerID="66de9a18bedd6709bdbbf2bff81f9b590b839400af8e3bee6f3d3ff48fd3b44a" Oct 31 04:27:11.794354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:11.793749 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff43e3-5d974c979f-gz8bs" Oct 31 04:27:11.809195 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:11.809159 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c95c1e20-gvw7v" event={"ID":"50b851f6-c51d-4482-a1f5-94cfdfbc1741","Type":"ContainerStarted","Data":"c609be38718d0a5299cb614494c4c0545485fab425ad60961f6aad04c2340add"} Oct 31 04:27:11.852636 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:11.852598 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ff43e3-5d974c979f-gz8bs"] Oct 31 04:27:11.861108 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:11.861077 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ff43e3-5d974c979f-gz8bs"] Oct 31 04:27:11.898537 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:11.898439 5019 scope.go:122] "RemoveContainer" containerID="66de9a18bedd6709bdbbf2bff81f9b590b839400af8e3bee6f3d3ff48fd3b44a" Oct 31 04:27:11.901415 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:27:11.898860 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66de9a18bedd6709bdbbf2bff81f9b590b839400af8e3bee6f3d3ff48fd3b44a\": container with ID starting with 66de9a18bedd6709bdbbf2bff81f9b590b839400af8e3bee6f3d3ff48fd3b44a not found: ID does not exist" containerID="66de9a18bedd6709bdbbf2bff81f9b590b839400af8e3bee6f3d3ff48fd3b44a" Oct 31 04:27:11.901415 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:11.898898 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66de9a18bedd6709bdbbf2bff81f9b590b839400af8e3bee6f3d3ff48fd3b44a"} err="failed to get container status \"66de9a18bedd6709bdbbf2bff81f9b590b839400af8e3bee6f3d3ff48fd3b44a\": rpc error: code = NotFound desc = could not find container \"66de9a18bedd6709bdbbf2bff81f9b590b839400af8e3bee6f3d3ff48fd3b44a\": container with ID starting with 66de9a18bedd6709bdbbf2bff81f9b590b839400af8e3bee6f3d3ff48fd3b44a not found: ID does not exist" Oct 31 04:27:12.819858 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:12.819814 5019 generic.go:356] "Generic (PLEG): container finished" podID="50b851f6-c51d-4482-a1f5-94cfdfbc1741" containerID="c8070e1d0199dde9e4637f7bcc0c68218eaa11e1f91dfbc6eee8b82dd4913fb7" exitCode=0 Oct 31 04:27:12.820362 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:12.819888 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c95c1e20-gvw7v" event={"ID":"50b851f6-c51d-4482-a1f5-94cfdfbc1741","Type":"ContainerDied","Data":"c8070e1d0199dde9e4637f7bcc0c68218eaa11e1f91dfbc6eee8b82dd4913fb7"} Oct 31 04:27:13.412372 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:13.412267 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ef78a46-e363-4411-90b8-df50a676c9e5" path="/var/lib/kubelet/pods/3ef78a46-e363-4411-90b8-df50a676c9e5/volumes" Oct 31 04:27:13.832560 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:13.832529 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d1d441-669694c87-fhkbx"] Oct 31 04:27:13.833180 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:13.833165 5019 cpu_manager.go:414] "containerMap: removing container" podUID="3ef78a46-e363-4411-90b8-df50a676c9e5" containerName="payload-pre-processing" Oct 31 04:27:13.833180 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:13.833179 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ef78a46-e363-4411-90b8-df50a676c9e5" containerName="payload-pre-processing" Oct 31 04:27:13.833301 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:13.833289 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1233843-bebe-44fc-82f5-6120e6302e4e" containerName="istio-proxy" Oct 31 04:27:13.833370 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:13.833301 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ef78a46-e363-4411-90b8-df50a676c9e5" containerName="payload-pre-processing" Oct 31 04:27:13.837083 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:13.837061 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d1d441-669694c87-fhkbx" Oct 31 04:27:13.903355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:13.892684 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-d1d441-dockercfg-25dnt\"" Oct 31 04:27:13.903355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:13.892857 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-d1d441\"" Oct 31 04:27:13.954624 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:13.954571 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d1d441-669694c87-fhkbx"] Oct 31 04:27:13.955549 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:13.955504 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66wkc\" (UniqueName: \"kubernetes.io/projected/52e6c267-5637-4fa5-9600-47b905bf75ee-kube-api-access-66wkc\") pod \"payload-pre-processing-e2e-praxis-a-d1d441-669694c87-fhkbx\" (UID: \"52e6c267-5637-4fa5-9600-47b905bf75ee\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d1d441-669694c87-fhkbx" Oct 31 04:27:13.955687 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:13.955672 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52e6c267-5637-4fa5-9600-47b905bf75ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-d1d441-669694c87-fhkbx\" (UID: \"52e6c267-5637-4fa5-9600-47b905bf75ee\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d1d441-669694c87-fhkbx" Oct 31 04:27:14.024575 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:14.023669 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d1d441-588dbc6899-4rnfz"] Oct 31 04:27:14.030014 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:14.029266 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d1d441-588dbc6899-4rnfz" Oct 31 04:27:14.052576 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:14.052122 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d1d441-588dbc6899-4rnfz"] Oct 31 04:27:14.059201 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:14.058424 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52e6c267-5637-4fa5-9600-47b905bf75ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-d1d441-669694c87-fhkbx\" (UID: \"52e6c267-5637-4fa5-9600-47b905bf75ee\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d1d441-669694c87-fhkbx" Oct 31 04:27:14.059201 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:14.058542 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt5qf\" (UniqueName: \"kubernetes.io/projected/63386d30-7d65-4c86-9362-0ee7b8143bb1-kube-api-access-rt5qf\") pod \"payload-processing-e2e-praxis-a-d1d441-588dbc6899-4rnfz\" (UID: \"63386d30-7d65-4c86-9362-0ee7b8143bb1\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d1d441-588dbc6899-4rnfz" Oct 31 04:27:14.059201 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:14.058636 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63386d30-7d65-4c86-9362-0ee7b8143bb1-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-d1d441-588dbc6899-4rnfz\" (UID: \"63386d30-7d65-4c86-9362-0ee7b8143bb1\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d1d441-588dbc6899-4rnfz" Oct 31 04:27:14.059201 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:14.058726 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66wkc\" (UniqueName: \"kubernetes.io/projected/52e6c267-5637-4fa5-9600-47b905bf75ee-kube-api-access-66wkc\") pod \"payload-pre-processing-e2e-praxis-a-d1d441-669694c87-fhkbx\" (UID: \"52e6c267-5637-4fa5-9600-47b905bf75ee\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d1d441-669694c87-fhkbx" Oct 31 04:27:14.059557 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:14.059344 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52e6c267-5637-4fa5-9600-47b905bf75ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-d1d441-669694c87-fhkbx\" (UID: \"52e6c267-5637-4fa5-9600-47b905bf75ee\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d1d441-669694c87-fhkbx" Oct 31 04:27:14.104173 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:14.104071 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66wkc\" (UniqueName: \"kubernetes.io/projected/52e6c267-5637-4fa5-9600-47b905bf75ee-kube-api-access-66wkc\") pod \"payload-pre-processing-e2e-praxis-a-d1d441-669694c87-fhkbx\" (UID: \"52e6c267-5637-4fa5-9600-47b905bf75ee\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d1d441-669694c87-fhkbx" Oct 31 04:27:14.155198 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:14.155167 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d1d441-669694c87-fhkbx" Oct 31 04:27:14.160440 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:14.160411 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt5qf\" (UniqueName: \"kubernetes.io/projected/63386d30-7d65-4c86-9362-0ee7b8143bb1-kube-api-access-rt5qf\") pod \"payload-processing-e2e-praxis-a-d1d441-588dbc6899-4rnfz\" (UID: \"63386d30-7d65-4c86-9362-0ee7b8143bb1\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d1d441-588dbc6899-4rnfz" Oct 31 04:27:14.160563 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:14.160502 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63386d30-7d65-4c86-9362-0ee7b8143bb1-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-d1d441-588dbc6899-4rnfz\" (UID: \"63386d30-7d65-4c86-9362-0ee7b8143bb1\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d1d441-588dbc6899-4rnfz" Oct 31 04:27:14.161380 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:14.161354 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63386d30-7d65-4c86-9362-0ee7b8143bb1-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-d1d441-588dbc6899-4rnfz\" (UID: \"63386d30-7d65-4c86-9362-0ee7b8143bb1\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d1d441-588dbc6899-4rnfz" Oct 31 04:27:14.187020 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:14.186994 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt5qf\" (UniqueName: \"kubernetes.io/projected/63386d30-7d65-4c86-9362-0ee7b8143bb1-kube-api-access-rt5qf\") pod \"payload-processing-e2e-praxis-a-d1d441-588dbc6899-4rnfz\" (UID: \"63386d30-7d65-4c86-9362-0ee7b8143bb1\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d1d441-588dbc6899-4rnfz" Oct 31 04:27:14.231268 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:14.230973 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c95c1e20-gvw7v" Oct 31 04:27:14.263606 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:14.261872 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50b851f6-c51d-4482-a1f5-94cfdfbc1741-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50b851f6-c51d-4482-a1f5-94cfdfbc1741-maas-api-openshift-ca\") pod \"50b851f6-c51d-4482-a1f5-94cfdfbc1741\" (UID: \"50b851f6-c51d-4482-a1f5-94cfdfbc1741\") " Oct 31 04:27:14.263606 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:14.261939 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50b851f6-c51d-4482-a1f5-94cfdfbc1741-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50b851f6-c51d-4482-a1f5-94cfdfbc1741-maas-api-xks-ca\") pod \"50b851f6-c51d-4482-a1f5-94cfdfbc1741\" (UID: \"50b851f6-c51d-4482-a1f5-94cfdfbc1741\") " Oct 31 04:27:14.263606 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:14.262037 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50b851f6-c51d-4482-a1f5-94cfdfbc1741-kube-api-access-24gv5\" (UniqueName: \"kubernetes.io/projected/50b851f6-c51d-4482-a1f5-94cfdfbc1741-kube-api-access-24gv5\") pod \"50b851f6-c51d-4482-a1f5-94cfdfbc1741\" (UID: \"50b851f6-c51d-4482-a1f5-94cfdfbc1741\") " Oct 31 04:27:14.263606 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:14.263566 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50b851f6-c51d-4482-a1f5-94cfdfbc1741-maas-api-openshift-ca" pod "50b851f6-c51d-4482-a1f5-94cfdfbc1741" (UID: "50b851f6-c51d-4482-a1f5-94cfdfbc1741"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:27:14.269589 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:14.268996 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50b851f6-c51d-4482-a1f5-94cfdfbc1741-kube-api-access-24gv5" pod "50b851f6-c51d-4482-a1f5-94cfdfbc1741" (UID: "50b851f6-c51d-4482-a1f5-94cfdfbc1741"). InnerVolumeSpecName "kube-api-access-24gv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:27:14.293985 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:14.293938 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50b851f6-c51d-4482-a1f5-94cfdfbc1741-maas-api-xks-ca" pod "50b851f6-c51d-4482-a1f5-94cfdfbc1741" (UID: "50b851f6-c51d-4482-a1f5-94cfdfbc1741"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:27:14.358433 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:14.358324 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d1d441-588dbc6899-4rnfz" Oct 31 04:27:14.364664 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:14.364631 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24gv5\" (UniqueName: \"kubernetes.io/projected/50b851f6-c51d-4482-a1f5-94cfdfbc1741-kube-api-access-24gv5\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:27:14.364664 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:14.364665 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50b851f6-c51d-4482-a1f5-94cfdfbc1741-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:27:14.364858 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:14.364678 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50b851f6-c51d-4482-a1f5-94cfdfbc1741-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:27:14.496055 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:14.495904 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d1d441-669694c87-fhkbx"] Oct 31 04:27:14.498935 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:27:14.498895 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52e6c267_5637_4fa5_9600_47b905bf75ee.slice/crio-02208dffebe04c90a42933964e85bf762a793a66b734dba99f2d242dd0128887 WatchSource:0}: Error finding container 02208dffebe04c90a42933964e85bf762a793a66b734dba99f2d242dd0128887: Status 404 returned error can't find the container with id 02208dffebe04c90a42933964e85bf762a793a66b734dba99f2d242dd0128887 Oct 31 04:27:14.663321 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:14.663292 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d1d441-588dbc6899-4rnfz"] Oct 31 04:27:14.663878 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:27:14.663851 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63386d30_7d65_4c86_9362_0ee7b8143bb1.slice/crio-c8c6e25a97e83932a3f12134e430994e84afcdf751bb33090c6631b46ca13cc5 WatchSource:0}: Error finding container c8c6e25a97e83932a3f12134e430994e84afcdf751bb33090c6631b46ca13cc5: Status 404 returned error can't find the container with id c8c6e25a97e83932a3f12134e430994e84afcdf751bb33090c6631b46ca13cc5 Oct 31 04:27:14.835413 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:14.835371 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d1d441-669694c87-fhkbx" event={"ID":"52e6c267-5637-4fa5-9600-47b905bf75ee","Type":"ContainerStarted","Data":"61c7da3fb92f673ff6b9332476bb7321bde96e0476b6509ff058a44caee7e765"} Oct 31 04:27:14.835919 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:14.835440 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d1d441-669694c87-fhkbx" event={"ID":"52e6c267-5637-4fa5-9600-47b905bf75ee","Type":"ContainerStarted","Data":"02208dffebe04c90a42933964e85bf762a793a66b734dba99f2d242dd0128887"} Oct 31 04:27:14.835919 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:14.835594 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d1d441-669694c87-fhkbx" Oct 31 04:27:14.837558 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:14.837532 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d1d441-588dbc6899-4rnfz" event={"ID":"63386d30-7d65-4c86-9362-0ee7b8143bb1","Type":"ContainerStarted","Data":"dfbccaa07cd67f232fb76ed4abf8d821cedb480b743cf070f3c3073e8dcb3c19"} Oct 31 04:27:14.837701 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:14.837571 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d1d441-588dbc6899-4rnfz" event={"ID":"63386d30-7d65-4c86-9362-0ee7b8143bb1","Type":"ContainerStarted","Data":"c8c6e25a97e83932a3f12134e430994e84afcdf751bb33090c6631b46ca13cc5"} Oct 31 04:27:14.837701 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:14.837607 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-d1d441-588dbc6899-4rnfz" Oct 31 04:27:14.838933 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:14.838910 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c95c1e20-gvw7v" event={"ID":"50b851f6-c51d-4482-a1f5-94cfdfbc1741","Type":"ContainerDied","Data":"c609be38718d0a5299cb614494c4c0545485fab425ad60961f6aad04c2340add"} Oct 31 04:27:14.839035 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:14.838947 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c609be38718d0a5299cb614494c4c0545485fab425ad60961f6aad04c2340add" Oct 31 04:27:14.839035 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:14.838917 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c95c1e20-gvw7v" Oct 31 04:27:14.876908 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:14.876843 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d1d441-669694c87-fhkbx" podStartSLOduration=1.876824344 podStartE2EDuration="1.876824344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:27:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:27:14.874674213 +0000 UTC m=+1389.631012253" watchObservedRunningTime="2026-10-31 04:27:14.876824344 +0000 UTC m=+1389.633162383" Oct 31 04:27:14.917363 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:14.917223 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-d1d441-588dbc6899-4rnfz" podStartSLOduration=1.917202566 podStartE2EDuration="1.917202566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:27:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:27:14.91495051 +0000 UTC m=+1389.671288552" watchObservedRunningTime="2026-10-31 04:27:14.917202566 +0000 UTC m=+1389.673540604" Oct 31 04:27:15.195840 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:15.195757 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4" Oct 31 04:27:15.286583 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:15.286539 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b35297c9-af92-4356-9951-2a90b09fb3cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b35297c9-af92-4356-9951-2a90b09fb3cd-istio-podinfo\") pod \"b35297c9-af92-4356-9951-2a90b09fb3cd\" (UID: \"b35297c9-af92-4356-9951-2a90b09fb3cd\") " Oct 31 04:27:15.286777 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:15.286696 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b35297c9-af92-4356-9951-2a90b09fb3cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b35297c9-af92-4356-9951-2a90b09fb3cd-istiod-ca-cert\") pod \"b35297c9-af92-4356-9951-2a90b09fb3cd\" (UID: \"b35297c9-af92-4356-9951-2a90b09fb3cd\") " Oct 31 04:27:15.286777 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:15.286771 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b35297c9-af92-4356-9951-2a90b09fb3cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b35297c9-af92-4356-9951-2a90b09fb3cd-workload-certs\") pod \"b35297c9-af92-4356-9951-2a90b09fb3cd\" (UID: \"b35297c9-af92-4356-9951-2a90b09fb3cd\") " Oct 31 04:27:15.286897 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:15.286837 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b35297c9-af92-4356-9951-2a90b09fb3cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b35297c9-af92-4356-9951-2a90b09fb3cd-workload-socket\") pod \"b35297c9-af92-4356-9951-2a90b09fb3cd\" (UID: \"b35297c9-af92-4356-9951-2a90b09fb3cd\") " Oct 31 04:27:15.286951 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:15.286922 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b35297c9-af92-4356-9951-2a90b09fb3cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b35297c9-af92-4356-9951-2a90b09fb3cd-istio-data\") pod \"b35297c9-af92-4356-9951-2a90b09fb3cd\" (UID: \"b35297c9-af92-4356-9951-2a90b09fb3cd\") " Oct 31 04:27:15.287009 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:15.286954 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b35297c9-af92-4356-9951-2a90b09fb3cd-kube-api-access-qnfzt\" (UniqueName: \"kubernetes.io/projected/b35297c9-af92-4356-9951-2a90b09fb3cd-kube-api-access-qnfzt\") pod \"b35297c9-af92-4356-9951-2a90b09fb3cd\" (UID: \"b35297c9-af92-4356-9951-2a90b09fb3cd\") " Oct 31 04:27:15.287009 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:15.286999 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b35297c9-af92-4356-9951-2a90b09fb3cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b35297c9-af92-4356-9951-2a90b09fb3cd-credential-socket\") pod \"b35297c9-af92-4356-9951-2a90b09fb3cd\" (UID: \"b35297c9-af92-4356-9951-2a90b09fb3cd\") " Oct 31 04:27:15.287112 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:15.287077 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b35297c9-af92-4356-9951-2a90b09fb3cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b35297c9-af92-4356-9951-2a90b09fb3cd-istio-envoy\") pod \"b35297c9-af92-4356-9951-2a90b09fb3cd\" (UID: \"b35297c9-af92-4356-9951-2a90b09fb3cd\") " Oct 31 04:27:15.287165 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:15.287144 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b35297c9-af92-4356-9951-2a90b09fb3cd-istio-token\" (UniqueName: \"kubernetes.io/projected/b35297c9-af92-4356-9951-2a90b09fb3cd-istio-token\") pod \"b35297c9-af92-4356-9951-2a90b09fb3cd\" (UID: \"b35297c9-af92-4356-9951-2a90b09fb3cd\") " Oct 31 04:27:15.288420 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:15.288388 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b35297c9-af92-4356-9951-2a90b09fb3cd-workload-socket" pod "b35297c9-af92-4356-9951-2a90b09fb3cd" (UID: "b35297c9-af92-4356-9951-2a90b09fb3cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:27:15.289309 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:15.289280 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b35297c9-af92-4356-9951-2a90b09fb3cd-workload-certs" pod "b35297c9-af92-4356-9951-2a90b09fb3cd" (UID: "b35297c9-af92-4356-9951-2a90b09fb3cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:27:15.289446 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:15.289367 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b35297c9-af92-4356-9951-2a90b09fb3cd-credential-socket" pod "b35297c9-af92-4356-9951-2a90b09fb3cd" (UID: "b35297c9-af92-4356-9951-2a90b09fb3cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:27:15.289446 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:15.289394 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b35297c9-af92-4356-9951-2a90b09fb3cd-istio-data" pod "b35297c9-af92-4356-9951-2a90b09fb3cd" (UID: "b35297c9-af92-4356-9951-2a90b09fb3cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:27:15.289979 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:15.289949 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b35297c9-af92-4356-9951-2a90b09fb3cd-istiod-ca-cert" pod "b35297c9-af92-4356-9951-2a90b09fb3cd" (UID: "b35297c9-af92-4356-9951-2a90b09fb3cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:27:15.304982 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:15.304948 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b35297c9-af92-4356-9951-2a90b09fb3cd-kube-api-access-qnfzt" pod "b35297c9-af92-4356-9951-2a90b09fb3cd" (UID: "b35297c9-af92-4356-9951-2a90b09fb3cd"). InnerVolumeSpecName "kube-api-access-qnfzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:27:15.305146 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:15.305101 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b35297c9-af92-4356-9951-2a90b09fb3cd-istio-podinfo" pod "b35297c9-af92-4356-9951-2a90b09fb3cd" (UID: "b35297c9-af92-4356-9951-2a90b09fb3cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:27:15.305250 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:15.305231 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b35297c9-af92-4356-9951-2a90b09fb3cd-istio-token" pod "b35297c9-af92-4356-9951-2a90b09fb3cd" (UID: "b35297c9-af92-4356-9951-2a90b09fb3cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:27:15.307467 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:15.307424 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b35297c9-af92-4356-9951-2a90b09fb3cd-istio-envoy" pod "b35297c9-af92-4356-9951-2a90b09fb3cd" (UID: "b35297c9-af92-4356-9951-2a90b09fb3cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:27:15.389430 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:15.389317 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b35297c9-af92-4356-9951-2a90b09fb3cd-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:27:15.389430 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:15.389372 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b35297c9-af92-4356-9951-2a90b09fb3cd-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:27:15.389430 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:15.389387 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b35297c9-af92-4356-9951-2a90b09fb3cd-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:27:15.389430 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:15.389402 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b35297c9-af92-4356-9951-2a90b09fb3cd-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:27:15.389430 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:15.389414 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b35297c9-af92-4356-9951-2a90b09fb3cd-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:27:15.389430 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:15.389428 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b35297c9-af92-4356-9951-2a90b09fb3cd-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:27:15.389842 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:15.389441 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnfzt\" (UniqueName: \"kubernetes.io/projected/b35297c9-af92-4356-9951-2a90b09fb3cd-kube-api-access-qnfzt\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:27:15.389842 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:15.389455 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b35297c9-af92-4356-9951-2a90b09fb3cd-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:27:15.389842 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:15.389470 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b35297c9-af92-4356-9951-2a90b09fb3cd-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:27:15.849405 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:15.849367 5019 generic.go:356] "Generic (PLEG): container finished" podID="b35297c9-af92-4356-9951-2a90b09fb3cd" containerID="eb9ef275fc5a81b1ef005c5b04d7a927d89d420a8dda42bb37679e3d305ba699" exitCode=0 Oct 31 04:27:15.849814 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:15.849428 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4" Oct 31 04:27:15.849814 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:15.849463 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4" event={"ID":"b35297c9-af92-4356-9951-2a90b09fb3cd","Type":"ContainerDied","Data":"eb9ef275fc5a81b1ef005c5b04d7a927d89d420a8dda42bb37679e3d305ba699"} Oct 31 04:27:15.849814 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:15.849528 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4" event={"ID":"b35297c9-af92-4356-9951-2a90b09fb3cd","Type":"ContainerDied","Data":"8c27acbf208c35d6b3a120e7a5aa367dae0cd8bce64effde3a7740eb8267656b"} Oct 31 04:27:15.849814 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:15.849580 5019 scope.go:122] "RemoveContainer" containerID="eb9ef275fc5a81b1ef005c5b04d7a927d89d420a8dda42bb37679e3d305ba699" Oct 31 04:27:15.863572 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:15.863553 5019 scope.go:122] "RemoveContainer" containerID="eb9ef275fc5a81b1ef005c5b04d7a927d89d420a8dda42bb37679e3d305ba699" Oct 31 04:27:15.863832 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:27:15.863813 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb9ef275fc5a81b1ef005c5b04d7a927d89d420a8dda42bb37679e3d305ba699\": container with ID starting with eb9ef275fc5a81b1ef005c5b04d7a927d89d420a8dda42bb37679e3d305ba699 not found: ID does not exist" containerID="eb9ef275fc5a81b1ef005c5b04d7a927d89d420a8dda42bb37679e3d305ba699" Oct 31 04:27:15.863905 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:15.863844 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb9ef275fc5a81b1ef005c5b04d7a927d89d420a8dda42bb37679e3d305ba699"} err="failed to get container status \"eb9ef275fc5a81b1ef005c5b04d7a927d89d420a8dda42bb37679e3d305ba699\": rpc error: code = NotFound desc = could not find container \"eb9ef275fc5a81b1ef005c5b04d7a927d89d420a8dda42bb37679e3d305ba699\": container with ID starting with eb9ef275fc5a81b1ef005c5b04d7a927d89d420a8dda42bb37679e3d305ba699 not found: ID does not exist" Oct 31 04:27:15.904410 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:15.904381 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4"] Oct 31 04:27:15.914520 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:15.914491 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2kmj4"] Oct 31 04:27:16.687603 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:16.687566 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-9ee727-7f9ccf9d5-r5j4l" Oct 31 04:27:16.687762 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:16.687671 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9ee727-d7d75f767-qq7nn" Oct 31 04:27:17.417485 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:17.417377 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b35297c9-af92-4356-9951-2a90b09fb3cd" path="/var/lib/kubelet/pods/b35297c9-af92-4356-9951-2a90b09fb3cd/volumes" Oct 31 04:27:17.866825 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:17.866790 5019 generic.go:356] "Generic (PLEG): container finished" podID="a473bd3b-e50d-4a3d-ad54-fd49ee49f144" containerID="ae4735ff03902e2365dcc8f480457ab57d0e20248b7405414b828b8d3667dca5" exitCode=0 Oct 31 04:27:17.867006 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:17.866860 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd" event={"ID":"a473bd3b-e50d-4a3d-ad54-fd49ee49f144","Type":"ContainerDied","Data":"ae4735ff03902e2365dcc8f480457ab57d0e20248b7405414b828b8d3667dca5"} Oct 31 04:27:17.867298 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:17.867283 5019 scope.go:122] "RemoveContainer" containerID="ae4735ff03902e2365dcc8f480457ab57d0e20248b7405414b828b8d3667dca5" Oct 31 04:27:17.868762 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:17.868734 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:27:17.868838 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:17.868800 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:27:17.868838 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:17.868832 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:27:18.003902 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:17.998601 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd" Oct 31 04:27:18.897760 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:18.897707 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd" event={"ID":"a473bd3b-e50d-4a3d-ad54-fd49ee49f144","Type":"ContainerStarted","Data":"e8878179de9af734b48958325224021be55c4d6c045b1c39fa15d2c27a9da8b3"} Oct 31 04:27:19.735142 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:19.735107 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd" Oct 31 04:27:19.736515 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:19.736489 5019 patch_prober.go:29] interesting pod/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Oct 31 04:27:19.736642 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:19.736538 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd" podUID="a473bd3b-e50d-4a3d-ad54-fd49ee49f144" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Oct 31 04:27:19.759431 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:19.759406 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-c6859a-5f796cd5cf-gmshg" Oct 31 04:27:20.734652 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:20.734616 5019 patch_prober.go:29] interesting pod/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Oct 31 04:27:20.735047 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:20.734665 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd" podUID="a473bd3b-e50d-4a3d-ad54-fd49ee49f144" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Oct 31 04:27:21.735471 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:21.735429 5019 patch_prober.go:29] interesting pod/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Oct 31 04:27:21.735898 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:21.735484 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd" podUID="a473bd3b-e50d-4a3d-ad54-fd49ee49f144" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Oct 31 04:27:22.417904 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:27:22.417865 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:27:22.735422 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:22.735384 5019 patch_prober.go:29] interesting pod/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Oct 31 04:27:22.735624 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:22.735450 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd" podUID="a473bd3b-e50d-4a3d-ad54-fd49ee49f144" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Oct 31 04:27:23.735109 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:23.735071 5019 patch_prober.go:29] interesting pod/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Oct 31 04:27:23.735295 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:23.735132 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd" podUID="a473bd3b-e50d-4a3d-ad54-fd49ee49f144" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Oct 31 04:27:24.740227 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:24.740198 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd" Oct 31 04:27:24.740784 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:24.740545 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd" Oct 31 04:27:24.741434 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:24.741413 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd" Oct 31 04:27:25.361190 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:25.361154 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ff43e3-67d8bf5499-ttklw"] Oct 31 04:27:25.362455 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:25.362427 5019 cpu_manager.go:414] "containerMap: removing container" podUID="b35297c9-af92-4356-9951-2a90b09fb3cd" containerName="istio-proxy" Oct 31 04:27:25.362637 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:25.362621 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b35297c9-af92-4356-9951-2a90b09fb3cd" containerName="istio-proxy" Oct 31 04:27:25.362763 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:25.362752 5019 cpu_manager.go:414] "containerMap: removing container" podUID="50b851f6-c51d-4482-a1f5-94cfdfbc1741" containerName="revoke-keys" Oct 31 04:27:25.362838 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:25.362828 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50b851f6-c51d-4482-a1f5-94cfdfbc1741" containerName="revoke-keys" Oct 31 04:27:25.363077 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:25.363063 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="50b851f6-c51d-4482-a1f5-94cfdfbc1741" containerName="revoke-keys" Oct 31 04:27:25.363181 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:25.363171 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="b35297c9-af92-4356-9951-2a90b09fb3cd" containerName="istio-proxy" Oct 31 04:27:25.367997 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:25.367977 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff43e3-67d8bf5499-ttklw" Oct 31 04:27:25.386252 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:25.386222 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-ff43e3-dockercfg-gfq7g\"" Oct 31 04:27:25.386643 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:25.386592 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-ff43e3\"" Oct 31 04:27:25.391320 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:25.391252 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ff43e3-67d8bf5499-ttklw"] Oct 31 04:27:25.473187 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:25.473130 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28fc13fa-c89f-459e-9ffc-19a9f85bfec9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ff43e3-67d8bf5499-ttklw\" (UID: \"28fc13fa-c89f-459e-9ffc-19a9f85bfec9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff43e3-67d8bf5499-ttklw" Oct 31 04:27:25.483262 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:25.483226 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v8vc\" (UniqueName: \"kubernetes.io/projected/28fc13fa-c89f-459e-9ffc-19a9f85bfec9-kube-api-access-5v8vc\") pod \"payload-pre-processing-e2e-ipp-a-ff43e3-67d8bf5499-ttklw\" (UID: \"28fc13fa-c89f-459e-9ffc-19a9f85bfec9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff43e3-67d8bf5499-ttklw" Oct 31 04:27:25.509639 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:25.507384 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ff43e3-944445948-ph2kl"] Oct 31 04:27:25.512699 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:25.512678 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff43e3-944445948-ph2kl" Oct 31 04:27:25.550804 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:25.550769 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ff43e3-944445948-ph2kl"] Oct 31 04:27:25.590968 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:25.590940 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79838bec-a733-4da9-9f5b-d476f8de3952-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ff43e3-944445948-ph2kl\" (UID: \"79838bec-a733-4da9-9f5b-d476f8de3952\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ff43e3-944445948-ph2kl" Oct 31 04:27:25.591266 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:25.591241 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28fc13fa-c89f-459e-9ffc-19a9f85bfec9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ff43e3-67d8bf5499-ttklw\" (UID: \"28fc13fa-c89f-459e-9ffc-19a9f85bfec9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff43e3-67d8bf5499-ttklw" Oct 31 04:27:25.591373 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:25.591280 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nb7s\" (UniqueName: \"kubernetes.io/projected/79838bec-a733-4da9-9f5b-d476f8de3952-kube-api-access-8nb7s\") pod \"payload-processing-e2e-ipp-a-ff43e3-944445948-ph2kl\" (UID: \"79838bec-a733-4da9-9f5b-d476f8de3952\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ff43e3-944445948-ph2kl" Oct 31 04:27:25.591438 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:25.591397 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v8vc\" (UniqueName: \"kubernetes.io/projected/28fc13fa-c89f-459e-9ffc-19a9f85bfec9-kube-api-access-5v8vc\") pod \"payload-pre-processing-e2e-ipp-a-ff43e3-67d8bf5499-ttklw\" (UID: \"28fc13fa-c89f-459e-9ffc-19a9f85bfec9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff43e3-67d8bf5499-ttklw" Oct 31 04:27:25.592407 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:25.592384 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28fc13fa-c89f-459e-9ffc-19a9f85bfec9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ff43e3-67d8bf5499-ttklw\" (UID: \"28fc13fa-c89f-459e-9ffc-19a9f85bfec9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff43e3-67d8bf5499-ttklw" Oct 31 04:27:25.614683 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:25.614618 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v8vc\" (UniqueName: \"kubernetes.io/projected/28fc13fa-c89f-459e-9ffc-19a9f85bfec9-kube-api-access-5v8vc\") pod \"payload-pre-processing-e2e-ipp-a-ff43e3-67d8bf5499-ttklw\" (UID: \"28fc13fa-c89f-459e-9ffc-19a9f85bfec9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff43e3-67d8bf5499-ttklw" Oct 31 04:27:25.693289 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:25.693260 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nb7s\" (UniqueName: \"kubernetes.io/projected/79838bec-a733-4da9-9f5b-d476f8de3952-kube-api-access-8nb7s\") pod \"payload-processing-e2e-ipp-a-ff43e3-944445948-ph2kl\" (UID: \"79838bec-a733-4da9-9f5b-d476f8de3952\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ff43e3-944445948-ph2kl" Oct 31 04:27:25.693529 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:25.693513 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79838bec-a733-4da9-9f5b-d476f8de3952-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ff43e3-944445948-ph2kl\" (UID: \"79838bec-a733-4da9-9f5b-d476f8de3952\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ff43e3-944445948-ph2kl" Oct 31 04:27:25.694359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:25.694320 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79838bec-a733-4da9-9f5b-d476f8de3952-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ff43e3-944445948-ph2kl\" (UID: \"79838bec-a733-4da9-9f5b-d476f8de3952\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ff43e3-944445948-ph2kl" Oct 31 04:27:25.716137 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:25.716105 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff43e3-67d8bf5499-ttklw" Oct 31 04:27:25.716468 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:25.716449 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nb7s\" (UniqueName: \"kubernetes.io/projected/79838bec-a733-4da9-9f5b-d476f8de3952-kube-api-access-8nb7s\") pod \"payload-processing-e2e-ipp-a-ff43e3-944445948-ph2kl\" (UID: \"79838bec-a733-4da9-9f5b-d476f8de3952\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ff43e3-944445948-ph2kl" Oct 31 04:27:25.852457 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:25.852425 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff43e3-944445948-ph2kl" Oct 31 04:27:25.852880 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:25.852765 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-d1d441-588dbc6899-4rnfz" Oct 31 04:27:25.852880 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:25.852822 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d1d441-669694c87-fhkbx" Oct 31 04:27:26.022401 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:26.022370 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ff43e3-67d8bf5499-ttklw"] Oct 31 04:27:26.023601 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:27:26.023576 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28fc13fa_c89f_459e_9ffc_19a9f85bfec9.slice/crio-17a6f66adc58e43b5464329f2e2c5099da1ec33f4c96feb28d2b0c0083884d7b WatchSource:0}: Error finding container 17a6f66adc58e43b5464329f2e2c5099da1ec33f4c96feb28d2b0c0083884d7b: Status 404 returned error can't find the container with id 17a6f66adc58e43b5464329f2e2c5099da1ec33f4c96feb28d2b0c0083884d7b Oct 31 04:27:26.158162 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:26.158139 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ff43e3-944445948-ph2kl"] Oct 31 04:27:26.158543 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:27:26.158518 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79838bec_a733_4da9_9f5b_d476f8de3952.slice/crio-93fa06a3a81bb89d9a087c5a79e6951ac0e1342e4834ca940af63d341b84d348 WatchSource:0}: Error finding container 93fa06a3a81bb89d9a087c5a79e6951ac0e1342e4834ca940af63d341b84d348: Status 404 returned error can't find the container with id 93fa06a3a81bb89d9a087c5a79e6951ac0e1342e4834ca940af63d341b84d348 Oct 31 04:27:26.984368 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:26.984293 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff43e3-67d8bf5499-ttklw" event={"ID":"28fc13fa-c89f-459e-9ffc-19a9f85bfec9","Type":"ContainerStarted","Data":"fb6588baf0950aecb87ceb0c82a23946da67ed7df45a147d2ab3d90bf069b1b2"} Oct 31 04:27:26.984844 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:26.984398 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff43e3-67d8bf5499-ttklw" event={"ID":"28fc13fa-c89f-459e-9ffc-19a9f85bfec9","Type":"ContainerStarted","Data":"17a6f66adc58e43b5464329f2e2c5099da1ec33f4c96feb28d2b0c0083884d7b"} Oct 31 04:27:26.984935 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:26.984914 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff43e3-67d8bf5499-ttklw" Oct 31 04:27:26.986635 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:26.986596 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff43e3-944445948-ph2kl" event={"ID":"79838bec-a733-4da9-9f5b-d476f8de3952","Type":"ContainerStarted","Data":"a628d3f0370ce429848f03771fa47fea4509d04c289664a1739302b165e8ff31"} Oct 31 04:27:26.986778 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:26.986644 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff43e3-944445948-ph2kl" event={"ID":"79838bec-a733-4da9-9f5b-d476f8de3952","Type":"ContainerStarted","Data":"93fa06a3a81bb89d9a087c5a79e6951ac0e1342e4834ca940af63d341b84d348"} Oct 31 04:27:26.986778 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:26.986745 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff43e3-944445948-ph2kl" Oct 31 04:27:27.028568 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:27.028526 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff43e3-67d8bf5499-ttklw" podStartSLOduration=2.028514587 podStartE2EDuration="2.028514587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:27:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:27:27.027317085 +0000 UTC m=+1401.783655123" watchObservedRunningTime="2026-10-31 04:27:27.028514587 +0000 UTC m=+1401.784852661" Oct 31 04:27:27.072452 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:27.072401 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff43e3-944445948-ph2kl" podStartSLOduration=2.072386432 podStartE2EDuration="2.072386432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:27:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:27:27.068232104 +0000 UTC m=+1401.824570138" watchObservedRunningTime="2026-10-31 04:27:27.072386432 +0000 UTC m=+1401.828724472" Oct 31 04:27:33.750356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:33.734305 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f91cd1-7f59958f95-c2qrv"] Oct 31 04:27:33.750356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:33.734588 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91cd1-7f59958f95-c2qrv" podUID="5c95c936-5fcf-4bb0-8701-09f544af2fe0" containerName="payload-pre-processing" containerID="cri-o://8ba7f92a6cfc96cd7b27986b3603109da5ee467c6ec21fdfb8e261a4529e02a9" gracePeriod=30 Oct 31 04:27:34.238582 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:34.238544 5019 generic.go:356] "Generic (PLEG): container finished" podID="5c95c936-5fcf-4bb0-8701-09f544af2fe0" containerID="8ba7f92a6cfc96cd7b27986b3603109da5ee467c6ec21fdfb8e261a4529e02a9" exitCode=0 Oct 31 04:27:34.244352 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:34.238823 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91cd1-7f59958f95-c2qrv" event={"ID":"5c95c936-5fcf-4bb0-8701-09f544af2fe0","Type":"ContainerDied","Data":"8ba7f92a6cfc96cd7b27986b3603109da5ee467c6ec21fdfb8e261a4529e02a9"} Oct 31 04:27:34.647232 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:34.647198 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91cd1-7f59958f95-c2qrv" Oct 31 04:27:34.736377 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:34.732826 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c95c936-5fcf-4bb0-8701-09f544af2fe0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c95c936-5fcf-4bb0-8701-09f544af2fe0-plugins-config-volume\") pod \"5c95c936-5fcf-4bb0-8701-09f544af2fe0\" (UID: \"5c95c936-5fcf-4bb0-8701-09f544af2fe0\") " Oct 31 04:27:34.736377 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:34.732903 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c95c936-5fcf-4bb0-8701-09f544af2fe0-kube-api-access-7g6vl\" (UniqueName: \"kubernetes.io/projected/5c95c936-5fcf-4bb0-8701-09f544af2fe0-kube-api-access-7g6vl\") pod \"5c95c936-5fcf-4bb0-8701-09f544af2fe0\" (UID: \"5c95c936-5fcf-4bb0-8701-09f544af2fe0\") " Oct 31 04:27:34.744458 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:34.742731 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c95c936-5fcf-4bb0-8701-09f544af2fe0-plugins-config-volume" pod "5c95c936-5fcf-4bb0-8701-09f544af2fe0" (UID: "5c95c936-5fcf-4bb0-8701-09f544af2fe0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:27:34.746350 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:34.745773 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c95c936-5fcf-4bb0-8701-09f544af2fe0-kube-api-access-7g6vl" pod "5c95c936-5fcf-4bb0-8701-09f544af2fe0" (UID: "5c95c936-5fcf-4bb0-8701-09f544af2fe0"). InnerVolumeSpecName "kube-api-access-7g6vl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:27:34.836367 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:34.836238 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c95c936-5fcf-4bb0-8701-09f544af2fe0-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:27:34.836367 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:34.836275 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g6vl\" (UniqueName: \"kubernetes.io/projected/5c95c936-5fcf-4bb0-8701-09f544af2fe0-kube-api-access-7g6vl\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:27:35.254982 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:35.254955 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91cd1-7f59958f95-c2qrv" Oct 31 04:27:35.255160 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:35.254989 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91cd1-7f59958f95-c2qrv" event={"ID":"5c95c936-5fcf-4bb0-8701-09f544af2fe0","Type":"ContainerDied","Data":"4862742438d9e9d75fa32f77caf40596ee228c9d203c715a80adc0f10cb5472c"} Oct 31 04:27:35.255160 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:35.255048 5019 scope.go:122] "RemoveContainer" containerID="8ba7f92a6cfc96cd7b27986b3603109da5ee467c6ec21fdfb8e261a4529e02a9" Oct 31 04:27:35.334889 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:35.334845 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f91cd1-7f59958f95-c2qrv"] Oct 31 04:27:35.345939 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:35.345905 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f91cd1-7f59958f95-c2qrv"] Oct 31 04:27:35.435223 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:35.435186 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c95c936-5fcf-4bb0-8701-09f544af2fe0" path="/var/lib/kubelet/pods/5c95c936-5fcf-4bb0-8701-09f544af2fe0/volumes" Oct 31 04:27:37.999142 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:37.999050 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff43e3-944445948-ph2kl" Oct 31 04:27:39.007407 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:39.007267 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff43e3-67d8bf5499-ttklw" Oct 31 04:27:40.118379 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:40.118343 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-093363ca-hl9xf"] Oct 31 04:27:40.164846 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:40.164818 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-093363ca-hl9xf"] Oct 31 04:27:41.021520 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:41.021484 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x"] Oct 31 04:27:41.021696 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:41.021674 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x" podUID="9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb" containerName="istio-proxy" containerID="cri-o://1f1b0319462ddc3cbb721c9c282e88d9d333816bf3ecf1b1049d48af5dc00cfa" gracePeriod=30 Oct 31 04:27:41.033597 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:41.033570 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-srnk2"] Oct 31 04:27:41.033789 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:41.033765 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-srnk2" podUID="19032517-0eb5-47e6-86f3-a2772a21b1b8" containerName="istio-proxy" containerID="cri-o://37a02397ad9328319794fc667f46249918a42440b5fe7af0bcc2c6e2d2c98cfe" gracePeriod=30 Oct 31 04:27:41.407495 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:41.407423 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="679c7d7b-2edf-4359-be21-d5b5010459cb" path="/var/lib/kubelet/pods/679c7d7b-2edf-4359-be21-d5b5010459cb/volumes" Oct 31 04:27:42.341624 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:42.341589 5019 generic.go:356] "Generic (PLEG): container finished" podID="92398da3-abea-427b-ad63-1d9fbd4385e5" containerID="33fec37f6afc9a22df2004dd4083eb96dbf8e4d306eaf1eeffbfa190e52571e0" exitCode=0 Oct 31 04:27:42.341800 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:42.341658 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8" event={"ID":"92398da3-abea-427b-ad63-1d9fbd4385e5","Type":"ContainerDied","Data":"33fec37f6afc9a22df2004dd4083eb96dbf8e4d306eaf1eeffbfa190e52571e0"} Oct 31 04:27:42.342150 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:42.342134 5019 scope.go:122] "RemoveContainer" containerID="33fec37f6afc9a22df2004dd4083eb96dbf8e4d306eaf1eeffbfa190e52571e0" Oct 31 04:27:42.343669 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:42.343641 5019 generic.go:356] "Generic (PLEG): container finished" podID="0165c2c6-76a2-42fc-a3f4-50674c99bc88" containerID="f9f0f9d9da35d1e56014f64fe194e7b84639347ad128aaac04c83308e683cb00" exitCode=0 Oct 31 04:27:42.343793 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:42.343679 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g" event={"ID":"0165c2c6-76a2-42fc-a3f4-50674c99bc88","Type":"ContainerDied","Data":"f9f0f9d9da35d1e56014f64fe194e7b84639347ad128aaac04c83308e683cb00"} Oct 31 04:27:42.343878 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:42.343846 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:27:42.343943 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:42.343916 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:27:42.344002 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:42.343946 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:27:42.344140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:42.344124 5019 scope.go:122] "RemoveContainer" containerID="f9f0f9d9da35d1e56014f64fe194e7b84639347ad128aaac04c83308e683cb00" Oct 31 04:27:42.345591 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:42.345566 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:27:42.345653 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:42.345623 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:27:42.345691 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:42.345656 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:27:42.883351 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:42.883224 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn"] Oct 31 04:27:42.887349 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:42.884249 5019 cpu_manager.go:414] "containerMap: removing container" podUID="5c95c936-5fcf-4bb0-8701-09f544af2fe0" containerName="payload-pre-processing" Oct 31 04:27:42.887349 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:42.884273 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c95c936-5fcf-4bb0-8701-09f544af2fe0" containerName="payload-pre-processing" Oct 31 04:27:42.887349 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:42.884424 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c95c936-5fcf-4bb0-8701-09f544af2fe0" containerName="payload-pre-processing" Oct 31 04:27:42.906350 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:42.896549 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn" Oct 31 04:27:42.928388 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:42.927591 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-24cd29-openshift-default-dockercfg-6frgx\"" Oct 31 04:27:42.997459 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:42.969298 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-istio-podinfo\") pod \"e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn\" (UID: \"042cfe03-4fa9-4b8d-a3e3-73283e3149d4\") " pod="openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn" Oct 31 04:27:42.997459 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:42.969395 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-istio-data\") pod \"e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn\" (UID: \"042cfe03-4fa9-4b8d-a3e3-73283e3149d4\") " pod="openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn" Oct 31 04:27:42.997459 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:42.969428 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tnt9\" (UniqueName: \"kubernetes.io/projected/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-kube-api-access-4tnt9\") pod \"e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn\" (UID: \"042cfe03-4fa9-4b8d-a3e3-73283e3149d4\") " pod="openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn" Oct 31 04:27:42.997459 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:42.969468 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-workload-certs\") pod \"e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn\" (UID: \"042cfe03-4fa9-4b8d-a3e3-73283e3149d4\") " pod="openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn" Oct 31 04:27:42.997459 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:42.969511 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-workload-socket\") pod \"e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn\" (UID: \"042cfe03-4fa9-4b8d-a3e3-73283e3149d4\") " pod="openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn" Oct 31 04:27:42.997459 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:42.969554 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-istio-envoy\") pod \"e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn\" (UID: \"042cfe03-4fa9-4b8d-a3e3-73283e3149d4\") " pod="openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn" Oct 31 04:27:42.997459 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:42.969635 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-credential-socket\") pod \"e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn\" (UID: \"042cfe03-4fa9-4b8d-a3e3-73283e3149d4\") " pod="openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn" Oct 31 04:27:42.997459 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:42.969661 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-istio-token\") pod \"e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn\" (UID: \"042cfe03-4fa9-4b8d-a3e3-73283e3149d4\") " pod="openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn" Oct 31 04:27:42.997459 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:42.969698 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-istiod-ca-cert\") pod \"e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn\" (UID: \"042cfe03-4fa9-4b8d-a3e3-73283e3149d4\") " pod="openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn" Oct 31 04:27:43.031880 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:43.031395 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn"] Oct 31 04:27:43.150703 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:43.150619 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-workload-certs\") pod \"e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn\" (UID: \"042cfe03-4fa9-4b8d-a3e3-73283e3149d4\") " pod="openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn" Oct 31 04:27:43.150943 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:43.150925 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-workload-socket\") pod \"e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn\" (UID: \"042cfe03-4fa9-4b8d-a3e3-73283e3149d4\") " pod="openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn" Oct 31 04:27:43.151797 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:43.151774 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-istio-envoy\") pod \"e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn\" (UID: \"042cfe03-4fa9-4b8d-a3e3-73283e3149d4\") " pod="openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn" Oct 31 04:27:43.152404 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:43.152384 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-credential-socket\") pod \"e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn\" (UID: \"042cfe03-4fa9-4b8d-a3e3-73283e3149d4\") " pod="openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn" Oct 31 04:27:43.152540 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:43.152526 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-istio-token\") pod \"e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn\" (UID: \"042cfe03-4fa9-4b8d-a3e3-73283e3149d4\") " pod="openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn" Oct 31 04:27:43.152660 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:43.152648 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-istiod-ca-cert\") pod \"e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn\" (UID: \"042cfe03-4fa9-4b8d-a3e3-73283e3149d4\") " pod="openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn" Oct 31 04:27:43.152950 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:43.152933 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-istio-podinfo\") pod \"e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn\" (UID: \"042cfe03-4fa9-4b8d-a3e3-73283e3149d4\") " pod="openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn" Oct 31 04:27:43.153095 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:43.153081 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-istio-data\") pod \"e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn\" (UID: \"042cfe03-4fa9-4b8d-a3e3-73283e3149d4\") " pod="openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn" Oct 31 04:27:43.153197 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:43.153184 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tnt9\" (UniqueName: \"kubernetes.io/projected/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-kube-api-access-4tnt9\") pod \"e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn\" (UID: \"042cfe03-4fa9-4b8d-a3e3-73283e3149d4\") " pod="openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn" Oct 31 04:27:43.153650 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:43.151691 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-workload-socket\") pod \"e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn\" (UID: \"042cfe03-4fa9-4b8d-a3e3-73283e3149d4\") " pod="openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn" Oct 31 04:27:43.155276 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:43.154223 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-credential-socket\") pod \"e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn\" (UID: \"042cfe03-4fa9-4b8d-a3e3-73283e3149d4\") " pod="openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn" Oct 31 04:27:43.155441 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:43.151399 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-workload-certs\") pod \"e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn\" (UID: \"042cfe03-4fa9-4b8d-a3e3-73283e3149d4\") " pod="openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn" Oct 31 04:27:43.155551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:43.154935 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-istiod-ca-cert\") pod \"e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn\" (UID: \"042cfe03-4fa9-4b8d-a3e3-73283e3149d4\") " pod="openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn" Oct 31 04:27:43.155655 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:43.155227 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-istio-data\") pod \"e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn\" (UID: \"042cfe03-4fa9-4b8d-a3e3-73283e3149d4\") " pod="openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn" Oct 31 04:27:43.188663 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:43.188595 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tnt9\" (UniqueName: \"kubernetes.io/projected/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-kube-api-access-4tnt9\") pod \"e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn\" (UID: \"042cfe03-4fa9-4b8d-a3e3-73283e3149d4\") " pod="openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn" Oct 31 04:27:43.208805 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:43.208775 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-istio-token\") pod \"e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn\" (UID: \"042cfe03-4fa9-4b8d-a3e3-73283e3149d4\") " pod="openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn" Oct 31 04:27:43.220790 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:43.220566 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-istio-envoy\") pod \"e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn\" (UID: \"042cfe03-4fa9-4b8d-a3e3-73283e3149d4\") " pod="openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn" Oct 31 04:27:43.246146 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:43.246097 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-istio-podinfo\") pod \"e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn\" (UID: \"042cfe03-4fa9-4b8d-a3e3-73283e3149d4\") " pod="openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn" Oct 31 04:27:43.354282 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:43.354079 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g" event={"ID":"0165c2c6-76a2-42fc-a3f4-50674c99bc88","Type":"ContainerStarted","Data":"5af8047060e9c391d0aa7adec8dacf4c8ea8445e596d8262cdacff7ae01e8483"} Oct 31 04:27:43.356496 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:43.356469 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8" event={"ID":"92398da3-abea-427b-ad63-1d9fbd4385e5","Type":"ContainerStarted","Data":"0ebf790fd3da68e5d821df8b7cc619b150466ace914bbfa5ee8dbb29b8d79c8c"} Oct 31 04:27:43.448942 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:43.448870 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn" Oct 31 04:27:43.454184 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:43.454159 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8" Oct 31 04:27:43.455952 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:43.455924 5019 patch_prober.go:29] interesting pod/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 31 04:27:43.456080 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:43.455982 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8" podUID="92398da3-abea-427b-ad63-1d9fbd4385e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 31 04:27:43.539563 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:43.539524 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g" Oct 31 04:27:43.540529 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:43.540483 5019 patch_prober.go:29] interesting pod/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 31 04:27:43.540690 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:43.540545 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g" podUID="0165c2c6-76a2-42fc-a3f4-50674c99bc88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 31 04:27:43.846584 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:27:43.846554 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod042cfe03_4fa9_4b8d_a3e3_73283e3149d4.slice/crio-822820128f068c8a8c9379522827541da1721f597d94e4d1f0ba2101d9a37260 WatchSource:0}: Error finding container 822820128f068c8a8c9379522827541da1721f597d94e4d1f0ba2101d9a37260: Status 404 returned error can't find the container with id 822820128f068c8a8c9379522827541da1721f597d94e4d1f0ba2101d9a37260 Oct 31 04:27:43.846710 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:43.846619 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn"] Oct 31 04:27:43.849170 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:43.849138 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:27:43.849291 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:43.849199 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:27:43.849291 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:43.849229 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:27:44.368469 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:44.367616 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn" event={"ID":"042cfe03-4fa9-4b8d-a3e3-73283e3149d4","Type":"ContainerStarted","Data":"1fef3af8cfb9cc006f1e2898170d11ff3e6ddb8f422c34535fb185ce588ad78b"} Oct 31 04:27:44.368469 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:44.367686 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn" event={"ID":"042cfe03-4fa9-4b8d-a3e3-73283e3149d4","Type":"ContainerStarted","Data":"822820128f068c8a8c9379522827541da1721f597d94e4d1f0ba2101d9a37260"} Oct 31 04:27:44.449308 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:44.449242 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn" Oct 31 04:27:44.450627 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:44.450595 5019 patch_prober.go:29] interesting pod/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 31 04:27:44.450757 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:44.450638 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn" podUID="042cfe03-4fa9-4b8d-a3e3-73283e3149d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 31 04:27:44.455210 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:44.455184 5019 patch_prober.go:29] interesting pod/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 31 04:27:44.455316 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:44.455230 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8" podUID="92398da3-abea-427b-ad63-1d9fbd4385e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 31 04:27:44.539943 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:44.539598 5019 patch_prober.go:29] interesting pod/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 31 04:27:44.539943 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:44.539650 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g" podUID="0165c2c6-76a2-42fc-a3f4-50674c99bc88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 31 04:27:45.454627 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:45.454588 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn" Oct 31 04:27:45.455056 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:45.454747 5019 patch_prober.go:29] interesting pod/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 31 04:27:45.455056 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:45.454793 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8" podUID="92398da3-abea-427b-ad63-1d9fbd4385e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 31 04:27:45.525358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:45.525126 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn" podStartSLOduration=3.525108202 podStartE2EDuration="3.525108202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:27:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:27:44.426712704 +0000 UTC m=+1419.183050743" watchObservedRunningTime="2026-10-31 04:27:45.525108202 +0000 UTC m=+1420.281446241" Oct 31 04:27:45.540092 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:45.540066 5019 patch_prober.go:29] interesting pod/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 31 04:27:45.540270 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:45.540249 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g" podUID="0165c2c6-76a2-42fc-a3f4-50674c99bc88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 31 04:27:45.587352 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:45.579403 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g" Oct 31 04:27:46.440593 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.439453 5019 generic.go:356] "Generic (PLEG): container finished" podID="19032517-0eb5-47e6-86f3-a2772a21b1b8" containerID="37a02397ad9328319794fc667f46249918a42440b5fe7af0bcc2c6e2d2c98cfe" exitCode=0 Oct 31 04:27:46.440593 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.439560 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-srnk2" event={"ID":"19032517-0eb5-47e6-86f3-a2772a21b1b8","Type":"ContainerDied","Data":"37a02397ad9328319794fc667f46249918a42440b5fe7af0bcc2c6e2d2c98cfe"} Oct 31 04:27:46.441886 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.441859 5019 generic.go:356] "Generic (PLEG): container finished" podID="9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb" containerID="1f1b0319462ddc3cbb721c9c282e88d9d333816bf3ecf1b1049d48af5dc00cfa" exitCode=0 Oct 31 04:27:46.443208 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.443185 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x" event={"ID":"9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb","Type":"ContainerDied","Data":"1f1b0319462ddc3cbb721c9c282e88d9d333816bf3ecf1b1049d48af5dc00cfa"} Oct 31 04:27:46.443286 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.443251 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn" Oct 31 04:27:46.444309 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.444290 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn" Oct 31 04:27:46.454474 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.454445 5019 patch_prober.go:29] interesting pod/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 31 04:27:46.454577 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.454497 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8" podUID="92398da3-abea-427b-ad63-1d9fbd4385e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 31 04:27:46.548431 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.548238 5019 patch_prober.go:29] interesting pod/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 31 04:27:46.548974 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.548942 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g" podUID="0165c2c6-76a2-42fc-a3f4-50674c99bc88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 31 04:27:46.656621 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.656594 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8" Oct 31 04:27:46.798673 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.793547 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-srnk2" Oct 31 04:27:46.830253 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.830087 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x" Oct 31 04:27:46.850628 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.850597 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19032517-0eb5-47e6-86f3-a2772a21b1b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19032517-0eb5-47e6-86f3-a2772a21b1b8-istiod-ca-cert\") pod \"19032517-0eb5-47e6-86f3-a2772a21b1b8\" (UID: \"19032517-0eb5-47e6-86f3-a2772a21b1b8\") " Oct 31 04:27:46.850796 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.850647 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-kube-api-access-8mkg6\" (UniqueName: \"kubernetes.io/projected/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-kube-api-access-8mkg6\") pod \"9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb\" (UID: \"9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb\") " Oct 31 04:27:46.850796 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.850727 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19032517-0eb5-47e6-86f3-a2772a21b1b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19032517-0eb5-47e6-86f3-a2772a21b1b8-workload-certs\") pod \"19032517-0eb5-47e6-86f3-a2772a21b1b8\" (UID: \"19032517-0eb5-47e6-86f3-a2772a21b1b8\") " Oct 31 04:27:46.850796 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.850754 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-istio-token\" (UniqueName: \"kubernetes.io/projected/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-istio-token\") pod \"9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb\" (UID: \"9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb\") " Oct 31 04:27:46.850974 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.850796 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19032517-0eb5-47e6-86f3-a2772a21b1b8-kube-api-access-gzt62\" (UniqueName: \"kubernetes.io/projected/19032517-0eb5-47e6-86f3-a2772a21b1b8-kube-api-access-gzt62\") pod \"19032517-0eb5-47e6-86f3-a2772a21b1b8\" (UID: \"19032517-0eb5-47e6-86f3-a2772a21b1b8\") " Oct 31 04:27:46.850974 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.850830 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19032517-0eb5-47e6-86f3-a2772a21b1b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19032517-0eb5-47e6-86f3-a2772a21b1b8-istio-data\") pod \"19032517-0eb5-47e6-86f3-a2772a21b1b8\" (UID: \"19032517-0eb5-47e6-86f3-a2772a21b1b8\") " Oct 31 04:27:46.851124 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.851086 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-istio-podinfo\") pod \"9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb\" (UID: \"9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb\") " Oct 31 04:27:46.851237 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.851219 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19032517-0eb5-47e6-86f3-a2772a21b1b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19032517-0eb5-47e6-86f3-a2772a21b1b8-istio-envoy\") pod \"19032517-0eb5-47e6-86f3-a2772a21b1b8\" (UID: \"19032517-0eb5-47e6-86f3-a2772a21b1b8\") " Oct 31 04:27:46.851300 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.851276 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-istio-data\") pod \"9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb\" (UID: \"9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb\") " Oct 31 04:27:46.851367 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.851318 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19032517-0eb5-47e6-86f3-a2772a21b1b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19032517-0eb5-47e6-86f3-a2772a21b1b8-credential-socket\") pod \"19032517-0eb5-47e6-86f3-a2772a21b1b8\" (UID: \"19032517-0eb5-47e6-86f3-a2772a21b1b8\") " Oct 31 04:27:46.851438 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.851411 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19032517-0eb5-47e6-86f3-a2772a21b1b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19032517-0eb5-47e6-86f3-a2772a21b1b8-istio-podinfo\") pod \"19032517-0eb5-47e6-86f3-a2772a21b1b8\" (UID: \"19032517-0eb5-47e6-86f3-a2772a21b1b8\") " Oct 31 04:27:46.851511 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.851495 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-workload-certs\") pod \"9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb\" (UID: \"9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb\") " Oct 31 04:27:46.851574 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.851548 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-istio-envoy\") pod \"9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb\" (UID: \"9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb\") " Oct 31 04:27:46.851654 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.851636 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-istiod-ca-cert\") pod \"9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb\" (UID: \"9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb\") " Oct 31 04:27:46.856016 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.851686 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19032517-0eb5-47e6-86f3-a2772a21b1b8-istio-token\" (UniqueName: \"kubernetes.io/projected/19032517-0eb5-47e6-86f3-a2772a21b1b8-istio-token\") pod \"19032517-0eb5-47e6-86f3-a2772a21b1b8\" (UID: \"19032517-0eb5-47e6-86f3-a2772a21b1b8\") " Oct 31 04:27:46.856016 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.855841 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-credential-socket\") pod \"9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb\" (UID: \"9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb\") " Oct 31 04:27:46.856016 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.855881 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19032517-0eb5-47e6-86f3-a2772a21b1b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19032517-0eb5-47e6-86f3-a2772a21b1b8-workload-socket\") pod \"19032517-0eb5-47e6-86f3-a2772a21b1b8\" (UID: \"19032517-0eb5-47e6-86f3-a2772a21b1b8\") " Oct 31 04:27:46.856016 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.855920 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-workload-socket\") pod \"9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb\" (UID: \"9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb\") " Oct 31 04:27:46.857984 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.851094 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19032517-0eb5-47e6-86f3-a2772a21b1b8-workload-certs" pod "19032517-0eb5-47e6-86f3-a2772a21b1b8" (UID: "19032517-0eb5-47e6-86f3-a2772a21b1b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:27:46.857984 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.851415 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19032517-0eb5-47e6-86f3-a2772a21b1b8-istiod-ca-cert" pod "19032517-0eb5-47e6-86f3-a2772a21b1b8" (UID: "19032517-0eb5-47e6-86f3-a2772a21b1b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:27:46.857984 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.852083 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19032517-0eb5-47e6-86f3-a2772a21b1b8-istio-data" pod "19032517-0eb5-47e6-86f3-a2772a21b1b8" (UID: "19032517-0eb5-47e6-86f3-a2772a21b1b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:27:46.858200 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.857982 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-istio-envoy" pod "9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb" (UID: "9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:27:46.858200 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.858172 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-workload-socket" pod "9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb" (UID: "9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:27:46.858200 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.852745 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19032517-0eb5-47e6-86f3-a2772a21b1b8-credential-socket" pod "19032517-0eb5-47e6-86f3-a2772a21b1b8" (UID: "19032517-0eb5-47e6-86f3-a2772a21b1b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:27:46.858200 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.853208 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-istio-data" pod "9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb" (UID: "9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:27:46.858200 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.853505 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-istiod-ca-cert" pod "9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb" (UID: "9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:27:46.858200 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.854166 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-workload-certs" pod "9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb" (UID: "9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:27:46.858200 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.854768 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-kube-api-access-8mkg6" pod "9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb" (UID: "9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb"). InnerVolumeSpecName "kube-api-access-8mkg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:27:46.858642 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.856349 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-credential-socket" pod "9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb" (UID: "9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:27:46.858642 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.857945 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-istio-token" pod "9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb" (UID: "9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:27:46.859273 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.859248 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19032517-0eb5-47e6-86f3-a2772a21b1b8-workload-socket" pod "19032517-0eb5-47e6-86f3-a2772a21b1b8" (UID: "19032517-0eb5-47e6-86f3-a2772a21b1b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:27:46.859841 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.859819 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19032517-0eb5-47e6-86f3-a2772a21b1b8-istio-podinfo" pod "19032517-0eb5-47e6-86f3-a2772a21b1b8" (UID: "19032517-0eb5-47e6-86f3-a2772a21b1b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:27:46.860946 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.860836 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19032517-0eb5-47e6-86f3-a2772a21b1b8-istio-envoy" pod "19032517-0eb5-47e6-86f3-a2772a21b1b8" (UID: "19032517-0eb5-47e6-86f3-a2772a21b1b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:27:46.861446 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.861423 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-istio-podinfo" pod "9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb" (UID: "9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:27:46.861551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.861460 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19032517-0eb5-47e6-86f3-a2772a21b1b8-istio-token" pod "19032517-0eb5-47e6-86f3-a2772a21b1b8" (UID: "19032517-0eb5-47e6-86f3-a2772a21b1b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:27:46.861551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.861498 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19032517-0eb5-47e6-86f3-a2772a21b1b8-kube-api-access-gzt62" pod "19032517-0eb5-47e6-86f3-a2772a21b1b8" (UID: "19032517-0eb5-47e6-86f3-a2772a21b1b8"). InnerVolumeSpecName "kube-api-access-gzt62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:27:46.958704 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.958618 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:27:46.958704 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.958649 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19032517-0eb5-47e6-86f3-a2772a21b1b8-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:27:46.958704 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.958664 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:27:46.958704 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.958677 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19032517-0eb5-47e6-86f3-a2772a21b1b8-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:27:46.958704 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.958691 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:27:46.958704 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.958705 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19032517-0eb5-47e6-86f3-a2772a21b1b8-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:27:46.959121 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.958718 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mkg6\" (UniqueName: \"kubernetes.io/projected/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-kube-api-access-8mkg6\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:27:46.959121 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.958731 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19032517-0eb5-47e6-86f3-a2772a21b1b8-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:27:46.959121 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.958746 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:27:46.959121 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.958758 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzt62\" (UniqueName: \"kubernetes.io/projected/19032517-0eb5-47e6-86f3-a2772a21b1b8-kube-api-access-gzt62\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:27:46.959121 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.958770 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19032517-0eb5-47e6-86f3-a2772a21b1b8-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:27:46.959121 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.958782 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:27:46.959121 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.958795 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19032517-0eb5-47e6-86f3-a2772a21b1b8-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:27:46.959121 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.958808 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:27:46.959121 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.958820 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19032517-0eb5-47e6-86f3-a2772a21b1b8-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:27:46.959121 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.958832 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19032517-0eb5-47e6-86f3-a2772a21b1b8-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:27:46.959121 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.958845 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:27:46.959121 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:46.958857 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:27:47.451774 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:47.451737 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-srnk2" event={"ID":"19032517-0eb5-47e6-86f3-a2772a21b1b8","Type":"ContainerDied","Data":"ca9e81705b63d42254e5674a4b7d2856000bcfc34a5e8e4a66394e0e2c7bc72e"} Oct 31 04:27:47.451909 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:47.451802 5019 scope.go:122] "RemoveContainer" containerID="37a02397ad9328319794fc667f46249918a42440b5fe7af0bcc2c6e2d2c98cfe" Oct 31 04:27:47.451992 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:47.451974 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-srnk2" Oct 31 04:27:47.454913 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:47.454855 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x" Oct 31 04:27:47.454913 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:47.454870 5019 patch_prober.go:29] interesting pod/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 31 04:27:47.455062 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:47.454913 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8" podUID="92398da3-abea-427b-ad63-1d9fbd4385e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 31 04:27:47.455062 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:47.454925 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x" event={"ID":"9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb","Type":"ContainerDied","Data":"769a9d91d01bfc482c31e09e74101c1d71b1e079b8e0adbf9f2c83b3d33f6f06"} Oct 31 04:27:47.464517 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:47.464496 5019 scope.go:122] "RemoveContainer" containerID="1f1b0319462ddc3cbb721c9c282e88d9d333816bf3ecf1b1049d48af5dc00cfa" Oct 31 04:27:47.534589 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:47.534565 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x"] Oct 31 04:27:47.537187 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:47.537161 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-p6f5x"] Oct 31 04:27:47.544578 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:47.539244 5019 patch_prober.go:29] interesting pod/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 31 04:27:47.544578 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:47.539304 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g" podUID="0165c2c6-76a2-42fc-a3f4-50674c99bc88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 31 04:27:47.646727 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:47.646688 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-srnk2"] Oct 31 04:27:47.650935 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:47.650911 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-srnk2"] Oct 31 04:27:48.454806 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:48.454763 5019 patch_prober.go:29] interesting pod/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 31 04:27:48.454952 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:48.454836 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8" podUID="92398da3-abea-427b-ad63-1d9fbd4385e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 31 04:27:48.539300 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:48.539242 5019 patch_prober.go:29] interesting pod/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 31 04:27:48.539523 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:48.539353 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g" podUID="0165c2c6-76a2-42fc-a3f4-50674c99bc88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 31 04:27:48.776790 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:48.776755 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-24cd29-6d745d66c9-b2hdr"] Oct 31 04:27:48.777716 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:48.777695 5019 cpu_manager.go:414] "containerMap: removing container" podUID="9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb" containerName="istio-proxy" Oct 31 04:27:48.777780 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:48.777721 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb" containerName="istio-proxy" Oct 31 04:27:48.777780 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:48.777761 5019 cpu_manager.go:414] "containerMap: removing container" podUID="19032517-0eb5-47e6-86f3-a2772a21b1b8" containerName="istio-proxy" Oct 31 04:27:48.777780 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:48.777770 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19032517-0eb5-47e6-86f3-a2772a21b1b8" containerName="istio-proxy" Oct 31 04:27:48.778024 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:48.778007 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb" containerName="istio-proxy" Oct 31 04:27:48.778085 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:48.778036 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="19032517-0eb5-47e6-86f3-a2772a21b1b8" containerName="istio-proxy" Oct 31 04:27:48.783940 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:48.783918 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24cd29-6d745d66c9-b2hdr" Oct 31 04:27:48.804510 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:48.804488 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-24cd29-6d745d66c9-b2hdr"] Oct 31 04:27:48.920357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:48.910036 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-24cd29\"" Oct 31 04:27:48.920357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:48.910201 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-24cd29-dockercfg-j8pkg\"" Oct 31 04:27:48.961911 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:48.951586 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4llv5\" (UniqueName: \"kubernetes.io/projected/dd0acc5e-e7da-48c2-b24b-fc0701773a6a-kube-api-access-4llv5\") pod \"payload-pre-processing-e2e-ipp-b-24cd29-6d745d66c9-b2hdr\" (UID: \"dd0acc5e-e7da-48c2-b24b-fc0701773a6a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24cd29-6d745d66c9-b2hdr" Oct 31 04:27:48.962697 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:48.962665 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd0acc5e-e7da-48c2-b24b-fc0701773a6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-24cd29-6d745d66c9-b2hdr\" (UID: \"dd0acc5e-e7da-48c2-b24b-fc0701773a6a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24cd29-6d745d66c9-b2hdr" Oct 31 04:27:48.989372 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:48.981371 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-24cd29-69d9789c79-fh6c4"] Oct 31 04:27:48.989372 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:48.986590 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-24cd29-69d9789c79-fh6c4" Oct 31 04:27:49.017360 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:49.013041 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-24cd29-69d9789c79-fh6c4"] Oct 31 04:27:49.066702 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:49.066600 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd0acc5e-e7da-48c2-b24b-fc0701773a6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-24cd29-6d745d66c9-b2hdr\" (UID: \"dd0acc5e-e7da-48c2-b24b-fc0701773a6a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24cd29-6d745d66c9-b2hdr" Oct 31 04:27:49.067060 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:49.066727 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c91c5252-fbf0-48b5-9ddb-124322049a54-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-24cd29-69d9789c79-fh6c4\" (UID: \"c91c5252-fbf0-48b5-9ddb-124322049a54\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-24cd29-69d9789c79-fh6c4" Oct 31 04:27:49.067060 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:49.066951 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4llv5\" (UniqueName: \"kubernetes.io/projected/dd0acc5e-e7da-48c2-b24b-fc0701773a6a-kube-api-access-4llv5\") pod \"payload-pre-processing-e2e-ipp-b-24cd29-6d745d66c9-b2hdr\" (UID: \"dd0acc5e-e7da-48c2-b24b-fc0701773a6a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24cd29-6d745d66c9-b2hdr" Oct 31 04:27:49.067060 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:49.066983 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj7ll\" (UniqueName: \"kubernetes.io/projected/c91c5252-fbf0-48b5-9ddb-124322049a54-kube-api-access-sj7ll\") pod \"payload-processing-e2e-ipp-b-24cd29-69d9789c79-fh6c4\" (UID: \"c91c5252-fbf0-48b5-9ddb-124322049a54\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-24cd29-69d9789c79-fh6c4" Oct 31 04:27:49.068495 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:49.068472 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd0acc5e-e7da-48c2-b24b-fc0701773a6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-24cd29-6d745d66c9-b2hdr\" (UID: \"dd0acc5e-e7da-48c2-b24b-fc0701773a6a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24cd29-6d745d66c9-b2hdr" Oct 31 04:27:49.091385 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:49.091349 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4llv5\" (UniqueName: \"kubernetes.io/projected/dd0acc5e-e7da-48c2-b24b-fc0701773a6a-kube-api-access-4llv5\") pod \"payload-pre-processing-e2e-ipp-b-24cd29-6d745d66c9-b2hdr\" (UID: \"dd0acc5e-e7da-48c2-b24b-fc0701773a6a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24cd29-6d745d66c9-b2hdr" Oct 31 04:27:49.100080 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:49.100055 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24cd29-6d745d66c9-b2hdr" Oct 31 04:27:49.169049 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:49.169012 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c91c5252-fbf0-48b5-9ddb-124322049a54-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-24cd29-69d9789c79-fh6c4\" (UID: \"c91c5252-fbf0-48b5-9ddb-124322049a54\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-24cd29-69d9789c79-fh6c4" Oct 31 04:27:49.169205 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:49.169186 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj7ll\" (UniqueName: \"kubernetes.io/projected/c91c5252-fbf0-48b5-9ddb-124322049a54-kube-api-access-sj7ll\") pod \"payload-processing-e2e-ipp-b-24cd29-69d9789c79-fh6c4\" (UID: \"c91c5252-fbf0-48b5-9ddb-124322049a54\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-24cd29-69d9789c79-fh6c4" Oct 31 04:27:49.169951 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:49.169709 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c91c5252-fbf0-48b5-9ddb-124322049a54-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-24cd29-69d9789c79-fh6c4\" (UID: \"c91c5252-fbf0-48b5-9ddb-124322049a54\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-24cd29-69d9789c79-fh6c4" Oct 31 04:27:49.196222 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:49.196190 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj7ll\" (UniqueName: \"kubernetes.io/projected/c91c5252-fbf0-48b5-9ddb-124322049a54-kube-api-access-sj7ll\") pod \"payload-processing-e2e-ipp-b-24cd29-69d9789c79-fh6c4\" (UID: \"c91c5252-fbf0-48b5-9ddb-124322049a54\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-24cd29-69d9789c79-fh6c4" Oct 31 04:27:49.250657 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:49.250616 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8"] Oct 31 04:27:49.257350 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:49.257308 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8" Oct 31 04:27:49.280083 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:49.280049 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8"] Oct 31 04:27:49.293651 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:49.293626 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-b44631-openshift-default-dockercfg-jzp9n\"" Oct 31 04:27:49.315881 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:49.315855 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-24cd29-69d9789c79-fh6c4" Oct 31 04:27:49.376481 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:49.376068 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73613ab5-a053-42d6-b3f4-6e4a381da1ac-istio-token\") pod \"e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8\" (UID: \"73613ab5-a053-42d6-b3f4-6e4a381da1ac\") " pod="openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8" Oct 31 04:27:49.376481 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:49.376107 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swvp4\" (UniqueName: \"kubernetes.io/projected/73613ab5-a053-42d6-b3f4-6e4a381da1ac-kube-api-access-swvp4\") pod \"e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8\" (UID: \"73613ab5-a053-42d6-b3f4-6e4a381da1ac\") " pod="openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8" Oct 31 04:27:49.376481 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:49.376153 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73613ab5-a053-42d6-b3f4-6e4a381da1ac-workload-certs\") pod \"e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8\" (UID: \"73613ab5-a053-42d6-b3f4-6e4a381da1ac\") " pod="openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8" Oct 31 04:27:49.376481 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:49.376173 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73613ab5-a053-42d6-b3f4-6e4a381da1ac-workload-socket\") pod \"e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8\" (UID: \"73613ab5-a053-42d6-b3f4-6e4a381da1ac\") " pod="openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8" Oct 31 04:27:49.376481 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:49.376195 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73613ab5-a053-42d6-b3f4-6e4a381da1ac-istio-podinfo\") pod \"e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8\" (UID: \"73613ab5-a053-42d6-b3f4-6e4a381da1ac\") " pod="openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8" Oct 31 04:27:49.376481 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:49.376211 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73613ab5-a053-42d6-b3f4-6e4a381da1ac-istio-envoy\") pod \"e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8\" (UID: \"73613ab5-a053-42d6-b3f4-6e4a381da1ac\") " pod="openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8" Oct 31 04:27:49.376481 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:49.376231 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73613ab5-a053-42d6-b3f4-6e4a381da1ac-istiod-ca-cert\") pod \"e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8\" (UID: \"73613ab5-a053-42d6-b3f4-6e4a381da1ac\") " pod="openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8" Oct 31 04:27:49.376481 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:49.376259 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73613ab5-a053-42d6-b3f4-6e4a381da1ac-credential-socket\") pod \"e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8\" (UID: \"73613ab5-a053-42d6-b3f4-6e4a381da1ac\") " pod="openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8" Oct 31 04:27:49.376481 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:49.376282 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73613ab5-a053-42d6-b3f4-6e4a381da1ac-istio-data\") pod \"e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8\" (UID: \"73613ab5-a053-42d6-b3f4-6e4a381da1ac\") " pod="openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8" Oct 31 04:27:49.411691 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:27:49.411656 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd0acc5e_e7da_48c2_b24b_fc0701773a6a.slice/crio-52b7fa7d0bfd7e7e32cf074abe6f87fb0d424565508104fe2530413fa16842dd WatchSource:0}: Error finding container 52b7fa7d0bfd7e7e32cf074abe6f87fb0d424565508104fe2530413fa16842dd: Status 404 returned error can't find the container with id 52b7fa7d0bfd7e7e32cf074abe6f87fb0d424565508104fe2530413fa16842dd Oct 31 04:27:49.415164 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:49.415132 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19032517-0eb5-47e6-86f3-a2772a21b1b8" path="/var/lib/kubelet/pods/19032517-0eb5-47e6-86f3-a2772a21b1b8/volumes" Oct 31 04:27:49.415950 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:49.415930 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb" path="/var/lib/kubelet/pods/9f67449a-4ed1-4c5e-bdad-decbb3b1b6cb/volumes" Oct 31 04:27:49.416314 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:49.416298 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-24cd29-6d745d66c9-b2hdr"] Oct 31 04:27:49.459984 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:49.459960 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8" Oct 31 04:27:49.461080 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:49.461064 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8" Oct 31 04:27:49.475630 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:49.475608 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24cd29-6d745d66c9-b2hdr" event={"ID":"dd0acc5e-e7da-48c2-b24b-fc0701773a6a","Type":"ContainerStarted","Data":"52b7fa7d0bfd7e7e32cf074abe6f87fb0d424565508104fe2530413fa16842dd"} Oct 31 04:27:49.477452 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:49.477428 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73613ab5-a053-42d6-b3f4-6e4a381da1ac-istio-token\") pod \"e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8\" (UID: \"73613ab5-a053-42d6-b3f4-6e4a381da1ac\") " pod="openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8" Oct 31 04:27:49.477531 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:49.477487 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swvp4\" (UniqueName: \"kubernetes.io/projected/73613ab5-a053-42d6-b3f4-6e4a381da1ac-kube-api-access-swvp4\") pod \"e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8\" (UID: \"73613ab5-a053-42d6-b3f4-6e4a381da1ac\") " pod="openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8" Oct 31 04:27:49.477585 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:49.477552 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73613ab5-a053-42d6-b3f4-6e4a381da1ac-workload-certs\") pod \"e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8\" (UID: \"73613ab5-a053-42d6-b3f4-6e4a381da1ac\") " pod="openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8" Oct 31 04:27:49.477623 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:49.477592 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73613ab5-a053-42d6-b3f4-6e4a381da1ac-workload-socket\") pod \"e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8\" (UID: \"73613ab5-a053-42d6-b3f4-6e4a381da1ac\") " pod="openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8" Oct 31 04:27:49.477681 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:49.477624 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73613ab5-a053-42d6-b3f4-6e4a381da1ac-istio-podinfo\") pod \"e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8\" (UID: \"73613ab5-a053-42d6-b3f4-6e4a381da1ac\") " pod="openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8" Oct 31 04:27:49.477681 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:49.477650 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73613ab5-a053-42d6-b3f4-6e4a381da1ac-istio-envoy\") pod \"e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8\" (UID: \"73613ab5-a053-42d6-b3f4-6e4a381da1ac\") " pod="openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8" Oct 31 04:27:49.477681 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:49.477682 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73613ab5-a053-42d6-b3f4-6e4a381da1ac-istiod-ca-cert\") pod \"e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8\" (UID: \"73613ab5-a053-42d6-b3f4-6e4a381da1ac\") " pod="openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8" Oct 31 04:27:49.477841 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:49.477726 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73613ab5-a053-42d6-b3f4-6e4a381da1ac-credential-socket\") pod \"e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8\" (UID: \"73613ab5-a053-42d6-b3f4-6e4a381da1ac\") " pod="openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8" Oct 31 04:27:49.477841 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:49.477765 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73613ab5-a053-42d6-b3f4-6e4a381da1ac-istio-data\") pod \"e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8\" (UID: \"73613ab5-a053-42d6-b3f4-6e4a381da1ac\") " pod="openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8" Oct 31 04:27:49.477994 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:49.477924 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73613ab5-a053-42d6-b3f4-6e4a381da1ac-workload-socket\") pod \"e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8\" (UID: \"73613ab5-a053-42d6-b3f4-6e4a381da1ac\") " pod="openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8" Oct 31 04:27:49.478061 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:49.477987 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73613ab5-a053-42d6-b3f4-6e4a381da1ac-workload-certs\") pod \"e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8\" (UID: \"73613ab5-a053-42d6-b3f4-6e4a381da1ac\") " pod="openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8" Oct 31 04:27:49.478212 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:49.478188 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73613ab5-a053-42d6-b3f4-6e4a381da1ac-istio-data\") pod \"e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8\" (UID: \"73613ab5-a053-42d6-b3f4-6e4a381da1ac\") " pod="openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8" Oct 31 04:27:49.478281 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:49.478264 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73613ab5-a053-42d6-b3f4-6e4a381da1ac-credential-socket\") pod \"e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8\" (UID: \"73613ab5-a053-42d6-b3f4-6e4a381da1ac\") " pod="openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8" Oct 31 04:27:49.478692 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:49.478671 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73613ab5-a053-42d6-b3f4-6e4a381da1ac-istiod-ca-cert\") pod \"e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8\" (UID: \"73613ab5-a053-42d6-b3f4-6e4a381da1ac\") " pod="openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8" Oct 31 04:27:49.479758 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:49.479736 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73613ab5-a053-42d6-b3f4-6e4a381da1ac-istio-envoy\") pod \"e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8\" (UID: \"73613ab5-a053-42d6-b3f4-6e4a381da1ac\") " pod="openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8" Oct 31 04:27:49.480152 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:49.480136 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73613ab5-a053-42d6-b3f4-6e4a381da1ac-istio-podinfo\") pod \"e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8\" (UID: \"73613ab5-a053-42d6-b3f4-6e4a381da1ac\") " pod="openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8" Oct 31 04:27:49.558877 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:49.557635 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g" Oct 31 04:27:49.560700 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:49.560636 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g" Oct 31 04:27:49.569190 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:49.567229 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73613ab5-a053-42d6-b3f4-6e4a381da1ac-istio-token\") pod \"e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8\" (UID: \"73613ab5-a053-42d6-b3f4-6e4a381da1ac\") " pod="openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8" Oct 31 04:27:49.576760 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:49.576663 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swvp4\" (UniqueName: \"kubernetes.io/projected/73613ab5-a053-42d6-b3f4-6e4a381da1ac-kube-api-access-swvp4\") pod \"e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8\" (UID: \"73613ab5-a053-42d6-b3f4-6e4a381da1ac\") " pod="openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8" Oct 31 04:27:49.636468 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:49.625170 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8" Oct 31 04:27:49.653591 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:49.650091 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-24cd29-69d9789c79-fh6c4"] Oct 31 04:27:50.002014 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:50.001987 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8"] Oct 31 04:27:50.002886 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:27:50.002860 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73613ab5_a053_42d6_b3f4_6e4a381da1ac.slice/crio-3dedd1ea18ac24bd3b7048d1d9defe960de3ca675744efe0f66e42f24634dfd3 WatchSource:0}: Error finding container 3dedd1ea18ac24bd3b7048d1d9defe960de3ca675744efe0f66e42f24634dfd3: Status 404 returned error can't find the container with id 3dedd1ea18ac24bd3b7048d1d9defe960de3ca675744efe0f66e42f24634dfd3 Oct 31 04:27:50.005255 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:50.005220 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:27:50.005350 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:50.005295 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:27:50.005350 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:50.005323 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:27:50.533352 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:50.532530 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24cd29-6d745d66c9-b2hdr" event={"ID":"dd0acc5e-e7da-48c2-b24b-fc0701773a6a","Type":"ContainerStarted","Data":"1155993ffe9d6670a538888ead2820e27d2ec357863998fe8d9529228d0f2058"} Oct 31 04:27:50.533589 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:50.533556 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24cd29-6d745d66c9-b2hdr" Oct 31 04:27:50.536577 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:50.536544 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8" event={"ID":"73613ab5-a053-42d6-b3f4-6e4a381da1ac","Type":"ContainerStarted","Data":"0d1dc38f376d2b8e894ce19f102caa980e174d6b88767afec2d6703ff7913cb5"} Oct 31 04:27:50.541319 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:50.541292 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8" event={"ID":"73613ab5-a053-42d6-b3f4-6e4a381da1ac","Type":"ContainerStarted","Data":"3dedd1ea18ac24bd3b7048d1d9defe960de3ca675744efe0f66e42f24634dfd3"} Oct 31 04:27:50.545423 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:50.545390 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-24cd29-69d9789c79-fh6c4" event={"ID":"c91c5252-fbf0-48b5-9ddb-124322049a54","Type":"ContainerStarted","Data":"6fd9bfa0d5511c0b046770cc316761a9598c86ff645291ea2ec04f993269b3ee"} Oct 31 04:27:50.545580 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:50.545564 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-24cd29-69d9789c79-fh6c4" event={"ID":"c91c5252-fbf0-48b5-9ddb-124322049a54","Type":"ContainerStarted","Data":"4d7b52cedd1c92f057ceb8cc60f0f0aa0e1de43462cf3584f5866375e0163eb7"} Oct 31 04:27:50.546098 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:50.546082 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-24cd29-69d9789c79-fh6c4" Oct 31 04:27:50.626604 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:50.626549 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8" podStartSLOduration=1.62653259 podStartE2EDuration="1.62653259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:27:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:27:50.625829387 +0000 UTC m=+1425.382167427" watchObservedRunningTime="2026-10-31 04:27:50.62653259 +0000 UTC m=+1425.382870630" Oct 31 04:27:50.629069 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:50.629020 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24cd29-6d745d66c9-b2hdr" podStartSLOduration=2.629004617 podStartE2EDuration="2.629004617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:27:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:27:50.581751798 +0000 UTC m=+1425.338089837" watchObservedRunningTime="2026-10-31 04:27:50.629004617 +0000 UTC m=+1425.385342661" Oct 31 04:27:50.646893 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:50.646862 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8" Oct 31 04:27:50.667516 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:50.667468 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-24cd29-69d9789c79-fh6c4" podStartSLOduration=2.667455097 podStartE2EDuration="2.667455097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:27:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:27:50.66628563 +0000 UTC m=+1425.422623670" watchObservedRunningTime="2026-10-31 04:27:50.667455097 +0000 UTC m=+1425.423793135" Oct 31 04:27:51.652024 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:51.651994 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8" Oct 31 04:27:52.391421 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:27:52.391384 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:27:52.560200 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:52.560167 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8" Oct 31 04:27:52.561012 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:52.560996 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8" Oct 31 04:27:54.851407 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:54.851374 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b44631-6b47595bc-b7rtc"] Oct 31 04:27:54.855563 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:54.855545 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b44631-6b47595bc-b7rtc" Oct 31 04:27:54.884905 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:54.884882 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-b44631-dockercfg-qz2jp\"" Oct 31 04:27:54.885062 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:54.884919 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-b44631\"" Oct 31 04:27:54.896352 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:54.889581 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b44631-6b47595bc-b7rtc"] Oct 31 04:27:54.958353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:54.955077 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b44631-8456d66fbf-kb6zn"] Oct 31 04:27:54.980353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:54.969629 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b44631-8456d66fbf-kb6zn" Oct 31 04:27:54.999963 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:54.996230 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/454f6067-ffa0-41cb-9716-d6bacd72abbf-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-b44631-6b47595bc-b7rtc\" (UID: \"454f6067-ffa0-41cb-9716-d6bacd72abbf\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b44631-6b47595bc-b7rtc" Oct 31 04:27:54.999963 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:54.996400 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j566c\" (UniqueName: \"kubernetes.io/projected/454f6067-ffa0-41cb-9716-d6bacd72abbf-kube-api-access-j566c\") pod \"payload-pre-processing-e2e-praxis-b-b44631-6b47595bc-b7rtc\" (UID: \"454f6067-ffa0-41cb-9716-d6bacd72abbf\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b44631-6b47595bc-b7rtc" Oct 31 04:27:55.036420 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:55.036377 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b44631-8456d66fbf-kb6zn"] Oct 31 04:27:55.097709 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:55.097376 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/454f6067-ffa0-41cb-9716-d6bacd72abbf-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-b44631-6b47595bc-b7rtc\" (UID: \"454f6067-ffa0-41cb-9716-d6bacd72abbf\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b44631-6b47595bc-b7rtc" Oct 31 04:27:55.097709 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:55.097528 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmp4f\" (UniqueName: \"kubernetes.io/projected/51c854e6-11ce-48a1-97a4-fb77b57f5419-kube-api-access-pmp4f\") pod \"payload-processing-e2e-praxis-b-b44631-8456d66fbf-kb6zn\" (UID: \"51c854e6-11ce-48a1-97a4-fb77b57f5419\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b44631-8456d66fbf-kb6zn" Oct 31 04:27:55.097709 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:55.097572 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j566c\" (UniqueName: \"kubernetes.io/projected/454f6067-ffa0-41cb-9716-d6bacd72abbf-kube-api-access-j566c\") pod \"payload-pre-processing-e2e-praxis-b-b44631-6b47595bc-b7rtc\" (UID: \"454f6067-ffa0-41cb-9716-d6bacd72abbf\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b44631-6b47595bc-b7rtc" Oct 31 04:27:55.097709 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:55.097713 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51c854e6-11ce-48a1-97a4-fb77b57f5419-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-b44631-8456d66fbf-kb6zn\" (UID: \"51c854e6-11ce-48a1-97a4-fb77b57f5419\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b44631-8456d66fbf-kb6zn" Oct 31 04:27:55.112428 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:55.098684 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/454f6067-ffa0-41cb-9716-d6bacd72abbf-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-b44631-6b47595bc-b7rtc\" (UID: \"454f6067-ffa0-41cb-9716-d6bacd72abbf\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b44631-6b47595bc-b7rtc" Oct 31 04:27:55.125586 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:55.125553 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j566c\" (UniqueName: \"kubernetes.io/projected/454f6067-ffa0-41cb-9716-d6bacd72abbf-kube-api-access-j566c\") pod \"payload-pre-processing-e2e-praxis-b-b44631-6b47595bc-b7rtc\" (UID: \"454f6067-ffa0-41cb-9716-d6bacd72abbf\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b44631-6b47595bc-b7rtc" Oct 31 04:27:55.170778 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:55.170748 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b44631-6b47595bc-b7rtc" Oct 31 04:27:55.199150 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:55.198756 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmp4f\" (UniqueName: \"kubernetes.io/projected/51c854e6-11ce-48a1-97a4-fb77b57f5419-kube-api-access-pmp4f\") pod \"payload-processing-e2e-praxis-b-b44631-8456d66fbf-kb6zn\" (UID: \"51c854e6-11ce-48a1-97a4-fb77b57f5419\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b44631-8456d66fbf-kb6zn" Oct 31 04:27:55.199150 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:55.198939 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51c854e6-11ce-48a1-97a4-fb77b57f5419-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-b44631-8456d66fbf-kb6zn\" (UID: \"51c854e6-11ce-48a1-97a4-fb77b57f5419\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b44631-8456d66fbf-kb6zn" Oct 31 04:27:55.200086 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:55.200057 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51c854e6-11ce-48a1-97a4-fb77b57f5419-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-b44631-8456d66fbf-kb6zn\" (UID: \"51c854e6-11ce-48a1-97a4-fb77b57f5419\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b44631-8456d66fbf-kb6zn" Oct 31 04:27:55.223093 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:55.223062 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmp4f\" (UniqueName: \"kubernetes.io/projected/51c854e6-11ce-48a1-97a4-fb77b57f5419-kube-api-access-pmp4f\") pod \"payload-processing-e2e-praxis-b-b44631-8456d66fbf-kb6zn\" (UID: \"51c854e6-11ce-48a1-97a4-fb77b57f5419\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b44631-8456d66fbf-kb6zn" Oct 31 04:27:55.363960 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:55.363892 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b44631-8456d66fbf-kb6zn" Oct 31 04:27:55.479298 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:27:55.479253 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod454f6067_ffa0_41cb_9716_d6bacd72abbf.slice/crio-0586f4fa2c5ff99628498078ab62613f1564694f8ac2fe54e468b23129acc1b4 WatchSource:0}: Error finding container 0586f4fa2c5ff99628498078ab62613f1564694f8ac2fe54e468b23129acc1b4: Status 404 returned error can't find the container with id 0586f4fa2c5ff99628498078ab62613f1564694f8ac2fe54e468b23129acc1b4 Oct 31 04:27:55.479789 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:55.479768 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b44631-6b47595bc-b7rtc"] Oct 31 04:27:55.635662 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:55.635568 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b44631-6b47595bc-b7rtc" event={"ID":"454f6067-ffa0-41cb-9716-d6bacd72abbf","Type":"ContainerStarted","Data":"6811e710aabd79bfc7a1ec9c065dea78a872f8a469788159a34a2d7237ea30cb"} Oct 31 04:27:55.635968 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:55.635940 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b44631-6b47595bc-b7rtc" event={"ID":"454f6067-ffa0-41cb-9716-d6bacd72abbf","Type":"ContainerStarted","Data":"0586f4fa2c5ff99628498078ab62613f1564694f8ac2fe54e468b23129acc1b4"} Oct 31 04:27:55.637197 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:55.637178 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b44631-6b47595bc-b7rtc" Oct 31 04:27:55.682403 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:55.682270 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b44631-8456d66fbf-kb6zn"] Oct 31 04:27:55.685110 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:55.685060 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b44631-6b47595bc-b7rtc" podStartSLOduration=1.685044016 podStartE2EDuration="1.685044016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:27:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:27:55.680619744 +0000 UTC m=+1430.436957782" watchObservedRunningTime="2026-10-31 04:27:55.685044016 +0000 UTC m=+1430.441382056" Oct 31 04:27:56.645592 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:56.645495 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b44631-8456d66fbf-kb6zn" event={"ID":"51c854e6-11ce-48a1-97a4-fb77b57f5419","Type":"ContainerStarted","Data":"3f8494a84c9d4afa7134dada27f952aa76a5879de3d0a65dd200178767121248"} Oct 31 04:27:56.646045 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:56.645593 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b44631-8456d66fbf-kb6zn" event={"ID":"51c854e6-11ce-48a1-97a4-fb77b57f5419","Type":"ContainerStarted","Data":"4f60e05f66c9e6c58718350233df5fdda317ddffba121ab3a9cec0847c504297"} Oct 31 04:27:56.646045 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:56.645719 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-b44631-8456d66fbf-kb6zn" Oct 31 04:27:56.656820 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:56.656796 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-edc450-8c5754587-sfsfv"] Oct 31 04:27:56.661427 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:56.661408 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-edc450-8c5754587-sfsfv" Oct 31 04:27:56.678790 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:56.678766 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-edc450-dockercfg-6kwjc\"" Oct 31 04:27:56.678957 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:56.678940 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-edc450\"" Oct 31 04:27:56.680575 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:56.680511 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-edc450-8c5754587-sfsfv"] Oct 31 04:27:56.701920 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:56.701870 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-b44631-8456d66fbf-kb6zn" podStartSLOduration=2.701854612 podStartE2EDuration="2.701854612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:27:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:27:56.700695424 +0000 UTC m=+1431.457033464" watchObservedRunningTime="2026-10-31 04:27:56.701854612 +0000 UTC m=+1431.458192652" Oct 31 04:27:56.735685 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:56.735660 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgj5k\" (UniqueName: \"kubernetes.io/projected/c0d4b688-163d-43b4-af30-0a834f972117-kube-api-access-fgj5k\") pod \"payload-processing-e2e-praxis-b-edc450-8c5754587-sfsfv\" (UID: \"c0d4b688-163d-43b4-af30-0a834f972117\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-edc450-8c5754587-sfsfv" Oct 31 04:27:56.735811 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:56.735763 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0d4b688-163d-43b4-af30-0a834f972117-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-edc450-8c5754587-sfsfv\" (UID: \"c0d4b688-163d-43b4-af30-0a834f972117\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-edc450-8c5754587-sfsfv" Oct 31 04:27:56.869357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:56.864553 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgj5k\" (UniqueName: \"kubernetes.io/projected/c0d4b688-163d-43b4-af30-0a834f972117-kube-api-access-fgj5k\") pod \"payload-processing-e2e-praxis-b-edc450-8c5754587-sfsfv\" (UID: \"c0d4b688-163d-43b4-af30-0a834f972117\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-edc450-8c5754587-sfsfv" Oct 31 04:27:56.869357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:56.864655 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0d4b688-163d-43b4-af30-0a834f972117-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-edc450-8c5754587-sfsfv\" (UID: \"c0d4b688-163d-43b4-af30-0a834f972117\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-edc450-8c5754587-sfsfv" Oct 31 04:27:56.869357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:56.868945 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0d4b688-163d-43b4-af30-0a834f972117-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-edc450-8c5754587-sfsfv\" (UID: \"c0d4b688-163d-43b4-af30-0a834f972117\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-edc450-8c5754587-sfsfv" Oct 31 04:27:56.934151 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:56.933514 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgj5k\" (UniqueName: \"kubernetes.io/projected/c0d4b688-163d-43b4-af30-0a834f972117-kube-api-access-fgj5k\") pod \"payload-processing-e2e-praxis-b-edc450-8c5754587-sfsfv\" (UID: \"c0d4b688-163d-43b4-af30-0a834f972117\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-edc450-8c5754587-sfsfv" Oct 31 04:27:56.978209 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:56.978174 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-edc450-8c5754587-sfsfv" Oct 31 04:27:57.285572 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:57.285537 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-edc450-8c5754587-sfsfv"] Oct 31 04:27:57.285745 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:27:57.285707 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0d4b688_163d_43b4_af30_0a834f972117.slice/crio-f530272c69289e78c1264fa13b5b474c53df9ba3ececd582ce52269d7b6d44ec WatchSource:0}: Error finding container f530272c69289e78c1264fa13b5b474c53df9ba3ececd582ce52269d7b6d44ec: Status 404 returned error can't find the container with id f530272c69289e78c1264fa13b5b474c53df9ba3ececd582ce52269d7b6d44ec Oct 31 04:27:57.654927 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:57.654844 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-edc450-8c5754587-sfsfv" event={"ID":"c0d4b688-163d-43b4-af30-0a834f972117","Type":"ContainerStarted","Data":"d13f0b2a5d0dc0dd4e33d44e778ac26b757fc035b498cfeabe66ed487b438fcd"} Oct 31 04:27:57.655309 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:57.655176 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-edc450-8c5754587-sfsfv" Oct 31 04:27:57.655309 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:57.655207 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-edc450-8c5754587-sfsfv" event={"ID":"c0d4b688-163d-43b4-af30-0a834f972117","Type":"ContainerStarted","Data":"f530272c69289e78c1264fa13b5b474c53df9ba3ececd582ce52269d7b6d44ec"} Oct 31 04:27:57.697446 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:27:57.697395 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-edc450-8c5754587-sfsfv" podStartSLOduration=1.6973784859999999 podStartE2EDuration="1.697378486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:27:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:27:57.696321806 +0000 UTC m=+1432.452659846" watchObservedRunningTime="2026-10-31 04:27:57.697378486 +0000 UTC m=+1432.453716526" Oct 31 04:28:01.455456 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:01.455425 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b8bb1e-6d9649764-p2b84"] Oct 31 04:28:01.455906 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:01.455607 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-b8bb1e-6d9649764-p2b84" podUID="abc129a9-efa9-45f0-960d-27a21a68f326" containerName="payload-processing" containerID="cri-o://9d647a62296dd426f25935cd623c8864f548804248b8e094cebbc4b80d0c8521" gracePeriod=30 Oct 31 04:28:01.556274 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:01.556247 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-24cd29-69d9789c79-fh6c4" Oct 31 04:28:01.702135 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:01.702099 5019 generic.go:356] "Generic (PLEG): container finished" podID="abc129a9-efa9-45f0-960d-27a21a68f326" containerID="9d647a62296dd426f25935cd623c8864f548804248b8e094cebbc4b80d0c8521" exitCode=0 Oct 31 04:28:01.702284 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:01.702272 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b8bb1e-6d9649764-p2b84" event={"ID":"abc129a9-efa9-45f0-960d-27a21a68f326","Type":"ContainerDied","Data":"9d647a62296dd426f25935cd623c8864f548804248b8e094cebbc4b80d0c8521"} Oct 31 04:28:01.795420 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:01.795394 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b8bb1e-6d9649764-p2b84" Oct 31 04:28:01.877430 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:01.855757 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abc129a9-efa9-45f0-960d-27a21a68f326-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abc129a9-efa9-45f0-960d-27a21a68f326-plugins-config-volume\") pod \"abc129a9-efa9-45f0-960d-27a21a68f326\" (UID: \"abc129a9-efa9-45f0-960d-27a21a68f326\") " Oct 31 04:28:01.877430 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:01.855924 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abc129a9-efa9-45f0-960d-27a21a68f326-kube-api-access-mctqf\" (UniqueName: \"kubernetes.io/projected/abc129a9-efa9-45f0-960d-27a21a68f326-kube-api-access-mctqf\") pod \"abc129a9-efa9-45f0-960d-27a21a68f326\" (UID: \"abc129a9-efa9-45f0-960d-27a21a68f326\") " Oct 31 04:28:01.894085 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:01.894045 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abc129a9-efa9-45f0-960d-27a21a68f326-plugins-config-volume" pod "abc129a9-efa9-45f0-960d-27a21a68f326" (UID: "abc129a9-efa9-45f0-960d-27a21a68f326"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:28:01.926869 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:01.926519 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abc129a9-efa9-45f0-960d-27a21a68f326-kube-api-access-mctqf" pod "abc129a9-efa9-45f0-960d-27a21a68f326" (UID: "abc129a9-efa9-45f0-960d-27a21a68f326"). InnerVolumeSpecName "kube-api-access-mctqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:28:01.958071 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:01.958039 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abc129a9-efa9-45f0-960d-27a21a68f326-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:28:01.958071 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:01.958068 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mctqf\" (UniqueName: \"kubernetes.io/projected/abc129a9-efa9-45f0-960d-27a21a68f326-kube-api-access-mctqf\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:28:02.563209 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:02.563179 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24cd29-6d745d66c9-b2hdr" Oct 31 04:28:02.713360 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:02.713310 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b8bb1e-6d9649764-p2b84" event={"ID":"abc129a9-efa9-45f0-960d-27a21a68f326","Type":"ContainerDied","Data":"dd8279091bb12e41a07660294c18f968b642ceaf22d111c598ba545b78f18e9b"} Oct 31 04:28:02.713501 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:02.713389 5019 scope.go:122] "RemoveContainer" containerID="9d647a62296dd426f25935cd623c8864f548804248b8e094cebbc4b80d0c8521" Oct 31 04:28:02.713501 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:02.713324 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b8bb1e-6d9649764-p2b84" Oct 31 04:28:02.767855 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:02.767827 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b8bb1e-6d9649764-p2b84"] Oct 31 04:28:02.772913 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:02.772889 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b8bb1e-6d9649764-p2b84"] Oct 31 04:28:03.408163 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:03.408134 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abc129a9-efa9-45f0-960d-27a21a68f326" path="/var/lib/kubelet/pods/abc129a9-efa9-45f0-960d-27a21a68f326/volumes" Oct 31 04:28:07.658076 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:07.658041 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b44631-6b47595bc-b7rtc" Oct 31 04:28:07.658532 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:07.658122 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-b44631-8456d66fbf-kb6zn" Oct 31 04:28:08.157835 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:08.157803 5019 scope.go:122] "RemoveContainer" containerID="9bdcd4d3dd00e3db2f82875cfe06b7454b451411ad8e7c4827a8c8f3bcae22cb" Oct 31 04:28:08.167936 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:08.167912 5019 scope.go:122] "RemoveContainer" containerID="a780e1cde60cb486f89fa80c87c5048d248bb903a6c0abeb02934c5b3fa71545" Oct 31 04:28:08.177090 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:08.177065 5019 scope.go:122] "RemoveContainer" containerID="3733ebda239dbfa78f06d0e05cef44efd34de07205294fe1b927681cb03c0fd3" Oct 31 04:28:08.668184 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:08.668159 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-edc450-8c5754587-sfsfv" Oct 31 04:28:10.509146 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:10.509115 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-81c810-666bd7cfc5-k862v"] Oct 31 04:28:10.509757 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:10.509744 5019 cpu_manager.go:414] "containerMap: removing container" podUID="abc129a9-efa9-45f0-960d-27a21a68f326" containerName="payload-processing" Oct 31 04:28:10.509818 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:10.509758 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abc129a9-efa9-45f0-960d-27a21a68f326" containerName="payload-processing" Oct 31 04:28:10.509900 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:10.509891 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="abc129a9-efa9-45f0-960d-27a21a68f326" containerName="payload-processing" Oct 31 04:28:10.514758 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:10.514739 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-81c810-666bd7cfc5-k862v" Oct 31 04:28:10.533192 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:10.533169 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-81c810-666bd7cfc5-k862v"] Oct 31 04:28:10.543378 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:10.540552 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-81c810\"" Oct 31 04:28:10.543378 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:10.540814 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-81c810-dockercfg-jfssc\"" Oct 31 04:28:10.592353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:10.589730 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l268k\" (UniqueName: \"kubernetes.io/projected/97d7ee87-7714-4ee7-99cc-350ca67dbbe7-kube-api-access-l268k\") pod \"payload-pre-processing-e2e-praxis-b-81c810-666bd7cfc5-k862v\" (UID: \"97d7ee87-7714-4ee7-99cc-350ca67dbbe7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-81c810-666bd7cfc5-k862v" Oct 31 04:28:10.592353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:10.589849 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97d7ee87-7714-4ee7-99cc-350ca67dbbe7-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-81c810-666bd7cfc5-k862v\" (UID: \"97d7ee87-7714-4ee7-99cc-350ca67dbbe7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-81c810-666bd7cfc5-k862v" Oct 31 04:28:10.691930 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:10.691757 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l268k\" (UniqueName: \"kubernetes.io/projected/97d7ee87-7714-4ee7-99cc-350ca67dbbe7-kube-api-access-l268k\") pod \"payload-pre-processing-e2e-praxis-b-81c810-666bd7cfc5-k862v\" (UID: \"97d7ee87-7714-4ee7-99cc-350ca67dbbe7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-81c810-666bd7cfc5-k862v" Oct 31 04:28:10.691930 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:10.691906 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97d7ee87-7714-4ee7-99cc-350ca67dbbe7-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-81c810-666bd7cfc5-k862v\" (UID: \"97d7ee87-7714-4ee7-99cc-350ca67dbbe7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-81c810-666bd7cfc5-k862v" Oct 31 04:28:10.693643 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:10.692863 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97d7ee87-7714-4ee7-99cc-350ca67dbbe7-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-81c810-666bd7cfc5-k862v\" (UID: \"97d7ee87-7714-4ee7-99cc-350ca67dbbe7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-81c810-666bd7cfc5-k862v" Oct 31 04:28:10.736555 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:10.726779 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l268k\" (UniqueName: \"kubernetes.io/projected/97d7ee87-7714-4ee7-99cc-350ca67dbbe7-kube-api-access-l268k\") pod \"payload-pre-processing-e2e-praxis-b-81c810-666bd7cfc5-k862v\" (UID: \"97d7ee87-7714-4ee7-99cc-350ca67dbbe7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-81c810-666bd7cfc5-k862v" Oct 31 04:28:10.831596 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:10.831509 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-81c810-666bd7cfc5-k862v" Oct 31 04:28:11.138296 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:28:11.138227 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97d7ee87_7714_4ee7_99cc_350ca67dbbe7.slice/crio-62400b252cc7ec429bc560209e8db2dfbab5150bb961933449344cb3cbc9885c WatchSource:0}: Error finding container 62400b252cc7ec429bc560209e8db2dfbab5150bb961933449344cb3cbc9885c: Status 404 returned error can't find the container with id 62400b252cc7ec429bc560209e8db2dfbab5150bb961933449344cb3cbc9885c Oct 31 04:28:11.138680 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:11.138651 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-81c810-666bd7cfc5-k862v"] Oct 31 04:28:11.800403 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:11.800315 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-81c810-666bd7cfc5-k862v" event={"ID":"97d7ee87-7714-4ee7-99cc-350ca67dbbe7","Type":"ContainerStarted","Data":"1a120be7059b2cbe70faf0bbd934ef8ae551ae364635a7576d33324411745882"} Oct 31 04:28:11.800856 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:11.800431 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-81c810-666bd7cfc5-k862v" event={"ID":"97d7ee87-7714-4ee7-99cc-350ca67dbbe7","Type":"ContainerStarted","Data":"62400b252cc7ec429bc560209e8db2dfbab5150bb961933449344cb3cbc9885c"} Oct 31 04:28:11.800856 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:11.800572 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-81c810-666bd7cfc5-k862v" Oct 31 04:28:11.842875 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:11.842792 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-81c810-666bd7cfc5-k862v" podStartSLOduration=1.8427795850000002 podStartE2EDuration="1.842779585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:28:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:28:11.841927212 +0000 UTC m=+1446.598265250" watchObservedRunningTime="2026-10-31 04:28:11.842779585 +0000 UTC m=+1446.599117626" Oct 31 04:28:15.346242 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:15.346209 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj"] Oct 31 04:28:15.355441 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:15.355407 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj" Oct 31 04:28:15.380499 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:15.380458 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj"] Oct 31 04:28:15.385485 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:15.385461 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-b3ca79-openshift-default-dockercfg-67gkr\"" Oct 31 04:28:15.463375 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:15.459612 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/618ff6b5-9d72-4ac1-9132-89c0697dfc52-workload-socket\") pod \"e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj\" (UID: \"618ff6b5-9d72-4ac1-9132-89c0697dfc52\") " pod="openshift-ingress/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj" Oct 31 04:28:15.463375 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:15.459684 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/618ff6b5-9d72-4ac1-9132-89c0697dfc52-istio-token\") pod \"e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj\" (UID: \"618ff6b5-9d72-4ac1-9132-89c0697dfc52\") " pod="openshift-ingress/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj" Oct 31 04:28:15.463375 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:15.459740 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/618ff6b5-9d72-4ac1-9132-89c0697dfc52-istio-envoy\") pod \"e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj\" (UID: \"618ff6b5-9d72-4ac1-9132-89c0697dfc52\") " pod="openshift-ingress/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj" Oct 31 04:28:15.463375 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:15.459782 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/618ff6b5-9d72-4ac1-9132-89c0697dfc52-istio-data\") pod \"e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj\" (UID: \"618ff6b5-9d72-4ac1-9132-89c0697dfc52\") " pod="openshift-ingress/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj" Oct 31 04:28:15.463375 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:15.459860 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/618ff6b5-9d72-4ac1-9132-89c0697dfc52-workload-certs\") pod \"e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj\" (UID: \"618ff6b5-9d72-4ac1-9132-89c0697dfc52\") " pod="openshift-ingress/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj" Oct 31 04:28:15.463375 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:15.459885 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqkfr\" (UniqueName: \"kubernetes.io/projected/618ff6b5-9d72-4ac1-9132-89c0697dfc52-kube-api-access-vqkfr\") pod \"e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj\" (UID: \"618ff6b5-9d72-4ac1-9132-89c0697dfc52\") " pod="openshift-ingress/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj" Oct 31 04:28:15.463375 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:15.459954 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/618ff6b5-9d72-4ac1-9132-89c0697dfc52-istio-podinfo\") pod \"e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj\" (UID: \"618ff6b5-9d72-4ac1-9132-89c0697dfc52\") " pod="openshift-ingress/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj" Oct 31 04:28:15.463375 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:15.459986 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/618ff6b5-9d72-4ac1-9132-89c0697dfc52-istiod-ca-cert\") pod \"e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj\" (UID: \"618ff6b5-9d72-4ac1-9132-89c0697dfc52\") " pod="openshift-ingress/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj" Oct 31 04:28:15.463375 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:15.460099 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/618ff6b5-9d72-4ac1-9132-89c0697dfc52-credential-socket\") pod \"e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj\" (UID: \"618ff6b5-9d72-4ac1-9132-89c0697dfc52\") " pod="openshift-ingress/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj" Oct 31 04:28:15.566678 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:15.566268 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/618ff6b5-9d72-4ac1-9132-89c0697dfc52-workload-certs\") pod \"e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj\" (UID: \"618ff6b5-9d72-4ac1-9132-89c0697dfc52\") " pod="openshift-ingress/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj" Oct 31 04:28:15.566678 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:15.566350 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqkfr\" (UniqueName: \"kubernetes.io/projected/618ff6b5-9d72-4ac1-9132-89c0697dfc52-kube-api-access-vqkfr\") pod \"e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj\" (UID: \"618ff6b5-9d72-4ac1-9132-89c0697dfc52\") " pod="openshift-ingress/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj" Oct 31 04:28:15.566678 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:15.566446 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/618ff6b5-9d72-4ac1-9132-89c0697dfc52-istio-podinfo\") pod \"e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj\" (UID: \"618ff6b5-9d72-4ac1-9132-89c0697dfc52\") " pod="openshift-ingress/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj" Oct 31 04:28:15.566678 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:15.566486 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/618ff6b5-9d72-4ac1-9132-89c0697dfc52-istiod-ca-cert\") pod \"e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj\" (UID: \"618ff6b5-9d72-4ac1-9132-89c0697dfc52\") " pod="openshift-ingress/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj" Oct 31 04:28:15.566678 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:15.566619 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/618ff6b5-9d72-4ac1-9132-89c0697dfc52-credential-socket\") pod \"e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj\" (UID: \"618ff6b5-9d72-4ac1-9132-89c0697dfc52\") " pod="openshift-ingress/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj" Oct 31 04:28:15.567083 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:15.566751 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/618ff6b5-9d72-4ac1-9132-89c0697dfc52-workload-socket\") pod \"e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj\" (UID: \"618ff6b5-9d72-4ac1-9132-89c0697dfc52\") " pod="openshift-ingress/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj" Oct 31 04:28:15.567083 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:15.566780 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/618ff6b5-9d72-4ac1-9132-89c0697dfc52-istio-token\") pod \"e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj\" (UID: \"618ff6b5-9d72-4ac1-9132-89c0697dfc52\") " pod="openshift-ingress/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj" Oct 31 04:28:15.567083 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:15.566830 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/618ff6b5-9d72-4ac1-9132-89c0697dfc52-istio-envoy\") pod \"e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj\" (UID: \"618ff6b5-9d72-4ac1-9132-89c0697dfc52\") " pod="openshift-ingress/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj" Oct 31 04:28:15.567083 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:15.566873 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/618ff6b5-9d72-4ac1-9132-89c0697dfc52-istio-data\") pod \"e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj\" (UID: \"618ff6b5-9d72-4ac1-9132-89c0697dfc52\") " pod="openshift-ingress/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj" Oct 31 04:28:15.567454 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:15.567423 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/618ff6b5-9d72-4ac1-9132-89c0697dfc52-istio-data\") pod \"e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj\" (UID: \"618ff6b5-9d72-4ac1-9132-89c0697dfc52\") " pod="openshift-ingress/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj" Oct 31 04:28:15.567999 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:15.567972 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/618ff6b5-9d72-4ac1-9132-89c0697dfc52-workload-certs\") pod \"e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj\" (UID: \"618ff6b5-9d72-4ac1-9132-89c0697dfc52\") " pod="openshift-ingress/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj" Oct 31 04:28:15.568902 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:15.568872 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/618ff6b5-9d72-4ac1-9132-89c0697dfc52-workload-socket\") pod \"e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj\" (UID: \"618ff6b5-9d72-4ac1-9132-89c0697dfc52\") " pod="openshift-ingress/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj" Oct 31 04:28:15.576357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:15.569456 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/618ff6b5-9d72-4ac1-9132-89c0697dfc52-istiod-ca-cert\") pod \"e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj\" (UID: \"618ff6b5-9d72-4ac1-9132-89c0697dfc52\") " pod="openshift-ingress/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj" Oct 31 04:28:15.576357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:15.569673 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/618ff6b5-9d72-4ac1-9132-89c0697dfc52-credential-socket\") pod \"e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj\" (UID: \"618ff6b5-9d72-4ac1-9132-89c0697dfc52\") " pod="openshift-ingress/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj" Oct 31 04:28:15.576357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:15.573068 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/618ff6b5-9d72-4ac1-9132-89c0697dfc52-istio-envoy\") pod \"e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj\" (UID: \"618ff6b5-9d72-4ac1-9132-89c0697dfc52\") " pod="openshift-ingress/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj" Oct 31 04:28:15.606394 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:15.598986 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqkfr\" (UniqueName: \"kubernetes.io/projected/618ff6b5-9d72-4ac1-9132-89c0697dfc52-kube-api-access-vqkfr\") pod \"e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj\" (UID: \"618ff6b5-9d72-4ac1-9132-89c0697dfc52\") " pod="openshift-ingress/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj" Oct 31 04:28:15.606394 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:15.600918 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/618ff6b5-9d72-4ac1-9132-89c0697dfc52-istio-podinfo\") pod \"e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj\" (UID: \"618ff6b5-9d72-4ac1-9132-89c0697dfc52\") " pod="openshift-ingress/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj" Oct 31 04:28:15.622387 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:15.617460 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/618ff6b5-9d72-4ac1-9132-89c0697dfc52-istio-token\") pod \"e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj\" (UID: \"618ff6b5-9d72-4ac1-9132-89c0697dfc52\") " pod="openshift-ingress/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj" Oct 31 04:28:15.689780 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:15.689592 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj" Oct 31 04:28:16.010669 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:16.010643 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj"] Oct 31 04:28:16.011368 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:28:16.011316 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod618ff6b5_9d72_4ac1_9132_89c0697dfc52.slice/crio-b39d27aecd1aba59b5d445f818cfac9636e2d72e7875cdb39295f822aaeb0b5f WatchSource:0}: Error finding container b39d27aecd1aba59b5d445f818cfac9636e2d72e7875cdb39295f822aaeb0b5f: Status 404 returned error can't find the container with id b39d27aecd1aba59b5d445f818cfac9636e2d72e7875cdb39295f822aaeb0b5f Oct 31 04:28:16.014412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:16.014327 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:28:16.014522 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:16.014462 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:28:16.014522 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:16.014505 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:28:16.839129 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:16.839096 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj" event={"ID":"618ff6b5-9d72-4ac1-9132-89c0697dfc52","Type":"ContainerStarted","Data":"6de2c0547b9f30b5aa6c9ef198c8c8b2f8765b2d11e5a5c4f95bd38e3a58e368"} Oct 31 04:28:16.839555 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:16.839156 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj" event={"ID":"618ff6b5-9d72-4ac1-9132-89c0697dfc52","Type":"ContainerStarted","Data":"b39d27aecd1aba59b5d445f818cfac9636e2d72e7875cdb39295f822aaeb0b5f"} Oct 31 04:28:16.899017 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:16.898968 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj" podStartSLOduration=1.898954555 podStartE2EDuration="1.898954555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:28:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:28:16.895191815 +0000 UTC m=+1451.651529857" watchObservedRunningTime="2026-10-31 04:28:16.898954555 +0000 UTC m=+1451.655292594" Oct 31 04:28:17.690239 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:17.690211 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj" Oct 31 04:28:17.695455 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:17.695431 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj" Oct 31 04:28:17.846757 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:17.846729 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj" Oct 31 04:28:17.847547 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:17.847524 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj" Oct 31 04:28:22.417354 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:28:22.416663 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:28:22.811278 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:22.811239 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-81c810-666bd7cfc5-k862v" Oct 31 04:28:31.003706 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:31.003666 5019 generic.go:356] "Generic (PLEG): container finished" podID="042cfe03-4fa9-4b8d-a3e3-73283e3149d4" containerID="1fef3af8cfb9cc006f1e2898170d11ff3e6ddb8f422c34535fb185ce588ad78b" exitCode=0 Oct 31 04:28:31.004119 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:31.003732 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn" event={"ID":"042cfe03-4fa9-4b8d-a3e3-73283e3149d4","Type":"ContainerDied","Data":"1fef3af8cfb9cc006f1e2898170d11ff3e6ddb8f422c34535fb185ce588ad78b"} Oct 31 04:28:31.004161 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:31.004143 5019 scope.go:122] "RemoveContainer" containerID="1fef3af8cfb9cc006f1e2898170d11ff3e6ddb8f422c34535fb185ce588ad78b" Oct 31 04:28:31.005762 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:31.005733 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:28:31.005823 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:31.005803 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:28:31.005860 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:31.005834 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:28:31.443852 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:31.443766 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn" Oct 31 04:28:32.013095 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:32.013050 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn" event={"ID":"042cfe03-4fa9-4b8d-a3e3-73283e3149d4","Type":"ContainerStarted","Data":"43fa69192de2b18cc6cf75c40589998c7b44762e0ca976123955ae8ccdcb6f8d"} Oct 31 04:28:32.449201 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:32.449116 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn" Oct 31 04:28:32.450127 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:32.450097 5019 patch_prober.go:29] interesting pod/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 31 04:28:32.450262 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:32.450154 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn" podUID="042cfe03-4fa9-4b8d-a3e3-73283e3149d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 31 04:28:33.450315 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:33.450277 5019 patch_prober.go:29] interesting pod/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 31 04:28:33.450714 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:33.450366 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn" podUID="042cfe03-4fa9-4b8d-a3e3-73283e3149d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 31 04:28:34.449575 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:34.449538 5019 patch_prober.go:29] interesting pod/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 31 04:28:34.449740 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:34.449592 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn" podUID="042cfe03-4fa9-4b8d-a3e3-73283e3149d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 31 04:28:35.449945 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:35.449915 5019 patch_prober.go:29] interesting pod/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 31 04:28:35.450382 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:35.449962 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn" podUID="042cfe03-4fa9-4b8d-a3e3-73283e3149d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 31 04:28:36.450279 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:36.450241 5019 patch_prober.go:29] interesting pod/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 31 04:28:36.450844 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:36.450297 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn" podUID="042cfe03-4fa9-4b8d-a3e3-73283e3149d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 31 04:28:37.453958 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:37.453930 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn" Oct 31 04:28:37.454457 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:37.454208 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn" Oct 31 04:28:37.454926 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:37.454835 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn" Oct 31 04:28:42.876516 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:42.876484 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-24cd29-69d9789c79-fh6c4"] Oct 31 04:28:42.911524 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:42.911491 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-24cd29-6d745d66c9-b2hdr"] Oct 31 04:28:47.882496 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:47.882393 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-24cd29-69d9789c79-fh6c4" podUID="c91c5252-fbf0-48b5-9ddb-124322049a54" containerName="payload-processing" containerID="cri-o://6fd9bfa0d5511c0b046770cc316761a9598c86ff645291ea2ec04f993269b3ee" gracePeriod=25 Oct 31 04:28:47.912821 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:47.912758 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24cd29-6d745d66c9-b2hdr" podUID="dd0acc5e-e7da-48c2-b24b-fc0701773a6a" containerName="payload-pre-processing" containerID="cri-o://1155993ffe9d6670a538888ead2820e27d2ec357863998fe8d9529228d0f2058" gracePeriod=25 Oct 31 04:28:48.366495 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:48.366243 5019 generic.go:356] "Generic (PLEG): container finished" podID="dd0acc5e-e7da-48c2-b24b-fc0701773a6a" containerID="1155993ffe9d6670a538888ead2820e27d2ec357863998fe8d9529228d0f2058" exitCode=0 Oct 31 04:28:48.366495 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:48.366434 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24cd29-6d745d66c9-b2hdr" event={"ID":"dd0acc5e-e7da-48c2-b24b-fc0701773a6a","Type":"ContainerDied","Data":"1155993ffe9d6670a538888ead2820e27d2ec357863998fe8d9529228d0f2058"} Oct 31 04:28:48.367946 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:48.367925 5019 generic.go:356] "Generic (PLEG): container finished" podID="c91c5252-fbf0-48b5-9ddb-124322049a54" containerID="6fd9bfa0d5511c0b046770cc316761a9598c86ff645291ea2ec04f993269b3ee" exitCode=0 Oct 31 04:28:48.368126 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:48.368085 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-24cd29-69d9789c79-fh6c4" event={"ID":"c91c5252-fbf0-48b5-9ddb-124322049a54","Type":"ContainerDied","Data":"6fd9bfa0d5511c0b046770cc316761a9598c86ff645291ea2ec04f993269b3ee"} Oct 31 04:28:48.423458 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:48.423430 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-24cd29-69d9789c79-fh6c4" Oct 31 04:28:48.428568 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:48.428546 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24cd29-6d745d66c9-b2hdr" Oct 31 04:28:48.558543 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:48.558513 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd0acc5e-e7da-48c2-b24b-fc0701773a6a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd0acc5e-e7da-48c2-b24b-fc0701773a6a-plugins-config-volume\") pod \"dd0acc5e-e7da-48c2-b24b-fc0701773a6a\" (UID: \"dd0acc5e-e7da-48c2-b24b-fc0701773a6a\") " Oct 31 04:28:48.558710 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:48.558649 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd0acc5e-e7da-48c2-b24b-fc0701773a6a-kube-api-access-4llv5\" (UniqueName: \"kubernetes.io/projected/dd0acc5e-e7da-48c2-b24b-fc0701773a6a-kube-api-access-4llv5\") pod \"dd0acc5e-e7da-48c2-b24b-fc0701773a6a\" (UID: \"dd0acc5e-e7da-48c2-b24b-fc0701773a6a\") " Oct 31 04:28:48.558876 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:48.558852 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd0acc5e-e7da-48c2-b24b-fc0701773a6a-plugins-config-volume" pod "dd0acc5e-e7da-48c2-b24b-fc0701773a6a" (UID: "dd0acc5e-e7da-48c2-b24b-fc0701773a6a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:28:48.559237 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:48.559216 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c91c5252-fbf0-48b5-9ddb-124322049a54-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c91c5252-fbf0-48b5-9ddb-124322049a54-plugins-config-volume\") pod \"c91c5252-fbf0-48b5-9ddb-124322049a54\" (UID: \"c91c5252-fbf0-48b5-9ddb-124322049a54\") " Oct 31 04:28:48.559489 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:48.559475 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c91c5252-fbf0-48b5-9ddb-124322049a54-kube-api-access-sj7ll\" (UniqueName: \"kubernetes.io/projected/c91c5252-fbf0-48b5-9ddb-124322049a54-kube-api-access-sj7ll\") pod \"c91c5252-fbf0-48b5-9ddb-124322049a54\" (UID: \"c91c5252-fbf0-48b5-9ddb-124322049a54\") " Oct 31 04:28:48.559608 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:48.559589 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c91c5252-fbf0-48b5-9ddb-124322049a54-plugins-config-volume" pod "c91c5252-fbf0-48b5-9ddb-124322049a54" (UID: "c91c5252-fbf0-48b5-9ddb-124322049a54"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:28:48.560102 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:48.560086 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c91c5252-fbf0-48b5-9ddb-124322049a54-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:28:48.560184 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:48.560176 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd0acc5e-e7da-48c2-b24b-fc0701773a6a-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:28:48.561035 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:48.561010 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd0acc5e-e7da-48c2-b24b-fc0701773a6a-kube-api-access-4llv5" pod "dd0acc5e-e7da-48c2-b24b-fc0701773a6a" (UID: "dd0acc5e-e7da-48c2-b24b-fc0701773a6a"). InnerVolumeSpecName "kube-api-access-4llv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:28:48.561564 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:48.561539 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c91c5252-fbf0-48b5-9ddb-124322049a54-kube-api-access-sj7ll" pod "c91c5252-fbf0-48b5-9ddb-124322049a54" (UID: "c91c5252-fbf0-48b5-9ddb-124322049a54"). InnerVolumeSpecName "kube-api-access-sj7ll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:28:48.661896 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:48.661818 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4llv5\" (UniqueName: \"kubernetes.io/projected/dd0acc5e-e7da-48c2-b24b-fc0701773a6a-kube-api-access-4llv5\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:28:48.661896 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:48.661848 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sj7ll\" (UniqueName: \"kubernetes.io/projected/c91c5252-fbf0-48b5-9ddb-124322049a54-kube-api-access-sj7ll\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:28:49.394819 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:49.394781 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24cd29-6d745d66c9-b2hdr" event={"ID":"dd0acc5e-e7da-48c2-b24b-fc0701773a6a","Type":"ContainerDied","Data":"52b7fa7d0bfd7e7e32cf074abe6f87fb0d424565508104fe2530413fa16842dd"} Oct 31 04:28:49.395471 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:49.395453 5019 scope.go:122] "RemoveContainer" containerID="1155993ffe9d6670a538888ead2820e27d2ec357863998fe8d9529228d0f2058" Oct 31 04:28:49.395699 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:49.395393 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24cd29-6d745d66c9-b2hdr" Oct 31 04:28:49.408925 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:49.408897 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-24cd29-69d9789c79-fh6c4" Oct 31 04:28:49.428721 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:49.425429 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-24cd29-69d9789c79-fh6c4" event={"ID":"c91c5252-fbf0-48b5-9ddb-124322049a54","Type":"ContainerDied","Data":"4d7b52cedd1c92f057ceb8cc60f0f0aa0e1de43462cf3584f5866375e0163eb7"} Oct 31 04:28:49.432413 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:49.432386 5019 scope.go:122] "RemoveContainer" containerID="6fd9bfa0d5511c0b046770cc316761a9598c86ff645291ea2ec04f993269b3ee" Oct 31 04:28:49.516373 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:49.512978 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-24cd29-69d9789c79-fh6c4"] Oct 31 04:28:49.518633 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:49.518601 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-24cd29-69d9789c79-fh6c4"] Oct 31 04:28:49.543775 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:49.543744 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-24cd29-6d745d66c9-b2hdr"] Oct 31 04:28:49.547409 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:49.547384 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-24cd29-6d745d66c9-b2hdr"] Oct 31 04:28:51.412687 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:51.412653 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c91c5252-fbf0-48b5-9ddb-124322049a54" path="/var/lib/kubelet/pods/c91c5252-fbf0-48b5-9ddb-124322049a54/volumes" Oct 31 04:28:51.413380 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:51.413356 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd0acc5e-e7da-48c2-b24b-fc0701773a6a" path="/var/lib/kubelet/pods/dd0acc5e-e7da-48c2-b24b-fc0701773a6a/volumes" Oct 31 04:28:52.393246 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:28:52.393208 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:28:53.601541 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:53.601502 5019 patch_prober.go:29] interesting pod/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:28:53.601964 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:53.601586 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8" podUID="73613ab5-a053-42d6-b3f4-6e4a381da1ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:28:55.461180 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:55.461085 5019 generic.go:356] "Generic (PLEG): container finished" podID="73613ab5-a053-42d6-b3f4-6e4a381da1ac" containerID="0d1dc38f376d2b8e894ce19f102caa980e174d6b88767afec2d6703ff7913cb5" exitCode=0 Oct 31 04:28:55.461180 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:55.461156 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8" event={"ID":"73613ab5-a053-42d6-b3f4-6e4a381da1ac","Type":"ContainerDied","Data":"0d1dc38f376d2b8e894ce19f102caa980e174d6b88767afec2d6703ff7913cb5"} Oct 31 04:28:55.461774 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:55.461659 5019 scope.go:122] "RemoveContainer" containerID="0d1dc38f376d2b8e894ce19f102caa980e174d6b88767afec2d6703ff7913cb5" Oct 31 04:28:55.463225 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:55.463196 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:28:55.463314 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:55.463284 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:28:55.463388 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:55.463316 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:28:56.497922 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:56.497682 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8" event={"ID":"73613ab5-a053-42d6-b3f4-6e4a381da1ac","Type":"ContainerStarted","Data":"802b87ce7cf6172d586899fd6236c109f1a537f16652ae280694b6af09afe643"} Oct 31 04:28:56.606672 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:56.606637 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-9ee727-7f9ccf9d5-r5j4l"] Oct 31 04:28:56.634469 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:56.634391 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-9ee727-d7d75f767-qq7nn"] Oct 31 04:28:56.647127 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:56.647097 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8" Oct 31 04:28:56.647282 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:56.647259 5019 patch_prober.go:29] interesting pod/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 31 04:28:56.647328 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:56.647308 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8" podUID="73613ab5-a053-42d6-b3f4-6e4a381da1ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 31 04:28:57.647678 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:57.647645 5019 patch_prober.go:29] interesting pod/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 31 04:28:57.648230 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:57.647696 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8" podUID="73613ab5-a053-42d6-b3f4-6e4a381da1ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 31 04:28:58.647379 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:58.647348 5019 patch_prober.go:29] interesting pod/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 31 04:28:58.647567 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:58.647399 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8" podUID="73613ab5-a053-42d6-b3f4-6e4a381da1ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 31 04:28:59.647811 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:59.647774 5019 patch_prober.go:29] interesting pod/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 31 04:28:59.648215 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:28:59.647831 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8" podUID="73613ab5-a053-42d6-b3f4-6e4a381da1ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 31 04:29:00.647541 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:00.647501 5019 patch_prober.go:29] interesting pod/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 31 04:29:00.647743 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:00.647560 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8" podUID="73613ab5-a053-42d6-b3f4-6e4a381da1ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 31 04:29:01.607856 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:01.607798 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-9ee727-7f9ccf9d5-r5j4l" podUID="ba7c4e82-982a-435c-9cc5-36042d5c88d5" containerName="payload-processing" containerID="cri-o://49daf298ba117bafd3029084a99ad703ceaec49f2f1ebc5ace390c1de370f03c" gracePeriod=25 Oct 31 04:29:01.635414 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:01.635375 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9ee727-d7d75f767-qq7nn" podUID="4cc24bed-f9fb-43a0-80f4-411f33eb3f97" containerName="payload-pre-processing" containerID="cri-o://efd16cbe95f4c7efc9e7bbde4711284635b88503ed33a0abec5981b8f7d74eab" gracePeriod=25 Oct 31 04:29:01.653254 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:01.653229 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8" Oct 31 04:29:01.654038 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:01.654013 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8" Oct 31 04:29:01.658353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:01.657963 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8" Oct 31 04:29:02.024072 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:02.023718 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-9ee727-7f9ccf9d5-r5j4l" Oct 31 04:29:02.153403 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:02.150832 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba7c4e82-982a-435c-9cc5-36042d5c88d5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba7c4e82-982a-435c-9cc5-36042d5c88d5-plugins-config-volume\") pod \"ba7c4e82-982a-435c-9cc5-36042d5c88d5\" (UID: \"ba7c4e82-982a-435c-9cc5-36042d5c88d5\") " Oct 31 04:29:02.153403 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:02.151086 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba7c4e82-982a-435c-9cc5-36042d5c88d5-kube-api-access-htqzv\" (UniqueName: \"kubernetes.io/projected/ba7c4e82-982a-435c-9cc5-36042d5c88d5-kube-api-access-htqzv\") pod \"ba7c4e82-982a-435c-9cc5-36042d5c88d5\" (UID: \"ba7c4e82-982a-435c-9cc5-36042d5c88d5\") " Oct 31 04:29:02.158974 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:02.158939 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba7c4e82-982a-435c-9cc5-36042d5c88d5-plugins-config-volume" pod "ba7c4e82-982a-435c-9cc5-36042d5c88d5" (UID: "ba7c4e82-982a-435c-9cc5-36042d5c88d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:29:02.178120 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:02.175311 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba7c4e82-982a-435c-9cc5-36042d5c88d5-kube-api-access-htqzv" pod "ba7c4e82-982a-435c-9cc5-36042d5c88d5" (UID: "ba7c4e82-982a-435c-9cc5-36042d5c88d5"). InnerVolumeSpecName "kube-api-access-htqzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:29:02.203436 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:02.203406 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9ee727-d7d75f767-qq7nn" Oct 31 04:29:02.253061 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:02.253022 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htqzv\" (UniqueName: \"kubernetes.io/projected/ba7c4e82-982a-435c-9cc5-36042d5c88d5-kube-api-access-htqzv\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:29:02.253061 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:02.253061 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba7c4e82-982a-435c-9cc5-36042d5c88d5-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:29:02.354456 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:02.354425 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cc24bed-f9fb-43a0-80f4-411f33eb3f97-kube-api-access-5cbg4\" (UniqueName: \"kubernetes.io/projected/4cc24bed-f9fb-43a0-80f4-411f33eb3f97-kube-api-access-5cbg4\") pod \"4cc24bed-f9fb-43a0-80f4-411f33eb3f97\" (UID: \"4cc24bed-f9fb-43a0-80f4-411f33eb3f97\") " Oct 31 04:29:02.354622 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:02.354543 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cc24bed-f9fb-43a0-80f4-411f33eb3f97-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cc24bed-f9fb-43a0-80f4-411f33eb3f97-plugins-config-volume\") pod \"4cc24bed-f9fb-43a0-80f4-411f33eb3f97\" (UID: \"4cc24bed-f9fb-43a0-80f4-411f33eb3f97\") " Oct 31 04:29:02.354918 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:02.354886 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cc24bed-f9fb-43a0-80f4-411f33eb3f97-plugins-config-volume" pod "4cc24bed-f9fb-43a0-80f4-411f33eb3f97" (UID: "4cc24bed-f9fb-43a0-80f4-411f33eb3f97"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:29:02.355042 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:02.354976 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cc24bed-f9fb-43a0-80f4-411f33eb3f97-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:29:02.356544 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:02.356526 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cc24bed-f9fb-43a0-80f4-411f33eb3f97-kube-api-access-5cbg4" pod "4cc24bed-f9fb-43a0-80f4-411f33eb3f97" (UID: "4cc24bed-f9fb-43a0-80f4-411f33eb3f97"). InnerVolumeSpecName "kube-api-access-5cbg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:29:02.456150 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:02.456087 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cbg4\" (UniqueName: \"kubernetes.io/projected/4cc24bed-f9fb-43a0-80f4-411f33eb3f97-kube-api-access-5cbg4\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:29:02.572380 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:02.572346 5019 generic.go:356] "Generic (PLEG): container finished" podID="ba7c4e82-982a-435c-9cc5-36042d5c88d5" containerID="49daf298ba117bafd3029084a99ad703ceaec49f2f1ebc5ace390c1de370f03c" exitCode=0 Oct 31 04:29:02.572539 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:02.572387 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-9ee727-7f9ccf9d5-r5j4l" Oct 31 04:29:02.572539 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:02.572426 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-9ee727-7f9ccf9d5-r5j4l" event={"ID":"ba7c4e82-982a-435c-9cc5-36042d5c88d5","Type":"ContainerDied","Data":"49daf298ba117bafd3029084a99ad703ceaec49f2f1ebc5ace390c1de370f03c"} Oct 31 04:29:02.572539 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:02.572490 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-9ee727-7f9ccf9d5-r5j4l" event={"ID":"ba7c4e82-982a-435c-9cc5-36042d5c88d5","Type":"ContainerDied","Data":"fdfcd096487696500da8260413a5a5730f68b5231318c6bb0ba0e497b017d38c"} Oct 31 04:29:02.572539 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:02.572521 5019 scope.go:122] "RemoveContainer" containerID="49daf298ba117bafd3029084a99ad703ceaec49f2f1ebc5ace390c1de370f03c" Oct 31 04:29:02.574130 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:02.574109 5019 generic.go:356] "Generic (PLEG): container finished" podID="4cc24bed-f9fb-43a0-80f4-411f33eb3f97" containerID="efd16cbe95f4c7efc9e7bbde4711284635b88503ed33a0abec5981b8f7d74eab" exitCode=0 Oct 31 04:29:02.574239 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:02.574163 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9ee727-d7d75f767-qq7nn" Oct 31 04:29:02.574239 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:02.574198 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9ee727-d7d75f767-qq7nn" event={"ID":"4cc24bed-f9fb-43a0-80f4-411f33eb3f97","Type":"ContainerDied","Data":"efd16cbe95f4c7efc9e7bbde4711284635b88503ed33a0abec5981b8f7d74eab"} Oct 31 04:29:02.574328 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:02.574247 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9ee727-d7d75f767-qq7nn" event={"ID":"4cc24bed-f9fb-43a0-80f4-411f33eb3f97","Type":"ContainerDied","Data":"68a85e335d8b7b451a29c269755a6bc813acc6715972660dbc31d0539d88b279"} Oct 31 04:29:02.585684 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:02.585637 5019 scope.go:122] "RemoveContainer" containerID="49daf298ba117bafd3029084a99ad703ceaec49f2f1ebc5ace390c1de370f03c" Oct 31 04:29:02.585949 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:29:02.585930 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49daf298ba117bafd3029084a99ad703ceaec49f2f1ebc5ace390c1de370f03c\": container with ID starting with 49daf298ba117bafd3029084a99ad703ceaec49f2f1ebc5ace390c1de370f03c not found: ID does not exist" containerID="49daf298ba117bafd3029084a99ad703ceaec49f2f1ebc5ace390c1de370f03c" Oct 31 04:29:02.586017 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:02.585963 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49daf298ba117bafd3029084a99ad703ceaec49f2f1ebc5ace390c1de370f03c"} err="failed to get container status \"49daf298ba117bafd3029084a99ad703ceaec49f2f1ebc5ace390c1de370f03c\": rpc error: code = NotFound desc = could not find container \"49daf298ba117bafd3029084a99ad703ceaec49f2f1ebc5ace390c1de370f03c\": container with ID starting with 49daf298ba117bafd3029084a99ad703ceaec49f2f1ebc5ace390c1de370f03c not found: ID does not exist" Oct 31 04:29:02.586017 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:02.585985 5019 scope.go:122] "RemoveContainer" containerID="efd16cbe95f4c7efc9e7bbde4711284635b88503ed33a0abec5981b8f7d74eab" Oct 31 04:29:02.595395 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:02.595377 5019 scope.go:122] "RemoveContainer" containerID="efd16cbe95f4c7efc9e7bbde4711284635b88503ed33a0abec5981b8f7d74eab" Oct 31 04:29:02.595655 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:29:02.595635 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efd16cbe95f4c7efc9e7bbde4711284635b88503ed33a0abec5981b8f7d74eab\": container with ID starting with efd16cbe95f4c7efc9e7bbde4711284635b88503ed33a0abec5981b8f7d74eab not found: ID does not exist" containerID="efd16cbe95f4c7efc9e7bbde4711284635b88503ed33a0abec5981b8f7d74eab" Oct 31 04:29:02.595718 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:02.595659 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efd16cbe95f4c7efc9e7bbde4711284635b88503ed33a0abec5981b8f7d74eab"} err="failed to get container status \"efd16cbe95f4c7efc9e7bbde4711284635b88503ed33a0abec5981b8f7d74eab\": rpc error: code = NotFound desc = could not find container \"efd16cbe95f4c7efc9e7bbde4711284635b88503ed33a0abec5981b8f7d74eab\": container with ID starting with efd16cbe95f4c7efc9e7bbde4711284635b88503ed33a0abec5981b8f7d74eab not found: ID does not exist" Oct 31 04:29:02.625226 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:02.625106 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-9ee727-7f9ccf9d5-r5j4l"] Oct 31 04:29:02.630183 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:02.630122 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-9ee727-7f9ccf9d5-r5j4l"] Oct 31 04:29:02.667421 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:02.667391 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-9ee727-d7d75f767-qq7nn"] Oct 31 04:29:02.670844 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:02.670816 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-9ee727-d7d75f767-qq7nn"] Oct 31 04:29:03.411312 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:03.411280 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cc24bed-f9fb-43a0-80f4-411f33eb3f97" path="/var/lib/kubelet/pods/4cc24bed-f9fb-43a0-80f4-411f33eb3f97/volumes" Oct 31 04:29:03.411758 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:03.411742 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba7c4e82-982a-435c-9cc5-36042d5c88d5" path="/var/lib/kubelet/pods/ba7c4e82-982a-435c-9cc5-36042d5c88d5/volumes" Oct 31 04:29:04.859362 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:04.859318 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c6859a-5f796cd5cf-gmshg"] Oct 31 04:29:05.109094 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:05.109065 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g"] Oct 31 04:29:05.109313 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:05.109286 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g" podUID="0165c2c6-76a2-42fc-a3f4-50674c99bc88" containerName="istio-proxy" containerID="cri-o://5af8047060e9c391d0aa7adec8dacf4c8ea8445e596d8262cdacff7ae01e8483" gracePeriod=30 Oct 31 04:29:05.741634 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:05.741606 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s88kp_3945ff65-e877-4372-9334-3968c68b7295/console-operator/2.log" Oct 31 04:29:05.754064 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:05.754030 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s88kp_3945ff65-e877-4372-9334-3968c68b7295/console-operator/2.log" Oct 31 04:29:08.265053 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:08.265024 5019 scope.go:122] "RemoveContainer" containerID="8a69b339b42f22e7c3da86e0aabb4ee1bd92a777fe72201f5208c3b4eb31cab8" Oct 31 04:29:08.275753 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:08.275729 5019 scope.go:122] "RemoveContainer" containerID="fe497322f646aa422a46e12333f90173e11208df8911866d50631f9169b9adaf" Oct 31 04:29:09.163366 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:09.163320 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-24cd29-5c5cb6887b-chxzf"] Oct 31 04:29:09.164010 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:09.163994 5019 cpu_manager.go:414] "containerMap: removing container" podUID="c91c5252-fbf0-48b5-9ddb-124322049a54" containerName="payload-processing" Oct 31 04:29:09.164062 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:09.164011 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c91c5252-fbf0-48b5-9ddb-124322049a54" containerName="payload-processing" Oct 31 04:29:09.164062 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:09.164035 5019 cpu_manager.go:414] "containerMap: removing container" podUID="4cc24bed-f9fb-43a0-80f4-411f33eb3f97" containerName="payload-pre-processing" Oct 31 04:29:09.164062 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:09.164040 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cc24bed-f9fb-43a0-80f4-411f33eb3f97" containerName="payload-pre-processing" Oct 31 04:29:09.164062 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:09.164056 5019 cpu_manager.go:414] "containerMap: removing container" podUID="dd0acc5e-e7da-48c2-b24b-fc0701773a6a" containerName="payload-pre-processing" Oct 31 04:29:09.164062 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:09.164061 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd0acc5e-e7da-48c2-b24b-fc0701773a6a" containerName="payload-pre-processing" Oct 31 04:29:09.164213 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:09.164073 5019 cpu_manager.go:414] "containerMap: removing container" podUID="ba7c4e82-982a-435c-9cc5-36042d5c88d5" containerName="payload-processing" Oct 31 04:29:09.164213 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:09.164077 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba7c4e82-982a-435c-9cc5-36042d5c88d5" containerName="payload-processing" Oct 31 04:29:09.164213 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:09.164175 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="c91c5252-fbf0-48b5-9ddb-124322049a54" containerName="payload-processing" Oct 31 04:29:09.164213 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:09.164187 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cc24bed-f9fb-43a0-80f4-411f33eb3f97" containerName="payload-pre-processing" Oct 31 04:29:09.164213 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:09.164194 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd0acc5e-e7da-48c2-b24b-fc0701773a6a" containerName="payload-pre-processing" Oct 31 04:29:09.164213 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:09.164201 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba7c4e82-982a-435c-9cc5-36042d5c88d5" containerName="payload-processing" Oct 31 04:29:09.169149 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:09.169132 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-24cd29-5c5cb6887b-chxzf" Oct 31 04:29:09.193261 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:09.193234 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-24cd29-5c5cb6887b-chxzf"] Oct 31 04:29:09.197836 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:09.197814 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-24cd29-dockercfg-l9cnp\"" Oct 31 04:29:09.241021 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:09.240975 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-24cd29\"" Oct 31 04:29:09.269821 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:09.269796 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-24cd29-67c5899cb7-qfvwr"] Oct 31 04:29:09.274017 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:09.273997 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24cd29-67c5899cb7-qfvwr" Oct 31 04:29:09.281421 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:09.281398 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/530d712c-fe37-4d92-ae54-be36babd2335-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-24cd29-5c5cb6887b-chxzf\" (UID: \"530d712c-fe37-4d92-ae54-be36babd2335\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-24cd29-5c5cb6887b-chxzf" Oct 31 04:29:09.281637 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:09.281603 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b624\" (UniqueName: \"kubernetes.io/projected/530d712c-fe37-4d92-ae54-be36babd2335-kube-api-access-4b624\") pod \"payload-processing-e2e-ipp-b-24cd29-5c5cb6887b-chxzf\" (UID: \"530d712c-fe37-4d92-ae54-be36babd2335\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-24cd29-5c5cb6887b-chxzf" Oct 31 04:29:09.293316 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:09.293292 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-24cd29-67c5899cb7-qfvwr"] Oct 31 04:29:09.383392 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:09.383362 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b624\" (UniqueName: \"kubernetes.io/projected/530d712c-fe37-4d92-ae54-be36babd2335-kube-api-access-4b624\") pod \"payload-processing-e2e-ipp-b-24cd29-5c5cb6887b-chxzf\" (UID: \"530d712c-fe37-4d92-ae54-be36babd2335\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-24cd29-5c5cb6887b-chxzf" Oct 31 04:29:09.383556 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:09.383519 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crqdv\" (UniqueName: \"kubernetes.io/projected/37afe0fa-7748-4624-a783-4dd54c30b28d-kube-api-access-crqdv\") pod \"payload-pre-processing-e2e-ipp-b-24cd29-67c5899cb7-qfvwr\" (UID: \"37afe0fa-7748-4624-a783-4dd54c30b28d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24cd29-67c5899cb7-qfvwr" Oct 31 04:29:09.383634 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:09.383623 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/530d712c-fe37-4d92-ae54-be36babd2335-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-24cd29-5c5cb6887b-chxzf\" (UID: \"530d712c-fe37-4d92-ae54-be36babd2335\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-24cd29-5c5cb6887b-chxzf" Oct 31 04:29:09.383691 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:09.383649 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37afe0fa-7748-4624-a783-4dd54c30b28d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-24cd29-67c5899cb7-qfvwr\" (UID: \"37afe0fa-7748-4624-a783-4dd54c30b28d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24cd29-67c5899cb7-qfvwr" Oct 31 04:29:09.384282 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:09.384262 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/530d712c-fe37-4d92-ae54-be36babd2335-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-24cd29-5c5cb6887b-chxzf\" (UID: \"530d712c-fe37-4d92-ae54-be36babd2335\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-24cd29-5c5cb6887b-chxzf" Oct 31 04:29:09.407187 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:09.407161 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b624\" (UniqueName: \"kubernetes.io/projected/530d712c-fe37-4d92-ae54-be36babd2335-kube-api-access-4b624\") pod \"payload-processing-e2e-ipp-b-24cd29-5c5cb6887b-chxzf\" (UID: \"530d712c-fe37-4d92-ae54-be36babd2335\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-24cd29-5c5cb6887b-chxzf" Oct 31 04:29:09.484479 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:09.484457 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-24cd29-5c5cb6887b-chxzf" Oct 31 04:29:09.484885 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:09.484854 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crqdv\" (UniqueName: \"kubernetes.io/projected/37afe0fa-7748-4624-a783-4dd54c30b28d-kube-api-access-crqdv\") pod \"payload-pre-processing-e2e-ipp-b-24cd29-67c5899cb7-qfvwr\" (UID: \"37afe0fa-7748-4624-a783-4dd54c30b28d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24cd29-67c5899cb7-qfvwr" Oct 31 04:29:09.484968 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:09.484935 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37afe0fa-7748-4624-a783-4dd54c30b28d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-24cd29-67c5899cb7-qfvwr\" (UID: \"37afe0fa-7748-4624-a783-4dd54c30b28d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24cd29-67c5899cb7-qfvwr" Oct 31 04:29:09.485736 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:09.485713 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37afe0fa-7748-4624-a783-4dd54c30b28d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-24cd29-67c5899cb7-qfvwr\" (UID: \"37afe0fa-7748-4624-a783-4dd54c30b28d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24cd29-67c5899cb7-qfvwr" Oct 31 04:29:09.508679 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:09.508654 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crqdv\" (UniqueName: \"kubernetes.io/projected/37afe0fa-7748-4624-a783-4dd54c30b28d-kube-api-access-crqdv\") pod \"payload-pre-processing-e2e-ipp-b-24cd29-67c5899cb7-qfvwr\" (UID: \"37afe0fa-7748-4624-a783-4dd54c30b28d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24cd29-67c5899cb7-qfvwr" Oct 31 04:29:09.592602 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:09.592566 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24cd29-67c5899cb7-qfvwr" Oct 31 04:29:09.794757 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:09.794682 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-24cd29-5c5cb6887b-chxzf"] Oct 31 04:29:09.807586 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:29:09.807556 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod530d712c_fe37_4d92_ae54_be36babd2335.slice/crio-ef25d268f1e5bffd7e7fbfe539573a81105bb0ae7787af499bbd56d01e4bee17 WatchSource:0}: Error finding container ef25d268f1e5bffd7e7fbfe539573a81105bb0ae7787af499bbd56d01e4bee17: Status 404 returned error can't find the container with id ef25d268f1e5bffd7e7fbfe539573a81105bb0ae7787af499bbd56d01e4bee17 Oct 31 04:29:09.859823 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:09.859773 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-c6859a-5f796cd5cf-gmshg" podUID="e4a67a18-4432-4411-9c7f-09b505667a44" containerName="payload-processing" containerID="cri-o://b5dfcc18b00d87e4f0aaf36a80c5b9a0f9ed2969b3e1ea11b7126b0a962e56b7" gracePeriod=25 Oct 31 04:29:09.908185 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:09.908154 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-24cd29-67c5899cb7-qfvwr"] Oct 31 04:29:09.915130 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:29:09.915100 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37afe0fa_7748_4624_a783_4dd54c30b28d.slice/crio-4272dbe7b0d2bd2b0d313fc5795a97d6680ca23d9a7f7a5075ffedf73326cb8c WatchSource:0}: Error finding container 4272dbe7b0d2bd2b0d313fc5795a97d6680ca23d9a7f7a5075ffedf73326cb8c: Status 404 returned error can't find the container with id 4272dbe7b0d2bd2b0d313fc5795a97d6680ca23d9a7f7a5075ffedf73326cb8c Oct 31 04:29:10.224483 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.224447 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c6859a-5f796cd5cf-gmshg" Oct 31 04:29:10.327938 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.327907 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4a67a18-4432-4411-9c7f-09b505667a44-kube-api-access-nrqn5\" (UniqueName: \"kubernetes.io/projected/e4a67a18-4432-4411-9c7f-09b505667a44-kube-api-access-nrqn5\") pod \"e4a67a18-4432-4411-9c7f-09b505667a44\" (UID: \"e4a67a18-4432-4411-9c7f-09b505667a44\") " Oct 31 04:29:10.328366 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.328025 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4a67a18-4432-4411-9c7f-09b505667a44-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4a67a18-4432-4411-9c7f-09b505667a44-plugins-config-volume\") pod \"e4a67a18-4432-4411-9c7f-09b505667a44\" (UID: \"e4a67a18-4432-4411-9c7f-09b505667a44\") " Oct 31 04:29:10.329048 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.329017 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4a67a18-4432-4411-9c7f-09b505667a44-plugins-config-volume" pod "e4a67a18-4432-4411-9c7f-09b505667a44" (UID: "e4a67a18-4432-4411-9c7f-09b505667a44"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:29:10.329271 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.329249 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4a67a18-4432-4411-9c7f-09b505667a44-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:29:10.331010 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.330983 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4a67a18-4432-4411-9c7f-09b505667a44-kube-api-access-nrqn5" pod "e4a67a18-4432-4411-9c7f-09b505667a44" (UID: "e4a67a18-4432-4411-9c7f-09b505667a44"). InnerVolumeSpecName "kube-api-access-nrqn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:29:10.430744 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.430718 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrqn5\" (UniqueName: \"kubernetes.io/projected/e4a67a18-4432-4411-9c7f-09b505667a44-kube-api-access-nrqn5\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:29:10.460258 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.460233 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g" Oct 31 04:29:10.531867 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.531832 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0165c2c6-76a2-42fc-a3f4-50674c99bc88-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0165c2c6-76a2-42fc-a3f4-50674c99bc88-istio-data\") pod \"0165c2c6-76a2-42fc-a3f4-50674c99bc88\" (UID: \"0165c2c6-76a2-42fc-a3f4-50674c99bc88\") " Oct 31 04:29:10.532034 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.531897 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0165c2c6-76a2-42fc-a3f4-50674c99bc88-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0165c2c6-76a2-42fc-a3f4-50674c99bc88-workload-certs\") pod \"0165c2c6-76a2-42fc-a3f4-50674c99bc88\" (UID: \"0165c2c6-76a2-42fc-a3f4-50674c99bc88\") " Oct 31 04:29:10.532034 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.531948 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0165c2c6-76a2-42fc-a3f4-50674c99bc88-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0165c2c6-76a2-42fc-a3f4-50674c99bc88-istio-podinfo\") pod \"0165c2c6-76a2-42fc-a3f4-50674c99bc88\" (UID: \"0165c2c6-76a2-42fc-a3f4-50674c99bc88\") " Oct 31 04:29:10.532162 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.532079 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0165c2c6-76a2-42fc-a3f4-50674c99bc88-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0165c2c6-76a2-42fc-a3f4-50674c99bc88-credential-socket\") pod \"0165c2c6-76a2-42fc-a3f4-50674c99bc88\" (UID: \"0165c2c6-76a2-42fc-a3f4-50674c99bc88\") " Oct 31 04:29:10.532162 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.532134 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0165c2c6-76a2-42fc-a3f4-50674c99bc88-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0165c2c6-76a2-42fc-a3f4-50674c99bc88-workload-socket\") pod \"0165c2c6-76a2-42fc-a3f4-50674c99bc88\" (UID: \"0165c2c6-76a2-42fc-a3f4-50674c99bc88\") " Oct 31 04:29:10.532162 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.532138 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0165c2c6-76a2-42fc-a3f4-50674c99bc88-istio-data" pod "0165c2c6-76a2-42fc-a3f4-50674c99bc88" (UID: "0165c2c6-76a2-42fc-a3f4-50674c99bc88"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:29:10.532327 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.532168 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0165c2c6-76a2-42fc-a3f4-50674c99bc88-kube-api-access-b4c6x\" (UniqueName: \"kubernetes.io/projected/0165c2c6-76a2-42fc-a3f4-50674c99bc88-kube-api-access-b4c6x\") pod \"0165c2c6-76a2-42fc-a3f4-50674c99bc88\" (UID: \"0165c2c6-76a2-42fc-a3f4-50674c99bc88\") " Oct 31 04:29:10.532327 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.532203 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0165c2c6-76a2-42fc-a3f4-50674c99bc88-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0165c2c6-76a2-42fc-a3f4-50674c99bc88-istiod-ca-cert\") pod \"0165c2c6-76a2-42fc-a3f4-50674c99bc88\" (UID: \"0165c2c6-76a2-42fc-a3f4-50674c99bc88\") " Oct 31 04:29:10.532327 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.532247 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0165c2c6-76a2-42fc-a3f4-50674c99bc88-istio-token\" (UniqueName: \"kubernetes.io/projected/0165c2c6-76a2-42fc-a3f4-50674c99bc88-istio-token\") pod \"0165c2c6-76a2-42fc-a3f4-50674c99bc88\" (UID: \"0165c2c6-76a2-42fc-a3f4-50674c99bc88\") " Oct 31 04:29:10.532504 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.532419 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0165c2c6-76a2-42fc-a3f4-50674c99bc88-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0165c2c6-76a2-42fc-a3f4-50674c99bc88-istio-envoy\") pod \"0165c2c6-76a2-42fc-a3f4-50674c99bc88\" (UID: \"0165c2c6-76a2-42fc-a3f4-50674c99bc88\") " Oct 31 04:29:10.533101 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.533074 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0165c2c6-76a2-42fc-a3f4-50674c99bc88-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:29:10.533456 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.533419 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0165c2c6-76a2-42fc-a3f4-50674c99bc88-workload-socket" pod "0165c2c6-76a2-42fc-a3f4-50674c99bc88" (UID: "0165c2c6-76a2-42fc-a3f4-50674c99bc88"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:29:10.533574 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.533440 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0165c2c6-76a2-42fc-a3f4-50674c99bc88-credential-socket" pod "0165c2c6-76a2-42fc-a3f4-50674c99bc88" (UID: "0165c2c6-76a2-42fc-a3f4-50674c99bc88"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:29:10.533629 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.533600 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0165c2c6-76a2-42fc-a3f4-50674c99bc88-istiod-ca-cert" pod "0165c2c6-76a2-42fc-a3f4-50674c99bc88" (UID: "0165c2c6-76a2-42fc-a3f4-50674c99bc88"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:29:10.534113 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.534093 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0165c2c6-76a2-42fc-a3f4-50674c99bc88-workload-certs" pod "0165c2c6-76a2-42fc-a3f4-50674c99bc88" (UID: "0165c2c6-76a2-42fc-a3f4-50674c99bc88"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:29:10.543659 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.543624 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0165c2c6-76a2-42fc-a3f4-50674c99bc88-istio-token" pod "0165c2c6-76a2-42fc-a3f4-50674c99bc88" (UID: "0165c2c6-76a2-42fc-a3f4-50674c99bc88"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:29:10.543748 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.543663 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0165c2c6-76a2-42fc-a3f4-50674c99bc88-kube-api-access-b4c6x" pod "0165c2c6-76a2-42fc-a3f4-50674c99bc88" (UID: "0165c2c6-76a2-42fc-a3f4-50674c99bc88"). InnerVolumeSpecName "kube-api-access-b4c6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:29:10.543748 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.543701 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0165c2c6-76a2-42fc-a3f4-50674c99bc88-istio-podinfo" pod "0165c2c6-76a2-42fc-a3f4-50674c99bc88" (UID: "0165c2c6-76a2-42fc-a3f4-50674c99bc88"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:29:10.545322 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.545299 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0165c2c6-76a2-42fc-a3f4-50674c99bc88-istio-envoy" pod "0165c2c6-76a2-42fc-a3f4-50674c99bc88" (UID: "0165c2c6-76a2-42fc-a3f4-50674c99bc88"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:29:10.634184 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.634130 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0165c2c6-76a2-42fc-a3f4-50674c99bc88-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:29:10.634184 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.634153 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0165c2c6-76a2-42fc-a3f4-50674c99bc88-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:29:10.634184 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.634161 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4c6x\" (UniqueName: \"kubernetes.io/projected/0165c2c6-76a2-42fc-a3f4-50674c99bc88-kube-api-access-b4c6x\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:29:10.634184 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.634169 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0165c2c6-76a2-42fc-a3f4-50674c99bc88-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:29:10.634184 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.634180 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0165c2c6-76a2-42fc-a3f4-50674c99bc88-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:29:10.634184 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.634187 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0165c2c6-76a2-42fc-a3f4-50674c99bc88-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:29:10.634451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.634196 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0165c2c6-76a2-42fc-a3f4-50674c99bc88-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:29:10.634451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.634204 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0165c2c6-76a2-42fc-a3f4-50674c99bc88-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:29:10.708374 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.706918 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24cd29-67c5899cb7-qfvwr" event={"ID":"37afe0fa-7748-4624-a783-4dd54c30b28d","Type":"ContainerStarted","Data":"f02eed02bd8f997a168e0972a6d01104f4b4b1fde279eb0cd0e76c554e1e92d7"} Oct 31 04:29:10.708374 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.706972 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24cd29-67c5899cb7-qfvwr" event={"ID":"37afe0fa-7748-4624-a783-4dd54c30b28d","Type":"ContainerStarted","Data":"4272dbe7b0d2bd2b0d313fc5795a97d6680ca23d9a7f7a5075ffedf73326cb8c"} Oct 31 04:29:10.708374 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.707030 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24cd29-67c5899cb7-qfvwr" Oct 31 04:29:10.709116 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.708844 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-24cd29-5c5cb6887b-chxzf" event={"ID":"530d712c-fe37-4d92-ae54-be36babd2335","Type":"ContainerStarted","Data":"3d57b852ef55fddb48af92ed9f2d49a5f0deeb3341cf09a70bfee6be2405fb86"} Oct 31 04:29:10.709116 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.708889 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-24cd29-5c5cb6887b-chxzf" event={"ID":"530d712c-fe37-4d92-ae54-be36babd2335","Type":"ContainerStarted","Data":"ef25d268f1e5bffd7e7fbfe539573a81105bb0ae7787af499bbd56d01e4bee17"} Oct 31 04:29:10.709116 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.708940 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-24cd29-5c5cb6887b-chxzf" Oct 31 04:29:10.711061 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.711034 5019 generic.go:356] "Generic (PLEG): container finished" podID="0165c2c6-76a2-42fc-a3f4-50674c99bc88" containerID="5af8047060e9c391d0aa7adec8dacf4c8ea8445e596d8262cdacff7ae01e8483" exitCode=0 Oct 31 04:29:10.711164 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.711085 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g" Oct 31 04:29:10.711164 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.711111 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g" event={"ID":"0165c2c6-76a2-42fc-a3f4-50674c99bc88","Type":"ContainerDied","Data":"5af8047060e9c391d0aa7adec8dacf4c8ea8445e596d8262cdacff7ae01e8483"} Oct 31 04:29:10.711164 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.711155 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g" event={"ID":"0165c2c6-76a2-42fc-a3f4-50674c99bc88","Type":"ContainerDied","Data":"1e60ff7bb346cac0bcdc83a22cc74a9af3f09d146509c5a14c017a59dc5f3145"} Oct 31 04:29:10.711323 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.711186 5019 scope.go:122] "RemoveContainer" containerID="5af8047060e9c391d0aa7adec8dacf4c8ea8445e596d8262cdacff7ae01e8483" Oct 31 04:29:10.717708 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.717539 5019 generic.go:356] "Generic (PLEG): container finished" podID="e4a67a18-4432-4411-9c7f-09b505667a44" containerID="b5dfcc18b00d87e4f0aaf36a80c5b9a0f9ed2969b3e1ea11b7126b0a962e56b7" exitCode=0 Oct 31 04:29:10.717708 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.717602 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c6859a-5f796cd5cf-gmshg" event={"ID":"e4a67a18-4432-4411-9c7f-09b505667a44","Type":"ContainerDied","Data":"b5dfcc18b00d87e4f0aaf36a80c5b9a0f9ed2969b3e1ea11b7126b0a962e56b7"} Oct 31 04:29:10.717708 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.717636 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c6859a-5f796cd5cf-gmshg" event={"ID":"e4a67a18-4432-4411-9c7f-09b505667a44","Type":"ContainerDied","Data":"0ffc78f8b6743dc89de31ebff6e7c61a3e2fce2c1a1b353287e23b758a5cb377"} Oct 31 04:29:10.717708 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.717698 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c6859a-5f796cd5cf-gmshg" Oct 31 04:29:10.724535 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.724511 5019 scope.go:122] "RemoveContainer" containerID="f9f0f9d9da35d1e56014f64fe194e7b84639347ad128aaac04c83308e683cb00" Oct 31 04:29:10.741246 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.741224 5019 scope.go:122] "RemoveContainer" containerID="5af8047060e9c391d0aa7adec8dacf4c8ea8445e596d8262cdacff7ae01e8483" Oct 31 04:29:10.741539 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:29:10.741520 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5af8047060e9c391d0aa7adec8dacf4c8ea8445e596d8262cdacff7ae01e8483\": container with ID starting with 5af8047060e9c391d0aa7adec8dacf4c8ea8445e596d8262cdacff7ae01e8483 not found: ID does not exist" containerID="5af8047060e9c391d0aa7adec8dacf4c8ea8445e596d8262cdacff7ae01e8483" Oct 31 04:29:10.741612 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.741545 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5af8047060e9c391d0aa7adec8dacf4c8ea8445e596d8262cdacff7ae01e8483"} err="failed to get container status \"5af8047060e9c391d0aa7adec8dacf4c8ea8445e596d8262cdacff7ae01e8483\": rpc error: code = NotFound desc = could not find container \"5af8047060e9c391d0aa7adec8dacf4c8ea8445e596d8262cdacff7ae01e8483\": container with ID starting with 5af8047060e9c391d0aa7adec8dacf4c8ea8445e596d8262cdacff7ae01e8483 not found: ID does not exist" Oct 31 04:29:10.741612 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.741562 5019 scope.go:122] "RemoveContainer" containerID="f9f0f9d9da35d1e56014f64fe194e7b84639347ad128aaac04c83308e683cb00" Oct 31 04:29:10.741823 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:29:10.741807 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9f0f9d9da35d1e56014f64fe194e7b84639347ad128aaac04c83308e683cb00\": container with ID starting with f9f0f9d9da35d1e56014f64fe194e7b84639347ad128aaac04c83308e683cb00 not found: ID does not exist" containerID="f9f0f9d9da35d1e56014f64fe194e7b84639347ad128aaac04c83308e683cb00" Oct 31 04:29:10.741860 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.741827 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9f0f9d9da35d1e56014f64fe194e7b84639347ad128aaac04c83308e683cb00"} err="failed to get container status \"f9f0f9d9da35d1e56014f64fe194e7b84639347ad128aaac04c83308e683cb00\": rpc error: code = NotFound desc = could not find container \"f9f0f9d9da35d1e56014f64fe194e7b84639347ad128aaac04c83308e683cb00\": container with ID starting with f9f0f9d9da35d1e56014f64fe194e7b84639347ad128aaac04c83308e683cb00 not found: ID does not exist" Oct 31 04:29:10.741860 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.741840 5019 scope.go:122] "RemoveContainer" containerID="b5dfcc18b00d87e4f0aaf36a80c5b9a0f9ed2969b3e1ea11b7126b0a962e56b7" Oct 31 04:29:10.750401 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.750382 5019 scope.go:122] "RemoveContainer" containerID="b5dfcc18b00d87e4f0aaf36a80c5b9a0f9ed2969b3e1ea11b7126b0a962e56b7" Oct 31 04:29:10.750667 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:29:10.750643 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5dfcc18b00d87e4f0aaf36a80c5b9a0f9ed2969b3e1ea11b7126b0a962e56b7\": container with ID starting with b5dfcc18b00d87e4f0aaf36a80c5b9a0f9ed2969b3e1ea11b7126b0a962e56b7 not found: ID does not exist" containerID="b5dfcc18b00d87e4f0aaf36a80c5b9a0f9ed2969b3e1ea11b7126b0a962e56b7" Oct 31 04:29:10.750819 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.750786 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5dfcc18b00d87e4f0aaf36a80c5b9a0f9ed2969b3e1ea11b7126b0a962e56b7"} err="failed to get container status \"b5dfcc18b00d87e4f0aaf36a80c5b9a0f9ed2969b3e1ea11b7126b0a962e56b7\": rpc error: code = NotFound desc = could not find container \"b5dfcc18b00d87e4f0aaf36a80c5b9a0f9ed2969b3e1ea11b7126b0a962e56b7\": container with ID starting with b5dfcc18b00d87e4f0aaf36a80c5b9a0f9ed2969b3e1ea11b7126b0a962e56b7 not found: ID does not exist" Oct 31 04:29:10.753283 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.753241 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24cd29-67c5899cb7-qfvwr" podStartSLOduration=1.753230253 podStartE2EDuration="1.753230253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:29:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:29:10.750902208 +0000 UTC m=+1505.507240248" watchObservedRunningTime="2026-10-31 04:29:10.753230253 +0000 UTC m=+1505.509568291" Oct 31 04:29:10.791581 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.791554 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g"] Oct 31 04:29:10.799476 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.799455 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-9ee727-openshift-default-745c55b44f-z5w2g"] Oct 31 04:29:10.831676 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.831648 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c6859a-5f796cd5cf-gmshg"] Oct 31 04:29:10.836143 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.836119 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c6859a-5f796cd5cf-gmshg"] Oct 31 04:29:10.877168 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:10.877104 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-24cd29-5c5cb6887b-chxzf" podStartSLOduration=1.877084006 podStartE2EDuration="1.877084006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:29:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:29:10.874810455 +0000 UTC m=+1505.631148501" watchObservedRunningTime="2026-10-31 04:29:10.877084006 +0000 UTC m=+1505.633422045" Oct 31 04:29:11.413494 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:11.413457 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0165c2c6-76a2-42fc-a3f4-50674c99bc88" path="/var/lib/kubelet/pods/0165c2c6-76a2-42fc-a3f4-50674c99bc88/volumes" Oct 31 04:29:11.414238 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:11.414215 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4a67a18-4432-4411-9c7f-09b505667a44" path="/var/lib/kubelet/pods/e4a67a18-4432-4411-9c7f-09b505667a44/volumes" Oct 31 04:29:13.267981 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:13.267933 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b44631-8456d66fbf-kb6zn"] Oct 31 04:29:13.301060 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:13.301018 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b44631-6b47595bc-b7rtc"] Oct 31 04:29:18.269263 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:18.269208 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-b44631-8456d66fbf-kb6zn" podUID="51c854e6-11ce-48a1-97a4-fb77b57f5419" containerName="payload-processing" containerID="cri-o://3f8494a84c9d4afa7134dada27f952aa76a5879de3d0a65dd200178767121248" gracePeriod=25 Oct 31 04:29:18.301723 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:18.301467 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b44631-6b47595bc-b7rtc" podUID="454f6067-ffa0-41cb-9716-d6bacd72abbf" containerName="payload-pre-processing" containerID="cri-o://6811e710aabd79bfc7a1ec9c065dea78a872f8a469788159a34a2d7237ea30cb" gracePeriod=25 Oct 31 04:29:18.858506 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:18.854965 5019 generic.go:356] "Generic (PLEG): container finished" podID="51c854e6-11ce-48a1-97a4-fb77b57f5419" containerID="3f8494a84c9d4afa7134dada27f952aa76a5879de3d0a65dd200178767121248" exitCode=0 Oct 31 04:29:18.858506 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:18.856238 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b44631-8456d66fbf-kb6zn" event={"ID":"51c854e6-11ce-48a1-97a4-fb77b57f5419","Type":"ContainerDied","Data":"3f8494a84c9d4afa7134dada27f952aa76a5879de3d0a65dd200178767121248"} Oct 31 04:29:18.865185 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:18.859768 5019 generic.go:356] "Generic (PLEG): container finished" podID="454f6067-ffa0-41cb-9716-d6bacd72abbf" containerID="6811e710aabd79bfc7a1ec9c065dea78a872f8a469788159a34a2d7237ea30cb" exitCode=0 Oct 31 04:29:18.865185 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:18.859829 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b44631-6b47595bc-b7rtc" event={"ID":"454f6067-ffa0-41cb-9716-d6bacd72abbf","Type":"ContainerDied","Data":"6811e710aabd79bfc7a1ec9c065dea78a872f8a469788159a34a2d7237ea30cb"} Oct 31 04:29:18.936250 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:18.935967 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b44631-8456d66fbf-kb6zn" Oct 31 04:29:18.950959 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:18.950882 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b44631-6b47595bc-b7rtc" Oct 31 04:29:18.951683 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:18.951656 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51c854e6-11ce-48a1-97a4-fb77b57f5419-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51c854e6-11ce-48a1-97a4-fb77b57f5419-plugins-config-volume\") pod \"51c854e6-11ce-48a1-97a4-fb77b57f5419\" (UID: \"51c854e6-11ce-48a1-97a4-fb77b57f5419\") " Oct 31 04:29:18.951817 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:18.951713 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51c854e6-11ce-48a1-97a4-fb77b57f5419-kube-api-access-pmp4f\" (UniqueName: \"kubernetes.io/projected/51c854e6-11ce-48a1-97a4-fb77b57f5419-kube-api-access-pmp4f\") pod \"51c854e6-11ce-48a1-97a4-fb77b57f5419\" (UID: \"51c854e6-11ce-48a1-97a4-fb77b57f5419\") " Oct 31 04:29:18.952209 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:18.952182 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51c854e6-11ce-48a1-97a4-fb77b57f5419-plugins-config-volume" pod "51c854e6-11ce-48a1-97a4-fb77b57f5419" (UID: "51c854e6-11ce-48a1-97a4-fb77b57f5419"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:29:18.954138 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:18.954105 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51c854e6-11ce-48a1-97a4-fb77b57f5419-kube-api-access-pmp4f" pod "51c854e6-11ce-48a1-97a4-fb77b57f5419" (UID: "51c854e6-11ce-48a1-97a4-fb77b57f5419"). InnerVolumeSpecName "kube-api-access-pmp4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:29:19.053635 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:19.053585 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/454f6067-ffa0-41cb-9716-d6bacd72abbf-kube-api-access-j566c\" (UniqueName: \"kubernetes.io/projected/454f6067-ffa0-41cb-9716-d6bacd72abbf-kube-api-access-j566c\") pod \"454f6067-ffa0-41cb-9716-d6bacd72abbf\" (UID: \"454f6067-ffa0-41cb-9716-d6bacd72abbf\") " Oct 31 04:29:19.053831 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:19.053770 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/454f6067-ffa0-41cb-9716-d6bacd72abbf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/454f6067-ffa0-41cb-9716-d6bacd72abbf-plugins-config-volume\") pod \"454f6067-ffa0-41cb-9716-d6bacd72abbf\" (UID: \"454f6067-ffa0-41cb-9716-d6bacd72abbf\") " Oct 31 04:29:19.054126 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:19.054102 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51c854e6-11ce-48a1-97a4-fb77b57f5419-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:29:19.054126 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:19.054118 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmp4f\" (UniqueName: \"kubernetes.io/projected/51c854e6-11ce-48a1-97a4-fb77b57f5419-kube-api-access-pmp4f\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:29:19.054505 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:19.054482 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/454f6067-ffa0-41cb-9716-d6bacd72abbf-plugins-config-volume" pod "454f6067-ffa0-41cb-9716-d6bacd72abbf" (UID: "454f6067-ffa0-41cb-9716-d6bacd72abbf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:29:19.056953 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:19.056916 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/454f6067-ffa0-41cb-9716-d6bacd72abbf-kube-api-access-j566c" pod "454f6067-ffa0-41cb-9716-d6bacd72abbf" (UID: "454f6067-ffa0-41cb-9716-d6bacd72abbf"). InnerVolumeSpecName "kube-api-access-j566c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:29:19.155748 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:19.155716 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/454f6067-ffa0-41cb-9716-d6bacd72abbf-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:29:19.155748 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:19.155747 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j566c\" (UniqueName: \"kubernetes.io/projected/454f6067-ffa0-41cb-9716-d6bacd72abbf-kube-api-access-j566c\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:29:19.880658 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:19.880611 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b44631-6b47595bc-b7rtc" event={"ID":"454f6067-ffa0-41cb-9716-d6bacd72abbf","Type":"ContainerDied","Data":"0586f4fa2c5ff99628498078ab62613f1564694f8ac2fe54e468b23129acc1b4"} Oct 31 04:29:19.881405 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:19.881313 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b44631-6b47595bc-b7rtc" Oct 31 04:29:19.881580 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:19.881563 5019 scope.go:122] "RemoveContainer" containerID="6811e710aabd79bfc7a1ec9c065dea78a872f8a469788159a34a2d7237ea30cb" Oct 31 04:29:19.889074 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:19.889028 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b44631-8456d66fbf-kb6zn" Oct 31 04:29:19.889274 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:19.889017 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b44631-8456d66fbf-kb6zn" event={"ID":"51c854e6-11ce-48a1-97a4-fb77b57f5419","Type":"ContainerDied","Data":"4f60e05f66c9e6c58718350233df5fdda317ddffba121ab3a9cec0847c504297"} Oct 31 04:29:19.917553 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:19.917515 5019 scope.go:122] "RemoveContainer" containerID="3f8494a84c9d4afa7134dada27f952aa76a5879de3d0a65dd200178767121248" Oct 31 04:29:19.940881 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:19.940844 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b44631-6b47595bc-b7rtc"] Oct 31 04:29:19.952117 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:19.952080 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b44631-6b47595bc-b7rtc"] Oct 31 04:29:19.978651 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:19.978605 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b44631-8456d66fbf-kb6zn"] Oct 31 04:29:19.986622 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:19.986581 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b44631-8456d66fbf-kb6zn"] Oct 31 04:29:20.463561 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:20.463519 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8"] Oct 31 04:29:20.463808 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:20.463782 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8" podUID="73613ab5-a053-42d6-b3f4-6e4a381da1ac" containerName="istio-proxy" containerID="cri-o://802b87ce7cf6172d586899fd6236c109f1a537f16652ae280694b6af09afe643" gracePeriod=30 Oct 31 04:29:21.412958 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:21.412915 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="454f6067-ffa0-41cb-9716-d6bacd72abbf" path="/var/lib/kubelet/pods/454f6067-ffa0-41cb-9716-d6bacd72abbf/volumes" Oct 31 04:29:21.413649 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:21.413627 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51c854e6-11ce-48a1-97a4-fb77b57f5419" path="/var/lib/kubelet/pods/51c854e6-11ce-48a1-97a4-fb77b57f5419/volumes" Oct 31 04:29:21.729736 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:21.729703 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24cd29-67c5899cb7-qfvwr" Oct 31 04:29:21.729938 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:21.729860 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-24cd29-5c5cb6887b-chxzf" Oct 31 04:29:22.300594 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:22.300535 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr"] Oct 31 04:29:22.301378 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:22.301354 5019 cpu_manager.go:414] "containerMap: removing container" podUID="454f6067-ffa0-41cb-9716-d6bacd72abbf" containerName="payload-pre-processing" Oct 31 04:29:22.301378 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:22.301376 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="454f6067-ffa0-41cb-9716-d6bacd72abbf" containerName="payload-pre-processing" Oct 31 04:29:22.301572 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:22.301392 5019 cpu_manager.go:414] "containerMap: removing container" podUID="0165c2c6-76a2-42fc-a3f4-50674c99bc88" containerName="istio-proxy" Oct 31 04:29:22.301572 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:22.301398 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0165c2c6-76a2-42fc-a3f4-50674c99bc88" containerName="istio-proxy" Oct 31 04:29:22.301572 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:22.301410 5019 cpu_manager.go:414] "containerMap: removing container" podUID="51c854e6-11ce-48a1-97a4-fb77b57f5419" containerName="payload-processing" Oct 31 04:29:22.301572 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:22.301415 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51c854e6-11ce-48a1-97a4-fb77b57f5419" containerName="payload-processing" Oct 31 04:29:22.301572 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:22.301424 5019 cpu_manager.go:414] "containerMap: removing container" podUID="0165c2c6-76a2-42fc-a3f4-50674c99bc88" containerName="istio-proxy" Oct 31 04:29:22.301572 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:22.301429 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0165c2c6-76a2-42fc-a3f4-50674c99bc88" containerName="istio-proxy" Oct 31 04:29:22.301572 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:22.301451 5019 cpu_manager.go:414] "containerMap: removing container" podUID="e4a67a18-4432-4411-9c7f-09b505667a44" containerName="payload-processing" Oct 31 04:29:22.301572 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:22.301455 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4a67a18-4432-4411-9c7f-09b505667a44" containerName="payload-processing" Oct 31 04:29:22.301572 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:22.301546 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4a67a18-4432-4411-9c7f-09b505667a44" containerName="payload-processing" Oct 31 04:29:22.301572 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:22.301558 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="454f6067-ffa0-41cb-9716-d6bacd72abbf" containerName="payload-pre-processing" Oct 31 04:29:22.301572 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:22.301566 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="0165c2c6-76a2-42fc-a3f4-50674c99bc88" containerName="istio-proxy" Oct 31 04:29:22.301572 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:22.301573 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="51c854e6-11ce-48a1-97a4-fb77b57f5419" containerName="payload-processing" Oct 31 04:29:22.301572 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:22.301580 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="0165c2c6-76a2-42fc-a3f4-50674c99bc88" containerName="istio-proxy" Oct 31 04:29:22.309350 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:22.308325 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr" Oct 31 04:29:22.346519 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:22.346475 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr"] Oct 31 04:29:22.352593 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:22.352556 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-7cf9ef-openshift-default-dockercfg-7g5ss\"" Oct 31 04:29:22.414801 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:22.414758 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16ca6a86-9186-4d31-a3ac-07f81f318a42-istio-data\") pod \"e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr\" (UID: \"16ca6a86-9186-4d31-a3ac-07f81f318a42\") " pod="openshift-ingress/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr" Oct 31 04:29:22.419406 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:22.417844 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16ca6a86-9186-4d31-a3ac-07f81f318a42-istio-token\") pod \"e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr\" (UID: \"16ca6a86-9186-4d31-a3ac-07f81f318a42\") " pod="openshift-ingress/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr" Oct 31 04:29:22.419406 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:22.419058 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16ca6a86-9186-4d31-a3ac-07f81f318a42-workload-socket\") pod \"e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr\" (UID: \"16ca6a86-9186-4d31-a3ac-07f81f318a42\") " pod="openshift-ingress/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr" Oct 31 04:29:22.419406 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:22.419166 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16ca6a86-9186-4d31-a3ac-07f81f318a42-istio-envoy\") pod \"e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr\" (UID: \"16ca6a86-9186-4d31-a3ac-07f81f318a42\") " pod="openshift-ingress/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr" Oct 31 04:29:22.419406 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:22.419224 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16ca6a86-9186-4d31-a3ac-07f81f318a42-credential-socket\") pod \"e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr\" (UID: \"16ca6a86-9186-4d31-a3ac-07f81f318a42\") " pod="openshift-ingress/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr" Oct 31 04:29:22.419899 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:22.419868 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16ca6a86-9186-4d31-a3ac-07f81f318a42-workload-certs\") pod \"e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr\" (UID: \"16ca6a86-9186-4d31-a3ac-07f81f318a42\") " pod="openshift-ingress/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr" Oct 31 04:29:22.420080 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:22.420065 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16ca6a86-9186-4d31-a3ac-07f81f318a42-istiod-ca-cert\") pod \"e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr\" (UID: \"16ca6a86-9186-4d31-a3ac-07f81f318a42\") " pod="openshift-ingress/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr" Oct 31 04:29:22.420176 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:22.420162 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gz74\" (UniqueName: \"kubernetes.io/projected/16ca6a86-9186-4d31-a3ac-07f81f318a42-kube-api-access-7gz74\") pod \"e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr\" (UID: \"16ca6a86-9186-4d31-a3ac-07f81f318a42\") " pod="openshift-ingress/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr" Oct 31 04:29:22.420283 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:22.420270 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16ca6a86-9186-4d31-a3ac-07f81f318a42-istio-podinfo\") pod \"e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr\" (UID: \"16ca6a86-9186-4d31-a3ac-07f81f318a42\") " pod="openshift-ingress/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr" Oct 31 04:29:22.422937 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:29:22.422896 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:29:22.522293 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:22.522248 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16ca6a86-9186-4d31-a3ac-07f81f318a42-istio-token\") pod \"e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr\" (UID: \"16ca6a86-9186-4d31-a3ac-07f81f318a42\") " pod="openshift-ingress/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr" Oct 31 04:29:22.522615 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:22.522589 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16ca6a86-9186-4d31-a3ac-07f81f318a42-workload-socket\") pod \"e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr\" (UID: \"16ca6a86-9186-4d31-a3ac-07f81f318a42\") " pod="openshift-ingress/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr" Oct 31 04:29:22.522794 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:22.522781 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16ca6a86-9186-4d31-a3ac-07f81f318a42-istio-envoy\") pod \"e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr\" (UID: \"16ca6a86-9186-4d31-a3ac-07f81f318a42\") " pod="openshift-ingress/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr" Oct 31 04:29:22.522928 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:22.522914 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16ca6a86-9186-4d31-a3ac-07f81f318a42-credential-socket\") pod \"e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr\" (UID: \"16ca6a86-9186-4d31-a3ac-07f81f318a42\") " pod="openshift-ingress/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr" Oct 31 04:29:22.523158 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:22.523144 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16ca6a86-9186-4d31-a3ac-07f81f318a42-workload-certs\") pod \"e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr\" (UID: \"16ca6a86-9186-4d31-a3ac-07f81f318a42\") " pod="openshift-ingress/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr" Oct 31 04:29:22.523307 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:22.523295 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16ca6a86-9186-4d31-a3ac-07f81f318a42-istiod-ca-cert\") pod \"e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr\" (UID: \"16ca6a86-9186-4d31-a3ac-07f81f318a42\") " pod="openshift-ingress/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr" Oct 31 04:29:22.523445 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:22.523431 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gz74\" (UniqueName: \"kubernetes.io/projected/16ca6a86-9186-4d31-a3ac-07f81f318a42-kube-api-access-7gz74\") pod \"e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr\" (UID: \"16ca6a86-9186-4d31-a3ac-07f81f318a42\") " pod="openshift-ingress/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr" Oct 31 04:29:22.539409 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:22.539304 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16ca6a86-9186-4d31-a3ac-07f81f318a42-istio-podinfo\") pod \"e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr\" (UID: \"16ca6a86-9186-4d31-a3ac-07f81f318a42\") " pod="openshift-ingress/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr" Oct 31 04:29:22.541007 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:22.540977 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16ca6a86-9186-4d31-a3ac-07f81f318a42-istio-data\") pod \"e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr\" (UID: \"16ca6a86-9186-4d31-a3ac-07f81f318a42\") " pod="openshift-ingress/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr" Oct 31 04:29:22.541413 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:22.534713 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16ca6a86-9186-4d31-a3ac-07f81f318a42-istio-envoy\") pod \"e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr\" (UID: \"16ca6a86-9186-4d31-a3ac-07f81f318a42\") " pod="openshift-ingress/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr" Oct 31 04:29:22.541413 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:22.536324 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16ca6a86-9186-4d31-a3ac-07f81f318a42-istiod-ca-cert\") pod \"e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr\" (UID: \"16ca6a86-9186-4d31-a3ac-07f81f318a42\") " pod="openshift-ingress/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr" Oct 31 04:29:22.541413 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:22.524064 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16ca6a86-9186-4d31-a3ac-07f81f318a42-workload-socket\") pod \"e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr\" (UID: \"16ca6a86-9186-4d31-a3ac-07f81f318a42\") " pod="openshift-ingress/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr" Oct 31 04:29:22.541413 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:22.540876 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16ca6a86-9186-4d31-a3ac-07f81f318a42-credential-socket\") pod \"e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr\" (UID: \"16ca6a86-9186-4d31-a3ac-07f81f318a42\") " pod="openshift-ingress/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr" Oct 31 04:29:22.541689 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:22.535722 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16ca6a86-9186-4d31-a3ac-07f81f318a42-workload-certs\") pod \"e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr\" (UID: \"16ca6a86-9186-4d31-a3ac-07f81f318a42\") " pod="openshift-ingress/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr" Oct 31 04:29:22.542102 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:22.542080 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16ca6a86-9186-4d31-a3ac-07f81f318a42-istio-data\") pod \"e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr\" (UID: \"16ca6a86-9186-4d31-a3ac-07f81f318a42\") " pod="openshift-ingress/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr" Oct 31 04:29:22.544974 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:22.544953 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16ca6a86-9186-4d31-a3ac-07f81f318a42-istio-podinfo\") pod \"e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr\" (UID: \"16ca6a86-9186-4d31-a3ac-07f81f318a42\") " pod="openshift-ingress/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr" Oct 31 04:29:22.548445 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:22.548397 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16ca6a86-9186-4d31-a3ac-07f81f318a42-istio-token\") pod \"e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr\" (UID: \"16ca6a86-9186-4d31-a3ac-07f81f318a42\") " pod="openshift-ingress/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr" Oct 31 04:29:22.567453 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:22.564487 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gz74\" (UniqueName: \"kubernetes.io/projected/16ca6a86-9186-4d31-a3ac-07f81f318a42-kube-api-access-7gz74\") pod \"e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr\" (UID: \"16ca6a86-9186-4d31-a3ac-07f81f318a42\") " pod="openshift-ingress/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr" Oct 31 04:29:22.652637 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:22.652598 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr" Oct 31 04:29:22.995220 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:22.995163 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr"] Oct 31 04:29:23.001925 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:23.001844 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:29:23.002137 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:23.002092 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:29:23.002238 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:23.002185 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:29:23.931893 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:23.931781 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr" event={"ID":"16ca6a86-9186-4d31-a3ac-07f81f318a42","Type":"ContainerStarted","Data":"c7e02b380f069e182c5ee70b3d5189736fe64a69078b3152081daf0225692873"} Oct 31 04:29:23.931893 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:23.931853 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr" event={"ID":"16ca6a86-9186-4d31-a3ac-07f81f318a42","Type":"ContainerStarted","Data":"fc8090995931b6fcd0e12cb3ad0be5c502d3be12d1be93c38bcebfdabfcda4b8"} Oct 31 04:29:23.933657 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:23.933629 5019 generic.go:356] "Generic (PLEG): container finished" podID="618ff6b5-9d72-4ac1-9132-89c0697dfc52" containerID="6de2c0547b9f30b5aa6c9ef198c8c8b2f8765b2d11e5a5c4f95bd38e3a58e368" exitCode=0 Oct 31 04:29:23.933800 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:23.933696 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj" event={"ID":"618ff6b5-9d72-4ac1-9132-89c0697dfc52","Type":"ContainerDied","Data":"6de2c0547b9f30b5aa6c9ef198c8c8b2f8765b2d11e5a5c4f95bd38e3a58e368"} Oct 31 04:29:23.934166 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:23.934151 5019 scope.go:122] "RemoveContainer" containerID="6de2c0547b9f30b5aa6c9ef198c8c8b2f8765b2d11e5a5c4f95bd38e3a58e368" Oct 31 04:29:23.935966 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:23.935934 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:29:23.936056 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:23.936010 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:29:23.936056 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:23.936046 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:29:24.004318 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:24.004257 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr" podStartSLOduration=2.004236412 podStartE2EDuration="2.004236412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:29:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:29:23.990429271 +0000 UTC m=+1518.746767310" watchObservedRunningTime="2026-10-31 04:29:24.004236412 +0000 UTC m=+1518.760574450" Oct 31 04:29:24.653263 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:24.653225 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr" Oct 31 04:29:24.659544 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:24.659503 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr" Oct 31 04:29:24.944937 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:24.944843 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj" event={"ID":"618ff6b5-9d72-4ac1-9132-89c0697dfc52","Type":"ContainerStarted","Data":"b62294855ceca27bee0fc4caca551009560e7ec849df8d151b99df8b28d3cb55"} Oct 31 04:29:24.945494 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:24.945267 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr" Oct 31 04:29:24.946446 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:24.946429 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr" Oct 31 04:29:25.712171 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:25.712140 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj" Oct 31 04:29:25.712395 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:25.712263 5019 patch_prober.go:29] interesting pod/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 31 04:29:25.712395 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:25.712305 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj" podUID="618ff6b5-9d72-4ac1-9132-89c0697dfc52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 31 04:29:25.827911 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:25.827879 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8" Oct 31 04:29:25.924614 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:25.924570 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73613ab5-a053-42d6-b3f4-6e4a381da1ac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73613ab5-a053-42d6-b3f4-6e4a381da1ac-workload-socket\") pod \"73613ab5-a053-42d6-b3f4-6e4a381da1ac\" (UID: \"73613ab5-a053-42d6-b3f4-6e4a381da1ac\") " Oct 31 04:29:25.924852 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:25.924671 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73613ab5-a053-42d6-b3f4-6e4a381da1ac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73613ab5-a053-42d6-b3f4-6e4a381da1ac-credential-socket\") pod \"73613ab5-a053-42d6-b3f4-6e4a381da1ac\" (UID: \"73613ab5-a053-42d6-b3f4-6e4a381da1ac\") " Oct 31 04:29:25.924852 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:25.924765 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73613ab5-a053-42d6-b3f4-6e4a381da1ac-kube-api-access-swvp4\" (UniqueName: \"kubernetes.io/projected/73613ab5-a053-42d6-b3f4-6e4a381da1ac-kube-api-access-swvp4\") pod \"73613ab5-a053-42d6-b3f4-6e4a381da1ac\" (UID: \"73613ab5-a053-42d6-b3f4-6e4a381da1ac\") " Oct 31 04:29:25.924852 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:25.924806 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73613ab5-a053-42d6-b3f4-6e4a381da1ac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73613ab5-a053-42d6-b3f4-6e4a381da1ac-workload-certs\") pod \"73613ab5-a053-42d6-b3f4-6e4a381da1ac\" (UID: \"73613ab5-a053-42d6-b3f4-6e4a381da1ac\") " Oct 31 04:29:25.925021 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:25.924903 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73613ab5-a053-42d6-b3f4-6e4a381da1ac-istio-token\" (UniqueName: \"kubernetes.io/projected/73613ab5-a053-42d6-b3f4-6e4a381da1ac-istio-token\") pod \"73613ab5-a053-42d6-b3f4-6e4a381da1ac\" (UID: \"73613ab5-a053-42d6-b3f4-6e4a381da1ac\") " Oct 31 04:29:25.925075 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:25.925059 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73613ab5-a053-42d6-b3f4-6e4a381da1ac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73613ab5-a053-42d6-b3f4-6e4a381da1ac-istiod-ca-cert\") pod \"73613ab5-a053-42d6-b3f4-6e4a381da1ac\" (UID: \"73613ab5-a053-42d6-b3f4-6e4a381da1ac\") " Oct 31 04:29:25.925126 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:25.925092 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73613ab5-a053-42d6-b3f4-6e4a381da1ac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73613ab5-a053-42d6-b3f4-6e4a381da1ac-istio-data\") pod \"73613ab5-a053-42d6-b3f4-6e4a381da1ac\" (UID: \"73613ab5-a053-42d6-b3f4-6e4a381da1ac\") " Oct 31 04:29:25.925187 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:25.925128 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73613ab5-a053-42d6-b3f4-6e4a381da1ac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73613ab5-a053-42d6-b3f4-6e4a381da1ac-istio-envoy\") pod \"73613ab5-a053-42d6-b3f4-6e4a381da1ac\" (UID: \"73613ab5-a053-42d6-b3f4-6e4a381da1ac\") " Oct 31 04:29:25.925187 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:25.925166 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/73613ab5-a053-42d6-b3f4-6e4a381da1ac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73613ab5-a053-42d6-b3f4-6e4a381da1ac-istio-podinfo\") pod \"73613ab5-a053-42d6-b3f4-6e4a381da1ac\" (UID: \"73613ab5-a053-42d6-b3f4-6e4a381da1ac\") " Oct 31 04:29:25.926299 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:25.925963 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73613ab5-a053-42d6-b3f4-6e4a381da1ac-credential-socket" pod "73613ab5-a053-42d6-b3f4-6e4a381da1ac" (UID: "73613ab5-a053-42d6-b3f4-6e4a381da1ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:29:25.926299 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:25.926107 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73613ab5-a053-42d6-b3f4-6e4a381da1ac-workload-certs" pod "73613ab5-a053-42d6-b3f4-6e4a381da1ac" (UID: "73613ab5-a053-42d6-b3f4-6e4a381da1ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:29:25.926299 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:25.926182 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73613ab5-a053-42d6-b3f4-6e4a381da1ac-workload-socket" pod "73613ab5-a053-42d6-b3f4-6e4a381da1ac" (UID: "73613ab5-a053-42d6-b3f4-6e4a381da1ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:29:25.927037 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:25.926782 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73613ab5-a053-42d6-b3f4-6e4a381da1ac-istio-data" pod "73613ab5-a053-42d6-b3f4-6e4a381da1ac" (UID: "73613ab5-a053-42d6-b3f4-6e4a381da1ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:29:25.927496 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:25.927476 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73613ab5-a053-42d6-b3f4-6e4a381da1ac-istiod-ca-cert" pod "73613ab5-a053-42d6-b3f4-6e4a381da1ac" (UID: "73613ab5-a053-42d6-b3f4-6e4a381da1ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:29:25.928641 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:25.928579 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73613ab5-a053-42d6-b3f4-6e4a381da1ac-istio-podinfo" pod "73613ab5-a053-42d6-b3f4-6e4a381da1ac" (UID: "73613ab5-a053-42d6-b3f4-6e4a381da1ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:29:25.930507 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:25.930384 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73613ab5-a053-42d6-b3f4-6e4a381da1ac-kube-api-access-swvp4" pod "73613ab5-a053-42d6-b3f4-6e4a381da1ac" (UID: "73613ab5-a053-42d6-b3f4-6e4a381da1ac"). InnerVolumeSpecName "kube-api-access-swvp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:29:25.930702 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:25.930677 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73613ab5-a053-42d6-b3f4-6e4a381da1ac-istio-envoy" pod "73613ab5-a053-42d6-b3f4-6e4a381da1ac" (UID: "73613ab5-a053-42d6-b3f4-6e4a381da1ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:29:25.930958 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:25.930934 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73613ab5-a053-42d6-b3f4-6e4a381da1ac-istio-token" pod "73613ab5-a053-42d6-b3f4-6e4a381da1ac" (UID: "73613ab5-a053-42d6-b3f4-6e4a381da1ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:29:25.959504 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:25.959464 5019 generic.go:356] "Generic (PLEG): container finished" podID="73613ab5-a053-42d6-b3f4-6e4a381da1ac" containerID="802b87ce7cf6172d586899fd6236c109f1a537f16652ae280694b6af09afe643" exitCode=0 Oct 31 04:29:25.960497 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:25.960475 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8" Oct 31 04:29:25.960857 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:25.960665 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8" event={"ID":"73613ab5-a053-42d6-b3f4-6e4a381da1ac","Type":"ContainerDied","Data":"802b87ce7cf6172d586899fd6236c109f1a537f16652ae280694b6af09afe643"} Oct 31 04:29:25.960857 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:25.960722 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8" event={"ID":"73613ab5-a053-42d6-b3f4-6e4a381da1ac","Type":"ContainerDied","Data":"3dedd1ea18ac24bd3b7048d1d9defe960de3ca675744efe0f66e42f24634dfd3"} Oct 31 04:29:25.960857 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:25.960748 5019 scope.go:122] "RemoveContainer" containerID="802b87ce7cf6172d586899fd6236c109f1a537f16652ae280694b6af09afe643" Oct 31 04:29:26.005153 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:26.005133 5019 scope.go:122] "RemoveContainer" containerID="0d1dc38f376d2b8e894ce19f102caa980e174d6b88767afec2d6703ff7913cb5" Oct 31 04:29:26.017170 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:26.017145 5019 scope.go:122] "RemoveContainer" containerID="802b87ce7cf6172d586899fd6236c109f1a537f16652ae280694b6af09afe643" Oct 31 04:29:26.017565 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:29:26.017543 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"802b87ce7cf6172d586899fd6236c109f1a537f16652ae280694b6af09afe643\": container with ID starting with 802b87ce7cf6172d586899fd6236c109f1a537f16652ae280694b6af09afe643 not found: ID does not exist" containerID="802b87ce7cf6172d586899fd6236c109f1a537f16652ae280694b6af09afe643" Oct 31 04:29:26.017688 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:26.017576 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"802b87ce7cf6172d586899fd6236c109f1a537f16652ae280694b6af09afe643"} err="failed to get container status \"802b87ce7cf6172d586899fd6236c109f1a537f16652ae280694b6af09afe643\": rpc error: code = NotFound desc = could not find container \"802b87ce7cf6172d586899fd6236c109f1a537f16652ae280694b6af09afe643\": container with ID starting with 802b87ce7cf6172d586899fd6236c109f1a537f16652ae280694b6af09afe643 not found: ID does not exist" Oct 31 04:29:26.017688 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:26.017600 5019 scope.go:122] "RemoveContainer" containerID="0d1dc38f376d2b8e894ce19f102caa980e174d6b88767afec2d6703ff7913cb5" Oct 31 04:29:26.017871 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:29:26.017849 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d1dc38f376d2b8e894ce19f102caa980e174d6b88767afec2d6703ff7913cb5\": container with ID starting with 0d1dc38f376d2b8e894ce19f102caa980e174d6b88767afec2d6703ff7913cb5 not found: ID does not exist" containerID="0d1dc38f376d2b8e894ce19f102caa980e174d6b88767afec2d6703ff7913cb5" Oct 31 04:29:26.017951 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:26.017879 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d1dc38f376d2b8e894ce19f102caa980e174d6b88767afec2d6703ff7913cb5"} err="failed to get container status \"0d1dc38f376d2b8e894ce19f102caa980e174d6b88767afec2d6703ff7913cb5\": rpc error: code = NotFound desc = could not find container \"0d1dc38f376d2b8e894ce19f102caa980e174d6b88767afec2d6703ff7913cb5\": container with ID starting with 0d1dc38f376d2b8e894ce19f102caa980e174d6b88767afec2d6703ff7913cb5 not found: ID does not exist" Oct 31 04:29:26.027557 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:26.027501 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swvp4\" (UniqueName: \"kubernetes.io/projected/73613ab5-a053-42d6-b3f4-6e4a381da1ac-kube-api-access-swvp4\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:29:26.027759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:26.027558 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73613ab5-a053-42d6-b3f4-6e4a381da1ac-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:29:26.027759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:26.027586 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73613ab5-a053-42d6-b3f4-6e4a381da1ac-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:29:26.027759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:26.027598 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73613ab5-a053-42d6-b3f4-6e4a381da1ac-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:29:26.027759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:26.027610 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73613ab5-a053-42d6-b3f4-6e4a381da1ac-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:29:26.027759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:26.027622 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73613ab5-a053-42d6-b3f4-6e4a381da1ac-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:29:26.027759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:26.027634 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73613ab5-a053-42d6-b3f4-6e4a381da1ac-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:29:26.027759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:26.027659 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73613ab5-a053-42d6-b3f4-6e4a381da1ac-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:29:26.027759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:26.027674 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73613ab5-a053-42d6-b3f4-6e4a381da1ac-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:29:26.047428 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:26.047383 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8"] Oct 31 04:29:26.050982 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:26.050936 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-b44631-openshift-default-5bbbcc5f7-896p8"] Oct 31 04:29:26.694944 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:26.694897 5019 patch_prober.go:29] interesting pod/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 31 04:29:26.695206 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:26.694993 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj" podUID="618ff6b5-9d72-4ac1-9132-89c0697dfc52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 31 04:29:27.408674 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:27.408638 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73613ab5-a053-42d6-b3f4-6e4a381da1ac" path="/var/lib/kubelet/pods/73613ab5-a053-42d6-b3f4-6e4a381da1ac/volumes" Oct 31 04:29:27.694419 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:27.693828 5019 patch_prober.go:29] interesting pod/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 31 04:29:27.694419 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:27.693899 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj" podUID="618ff6b5-9d72-4ac1-9132-89c0697dfc52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 31 04:29:28.290357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:28.289885 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-24cd29-5c5cb6887b-chxzf_530d712c-fe37-4d92-ae54-be36babd2335/payload-processing/0.log" Oct 31 04:29:28.391018 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:28.390895 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7cf9ef-699d468f47-5cpz6"] Oct 31 04:29:28.400428 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:28.391944 5019 cpu_manager.go:414] "containerMap: removing container" podUID="73613ab5-a053-42d6-b3f4-6e4a381da1ac" containerName="istio-proxy" Oct 31 04:29:28.400428 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:28.391973 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73613ab5-a053-42d6-b3f4-6e4a381da1ac" containerName="istio-proxy" Oct 31 04:29:28.400428 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:28.392034 5019 cpu_manager.go:414] "containerMap: removing container" podUID="73613ab5-a053-42d6-b3f4-6e4a381da1ac" containerName="istio-proxy" Oct 31 04:29:28.400428 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:28.392041 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73613ab5-a053-42d6-b3f4-6e4a381da1ac" containerName="istio-proxy" Oct 31 04:29:28.400428 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:28.392192 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="73613ab5-a053-42d6-b3f4-6e4a381da1ac" containerName="istio-proxy" Oct 31 04:29:28.400428 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:28.392215 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="73613ab5-a053-42d6-b3f4-6e4a381da1ac" containerName="istio-proxy" Oct 31 04:29:28.403281 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:28.403247 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7cf9ef-699d468f47-5cpz6" Oct 31 04:29:28.410763 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:28.410725 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7cf9ef-699d468f47-5cpz6"] Oct 31 04:29:28.434556 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:28.431702 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-7cf9ef-dockercfg-ll7kw\"" Oct 31 04:29:28.434556 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:28.431877 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-7cf9ef\"" Oct 31 04:29:28.474369 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:28.470077 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13fbd89f-fe6e-4b64-876e-ac8d3db07113-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-7cf9ef-699d468f47-5cpz6\" (UID: \"13fbd89f-fe6e-4b64-876e-ac8d3db07113\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7cf9ef-699d468f47-5cpz6" Oct 31 04:29:28.474369 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:28.470267 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txpjb\" (UniqueName: \"kubernetes.io/projected/13fbd89f-fe6e-4b64-876e-ac8d3db07113-kube-api-access-txpjb\") pod \"payload-pre-processing-e2e-praxis-b-7cf9ef-699d468f47-5cpz6\" (UID: \"13fbd89f-fe6e-4b64-876e-ac8d3db07113\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7cf9ef-699d468f47-5cpz6" Oct 31 04:29:28.571420 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:28.571295 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txpjb\" (UniqueName: \"kubernetes.io/projected/13fbd89f-fe6e-4b64-876e-ac8d3db07113-kube-api-access-txpjb\") pod \"payload-pre-processing-e2e-praxis-b-7cf9ef-699d468f47-5cpz6\" (UID: \"13fbd89f-fe6e-4b64-876e-ac8d3db07113\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7cf9ef-699d468f47-5cpz6" Oct 31 04:29:28.585789 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:28.585736 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13fbd89f-fe6e-4b64-876e-ac8d3db07113-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-7cf9ef-699d468f47-5cpz6\" (UID: \"13fbd89f-fe6e-4b64-876e-ac8d3db07113\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7cf9ef-699d468f47-5cpz6" Oct 31 04:29:28.606645 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:28.606606 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13fbd89f-fe6e-4b64-876e-ac8d3db07113-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-7cf9ef-699d468f47-5cpz6\" (UID: \"13fbd89f-fe6e-4b64-876e-ac8d3db07113\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7cf9ef-699d468f47-5cpz6" Oct 31 04:29:28.625537 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:28.625466 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7cf9ef-7b7cbd9689-srt2f"] Oct 31 04:29:28.646415 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:28.640436 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-7cf9ef-7b7cbd9689-srt2f" Oct 31 04:29:28.646415 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:28.640597 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7cf9ef-7b7cbd9689-srt2f"] Oct 31 04:29:28.646415 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:28.644615 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txpjb\" (UniqueName: \"kubernetes.io/projected/13fbd89f-fe6e-4b64-876e-ac8d3db07113-kube-api-access-txpjb\") pod \"payload-pre-processing-e2e-praxis-b-7cf9ef-699d468f47-5cpz6\" (UID: \"13fbd89f-fe6e-4b64-876e-ac8d3db07113\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7cf9ef-699d468f47-5cpz6" Oct 31 04:29:28.719750 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:28.714191 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6434973f-f51b-42df-8f28-97d9f310fda0-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-7cf9ef-7b7cbd9689-srt2f\" (UID: \"6434973f-f51b-42df-8f28-97d9f310fda0\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7cf9ef-7b7cbd9689-srt2f" Oct 31 04:29:28.720125 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:28.720087 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24nlz\" (UniqueName: \"kubernetes.io/projected/6434973f-f51b-42df-8f28-97d9f310fda0-kube-api-access-24nlz\") pod \"payload-processing-e2e-praxis-b-7cf9ef-7b7cbd9689-srt2f\" (UID: \"6434973f-f51b-42df-8f28-97d9f310fda0\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7cf9ef-7b7cbd9689-srt2f" Oct 31 04:29:28.720264 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:28.714507 5019 patch_prober.go:29] interesting pod/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 31 04:29:28.720684 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:28.720631 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj" podUID="618ff6b5-9d72-4ac1-9132-89c0697dfc52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 31 04:29:28.749254 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:28.747350 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7cf9ef-699d468f47-5cpz6" Oct 31 04:29:28.823402 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:28.823243 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6434973f-f51b-42df-8f28-97d9f310fda0-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-7cf9ef-7b7cbd9689-srt2f\" (UID: \"6434973f-f51b-42df-8f28-97d9f310fda0\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7cf9ef-7b7cbd9689-srt2f" Oct 31 04:29:28.823402 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:28.823317 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24nlz\" (UniqueName: \"kubernetes.io/projected/6434973f-f51b-42df-8f28-97d9f310fda0-kube-api-access-24nlz\") pod \"payload-processing-e2e-praxis-b-7cf9ef-7b7cbd9689-srt2f\" (UID: \"6434973f-f51b-42df-8f28-97d9f310fda0\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7cf9ef-7b7cbd9689-srt2f" Oct 31 04:29:28.824447 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:28.824422 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6434973f-f51b-42df-8f28-97d9f310fda0-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-7cf9ef-7b7cbd9689-srt2f\" (UID: \"6434973f-f51b-42df-8f28-97d9f310fda0\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7cf9ef-7b7cbd9689-srt2f" Oct 31 04:29:28.847243 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:28.847208 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24nlz\" (UniqueName: \"kubernetes.io/projected/6434973f-f51b-42df-8f28-97d9f310fda0-kube-api-access-24nlz\") pod \"payload-processing-e2e-praxis-b-7cf9ef-7b7cbd9689-srt2f\" (UID: \"6434973f-f51b-42df-8f28-97d9f310fda0\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7cf9ef-7b7cbd9689-srt2f" Oct 31 04:29:29.021515 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:29.021468 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-7cf9ef-7b7cbd9689-srt2f" Oct 31 04:29:29.189596 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:29.189475 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7cf9ef-699d468f47-5cpz6"] Oct 31 04:29:29.367627 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:29.367590 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7cf9ef-7b7cbd9689-srt2f"] Oct 31 04:29:29.700020 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:29.699872 5019 patch_prober.go:29] interesting pod/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 31 04:29:29.700020 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:29.699955 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj" podUID="618ff6b5-9d72-4ac1-9132-89c0697dfc52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 31 04:29:30.022196 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:30.022147 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7cf9ef-699d468f47-5cpz6" event={"ID":"13fbd89f-fe6e-4b64-876e-ac8d3db07113","Type":"ContainerStarted","Data":"8d81ecb730e561317698ca0cd7f826ef33aa1f234f4c747c72983f49f6d6729b"} Oct 31 04:29:30.022458 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:30.022266 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7cf9ef-699d468f47-5cpz6" Oct 31 04:29:30.022458 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:30.022296 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7cf9ef-699d468f47-5cpz6" event={"ID":"13fbd89f-fe6e-4b64-876e-ac8d3db07113","Type":"ContainerStarted","Data":"6ba1e86e92e355cd00434ce41607cf2c2bf71ea8e9ae461fd0b42e26ea7391bc"} Oct 31 04:29:30.024440 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:30.024396 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-7cf9ef-7b7cbd9689-srt2f" event={"ID":"6434973f-f51b-42df-8f28-97d9f310fda0","Type":"ContainerStarted","Data":"af339036524abd3d71adbc9544a8fefb5cb7b0acefee2782b217699f1a01c0c6"} Oct 31 04:29:30.024632 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:30.024455 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-7cf9ef-7b7cbd9689-srt2f" event={"ID":"6434973f-f51b-42df-8f28-97d9f310fda0","Type":"ContainerStarted","Data":"cbde5e14c00b482f0a72c3eff9502fd34412c73afc14ca2d6bbc82e4af009943"} Oct 31 04:29:30.024632 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:30.024479 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-7cf9ef-7b7cbd9689-srt2f" Oct 31 04:29:30.068664 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:30.068583 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7cf9ef-699d468f47-5cpz6" podStartSLOduration=2.068560208 podStartE2EDuration="2.068560208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:29:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:29:30.061803734 +0000 UTC m=+1524.818141796" watchObservedRunningTime="2026-10-31 04:29:30.068560208 +0000 UTC m=+1524.824898248" Oct 31 04:29:30.095468 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:30.095422 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b11c770-bfxfp"] Oct 31 04:29:30.100700 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:30.100668 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b11c770-bfxfp" Oct 31 04:29:30.106212 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:30.106151 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-7cf9ef-7b7cbd9689-srt2f" podStartSLOduration=2.106128372 podStartE2EDuration="2.106128372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:29:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:29:30.101488844 +0000 UTC m=+1524.857826884" watchObservedRunningTime="2026-10-31 04:29:30.106128372 +0000 UTC m=+1524.862466412" Oct 31 04:29:30.116920 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:30.115442 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:29:30.116920 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:30.115490 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:29:30.116920 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:30.115593 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:29:30.116920 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:30.115596 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:29:30.120992 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:30.120956 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b11c770-bfxfp"] Oct 31 04:29:30.160576 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:30.160519 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/692297c2-7276-498d-9b3b-5941673eacd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b11c770-bfxfp\" (UID: \"692297c2-7276-498d-9b3b-5941673eacd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b11c770-bfxfp" Oct 31 04:29:30.160795 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:30.160667 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/692297c2-7276-498d-9b3b-5941673eacd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b11c770-bfxfp\" (UID: \"692297c2-7276-498d-9b3b-5941673eacd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b11c770-bfxfp" Oct 31 04:29:30.160880 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:30.160851 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dcdr\" (UniqueName: \"kubernetes.io/projected/692297c2-7276-498d-9b3b-5941673eacd1-kube-api-access-7dcdr\") pod \"maas-api-revoke-sub-2b11c770-bfxfp\" (UID: \"692297c2-7276-498d-9b3b-5941673eacd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b11c770-bfxfp" Oct 31 04:29:30.262726 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:30.262687 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dcdr\" (UniqueName: \"kubernetes.io/projected/692297c2-7276-498d-9b3b-5941673eacd1-kube-api-access-7dcdr\") pod \"maas-api-revoke-sub-2b11c770-bfxfp\" (UID: \"692297c2-7276-498d-9b3b-5941673eacd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b11c770-bfxfp" Oct 31 04:29:30.262892 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:30.262826 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/692297c2-7276-498d-9b3b-5941673eacd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b11c770-bfxfp\" (UID: \"692297c2-7276-498d-9b3b-5941673eacd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b11c770-bfxfp" Oct 31 04:29:30.262892 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:30.262875 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/692297c2-7276-498d-9b3b-5941673eacd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b11c770-bfxfp\" (UID: \"692297c2-7276-498d-9b3b-5941673eacd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b11c770-bfxfp" Oct 31 04:29:30.263553 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:30.263522 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/692297c2-7276-498d-9b3b-5941673eacd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b11c770-bfxfp\" (UID: \"692297c2-7276-498d-9b3b-5941673eacd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b11c770-bfxfp" Oct 31 04:29:30.265401 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:30.265381 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/692297c2-7276-498d-9b3b-5941673eacd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b11c770-bfxfp\" (UID: \"692297c2-7276-498d-9b3b-5941673eacd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b11c770-bfxfp" Oct 31 04:29:30.286798 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:30.286719 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dcdr\" (UniqueName: \"kubernetes.io/projected/692297c2-7276-498d-9b3b-5941673eacd1-kube-api-access-7dcdr\") pod \"maas-api-revoke-sub-2b11c770-bfxfp\" (UID: \"692297c2-7276-498d-9b3b-5941673eacd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b11c770-bfxfp" Oct 31 04:29:30.421291 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:30.421252 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b11c770-bfxfp" Oct 31 04:29:30.694847 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:30.694751 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj" Oct 31 04:29:30.695289 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:30.695268 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj" Oct 31 04:29:30.696100 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:30.696080 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj" Oct 31 04:29:30.725431 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:30.725401 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b11c770-bfxfp"] Oct 31 04:29:30.726622 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:29:30.726584 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod692297c2_7276_498d_9b3b_5941673eacd1.slice/crio-10f770d5656a31795af8f85eacd91cc04c763593c673b95c44aad8c3f4394aeb WatchSource:0}: Error finding container 10f770d5656a31795af8f85eacd91cc04c763593c673b95c44aad8c3f4394aeb: Status 404 returned error can't find the container with id 10f770d5656a31795af8f85eacd91cc04c763593c673b95c44aad8c3f4394aeb Oct 31 04:29:31.033842 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:31.033782 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b11c770-bfxfp" event={"ID":"692297c2-7276-498d-9b3b-5941673eacd1","Type":"ContainerStarted","Data":"10f770d5656a31795af8f85eacd91cc04c763593c673b95c44aad8c3f4394aeb"} Oct 31 04:29:32.042983 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:32.042945 5019 generic.go:356] "Generic (PLEG): container finished" podID="692297c2-7276-498d-9b3b-5941673eacd1" containerID="24efb5a8f7884be7d98fafa7151321897a5538ccf8c8694f68b698444b60553b" exitCode=0 Oct 31 04:29:32.043425 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:32.043033 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b11c770-bfxfp" event={"ID":"692297c2-7276-498d-9b3b-5941673eacd1","Type":"ContainerDied","Data":"24efb5a8f7884be7d98fafa7151321897a5538ccf8c8694f68b698444b60553b"} Oct 31 04:29:33.257306 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:33.257268 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b11c770-bfxfp" Oct 31 04:29:33.317524 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:33.317431 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/692297c2-7276-498d-9b3b-5941673eacd1-kube-api-access-7dcdr\" (UniqueName: \"kubernetes.io/projected/692297c2-7276-498d-9b3b-5941673eacd1-kube-api-access-7dcdr\") pod \"692297c2-7276-498d-9b3b-5941673eacd1\" (UID: \"692297c2-7276-498d-9b3b-5941673eacd1\") " Oct 31 04:29:33.317701 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:33.317538 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/692297c2-7276-498d-9b3b-5941673eacd1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/692297c2-7276-498d-9b3b-5941673eacd1-maas-api-openshift-ca\") pod \"692297c2-7276-498d-9b3b-5941673eacd1\" (UID: \"692297c2-7276-498d-9b3b-5941673eacd1\") " Oct 31 04:29:33.317701 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:33.317670 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/692297c2-7276-498d-9b3b-5941673eacd1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/692297c2-7276-498d-9b3b-5941673eacd1-maas-api-xks-ca\") pod \"692297c2-7276-498d-9b3b-5941673eacd1\" (UID: \"692297c2-7276-498d-9b3b-5941673eacd1\") " Oct 31 04:29:33.318411 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:33.318372 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/692297c2-7276-498d-9b3b-5941673eacd1-maas-api-openshift-ca" pod "692297c2-7276-498d-9b3b-5941673eacd1" (UID: "692297c2-7276-498d-9b3b-5941673eacd1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:29:33.319926 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:33.319894 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/692297c2-7276-498d-9b3b-5941673eacd1-kube-api-access-7dcdr" pod "692297c2-7276-498d-9b3b-5941673eacd1" (UID: "692297c2-7276-498d-9b3b-5941673eacd1"). InnerVolumeSpecName "kube-api-access-7dcdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:29:33.320439 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:33.320411 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/692297c2-7276-498d-9b3b-5941673eacd1-maas-api-xks-ca" pod "692297c2-7276-498d-9b3b-5941673eacd1" (UID: "692297c2-7276-498d-9b3b-5941673eacd1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:29:33.419528 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:33.419490 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/692297c2-7276-498d-9b3b-5941673eacd1-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:29:33.419528 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:33.419529 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/692297c2-7276-498d-9b3b-5941673eacd1-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:29:33.419722 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:33.419551 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dcdr\" (UniqueName: \"kubernetes.io/projected/692297c2-7276-498d-9b3b-5941673eacd1-kube-api-access-7dcdr\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:29:34.062030 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:34.061999 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b11c770-bfxfp" Oct 31 04:29:34.062220 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:34.061993 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b11c770-bfxfp" event={"ID":"692297c2-7276-498d-9b3b-5941673eacd1","Type":"ContainerDied","Data":"10f770d5656a31795af8f85eacd91cc04c763593c673b95c44aad8c3f4394aeb"} Oct 31 04:29:34.062220 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:34.062099 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10f770d5656a31795af8f85eacd91cc04c763593c673b95c44aad8c3f4394aeb" Oct 31 04:29:35.825379 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:35.825327 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ff43e3-944445948-ph2kl"] Oct 31 04:29:35.825799 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:35.825565 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff43e3-944445948-ph2kl" podUID="79838bec-a733-4da9-9f5b-d476f8de3952" containerName="payload-processing" containerID="cri-o://a628d3f0370ce429848f03771fa47fea4509d04c289664a1739302b165e8ff31" gracePeriod=30 Oct 31 04:29:35.856861 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:35.856827 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ff43e3-67d8bf5499-ttklw"] Oct 31 04:29:35.857088 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:35.857059 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff43e3-67d8bf5499-ttklw" podUID="28fc13fa-c89f-459e-9ffc-19a9f85bfec9" containerName="payload-pre-processing" containerID="cri-o://fb6588baf0950aecb87ceb0c82a23946da67ed7df45a147d2ab3d90bf069b1b2" gracePeriod=30 Oct 31 04:29:36.091112 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:36.091012 5019 generic.go:356] "Generic (PLEG): container finished" podID="28fc13fa-c89f-459e-9ffc-19a9f85bfec9" containerID="fb6588baf0950aecb87ceb0c82a23946da67ed7df45a147d2ab3d90bf069b1b2" exitCode=0 Oct 31 04:29:36.091112 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:36.091110 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff43e3-67d8bf5499-ttklw" event={"ID":"28fc13fa-c89f-459e-9ffc-19a9f85bfec9","Type":"ContainerDied","Data":"fb6588baf0950aecb87ceb0c82a23946da67ed7df45a147d2ab3d90bf069b1b2"} Oct 31 04:29:36.093012 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:36.092930 5019 generic.go:356] "Generic (PLEG): container finished" podID="79838bec-a733-4da9-9f5b-d476f8de3952" containerID="a628d3f0370ce429848f03771fa47fea4509d04c289664a1739302b165e8ff31" exitCode=0 Oct 31 04:29:36.093673 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:36.093171 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff43e3-944445948-ph2kl" event={"ID":"79838bec-a733-4da9-9f5b-d476f8de3952","Type":"ContainerDied","Data":"a628d3f0370ce429848f03771fa47fea4509d04c289664a1739302b165e8ff31"} Oct 31 04:29:36.304859 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:36.304582 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff43e3-944445948-ph2kl" Oct 31 04:29:36.337703 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:36.337673 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff43e3-67d8bf5499-ttklw" Oct 31 04:29:36.419403 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:36.419289 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79838bec-a733-4da9-9f5b-d476f8de3952-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79838bec-a733-4da9-9f5b-d476f8de3952-plugins-config-volume\") pod \"79838bec-a733-4da9-9f5b-d476f8de3952\" (UID: \"79838bec-a733-4da9-9f5b-d476f8de3952\") " Oct 31 04:29:36.419700 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:36.419676 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79838bec-a733-4da9-9f5b-d476f8de3952-kube-api-access-8nb7s\" (UniqueName: \"kubernetes.io/projected/79838bec-a733-4da9-9f5b-d476f8de3952-kube-api-access-8nb7s\") pod \"79838bec-a733-4da9-9f5b-d476f8de3952\" (UID: \"79838bec-a733-4da9-9f5b-d476f8de3952\") " Oct 31 04:29:36.419797 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:36.419731 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79838bec-a733-4da9-9f5b-d476f8de3952-plugins-config-volume" pod "79838bec-a733-4da9-9f5b-d476f8de3952" (UID: "79838bec-a733-4da9-9f5b-d476f8de3952"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:29:36.420111 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:36.420092 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79838bec-a733-4da9-9f5b-d476f8de3952-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:29:36.421927 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:36.421893 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79838bec-a733-4da9-9f5b-d476f8de3952-kube-api-access-8nb7s" pod "79838bec-a733-4da9-9f5b-d476f8de3952" (UID: "79838bec-a733-4da9-9f5b-d476f8de3952"). InnerVolumeSpecName "kube-api-access-8nb7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:29:36.521595 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:36.521559 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28fc13fa-c89f-459e-9ffc-19a9f85bfec9-kube-api-access-5v8vc\" (UniqueName: \"kubernetes.io/projected/28fc13fa-c89f-459e-9ffc-19a9f85bfec9-kube-api-access-5v8vc\") pod \"28fc13fa-c89f-459e-9ffc-19a9f85bfec9\" (UID: \"28fc13fa-c89f-459e-9ffc-19a9f85bfec9\") " Oct 31 04:29:36.521792 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:36.521618 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28fc13fa-c89f-459e-9ffc-19a9f85bfec9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28fc13fa-c89f-459e-9ffc-19a9f85bfec9-plugins-config-volume\") pod \"28fc13fa-c89f-459e-9ffc-19a9f85bfec9\" (UID: \"28fc13fa-c89f-459e-9ffc-19a9f85bfec9\") " Oct 31 04:29:36.522052 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:36.522029 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28fc13fa-c89f-459e-9ffc-19a9f85bfec9-plugins-config-volume" pod "28fc13fa-c89f-459e-9ffc-19a9f85bfec9" (UID: "28fc13fa-c89f-459e-9ffc-19a9f85bfec9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:29:36.522453 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:36.522430 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28fc13fa-c89f-459e-9ffc-19a9f85bfec9-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:29:36.522532 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:36.522459 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nb7s\" (UniqueName: \"kubernetes.io/projected/79838bec-a733-4da9-9f5b-d476f8de3952-kube-api-access-8nb7s\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:29:36.524055 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:36.524026 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28fc13fa-c89f-459e-9ffc-19a9f85bfec9-kube-api-access-5v8vc" pod "28fc13fa-c89f-459e-9ffc-19a9f85bfec9" (UID: "28fc13fa-c89f-459e-9ffc-19a9f85bfec9"). InnerVolumeSpecName "kube-api-access-5v8vc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:29:36.624605 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:36.624558 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v8vc\" (UniqueName: \"kubernetes.io/projected/28fc13fa-c89f-459e-9ffc-19a9f85bfec9-kube-api-access-5v8vc\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:29:37.105292 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:37.105188 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff43e3-67d8bf5499-ttklw" event={"ID":"28fc13fa-c89f-459e-9ffc-19a9f85bfec9","Type":"ContainerDied","Data":"17a6f66adc58e43b5464329f2e2c5099da1ec33f4c96feb28d2b0c0083884d7b"} Oct 31 04:29:37.105810 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:37.105208 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff43e3-67d8bf5499-ttklw" Oct 31 04:29:37.105810 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:37.105311 5019 scope.go:122] "RemoveContainer" containerID="fb6588baf0950aecb87ceb0c82a23946da67ed7df45a147d2ab3d90bf069b1b2" Oct 31 04:29:37.108385 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:37.108360 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff43e3-944445948-ph2kl" Oct 31 04:29:37.108949 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:37.108325 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff43e3-944445948-ph2kl" event={"ID":"79838bec-a733-4da9-9f5b-d476f8de3952","Type":"ContainerDied","Data":"93fa06a3a81bb89d9a087c5a79e6951ac0e1342e4834ca940af63d341b84d348"} Oct 31 04:29:37.121889 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:37.121817 5019 scope.go:122] "RemoveContainer" containerID="a628d3f0370ce429848f03771fa47fea4509d04c289664a1739302b165e8ff31" Oct 31 04:29:37.159827 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:37.159791 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ff43e3-67d8bf5499-ttklw"] Oct 31 04:29:37.172004 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:37.171963 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ff43e3-67d8bf5499-ttklw"] Oct 31 04:29:37.218018 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:37.217324 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ff43e3-944445948-ph2kl"] Oct 31 04:29:37.221511 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:37.221485 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ff43e3-944445948-ph2kl"] Oct 31 04:29:37.411205 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:37.411122 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28fc13fa-c89f-459e-9ffc-19a9f85bfec9" path="/var/lib/kubelet/pods/28fc13fa-c89f-459e-9ffc-19a9f85bfec9/volumes" Oct 31 04:29:37.411640 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:37.411625 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79838bec-a733-4da9-9f5b-d476f8de3952" path="/var/lib/kubelet/pods/79838bec-a733-4da9-9f5b-d476f8de3952/volumes" Oct 31 04:29:39.735456 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:39.735417 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8"] Oct 31 04:29:39.736099 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:39.735688 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8" podUID="92398da3-abea-427b-ad63-1d9fbd4385e5" containerName="istio-proxy" containerID="cri-o://0ebf790fd3da68e5d821df8b7cc619b150466ace914bbfa5ee8dbb29b8d79c8c" gracePeriod=30 Oct 31 04:29:41.036941 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:41.036911 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-7cf9ef-7b7cbd9689-srt2f" Oct 31 04:29:41.037387 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:41.036977 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7cf9ef-699d468f47-5cpz6" Oct 31 04:29:42.277282 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:42.277240 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd"] Oct 31 04:29:42.278118 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:42.278059 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd" podUID="a473bd3b-e50d-4a3d-ad54-fd49ee49f144" containerName="istio-proxy" containerID="cri-o://e8878179de9af734b48958325224021be55c4d6c045b1c39fa15d2c27a9da8b3" gracePeriod=30 Oct 31 04:29:44.118534 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:44.118493 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a2e7e0a-zf829"] Oct 31 04:29:44.119230 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:44.119211 5019 cpu_manager.go:414] "containerMap: removing container" podUID="28fc13fa-c89f-459e-9ffc-19a9f85bfec9" containerName="payload-pre-processing" Oct 31 04:29:44.119285 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:44.119233 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28fc13fa-c89f-459e-9ffc-19a9f85bfec9" containerName="payload-pre-processing" Oct 31 04:29:44.119285 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:44.119249 5019 cpu_manager.go:414] "containerMap: removing container" podUID="79838bec-a733-4da9-9f5b-d476f8de3952" containerName="payload-processing" Oct 31 04:29:44.119285 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:44.119254 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79838bec-a733-4da9-9f5b-d476f8de3952" containerName="payload-processing" Oct 31 04:29:44.119285 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:44.119274 5019 cpu_manager.go:414] "containerMap: removing container" podUID="692297c2-7276-498d-9b3b-5941673eacd1" containerName="revoke-keys" Oct 31 04:29:44.119285 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:44.119279 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="692297c2-7276-498d-9b3b-5941673eacd1" containerName="revoke-keys" Oct 31 04:29:44.119499 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:44.119407 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="692297c2-7276-498d-9b3b-5941673eacd1" containerName="revoke-keys" Oct 31 04:29:44.119499 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:44.119424 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="28fc13fa-c89f-459e-9ffc-19a9f85bfec9" containerName="payload-pre-processing" Oct 31 04:29:44.119499 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:44.119432 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="79838bec-a733-4da9-9f5b-d476f8de3952" containerName="payload-processing" Oct 31 04:29:44.124665 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:44.124637 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a2e7e0a-zf829" Oct 31 04:29:44.142605 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:44.142570 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a2e7e0a-zf829"] Oct 31 04:29:44.149452 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:44.149420 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:29:44.149452 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:44.149414 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:29:44.149452 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:44.149431 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:29:44.151830 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:44.151798 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:29:44.167054 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:44.167013 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7qbq\" (UniqueName: \"kubernetes.io/projected/34ec763e-a473-4d95-a094-6868c8e8f95d-kube-api-access-p7qbq\") pod \"maas-api-revoke-sub-1a2e7e0a-zf829\" (UID: \"34ec763e-a473-4d95-a094-6868c8e8f95d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a2e7e0a-zf829" Oct 31 04:29:44.167247 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:44.167074 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34ec763e-a473-4d95-a094-6868c8e8f95d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a2e7e0a-zf829\" (UID: \"34ec763e-a473-4d95-a094-6868c8e8f95d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a2e7e0a-zf829" Oct 31 04:29:44.167289 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:44.167262 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34ec763e-a473-4d95-a094-6868c8e8f95d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a2e7e0a-zf829\" (UID: \"34ec763e-a473-4d95-a094-6868c8e8f95d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a2e7e0a-zf829" Oct 31 04:29:44.269141 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:44.269097 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7qbq\" (UniqueName: \"kubernetes.io/projected/34ec763e-a473-4d95-a094-6868c8e8f95d-kube-api-access-p7qbq\") pod \"maas-api-revoke-sub-1a2e7e0a-zf829\" (UID: \"34ec763e-a473-4d95-a094-6868c8e8f95d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a2e7e0a-zf829" Oct 31 04:29:44.269430 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:44.269159 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34ec763e-a473-4d95-a094-6868c8e8f95d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a2e7e0a-zf829\" (UID: \"34ec763e-a473-4d95-a094-6868c8e8f95d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a2e7e0a-zf829" Oct 31 04:29:44.269430 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:44.269261 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34ec763e-a473-4d95-a094-6868c8e8f95d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a2e7e0a-zf829\" (UID: \"34ec763e-a473-4d95-a094-6868c8e8f95d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a2e7e0a-zf829" Oct 31 04:29:44.270153 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:44.270125 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34ec763e-a473-4d95-a094-6868c8e8f95d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a2e7e0a-zf829\" (UID: \"34ec763e-a473-4d95-a094-6868c8e8f95d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a2e7e0a-zf829" Oct 31 04:29:44.271972 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:44.271943 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34ec763e-a473-4d95-a094-6868c8e8f95d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a2e7e0a-zf829\" (UID: \"34ec763e-a473-4d95-a094-6868c8e8f95d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a2e7e0a-zf829" Oct 31 04:29:44.292777 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:44.292730 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7qbq\" (UniqueName: \"kubernetes.io/projected/34ec763e-a473-4d95-a094-6868c8e8f95d-kube-api-access-p7qbq\") pod \"maas-api-revoke-sub-1a2e7e0a-zf829\" (UID: \"34ec763e-a473-4d95-a094-6868c8e8f95d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a2e7e0a-zf829" Oct 31 04:29:44.439977 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:44.439891 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a2e7e0a-zf829" Oct 31 04:29:44.751677 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:44.751645 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a2e7e0a-zf829"] Oct 31 04:29:44.753506 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:29:44.753472 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34ec763e_a473_4d95_a094_6868c8e8f95d.slice/crio-1b925210f6259df21f2dbc4563f86498a5c350767bba0210526e3addaca8caed WatchSource:0}: Error finding container 1b925210f6259df21f2dbc4563f86498a5c350767bba0210526e3addaca8caed: Status 404 returned error can't find the container with id 1b925210f6259df21f2dbc4563f86498a5c350767bba0210526e3addaca8caed Oct 31 04:29:45.036912 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:45.036883 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8" Oct 31 04:29:45.081128 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:45.081090 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92398da3-abea-427b-ad63-1d9fbd4385e5-kube-api-access-gqrtn\" (UniqueName: \"kubernetes.io/projected/92398da3-abea-427b-ad63-1d9fbd4385e5-kube-api-access-gqrtn\") pod \"92398da3-abea-427b-ad63-1d9fbd4385e5\" (UID: \"92398da3-abea-427b-ad63-1d9fbd4385e5\") " Oct 31 04:29:45.082260 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:45.081209 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/92398da3-abea-427b-ad63-1d9fbd4385e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92398da3-abea-427b-ad63-1d9fbd4385e5-istio-podinfo\") pod \"92398da3-abea-427b-ad63-1d9fbd4385e5\" (UID: \"92398da3-abea-427b-ad63-1d9fbd4385e5\") " Oct 31 04:29:45.082260 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:45.081293 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92398da3-abea-427b-ad63-1d9fbd4385e5-istio-token\" (UniqueName: \"kubernetes.io/projected/92398da3-abea-427b-ad63-1d9fbd4385e5-istio-token\") pod \"92398da3-abea-427b-ad63-1d9fbd4385e5\" (UID: \"92398da3-abea-427b-ad63-1d9fbd4385e5\") " Oct 31 04:29:45.082260 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:45.081370 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92398da3-abea-427b-ad63-1d9fbd4385e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92398da3-abea-427b-ad63-1d9fbd4385e5-workload-certs\") pod \"92398da3-abea-427b-ad63-1d9fbd4385e5\" (UID: \"92398da3-abea-427b-ad63-1d9fbd4385e5\") " Oct 31 04:29:45.082260 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:45.081426 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92398da3-abea-427b-ad63-1d9fbd4385e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92398da3-abea-427b-ad63-1d9fbd4385e5-istio-data\") pod \"92398da3-abea-427b-ad63-1d9fbd4385e5\" (UID: \"92398da3-abea-427b-ad63-1d9fbd4385e5\") " Oct 31 04:29:45.082260 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:45.081501 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92398da3-abea-427b-ad63-1d9fbd4385e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92398da3-abea-427b-ad63-1d9fbd4385e5-istio-envoy\") pod \"92398da3-abea-427b-ad63-1d9fbd4385e5\" (UID: \"92398da3-abea-427b-ad63-1d9fbd4385e5\") " Oct 31 04:29:45.082260 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:45.081530 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92398da3-abea-427b-ad63-1d9fbd4385e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92398da3-abea-427b-ad63-1d9fbd4385e5-workload-socket\") pod \"92398da3-abea-427b-ad63-1d9fbd4385e5\" (UID: \"92398da3-abea-427b-ad63-1d9fbd4385e5\") " Oct 31 04:29:45.082260 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:45.081588 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92398da3-abea-427b-ad63-1d9fbd4385e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92398da3-abea-427b-ad63-1d9fbd4385e5-istiod-ca-cert\") pod \"92398da3-abea-427b-ad63-1d9fbd4385e5\" (UID: \"92398da3-abea-427b-ad63-1d9fbd4385e5\") " Oct 31 04:29:45.082260 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:45.081656 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92398da3-abea-427b-ad63-1d9fbd4385e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92398da3-abea-427b-ad63-1d9fbd4385e5-credential-socket\") pod \"92398da3-abea-427b-ad63-1d9fbd4385e5\" (UID: \"92398da3-abea-427b-ad63-1d9fbd4385e5\") " Oct 31 04:29:45.082882 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:45.082629 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92398da3-abea-427b-ad63-1d9fbd4385e5-credential-socket" pod "92398da3-abea-427b-ad63-1d9fbd4385e5" (UID: "92398da3-abea-427b-ad63-1d9fbd4385e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:29:45.083694 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:45.082966 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92398da3-abea-427b-ad63-1d9fbd4385e5-istio-data" pod "92398da3-abea-427b-ad63-1d9fbd4385e5" (UID: "92398da3-abea-427b-ad63-1d9fbd4385e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:29:45.083694 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:45.083191 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92398da3-abea-427b-ad63-1d9fbd4385e5-workload-certs" pod "92398da3-abea-427b-ad63-1d9fbd4385e5" (UID: "92398da3-abea-427b-ad63-1d9fbd4385e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:29:45.083694 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:45.083461 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92398da3-abea-427b-ad63-1d9fbd4385e5-workload-socket" pod "92398da3-abea-427b-ad63-1d9fbd4385e5" (UID: "92398da3-abea-427b-ad63-1d9fbd4385e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:29:45.083694 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:45.083666 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92398da3-abea-427b-ad63-1d9fbd4385e5-istiod-ca-cert" pod "92398da3-abea-427b-ad63-1d9fbd4385e5" (UID: "92398da3-abea-427b-ad63-1d9fbd4385e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:29:45.086652 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:45.086590 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92398da3-abea-427b-ad63-1d9fbd4385e5-kube-api-access-gqrtn" pod "92398da3-abea-427b-ad63-1d9fbd4385e5" (UID: "92398da3-abea-427b-ad63-1d9fbd4385e5"). InnerVolumeSpecName "kube-api-access-gqrtn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:29:45.088661 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:45.088624 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92398da3-abea-427b-ad63-1d9fbd4385e5-istio-token" pod "92398da3-abea-427b-ad63-1d9fbd4385e5" (UID: "92398da3-abea-427b-ad63-1d9fbd4385e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:29:45.088867 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:45.088830 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92398da3-abea-427b-ad63-1d9fbd4385e5-istio-podinfo" pod "92398da3-abea-427b-ad63-1d9fbd4385e5" (UID: "92398da3-abea-427b-ad63-1d9fbd4385e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:29:45.089622 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:45.089425 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92398da3-abea-427b-ad63-1d9fbd4385e5-istio-envoy" pod "92398da3-abea-427b-ad63-1d9fbd4385e5" (UID: "92398da3-abea-427b-ad63-1d9fbd4385e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:29:45.184024 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:45.183981 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92398da3-abea-427b-ad63-1d9fbd4385e5-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:29:45.184024 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:45.184017 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92398da3-abea-427b-ad63-1d9fbd4385e5-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:29:45.184024 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:45.184027 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92398da3-abea-427b-ad63-1d9fbd4385e5-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:29:45.184024 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:45.184034 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92398da3-abea-427b-ad63-1d9fbd4385e5-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:29:45.184530 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:45.184042 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92398da3-abea-427b-ad63-1d9fbd4385e5-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:29:45.184530 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:45.184050 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92398da3-abea-427b-ad63-1d9fbd4385e5-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:29:45.184530 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:45.184058 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92398da3-abea-427b-ad63-1d9fbd4385e5-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:29:45.184530 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:45.184066 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqrtn\" (UniqueName: \"kubernetes.io/projected/92398da3-abea-427b-ad63-1d9fbd4385e5-kube-api-access-gqrtn\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:29:45.184530 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:45.184076 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92398da3-abea-427b-ad63-1d9fbd4385e5-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:29:45.204156 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:45.204121 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a2e7e0a-zf829" event={"ID":"34ec763e-a473-4d95-a094-6868c8e8f95d","Type":"ContainerStarted","Data":"1b925210f6259df21f2dbc4563f86498a5c350767bba0210526e3addaca8caed"} Oct 31 04:29:45.206197 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:45.206171 5019 generic.go:356] "Generic (PLEG): container finished" podID="92398da3-abea-427b-ad63-1d9fbd4385e5" containerID="0ebf790fd3da68e5d821df8b7cc619b150466ace914bbfa5ee8dbb29b8d79c8c" exitCode=0 Oct 31 04:29:45.206359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:45.206235 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8" Oct 31 04:29:45.206359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:45.206262 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8" event={"ID":"92398da3-abea-427b-ad63-1d9fbd4385e5","Type":"ContainerDied","Data":"0ebf790fd3da68e5d821df8b7cc619b150466ace914bbfa5ee8dbb29b8d79c8c"} Oct 31 04:29:45.206359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:45.206317 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8" event={"ID":"92398da3-abea-427b-ad63-1d9fbd4385e5","Type":"ContainerDied","Data":"27d0ece5079b77213cc93f2d5db609e5ca4c9c42f170c6fbbd8c44593eaab633"} Oct 31 04:29:45.206500 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:45.206362 5019 scope.go:122] "RemoveContainer" containerID="0ebf790fd3da68e5d821df8b7cc619b150466ace914bbfa5ee8dbb29b8d79c8c" Oct 31 04:29:45.221968 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:45.221947 5019 scope.go:122] "RemoveContainer" containerID="33fec37f6afc9a22df2004dd4083eb96dbf8e4d306eaf1eeffbfa190e52571e0" Oct 31 04:29:45.233609 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:45.233586 5019 scope.go:122] "RemoveContainer" containerID="0ebf790fd3da68e5d821df8b7cc619b150466ace914bbfa5ee8dbb29b8d79c8c" Oct 31 04:29:45.233908 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:29:45.233875 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ebf790fd3da68e5d821df8b7cc619b150466ace914bbfa5ee8dbb29b8d79c8c\": container with ID starting with 0ebf790fd3da68e5d821df8b7cc619b150466ace914bbfa5ee8dbb29b8d79c8c not found: ID does not exist" containerID="0ebf790fd3da68e5d821df8b7cc619b150466ace914bbfa5ee8dbb29b8d79c8c" Oct 31 04:29:45.234022 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:45.233917 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ebf790fd3da68e5d821df8b7cc619b150466ace914bbfa5ee8dbb29b8d79c8c"} err="failed to get container status \"0ebf790fd3da68e5d821df8b7cc619b150466ace914bbfa5ee8dbb29b8d79c8c\": rpc error: code = NotFound desc = could not find container \"0ebf790fd3da68e5d821df8b7cc619b150466ace914bbfa5ee8dbb29b8d79c8c\": container with ID starting with 0ebf790fd3da68e5d821df8b7cc619b150466ace914bbfa5ee8dbb29b8d79c8c not found: ID does not exist" Oct 31 04:29:45.234022 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:45.233941 5019 scope.go:122] "RemoveContainer" containerID="33fec37f6afc9a22df2004dd4083eb96dbf8e4d306eaf1eeffbfa190e52571e0" Oct 31 04:29:45.234270 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:29:45.234246 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33fec37f6afc9a22df2004dd4083eb96dbf8e4d306eaf1eeffbfa190e52571e0\": container with ID starting with 33fec37f6afc9a22df2004dd4083eb96dbf8e4d306eaf1eeffbfa190e52571e0 not found: ID does not exist" containerID="33fec37f6afc9a22df2004dd4083eb96dbf8e4d306eaf1eeffbfa190e52571e0" Oct 31 04:29:45.234408 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:45.234275 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33fec37f6afc9a22df2004dd4083eb96dbf8e4d306eaf1eeffbfa190e52571e0"} err="failed to get container status \"33fec37f6afc9a22df2004dd4083eb96dbf8e4d306eaf1eeffbfa190e52571e0\": rpc error: code = NotFound desc = could not find container \"33fec37f6afc9a22df2004dd4083eb96dbf8e4d306eaf1eeffbfa190e52571e0\": container with ID starting with 33fec37f6afc9a22df2004dd4083eb96dbf8e4d306eaf1eeffbfa190e52571e0 not found: ID does not exist" Oct 31 04:29:45.260287 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:45.260260 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8"] Oct 31 04:29:45.264401 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:45.264379 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-c6859a-openshift-default-6d77b94c97-s7ld8"] Oct 31 04:29:45.411423 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:45.411384 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92398da3-abea-427b-ad63-1d9fbd4385e5" path="/var/lib/kubelet/pods/92398da3-abea-427b-ad63-1d9fbd4385e5/volumes" Oct 31 04:29:46.218066 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:46.218031 5019 generic.go:356] "Generic (PLEG): container finished" podID="34ec763e-a473-4d95-a094-6868c8e8f95d" containerID="09757164f5be9a87f68e6ab824bb3db2b7896effd161947f9522309365a19dff" exitCode=0 Oct 31 04:29:46.218535 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:46.218119 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a2e7e0a-zf829" event={"ID":"34ec763e-a473-4d95-a094-6868c8e8f95d","Type":"ContainerDied","Data":"09757164f5be9a87f68e6ab824bb3db2b7896effd161947f9522309365a19dff"} Oct 31 04:29:46.685763 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:46.685671 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-edc450-8c5754587-sfsfv"] Oct 31 04:29:47.445413 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:29:47.436609 5019 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda473bd3b_e50d_4a3d_ad54_fd49ee49f144.slice/crio-conmon-e8878179de9af734b48958325224021be55c4d6c045b1c39fa15d2c27a9da8b3.scope\": RecentStats: unable to find data in memory cache]" Oct 31 04:29:47.445413 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:29:47.436639 5019 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda473bd3b_e50d_4a3d_ad54_fd49ee49f144.slice/crio-conmon-e8878179de9af734b48958325224021be55c4d6c045b1c39fa15d2c27a9da8b3.scope\": RecentStats: unable to find data in memory cache]" Oct 31 04:29:47.502849 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:47.502818 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a2e7e0a-zf829" Oct 31 04:29:47.621960 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:47.621933 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd" Oct 31 04:29:47.628309 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:47.628283 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34ec763e-a473-4d95-a094-6868c8e8f95d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34ec763e-a473-4d95-a094-6868c8e8f95d-maas-api-openshift-ca\") pod \"34ec763e-a473-4d95-a094-6868c8e8f95d\" (UID: \"34ec763e-a473-4d95-a094-6868c8e8f95d\") " Oct 31 04:29:47.628465 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:47.628371 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34ec763e-a473-4d95-a094-6868c8e8f95d-kube-api-access-p7qbq\" (UniqueName: \"kubernetes.io/projected/34ec763e-a473-4d95-a094-6868c8e8f95d-kube-api-access-p7qbq\") pod \"34ec763e-a473-4d95-a094-6868c8e8f95d\" (UID: \"34ec763e-a473-4d95-a094-6868c8e8f95d\") " Oct 31 04:29:47.628556 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:47.628544 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34ec763e-a473-4d95-a094-6868c8e8f95d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34ec763e-a473-4d95-a094-6868c8e8f95d-maas-api-xks-ca\") pod \"34ec763e-a473-4d95-a094-6868c8e8f95d\" (UID: \"34ec763e-a473-4d95-a094-6868c8e8f95d\") " Oct 31 04:29:47.628668 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:47.628646 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34ec763e-a473-4d95-a094-6868c8e8f95d-maas-api-openshift-ca" pod "34ec763e-a473-4d95-a094-6868c8e8f95d" (UID: "34ec763e-a473-4d95-a094-6868c8e8f95d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:29:47.629053 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:47.629031 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34ec763e-a473-4d95-a094-6868c8e8f95d-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:29:47.630594 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:47.630566 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34ec763e-a473-4d95-a094-6868c8e8f95d-maas-api-xks-ca" pod "34ec763e-a473-4d95-a094-6868c8e8f95d" (UID: "34ec763e-a473-4d95-a094-6868c8e8f95d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:29:47.630795 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:47.630780 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34ec763e-a473-4d95-a094-6868c8e8f95d-kube-api-access-p7qbq" pod "34ec763e-a473-4d95-a094-6868c8e8f95d" (UID: "34ec763e-a473-4d95-a094-6868c8e8f95d"). InnerVolumeSpecName "kube-api-access-p7qbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:29:47.729906 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:47.729821 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-istio-podinfo\") pod \"a473bd3b-e50d-4a3d-ad54-fd49ee49f144\" (UID: \"a473bd3b-e50d-4a3d-ad54-fd49ee49f144\") " Oct 31 04:29:47.729906 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:47.729897 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-istio-data\") pod \"a473bd3b-e50d-4a3d-ad54-fd49ee49f144\" (UID: \"a473bd3b-e50d-4a3d-ad54-fd49ee49f144\") " Oct 31 04:29:47.730103 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:47.729934 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-istio-envoy\") pod \"a473bd3b-e50d-4a3d-ad54-fd49ee49f144\" (UID: \"a473bd3b-e50d-4a3d-ad54-fd49ee49f144\") " Oct 31 04:29:47.730151 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:47.730135 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-istio-token\" (UniqueName: \"kubernetes.io/projected/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-istio-token\") pod \"a473bd3b-e50d-4a3d-ad54-fd49ee49f144\" (UID: \"a473bd3b-e50d-4a3d-ad54-fd49ee49f144\") " Oct 31 04:29:47.730192 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:47.730174 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-istio-data" pod "a473bd3b-e50d-4a3d-ad54-fd49ee49f144" (UID: "a473bd3b-e50d-4a3d-ad54-fd49ee49f144"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:29:47.730246 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:47.730198 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-workload-socket\") pod \"a473bd3b-e50d-4a3d-ad54-fd49ee49f144\" (UID: \"a473bd3b-e50d-4a3d-ad54-fd49ee49f144\") " Oct 31 04:29:47.730246 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:47.730240 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-istiod-ca-cert\") pod \"a473bd3b-e50d-4a3d-ad54-fd49ee49f144\" (UID: \"a473bd3b-e50d-4a3d-ad54-fd49ee49f144\") " Oct 31 04:29:47.730393 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:47.730264 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-kube-api-access-k8bk2\" (UniqueName: \"kubernetes.io/projected/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-kube-api-access-k8bk2\") pod \"a473bd3b-e50d-4a3d-ad54-fd49ee49f144\" (UID: \"a473bd3b-e50d-4a3d-ad54-fd49ee49f144\") " Oct 31 04:29:47.730393 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:47.730317 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-workload-certs\") pod \"a473bd3b-e50d-4a3d-ad54-fd49ee49f144\" (UID: \"a473bd3b-e50d-4a3d-ad54-fd49ee49f144\") " Oct 31 04:29:47.730393 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:47.730369 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-credential-socket\") pod \"a473bd3b-e50d-4a3d-ad54-fd49ee49f144\" (UID: \"a473bd3b-e50d-4a3d-ad54-fd49ee49f144\") " Oct 31 04:29:47.730582 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:47.730510 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-workload-socket" pod "a473bd3b-e50d-4a3d-ad54-fd49ee49f144" (UID: "a473bd3b-e50d-4a3d-ad54-fd49ee49f144"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:29:47.730664 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:47.730640 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-workload-certs" pod "a473bd3b-e50d-4a3d-ad54-fd49ee49f144" (UID: "a473bd3b-e50d-4a3d-ad54-fd49ee49f144"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:29:47.730798 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:47.730775 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-credential-socket" pod "a473bd3b-e50d-4a3d-ad54-fd49ee49f144" (UID: "a473bd3b-e50d-4a3d-ad54-fd49ee49f144"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:29:47.731007 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:47.730987 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:29:47.731099 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:47.731008 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:29:47.731099 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:47.731017 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:29:47.731099 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:47.731035 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:29:47.731099 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:47.731031 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-istiod-ca-cert" pod "a473bd3b-e50d-4a3d-ad54-fd49ee49f144" (UID: "a473bd3b-e50d-4a3d-ad54-fd49ee49f144"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:29:47.731099 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:47.731043 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34ec763e-a473-4d95-a094-6868c8e8f95d-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:29:47.731099 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:47.731052 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7qbq\" (UniqueName: \"kubernetes.io/projected/34ec763e-a473-4d95-a094-6868c8e8f95d-kube-api-access-p7qbq\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:29:47.732410 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:47.732383 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-kube-api-access-k8bk2" pod "a473bd3b-e50d-4a3d-ad54-fd49ee49f144" (UID: "a473bd3b-e50d-4a3d-ad54-fd49ee49f144"). InnerVolumeSpecName "kube-api-access-k8bk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:29:47.732511 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:47.732449 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-istio-token" pod "a473bd3b-e50d-4a3d-ad54-fd49ee49f144" (UID: "a473bd3b-e50d-4a3d-ad54-fd49ee49f144"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:29:47.732632 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:47.732609 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-istio-envoy" pod "a473bd3b-e50d-4a3d-ad54-fd49ee49f144" (UID: "a473bd3b-e50d-4a3d-ad54-fd49ee49f144"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:29:47.732681 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:47.732615 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-istio-podinfo" pod "a473bd3b-e50d-4a3d-ad54-fd49ee49f144" (UID: "a473bd3b-e50d-4a3d-ad54-fd49ee49f144"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:29:47.832267 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:47.832232 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:29:47.832267 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:47.832260 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8bk2\" (UniqueName: \"kubernetes.io/projected/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-kube-api-access-k8bk2\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:29:47.832267 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:47.832270 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:29:47.832538 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:47.832279 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:29:47.832538 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:47.832287 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a473bd3b-e50d-4a3d-ad54-fd49ee49f144-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:29:48.235161 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:48.235127 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a2e7e0a-zf829" Oct 31 04:29:48.235391 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:48.235131 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a2e7e0a-zf829" event={"ID":"34ec763e-a473-4d95-a094-6868c8e8f95d","Type":"ContainerDied","Data":"1b925210f6259df21f2dbc4563f86498a5c350767bba0210526e3addaca8caed"} Oct 31 04:29:48.235391 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:48.235260 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b925210f6259df21f2dbc4563f86498a5c350767bba0210526e3addaca8caed" Oct 31 04:29:48.237088 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:48.237063 5019 generic.go:356] "Generic (PLEG): container finished" podID="a473bd3b-e50d-4a3d-ad54-fd49ee49f144" containerID="e8878179de9af734b48958325224021be55c4d6c045b1c39fa15d2c27a9da8b3" exitCode=0 Oct 31 04:29:48.237200 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:48.237107 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd" Oct 31 04:29:48.237200 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:48.237151 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd" event={"ID":"a473bd3b-e50d-4a3d-ad54-fd49ee49f144","Type":"ContainerDied","Data":"e8878179de9af734b48958325224021be55c4d6c045b1c39fa15d2c27a9da8b3"} Oct 31 04:29:48.237285 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:48.237219 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd" event={"ID":"a473bd3b-e50d-4a3d-ad54-fd49ee49f144","Type":"ContainerDied","Data":"b803dc928a469e94e5e1ef458f1eea779d5f91fd93d8a96bef719f44eda633bb"} Oct 31 04:29:48.237285 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:48.237249 5019 scope.go:122] "RemoveContainer" containerID="e8878179de9af734b48958325224021be55c4d6c045b1c39fa15d2c27a9da8b3" Oct 31 04:29:48.248201 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:48.248107 5019 scope.go:122] "RemoveContainer" containerID="ae4735ff03902e2365dcc8f480457ab57d0e20248b7405414b828b8d3667dca5" Oct 31 04:29:48.257640 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:48.257623 5019 scope.go:122] "RemoveContainer" containerID="e8878179de9af734b48958325224021be55c4d6c045b1c39fa15d2c27a9da8b3" Oct 31 04:29:48.257889 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:29:48.257872 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8878179de9af734b48958325224021be55c4d6c045b1c39fa15d2c27a9da8b3\": container with ID starting with e8878179de9af734b48958325224021be55c4d6c045b1c39fa15d2c27a9da8b3 not found: ID does not exist" containerID="e8878179de9af734b48958325224021be55c4d6c045b1c39fa15d2c27a9da8b3" Oct 31 04:29:48.257977 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:48.257901 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8878179de9af734b48958325224021be55c4d6c045b1c39fa15d2c27a9da8b3"} err="failed to get container status \"e8878179de9af734b48958325224021be55c4d6c045b1c39fa15d2c27a9da8b3\": rpc error: code = NotFound desc = could not find container \"e8878179de9af734b48958325224021be55c4d6c045b1c39fa15d2c27a9da8b3\": container with ID starting with e8878179de9af734b48958325224021be55c4d6c045b1c39fa15d2c27a9da8b3 not found: ID does not exist" Oct 31 04:29:48.257977 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:48.257922 5019 scope.go:122] "RemoveContainer" containerID="ae4735ff03902e2365dcc8f480457ab57d0e20248b7405414b828b8d3667dca5" Oct 31 04:29:48.258143 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:29:48.258125 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae4735ff03902e2365dcc8f480457ab57d0e20248b7405414b828b8d3667dca5\": container with ID starting with ae4735ff03902e2365dcc8f480457ab57d0e20248b7405414b828b8d3667dca5 not found: ID does not exist" containerID="ae4735ff03902e2365dcc8f480457ab57d0e20248b7405414b828b8d3667dca5" Oct 31 04:29:48.258181 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:48.258149 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae4735ff03902e2365dcc8f480457ab57d0e20248b7405414b828b8d3667dca5"} err="failed to get container status \"ae4735ff03902e2365dcc8f480457ab57d0e20248b7405414b828b8d3667dca5\": rpc error: code = NotFound desc = could not find container \"ae4735ff03902e2365dcc8f480457ab57d0e20248b7405414b828b8d3667dca5\": container with ID starting with ae4735ff03902e2365dcc8f480457ab57d0e20248b7405414b828b8d3667dca5 not found: ID does not exist" Oct 31 04:29:48.306705 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:48.306676 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd"] Oct 31 04:29:48.317533 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:48.317505 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-ff43e3-openshift-default-6bcc4fdd9-wt9wd"] Oct 31 04:29:49.410773 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:49.410737 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a473bd3b-e50d-4a3d-ad54-fd49ee49f144" path="/var/lib/kubelet/pods/a473bd3b-e50d-4a3d-ad54-fd49ee49f144/volumes" Oct 31 04:29:51.686379 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:51.686326 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-edc450-8c5754587-sfsfv" podUID="c0d4b688-163d-43b4-af30-0a834f972117" containerName="payload-processing" containerID="cri-o://d13f0b2a5d0dc0dd4e33d44e778ac26b757fc035b498cfeabe66ed487b438fcd" gracePeriod=25 Oct 31 04:29:52.012589 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:52.012561 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-edc450-8c5754587-sfsfv" Oct 31 04:29:52.087911 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:52.087872 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0d4b688-163d-43b4-af30-0a834f972117-kube-api-access-fgj5k\" (UniqueName: \"kubernetes.io/projected/c0d4b688-163d-43b4-af30-0a834f972117-kube-api-access-fgj5k\") pod \"c0d4b688-163d-43b4-af30-0a834f972117\" (UID: \"c0d4b688-163d-43b4-af30-0a834f972117\") " Oct 31 04:29:52.088086 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:52.087945 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0d4b688-163d-43b4-af30-0a834f972117-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0d4b688-163d-43b4-af30-0a834f972117-plugins-config-volume\") pod \"c0d4b688-163d-43b4-af30-0a834f972117\" (UID: \"c0d4b688-163d-43b4-af30-0a834f972117\") " Oct 31 04:29:52.088452 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:52.088425 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0d4b688-163d-43b4-af30-0a834f972117-plugins-config-volume" pod "c0d4b688-163d-43b4-af30-0a834f972117" (UID: "c0d4b688-163d-43b4-af30-0a834f972117"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:29:52.088661 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:52.088629 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0d4b688-163d-43b4-af30-0a834f972117-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:29:52.090783 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:52.090750 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0d4b688-163d-43b4-af30-0a834f972117-kube-api-access-fgj5k" pod "c0d4b688-163d-43b4-af30-0a834f972117" (UID: "c0d4b688-163d-43b4-af30-0a834f972117"). InnerVolumeSpecName "kube-api-access-fgj5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:29:52.190256 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:52.190213 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgj5k\" (UniqueName: \"kubernetes.io/projected/c0d4b688-163d-43b4-af30-0a834f972117-kube-api-access-fgj5k\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:29:52.272785 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:52.272679 5019 generic.go:356] "Generic (PLEG): container finished" podID="c0d4b688-163d-43b4-af30-0a834f972117" containerID="d13f0b2a5d0dc0dd4e33d44e778ac26b757fc035b498cfeabe66ed487b438fcd" exitCode=0 Oct 31 04:29:52.272959 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:52.272927 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-edc450-8c5754587-sfsfv" event={"ID":"c0d4b688-163d-43b4-af30-0a834f972117","Type":"ContainerDied","Data":"d13f0b2a5d0dc0dd4e33d44e778ac26b757fc035b498cfeabe66ed487b438fcd"} Oct 31 04:29:52.273036 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:52.273017 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-edc450-8c5754587-sfsfv" event={"ID":"c0d4b688-163d-43b4-af30-0a834f972117","Type":"ContainerDied","Data":"f530272c69289e78c1264fa13b5b474c53df9ba3ececd582ce52269d7b6d44ec"} Oct 31 04:29:52.273100 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:52.273061 5019 scope.go:122] "RemoveContainer" containerID="d13f0b2a5d0dc0dd4e33d44e778ac26b757fc035b498cfeabe66ed487b438fcd" Oct 31 04:29:52.273242 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:52.273225 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-edc450-8c5754587-sfsfv" Oct 31 04:29:52.289816 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:52.289789 5019 scope.go:122] "RemoveContainer" containerID="d13f0b2a5d0dc0dd4e33d44e778ac26b757fc035b498cfeabe66ed487b438fcd" Oct 31 04:29:52.290186 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:29:52.290164 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d13f0b2a5d0dc0dd4e33d44e778ac26b757fc035b498cfeabe66ed487b438fcd\": container with ID starting with d13f0b2a5d0dc0dd4e33d44e778ac26b757fc035b498cfeabe66ed487b438fcd not found: ID does not exist" containerID="d13f0b2a5d0dc0dd4e33d44e778ac26b757fc035b498cfeabe66ed487b438fcd" Oct 31 04:29:52.290256 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:52.290199 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d13f0b2a5d0dc0dd4e33d44e778ac26b757fc035b498cfeabe66ed487b438fcd"} err="failed to get container status \"d13f0b2a5d0dc0dd4e33d44e778ac26b757fc035b498cfeabe66ed487b438fcd\": rpc error: code = NotFound desc = could not find container \"d13f0b2a5d0dc0dd4e33d44e778ac26b757fc035b498cfeabe66ed487b438fcd\": container with ID starting with d13f0b2a5d0dc0dd4e33d44e778ac26b757fc035b498cfeabe66ed487b438fcd not found: ID does not exist" Oct 31 04:29:52.329961 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:52.329918 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-edc450-8c5754587-sfsfv"] Oct 31 04:29:52.333780 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:52.333755 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-edc450-8c5754587-sfsfv"] Oct 31 04:29:52.399084 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:29:52.399049 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:29:53.407926 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:53.407890 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0d4b688-163d-43b4-af30-0a834f972117" path="/var/lib/kubelet/pods/c0d4b688-163d-43b4-af30-0a834f972117/volumes" Oct 31 04:29:55.326509 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.326474 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8"] Oct 31 04:29:55.328196 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.328003 5019 cpu_manager.go:414] "containerMap: removing container" podUID="c0d4b688-163d-43b4-af30-0a834f972117" containerName="payload-processing" Oct 31 04:29:55.328418 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.328394 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0d4b688-163d-43b4-af30-0a834f972117" containerName="payload-processing" Oct 31 04:29:55.328557 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.328429 5019 cpu_manager.go:414] "containerMap: removing container" podUID="92398da3-abea-427b-ad63-1d9fbd4385e5" containerName="istio-proxy" Oct 31 04:29:55.328557 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.328438 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92398da3-abea-427b-ad63-1d9fbd4385e5" containerName="istio-proxy" Oct 31 04:29:55.328557 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.328487 5019 cpu_manager.go:414] "containerMap: removing container" podUID="a473bd3b-e50d-4a3d-ad54-fd49ee49f144" containerName="istio-proxy" Oct 31 04:29:55.328557 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.328496 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a473bd3b-e50d-4a3d-ad54-fd49ee49f144" containerName="istio-proxy" Oct 31 04:29:55.328557 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.328532 5019 cpu_manager.go:414] "containerMap: removing container" podUID="a473bd3b-e50d-4a3d-ad54-fd49ee49f144" containerName="istio-proxy" Oct 31 04:29:55.328557 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.328541 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a473bd3b-e50d-4a3d-ad54-fd49ee49f144" containerName="istio-proxy" Oct 31 04:29:55.328848 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.328564 5019 cpu_manager.go:414] "containerMap: removing container" podUID="34ec763e-a473-4d95-a094-6868c8e8f95d" containerName="revoke-keys" Oct 31 04:29:55.328848 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.328570 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34ec763e-a473-4d95-a094-6868c8e8f95d" containerName="revoke-keys" Oct 31 04:29:55.329244 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.329222 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="a473bd3b-e50d-4a3d-ad54-fd49ee49f144" containerName="istio-proxy" Oct 31 04:29:55.329397 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.329382 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="92398da3-abea-427b-ad63-1d9fbd4385e5" containerName="istio-proxy" Oct 31 04:29:55.329522 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.329510 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="a473bd3b-e50d-4a3d-ad54-fd49ee49f144" containerName="istio-proxy" Oct 31 04:29:55.329627 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.329616 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="34ec763e-a473-4d95-a094-6868c8e8f95d" containerName="revoke-keys" Oct 31 04:29:55.329714 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.329704 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0d4b688-163d-43b4-af30-0a834f972117" containerName="payload-processing" Oct 31 04:29:55.329787 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.329778 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="92398da3-abea-427b-ad63-1d9fbd4385e5" containerName="istio-proxy" Oct 31 04:29:55.338926 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.338889 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8" Oct 31 04:29:55.352660 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.350901 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp"] Oct 31 04:29:55.352660 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.351904 5019 cpu_manager.go:414] "containerMap: removing container" podUID="92398da3-abea-427b-ad63-1d9fbd4385e5" containerName="istio-proxy" Oct 31 04:29:55.352660 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.351924 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92398da3-abea-427b-ad63-1d9fbd4385e5" containerName="istio-proxy" Oct 31 04:29:55.366579 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.356264 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp" Oct 31 04:29:55.366579 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.357955 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8"] Oct 31 04:29:55.422400 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.422368 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp"] Oct 31 04:29:55.433636 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.433595 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75bc7ee5-eb66-43cf-8bca-5322159afbb0-istiod-ca-cert\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8\" (UID: \"75bc7ee5-eb66-43cf-8bca-5322159afbb0\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8" Oct 31 04:29:55.433834 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.433734 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75bc7ee5-eb66-43cf-8bca-5322159afbb0-workload-certs\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8\" (UID: \"75bc7ee5-eb66-43cf-8bca-5322159afbb0\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8" Oct 31 04:29:55.433834 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.433777 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75bc7ee5-eb66-43cf-8bca-5322159afbb0-istio-envoy\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8\" (UID: \"75bc7ee5-eb66-43cf-8bca-5322159afbb0\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8" Oct 31 04:29:55.433834 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.433821 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5pgf\" (UniqueName: \"kubernetes.io/projected/75bc7ee5-eb66-43cf-8bca-5322159afbb0-kube-api-access-n5pgf\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8\" (UID: \"75bc7ee5-eb66-43cf-8bca-5322159afbb0\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8" Oct 31 04:29:55.434016 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.433930 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75bc7ee5-eb66-43cf-8bca-5322159afbb0-credential-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8\" (UID: \"75bc7ee5-eb66-43cf-8bca-5322159afbb0\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8" Oct 31 04:29:55.434016 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.433969 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75bc7ee5-eb66-43cf-8bca-5322159afbb0-istio-data\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8\" (UID: \"75bc7ee5-eb66-43cf-8bca-5322159afbb0\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8" Oct 31 04:29:55.434016 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.433988 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75bc7ee5-eb66-43cf-8bca-5322159afbb0-istio-token\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8\" (UID: \"75bc7ee5-eb66-43cf-8bca-5322159afbb0\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8" Oct 31 04:29:55.434175 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.434046 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75bc7ee5-eb66-43cf-8bca-5322159afbb0-istio-podinfo\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8\" (UID: \"75bc7ee5-eb66-43cf-8bca-5322159afbb0\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8" Oct 31 04:29:55.434175 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.434096 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75bc7ee5-eb66-43cf-8bca-5322159afbb0-workload-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8\" (UID: \"75bc7ee5-eb66-43cf-8bca-5322159afbb0\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8" Oct 31 04:29:55.535051 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.535009 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d85d51e-dcd7-4e26-b793-70db760b09a8-istiod-ca-cert\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp\" (UID: \"6d85d51e-dcd7-4e26-b793-70db760b09a8\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp" Oct 31 04:29:55.535262 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.535128 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75bc7ee5-eb66-43cf-8bca-5322159afbb0-istiod-ca-cert\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8\" (UID: \"75bc7ee5-eb66-43cf-8bca-5322159afbb0\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8" Oct 31 04:29:55.535262 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.535176 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d85d51e-dcd7-4e26-b793-70db760b09a8-istio-token\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp\" (UID: \"6d85d51e-dcd7-4e26-b793-70db760b09a8\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp" Oct 31 04:29:55.535417 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.535299 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d85d51e-dcd7-4e26-b793-70db760b09a8-workload-certs\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp\" (UID: \"6d85d51e-dcd7-4e26-b793-70db760b09a8\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp" Oct 31 04:29:55.535417 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.535369 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjmvz\" (UniqueName: \"kubernetes.io/projected/6d85d51e-dcd7-4e26-b793-70db760b09a8-kube-api-access-bjmvz\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp\" (UID: \"6d85d51e-dcd7-4e26-b793-70db760b09a8\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp" Oct 31 04:29:55.535526 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.535453 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75bc7ee5-eb66-43cf-8bca-5322159afbb0-workload-certs\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8\" (UID: \"75bc7ee5-eb66-43cf-8bca-5322159afbb0\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8" Oct 31 04:29:55.535526 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.535502 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75bc7ee5-eb66-43cf-8bca-5322159afbb0-istio-envoy\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8\" (UID: \"75bc7ee5-eb66-43cf-8bca-5322159afbb0\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8" Oct 31 04:29:55.535636 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.535568 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5pgf\" (UniqueName: \"kubernetes.io/projected/75bc7ee5-eb66-43cf-8bca-5322159afbb0-kube-api-access-n5pgf\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8\" (UID: \"75bc7ee5-eb66-43cf-8bca-5322159afbb0\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8" Oct 31 04:29:55.535636 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.535618 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75bc7ee5-eb66-43cf-8bca-5322159afbb0-credential-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8\" (UID: \"75bc7ee5-eb66-43cf-8bca-5322159afbb0\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8" Oct 31 04:29:55.535732 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.535655 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d85d51e-dcd7-4e26-b793-70db760b09a8-istio-data\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp\" (UID: \"6d85d51e-dcd7-4e26-b793-70db760b09a8\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp" Oct 31 04:29:55.535732 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.535687 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75bc7ee5-eb66-43cf-8bca-5322159afbb0-istio-data\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8\" (UID: \"75bc7ee5-eb66-43cf-8bca-5322159afbb0\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8" Oct 31 04:29:55.535732 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.535712 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75bc7ee5-eb66-43cf-8bca-5322159afbb0-istio-token\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8\" (UID: \"75bc7ee5-eb66-43cf-8bca-5322159afbb0\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8" Oct 31 04:29:55.535877 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.535744 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75bc7ee5-eb66-43cf-8bca-5322159afbb0-istio-podinfo\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8\" (UID: \"75bc7ee5-eb66-43cf-8bca-5322159afbb0\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8" Oct 31 04:29:55.535877 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.535785 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d85d51e-dcd7-4e26-b793-70db760b09a8-istio-podinfo\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp\" (UID: \"6d85d51e-dcd7-4e26-b793-70db760b09a8\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp" Oct 31 04:29:55.535877 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.535822 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d85d51e-dcd7-4e26-b793-70db760b09a8-workload-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp\" (UID: \"6d85d51e-dcd7-4e26-b793-70db760b09a8\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp" Oct 31 04:29:55.535877 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.535864 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d85d51e-dcd7-4e26-b793-70db760b09a8-credential-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp\" (UID: \"6d85d51e-dcd7-4e26-b793-70db760b09a8\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp" Oct 31 04:29:55.536066 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.535881 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75bc7ee5-eb66-43cf-8bca-5322159afbb0-workload-certs\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8\" (UID: \"75bc7ee5-eb66-43cf-8bca-5322159afbb0\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8" Oct 31 04:29:55.536066 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.535897 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75bc7ee5-eb66-43cf-8bca-5322159afbb0-workload-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8\" (UID: \"75bc7ee5-eb66-43cf-8bca-5322159afbb0\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8" Oct 31 04:29:55.536066 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.535949 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d85d51e-dcd7-4e26-b793-70db760b09a8-istio-envoy\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp\" (UID: \"6d85d51e-dcd7-4e26-b793-70db760b09a8\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp" Oct 31 04:29:55.536066 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.535996 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75bc7ee5-eb66-43cf-8bca-5322159afbb0-istiod-ca-cert\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8\" (UID: \"75bc7ee5-eb66-43cf-8bca-5322159afbb0\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8" Oct 31 04:29:55.536267 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.536139 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75bc7ee5-eb66-43cf-8bca-5322159afbb0-workload-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8\" (UID: \"75bc7ee5-eb66-43cf-8bca-5322159afbb0\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8" Oct 31 04:29:55.536372 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.536349 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75bc7ee5-eb66-43cf-8bca-5322159afbb0-istio-data\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8\" (UID: \"75bc7ee5-eb66-43cf-8bca-5322159afbb0\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8" Oct 31 04:29:55.536537 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.536477 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75bc7ee5-eb66-43cf-8bca-5322159afbb0-credential-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8\" (UID: \"75bc7ee5-eb66-43cf-8bca-5322159afbb0\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8" Oct 31 04:29:55.538824 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.538800 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75bc7ee5-eb66-43cf-8bca-5322159afbb0-istio-envoy\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8\" (UID: \"75bc7ee5-eb66-43cf-8bca-5322159afbb0\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8" Oct 31 04:29:55.539239 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.539219 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75bc7ee5-eb66-43cf-8bca-5322159afbb0-istio-podinfo\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8\" (UID: \"75bc7ee5-eb66-43cf-8bca-5322159afbb0\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8" Oct 31 04:29:55.559940 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.559905 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75bc7ee5-eb66-43cf-8bca-5322159afbb0-istio-token\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8\" (UID: \"75bc7ee5-eb66-43cf-8bca-5322159afbb0\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8" Oct 31 04:29:55.560409 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.560383 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5pgf\" (UniqueName: \"kubernetes.io/projected/75bc7ee5-eb66-43cf-8bca-5322159afbb0-kube-api-access-n5pgf\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8\" (UID: \"75bc7ee5-eb66-43cf-8bca-5322159afbb0\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8" Oct 31 04:29:55.636956 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.636856 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d85d51e-dcd7-4e26-b793-70db760b09a8-istiod-ca-cert\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp\" (UID: \"6d85d51e-dcd7-4e26-b793-70db760b09a8\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp" Oct 31 04:29:55.636956 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.636924 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d85d51e-dcd7-4e26-b793-70db760b09a8-istio-token\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp\" (UID: \"6d85d51e-dcd7-4e26-b793-70db760b09a8\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp" Oct 31 04:29:55.637191 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.636996 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d85d51e-dcd7-4e26-b793-70db760b09a8-workload-certs\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp\" (UID: \"6d85d51e-dcd7-4e26-b793-70db760b09a8\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp" Oct 31 04:29:55.637191 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.637025 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjmvz\" (UniqueName: \"kubernetes.io/projected/6d85d51e-dcd7-4e26-b793-70db760b09a8-kube-api-access-bjmvz\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp\" (UID: \"6d85d51e-dcd7-4e26-b793-70db760b09a8\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp" Oct 31 04:29:55.637191 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.637153 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d85d51e-dcd7-4e26-b793-70db760b09a8-istio-data\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp\" (UID: \"6d85d51e-dcd7-4e26-b793-70db760b09a8\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp" Oct 31 04:29:55.637419 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.637209 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d85d51e-dcd7-4e26-b793-70db760b09a8-istio-podinfo\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp\" (UID: \"6d85d51e-dcd7-4e26-b793-70db760b09a8\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp" Oct 31 04:29:55.637419 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.637320 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d85d51e-dcd7-4e26-b793-70db760b09a8-workload-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp\" (UID: \"6d85d51e-dcd7-4e26-b793-70db760b09a8\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp" Oct 31 04:29:55.637419 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.637371 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d85d51e-dcd7-4e26-b793-70db760b09a8-credential-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp\" (UID: \"6d85d51e-dcd7-4e26-b793-70db760b09a8\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp" Oct 31 04:29:55.637419 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.637376 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d85d51e-dcd7-4e26-b793-70db760b09a8-workload-certs\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp\" (UID: \"6d85d51e-dcd7-4e26-b793-70db760b09a8\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp" Oct 31 04:29:55.637419 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.637414 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d85d51e-dcd7-4e26-b793-70db760b09a8-istio-envoy\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp\" (UID: \"6d85d51e-dcd7-4e26-b793-70db760b09a8\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp" Oct 31 04:29:55.637740 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.637568 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d85d51e-dcd7-4e26-b793-70db760b09a8-istio-data\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp\" (UID: \"6d85d51e-dcd7-4e26-b793-70db760b09a8\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp" Oct 31 04:29:55.637740 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.637646 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d85d51e-dcd7-4e26-b793-70db760b09a8-workload-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp\" (UID: \"6d85d51e-dcd7-4e26-b793-70db760b09a8\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp" Oct 31 04:29:55.637740 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.637692 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d85d51e-dcd7-4e26-b793-70db760b09a8-istiod-ca-cert\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp\" (UID: \"6d85d51e-dcd7-4e26-b793-70db760b09a8\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp" Oct 31 04:29:55.637893 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.637827 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d85d51e-dcd7-4e26-b793-70db760b09a8-credential-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp\" (UID: \"6d85d51e-dcd7-4e26-b793-70db760b09a8\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp" Oct 31 04:29:55.639806 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.639780 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d85d51e-dcd7-4e26-b793-70db760b09a8-istio-envoy\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp\" (UID: \"6d85d51e-dcd7-4e26-b793-70db760b09a8\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp" Oct 31 04:29:55.640070 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.640050 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d85d51e-dcd7-4e26-b793-70db760b09a8-istio-podinfo\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp\" (UID: \"6d85d51e-dcd7-4e26-b793-70db760b09a8\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp" Oct 31 04:29:55.660379 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.660295 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d85d51e-dcd7-4e26-b793-70db760b09a8-istio-token\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp\" (UID: \"6d85d51e-dcd7-4e26-b793-70db760b09a8\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp" Oct 31 04:29:55.660926 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.660904 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjmvz\" (UniqueName: \"kubernetes.io/projected/6d85d51e-dcd7-4e26-b793-70db760b09a8-kube-api-access-bjmvz\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp\" (UID: \"6d85d51e-dcd7-4e26-b793-70db760b09a8\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp" Oct 31 04:29:55.715743 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.715710 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8" Oct 31 04:29:55.723560 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:55.723525 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp" Oct 31 04:29:56.055601 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:56.055573 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8"] Oct 31 04:29:56.056295 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:29:56.056258 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75bc7ee5_eb66_43cf_8bca_5322159afbb0.slice/crio-64dce2f6a76177c2cacf300c29d87a39c6993417fa435ad058147ee410781371 WatchSource:0}: Error finding container 64dce2f6a76177c2cacf300c29d87a39c6993417fa435ad058147ee410781371: Status 404 returned error can't find the container with id 64dce2f6a76177c2cacf300c29d87a39c6993417fa435ad058147ee410781371 Oct 31 04:29:56.058478 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:29:56.058451 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d85d51e_dcd7_4e26_b793_70db760b09a8.slice/crio-57f400a0aca34b4ba068444b1d07aa1ae583252140a88b5172f4163fb2118263 WatchSource:0}: Error finding container 57f400a0aca34b4ba068444b1d07aa1ae583252140a88b5172f4163fb2118263: Status 404 returned error can't find the container with id 57f400a0aca34b4ba068444b1d07aa1ae583252140a88b5172f4163fb2118263 Oct 31 04:29:56.058617 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:56.058585 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:29:56.058665 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:56.058649 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:29:56.058701 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:56.058679 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:29:56.059500 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:56.059474 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp"] Oct 31 04:29:56.060874 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:56.060841 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:29:56.060977 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:56.060897 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:29:56.060977 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:56.060928 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:29:56.310466 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:56.310302 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp" event={"ID":"6d85d51e-dcd7-4e26-b793-70db760b09a8","Type":"ContainerStarted","Data":"a70d56685a0531cce0c8dd173dca0076ffe53830bc1eb3293caeb17e71a7ede8"} Oct 31 04:29:56.310466 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:56.310382 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp" event={"ID":"6d85d51e-dcd7-4e26-b793-70db760b09a8","Type":"ContainerStarted","Data":"57f400a0aca34b4ba068444b1d07aa1ae583252140a88b5172f4163fb2118263"} Oct 31 04:29:56.312429 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:56.312402 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8" event={"ID":"75bc7ee5-eb66-43cf-8bca-5322159afbb0","Type":"ContainerStarted","Data":"d82b084680c042ef089fa6846c20e46586cb28301fdb17e6e67d6ee7fcb37da2"} Oct 31 04:29:56.312571 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:56.312448 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8" event={"ID":"75bc7ee5-eb66-43cf-8bca-5322159afbb0","Type":"ContainerStarted","Data":"64dce2f6a76177c2cacf300c29d87a39c6993417fa435ad058147ee410781371"} Oct 31 04:29:56.362721 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:56.362667 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp" podStartSLOduration=1.362650877 podStartE2EDuration="1.362650877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:29:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:29:56.355378949 +0000 UTC m=+1551.111717013" watchObservedRunningTime="2026-10-31 04:29:56.362650877 +0000 UTC m=+1551.118988915" Oct 31 04:29:56.406622 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:56.406565 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8" podStartSLOduration=1.406546187 podStartE2EDuration="1.406546187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:29:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:29:56.404667153 +0000 UTC m=+1551.161005194" watchObservedRunningTime="2026-10-31 04:29:56.406546187 +0000 UTC m=+1551.162884227" Oct 31 04:29:56.718412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:56.716420 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8" Oct 31 04:29:56.725535 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:56.725408 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp" Oct 31 04:29:57.727477 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:57.727437 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 31 04:29:57.727980 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:57.727498 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8" podUID="75bc7ee5-eb66-43cf-8bca-5322159afbb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 31 04:29:57.729216 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:57.729182 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Oct 31 04:29:57.729373 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:57.729232 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp" podUID="6d85d51e-dcd7-4e26-b793-70db760b09a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Oct 31 04:29:58.731291 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:58.722985 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 31 04:29:58.731291 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:58.723114 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8" podUID="75bc7ee5-eb66-43cf-8bca-5322159afbb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 31 04:29:58.731291 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:58.729244 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Oct 31 04:29:58.731291 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:58.729291 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp" podUID="6d85d51e-dcd7-4e26-b793-70db760b09a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Oct 31 04:29:59.033569 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:59.033481 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-24cd29-5c5cb6887b-chxzf"] Oct 31 04:29:59.034063 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:59.034026 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-24cd29-5c5cb6887b-chxzf" podUID="530d712c-fe37-4d92-ae54-be36babd2335" containerName="payload-processing" containerID="cri-o://3d57b852ef55fddb48af92ed9f2d49a5f0deeb3341cf09a70bfee6be2405fb86" gracePeriod=30 Oct 31 04:29:59.039534 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:59.039499 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-24cd29-67c5899cb7-qfvwr"] Oct 31 04:29:59.039893 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:59.039868 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24cd29-67c5899cb7-qfvwr" podUID="37afe0fa-7748-4624-a783-4dd54c30b28d" containerName="payload-pre-processing" containerID="cri-o://f02eed02bd8f997a168e0972a6d01104f4b4b1fde279eb0cd0e76c554e1e92d7" gracePeriod=30 Oct 31 04:29:59.364355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:59.360608 5019 generic.go:356] "Generic (PLEG): container finished" podID="37afe0fa-7748-4624-a783-4dd54c30b28d" containerID="f02eed02bd8f997a168e0972a6d01104f4b4b1fde279eb0cd0e76c554e1e92d7" exitCode=0 Oct 31 04:29:59.364355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:59.360682 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24cd29-67c5899cb7-qfvwr" event={"ID":"37afe0fa-7748-4624-a783-4dd54c30b28d","Type":"ContainerDied","Data":"f02eed02bd8f997a168e0972a6d01104f4b4b1fde279eb0cd0e76c554e1e92d7"} Oct 31 04:29:59.364355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:59.362681 5019 generic.go:356] "Generic (PLEG): container finished" podID="530d712c-fe37-4d92-ae54-be36babd2335" containerID="3d57b852ef55fddb48af92ed9f2d49a5f0deeb3341cf09a70bfee6be2405fb86" exitCode=0 Oct 31 04:29:59.364355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:59.362880 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-24cd29-5c5cb6887b-chxzf" event={"ID":"530d712c-fe37-4d92-ae54-be36babd2335","Type":"ContainerDied","Data":"3d57b852ef55fddb48af92ed9f2d49a5f0deeb3341cf09a70bfee6be2405fb86"} Oct 31 04:29:59.723352 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:59.721744 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 31 04:29:59.723352 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:59.721804 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8" podUID="75bc7ee5-eb66-43cf-8bca-5322159afbb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 31 04:29:59.727357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:59.725537 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Oct 31 04:29:59.727357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:29:59.725598 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp" podUID="6d85d51e-dcd7-4e26-b793-70db760b09a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Oct 31 04:30:00.161789 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.161760 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890350-ptzdp"] Oct 31 04:30:00.238471 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.235594 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-24cd29-5c5cb6887b-chxzf" Oct 31 04:30:00.398540 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.319765 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b3ca79-29890350-zvb5r"] Oct 31 04:30:00.398540 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.320761 5019 cpu_manager.go:414] "containerMap: removing container" podUID="530d712c-fe37-4d92-ae54-be36babd2335" containerName="payload-processing" Oct 31 04:30:00.398540 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.320785 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="530d712c-fe37-4d92-ae54-be36babd2335" containerName="payload-processing" Oct 31 04:30:00.398540 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.320997 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="530d712c-fe37-4d92-ae54-be36babd2335" containerName="payload-processing" Oct 31 04:30:00.398540 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.330777 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890350-ptzdp" Oct 31 04:30:00.398540 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.353661 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/530d712c-fe37-4d92-ae54-be36babd2335-kube-api-access-4b624\" (UniqueName: \"kubernetes.io/projected/530d712c-fe37-4d92-ae54-be36babd2335-kube-api-access-4b624\") pod \"530d712c-fe37-4d92-ae54-be36babd2335\" (UID: \"530d712c-fe37-4d92-ae54-be36babd2335\") " Oct 31 04:30:00.398540 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.353708 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/530d712c-fe37-4d92-ae54-be36babd2335-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/530d712c-fe37-4d92-ae54-be36babd2335-plugins-config-volume\") pod \"530d712c-fe37-4d92-ae54-be36babd2335\" (UID: \"530d712c-fe37-4d92-ae54-be36babd2335\") " Oct 31 04:30:00.398540 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.361657 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/530d712c-fe37-4d92-ae54-be36babd2335-plugins-config-volume" pod "530d712c-fe37-4d92-ae54-be36babd2335" (UID: "530d712c-fe37-4d92-ae54-be36babd2335"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:30:00.398540 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.365385 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:30:00.398540 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.365515 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:30:00.398540 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.366098 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-edc450-29890350-h9rjv"] Oct 31 04:30:00.398540 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.366341 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b3ca79-29890350-zvb5r" Oct 31 04:30:00.398540 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.367320 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5t6t\" (UniqueName: \"kubernetes.io/projected/98654db8-3fbb-4228-bec4-c84ab84ef439-kube-api-access-n5t6t\") pod \"maas-api-key-cleanup-e2e-worker-w3-4c2387-29890350-ptzdp\" (UID: \"98654db8-3fbb-4228-bec4-c84ab84ef439\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890350-ptzdp" Oct 31 04:30:00.398540 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.367399 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98654db8-3fbb-4228-bec4-c84ab84ef439-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-4c2387-29890350-ptzdp\" (UID: \"98654db8-3fbb-4228-bec4-c84ab84ef439\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890350-ptzdp" Oct 31 04:30:00.398540 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.367894 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/530d712c-fe37-4d92-ae54-be36babd2335-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:00.399789 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.399731 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/530d712c-fe37-4d92-ae54-be36babd2335-kube-api-access-4b624" pod "530d712c-fe37-4d92-ae54-be36babd2335" (UID: "530d712c-fe37-4d92-ae54-be36babd2335"). InnerVolumeSpecName "kube-api-access-4b624". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:30:00.449814 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.435726 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:30:00.459404 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.459380 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-24cd29-5c5cb6887b-chxzf" Oct 31 04:30:00.465722 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.462240 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-24cd29-5c5cb6887b-chxzf" event={"ID":"530d712c-fe37-4d92-ae54-be36babd2335","Type":"ContainerDied","Data":"ef25d268f1e5bffd7e7fbfe539573a81105bb0ae7787af499bbd56d01e4bee17"} Oct 31 04:30:00.465722 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.462293 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d1d441-29890350-qwxpl"] Oct 31 04:30:00.504817 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.504774 5019 scope.go:122] "RemoveContainer" containerID="3d57b852ef55fddb48af92ed9f2d49a5f0deeb3341cf09a70bfee6be2405fb86" Oct 31 04:30:00.505276 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.505249 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-edc450-29890350-h9rjv" Oct 31 04:30:00.562368 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.555225 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98654db8-3fbb-4228-bec4-c84ab84ef439-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-4c2387-29890350-ptzdp\" (UID: \"98654db8-3fbb-4228-bec4-c84ab84ef439\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890350-ptzdp" Oct 31 04:30:00.638694 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.638664 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890350-ncpb2"] Oct 31 04:30:00.640240 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.640213 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nvbv\" (UniqueName: \"kubernetes.io/projected/1e8202ef-9e6c-4a81-9cea-49feea25ebc5-kube-api-access-5nvbv\") pod \"maas-api-key-cleanup-e2e-praxis-b-edc450-29890350-h9rjv\" (UID: \"1e8202ef-9e6c-4a81-9cea-49feea25ebc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-edc450-29890350-h9rjv" Oct 31 04:30:00.640424 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.640403 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e8202ef-9e6c-4a81-9cea-49feea25ebc5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-edc450-29890350-h9rjv\" (UID: \"1e8202ef-9e6c-4a81-9cea-49feea25ebc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-edc450-29890350-h9rjv" Oct 31 04:30:00.640605 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.640589 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5t6t\" (UniqueName: \"kubernetes.io/projected/98654db8-3fbb-4228-bec4-c84ab84ef439-kube-api-access-n5t6t\") pod \"maas-api-key-cleanup-e2e-worker-w3-4c2387-29890350-ptzdp\" (UID: \"98654db8-3fbb-4228-bec4-c84ab84ef439\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890350-ptzdp" Oct 31 04:30:00.640820 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.640243 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98654db8-3fbb-4228-bec4-c84ab84ef439-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-4c2387-29890350-ptzdp\" (UID: \"98654db8-3fbb-4228-bec4-c84ab84ef439\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890350-ptzdp" Oct 31 04:30:00.640888 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.640798 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b624\" (UniqueName: \"kubernetes.io/projected/530d712c-fe37-4d92-ae54-be36babd2335-kube-api-access-4b624\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:00.643032 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.642905 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d1d441-29890350-qwxpl" Oct 31 04:30:00.659047 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.659000 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24cd29-67c5899cb7-qfvwr" Oct 31 04:30:00.660039 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.660020 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7cf9ef-29890350-8zht5"] Oct 31 04:30:00.660959 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.660937 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890350-ncpb2" Oct 31 04:30:00.661161 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.661147 5019 cpu_manager.go:414] "containerMap: removing container" podUID="37afe0fa-7748-4624-a783-4dd54c30b28d" containerName="payload-pre-processing" Oct 31 04:30:00.661242 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.661162 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37afe0fa-7748-4624-a783-4dd54c30b28d" containerName="payload-pre-processing" Oct 31 04:30:00.661303 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.661294 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="37afe0fa-7748-4624-a783-4dd54c30b28d" containerName="payload-pre-processing" Oct 31 04:30:00.668741 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.668058 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5t6t\" (UniqueName: \"kubernetes.io/projected/98654db8-3fbb-4228-bec4-c84ab84ef439-kube-api-access-n5t6t\") pod \"maas-api-key-cleanup-e2e-worker-w3-4c2387-29890350-ptzdp\" (UID: \"98654db8-3fbb-4228-bec4-c84ab84ef439\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890350-ptzdp" Oct 31 04:30:00.679357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.673532 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890350-ptzdp"] Oct 31 04:30:00.679357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.673574 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-81c810-29890350-mth8b"] Oct 31 04:30:00.700420 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.695214 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7cf9ef-29890350-8zht5" Oct 31 04:30:00.732429 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.731609 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 31 04:30:00.732429 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.731662 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8" podUID="75bc7ee5-eb66-43cf-8bca-5322159afbb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 31 04:30:00.733707 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.733554 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Oct 31 04:30:00.733707 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.733599 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp" podUID="6d85d51e-dcd7-4e26-b793-70db760b09a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Oct 31 04:30:00.743650 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.743623 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37afe0fa-7748-4624-a783-4dd54c30b28d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37afe0fa-7748-4624-a783-4dd54c30b28d-plugins-config-volume\") pod \"37afe0fa-7748-4624-a783-4dd54c30b28d\" (UID: \"37afe0fa-7748-4624-a783-4dd54c30b28d\") " Oct 31 04:30:00.744394 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.744373 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37afe0fa-7748-4624-a783-4dd54c30b28d-kube-api-access-crqdv\" (UniqueName: \"kubernetes.io/projected/37afe0fa-7748-4624-a783-4dd54c30b28d-kube-api-access-crqdv\") pod \"37afe0fa-7748-4624-a783-4dd54c30b28d\" (UID: \"37afe0fa-7748-4624-a783-4dd54c30b28d\") " Oct 31 04:30:00.744607 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.743996 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-81c810-29890350-mth8b" Oct 31 04:30:00.744722 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.743929 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b3ca79-29890350-zvb5r"] Oct 31 04:30:00.744797 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.744774 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890350-42rwc"] Oct 31 04:30:00.745024 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.745004 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cr44\" (UniqueName: \"kubernetes.io/projected/1d8fff90-ca49-400c-9aa8-3e6676aa5428-kube-api-access-5cr44\") pod \"maas-api-key-cleanup-e2e-praxis-a-d1d441-29890350-qwxpl\" (UID: \"1d8fff90-ca49-400c-9aa8-3e6676aa5428\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d1d441-29890350-qwxpl" Oct 31 04:30:00.745643 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.745622 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8255b0b4-8c46-4190-b88b-c7a1da3b72b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-b3ca79-29890350-zvb5r\" (UID: \"8255b0b4-8c46-4190-b88b-c7a1da3b72b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b3ca79-29890350-zvb5r" Oct 31 04:30:00.745799 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.745781 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdxgk\" (UniqueName: \"kubernetes.io/projected/8cb938bd-e510-49fa-8183-a91749feeb5b-kube-api-access-tdxgk\") pod \"maas-api-key-cleanup-e2e-worker-w4-c4006d-29890350-ncpb2\" (UID: \"8cb938bd-e510-49fa-8183-a91749feeb5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890350-ncpb2" Oct 31 04:30:00.745929 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.745913 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzfdj\" (UniqueName: \"kubernetes.io/projected/c931f349-7638-4cf2-9329-a9fb5d21e6eb-kube-api-access-pzfdj\") pod \"maas-api-key-cleanup-e2e-praxis-b-7cf9ef-29890350-8zht5\" (UID: \"c931f349-7638-4cf2-9329-a9fb5d21e6eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7cf9ef-29890350-8zht5" Oct 31 04:30:00.746058 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.746040 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx2cf\" (UniqueName: \"kubernetes.io/projected/8255b0b4-8c46-4190-b88b-c7a1da3b72b7-kube-api-access-jx2cf\") pod \"maas-api-key-cleanup-e2e-praxis-a-b3ca79-29890350-zvb5r\" (UID: \"8255b0b4-8c46-4190-b88b-c7a1da3b72b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b3ca79-29890350-zvb5r" Oct 31 04:30:00.746274 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.746256 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d8fff90-ca49-400c-9aa8-3e6676aa5428-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-d1d441-29890350-qwxpl\" (UID: \"1d8fff90-ca49-400c-9aa8-3e6676aa5428\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d1d441-29890350-qwxpl" Oct 31 04:30:00.747558 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.747520 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cb938bd-e510-49fa-8183-a91749feeb5b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c4006d-29890350-ncpb2\" (UID: \"8cb938bd-e510-49fa-8183-a91749feeb5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890350-ncpb2" Oct 31 04:30:00.747749 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.747731 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nvbv\" (UniqueName: \"kubernetes.io/projected/1e8202ef-9e6c-4a81-9cea-49feea25ebc5-kube-api-access-5nvbv\") pod \"maas-api-key-cleanup-e2e-praxis-b-edc450-29890350-h9rjv\" (UID: \"1e8202ef-9e6c-4a81-9cea-49feea25ebc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-edc450-29890350-h9rjv" Oct 31 04:30:00.747870 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.747856 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e8202ef-9e6c-4a81-9cea-49feea25ebc5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-edc450-29890350-h9rjv\" (UID: \"1e8202ef-9e6c-4a81-9cea-49feea25ebc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-edc450-29890350-h9rjv" Oct 31 04:30:00.747992 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.745234 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37afe0fa-7748-4624-a783-4dd54c30b28d-plugins-config-volume" pod "37afe0fa-7748-4624-a783-4dd54c30b28d" (UID: "37afe0fa-7748-4624-a783-4dd54c30b28d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:30:00.748049 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.747104 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37afe0fa-7748-4624-a783-4dd54c30b28d-kube-api-access-crqdv" pod "37afe0fa-7748-4624-a783-4dd54c30b28d" (UID: "37afe0fa-7748-4624-a783-4dd54c30b28d"). InnerVolumeSpecName "kube-api-access-crqdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:30:00.748256 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.748239 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c931f349-7638-4cf2-9329-a9fb5d21e6eb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-7cf9ef-29890350-8zht5\" (UID: \"c931f349-7638-4cf2-9329-a9fb5d21e6eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7cf9ef-29890350-8zht5" Oct 31 04:30:00.748678 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.748660 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crqdv\" (UniqueName: \"kubernetes.io/projected/37afe0fa-7748-4624-a783-4dd54c30b28d-kube-api-access-crqdv\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:00.748804 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.748791 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37afe0fa-7748-4624-a783-4dd54c30b28d-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:00.748876 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.748708 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e8202ef-9e6c-4a81-9cea-49feea25ebc5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-edc450-29890350-h9rjv\" (UID: \"1e8202ef-9e6c-4a81-9cea-49feea25ebc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-edc450-29890350-h9rjv" Oct 31 04:30:00.750528 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.750212 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-edc450-29890350-h9rjv"] Oct 31 04:30:00.750528 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.750250 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890350-42rwc"] Oct 31 04:30:00.750528 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.750276 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890350-ncpb2"] Oct 31 04:30:00.750528 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.750371 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890350-42rwc" Oct 31 04:30:00.826353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.820096 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d1d441-29890350-qwxpl"] Oct 31 04:30:00.843165 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.838142 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nvbv\" (UniqueName: \"kubernetes.io/projected/1e8202ef-9e6c-4a81-9cea-49feea25ebc5-kube-api-access-5nvbv\") pod \"maas-api-key-cleanup-e2e-praxis-b-edc450-29890350-h9rjv\" (UID: \"1e8202ef-9e6c-4a81-9cea-49feea25ebc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-edc450-29890350-h9rjv" Oct 31 04:30:00.850091 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.850061 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c931f349-7638-4cf2-9329-a9fb5d21e6eb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-7cf9ef-29890350-8zht5\" (UID: \"c931f349-7638-4cf2-9329-a9fb5d21e6eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7cf9ef-29890350-8zht5" Oct 31 04:30:00.850255 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.850127 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cr44\" (UniqueName: \"kubernetes.io/projected/1d8fff90-ca49-400c-9aa8-3e6676aa5428-kube-api-access-5cr44\") pod \"maas-api-key-cleanup-e2e-praxis-a-d1d441-29890350-qwxpl\" (UID: \"1d8fff90-ca49-400c-9aa8-3e6676aa5428\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d1d441-29890350-qwxpl" Oct 31 04:30:00.850255 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.850165 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8255b0b4-8c46-4190-b88b-c7a1da3b72b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-b3ca79-29890350-zvb5r\" (UID: \"8255b0b4-8c46-4190-b88b-c7a1da3b72b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b3ca79-29890350-zvb5r" Oct 31 04:30:00.850255 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.850195 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdxgk\" (UniqueName: \"kubernetes.io/projected/8cb938bd-e510-49fa-8183-a91749feeb5b-kube-api-access-tdxgk\") pod \"maas-api-key-cleanup-e2e-worker-w4-c4006d-29890350-ncpb2\" (UID: \"8cb938bd-e510-49fa-8183-a91749feeb5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890350-ncpb2" Oct 31 04:30:00.850255 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.850221 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzfdj\" (UniqueName: \"kubernetes.io/projected/c931f349-7638-4cf2-9329-a9fb5d21e6eb-kube-api-access-pzfdj\") pod \"maas-api-key-cleanup-e2e-praxis-b-7cf9ef-29890350-8zht5\" (UID: \"c931f349-7638-4cf2-9329-a9fb5d21e6eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7cf9ef-29890350-8zht5" Oct 31 04:30:00.850508 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.850260 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx2cf\" (UniqueName: \"kubernetes.io/projected/8255b0b4-8c46-4190-b88b-c7a1da3b72b7-kube-api-access-jx2cf\") pod \"maas-api-key-cleanup-e2e-praxis-a-b3ca79-29890350-zvb5r\" (UID: \"8255b0b4-8c46-4190-b88b-c7a1da3b72b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b3ca79-29890350-zvb5r" Oct 31 04:30:00.850508 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.850375 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be7a6f4e-acc0-47dc-a962-a42eb116ab82-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890350-42rwc\" (UID: \"be7a6f4e-acc0-47dc-a962-a42eb116ab82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890350-42rwc" Oct 31 04:30:00.850508 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.850423 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfkps\" (UniqueName: \"kubernetes.io/projected/be7a6f4e-acc0-47dc-a962-a42eb116ab82-kube-api-access-cfkps\") pod \"maas-api-key-cleanup-29890350-42rwc\" (UID: \"be7a6f4e-acc0-47dc-a962-a42eb116ab82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890350-42rwc" Oct 31 04:30:00.850508 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.850463 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2lhz\" (UniqueName: \"kubernetes.io/projected/e9df73df-d6a4-4152-915e-2611831bcf90-kube-api-access-g2lhz\") pod \"maas-api-key-cleanup-e2e-praxis-b-81c810-29890350-mth8b\" (UID: \"e9df73df-d6a4-4152-915e-2611831bcf90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-81c810-29890350-mth8b" Oct 31 04:30:00.850959 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.850507 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d8fff90-ca49-400c-9aa8-3e6676aa5428-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-d1d441-29890350-qwxpl\" (UID: \"1d8fff90-ca49-400c-9aa8-3e6676aa5428\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d1d441-29890350-qwxpl" Oct 31 04:30:00.850959 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.850538 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cb938bd-e510-49fa-8183-a91749feeb5b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c4006d-29890350-ncpb2\" (UID: \"8cb938bd-e510-49fa-8183-a91749feeb5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890350-ncpb2" Oct 31 04:30:00.850959 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.850624 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9df73df-d6a4-4152-915e-2611831bcf90-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-81c810-29890350-mth8b\" (UID: \"e9df73df-d6a4-4152-915e-2611831bcf90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-81c810-29890350-mth8b" Oct 31 04:30:00.851153 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.851023 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c931f349-7638-4cf2-9329-a9fb5d21e6eb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-7cf9ef-29890350-8zht5\" (UID: \"c931f349-7638-4cf2-9329-a9fb5d21e6eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7cf9ef-29890350-8zht5" Oct 31 04:30:00.851324 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.851299 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8255b0b4-8c46-4190-b88b-c7a1da3b72b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-b3ca79-29890350-zvb5r\" (UID: \"8255b0b4-8c46-4190-b88b-c7a1da3b72b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b3ca79-29890350-zvb5r" Oct 31 04:30:00.851679 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.851657 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d8fff90-ca49-400c-9aa8-3e6676aa5428-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-d1d441-29890350-qwxpl\" (UID: \"1d8fff90-ca49-400c-9aa8-3e6676aa5428\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d1d441-29890350-qwxpl" Oct 31 04:30:00.851804 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.851786 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cb938bd-e510-49fa-8183-a91749feeb5b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c4006d-29890350-ncpb2\" (UID: \"8cb938bd-e510-49fa-8183-a91749feeb5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890350-ncpb2" Oct 31 04:30:00.852896 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.852877 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-81c810-29890350-mth8b"] Oct 31 04:30:00.861775 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.861755 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7cf9ef-29890350-8zht5"] Oct 31 04:30:00.916404 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.878521 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-24cd29-5c5cb6887b-chxzf"] Oct 31 04:30:00.927518 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.925648 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-24cd29-5c5cb6887b-chxzf"] Oct 31 04:30:00.936354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.934075 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdxgk\" (UniqueName: \"kubernetes.io/projected/8cb938bd-e510-49fa-8183-a91749feeb5b-kube-api-access-tdxgk\") pod \"maas-api-key-cleanup-e2e-worker-w4-c4006d-29890350-ncpb2\" (UID: \"8cb938bd-e510-49fa-8183-a91749feeb5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890350-ncpb2" Oct 31 04:30:00.954379 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.953221 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890350-ptzdp" Oct 31 04:30:00.954379 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.954187 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9df73df-d6a4-4152-915e-2611831bcf90-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-81c810-29890350-mth8b\" (UID: \"e9df73df-d6a4-4152-915e-2611831bcf90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-81c810-29890350-mth8b" Oct 31 04:30:00.954613 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.954542 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be7a6f4e-acc0-47dc-a962-a42eb116ab82-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890350-42rwc\" (UID: \"be7a6f4e-acc0-47dc-a962-a42eb116ab82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890350-42rwc" Oct 31 04:30:00.954613 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.954598 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfkps\" (UniqueName: \"kubernetes.io/projected/be7a6f4e-acc0-47dc-a962-a42eb116ab82-kube-api-access-cfkps\") pod \"maas-api-key-cleanup-29890350-42rwc\" (UID: \"be7a6f4e-acc0-47dc-a962-a42eb116ab82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890350-42rwc" Oct 31 04:30:00.954754 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.954637 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2lhz\" (UniqueName: \"kubernetes.io/projected/e9df73df-d6a4-4152-915e-2611831bcf90-kube-api-access-g2lhz\") pod \"maas-api-key-cleanup-e2e-praxis-b-81c810-29890350-mth8b\" (UID: \"e9df73df-d6a4-4152-915e-2611831bcf90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-81c810-29890350-mth8b" Oct 31 04:30:00.955052 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.955022 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9df73df-d6a4-4152-915e-2611831bcf90-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-81c810-29890350-mth8b\" (UID: \"e9df73df-d6a4-4152-915e-2611831bcf90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-81c810-29890350-mth8b" Oct 31 04:30:00.955726 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.955698 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be7a6f4e-acc0-47dc-a962-a42eb116ab82-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890350-42rwc\" (UID: \"be7a6f4e-acc0-47dc-a962-a42eb116ab82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890350-42rwc" Oct 31 04:30:00.957609 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.957560 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx2cf\" (UniqueName: \"kubernetes.io/projected/8255b0b4-8c46-4190-b88b-c7a1da3b72b7-kube-api-access-jx2cf\") pod \"maas-api-key-cleanup-e2e-praxis-a-b3ca79-29890350-zvb5r\" (UID: \"8255b0b4-8c46-4190-b88b-c7a1da3b72b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b3ca79-29890350-zvb5r" Oct 31 04:30:00.958152 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.958137 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cr44\" (UniqueName: \"kubernetes.io/projected/1d8fff90-ca49-400c-9aa8-3e6676aa5428-kube-api-access-5cr44\") pod \"maas-api-key-cleanup-e2e-praxis-a-d1d441-29890350-qwxpl\" (UID: \"1d8fff90-ca49-400c-9aa8-3e6676aa5428\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d1d441-29890350-qwxpl" Oct 31 04:30:00.958716 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:00.958606 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzfdj\" (UniqueName: \"kubernetes.io/projected/c931f349-7638-4cf2-9329-a9fb5d21e6eb-kube-api-access-pzfdj\") pod \"maas-api-key-cleanup-e2e-praxis-b-7cf9ef-29890350-8zht5\" (UID: \"c931f349-7638-4cf2-9329-a9fb5d21e6eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7cf9ef-29890350-8zht5" Oct 31 04:30:01.002290 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:01.002228 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2lhz\" (UniqueName: \"kubernetes.io/projected/e9df73df-d6a4-4152-915e-2611831bcf90-kube-api-access-g2lhz\") pod \"maas-api-key-cleanup-e2e-praxis-b-81c810-29890350-mth8b\" (UID: \"e9df73df-d6a4-4152-915e-2611831bcf90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-81c810-29890350-mth8b" Oct 31 04:30:01.016403 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:01.002654 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfkps\" (UniqueName: \"kubernetes.io/projected/be7a6f4e-acc0-47dc-a962-a42eb116ab82-kube-api-access-cfkps\") pod \"maas-api-key-cleanup-29890350-42rwc\" (UID: \"be7a6f4e-acc0-47dc-a962-a42eb116ab82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890350-42rwc" Oct 31 04:30:01.041977 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:01.041943 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-edc450-29890350-h9rjv" Oct 31 04:30:01.061527 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:01.061496 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d1d441-29890350-qwxpl" Oct 31 04:30:01.152313 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:01.152281 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890350-ncpb2" Oct 31 04:30:01.162747 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:01.162714 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7cf9ef-29890350-8zht5" Oct 31 04:30:01.255408 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:01.199501 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-81c810-29890350-mth8b" Oct 31 04:30:01.255408 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:01.213597 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890350-42rwc" Oct 31 04:30:01.259806 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:01.258362 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b3ca79-29890350-zvb5r" Oct 31 04:30:01.459442 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:01.459409 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="530d712c-fe37-4d92-ae54-be36babd2335" path="/var/lib/kubelet/pods/530d712c-fe37-4d92-ae54-be36babd2335/volumes" Oct 31 04:30:01.515292 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:01.514485 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24cd29-67c5899cb7-qfvwr" event={"ID":"37afe0fa-7748-4624-a783-4dd54c30b28d","Type":"ContainerDied","Data":"4272dbe7b0d2bd2b0d313fc5795a97d6680ca23d9a7f7a5075ffedf73326cb8c"} Oct 31 04:30:01.515292 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:01.514564 5019 scope.go:122] "RemoveContainer" containerID="f02eed02bd8f997a168e0972a6d01104f4b4b1fde279eb0cd0e76c554e1e92d7" Oct 31 04:30:01.515292 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:01.514760 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24cd29-67c5899cb7-qfvwr" Oct 31 04:30:01.655103 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:30:01.655064 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98654db8_3fbb_4228_bec4_c84ab84ef439.slice/crio-36e10812bbf084e60a54ac8ef39e3974b4f91aa60becd512beb9e359d75426ed WatchSource:0}: Error finding container 36e10812bbf084e60a54ac8ef39e3974b4f91aa60becd512beb9e359d75426ed: Status 404 returned error can't find the container with id 36e10812bbf084e60a54ac8ef39e3974b4f91aa60becd512beb9e359d75426ed Oct 31 04:30:01.725147 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:01.670580 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890350-ptzdp"] Oct 31 04:30:01.759522 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:01.743485 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 31 04:30:01.759522 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:01.743553 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8" podUID="75bc7ee5-eb66-43cf-8bca-5322159afbb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 31 04:30:01.759522 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:01.743970 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Oct 31 04:30:01.759522 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:01.744009 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp" podUID="6d85d51e-dcd7-4e26-b793-70db760b09a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Oct 31 04:30:01.759522 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:01.754781 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-24cd29-67c5899cb7-qfvwr"] Oct 31 04:30:01.760361 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:01.760268 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-24cd29-67c5899cb7-qfvwr"] Oct 31 04:30:01.955093 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:30:01.955062 5019 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37afe0fa_7748_4624_a783_4dd54c30b28d.slice\": RecentStats: unable to find data in memory cache]" Oct 31 04:30:02.131790 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:02.131761 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-edc450-29890350-h9rjv"] Oct 31 04:30:02.132248 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:30:02.132219 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e8202ef_9e6c_4a81_9cea_49feea25ebc5.slice/crio-a6fdb42b30583d8bd92a96c762923e7527cd30e6d821e2eb6a50a5765b263257 WatchSource:0}: Error finding container a6fdb42b30583d8bd92a96c762923e7527cd30e6d821e2eb6a50a5765b263257: Status 404 returned error can't find the container with id a6fdb42b30583d8bd92a96c762923e7527cd30e6d821e2eb6a50a5765b263257 Oct 31 04:30:02.166241 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:02.166208 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d1d441-29890350-qwxpl"] Oct 31 04:30:02.271862 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:02.271827 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890350-ncpb2"] Oct 31 04:30:02.326246 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:02.326217 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7cf9ef-29890350-8zht5"] Oct 31 04:30:02.326736 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:30:02.326705 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc931f349_7638_4cf2_9329_a9fb5d21e6eb.slice/crio-a618708f67bb9f14b3d256ee6c56c066e1e322a1084b6cbeedab78ca1a8b624b WatchSource:0}: Error finding container a618708f67bb9f14b3d256ee6c56c066e1e322a1084b6cbeedab78ca1a8b624b: Status 404 returned error can't find the container with id a618708f67bb9f14b3d256ee6c56c066e1e322a1084b6cbeedab78ca1a8b624b Oct 31 04:30:02.339486 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:02.339462 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890350-42rwc"] Oct 31 04:30:02.343504 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:02.342745 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b3ca79-29890350-zvb5r"] Oct 31 04:30:02.344487 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:30:02.344460 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe7a6f4e_acc0_47dc_a962_a42eb116ab82.slice/crio-0604b5459456e58dcb6846cdb506c1ad74e204efb372f63f9639ec6f5e5d5053 WatchSource:0}: Error finding container 0604b5459456e58dcb6846cdb506c1ad74e204efb372f63f9639ec6f5e5d5053: Status 404 returned error can't find the container with id 0604b5459456e58dcb6846cdb506c1ad74e204efb372f63f9639ec6f5e5d5053 Oct 31 04:30:02.348574 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:30:02.348546 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8255b0b4_8c46_4190_b88b_c7a1da3b72b7.slice/crio-2d55063dd93d7a45b617fab38b0d95d91d670ee2be2d380119e58887fd481b35 WatchSource:0}: Error finding container 2d55063dd93d7a45b617fab38b0d95d91d670ee2be2d380119e58887fd481b35: Status 404 returned error can't find the container with id 2d55063dd93d7a45b617fab38b0d95d91d670ee2be2d380119e58887fd481b35 Oct 31 04:30:02.436270 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:02.436241 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-81c810-29890350-mth8b"] Oct 31 04:30:02.480673 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:30:02.480639 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9df73df_d6a4_4152_915e_2611831bcf90.slice/crio-0e80e6fe9f1264c49988ba3b1f3cfbea4f5cd564199a759a5978e60971b31d8c WatchSource:0}: Error finding container 0e80e6fe9f1264c49988ba3b1f3cfbea4f5cd564199a759a5978e60971b31d8c: Status 404 returned error can't find the container with id 0e80e6fe9f1264c49988ba3b1f3cfbea4f5cd564199a759a5978e60971b31d8c Oct 31 04:30:02.551050 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:02.551017 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-edc450-29890350-h9rjv" event={"ID":"1e8202ef-9e6c-4a81-9cea-49feea25ebc5","Type":"ContainerStarted","Data":"a6fdb42b30583d8bd92a96c762923e7527cd30e6d821e2eb6a50a5765b263257"} Oct 31 04:30:02.552775 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:02.552739 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b3ca79-29890350-zvb5r" event={"ID":"8255b0b4-8c46-4190-b88b-c7a1da3b72b7","Type":"ContainerStarted","Data":"2d55063dd93d7a45b617fab38b0d95d91d670ee2be2d380119e58887fd481b35"} Oct 31 04:30:02.554691 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:02.554664 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890350-ptzdp" event={"ID":"98654db8-3fbb-4228-bec4-c84ab84ef439","Type":"ContainerStarted","Data":"cb6f75c738f33409c7160d06f31680332ee3730ee5f674a88226d293e802adc7"} Oct 31 04:30:02.554807 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:02.554712 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890350-ptzdp" event={"ID":"98654db8-3fbb-4228-bec4-c84ab84ef439","Type":"ContainerStarted","Data":"36e10812bbf084e60a54ac8ef39e3974b4f91aa60becd512beb9e359d75426ed"} Oct 31 04:30:02.557508 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:02.557483 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-81c810-29890350-mth8b" event={"ID":"e9df73df-d6a4-4152-915e-2611831bcf90","Type":"ContainerStarted","Data":"0e80e6fe9f1264c49988ba3b1f3cfbea4f5cd564199a759a5978e60971b31d8c"} Oct 31 04:30:02.560088 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:02.560035 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7cf9ef-29890350-8zht5" event={"ID":"c931f349-7638-4cf2-9329-a9fb5d21e6eb","Type":"ContainerStarted","Data":"a618708f67bb9f14b3d256ee6c56c066e1e322a1084b6cbeedab78ca1a8b624b"} Oct 31 04:30:02.562157 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:02.562129 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890350-ncpb2" event={"ID":"8cb938bd-e510-49fa-8183-a91749feeb5b","Type":"ContainerStarted","Data":"10df3c0d4985a801df34458c1ece7bc9c4140c77a7f7feb99535e59985d6112a"} Oct 31 04:30:02.571362 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:02.569897 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d1d441-29890350-qwxpl" event={"ID":"1d8fff90-ca49-400c-9aa8-3e6676aa5428","Type":"ContainerStarted","Data":"ccc6fc20e313c99f3cd8c01fdd6928e266148f0fa6f5df178b814478f2a6d490"} Oct 31 04:30:02.602238 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:02.602205 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890350-42rwc" event={"ID":"be7a6f4e-acc0-47dc-a962-a42eb116ab82","Type":"ContainerStarted","Data":"0604b5459456e58dcb6846cdb506c1ad74e204efb372f63f9639ec6f5e5d5053"} Oct 31 04:30:02.627183 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:02.627116 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890350-ptzdp" podStartSLOduration=2.627100971 podStartE2EDuration="2.627100971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:30:02.620428585 +0000 UTC m=+1557.376766624" watchObservedRunningTime="2026-10-31 04:30:02.627100971 +0000 UTC m=+1557.383439009" Oct 31 04:30:02.716984 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:02.716950 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 31 04:30:02.717161 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:02.717001 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8" podUID="75bc7ee5-eb66-43cf-8bca-5322159afbb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 31 04:30:02.724659 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:02.724630 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Oct 31 04:30:02.724780 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:02.724680 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp" podUID="6d85d51e-dcd7-4e26-b793-70db760b09a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Oct 31 04:30:03.426382 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:03.425933 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37afe0fa-7748-4624-a783-4dd54c30b28d" path="/var/lib/kubelet/pods/37afe0fa-7748-4624-a783-4dd54c30b28d/volumes" Oct 31 04:30:03.617744 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:03.617557 5019 generic.go:356] "Generic (PLEG): container finished" podID="8cb938bd-e510-49fa-8183-a91749feeb5b" containerID="4ba488397e6710765a3a92df8bd79186a2599c05247893f8e6b96515354086cf" exitCode=60 Oct 31 04:30:03.617744 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:03.617635 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890350-ncpb2" event={"ID":"8cb938bd-e510-49fa-8183-a91749feeb5b","Type":"ContainerDied","Data":"4ba488397e6710765a3a92df8bd79186a2599c05247893f8e6b96515354086cf"} Oct 31 04:30:03.618070 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:03.618023 5019 scope.go:122] "RemoveContainer" containerID="4ba488397e6710765a3a92df8bd79186a2599c05247893f8e6b96515354086cf" Oct 31 04:30:03.623593 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:03.622710 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d1d441-29890350-qwxpl" event={"ID":"1d8fff90-ca49-400c-9aa8-3e6676aa5428","Type":"ContainerStarted","Data":"487ddb6bee839bfae96ecaaf2306b372fd45b0f6a6897d1220af500204047835"} Oct 31 04:30:03.623593 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:03.623173 5019 scope.go:122] "RemoveContainer" containerID="487ddb6bee839bfae96ecaaf2306b372fd45b0f6a6897d1220af500204047835" Oct 31 04:30:03.633833 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:03.633012 5019 generic.go:356] "Generic (PLEG): container finished" podID="be7a6f4e-acc0-47dc-a962-a42eb116ab82" containerID="477023509da95ba0c4692db69f4b3216b86cb9a64fc09d1e41ab562dec4b7f0a" exitCode=60 Oct 31 04:30:03.633833 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:03.633187 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890350-42rwc" event={"ID":"be7a6f4e-acc0-47dc-a962-a42eb116ab82","Type":"ContainerDied","Data":"477023509da95ba0c4692db69f4b3216b86cb9a64fc09d1e41ab562dec4b7f0a"} Oct 31 04:30:03.633833 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:03.633547 5019 scope.go:122] "RemoveContainer" containerID="477023509da95ba0c4692db69f4b3216b86cb9a64fc09d1e41ab562dec4b7f0a" Oct 31 04:30:03.636661 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:03.635805 5019 generic.go:356] "Generic (PLEG): container finished" podID="1e8202ef-9e6c-4a81-9cea-49feea25ebc5" containerID="ebb1086cea6a742482af5182e13cf86635d82c079a0bd78a5cd7a18b0518cdcb" exitCode=60 Oct 31 04:30:03.636661 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:03.635922 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-edc450-29890350-h9rjv" event={"ID":"1e8202ef-9e6c-4a81-9cea-49feea25ebc5","Type":"ContainerDied","Data":"ebb1086cea6a742482af5182e13cf86635d82c079a0bd78a5cd7a18b0518cdcb"} Oct 31 04:30:03.636661 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:03.636303 5019 scope.go:122] "RemoveContainer" containerID="ebb1086cea6a742482af5182e13cf86635d82c079a0bd78a5cd7a18b0518cdcb" Oct 31 04:30:03.640548 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:03.639755 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b3ca79-29890350-zvb5r" event={"ID":"8255b0b4-8c46-4190-b88b-c7a1da3b72b7","Type":"ContainerStarted","Data":"2609b64426c7f7444f731f07f126aac5a885aec91d95e363ec0e5bad787172dc"} Oct 31 04:30:03.640548 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:03.640097 5019 scope.go:122] "RemoveContainer" containerID="2609b64426c7f7444f731f07f126aac5a885aec91d95e363ec0e5bad787172dc" Oct 31 04:30:03.643541 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:03.643508 5019 generic.go:356] "Generic (PLEG): container finished" podID="98654db8-3fbb-4228-bec4-c84ab84ef439" containerID="cb6f75c738f33409c7160d06f31680332ee3730ee5f674a88226d293e802adc7" exitCode=60 Oct 31 04:30:03.643693 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:03.643644 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890350-ptzdp" event={"ID":"98654db8-3fbb-4228-bec4-c84ab84ef439","Type":"ContainerDied","Data":"cb6f75c738f33409c7160d06f31680332ee3730ee5f674a88226d293e802adc7"} Oct 31 04:30:03.644068 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:03.644050 5019 scope.go:122] "RemoveContainer" containerID="cb6f75c738f33409c7160d06f31680332ee3730ee5f674a88226d293e802adc7" Oct 31 04:30:03.652967 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:03.652933 5019 generic.go:356] "Generic (PLEG): container finished" podID="e9df73df-d6a4-4152-915e-2611831bcf90" containerID="a94042d8ea08deb27df16eaa95abf2c0abd96e4c9b4b67804bc18b47b7ad0107" exitCode=60 Oct 31 04:30:03.653110 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:03.653053 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-81c810-29890350-mth8b" event={"ID":"e9df73df-d6a4-4152-915e-2611831bcf90","Type":"ContainerDied","Data":"a94042d8ea08deb27df16eaa95abf2c0abd96e4c9b4b67804bc18b47b7ad0107"} Oct 31 04:30:03.653465 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:03.653442 5019 scope.go:122] "RemoveContainer" containerID="a94042d8ea08deb27df16eaa95abf2c0abd96e4c9b4b67804bc18b47b7ad0107" Oct 31 04:30:03.689355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:03.688419 5019 generic.go:356] "Generic (PLEG): container finished" podID="c931f349-7638-4cf2-9329-a9fb5d21e6eb" containerID="123d2d8b003afa18ba0271b310d5ba04a6420cc958094702ab4f2c57c5adb2de" exitCode=60 Oct 31 04:30:03.689355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:03.688476 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7cf9ef-29890350-8zht5" event={"ID":"c931f349-7638-4cf2-9329-a9fb5d21e6eb","Type":"ContainerDied","Data":"123d2d8b003afa18ba0271b310d5ba04a6420cc958094702ab4f2c57c5adb2de"} Oct 31 04:30:03.689355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:03.688900 5019 scope.go:122] "RemoveContainer" containerID="123d2d8b003afa18ba0271b310d5ba04a6420cc958094702ab4f2c57c5adb2de" Oct 31 04:30:03.731197 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:03.731167 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8" Oct 31 04:30:03.732307 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:03.732236 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8" Oct 31 04:30:03.733180 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:03.733159 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8" Oct 31 04:30:03.797257 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:03.797226 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp" Oct 31 04:30:03.798713 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:03.798690 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp" Oct 31 04:30:03.802264 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:03.800740 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp" Oct 31 04:30:04.701295 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:04.701201 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7cf9ef-29890350-8zht5" event={"ID":"c931f349-7638-4cf2-9329-a9fb5d21e6eb","Type":"ContainerStarted","Data":"8f8b5698032446790a2c3ae794f6e0e628856bce67f44f7447df9409f83f97cd"} Oct 31 04:30:04.703323 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:04.703293 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890350-ncpb2" event={"ID":"8cb938bd-e510-49fa-8183-a91749feeb5b","Type":"ContainerStarted","Data":"469190e63125da3db7a648dc6f7e172275ad121eb5dffd8a6622f3d489647d16"} Oct 31 04:30:04.705368 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:04.705342 5019 generic.go:356] "Generic (PLEG): container finished" podID="1d8fff90-ca49-400c-9aa8-3e6676aa5428" containerID="487ddb6bee839bfae96ecaaf2306b372fd45b0f6a6897d1220af500204047835" exitCode=60 Oct 31 04:30:04.705478 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:04.705415 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d1d441-29890350-qwxpl" event={"ID":"1d8fff90-ca49-400c-9aa8-3e6676aa5428","Type":"ContainerDied","Data":"487ddb6bee839bfae96ecaaf2306b372fd45b0f6a6897d1220af500204047835"} Oct 31 04:30:04.705478 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:04.705454 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d1d441-29890350-qwxpl" event={"ID":"1d8fff90-ca49-400c-9aa8-3e6676aa5428","Type":"ContainerStarted","Data":"922e99bd510cad9e6647f4953546476f53b89fc7203eadf2edc041b5de73eec1"} Oct 31 04:30:04.707718 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:04.707690 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890350-42rwc" event={"ID":"be7a6f4e-acc0-47dc-a962-a42eb116ab82","Type":"ContainerStarted","Data":"d5bb378ea62e4fb4a14d3b9e3219569dc63304acb312a5dc687b5ddc69591d8a"} Oct 31 04:30:04.710519 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:04.710462 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-edc450-29890350-h9rjv" event={"ID":"1e8202ef-9e6c-4a81-9cea-49feea25ebc5","Type":"ContainerStarted","Data":"b8c769d15eca8bf05ed5ec286f15fc860006a743ddb87c12789ed7f4a84b66d9"} Oct 31 04:30:04.715081 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:04.715052 5019 generic.go:356] "Generic (PLEG): container finished" podID="8255b0b4-8c46-4190-b88b-c7a1da3b72b7" containerID="2609b64426c7f7444f731f07f126aac5a885aec91d95e363ec0e5bad787172dc" exitCode=60 Oct 31 04:30:04.715364 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:04.715293 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b3ca79-29890350-zvb5r" event={"ID":"8255b0b4-8c46-4190-b88b-c7a1da3b72b7","Type":"ContainerDied","Data":"2609b64426c7f7444f731f07f126aac5a885aec91d95e363ec0e5bad787172dc"} Oct 31 04:30:04.715472 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:04.715400 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b3ca79-29890350-zvb5r" event={"ID":"8255b0b4-8c46-4190-b88b-c7a1da3b72b7","Type":"ContainerStarted","Data":"7cf98fb48d86ce4be77ed1bd07ae39e390004317b69ca386ff1b1faf9bd6e510"} Oct 31 04:30:04.718083 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:04.718057 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890350-ptzdp" event={"ID":"98654db8-3fbb-4228-bec4-c84ab84ef439","Type":"ContainerStarted","Data":"5b778d6dc950e6bd6537290053bf4172f4df44e21523ed539e2433f658419a2c"} Oct 31 04:30:04.720093 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:04.720069 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-81c810-29890350-mth8b" event={"ID":"e9df73df-d6a4-4152-915e-2611831bcf90","Type":"ContainerStarted","Data":"ad3e68f8f114cbf1df116fdf94c8950b216b498fb35b5ad2957140b2dd9c5566"} Oct 31 04:30:04.742135 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:04.742041 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7cf9ef-29890350-8zht5" podStartSLOduration=4.7420289780000004 podStartE2EDuration="4.742028978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:30:04.74087877 +0000 UTC m=+1559.497216820" watchObservedRunningTime="2026-10-31 04:30:04.742028978 +0000 UTC m=+1559.498367059" Oct 31 04:30:04.791958 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:04.791583 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-edc450-29890350-h9rjv" podStartSLOduration=4.791562372 podStartE2EDuration="4.791562372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:30:04.789785479 +0000 UTC m=+1559.546123519" watchObservedRunningTime="2026-10-31 04:30:04.791562372 +0000 UTC m=+1559.547900411" Oct 31 04:30:04.830046 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:04.829980 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890350-42rwc" podStartSLOduration=4.829960841 podStartE2EDuration="4.829960841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:30:04.829258408 +0000 UTC m=+1559.585596465" watchObservedRunningTime="2026-10-31 04:30:04.829960841 +0000 UTC m=+1559.586298880" Oct 31 04:30:04.873251 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:04.871359 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890350-ncpb2" podStartSLOduration=4.871328253 podStartE2EDuration="4.871328253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:30:04.867993485 +0000 UTC m=+1559.624331528" watchObservedRunningTime="2026-10-31 04:30:04.871328253 +0000 UTC m=+1559.627666394" Oct 31 04:30:04.911791 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:04.911731 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b3ca79-29890350-zvb5r" podStartSLOduration=4.911713692 podStartE2EDuration="4.911713692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:30:04.909668479 +0000 UTC m=+1559.666006544" watchObservedRunningTime="2026-10-31 04:30:04.911713692 +0000 UTC m=+1559.668051732" Oct 31 04:30:04.990796 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:04.990739 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-81c810-29890350-mth8b" podStartSLOduration=4.990725522 podStartE2EDuration="4.990725522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:30:04.987107047 +0000 UTC m=+1559.743445086" watchObservedRunningTime="2026-10-31 04:30:04.990725522 +0000 UTC m=+1559.747063612" Oct 31 04:30:05.028810 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:05.028745 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d1d441-29890350-qwxpl" podStartSLOduration=5.028729853 podStartE2EDuration="5.028729853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:30:05.027641557 +0000 UTC m=+1559.783979602" watchObservedRunningTime="2026-10-31 04:30:05.028729853 +0000 UTC m=+1559.785067892" Oct 31 04:30:05.735311 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:05.735268 5019 generic.go:356] "Generic (PLEG): container finished" podID="8cb938bd-e510-49fa-8183-a91749feeb5b" containerID="469190e63125da3db7a648dc6f7e172275ad121eb5dffd8a6622f3d489647d16" exitCode=60 Oct 31 04:30:05.735830 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:05.735318 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890350-ncpb2" event={"ID":"8cb938bd-e510-49fa-8183-a91749feeb5b","Type":"ContainerDied","Data":"469190e63125da3db7a648dc6f7e172275ad121eb5dffd8a6622f3d489647d16"} Oct 31 04:30:05.735830 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:05.735407 5019 scope.go:122] "RemoveContainer" containerID="4ba488397e6710765a3a92df8bd79186a2599c05247893f8e6b96515354086cf" Oct 31 04:30:05.735830 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:05.735750 5019 scope.go:122] "RemoveContainer" containerID="469190e63125da3db7a648dc6f7e172275ad121eb5dffd8a6622f3d489647d16" Oct 31 04:30:05.736106 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:30:05.736081 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c4006d-29890350-ncpb2_odh-ai-gateway-infra(8cb938bd-e510-49fa-8183-a91749feeb5b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890350-ncpb2" podUID="8cb938bd-e510-49fa-8183-a91749feeb5b" Oct 31 04:30:05.737756 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:05.737732 5019 generic.go:356] "Generic (PLEG): container finished" podID="1d8fff90-ca49-400c-9aa8-3e6676aa5428" containerID="922e99bd510cad9e6647f4953546476f53b89fc7203eadf2edc041b5de73eec1" exitCode=60 Oct 31 04:30:05.737876 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:05.737800 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d1d441-29890350-qwxpl" event={"ID":"1d8fff90-ca49-400c-9aa8-3e6676aa5428","Type":"ContainerDied","Data":"922e99bd510cad9e6647f4953546476f53b89fc7203eadf2edc041b5de73eec1"} Oct 31 04:30:05.738138 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:05.738118 5019 scope.go:122] "RemoveContainer" containerID="922e99bd510cad9e6647f4953546476f53b89fc7203eadf2edc041b5de73eec1" Oct 31 04:30:05.738521 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:30:05.738500 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-d1d441-29890350-qwxpl_odh-ai-gateway-infra(1d8fff90-ca49-400c-9aa8-3e6676aa5428)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d1d441-29890350-qwxpl" podUID="1d8fff90-ca49-400c-9aa8-3e6676aa5428" Oct 31 04:30:05.739786 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:05.739767 5019 generic.go:356] "Generic (PLEG): container finished" podID="be7a6f4e-acc0-47dc-a962-a42eb116ab82" containerID="d5bb378ea62e4fb4a14d3b9e3219569dc63304acb312a5dc687b5ddc69591d8a" exitCode=60 Oct 31 04:30:05.739869 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:05.739846 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890350-42rwc" event={"ID":"be7a6f4e-acc0-47dc-a962-a42eb116ab82","Type":"ContainerDied","Data":"d5bb378ea62e4fb4a14d3b9e3219569dc63304acb312a5dc687b5ddc69591d8a"} Oct 31 04:30:05.740105 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:05.740091 5019 scope.go:122] "RemoveContainer" containerID="d5bb378ea62e4fb4a14d3b9e3219569dc63304acb312a5dc687b5ddc69591d8a" Oct 31 04:30:05.740474 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:30:05.740455 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890350-42rwc_odh-ai-gateway-infra(be7a6f4e-acc0-47dc-a962-a42eb116ab82)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890350-42rwc" podUID="be7a6f4e-acc0-47dc-a962-a42eb116ab82" Oct 31 04:30:05.741836 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:05.741815 5019 generic.go:356] "Generic (PLEG): container finished" podID="1e8202ef-9e6c-4a81-9cea-49feea25ebc5" containerID="b8c769d15eca8bf05ed5ec286f15fc860006a743ddb87c12789ed7f4a84b66d9" exitCode=60 Oct 31 04:30:05.741928 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:05.741904 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-edc450-29890350-h9rjv" event={"ID":"1e8202ef-9e6c-4a81-9cea-49feea25ebc5","Type":"ContainerDied","Data":"b8c769d15eca8bf05ed5ec286f15fc860006a743ddb87c12789ed7f4a84b66d9"} Oct 31 04:30:05.742135 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:05.742117 5019 scope.go:122] "RemoveContainer" containerID="b8c769d15eca8bf05ed5ec286f15fc860006a743ddb87c12789ed7f4a84b66d9" Oct 31 04:30:05.742468 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:30:05.742443 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-edc450-29890350-h9rjv_odh-ai-gateway-infra(1e8202ef-9e6c-4a81-9cea-49feea25ebc5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-edc450-29890350-h9rjv" podUID="1e8202ef-9e6c-4a81-9cea-49feea25ebc5" Oct 31 04:30:05.743856 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:05.743836 5019 generic.go:356] "Generic (PLEG): container finished" podID="8255b0b4-8c46-4190-b88b-c7a1da3b72b7" containerID="7cf98fb48d86ce4be77ed1bd07ae39e390004317b69ca386ff1b1faf9bd6e510" exitCode=60 Oct 31 04:30:05.743939 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:05.743908 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b3ca79-29890350-zvb5r" event={"ID":"8255b0b4-8c46-4190-b88b-c7a1da3b72b7","Type":"ContainerDied","Data":"7cf98fb48d86ce4be77ed1bd07ae39e390004317b69ca386ff1b1faf9bd6e510"} Oct 31 04:30:05.744142 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:05.744124 5019 scope.go:122] "RemoveContainer" containerID="7cf98fb48d86ce4be77ed1bd07ae39e390004317b69ca386ff1b1faf9bd6e510" Oct 31 04:30:05.744471 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:30:05.744448 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-b3ca79-29890350-zvb5r_odh-ai-gateway-infra(8255b0b4-8c46-4190-b88b-c7a1da3b72b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b3ca79-29890350-zvb5r" podUID="8255b0b4-8c46-4190-b88b-c7a1da3b72b7" Oct 31 04:30:05.745959 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:05.745941 5019 generic.go:356] "Generic (PLEG): container finished" podID="98654db8-3fbb-4228-bec4-c84ab84ef439" containerID="5b778d6dc950e6bd6537290053bf4172f4df44e21523ed539e2433f658419a2c" exitCode=60 Oct 31 04:30:05.746053 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:05.746041 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890350-ptzdp" event={"ID":"98654db8-3fbb-4228-bec4-c84ab84ef439","Type":"ContainerDied","Data":"5b778d6dc950e6bd6537290053bf4172f4df44e21523ed539e2433f658419a2c"} Oct 31 04:30:05.746293 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:05.746279 5019 scope.go:122] "RemoveContainer" containerID="5b778d6dc950e6bd6537290053bf4172f4df44e21523ed539e2433f658419a2c" Oct 31 04:30:05.746535 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:30:05.746516 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-4c2387-29890350-ptzdp_odh-ai-gateway-infra(98654db8-3fbb-4228-bec4-c84ab84ef439)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890350-ptzdp" podUID="98654db8-3fbb-4228-bec4-c84ab84ef439" Oct 31 04:30:05.748049 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:05.748016 5019 generic.go:356] "Generic (PLEG): container finished" podID="e9df73df-d6a4-4152-915e-2611831bcf90" containerID="ad3e68f8f114cbf1df116fdf94c8950b216b498fb35b5ad2957140b2dd9c5566" exitCode=60 Oct 31 04:30:05.748134 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:05.748063 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-81c810-29890350-mth8b" event={"ID":"e9df73df-d6a4-4152-915e-2611831bcf90","Type":"ContainerDied","Data":"ad3e68f8f114cbf1df116fdf94c8950b216b498fb35b5ad2957140b2dd9c5566"} Oct 31 04:30:05.748384 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:05.748366 5019 scope.go:122] "RemoveContainer" containerID="ad3e68f8f114cbf1df116fdf94c8950b216b498fb35b5ad2957140b2dd9c5566" Oct 31 04:30:05.748688 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:30:05.748662 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-81c810-29890350-mth8b_odh-ai-gateway-infra(e9df73df-d6a4-4152-915e-2611831bcf90)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-81c810-29890350-mth8b" podUID="e9df73df-d6a4-4152-915e-2611831bcf90" Oct 31 04:30:05.750154 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:05.750135 5019 generic.go:356] "Generic (PLEG): container finished" podID="c931f349-7638-4cf2-9329-a9fb5d21e6eb" containerID="8f8b5698032446790a2c3ae794f6e0e628856bce67f44f7447df9409f83f97cd" exitCode=60 Oct 31 04:30:05.750237 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:05.750160 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7cf9ef-29890350-8zht5" event={"ID":"c931f349-7638-4cf2-9329-a9fb5d21e6eb","Type":"ContainerDied","Data":"8f8b5698032446790a2c3ae794f6e0e628856bce67f44f7447df9409f83f97cd"} Oct 31 04:30:05.750507 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:05.750483 5019 scope.go:122] "RemoveContainer" containerID="8f8b5698032446790a2c3ae794f6e0e628856bce67f44f7447df9409f83f97cd" Oct 31 04:30:05.750761 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:30:05.750743 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-7cf9ef-29890350-8zht5_odh-ai-gateway-infra(c931f349-7638-4cf2-9329-a9fb5d21e6eb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7cf9ef-29890350-8zht5" podUID="c931f349-7638-4cf2-9329-a9fb5d21e6eb" Oct 31 04:30:05.752323 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:05.752307 5019 scope.go:122] "RemoveContainer" containerID="487ddb6bee839bfae96ecaaf2306b372fd45b0f6a6897d1220af500204047835" Oct 31 04:30:05.767405 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:05.767310 5019 scope.go:122] "RemoveContainer" containerID="477023509da95ba0c4692db69f4b3216b86cb9a64fc09d1e41ab562dec4b7f0a" Oct 31 04:30:05.784067 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:05.784036 5019 scope.go:122] "RemoveContainer" containerID="ebb1086cea6a742482af5182e13cf86635d82c079a0bd78a5cd7a18b0518cdcb" Oct 31 04:30:05.801478 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:05.801454 5019 scope.go:122] "RemoveContainer" containerID="2609b64426c7f7444f731f07f126aac5a885aec91d95e363ec0e5bad787172dc" Oct 31 04:30:05.821773 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:05.821740 5019 scope.go:122] "RemoveContainer" containerID="cb6f75c738f33409c7160d06f31680332ee3730ee5f674a88226d293e802adc7" Oct 31 04:30:05.840396 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:05.840312 5019 scope.go:122] "RemoveContainer" containerID="a94042d8ea08deb27df16eaa95abf2c0abd96e4c9b4b67804bc18b47b7ad0107" Oct 31 04:30:05.862546 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:05.862071 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn"] Oct 31 04:30:05.862546 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:05.862312 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn" podUID="042cfe03-4fa9-4b8d-a3e3-73283e3149d4" containerName="istio-proxy" containerID="cri-o://43fa69192de2b18cc6cf75c40589998c7b44762e0ca976123955ae8ccdcb6f8d" gracePeriod=30 Oct 31 04:30:05.871830 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:05.869297 5019 scope.go:122] "RemoveContainer" containerID="123d2d8b003afa18ba0271b310d5ba04a6420cc958094702ab4f2c57c5adb2de" Oct 31 04:30:06.340712 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:06.340674 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-edc450-29890350-h9rjv"] Oct 31 04:30:06.766389 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:06.766357 5019 scope.go:122] "RemoveContainer" containerID="ad3e68f8f114cbf1df116fdf94c8950b216b498fb35b5ad2957140b2dd9c5566" Oct 31 04:30:06.767122 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:30:06.766737 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-81c810-29890350-mth8b_odh-ai-gateway-infra(e9df73df-d6a4-4152-915e-2611831bcf90)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-81c810-29890350-mth8b" podUID="e9df73df-d6a4-4152-915e-2611831bcf90" Oct 31 04:30:06.770189 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:06.770165 5019 scope.go:122] "RemoveContainer" containerID="8f8b5698032446790a2c3ae794f6e0e628856bce67f44f7447df9409f83f97cd" Oct 31 04:30:06.770707 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:30:06.770679 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-7cf9ef-29890350-8zht5_odh-ai-gateway-infra(c931f349-7638-4cf2-9329-a9fb5d21e6eb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7cf9ef-29890350-8zht5" podUID="c931f349-7638-4cf2-9329-a9fb5d21e6eb" Oct 31 04:30:06.773770 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:06.773749 5019 scope.go:122] "RemoveContainer" containerID="469190e63125da3db7a648dc6f7e172275ad121eb5dffd8a6622f3d489647d16" Oct 31 04:30:06.774236 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:30:06.774211 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c4006d-29890350-ncpb2_odh-ai-gateway-infra(8cb938bd-e510-49fa-8183-a91749feeb5b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890350-ncpb2" podUID="8cb938bd-e510-49fa-8183-a91749feeb5b" Oct 31 04:30:06.779673 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:06.779653 5019 scope.go:122] "RemoveContainer" containerID="922e99bd510cad9e6647f4953546476f53b89fc7203eadf2edc041b5de73eec1" Oct 31 04:30:06.780119 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:30:06.780095 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-d1d441-29890350-qwxpl_odh-ai-gateway-infra(1d8fff90-ca49-400c-9aa8-3e6676aa5428)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d1d441-29890350-qwxpl" podUID="1d8fff90-ca49-400c-9aa8-3e6676aa5428" Oct 31 04:30:06.788062 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:06.787324 5019 scope.go:122] "RemoveContainer" containerID="d5bb378ea62e4fb4a14d3b9e3219569dc63304acb312a5dc687b5ddc69591d8a" Oct 31 04:30:06.788442 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:30:06.788418 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890350-42rwc_odh-ai-gateway-infra(be7a6f4e-acc0-47dc-a962-a42eb116ab82)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890350-42rwc" podUID="be7a6f4e-acc0-47dc-a962-a42eb116ab82" Oct 31 04:30:06.795975 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:06.795957 5019 scope.go:122] "RemoveContainer" containerID="b8c769d15eca8bf05ed5ec286f15fc860006a743ddb87c12789ed7f4a84b66d9" Oct 31 04:30:06.797306 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:30:06.797280 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-edc450-29890350-h9rjv_odh-ai-gateway-infra(1e8202ef-9e6c-4a81-9cea-49feea25ebc5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-edc450-29890350-h9rjv" podUID="1e8202ef-9e6c-4a81-9cea-49feea25ebc5" Oct 31 04:30:06.797525 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:06.797509 5019 scope.go:122] "RemoveContainer" containerID="7cf98fb48d86ce4be77ed1bd07ae39e390004317b69ca386ff1b1faf9bd6e510" Oct 31 04:30:06.797892 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:30:06.797871 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-b3ca79-29890350-zvb5r_odh-ai-gateway-infra(8255b0b4-8c46-4190-b88b-c7a1da3b72b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b3ca79-29890350-zvb5r" podUID="8255b0b4-8c46-4190-b88b-c7a1da3b72b7" Oct 31 04:30:06.798005 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:06.797992 5019 scope.go:122] "RemoveContainer" containerID="5b778d6dc950e6bd6537290053bf4172f4df44e21523ed539e2433f658419a2c" Oct 31 04:30:06.798411 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:30:06.798265 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-4c2387-29890350-ptzdp_odh-ai-gateway-infra(98654db8-3fbb-4228-bec4-c84ab84ef439)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890350-ptzdp" podUID="98654db8-3fbb-4228-bec4-c84ab84ef439" Oct 31 04:30:07.456499 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:07.456466 5019 patch_prober.go:29] interesting pod/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 04:30:07.456657 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:07.456531 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn" podUID="042cfe03-4fa9-4b8d-a3e3-73283e3149d4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 04:30:07.836820 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:07.836786 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f"] Oct 31 04:30:07.849351 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:07.845765 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f" Oct 31 04:30:07.879377 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:07.873799 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f"] Oct 31 04:30:07.879377 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:07.874418 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-e37d27-openshift-default-dockercfg-djj7j\"" Oct 31 04:30:07.938359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:07.938096 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/116c8b42-2784-4f46-b534-ea35159e3cac-istio-podinfo\") pod \"e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f\" (UID: \"116c8b42-2784-4f46-b534-ea35159e3cac\") " pod="openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f" Oct 31 04:30:07.938359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:07.938176 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/116c8b42-2784-4f46-b534-ea35159e3cac-istio-envoy\") pod \"e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f\" (UID: \"116c8b42-2784-4f46-b534-ea35159e3cac\") " pod="openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f" Oct 31 04:30:07.938359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:07.938249 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/116c8b42-2784-4f46-b534-ea35159e3cac-istio-data\") pod \"e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f\" (UID: \"116c8b42-2784-4f46-b534-ea35159e3cac\") " pod="openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f" Oct 31 04:30:07.938359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:07.938288 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/116c8b42-2784-4f46-b534-ea35159e3cac-workload-socket\") pod \"e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f\" (UID: \"116c8b42-2784-4f46-b534-ea35159e3cac\") " pod="openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f" Oct 31 04:30:07.938359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:07.938356 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/116c8b42-2784-4f46-b534-ea35159e3cac-istiod-ca-cert\") pod \"e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f\" (UID: \"116c8b42-2784-4f46-b534-ea35159e3cac\") " pod="openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f" Oct 31 04:30:07.938764 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:07.938420 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/116c8b42-2784-4f46-b534-ea35159e3cac-istio-token\") pod \"e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f\" (UID: \"116c8b42-2784-4f46-b534-ea35159e3cac\") " pod="openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f" Oct 31 04:30:07.938764 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:07.938452 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr9m8\" (UniqueName: \"kubernetes.io/projected/116c8b42-2784-4f46-b534-ea35159e3cac-kube-api-access-dr9m8\") pod \"e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f\" (UID: \"116c8b42-2784-4f46-b534-ea35159e3cac\") " pod="openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f" Oct 31 04:30:07.938764 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:07.938486 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/116c8b42-2784-4f46-b534-ea35159e3cac-credential-socket\") pod \"e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f\" (UID: \"116c8b42-2784-4f46-b534-ea35159e3cac\") " pod="openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f" Oct 31 04:30:07.938764 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:07.938533 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/116c8b42-2784-4f46-b534-ea35159e3cac-workload-certs\") pod \"e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f\" (UID: \"116c8b42-2784-4f46-b534-ea35159e3cac\") " pod="openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f" Oct 31 04:30:08.096475 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.093523 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/116c8b42-2784-4f46-b534-ea35159e3cac-istio-data\") pod \"e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f\" (UID: \"116c8b42-2784-4f46-b534-ea35159e3cac\") " pod="openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f" Oct 31 04:30:08.096475 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.093639 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/116c8b42-2784-4f46-b534-ea35159e3cac-workload-socket\") pod \"e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f\" (UID: \"116c8b42-2784-4f46-b534-ea35159e3cac\") " pod="openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f" Oct 31 04:30:08.096475 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.093692 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/116c8b42-2784-4f46-b534-ea35159e3cac-istiod-ca-cert\") pod \"e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f\" (UID: \"116c8b42-2784-4f46-b534-ea35159e3cac\") " pod="openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f" Oct 31 04:30:08.096475 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.093741 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/116c8b42-2784-4f46-b534-ea35159e3cac-istio-token\") pod \"e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f\" (UID: \"116c8b42-2784-4f46-b534-ea35159e3cac\") " pod="openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f" Oct 31 04:30:08.096475 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.093785 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr9m8\" (UniqueName: \"kubernetes.io/projected/116c8b42-2784-4f46-b534-ea35159e3cac-kube-api-access-dr9m8\") pod \"e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f\" (UID: \"116c8b42-2784-4f46-b534-ea35159e3cac\") " pod="openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f" Oct 31 04:30:08.096475 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.093816 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/116c8b42-2784-4f46-b534-ea35159e3cac-credential-socket\") pod \"e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f\" (UID: \"116c8b42-2784-4f46-b534-ea35159e3cac\") " pod="openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f" Oct 31 04:30:08.096475 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.093889 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/116c8b42-2784-4f46-b534-ea35159e3cac-workload-certs\") pod \"e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f\" (UID: \"116c8b42-2784-4f46-b534-ea35159e3cac\") " pod="openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f" Oct 31 04:30:08.096475 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.094051 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/116c8b42-2784-4f46-b534-ea35159e3cac-istio-podinfo\") pod \"e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f\" (UID: \"116c8b42-2784-4f46-b534-ea35159e3cac\") " pod="openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f" Oct 31 04:30:08.096475 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.094125 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/116c8b42-2784-4f46-b534-ea35159e3cac-istio-envoy\") pod \"e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f\" (UID: \"116c8b42-2784-4f46-b534-ea35159e3cac\") " pod="openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f" Oct 31 04:30:08.100402 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.097760 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/116c8b42-2784-4f46-b534-ea35159e3cac-istio-data\") pod \"e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f\" (UID: \"116c8b42-2784-4f46-b534-ea35159e3cac\") " pod="openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f" Oct 31 04:30:08.100402 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.098003 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/116c8b42-2784-4f46-b534-ea35159e3cac-workload-socket\") pod \"e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f\" (UID: \"116c8b42-2784-4f46-b534-ea35159e3cac\") " pod="openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f" Oct 31 04:30:08.100402 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.098772 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/116c8b42-2784-4f46-b534-ea35159e3cac-istiod-ca-cert\") pod \"e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f\" (UID: \"116c8b42-2784-4f46-b534-ea35159e3cac\") " pod="openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f" Oct 31 04:30:08.100402 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.099030 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/116c8b42-2784-4f46-b534-ea35159e3cac-credential-socket\") pod \"e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f\" (UID: \"116c8b42-2784-4f46-b534-ea35159e3cac\") " pod="openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f" Oct 31 04:30:08.100402 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.099594 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/116c8b42-2784-4f46-b534-ea35159e3cac-workload-certs\") pod \"e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f\" (UID: \"116c8b42-2784-4f46-b534-ea35159e3cac\") " pod="openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f" Oct 31 04:30:08.117359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.115843 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/116c8b42-2784-4f46-b534-ea35159e3cac-istio-podinfo\") pod \"e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f\" (UID: \"116c8b42-2784-4f46-b534-ea35159e3cac\") " pod="openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f" Oct 31 04:30:08.140650 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.138943 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/116c8b42-2784-4f46-b534-ea35159e3cac-istio-envoy\") pod \"e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f\" (UID: \"116c8b42-2784-4f46-b534-ea35159e3cac\") " pod="openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f" Oct 31 04:30:08.148844 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.142732 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr9m8\" (UniqueName: \"kubernetes.io/projected/116c8b42-2784-4f46-b534-ea35159e3cac-kube-api-access-dr9m8\") pod \"e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f\" (UID: \"116c8b42-2784-4f46-b534-ea35159e3cac\") " pod="openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f" Oct 31 04:30:08.148844 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.148745 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/116c8b42-2784-4f46-b534-ea35159e3cac-istio-token\") pod \"e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f\" (UID: \"116c8b42-2784-4f46-b534-ea35159e3cac\") " pod="openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f" Oct 31 04:30:08.183018 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.182973 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz"] Oct 31 04:30:08.183697 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.183665 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f" Oct 31 04:30:08.191941 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.191912 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz" Oct 31 04:30:08.193482 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.193399 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-edc450-29890350-h9rjv" Oct 31 04:30:08.214190 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.214135 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz"] Oct 31 04:30:08.214863 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.214832 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-d1d441-openshift-default-dockercfg-vd7l5\"" Oct 31 04:30:08.304586 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.304477 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e8202ef-9e6c-4a81-9cea-49feea25ebc5-kube-api-access-5nvbv\" (UniqueName: \"kubernetes.io/projected/1e8202ef-9e6c-4a81-9cea-49feea25ebc5-kube-api-access-5nvbv\") pod \"1e8202ef-9e6c-4a81-9cea-49feea25ebc5\" (UID: \"1e8202ef-9e6c-4a81-9cea-49feea25ebc5\") " Oct 31 04:30:08.304586 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.304564 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e8202ef-9e6c-4a81-9cea-49feea25ebc5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e8202ef-9e6c-4a81-9cea-49feea25ebc5-maas-api-service-ca\") pod \"1e8202ef-9e6c-4a81-9cea-49feea25ebc5\" (UID: \"1e8202ef-9e6c-4a81-9cea-49feea25ebc5\") " Oct 31 04:30:08.305034 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.305005 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-workload-certs\") pod \"e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz\" (UID: \"ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd\") " pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz" Oct 31 04:30:08.305126 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.305061 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-istio-token\") pod \"e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz\" (UID: \"ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd\") " pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz" Oct 31 04:30:08.305187 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.305173 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-workload-socket\") pod \"e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz\" (UID: \"ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd\") " pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz" Oct 31 04:30:08.305246 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.305202 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-istiod-ca-cert\") pod \"e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz\" (UID: \"ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd\") " pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz" Oct 31 04:30:08.305246 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.305216 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e8202ef-9e6c-4a81-9cea-49feea25ebc5-maas-api-service-ca" pod "1e8202ef-9e6c-4a81-9cea-49feea25ebc5" (UID: "1e8202ef-9e6c-4a81-9cea-49feea25ebc5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:30:08.305374 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.305356 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-istio-podinfo\") pod \"e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz\" (UID: \"ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd\") " pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz" Oct 31 04:30:08.305509 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.305392 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-istio-data\") pod \"e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz\" (UID: \"ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd\") " pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz" Oct 31 04:30:08.305509 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.305432 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9fc6\" (UniqueName: \"kubernetes.io/projected/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-kube-api-access-j9fc6\") pod \"e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz\" (UID: \"ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd\") " pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz" Oct 31 04:30:08.305628 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.305521 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-credential-socket\") pod \"e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz\" (UID: \"ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd\") " pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz" Oct 31 04:30:08.305628 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.305571 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-istio-envoy\") pod \"e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz\" (UID: \"ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd\") " pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz" Oct 31 04:30:08.305712 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.305664 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e8202ef-9e6c-4a81-9cea-49feea25ebc5-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:08.318830 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.318765 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e8202ef-9e6c-4a81-9cea-49feea25ebc5-kube-api-access-5nvbv" pod "1e8202ef-9e6c-4a81-9cea-49feea25ebc5" (UID: "1e8202ef-9e6c-4a81-9cea-49feea25ebc5"). InnerVolumeSpecName "kube-api-access-5nvbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:30:08.407310 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.407273 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-credential-socket\") pod \"e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz\" (UID: \"ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd\") " pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz" Oct 31 04:30:08.407467 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.407375 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-istio-envoy\") pod \"e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz\" (UID: \"ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd\") " pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz" Oct 31 04:30:08.407467 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.407455 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-workload-certs\") pod \"e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz\" (UID: \"ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd\") " pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz" Oct 31 04:30:08.407558 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.407494 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-istio-token\") pod \"e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz\" (UID: \"ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd\") " pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz" Oct 31 04:30:08.407643 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.407623 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-workload-socket\") pod \"e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz\" (UID: \"ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd\") " pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz" Oct 31 04:30:08.407714 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.407658 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-istiod-ca-cert\") pod \"e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz\" (UID: \"ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd\") " pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz" Oct 31 04:30:08.407749 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.407729 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-credential-socket\") pod \"e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz\" (UID: \"ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd\") " pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz" Oct 31 04:30:08.407786 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.407753 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-istio-podinfo\") pod \"e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz\" (UID: \"ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd\") " pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz" Oct 31 04:30:08.407834 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.407788 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-istio-data\") pod \"e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz\" (UID: \"ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd\") " pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz" Oct 31 04:30:08.407834 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.407829 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9fc6\" (UniqueName: \"kubernetes.io/projected/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-kube-api-access-j9fc6\") pod \"e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz\" (UID: \"ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd\") " pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz" Oct 31 04:30:08.407981 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.407964 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nvbv\" (UniqueName: \"kubernetes.io/projected/1e8202ef-9e6c-4a81-9cea-49feea25ebc5-kube-api-access-5nvbv\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:08.408119 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.408080 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-workload-certs\") pod \"e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz\" (UID: \"ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd\") " pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz" Oct 31 04:30:08.408535 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.408264 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-istio-data\") pod \"e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz\" (UID: \"ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd\") " pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz" Oct 31 04:30:08.408535 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.408504 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-workload-socket\") pod \"e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz\" (UID: \"ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd\") " pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz" Oct 31 04:30:08.408656 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.408623 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-istiod-ca-cert\") pod \"e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz\" (UID: \"ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd\") " pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz" Oct 31 04:30:08.410650 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.410604 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-istio-podinfo\") pod \"e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz\" (UID: \"ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd\") " pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz" Oct 31 04:30:08.410650 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.410608 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-istio-envoy\") pod \"e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz\" (UID: \"ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd\") " pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz" Oct 31 04:30:08.431738 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.431705 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-istio-token\") pod \"e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz\" (UID: \"ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd\") " pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz" Oct 31 04:30:08.431906 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.431856 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9fc6\" (UniqueName: \"kubernetes.io/projected/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-kube-api-access-j9fc6\") pod \"e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz\" (UID: \"ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd\") " pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz" Oct 31 04:30:08.554393 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:30:08.554360 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod116c8b42_2784_4f46_b534_ea35159e3cac.slice/crio-8864375a14a4ec3883250543080344ce696cebfbe58acff4192d85d1bb650410 WatchSource:0}: Error finding container 8864375a14a4ec3883250543080344ce696cebfbe58acff4192d85d1bb650410: Status 404 returned error can't find the container with id 8864375a14a4ec3883250543080344ce696cebfbe58acff4192d85d1bb650410 Oct 31 04:30:08.554781 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.554758 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f"] Oct 31 04:30:08.557010 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.556974 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:30:08.557108 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.557054 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:30:08.557108 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.557095 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:30:08.572944 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.572916 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz" Oct 31 04:30:08.818601 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.818554 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f" event={"ID":"116c8b42-2784-4f46-b534-ea35159e3cac","Type":"ContainerStarted","Data":"a1f2a6d8ebb597400c5d9b8695bc42b7c367b59a6adb50e297ccef83a682760a"} Oct 31 04:30:08.818777 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.818623 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f" event={"ID":"116c8b42-2784-4f46-b534-ea35159e3cac","Type":"ContainerStarted","Data":"8864375a14a4ec3883250543080344ce696cebfbe58acff4192d85d1bb650410"} Oct 31 04:30:08.821129 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.821102 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-edc450-29890350-h9rjv" Oct 31 04:30:08.821129 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.821116 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-edc450-29890350-h9rjv" event={"ID":"1e8202ef-9e6c-4a81-9cea-49feea25ebc5","Type":"ContainerDied","Data":"a6fdb42b30583d8bd92a96c762923e7527cd30e6d821e2eb6a50a5765b263257"} Oct 31 04:30:08.821387 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.821196 5019 scope.go:122] "RemoveContainer" containerID="b8c769d15eca8bf05ed5ec286f15fc860006a743ddb87c12789ed7f4a84b66d9" Oct 31 04:30:08.865316 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.865041 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f" podStartSLOduration=1.865025561 podStartE2EDuration="1.865025561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:30:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:30:08.863741959 +0000 UTC m=+1563.620079999" watchObservedRunningTime="2026-10-31 04:30:08.865025561 +0000 UTC m=+1563.621363694" Oct 31 04:30:08.872205 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:30:08.872165 5019 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e8202ef_9e6c_4a81_9cea_49feea25ebc5.slice/crio-a6fdb42b30583d8bd92a96c762923e7527cd30e6d821e2eb6a50a5765b263257\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e8202ef_9e6c_4a81_9cea_49feea25ebc5.slice\": RecentStats: unable to find data in memory cache]" Oct 31 04:30:08.902456 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.902411 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-edc450-29890350-h9rjv"] Oct 31 04:30:08.910865 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.910810 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-edc450-29890350-h9rjv"] Oct 31 04:30:08.914298 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.914267 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz"] Oct 31 04:30:08.918593 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:30:08.918551 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab0ba839_9c46_40b2_b4ff_7cfc7249e8cd.slice/crio-406aa3896d3bef7d44e30014e54461d08c38204cf12b35f1c2f175ff3ba5aa2e WatchSource:0}: Error finding container 406aa3896d3bef7d44e30014e54461d08c38204cf12b35f1c2f175ff3ba5aa2e: Status 404 returned error can't find the container with id 406aa3896d3bef7d44e30014e54461d08c38204cf12b35f1c2f175ff3ba5aa2e Oct 31 04:30:08.921095 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.921056 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:30:08.921228 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.921148 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:30:08.921228 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:08.921191 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:30:09.184855 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:09.184821 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f" Oct 31 04:30:09.191741 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:09.191706 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f" Oct 31 04:30:09.412889 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:09.412804 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e8202ef-9e6c-4a81-9cea-49feea25ebc5" path="/var/lib/kubelet/pods/1e8202ef-9e6c-4a81-9cea-49feea25ebc5/volumes" Oct 31 04:30:09.832299 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:09.832255 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz" event={"ID":"ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd","Type":"ContainerStarted","Data":"22ef539616a122de2c4f5d74777ee579535cc7ed627f7e303c3fbed63ea48178"} Oct 31 04:30:09.832493 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:09.832362 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz" event={"ID":"ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd","Type":"ContainerStarted","Data":"406aa3896d3bef7d44e30014e54461d08c38204cf12b35f1c2f175ff3ba5aa2e"} Oct 31 04:30:09.832691 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:09.832673 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f" Oct 31 04:30:09.833608 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:09.833586 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f" Oct 31 04:30:09.939249 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:09.939196 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz" podStartSLOduration=1.939178708 podStartE2EDuration="1.939178708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:30:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:30:09.889716093 +0000 UTC m=+1564.646054133" watchObservedRunningTime="2026-10-31 04:30:09.939178708 +0000 UTC m=+1564.695516748" Oct 31 04:30:10.655435 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:10.655398 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz" Oct 31 04:30:10.725355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:10.722700 5019 patch_prober.go:29] interesting pod/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 31 04:30:10.725355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:10.722786 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz" podUID="ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 31 04:30:11.537920 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:11.518071 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:30:11.544977 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:11.538673 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" podUID="643433d8-418e-4102-8bc3-cd544932614a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 04:30:11.544977 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:11.539541 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 31 04:30:11.583042 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:11.581299 5019 patch_prober.go:29] interesting pod/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 31 04:30:11.583831 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:11.583272 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz" podUID="ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 31 04:30:11.854626 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:11.794913 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" podUID="e0f4b8c2-14f4-4052-bf54-dd65bf59e413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 04:30:11.987023 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:11.854873 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 31 04:30:12.068523 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:11.904809 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" podUID="dccd15a9-47f5-4bd1-b21b-9d5ca8b05845" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 31 04:30:12.068523 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:11.960417 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" podUID="8e2c7041-ad52-49f4-8b55-d16691f4938f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 31 04:30:12.068523 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:11.986287 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" podUID="b4ced3ab-41e4-4c54-93eb-95eddaae992d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 04:30:12.116621 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:12.116488 5019 generic.go:356] "Generic (PLEG): container finished" podID="042cfe03-4fa9-4b8d-a3e3-73283e3149d4" containerID="43fa69192de2b18cc6cf75c40589998c7b44762e0ca976123955ae8ccdcb6f8d" exitCode=0 Oct 31 04:30:12.117350 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:12.117229 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn" event={"ID":"042cfe03-4fa9-4b8d-a3e3-73283e3149d4","Type":"ContainerDied","Data":"43fa69192de2b18cc6cf75c40589998c7b44762e0ca976123955ae8ccdcb6f8d"} Oct 31 04:30:12.117350 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:12.117290 5019 scope.go:122] "RemoveContainer" containerID="1fef3af8cfb9cc006f1e2898170d11ff3e6ddb8f422c34535fb185ce588ad78b" Oct 31 04:30:12.486482 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:30:12.486435 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fef3af8cfb9cc006f1e2898170d11ff3e6ddb8f422c34535fb185ce588ad78b\": container with ID starting with 1fef3af8cfb9cc006f1e2898170d11ff3e6ddb8f422c34535fb185ce588ad78b not found: ID does not exist" containerID="1fef3af8cfb9cc006f1e2898170d11ff3e6ddb8f422c34535fb185ce588ad78b" Oct 31 04:30:12.516289 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:12.515929 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn" Oct 31 04:30:12.546357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:12.543424 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:30:12.546357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:12.544969 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" Oct 31 04:30:12.709107 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:12.709025 5019 patch_prober.go:29] interesting pod/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 31 04:30:12.709539 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:12.709504 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz" podUID="ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 31 04:30:12.710543 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:12.710515 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-istio-podinfo\") pod \"042cfe03-4fa9-4b8d-a3e3-73283e3149d4\" (UID: \"042cfe03-4fa9-4b8d-a3e3-73283e3149d4\") " Oct 31 04:30:12.710716 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:12.710695 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-kube-api-access-4tnt9\" (UniqueName: \"kubernetes.io/projected/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-kube-api-access-4tnt9\") pod \"042cfe03-4fa9-4b8d-a3e3-73283e3149d4\" (UID: \"042cfe03-4fa9-4b8d-a3e3-73283e3149d4\") " Oct 31 04:30:12.710908 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:12.710890 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-credential-socket\") pod \"042cfe03-4fa9-4b8d-a3e3-73283e3149d4\" (UID: \"042cfe03-4fa9-4b8d-a3e3-73283e3149d4\") " Oct 31 04:30:12.711419 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:12.711400 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-istiod-ca-cert\") pod \"042cfe03-4fa9-4b8d-a3e3-73283e3149d4\" (UID: \"042cfe03-4fa9-4b8d-a3e3-73283e3149d4\") " Oct 31 04:30:12.711583 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:12.711565 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-istio-data\") pod \"042cfe03-4fa9-4b8d-a3e3-73283e3149d4\" (UID: \"042cfe03-4fa9-4b8d-a3e3-73283e3149d4\") " Oct 31 04:30:12.716436 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:12.711700 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-workload-socket\") pod \"042cfe03-4fa9-4b8d-a3e3-73283e3149d4\" (UID: \"042cfe03-4fa9-4b8d-a3e3-73283e3149d4\") " Oct 31 04:30:12.719891 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:12.719868 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-workload-certs\") pod \"042cfe03-4fa9-4b8d-a3e3-73283e3149d4\" (UID: \"042cfe03-4fa9-4b8d-a3e3-73283e3149d4\") " Oct 31 04:30:12.720083 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:12.720066 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-istio-envoy\") pod \"042cfe03-4fa9-4b8d-a3e3-73283e3149d4\" (UID: \"042cfe03-4fa9-4b8d-a3e3-73283e3149d4\") " Oct 31 04:30:12.720201 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:12.720185 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-istio-token\" (UniqueName: \"kubernetes.io/projected/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-istio-token\") pod \"042cfe03-4fa9-4b8d-a3e3-73283e3149d4\" (UID: \"042cfe03-4fa9-4b8d-a3e3-73283e3149d4\") " Oct 31 04:30:12.727488 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:12.719417 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-workload-socket" pod "042cfe03-4fa9-4b8d-a3e3-73283e3149d4" (UID: "042cfe03-4fa9-4b8d-a3e3-73283e3149d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:30:12.727831 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:12.727810 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-workload-certs" pod "042cfe03-4fa9-4b8d-a3e3-73283e3149d4" (UID: "042cfe03-4fa9-4b8d-a3e3-73283e3149d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:30:12.756459 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:12.733403 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-credential-socket" pod "042cfe03-4fa9-4b8d-a3e3-73283e3149d4" (UID: "042cfe03-4fa9-4b8d-a3e3-73283e3149d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:30:12.758055 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:12.757776 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-istiod-ca-cert" pod "042cfe03-4fa9-4b8d-a3e3-73283e3149d4" (UID: "042cfe03-4fa9-4b8d-a3e3-73283e3149d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:30:12.758753 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:12.758238 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-istio-data" pod "042cfe03-4fa9-4b8d-a3e3-73283e3149d4" (UID: "042cfe03-4fa9-4b8d-a3e3-73283e3149d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:30:12.837383 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:12.825537 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:12.837383 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:12.825568 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:12.837383 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:12.825584 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:12.837383 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:12.825599 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:12.837383 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:12.825611 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:12.923811 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:12.923773 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-istio-envoy" pod "042cfe03-4fa9-4b8d-a3e3-73283e3149d4" (UID: "042cfe03-4fa9-4b8d-a3e3-73283e3149d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:30:12.997051 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:12.996934 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:13.013047 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:13.013013 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-istio-podinfo" pod "042cfe03-4fa9-4b8d-a3e3-73283e3149d4" (UID: "042cfe03-4fa9-4b8d-a3e3-73283e3149d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:30:13.013355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:13.013317 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-kube-api-access-4tnt9" pod "042cfe03-4fa9-4b8d-a3e3-73283e3149d4" (UID: "042cfe03-4fa9-4b8d-a3e3-73283e3149d4"). InnerVolumeSpecName "kube-api-access-4tnt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:30:13.013519 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:13.013493 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" podUID="e0f4b8c2-14f4-4052-bf54-dd65bf59e413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 04:30:13.014294 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:13.014274 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" Oct 31 04:30:13.025380 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:13.016455 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-istio-token" pod "042cfe03-4fa9-4b8d-a3e3-73283e3149d4" (UID: "042cfe03-4fa9-4b8d-a3e3-73283e3149d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:30:13.025380 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:13.019499 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" podUID="e0f4b8c2-14f4-4052-bf54-dd65bf59e413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": context deadline exceeded" Oct 31 04:30:13.078356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:13.063417 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" podUID="b4ced3ab-41e4-4c54-93eb-95eddaae992d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:30:13.078356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:13.063792 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:30:13.078356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:13.064061 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" podUID="dccd15a9-47f5-4bd1-b21b-9d5ca8b05845" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 31 04:30:13.078356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:13.064304 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" podUID="8e2c7041-ad52-49f4-8b55-d16691f4938f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:30:13.078356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:13.064696 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" Oct 31 04:30:13.078356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:13.064752 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" Oct 31 04:30:13.078356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:13.064789 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" Oct 31 04:30:13.078356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:13.064823 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" Oct 31 04:30:13.185558 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:13.185524 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:13.203223 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:13.203198 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:13.221434 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:13.221395 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tnt9\" (UniqueName: \"kubernetes.io/projected/042cfe03-4fa9-4b8d-a3e3-73283e3149d4-kube-api-access-4tnt9\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:13.301141 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:13.300066 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" podUID="dccd15a9-47f5-4bd1-b21b-9d5ca8b05845" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": context deadline exceeded" Oct 31 04:30:13.378380 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:13.361009 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn" event={"ID":"042cfe03-4fa9-4b8d-a3e3-73283e3149d4","Type":"ContainerDied","Data":"822820128f068c8a8c9379522827541da1721f597d94e4d1f0ba2101d9a37260"} Oct 31 04:30:13.388217 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:13.388189 5019 scope.go:122] "RemoveContainer" containerID="43fa69192de2b18cc6cf75c40589998c7b44762e0ca976123955ae8ccdcb6f8d" Oct 31 04:30:13.388560 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:13.380279 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn" Oct 31 04:30:13.519415 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:13.514951 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:30:13.554305 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:13.554206 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" podUID="643433d8-418e-4102-8bc3-cd544932614a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:30:13.688376 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:13.654206 5019 patch_prober.go:29] interesting pod/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 31 04:30:13.688376 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:13.654265 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz" podUID="ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 31 04:30:13.688376 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:13.663627 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" Oct 31 04:30:13.865418 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:13.839384 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn"] Oct 31 04:30:13.865418 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:13.844798 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-24cd29-openshift-default-6c7dfb89d9-plhmn"] Oct 31 04:30:14.019968 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:14.019924 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" podUID="e0f4b8c2-14f4-4052-bf54-dd65bf59e413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:30:14.098358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:14.079508 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" podUID="8e2c7041-ad52-49f4-8b55-d16691f4938f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 31 04:30:14.098358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:14.079577 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" podUID="b4ced3ab-41e4-4c54-93eb-95eddaae992d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Oct 31 04:30:14.120418 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:14.079599 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 31 04:30:14.120418 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:14.079629 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" podUID="dccd15a9-47f5-4bd1-b21b-9d5ca8b05845" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 31 04:30:14.120418 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:14.080456 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:30:14.164355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:14.163075 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" Oct 31 04:30:14.458560 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:14.458472 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" Oct 31 04:30:14.511377 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:14.511071 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" Oct 31 04:30:14.616356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:14.615911 5019 patch_prober.go:29] interesting pod/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 31 04:30:14.616356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:14.616009 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz" podUID="ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 31 04:30:14.662788 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:14.662754 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" Oct 31 04:30:14.666850 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:14.666820 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" Oct 31 04:30:14.942036 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:14.942000 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-e37d27-c8547cdf9-v856h"] Oct 31 04:30:14.943440 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:14.943402 5019 cpu_manager.go:414] "containerMap: removing container" podUID="1e8202ef-9e6c-4a81-9cea-49feea25ebc5" containerName="cleanup" Oct 31 04:30:14.943615 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:14.943599 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e8202ef-9e6c-4a81-9cea-49feea25ebc5" containerName="cleanup" Oct 31 04:30:14.943731 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:14.943720 5019 cpu_manager.go:414] "containerMap: removing container" podUID="042cfe03-4fa9-4b8d-a3e3-73283e3149d4" containerName="istio-proxy" Oct 31 04:30:14.943815 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:14.943803 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="042cfe03-4fa9-4b8d-a3e3-73283e3149d4" containerName="istio-proxy" Oct 31 04:30:14.943929 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:14.943918 5019 cpu_manager.go:414] "containerMap: removing container" podUID="1e8202ef-9e6c-4a81-9cea-49feea25ebc5" containerName="cleanup" Oct 31 04:30:14.944012 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:14.944002 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e8202ef-9e6c-4a81-9cea-49feea25ebc5" containerName="cleanup" Oct 31 04:30:14.944999 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:14.944317 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="042cfe03-4fa9-4b8d-a3e3-73283e3149d4" containerName="istio-proxy" Oct 31 04:30:14.945110 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:14.945097 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e8202ef-9e6c-4a81-9cea-49feea25ebc5" containerName="cleanup" Oct 31 04:30:14.954564 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:14.954535 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-e37d27-c8547cdf9-v856h"] Oct 31 04:30:14.954689 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:14.954663 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e37d27-c8547cdf9-v856h" Oct 31 04:30:15.022353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:15.021672 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-e37d27\"" Oct 31 04:30:15.071351 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:15.063604 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-e37d27-dockercfg-9xd7h\"" Oct 31 04:30:15.140410 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:15.127816 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cghxj\" (UniqueName: \"kubernetes.io/projected/c0570273-666f-4052-9713-00813a62c291-kube-api-access-cghxj\") pod \"payload-pre-processing-e2e-praxis-a-e37d27-c8547cdf9-v856h\" (UID: \"c0570273-666f-4052-9713-00813a62c291\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e37d27-c8547cdf9-v856h" Oct 31 04:30:15.140410 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:15.128082 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0570273-666f-4052-9713-00813a62c291-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-e37d27-c8547cdf9-v856h\" (UID: \"c0570273-666f-4052-9713-00813a62c291\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e37d27-c8547cdf9-v856h" Oct 31 04:30:15.192559 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:15.192441 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-e37d27-6ddbbfbc47-gb52r"] Oct 31 04:30:15.250359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:15.247867 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0570273-666f-4052-9713-00813a62c291-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-e37d27-c8547cdf9-v856h\" (UID: \"c0570273-666f-4052-9713-00813a62c291\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e37d27-c8547cdf9-v856h" Oct 31 04:30:15.250359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:15.247960 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cghxj\" (UniqueName: \"kubernetes.io/projected/c0570273-666f-4052-9713-00813a62c291-kube-api-access-cghxj\") pod \"payload-pre-processing-e2e-praxis-a-e37d27-c8547cdf9-v856h\" (UID: \"c0570273-666f-4052-9713-00813a62c291\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e37d27-c8547cdf9-v856h" Oct 31 04:30:15.250359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:15.249505 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0570273-666f-4052-9713-00813a62c291-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-e37d27-c8547cdf9-v856h\" (UID: \"c0570273-666f-4052-9713-00813a62c291\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e37d27-c8547cdf9-v856h" Oct 31 04:30:15.251571 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:15.251545 5019 cpu_manager.go:414] "containerMap: removing container" podUID="042cfe03-4fa9-4b8d-a3e3-73283e3149d4" containerName="istio-proxy" Oct 31 04:30:15.265692 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:15.263397 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="042cfe03-4fa9-4b8d-a3e3-73283e3149d4" containerName="istio-proxy" Oct 31 04:30:15.265692 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:15.263687 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="042cfe03-4fa9-4b8d-a3e3-73283e3149d4" containerName="istio-proxy" Oct 31 04:30:15.265692 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:15.263703 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e8202ef-9e6c-4a81-9cea-49feea25ebc5" containerName="cleanup" Oct 31 04:30:15.268796 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:15.268774 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-e37d27-6ddbbfbc47-gb52r"] Oct 31 04:30:15.268951 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:15.268895 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-e37d27-6ddbbfbc47-gb52r" Oct 31 04:30:15.343357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:15.342751 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cghxj\" (UniqueName: \"kubernetes.io/projected/c0570273-666f-4052-9713-00813a62c291-kube-api-access-cghxj\") pod \"payload-pre-processing-e2e-praxis-a-e37d27-c8547cdf9-v856h\" (UID: \"c0570273-666f-4052-9713-00813a62c291\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e37d27-c8547cdf9-v856h" Oct 31 04:30:15.446031 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:15.445832 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="042cfe03-4fa9-4b8d-a3e3-73283e3149d4" path="/var/lib/kubelet/pods/042cfe03-4fa9-4b8d-a3e3-73283e3149d4/volumes" Oct 31 04:30:15.450809 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:15.450774 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvjdn\" (UniqueName: \"kubernetes.io/projected/82b999d8-91cd-4021-8fc0-c2e61ae1a8f2-kube-api-access-xvjdn\") pod \"payload-processing-e2e-praxis-a-e37d27-6ddbbfbc47-gb52r\" (UID: \"82b999d8-91cd-4021-8fc0-c2e61ae1a8f2\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-e37d27-6ddbbfbc47-gb52r" Oct 31 04:30:15.450949 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:15.450830 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82b999d8-91cd-4021-8fc0-c2e61ae1a8f2-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-e37d27-6ddbbfbc47-gb52r\" (UID: \"82b999d8-91cd-4021-8fc0-c2e61ae1a8f2\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-e37d27-6ddbbfbc47-gb52r" Oct 31 04:30:15.543730 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:15.540275 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e37d27-c8547cdf9-v856h" Oct 31 04:30:15.551899 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:15.551859 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvjdn\" (UniqueName: \"kubernetes.io/projected/82b999d8-91cd-4021-8fc0-c2e61ae1a8f2-kube-api-access-xvjdn\") pod \"payload-processing-e2e-praxis-a-e37d27-6ddbbfbc47-gb52r\" (UID: \"82b999d8-91cd-4021-8fc0-c2e61ae1a8f2\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-e37d27-6ddbbfbc47-gb52r" Oct 31 04:30:15.552032 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:15.551938 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82b999d8-91cd-4021-8fc0-c2e61ae1a8f2-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-e37d27-6ddbbfbc47-gb52r\" (UID: \"82b999d8-91cd-4021-8fc0-c2e61ae1a8f2\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-e37d27-6ddbbfbc47-gb52r" Oct 31 04:30:15.553717 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:15.553697 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82b999d8-91cd-4021-8fc0-c2e61ae1a8f2-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-e37d27-6ddbbfbc47-gb52r\" (UID: \"82b999d8-91cd-4021-8fc0-c2e61ae1a8f2\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-e37d27-6ddbbfbc47-gb52r" Oct 31 04:30:15.601772 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:15.601293 5019 patch_prober.go:29] interesting pod/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 31 04:30:15.601772 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:15.601371 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz" podUID="ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 31 04:30:15.605100 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:15.605039 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvjdn\" (UniqueName: \"kubernetes.io/projected/82b999d8-91cd-4021-8fc0-c2e61ae1a8f2-kube-api-access-xvjdn\") pod \"payload-processing-e2e-praxis-a-e37d27-6ddbbfbc47-gb52r\" (UID: \"82b999d8-91cd-4021-8fc0-c2e61ae1a8f2\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-e37d27-6ddbbfbc47-gb52r" Oct 31 04:30:15.658004 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:15.657903 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-e37d27-6ddbbfbc47-gb52r" Oct 31 04:30:16.368223 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:16.367071 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-e37d27-c8547cdf9-v856h"] Oct 31 04:30:16.440211 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:16.440107 5019 scope.go:122] "RemoveContainer" containerID="922e99bd510cad9e6647f4953546476f53b89fc7203eadf2edc041b5de73eec1" Oct 31 04:30:16.449010 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:16.448789 5019 scope.go:122] "RemoveContainer" containerID="d5bb378ea62e4fb4a14d3b9e3219569dc63304acb312a5dc687b5ddc69591d8a" Oct 31 04:30:16.561272 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:16.561240 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e37d27-c8547cdf9-v856h" event={"ID":"c0570273-666f-4052-9713-00813a62c291","Type":"ContainerStarted","Data":"46aceb98e784b5db5dd8349a7fe272a27d318f467959af46bce38acc01f8c080"} Oct 31 04:30:16.611059 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:16.604367 5019 patch_prober.go:29] interesting pod/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 31 04:30:16.611059 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:16.604429 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz" podUID="ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 31 04:30:16.659642 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:30:16.659572 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82b999d8_91cd_4021_8fc0_c2e61ae1a8f2.slice/crio-2a34e926d7165ae4f66da12dc451aef17a465accc91600a988e2ac3c54f2339d WatchSource:0}: Error finding container 2a34e926d7165ae4f66da12dc451aef17a465accc91600a988e2ac3c54f2339d: Status 404 returned error can't find the container with id 2a34e926d7165ae4f66da12dc451aef17a465accc91600a988e2ac3c54f2339d Oct 31 04:30:16.722602 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:16.720296 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-e37d27-6ddbbfbc47-gb52r"] Oct 31 04:30:17.587357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:17.585620 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d1d441-29890350-qwxpl" event={"ID":"1d8fff90-ca49-400c-9aa8-3e6676aa5428","Type":"ContainerStarted","Data":"4c09a009c7c74cc6502a2b7eaff5ad0ba39e6329682090e0cf76dfbc26eef0ab"} Oct 31 04:30:17.608779 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:17.608750 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890350-42rwc" event={"ID":"be7a6f4e-acc0-47dc-a962-a42eb116ab82","Type":"ContainerStarted","Data":"46aff12a6deafb2f94027a4cafc9b095bdf403d0d9a7f1bb8a8466e0cd288cc9"} Oct 31 04:30:17.610827 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:17.610800 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz" Oct 31 04:30:17.611148 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:17.611116 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e37d27-c8547cdf9-v856h" event={"ID":"c0570273-666f-4052-9713-00813a62c291","Type":"ContainerStarted","Data":"4eb06d0f10f1623e369e8da90eca8e46ffbbcbae5b5cdefc23f82731378df35a"} Oct 31 04:30:17.611249 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:17.611177 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz" Oct 31 04:30:17.611249 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:17.611226 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e37d27-c8547cdf9-v856h" Oct 31 04:30:17.612079 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:17.612058 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz" Oct 31 04:30:17.613189 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:17.613161 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-e37d27-6ddbbfbc47-gb52r" event={"ID":"82b999d8-91cd-4021-8fc0-c2e61ae1a8f2","Type":"ContainerStarted","Data":"c12841823a893abfb6ad1afaed67c1e7c788229cbea72e588213881dcb69cced"} Oct 31 04:30:17.613269 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:17.613209 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-e37d27-6ddbbfbc47-gb52r" event={"ID":"82b999d8-91cd-4021-8fc0-c2e61ae1a8f2","Type":"ContainerStarted","Data":"2a34e926d7165ae4f66da12dc451aef17a465accc91600a988e2ac3c54f2339d"} Oct 31 04:30:17.613309 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:17.613266 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-e37d27-6ddbbfbc47-gb52r" Oct 31 04:30:17.748436 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:17.748132 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-e37d27-6ddbbfbc47-gb52r" podStartSLOduration=3.748110639 podStartE2EDuration="3.748110639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:30:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:30:17.744985386 +0000 UTC m=+1572.501323424" watchObservedRunningTime="2026-10-31 04:30:17.748110639 +0000 UTC m=+1572.504448678" Oct 31 04:30:17.835919 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:17.835442 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e37d27-c8547cdf9-v856h" podStartSLOduration=3.835420611 podStartE2EDuration="3.835420611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:30:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:30:17.834954926 +0000 UTC m=+1572.591292966" watchObservedRunningTime="2026-10-31 04:30:17.835420611 +0000 UTC m=+1572.591758652" Oct 31 04:30:18.400771 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:18.400676 5019 scope.go:122] "RemoveContainer" containerID="8f8b5698032446790a2c3ae794f6e0e628856bce67f44f7447df9409f83f97cd" Oct 31 04:30:18.400944 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:18.400828 5019 scope.go:122] "RemoveContainer" containerID="5b778d6dc950e6bd6537290053bf4172f4df44e21523ed539e2433f658419a2c" Oct 31 04:30:18.401006 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:18.400988 5019 scope.go:122] "RemoveContainer" containerID="7cf98fb48d86ce4be77ed1bd07ae39e390004317b69ca386ff1b1faf9bd6e510" Oct 31 04:30:18.635142 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:18.635101 5019 generic.go:356] "Generic (PLEG): container finished" podID="1d8fff90-ca49-400c-9aa8-3e6676aa5428" containerID="4c09a009c7c74cc6502a2b7eaff5ad0ba39e6329682090e0cf76dfbc26eef0ab" exitCode=60 Oct 31 04:30:18.635509 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:18.635320 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d1d441-29890350-qwxpl" event={"ID":"1d8fff90-ca49-400c-9aa8-3e6676aa5428","Type":"ContainerDied","Data":"4c09a009c7c74cc6502a2b7eaff5ad0ba39e6329682090e0cf76dfbc26eef0ab"} Oct 31 04:30:18.635667 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:18.635573 5019 scope.go:122] "RemoveContainer" containerID="922e99bd510cad9e6647f4953546476f53b89fc7203eadf2edc041b5de73eec1" Oct 31 04:30:18.636163 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:18.636142 5019 scope.go:122] "RemoveContainer" containerID="4c09a009c7c74cc6502a2b7eaff5ad0ba39e6329682090e0cf76dfbc26eef0ab" Oct 31 04:30:18.636967 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:30:18.636589 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-d1d441-29890350-qwxpl_odh-ai-gateway-infra(1d8fff90-ca49-400c-9aa8-3e6676aa5428)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d1d441-29890350-qwxpl" podUID="1d8fff90-ca49-400c-9aa8-3e6676aa5428" Oct 31 04:30:18.640257 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:18.639772 5019 generic.go:356] "Generic (PLEG): container finished" podID="be7a6f4e-acc0-47dc-a962-a42eb116ab82" containerID="46aff12a6deafb2f94027a4cafc9b095bdf403d0d9a7f1bb8a8466e0cd288cc9" exitCode=60 Oct 31 04:30:18.640436 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:18.640405 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890350-42rwc" event={"ID":"be7a6f4e-acc0-47dc-a962-a42eb116ab82","Type":"ContainerDied","Data":"46aff12a6deafb2f94027a4cafc9b095bdf403d0d9a7f1bb8a8466e0cd288cc9"} Oct 31 04:30:18.640860 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:18.640836 5019 scope.go:122] "RemoveContainer" containerID="46aff12a6deafb2f94027a4cafc9b095bdf403d0d9a7f1bb8a8466e0cd288cc9" Oct 31 04:30:18.641980 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:30:18.641155 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890350-42rwc_odh-ai-gateway-infra(be7a6f4e-acc0-47dc-a962-a42eb116ab82)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890350-42rwc" podUID="be7a6f4e-acc0-47dc-a962-a42eb116ab82" Oct 31 04:30:18.648139 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:18.648113 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d1d441-29890350-qwxpl"] Oct 31 04:30:18.654899 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:18.654878 5019 scope.go:122] "RemoveContainer" containerID="d5bb378ea62e4fb4a14d3b9e3219569dc63304acb312a5dc687b5ddc69591d8a" Oct 31 04:30:18.803718 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:18.803678 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890350-42rwc"] Oct 31 04:30:19.399861 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:19.399833 5019 scope.go:122] "RemoveContainer" containerID="469190e63125da3db7a648dc6f7e172275ad121eb5dffd8a6622f3d489647d16" Oct 31 04:30:19.400027 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:19.399961 5019 scope.go:122] "RemoveContainer" containerID="ad3e68f8f114cbf1df116fdf94c8950b216b498fb35b5ad2957140b2dd9c5566" Oct 31 04:30:19.654824 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:19.654790 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b3ca79-29890350-zvb5r" event={"ID":"8255b0b4-8c46-4190-b88b-c7a1da3b72b7","Type":"ContainerStarted","Data":"b906d6162683a53d8fea6feb8e65ed3894fdd53a30b1456667f03d42a58c8ae6"} Oct 31 04:30:19.656725 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:19.656699 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890350-ptzdp" event={"ID":"98654db8-3fbb-4228-bec4-c84ab84ef439","Type":"ContainerStarted","Data":"08c8775322e7551a24e379ba26b373e85d3a651a69828c05c3a4d329ea9b75f5"} Oct 31 04:30:19.658652 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:19.658623 5019 generic.go:356] "Generic (PLEG): container finished" podID="c931f349-7638-4cf2-9329-a9fb5d21e6eb" containerID="1ad87fb0b65d43380d118caa47c0463699d88296ca2f11571de24e878546ba6c" exitCode=60 Oct 31 04:30:19.659157 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:19.659136 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7cf9ef-29890350-8zht5" event={"ID":"c931f349-7638-4cf2-9329-a9fb5d21e6eb","Type":"ContainerDied","Data":"1ad87fb0b65d43380d118caa47c0463699d88296ca2f11571de24e878546ba6c"} Oct 31 04:30:19.659242 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:19.659179 5019 scope.go:122] "RemoveContainer" containerID="8f8b5698032446790a2c3ae794f6e0e628856bce67f44f7447df9409f83f97cd" Oct 31 04:30:19.661410 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:19.660522 5019 scope.go:122] "RemoveContainer" containerID="1ad87fb0b65d43380d118caa47c0463699d88296ca2f11571de24e878546ba6c" Oct 31 04:30:19.661410 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:30:19.660760 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-7cf9ef-29890350-8zht5_odh-ai-gateway-infra(c931f349-7638-4cf2-9329-a9fb5d21e6eb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7cf9ef-29890350-8zht5" podUID="c931f349-7638-4cf2-9329-a9fb5d21e6eb" Oct 31 04:30:19.907232 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:19.907203 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d1d441-29890350-qwxpl" Oct 31 04:30:19.925911 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:19.925885 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890350-42rwc" Oct 31 04:30:20.087952 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:20.087874 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d8fff90-ca49-400c-9aa8-3e6676aa5428-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d8fff90-ca49-400c-9aa8-3e6676aa5428-maas-api-service-ca\") pod \"1d8fff90-ca49-400c-9aa8-3e6676aa5428\" (UID: \"1d8fff90-ca49-400c-9aa8-3e6676aa5428\") " Oct 31 04:30:20.088190 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:20.088038 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d8fff90-ca49-400c-9aa8-3e6676aa5428-kube-api-access-5cr44\" (UniqueName: \"kubernetes.io/projected/1d8fff90-ca49-400c-9aa8-3e6676aa5428-kube-api-access-5cr44\") pod \"1d8fff90-ca49-400c-9aa8-3e6676aa5428\" (UID: \"1d8fff90-ca49-400c-9aa8-3e6676aa5428\") " Oct 31 04:30:20.088190 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:20.088071 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be7a6f4e-acc0-47dc-a962-a42eb116ab82-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be7a6f4e-acc0-47dc-a962-a42eb116ab82-maas-api-service-ca\") pod \"be7a6f4e-acc0-47dc-a962-a42eb116ab82\" (UID: \"be7a6f4e-acc0-47dc-a962-a42eb116ab82\") " Oct 31 04:30:20.088190 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:20.088151 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be7a6f4e-acc0-47dc-a962-a42eb116ab82-kube-api-access-cfkps\" (UniqueName: \"kubernetes.io/projected/be7a6f4e-acc0-47dc-a962-a42eb116ab82-kube-api-access-cfkps\") pod \"be7a6f4e-acc0-47dc-a962-a42eb116ab82\" (UID: \"be7a6f4e-acc0-47dc-a962-a42eb116ab82\") " Oct 31 04:30:20.088382 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:20.088291 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d8fff90-ca49-400c-9aa8-3e6676aa5428-maas-api-service-ca" pod "1d8fff90-ca49-400c-9aa8-3e6676aa5428" (UID: "1d8fff90-ca49-400c-9aa8-3e6676aa5428"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:30:20.088540 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:20.088516 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be7a6f4e-acc0-47dc-a962-a42eb116ab82-maas-api-service-ca" pod "be7a6f4e-acc0-47dc-a962-a42eb116ab82" (UID: "be7a6f4e-acc0-47dc-a962-a42eb116ab82"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:30:20.088796 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:20.088774 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d8fff90-ca49-400c-9aa8-3e6676aa5428-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:20.088884 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:20.088800 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be7a6f4e-acc0-47dc-a962-a42eb116ab82-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:20.090543 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:20.090513 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be7a6f4e-acc0-47dc-a962-a42eb116ab82-kube-api-access-cfkps" pod "be7a6f4e-acc0-47dc-a962-a42eb116ab82" (UID: "be7a6f4e-acc0-47dc-a962-a42eb116ab82"). InnerVolumeSpecName "kube-api-access-cfkps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:30:20.090648 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:20.090575 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d8fff90-ca49-400c-9aa8-3e6676aa5428-kube-api-access-5cr44" pod "1d8fff90-ca49-400c-9aa8-3e6676aa5428" (UID: "1d8fff90-ca49-400c-9aa8-3e6676aa5428"). InnerVolumeSpecName "kube-api-access-5cr44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:30:20.189722 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:20.189689 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cr44\" (UniqueName: \"kubernetes.io/projected/1d8fff90-ca49-400c-9aa8-3e6676aa5428-kube-api-access-5cr44\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:20.189722 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:20.189720 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfkps\" (UniqueName: \"kubernetes.io/projected/be7a6f4e-acc0-47dc-a962-a42eb116ab82-kube-api-access-cfkps\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:20.670612 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:20.670525 5019 generic.go:356] "Generic (PLEG): container finished" podID="8cb938bd-e510-49fa-8183-a91749feeb5b" containerID="e4a630c61b510e9b4efb9753edb41af00c097d3405698cd6682b65be3b1f31bb" exitCode=60 Oct 31 04:30:20.670831 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:20.670602 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890350-ncpb2" event={"ID":"8cb938bd-e510-49fa-8183-a91749feeb5b","Type":"ContainerDied","Data":"e4a630c61b510e9b4efb9753edb41af00c097d3405698cd6682b65be3b1f31bb"} Oct 31 04:30:20.670831 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:20.670672 5019 scope.go:122] "RemoveContainer" containerID="469190e63125da3db7a648dc6f7e172275ad121eb5dffd8a6622f3d489647d16" Oct 31 04:30:20.671077 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:20.671060 5019 scope.go:122] "RemoveContainer" containerID="e4a630c61b510e9b4efb9753edb41af00c097d3405698cd6682b65be3b1f31bb" Oct 31 04:30:20.671438 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:30:20.671416 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c4006d-29890350-ncpb2_odh-ai-gateway-infra(8cb938bd-e510-49fa-8183-a91749feeb5b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890350-ncpb2" podUID="8cb938bd-e510-49fa-8183-a91749feeb5b" Oct 31 04:30:20.672836 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:20.672816 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d1d441-29890350-qwxpl" event={"ID":"1d8fff90-ca49-400c-9aa8-3e6676aa5428","Type":"ContainerDied","Data":"ccc6fc20e313c99f3cd8c01fdd6928e266148f0fa6f5df178b814478f2a6d490"} Oct 31 04:30:20.672954 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:20.672823 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d1d441-29890350-qwxpl" Oct 31 04:30:20.674546 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:20.674528 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890350-42rwc" Oct 31 04:30:20.674702 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:20.674573 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890350-42rwc" event={"ID":"be7a6f4e-acc0-47dc-a962-a42eb116ab82","Type":"ContainerDied","Data":"0604b5459456e58dcb6846cdb506c1ad74e204efb372f63f9639ec6f5e5d5053"} Oct 31 04:30:20.677138 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:20.677107 5019 generic.go:356] "Generic (PLEG): container finished" podID="8255b0b4-8c46-4190-b88b-c7a1da3b72b7" containerID="b906d6162683a53d8fea6feb8e65ed3894fdd53a30b1456667f03d42a58c8ae6" exitCode=60 Oct 31 04:30:20.677249 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:20.677198 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b3ca79-29890350-zvb5r" event={"ID":"8255b0b4-8c46-4190-b88b-c7a1da3b72b7","Type":"ContainerDied","Data":"b906d6162683a53d8fea6feb8e65ed3894fdd53a30b1456667f03d42a58c8ae6"} Oct 31 04:30:20.677651 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:20.677631 5019 scope.go:122] "RemoveContainer" containerID="b906d6162683a53d8fea6feb8e65ed3894fdd53a30b1456667f03d42a58c8ae6" Oct 31 04:30:20.678044 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:30:20.677935 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-b3ca79-29890350-zvb5r_odh-ai-gateway-infra(8255b0b4-8c46-4190-b88b-c7a1da3b72b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b3ca79-29890350-zvb5r" podUID="8255b0b4-8c46-4190-b88b-c7a1da3b72b7" Oct 31 04:30:20.681730 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:20.681706 5019 generic.go:356] "Generic (PLEG): container finished" podID="98654db8-3fbb-4228-bec4-c84ab84ef439" containerID="08c8775322e7551a24e379ba26b373e85d3a651a69828c05c3a4d329ea9b75f5" exitCode=60 Oct 31 04:30:20.681838 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:20.681775 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890350-ptzdp" event={"ID":"98654db8-3fbb-4228-bec4-c84ab84ef439","Type":"ContainerDied","Data":"08c8775322e7551a24e379ba26b373e85d3a651a69828c05c3a4d329ea9b75f5"} Oct 31 04:30:20.682159 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:20.682140 5019 scope.go:122] "RemoveContainer" containerID="08c8775322e7551a24e379ba26b373e85d3a651a69828c05c3a4d329ea9b75f5" Oct 31 04:30:20.682462 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:30:20.682435 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-4c2387-29890350-ptzdp_odh-ai-gateway-infra(98654db8-3fbb-4228-bec4-c84ab84ef439)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890350-ptzdp" podUID="98654db8-3fbb-4228-bec4-c84ab84ef439" Oct 31 04:30:20.684766 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:20.684745 5019 generic.go:356] "Generic (PLEG): container finished" podID="e9df73df-d6a4-4152-915e-2611831bcf90" containerID="ed76f0b4f95365adff6df3b284fb82339f4d5746eaea01e541500dc7bce568b5" exitCode=60 Oct 31 04:30:20.684843 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:20.684835 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-81c810-29890350-mth8b" event={"ID":"e9df73df-d6a4-4152-915e-2611831bcf90","Type":"ContainerDied","Data":"ed76f0b4f95365adff6df3b284fb82339f4d5746eaea01e541500dc7bce568b5"} Oct 31 04:30:20.685163 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:20.685146 5019 scope.go:122] "RemoveContainer" containerID="ed76f0b4f95365adff6df3b284fb82339f4d5746eaea01e541500dc7bce568b5" Oct 31 04:30:20.685406 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:30:20.685326 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-81c810-29890350-mth8b_odh-ai-gateway-infra(e9df73df-d6a4-4152-915e-2611831bcf90)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-81c810-29890350-mth8b" podUID="e9df73df-d6a4-4152-915e-2611831bcf90" Oct 31 04:30:20.688422 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:20.688403 5019 scope.go:122] "RemoveContainer" containerID="4c09a009c7c74cc6502a2b7eaff5ad0ba39e6329682090e0cf76dfbc26eef0ab" Oct 31 04:30:20.699277 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:20.699252 5019 scope.go:122] "RemoveContainer" containerID="46aff12a6deafb2f94027a4cafc9b095bdf403d0d9a7f1bb8a8466e0cd288cc9" Oct 31 04:30:20.708191 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:20.708175 5019 scope.go:122] "RemoveContainer" containerID="7cf98fb48d86ce4be77ed1bd07ae39e390004317b69ca386ff1b1faf9bd6e510" Oct 31 04:30:20.721978 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:20.721957 5019 scope.go:122] "RemoveContainer" containerID="5b778d6dc950e6bd6537290053bf4172f4df44e21523ed539e2433f658419a2c" Oct 31 04:30:20.737880 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:20.737789 5019 scope.go:122] "RemoveContainer" containerID="ad3e68f8f114cbf1df116fdf94c8950b216b498fb35b5ad2957140b2dd9c5566" Oct 31 04:30:20.755466 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:20.755440 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7cf9ef-29890350-8zht5"] Oct 31 04:30:20.793398 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:20.793367 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d1d441-29890350-qwxpl"] Oct 31 04:30:20.800389 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:20.800351 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d1d441-29890350-qwxpl"] Oct 31 04:30:20.837099 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:20.837061 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890350-42rwc"] Oct 31 04:30:20.845677 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:20.845647 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890350-42rwc"] Oct 31 04:30:20.849746 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:20.849717 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b3ca79-29890350-zvb5r"] Oct 31 04:30:20.898787 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:20.898757 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890350-ptzdp"] Oct 31 04:30:20.925227 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:20.925170 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7cf9ef-29890350-8zht5" Oct 31 04:30:21.106223 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:21.106186 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c931f349-7638-4cf2-9329-a9fb5d21e6eb-kube-api-access-pzfdj\" (UniqueName: \"kubernetes.io/projected/c931f349-7638-4cf2-9329-a9fb5d21e6eb-kube-api-access-pzfdj\") pod \"c931f349-7638-4cf2-9329-a9fb5d21e6eb\" (UID: \"c931f349-7638-4cf2-9329-a9fb5d21e6eb\") " Oct 31 04:30:21.106468 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:21.106445 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c931f349-7638-4cf2-9329-a9fb5d21e6eb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c931f349-7638-4cf2-9329-a9fb5d21e6eb-maas-api-service-ca\") pod \"c931f349-7638-4cf2-9329-a9fb5d21e6eb\" (UID: \"c931f349-7638-4cf2-9329-a9fb5d21e6eb\") " Oct 31 04:30:21.106771 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:21.106734 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c931f349-7638-4cf2-9329-a9fb5d21e6eb-maas-api-service-ca" pod "c931f349-7638-4cf2-9329-a9fb5d21e6eb" (UID: "c931f349-7638-4cf2-9329-a9fb5d21e6eb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:30:21.107249 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:21.107224 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c931f349-7638-4cf2-9329-a9fb5d21e6eb-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:21.109021 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:21.108988 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c931f349-7638-4cf2-9329-a9fb5d21e6eb-kube-api-access-pzfdj" pod "c931f349-7638-4cf2-9329-a9fb5d21e6eb" (UID: "c931f349-7638-4cf2-9329-a9fb5d21e6eb"). InnerVolumeSpecName "kube-api-access-pzfdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:30:21.208761 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:21.208676 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzfdj\" (UniqueName: \"kubernetes.io/projected/c931f349-7638-4cf2-9329-a9fb5d21e6eb-kube-api-access-pzfdj\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:21.410612 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:21.410578 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d8fff90-ca49-400c-9aa8-3e6676aa5428" path="/var/lib/kubelet/pods/1d8fff90-ca49-400c-9aa8-3e6676aa5428/volumes" Oct 31 04:30:21.411062 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:21.411042 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be7a6f4e-acc0-47dc-a962-a42eb116ab82" path="/var/lib/kubelet/pods/be7a6f4e-acc0-47dc-a962-a42eb116ab82/volumes" Oct 31 04:30:21.704017 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:21.703984 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7cf9ef-29890350-8zht5" Oct 31 04:30:21.704154 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:21.703986 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7cf9ef-29890350-8zht5" event={"ID":"c931f349-7638-4cf2-9329-a9fb5d21e6eb","Type":"ContainerDied","Data":"a618708f67bb9f14b3d256ee6c56c066e1e322a1084b6cbeedab78ca1a8b624b"} Oct 31 04:30:21.704217 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:21.704202 5019 scope.go:122] "RemoveContainer" containerID="1ad87fb0b65d43380d118caa47c0463699d88296ca2f11571de24e878546ba6c" Oct 31 04:30:21.729428 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:21.729387 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890350-ncpb2"] Oct 31 04:30:21.778000 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:21.769382 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-81c810-29890350-mth8b"] Oct 31 04:30:21.786992 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:21.786953 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7cf9ef-29890350-8zht5"] Oct 31 04:30:21.801412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:21.801317 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7cf9ef-29890350-8zht5"] Oct 31 04:30:21.925498 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:21.925462 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64"] Oct 31 04:30:21.926213 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:21.926181 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64" podUID="735acd81-e3db-44fa-aada-9632da8c3e68" containerName="istio-proxy" containerID="cri-o://32bc2aebe88b04bc28ea6d7269774a0b5354f0f72da4bc4d9fe5892be2191eff" gracePeriod=30 Oct 31 04:30:21.939362 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:21.939319 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b3ca79-29890350-zvb5r" Oct 31 04:30:22.005174 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:22.005077 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890350-ptzdp" Oct 31 04:30:22.032633 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:22.027123 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8255b0b4-8c46-4190-b88b-c7a1da3b72b7-kube-api-access-jx2cf\" (UniqueName: \"kubernetes.io/projected/8255b0b4-8c46-4190-b88b-c7a1da3b72b7-kube-api-access-jx2cf\") pod \"8255b0b4-8c46-4190-b88b-c7a1da3b72b7\" (UID: \"8255b0b4-8c46-4190-b88b-c7a1da3b72b7\") " Oct 31 04:30:22.032633 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:22.027345 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8255b0b4-8c46-4190-b88b-c7a1da3b72b7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8255b0b4-8c46-4190-b88b-c7a1da3b72b7-maas-api-service-ca\") pod \"8255b0b4-8c46-4190-b88b-c7a1da3b72b7\" (UID: \"8255b0b4-8c46-4190-b88b-c7a1da3b72b7\") " Oct 31 04:30:22.036102 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:22.035553 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8255b0b4-8c46-4190-b88b-c7a1da3b72b7-maas-api-service-ca" pod "8255b0b4-8c46-4190-b88b-c7a1da3b72b7" (UID: "8255b0b4-8c46-4190-b88b-c7a1da3b72b7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:30:22.037424 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:22.037230 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8255b0b4-8c46-4190-b88b-c7a1da3b72b7-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:22.044531 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:22.044046 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8255b0b4-8c46-4190-b88b-c7a1da3b72b7-kube-api-access-jx2cf" pod "8255b0b4-8c46-4190-b88b-c7a1da3b72b7" (UID: "8255b0b4-8c46-4190-b88b-c7a1da3b72b7"). InnerVolumeSpecName "kube-api-access-jx2cf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:30:22.074396 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:22.067069 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890350-ncpb2" Oct 31 04:30:22.138425 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:22.138387 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98654db8-3fbb-4228-bec4-c84ab84ef439-kube-api-access-n5t6t\" (UniqueName: \"kubernetes.io/projected/98654db8-3fbb-4228-bec4-c84ab84ef439-kube-api-access-n5t6t\") pod \"98654db8-3fbb-4228-bec4-c84ab84ef439\" (UID: \"98654db8-3fbb-4228-bec4-c84ab84ef439\") " Oct 31 04:30:22.138613 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:22.138488 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98654db8-3fbb-4228-bec4-c84ab84ef439-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98654db8-3fbb-4228-bec4-c84ab84ef439-maas-api-service-ca\") pod \"98654db8-3fbb-4228-bec4-c84ab84ef439\" (UID: \"98654db8-3fbb-4228-bec4-c84ab84ef439\") " Oct 31 04:30:22.138613 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:22.138509 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-81c810-29890350-mth8b" Oct 31 04:30:22.138953 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:22.138930 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98654db8-3fbb-4228-bec4-c84ab84ef439-maas-api-service-ca" pod "98654db8-3fbb-4228-bec4-c84ab84ef439" (UID: "98654db8-3fbb-4228-bec4-c84ab84ef439"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:30:22.139270 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:22.139249 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jx2cf\" (UniqueName: \"kubernetes.io/projected/8255b0b4-8c46-4190-b88b-c7a1da3b72b7-kube-api-access-jx2cf\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:22.139357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:22.139276 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98654db8-3fbb-4228-bec4-c84ab84ef439-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:22.140854 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:22.140828 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98654db8-3fbb-4228-bec4-c84ab84ef439-kube-api-access-n5t6t" pod "98654db8-3fbb-4228-bec4-c84ab84ef439" (UID: "98654db8-3fbb-4228-bec4-c84ab84ef439"). InnerVolumeSpecName "kube-api-access-n5t6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:30:22.240515 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:22.240471 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9df73df-d6a4-4152-915e-2611831bcf90-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9df73df-d6a4-4152-915e-2611831bcf90-maas-api-service-ca\") pod \"e9df73df-d6a4-4152-915e-2611831bcf90\" (UID: \"e9df73df-d6a4-4152-915e-2611831bcf90\") " Oct 31 04:30:22.240804 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:22.240536 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cb938bd-e510-49fa-8183-a91749feeb5b-kube-api-access-tdxgk\" (UniqueName: \"kubernetes.io/projected/8cb938bd-e510-49fa-8183-a91749feeb5b-kube-api-access-tdxgk\") pod \"8cb938bd-e510-49fa-8183-a91749feeb5b\" (UID: \"8cb938bd-e510-49fa-8183-a91749feeb5b\") " Oct 31 04:30:22.240804 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:22.240705 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9df73df-d6a4-4152-915e-2611831bcf90-kube-api-access-g2lhz\" (UniqueName: \"kubernetes.io/projected/e9df73df-d6a4-4152-915e-2611831bcf90-kube-api-access-g2lhz\") pod \"e9df73df-d6a4-4152-915e-2611831bcf90\" (UID: \"e9df73df-d6a4-4152-915e-2611831bcf90\") " Oct 31 04:30:22.240804 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:22.240785 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cb938bd-e510-49fa-8183-a91749feeb5b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cb938bd-e510-49fa-8183-a91749feeb5b-maas-api-service-ca\") pod \"8cb938bd-e510-49fa-8183-a91749feeb5b\" (UID: \"8cb938bd-e510-49fa-8183-a91749feeb5b\") " Oct 31 04:30:22.241237 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:22.241215 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5t6t\" (UniqueName: \"kubernetes.io/projected/98654db8-3fbb-4228-bec4-c84ab84ef439-kube-api-access-n5t6t\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:22.241604 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:22.241578 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cb938bd-e510-49fa-8183-a91749feeb5b-maas-api-service-ca" pod "8cb938bd-e510-49fa-8183-a91749feeb5b" (UID: "8cb938bd-e510-49fa-8183-a91749feeb5b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:30:22.242376 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:22.242349 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9df73df-d6a4-4152-915e-2611831bcf90-maas-api-service-ca" pod "e9df73df-d6a4-4152-915e-2611831bcf90" (UID: "e9df73df-d6a4-4152-915e-2611831bcf90"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:30:22.244129 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:22.244103 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9df73df-d6a4-4152-915e-2611831bcf90-kube-api-access-g2lhz" pod "e9df73df-d6a4-4152-915e-2611831bcf90" (UID: "e9df73df-d6a4-4152-915e-2611831bcf90"). InnerVolumeSpecName "kube-api-access-g2lhz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:30:22.244302 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:22.244273 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cb938bd-e510-49fa-8183-a91749feeb5b-kube-api-access-tdxgk" pod "8cb938bd-e510-49fa-8183-a91749feeb5b" (UID: "8cb938bd-e510-49fa-8183-a91749feeb5b"). InnerVolumeSpecName "kube-api-access-tdxgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:30:22.344564 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:22.344512 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2lhz\" (UniqueName: \"kubernetes.io/projected/e9df73df-d6a4-4152-915e-2611831bcf90-kube-api-access-g2lhz\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:22.344564 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:22.344552 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cb938bd-e510-49fa-8183-a91749feeb5b-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:22.344564 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:22.344567 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9df73df-d6a4-4152-915e-2611831bcf90-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:22.344564 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:22.344581 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdxgk\" (UniqueName: \"kubernetes.io/projected/8cb938bd-e510-49fa-8183-a91749feeb5b-kube-api-access-tdxgk\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:22.405604 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:30:22.405563 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:30:22.719568 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:22.719511 5019 generic.go:356] "Generic (PLEG): container finished" podID="16ca6a86-9186-4d31-a3ac-07f81f318a42" containerID="c7e02b380f069e182c5ee70b3d5189736fe64a69078b3152081daf0225692873" exitCode=0 Oct 31 04:30:22.719774 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:22.719582 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr" event={"ID":"16ca6a86-9186-4d31-a3ac-07f81f318a42","Type":"ContainerDied","Data":"c7e02b380f069e182c5ee70b3d5189736fe64a69078b3152081daf0225692873"} Oct 31 04:30:22.720310 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:22.720103 5019 scope.go:122] "RemoveContainer" containerID="c7e02b380f069e182c5ee70b3d5189736fe64a69078b3152081daf0225692873" Oct 31 04:30:22.721554 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:22.721408 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b3ca79-29890350-zvb5r" Oct 31 04:30:22.721554 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:22.721514 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b3ca79-29890350-zvb5r" event={"ID":"8255b0b4-8c46-4190-b88b-c7a1da3b72b7","Type":"ContainerDied","Data":"2d55063dd93d7a45b617fab38b0d95d91d670ee2be2d380119e58887fd481b35"} Oct 31 04:30:22.721732 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:22.721582 5019 scope.go:122] "RemoveContainer" containerID="b906d6162683a53d8fea6feb8e65ed3894fdd53a30b1456667f03d42a58c8ae6" Oct 31 04:30:22.722155 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:22.722120 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:30:22.722283 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:22.722212 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:30:22.722283 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:22.722264 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:30:22.724633 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:22.724606 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890350-ptzdp" event={"ID":"98654db8-3fbb-4228-bec4-c84ab84ef439","Type":"ContainerDied","Data":"36e10812bbf084e60a54ac8ef39e3974b4f91aa60becd512beb9e359d75426ed"} Oct 31 04:30:22.724730 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:22.724706 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890350-ptzdp" Oct 31 04:30:22.726755 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:22.726735 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-81c810-29890350-mth8b" Oct 31 04:30:22.727063 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:22.727047 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-81c810-29890350-mth8b" event={"ID":"e9df73df-d6a4-4152-915e-2611831bcf90","Type":"ContainerDied","Data":"0e80e6fe9f1264c49988ba3b1f3cfbea4f5cd564199a759a5978e60971b31d8c"} Oct 31 04:30:22.730659 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:22.730414 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890350-ncpb2" event={"ID":"8cb938bd-e510-49fa-8183-a91749feeb5b","Type":"ContainerDied","Data":"10df3c0d4985a801df34458c1ece7bc9c4140c77a7f7feb99535e59985d6112a"} Oct 31 04:30:22.731039 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:22.730830 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890350-ncpb2" Oct 31 04:30:22.750752 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:22.750726 5019 scope.go:122] "RemoveContainer" containerID="08c8775322e7551a24e379ba26b373e85d3a651a69828c05c3a4d329ea9b75f5" Oct 31 04:30:22.763040 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:22.763014 5019 scope.go:122] "RemoveContainer" containerID="ed76f0b4f95365adff6df3b284fb82339f4d5746eaea01e541500dc7bce568b5" Oct 31 04:30:22.778694 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:22.778670 5019 scope.go:122] "RemoveContainer" containerID="e4a630c61b510e9b4efb9753edb41af00c097d3405698cd6682b65be3b1f31bb" Oct 31 04:30:22.815024 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:22.814985 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890350-ncpb2"] Oct 31 04:30:22.824001 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:22.823971 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890350-ncpb2"] Oct 31 04:30:22.855120 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:22.855084 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b3ca79-29890350-zvb5r"] Oct 31 04:30:22.861026 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:22.860992 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b3ca79-29890350-zvb5r"] Oct 31 04:30:22.894563 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:22.894528 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890350-ptzdp"] Oct 31 04:30:22.904844 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:22.904814 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890350-ptzdp"] Oct 31 04:30:22.939053 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:22.939016 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-81c810-29890350-mth8b"] Oct 31 04:30:22.942940 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:22.942914 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-81c810-29890350-mth8b"] Oct 31 04:30:23.409143 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:23.409109 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8255b0b4-8c46-4190-b88b-c7a1da3b72b7" path="/var/lib/kubelet/pods/8255b0b4-8c46-4190-b88b-c7a1da3b72b7/volumes" Oct 31 04:30:23.409664 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:23.409644 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cb938bd-e510-49fa-8183-a91749feeb5b" path="/var/lib/kubelet/pods/8cb938bd-e510-49fa-8183-a91749feeb5b/volumes" Oct 31 04:30:23.410039 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:23.410024 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98654db8-3fbb-4228-bec4-c84ab84ef439" path="/var/lib/kubelet/pods/98654db8-3fbb-4228-bec4-c84ab84ef439/volumes" Oct 31 04:30:23.410751 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:23.410730 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c931f349-7638-4cf2-9329-a9fb5d21e6eb" path="/var/lib/kubelet/pods/c931f349-7638-4cf2-9329-a9fb5d21e6eb/volumes" Oct 31 04:30:23.411272 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:23.411248 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9df73df-d6a4-4152-915e-2611831bcf90" path="/var/lib/kubelet/pods/e9df73df-d6a4-4152-915e-2611831bcf90/volumes" Oct 31 04:30:23.739634 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:23.739565 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr" event={"ID":"16ca6a86-9186-4d31-a3ac-07f81f318a42","Type":"ContainerStarted","Data":"7dc9ed89fc773d4b3760de30de1528182a467d913f7b0561e3c954a14e29caf8"} Oct 31 04:30:24.653545 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:24.653513 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr" Oct 31 04:30:24.655192 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:24.655164 5019 patch_prober.go:29] interesting pod/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Oct 31 04:30:24.655394 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:24.655214 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr" podUID="16ca6a86-9186-4d31-a3ac-07f81f318a42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Oct 31 04:30:24.938571 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:24.938476 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f6277de-fkjtd"] Oct 31 04:30:24.939103 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:24.939089 5019 cpu_manager.go:414] "containerMap: removing container" podUID="8cb938bd-e510-49fa-8183-a91749feeb5b" containerName="cleanup" Oct 31 04:30:24.939143 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:24.939104 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cb938bd-e510-49fa-8183-a91749feeb5b" containerName="cleanup" Oct 31 04:30:24.939143 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:24.939123 5019 cpu_manager.go:414] "containerMap: removing container" podUID="98654db8-3fbb-4228-bec4-c84ab84ef439" containerName="cleanup" Oct 31 04:30:24.939143 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:24.939128 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98654db8-3fbb-4228-bec4-c84ab84ef439" containerName="cleanup" Oct 31 04:30:24.939143 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:24.939140 5019 cpu_manager.go:414] "containerMap: removing container" podUID="e9df73df-d6a4-4152-915e-2611831bcf90" containerName="cleanup" Oct 31 04:30:24.939451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:24.939145 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9df73df-d6a4-4152-915e-2611831bcf90" containerName="cleanup" Oct 31 04:30:24.939451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:24.939156 5019 cpu_manager.go:414] "containerMap: removing container" podUID="be7a6f4e-acc0-47dc-a962-a42eb116ab82" containerName="cleanup" Oct 31 04:30:24.939451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:24.939160 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be7a6f4e-acc0-47dc-a962-a42eb116ab82" containerName="cleanup" Oct 31 04:30:24.939451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:24.939173 5019 cpu_manager.go:414] "containerMap: removing container" podUID="1d8fff90-ca49-400c-9aa8-3e6676aa5428" containerName="cleanup" Oct 31 04:30:24.939451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:24.939178 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d8fff90-ca49-400c-9aa8-3e6676aa5428" containerName="cleanup" Oct 31 04:30:24.939451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:24.939183 5019 cpu_manager.go:414] "containerMap: removing container" podUID="be7a6f4e-acc0-47dc-a962-a42eb116ab82" containerName="cleanup" Oct 31 04:30:24.939451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:24.939187 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be7a6f4e-acc0-47dc-a962-a42eb116ab82" containerName="cleanup" Oct 31 04:30:24.939451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:24.939197 5019 cpu_manager.go:414] "containerMap: removing container" podUID="8255b0b4-8c46-4190-b88b-c7a1da3b72b7" containerName="cleanup" Oct 31 04:30:24.939451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:24.939201 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8255b0b4-8c46-4190-b88b-c7a1da3b72b7" containerName="cleanup" Oct 31 04:30:24.939451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:24.939208 5019 cpu_manager.go:414] "containerMap: removing container" podUID="8255b0b4-8c46-4190-b88b-c7a1da3b72b7" containerName="cleanup" Oct 31 04:30:24.939451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:24.939212 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8255b0b4-8c46-4190-b88b-c7a1da3b72b7" containerName="cleanup" Oct 31 04:30:24.939451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:24.939223 5019 cpu_manager.go:414] "containerMap: removing container" podUID="c931f349-7638-4cf2-9329-a9fb5d21e6eb" containerName="cleanup" Oct 31 04:30:24.939451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:24.939228 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c931f349-7638-4cf2-9329-a9fb5d21e6eb" containerName="cleanup" Oct 31 04:30:24.939451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:24.939236 5019 cpu_manager.go:414] "containerMap: removing container" podUID="c931f349-7638-4cf2-9329-a9fb5d21e6eb" containerName="cleanup" Oct 31 04:30:24.939451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:24.939240 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c931f349-7638-4cf2-9329-a9fb5d21e6eb" containerName="cleanup" Oct 31 04:30:24.939451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:24.939253 5019 cpu_manager.go:414] "containerMap: removing container" podUID="8cb938bd-e510-49fa-8183-a91749feeb5b" containerName="cleanup" Oct 31 04:30:24.939451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:24.939258 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cb938bd-e510-49fa-8183-a91749feeb5b" containerName="cleanup" Oct 31 04:30:24.939451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:24.939264 5019 cpu_manager.go:414] "containerMap: removing container" podUID="c931f349-7638-4cf2-9329-a9fb5d21e6eb" containerName="cleanup" Oct 31 04:30:24.939451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:24.939268 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c931f349-7638-4cf2-9329-a9fb5d21e6eb" containerName="cleanup" Oct 31 04:30:24.939451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:24.939275 5019 cpu_manager.go:414] "containerMap: removing container" podUID="e9df73df-d6a4-4152-915e-2611831bcf90" containerName="cleanup" Oct 31 04:30:24.939451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:24.939281 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9df73df-d6a4-4152-915e-2611831bcf90" containerName="cleanup" Oct 31 04:30:24.939451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:24.939287 5019 cpu_manager.go:414] "containerMap: removing container" podUID="98654db8-3fbb-4228-bec4-c84ab84ef439" containerName="cleanup" Oct 31 04:30:24.939451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:24.939291 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98654db8-3fbb-4228-bec4-c84ab84ef439" containerName="cleanup" Oct 31 04:30:24.939451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:24.939299 5019 cpu_manager.go:414] "containerMap: removing container" podUID="1d8fff90-ca49-400c-9aa8-3e6676aa5428" containerName="cleanup" Oct 31 04:30:24.939451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:24.939303 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d8fff90-ca49-400c-9aa8-3e6676aa5428" containerName="cleanup" Oct 31 04:30:24.939451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:24.939397 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="8255b0b4-8c46-4190-b88b-c7a1da3b72b7" containerName="cleanup" Oct 31 04:30:24.939451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:24.939406 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="98654db8-3fbb-4228-bec4-c84ab84ef439" containerName="cleanup" Oct 31 04:30:24.939451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:24.939413 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9df73df-d6a4-4152-915e-2611831bcf90" containerName="cleanup" Oct 31 04:30:24.939451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:24.939420 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cb938bd-e510-49fa-8183-a91749feeb5b" containerName="cleanup" Oct 31 04:30:24.939451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:24.939426 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="c931f349-7638-4cf2-9329-a9fb5d21e6eb" containerName="cleanup" Oct 31 04:30:24.939451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:24.939431 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d8fff90-ca49-400c-9aa8-3e6676aa5428" containerName="cleanup" Oct 31 04:30:24.939451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:24.939438 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="be7a6f4e-acc0-47dc-a962-a42eb116ab82" containerName="cleanup" Oct 31 04:30:24.939451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:24.939444 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cb938bd-e510-49fa-8183-a91749feeb5b" containerName="cleanup" Oct 31 04:30:24.939451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:24.939452 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="c931f349-7638-4cf2-9329-a9fb5d21e6eb" containerName="cleanup" Oct 31 04:30:24.939451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:24.939461 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9df73df-d6a4-4152-915e-2611831bcf90" containerName="cleanup" Oct 31 04:30:24.939451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:24.939468 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="8255b0b4-8c46-4190-b88b-c7a1da3b72b7" containerName="cleanup" Oct 31 04:30:24.940821 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:24.939476 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d8fff90-ca49-400c-9aa8-3e6676aa5428" containerName="cleanup" Oct 31 04:30:24.940821 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:24.939485 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cb938bd-e510-49fa-8183-a91749feeb5b" containerName="cleanup" Oct 31 04:30:24.940821 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:24.939495 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="98654db8-3fbb-4228-bec4-c84ab84ef439" containerName="cleanup" Oct 31 04:30:24.940821 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:24.939513 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="be7a6f4e-acc0-47dc-a962-a42eb116ab82" containerName="cleanup" Oct 31 04:30:24.943120 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:24.943102 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6277de-fkjtd" Oct 31 04:30:24.945398 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:24.945376 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr" Oct 31 04:30:24.961124 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:24.961104 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f6277de-fkjtd"] Oct 31 04:30:24.969129 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:24.969105 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:30:24.969129 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:24.969110 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:30:25.000715 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:25.000678 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da002f29-2b99-451b-ada8-5c6d455341b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f6277de-fkjtd\" (UID: \"da002f29-2b99-451b-ada8-5c6d455341b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6277de-fkjtd" Oct 31 04:30:25.000886 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:25.000796 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwvcc\" (UniqueName: \"kubernetes.io/projected/da002f29-2b99-451b-ada8-5c6d455341b8-kube-api-access-pwvcc\") pod \"maas-api-revoke-sub-8f6277de-fkjtd\" (UID: \"da002f29-2b99-451b-ada8-5c6d455341b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6277de-fkjtd" Oct 31 04:30:25.000977 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:25.000904 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da002f29-2b99-451b-ada8-5c6d455341b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f6277de-fkjtd\" (UID: \"da002f29-2b99-451b-ada8-5c6d455341b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6277de-fkjtd" Oct 31 04:30:25.048756 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:25.048724 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:30:25.048922 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:25.048771 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:30:25.101771 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:25.101734 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwvcc\" (UniqueName: \"kubernetes.io/projected/da002f29-2b99-451b-ada8-5c6d455341b8-kube-api-access-pwvcc\") pod \"maas-api-revoke-sub-8f6277de-fkjtd\" (UID: \"da002f29-2b99-451b-ada8-5c6d455341b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6277de-fkjtd" Oct 31 04:30:25.101983 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:25.101869 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da002f29-2b99-451b-ada8-5c6d455341b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f6277de-fkjtd\" (UID: \"da002f29-2b99-451b-ada8-5c6d455341b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6277de-fkjtd" Oct 31 04:30:25.102045 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:25.101994 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da002f29-2b99-451b-ada8-5c6d455341b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f6277de-fkjtd\" (UID: \"da002f29-2b99-451b-ada8-5c6d455341b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6277de-fkjtd" Oct 31 04:30:25.102864 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:25.102828 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da002f29-2b99-451b-ada8-5c6d455341b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f6277de-fkjtd\" (UID: \"da002f29-2b99-451b-ada8-5c6d455341b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6277de-fkjtd" Oct 31 04:30:25.104785 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:25.104751 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da002f29-2b99-451b-ada8-5c6d455341b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f6277de-fkjtd\" (UID: \"da002f29-2b99-451b-ada8-5c6d455341b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6277de-fkjtd" Oct 31 04:30:25.125309 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:25.125280 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwvcc\" (UniqueName: \"kubernetes.io/projected/da002f29-2b99-451b-ada8-5c6d455341b8-kube-api-access-pwvcc\") pod \"maas-api-revoke-sub-8f6277de-fkjtd\" (UID: \"da002f29-2b99-451b-ada8-5c6d455341b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6277de-fkjtd" Oct 31 04:30:25.257289 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:25.257259 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6277de-fkjtd" Oct 31 04:30:25.599687 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:25.599394 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f6277de-fkjtd"] Oct 31 04:30:25.602927 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:30:25.602899 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda002f29_2b99_451b_ada8_5c6d455341b8.slice/crio-399196af0a2938ef9e124a55d349906fcf8da077b66911a4406136627d307bbe WatchSource:0}: Error finding container 399196af0a2938ef9e124a55d349906fcf8da077b66911a4406136627d307bbe: Status 404 returned error can't find the container with id 399196af0a2938ef9e124a55d349906fcf8da077b66911a4406136627d307bbe Oct 31 04:30:25.654285 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:25.654257 5019 patch_prober.go:29] interesting pod/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Oct 31 04:30:25.654603 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:25.654300 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr" podUID="16ca6a86-9186-4d31-a3ac-07f81f318a42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Oct 31 04:30:25.768540 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:25.766417 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6277de-fkjtd" event={"ID":"da002f29-2b99-451b-ada8-5c6d455341b8","Type":"ContainerStarted","Data":"399196af0a2938ef9e124a55d349906fcf8da077b66911a4406136627d307bbe"} Oct 31 04:30:26.654033 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:26.653987 5019 patch_prober.go:29] interesting pod/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Oct 31 04:30:26.654218 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:26.654054 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr" podUID="16ca6a86-9186-4d31-a3ac-07f81f318a42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Oct 31 04:30:26.808691 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:26.808652 5019 generic.go:356] "Generic (PLEG): container finished" podID="da002f29-2b99-451b-ada8-5c6d455341b8" containerID="7f030fb95c24d49d9c02c6bee389ee146f4c1e1d95f6e13490c1023d0ff0f963" exitCode=0 Oct 31 04:30:26.809323 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:26.809280 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6277de-fkjtd" event={"ID":"da002f29-2b99-451b-ada8-5c6d455341b8","Type":"ContainerDied","Data":"7f030fb95c24d49d9c02c6bee389ee146f4c1e1d95f6e13490c1023d0ff0f963"} Oct 31 04:30:27.653611 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:27.653575 5019 patch_prober.go:29] interesting pod/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Oct 31 04:30:27.653769 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:27.653635 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr" podUID="16ca6a86-9186-4d31-a3ac-07f81f318a42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Oct 31 04:30:27.704123 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:27.704042 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64" Oct 31 04:30:27.811511 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:27.811084 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/735acd81-e3db-44fa-aada-9632da8c3e68-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/735acd81-e3db-44fa-aada-9632da8c3e68-credential-socket\") pod \"735acd81-e3db-44fa-aada-9632da8c3e68\" (UID: \"735acd81-e3db-44fa-aada-9632da8c3e68\") " Oct 31 04:30:27.811511 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:27.811187 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/735acd81-e3db-44fa-aada-9632da8c3e68-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/735acd81-e3db-44fa-aada-9632da8c3e68-istio-podinfo\") pod \"735acd81-e3db-44fa-aada-9632da8c3e68\" (UID: \"735acd81-e3db-44fa-aada-9632da8c3e68\") " Oct 31 04:30:27.811511 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:27.811228 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/735acd81-e3db-44fa-aada-9632da8c3e68-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/735acd81-e3db-44fa-aada-9632da8c3e68-workload-socket\") pod \"735acd81-e3db-44fa-aada-9632da8c3e68\" (UID: \"735acd81-e3db-44fa-aada-9632da8c3e68\") " Oct 31 04:30:27.812404 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:27.812184 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/735acd81-e3db-44fa-aada-9632da8c3e68-istio-data" pod "735acd81-e3db-44fa-aada-9632da8c3e68" (UID: "735acd81-e3db-44fa-aada-9632da8c3e68"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:30:27.812897 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:27.812554 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/735acd81-e3db-44fa-aada-9632da8c3e68-credential-socket" pod "735acd81-e3db-44fa-aada-9632da8c3e68" (UID: "735acd81-e3db-44fa-aada-9632da8c3e68"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:30:27.812897 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:27.811256 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/735acd81-e3db-44fa-aada-9632da8c3e68-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/735acd81-e3db-44fa-aada-9632da8c3e68-istio-data\") pod \"735acd81-e3db-44fa-aada-9632da8c3e68\" (UID: \"735acd81-e3db-44fa-aada-9632da8c3e68\") " Oct 31 04:30:27.812897 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:27.812812 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/735acd81-e3db-44fa-aada-9632da8c3e68-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/735acd81-e3db-44fa-aada-9632da8c3e68-istio-envoy\") pod \"735acd81-e3db-44fa-aada-9632da8c3e68\" (UID: \"735acd81-e3db-44fa-aada-9632da8c3e68\") " Oct 31 04:30:27.813095 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:27.812903 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/735acd81-e3db-44fa-aada-9632da8c3e68-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/735acd81-e3db-44fa-aada-9632da8c3e68-workload-certs\") pod \"735acd81-e3db-44fa-aada-9632da8c3e68\" (UID: \"735acd81-e3db-44fa-aada-9632da8c3e68\") " Oct 31 04:30:27.813095 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:27.812971 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/735acd81-e3db-44fa-aada-9632da8c3e68-istio-token\" (UniqueName: \"kubernetes.io/projected/735acd81-e3db-44fa-aada-9632da8c3e68-istio-token\") pod \"735acd81-e3db-44fa-aada-9632da8c3e68\" (UID: \"735acd81-e3db-44fa-aada-9632da8c3e68\") " Oct 31 04:30:27.813095 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:27.813024 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/735acd81-e3db-44fa-aada-9632da8c3e68-kube-api-access-cznsb\" (UniqueName: \"kubernetes.io/projected/735acd81-e3db-44fa-aada-9632da8c3e68-kube-api-access-cznsb\") pod \"735acd81-e3db-44fa-aada-9632da8c3e68\" (UID: \"735acd81-e3db-44fa-aada-9632da8c3e68\") " Oct 31 04:30:27.813262 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:27.813092 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/735acd81-e3db-44fa-aada-9632da8c3e68-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/735acd81-e3db-44fa-aada-9632da8c3e68-istiod-ca-cert\") pod \"735acd81-e3db-44fa-aada-9632da8c3e68\" (UID: \"735acd81-e3db-44fa-aada-9632da8c3e68\") " Oct 31 04:30:27.816610 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:27.813723 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/735acd81-e3db-44fa-aada-9632da8c3e68-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:27.816610 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:27.813751 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/735acd81-e3db-44fa-aada-9632da8c3e68-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:27.816610 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:27.813879 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/735acd81-e3db-44fa-aada-9632da8c3e68-workload-certs" pod "735acd81-e3db-44fa-aada-9632da8c3e68" (UID: "735acd81-e3db-44fa-aada-9632da8c3e68"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:30:27.816610 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:27.814077 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/735acd81-e3db-44fa-aada-9632da8c3e68-istiod-ca-cert" pod "735acd81-e3db-44fa-aada-9632da8c3e68" (UID: "735acd81-e3db-44fa-aada-9632da8c3e68"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:30:27.816906 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:27.816797 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/735acd81-e3db-44fa-aada-9632da8c3e68-istio-envoy" pod "735acd81-e3db-44fa-aada-9632da8c3e68" (UID: "735acd81-e3db-44fa-aada-9632da8c3e68"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:30:27.817138 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:27.817115 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/735acd81-e3db-44fa-aada-9632da8c3e68-workload-socket" pod "735acd81-e3db-44fa-aada-9632da8c3e68" (UID: "735acd81-e3db-44fa-aada-9632da8c3e68"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:30:27.818425 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:27.817358 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/735acd81-e3db-44fa-aada-9632da8c3e68-istio-token" pod "735acd81-e3db-44fa-aada-9632da8c3e68" (UID: "735acd81-e3db-44fa-aada-9632da8c3e68"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:30:27.818425 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:27.817545 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/735acd81-e3db-44fa-aada-9632da8c3e68-kube-api-access-cznsb" pod "735acd81-e3db-44fa-aada-9632da8c3e68" (UID: "735acd81-e3db-44fa-aada-9632da8c3e68"). InnerVolumeSpecName "kube-api-access-cznsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:30:27.818986 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:27.818963 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/735acd81-e3db-44fa-aada-9632da8c3e68-istio-podinfo" pod "735acd81-e3db-44fa-aada-9632da8c3e68" (UID: "735acd81-e3db-44fa-aada-9632da8c3e68"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:30:27.827077 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:27.826907 5019 generic.go:356] "Generic (PLEG): container finished" podID="735acd81-e3db-44fa-aada-9632da8c3e68" containerID="32bc2aebe88b04bc28ea6d7269774a0b5354f0f72da4bc4d9fe5892be2191eff" exitCode=0 Oct 31 04:30:27.827077 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:27.826966 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64" Oct 31 04:30:27.827077 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:27.826981 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64" event={"ID":"735acd81-e3db-44fa-aada-9632da8c3e68","Type":"ContainerDied","Data":"32bc2aebe88b04bc28ea6d7269774a0b5354f0f72da4bc4d9fe5892be2191eff"} Oct 31 04:30:27.827077 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:27.827035 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64" event={"ID":"735acd81-e3db-44fa-aada-9632da8c3e68","Type":"ContainerDied","Data":"6b7ed979b83ca83615f7ab66f8db6585fd2141340c472a246aeef45519fc861a"} Oct 31 04:30:27.827077 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:27.827057 5019 scope.go:122] "RemoveContainer" containerID="32bc2aebe88b04bc28ea6d7269774a0b5354f0f72da4bc4d9fe5892be2191eff" Oct 31 04:30:27.860541 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:27.860518 5019 scope.go:122] "RemoveContainer" containerID="32bc2aebe88b04bc28ea6d7269774a0b5354f0f72da4bc4d9fe5892be2191eff" Oct 31 04:30:27.860897 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:30:27.860874 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32bc2aebe88b04bc28ea6d7269774a0b5354f0f72da4bc4d9fe5892be2191eff\": container with ID starting with 32bc2aebe88b04bc28ea6d7269774a0b5354f0f72da4bc4d9fe5892be2191eff not found: ID does not exist" containerID="32bc2aebe88b04bc28ea6d7269774a0b5354f0f72da4bc4d9fe5892be2191eff" Oct 31 04:30:27.860959 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:27.860914 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32bc2aebe88b04bc28ea6d7269774a0b5354f0f72da4bc4d9fe5892be2191eff"} err="failed to get container status \"32bc2aebe88b04bc28ea6d7269774a0b5354f0f72da4bc4d9fe5892be2191eff\": rpc error: code = NotFound desc = could not find container \"32bc2aebe88b04bc28ea6d7269774a0b5354f0f72da4bc4d9fe5892be2191eff\": container with ID starting with 32bc2aebe88b04bc28ea6d7269774a0b5354f0f72da4bc4d9fe5892be2191eff not found: ID does not exist" Oct 31 04:30:27.910443 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:27.910412 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64"] Oct 31 04:30:27.915721 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:27.915691 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/735acd81-e3db-44fa-aada-9632da8c3e68-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:27.915867 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:27.915725 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/735acd81-e3db-44fa-aada-9632da8c3e68-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:27.915867 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:27.915740 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/735acd81-e3db-44fa-aada-9632da8c3e68-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:27.915867 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:27.915754 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/735acd81-e3db-44fa-aada-9632da8c3e68-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:27.915867 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:27.915767 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/735acd81-e3db-44fa-aada-9632da8c3e68-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:27.915867 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:27.915780 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cznsb\" (UniqueName: \"kubernetes.io/projected/735acd81-e3db-44fa-aada-9632da8c3e68-kube-api-access-cznsb\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:27.915867 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:27.915798 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/735acd81-e3db-44fa-aada-9632da8c3e68-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:27.916092 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:27.916070 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-vmr64"] Oct 31 04:30:28.044277 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:28.044248 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6277de-fkjtd" Oct 31 04:30:28.117376 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:28.117310 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da002f29-2b99-451b-ada8-5c6d455341b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da002f29-2b99-451b-ada8-5c6d455341b8-maas-api-openshift-ca\") pod \"da002f29-2b99-451b-ada8-5c6d455341b8\" (UID: \"da002f29-2b99-451b-ada8-5c6d455341b8\") " Oct 31 04:30:28.118636 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:28.117702 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da002f29-2b99-451b-ada8-5c6d455341b8-kube-api-access-pwvcc\" (UniqueName: \"kubernetes.io/projected/da002f29-2b99-451b-ada8-5c6d455341b8-kube-api-access-pwvcc\") pod \"da002f29-2b99-451b-ada8-5c6d455341b8\" (UID: \"da002f29-2b99-451b-ada8-5c6d455341b8\") " Oct 31 04:30:28.118636 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:28.117920 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da002f29-2b99-451b-ada8-5c6d455341b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da002f29-2b99-451b-ada8-5c6d455341b8-maas-api-xks-ca\") pod \"da002f29-2b99-451b-ada8-5c6d455341b8\" (UID: \"da002f29-2b99-451b-ada8-5c6d455341b8\") " Oct 31 04:30:28.118636 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:28.118581 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da002f29-2b99-451b-ada8-5c6d455341b8-maas-api-openshift-ca" pod "da002f29-2b99-451b-ada8-5c6d455341b8" (UID: "da002f29-2b99-451b-ada8-5c6d455341b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:30:28.121310 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:28.121282 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da002f29-2b99-451b-ada8-5c6d455341b8-maas-api-xks-ca" pod "da002f29-2b99-451b-ada8-5c6d455341b8" (UID: "da002f29-2b99-451b-ada8-5c6d455341b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:30:28.121982 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:28.121949 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da002f29-2b99-451b-ada8-5c6d455341b8-kube-api-access-pwvcc" pod "da002f29-2b99-451b-ada8-5c6d455341b8" (UID: "da002f29-2b99-451b-ada8-5c6d455341b8"). InnerVolumeSpecName "kube-api-access-pwvcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:30:28.219808 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:28.219771 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da002f29-2b99-451b-ada8-5c6d455341b8-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:28.219808 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:28.219800 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da002f29-2b99-451b-ada8-5c6d455341b8-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:28.219808 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:28.219814 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwvcc\" (UniqueName: \"kubernetes.io/projected/da002f29-2b99-451b-ada8-5c6d455341b8-kube-api-access-pwvcc\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:28.644568 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:28.644530 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e37d27-c8547cdf9-v856h" Oct 31 04:30:28.644747 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:28.644642 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-e37d27-6ddbbfbc47-gb52r" Oct 31 04:30:28.653440 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:28.653409 5019 patch_prober.go:29] interesting pod/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Oct 31 04:30:28.653545 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:28.653468 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr" podUID="16ca6a86-9186-4d31-a3ac-07f81f318a42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Oct 31 04:30:28.848591 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:28.836494 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6277de-fkjtd" Oct 31 04:30:28.848591 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:28.836506 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6277de-fkjtd" event={"ID":"da002f29-2b99-451b-ada8-5c6d455341b8","Type":"ContainerDied","Data":"399196af0a2938ef9e124a55d349906fcf8da077b66911a4406136627d307bbe"} Oct 31 04:30:28.848591 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:28.836563 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="399196af0a2938ef9e124a55d349906fcf8da077b66911a4406136627d307bbe" Oct 31 04:30:29.408316 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:29.408278 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="735acd81-e3db-44fa-aada-9632da8c3e68" path="/var/lib/kubelet/pods/735acd81-e3db-44fa-aada-9632da8c3e68/volumes" Oct 31 04:30:29.657414 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:29.657384 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr" Oct 31 04:30:29.658699 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:29.658639 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr" Oct 31 04:30:30.118013 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:30.117981 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8d161cf-8d8z4"] Oct 31 04:30:30.118664 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:30.118648 5019 cpu_manager.go:414] "containerMap: removing container" podUID="735acd81-e3db-44fa-aada-9632da8c3e68" containerName="istio-proxy" Oct 31 04:30:30.118709 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:30.118665 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="735acd81-e3db-44fa-aada-9632da8c3e68" containerName="istio-proxy" Oct 31 04:30:30.118709 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:30.118676 5019 cpu_manager.go:414] "containerMap: removing container" podUID="98654db8-3fbb-4228-bec4-c84ab84ef439" containerName="cleanup" Oct 31 04:30:30.118709 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:30.118681 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98654db8-3fbb-4228-bec4-c84ab84ef439" containerName="cleanup" Oct 31 04:30:30.118709 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:30.118704 5019 cpu_manager.go:414] "containerMap: removing container" podUID="1d8fff90-ca49-400c-9aa8-3e6676aa5428" containerName="cleanup" Oct 31 04:30:30.118709 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:30.118708 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d8fff90-ca49-400c-9aa8-3e6676aa5428" containerName="cleanup" Oct 31 04:30:30.118871 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:30.118718 5019 cpu_manager.go:414] "containerMap: removing container" podUID="be7a6f4e-acc0-47dc-a962-a42eb116ab82" containerName="cleanup" Oct 31 04:30:30.118871 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:30.118722 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be7a6f4e-acc0-47dc-a962-a42eb116ab82" containerName="cleanup" Oct 31 04:30:30.118871 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:30.118731 5019 cpu_manager.go:414] "containerMap: removing container" podUID="8255b0b4-8c46-4190-b88b-c7a1da3b72b7" containerName="cleanup" Oct 31 04:30:30.118871 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:30.118735 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8255b0b4-8c46-4190-b88b-c7a1da3b72b7" containerName="cleanup" Oct 31 04:30:30.118871 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:30.118757 5019 cpu_manager.go:414] "containerMap: removing container" podUID="e9df73df-d6a4-4152-915e-2611831bcf90" containerName="cleanup" Oct 31 04:30:30.118871 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:30.118762 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9df73df-d6a4-4152-915e-2611831bcf90" containerName="cleanup" Oct 31 04:30:30.118871 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:30.118771 5019 cpu_manager.go:414] "containerMap: removing container" podUID="da002f29-2b99-451b-ada8-5c6d455341b8" containerName="revoke-keys" Oct 31 04:30:30.118871 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:30.118775 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da002f29-2b99-451b-ada8-5c6d455341b8" containerName="revoke-keys" Oct 31 04:30:30.118871 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:30.118782 5019 cpu_manager.go:414] "containerMap: removing container" podUID="8cb938bd-e510-49fa-8183-a91749feeb5b" containerName="cleanup" Oct 31 04:30:30.118871 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:30.118786 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cb938bd-e510-49fa-8183-a91749feeb5b" containerName="cleanup" Oct 31 04:30:30.118871 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:30.118870 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="be7a6f4e-acc0-47dc-a962-a42eb116ab82" containerName="cleanup" Oct 31 04:30:30.119186 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:30.118881 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="98654db8-3fbb-4228-bec4-c84ab84ef439" containerName="cleanup" Oct 31 04:30:30.119186 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:30.118890 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="c931f349-7638-4cf2-9329-a9fb5d21e6eb" containerName="cleanup" Oct 31 04:30:30.119186 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:30.118897 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9df73df-d6a4-4152-915e-2611831bcf90" containerName="cleanup" Oct 31 04:30:30.119186 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:30.118902 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="8255b0b4-8c46-4190-b88b-c7a1da3b72b7" containerName="cleanup" Oct 31 04:30:30.119186 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:30.118907 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d8fff90-ca49-400c-9aa8-3e6676aa5428" containerName="cleanup" Oct 31 04:30:30.119186 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:30.118914 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="da002f29-2b99-451b-ada8-5c6d455341b8" containerName="revoke-keys" Oct 31 04:30:30.119186 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:30.118920 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="735acd81-e3db-44fa-aada-9632da8c3e68" containerName="istio-proxy" Oct 31 04:30:30.123620 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:30.123600 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8d161cf-8d8z4" Oct 31 04:30:30.138699 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:30.138511 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:30:30.138699 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:30.138584 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:30:30.140031 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:30.140008 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8d161cf-8d8z4"] Oct 31 04:30:30.200393 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:30.200355 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:30:30.200622 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:30.200552 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:30:30.248216 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:30.248171 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8d1a346-e7a0-4207-9764-6a429a868977-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8d161cf-8d8z4\" (UID: \"d8d1a346-e7a0-4207-9764-6a429a868977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8d161cf-8d8z4" Oct 31 04:30:30.248444 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:30.248254 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8d1a346-e7a0-4207-9764-6a429a868977-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8d161cf-8d8z4\" (UID: \"d8d1a346-e7a0-4207-9764-6a429a868977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8d161cf-8d8z4" Oct 31 04:30:30.248537 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:30.248520 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qssxl\" (UniqueName: \"kubernetes.io/projected/d8d1a346-e7a0-4207-9764-6a429a868977-kube-api-access-qssxl\") pod \"maas-api-revoke-sub-b8d161cf-8d8z4\" (UID: \"d8d1a346-e7a0-4207-9764-6a429a868977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8d161cf-8d8z4" Oct 31 04:30:30.349923 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:30.349880 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8d1a346-e7a0-4207-9764-6a429a868977-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8d161cf-8d8z4\" (UID: \"d8d1a346-e7a0-4207-9764-6a429a868977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8d161cf-8d8z4" Oct 31 04:30:30.350132 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:30.350049 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qssxl\" (UniqueName: \"kubernetes.io/projected/d8d1a346-e7a0-4207-9764-6a429a868977-kube-api-access-qssxl\") pod \"maas-api-revoke-sub-b8d161cf-8d8z4\" (UID: \"d8d1a346-e7a0-4207-9764-6a429a868977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8d161cf-8d8z4" Oct 31 04:30:30.350220 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:30.350204 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8d1a346-e7a0-4207-9764-6a429a868977-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8d161cf-8d8z4\" (UID: \"d8d1a346-e7a0-4207-9764-6a429a868977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8d161cf-8d8z4" Oct 31 04:30:30.350659 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:30.350637 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8d1a346-e7a0-4207-9764-6a429a868977-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8d161cf-8d8z4\" (UID: \"d8d1a346-e7a0-4207-9764-6a429a868977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8d161cf-8d8z4" Oct 31 04:30:30.352516 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:30.352499 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8d1a346-e7a0-4207-9764-6a429a868977-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8d161cf-8d8z4\" (UID: \"d8d1a346-e7a0-4207-9764-6a429a868977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8d161cf-8d8z4" Oct 31 04:30:30.373864 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:30.373788 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qssxl\" (UniqueName: \"kubernetes.io/projected/d8d1a346-e7a0-4207-9764-6a429a868977-kube-api-access-qssxl\") pod \"maas-api-revoke-sub-b8d161cf-8d8z4\" (UID: \"d8d1a346-e7a0-4207-9764-6a429a868977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8d161cf-8d8z4" Oct 31 04:30:30.439180 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:30.439144 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8d161cf-8d8z4" Oct 31 04:30:30.766482 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:30:30.766445 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8d1a346_e7a0_4207_9764_6a429a868977.slice/crio-ee1d5bf7c181b346efeb87ae08e96d3c2bce3d1ddd7f870a6cebcae2b048f070 WatchSource:0}: Error finding container ee1d5bf7c181b346efeb87ae08e96d3c2bce3d1ddd7f870a6cebcae2b048f070: Status 404 returned error can't find the container with id ee1d5bf7c181b346efeb87ae08e96d3c2bce3d1ddd7f870a6cebcae2b048f070 Oct 31 04:30:30.767636 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:30.767607 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8d161cf-8d8z4"] Oct 31 04:30:30.890704 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:30.890659 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8d161cf-8d8z4" event={"ID":"d8d1a346-e7a0-4207-9764-6a429a868977","Type":"ContainerStarted","Data":"ee1d5bf7c181b346efeb87ae08e96d3c2bce3d1ddd7f870a6cebcae2b048f070"} Oct 31 04:30:31.900313 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:31.900214 5019 generic.go:356] "Generic (PLEG): container finished" podID="d8d1a346-e7a0-4207-9764-6a429a868977" containerID="0a20045ea6fb6b32f66a62b9e335ba3249e8592a5f4db52ee2ce4f3ef9b37c12" exitCode=0 Oct 31 04:30:31.900313 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:31.900277 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8d161cf-8d8z4" event={"ID":"d8d1a346-e7a0-4207-9764-6a429a868977","Type":"ContainerDied","Data":"0a20045ea6fb6b32f66a62b9e335ba3249e8592a5f4db52ee2ce4f3ef9b37c12"} Oct 31 04:30:32.838011 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:32.837980 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj"] Oct 31 04:30:32.838291 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:32.838234 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj" podUID="618ff6b5-9d72-4ac1-9132-89c0697dfc52" containerName="istio-proxy" containerID="cri-o://b62294855ceca27bee0fc4caca551009560e7ec849df8d151b99df8b28d3cb55" gracePeriod=30 Oct 31 04:30:33.154262 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:33.154238 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8d161cf-8d8z4" Oct 31 04:30:33.308689 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:33.308657 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8d1a346-e7a0-4207-9764-6a429a868977-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8d1a346-e7a0-4207-9764-6a429a868977-maas-api-openshift-ca\") pod \"d8d1a346-e7a0-4207-9764-6a429a868977\" (UID: \"d8d1a346-e7a0-4207-9764-6a429a868977\") " Oct 31 04:30:33.308895 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:33.308882 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8d1a346-e7a0-4207-9764-6a429a868977-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8d1a346-e7a0-4207-9764-6a429a868977-maas-api-xks-ca\") pod \"d8d1a346-e7a0-4207-9764-6a429a868977\" (UID: \"d8d1a346-e7a0-4207-9764-6a429a868977\") " Oct 31 04:30:33.308946 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:33.308914 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8d1a346-e7a0-4207-9764-6a429a868977-kube-api-access-qssxl\" (UniqueName: \"kubernetes.io/projected/d8d1a346-e7a0-4207-9764-6a429a868977-kube-api-access-qssxl\") pod \"d8d1a346-e7a0-4207-9764-6a429a868977\" (UID: \"d8d1a346-e7a0-4207-9764-6a429a868977\") " Oct 31 04:30:33.309091 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:33.309065 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8d1a346-e7a0-4207-9764-6a429a868977-maas-api-openshift-ca" pod "d8d1a346-e7a0-4207-9764-6a429a868977" (UID: "d8d1a346-e7a0-4207-9764-6a429a868977"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:30:33.309436 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:33.309413 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8d1a346-e7a0-4207-9764-6a429a868977-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:33.310982 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:33.310954 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8d1a346-e7a0-4207-9764-6a429a868977-maas-api-xks-ca" pod "d8d1a346-e7a0-4207-9764-6a429a868977" (UID: "d8d1a346-e7a0-4207-9764-6a429a868977"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:30:33.311075 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:33.311001 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8d1a346-e7a0-4207-9764-6a429a868977-kube-api-access-qssxl" pod "d8d1a346-e7a0-4207-9764-6a429a868977" (UID: "d8d1a346-e7a0-4207-9764-6a429a868977"). InnerVolumeSpecName "kube-api-access-qssxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:30:33.410504 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:33.410437 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8d1a346-e7a0-4207-9764-6a429a868977-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:33.410504 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:33.410463 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qssxl\" (UniqueName: \"kubernetes.io/projected/d8d1a346-e7a0-4207-9764-6a429a868977-kube-api-access-qssxl\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:33.919149 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:33.919116 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8d161cf-8d8z4" Oct 31 04:30:33.919359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:33.919142 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8d161cf-8d8z4" event={"ID":"d8d1a346-e7a0-4207-9764-6a429a868977","Type":"ContainerDied","Data":"ee1d5bf7c181b346efeb87ae08e96d3c2bce3d1ddd7f870a6cebcae2b048f070"} Oct 31 04:30:33.919359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:33.919194 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee1d5bf7c181b346efeb87ae08e96d3c2bce3d1ddd7f870a6cebcae2b048f070" Oct 31 04:30:35.330395 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:35.330353 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57d2fa8f-mzcpp"] Oct 31 04:30:35.331032 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:35.331018 5019 cpu_manager.go:414] "containerMap: removing container" podUID="d8d1a346-e7a0-4207-9764-6a429a868977" containerName="revoke-keys" Oct 31 04:30:35.331032 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:35.331033 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8d1a346-e7a0-4207-9764-6a429a868977" containerName="revoke-keys" Oct 31 04:30:35.331141 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:35.331132 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8d1a346-e7a0-4207-9764-6a429a868977" containerName="revoke-keys" Oct 31 04:30:35.335809 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:35.335792 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d2fa8f-mzcpp" Oct 31 04:30:35.354372 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:35.354350 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57d2fa8f-mzcpp"] Oct 31 04:30:35.361951 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:35.361929 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:30:35.362185 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:35.362164 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:30:35.362297 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:35.362172 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:30:35.362297 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:35.362228 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:30:35.435120 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:35.435090 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nltr\" (UniqueName: \"kubernetes.io/projected/b7c75409-79c8-4343-8cb6-117d0ae8f7e6-kube-api-access-6nltr\") pod \"maas-api-revoke-sub-57d2fa8f-mzcpp\" (UID: \"b7c75409-79c8-4343-8cb6-117d0ae8f7e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d2fa8f-mzcpp" Oct 31 04:30:35.435287 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:35.435273 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7c75409-79c8-4343-8cb6-117d0ae8f7e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57d2fa8f-mzcpp\" (UID: \"b7c75409-79c8-4343-8cb6-117d0ae8f7e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d2fa8f-mzcpp" Oct 31 04:30:35.435408 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:35.435359 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7c75409-79c8-4343-8cb6-117d0ae8f7e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57d2fa8f-mzcpp\" (UID: \"b7c75409-79c8-4343-8cb6-117d0ae8f7e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d2fa8f-mzcpp" Oct 31 04:30:35.537011 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:35.536979 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7c75409-79c8-4343-8cb6-117d0ae8f7e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57d2fa8f-mzcpp\" (UID: \"b7c75409-79c8-4343-8cb6-117d0ae8f7e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d2fa8f-mzcpp" Oct 31 04:30:35.537186 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:35.537037 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7c75409-79c8-4343-8cb6-117d0ae8f7e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57d2fa8f-mzcpp\" (UID: \"b7c75409-79c8-4343-8cb6-117d0ae8f7e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d2fa8f-mzcpp" Oct 31 04:30:35.537186 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:35.537122 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nltr\" (UniqueName: \"kubernetes.io/projected/b7c75409-79c8-4343-8cb6-117d0ae8f7e6-kube-api-access-6nltr\") pod \"maas-api-revoke-sub-57d2fa8f-mzcpp\" (UID: \"b7c75409-79c8-4343-8cb6-117d0ae8f7e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d2fa8f-mzcpp" Oct 31 04:30:35.537713 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:35.537688 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7c75409-79c8-4343-8cb6-117d0ae8f7e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57d2fa8f-mzcpp\" (UID: \"b7c75409-79c8-4343-8cb6-117d0ae8f7e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d2fa8f-mzcpp" Oct 31 04:30:35.539325 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:35.539301 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7c75409-79c8-4343-8cb6-117d0ae8f7e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57d2fa8f-mzcpp\" (UID: \"b7c75409-79c8-4343-8cb6-117d0ae8f7e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d2fa8f-mzcpp" Oct 31 04:30:35.560284 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:35.560259 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nltr\" (UniqueName: \"kubernetes.io/projected/b7c75409-79c8-4343-8cb6-117d0ae8f7e6-kube-api-access-6nltr\") pod \"maas-api-revoke-sub-57d2fa8f-mzcpp\" (UID: \"b7c75409-79c8-4343-8cb6-117d0ae8f7e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d2fa8f-mzcpp" Oct 31 04:30:35.649169 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:35.649088 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d2fa8f-mzcpp" Oct 31 04:30:35.960785 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:35.960751 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57d2fa8f-mzcpp"] Oct 31 04:30:35.961047 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:30:35.961019 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7c75409_79c8_4343_8cb6_117d0ae8f7e6.slice/crio-f13f50ce64782947207db519903c9347148e51aab73fd7b1ea40f6377610eca5 WatchSource:0}: Error finding container f13f50ce64782947207db519903c9347148e51aab73fd7b1ea40f6377610eca5: Status 404 returned error can't find the container with id f13f50ce64782947207db519903c9347148e51aab73fd7b1ea40f6377610eca5 Oct 31 04:30:36.943007 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:36.942975 5019 generic.go:356] "Generic (PLEG): container finished" podID="b7c75409-79c8-4343-8cb6-117d0ae8f7e6" containerID="3d91f2721a05811c1ba9501dcdcf30b980d2428f82fa7ef716ba3b40cb487519" exitCode=0 Oct 31 04:30:36.943391 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:36.943047 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d2fa8f-mzcpp" event={"ID":"b7c75409-79c8-4343-8cb6-117d0ae8f7e6","Type":"ContainerDied","Data":"3d91f2721a05811c1ba9501dcdcf30b980d2428f82fa7ef716ba3b40cb487519"} Oct 31 04:30:36.943391 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:36.943089 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d2fa8f-mzcpp" event={"ID":"b7c75409-79c8-4343-8cb6-117d0ae8f7e6","Type":"ContainerStarted","Data":"f13f50ce64782947207db519903c9347148e51aab73fd7b1ea40f6377610eca5"} Oct 31 04:30:38.782955 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:38.782910 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj" Oct 31 04:30:38.789568 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:38.789534 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d2fa8f-mzcpp" Oct 31 04:30:38.882598 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:38.857211 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/618ff6b5-9d72-4ac1-9132-89c0697dfc52-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/618ff6b5-9d72-4ac1-9132-89c0697dfc52-workload-certs\") pod \"618ff6b5-9d72-4ac1-9132-89c0697dfc52\" (UID: \"618ff6b5-9d72-4ac1-9132-89c0697dfc52\") " Oct 31 04:30:38.882598 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:38.857306 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/618ff6b5-9d72-4ac1-9132-89c0697dfc52-kube-api-access-vqkfr\" (UniqueName: \"kubernetes.io/projected/618ff6b5-9d72-4ac1-9132-89c0697dfc52-kube-api-access-vqkfr\") pod \"618ff6b5-9d72-4ac1-9132-89c0697dfc52\" (UID: \"618ff6b5-9d72-4ac1-9132-89c0697dfc52\") " Oct 31 04:30:38.882598 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:38.857386 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/618ff6b5-9d72-4ac1-9132-89c0697dfc52-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/618ff6b5-9d72-4ac1-9132-89c0697dfc52-workload-socket\") pod \"618ff6b5-9d72-4ac1-9132-89c0697dfc52\" (UID: \"618ff6b5-9d72-4ac1-9132-89c0697dfc52\") " Oct 31 04:30:38.882598 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:38.857418 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7c75409-79c8-4343-8cb6-117d0ae8f7e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7c75409-79c8-4343-8cb6-117d0ae8f7e6-maas-api-openshift-ca\") pod \"b7c75409-79c8-4343-8cb6-117d0ae8f7e6\" (UID: \"b7c75409-79c8-4343-8cb6-117d0ae8f7e6\") " Oct 31 04:30:38.882598 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:38.857450 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/618ff6b5-9d72-4ac1-9132-89c0697dfc52-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/618ff6b5-9d72-4ac1-9132-89c0697dfc52-istio-data\") pod \"618ff6b5-9d72-4ac1-9132-89c0697dfc52\" (UID: \"618ff6b5-9d72-4ac1-9132-89c0697dfc52\") " Oct 31 04:30:38.882598 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:38.857528 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/618ff6b5-9d72-4ac1-9132-89c0697dfc52-istio-token\" (UniqueName: \"kubernetes.io/projected/618ff6b5-9d72-4ac1-9132-89c0697dfc52-istio-token\") pod \"618ff6b5-9d72-4ac1-9132-89c0697dfc52\" (UID: \"618ff6b5-9d72-4ac1-9132-89c0697dfc52\") " Oct 31 04:30:38.882598 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:38.857575 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/618ff6b5-9d72-4ac1-9132-89c0697dfc52-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/618ff6b5-9d72-4ac1-9132-89c0697dfc52-istiod-ca-cert\") pod \"618ff6b5-9d72-4ac1-9132-89c0697dfc52\" (UID: \"618ff6b5-9d72-4ac1-9132-89c0697dfc52\") " Oct 31 04:30:38.882598 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:38.857612 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7c75409-79c8-4343-8cb6-117d0ae8f7e6-kube-api-access-6nltr\" (UniqueName: \"kubernetes.io/projected/b7c75409-79c8-4343-8cb6-117d0ae8f7e6-kube-api-access-6nltr\") pod \"b7c75409-79c8-4343-8cb6-117d0ae8f7e6\" (UID: \"b7c75409-79c8-4343-8cb6-117d0ae8f7e6\") " Oct 31 04:30:38.882598 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:38.857693 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7c75409-79c8-4343-8cb6-117d0ae8f7e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7c75409-79c8-4343-8cb6-117d0ae8f7e6-maas-api-xks-ca\") pod \"b7c75409-79c8-4343-8cb6-117d0ae8f7e6\" (UID: \"b7c75409-79c8-4343-8cb6-117d0ae8f7e6\") " Oct 31 04:30:38.882598 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:38.857757 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/618ff6b5-9d72-4ac1-9132-89c0697dfc52-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/618ff6b5-9d72-4ac1-9132-89c0697dfc52-credential-socket\") pod \"618ff6b5-9d72-4ac1-9132-89c0697dfc52\" (UID: \"618ff6b5-9d72-4ac1-9132-89c0697dfc52\") " Oct 31 04:30:38.882598 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:38.857819 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/618ff6b5-9d72-4ac1-9132-89c0697dfc52-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/618ff6b5-9d72-4ac1-9132-89c0697dfc52-istio-podinfo\") pod \"618ff6b5-9d72-4ac1-9132-89c0697dfc52\" (UID: \"618ff6b5-9d72-4ac1-9132-89c0697dfc52\") " Oct 31 04:30:38.882598 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:38.857866 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/618ff6b5-9d72-4ac1-9132-89c0697dfc52-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/618ff6b5-9d72-4ac1-9132-89c0697dfc52-istio-envoy\") pod \"618ff6b5-9d72-4ac1-9132-89c0697dfc52\" (UID: \"618ff6b5-9d72-4ac1-9132-89c0697dfc52\") " Oct 31 04:30:38.882598 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:38.873523 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/618ff6b5-9d72-4ac1-9132-89c0697dfc52-istiod-ca-cert" pod "618ff6b5-9d72-4ac1-9132-89c0697dfc52" (UID: "618ff6b5-9d72-4ac1-9132-89c0697dfc52"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:30:38.882598 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:38.879683 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7c75409-79c8-4343-8cb6-117d0ae8f7e6-maas-api-openshift-ca" pod "b7c75409-79c8-4343-8cb6-117d0ae8f7e6" (UID: "b7c75409-79c8-4343-8cb6-117d0ae8f7e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:30:38.882598 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:38.879889 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/618ff6b5-9d72-4ac1-9132-89c0697dfc52-istio-data" pod "618ff6b5-9d72-4ac1-9132-89c0697dfc52" (UID: "618ff6b5-9d72-4ac1-9132-89c0697dfc52"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:30:38.882598 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:38.879927 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/618ff6b5-9d72-4ac1-9132-89c0697dfc52-workload-certs" pod "618ff6b5-9d72-4ac1-9132-89c0697dfc52" (UID: "618ff6b5-9d72-4ac1-9132-89c0697dfc52"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:30:38.882598 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:38.880076 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/618ff6b5-9d72-4ac1-9132-89c0697dfc52-workload-socket" pod "618ff6b5-9d72-4ac1-9132-89c0697dfc52" (UID: "618ff6b5-9d72-4ac1-9132-89c0697dfc52"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:30:38.882598 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:38.880257 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/618ff6b5-9d72-4ac1-9132-89c0697dfc52-credential-socket" pod "618ff6b5-9d72-4ac1-9132-89c0697dfc52" (UID: "618ff6b5-9d72-4ac1-9132-89c0697dfc52"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:30:38.928870 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:38.885512 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/618ff6b5-9d72-4ac1-9132-89c0697dfc52-istio-token" pod "618ff6b5-9d72-4ac1-9132-89c0697dfc52" (UID: "618ff6b5-9d72-4ac1-9132-89c0697dfc52"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:30:39.062728 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:39.060751 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/618ff6b5-9d72-4ac1-9132-89c0697dfc52-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:39.062728 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:39.060792 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7c75409-79c8-4343-8cb6-117d0ae8f7e6-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:39.062728 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:39.060807 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/618ff6b5-9d72-4ac1-9132-89c0697dfc52-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:39.062728 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:39.060821 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/618ff6b5-9d72-4ac1-9132-89c0697dfc52-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:39.062728 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:39.060835 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/618ff6b5-9d72-4ac1-9132-89c0697dfc52-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:39.062728 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:39.060850 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/618ff6b5-9d72-4ac1-9132-89c0697dfc52-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:39.062728 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:39.060866 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/618ff6b5-9d72-4ac1-9132-89c0697dfc52-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:39.349367 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:39.314507 5019 generic.go:356] "Generic (PLEG): container finished" podID="618ff6b5-9d72-4ac1-9132-89c0697dfc52" containerID="b62294855ceca27bee0fc4caca551009560e7ec849df8d151b99df8b28d3cb55" exitCode=0 Oct 31 04:30:39.349367 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:39.314695 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj" Oct 31 04:30:39.349367 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:39.315070 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj" event={"ID":"618ff6b5-9d72-4ac1-9132-89c0697dfc52","Type":"ContainerDied","Data":"b62294855ceca27bee0fc4caca551009560e7ec849df8d151b99df8b28d3cb55"} Oct 31 04:30:39.349367 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:39.315117 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj" event={"ID":"618ff6b5-9d72-4ac1-9132-89c0697dfc52","Type":"ContainerDied","Data":"b39d27aecd1aba59b5d445f818cfac9636e2d72e7875cdb39295f822aaeb0b5f"} Oct 31 04:30:39.349367 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:39.315147 5019 scope.go:122] "RemoveContainer" containerID="b62294855ceca27bee0fc4caca551009560e7ec849df8d151b99df8b28d3cb55" Oct 31 04:30:39.387613 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:39.374632 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/618ff6b5-9d72-4ac1-9132-89c0697dfc52-istio-podinfo" pod "618ff6b5-9d72-4ac1-9132-89c0697dfc52" (UID: "618ff6b5-9d72-4ac1-9132-89c0697dfc52"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:30:39.415403 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:39.414110 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7c75409-79c8-4343-8cb6-117d0ae8f7e6-maas-api-xks-ca" pod "b7c75409-79c8-4343-8cb6-117d0ae8f7e6" (UID: "b7c75409-79c8-4343-8cb6-117d0ae8f7e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:30:39.418074 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:39.418047 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7c75409-79c8-4343-8cb6-117d0ae8f7e6-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:39.418261 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:39.418245 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/618ff6b5-9d72-4ac1-9132-89c0697dfc52-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:39.556741 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:39.521501 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/618ff6b5-9d72-4ac1-9132-89c0697dfc52-kube-api-access-vqkfr" pod "618ff6b5-9d72-4ac1-9132-89c0697dfc52" (UID: "618ff6b5-9d72-4ac1-9132-89c0697dfc52"). InnerVolumeSpecName "kube-api-access-vqkfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:30:39.556741 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:39.522714 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqkfr\" (UniqueName: \"kubernetes.io/projected/618ff6b5-9d72-4ac1-9132-89c0697dfc52-kube-api-access-vqkfr\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:39.609850 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:39.609769 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7c75409-79c8-4343-8cb6-117d0ae8f7e6-kube-api-access-6nltr" pod "b7c75409-79c8-4343-8cb6-117d0ae8f7e6" (UID: "b7c75409-79c8-4343-8cb6-117d0ae8f7e6"). InnerVolumeSpecName "kube-api-access-6nltr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:30:39.625649 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:39.623465 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/618ff6b5-9d72-4ac1-9132-89c0697dfc52-istio-envoy" pod "618ff6b5-9d72-4ac1-9132-89c0697dfc52" (UID: "618ff6b5-9d72-4ac1-9132-89c0697dfc52"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:30:39.625649 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:39.624888 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nltr\" (UniqueName: \"kubernetes.io/projected/b7c75409-79c8-4343-8cb6-117d0ae8f7e6-kube-api-access-6nltr\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:39.625649 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:39.624919 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/618ff6b5-9d72-4ac1-9132-89c0697dfc52-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:39.856363 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:39.834044 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" podUID="8e2c7041-ad52-49f4-8b55-d16691f4938f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 31 04:30:39.891474 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:39.883250 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d2fa8f-mzcpp" event={"ID":"b7c75409-79c8-4343-8cb6-117d0ae8f7e6","Type":"ContainerDied","Data":"f13f50ce64782947207db519903c9347148e51aab73fd7b1ea40f6377610eca5"} Oct 31 04:30:39.891751 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:39.891728 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f13f50ce64782947207db519903c9347148e51aab73fd7b1ea40f6377610eca5" Oct 31 04:30:39.921615 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:39.920463 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d2fa8f-mzcpp" Oct 31 04:30:39.948627 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:39.948600 5019 scope.go:122] "RemoveContainer" containerID="6de2c0547b9f30b5aa6c9ef198c8c8b2f8765b2d11e5a5c4f95bd38e3a58e368" Oct 31 04:30:39.975646 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:39.975622 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj"] Oct 31 04:30:39.990604 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:30:39.990565 5019 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7c75409_79c8_4343_8cb6_117d0ae8f7e6.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod618ff6b5_9d72_4ac1_9132_89c0697dfc52.slice/crio-b39d27aecd1aba59b5d445f818cfac9636e2d72e7875cdb39295f822aaeb0b5f\": RecentStats: unable to find data in memory cache]" Oct 31 04:30:39.995515 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:39.995490 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-b3ca79-openshift-default-745989d85-6nszj"] Oct 31 04:30:39.999073 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:39.998818 5019 scope.go:122] "RemoveContainer" containerID="b62294855ceca27bee0fc4caca551009560e7ec849df8d151b99df8b28d3cb55" Oct 31 04:30:39.999497 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:30:39.999448 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b62294855ceca27bee0fc4caca551009560e7ec849df8d151b99df8b28d3cb55\": container with ID starting with b62294855ceca27bee0fc4caca551009560e7ec849df8d151b99df8b28d3cb55 not found: ID does not exist" containerID="b62294855ceca27bee0fc4caca551009560e7ec849df8d151b99df8b28d3cb55" Oct 31 04:30:39.999608 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:39.999490 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b62294855ceca27bee0fc4caca551009560e7ec849df8d151b99df8b28d3cb55"} err="failed to get container status \"b62294855ceca27bee0fc4caca551009560e7ec849df8d151b99df8b28d3cb55\": rpc error: code = NotFound desc = could not find container \"b62294855ceca27bee0fc4caca551009560e7ec849df8d151b99df8b28d3cb55\": container with ID starting with b62294855ceca27bee0fc4caca551009560e7ec849df8d151b99df8b28d3cb55 not found: ID does not exist" Oct 31 04:30:39.999608 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:39.999514 5019 scope.go:122] "RemoveContainer" containerID="6de2c0547b9f30b5aa6c9ef198c8c8b2f8765b2d11e5a5c4f95bd38e3a58e368" Oct 31 04:30:40.000011 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:30:39.999950 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6de2c0547b9f30b5aa6c9ef198c8c8b2f8765b2d11e5a5c4f95bd38e3a58e368\": container with ID starting with 6de2c0547b9f30b5aa6c9ef198c8c8b2f8765b2d11e5a5c4f95bd38e3a58e368 not found: ID does not exist" containerID="6de2c0547b9f30b5aa6c9ef198c8c8b2f8765b2d11e5a5c4f95bd38e3a58e368" Oct 31 04:30:40.000011 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:39.999985 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6de2c0547b9f30b5aa6c9ef198c8c8b2f8765b2d11e5a5c4f95bd38e3a58e368"} err="failed to get container status \"6de2c0547b9f30b5aa6c9ef198c8c8b2f8765b2d11e5a5c4f95bd38e3a58e368\": rpc error: code = NotFound desc = could not find container \"6de2c0547b9f30b5aa6c9ef198c8c8b2f8765b2d11e5a5c4f95bd38e3a58e368\": container with ID starting with 6de2c0547b9f30b5aa6c9ef198c8c8b2f8765b2d11e5a5c4f95bd38e3a58e368 not found: ID does not exist" Oct 31 04:30:40.097688 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:40.097649 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7cf9ef-7b7cbd9689-srt2f"] Oct 31 04:30:40.138402 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:40.138369 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7cf9ef-699d468f47-5cpz6"] Oct 31 04:30:41.407920 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:41.407885 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="618ff6b5-9d72-4ac1-9132-89c0697dfc52" path="/var/lib/kubelet/pods/618ff6b5-9d72-4ac1-9132-89c0697dfc52/volumes" Oct 31 04:30:45.098693 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:45.098644 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-7cf9ef-7b7cbd9689-srt2f" podUID="6434973f-f51b-42df-8f28-97d9f310fda0" containerName="payload-processing" containerID="cri-o://af339036524abd3d71adbc9544a8fefb5cb7b0acefee2782b217699f1a01c0c6" gracePeriod=25 Oct 31 04:30:45.138876 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:45.138838 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7cf9ef-699d468f47-5cpz6" podUID="13fbd89f-fe6e-4b64-876e-ac8d3db07113" containerName="payload-pre-processing" containerID="cri-o://8d81ecb730e561317698ca0cd7f826ef33aa1f234f4c747c72983f49f6d6729b" gracePeriod=25 Oct 31 04:30:45.442066 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:45.442038 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-7cf9ef-7b7cbd9689-srt2f" Oct 31 04:30:45.468290 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:45.468268 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7cf9ef-699d468f47-5cpz6" Oct 31 04:30:45.531781 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:45.531744 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6434973f-f51b-42df-8f28-97d9f310fda0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6434973f-f51b-42df-8f28-97d9f310fda0-plugins-config-volume\") pod \"6434973f-f51b-42df-8f28-97d9f310fda0\" (UID: \"6434973f-f51b-42df-8f28-97d9f310fda0\") " Oct 31 04:30:45.531973 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:45.531799 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6434973f-f51b-42df-8f28-97d9f310fda0-kube-api-access-24nlz\" (UniqueName: \"kubernetes.io/projected/6434973f-f51b-42df-8f28-97d9f310fda0-kube-api-access-24nlz\") pod \"6434973f-f51b-42df-8f28-97d9f310fda0\" (UID: \"6434973f-f51b-42df-8f28-97d9f310fda0\") " Oct 31 04:30:45.531973 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:45.531876 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13fbd89f-fe6e-4b64-876e-ac8d3db07113-kube-api-access-txpjb\" (UniqueName: \"kubernetes.io/projected/13fbd89f-fe6e-4b64-876e-ac8d3db07113-kube-api-access-txpjb\") pod \"13fbd89f-fe6e-4b64-876e-ac8d3db07113\" (UID: \"13fbd89f-fe6e-4b64-876e-ac8d3db07113\") " Oct 31 04:30:45.531973 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:45.531916 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13fbd89f-fe6e-4b64-876e-ac8d3db07113-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13fbd89f-fe6e-4b64-876e-ac8d3db07113-plugins-config-volume\") pod \"13fbd89f-fe6e-4b64-876e-ac8d3db07113\" (UID: \"13fbd89f-fe6e-4b64-876e-ac8d3db07113\") " Oct 31 04:30:45.533249 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:45.533217 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6434973f-f51b-42df-8f28-97d9f310fda0-plugins-config-volume" pod "6434973f-f51b-42df-8f28-97d9f310fda0" (UID: "6434973f-f51b-42df-8f28-97d9f310fda0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:30:45.534531 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:45.534510 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13fbd89f-fe6e-4b64-876e-ac8d3db07113-plugins-config-volume" pod "13fbd89f-fe6e-4b64-876e-ac8d3db07113" (UID: "13fbd89f-fe6e-4b64-876e-ac8d3db07113"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:30:45.536136 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:45.536114 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13fbd89f-fe6e-4b64-876e-ac8d3db07113-kube-api-access-txpjb" pod "13fbd89f-fe6e-4b64-876e-ac8d3db07113" (UID: "13fbd89f-fe6e-4b64-876e-ac8d3db07113"). InnerVolumeSpecName "kube-api-access-txpjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:30:45.541925 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:45.541905 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6434973f-f51b-42df-8f28-97d9f310fda0-kube-api-access-24nlz" pod "6434973f-f51b-42df-8f28-97d9f310fda0" (UID: "6434973f-f51b-42df-8f28-97d9f310fda0"). InnerVolumeSpecName "kube-api-access-24nlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:30:45.633446 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:45.633408 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6434973f-f51b-42df-8f28-97d9f310fda0-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:45.633446 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:45.633442 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24nlz\" (UniqueName: \"kubernetes.io/projected/6434973f-f51b-42df-8f28-97d9f310fda0-kube-api-access-24nlz\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:45.633446 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:45.633454 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txpjb\" (UniqueName: \"kubernetes.io/projected/13fbd89f-fe6e-4b64-876e-ac8d3db07113-kube-api-access-txpjb\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:45.633702 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:45.633465 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13fbd89f-fe6e-4b64-876e-ac8d3db07113-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:45.933106 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:45.933070 5019 generic.go:356] "Generic (PLEG): container finished" podID="6434973f-f51b-42df-8f28-97d9f310fda0" containerID="af339036524abd3d71adbc9544a8fefb5cb7b0acefee2782b217699f1a01c0c6" exitCode=0 Oct 31 04:30:45.933311 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:45.933121 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-7cf9ef-7b7cbd9689-srt2f" Oct 31 04:30:45.933311 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:45.933157 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-7cf9ef-7b7cbd9689-srt2f" event={"ID":"6434973f-f51b-42df-8f28-97d9f310fda0","Type":"ContainerDied","Data":"af339036524abd3d71adbc9544a8fefb5cb7b0acefee2782b217699f1a01c0c6"} Oct 31 04:30:45.933311 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:45.933216 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-7cf9ef-7b7cbd9689-srt2f" event={"ID":"6434973f-f51b-42df-8f28-97d9f310fda0","Type":"ContainerDied","Data":"cbde5e14c00b482f0a72c3eff9502fd34412c73afc14ca2d6bbc82e4af009943"} Oct 31 04:30:45.933311 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:45.933247 5019 scope.go:122] "RemoveContainer" containerID="af339036524abd3d71adbc9544a8fefb5cb7b0acefee2782b217699f1a01c0c6" Oct 31 04:30:45.934812 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:45.934788 5019 generic.go:356] "Generic (PLEG): container finished" podID="13fbd89f-fe6e-4b64-876e-ac8d3db07113" containerID="8d81ecb730e561317698ca0cd7f826ef33aa1f234f4c747c72983f49f6d6729b" exitCode=0 Oct 31 04:30:45.934934 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:45.934820 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7cf9ef-699d468f47-5cpz6" event={"ID":"13fbd89f-fe6e-4b64-876e-ac8d3db07113","Type":"ContainerDied","Data":"8d81ecb730e561317698ca0cd7f826ef33aa1f234f4c747c72983f49f6d6729b"} Oct 31 04:30:45.934934 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:45.934837 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7cf9ef-699d468f47-5cpz6" Oct 31 04:30:45.934934 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:45.934854 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7cf9ef-699d468f47-5cpz6" event={"ID":"13fbd89f-fe6e-4b64-876e-ac8d3db07113","Type":"ContainerDied","Data":"6ba1e86e92e355cd00434ce41607cf2c2bf71ea8e9ae461fd0b42e26ea7391bc"} Oct 31 04:30:45.945260 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:45.945211 5019 scope.go:122] "RemoveContainer" containerID="af339036524abd3d71adbc9544a8fefb5cb7b0acefee2782b217699f1a01c0c6" Oct 31 04:30:45.945528 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:30:45.945508 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af339036524abd3d71adbc9544a8fefb5cb7b0acefee2782b217699f1a01c0c6\": container with ID starting with af339036524abd3d71adbc9544a8fefb5cb7b0acefee2782b217699f1a01c0c6 not found: ID does not exist" containerID="af339036524abd3d71adbc9544a8fefb5cb7b0acefee2782b217699f1a01c0c6" Oct 31 04:30:45.945625 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:45.945538 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af339036524abd3d71adbc9544a8fefb5cb7b0acefee2782b217699f1a01c0c6"} err="failed to get container status \"af339036524abd3d71adbc9544a8fefb5cb7b0acefee2782b217699f1a01c0c6\": rpc error: code = NotFound desc = could not find container \"af339036524abd3d71adbc9544a8fefb5cb7b0acefee2782b217699f1a01c0c6\": container with ID starting with af339036524abd3d71adbc9544a8fefb5cb7b0acefee2782b217699f1a01c0c6 not found: ID does not exist" Oct 31 04:30:45.945625 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:45.945560 5019 scope.go:122] "RemoveContainer" containerID="8d81ecb730e561317698ca0cd7f826ef33aa1f234f4c747c72983f49f6d6729b" Oct 31 04:30:45.954567 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:45.954546 5019 scope.go:122] "RemoveContainer" containerID="8d81ecb730e561317698ca0cd7f826ef33aa1f234f4c747c72983f49f6d6729b" Oct 31 04:30:45.954833 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:30:45.954813 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d81ecb730e561317698ca0cd7f826ef33aa1f234f4c747c72983f49f6d6729b\": container with ID starting with 8d81ecb730e561317698ca0cd7f826ef33aa1f234f4c747c72983f49f6d6729b not found: ID does not exist" containerID="8d81ecb730e561317698ca0cd7f826ef33aa1f234f4c747c72983f49f6d6729b" Oct 31 04:30:45.954889 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:45.954840 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d81ecb730e561317698ca0cd7f826ef33aa1f234f4c747c72983f49f6d6729b"} err="failed to get container status \"8d81ecb730e561317698ca0cd7f826ef33aa1f234f4c747c72983f49f6d6729b\": rpc error: code = NotFound desc = could not find container \"8d81ecb730e561317698ca0cd7f826ef33aa1f234f4c747c72983f49f6d6729b\": container with ID starting with 8d81ecb730e561317698ca0cd7f826ef33aa1f234f4c747c72983f49f6d6729b not found: ID does not exist" Oct 31 04:30:45.985653 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:45.985622 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7cf9ef-7b7cbd9689-srt2f"] Oct 31 04:30:45.991728 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:45.991696 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7cf9ef-7b7cbd9689-srt2f"] Oct 31 04:30:46.025539 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:46.025496 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7cf9ef-699d468f47-5cpz6"] Oct 31 04:30:46.029899 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:46.029872 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7cf9ef-699d468f47-5cpz6"] Oct 31 04:30:46.623643 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:46.623598 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92d74b5e-q7pb7"] Oct 31 04:30:46.624362 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:46.624300 5019 cpu_manager.go:414] "containerMap: removing container" podUID="13fbd89f-fe6e-4b64-876e-ac8d3db07113" containerName="payload-pre-processing" Oct 31 04:30:46.624362 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:46.624319 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13fbd89f-fe6e-4b64-876e-ac8d3db07113" containerName="payload-pre-processing" Oct 31 04:30:46.624362 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:46.624351 5019 cpu_manager.go:414] "containerMap: removing container" podUID="b7c75409-79c8-4343-8cb6-117d0ae8f7e6" containerName="revoke-keys" Oct 31 04:30:46.624362 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:46.624359 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7c75409-79c8-4343-8cb6-117d0ae8f7e6" containerName="revoke-keys" Oct 31 04:30:46.624593 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:46.624375 5019 cpu_manager.go:414] "containerMap: removing container" podUID="618ff6b5-9d72-4ac1-9132-89c0697dfc52" containerName="istio-proxy" Oct 31 04:30:46.624593 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:46.624380 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="618ff6b5-9d72-4ac1-9132-89c0697dfc52" containerName="istio-proxy" Oct 31 04:30:46.624593 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:46.624399 5019 cpu_manager.go:414] "containerMap: removing container" podUID="6434973f-f51b-42df-8f28-97d9f310fda0" containerName="payload-processing" Oct 31 04:30:46.624593 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:46.624404 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6434973f-f51b-42df-8f28-97d9f310fda0" containerName="payload-processing" Oct 31 04:30:46.624593 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:46.624411 5019 cpu_manager.go:414] "containerMap: removing container" podUID="618ff6b5-9d72-4ac1-9132-89c0697dfc52" containerName="istio-proxy" Oct 31 04:30:46.624593 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:46.624416 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="618ff6b5-9d72-4ac1-9132-89c0697dfc52" containerName="istio-proxy" Oct 31 04:30:46.624593 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:46.624515 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="13fbd89f-fe6e-4b64-876e-ac8d3db07113" containerName="payload-pre-processing" Oct 31 04:30:46.624593 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:46.624523 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7c75409-79c8-4343-8cb6-117d0ae8f7e6" containerName="revoke-keys" Oct 31 04:30:46.624593 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:46.624530 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="618ff6b5-9d72-4ac1-9132-89c0697dfc52" containerName="istio-proxy" Oct 31 04:30:46.624593 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:46.624540 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="6434973f-f51b-42df-8f28-97d9f310fda0" containerName="payload-processing" Oct 31 04:30:46.624593 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:46.624546 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="618ff6b5-9d72-4ac1-9132-89c0697dfc52" containerName="istio-proxy" Oct 31 04:30:46.629847 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:46.629824 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92d74b5e-q7pb7" Oct 31 04:30:46.647287 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:46.647255 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92d74b5e-q7pb7"] Oct 31 04:30:46.654981 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:46.654947 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:30:46.654981 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:46.654952 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:30:46.655183 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:46.654978 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:30:46.656658 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:46.656634 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:30:46.748181 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:46.748145 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ea1146c-92d8-42f1-b7e2-5b03b24e0dae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92d74b5e-q7pb7\" (UID: \"2ea1146c-92d8-42f1-b7e2-5b03b24e0dae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92d74b5e-q7pb7" Oct 31 04:30:46.748181 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:46.748193 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tz5s\" (UniqueName: \"kubernetes.io/projected/2ea1146c-92d8-42f1-b7e2-5b03b24e0dae-kube-api-access-9tz5s\") pod \"maas-api-revoke-sub-92d74b5e-q7pb7\" (UID: \"2ea1146c-92d8-42f1-b7e2-5b03b24e0dae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92d74b5e-q7pb7" Oct 31 04:30:46.748527 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:46.748435 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ea1146c-92d8-42f1-b7e2-5b03b24e0dae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92d74b5e-q7pb7\" (UID: \"2ea1146c-92d8-42f1-b7e2-5b03b24e0dae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92d74b5e-q7pb7" Oct 31 04:30:46.850163 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:46.850118 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ea1146c-92d8-42f1-b7e2-5b03b24e0dae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92d74b5e-q7pb7\" (UID: \"2ea1146c-92d8-42f1-b7e2-5b03b24e0dae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92d74b5e-q7pb7" Oct 31 04:30:46.850163 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:46.850167 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tz5s\" (UniqueName: \"kubernetes.io/projected/2ea1146c-92d8-42f1-b7e2-5b03b24e0dae-kube-api-access-9tz5s\") pod \"maas-api-revoke-sub-92d74b5e-q7pb7\" (UID: \"2ea1146c-92d8-42f1-b7e2-5b03b24e0dae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92d74b5e-q7pb7" Oct 31 04:30:46.850482 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:46.850250 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ea1146c-92d8-42f1-b7e2-5b03b24e0dae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92d74b5e-q7pb7\" (UID: \"2ea1146c-92d8-42f1-b7e2-5b03b24e0dae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92d74b5e-q7pb7" Oct 31 04:30:46.850891 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:46.850858 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ea1146c-92d8-42f1-b7e2-5b03b24e0dae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92d74b5e-q7pb7\" (UID: \"2ea1146c-92d8-42f1-b7e2-5b03b24e0dae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92d74b5e-q7pb7" Oct 31 04:30:46.852664 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:46.852643 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ea1146c-92d8-42f1-b7e2-5b03b24e0dae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92d74b5e-q7pb7\" (UID: \"2ea1146c-92d8-42f1-b7e2-5b03b24e0dae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92d74b5e-q7pb7" Oct 31 04:30:46.874116 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:46.874041 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tz5s\" (UniqueName: \"kubernetes.io/projected/2ea1146c-92d8-42f1-b7e2-5b03b24e0dae-kube-api-access-9tz5s\") pod \"maas-api-revoke-sub-92d74b5e-q7pb7\" (UID: \"2ea1146c-92d8-42f1-b7e2-5b03b24e0dae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92d74b5e-q7pb7" Oct 31 04:30:46.945356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:46.945298 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92d74b5e-q7pb7" Oct 31 04:30:47.249655 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:47.249627 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92d74b5e-q7pb7"] Oct 31 04:30:47.251419 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:30:47.251393 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ea1146c_92d8_42f1_b7e2_5b03b24e0dae.slice/crio-95588b8e7b1f833de9c5dd27c0958cb56f048975f3460c2f8547b76f4ea0f181 WatchSource:0}: Error finding container 95588b8e7b1f833de9c5dd27c0958cb56f048975f3460c2f8547b76f4ea0f181: Status 404 returned error can't find the container with id 95588b8e7b1f833de9c5dd27c0958cb56f048975f3460c2f8547b76f4ea0f181 Oct 31 04:30:47.417852 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:47.417807 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13fbd89f-fe6e-4b64-876e-ac8d3db07113" path="/var/lib/kubelet/pods/13fbd89f-fe6e-4b64-876e-ac8d3db07113/volumes" Oct 31 04:30:47.418447 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:47.418430 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6434973f-f51b-42df-8f28-97d9f310fda0" path="/var/lib/kubelet/pods/6434973f-f51b-42df-8f28-97d9f310fda0/volumes" Oct 31 04:30:47.954062 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:47.953975 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92d74b5e-q7pb7" event={"ID":"2ea1146c-92d8-42f1-b7e2-5b03b24e0dae","Type":"ContainerStarted","Data":"de6b3d01e5343f1bc53229b51db6da07c35751ce45800f098ccb87b2e3176698"} Oct 31 04:30:47.954062 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:47.954032 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92d74b5e-q7pb7" event={"ID":"2ea1146c-92d8-42f1-b7e2-5b03b24e0dae","Type":"ContainerStarted","Data":"95588b8e7b1f833de9c5dd27c0958cb56f048975f3460c2f8547b76f4ea0f181"} Oct 31 04:30:48.063410 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:48.063354 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92d74b5e-q7pb7" podStartSLOduration=2.063324647 podStartE2EDuration="2.063324647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:30:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:30:48.059124061 +0000 UTC m=+1602.815462105" watchObservedRunningTime="2026-10-31 04:30:48.063324647 +0000 UTC m=+1602.819662685" Oct 31 04:30:48.356521 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:48.356484 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f393145-c6hh2"] Oct 31 04:30:48.360925 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:48.360903 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f393145-c6hh2" Oct 31 04:30:48.381425 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:48.381388 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f393145-c6hh2"] Oct 31 04:30:48.469615 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:48.469570 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8da94f34-3152-486f-9da7-63ffb4186238-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f393145-c6hh2\" (UID: \"8da94f34-3152-486f-9da7-63ffb4186238\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f393145-c6hh2" Oct 31 04:30:48.469828 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:48.469685 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q6lx\" (UniqueName: \"kubernetes.io/projected/8da94f34-3152-486f-9da7-63ffb4186238-kube-api-access-2q6lx\") pod \"maas-api-revoke-sub-9f393145-c6hh2\" (UID: \"8da94f34-3152-486f-9da7-63ffb4186238\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f393145-c6hh2" Oct 31 04:30:48.469939 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:48.469913 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8da94f34-3152-486f-9da7-63ffb4186238-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f393145-c6hh2\" (UID: \"8da94f34-3152-486f-9da7-63ffb4186238\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f393145-c6hh2" Oct 31 04:30:48.571772 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:48.571737 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8da94f34-3152-486f-9da7-63ffb4186238-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f393145-c6hh2\" (UID: \"8da94f34-3152-486f-9da7-63ffb4186238\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f393145-c6hh2" Oct 31 04:30:48.571986 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:48.571936 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8da94f34-3152-486f-9da7-63ffb4186238-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f393145-c6hh2\" (UID: \"8da94f34-3152-486f-9da7-63ffb4186238\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f393145-c6hh2" Oct 31 04:30:48.572057 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:48.572033 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q6lx\" (UniqueName: \"kubernetes.io/projected/8da94f34-3152-486f-9da7-63ffb4186238-kube-api-access-2q6lx\") pod \"maas-api-revoke-sub-9f393145-c6hh2\" (UID: \"8da94f34-3152-486f-9da7-63ffb4186238\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f393145-c6hh2" Oct 31 04:30:48.572664 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:48.572634 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8da94f34-3152-486f-9da7-63ffb4186238-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f393145-c6hh2\" (UID: \"8da94f34-3152-486f-9da7-63ffb4186238\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f393145-c6hh2" Oct 31 04:30:48.574386 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:48.574364 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8da94f34-3152-486f-9da7-63ffb4186238-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f393145-c6hh2\" (UID: \"8da94f34-3152-486f-9da7-63ffb4186238\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f393145-c6hh2" Oct 31 04:30:48.596120 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:48.596089 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q6lx\" (UniqueName: \"kubernetes.io/projected/8da94f34-3152-486f-9da7-63ffb4186238-kube-api-access-2q6lx\") pod \"maas-api-revoke-sub-9f393145-c6hh2\" (UID: \"8da94f34-3152-486f-9da7-63ffb4186238\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f393145-c6hh2" Oct 31 04:30:48.674881 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:48.674845 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f393145-c6hh2" Oct 31 04:30:48.963426 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:48.963390 5019 generic.go:356] "Generic (PLEG): container finished" podID="2ea1146c-92d8-42f1-b7e2-5b03b24e0dae" containerID="de6b3d01e5343f1bc53229b51db6da07c35751ce45800f098ccb87b2e3176698" exitCode=0 Oct 31 04:30:48.963426 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:48.963431 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92d74b5e-q7pb7" event={"ID":"2ea1146c-92d8-42f1-b7e2-5b03b24e0dae","Type":"ContainerDied","Data":"de6b3d01e5343f1bc53229b51db6da07c35751ce45800f098ccb87b2e3176698"} Oct 31 04:30:48.984163 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:48.984092 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f393145-c6hh2"] Oct 31 04:30:48.985137 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:30:48.985112 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8da94f34_3152_486f_9da7_63ffb4186238.slice/crio-52be73995b7299f043d0bac7a55b45defd844f5671b725917fd27a56224d0858 WatchSource:0}: Error finding container 52be73995b7299f043d0bac7a55b45defd844f5671b725917fd27a56224d0858: Status 404 returned error can't find the container with id 52be73995b7299f043d0bac7a55b45defd844f5671b725917fd27a56224d0858 Oct 31 04:30:49.974718 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:49.974676 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f393145-c6hh2" event={"ID":"8da94f34-3152-486f-9da7-63ffb4186238","Type":"ContainerStarted","Data":"5874438da227c29d36da52f00273f685a5d85d8d149ff5bdb4f0647db508be3c"} Oct 31 04:30:49.975181 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:49.974744 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f393145-c6hh2" event={"ID":"8da94f34-3152-486f-9da7-63ffb4186238","Type":"ContainerStarted","Data":"52be73995b7299f043d0bac7a55b45defd844f5671b725917fd27a56224d0858"} Oct 31 04:30:50.016878 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:50.016817 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f393145-c6hh2" podStartSLOduration=2.016802348 podStartE2EDuration="2.016802348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:30:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:30:50.013702218 +0000 UTC m=+1604.770040256" watchObservedRunningTime="2026-10-31 04:30:50.016802348 +0000 UTC m=+1604.773140386" Oct 31 04:30:50.183276 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:50.183246 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92d74b5e-q7pb7" Oct 31 04:30:50.294740 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:50.294651 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ea1146c-92d8-42f1-b7e2-5b03b24e0dae-kube-api-access-9tz5s\" (UniqueName: \"kubernetes.io/projected/2ea1146c-92d8-42f1-b7e2-5b03b24e0dae-kube-api-access-9tz5s\") pod \"2ea1146c-92d8-42f1-b7e2-5b03b24e0dae\" (UID: \"2ea1146c-92d8-42f1-b7e2-5b03b24e0dae\") " Oct 31 04:30:50.294898 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:50.294803 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ea1146c-92d8-42f1-b7e2-5b03b24e0dae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ea1146c-92d8-42f1-b7e2-5b03b24e0dae-maas-api-xks-ca\") pod \"2ea1146c-92d8-42f1-b7e2-5b03b24e0dae\" (UID: \"2ea1146c-92d8-42f1-b7e2-5b03b24e0dae\") " Oct 31 04:30:50.294898 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:50.294848 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ea1146c-92d8-42f1-b7e2-5b03b24e0dae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ea1146c-92d8-42f1-b7e2-5b03b24e0dae-maas-api-openshift-ca\") pod \"2ea1146c-92d8-42f1-b7e2-5b03b24e0dae\" (UID: \"2ea1146c-92d8-42f1-b7e2-5b03b24e0dae\") " Oct 31 04:30:50.295242 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:50.295205 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ea1146c-92d8-42f1-b7e2-5b03b24e0dae-maas-api-openshift-ca" pod "2ea1146c-92d8-42f1-b7e2-5b03b24e0dae" (UID: "2ea1146c-92d8-42f1-b7e2-5b03b24e0dae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:30:50.295511 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:50.295488 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ea1146c-92d8-42f1-b7e2-5b03b24e0dae-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:50.297387 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:50.297365 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ea1146c-92d8-42f1-b7e2-5b03b24e0dae-maas-api-xks-ca" pod "2ea1146c-92d8-42f1-b7e2-5b03b24e0dae" (UID: "2ea1146c-92d8-42f1-b7e2-5b03b24e0dae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:30:50.297482 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:50.297396 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ea1146c-92d8-42f1-b7e2-5b03b24e0dae-kube-api-access-9tz5s" pod "2ea1146c-92d8-42f1-b7e2-5b03b24e0dae" (UID: "2ea1146c-92d8-42f1-b7e2-5b03b24e0dae"). InnerVolumeSpecName "kube-api-access-9tz5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:30:50.397374 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:50.397287 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tz5s\" (UniqueName: \"kubernetes.io/projected/2ea1146c-92d8-42f1-b7e2-5b03b24e0dae-kube-api-access-9tz5s\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:50.397374 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:50.397317 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ea1146c-92d8-42f1-b7e2-5b03b24e0dae-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:50.984849 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:50.984818 5019 generic.go:356] "Generic (PLEG): container finished" podID="8da94f34-3152-486f-9da7-63ffb4186238" containerID="5874438da227c29d36da52f00273f685a5d85d8d149ff5bdb4f0647db508be3c" exitCode=0 Oct 31 04:30:50.985305 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:50.984908 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f393145-c6hh2" event={"ID":"8da94f34-3152-486f-9da7-63ffb4186238","Type":"ContainerDied","Data":"5874438da227c29d36da52f00273f685a5d85d8d149ff5bdb4f0647db508be3c"} Oct 31 04:30:50.986309 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:50.986290 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92d74b5e-q7pb7" Oct 31 04:30:50.986435 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:50.986365 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92d74b5e-q7pb7" event={"ID":"2ea1146c-92d8-42f1-b7e2-5b03b24e0dae","Type":"ContainerDied","Data":"95588b8e7b1f833de9c5dd27c0958cb56f048975f3460c2f8547b76f4ea0f181"} Oct 31 04:30:50.986435 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:50.986405 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95588b8e7b1f833de9c5dd27c0958cb56f048975f3460c2f8547b76f4ea0f181" Oct 31 04:30:52.164364 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:52.164314 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f393145-c6hh2" Oct 31 04:30:52.330328 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:52.330257 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8da94f34-3152-486f-9da7-63ffb4186238-kube-api-access-2q6lx\" (UniqueName: \"kubernetes.io/projected/8da94f34-3152-486f-9da7-63ffb4186238-kube-api-access-2q6lx\") pod \"8da94f34-3152-486f-9da7-63ffb4186238\" (UID: \"8da94f34-3152-486f-9da7-63ffb4186238\") " Oct 31 04:30:52.330490 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:52.330439 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8da94f34-3152-486f-9da7-63ffb4186238-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8da94f34-3152-486f-9da7-63ffb4186238-maas-api-xks-ca\") pod \"8da94f34-3152-486f-9da7-63ffb4186238\" (UID: \"8da94f34-3152-486f-9da7-63ffb4186238\") " Oct 31 04:30:52.330553 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:52.330537 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8da94f34-3152-486f-9da7-63ffb4186238-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8da94f34-3152-486f-9da7-63ffb4186238-maas-api-openshift-ca\") pod \"8da94f34-3152-486f-9da7-63ffb4186238\" (UID: \"8da94f34-3152-486f-9da7-63ffb4186238\") " Oct 31 04:30:52.330904 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:52.330877 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8da94f34-3152-486f-9da7-63ffb4186238-maas-api-openshift-ca" pod "8da94f34-3152-486f-9da7-63ffb4186238" (UID: "8da94f34-3152-486f-9da7-63ffb4186238"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:30:52.331087 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:52.331065 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8da94f34-3152-486f-9da7-63ffb4186238-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:52.332463 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:52.332437 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8da94f34-3152-486f-9da7-63ffb4186238-maas-api-xks-ca" pod "8da94f34-3152-486f-9da7-63ffb4186238" (UID: "8da94f34-3152-486f-9da7-63ffb4186238"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:30:52.332535 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:52.332442 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8da94f34-3152-486f-9da7-63ffb4186238-kube-api-access-2q6lx" pod "8da94f34-3152-486f-9da7-63ffb4186238" (UID: "8da94f34-3152-486f-9da7-63ffb4186238"). InnerVolumeSpecName "kube-api-access-2q6lx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:30:52.389877 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:30:52.389840 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:30:52.431896 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:52.431867 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q6lx\" (UniqueName: \"kubernetes.io/projected/8da94f34-3152-486f-9da7-63ffb4186238-kube-api-access-2q6lx\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:52.431896 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:52.431899 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8da94f34-3152-486f-9da7-63ffb4186238-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:30:53.005926 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:53.005894 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f393145-c6hh2" Oct 31 04:30:53.006108 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:53.005941 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f393145-c6hh2" event={"ID":"8da94f34-3152-486f-9da7-63ffb4186238","Type":"ContainerDied","Data":"52be73995b7299f043d0bac7a55b45defd844f5671b725917fd27a56224d0858"} Oct 31 04:30:53.006108 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:53.005993 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52be73995b7299f043d0bac7a55b45defd844f5671b725917fd27a56224d0858" Oct 31 04:30:58.561509 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:58.561470 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s"] Oct 31 04:30:58.562388 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:58.562364 5019 cpu_manager.go:414] "containerMap: removing container" podUID="2ea1146c-92d8-42f1-b7e2-5b03b24e0dae" containerName="revoke-keys" Oct 31 04:30:58.562388 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:58.562386 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ea1146c-92d8-42f1-b7e2-5b03b24e0dae" containerName="revoke-keys" Oct 31 04:30:58.562583 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:58.562401 5019 cpu_manager.go:414] "containerMap: removing container" podUID="8da94f34-3152-486f-9da7-63ffb4186238" containerName="revoke-keys" Oct 31 04:30:58.562583 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:58.562406 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8da94f34-3152-486f-9da7-63ffb4186238" containerName="revoke-keys" Oct 31 04:30:58.562583 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:58.562546 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ea1146c-92d8-42f1-b7e2-5b03b24e0dae" containerName="revoke-keys" Oct 31 04:30:58.562583 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:58.562556 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="8da94f34-3152-486f-9da7-63ffb4186238" containerName="revoke-keys" Oct 31 04:30:58.565826 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:58.565802 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s" Oct 31 04:30:58.586768 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:58.586734 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-e6a0de-openshift-default-dockercfg-2sblj\"" Oct 31 04:30:58.607433 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:58.607401 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s"] Oct 31 04:30:58.618167 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:58.618126 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eb20738-52aa-4cc7-990a-ddd8c704e683-istio-podinfo\") pod \"e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s\" (UID: \"9eb20738-52aa-4cc7-990a-ddd8c704e683\") " pod="openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s" Oct 31 04:30:58.618367 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:58.618245 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb20738-52aa-4cc7-990a-ddd8c704e683-workload-socket\") pod \"e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s\" (UID: \"9eb20738-52aa-4cc7-990a-ddd8c704e683\") " pod="openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s" Oct 31 04:30:58.618367 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:58.618358 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eb20738-52aa-4cc7-990a-ddd8c704e683-istio-token\") pod \"e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s\" (UID: \"9eb20738-52aa-4cc7-990a-ddd8c704e683\") " pod="openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s" Oct 31 04:30:58.618520 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:58.618410 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb20738-52aa-4cc7-990a-ddd8c704e683-credential-socket\") pod \"e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s\" (UID: \"9eb20738-52aa-4cc7-990a-ddd8c704e683\") " pod="openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s" Oct 31 04:30:58.618520 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:58.618470 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eb20738-52aa-4cc7-990a-ddd8c704e683-istio-envoy\") pod \"e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s\" (UID: \"9eb20738-52aa-4cc7-990a-ddd8c704e683\") " pod="openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s" Oct 31 04:30:58.618623 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:58.618535 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eb20738-52aa-4cc7-990a-ddd8c704e683-istio-data\") pod \"e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s\" (UID: \"9eb20738-52aa-4cc7-990a-ddd8c704e683\") " pod="openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s" Oct 31 04:30:58.618623 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:58.618570 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj8bz\" (UniqueName: \"kubernetes.io/projected/9eb20738-52aa-4cc7-990a-ddd8c704e683-kube-api-access-bj8bz\") pod \"e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s\" (UID: \"9eb20738-52aa-4cc7-990a-ddd8c704e683\") " pod="openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s" Oct 31 04:30:58.618623 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:58.618610 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eb20738-52aa-4cc7-990a-ddd8c704e683-workload-certs\") pod \"e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s\" (UID: \"9eb20738-52aa-4cc7-990a-ddd8c704e683\") " pod="openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s" Oct 31 04:30:58.618775 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:58.618636 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eb20738-52aa-4cc7-990a-ddd8c704e683-istiod-ca-cert\") pod \"e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s\" (UID: \"9eb20738-52aa-4cc7-990a-ddd8c704e683\") " pod="openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s" Oct 31 04:30:58.720477 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:58.720435 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eb20738-52aa-4cc7-990a-ddd8c704e683-istio-podinfo\") pod \"e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s\" (UID: \"9eb20738-52aa-4cc7-990a-ddd8c704e683\") " pod="openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s" Oct 31 04:30:58.720673 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:58.720559 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb20738-52aa-4cc7-990a-ddd8c704e683-workload-socket\") pod \"e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s\" (UID: \"9eb20738-52aa-4cc7-990a-ddd8c704e683\") " pod="openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s" Oct 31 04:30:58.720673 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:58.720658 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eb20738-52aa-4cc7-990a-ddd8c704e683-istio-token\") pod \"e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s\" (UID: \"9eb20738-52aa-4cc7-990a-ddd8c704e683\") " pod="openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s" Oct 31 04:30:58.720802 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:58.720710 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb20738-52aa-4cc7-990a-ddd8c704e683-credential-socket\") pod \"e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s\" (UID: \"9eb20738-52aa-4cc7-990a-ddd8c704e683\") " pod="openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s" Oct 31 04:30:58.720802 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:58.720769 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eb20738-52aa-4cc7-990a-ddd8c704e683-istio-envoy\") pod \"e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s\" (UID: \"9eb20738-52aa-4cc7-990a-ddd8c704e683\") " pod="openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s" Oct 31 04:30:58.720917 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:58.720840 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eb20738-52aa-4cc7-990a-ddd8c704e683-istio-data\") pod \"e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s\" (UID: \"9eb20738-52aa-4cc7-990a-ddd8c704e683\") " pod="openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s" Oct 31 04:30:58.720917 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:58.720874 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj8bz\" (UniqueName: \"kubernetes.io/projected/9eb20738-52aa-4cc7-990a-ddd8c704e683-kube-api-access-bj8bz\") pod \"e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s\" (UID: \"9eb20738-52aa-4cc7-990a-ddd8c704e683\") " pod="openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s" Oct 31 04:30:58.721032 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:58.720919 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eb20738-52aa-4cc7-990a-ddd8c704e683-workload-certs\") pod \"e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s\" (UID: \"9eb20738-52aa-4cc7-990a-ddd8c704e683\") " pod="openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s" Oct 31 04:30:58.721032 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:58.720949 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eb20738-52aa-4cc7-990a-ddd8c704e683-istiod-ca-cert\") pod \"e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s\" (UID: \"9eb20738-52aa-4cc7-990a-ddd8c704e683\") " pod="openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s" Oct 31 04:30:58.721143 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:58.721064 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb20738-52aa-4cc7-990a-ddd8c704e683-workload-socket\") pod \"e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s\" (UID: \"9eb20738-52aa-4cc7-990a-ddd8c704e683\") " pod="openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s" Oct 31 04:30:58.721143 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:58.721070 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb20738-52aa-4cc7-990a-ddd8c704e683-credential-socket\") pod \"e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s\" (UID: \"9eb20738-52aa-4cc7-990a-ddd8c704e683\") " pod="openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s" Oct 31 04:30:58.721289 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:58.721263 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eb20738-52aa-4cc7-990a-ddd8c704e683-istio-data\") pod \"e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s\" (UID: \"9eb20738-52aa-4cc7-990a-ddd8c704e683\") " pod="openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s" Oct 31 04:30:58.721450 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:58.721426 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eb20738-52aa-4cc7-990a-ddd8c704e683-workload-certs\") pod \"e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s\" (UID: \"9eb20738-52aa-4cc7-990a-ddd8c704e683\") " pod="openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s" Oct 31 04:30:58.721725 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:58.721701 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eb20738-52aa-4cc7-990a-ddd8c704e683-istiod-ca-cert\") pod \"e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s\" (UID: \"9eb20738-52aa-4cc7-990a-ddd8c704e683\") " pod="openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s" Oct 31 04:30:58.723251 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:58.723226 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eb20738-52aa-4cc7-990a-ddd8c704e683-istio-podinfo\") pod \"e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s\" (UID: \"9eb20738-52aa-4cc7-990a-ddd8c704e683\") " pod="openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s" Oct 31 04:30:58.723558 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:58.723538 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eb20738-52aa-4cc7-990a-ddd8c704e683-istio-envoy\") pod \"e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s\" (UID: \"9eb20738-52aa-4cc7-990a-ddd8c704e683\") " pod="openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s" Oct 31 04:30:58.744046 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:58.744017 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eb20738-52aa-4cc7-990a-ddd8c704e683-istio-token\") pod \"e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s\" (UID: \"9eb20738-52aa-4cc7-990a-ddd8c704e683\") " pod="openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s" Oct 31 04:30:58.744792 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:58.744765 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj8bz\" (UniqueName: \"kubernetes.io/projected/9eb20738-52aa-4cc7-990a-ddd8c704e683-kube-api-access-bj8bz\") pod \"e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s\" (UID: \"9eb20738-52aa-4cc7-990a-ddd8c704e683\") " pod="openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s" Oct 31 04:30:58.884836 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:58.884752 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s" Oct 31 04:30:59.285232 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:59.285204 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s"] Oct 31 04:30:59.287177 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:30:59.287146 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eb20738_52aa_4cc7_990a_ddd8c704e683.slice/crio-a245874a62a57850ef1d2f1b7a663271d42a68128d44df4ecc87355e267c6d97 WatchSource:0}: Error finding container a245874a62a57850ef1d2f1b7a663271d42a68128d44df4ecc87355e267c6d97: Status 404 returned error can't find the container with id a245874a62a57850ef1d2f1b7a663271d42a68128d44df4ecc87355e267c6d97 Oct 31 04:30:59.290430 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:59.290387 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:30:59.290513 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:59.290477 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:30:59.290555 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:30:59.290520 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:31:00.112358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:00.111988 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s" event={"ID":"9eb20738-52aa-4cc7-990a-ddd8c704e683","Type":"ContainerStarted","Data":"0e312c3b42b8ad05f96fb8246c9830331d8048aae53047546848409bf98da983"} Oct 31 04:31:00.112358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:00.112075 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s" event={"ID":"9eb20738-52aa-4cc7-990a-ddd8c704e683","Type":"ContainerStarted","Data":"a245874a62a57850ef1d2f1b7a663271d42a68128d44df4ecc87355e267c6d97"} Oct 31 04:31:00.241356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:00.232218 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s" podStartSLOduration=2.232197719 podStartE2EDuration="2.232197719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:30:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:31:00.205678975 +0000 UTC m=+1614.962017015" watchObservedRunningTime="2026-10-31 04:31:00.232197719 +0000 UTC m=+1614.988535759" Oct 31 04:31:00.422420 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:00.408365 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-549ca28e-zhcw4"] Oct 31 04:31:00.434017 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:00.433987 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-549ca28e-zhcw4" Oct 31 04:31:00.465354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:00.464666 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:31:00.486350 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:00.483939 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:31:00.486350 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:00.484580 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:31:00.486350 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:00.484699 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:31:00.494351 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:00.492172 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-549ca28e-zhcw4"] Oct 31 04:31:00.620508 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:00.620470 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9098da5-5571-4cfb-8c10-a1541763c969-maas-api-xks-ca\") pod \"maas-api-revoke-sub-549ca28e-zhcw4\" (UID: \"d9098da5-5571-4cfb-8c10-a1541763c969\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-549ca28e-zhcw4" Oct 31 04:31:00.620853 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:00.620800 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzpbh\" (UniqueName: \"kubernetes.io/projected/d9098da5-5571-4cfb-8c10-a1541763c969-kube-api-access-vzpbh\") pod \"maas-api-revoke-sub-549ca28e-zhcw4\" (UID: \"d9098da5-5571-4cfb-8c10-a1541763c969\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-549ca28e-zhcw4" Oct 31 04:31:00.621456 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:00.621432 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9098da5-5571-4cfb-8c10-a1541763c969-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-549ca28e-zhcw4\" (UID: \"d9098da5-5571-4cfb-8c10-a1541763c969\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-549ca28e-zhcw4" Oct 31 04:31:00.817369 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:00.812922 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9098da5-5571-4cfb-8c10-a1541763c969-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-549ca28e-zhcw4\" (UID: \"d9098da5-5571-4cfb-8c10-a1541763c969\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-549ca28e-zhcw4" Oct 31 04:31:00.817369 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:00.813125 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9098da5-5571-4cfb-8c10-a1541763c969-maas-api-xks-ca\") pod \"maas-api-revoke-sub-549ca28e-zhcw4\" (UID: \"d9098da5-5571-4cfb-8c10-a1541763c969\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-549ca28e-zhcw4" Oct 31 04:31:00.817369 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:00.813164 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzpbh\" (UniqueName: \"kubernetes.io/projected/d9098da5-5571-4cfb-8c10-a1541763c969-kube-api-access-vzpbh\") pod \"maas-api-revoke-sub-549ca28e-zhcw4\" (UID: \"d9098da5-5571-4cfb-8c10-a1541763c969\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-549ca28e-zhcw4" Oct 31 04:31:00.817369 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:00.839689 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9098da5-5571-4cfb-8c10-a1541763c969-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-549ca28e-zhcw4\" (UID: \"d9098da5-5571-4cfb-8c10-a1541763c969\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-549ca28e-zhcw4" Oct 31 04:31:00.886768 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:00.885926 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzpbh\" (UniqueName: \"kubernetes.io/projected/d9098da5-5571-4cfb-8c10-a1541763c969-kube-api-access-vzpbh\") pod \"maas-api-revoke-sub-549ca28e-zhcw4\" (UID: \"d9098da5-5571-4cfb-8c10-a1541763c969\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-549ca28e-zhcw4" Oct 31 04:31:00.918368 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:00.894546 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s" Oct 31 04:31:00.918368 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:00.910584 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s" Oct 31 04:31:00.971356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:00.970839 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9098da5-5571-4cfb-8c10-a1541763c969-maas-api-xks-ca\") pod \"maas-api-revoke-sub-549ca28e-zhcw4\" (UID: \"d9098da5-5571-4cfb-8c10-a1541763c969\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-549ca28e-zhcw4" Oct 31 04:31:01.121406 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:01.120560 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-549ca28e-zhcw4" Oct 31 04:31:01.169901 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:01.166601 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" podUID="dccd15a9-47f5-4bd1-b21b-9d5ca8b05845" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:31:01.201386 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:01.191444 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" podUID="e0f4b8c2-14f4-4052-bf54-dd65bf59e413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 04:31:01.393402 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:01.389812 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s" Oct 31 04:31:01.487356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:01.485792 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:31:01.513357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:01.512475 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:31:01.635367 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:01.633681 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" podUID="643433d8-418e-4102-8bc3-cd544932614a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:31:01.688417 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:01.687458 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:31:01.899592 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:01.829532 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s" Oct 31 04:31:02.169663 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:02.169606 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" podUID="b4ced3ab-41e4-4c54-93eb-95eddaae992d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 04:31:02.190973 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:02.186431 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" podUID="dccd15a9-47f5-4bd1-b21b-9d5ca8b05845" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:31:02.190973 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:02.186843 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" Oct 31 04:31:02.220707 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:02.206424 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" podUID="e0f4b8c2-14f4-4052-bf54-dd65bf59e413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:31:02.220707 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:02.206843 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" Oct 31 04:31:02.586390 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:02.511514 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:31:02.586390 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:02.512215 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" Oct 31 04:31:02.586390 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:02.575428 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:31:02.586390 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:02.575870 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" Oct 31 04:31:02.658432 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:02.658393 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" podUID="643433d8-418e-4102-8bc3-cd544932614a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 04:31:02.659298 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:02.659275 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" Oct 31 04:31:02.709498 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:02.663264 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" podUID="8e2c7041-ad52-49f4-8b55-d16691f4938f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:31:02.720797 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:02.710759 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:31:02.905256 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:02.900944 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" Oct 31 04:31:02.962717 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:02.962635 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" podUID="e0f4b8c2-14f4-4052-bf54-dd65bf59e413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:31:03.362657 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:03.362615 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" podUID="dccd15a9-47f5-4bd1-b21b-9d5ca8b05845" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": context deadline exceeded" Oct 31 04:31:03.363326 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:03.363292 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" podUID="dccd15a9-47f5-4bd1-b21b-9d5ca8b05845" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 31 04:31:03.449594 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:03.449460 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" podUID="b4ced3ab-41e4-4c54-93eb-95eddaae992d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Oct 31 04:31:03.484815 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:03.484774 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" podUID="e0f4b8c2-14f4-4052-bf54-dd65bf59e413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:31:03.556381 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:03.509085 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr"] Oct 31 04:31:03.556381 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:03.538714 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr" podUID="16ca6a86-9186-4d31-a3ac-07f81f318a42" containerName="istio-proxy" containerID="cri-o://7dc9ed89fc773d4b3760de30de1528182a467d913f7b0561e3c954a14e29caf8" gracePeriod=30 Oct 31 04:31:03.556381 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:03.544521 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 04:31:03.583357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:03.575711 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:31:03.583568 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:03.583484 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" Oct 31 04:31:03.583568 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:03.583535 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-549ca28e-zhcw4"] Oct 31 04:31:03.583682 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:03.583601 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" Oct 31 04:31:03.583682 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:03.583635 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" Oct 31 04:31:03.614504 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:03.614463 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" Oct 31 04:31:03.636961 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:03.622804 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" Oct 31 04:31:03.636961 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:03.636447 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" Oct 31 04:31:03.637279 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:03.637220 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" Oct 31 04:31:03.639695 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:03.639673 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" Oct 31 04:31:03.711677 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:03.711580 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-549ca28e-zhcw4" event={"ID":"d9098da5-5571-4cfb-8c10-a1541763c969","Type":"ContainerStarted","Data":"05e1d531613a9409fdc4d0e8a03e7360cef6659c916bd7aa8da0670a28e06b67"} Oct 31 04:31:04.728107 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:04.728067 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-549ca28e-zhcw4" event={"ID":"d9098da5-5571-4cfb-8c10-a1541763c969","Type":"ContainerStarted","Data":"7473b80a497d8c89a2afcc061869f56df510bd4abe0df63458d50c6d5657f5e2"} Oct 31 04:31:04.855924 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:04.852364 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-549ca28e-zhcw4" podStartSLOduration=4.852323339 podStartE2EDuration="4.852323339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:31:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:31:04.829850028 +0000 UTC m=+1619.586188068" watchObservedRunningTime="2026-10-31 04:31:04.852323339 +0000 UTC m=+1619.608661381" Oct 31 04:31:05.736945 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:05.736914 5019 generic.go:356] "Generic (PLEG): container finished" podID="d9098da5-5571-4cfb-8c10-a1541763c969" containerID="7473b80a497d8c89a2afcc061869f56df510bd4abe0df63458d50c6d5657f5e2" exitCode=0 Oct 31 04:31:05.737405 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:05.736997 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-549ca28e-zhcw4" event={"ID":"d9098da5-5571-4cfb-8c10-a1541763c969","Type":"ContainerDied","Data":"7473b80a497d8c89a2afcc061869f56df510bd4abe0df63458d50c6d5657f5e2"} Oct 31 04:31:06.916842 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:06.916793 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n"] Oct 31 04:31:06.924543 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:06.924513 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n" Oct 31 04:31:06.942832 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:06.942238 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-549ca28e-zhcw4" Oct 31 04:31:06.978283 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:06.961751 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv"] Oct 31 04:31:06.978283 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:06.962905 5019 cpu_manager.go:414] "containerMap: removing container" podUID="d9098da5-5571-4cfb-8c10-a1541763c969" containerName="revoke-keys" Oct 31 04:31:06.978283 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:06.962927 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9098da5-5571-4cfb-8c10-a1541763c969" containerName="revoke-keys" Oct 31 04:31:06.978283 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:06.963096 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9098da5-5571-4cfb-8c10-a1541763c969" containerName="revoke-keys" Oct 31 04:31:06.978283 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:06.967739 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n"] Oct 31 04:31:06.978283 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:06.967858 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv" Oct 31 04:31:06.995280 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:06.995214 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv"] Oct 31 04:31:07.042271 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.042186 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9098da5-5571-4cfb-8c10-a1541763c969-kube-api-access-vzpbh\" (UniqueName: \"kubernetes.io/projected/d9098da5-5571-4cfb-8c10-a1541763c969-kube-api-access-vzpbh\") pod \"d9098da5-5571-4cfb-8c10-a1541763c969\" (UID: \"d9098da5-5571-4cfb-8c10-a1541763c969\") " Oct 31 04:31:07.042271 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.042261 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9098da5-5571-4cfb-8c10-a1541763c969-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9098da5-5571-4cfb-8c10-a1541763c969-maas-api-xks-ca\") pod \"d9098da5-5571-4cfb-8c10-a1541763c969\" (UID: \"d9098da5-5571-4cfb-8c10-a1541763c969\") " Oct 31 04:31:07.042630 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.042608 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9098da5-5571-4cfb-8c10-a1541763c969-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9098da5-5571-4cfb-8c10-a1541763c969-maas-api-openshift-ca\") pod \"d9098da5-5571-4cfb-8c10-a1541763c969\" (UID: \"d9098da5-5571-4cfb-8c10-a1541763c969\") " Oct 31 04:31:07.042868 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.042847 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-istio-envoy\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n\" (UID: \"88b7c3d5-09b5-4eda-a752-aac0c1f0a11c\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n" Oct 31 04:31:07.042980 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.042929 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-workload-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n\" (UID: \"88b7c3d5-09b5-4eda-a752-aac0c1f0a11c\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n" Oct 31 04:31:07.042980 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.042948 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9098da5-5571-4cfb-8c10-a1541763c969-maas-api-openshift-ca" pod "d9098da5-5571-4cfb-8c10-a1541763c969" (UID: "d9098da5-5571-4cfb-8c10-a1541763c969"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:31:07.043088 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.043040 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e308d744-11e3-4fef-bc86-eb0a7d9730b2-istio-data\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv\" (UID: \"e308d744-11e3-4fef-bc86-eb0a7d9730b2\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv" Oct 31 04:31:07.043088 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.043079 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-istiod-ca-cert\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n\" (UID: \"88b7c3d5-09b5-4eda-a752-aac0c1f0a11c\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n" Oct 31 04:31:07.043169 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.043109 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-credential-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n\" (UID: \"88b7c3d5-09b5-4eda-a752-aac0c1f0a11c\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n" Oct 31 04:31:07.043207 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.043179 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e308d744-11e3-4fef-bc86-eb0a7d9730b2-workload-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv\" (UID: \"e308d744-11e3-4fef-bc86-eb0a7d9730b2\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv" Oct 31 04:31:07.043327 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.043312 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-istio-podinfo\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n\" (UID: \"88b7c3d5-09b5-4eda-a752-aac0c1f0a11c\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n" Oct 31 04:31:07.043414 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.043357 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-istio-token\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n\" (UID: \"88b7c3d5-09b5-4eda-a752-aac0c1f0a11c\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n" Oct 31 04:31:07.043503 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.043475 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e308d744-11e3-4fef-bc86-eb0a7d9730b2-istio-podinfo\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv\" (UID: \"e308d744-11e3-4fef-bc86-eb0a7d9730b2\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv" Oct 31 04:31:07.043594 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.043574 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e308d744-11e3-4fef-bc86-eb0a7d9730b2-credential-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv\" (UID: \"e308d744-11e3-4fef-bc86-eb0a7d9730b2\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv" Oct 31 04:31:07.043650 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.043609 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-istio-data\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n\" (UID: \"88b7c3d5-09b5-4eda-a752-aac0c1f0a11c\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n" Oct 31 04:31:07.043723 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.043710 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-workload-certs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n\" (UID: \"88b7c3d5-09b5-4eda-a752-aac0c1f0a11c\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n" Oct 31 04:31:07.043799 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.043769 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e308d744-11e3-4fef-bc86-eb0a7d9730b2-istio-token\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv\" (UID: \"e308d744-11e3-4fef-bc86-eb0a7d9730b2\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv" Oct 31 04:31:07.043853 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.043816 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ktg7\" (UniqueName: \"kubernetes.io/projected/e308d744-11e3-4fef-bc86-eb0a7d9730b2-kube-api-access-8ktg7\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv\" (UID: \"e308d744-11e3-4fef-bc86-eb0a7d9730b2\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv" Oct 31 04:31:07.043853 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.043842 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e308d744-11e3-4fef-bc86-eb0a7d9730b2-workload-certs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv\" (UID: \"e308d744-11e3-4fef-bc86-eb0a7d9730b2\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv" Oct 31 04:31:07.043952 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.043878 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e308d744-11e3-4fef-bc86-eb0a7d9730b2-istio-envoy\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv\" (UID: \"e308d744-11e3-4fef-bc86-eb0a7d9730b2\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv" Oct 31 04:31:07.043952 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.043934 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p799z\" (UniqueName: \"kubernetes.io/projected/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-kube-api-access-p799z\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n\" (UID: \"88b7c3d5-09b5-4eda-a752-aac0c1f0a11c\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n" Oct 31 04:31:07.044050 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.043990 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e308d744-11e3-4fef-bc86-eb0a7d9730b2-istiod-ca-cert\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv\" (UID: \"e308d744-11e3-4fef-bc86-eb0a7d9730b2\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv" Oct 31 04:31:07.044102 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.044093 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9098da5-5571-4cfb-8c10-a1541763c969-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:31:07.044818 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.044793 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9098da5-5571-4cfb-8c10-a1541763c969-maas-api-xks-ca" pod "d9098da5-5571-4cfb-8c10-a1541763c969" (UID: "d9098da5-5571-4cfb-8c10-a1541763c969"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:31:07.045122 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.045102 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9098da5-5571-4cfb-8c10-a1541763c969-kube-api-access-vzpbh" pod "d9098da5-5571-4cfb-8c10-a1541763c969" (UID: "d9098da5-5571-4cfb-8c10-a1541763c969"). InnerVolumeSpecName "kube-api-access-vzpbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:31:07.145803 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.145754 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p799z\" (UniqueName: \"kubernetes.io/projected/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-kube-api-access-p799z\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n\" (UID: \"88b7c3d5-09b5-4eda-a752-aac0c1f0a11c\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n" Oct 31 04:31:07.146006 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.145819 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e308d744-11e3-4fef-bc86-eb0a7d9730b2-istiod-ca-cert\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv\" (UID: \"e308d744-11e3-4fef-bc86-eb0a7d9730b2\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv" Oct 31 04:31:07.146006 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.145853 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-istio-envoy\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n\" (UID: \"88b7c3d5-09b5-4eda-a752-aac0c1f0a11c\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n" Oct 31 04:31:07.146006 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.145877 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-workload-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n\" (UID: \"88b7c3d5-09b5-4eda-a752-aac0c1f0a11c\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n" Oct 31 04:31:07.146006 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.145926 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e308d744-11e3-4fef-bc86-eb0a7d9730b2-istio-data\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv\" (UID: \"e308d744-11e3-4fef-bc86-eb0a7d9730b2\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv" Oct 31 04:31:07.146006 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.145961 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-istiod-ca-cert\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n\" (UID: \"88b7c3d5-09b5-4eda-a752-aac0c1f0a11c\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n" Oct 31 04:31:07.146006 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.146001 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-credential-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n\" (UID: \"88b7c3d5-09b5-4eda-a752-aac0c1f0a11c\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n" Oct 31 04:31:07.146400 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.146047 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e308d744-11e3-4fef-bc86-eb0a7d9730b2-workload-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv\" (UID: \"e308d744-11e3-4fef-bc86-eb0a7d9730b2\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv" Oct 31 04:31:07.146462 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.146418 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-istio-podinfo\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n\" (UID: \"88b7c3d5-09b5-4eda-a752-aac0c1f0a11c\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n" Oct 31 04:31:07.146512 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.146459 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e308d744-11e3-4fef-bc86-eb0a7d9730b2-istio-data\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv\" (UID: \"e308d744-11e3-4fef-bc86-eb0a7d9730b2\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv" Oct 31 04:31:07.146512 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.146473 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-istio-token\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n\" (UID: \"88b7c3d5-09b5-4eda-a752-aac0c1f0a11c\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n" Oct 31 04:31:07.146512 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.146476 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-credential-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n\" (UID: \"88b7c3d5-09b5-4eda-a752-aac0c1f0a11c\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n" Oct 31 04:31:07.146512 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.146506 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e308d744-11e3-4fef-bc86-eb0a7d9730b2-istio-podinfo\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv\" (UID: \"e308d744-11e3-4fef-bc86-eb0a7d9730b2\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv" Oct 31 04:31:07.146738 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.146554 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-workload-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n\" (UID: \"88b7c3d5-09b5-4eda-a752-aac0c1f0a11c\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n" Oct 31 04:31:07.146738 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.146724 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e308d744-11e3-4fef-bc86-eb0a7d9730b2-workload-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv\" (UID: \"e308d744-11e3-4fef-bc86-eb0a7d9730b2\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv" Oct 31 04:31:07.146846 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.146797 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e308d744-11e3-4fef-bc86-eb0a7d9730b2-credential-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv\" (UID: \"e308d744-11e3-4fef-bc86-eb0a7d9730b2\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv" Oct 31 04:31:07.146846 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.146824 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-istio-data\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n\" (UID: \"88b7c3d5-09b5-4eda-a752-aac0c1f0a11c\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n" Oct 31 04:31:07.146961 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.146897 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-workload-certs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n\" (UID: \"88b7c3d5-09b5-4eda-a752-aac0c1f0a11c\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n" Oct 31 04:31:07.147024 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.146959 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e308d744-11e3-4fef-bc86-eb0a7d9730b2-istio-token\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv\" (UID: \"e308d744-11e3-4fef-bc86-eb0a7d9730b2\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv" Oct 31 04:31:07.147024 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.147007 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ktg7\" (UniqueName: \"kubernetes.io/projected/e308d744-11e3-4fef-bc86-eb0a7d9730b2-kube-api-access-8ktg7\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv\" (UID: \"e308d744-11e3-4fef-bc86-eb0a7d9730b2\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv" Oct 31 04:31:07.147144 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.147035 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e308d744-11e3-4fef-bc86-eb0a7d9730b2-workload-certs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv\" (UID: \"e308d744-11e3-4fef-bc86-eb0a7d9730b2\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv" Oct 31 04:31:07.147144 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.147068 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e308d744-11e3-4fef-bc86-eb0a7d9730b2-istio-envoy\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv\" (UID: \"e308d744-11e3-4fef-bc86-eb0a7d9730b2\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv" Oct 31 04:31:07.147144 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.147076 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-istiod-ca-cert\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n\" (UID: \"88b7c3d5-09b5-4eda-a752-aac0c1f0a11c\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n" Oct 31 04:31:07.147312 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.147182 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e308d744-11e3-4fef-bc86-eb0a7d9730b2-istiod-ca-cert\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv\" (UID: \"e308d744-11e3-4fef-bc86-eb0a7d9730b2\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv" Oct 31 04:31:07.147500 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.147475 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-istio-data\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n\" (UID: \"88b7c3d5-09b5-4eda-a752-aac0c1f0a11c\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n" Oct 31 04:31:07.147651 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.147475 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e308d744-11e3-4fef-bc86-eb0a7d9730b2-credential-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv\" (UID: \"e308d744-11e3-4fef-bc86-eb0a7d9730b2\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv" Oct 31 04:31:07.147651 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.147594 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzpbh\" (UniqueName: \"kubernetes.io/projected/d9098da5-5571-4cfb-8c10-a1541763c969-kube-api-access-vzpbh\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:31:07.147767 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.147685 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9098da5-5571-4cfb-8c10-a1541763c969-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:31:07.147767 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.147753 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-workload-certs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n\" (UID: \"88b7c3d5-09b5-4eda-a752-aac0c1f0a11c\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n" Oct 31 04:31:07.147878 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.147849 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e308d744-11e3-4fef-bc86-eb0a7d9730b2-workload-certs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv\" (UID: \"e308d744-11e3-4fef-bc86-eb0a7d9730b2\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv" Oct 31 04:31:07.149474 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.149448 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-istio-envoy\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n\" (UID: \"88b7c3d5-09b5-4eda-a752-aac0c1f0a11c\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n" Oct 31 04:31:07.149571 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.149460 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e308d744-11e3-4fef-bc86-eb0a7d9730b2-istio-envoy\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv\" (UID: \"e308d744-11e3-4fef-bc86-eb0a7d9730b2\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv" Oct 31 04:31:07.150190 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.150157 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-istio-podinfo\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n\" (UID: \"88b7c3d5-09b5-4eda-a752-aac0c1f0a11c\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n" Oct 31 04:31:07.150860 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.150839 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e308d744-11e3-4fef-bc86-eb0a7d9730b2-istio-podinfo\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv\" (UID: \"e308d744-11e3-4fef-bc86-eb0a7d9730b2\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv" Oct 31 04:31:07.169556 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.169516 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-istio-token\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n\" (UID: \"88b7c3d5-09b5-4eda-a752-aac0c1f0a11c\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n" Oct 31 04:31:07.170001 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.169976 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p799z\" (UniqueName: \"kubernetes.io/projected/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-kube-api-access-p799z\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n\" (UID: \"88b7c3d5-09b5-4eda-a752-aac0c1f0a11c\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n" Oct 31 04:31:07.170046 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.170035 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e308d744-11e3-4fef-bc86-eb0a7d9730b2-istio-token\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv\" (UID: \"e308d744-11e3-4fef-bc86-eb0a7d9730b2\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv" Oct 31 04:31:07.170491 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.170473 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ktg7\" (UniqueName: \"kubernetes.io/projected/e308d744-11e3-4fef-bc86-eb0a7d9730b2-kube-api-access-8ktg7\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv\" (UID: \"e308d744-11e3-4fef-bc86-eb0a7d9730b2\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv" Oct 31 04:31:07.262105 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.262069 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n" Oct 31 04:31:07.297158 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.297076 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv" Oct 31 04:31:07.590949 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.590918 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n"] Oct 31 04:31:07.599561 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.599511 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:31:07.599688 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.599601 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:31:07.599688 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.599632 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:31:07.646089 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:31:07.646056 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode308d744_11e3_4fef_bc86_eb0a7d9730b2.slice/crio-fda7b873f1a4ec5d1f5a13cbc11a117d3ab4ecff8252544ff3f7ef5805c30f53 WatchSource:0}: Error finding container fda7b873f1a4ec5d1f5a13cbc11a117d3ab4ecff8252544ff3f7ef5805c30f53: Status 404 returned error can't find the container with id fda7b873f1a4ec5d1f5a13cbc11a117d3ab4ecff8252544ff3f7ef5805c30f53 Oct 31 04:31:07.647267 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.647239 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv"] Oct 31 04:31:07.649027 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.648993 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:31:07.649117 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.649063 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:31:07.649117 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.649091 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:31:07.766412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.765243 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv" event={"ID":"e308d744-11e3-4fef-bc86-eb0a7d9730b2","Type":"ContainerStarted","Data":"6f39ce4b9eca7f0e45ef638219c03e5402afb389d2d0bf222fdbf078e51b73d8"} Oct 31 04:31:07.766412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.765312 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv" event={"ID":"e308d744-11e3-4fef-bc86-eb0a7d9730b2","Type":"ContainerStarted","Data":"fda7b873f1a4ec5d1f5a13cbc11a117d3ab4ecff8252544ff3f7ef5805c30f53"} Oct 31 04:31:07.770760 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.770082 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n" event={"ID":"88b7c3d5-09b5-4eda-a752-aac0c1f0a11c","Type":"ContainerStarted","Data":"e11593db6c87ea30fb2cc41c61641792a4ef7fb1d40533ee40fb5a297b6bb4b6"} Oct 31 04:31:07.770760 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.770137 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n" event={"ID":"88b7c3d5-09b5-4eda-a752-aac0c1f0a11c","Type":"ContainerStarted","Data":"a6c5d4441a902498dc0e1f70d0a54d9c2122a399b7eeef2c8714fc524d4d1f1f"} Oct 31 04:31:07.776621 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.772768 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-549ca28e-zhcw4" event={"ID":"d9098da5-5571-4cfb-8c10-a1541763c969","Type":"ContainerDied","Data":"05e1d531613a9409fdc4d0e8a03e7360cef6659c916bd7aa8da0670a28e06b67"} Oct 31 04:31:07.776621 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.772814 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05e1d531613a9409fdc4d0e8a03e7360cef6659c916bd7aa8da0670a28e06b67" Oct 31 04:31:07.776621 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.772871 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-549ca28e-zhcw4" Oct 31 04:31:07.816496 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.816434 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv" podStartSLOduration=1.816412439 podStartE2EDuration="1.816412439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:31:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:31:07.811277197 +0000 UTC m=+1622.567615237" watchObservedRunningTime="2026-10-31 04:31:07.816412439 +0000 UTC m=+1622.572750479" Oct 31 04:31:07.876396 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:07.875476 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n" podStartSLOduration=1.875447436 podStartE2EDuration="1.875447436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:31:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:31:07.870416038 +0000 UTC m=+1622.626754076" watchObservedRunningTime="2026-10-31 04:31:07.875447436 +0000 UTC m=+1622.631785476" Oct 31 04:31:08.262835 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:08.262798 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n" Oct 31 04:31:08.264631 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:08.264604 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 31 04:31:08.264775 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:08.264657 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n" podUID="88b7c3d5-09b5-4eda-a752-aac0c1f0a11c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 31 04:31:08.297978 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:08.297940 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv" Oct 31 04:31:08.299689 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:08.299660 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 31 04:31:08.299851 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:08.299709 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv" podUID="e308d744-11e3-4fef-bc86-eb0a7d9730b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 31 04:31:08.429757 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:08.429726 5019 scope.go:122] "RemoveContainer" containerID="b845341f9712881c3562b23304768f72f641e1ee11865158a230e336d08ea91a" Oct 31 04:31:08.838938 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:08.838470 5019 generic.go:356] "Generic (PLEG): container finished" podID="16ca6a86-9186-4d31-a3ac-07f81f318a42" containerID="7dc9ed89fc773d4b3760de30de1528182a467d913f7b0561e3c954a14e29caf8" exitCode=0 Oct 31 04:31:08.839449 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:08.839424 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr" event={"ID":"16ca6a86-9186-4d31-a3ac-07f81f318a42","Type":"ContainerDied","Data":"7dc9ed89fc773d4b3760de30de1528182a467d913f7b0561e3c954a14e29caf8"} Oct 31 04:31:08.839636 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:08.839492 5019 scope.go:122] "RemoveContainer" containerID="c7e02b380f069e182c5ee70b3d5189736fe64a69078b3152081daf0225692873" Oct 31 04:31:08.968078 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:08.968048 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr" Oct 31 04:31:09.139837 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:09.139741 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16ca6a86-9186-4d31-a3ac-07f81f318a42-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16ca6a86-9186-4d31-a3ac-07f81f318a42-istiod-ca-cert\") pod \"16ca6a86-9186-4d31-a3ac-07f81f318a42\" (UID: \"16ca6a86-9186-4d31-a3ac-07f81f318a42\") " Oct 31 04:31:09.139837 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:09.139825 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16ca6a86-9186-4d31-a3ac-07f81f318a42-kube-api-access-7gz74\" (UniqueName: \"kubernetes.io/projected/16ca6a86-9186-4d31-a3ac-07f81f318a42-kube-api-access-7gz74\") pod \"16ca6a86-9186-4d31-a3ac-07f81f318a42\" (UID: \"16ca6a86-9186-4d31-a3ac-07f81f318a42\") " Oct 31 04:31:09.140159 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:09.139857 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16ca6a86-9186-4d31-a3ac-07f81f318a42-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16ca6a86-9186-4d31-a3ac-07f81f318a42-credential-socket\") pod \"16ca6a86-9186-4d31-a3ac-07f81f318a42\" (UID: \"16ca6a86-9186-4d31-a3ac-07f81f318a42\") " Oct 31 04:31:09.140159 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:09.139894 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16ca6a86-9186-4d31-a3ac-07f81f318a42-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16ca6a86-9186-4d31-a3ac-07f81f318a42-istio-envoy\") pod \"16ca6a86-9186-4d31-a3ac-07f81f318a42\" (UID: \"16ca6a86-9186-4d31-a3ac-07f81f318a42\") " Oct 31 04:31:09.140159 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:09.139926 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16ca6a86-9186-4d31-a3ac-07f81f318a42-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16ca6a86-9186-4d31-a3ac-07f81f318a42-workload-certs\") pod \"16ca6a86-9186-4d31-a3ac-07f81f318a42\" (UID: \"16ca6a86-9186-4d31-a3ac-07f81f318a42\") " Oct 31 04:31:09.140159 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:09.140043 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16ca6a86-9186-4d31-a3ac-07f81f318a42-istio-token\" (UniqueName: \"kubernetes.io/projected/16ca6a86-9186-4d31-a3ac-07f81f318a42-istio-token\") pod \"16ca6a86-9186-4d31-a3ac-07f81f318a42\" (UID: \"16ca6a86-9186-4d31-a3ac-07f81f318a42\") " Oct 31 04:31:09.140159 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:09.140111 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16ca6a86-9186-4d31-a3ac-07f81f318a42-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16ca6a86-9186-4d31-a3ac-07f81f318a42-istio-data\") pod \"16ca6a86-9186-4d31-a3ac-07f81f318a42\" (UID: \"16ca6a86-9186-4d31-a3ac-07f81f318a42\") " Oct 31 04:31:09.140475 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:09.140165 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16ca6a86-9186-4d31-a3ac-07f81f318a42-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16ca6a86-9186-4d31-a3ac-07f81f318a42-istio-podinfo\") pod \"16ca6a86-9186-4d31-a3ac-07f81f318a42\" (UID: \"16ca6a86-9186-4d31-a3ac-07f81f318a42\") " Oct 31 04:31:09.140475 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:09.140276 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16ca6a86-9186-4d31-a3ac-07f81f318a42-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16ca6a86-9186-4d31-a3ac-07f81f318a42-workload-socket\") pod \"16ca6a86-9186-4d31-a3ac-07f81f318a42\" (UID: \"16ca6a86-9186-4d31-a3ac-07f81f318a42\") " Oct 31 04:31:09.143156 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:09.142783 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16ca6a86-9186-4d31-a3ac-07f81f318a42-workload-certs" pod "16ca6a86-9186-4d31-a3ac-07f81f318a42" (UID: "16ca6a86-9186-4d31-a3ac-07f81f318a42"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:31:09.143664 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:09.143641 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16ca6a86-9186-4d31-a3ac-07f81f318a42-istio-data" pod "16ca6a86-9186-4d31-a3ac-07f81f318a42" (UID: "16ca6a86-9186-4d31-a3ac-07f81f318a42"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:31:09.144288 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:09.144250 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16ca6a86-9186-4d31-a3ac-07f81f318a42-credential-socket" pod "16ca6a86-9186-4d31-a3ac-07f81f318a42" (UID: "16ca6a86-9186-4d31-a3ac-07f81f318a42"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:31:09.144624 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:09.144584 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16ca6a86-9186-4d31-a3ac-07f81f318a42-workload-socket" pod "16ca6a86-9186-4d31-a3ac-07f81f318a42" (UID: "16ca6a86-9186-4d31-a3ac-07f81f318a42"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:31:09.145227 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:09.145203 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16ca6a86-9186-4d31-a3ac-07f81f318a42-istiod-ca-cert" pod "16ca6a86-9186-4d31-a3ac-07f81f318a42" (UID: "16ca6a86-9186-4d31-a3ac-07f81f318a42"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:31:09.147018 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:09.146991 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16ca6a86-9186-4d31-a3ac-07f81f318a42-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:31:09.147117 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:09.147023 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16ca6a86-9186-4d31-a3ac-07f81f318a42-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:31:09.147117 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:09.147041 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16ca6a86-9186-4d31-a3ac-07f81f318a42-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:31:09.147117 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:09.147056 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16ca6a86-9186-4d31-a3ac-07f81f318a42-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:31:09.147117 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:09.147067 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16ca6a86-9186-4d31-a3ac-07f81f318a42-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:31:09.160281 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:09.158851 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16ca6a86-9186-4d31-a3ac-07f81f318a42-kube-api-access-7gz74" pod "16ca6a86-9186-4d31-a3ac-07f81f318a42" (UID: "16ca6a86-9186-4d31-a3ac-07f81f318a42"). InnerVolumeSpecName "kube-api-access-7gz74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:31:09.160281 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:09.159418 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16ca6a86-9186-4d31-a3ac-07f81f318a42-istio-podinfo" pod "16ca6a86-9186-4d31-a3ac-07f81f318a42" (UID: "16ca6a86-9186-4d31-a3ac-07f81f318a42"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:31:09.160281 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:09.160252 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16ca6a86-9186-4d31-a3ac-07f81f318a42-istio-token" pod "16ca6a86-9186-4d31-a3ac-07f81f318a42" (UID: "16ca6a86-9186-4d31-a3ac-07f81f318a42"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:31:09.160613 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:09.160590 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16ca6a86-9186-4d31-a3ac-07f81f318a42-istio-envoy" pod "16ca6a86-9186-4d31-a3ac-07f81f318a42" (UID: "16ca6a86-9186-4d31-a3ac-07f81f318a42"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:31:09.248731 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:09.248688 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gz74\" (UniqueName: \"kubernetes.io/projected/16ca6a86-9186-4d31-a3ac-07f81f318a42-kube-api-access-7gz74\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:31:09.248731 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:09.248723 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16ca6a86-9186-4d31-a3ac-07f81f318a42-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:31:09.248731 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:09.248734 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16ca6a86-9186-4d31-a3ac-07f81f318a42-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:31:09.248731 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:09.248744 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16ca6a86-9186-4d31-a3ac-07f81f318a42-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:31:09.263237 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:09.263189 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 31 04:31:09.263717 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:09.263248 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n" podUID="88b7c3d5-09b5-4eda-a752-aac0c1f0a11c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 31 04:31:09.301355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:09.300889 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 31 04:31:09.301355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:09.300966 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv" podUID="e308d744-11e3-4fef-bc86-eb0a7d9730b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 31 04:31:09.849816 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:09.849784 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr" Oct 31 04:31:09.850014 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:09.849781 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr" event={"ID":"16ca6a86-9186-4d31-a3ac-07f81f318a42","Type":"ContainerDied","Data":"fc8090995931b6fcd0e12cb3ad0be5c502d3be12d1be93c38bcebfdabfcda4b8"} Oct 31 04:31:09.850014 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:09.849948 5019 scope.go:122] "RemoveContainer" containerID="7dc9ed89fc773d4b3760de30de1528182a467d913f7b0561e3c954a14e29caf8" Oct 31 04:31:09.940723 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:09.940647 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr"] Oct 31 04:31:09.947198 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:09.947158 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-7cf9ef-openshift-default-757899c4f-ht8jr"] Oct 31 04:31:10.263194 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:10.263159 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 31 04:31:10.263407 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:10.263229 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n" podUID="88b7c3d5-09b5-4eda-a752-aac0c1f0a11c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 31 04:31:10.302361 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:10.301762 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 31 04:31:10.302361 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:10.301824 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv" podUID="e308d744-11e3-4fef-bc86-eb0a7d9730b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 31 04:31:10.758888 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:10.758855 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f20a4797-csz4v"] Oct 31 04:31:10.759503 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:10.759489 5019 cpu_manager.go:414] "containerMap: removing container" podUID="16ca6a86-9186-4d31-a3ac-07f81f318a42" containerName="istio-proxy" Oct 31 04:31:10.759576 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:10.759504 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16ca6a86-9186-4d31-a3ac-07f81f318a42" containerName="istio-proxy" Oct 31 04:31:10.759645 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:10.759625 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="16ca6a86-9186-4d31-a3ac-07f81f318a42" containerName="istio-proxy" Oct 31 04:31:10.764636 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:10.764618 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20a4797-csz4v" Oct 31 04:31:10.800355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:10.797691 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:31:10.812357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:10.809378 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:31:10.812357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:10.810098 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:31:10.812357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:10.810162 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:31:10.843362 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:10.834401 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f20a4797-csz4v"] Oct 31 04:31:10.889433 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:10.881460 5019 patch_prober.go:29] interesting pod/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:31:10.889433 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:10.881532 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f" podUID="116c8b42-2784-4f46-b534-ea35159e3cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:31:10.912376 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:10.891703 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c0927ea-ba50-457d-9cdc-c289169d44f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f20a4797-csz4v\" (UID: \"9c0927ea-ba50-457d-9cdc-c289169d44f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20a4797-csz4v" Oct 31 04:31:10.912376 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:10.891854 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmmnc\" (UniqueName: \"kubernetes.io/projected/9c0927ea-ba50-457d-9cdc-c289169d44f7-kube-api-access-cmmnc\") pod \"maas-api-revoke-sub-f20a4797-csz4v\" (UID: \"9c0927ea-ba50-457d-9cdc-c289169d44f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20a4797-csz4v" Oct 31 04:31:10.912376 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:10.891948 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c0927ea-ba50-457d-9cdc-c289169d44f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f20a4797-csz4v\" (UID: \"9c0927ea-ba50-457d-9cdc-c289169d44f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20a4797-csz4v" Oct 31 04:31:10.997357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:10.997218 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c0927ea-ba50-457d-9cdc-c289169d44f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f20a4797-csz4v\" (UID: \"9c0927ea-ba50-457d-9cdc-c289169d44f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20a4797-csz4v" Oct 31 04:31:10.997357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:10.997357 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c0927ea-ba50-457d-9cdc-c289169d44f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f20a4797-csz4v\" (UID: \"9c0927ea-ba50-457d-9cdc-c289169d44f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20a4797-csz4v" Oct 31 04:31:10.997640 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:10.997539 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmmnc\" (UniqueName: \"kubernetes.io/projected/9c0927ea-ba50-457d-9cdc-c289169d44f7-kube-api-access-cmmnc\") pod \"maas-api-revoke-sub-f20a4797-csz4v\" (UID: \"9c0927ea-ba50-457d-9cdc-c289169d44f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20a4797-csz4v" Oct 31 04:31:11.001356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:10.998791 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c0927ea-ba50-457d-9cdc-c289169d44f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f20a4797-csz4v\" (UID: \"9c0927ea-ba50-457d-9cdc-c289169d44f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20a4797-csz4v" Oct 31 04:31:11.041462 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:11.040201 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c0927ea-ba50-457d-9cdc-c289169d44f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f20a4797-csz4v\" (UID: \"9c0927ea-ba50-457d-9cdc-c289169d44f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20a4797-csz4v" Oct 31 04:31:11.041959 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:11.041931 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmmnc\" (UniqueName: \"kubernetes.io/projected/9c0927ea-ba50-457d-9cdc-c289169d44f7-kube-api-access-cmmnc\") pod \"maas-api-revoke-sub-f20a4797-csz4v\" (UID: \"9c0927ea-ba50-457d-9cdc-c289169d44f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20a4797-csz4v" Oct 31 04:31:11.120629 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:11.120586 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20a4797-csz4v" Oct 31 04:31:11.263259 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:11.262764 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 31 04:31:11.263259 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:11.262833 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n" podUID="88b7c3d5-09b5-4eda-a752-aac0c1f0a11c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 31 04:31:11.339255 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:11.338719 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 31 04:31:11.339255 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:11.338787 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv" podUID="e308d744-11e3-4fef-bc86-eb0a7d9730b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 31 04:31:11.467900 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:11.467857 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16ca6a86-9186-4d31-a3ac-07f81f318a42" path="/var/lib/kubelet/pods/16ca6a86-9186-4d31-a3ac-07f81f318a42/volumes" Oct 31 04:31:11.545443 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:11.544126 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f20a4797-csz4v"] Oct 31 04:31:11.899400 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:11.899118 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20a4797-csz4v" event={"ID":"9c0927ea-ba50-457d-9cdc-c289169d44f7","Type":"ContainerStarted","Data":"b8c4ae932150f52955aa4769dde379ebb78b45767f2738551a99d61a3023c0a4"} Oct 31 04:31:12.263122 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:12.263073 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 31 04:31:12.263296 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:12.263152 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n" podUID="88b7c3d5-09b5-4eda-a752-aac0c1f0a11c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 31 04:31:12.300353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:12.299688 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 31 04:31:12.300353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:12.299763 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv" podUID="e308d744-11e3-4fef-bc86-eb0a7d9730b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 31 04:31:12.943198 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:12.943162 5019 generic.go:356] "Generic (PLEG): container finished" podID="116c8b42-2784-4f46-b534-ea35159e3cac" containerID="a1f2a6d8ebb597400c5d9b8695bc42b7c367b59a6adb50e297ccef83a682760a" exitCode=0 Oct 31 04:31:12.943456 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:12.943236 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f" event={"ID":"116c8b42-2784-4f46-b534-ea35159e3cac","Type":"ContainerDied","Data":"a1f2a6d8ebb597400c5d9b8695bc42b7c367b59a6adb50e297ccef83a682760a"} Oct 31 04:31:12.943765 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:12.943749 5019 scope.go:122] "RemoveContainer" containerID="a1f2a6d8ebb597400c5d9b8695bc42b7c367b59a6adb50e297ccef83a682760a" Oct 31 04:31:12.945475 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:12.945437 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:31:12.945591 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:12.945555 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:31:12.945661 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:12.945639 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:31:12.945984 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:12.945967 5019 generic.go:356] "Generic (PLEG): container finished" podID="9c0927ea-ba50-457d-9cdc-c289169d44f7" containerID="5a858458189206a917c8c333f4ffed762f4cb44fc15f2164c02c051ad0404c69" exitCode=0 Oct 31 04:31:12.946080 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:12.946056 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20a4797-csz4v" event={"ID":"9c0927ea-ba50-457d-9cdc-c289169d44f7","Type":"ContainerDied","Data":"5a858458189206a917c8c333f4ffed762f4cb44fc15f2164c02c051ad0404c69"} Oct 31 04:31:13.262717 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:13.262688 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 31 04:31:13.262941 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:13.262922 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n" podUID="88b7c3d5-09b5-4eda-a752-aac0c1f0a11c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 31 04:31:13.309414 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:13.308587 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 31 04:31:13.309414 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:13.308658 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv" podUID="e308d744-11e3-4fef-bc86-eb0a7d9730b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 31 04:31:13.955435 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:13.955401 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f" event={"ID":"116c8b42-2784-4f46-b534-ea35159e3cac","Type":"ContainerStarted","Data":"4a06bbf73f055024ff0141448cf35493927a32b8e54fa13559532d8448dc50e4"} Oct 31 04:31:14.201356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:14.200711 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f" Oct 31 04:31:14.260387 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:14.242541 5019 patch_prober.go:29] interesting pod/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Oct 31 04:31:14.260387 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:14.242618 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f" podUID="116c8b42-2784-4f46-b534-ea35159e3cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Oct 31 04:31:14.265394 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:14.265324 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 31 04:31:14.265720 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:14.265677 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n" podUID="88b7c3d5-09b5-4eda-a752-aac0c1f0a11c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 31 04:31:14.315356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:14.307738 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 31 04:31:14.315356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:14.307798 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv" podUID="e308d744-11e3-4fef-bc86-eb0a7d9730b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 31 04:31:14.442897 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:14.442521 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20a4797-csz4v" Oct 31 04:31:14.557608 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:14.557448 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c0927ea-ba50-457d-9cdc-c289169d44f7-kube-api-access-cmmnc\" (UniqueName: \"kubernetes.io/projected/9c0927ea-ba50-457d-9cdc-c289169d44f7-kube-api-access-cmmnc\") pod \"9c0927ea-ba50-457d-9cdc-c289169d44f7\" (UID: \"9c0927ea-ba50-457d-9cdc-c289169d44f7\") " Oct 31 04:31:14.557742 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:14.557617 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c0927ea-ba50-457d-9cdc-c289169d44f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c0927ea-ba50-457d-9cdc-c289169d44f7-maas-api-openshift-ca\") pod \"9c0927ea-ba50-457d-9cdc-c289169d44f7\" (UID: \"9c0927ea-ba50-457d-9cdc-c289169d44f7\") " Oct 31 04:31:14.557742 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:14.557721 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c0927ea-ba50-457d-9cdc-c289169d44f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c0927ea-ba50-457d-9cdc-c289169d44f7-maas-api-xks-ca\") pod \"9c0927ea-ba50-457d-9cdc-c289169d44f7\" (UID: \"9c0927ea-ba50-457d-9cdc-c289169d44f7\") " Oct 31 04:31:14.558648 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:14.558628 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c0927ea-ba50-457d-9cdc-c289169d44f7-maas-api-openshift-ca" pod "9c0927ea-ba50-457d-9cdc-c289169d44f7" (UID: "9c0927ea-ba50-457d-9cdc-c289169d44f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:31:14.564371 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:14.560995 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c0927ea-ba50-457d-9cdc-c289169d44f7-maas-api-xks-ca" pod "9c0927ea-ba50-457d-9cdc-c289169d44f7" (UID: "9c0927ea-ba50-457d-9cdc-c289169d44f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:31:14.564371 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:14.561131 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c0927ea-ba50-457d-9cdc-c289169d44f7-kube-api-access-cmmnc" pod "9c0927ea-ba50-457d-9cdc-c289169d44f7" (UID: "9c0927ea-ba50-457d-9cdc-c289169d44f7"). InnerVolumeSpecName "kube-api-access-cmmnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:31:14.660763 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:14.660310 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmmnc\" (UniqueName: \"kubernetes.io/projected/9c0927ea-ba50-457d-9cdc-c289169d44f7-kube-api-access-cmmnc\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:31:14.660763 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:14.660380 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c0927ea-ba50-457d-9cdc-c289169d44f7-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:31:14.660763 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:14.660398 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c0927ea-ba50-457d-9cdc-c289169d44f7-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:31:14.968730 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:14.968700 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20a4797-csz4v" Oct 31 04:31:14.968925 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:14.968742 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20a4797-csz4v" event={"ID":"9c0927ea-ba50-457d-9cdc-c289169d44f7","Type":"ContainerDied","Data":"b8c4ae932150f52955aa4769dde379ebb78b45767f2738551a99d61a3023c0a4"} Oct 31 04:31:14.968925 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:14.968838 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8c4ae932150f52955aa4769dde379ebb78b45767f2738551a99d61a3023c0a4" Oct 31 04:31:15.189358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:15.188428 5019 patch_prober.go:29] interesting pod/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Oct 31 04:31:15.189358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:15.188490 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f" podUID="116c8b42-2784-4f46-b534-ea35159e3cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Oct 31 04:31:15.262984 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:15.262894 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 31 04:31:15.262984 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:15.262965 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n" podUID="88b7c3d5-09b5-4eda-a752-aac0c1f0a11c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 31 04:31:15.302356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:15.300759 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 31 04:31:15.302356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:15.300823 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv" podUID="e308d744-11e3-4fef-bc86-eb0a7d9730b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 31 04:31:16.188612 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:16.188570 5019 patch_prober.go:29] interesting pod/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Oct 31 04:31:16.189245 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:16.189212 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f" podUID="116c8b42-2784-4f46-b534-ea35159e3cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Oct 31 04:31:16.315713 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:16.315662 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv" Oct 31 04:31:16.317069 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:16.317043 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv" Oct 31 04:31:16.317897 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:16.317868 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv" Oct 31 04:31:16.522889 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:16.520400 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n" Oct 31 04:31:16.522889 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:16.522136 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n" Oct 31 04:31:16.523617 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:16.523593 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n" Oct 31 04:31:17.189354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:17.187268 5019 patch_prober.go:29] interesting pod/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Oct 31 04:31:17.189354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:17.187363 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f" podUID="116c8b42-2784-4f46-b534-ea35159e3cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Oct 31 04:31:17.367514 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:17.367460 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d033c6a-bgnt7"] Oct 31 04:31:17.368107 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:17.368090 5019 cpu_manager.go:414] "containerMap: removing container" podUID="16ca6a86-9186-4d31-a3ac-07f81f318a42" containerName="istio-proxy" Oct 31 04:31:17.368169 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:17.368108 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16ca6a86-9186-4d31-a3ac-07f81f318a42" containerName="istio-proxy" Oct 31 04:31:17.368169 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:17.368144 5019 cpu_manager.go:414] "containerMap: removing container" podUID="9c0927ea-ba50-457d-9cdc-c289169d44f7" containerName="revoke-keys" Oct 31 04:31:17.368169 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:17.368150 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c0927ea-ba50-457d-9cdc-c289169d44f7" containerName="revoke-keys" Oct 31 04:31:17.368343 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:17.368255 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="16ca6a86-9186-4d31-a3ac-07f81f318a42" containerName="istio-proxy" Oct 31 04:31:17.368343 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:17.368270 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c0927ea-ba50-457d-9cdc-c289169d44f7" containerName="revoke-keys" Oct 31 04:31:17.383356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:17.379791 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d033c6a-bgnt7" Oct 31 04:31:17.397130 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:17.397099 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:31:17.407585 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:17.406643 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:31:17.407585 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:17.406845 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:31:17.407585 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:17.406950 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:31:17.415141 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:17.415113 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d033c6a-bgnt7"] Oct 31 04:31:17.546545 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:17.546508 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8wdh\" (UniqueName: \"kubernetes.io/projected/3ace7b49-3ea3-4594-925a-09e1c3ebeb01-kube-api-access-g8wdh\") pod \"maas-api-revoke-sub-5d033c6a-bgnt7\" (UID: \"3ace7b49-3ea3-4594-925a-09e1c3ebeb01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d033c6a-bgnt7" Oct 31 04:31:17.546728 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:17.546566 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ace7b49-3ea3-4594-925a-09e1c3ebeb01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d033c6a-bgnt7\" (UID: \"3ace7b49-3ea3-4594-925a-09e1c3ebeb01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d033c6a-bgnt7" Oct 31 04:31:17.546775 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:17.546752 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ace7b49-3ea3-4594-925a-09e1c3ebeb01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d033c6a-bgnt7\" (UID: \"3ace7b49-3ea3-4594-925a-09e1c3ebeb01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d033c6a-bgnt7" Oct 31 04:31:17.648416 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:17.648377 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8wdh\" (UniqueName: \"kubernetes.io/projected/3ace7b49-3ea3-4594-925a-09e1c3ebeb01-kube-api-access-g8wdh\") pod \"maas-api-revoke-sub-5d033c6a-bgnt7\" (UID: \"3ace7b49-3ea3-4594-925a-09e1c3ebeb01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d033c6a-bgnt7" Oct 31 04:31:17.649174 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:17.649131 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ace7b49-3ea3-4594-925a-09e1c3ebeb01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d033c6a-bgnt7\" (UID: \"3ace7b49-3ea3-4594-925a-09e1c3ebeb01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d033c6a-bgnt7" Oct 31 04:31:17.653597 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:17.653568 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ace7b49-3ea3-4594-925a-09e1c3ebeb01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d033c6a-bgnt7\" (UID: \"3ace7b49-3ea3-4594-925a-09e1c3ebeb01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d033c6a-bgnt7" Oct 31 04:31:17.670371 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:17.650158 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ace7b49-3ea3-4594-925a-09e1c3ebeb01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d033c6a-bgnt7\" (UID: \"3ace7b49-3ea3-4594-925a-09e1c3ebeb01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d033c6a-bgnt7" Oct 31 04:31:17.699712 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:17.699681 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ace7b49-3ea3-4594-925a-09e1c3ebeb01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d033c6a-bgnt7\" (UID: \"3ace7b49-3ea3-4594-925a-09e1c3ebeb01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d033c6a-bgnt7" Oct 31 04:31:17.700610 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:17.700587 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8wdh\" (UniqueName: \"kubernetes.io/projected/3ace7b49-3ea3-4594-925a-09e1c3ebeb01-kube-api-access-g8wdh\") pod \"maas-api-revoke-sub-5d033c6a-bgnt7\" (UID: \"3ace7b49-3ea3-4594-925a-09e1c3ebeb01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d033c6a-bgnt7" Oct 31 04:31:17.719146 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:17.718866 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d033c6a-bgnt7" Oct 31 04:31:18.471612 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:18.471570 5019 patch_prober.go:29] interesting pod/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Oct 31 04:31:18.472216 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:18.472181 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f" podUID="116c8b42-2784-4f46-b534-ea35159e3cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Oct 31 04:31:18.641821 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:18.627427 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" podUID="b4ced3ab-41e4-4c54-93eb-95eddaae992d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:31:18.764410 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:18.760291 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" podUID="8e2c7041-ad52-49f4-8b55-d16691f4938f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:31:18.875373 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:18.841477 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:31:19.184678 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:19.177811 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d033c6a-bgnt7"] Oct 31 04:31:19.217520 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:19.217466 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mprn9" podUID="ffacd465-1055-444a-9f85-f46a57ac3724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:31:19.221288 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:19.221250 5019 patch_prober.go:29] interesting pod/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Oct 31 04:31:19.221778 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:19.221745 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f" podUID="116c8b42-2784-4f46-b534-ea35159e3cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Oct 31 04:31:19.556528 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:19.556430 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:31:19.577373 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:19.557060 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" podUID="e0f4b8c2-14f4-4052-bf54-dd65bf59e413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:31:19.577373 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:19.571884 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" podUID="643433d8-418e-4102-8bc3-cd544932614a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:31:19.650375 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:19.640747 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" podUID="b4ced3ab-41e4-4c54-93eb-95eddaae992d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:31:19.650375 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:19.641447 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" Oct 31 04:31:19.714011 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:19.713970 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d033c6a-bgnt7" event={"ID":"3ace7b49-3ea3-4594-925a-09e1c3ebeb01","Type":"ContainerStarted","Data":"6114e31d39a0e9cbbbec7a5ea3edb9e3bf19dce0cf8e1666e9cdd30310f3e9db"} Oct 31 04:31:19.735738 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:19.735691 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:31:19.780495 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:19.780090 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:31:19.780495 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:19.780152 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8" podUID="75bc7ee5-eb66-43cf-8bca-5322159afbb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:31:19.846639 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:19.846529 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:31:19.846937 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:19.846887 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp" podUID="6d85d51e-dcd7-4e26-b793-70db760b09a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:31:19.895726 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:19.895680 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 31 04:31:19.919958 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:19.919924 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" Oct 31 04:31:20.111510 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:20.086517 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" podUID="8e2c7041-ad52-49f4-8b55-d16691f4938f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:31:20.111510 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:20.087184 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" Oct 31 04:31:20.212356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:20.186728 5019 patch_prober.go:29] interesting pod/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Oct 31 04:31:20.212356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:20.186795 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f" podUID="116c8b42-2784-4f46-b534-ea35159e3cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Oct 31 04:31:20.413100 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:20.412991 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" podUID="dccd15a9-47f5-4bd1-b21b-9d5ca8b05845" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:31:20.415857 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:20.415829 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" Oct 31 04:31:20.578368 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:20.561903 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" podUID="e0f4b8c2-14f4-4052-bf54-dd65bf59e413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 04:31:20.639036 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:20.639004 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" Oct 31 04:31:20.639685 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:20.571463 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:31:20.646239 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:20.572566 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" podUID="643433d8-418e-4102-8bc3-cd544932614a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:31:20.649619 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:20.647067 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" Oct 31 04:31:20.653713 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:20.653685 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" Oct 31 04:31:20.768249 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:20.755201 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:31:20.768249 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:20.755801 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" Oct 31 04:31:20.931984 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:20.926809 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 31 04:31:20.935946 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:20.934266 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" Oct 31 04:31:20.984654 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:20.984346 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" Oct 31 04:31:20.984654 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:20.984440 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:31:21.040316 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:21.012875 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" Oct 31 04:31:21.044355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:21.044311 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" Oct 31 04:31:21.108866 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:21.108829 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" Oct 31 04:31:21.113201 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:21.113128 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" Oct 31 04:31:21.203515 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:21.203485 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f" Oct 31 04:31:21.205928 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:21.205878 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f" Oct 31 04:31:21.208168 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:21.208109 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f" Oct 31 04:31:21.811207 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:21.811105 5019 generic.go:356] "Generic (PLEG): container finished" podID="3ace7b49-3ea3-4594-925a-09e1c3ebeb01" containerID="0683db2c97930678af2fd272c0b753a5b83a8111f342bd7b1dff8d4d27a49d03" exitCode=0 Oct 31 04:31:21.811207 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:21.811190 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d033c6a-bgnt7" event={"ID":"3ace7b49-3ea3-4594-925a-09e1c3ebeb01","Type":"ContainerDied","Data":"0683db2c97930678af2fd272c0b753a5b83a8111f342bd7b1dff8d4d27a49d03"} Oct 31 04:31:22.392378 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:31:22.392322 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:31:23.049921 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:23.049894 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d033c6a-bgnt7" Oct 31 04:31:23.169042 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:23.168957 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ace7b49-3ea3-4594-925a-09e1c3ebeb01-kube-api-access-g8wdh\" (UniqueName: \"kubernetes.io/projected/3ace7b49-3ea3-4594-925a-09e1c3ebeb01-kube-api-access-g8wdh\") pod \"3ace7b49-3ea3-4594-925a-09e1c3ebeb01\" (UID: \"3ace7b49-3ea3-4594-925a-09e1c3ebeb01\") " Oct 31 04:31:23.169218 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:23.169197 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ace7b49-3ea3-4594-925a-09e1c3ebeb01-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ace7b49-3ea3-4594-925a-09e1c3ebeb01-maas-api-openshift-ca\") pod \"3ace7b49-3ea3-4594-925a-09e1c3ebeb01\" (UID: \"3ace7b49-3ea3-4594-925a-09e1c3ebeb01\") " Oct 31 04:31:23.169420 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:23.169390 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ace7b49-3ea3-4594-925a-09e1c3ebeb01-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ace7b49-3ea3-4594-925a-09e1c3ebeb01-maas-api-xks-ca\") pod \"3ace7b49-3ea3-4594-925a-09e1c3ebeb01\" (UID: \"3ace7b49-3ea3-4594-925a-09e1c3ebeb01\") " Oct 31 04:31:23.169780 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:23.169649 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ace7b49-3ea3-4594-925a-09e1c3ebeb01-maas-api-openshift-ca" pod "3ace7b49-3ea3-4594-925a-09e1c3ebeb01" (UID: "3ace7b49-3ea3-4594-925a-09e1c3ebeb01"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:31:23.170433 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:23.170170 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ace7b49-3ea3-4594-925a-09e1c3ebeb01-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:31:23.171495 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:23.171469 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ace7b49-3ea3-4594-925a-09e1c3ebeb01-kube-api-access-g8wdh" pod "3ace7b49-3ea3-4594-925a-09e1c3ebeb01" (UID: "3ace7b49-3ea3-4594-925a-09e1c3ebeb01"). InnerVolumeSpecName "kube-api-access-g8wdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:31:23.171825 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:23.171799 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ace7b49-3ea3-4594-925a-09e1c3ebeb01-maas-api-xks-ca" pod "3ace7b49-3ea3-4594-925a-09e1c3ebeb01" (UID: "3ace7b49-3ea3-4594-925a-09e1c3ebeb01"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:31:23.272080 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:23.272044 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ace7b49-3ea3-4594-925a-09e1c3ebeb01-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:31:23.272080 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:23.272072 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8wdh\" (UniqueName: \"kubernetes.io/projected/3ace7b49-3ea3-4594-925a-09e1c3ebeb01-kube-api-access-g8wdh\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:31:23.828392 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:23.828363 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d033c6a-bgnt7" Oct 31 04:31:23.828392 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:23.828375 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d033c6a-bgnt7" event={"ID":"3ace7b49-3ea3-4594-925a-09e1c3ebeb01","Type":"ContainerDied","Data":"6114e31d39a0e9cbbbec7a5ea3edb9e3bf19dce0cf8e1666e9cdd30310f3e9db"} Oct 31 04:31:23.828611 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:23.828424 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6114e31d39a0e9cbbbec7a5ea3edb9e3bf19dce0cf8e1666e9cdd30310f3e9db" Oct 31 04:31:25.301217 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:25.301176 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ce95e7f-gkxj9"] Oct 31 04:31:25.301857 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:25.301840 5019 cpu_manager.go:414] "containerMap: removing container" podUID="3ace7b49-3ea3-4594-925a-09e1c3ebeb01" containerName="revoke-keys" Oct 31 04:31:25.301857 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:25.301856 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ace7b49-3ea3-4594-925a-09e1c3ebeb01" containerName="revoke-keys" Oct 31 04:31:25.302010 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:25.301953 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ace7b49-3ea3-4594-925a-09e1c3ebeb01" containerName="revoke-keys" Oct 31 04:31:25.305505 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:25.305485 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce95e7f-gkxj9" Oct 31 04:31:25.320300 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:25.320276 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:31:25.320466 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:25.320306 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:31:25.324552 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:25.324523 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ce95e7f-gkxj9"] Oct 31 04:31:25.332020 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:25.331989 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:31:25.332209 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:25.332192 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:31:25.504431 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:25.504374 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq8mh\" (UniqueName: \"kubernetes.io/projected/69624fdb-1b77-4d58-a5bc-70daa1dd48e5-kube-api-access-bq8mh\") pod \"maas-api-revoke-sub-5ce95e7f-gkxj9\" (UID: \"69624fdb-1b77-4d58-a5bc-70daa1dd48e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce95e7f-gkxj9" Oct 31 04:31:25.504431 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:25.504433 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69624fdb-1b77-4d58-a5bc-70daa1dd48e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ce95e7f-gkxj9\" (UID: \"69624fdb-1b77-4d58-a5bc-70daa1dd48e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce95e7f-gkxj9" Oct 31 04:31:25.504772 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:25.504747 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69624fdb-1b77-4d58-a5bc-70daa1dd48e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ce95e7f-gkxj9\" (UID: \"69624fdb-1b77-4d58-a5bc-70daa1dd48e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce95e7f-gkxj9" Oct 31 04:31:25.606059 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:25.605960 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69624fdb-1b77-4d58-a5bc-70daa1dd48e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ce95e7f-gkxj9\" (UID: \"69624fdb-1b77-4d58-a5bc-70daa1dd48e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce95e7f-gkxj9" Oct 31 04:31:25.606251 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:25.606080 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq8mh\" (UniqueName: \"kubernetes.io/projected/69624fdb-1b77-4d58-a5bc-70daa1dd48e5-kube-api-access-bq8mh\") pod \"maas-api-revoke-sub-5ce95e7f-gkxj9\" (UID: \"69624fdb-1b77-4d58-a5bc-70daa1dd48e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce95e7f-gkxj9" Oct 31 04:31:25.606251 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:25.606116 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69624fdb-1b77-4d58-a5bc-70daa1dd48e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ce95e7f-gkxj9\" (UID: \"69624fdb-1b77-4d58-a5bc-70daa1dd48e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce95e7f-gkxj9" Oct 31 04:31:25.606692 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:25.606668 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69624fdb-1b77-4d58-a5bc-70daa1dd48e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ce95e7f-gkxj9\" (UID: \"69624fdb-1b77-4d58-a5bc-70daa1dd48e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce95e7f-gkxj9" Oct 31 04:31:25.608545 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:25.608520 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69624fdb-1b77-4d58-a5bc-70daa1dd48e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ce95e7f-gkxj9\" (UID: \"69624fdb-1b77-4d58-a5bc-70daa1dd48e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce95e7f-gkxj9" Oct 31 04:31:25.629178 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:25.629138 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq8mh\" (UniqueName: \"kubernetes.io/projected/69624fdb-1b77-4d58-a5bc-70daa1dd48e5-kube-api-access-bq8mh\") pod \"maas-api-revoke-sub-5ce95e7f-gkxj9\" (UID: \"69624fdb-1b77-4d58-a5bc-70daa1dd48e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce95e7f-gkxj9" Oct 31 04:31:25.922304 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:25.922214 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce95e7f-gkxj9" Oct 31 04:31:26.230281 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:26.230246 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ce95e7f-gkxj9"] Oct 31 04:31:26.855034 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:26.854940 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce95e7f-gkxj9" event={"ID":"69624fdb-1b77-4d58-a5bc-70daa1dd48e5","Type":"ContainerStarted","Data":"6c1572ef8da7e96145bca7460707d576f3bb5d5a7d8f856ff34acc06b6321ec0"} Oct 31 04:31:26.855034 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:26.854993 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce95e7f-gkxj9" event={"ID":"69624fdb-1b77-4d58-a5bc-70daa1dd48e5","Type":"ContainerStarted","Data":"8ad524787a6b4af2a99ca33c9c07d990064e6feb6f187a88aa45ab69be8621cd"} Oct 31 04:31:26.896880 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:26.896824 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce95e7f-gkxj9" podStartSLOduration=1.8968095470000002 podStartE2EDuration="1.896809547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:31:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:31:26.895094051 +0000 UTC m=+1641.651432089" watchObservedRunningTime="2026-10-31 04:31:26.896809547 +0000 UTC m=+1641.653147596" Oct 31 04:31:27.864601 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:27.864561 5019 generic.go:356] "Generic (PLEG): container finished" podID="69624fdb-1b77-4d58-a5bc-70daa1dd48e5" containerID="6c1572ef8da7e96145bca7460707d576f3bb5d5a7d8f856ff34acc06b6321ec0" exitCode=0 Oct 31 04:31:27.865026 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:27.864643 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce95e7f-gkxj9" event={"ID":"69624fdb-1b77-4d58-a5bc-70daa1dd48e5","Type":"ContainerDied","Data":"6c1572ef8da7e96145bca7460707d576f3bb5d5a7d8f856ff34acc06b6321ec0"} Oct 31 04:31:29.037354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:29.037303 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce95e7f-gkxj9" Oct 31 04:31:29.049598 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:29.049574 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69624fdb-1b77-4d58-a5bc-70daa1dd48e5-kube-api-access-bq8mh\" (UniqueName: \"kubernetes.io/projected/69624fdb-1b77-4d58-a5bc-70daa1dd48e5-kube-api-access-bq8mh\") pod \"69624fdb-1b77-4d58-a5bc-70daa1dd48e5\" (UID: \"69624fdb-1b77-4d58-a5bc-70daa1dd48e5\") " Oct 31 04:31:29.049730 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:29.049607 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69624fdb-1b77-4d58-a5bc-70daa1dd48e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69624fdb-1b77-4d58-a5bc-70daa1dd48e5-maas-api-xks-ca\") pod \"69624fdb-1b77-4d58-a5bc-70daa1dd48e5\" (UID: \"69624fdb-1b77-4d58-a5bc-70daa1dd48e5\") " Oct 31 04:31:29.049730 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:29.049674 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69624fdb-1b77-4d58-a5bc-70daa1dd48e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69624fdb-1b77-4d58-a5bc-70daa1dd48e5-maas-api-openshift-ca\") pod \"69624fdb-1b77-4d58-a5bc-70daa1dd48e5\" (UID: \"69624fdb-1b77-4d58-a5bc-70daa1dd48e5\") " Oct 31 04:31:29.050112 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:29.050082 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69624fdb-1b77-4d58-a5bc-70daa1dd48e5-maas-api-openshift-ca" pod "69624fdb-1b77-4d58-a5bc-70daa1dd48e5" (UID: "69624fdb-1b77-4d58-a5bc-70daa1dd48e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:31:29.050448 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:29.050427 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69624fdb-1b77-4d58-a5bc-70daa1dd48e5-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:31:29.051743 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:29.051718 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69624fdb-1b77-4d58-a5bc-70daa1dd48e5-kube-api-access-bq8mh" pod "69624fdb-1b77-4d58-a5bc-70daa1dd48e5" (UID: "69624fdb-1b77-4d58-a5bc-70daa1dd48e5"). InnerVolumeSpecName "kube-api-access-bq8mh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:31:29.051743 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:29.051727 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69624fdb-1b77-4d58-a5bc-70daa1dd48e5-maas-api-xks-ca" pod "69624fdb-1b77-4d58-a5bc-70daa1dd48e5" (UID: "69624fdb-1b77-4d58-a5bc-70daa1dd48e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:31:29.152156 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:29.152068 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq8mh\" (UniqueName: \"kubernetes.io/projected/69624fdb-1b77-4d58-a5bc-70daa1dd48e5-kube-api-access-bq8mh\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:31:29.152156 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:29.152096 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69624fdb-1b77-4d58-a5bc-70daa1dd48e5-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:31:29.880229 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:29.880202 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce95e7f-gkxj9" Oct 31 04:31:29.880562 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:29.880240 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce95e7f-gkxj9" event={"ID":"69624fdb-1b77-4d58-a5bc-70daa1dd48e5","Type":"ContainerDied","Data":"8ad524787a6b4af2a99ca33c9c07d990064e6feb6f187a88aa45ab69be8621cd"} Oct 31 04:31:29.880562 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:29.880293 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ad524787a6b4af2a99ca33c9c07d990064e6feb6f187a88aa45ab69be8621cd" Oct 31 04:31:33.911490 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:33.911449 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09a10af3-kzrw6"] Oct 31 04:31:33.912363 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:33.912327 5019 cpu_manager.go:414] "containerMap: removing container" podUID="69624fdb-1b77-4d58-a5bc-70daa1dd48e5" containerName="revoke-keys" Oct 31 04:31:33.912500 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:33.912365 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69624fdb-1b77-4d58-a5bc-70daa1dd48e5" containerName="revoke-keys" Oct 31 04:31:33.912587 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:33.912572 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="69624fdb-1b77-4d58-a5bc-70daa1dd48e5" containerName="revoke-keys" Oct 31 04:31:33.916950 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:33.916927 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a10af3-kzrw6" Oct 31 04:31:33.935061 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:33.935025 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09a10af3-kzrw6"] Oct 31 04:31:33.941266 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:33.941236 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:31:33.941503 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:33.941303 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:31:33.941503 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:33.941420 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:31:33.941503 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:33.941459 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:31:34.029520 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:34.029487 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33e5de72-d4a8-4cc1-ae49-7877abe669ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09a10af3-kzrw6\" (UID: \"33e5de72-d4a8-4cc1-ae49-7877abe669ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a10af3-kzrw6" Oct 31 04:31:34.029724 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:34.029564 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33e5de72-d4a8-4cc1-ae49-7877abe669ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09a10af3-kzrw6\" (UID: \"33e5de72-d4a8-4cc1-ae49-7877abe669ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a10af3-kzrw6" Oct 31 04:31:34.029807 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:34.029784 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hnw4\" (UniqueName: \"kubernetes.io/projected/33e5de72-d4a8-4cc1-ae49-7877abe669ce-kube-api-access-7hnw4\") pod \"maas-api-revoke-sub-09a10af3-kzrw6\" (UID: \"33e5de72-d4a8-4cc1-ae49-7877abe669ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a10af3-kzrw6" Oct 31 04:31:34.130703 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:34.130649 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33e5de72-d4a8-4cc1-ae49-7877abe669ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09a10af3-kzrw6\" (UID: \"33e5de72-d4a8-4cc1-ae49-7877abe669ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a10af3-kzrw6" Oct 31 04:31:34.130914 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:34.130793 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hnw4\" (UniqueName: \"kubernetes.io/projected/33e5de72-d4a8-4cc1-ae49-7877abe669ce-kube-api-access-7hnw4\") pod \"maas-api-revoke-sub-09a10af3-kzrw6\" (UID: \"33e5de72-d4a8-4cc1-ae49-7877abe669ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a10af3-kzrw6" Oct 31 04:31:34.130993 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:34.130941 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33e5de72-d4a8-4cc1-ae49-7877abe669ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09a10af3-kzrw6\" (UID: \"33e5de72-d4a8-4cc1-ae49-7877abe669ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a10af3-kzrw6" Oct 31 04:31:34.131399 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:34.131374 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33e5de72-d4a8-4cc1-ae49-7877abe669ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09a10af3-kzrw6\" (UID: \"33e5de72-d4a8-4cc1-ae49-7877abe669ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a10af3-kzrw6" Oct 31 04:31:34.133313 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:34.133291 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33e5de72-d4a8-4cc1-ae49-7877abe669ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09a10af3-kzrw6\" (UID: \"33e5de72-d4a8-4cc1-ae49-7877abe669ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a10af3-kzrw6" Oct 31 04:31:34.154947 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:34.154914 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hnw4\" (UniqueName: \"kubernetes.io/projected/33e5de72-d4a8-4cc1-ae49-7877abe669ce-kube-api-access-7hnw4\") pod \"maas-api-revoke-sub-09a10af3-kzrw6\" (UID: \"33e5de72-d4a8-4cc1-ae49-7877abe669ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a10af3-kzrw6" Oct 31 04:31:34.232677 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:34.232642 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a10af3-kzrw6" Oct 31 04:31:34.539480 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:31:34.539452 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33e5de72_d4a8_4cc1_ae49_7877abe669ce.slice/crio-21173e775fe387ef544ceaf6f2a2d898f02ddf5969ececf32cddcda9b04c874c WatchSource:0}: Error finding container 21173e775fe387ef544ceaf6f2a2d898f02ddf5969ececf32cddcda9b04c874c: Status 404 returned error can't find the container with id 21173e775fe387ef544ceaf6f2a2d898f02ddf5969ececf32cddcda9b04c874c Oct 31 04:31:34.539480 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:34.539465 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09a10af3-kzrw6"] Oct 31 04:31:34.924530 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:34.924493 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a10af3-kzrw6" event={"ID":"33e5de72-d4a8-4cc1-ae49-7877abe669ce","Type":"ContainerStarted","Data":"a951b2503c42e6c502fdb4fea75fb4f5b37c1a3d7e5997c9c79f6bfbb496f6fc"} Oct 31 04:31:34.924939 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:34.924551 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a10af3-kzrw6" event={"ID":"33e5de72-d4a8-4cc1-ae49-7877abe669ce","Type":"ContainerStarted","Data":"21173e775fe387ef544ceaf6f2a2d898f02ddf5969ececf32cddcda9b04c874c"} Oct 31 04:31:34.966830 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:34.966773 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a10af3-kzrw6" podStartSLOduration=1.96675724 podStartE2EDuration="1.96675724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:31:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:31:34.96494214 +0000 UTC m=+1649.721280181" watchObservedRunningTime="2026-10-31 04:31:34.96675724 +0000 UTC m=+1649.723095278" Oct 31 04:31:35.936457 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:35.936317 5019 generic.go:356] "Generic (PLEG): container finished" podID="33e5de72-d4a8-4cc1-ae49-7877abe669ce" containerID="a951b2503c42e6c502fdb4fea75fb4f5b37c1a3d7e5997c9c79f6bfbb496f6fc" exitCode=0 Oct 31 04:31:35.936925 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:35.936581 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a10af3-kzrw6" event={"ID":"33e5de72-d4a8-4cc1-ae49-7877abe669ce","Type":"ContainerDied","Data":"a951b2503c42e6c502fdb4fea75fb4f5b37c1a3d7e5997c9c79f6bfbb496f6fc"} Oct 31 04:31:37.124094 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:37.124065 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a10af3-kzrw6" Oct 31 04:31:37.277264 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:37.277222 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33e5de72-d4a8-4cc1-ae49-7877abe669ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33e5de72-d4a8-4cc1-ae49-7877abe669ce-maas-api-xks-ca\") pod \"33e5de72-d4a8-4cc1-ae49-7877abe669ce\" (UID: \"33e5de72-d4a8-4cc1-ae49-7877abe669ce\") " Oct 31 04:31:37.277481 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:37.277281 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33e5de72-d4a8-4cc1-ae49-7877abe669ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33e5de72-d4a8-4cc1-ae49-7877abe669ce-maas-api-openshift-ca\") pod \"33e5de72-d4a8-4cc1-ae49-7877abe669ce\" (UID: \"33e5de72-d4a8-4cc1-ae49-7877abe669ce\") " Oct 31 04:31:37.277481 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:37.277452 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33e5de72-d4a8-4cc1-ae49-7877abe669ce-kube-api-access-7hnw4\" (UniqueName: \"kubernetes.io/projected/33e5de72-d4a8-4cc1-ae49-7877abe669ce-kube-api-access-7hnw4\") pod \"33e5de72-d4a8-4cc1-ae49-7877abe669ce\" (UID: \"33e5de72-d4a8-4cc1-ae49-7877abe669ce\") " Oct 31 04:31:37.277783 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:37.277759 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33e5de72-d4a8-4cc1-ae49-7877abe669ce-maas-api-openshift-ca" pod "33e5de72-d4a8-4cc1-ae49-7877abe669ce" (UID: "33e5de72-d4a8-4cc1-ae49-7877abe669ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:31:37.278204 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:37.278178 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33e5de72-d4a8-4cc1-ae49-7877abe669ce-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:31:37.279764 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:37.279732 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33e5de72-d4a8-4cc1-ae49-7877abe669ce-maas-api-xks-ca" pod "33e5de72-d4a8-4cc1-ae49-7877abe669ce" (UID: "33e5de72-d4a8-4cc1-ae49-7877abe669ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:31:37.279861 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:37.279771 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33e5de72-d4a8-4cc1-ae49-7877abe669ce-kube-api-access-7hnw4" pod "33e5de72-d4a8-4cc1-ae49-7877abe669ce" (UID: "33e5de72-d4a8-4cc1-ae49-7877abe669ce"). InnerVolumeSpecName "kube-api-access-7hnw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:31:37.379938 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:37.379891 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hnw4\" (UniqueName: \"kubernetes.io/projected/33e5de72-d4a8-4cc1-ae49-7877abe669ce-kube-api-access-7hnw4\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:31:37.379938 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:37.379928 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33e5de72-d4a8-4cc1-ae49-7877abe669ce-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:31:37.954573 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:37.954546 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a10af3-kzrw6" Oct 31 04:31:37.954752 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:37.954545 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a10af3-kzrw6" event={"ID":"33e5de72-d4a8-4cc1-ae49-7877abe669ce","Type":"ContainerDied","Data":"21173e775fe387ef544ceaf6f2a2d898f02ddf5969ececf32cddcda9b04c874c"} Oct 31 04:31:37.954752 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:37.954670 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21173e775fe387ef544ceaf6f2a2d898f02ddf5969ececf32cddcda9b04c874c" Oct 31 04:31:41.708234 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:41.708198 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f8cb9ed-fb8mh"] Oct 31 04:31:41.708831 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:41.708817 5019 cpu_manager.go:414] "containerMap: removing container" podUID="33e5de72-d4a8-4cc1-ae49-7877abe669ce" containerName="revoke-keys" Oct 31 04:31:41.708884 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:41.708831 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33e5de72-d4a8-4cc1-ae49-7877abe669ce" containerName="revoke-keys" Oct 31 04:31:41.708946 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:41.708937 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="33e5de72-d4a8-4cc1-ae49-7877abe669ce" containerName="revoke-keys" Oct 31 04:31:41.712711 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:41.712688 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f8cb9ed-fb8mh" Oct 31 04:31:41.732297 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:41.732274 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f8cb9ed-fb8mh"] Oct 31 04:31:41.739585 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:41.739565 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:31:41.740014 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:41.739996 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:31:41.740120 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:41.740094 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:31:41.742231 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:41.742214 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:31:41.840364 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:41.837596 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b49996e2-beb8-4dcf-b709-ddc13df6976c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f8cb9ed-fb8mh\" (UID: \"b49996e2-beb8-4dcf-b709-ddc13df6976c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f8cb9ed-fb8mh" Oct 31 04:31:41.840364 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:41.837656 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b49996e2-beb8-4dcf-b709-ddc13df6976c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f8cb9ed-fb8mh\" (UID: \"b49996e2-beb8-4dcf-b709-ddc13df6976c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f8cb9ed-fb8mh" Oct 31 04:31:41.840364 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:41.837774 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vgdb\" (UniqueName: \"kubernetes.io/projected/b49996e2-beb8-4dcf-b709-ddc13df6976c-kube-api-access-7vgdb\") pod \"maas-api-revoke-sub-6f8cb9ed-fb8mh\" (UID: \"b49996e2-beb8-4dcf-b709-ddc13df6976c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f8cb9ed-fb8mh" Oct 31 04:31:41.952824 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:41.952773 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b49996e2-beb8-4dcf-b709-ddc13df6976c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f8cb9ed-fb8mh\" (UID: \"b49996e2-beb8-4dcf-b709-ddc13df6976c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f8cb9ed-fb8mh" Oct 31 04:31:41.953087 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:41.953069 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b49996e2-beb8-4dcf-b709-ddc13df6976c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f8cb9ed-fb8mh\" (UID: \"b49996e2-beb8-4dcf-b709-ddc13df6976c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f8cb9ed-fb8mh" Oct 31 04:31:41.953392 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:41.953319 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vgdb\" (UniqueName: \"kubernetes.io/projected/b49996e2-beb8-4dcf-b709-ddc13df6976c-kube-api-access-7vgdb\") pod \"maas-api-revoke-sub-6f8cb9ed-fb8mh\" (UID: \"b49996e2-beb8-4dcf-b709-ddc13df6976c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f8cb9ed-fb8mh" Oct 31 04:31:41.957347 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:41.956633 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b49996e2-beb8-4dcf-b709-ddc13df6976c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f8cb9ed-fb8mh\" (UID: \"b49996e2-beb8-4dcf-b709-ddc13df6976c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f8cb9ed-fb8mh" Oct 31 04:31:41.989175 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:41.989099 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b49996e2-beb8-4dcf-b709-ddc13df6976c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f8cb9ed-fb8mh\" (UID: \"b49996e2-beb8-4dcf-b709-ddc13df6976c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f8cb9ed-fb8mh" Oct 31 04:31:42.016514 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:42.015771 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vgdb\" (UniqueName: \"kubernetes.io/projected/b49996e2-beb8-4dcf-b709-ddc13df6976c-kube-api-access-7vgdb\") pod \"maas-api-revoke-sub-6f8cb9ed-fb8mh\" (UID: \"b49996e2-beb8-4dcf-b709-ddc13df6976c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f8cb9ed-fb8mh" Oct 31 04:31:42.027431 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:42.027237 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f8cb9ed-fb8mh" Oct 31 04:31:42.402844 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:31:42.402813 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb49996e2_beb8_4dcf_b709_ddc13df6976c.slice/crio-e4cce4f49f4a2b54e5936dde2f396d9b86281ce9d73bfa69db222e3538f1c34e WatchSource:0}: Error finding container e4cce4f49f4a2b54e5936dde2f396d9b86281ce9d73bfa69db222e3538f1c34e: Status 404 returned error can't find the container with id e4cce4f49f4a2b54e5936dde2f396d9b86281ce9d73bfa69db222e3538f1c34e Oct 31 04:31:42.422965 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:42.422675 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f8cb9ed-fb8mh"] Oct 31 04:31:43.044827 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:43.044789 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f8cb9ed-fb8mh" event={"ID":"b49996e2-beb8-4dcf-b709-ddc13df6976c","Type":"ContainerStarted","Data":"d55cace45d7e120bb4087a067f3dcf6d4ebed6cc1ff277845640a2c67520c19f"} Oct 31 04:31:43.045234 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:43.044851 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f8cb9ed-fb8mh" event={"ID":"b49996e2-beb8-4dcf-b709-ddc13df6976c","Type":"ContainerStarted","Data":"e4cce4f49f4a2b54e5936dde2f396d9b86281ce9d73bfa69db222e3538f1c34e"} Oct 31 04:31:43.091355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:43.088506 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f8cb9ed-fb8mh" podStartSLOduration=2.088486681 podStartE2EDuration="2.088486681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:31:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:31:43.087706964 +0000 UTC m=+1657.844045003" watchObservedRunningTime="2026-10-31 04:31:43.088486681 +0000 UTC m=+1657.844824720" Oct 31 04:31:43.701790 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:43.701751 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ab66a90-tl5cg"] Oct 31 04:31:43.705998 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:43.705979 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ab66a90-tl5cg" Oct 31 04:31:43.723718 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:43.723691 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ab66a90-tl5cg"] Oct 31 04:31:43.808233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:43.808145 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g6kk\" (UniqueName: \"kubernetes.io/projected/26ef4705-4fef-4182-ad47-d3d7c232e0a0-kube-api-access-5g6kk\") pod \"maas-api-revoke-sub-7ab66a90-tl5cg\" (UID: \"26ef4705-4fef-4182-ad47-d3d7c232e0a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ab66a90-tl5cg" Oct 31 04:31:43.808408 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:43.808297 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26ef4705-4fef-4182-ad47-d3d7c232e0a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ab66a90-tl5cg\" (UID: \"26ef4705-4fef-4182-ad47-d3d7c232e0a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ab66a90-tl5cg" Oct 31 04:31:43.808494 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:43.808476 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26ef4705-4fef-4182-ad47-d3d7c232e0a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ab66a90-tl5cg\" (UID: \"26ef4705-4fef-4182-ad47-d3d7c232e0a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ab66a90-tl5cg" Oct 31 04:31:43.909457 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:43.909422 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26ef4705-4fef-4182-ad47-d3d7c232e0a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ab66a90-tl5cg\" (UID: \"26ef4705-4fef-4182-ad47-d3d7c232e0a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ab66a90-tl5cg" Oct 31 04:31:43.909666 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:43.909542 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g6kk\" (UniqueName: \"kubernetes.io/projected/26ef4705-4fef-4182-ad47-d3d7c232e0a0-kube-api-access-5g6kk\") pod \"maas-api-revoke-sub-7ab66a90-tl5cg\" (UID: \"26ef4705-4fef-4182-ad47-d3d7c232e0a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ab66a90-tl5cg" Oct 31 04:31:43.909666 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:43.909644 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26ef4705-4fef-4182-ad47-d3d7c232e0a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ab66a90-tl5cg\" (UID: \"26ef4705-4fef-4182-ad47-d3d7c232e0a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ab66a90-tl5cg" Oct 31 04:31:43.910279 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:43.910254 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26ef4705-4fef-4182-ad47-d3d7c232e0a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ab66a90-tl5cg\" (UID: \"26ef4705-4fef-4182-ad47-d3d7c232e0a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ab66a90-tl5cg" Oct 31 04:31:43.912075 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:43.912048 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26ef4705-4fef-4182-ad47-d3d7c232e0a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ab66a90-tl5cg\" (UID: \"26ef4705-4fef-4182-ad47-d3d7c232e0a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ab66a90-tl5cg" Oct 31 04:31:43.933026 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:43.932998 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g6kk\" (UniqueName: \"kubernetes.io/projected/26ef4705-4fef-4182-ad47-d3d7c232e0a0-kube-api-access-5g6kk\") pod \"maas-api-revoke-sub-7ab66a90-tl5cg\" (UID: \"26ef4705-4fef-4182-ad47-d3d7c232e0a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ab66a90-tl5cg" Oct 31 04:31:44.020056 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:44.020031 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ab66a90-tl5cg" Oct 31 04:31:44.054293 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:44.054259 5019 generic.go:356] "Generic (PLEG): container finished" podID="b49996e2-beb8-4dcf-b709-ddc13df6976c" containerID="d55cace45d7e120bb4087a067f3dcf6d4ebed6cc1ff277845640a2c67520c19f" exitCode=0 Oct 31 04:31:44.054686 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:44.054475 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f8cb9ed-fb8mh" event={"ID":"b49996e2-beb8-4dcf-b709-ddc13df6976c","Type":"ContainerDied","Data":"d55cace45d7e120bb4087a067f3dcf6d4ebed6cc1ff277845640a2c67520c19f"} Oct 31 04:31:44.343254 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:44.343224 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ab66a90-tl5cg"] Oct 31 04:31:44.343994 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:31:44.343958 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26ef4705_4fef_4182_ad47_d3d7c232e0a0.slice/crio-56927645e53ce7033424d511c2aa984deb4919406607bca1def56e7ea59da10a WatchSource:0}: Error finding container 56927645e53ce7033424d511c2aa984deb4919406607bca1def56e7ea59da10a: Status 404 returned error can't find the container with id 56927645e53ce7033424d511c2aa984deb4919406607bca1def56e7ea59da10a Oct 31 04:31:45.064897 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:45.064860 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ab66a90-tl5cg" event={"ID":"26ef4705-4fef-4182-ad47-d3d7c232e0a0","Type":"ContainerStarted","Data":"92f839d89e8307063c82a3778a3792cf73070a8b574d42cf0804e4c4449404f6"} Oct 31 04:31:45.065318 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:45.064920 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ab66a90-tl5cg" event={"ID":"26ef4705-4fef-4182-ad47-d3d7c232e0a0","Type":"ContainerStarted","Data":"56927645e53ce7033424d511c2aa984deb4919406607bca1def56e7ea59da10a"} Oct 31 04:31:45.107039 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:45.106981 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ab66a90-tl5cg" podStartSLOduration=2.106962924 podStartE2EDuration="2.106962924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:31:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:31:45.105036838 +0000 UTC m=+1659.861374878" watchObservedRunningTime="2026-10-31 04:31:45.106962924 +0000 UTC m=+1659.863300966" Oct 31 04:31:45.293522 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:45.293496 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f8cb9ed-fb8mh" Oct 31 04:31:45.436040 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:45.436007 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b49996e2-beb8-4dcf-b709-ddc13df6976c-kube-api-access-7vgdb\" (UniqueName: \"kubernetes.io/projected/b49996e2-beb8-4dcf-b709-ddc13df6976c-kube-api-access-7vgdb\") pod \"b49996e2-beb8-4dcf-b709-ddc13df6976c\" (UID: \"b49996e2-beb8-4dcf-b709-ddc13df6976c\") " Oct 31 04:31:45.436200 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:45.436137 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b49996e2-beb8-4dcf-b709-ddc13df6976c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b49996e2-beb8-4dcf-b709-ddc13df6976c-maas-api-openshift-ca\") pod \"b49996e2-beb8-4dcf-b709-ddc13df6976c\" (UID: \"b49996e2-beb8-4dcf-b709-ddc13df6976c\") " Oct 31 04:31:45.436200 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:45.436181 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b49996e2-beb8-4dcf-b709-ddc13df6976c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b49996e2-beb8-4dcf-b709-ddc13df6976c-maas-api-xks-ca\") pod \"b49996e2-beb8-4dcf-b709-ddc13df6976c\" (UID: \"b49996e2-beb8-4dcf-b709-ddc13df6976c\") " Oct 31 04:31:45.436502 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:45.436470 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b49996e2-beb8-4dcf-b709-ddc13df6976c-maas-api-openshift-ca" pod "b49996e2-beb8-4dcf-b709-ddc13df6976c" (UID: "b49996e2-beb8-4dcf-b709-ddc13df6976c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:31:45.436682 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:45.436666 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b49996e2-beb8-4dcf-b709-ddc13df6976c-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:31:45.438039 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:45.438010 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b49996e2-beb8-4dcf-b709-ddc13df6976c-kube-api-access-7vgdb" pod "b49996e2-beb8-4dcf-b709-ddc13df6976c" (UID: "b49996e2-beb8-4dcf-b709-ddc13df6976c"). InnerVolumeSpecName "kube-api-access-7vgdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:31:45.438141 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:45.438109 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b49996e2-beb8-4dcf-b709-ddc13df6976c-maas-api-xks-ca" pod "b49996e2-beb8-4dcf-b709-ddc13df6976c" (UID: "b49996e2-beb8-4dcf-b709-ddc13df6976c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:31:45.538053 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:45.538027 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vgdb\" (UniqueName: \"kubernetes.io/projected/b49996e2-beb8-4dcf-b709-ddc13df6976c-kube-api-access-7vgdb\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:31:45.538053 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:45.538050 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b49996e2-beb8-4dcf-b709-ddc13df6976c-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:31:46.074444 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:46.074416 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f8cb9ed-fb8mh" Oct 31 04:31:46.074444 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:46.074425 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f8cb9ed-fb8mh" event={"ID":"b49996e2-beb8-4dcf-b709-ddc13df6976c","Type":"ContainerDied","Data":"e4cce4f49f4a2b54e5936dde2f396d9b86281ce9d73bfa69db222e3538f1c34e"} Oct 31 04:31:46.074946 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:46.074484 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4cce4f49f4a2b54e5936dde2f396d9b86281ce9d73bfa69db222e3538f1c34e" Oct 31 04:31:46.076322 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:46.076298 5019 generic.go:356] "Generic (PLEG): container finished" podID="26ef4705-4fef-4182-ad47-d3d7c232e0a0" containerID="92f839d89e8307063c82a3778a3792cf73070a8b574d42cf0804e4c4449404f6" exitCode=0 Oct 31 04:31:46.076454 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:46.076325 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ab66a90-tl5cg" event={"ID":"26ef4705-4fef-4182-ad47-d3d7c232e0a0","Type":"ContainerDied","Data":"92f839d89e8307063c82a3778a3792cf73070a8b574d42cf0804e4c4449404f6"} Oct 31 04:31:46.332247 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:46.332166 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85913ce3-mjwbv"] Oct 31 04:31:46.332816 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:46.332801 5019 cpu_manager.go:414] "containerMap: removing container" podUID="b49996e2-beb8-4dcf-b709-ddc13df6976c" containerName="revoke-keys" Oct 31 04:31:46.332887 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:46.332817 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b49996e2-beb8-4dcf-b709-ddc13df6976c" containerName="revoke-keys" Oct 31 04:31:46.332957 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:46.332947 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="b49996e2-beb8-4dcf-b709-ddc13df6976c" containerName="revoke-keys" Oct 31 04:31:46.336866 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:46.336840 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85913ce3-mjwbv" Oct 31 04:31:46.357437 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:46.357410 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85913ce3-mjwbv"] Oct 31 04:31:46.456303 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:46.456263 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2vbg\" (UniqueName: \"kubernetes.io/projected/8041aee9-d4f6-41e3-8309-b298131dbf49-kube-api-access-k2vbg\") pod \"maas-api-revoke-sub-85913ce3-mjwbv\" (UID: \"8041aee9-d4f6-41e3-8309-b298131dbf49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85913ce3-mjwbv" Oct 31 04:31:46.456493 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:46.456320 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8041aee9-d4f6-41e3-8309-b298131dbf49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85913ce3-mjwbv\" (UID: \"8041aee9-d4f6-41e3-8309-b298131dbf49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85913ce3-mjwbv" Oct 31 04:31:46.456544 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:46.456494 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8041aee9-d4f6-41e3-8309-b298131dbf49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85913ce3-mjwbv\" (UID: \"8041aee9-d4f6-41e3-8309-b298131dbf49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85913ce3-mjwbv" Oct 31 04:31:46.557672 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:46.557621 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2vbg\" (UniqueName: \"kubernetes.io/projected/8041aee9-d4f6-41e3-8309-b298131dbf49-kube-api-access-k2vbg\") pod \"maas-api-revoke-sub-85913ce3-mjwbv\" (UID: \"8041aee9-d4f6-41e3-8309-b298131dbf49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85913ce3-mjwbv" Oct 31 04:31:46.557672 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:46.557673 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8041aee9-d4f6-41e3-8309-b298131dbf49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85913ce3-mjwbv\" (UID: \"8041aee9-d4f6-41e3-8309-b298131dbf49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85913ce3-mjwbv" Oct 31 04:31:46.557966 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:46.557761 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8041aee9-d4f6-41e3-8309-b298131dbf49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85913ce3-mjwbv\" (UID: \"8041aee9-d4f6-41e3-8309-b298131dbf49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85913ce3-mjwbv" Oct 31 04:31:46.558749 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:46.558719 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8041aee9-d4f6-41e3-8309-b298131dbf49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85913ce3-mjwbv\" (UID: \"8041aee9-d4f6-41e3-8309-b298131dbf49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85913ce3-mjwbv" Oct 31 04:31:46.560720 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:46.560694 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8041aee9-d4f6-41e3-8309-b298131dbf49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85913ce3-mjwbv\" (UID: \"8041aee9-d4f6-41e3-8309-b298131dbf49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85913ce3-mjwbv" Oct 31 04:31:46.581467 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:46.581415 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2vbg\" (UniqueName: \"kubernetes.io/projected/8041aee9-d4f6-41e3-8309-b298131dbf49-kube-api-access-k2vbg\") pod \"maas-api-revoke-sub-85913ce3-mjwbv\" (UID: \"8041aee9-d4f6-41e3-8309-b298131dbf49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85913ce3-mjwbv" Oct 31 04:31:46.653907 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:46.653844 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85913ce3-mjwbv" Oct 31 04:31:47.022403 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:31:47.016113 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8041aee9_d4f6_41e3_8309_b298131dbf49.slice/crio-31995a40d3be1fee8d3df1b1c365787f71466111c3748049b3d324991010ebda WatchSource:0}: Error finding container 31995a40d3be1fee8d3df1b1c365787f71466111c3748049b3d324991010ebda: Status 404 returned error can't find the container with id 31995a40d3be1fee8d3df1b1c365787f71466111c3748049b3d324991010ebda Oct 31 04:31:47.022403 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:47.016835 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85913ce3-mjwbv"] Oct 31 04:31:47.087790 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:47.087756 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85913ce3-mjwbv" event={"ID":"8041aee9-d4f6-41e3-8309-b298131dbf49","Type":"ContainerStarted","Data":"31995a40d3be1fee8d3df1b1c365787f71466111c3748049b3d324991010ebda"} Oct 31 04:31:47.190680 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:47.190644 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d1d441-588dbc6899-4rnfz"] Oct 31 04:31:47.217207 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:47.217157 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d1d441-669694c87-fhkbx"] Oct 31 04:31:47.331376 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:47.330827 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ab66a90-tl5cg" Oct 31 04:31:47.457374 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:47.457320 5019 patch_prober.go:29] interesting pod/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:31:47.457556 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:47.457401 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s" podUID="9eb20738-52aa-4cc7-990a-ddd8c704e683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:31:47.483404 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:47.483369 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26ef4705-4fef-4182-ad47-d3d7c232e0a0-kube-api-access-5g6kk\" (UniqueName: \"kubernetes.io/projected/26ef4705-4fef-4182-ad47-d3d7c232e0a0-kube-api-access-5g6kk\") pod \"26ef4705-4fef-4182-ad47-d3d7c232e0a0\" (UID: \"26ef4705-4fef-4182-ad47-d3d7c232e0a0\") " Oct 31 04:31:47.483604 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:47.483485 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26ef4705-4fef-4182-ad47-d3d7c232e0a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26ef4705-4fef-4182-ad47-d3d7c232e0a0-maas-api-openshift-ca\") pod \"26ef4705-4fef-4182-ad47-d3d7c232e0a0\" (UID: \"26ef4705-4fef-4182-ad47-d3d7c232e0a0\") " Oct 31 04:31:47.483717 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:47.483697 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26ef4705-4fef-4182-ad47-d3d7c232e0a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26ef4705-4fef-4182-ad47-d3d7c232e0a0-maas-api-xks-ca\") pod \"26ef4705-4fef-4182-ad47-d3d7c232e0a0\" (UID: \"26ef4705-4fef-4182-ad47-d3d7c232e0a0\") " Oct 31 04:31:47.484102 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:47.484082 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26ef4705-4fef-4182-ad47-d3d7c232e0a0-maas-api-openshift-ca" pod "26ef4705-4fef-4182-ad47-d3d7c232e0a0" (UID: "26ef4705-4fef-4182-ad47-d3d7c232e0a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:31:47.486753 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:47.486733 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26ef4705-4fef-4182-ad47-d3d7c232e0a0-maas-api-xks-ca" pod "26ef4705-4fef-4182-ad47-d3d7c232e0a0" (UID: "26ef4705-4fef-4182-ad47-d3d7c232e0a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:31:47.493150 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:47.493122 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26ef4705-4fef-4182-ad47-d3d7c232e0a0-kube-api-access-5g6kk" pod "26ef4705-4fef-4182-ad47-d3d7c232e0a0" (UID: "26ef4705-4fef-4182-ad47-d3d7c232e0a0"). InnerVolumeSpecName "kube-api-access-5g6kk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:31:47.584891 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:47.584863 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5g6kk\" (UniqueName: \"kubernetes.io/projected/26ef4705-4fef-4182-ad47-d3d7c232e0a0-kube-api-access-5g6kk\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:31:47.584891 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:47.584892 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26ef4705-4fef-4182-ad47-d3d7c232e0a0-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:31:47.585052 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:47.584907 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26ef4705-4fef-4182-ad47-d3d7c232e0a0-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:31:48.035491 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:48.035459 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81af36e3-qnf9f"] Oct 31 04:31:48.036145 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:48.036130 5019 cpu_manager.go:414] "containerMap: removing container" podUID="26ef4705-4fef-4182-ad47-d3d7c232e0a0" containerName="revoke-keys" Oct 31 04:31:48.036191 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:48.036146 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26ef4705-4fef-4182-ad47-d3d7c232e0a0" containerName="revoke-keys" Oct 31 04:31:48.036270 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:48.036261 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="26ef4705-4fef-4182-ad47-d3d7c232e0a0" containerName="revoke-keys" Oct 31 04:31:48.040181 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:48.040163 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81af36e3-qnf9f" Oct 31 04:31:48.061054 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:48.060944 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81af36e3-qnf9f"] Oct 31 04:31:48.094771 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:48.094734 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80707894-d35d-43c3-8bbb-c443f7e9a4e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81af36e3-qnf9f\" (UID: \"80707894-d35d-43c3-8bbb-c443f7e9a4e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81af36e3-qnf9f" Oct 31 04:31:48.095384 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:48.095360 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gblq8\" (UniqueName: \"kubernetes.io/projected/80707894-d35d-43c3-8bbb-c443f7e9a4e2-kube-api-access-gblq8\") pod \"maas-api-revoke-sub-81af36e3-qnf9f\" (UID: \"80707894-d35d-43c3-8bbb-c443f7e9a4e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81af36e3-qnf9f" Oct 31 04:31:48.095607 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:48.095587 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80707894-d35d-43c3-8bbb-c443f7e9a4e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81af36e3-qnf9f\" (UID: \"80707894-d35d-43c3-8bbb-c443f7e9a4e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81af36e3-qnf9f" Oct 31 04:31:48.104240 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:48.104171 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ab66a90-tl5cg" event={"ID":"26ef4705-4fef-4182-ad47-d3d7c232e0a0","Type":"ContainerDied","Data":"56927645e53ce7033424d511c2aa984deb4919406607bca1def56e7ea59da10a"} Oct 31 04:31:48.104240 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:48.104216 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ab66a90-tl5cg" Oct 31 04:31:48.104877 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:48.104224 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56927645e53ce7033424d511c2aa984deb4919406607bca1def56e7ea59da10a" Oct 31 04:31:48.106827 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:48.106804 5019 generic.go:356] "Generic (PLEG): container finished" podID="8041aee9-d4f6-41e3-8309-b298131dbf49" containerID="c756020a4ac7c027dd90aea145781e56963fd24c14de885e0490798d6553e27b" exitCode=0 Oct 31 04:31:48.107078 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:48.107055 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85913ce3-mjwbv" event={"ID":"8041aee9-d4f6-41e3-8309-b298131dbf49","Type":"ContainerDied","Data":"c756020a4ac7c027dd90aea145781e56963fd24c14de885e0490798d6553e27b"} Oct 31 04:31:48.196900 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:48.196863 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80707894-d35d-43c3-8bbb-c443f7e9a4e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81af36e3-qnf9f\" (UID: \"80707894-d35d-43c3-8bbb-c443f7e9a4e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81af36e3-qnf9f" Oct 31 04:31:48.197087 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:48.196992 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gblq8\" (UniqueName: \"kubernetes.io/projected/80707894-d35d-43c3-8bbb-c443f7e9a4e2-kube-api-access-gblq8\") pod \"maas-api-revoke-sub-81af36e3-qnf9f\" (UID: \"80707894-d35d-43c3-8bbb-c443f7e9a4e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81af36e3-qnf9f" Oct 31 04:31:48.197087 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:48.197079 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80707894-d35d-43c3-8bbb-c443f7e9a4e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81af36e3-qnf9f\" (UID: \"80707894-d35d-43c3-8bbb-c443f7e9a4e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81af36e3-qnf9f" Oct 31 04:31:48.197865 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:48.197841 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80707894-d35d-43c3-8bbb-c443f7e9a4e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81af36e3-qnf9f\" (UID: \"80707894-d35d-43c3-8bbb-c443f7e9a4e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81af36e3-qnf9f" Oct 31 04:31:48.200001 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:48.199975 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80707894-d35d-43c3-8bbb-c443f7e9a4e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81af36e3-qnf9f\" (UID: \"80707894-d35d-43c3-8bbb-c443f7e9a4e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81af36e3-qnf9f" Oct 31 04:31:48.220103 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:48.220045 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gblq8\" (UniqueName: \"kubernetes.io/projected/80707894-d35d-43c3-8bbb-c443f7e9a4e2-kube-api-access-gblq8\") pod \"maas-api-revoke-sub-81af36e3-qnf9f\" (UID: \"80707894-d35d-43c3-8bbb-c443f7e9a4e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81af36e3-qnf9f" Oct 31 04:31:48.356343 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:48.356301 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81af36e3-qnf9f" Oct 31 04:31:48.704218 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:31:48.704187 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80707894_d35d_43c3_8bbb_c443f7e9a4e2.slice/crio-db49cb52196a49e8a472d34a4e998e9bce33b3b3c4feccaf470dec871cc86581 WatchSource:0}: Error finding container db49cb52196a49e8a472d34a4e998e9bce33b3b3c4feccaf470dec871cc86581: Status 404 returned error can't find the container with id db49cb52196a49e8a472d34a4e998e9bce33b3b3c4feccaf470dec871cc86581 Oct 31 04:31:48.706699 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:48.706671 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81af36e3-qnf9f"] Oct 31 04:31:49.117474 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:49.117437 5019 generic.go:356] "Generic (PLEG): container finished" podID="9eb20738-52aa-4cc7-990a-ddd8c704e683" containerID="0e312c3b42b8ad05f96fb8246c9830331d8048aae53047546848409bf98da983" exitCode=0 Oct 31 04:31:49.117899 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:49.117519 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s" event={"ID":"9eb20738-52aa-4cc7-990a-ddd8c704e683","Type":"ContainerDied","Data":"0e312c3b42b8ad05f96fb8246c9830331d8048aae53047546848409bf98da983"} Oct 31 04:31:49.118314 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:49.118293 5019 scope.go:122] "RemoveContainer" containerID="0e312c3b42b8ad05f96fb8246c9830331d8048aae53047546848409bf98da983" Oct 31 04:31:49.119791 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:49.119764 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81af36e3-qnf9f" event={"ID":"80707894-d35d-43c3-8bbb-c443f7e9a4e2","Type":"ContainerStarted","Data":"2a5379848e94cf984747f3d18aeaa5990a6186e519c125b63c7256f4b87726ac"} Oct 31 04:31:49.119907 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:49.119837 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81af36e3-qnf9f" event={"ID":"80707894-d35d-43c3-8bbb-c443f7e9a4e2","Type":"ContainerStarted","Data":"db49cb52196a49e8a472d34a4e998e9bce33b3b3c4feccaf470dec871cc86581"} Oct 31 04:31:49.119993 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:49.119967 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:31:49.120072 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:49.120053 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:31:49.120128 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:49.120110 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:31:49.209318 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:49.209249 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81af36e3-qnf9f" podStartSLOduration=1.20921822 podStartE2EDuration="1.20921822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:31:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:31:49.207144073 +0000 UTC m=+1663.963482111" watchObservedRunningTime="2026-10-31 04:31:49.20921822 +0000 UTC m=+1663.965556259" Oct 31 04:31:49.348165 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:49.347781 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85913ce3-mjwbv" Oct 31 04:31:49.534298 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:49.534269 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8041aee9-d4f6-41e3-8309-b298131dbf49-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8041aee9-d4f6-41e3-8309-b298131dbf49-maas-api-openshift-ca\") pod \"8041aee9-d4f6-41e3-8309-b298131dbf49\" (UID: \"8041aee9-d4f6-41e3-8309-b298131dbf49\") " Oct 31 04:31:49.534491 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:49.534395 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8041aee9-d4f6-41e3-8309-b298131dbf49-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8041aee9-d4f6-41e3-8309-b298131dbf49-maas-api-xks-ca\") pod \"8041aee9-d4f6-41e3-8309-b298131dbf49\" (UID: \"8041aee9-d4f6-41e3-8309-b298131dbf49\") " Oct 31 04:31:49.534573 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:49.534553 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8041aee9-d4f6-41e3-8309-b298131dbf49-kube-api-access-k2vbg\" (UniqueName: \"kubernetes.io/projected/8041aee9-d4f6-41e3-8309-b298131dbf49-kube-api-access-k2vbg\") pod \"8041aee9-d4f6-41e3-8309-b298131dbf49\" (UID: \"8041aee9-d4f6-41e3-8309-b298131dbf49\") " Oct 31 04:31:49.534740 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:49.534715 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8041aee9-d4f6-41e3-8309-b298131dbf49-maas-api-openshift-ca" pod "8041aee9-d4f6-41e3-8309-b298131dbf49" (UID: "8041aee9-d4f6-41e3-8309-b298131dbf49"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:31:49.535156 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:49.535131 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8041aee9-d4f6-41e3-8309-b298131dbf49-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:31:49.536486 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:49.536463 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8041aee9-d4f6-41e3-8309-b298131dbf49-maas-api-xks-ca" pod "8041aee9-d4f6-41e3-8309-b298131dbf49" (UID: "8041aee9-d4f6-41e3-8309-b298131dbf49"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:31:49.536674 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:49.536650 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8041aee9-d4f6-41e3-8309-b298131dbf49-kube-api-access-k2vbg" pod "8041aee9-d4f6-41e3-8309-b298131dbf49" (UID: "8041aee9-d4f6-41e3-8309-b298131dbf49"). InnerVolumeSpecName "kube-api-access-k2vbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:31:49.636254 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:49.636221 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8041aee9-d4f6-41e3-8309-b298131dbf49-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:31:49.636254 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:49.636254 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2vbg\" (UniqueName: \"kubernetes.io/projected/8041aee9-d4f6-41e3-8309-b298131dbf49-kube-api-access-k2vbg\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:31:50.131692 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:50.131655 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s" event={"ID":"9eb20738-52aa-4cc7-990a-ddd8c704e683","Type":"ContainerStarted","Data":"2fd2b448c613cc7efef4751099580e39ad6c7484d89578a86c30a80cf22ba576"} Oct 31 04:31:50.133410 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:50.133377 5019 generic.go:356] "Generic (PLEG): container finished" podID="80707894-d35d-43c3-8bbb-c443f7e9a4e2" containerID="2a5379848e94cf984747f3d18aeaa5990a6186e519c125b63c7256f4b87726ac" exitCode=0 Oct 31 04:31:50.133608 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:50.133456 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81af36e3-qnf9f" event={"ID":"80707894-d35d-43c3-8bbb-c443f7e9a4e2","Type":"ContainerDied","Data":"2a5379848e94cf984747f3d18aeaa5990a6186e519c125b63c7256f4b87726ac"} Oct 31 04:31:50.134836 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:50.134816 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85913ce3-mjwbv" event={"ID":"8041aee9-d4f6-41e3-8309-b298131dbf49","Type":"ContainerDied","Data":"31995a40d3be1fee8d3df1b1c365787f71466111c3748049b3d324991010ebda"} Oct 31 04:31:50.134957 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:50.134851 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31995a40d3be1fee8d3df1b1c365787f71466111c3748049b3d324991010ebda" Oct 31 04:31:50.134957 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:50.134824 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85913ce3-mjwbv" Oct 31 04:31:50.890355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:50.888678 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s" Oct 31 04:31:50.894353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:50.892423 5019 patch_prober.go:29] interesting pod/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 31 04:31:50.894353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:50.892476 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s" podUID="9eb20738-52aa-4cc7-990a-ddd8c704e683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 31 04:31:51.104071 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:51.103283 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-e37d27-6ddbbfbc47-gb52r"] Oct 31 04:31:51.123443 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:51.123403 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-e37d27-c8547cdf9-v856h"] Oct 31 04:31:51.554503 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:51.554479 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81af36e3-qnf9f" Oct 31 04:31:51.566957 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:51.566932 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80707894-d35d-43c3-8bbb-c443f7e9a4e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80707894-d35d-43c3-8bbb-c443f7e9a4e2-maas-api-openshift-ca\") pod \"80707894-d35d-43c3-8bbb-c443f7e9a4e2\" (UID: \"80707894-d35d-43c3-8bbb-c443f7e9a4e2\") " Oct 31 04:31:51.567091 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:51.567078 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80707894-d35d-43c3-8bbb-c443f7e9a4e2-kube-api-access-gblq8\" (UniqueName: \"kubernetes.io/projected/80707894-d35d-43c3-8bbb-c443f7e9a4e2-kube-api-access-gblq8\") pod \"80707894-d35d-43c3-8bbb-c443f7e9a4e2\" (UID: \"80707894-d35d-43c3-8bbb-c443f7e9a4e2\") " Oct 31 04:31:51.567206 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:51.567193 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80707894-d35d-43c3-8bbb-c443f7e9a4e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80707894-d35d-43c3-8bbb-c443f7e9a4e2-maas-api-xks-ca\") pod \"80707894-d35d-43c3-8bbb-c443f7e9a4e2\" (UID: \"80707894-d35d-43c3-8bbb-c443f7e9a4e2\") " Oct 31 04:31:51.567325 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:51.567301 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80707894-d35d-43c3-8bbb-c443f7e9a4e2-maas-api-openshift-ca" pod "80707894-d35d-43c3-8bbb-c443f7e9a4e2" (UID: "80707894-d35d-43c3-8bbb-c443f7e9a4e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:31:51.567557 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:51.567535 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80707894-d35d-43c3-8bbb-c443f7e9a4e2-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:31:51.576460 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:51.569430 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80707894-d35d-43c3-8bbb-c443f7e9a4e2-kube-api-access-gblq8" pod "80707894-d35d-43c3-8bbb-c443f7e9a4e2" (UID: "80707894-d35d-43c3-8bbb-c443f7e9a4e2"). InnerVolumeSpecName "kube-api-access-gblq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:31:51.576460 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:51.575406 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80707894-d35d-43c3-8bbb-c443f7e9a4e2-maas-api-xks-ca" pod "80707894-d35d-43c3-8bbb-c443f7e9a4e2" (UID: "80707894-d35d-43c3-8bbb-c443f7e9a4e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:31:51.668905 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:51.668874 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gblq8\" (UniqueName: \"kubernetes.io/projected/80707894-d35d-43c3-8bbb-c443f7e9a4e2-kube-api-access-gblq8\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:31:51.668905 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:51.668903 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80707894-d35d-43c3-8bbb-c443f7e9a4e2-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:31:51.897885 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:51.897792 5019 patch_prober.go:29] interesting pod/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 31 04:31:51.898142 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:51.898112 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s" podUID="9eb20738-52aa-4cc7-990a-ddd8c704e683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 31 04:31:52.152239 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:52.152133 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81af36e3-qnf9f" event={"ID":"80707894-d35d-43c3-8bbb-c443f7e9a4e2","Type":"ContainerDied","Data":"db49cb52196a49e8a472d34a4e998e9bce33b3b3c4feccaf470dec871cc86581"} Oct 31 04:31:52.154961 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:52.153401 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db49cb52196a49e8a472d34a4e998e9bce33b3b3c4feccaf470dec871cc86581" Oct 31 04:31:52.154961 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:52.153503 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81af36e3-qnf9f" Oct 31 04:31:52.224032 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:52.222394 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d1d441-669694c87-fhkbx" podUID="52e6c267-5637-4fa5-9600-47b905bf75ee" containerName="payload-pre-processing" containerID="cri-o://61c7da3fb92f673ff6b9332476bb7321bde96e0476b6509ff058a44caee7e765" gracePeriod=25 Oct 31 04:31:52.224032 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:52.223389 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-d1d441-588dbc6899-4rnfz" podUID="63386d30-7d65-4c86-9362-0ee7b8143bb1" containerName="payload-processing" containerID="cri-o://dfbccaa07cd67f232fb76ed4abf8d821cedb480b743cf070f3c3073e8dcb3c19" gracePeriod=25 Oct 31 04:31:52.423806 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:31:52.423719 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:31:52.893358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:52.889677 5019 patch_prober.go:29] interesting pod/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 31 04:31:52.893358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:52.889733 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s" podUID="9eb20738-52aa-4cc7-990a-ddd8c704e683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 31 04:31:53.890357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:53.889414 5019 patch_prober.go:29] interesting pod/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 31 04:31:53.890357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:53.889471 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s" podUID="9eb20738-52aa-4cc7-990a-ddd8c704e683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 31 04:31:54.083650 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:54.082298 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97790921-ljcjc"] Oct 31 04:31:54.137852 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:54.137815 5019 cpu_manager.go:414] "containerMap: removing container" podUID="8041aee9-d4f6-41e3-8309-b298131dbf49" containerName="revoke-keys" Oct 31 04:31:54.137852 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:54.137847 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8041aee9-d4f6-41e3-8309-b298131dbf49" containerName="revoke-keys" Oct 31 04:31:54.138109 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:54.137904 5019 cpu_manager.go:414] "containerMap: removing container" podUID="80707894-d35d-43c3-8bbb-c443f7e9a4e2" containerName="revoke-keys" Oct 31 04:31:54.138109 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:54.137912 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80707894-d35d-43c3-8bbb-c443f7e9a4e2" containerName="revoke-keys" Oct 31 04:31:54.138310 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:54.138293 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="8041aee9-d4f6-41e3-8309-b298131dbf49" containerName="revoke-keys" Oct 31 04:31:54.138382 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:54.138346 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="80707894-d35d-43c3-8bbb-c443f7e9a4e2" containerName="revoke-keys" Oct 31 04:31:54.153780 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:54.153714 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97790921-ljcjc"] Oct 31 04:31:54.153896 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:54.153818 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97790921-ljcjc" Oct 31 04:31:54.168598 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:54.168571 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:31:54.210359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:54.204460 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:31:54.210359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:54.204528 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:31:54.210359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:54.204630 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:31:54.250311 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:54.250287 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a912a29d-b513-4355-93b3-fee0059c902d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97790921-ljcjc\" (UID: \"a912a29d-b513-4355-93b3-fee0059c902d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97790921-ljcjc" Oct 31 04:31:54.250455 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:54.250320 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54wxw\" (UniqueName: \"kubernetes.io/projected/a912a29d-b513-4355-93b3-fee0059c902d-kube-api-access-54wxw\") pod \"maas-api-revoke-sub-97790921-ljcjc\" (UID: \"a912a29d-b513-4355-93b3-fee0059c902d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97790921-ljcjc" Oct 31 04:31:54.250509 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:54.250487 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a912a29d-b513-4355-93b3-fee0059c902d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97790921-ljcjc\" (UID: \"a912a29d-b513-4355-93b3-fee0059c902d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97790921-ljcjc" Oct 31 04:31:54.351255 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:54.351228 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a912a29d-b513-4355-93b3-fee0059c902d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97790921-ljcjc\" (UID: \"a912a29d-b513-4355-93b3-fee0059c902d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97790921-ljcjc" Oct 31 04:31:54.351423 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:54.351264 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54wxw\" (UniqueName: \"kubernetes.io/projected/a912a29d-b513-4355-93b3-fee0059c902d-kube-api-access-54wxw\") pod \"maas-api-revoke-sub-97790921-ljcjc\" (UID: \"a912a29d-b513-4355-93b3-fee0059c902d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97790921-ljcjc" Oct 31 04:31:54.351423 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:54.351364 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a912a29d-b513-4355-93b3-fee0059c902d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97790921-ljcjc\" (UID: \"a912a29d-b513-4355-93b3-fee0059c902d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97790921-ljcjc" Oct 31 04:31:54.351932 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:54.351904 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a912a29d-b513-4355-93b3-fee0059c902d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97790921-ljcjc\" (UID: \"a912a29d-b513-4355-93b3-fee0059c902d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97790921-ljcjc" Oct 31 04:31:54.353680 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:54.353659 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a912a29d-b513-4355-93b3-fee0059c902d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97790921-ljcjc\" (UID: \"a912a29d-b513-4355-93b3-fee0059c902d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97790921-ljcjc" Oct 31 04:31:54.407881 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:54.407823 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54wxw\" (UniqueName: \"kubernetes.io/projected/a912a29d-b513-4355-93b3-fee0059c902d-kube-api-access-54wxw\") pod \"maas-api-revoke-sub-97790921-ljcjc\" (UID: \"a912a29d-b513-4355-93b3-fee0059c902d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97790921-ljcjc" Oct 31 04:31:54.516005 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:54.515962 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97790921-ljcjc" Oct 31 04:31:54.838263 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:31:54.838223 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda912a29d_b513_4355_93b3_fee0059c902d.slice/crio-aa9979ad4dfb3e010a33f76e56a6df315d23db213aea38a81e915f9d28cb4a82 WatchSource:0}: Error finding container aa9979ad4dfb3e010a33f76e56a6df315d23db213aea38a81e915f9d28cb4a82: Status 404 returned error can't find the container with id aa9979ad4dfb3e010a33f76e56a6df315d23db213aea38a81e915f9d28cb4a82 Oct 31 04:31:54.838909 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:54.838881 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97790921-ljcjc"] Oct 31 04:31:54.889143 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:54.888735 5019 patch_prober.go:29] interesting pod/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 31 04:31:54.889143 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:54.888791 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s" podUID="9eb20738-52aa-4cc7-990a-ddd8c704e683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 31 04:31:55.207730 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.207694 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97790921-ljcjc" event={"ID":"a912a29d-b513-4355-93b3-fee0059c902d","Type":"ContainerStarted","Data":"aa9979ad4dfb3e010a33f76e56a6df315d23db213aea38a81e915f9d28cb4a82"} Oct 31 04:31:55.616738 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.616698 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc"] Oct 31 04:31:55.636757 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.636676 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj"] Oct 31 04:31:55.637504 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.637475 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc" Oct 31 04:31:55.644109 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.644058 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j"] Oct 31 04:31:55.649409 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.646993 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj" Oct 31 04:31:55.652837 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.652573 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc"] Oct 31 04:31:55.652986 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.652874 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j" Oct 31 04:31:55.657143 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.656065 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj"] Oct 31 04:31:55.662568 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.662543 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j"] Oct 31 04:31:55.662721 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.662703 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-c4006d-openshift-default-dockercfg-h2s7k\"" Oct 31 04:31:55.674060 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.673753 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-credential-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj\" (UID: \"ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj" Oct 31 04:31:55.674060 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.673852 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c071116-bc3c-4d68-9e80-ec887ee96be7-istiod-ca-cert\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc\" (UID: \"0c071116-bc3c-4d68-9e80-ec887ee96be7\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc" Oct 31 04:31:55.674060 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.673895 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-istio-podinfo\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj\" (UID: \"ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj" Oct 31 04:31:55.674060 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.673935 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c071116-bc3c-4d68-9e80-ec887ee96be7-workload-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc\" (UID: \"0c071116-bc3c-4d68-9e80-ec887ee96be7\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc" Oct 31 04:31:55.674060 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.673969 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c071116-bc3c-4d68-9e80-ec887ee96be7-istio-podinfo\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc\" (UID: \"0c071116-bc3c-4d68-9e80-ec887ee96be7\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc" Oct 31 04:31:55.674060 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.674033 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c071116-bc3c-4d68-9e80-ec887ee96be7-istio-token\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc\" (UID: \"0c071116-bc3c-4d68-9e80-ec887ee96be7\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc" Oct 31 04:31:55.674761 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.674085 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-workload-certs\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj\" (UID: \"ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj" Oct 31 04:31:55.674761 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.674128 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-istiod-ca-cert\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj\" (UID: \"ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj" Oct 31 04:31:55.674761 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.674167 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvnq2\" (UniqueName: \"kubernetes.io/projected/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-kube-api-access-xvnq2\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj\" (UID: \"ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj" Oct 31 04:31:55.674761 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.674190 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkdhh\" (UniqueName: \"kubernetes.io/projected/0c071116-bc3c-4d68-9e80-ec887ee96be7-kube-api-access-bkdhh\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc\" (UID: \"0c071116-bc3c-4d68-9e80-ec887ee96be7\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc" Oct 31 04:31:55.674761 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.674219 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c071116-bc3c-4d68-9e80-ec887ee96be7-workload-certs\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc\" (UID: \"0c071116-bc3c-4d68-9e80-ec887ee96be7\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc" Oct 31 04:31:55.674761 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.674249 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c071116-bc3c-4d68-9e80-ec887ee96be7-istio-data\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc\" (UID: \"0c071116-bc3c-4d68-9e80-ec887ee96be7\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc" Oct 31 04:31:55.674761 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.674297 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-workload-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj\" (UID: \"ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj" Oct 31 04:31:55.676386 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.674321 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-istio-envoy\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj\" (UID: \"ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj" Oct 31 04:31:55.676490 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.676431 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c071116-bc3c-4d68-9e80-ec887ee96be7-credential-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc\" (UID: \"0c071116-bc3c-4d68-9e80-ec887ee96be7\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc" Oct 31 04:31:55.676490 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.676478 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-istio-token\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj\" (UID: \"ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj" Oct 31 04:31:55.676622 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.676508 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c071116-bc3c-4d68-9e80-ec887ee96be7-istio-envoy\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc\" (UID: \"0c071116-bc3c-4d68-9e80-ec887ee96be7\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc" Oct 31 04:31:55.676622 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.676547 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-istio-data\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj\" (UID: \"ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj" Oct 31 04:31:55.777750 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.777718 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-istio-token\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj\" (UID: \"ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj" Oct 31 04:31:55.777902 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.777757 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c071116-bc3c-4d68-9e80-ec887ee96be7-istio-envoy\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc\" (UID: \"0c071116-bc3c-4d68-9e80-ec887ee96be7\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc" Oct 31 04:31:55.777902 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.777783 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-istio-data\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj\" (UID: \"ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj" Oct 31 04:31:55.777902 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.777853 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e107852e-0f1f-4705-bf39-a676808806ef-credential-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j\" (UID: \"e107852e-0f1f-4705-bf39-a676808806ef\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j" Oct 31 04:31:55.778091 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.777910 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-credential-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj\" (UID: \"ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj" Oct 31 04:31:55.778091 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.777946 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e107852e-0f1f-4705-bf39-a676808806ef-workload-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j\" (UID: \"e107852e-0f1f-4705-bf39-a676808806ef\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j" Oct 31 04:31:55.778091 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.778039 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e107852e-0f1f-4705-bf39-a676808806ef-istiod-ca-cert\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j\" (UID: \"e107852e-0f1f-4705-bf39-a676808806ef\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j" Oct 31 04:31:55.778226 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.778142 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c071116-bc3c-4d68-9e80-ec887ee96be7-istiod-ca-cert\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc\" (UID: \"0c071116-bc3c-4d68-9e80-ec887ee96be7\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc" Oct 31 04:31:55.778226 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.778186 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-istio-podinfo\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj\" (UID: \"ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj" Oct 31 04:31:55.778352 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.778224 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c071116-bc3c-4d68-9e80-ec887ee96be7-workload-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc\" (UID: \"0c071116-bc3c-4d68-9e80-ec887ee96be7\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc" Oct 31 04:31:55.778352 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.778264 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-istio-data\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj\" (UID: \"ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj" Oct 31 04:31:55.778352 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.778264 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c071116-bc3c-4d68-9e80-ec887ee96be7-istio-podinfo\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc\" (UID: \"0c071116-bc3c-4d68-9e80-ec887ee96be7\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc" Oct 31 04:31:55.778352 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.778289 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-credential-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj\" (UID: \"ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj" Oct 31 04:31:55.778352 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.778320 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e107852e-0f1f-4705-bf39-a676808806ef-istio-podinfo\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j\" (UID: \"e107852e-0f1f-4705-bf39-a676808806ef\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j" Oct 31 04:31:55.778631 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.778383 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e107852e-0f1f-4705-bf39-a676808806ef-workload-certs\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j\" (UID: \"e107852e-0f1f-4705-bf39-a676808806ef\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j" Oct 31 04:31:55.778631 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.778422 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e107852e-0f1f-4705-bf39-a676808806ef-istio-envoy\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j\" (UID: \"e107852e-0f1f-4705-bf39-a676808806ef\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j" Oct 31 04:31:55.778631 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.778458 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c071116-bc3c-4d68-9e80-ec887ee96be7-istio-token\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc\" (UID: \"0c071116-bc3c-4d68-9e80-ec887ee96be7\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc" Oct 31 04:31:55.778631 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.778498 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e107852e-0f1f-4705-bf39-a676808806ef-istio-data\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j\" (UID: \"e107852e-0f1f-4705-bf39-a676808806ef\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j" Oct 31 04:31:55.778631 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.778531 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-workload-certs\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj\" (UID: \"ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj" Oct 31 04:31:55.778631 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.778572 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-istiod-ca-cert\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj\" (UID: \"ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj" Oct 31 04:31:55.778631 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.778604 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e107852e-0f1f-4705-bf39-a676808806ef-istio-token\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j\" (UID: \"e107852e-0f1f-4705-bf39-a676808806ef\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j" Oct 31 04:31:55.779018 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.778637 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvnq2\" (UniqueName: \"kubernetes.io/projected/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-kube-api-access-xvnq2\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj\" (UID: \"ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj" Oct 31 04:31:55.779018 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.778659 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkdhh\" (UniqueName: \"kubernetes.io/projected/0c071116-bc3c-4d68-9e80-ec887ee96be7-kube-api-access-bkdhh\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc\" (UID: \"0c071116-bc3c-4d68-9e80-ec887ee96be7\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc" Oct 31 04:31:55.779018 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.778685 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c071116-bc3c-4d68-9e80-ec887ee96be7-workload-certs\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc\" (UID: \"0c071116-bc3c-4d68-9e80-ec887ee96be7\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc" Oct 31 04:31:55.779018 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.778715 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c071116-bc3c-4d68-9e80-ec887ee96be7-istio-data\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc\" (UID: \"0c071116-bc3c-4d68-9e80-ec887ee96be7\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc" Oct 31 04:31:55.779018 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.778757 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gkt8\" (UniqueName: \"kubernetes.io/projected/e107852e-0f1f-4705-bf39-a676808806ef-kube-api-access-6gkt8\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j\" (UID: \"e107852e-0f1f-4705-bf39-a676808806ef\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j" Oct 31 04:31:55.779018 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.778786 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-workload-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj\" (UID: \"ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj" Oct 31 04:31:55.779018 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.778810 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-istio-envoy\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj\" (UID: \"ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj" Oct 31 04:31:55.779018 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.778863 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c071116-bc3c-4d68-9e80-ec887ee96be7-credential-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc\" (UID: \"0c071116-bc3c-4d68-9e80-ec887ee96be7\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc" Oct 31 04:31:55.779018 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.778923 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c071116-bc3c-4d68-9e80-ec887ee96be7-istiod-ca-cert\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc\" (UID: \"0c071116-bc3c-4d68-9e80-ec887ee96be7\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc" Oct 31 04:31:55.779648 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.779190 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c071116-bc3c-4d68-9e80-ec887ee96be7-credential-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc\" (UID: \"0c071116-bc3c-4d68-9e80-ec887ee96be7\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc" Oct 31 04:31:55.779648 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.779212 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c071116-bc3c-4d68-9e80-ec887ee96be7-workload-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc\" (UID: \"0c071116-bc3c-4d68-9e80-ec887ee96be7\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc" Oct 31 04:31:55.779648 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.779571 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c071116-bc3c-4d68-9e80-ec887ee96be7-workload-certs\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc\" (UID: \"0c071116-bc3c-4d68-9e80-ec887ee96be7\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc" Oct 31 04:31:55.779811 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.779743 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-workload-certs\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj\" (UID: \"ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj" Oct 31 04:31:55.780414 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.780308 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c071116-bc3c-4d68-9e80-ec887ee96be7-istio-data\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc\" (UID: \"0c071116-bc3c-4d68-9e80-ec887ee96be7\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc" Oct 31 04:31:55.780649 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.780616 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-workload-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj\" (UID: \"ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj" Oct 31 04:31:55.780649 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.780620 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-istiod-ca-cert\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj\" (UID: \"ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj" Oct 31 04:31:55.781220 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.781197 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c071116-bc3c-4d68-9e80-ec887ee96be7-istio-envoy\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc\" (UID: \"0c071116-bc3c-4d68-9e80-ec887ee96be7\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc" Oct 31 04:31:55.781463 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.781240 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c071116-bc3c-4d68-9e80-ec887ee96be7-istio-podinfo\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc\" (UID: \"0c071116-bc3c-4d68-9e80-ec887ee96be7\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc" Oct 31 04:31:55.782232 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.782213 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-istio-podinfo\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj\" (UID: \"ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj" Oct 31 04:31:55.782521 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.782497 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-istio-envoy\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj\" (UID: \"ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj" Oct 31 04:31:55.804485 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.804461 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-istio-token\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj\" (UID: \"ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj" Oct 31 04:31:55.805600 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.805577 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c071116-bc3c-4d68-9e80-ec887ee96be7-istio-token\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc\" (UID: \"0c071116-bc3c-4d68-9e80-ec887ee96be7\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc" Oct 31 04:31:55.806292 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.806263 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvnq2\" (UniqueName: \"kubernetes.io/projected/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-kube-api-access-xvnq2\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj\" (UID: \"ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj" Oct 31 04:31:55.806973 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.806951 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkdhh\" (UniqueName: \"kubernetes.io/projected/0c071116-bc3c-4d68-9e80-ec887ee96be7-kube-api-access-bkdhh\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc\" (UID: \"0c071116-bc3c-4d68-9e80-ec887ee96be7\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc" Oct 31 04:31:55.851222 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.851170 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-d1d441-588dbc6899-4rnfz" podUID="63386d30-7d65-4c86-9362-0ee7b8143bb1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 04:31:55.851370 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.851199 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d1d441-669694c87-fhkbx" podUID="52e6c267-5637-4fa5-9600-47b905bf75ee" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 04:31:55.880523 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.880497 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e107852e-0f1f-4705-bf39-a676808806ef-credential-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j\" (UID: \"e107852e-0f1f-4705-bf39-a676808806ef\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j" Oct 31 04:31:55.880648 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.880576 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e107852e-0f1f-4705-bf39-a676808806ef-workload-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j\" (UID: \"e107852e-0f1f-4705-bf39-a676808806ef\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j" Oct 31 04:31:55.880648 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.880601 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e107852e-0f1f-4705-bf39-a676808806ef-istiod-ca-cert\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j\" (UID: \"e107852e-0f1f-4705-bf39-a676808806ef\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j" Oct 31 04:31:55.880763 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.880673 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e107852e-0f1f-4705-bf39-a676808806ef-istio-podinfo\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j\" (UID: \"e107852e-0f1f-4705-bf39-a676808806ef\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j" Oct 31 04:31:55.880763 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.880709 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e107852e-0f1f-4705-bf39-a676808806ef-workload-certs\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j\" (UID: \"e107852e-0f1f-4705-bf39-a676808806ef\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j" Oct 31 04:31:55.880763 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.880748 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e107852e-0f1f-4705-bf39-a676808806ef-istio-envoy\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j\" (UID: \"e107852e-0f1f-4705-bf39-a676808806ef\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j" Oct 31 04:31:55.880902 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.880798 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e107852e-0f1f-4705-bf39-a676808806ef-istio-data\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j\" (UID: \"e107852e-0f1f-4705-bf39-a676808806ef\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j" Oct 31 04:31:55.880956 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.880921 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e107852e-0f1f-4705-bf39-a676808806ef-credential-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j\" (UID: \"e107852e-0f1f-4705-bf39-a676808806ef\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j" Oct 31 04:31:55.881015 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.880951 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e107852e-0f1f-4705-bf39-a676808806ef-workload-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j\" (UID: \"e107852e-0f1f-4705-bf39-a676808806ef\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j" Oct 31 04:31:55.881076 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.881038 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e107852e-0f1f-4705-bf39-a676808806ef-istio-token\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j\" (UID: \"e107852e-0f1f-4705-bf39-a676808806ef\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j" Oct 31 04:31:55.881140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.881111 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gkt8\" (UniqueName: \"kubernetes.io/projected/e107852e-0f1f-4705-bf39-a676808806ef-kube-api-access-6gkt8\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j\" (UID: \"e107852e-0f1f-4705-bf39-a676808806ef\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j" Oct 31 04:31:55.881203 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.881155 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e107852e-0f1f-4705-bf39-a676808806ef-istio-data\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j\" (UID: \"e107852e-0f1f-4705-bf39-a676808806ef\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j" Oct 31 04:31:55.881362 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.881319 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e107852e-0f1f-4705-bf39-a676808806ef-workload-certs\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j\" (UID: \"e107852e-0f1f-4705-bf39-a676808806ef\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j" Oct 31 04:31:55.881495 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.881444 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e107852e-0f1f-4705-bf39-a676808806ef-istiod-ca-cert\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j\" (UID: \"e107852e-0f1f-4705-bf39-a676808806ef\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j" Oct 31 04:31:55.883400 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.883373 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e107852e-0f1f-4705-bf39-a676808806ef-istio-envoy\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j\" (UID: \"e107852e-0f1f-4705-bf39-a676808806ef\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j" Oct 31 04:31:55.885037 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.885017 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e107852e-0f1f-4705-bf39-a676808806ef-istio-podinfo\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j\" (UID: \"e107852e-0f1f-4705-bf39-a676808806ef\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j" Oct 31 04:31:55.891769 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.891717 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s" Oct 31 04:31:55.892521 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.892465 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s" Oct 31 04:31:55.893640 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.893612 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s" Oct 31 04:31:55.903889 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.903861 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e107852e-0f1f-4705-bf39-a676808806ef-istio-token\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j\" (UID: \"e107852e-0f1f-4705-bf39-a676808806ef\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j" Oct 31 04:31:55.903982 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.903921 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gkt8\" (UniqueName: \"kubernetes.io/projected/e107852e-0f1f-4705-bf39-a676808806ef-kube-api-access-6gkt8\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j\" (UID: \"e107852e-0f1f-4705-bf39-a676808806ef\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j" Oct 31 04:31:55.988619 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.988584 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj" Oct 31 04:31:55.989275 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:55.989250 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc" Oct 31 04:31:56.016359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:56.011688 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j" Oct 31 04:31:56.104355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:56.103861 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-e37d27-6ddbbfbc47-gb52r" podUID="82b999d8-91cd-4021-8fc0-c2e61ae1a8f2" containerName="payload-processing" containerID="cri-o://c12841823a893abfb6ad1afaed67c1e7c788229cbea72e588213881dcb69cced" gracePeriod=25 Oct 31 04:31:56.145669 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:56.142612 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e37d27-c8547cdf9-v856h" podUID="c0570273-666f-4052-9713-00813a62c291" containerName="payload-pre-processing" containerID="cri-o://4eb06d0f10f1623e369e8da90eca8e46ffbbcbae5b5cdefc23f82731378df35a" gracePeriod=25 Oct 31 04:31:56.222587 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:56.222026 5019 generic.go:356] "Generic (PLEG): container finished" podID="a912a29d-b513-4355-93b3-fee0059c902d" containerID="0e87eb6578ed42f510ce260bc026853df54829d892a91be664777009bfc0599f" exitCode=0 Oct 31 04:31:56.223025 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:56.222961 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97790921-ljcjc" event={"ID":"a912a29d-b513-4355-93b3-fee0059c902d","Type":"ContainerDied","Data":"0e87eb6578ed42f510ce260bc026853df54829d892a91be664777009bfc0599f"} Oct 31 04:31:57.426759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:57.308031 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:31:57.426759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:57.308349 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:31:57.426759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:57.308462 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:31:57.597699 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:57.594874 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:31:57.598218 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:57.598176 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:31:57.598502 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:57.598478 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:31:57.969353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:57.960154 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj"] Oct 31 04:31:57.991431 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:57.972904 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:31:57.995083 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:57.995032 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" podUID="8e2c7041-ad52-49f4-8b55-d16691f4938f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": context deadline exceeded" Oct 31 04:31:58.012918 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:58.008225 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:31:58.014923 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:58.014841 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 31 04:31:58.059354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:58.034090 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:31:58.160381 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:58.151765 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:31:58.160381 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:58.151893 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:31:58.160381 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:58.066254 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" podUID="643433d8-418e-4102-8bc3-cd544932614a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:31:58.162002 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:58.083405 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" podUID="643433d8-418e-4102-8bc3-cd544932614a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:31:58.162002 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:58.161906 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj" event={"ID":"ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77","Type":"ContainerStarted","Data":"77c11ebe6bde388cf63dd955d18b08c9c5ba158fe741db3a8a6e1dd4a3fdf04b"} Oct 31 04:31:58.165526 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:58.165179 5019 generic.go:356] "Generic (PLEG): container finished" podID="c0570273-666f-4052-9713-00813a62c291" containerID="4eb06d0f10f1623e369e8da90eca8e46ffbbcbae5b5cdefc23f82731378df35a" exitCode=0 Oct 31 04:31:58.165526 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:58.165434 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e37d27-c8547cdf9-v856h" event={"ID":"c0570273-666f-4052-9713-00813a62c291","Type":"ContainerDied","Data":"4eb06d0f10f1623e369e8da90eca8e46ffbbcbae5b5cdefc23f82731378df35a"} Oct 31 04:31:58.181417 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:58.181375 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-e37d27-6ddbbfbc47-gb52r" Oct 31 04:31:58.205269 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:58.205201 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-e37d27-6ddbbfbc47-gb52r" event={"ID":"82b999d8-91cd-4021-8fc0-c2e61ae1a8f2","Type":"ContainerDied","Data":"c12841823a893abfb6ad1afaed67c1e7c788229cbea72e588213881dcb69cced"} Oct 31 04:31:58.214497 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:58.214466 5019 scope.go:122] "RemoveContainer" containerID="c12841823a893abfb6ad1afaed67c1e7c788229cbea72e588213881dcb69cced" Oct 31 04:31:58.214982 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:58.205147 5019 generic.go:356] "Generic (PLEG): container finished" podID="82b999d8-91cd-4021-8fc0-c2e61ae1a8f2" containerID="c12841823a893abfb6ad1afaed67c1e7c788229cbea72e588213881dcb69cced" exitCode=0 Oct 31 04:31:58.230448 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:58.230418 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:31:58.344354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:58.324810 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82b999d8-91cd-4021-8fc0-c2e61ae1a8f2-kube-api-access-xvjdn\" (UniqueName: \"kubernetes.io/projected/82b999d8-91cd-4021-8fc0-c2e61ae1a8f2-kube-api-access-xvjdn\") pod \"82b999d8-91cd-4021-8fc0-c2e61ae1a8f2\" (UID: \"82b999d8-91cd-4021-8fc0-c2e61ae1a8f2\") " Oct 31 04:31:58.473612 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:58.473580 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82b999d8-91cd-4021-8fc0-c2e61ae1a8f2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82b999d8-91cd-4021-8fc0-c2e61ae1a8f2-plugins-config-volume\") pod \"82b999d8-91cd-4021-8fc0-c2e61ae1a8f2\" (UID: \"82b999d8-91cd-4021-8fc0-c2e61ae1a8f2\") " Oct 31 04:31:58.474233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:58.355663 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82b999d8-91cd-4021-8fc0-c2e61ae1a8f2-kube-api-access-xvjdn" pod "82b999d8-91cd-4021-8fc0-c2e61ae1a8f2" (UID: "82b999d8-91cd-4021-8fc0-c2e61ae1a8f2"). InnerVolumeSpecName "kube-api-access-xvjdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:31:58.552497 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:58.545435 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvjdn\" (UniqueName: \"kubernetes.io/projected/82b999d8-91cd-4021-8fc0-c2e61ae1a8f2-kube-api-access-xvjdn\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:31:58.556649 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:58.556609 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc" event={"ID":"0c071116-bc3c-4d68-9e80-ec887ee96be7","Type":"ContainerStarted","Data":"429cb2d026da0878527670dec00aaef2fa89cae98201ecb03767c1c5bcd2e56d"} Oct 31 04:31:58.642368 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:58.616143 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc"] Oct 31 04:31:58.642368 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:58.601534 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82b999d8-91cd-4021-8fc0-c2e61ae1a8f2-plugins-config-volume" pod "82b999d8-91cd-4021-8fc0-c2e61ae1a8f2" (UID: "82b999d8-91cd-4021-8fc0-c2e61ae1a8f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:31:58.642368 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:58.592549 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" podUID="b4ced3ab-41e4-4c54-93eb-95eddaae992d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:31:58.673353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:58.666704 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82b999d8-91cd-4021-8fc0-c2e61ae1a8f2-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:31:58.673353 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:31:58.666836 5019 logging.go:55] [core] [Channel #2526 SubChannel #2527]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.203:50052", ServerName: "10.132.0.203:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.203:50052: connect: connection refused" Oct 31 04:31:58.673353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:58.668789 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j"] Oct 31 04:31:59.025408 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:59.017488 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:31:59.025408 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:59.018242 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" Oct 31 04:31:59.137801 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:59.136864 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" podUID="8e2c7041-ad52-49f4-8b55-d16691f4938f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:31:59.257828 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:59.233714 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:31:59.257828 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:59.234208 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" Oct 31 04:31:59.647403 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:59.647321 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e37d27-c8547cdf9-v856h" podUID="c0570273-666f-4052-9713-00813a62c291" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.203:50052\" within 1s: context deadline exceeded" Oct 31 04:31:59.744634 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:59.733304 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-e37d27-6ddbbfbc47-gb52r" Oct 31 04:31:59.795023 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:59.784044 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" Oct 31 04:31:59.795023 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:59.784103 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" Oct 31 04:31:59.795023 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:59.784122 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-e37d27-6ddbbfbc47-gb52r" event={"ID":"82b999d8-91cd-4021-8fc0-c2e61ae1a8f2","Type":"ContainerDied","Data":"2a34e926d7165ae4f66da12dc451aef17a465accc91600a988e2ac3c54f2339d"} Oct 31 04:31:59.795023 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:59.784160 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc" event={"ID":"0c071116-bc3c-4d68-9e80-ec887ee96be7","Type":"ContainerStarted","Data":"134b6545aacd863720d0a6b9fcb2299399d73d2db685948a9ecf3c05102c3e3e"} Oct 31 04:31:59.795023 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:59.784186 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j" event={"ID":"e107852e-0f1f-4705-bf39-a676808806ef","Type":"ContainerStarted","Data":"e809418e6d2904b1514bfb9be8e45e2c5dddae293bb7d30fa1ea0a8d11967ced"} Oct 31 04:31:59.828068 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:59.827996 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc" podStartSLOduration=4.8279759559999995 podStartE2EDuration="4.827975956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:31:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:31:59.817418817 +0000 UTC m=+1674.573756856" watchObservedRunningTime="2026-10-31 04:31:59.827975956 +0000 UTC m=+1674.584313995" Oct 31 04:31:59.847756 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:59.847406 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e37d27-c8547cdf9-v856h" Oct 31 04:31:59.878909 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:59.878824 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97790921-ljcjc" Oct 31 04:31:59.884727 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:59.884629 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj" podStartSLOduration=4.884611919 podStartE2EDuration="4.884611919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:31:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:31:59.869901843 +0000 UTC m=+1674.626239884" watchObservedRunningTime="2026-10-31 04:31:59.884611919 +0000 UTC m=+1674.640949958" Oct 31 04:31:59.933321 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:59.933290 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a912a29d-b513-4355-93b3-fee0059c902d-kube-api-access-54wxw\" (UniqueName: \"kubernetes.io/projected/a912a29d-b513-4355-93b3-fee0059c902d-kube-api-access-54wxw\") pod \"a912a29d-b513-4355-93b3-fee0059c902d\" (UID: \"a912a29d-b513-4355-93b3-fee0059c902d\") " Oct 31 04:31:59.933795 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:59.933767 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0570273-666f-4052-9713-00813a62c291-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0570273-666f-4052-9713-00813a62c291-plugins-config-volume\") pod \"c0570273-666f-4052-9713-00813a62c291\" (UID: \"c0570273-666f-4052-9713-00813a62c291\") " Oct 31 04:31:59.933923 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:59.933879 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0570273-666f-4052-9713-00813a62c291-kube-api-access-cghxj\" (UniqueName: \"kubernetes.io/projected/c0570273-666f-4052-9713-00813a62c291-kube-api-access-cghxj\") pod \"c0570273-666f-4052-9713-00813a62c291\" (UID: \"c0570273-666f-4052-9713-00813a62c291\") " Oct 31 04:31:59.934155 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:59.934134 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0570273-666f-4052-9713-00813a62c291-plugins-config-volume" pod "c0570273-666f-4052-9713-00813a62c291" (UID: "c0570273-666f-4052-9713-00813a62c291"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:31:59.934298 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:59.934276 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a912a29d-b513-4355-93b3-fee0059c902d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a912a29d-b513-4355-93b3-fee0059c902d-maas-api-openshift-ca\") pod \"a912a29d-b513-4355-93b3-fee0059c902d\" (UID: \"a912a29d-b513-4355-93b3-fee0059c902d\") " Oct 31 04:31:59.934414 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:59.934396 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a912a29d-b513-4355-93b3-fee0059c902d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a912a29d-b513-4355-93b3-fee0059c902d-maas-api-xks-ca\") pod \"a912a29d-b513-4355-93b3-fee0059c902d\" (UID: \"a912a29d-b513-4355-93b3-fee0059c902d\") " Oct 31 04:31:59.934970 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:59.934946 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a912a29d-b513-4355-93b3-fee0059c902d-maas-api-openshift-ca" pod "a912a29d-b513-4355-93b3-fee0059c902d" (UID: "a912a29d-b513-4355-93b3-fee0059c902d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:31:59.935895 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:59.935875 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0570273-666f-4052-9713-00813a62c291-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:31:59.936041 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:59.936024 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a912a29d-b513-4355-93b3-fee0059c902d-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:31:59.936445 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:59.936418 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a912a29d-b513-4355-93b3-fee0059c902d-kube-api-access-54wxw" pod "a912a29d-b513-4355-93b3-fee0059c902d" (UID: "a912a29d-b513-4355-93b3-fee0059c902d"). InnerVolumeSpecName "kube-api-access-54wxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:31:59.938043 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:59.937927 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0570273-666f-4052-9713-00813a62c291-kube-api-access-cghxj" pod "c0570273-666f-4052-9713-00813a62c291" (UID: "c0570273-666f-4052-9713-00813a62c291"). InnerVolumeSpecName "kube-api-access-cghxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:31:59.938425 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:59.938394 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a912a29d-b513-4355-93b3-fee0059c902d-maas-api-xks-ca" pod "a912a29d-b513-4355-93b3-fee0059c902d" (UID: "a912a29d-b513-4355-93b3-fee0059c902d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:31:59.999024 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:59.991463 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj" Oct 31 04:31:59.999024 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:31:59.991560 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc" Oct 31 04:32:00.023281 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:00.023234 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-e37d27-6ddbbfbc47-gb52r"] Oct 31 04:32:00.030500 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:00.030474 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-e37d27-6ddbbfbc47-gb52r"] Oct 31 04:32:00.037607 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:00.037539 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cghxj\" (UniqueName: \"kubernetes.io/projected/c0570273-666f-4052-9713-00813a62c291-kube-api-access-cghxj\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:32:00.037607 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:00.037568 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a912a29d-b513-4355-93b3-fee0059c902d-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:32:00.037607 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:00.037586 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54wxw\" (UniqueName: \"kubernetes.io/projected/a912a29d-b513-4355-93b3-fee0059c902d-kube-api-access-54wxw\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:32:00.873428 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:00.873381 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj" event={"ID":"ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77","Type":"ContainerStarted","Data":"7eb4cea754b9697f55742b13bf2a141ed71dd6ee7b2237ba73c698a9f0452a45"} Oct 31 04:32:00.880366 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:00.877476 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e37d27-c8547cdf9-v856h" event={"ID":"c0570273-666f-4052-9713-00813a62c291","Type":"ContainerDied","Data":"46aceb98e784b5db5dd8349a7fe272a27d318f467959af46bce38acc01f8c080"} Oct 31 04:32:00.880366 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:00.877549 5019 scope.go:122] "RemoveContainer" containerID="4eb06d0f10f1623e369e8da90eca8e46ffbbcbae5b5cdefc23f82731378df35a" Oct 31 04:32:00.880366 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:00.877670 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e37d27-c8547cdf9-v856h" Oct 31 04:32:00.889256 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:00.889228 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97790921-ljcjc" event={"ID":"a912a29d-b513-4355-93b3-fee0059c902d","Type":"ContainerDied","Data":"aa9979ad4dfb3e010a33f76e56a6df315d23db213aea38a81e915f9d28cb4a82"} Oct 31 04:32:00.889592 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:00.889567 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa9979ad4dfb3e010a33f76e56a6df315d23db213aea38a81e915f9d28cb4a82" Oct 31 04:32:00.890376 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:00.890360 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97790921-ljcjc" Oct 31 04:32:00.955725 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:00.955684 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j" event={"ID":"e107852e-0f1f-4705-bf39-a676808806ef","Type":"ContainerStarted","Data":"b988f154cd8443fafa6814fc2771b9b20a9b97df6edae4aaa1d7b3b4e25f5950"} Oct 31 04:32:01.038277 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:01.038213 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 31 04:32:01.038604 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:01.038564 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j" podUID="e107852e-0f1f-4705-bf39-a676808806ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 31 04:32:01.039450 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:01.039421 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Oct 31 04:32:01.039646 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:01.039622 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc" podUID="0c071116-bc3c-4d68-9e80-ec887ee96be7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Oct 31 04:32:01.088548 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:01.088482 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Oct 31 04:32:01.112362 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:01.108599 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj" podUID="ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Oct 31 04:32:01.112362 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:01.108525 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j" Oct 31 04:32:01.608358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:01.608021 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82b999d8-91cd-4021-8fc0-c2e61ae1a8f2" path="/var/lib/kubelet/pods/82b999d8-91cd-4021-8fc0-c2e61ae1a8f2/volumes" Oct 31 04:32:01.611365 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:01.608576 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-e37d27-c8547cdf9-v856h"] Oct 31 04:32:01.690355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:01.689508 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-e37d27-c8547cdf9-v856h"] Oct 31 04:32:01.757402 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:01.756389 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j" podStartSLOduration=6.756367823 podStartE2EDuration="6.756367823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:31:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:32:01.126026375 +0000 UTC m=+1675.882364451" watchObservedRunningTime="2026-10-31 04:32:01.756367823 +0000 UTC m=+1676.512705863" Oct 31 04:32:01.763354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:01.760100 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 04:32:01.993354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:01.992796 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Oct 31 04:32:01.993354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:01.992853 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj" podUID="ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Oct 31 04:32:02.001374 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:02.001046 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Oct 31 04:32:02.001374 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:02.001097 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc" podUID="0c071116-bc3c-4d68-9e80-ec887ee96be7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Oct 31 04:32:02.017731 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:02.017644 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 31 04:32:02.017731 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:02.017692 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j" podUID="e107852e-0f1f-4705-bf39-a676808806ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 31 04:32:02.994753 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:02.994718 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Oct 31 04:32:02.995304 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:02.995272 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc" podUID="0c071116-bc3c-4d68-9e80-ec887ee96be7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Oct 31 04:32:02.995954 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:02.995916 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Oct 31 04:32:02.996112 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:02.996091 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj" podUID="ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Oct 31 04:32:03.019361 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:03.018693 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 31 04:32:03.019361 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:03.018744 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j" podUID="e107852e-0f1f-4705-bf39-a676808806ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 31 04:32:03.472400 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:03.455624 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0570273-666f-4052-9713-00813a62c291" path="/var/lib/kubelet/pods/c0570273-666f-4052-9713-00813a62c291/volumes" Oct 31 04:32:03.538363 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:03.537127 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2023e453-9kxs6"] Oct 31 04:32:03.538363 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:03.538096 5019 cpu_manager.go:414] "containerMap: removing container" podUID="a912a29d-b513-4355-93b3-fee0059c902d" containerName="revoke-keys" Oct 31 04:32:03.538363 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:03.538117 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a912a29d-b513-4355-93b3-fee0059c902d" containerName="revoke-keys" Oct 31 04:32:03.538363 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:03.538136 5019 cpu_manager.go:414] "containerMap: removing container" podUID="82b999d8-91cd-4021-8fc0-c2e61ae1a8f2" containerName="payload-processing" Oct 31 04:32:03.538363 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:03.538145 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82b999d8-91cd-4021-8fc0-c2e61ae1a8f2" containerName="payload-processing" Oct 31 04:32:03.538363 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:03.538177 5019 cpu_manager.go:414] "containerMap: removing container" podUID="c0570273-666f-4052-9713-00813a62c291" containerName="payload-pre-processing" Oct 31 04:32:03.538363 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:03.538184 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0570273-666f-4052-9713-00813a62c291" containerName="payload-pre-processing" Oct 31 04:32:03.538877 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:03.538386 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="a912a29d-b513-4355-93b3-fee0059c902d" containerName="revoke-keys" Oct 31 04:32:03.538877 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:03.538400 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="82b999d8-91cd-4021-8fc0-c2e61ae1a8f2" containerName="payload-processing" Oct 31 04:32:03.562365 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:03.538416 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0570273-666f-4052-9713-00813a62c291" containerName="payload-pre-processing" Oct 31 04:32:03.668380 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:03.578958 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2023e453-9kxs6"] Oct 31 04:32:03.668380 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:03.579080 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2023e453-9kxs6" Oct 31 04:32:03.668380 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:03.599649 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:32:03.668380 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:03.649819 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:32:03.668380 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:03.649922 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:32:03.728895 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:03.709123 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:32:03.728895 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:03.711699 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d1e5631-5416-4a0a-ab66-716b75268f1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2023e453-9kxs6\" (UID: \"4d1e5631-5416-4a0a-ab66-716b75268f1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2023e453-9kxs6" Oct 31 04:32:03.728895 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:03.711739 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d1e5631-5416-4a0a-ab66-716b75268f1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2023e453-9kxs6\" (UID: \"4d1e5631-5416-4a0a-ab66-716b75268f1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2023e453-9kxs6" Oct 31 04:32:03.728895 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:03.711925 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frjgc\" (UniqueName: \"kubernetes.io/projected/4d1e5631-5416-4a0a-ab66-716b75268f1b-kube-api-access-frjgc\") pod \"maas-api-revoke-sub-2023e453-9kxs6\" (UID: \"4d1e5631-5416-4a0a-ab66-716b75268f1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2023e453-9kxs6" Oct 31 04:32:03.819596 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:03.819519 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d1e5631-5416-4a0a-ab66-716b75268f1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2023e453-9kxs6\" (UID: \"4d1e5631-5416-4a0a-ab66-716b75268f1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2023e453-9kxs6" Oct 31 04:32:03.819903 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:03.819571 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d1e5631-5416-4a0a-ab66-716b75268f1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2023e453-9kxs6\" (UID: \"4d1e5631-5416-4a0a-ab66-716b75268f1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2023e453-9kxs6" Oct 31 04:32:03.828375 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:03.820162 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frjgc\" (UniqueName: \"kubernetes.io/projected/4d1e5631-5416-4a0a-ab66-716b75268f1b-kube-api-access-frjgc\") pod \"maas-api-revoke-sub-2023e453-9kxs6\" (UID: \"4d1e5631-5416-4a0a-ab66-716b75268f1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2023e453-9kxs6" Oct 31 04:32:03.828375 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:03.821193 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d1e5631-5416-4a0a-ab66-716b75268f1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2023e453-9kxs6\" (UID: \"4d1e5631-5416-4a0a-ab66-716b75268f1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2023e453-9kxs6" Oct 31 04:32:03.867393 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:03.866760 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d1e5631-5416-4a0a-ab66-716b75268f1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2023e453-9kxs6\" (UID: \"4d1e5631-5416-4a0a-ab66-716b75268f1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2023e453-9kxs6" Oct 31 04:32:03.891367 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:03.886321 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frjgc\" (UniqueName: \"kubernetes.io/projected/4d1e5631-5416-4a0a-ab66-716b75268f1b-kube-api-access-frjgc\") pod \"maas-api-revoke-sub-2023e453-9kxs6\" (UID: \"4d1e5631-5416-4a0a-ab66-716b75268f1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2023e453-9kxs6" Oct 31 04:32:03.965494 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:03.965445 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" podUID="e0f4b8c2-14f4-4052-bf54-dd65bf59e413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:32:03.991809 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:03.991717 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Oct 31 04:32:03.992071 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:03.992033 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc" podUID="0c071116-bc3c-4d68-9e80-ec887ee96be7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Oct 31 04:32:03.992921 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:03.992885 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Oct 31 04:32:03.993102 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:03.993082 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj" podUID="ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Oct 31 04:32:04.014433 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:04.014407 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 31 04:32:04.015182 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:04.015156 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j" podUID="e107852e-0f1f-4705-bf39-a676808806ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 31 04:32:04.063585 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:04.063555 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2023e453-9kxs6" Oct 31 04:32:04.835414 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:04.820411 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 31 04:32:05.066478 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:05.050892 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 31 04:32:05.066478 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:05.050969 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j" podUID="e107852e-0f1f-4705-bf39-a676808806ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 31 04:32:05.066478 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:05.051984 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Oct 31 04:32:05.066478 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:05.052027 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj" podUID="ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Oct 31 04:32:05.128201 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:05.128115 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Oct 31 04:32:05.128469 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:05.128441 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc" podUID="0c071116-bc3c-4d68-9e80-ec887ee96be7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Oct 31 04:32:05.489660 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:05.487629 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" podUID="dccd15a9-47f5-4bd1-b21b-9d5ca8b05845" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 31 04:32:05.868683 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:05.851452 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 31 04:32:05.992867 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:05.992778 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d1d441-669694c87-fhkbx" podUID="52e6c267-5637-4fa5-9600-47b905bf75ee" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 04:32:06.101355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:06.077160 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2023e453-9kxs6"] Oct 31 04:32:06.148719 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:06.083812 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 31 04:32:06.148719 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:06.148623 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j" podUID="e107852e-0f1f-4705-bf39-a676808806ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 31 04:32:06.148719 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:06.112716 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-d1d441-588dbc6899-4rnfz" podUID="63386d30-7d65-4c86-9362-0ee7b8143bb1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 04:32:06.148719 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:06.112764 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" podUID="8e2c7041-ad52-49f4-8b55-d16691f4938f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:32:06.149070 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:06.104531 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:32:06.149273 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:06.084412 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Oct 31 04:32:06.149401 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:06.149304 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc" podUID="0c071116-bc3c-4d68-9e80-ec887ee96be7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Oct 31 04:32:06.149466 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:06.084465 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Oct 31 04:32:06.149521 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:06.149468 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj" podUID="ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Oct 31 04:32:06.149895 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:06.148387 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" podUID="643433d8-418e-4102-8bc3-cd544932614a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 04:32:06.663104 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:06.663067 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2023e453-9kxs6" event={"ID":"4d1e5631-5416-4a0a-ab66-716b75268f1b","Type":"ContainerStarted","Data":"22dc606b9ac4e9b48264ff634e5565db4d8c614dcf75bfb02ca2a6aa3bd366d8"} Oct 31 04:32:06.663368 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:06.663345 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2023e453-9kxs6" event={"ID":"4d1e5631-5416-4a0a-ab66-716b75268f1b","Type":"ContainerStarted","Data":"9a6c2ba725b48378bb9e3c2688b143acfcbafa7b982c128a835e5aefed103b6e"} Oct 31 04:32:06.706133 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:06.706079 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2023e453-9kxs6" podStartSLOduration=3.706061883 podStartE2EDuration="3.706061883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:32:06.705580111 +0000 UTC m=+1681.461918151" watchObservedRunningTime="2026-10-31 04:32:06.706061883 +0000 UTC m=+1681.462399922" Oct 31 04:32:06.994355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:06.990018 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Oct 31 04:32:06.994355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:06.990072 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc" podUID="0c071116-bc3c-4d68-9e80-ec887ee96be7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Oct 31 04:32:06.994355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:06.990989 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Oct 31 04:32:06.994355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:06.991030 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj" podUID="ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Oct 31 04:32:07.024665 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:07.024630 5019 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 31 04:32:07.024782 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:07.024699 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j" podUID="e107852e-0f1f-4705-bf39-a676808806ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 31 04:32:07.807374 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:07.796423 5019 generic.go:356] "Generic (PLEG): container finished" podID="4d1e5631-5416-4a0a-ab66-716b75268f1b" containerID="22dc606b9ac4e9b48264ff634e5565db4d8c614dcf75bfb02ca2a6aa3bd366d8" exitCode=0 Oct 31 04:32:07.807374 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:07.796714 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2023e453-9kxs6" event={"ID":"4d1e5631-5416-4a0a-ab66-716b75268f1b","Type":"ContainerDied","Data":"22dc606b9ac4e9b48264ff634e5565db4d8c614dcf75bfb02ca2a6aa3bd366d8"} Oct 31 04:32:08.207732 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:08.207647 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j" Oct 31 04:32:08.209494 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:08.209473 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j" Oct 31 04:32:08.221385 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:08.221367 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j" Oct 31 04:32:08.320356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:08.319619 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc" Oct 31 04:32:08.323265 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:08.321104 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc" Oct 31 04:32:08.323265 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:08.322229 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc" Oct 31 04:32:08.394639 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:08.394607 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj" Oct 31 04:32:08.395444 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:08.395299 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj" Oct 31 04:32:08.396131 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:08.396108 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj" Oct 31 04:32:09.060164 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:09.060129 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2023e453-9kxs6" Oct 31 04:32:09.161112 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:09.161042 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d1e5631-5416-4a0a-ab66-716b75268f1b-kube-api-access-frjgc\" (UniqueName: \"kubernetes.io/projected/4d1e5631-5416-4a0a-ab66-716b75268f1b-kube-api-access-frjgc\") pod \"4d1e5631-5416-4a0a-ab66-716b75268f1b\" (UID: \"4d1e5631-5416-4a0a-ab66-716b75268f1b\") " Oct 31 04:32:09.161262 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:09.161150 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d1e5631-5416-4a0a-ab66-716b75268f1b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d1e5631-5416-4a0a-ab66-716b75268f1b-maas-api-xks-ca\") pod \"4d1e5631-5416-4a0a-ab66-716b75268f1b\" (UID: \"4d1e5631-5416-4a0a-ab66-716b75268f1b\") " Oct 31 04:32:09.161314 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:09.161302 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d1e5631-5416-4a0a-ab66-716b75268f1b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d1e5631-5416-4a0a-ab66-716b75268f1b-maas-api-openshift-ca\") pod \"4d1e5631-5416-4a0a-ab66-716b75268f1b\" (UID: \"4d1e5631-5416-4a0a-ab66-716b75268f1b\") " Oct 31 04:32:09.161668 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:09.161609 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d1e5631-5416-4a0a-ab66-716b75268f1b-maas-api-openshift-ca" pod "4d1e5631-5416-4a0a-ab66-716b75268f1b" (UID: "4d1e5631-5416-4a0a-ab66-716b75268f1b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:32:09.161975 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:09.161953 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d1e5631-5416-4a0a-ab66-716b75268f1b-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:32:09.163175 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:09.163141 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d1e5631-5416-4a0a-ab66-716b75268f1b-kube-api-access-frjgc" pod "4d1e5631-5416-4a0a-ab66-716b75268f1b" (UID: "4d1e5631-5416-4a0a-ab66-716b75268f1b"). InnerVolumeSpecName "kube-api-access-frjgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:32:09.163298 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:09.163251 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d1e5631-5416-4a0a-ab66-716b75268f1b-maas-api-xks-ca" pod "4d1e5631-5416-4a0a-ab66-716b75268f1b" (UID: "4d1e5631-5416-4a0a-ab66-716b75268f1b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:32:09.263509 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:09.263471 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frjgc\" (UniqueName: \"kubernetes.io/projected/4d1e5631-5416-4a0a-ab66-716b75268f1b-kube-api-access-frjgc\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:32:09.263509 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:09.263511 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d1e5631-5416-4a0a-ab66-716b75268f1b-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:32:09.365202 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:09.365170 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz"] Oct 31 04:32:09.365414 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:09.365390 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz" podUID="ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd" containerName="istio-proxy" containerID="cri-o://22ef539616a122de2c4f5d74777ee579535cc7ed627f7e303c3fbed63ea48178" gracePeriod=30 Oct 31 04:32:09.835275 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:09.835243 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2023e453-9kxs6" Oct 31 04:32:09.835499 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:09.835272 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2023e453-9kxs6" event={"ID":"4d1e5631-5416-4a0a-ab66-716b75268f1b","Type":"ContainerDied","Data":"9a6c2ba725b48378bb9e3c2688b143acfcbafa7b982c128a835e5aefed103b6e"} Oct 31 04:32:09.835499 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:09.835326 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a6c2ba725b48378bb9e3c2688b143acfcbafa7b982c128a835e5aefed103b6e" Oct 31 04:32:10.573225 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:10.573193 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-552fab68-mhm82"] Oct 31 04:32:10.573832 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:10.573817 5019 cpu_manager.go:414] "containerMap: removing container" podUID="4d1e5631-5416-4a0a-ab66-716b75268f1b" containerName="revoke-keys" Oct 31 04:32:10.573874 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:10.573833 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d1e5631-5416-4a0a-ab66-716b75268f1b" containerName="revoke-keys" Oct 31 04:32:10.573968 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:10.573959 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d1e5631-5416-4a0a-ab66-716b75268f1b" containerName="revoke-keys" Oct 31 04:32:10.577729 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:10.577713 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-552fab68-mhm82" Oct 31 04:32:10.599401 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:10.599376 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-552fab68-mhm82"] Oct 31 04:32:10.617400 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:10.610079 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:32:10.617400 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:10.610157 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:32:10.647080 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:10.647055 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:32:10.647225 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:10.647078 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:32:10.730892 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:10.724185 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32940c15-666b-42f4-ab4e-67a371895cfd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-552fab68-mhm82\" (UID: \"32940c15-666b-42f4-ab4e-67a371895cfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-552fab68-mhm82" Oct 31 04:32:10.730892 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:10.724267 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdtz6\" (UniqueName: \"kubernetes.io/projected/32940c15-666b-42f4-ab4e-67a371895cfd-kube-api-access-rdtz6\") pod \"maas-api-revoke-sub-552fab68-mhm82\" (UID: \"32940c15-666b-42f4-ab4e-67a371895cfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-552fab68-mhm82" Oct 31 04:32:10.730892 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:10.724357 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32940c15-666b-42f4-ab4e-67a371895cfd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-552fab68-mhm82\" (UID: \"32940c15-666b-42f4-ab4e-67a371895cfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-552fab68-mhm82" Oct 31 04:32:10.825990 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:10.825913 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32940c15-666b-42f4-ab4e-67a371895cfd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-552fab68-mhm82\" (UID: \"32940c15-666b-42f4-ab4e-67a371895cfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-552fab68-mhm82" Oct 31 04:32:10.825990 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:10.825974 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdtz6\" (UniqueName: \"kubernetes.io/projected/32940c15-666b-42f4-ab4e-67a371895cfd-kube-api-access-rdtz6\") pod \"maas-api-revoke-sub-552fab68-mhm82\" (UID: \"32940c15-666b-42f4-ab4e-67a371895cfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-552fab68-mhm82" Oct 31 04:32:10.826183 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:10.826045 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32940c15-666b-42f4-ab4e-67a371895cfd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-552fab68-mhm82\" (UID: \"32940c15-666b-42f4-ab4e-67a371895cfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-552fab68-mhm82" Oct 31 04:32:10.826745 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:10.826718 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32940c15-666b-42f4-ab4e-67a371895cfd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-552fab68-mhm82\" (UID: \"32940c15-666b-42f4-ab4e-67a371895cfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-552fab68-mhm82" Oct 31 04:32:10.828429 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:10.828406 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32940c15-666b-42f4-ab4e-67a371895cfd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-552fab68-mhm82\" (UID: \"32940c15-666b-42f4-ab4e-67a371895cfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-552fab68-mhm82" Oct 31 04:32:10.848948 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:10.848921 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdtz6\" (UniqueName: \"kubernetes.io/projected/32940c15-666b-42f4-ab4e-67a371895cfd-kube-api-access-rdtz6\") pod \"maas-api-revoke-sub-552fab68-mhm82\" (UID: \"32940c15-666b-42f4-ab4e-67a371895cfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-552fab68-mhm82" Oct 31 04:32:10.892575 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:10.892554 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-552fab68-mhm82" Oct 31 04:32:11.201113 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:11.198393 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-552fab68-mhm82"] Oct 31 04:32:11.853017 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:11.852945 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-552fab68-mhm82" event={"ID":"32940c15-666b-42f4-ab4e-67a371895cfd","Type":"ContainerStarted","Data":"637c16d6733b797aeb8e045bb9452c904d29084b0c3c9baa6f61f09add36a310"} Oct 31 04:32:11.853017 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:11.853000 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-552fab68-mhm82" event={"ID":"32940c15-666b-42f4-ab4e-67a371895cfd","Type":"ContainerStarted","Data":"e6c1ca896575617fce9f4633a76ee63e3ced5734e88f1a551e9c85ecf4e8a71a"} Oct 31 04:32:11.893557 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:11.893512 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-552fab68-mhm82" podStartSLOduration=1.893501653 podStartE2EDuration="1.893501653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:32:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:32:11.892447264 +0000 UTC m=+1686.648785305" watchObservedRunningTime="2026-10-31 04:32:11.893501653 +0000 UTC m=+1686.649839691" Oct 31 04:32:12.538930 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:12.538900 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d1d441-588dbc6899-4rnfz" Oct 31 04:32:12.593411 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:12.593383 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d1d441-669694c87-fhkbx" Oct 31 04:32:12.675824 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:12.675754 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63386d30-7d65-4c86-9362-0ee7b8143bb1-kube-api-access-rt5qf\" (UniqueName: \"kubernetes.io/projected/63386d30-7d65-4c86-9362-0ee7b8143bb1-kube-api-access-rt5qf\") pod \"63386d30-7d65-4c86-9362-0ee7b8143bb1\" (UID: \"63386d30-7d65-4c86-9362-0ee7b8143bb1\") " Oct 31 04:32:12.675978 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:12.675922 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63386d30-7d65-4c86-9362-0ee7b8143bb1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63386d30-7d65-4c86-9362-0ee7b8143bb1-plugins-config-volume\") pod \"63386d30-7d65-4c86-9362-0ee7b8143bb1\" (UID: \"63386d30-7d65-4c86-9362-0ee7b8143bb1\") " Oct 31 04:32:12.676299 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:12.676274 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63386d30-7d65-4c86-9362-0ee7b8143bb1-plugins-config-volume" pod "63386d30-7d65-4c86-9362-0ee7b8143bb1" (UID: "63386d30-7d65-4c86-9362-0ee7b8143bb1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:32:12.676469 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:12.676453 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63386d30-7d65-4c86-9362-0ee7b8143bb1-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:32:12.678278 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:12.678244 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63386d30-7d65-4c86-9362-0ee7b8143bb1-kube-api-access-rt5qf" pod "63386d30-7d65-4c86-9362-0ee7b8143bb1" (UID: "63386d30-7d65-4c86-9362-0ee7b8143bb1"). InnerVolumeSpecName "kube-api-access-rt5qf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:32:12.777275 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:12.777244 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52e6c267-5637-4fa5-9600-47b905bf75ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52e6c267-5637-4fa5-9600-47b905bf75ee-plugins-config-volume\") pod \"52e6c267-5637-4fa5-9600-47b905bf75ee\" (UID: \"52e6c267-5637-4fa5-9600-47b905bf75ee\") " Oct 31 04:32:12.777501 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:12.777378 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52e6c267-5637-4fa5-9600-47b905bf75ee-kube-api-access-66wkc\" (UniqueName: \"kubernetes.io/projected/52e6c267-5637-4fa5-9600-47b905bf75ee-kube-api-access-66wkc\") pod \"52e6c267-5637-4fa5-9600-47b905bf75ee\" (UID: \"52e6c267-5637-4fa5-9600-47b905bf75ee\") " Oct 31 04:32:12.777791 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:12.777759 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52e6c267-5637-4fa5-9600-47b905bf75ee-plugins-config-volume" pod "52e6c267-5637-4fa5-9600-47b905bf75ee" (UID: "52e6c267-5637-4fa5-9600-47b905bf75ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:32:12.777869 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:12.777829 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt5qf\" (UniqueName: \"kubernetes.io/projected/63386d30-7d65-4c86-9362-0ee7b8143bb1-kube-api-access-rt5qf\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:32:12.779549 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:12.779521 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52e6c267-5637-4fa5-9600-47b905bf75ee-kube-api-access-66wkc" pod "52e6c267-5637-4fa5-9600-47b905bf75ee" (UID: "52e6c267-5637-4fa5-9600-47b905bf75ee"). InnerVolumeSpecName "kube-api-access-66wkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:32:12.862970 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:12.862935 5019 generic.go:356] "Generic (PLEG): container finished" podID="52e6c267-5637-4fa5-9600-47b905bf75ee" containerID="61c7da3fb92f673ff6b9332476bb7321bde96e0476b6509ff058a44caee7e765" exitCode=0 Oct 31 04:32:12.863379 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:12.862977 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d1d441-669694c87-fhkbx" Oct 31 04:32:12.863379 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:12.863014 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d1d441-669694c87-fhkbx" event={"ID":"52e6c267-5637-4fa5-9600-47b905bf75ee","Type":"ContainerDied","Data":"61c7da3fb92f673ff6b9332476bb7321bde96e0476b6509ff058a44caee7e765"} Oct 31 04:32:12.863379 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:12.863084 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d1d441-669694c87-fhkbx" event={"ID":"52e6c267-5637-4fa5-9600-47b905bf75ee","Type":"ContainerDied","Data":"02208dffebe04c90a42933964e85bf762a793a66b734dba99f2d242dd0128887"} Oct 31 04:32:12.863379 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:12.863114 5019 scope.go:122] "RemoveContainer" containerID="61c7da3fb92f673ff6b9332476bb7321bde96e0476b6509ff058a44caee7e765" Oct 31 04:32:12.864612 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:12.864588 5019 generic.go:356] "Generic (PLEG): container finished" podID="63386d30-7d65-4c86-9362-0ee7b8143bb1" containerID="dfbccaa07cd67f232fb76ed4abf8d821cedb480b743cf070f3c3073e8dcb3c19" exitCode=0 Oct 31 04:32:12.864732 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:12.864623 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d1d441-588dbc6899-4rnfz" Oct 31 04:32:12.864732 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:12.864694 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d1d441-588dbc6899-4rnfz" event={"ID":"63386d30-7d65-4c86-9362-0ee7b8143bb1","Type":"ContainerDied","Data":"dfbccaa07cd67f232fb76ed4abf8d821cedb480b743cf070f3c3073e8dcb3c19"} Oct 31 04:32:12.864853 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:12.864746 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d1d441-588dbc6899-4rnfz" event={"ID":"63386d30-7d65-4c86-9362-0ee7b8143bb1","Type":"ContainerDied","Data":"c8c6e25a97e83932a3f12134e430994e84afcdf751bb33090c6631b46ca13cc5"} Oct 31 04:32:12.872313 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:12.872287 5019 generic.go:356] "Generic (PLEG): container finished" podID="32940c15-666b-42f4-ab4e-67a371895cfd" containerID="637c16d6733b797aeb8e045bb9452c904d29084b0c3c9baa6f61f09add36a310" exitCode=0 Oct 31 04:32:12.872559 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:12.872371 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-552fab68-mhm82" event={"ID":"32940c15-666b-42f4-ab4e-67a371895cfd","Type":"ContainerDied","Data":"637c16d6733b797aeb8e045bb9452c904d29084b0c3c9baa6f61f09add36a310"} Oct 31 04:32:12.876891 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:12.876874 5019 scope.go:122] "RemoveContainer" containerID="61c7da3fb92f673ff6b9332476bb7321bde96e0476b6509ff058a44caee7e765" Oct 31 04:32:12.877160 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:32:12.877140 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61c7da3fb92f673ff6b9332476bb7321bde96e0476b6509ff058a44caee7e765\": container with ID starting with 61c7da3fb92f673ff6b9332476bb7321bde96e0476b6509ff058a44caee7e765 not found: ID does not exist" containerID="61c7da3fb92f673ff6b9332476bb7321bde96e0476b6509ff058a44caee7e765" Oct 31 04:32:12.877262 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:12.877165 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61c7da3fb92f673ff6b9332476bb7321bde96e0476b6509ff058a44caee7e765"} err="failed to get container status \"61c7da3fb92f673ff6b9332476bb7321bde96e0476b6509ff058a44caee7e765\": rpc error: code = NotFound desc = could not find container \"61c7da3fb92f673ff6b9332476bb7321bde96e0476b6509ff058a44caee7e765\": container with ID starting with 61c7da3fb92f673ff6b9332476bb7321bde96e0476b6509ff058a44caee7e765 not found: ID does not exist" Oct 31 04:32:12.877262 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:12.877182 5019 scope.go:122] "RemoveContainer" containerID="dfbccaa07cd67f232fb76ed4abf8d821cedb480b743cf070f3c3073e8dcb3c19" Oct 31 04:32:12.878677 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:12.878655 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52e6c267-5637-4fa5-9600-47b905bf75ee-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:32:12.878677 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:12.878676 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66wkc\" (UniqueName: \"kubernetes.io/projected/52e6c267-5637-4fa5-9600-47b905bf75ee-kube-api-access-66wkc\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:32:12.887484 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:12.887468 5019 scope.go:122] "RemoveContainer" containerID="dfbccaa07cd67f232fb76ed4abf8d821cedb480b743cf070f3c3073e8dcb3c19" Oct 31 04:32:12.887732 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:32:12.887706 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfbccaa07cd67f232fb76ed4abf8d821cedb480b743cf070f3c3073e8dcb3c19\": container with ID starting with dfbccaa07cd67f232fb76ed4abf8d821cedb480b743cf070f3c3073e8dcb3c19 not found: ID does not exist" containerID="dfbccaa07cd67f232fb76ed4abf8d821cedb480b743cf070f3c3073e8dcb3c19" Oct 31 04:32:12.887808 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:12.887745 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfbccaa07cd67f232fb76ed4abf8d821cedb480b743cf070f3c3073e8dcb3c19"} err="failed to get container status \"dfbccaa07cd67f232fb76ed4abf8d821cedb480b743cf070f3c3073e8dcb3c19\": rpc error: code = NotFound desc = could not find container \"dfbccaa07cd67f232fb76ed4abf8d821cedb480b743cf070f3c3073e8dcb3c19\": container with ID starting with dfbccaa07cd67f232fb76ed4abf8d821cedb480b743cf070f3c3073e8dcb3c19 not found: ID does not exist" Oct 31 04:32:12.961472 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:12.961392 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d1d441-588dbc6899-4rnfz"] Oct 31 04:32:12.964371 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:12.964349 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d1d441-588dbc6899-4rnfz"] Oct 31 04:32:13.013675 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:13.013629 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d1d441-669694c87-fhkbx"] Oct 31 04:32:13.020389 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:13.020359 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d1d441-669694c87-fhkbx"] Oct 31 04:32:13.408469 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:13.408436 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52e6c267-5637-4fa5-9600-47b905bf75ee" path="/var/lib/kubelet/pods/52e6c267-5637-4fa5-9600-47b905bf75ee/volumes" Oct 31 04:32:13.408863 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:13.408843 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63386d30-7d65-4c86-9362-0ee7b8143bb1" path="/var/lib/kubelet/pods/63386d30-7d65-4c86-9362-0ee7b8143bb1/volumes" Oct 31 04:32:14.056779 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:14.056757 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-552fab68-mhm82" Oct 31 04:32:14.111502 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:14.111474 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f"] Oct 31 04:32:14.111783 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:14.111720 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f" podUID="116c8b42-2784-4f46-b534-ea35159e3cac" containerName="istio-proxy" containerID="cri-o://4a06bbf73f055024ff0141448cf35493927a32b8e54fa13559532d8448dc50e4" gracePeriod=30 Oct 31 04:32:14.215407 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:14.215248 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32940c15-666b-42f4-ab4e-67a371895cfd-kube-api-access-rdtz6\" (UniqueName: \"kubernetes.io/projected/32940c15-666b-42f4-ab4e-67a371895cfd-kube-api-access-rdtz6\") pod \"32940c15-666b-42f4-ab4e-67a371895cfd\" (UID: \"32940c15-666b-42f4-ab4e-67a371895cfd\") " Oct 31 04:32:14.215407 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:14.215298 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32940c15-666b-42f4-ab4e-67a371895cfd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32940c15-666b-42f4-ab4e-67a371895cfd-maas-api-xks-ca\") pod \"32940c15-666b-42f4-ab4e-67a371895cfd\" (UID: \"32940c15-666b-42f4-ab4e-67a371895cfd\") " Oct 31 04:32:14.240647 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:14.216707 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32940c15-666b-42f4-ab4e-67a371895cfd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32940c15-666b-42f4-ab4e-67a371895cfd-maas-api-openshift-ca\") pod \"32940c15-666b-42f4-ab4e-67a371895cfd\" (UID: \"32940c15-666b-42f4-ab4e-67a371895cfd\") " Oct 31 04:32:14.240647 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:14.217759 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32940c15-666b-42f4-ab4e-67a371895cfd-maas-api-openshift-ca" pod "32940c15-666b-42f4-ab4e-67a371895cfd" (UID: "32940c15-666b-42f4-ab4e-67a371895cfd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:32:14.265374 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:14.263624 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32940c15-666b-42f4-ab4e-67a371895cfd-kube-api-access-rdtz6" pod "32940c15-666b-42f4-ab4e-67a371895cfd" (UID: "32940c15-666b-42f4-ab4e-67a371895cfd"). InnerVolumeSpecName "kube-api-access-rdtz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:32:14.276355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:14.274472 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32940c15-666b-42f4-ab4e-67a371895cfd-maas-api-xks-ca" pod "32940c15-666b-42f4-ab4e-67a371895cfd" (UID: "32940c15-666b-42f4-ab4e-67a371895cfd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:32:14.324352 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:14.321605 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32940c15-666b-42f4-ab4e-67a371895cfd-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:32:14.324352 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:14.321641 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdtz6\" (UniqueName: \"kubernetes.io/projected/32940c15-666b-42f4-ab4e-67a371895cfd-kube-api-access-rdtz6\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:32:14.324352 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:14.321658 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32940c15-666b-42f4-ab4e-67a371895cfd-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:32:14.634439 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:14.634414 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz" Oct 31 04:32:14.726070 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:14.726026 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-workload-certs\") pod \"ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd\" (UID: \"ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd\") " Oct 31 04:32:14.726304 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:14.726096 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-istio-envoy\") pod \"ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd\" (UID: \"ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd\") " Oct 31 04:32:14.726304 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:14.726152 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-kube-api-access-j9fc6\" (UniqueName: \"kubernetes.io/projected/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-kube-api-access-j9fc6\") pod \"ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd\" (UID: \"ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd\") " Oct 31 04:32:14.726304 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:14.726297 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-istio-data\") pod \"ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd\" (UID: \"ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd\") " Oct 31 04:32:14.726585 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:14.726374 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-credential-socket\") pod \"ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd\" (UID: \"ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd\") " Oct 31 04:32:14.726585 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:14.726425 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-istio-token\" (UniqueName: \"kubernetes.io/projected/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-istio-token\") pod \"ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd\" (UID: \"ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd\") " Oct 31 04:32:14.726585 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:14.726464 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-istio-podinfo\") pod \"ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd\" (UID: \"ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd\") " Oct 31 04:32:14.726585 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:14.726527 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-workload-socket\") pod \"ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd\" (UID: \"ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd\") " Oct 31 04:32:14.726842 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:14.726601 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-istiod-ca-cert\") pod \"ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd\" (UID: \"ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd\") " Oct 31 04:32:14.727141 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:14.727114 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-istio-data" pod "ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd" (UID: "ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:32:14.727359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:14.727313 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:32:14.727543 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:14.727514 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-credential-socket" pod "ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd" (UID: "ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:32:14.727745 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:14.727724 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-workload-socket" pod "ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd" (UID: "ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:32:14.727831 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:14.727745 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-istiod-ca-cert" pod "ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd" (UID: "ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:32:14.728023 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:14.727998 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-workload-certs" pod "ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd" (UID: "ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:32:14.730112 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:14.730048 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-istio-podinfo" pod "ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd" (UID: "ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:32:14.730719 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:14.730665 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-istio-token" pod "ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd" (UID: "ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:32:14.731446 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:14.731420 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-istio-envoy" pod "ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd" (UID: "ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:32:14.733386 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:14.733273 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-kube-api-access-j9fc6" pod "ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd" (UID: "ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd"). InnerVolumeSpecName "kube-api-access-j9fc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:32:14.828766 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:14.828735 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:32:14.828766 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:14.828761 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:32:14.828766 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:14.828771 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:32:14.828966 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:14.828779 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:32:14.828966 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:14.828788 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:32:14.828966 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:14.828796 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:32:14.828966 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:14.828803 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:32:14.828966 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:14.828810 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9fc6\" (UniqueName: \"kubernetes.io/projected/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd-kube-api-access-j9fc6\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:32:14.897970 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:14.897937 5019 generic.go:356] "Generic (PLEG): container finished" podID="ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd" containerID="22ef539616a122de2c4f5d74777ee579535cc7ed627f7e303c3fbed63ea48178" exitCode=0 Oct 31 04:32:14.898253 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:14.898230 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz" event={"ID":"ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd","Type":"ContainerDied","Data":"22ef539616a122de2c4f5d74777ee579535cc7ed627f7e303c3fbed63ea48178"} Oct 31 04:32:14.898479 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:14.898461 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz" event={"ID":"ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd","Type":"ContainerDied","Data":"406aa3896d3bef7d44e30014e54461d08c38204cf12b35f1c2f175ff3ba5aa2e"} Oct 31 04:32:14.898631 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:14.898616 5019 scope.go:122] "RemoveContainer" containerID="22ef539616a122de2c4f5d74777ee579535cc7ed627f7e303c3fbed63ea48178" Oct 31 04:32:14.898905 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:14.898888 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz" Oct 31 04:32:14.901988 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:14.901961 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-552fab68-mhm82" event={"ID":"32940c15-666b-42f4-ab4e-67a371895cfd","Type":"ContainerDied","Data":"e6c1ca896575617fce9f4633a76ee63e3ced5734e88f1a551e9c85ecf4e8a71a"} Oct 31 04:32:14.902088 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:14.902014 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6c1ca896575617fce9f4633a76ee63e3ced5734e88f1a551e9c85ecf4e8a71a" Oct 31 04:32:14.902430 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:14.902200 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-552fab68-mhm82" Oct 31 04:32:14.914425 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:14.914408 5019 scope.go:122] "RemoveContainer" containerID="22ef539616a122de2c4f5d74777ee579535cc7ed627f7e303c3fbed63ea48178" Oct 31 04:32:14.914659 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:32:14.914639 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22ef539616a122de2c4f5d74777ee579535cc7ed627f7e303c3fbed63ea48178\": container with ID starting with 22ef539616a122de2c4f5d74777ee579535cc7ed627f7e303c3fbed63ea48178 not found: ID does not exist" containerID="22ef539616a122de2c4f5d74777ee579535cc7ed627f7e303c3fbed63ea48178" Oct 31 04:32:14.914715 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:14.914669 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22ef539616a122de2c4f5d74777ee579535cc7ed627f7e303c3fbed63ea48178"} err="failed to get container status \"22ef539616a122de2c4f5d74777ee579535cc7ed627f7e303c3fbed63ea48178\": rpc error: code = NotFound desc = could not find container \"22ef539616a122de2c4f5d74777ee579535cc7ed627f7e303c3fbed63ea48178\": container with ID starting with 22ef539616a122de2c4f5d74777ee579535cc7ed627f7e303c3fbed63ea48178 not found: ID does not exist" Oct 31 04:32:14.976499 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:14.976464 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz"] Oct 31 04:32:14.980750 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:14.980721 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-d1d441-openshift-default-c79997b97-7gslz"] Oct 31 04:32:15.038376 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:15.038234 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c95c1e20-gvw7v"] Oct 31 04:32:15.050394 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:15.045158 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c95c1e20-gvw7v"] Oct 31 04:32:15.408501 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:15.408425 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50b851f6-c51d-4482-a1f5-94cfdfbc1741" path="/var/lib/kubelet/pods/50b851f6-c51d-4482-a1f5-94cfdfbc1741/volumes" Oct 31 04:32:15.408969 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:15.408948 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd" path="/var/lib/kubelet/pods/ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd/volumes" Oct 31 04:32:15.428586 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:15.428297 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9"] Oct 31 04:32:15.429205 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:15.429181 5019 cpu_manager.go:414] "containerMap: removing container" podUID="32940c15-666b-42f4-ab4e-67a371895cfd" containerName="revoke-keys" Oct 31 04:32:15.429296 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:15.429208 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32940c15-666b-42f4-ab4e-67a371895cfd" containerName="revoke-keys" Oct 31 04:32:15.429296 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:15.429231 5019 cpu_manager.go:414] "containerMap: removing container" podUID="63386d30-7d65-4c86-9362-0ee7b8143bb1" containerName="payload-processing" Oct 31 04:32:15.429296 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:15.429239 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63386d30-7d65-4c86-9362-0ee7b8143bb1" containerName="payload-processing" Oct 31 04:32:15.429296 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:15.429252 5019 cpu_manager.go:414] "containerMap: removing container" podUID="ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd" containerName="istio-proxy" Oct 31 04:32:15.429296 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:15.429260 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd" containerName="istio-proxy" Oct 31 04:32:15.429658 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:15.429320 5019 cpu_manager.go:414] "containerMap: removing container" podUID="52e6c267-5637-4fa5-9600-47b905bf75ee" containerName="payload-pre-processing" Oct 31 04:32:15.429658 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:15.429327 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52e6c267-5637-4fa5-9600-47b905bf75ee" containerName="payload-pre-processing" Oct 31 04:32:15.429658 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:15.429489 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="52e6c267-5637-4fa5-9600-47b905bf75ee" containerName="payload-pre-processing" Oct 31 04:32:15.429658 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:15.429509 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="32940c15-666b-42f4-ab4e-67a371895cfd" containerName="revoke-keys" Oct 31 04:32:15.429658 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:15.429521 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="63386d30-7d65-4c86-9362-0ee7b8143bb1" containerName="payload-processing" Oct 31 04:32:15.429658 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:15.429530 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab0ba839-9c46-40b2-b4ff-7cfc7249e8cd" containerName="istio-proxy" Oct 31 04:32:15.435314 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:15.435295 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9" Oct 31 04:32:15.457848 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:15.457823 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9"] Oct 31 04:32:15.513588 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:15.513557 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-338f2e-openshift-default-dockercfg-lkdhm\"" Oct 31 04:32:15.550356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:15.547586 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b547bb0b-cae2-453b-aa56-4f37434a1ce8-istio-podinfo\") pod \"e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9\" (UID: \"b547bb0b-cae2-453b-aa56-4f37434a1ce8\") " pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9" Oct 31 04:32:15.550356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:15.547648 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b547bb0b-cae2-453b-aa56-4f37434a1ce8-istio-envoy\") pod \"e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9\" (UID: \"b547bb0b-cae2-453b-aa56-4f37434a1ce8\") " pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9" Oct 31 04:32:15.550356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:15.547770 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b547bb0b-cae2-453b-aa56-4f37434a1ce8-workload-certs\") pod \"e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9\" (UID: \"b547bb0b-cae2-453b-aa56-4f37434a1ce8\") " pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9" Oct 31 04:32:15.550356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:15.547804 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b547bb0b-cae2-453b-aa56-4f37434a1ce8-istiod-ca-cert\") pod \"e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9\" (UID: \"b547bb0b-cae2-453b-aa56-4f37434a1ce8\") " pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9" Oct 31 04:32:15.550356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:15.547988 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b547bb0b-cae2-453b-aa56-4f37434a1ce8-workload-socket\") pod \"e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9\" (UID: \"b547bb0b-cae2-453b-aa56-4f37434a1ce8\") " pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9" Oct 31 04:32:15.550356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:15.548034 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wmgk\" (UniqueName: \"kubernetes.io/projected/b547bb0b-cae2-453b-aa56-4f37434a1ce8-kube-api-access-6wmgk\") pod \"e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9\" (UID: \"b547bb0b-cae2-453b-aa56-4f37434a1ce8\") " pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9" Oct 31 04:32:15.550356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:15.548067 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b547bb0b-cae2-453b-aa56-4f37434a1ce8-istio-data\") pod \"e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9\" (UID: \"b547bb0b-cae2-453b-aa56-4f37434a1ce8\") " pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9" Oct 31 04:32:15.550356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:15.548113 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b547bb0b-cae2-453b-aa56-4f37434a1ce8-istio-token\") pod \"e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9\" (UID: \"b547bb0b-cae2-453b-aa56-4f37434a1ce8\") " pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9" Oct 31 04:32:15.550356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:15.548153 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b547bb0b-cae2-453b-aa56-4f37434a1ce8-credential-socket\") pod \"e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9\" (UID: \"b547bb0b-cae2-453b-aa56-4f37434a1ce8\") " pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9" Oct 31 04:32:15.653184 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:15.653051 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b547bb0b-cae2-453b-aa56-4f37434a1ce8-workload-socket\") pod \"e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9\" (UID: \"b547bb0b-cae2-453b-aa56-4f37434a1ce8\") " pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9" Oct 31 04:32:15.653184 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:15.653108 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wmgk\" (UniqueName: \"kubernetes.io/projected/b547bb0b-cae2-453b-aa56-4f37434a1ce8-kube-api-access-6wmgk\") pod \"e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9\" (UID: \"b547bb0b-cae2-453b-aa56-4f37434a1ce8\") " pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9" Oct 31 04:32:15.653184 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:15.653157 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b547bb0b-cae2-453b-aa56-4f37434a1ce8-istio-data\") pod \"e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9\" (UID: \"b547bb0b-cae2-453b-aa56-4f37434a1ce8\") " pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9" Oct 31 04:32:15.653521 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:15.653210 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b547bb0b-cae2-453b-aa56-4f37434a1ce8-istio-token\") pod \"e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9\" (UID: \"b547bb0b-cae2-453b-aa56-4f37434a1ce8\") " pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9" Oct 31 04:32:15.653521 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:15.653247 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b547bb0b-cae2-453b-aa56-4f37434a1ce8-credential-socket\") pod \"e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9\" (UID: \"b547bb0b-cae2-453b-aa56-4f37434a1ce8\") " pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9" Oct 31 04:32:15.655057 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:15.654069 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b547bb0b-cae2-453b-aa56-4f37434a1ce8-workload-socket\") pod \"e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9\" (UID: \"b547bb0b-cae2-453b-aa56-4f37434a1ce8\") " pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9" Oct 31 04:32:15.655057 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:15.654303 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b547bb0b-cae2-453b-aa56-4f37434a1ce8-istio-data\") pod \"e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9\" (UID: \"b547bb0b-cae2-453b-aa56-4f37434a1ce8\") " pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9" Oct 31 04:32:15.655057 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:15.654577 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b547bb0b-cae2-453b-aa56-4f37434a1ce8-istio-podinfo\") pod \"e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9\" (UID: \"b547bb0b-cae2-453b-aa56-4f37434a1ce8\") " pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9" Oct 31 04:32:15.655057 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:15.654632 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b547bb0b-cae2-453b-aa56-4f37434a1ce8-istio-envoy\") pod \"e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9\" (UID: \"b547bb0b-cae2-453b-aa56-4f37434a1ce8\") " pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9" Oct 31 04:32:15.655057 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:15.654786 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b547bb0b-cae2-453b-aa56-4f37434a1ce8-workload-certs\") pod \"e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9\" (UID: \"b547bb0b-cae2-453b-aa56-4f37434a1ce8\") " pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9" Oct 31 04:32:15.657314 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:15.657291 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b547bb0b-cae2-453b-aa56-4f37434a1ce8-credential-socket\") pod \"e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9\" (UID: \"b547bb0b-cae2-453b-aa56-4f37434a1ce8\") " pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9" Oct 31 04:32:15.657479 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:15.657458 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b547bb0b-cae2-453b-aa56-4f37434a1ce8-istiod-ca-cert\") pod \"e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9\" (UID: \"b547bb0b-cae2-453b-aa56-4f37434a1ce8\") " pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9" Oct 31 04:32:15.658088 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:15.658068 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b547bb0b-cae2-453b-aa56-4f37434a1ce8-istiod-ca-cert\") pod \"e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9\" (UID: \"b547bb0b-cae2-453b-aa56-4f37434a1ce8\") " pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9" Oct 31 04:32:15.658497 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:15.658443 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b547bb0b-cae2-453b-aa56-4f37434a1ce8-workload-certs\") pod \"e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9\" (UID: \"b547bb0b-cae2-453b-aa56-4f37434a1ce8\") " pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9" Oct 31 04:32:15.676057 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:15.675964 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b547bb0b-cae2-453b-aa56-4f37434a1ce8-istio-podinfo\") pod \"e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9\" (UID: \"b547bb0b-cae2-453b-aa56-4f37434a1ce8\") " pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9" Oct 31 04:32:15.682923 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:15.682900 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wmgk\" (UniqueName: \"kubernetes.io/projected/b547bb0b-cae2-453b-aa56-4f37434a1ce8-kube-api-access-6wmgk\") pod \"e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9\" (UID: \"b547bb0b-cae2-453b-aa56-4f37434a1ce8\") " pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9" Oct 31 04:32:15.689261 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:15.688685 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b547bb0b-cae2-453b-aa56-4f37434a1ce8-istio-envoy\") pod \"e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9\" (UID: \"b547bb0b-cae2-453b-aa56-4f37434a1ce8\") " pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9" Oct 31 04:32:15.708952 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:15.708918 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b547bb0b-cae2-453b-aa56-4f37434a1ce8-istio-token\") pod \"e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9\" (UID: \"b547bb0b-cae2-453b-aa56-4f37434a1ce8\") " pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9" Oct 31 04:32:15.758819 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:15.758782 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9" Oct 31 04:32:16.144584 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:32:16.144544 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb547bb0b_cae2_453b_aa56_4f37434a1ce8.slice/crio-1a0345cd5def1ae4e0cb237e4c08d2f51af0687b5cb1f8a7f98e2bae828f16e1 WatchSource:0}: Error finding container 1a0345cd5def1ae4e0cb237e4c08d2f51af0687b5cb1f8a7f98e2bae828f16e1: Status 404 returned error can't find the container with id 1a0345cd5def1ae4e0cb237e4c08d2f51af0687b5cb1f8a7f98e2bae828f16e1 Oct 31 04:32:16.162021 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:16.161709 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:32:16.162021 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:16.161800 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:32:16.162021 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:16.161837 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:32:16.263608 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:16.262842 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9"] Oct 31 04:32:16.924440 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:16.924403 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9" event={"ID":"b547bb0b-cae2-453b-aa56-4f37434a1ce8","Type":"ContainerStarted","Data":"07fe73b7b2ad40bd8c4007ec62b8acdccaf0a951bd6e972b1f4c02148cb75cd1"} Oct 31 04:32:16.924893 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:16.924464 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9" event={"ID":"b547bb0b-cae2-453b-aa56-4f37434a1ce8","Type":"ContainerStarted","Data":"1a0345cd5def1ae4e0cb237e4c08d2f51af0687b5cb1f8a7f98e2bae828f16e1"} Oct 31 04:32:16.974723 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:16.974669 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9" podStartSLOduration=1.9746540179999998 podStartE2EDuration="1.974654018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:32:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:32:16.968676796 +0000 UTC m=+1691.725014834" watchObservedRunningTime="2026-10-31 04:32:16.974654018 +0000 UTC m=+1691.730992060" Oct 31 04:32:17.759803 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:17.759771 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9" Oct 31 04:32:17.765529 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:17.765505 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9" Oct 31 04:32:17.931569 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:17.931539 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9" Oct 31 04:32:17.932379 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:17.932328 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9" Oct 31 04:32:19.416564 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:19.416535 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f" Oct 31 04:32:19.560639 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:19.560610 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/116c8b42-2784-4f46-b534-ea35159e3cac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/116c8b42-2784-4f46-b534-ea35159e3cac-workload-certs\") pod \"116c8b42-2784-4f46-b534-ea35159e3cac\" (UID: \"116c8b42-2784-4f46-b534-ea35159e3cac\") " Oct 31 04:32:19.560817 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:19.560659 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/116c8b42-2784-4f46-b534-ea35159e3cac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/116c8b42-2784-4f46-b534-ea35159e3cac-workload-socket\") pod \"116c8b42-2784-4f46-b534-ea35159e3cac\" (UID: \"116c8b42-2784-4f46-b534-ea35159e3cac\") " Oct 31 04:32:19.560817 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:19.560684 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/116c8b42-2784-4f46-b534-ea35159e3cac-istio-token\" (UniqueName: \"kubernetes.io/projected/116c8b42-2784-4f46-b534-ea35159e3cac-istio-token\") pod \"116c8b42-2784-4f46-b534-ea35159e3cac\" (UID: \"116c8b42-2784-4f46-b534-ea35159e3cac\") " Oct 31 04:32:19.560817 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:19.560781 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/116c8b42-2784-4f46-b534-ea35159e3cac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/116c8b42-2784-4f46-b534-ea35159e3cac-istio-data\") pod \"116c8b42-2784-4f46-b534-ea35159e3cac\" (UID: \"116c8b42-2784-4f46-b534-ea35159e3cac\") " Oct 31 04:32:19.561012 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:19.560818 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/116c8b42-2784-4f46-b534-ea35159e3cac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/116c8b42-2784-4f46-b534-ea35159e3cac-istiod-ca-cert\") pod \"116c8b42-2784-4f46-b534-ea35159e3cac\" (UID: \"116c8b42-2784-4f46-b534-ea35159e3cac\") " Oct 31 04:32:19.561012 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:19.560923 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/116c8b42-2784-4f46-b534-ea35159e3cac-workload-socket" pod "116c8b42-2784-4f46-b534-ea35159e3cac" (UID: "116c8b42-2784-4f46-b534-ea35159e3cac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:32:19.561012 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:19.560935 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/116c8b42-2784-4f46-b534-ea35159e3cac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/116c8b42-2784-4f46-b534-ea35159e3cac-istio-envoy\") pod \"116c8b42-2784-4f46-b534-ea35159e3cac\" (UID: \"116c8b42-2784-4f46-b534-ea35159e3cac\") " Oct 31 04:32:19.561012 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:19.560964 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/116c8b42-2784-4f46-b534-ea35159e3cac-workload-certs" pod "116c8b42-2784-4f46-b534-ea35159e3cac" (UID: "116c8b42-2784-4f46-b534-ea35159e3cac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:32:19.561309 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:19.561013 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/116c8b42-2784-4f46-b534-ea35159e3cac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/116c8b42-2784-4f46-b534-ea35159e3cac-istio-podinfo\") pod \"116c8b42-2784-4f46-b534-ea35159e3cac\" (UID: \"116c8b42-2784-4f46-b534-ea35159e3cac\") " Oct 31 04:32:19.561309 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:19.561069 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/116c8b42-2784-4f46-b534-ea35159e3cac-kube-api-access-dr9m8\" (UniqueName: \"kubernetes.io/projected/116c8b42-2784-4f46-b534-ea35159e3cac-kube-api-access-dr9m8\") pod \"116c8b42-2784-4f46-b534-ea35159e3cac\" (UID: \"116c8b42-2784-4f46-b534-ea35159e3cac\") " Oct 31 04:32:19.561309 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:19.561101 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/116c8b42-2784-4f46-b534-ea35159e3cac-istio-data" pod "116c8b42-2784-4f46-b534-ea35159e3cac" (UID: "116c8b42-2784-4f46-b534-ea35159e3cac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:32:19.561309 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:19.561115 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/116c8b42-2784-4f46-b534-ea35159e3cac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/116c8b42-2784-4f46-b534-ea35159e3cac-credential-socket\") pod \"116c8b42-2784-4f46-b534-ea35159e3cac\" (UID: \"116c8b42-2784-4f46-b534-ea35159e3cac\") " Oct 31 04:32:19.561309 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:19.561220 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/116c8b42-2784-4f46-b534-ea35159e3cac-istiod-ca-cert" pod "116c8b42-2784-4f46-b534-ea35159e3cac" (UID: "116c8b42-2784-4f46-b534-ea35159e3cac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:32:19.561601 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:19.561425 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/116c8b42-2784-4f46-b534-ea35159e3cac-credential-socket" pod "116c8b42-2784-4f46-b534-ea35159e3cac" (UID: "116c8b42-2784-4f46-b534-ea35159e3cac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:32:19.562089 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:19.562064 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/116c8b42-2784-4f46-b534-ea35159e3cac-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:32:19.562995 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:19.562551 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/116c8b42-2784-4f46-b534-ea35159e3cac-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:32:19.562995 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:19.562573 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/116c8b42-2784-4f46-b534-ea35159e3cac-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:32:19.562995 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:19.562587 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/116c8b42-2784-4f46-b534-ea35159e3cac-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:32:19.562995 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:19.562600 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/116c8b42-2784-4f46-b534-ea35159e3cac-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:32:19.563843 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:19.563811 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/116c8b42-2784-4f46-b534-ea35159e3cac-istio-token" pod "116c8b42-2784-4f46-b534-ea35159e3cac" (UID: "116c8b42-2784-4f46-b534-ea35159e3cac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:32:19.576209 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:19.576185 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/116c8b42-2784-4f46-b534-ea35159e3cac-kube-api-access-dr9m8" pod "116c8b42-2784-4f46-b534-ea35159e3cac" (UID: "116c8b42-2784-4f46-b534-ea35159e3cac"). InnerVolumeSpecName "kube-api-access-dr9m8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:32:19.576681 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:19.576654 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/116c8b42-2784-4f46-b534-ea35159e3cac-istio-podinfo" pod "116c8b42-2784-4f46-b534-ea35159e3cac" (UID: "116c8b42-2784-4f46-b534-ea35159e3cac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:32:19.576782 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:19.576658 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/116c8b42-2784-4f46-b534-ea35159e3cac-istio-envoy" pod "116c8b42-2784-4f46-b534-ea35159e3cac" (UID: "116c8b42-2784-4f46-b534-ea35159e3cac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:32:19.663769 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:19.663712 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/116c8b42-2784-4f46-b534-ea35159e3cac-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:32:19.663769 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:19.663735 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/116c8b42-2784-4f46-b534-ea35159e3cac-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:32:19.663769 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:19.663744 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dr9m8\" (UniqueName: \"kubernetes.io/projected/116c8b42-2784-4f46-b534-ea35159e3cac-kube-api-access-dr9m8\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:32:19.663769 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:19.663752 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/116c8b42-2784-4f46-b534-ea35159e3cac-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:32:19.949226 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:19.949150 5019 generic.go:356] "Generic (PLEG): container finished" podID="116c8b42-2784-4f46-b534-ea35159e3cac" containerID="4a06bbf73f055024ff0141448cf35493927a32b8e54fa13559532d8448dc50e4" exitCode=0 Oct 31 04:32:19.949226 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:19.949216 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f" Oct 31 04:32:19.949425 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:19.949233 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f" event={"ID":"116c8b42-2784-4f46-b534-ea35159e3cac","Type":"ContainerDied","Data":"4a06bbf73f055024ff0141448cf35493927a32b8e54fa13559532d8448dc50e4"} Oct 31 04:32:19.949425 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:19.949316 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f" event={"ID":"116c8b42-2784-4f46-b534-ea35159e3cac","Type":"ContainerDied","Data":"8864375a14a4ec3883250543080344ce696cebfbe58acff4192d85d1bb650410"} Oct 31 04:32:19.949425 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:19.949374 5019 scope.go:122] "RemoveContainer" containerID="4a06bbf73f055024ff0141448cf35493927a32b8e54fa13559532d8448dc50e4" Oct 31 04:32:19.960777 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:19.960756 5019 scope.go:122] "RemoveContainer" containerID="a1f2a6d8ebb597400c5d9b8695bc42b7c367b59a6adb50e297ccef83a682760a" Oct 31 04:32:19.970041 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:19.970016 5019 scope.go:122] "RemoveContainer" containerID="4a06bbf73f055024ff0141448cf35493927a32b8e54fa13559532d8448dc50e4" Oct 31 04:32:19.970320 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:32:19.970300 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a06bbf73f055024ff0141448cf35493927a32b8e54fa13559532d8448dc50e4\": container with ID starting with 4a06bbf73f055024ff0141448cf35493927a32b8e54fa13559532d8448dc50e4 not found: ID does not exist" containerID="4a06bbf73f055024ff0141448cf35493927a32b8e54fa13559532d8448dc50e4" Oct 31 04:32:19.970431 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:19.970346 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a06bbf73f055024ff0141448cf35493927a32b8e54fa13559532d8448dc50e4"} err="failed to get container status \"4a06bbf73f055024ff0141448cf35493927a32b8e54fa13559532d8448dc50e4\": rpc error: code = NotFound desc = could not find container \"4a06bbf73f055024ff0141448cf35493927a32b8e54fa13559532d8448dc50e4\": container with ID starting with 4a06bbf73f055024ff0141448cf35493927a32b8e54fa13559532d8448dc50e4 not found: ID does not exist" Oct 31 04:32:19.970431 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:19.970370 5019 scope.go:122] "RemoveContainer" containerID="a1f2a6d8ebb597400c5d9b8695bc42b7c367b59a6adb50e297ccef83a682760a" Oct 31 04:32:19.970649 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:32:19.970628 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1f2a6d8ebb597400c5d9b8695bc42b7c367b59a6adb50e297ccef83a682760a\": container with ID starting with a1f2a6d8ebb597400c5d9b8695bc42b7c367b59a6adb50e297ccef83a682760a not found: ID does not exist" containerID="a1f2a6d8ebb597400c5d9b8695bc42b7c367b59a6adb50e297ccef83a682760a" Oct 31 04:32:19.970689 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:19.970659 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1f2a6d8ebb597400c5d9b8695bc42b7c367b59a6adb50e297ccef83a682760a"} err="failed to get container status \"a1f2a6d8ebb597400c5d9b8695bc42b7c367b59a6adb50e297ccef83a682760a\": rpc error: code = NotFound desc = could not find container \"a1f2a6d8ebb597400c5d9b8695bc42b7c367b59a6adb50e297ccef83a682760a\": container with ID starting with a1f2a6d8ebb597400c5d9b8695bc42b7c367b59a6adb50e297ccef83a682760a not found: ID does not exist" Oct 31 04:32:20.015607 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:20.015573 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f"] Oct 31 04:32:20.027704 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:20.027682 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-e37d27-openshift-default-576b58b5-kk62f"] Oct 31 04:32:20.429683 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:20.429643 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-81c810-666bd7cfc5-k862v"] Oct 31 04:32:21.411619 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:21.411586 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="116c8b42-2784-4f46-b534-ea35159e3cac" path="/var/lib/kubelet/pods/116c8b42-2784-4f46-b534-ea35159e3cac/volumes" Oct 31 04:32:22.400284 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:32:22.400248 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:32:22.587257 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.587218 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-338f2e-6ccbc66d75-htxcd"] Oct 31 04:32:22.587885 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.587869 5019 cpu_manager.go:414] "containerMap: removing container" podUID="116c8b42-2784-4f46-b534-ea35159e3cac" containerName="istio-proxy" Oct 31 04:32:22.587932 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.587886 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="116c8b42-2784-4f46-b534-ea35159e3cac" containerName="istio-proxy" Oct 31 04:32:22.587932 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.587903 5019 cpu_manager.go:414] "containerMap: removing container" podUID="116c8b42-2784-4f46-b534-ea35159e3cac" containerName="istio-proxy" Oct 31 04:32:22.587932 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.587909 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="116c8b42-2784-4f46-b534-ea35159e3cac" containerName="istio-proxy" Oct 31 04:32:22.588047 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.588008 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="116c8b42-2784-4f46-b534-ea35159e3cac" containerName="istio-proxy" Oct 31 04:32:22.588047 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.588017 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="116c8b42-2784-4f46-b534-ea35159e3cac" containerName="istio-proxy" Oct 31 04:32:22.592932 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.592911 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-338f2e-6ccbc66d75-htxcd" Oct 31 04:32:22.622528 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.622493 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-338f2e-dockercfg-wlvz5\"" Oct 31 04:32:22.628564 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.628542 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-338f2e-6ccbc66d75-htxcd"] Oct 31 04:32:22.640843 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.640790 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-338f2e\"" Oct 31 04:32:22.647897 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.647870 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-338f2e-57f7757bf-mchhm"] Oct 31 04:32:22.653849 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.653797 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-338f2e-57f7757bf-mchhm" Oct 31 04:32:22.673300 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.672103 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-338f2e-57f7757bf-mchhm"] Oct 31 04:32:22.722065 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.722019 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rcqb\" (UniqueName: \"kubernetes.io/projected/53f77b18-7ca9-4e11-869d-0a1aa66d2057-kube-api-access-4rcqb\") pod \"payload-pre-processing-e2e-shared-a-w3-338f2e-6ccbc66d75-htxcd\" (UID: \"53f77b18-7ca9-4e11-869d-0a1aa66d2057\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-338f2e-6ccbc66d75-htxcd" Oct 31 04:32:22.722204 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.722086 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53f77b18-7ca9-4e11-869d-0a1aa66d2057-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-338f2e-6ccbc66d75-htxcd\" (UID: \"53f77b18-7ca9-4e11-869d-0a1aa66d2057\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-338f2e-6ccbc66d75-htxcd" Oct 31 04:32:22.724285 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.724261 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z"] Oct 31 04:32:22.729796 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.729772 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z" Oct 31 04:32:22.745283 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.745190 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-57dc6d-openshift-default-dockercfg-4gkzk\"" Oct 31 04:32:22.750276 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.750231 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z"] Oct 31 04:32:22.823350 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.823306 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z2ws\" (UniqueName: \"kubernetes.io/projected/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-kube-api-access-4z2ws\") pod \"e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z\" (UID: \"1132f111-a44a-4cc9-97bf-540c4f0dc0a5\") " pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z" Oct 31 04:32:22.823539 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.823419 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-istio-envoy\") pod \"e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z\" (UID: \"1132f111-a44a-4cc9-97bf-540c4f0dc0a5\") " pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z" Oct 31 04:32:22.823539 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.823440 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-istio-data\") pod \"e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z\" (UID: \"1132f111-a44a-4cc9-97bf-540c4f0dc0a5\") " pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z" Oct 31 04:32:22.823654 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.823556 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvc4m\" (UniqueName: \"kubernetes.io/projected/248dfc75-12b3-4535-91d1-5c604ac06498-kube-api-access-kvc4m\") pod \"payload-processing-e2e-shared-a-w3-338f2e-57f7757bf-mchhm\" (UID: \"248dfc75-12b3-4535-91d1-5c604ac06498\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-338f2e-57f7757bf-mchhm" Oct 31 04:32:22.823654 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.823593 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-istio-podinfo\") pod \"e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z\" (UID: \"1132f111-a44a-4cc9-97bf-540c4f0dc0a5\") " pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z" Oct 31 04:32:22.823768 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.823685 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rcqb\" (UniqueName: \"kubernetes.io/projected/53f77b18-7ca9-4e11-869d-0a1aa66d2057-kube-api-access-4rcqb\") pod \"payload-pre-processing-e2e-shared-a-w3-338f2e-6ccbc66d75-htxcd\" (UID: \"53f77b18-7ca9-4e11-869d-0a1aa66d2057\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-338f2e-6ccbc66d75-htxcd" Oct 31 04:32:22.823768 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.823732 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53f77b18-7ca9-4e11-869d-0a1aa66d2057-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-338f2e-6ccbc66d75-htxcd\" (UID: \"53f77b18-7ca9-4e11-869d-0a1aa66d2057\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-338f2e-6ccbc66d75-htxcd" Oct 31 04:32:22.823768 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.823764 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-workload-socket\") pod \"e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z\" (UID: \"1132f111-a44a-4cc9-97bf-540c4f0dc0a5\") " pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z" Oct 31 04:32:22.823926 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.823811 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-credential-socket\") pod \"e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z\" (UID: \"1132f111-a44a-4cc9-97bf-540c4f0dc0a5\") " pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z" Oct 31 04:32:22.823979 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.823951 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-workload-certs\") pod \"e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z\" (UID: \"1132f111-a44a-4cc9-97bf-540c4f0dc0a5\") " pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z" Oct 31 04:32:22.824036 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.824014 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-istiod-ca-cert\") pod \"e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z\" (UID: \"1132f111-a44a-4cc9-97bf-540c4f0dc0a5\") " pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z" Oct 31 04:32:22.824103 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.824086 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/248dfc75-12b3-4535-91d1-5c604ac06498-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-338f2e-57f7757bf-mchhm\" (UID: \"248dfc75-12b3-4535-91d1-5c604ac06498\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-338f2e-57f7757bf-mchhm" Oct 31 04:32:22.824157 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.824129 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-istio-token\") pod \"e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z\" (UID: \"1132f111-a44a-4cc9-97bf-540c4f0dc0a5\") " pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z" Oct 31 04:32:22.824469 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.824449 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53f77b18-7ca9-4e11-869d-0a1aa66d2057-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-338f2e-6ccbc66d75-htxcd\" (UID: \"53f77b18-7ca9-4e11-869d-0a1aa66d2057\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-338f2e-6ccbc66d75-htxcd" Oct 31 04:32:22.850674 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.850638 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rcqb\" (UniqueName: \"kubernetes.io/projected/53f77b18-7ca9-4e11-869d-0a1aa66d2057-kube-api-access-4rcqb\") pod \"payload-pre-processing-e2e-shared-a-w3-338f2e-6ccbc66d75-htxcd\" (UID: \"53f77b18-7ca9-4e11-869d-0a1aa66d2057\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-338f2e-6ccbc66d75-htxcd" Oct 31 04:32:22.907287 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.907219 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-338f2e-6ccbc66d75-htxcd" Oct 31 04:32:22.925713 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.925685 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-istio-envoy\") pod \"e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z\" (UID: \"1132f111-a44a-4cc9-97bf-540c4f0dc0a5\") " pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z" Oct 31 04:32:22.925713 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.925725 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-istio-data\") pod \"e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z\" (UID: \"1132f111-a44a-4cc9-97bf-540c4f0dc0a5\") " pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z" Oct 31 04:32:22.926083 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.925793 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvc4m\" (UniqueName: \"kubernetes.io/projected/248dfc75-12b3-4535-91d1-5c604ac06498-kube-api-access-kvc4m\") pod \"payload-processing-e2e-shared-a-w3-338f2e-57f7757bf-mchhm\" (UID: \"248dfc75-12b3-4535-91d1-5c604ac06498\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-338f2e-57f7757bf-mchhm" Oct 31 04:32:22.926083 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.925820 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-istio-podinfo\") pod \"e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z\" (UID: \"1132f111-a44a-4cc9-97bf-540c4f0dc0a5\") " pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z" Oct 31 04:32:22.926083 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.925887 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-workload-socket\") pod \"e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z\" (UID: \"1132f111-a44a-4cc9-97bf-540c4f0dc0a5\") " pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z" Oct 31 04:32:22.926083 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.925933 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-credential-socket\") pod \"e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z\" (UID: \"1132f111-a44a-4cc9-97bf-540c4f0dc0a5\") " pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z" Oct 31 04:32:22.926083 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.926044 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-workload-certs\") pod \"e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z\" (UID: \"1132f111-a44a-4cc9-97bf-540c4f0dc0a5\") " pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z" Oct 31 04:32:22.926083 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.926096 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-istiod-ca-cert\") pod \"e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z\" (UID: \"1132f111-a44a-4cc9-97bf-540c4f0dc0a5\") " pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z" Oct 31 04:32:22.926450 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.926152 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/248dfc75-12b3-4535-91d1-5c604ac06498-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-338f2e-57f7757bf-mchhm\" (UID: \"248dfc75-12b3-4535-91d1-5c604ac06498\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-338f2e-57f7757bf-mchhm" Oct 31 04:32:22.926450 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.926194 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-istio-token\") pod \"e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z\" (UID: \"1132f111-a44a-4cc9-97bf-540c4f0dc0a5\") " pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z" Oct 31 04:32:22.926450 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.926221 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z2ws\" (UniqueName: \"kubernetes.io/projected/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-kube-api-access-4z2ws\") pod \"e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z\" (UID: \"1132f111-a44a-4cc9-97bf-540c4f0dc0a5\") " pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z" Oct 31 04:32:22.926617 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.926455 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-istio-data\") pod \"e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z\" (UID: \"1132f111-a44a-4cc9-97bf-540c4f0dc0a5\") " pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z" Oct 31 04:32:22.926765 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.926743 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-workload-certs\") pod \"e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z\" (UID: \"1132f111-a44a-4cc9-97bf-540c4f0dc0a5\") " pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z" Oct 31 04:32:22.926902 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.926884 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-workload-socket\") pod \"e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z\" (UID: \"1132f111-a44a-4cc9-97bf-540c4f0dc0a5\") " pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z" Oct 31 04:32:22.927277 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.927027 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-credential-socket\") pod \"e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z\" (UID: \"1132f111-a44a-4cc9-97bf-540c4f0dc0a5\") " pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z" Oct 31 04:32:22.927277 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.927152 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-istiod-ca-cert\") pod \"e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z\" (UID: \"1132f111-a44a-4cc9-97bf-540c4f0dc0a5\") " pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z" Oct 31 04:32:22.927456 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.927387 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/248dfc75-12b3-4535-91d1-5c604ac06498-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-338f2e-57f7757bf-mchhm\" (UID: \"248dfc75-12b3-4535-91d1-5c604ac06498\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-338f2e-57f7757bf-mchhm" Oct 31 04:32:22.928573 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.928551 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-istio-envoy\") pod \"e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z\" (UID: \"1132f111-a44a-4cc9-97bf-540c4f0dc0a5\") " pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z" Oct 31 04:32:22.929762 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.929738 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-istio-podinfo\") pod \"e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z\" (UID: \"1132f111-a44a-4cc9-97bf-540c4f0dc0a5\") " pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z" Oct 31 04:32:22.957367 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.954425 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z2ws\" (UniqueName: \"kubernetes.io/projected/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-kube-api-access-4z2ws\") pod \"e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z\" (UID: \"1132f111-a44a-4cc9-97bf-540c4f0dc0a5\") " pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z" Oct 31 04:32:22.958915 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.958880 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvc4m\" (UniqueName: \"kubernetes.io/projected/248dfc75-12b3-4535-91d1-5c604ac06498-kube-api-access-kvc4m\") pod \"payload-processing-e2e-shared-a-w3-338f2e-57f7757bf-mchhm\" (UID: \"248dfc75-12b3-4535-91d1-5c604ac06498\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-338f2e-57f7757bf-mchhm" Oct 31 04:32:22.961098 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.960160 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-istio-token\") pod \"e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z\" (UID: \"1132f111-a44a-4cc9-97bf-540c4f0dc0a5\") " pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z" Oct 31 04:32:22.971776 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:22.971547 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-338f2e-57f7757bf-mchhm" Oct 31 04:32:23.056349 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:23.054743 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z" Oct 31 04:32:23.364022 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:23.362316 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-338f2e-6ccbc66d75-htxcd"] Oct 31 04:32:23.788367 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:23.788002 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-338f2e-57f7757bf-mchhm"] Oct 31 04:32:23.834880 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:23.834834 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z"] Oct 31 04:32:23.881926 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:23.881164 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:32:23.881926 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:23.881266 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:32:23.881926 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:23.881317 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:32:24.054149 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:24.054111 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-338f2e-57f7757bf-mchhm" event={"ID":"248dfc75-12b3-4535-91d1-5c604ac06498","Type":"ContainerStarted","Data":"82e6f10a302dede9d175fb383eb33f97ceb1f3eb2c5460045c832f42585aa944"} Oct 31 04:32:24.054823 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:24.054804 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-338f2e-57f7757bf-mchhm" Oct 31 04:32:24.056924 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:24.056884 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z" event={"ID":"1132f111-a44a-4cc9-97bf-540c4f0dc0a5","Type":"ContainerStarted","Data":"08dc0ed097de9b499ddef102e066251079af4159272bbc97341d759c5c9ce033"} Oct 31 04:32:24.059186 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:24.059163 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-338f2e-6ccbc66d75-htxcd" event={"ID":"53f77b18-7ca9-4e11-869d-0a1aa66d2057","Type":"ContainerStarted","Data":"7badcff5803238f686bc30f63db2530bff481b08d6668a94ae0a2623a2df8861"} Oct 31 04:32:24.059294 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:24.059203 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-338f2e-6ccbc66d75-htxcd" event={"ID":"53f77b18-7ca9-4e11-869d-0a1aa66d2057","Type":"ContainerStarted","Data":"61497f6d2b1b41e28a1517839ec71e4970fa9fe59b23de3f45ec26459b20b266"} Oct 31 04:32:24.059387 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:24.059297 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-338f2e-6ccbc66d75-htxcd" Oct 31 04:32:24.104381 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:24.101965 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-338f2e-57f7757bf-mchhm" podStartSLOduration=2.101945963 podStartE2EDuration="2.101945963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:32:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:32:24.100100074 +0000 UTC m=+1698.856438126" watchObservedRunningTime="2026-10-31 04:32:24.101945963 +0000 UTC m=+1698.858284000" Oct 31 04:32:24.152248 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:24.148773 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-338f2e-6ccbc66d75-htxcd" podStartSLOduration=2.14874881 podStartE2EDuration="2.14874881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:32:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:32:24.141645017 +0000 UTC m=+1698.897983058" watchObservedRunningTime="2026-10-31 04:32:24.14874881 +0000 UTC m=+1698.905086850" Oct 31 04:32:25.088728 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:25.088676 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-338f2e-57f7757bf-mchhm" event={"ID":"248dfc75-12b3-4535-91d1-5c604ac06498","Type":"ContainerStarted","Data":"f429c47c34413b5a965793d294fb185d27ba0c31a7f67323c08dca502c530375"} Oct 31 04:32:25.103577 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:25.103495 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z" event={"ID":"1132f111-a44a-4cc9-97bf-540c4f0dc0a5","Type":"ContainerStarted","Data":"f3f42daf2ba8bd5b75932b0e38fac44b2c54c9ea6b230874a15f71a6d01cb65c"} Oct 31 04:32:25.184353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:25.182484 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z" podStartSLOduration=3.18246449 podStartE2EDuration="3.18246449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:32:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:32:25.163545935 +0000 UTC m=+1699.919883976" watchObservedRunningTime="2026-10-31 04:32:25.18246449 +0000 UTC m=+1699.938802535" Oct 31 04:32:25.430098 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:25.430009 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-81c810-666bd7cfc5-k862v" podUID="97d7ee87-7714-4ee7-99cc-350ca67dbbe7" containerName="payload-pre-processing" containerID="cri-o://1a120be7059b2cbe70faf0bbd934ef8ae551ae364635a7576d33324411745882" gracePeriod=25 Oct 31 04:32:25.717676 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:25.717650 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-81c810-666bd7cfc5-k862v" Oct 31 04:32:25.834319 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:25.834283 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97d7ee87-7714-4ee7-99cc-350ca67dbbe7-kube-api-access-l268k\" (UniqueName: \"kubernetes.io/projected/97d7ee87-7714-4ee7-99cc-350ca67dbbe7-kube-api-access-l268k\") pod \"97d7ee87-7714-4ee7-99cc-350ca67dbbe7\" (UID: \"97d7ee87-7714-4ee7-99cc-350ca67dbbe7\") " Oct 31 04:32:25.834589 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:25.834571 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97d7ee87-7714-4ee7-99cc-350ca67dbbe7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97d7ee87-7714-4ee7-99cc-350ca67dbbe7-plugins-config-volume\") pod \"97d7ee87-7714-4ee7-99cc-350ca67dbbe7\" (UID: \"97d7ee87-7714-4ee7-99cc-350ca67dbbe7\") " Oct 31 04:32:25.834935 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:25.834914 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97d7ee87-7714-4ee7-99cc-350ca67dbbe7-plugins-config-volume" pod "97d7ee87-7714-4ee7-99cc-350ca67dbbe7" (UID: "97d7ee87-7714-4ee7-99cc-350ca67dbbe7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:32:25.835081 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:25.835067 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97d7ee87-7714-4ee7-99cc-350ca67dbbe7-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:32:25.836383 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:25.836365 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97d7ee87-7714-4ee7-99cc-350ca67dbbe7-kube-api-access-l268k" pod "97d7ee87-7714-4ee7-99cc-350ca67dbbe7" (UID: "97d7ee87-7714-4ee7-99cc-350ca67dbbe7"). InnerVolumeSpecName "kube-api-access-l268k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:32:25.936723 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:25.936649 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l268k\" (UniqueName: \"kubernetes.io/projected/97d7ee87-7714-4ee7-99cc-350ca67dbbe7-kube-api-access-l268k\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:32:26.057362 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:26.057314 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z" Oct 31 04:32:26.062745 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:26.062715 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z" Oct 31 04:32:26.118678 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:26.118643 5019 generic.go:356] "Generic (PLEG): container finished" podID="97d7ee87-7714-4ee7-99cc-350ca67dbbe7" containerID="1a120be7059b2cbe70faf0bbd934ef8ae551ae364635a7576d33324411745882" exitCode=0 Oct 31 04:32:26.119122 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:26.118699 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-81c810-666bd7cfc5-k862v" Oct 31 04:32:26.119122 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:26.118726 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-81c810-666bd7cfc5-k862v" event={"ID":"97d7ee87-7714-4ee7-99cc-350ca67dbbe7","Type":"ContainerDied","Data":"1a120be7059b2cbe70faf0bbd934ef8ae551ae364635a7576d33324411745882"} Oct 31 04:32:26.119122 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:26.118783 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-81c810-666bd7cfc5-k862v" event={"ID":"97d7ee87-7714-4ee7-99cc-350ca67dbbe7","Type":"ContainerDied","Data":"62400b252cc7ec429bc560209e8db2dfbab5150bb961933449344cb3cbc9885c"} Oct 31 04:32:26.119122 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:26.118798 5019 scope.go:122] "RemoveContainer" containerID="1a120be7059b2cbe70faf0bbd934ef8ae551ae364635a7576d33324411745882" Oct 31 04:32:26.119588 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:26.119558 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z" Oct 31 04:32:26.120250 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:26.120230 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z" Oct 31 04:32:26.133572 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:26.133552 5019 scope.go:122] "RemoveContainer" containerID="1a120be7059b2cbe70faf0bbd934ef8ae551ae364635a7576d33324411745882" Oct 31 04:32:26.133895 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:32:26.133876 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a120be7059b2cbe70faf0bbd934ef8ae551ae364635a7576d33324411745882\": container with ID starting with 1a120be7059b2cbe70faf0bbd934ef8ae551ae364635a7576d33324411745882 not found: ID does not exist" containerID="1a120be7059b2cbe70faf0bbd934ef8ae551ae364635a7576d33324411745882" Oct 31 04:32:26.133962 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:26.133905 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a120be7059b2cbe70faf0bbd934ef8ae551ae364635a7576d33324411745882"} err="failed to get container status \"1a120be7059b2cbe70faf0bbd934ef8ae551ae364635a7576d33324411745882\": rpc error: code = NotFound desc = could not find container \"1a120be7059b2cbe70faf0bbd934ef8ae551ae364635a7576d33324411745882\": container with ID starting with 1a120be7059b2cbe70faf0bbd934ef8ae551ae364635a7576d33324411745882 not found: ID does not exist" Oct 31 04:32:26.176794 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:26.176762 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-81c810-666bd7cfc5-k862v"] Oct 31 04:32:26.183546 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:26.183514 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-81c810-666bd7cfc5-k862v"] Oct 31 04:32:27.409545 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:27.409512 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97d7ee87-7714-4ee7-99cc-350ca67dbbe7" path="/var/lib/kubelet/pods/97d7ee87-7714-4ee7-99cc-350ca67dbbe7/volumes" Oct 31 04:32:35.124205 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:35.124171 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-338f2e-6ccbc66d75-htxcd" Oct 31 04:32:36.122564 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:36.122528 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-338f2e-57f7757bf-mchhm" Oct 31 04:32:36.397154 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:36.397065 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnhnx"] Oct 31 04:32:36.397662 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:36.397255 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnhnx" podUID="156f03ac-9365-42af-b18a-30f81ab01b7e" containerName="discovery" containerID="cri-o://2fbd4d02f2286b41d7f24edafdacf39cefdf3c9dc97da394d3cdd53eb5214e4a" gracePeriod=30 Oct 31 04:32:36.706468 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:36.706442 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnhnx" Oct 31 04:32:36.797548 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:36.797514 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/156f03ac-9365-42af-b18a-30f81ab01b7e-cacerts\" (UniqueName: \"kubernetes.io/secret/156f03ac-9365-42af-b18a-30f81ab01b7e-cacerts\") pod \"156f03ac-9365-42af-b18a-30f81ab01b7e\" (UID: \"156f03ac-9365-42af-b18a-30f81ab01b7e\") " Oct 31 04:32:36.797751 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:36.797563 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/156f03ac-9365-42af-b18a-30f81ab01b7e-istio-token\" (UniqueName: \"kubernetes.io/projected/156f03ac-9365-42af-b18a-30f81ab01b7e-istio-token\") pod \"156f03ac-9365-42af-b18a-30f81ab01b7e\" (UID: \"156f03ac-9365-42af-b18a-30f81ab01b7e\") " Oct 31 04:32:36.797751 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:36.797622 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/156f03ac-9365-42af-b18a-30f81ab01b7e-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/156f03ac-9365-42af-b18a-30f81ab01b7e-istio-csr-dns-cert\") pod \"156f03ac-9365-42af-b18a-30f81ab01b7e\" (UID: \"156f03ac-9365-42af-b18a-30f81ab01b7e\") " Oct 31 04:32:36.797751 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:36.797660 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/156f03ac-9365-42af-b18a-30f81ab01b7e-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/156f03ac-9365-42af-b18a-30f81ab01b7e-istio-kubeconfig\") pod \"156f03ac-9365-42af-b18a-30f81ab01b7e\" (UID: \"156f03ac-9365-42af-b18a-30f81ab01b7e\") " Oct 31 04:32:36.797751 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:36.797732 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/156f03ac-9365-42af-b18a-30f81ab01b7e-kube-api-access-dtgfc\" (UniqueName: \"kubernetes.io/projected/156f03ac-9365-42af-b18a-30f81ab01b7e-kube-api-access-dtgfc\") pod \"156f03ac-9365-42af-b18a-30f81ab01b7e\" (UID: \"156f03ac-9365-42af-b18a-30f81ab01b7e\") " Oct 31 04:32:36.798010 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:36.797863 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/156f03ac-9365-42af-b18a-30f81ab01b7e-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/156f03ac-9365-42af-b18a-30f81ab01b7e-local-certs\") pod \"156f03ac-9365-42af-b18a-30f81ab01b7e\" (UID: \"156f03ac-9365-42af-b18a-30f81ab01b7e\") " Oct 31 04:32:36.798071 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:36.798020 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/156f03ac-9365-42af-b18a-30f81ab01b7e-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/156f03ac-9365-42af-b18a-30f81ab01b7e-istio-csr-ca-configmap\") pod \"156f03ac-9365-42af-b18a-30f81ab01b7e\" (UID: \"156f03ac-9365-42af-b18a-30f81ab01b7e\") " Oct 31 04:32:36.798955 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:36.798910 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/156f03ac-9365-42af-b18a-30f81ab01b7e-istio-csr-ca-configmap" pod "156f03ac-9365-42af-b18a-30f81ab01b7e" (UID: "156f03ac-9365-42af-b18a-30f81ab01b7e"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:32:36.801372 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:36.801281 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/156f03ac-9365-42af-b18a-30f81ab01b7e-istio-csr-dns-cert" pod "156f03ac-9365-42af-b18a-30f81ab01b7e" (UID: "156f03ac-9365-42af-b18a-30f81ab01b7e"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:32:36.804798 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:36.804745 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/156f03ac-9365-42af-b18a-30f81ab01b7e-kube-api-access-dtgfc" pod "156f03ac-9365-42af-b18a-30f81ab01b7e" (UID: "156f03ac-9365-42af-b18a-30f81ab01b7e"). InnerVolumeSpecName "kube-api-access-dtgfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:32:36.804798 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:36.804777 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/156f03ac-9365-42af-b18a-30f81ab01b7e-istio-token" pod "156f03ac-9365-42af-b18a-30f81ab01b7e" (UID: "156f03ac-9365-42af-b18a-30f81ab01b7e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:32:36.805160 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:36.805070 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/156f03ac-9365-42af-b18a-30f81ab01b7e-istio-kubeconfig" pod "156f03ac-9365-42af-b18a-30f81ab01b7e" (UID: "156f03ac-9365-42af-b18a-30f81ab01b7e"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:32:36.805160 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:36.805138 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/156f03ac-9365-42af-b18a-30f81ab01b7e-cacerts" pod "156f03ac-9365-42af-b18a-30f81ab01b7e" (UID: "156f03ac-9365-42af-b18a-30f81ab01b7e"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:32:36.806915 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:36.806891 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/156f03ac-9365-42af-b18a-30f81ab01b7e-local-certs" pod "156f03ac-9365-42af-b18a-30f81ab01b7e" (UID: "156f03ac-9365-42af-b18a-30f81ab01b7e"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:32:36.899948 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:36.899915 5019 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/156f03ac-9365-42af-b18a-30f81ab01b7e-cacerts\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:32:36.899948 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:36.899943 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/156f03ac-9365-42af-b18a-30f81ab01b7e-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:32:36.899948 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:36.899953 5019 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/156f03ac-9365-42af-b18a-30f81ab01b7e-istio-csr-dns-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:32:36.900197 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:36.899962 5019 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/156f03ac-9365-42af-b18a-30f81ab01b7e-istio-kubeconfig\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:32:36.900197 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:36.899973 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtgfc\" (UniqueName: \"kubernetes.io/projected/156f03ac-9365-42af-b18a-30f81ab01b7e-kube-api-access-dtgfc\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:32:36.900197 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:36.899984 5019 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/156f03ac-9365-42af-b18a-30f81ab01b7e-local-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:32:36.900197 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:36.899993 5019 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/156f03ac-9365-42af-b18a-30f81ab01b7e-istio-csr-ca-configmap\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:32:37.250898 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:37.250848 5019 generic.go:356] "Generic (PLEG): container finished" podID="156f03ac-9365-42af-b18a-30f81ab01b7e" containerID="2fbd4d02f2286b41d7f24edafdacf39cefdf3c9dc97da394d3cdd53eb5214e4a" exitCode=0 Oct 31 04:32:37.250898 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:37.250893 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnhnx" event={"ID":"156f03ac-9365-42af-b18a-30f81ab01b7e","Type":"ContainerDied","Data":"2fbd4d02f2286b41d7f24edafdacf39cefdf3c9dc97da394d3cdd53eb5214e4a"} Oct 31 04:32:37.251172 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:37.250909 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnhnx" Oct 31 04:32:37.251172 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:37.250955 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnhnx" event={"ID":"156f03ac-9365-42af-b18a-30f81ab01b7e","Type":"ContainerDied","Data":"5f77de6d47e2a9bbcccc915f18a69f7f556b9250345b947268d2d73e501cf89b"} Oct 31 04:32:37.251172 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:37.250981 5019 scope.go:122] "RemoveContainer" containerID="2fbd4d02f2286b41d7f24edafdacf39cefdf3c9dc97da394d3cdd53eb5214e4a" Oct 31 04:32:37.266642 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:37.266620 5019 scope.go:122] "RemoveContainer" containerID="2fbd4d02f2286b41d7f24edafdacf39cefdf3c9dc97da394d3cdd53eb5214e4a" Oct 31 04:32:37.266922 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:32:37.266901 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fbd4d02f2286b41d7f24edafdacf39cefdf3c9dc97da394d3cdd53eb5214e4a\": container with ID starting with 2fbd4d02f2286b41d7f24edafdacf39cefdf3c9dc97da394d3cdd53eb5214e4a not found: ID does not exist" containerID="2fbd4d02f2286b41d7f24edafdacf39cefdf3c9dc97da394d3cdd53eb5214e4a" Oct 31 04:32:37.266971 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:37.266930 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fbd4d02f2286b41d7f24edafdacf39cefdf3c9dc97da394d3cdd53eb5214e4a"} err="failed to get container status \"2fbd4d02f2286b41d7f24edafdacf39cefdf3c9dc97da394d3cdd53eb5214e4a\": rpc error: code = NotFound desc = could not find container \"2fbd4d02f2286b41d7f24edafdacf39cefdf3c9dc97da394d3cdd53eb5214e4a\": container with ID starting with 2fbd4d02f2286b41d7f24edafdacf39cefdf3c9dc97da394d3cdd53eb5214e4a not found: ID does not exist" Oct 31 04:32:37.317971 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:37.317943 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnhnx"] Oct 31 04:32:37.330141 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:37.330113 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnhnx"] Oct 31 04:32:37.432179 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:37.432137 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="156f03ac-9365-42af-b18a-30f81ab01b7e" path="/var/lib/kubelet/pods/156f03ac-9365-42af-b18a-30f81ab01b7e/volumes" Oct 31 04:32:45.722409 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:45.721793 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s"] Oct 31 04:32:45.722409 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:45.722054 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s" podUID="9eb20738-52aa-4cc7-990a-ddd8c704e683" containerName="istio-proxy" containerID="cri-o://2fd2b448c613cc7efef4751099580e39ad6c7484d89578a86c30a80cf22ba576" gracePeriod=30 Oct 31 04:32:46.539592 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:46.539556 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f55a7cc5-6vpwm"] Oct 31 04:32:46.540214 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:46.540199 5019 cpu_manager.go:414] "containerMap: removing container" podUID="156f03ac-9365-42af-b18a-30f81ab01b7e" containerName="discovery" Oct 31 04:32:46.540214 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:46.540214 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="156f03ac-9365-42af-b18a-30f81ab01b7e" containerName="discovery" Oct 31 04:32:46.540421 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:46.540251 5019 cpu_manager.go:414] "containerMap: removing container" podUID="97d7ee87-7714-4ee7-99cc-350ca67dbbe7" containerName="payload-pre-processing" Oct 31 04:32:46.540421 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:46.540257 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97d7ee87-7714-4ee7-99cc-350ca67dbbe7" containerName="payload-pre-processing" Oct 31 04:32:46.540421 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:46.540360 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="156f03ac-9365-42af-b18a-30f81ab01b7e" containerName="discovery" Oct 31 04:32:46.540421 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:46.540375 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="97d7ee87-7714-4ee7-99cc-350ca67dbbe7" containerName="payload-pre-processing" Oct 31 04:32:46.545564 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:46.545545 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f55a7cc5-6vpwm" Oct 31 04:32:46.564848 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:46.563881 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f55a7cc5-6vpwm"] Oct 31 04:32:46.569886 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:46.569848 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:32:46.570045 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:46.569932 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:32:46.570045 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:46.569971 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:32:46.570045 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:46.569986 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:32:46.654312 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:46.654281 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c62cf027-5252-4913-933e-31d98e74eb01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f55a7cc5-6vpwm\" (UID: \"c62cf027-5252-4913-933e-31d98e74eb01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f55a7cc5-6vpwm" Oct 31 04:32:46.654517 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:46.654456 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sbqp\" (UniqueName: \"kubernetes.io/projected/c62cf027-5252-4913-933e-31d98e74eb01-kube-api-access-5sbqp\") pod \"maas-api-revoke-sub-f55a7cc5-6vpwm\" (UID: \"c62cf027-5252-4913-933e-31d98e74eb01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f55a7cc5-6vpwm" Oct 31 04:32:46.654517 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:46.654514 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c62cf027-5252-4913-933e-31d98e74eb01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f55a7cc5-6vpwm\" (UID: \"c62cf027-5252-4913-933e-31d98e74eb01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f55a7cc5-6vpwm" Oct 31 04:32:46.756421 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:46.756385 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sbqp\" (UniqueName: \"kubernetes.io/projected/c62cf027-5252-4913-933e-31d98e74eb01-kube-api-access-5sbqp\") pod \"maas-api-revoke-sub-f55a7cc5-6vpwm\" (UID: \"c62cf027-5252-4913-933e-31d98e74eb01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f55a7cc5-6vpwm" Oct 31 04:32:46.756879 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:46.756448 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c62cf027-5252-4913-933e-31d98e74eb01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f55a7cc5-6vpwm\" (UID: \"c62cf027-5252-4913-933e-31d98e74eb01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f55a7cc5-6vpwm" Oct 31 04:32:46.756879 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:46.756575 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c62cf027-5252-4913-933e-31d98e74eb01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f55a7cc5-6vpwm\" (UID: \"c62cf027-5252-4913-933e-31d98e74eb01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f55a7cc5-6vpwm" Oct 31 04:32:46.757464 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:46.757439 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c62cf027-5252-4913-933e-31d98e74eb01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f55a7cc5-6vpwm\" (UID: \"c62cf027-5252-4913-933e-31d98e74eb01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f55a7cc5-6vpwm" Oct 31 04:32:46.758934 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:46.758916 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c62cf027-5252-4913-933e-31d98e74eb01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f55a7cc5-6vpwm\" (UID: \"c62cf027-5252-4913-933e-31d98e74eb01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f55a7cc5-6vpwm" Oct 31 04:32:46.780657 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:46.780630 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sbqp\" (UniqueName: \"kubernetes.io/projected/c62cf027-5252-4913-933e-31d98e74eb01-kube-api-access-5sbqp\") pod \"maas-api-revoke-sub-f55a7cc5-6vpwm\" (UID: \"c62cf027-5252-4913-933e-31d98e74eb01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f55a7cc5-6vpwm" Oct 31 04:32:46.864730 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:46.864660 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f55a7cc5-6vpwm" Oct 31 04:32:47.181915 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:47.181877 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f55a7cc5-6vpwm"] Oct 31 04:32:47.183671 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:32:47.183640 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc62cf027_5252_4913_933e_31d98e74eb01.slice/crio-e5ecf557e2730f792914cdb3382c71d5e35aa9aa396b79d3f0ca6d458180252a WatchSource:0}: Error finding container e5ecf557e2730f792914cdb3382c71d5e35aa9aa396b79d3f0ca6d458180252a: Status 404 returned error can't find the container with id e5ecf557e2730f792914cdb3382c71d5e35aa9aa396b79d3f0ca6d458180252a Oct 31 04:32:47.561708 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:47.561675 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f55a7cc5-6vpwm" event={"ID":"c62cf027-5252-4913-933e-31d98e74eb01","Type":"ContainerStarted","Data":"e5ecf557e2730f792914cdb3382c71d5e35aa9aa396b79d3f0ca6d458180252a"} Oct 31 04:32:48.572675 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:48.572587 5019 generic.go:356] "Generic (PLEG): container finished" podID="c62cf027-5252-4913-933e-31d98e74eb01" containerID="cc00f52184f4fbccd1e6c56314d9e32013e6aabfe3a65d44ca2c670f7efbf0d8" exitCode=0 Oct 31 04:32:48.573108 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:48.572669 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f55a7cc5-6vpwm" event={"ID":"c62cf027-5252-4913-933e-31d98e74eb01","Type":"ContainerDied","Data":"cc00f52184f4fbccd1e6c56314d9e32013e6aabfe3a65d44ca2c670f7efbf0d8"} Oct 31 04:32:49.760162 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:49.760139 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f55a7cc5-6vpwm" Oct 31 04:32:49.799169 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:49.799140 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c62cf027-5252-4913-933e-31d98e74eb01-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c62cf027-5252-4913-933e-31d98e74eb01-maas-api-xks-ca\") pod \"c62cf027-5252-4913-933e-31d98e74eb01\" (UID: \"c62cf027-5252-4913-933e-31d98e74eb01\") " Oct 31 04:32:49.799285 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:49.799232 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c62cf027-5252-4913-933e-31d98e74eb01-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c62cf027-5252-4913-933e-31d98e74eb01-maas-api-openshift-ca\") pod \"c62cf027-5252-4913-933e-31d98e74eb01\" (UID: \"c62cf027-5252-4913-933e-31d98e74eb01\") " Oct 31 04:32:49.799451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:49.799431 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c62cf027-5252-4913-933e-31d98e74eb01-kube-api-access-5sbqp\" (UniqueName: \"kubernetes.io/projected/c62cf027-5252-4913-933e-31d98e74eb01-kube-api-access-5sbqp\") pod \"c62cf027-5252-4913-933e-31d98e74eb01\" (UID: \"c62cf027-5252-4913-933e-31d98e74eb01\") " Oct 31 04:32:49.799635 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:49.799616 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c62cf027-5252-4913-933e-31d98e74eb01-maas-api-openshift-ca" pod "c62cf027-5252-4913-933e-31d98e74eb01" (UID: "c62cf027-5252-4913-933e-31d98e74eb01"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:32:49.799986 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:49.799967 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c62cf027-5252-4913-933e-31d98e74eb01-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:32:49.802063 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:49.801891 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c62cf027-5252-4913-933e-31d98e74eb01-kube-api-access-5sbqp" pod "c62cf027-5252-4913-933e-31d98e74eb01" (UID: "c62cf027-5252-4913-933e-31d98e74eb01"). InnerVolumeSpecName "kube-api-access-5sbqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:32:49.802063 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:49.802002 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c62cf027-5252-4913-933e-31d98e74eb01-maas-api-xks-ca" pod "c62cf027-5252-4913-933e-31d98e74eb01" (UID: "c62cf027-5252-4913-933e-31d98e74eb01"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:32:49.901153 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:49.901073 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sbqp\" (UniqueName: \"kubernetes.io/projected/c62cf027-5252-4913-933e-31d98e74eb01-kube-api-access-5sbqp\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:32:49.901153 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:49.901110 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c62cf027-5252-4913-933e-31d98e74eb01-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:32:50.596138 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:50.596106 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f55a7cc5-6vpwm" Oct 31 04:32:50.596461 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:50.596135 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f55a7cc5-6vpwm" event={"ID":"c62cf027-5252-4913-933e-31d98e74eb01","Type":"ContainerDied","Data":"e5ecf557e2730f792914cdb3382c71d5e35aa9aa396b79d3f0ca6d458180252a"} Oct 31 04:32:50.596461 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:50.596188 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5ecf557e2730f792914cdb3382c71d5e35aa9aa396b79d3f0ca6d458180252a" Oct 31 04:32:51.110240 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:51.110181 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s" Oct 31 04:32:51.217708 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:51.217672 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eb20738-52aa-4cc7-990a-ddd8c704e683-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb20738-52aa-4cc7-990a-ddd8c704e683-workload-socket\") pod \"9eb20738-52aa-4cc7-990a-ddd8c704e683\" (UID: \"9eb20738-52aa-4cc7-990a-ddd8c704e683\") " Oct 31 04:32:51.217987 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:51.217963 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb20738-52aa-4cc7-990a-ddd8c704e683-workload-socket" pod "9eb20738-52aa-4cc7-990a-ddd8c704e683" (UID: "9eb20738-52aa-4cc7-990a-ddd8c704e683"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:32:51.218465 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:51.218441 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9eb20738-52aa-4cc7-990a-ddd8c704e683-istiod-ca-cert" pod "9eb20738-52aa-4cc7-990a-ddd8c704e683" (UID: "9eb20738-52aa-4cc7-990a-ddd8c704e683"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:32:51.218577 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:51.218064 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9eb20738-52aa-4cc7-990a-ddd8c704e683-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eb20738-52aa-4cc7-990a-ddd8c704e683-istiod-ca-cert\") pod \"9eb20738-52aa-4cc7-990a-ddd8c704e683\" (UID: \"9eb20738-52aa-4cc7-990a-ddd8c704e683\") " Oct 31 04:32:51.218577 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:51.218531 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eb20738-52aa-4cc7-990a-ddd8c704e683-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb20738-52aa-4cc7-990a-ddd8c704e683-credential-socket\") pod \"9eb20738-52aa-4cc7-990a-ddd8c704e683\" (UID: \"9eb20738-52aa-4cc7-990a-ddd8c704e683\") " Oct 31 04:32:51.218837 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:51.218817 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb20738-52aa-4cc7-990a-ddd8c704e683-credential-socket" pod "9eb20738-52aa-4cc7-990a-ddd8c704e683" (UID: "9eb20738-52aa-4cc7-990a-ddd8c704e683"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:32:51.218908 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:51.218891 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eb20738-52aa-4cc7-990a-ddd8c704e683-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eb20738-52aa-4cc7-990a-ddd8c704e683-istio-data\") pod \"9eb20738-52aa-4cc7-990a-ddd8c704e683\" (UID: \"9eb20738-52aa-4cc7-990a-ddd8c704e683\") " Oct 31 04:32:51.219148 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:51.219130 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb20738-52aa-4cc7-990a-ddd8c704e683-istio-data" pod "9eb20738-52aa-4cc7-990a-ddd8c704e683" (UID: "9eb20738-52aa-4cc7-990a-ddd8c704e683"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:32:51.219574 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:51.219551 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb20738-52aa-4cc7-990a-ddd8c704e683-workload-certs" pod "9eb20738-52aa-4cc7-990a-ddd8c704e683" (UID: "9eb20738-52aa-4cc7-990a-ddd8c704e683"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:32:51.219669 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:51.219596 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eb20738-52aa-4cc7-990a-ddd8c704e683-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eb20738-52aa-4cc7-990a-ddd8c704e683-workload-certs\") pod \"9eb20738-52aa-4cc7-990a-ddd8c704e683\" (UID: \"9eb20738-52aa-4cc7-990a-ddd8c704e683\") " Oct 31 04:32:51.219726 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:51.219695 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9eb20738-52aa-4cc7-990a-ddd8c704e683-istio-token\" (UniqueName: \"kubernetes.io/projected/9eb20738-52aa-4cc7-990a-ddd8c704e683-istio-token\") pod \"9eb20738-52aa-4cc7-990a-ddd8c704e683\" (UID: \"9eb20738-52aa-4cc7-990a-ddd8c704e683\") " Oct 31 04:32:51.220138 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:51.220094 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9eb20738-52aa-4cc7-990a-ddd8c704e683-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eb20738-52aa-4cc7-990a-ddd8c704e683-istio-podinfo\") pod \"9eb20738-52aa-4cc7-990a-ddd8c704e683\" (UID: \"9eb20738-52aa-4cc7-990a-ddd8c704e683\") " Oct 31 04:32:51.220214 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:51.220184 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9eb20738-52aa-4cc7-990a-ddd8c704e683-kube-api-access-bj8bz\" (UniqueName: \"kubernetes.io/projected/9eb20738-52aa-4cc7-990a-ddd8c704e683-kube-api-access-bj8bz\") pod \"9eb20738-52aa-4cc7-990a-ddd8c704e683\" (UID: \"9eb20738-52aa-4cc7-990a-ddd8c704e683\") " Oct 31 04:32:51.220347 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:51.220316 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eb20738-52aa-4cc7-990a-ddd8c704e683-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eb20738-52aa-4cc7-990a-ddd8c704e683-istio-envoy\") pod \"9eb20738-52aa-4cc7-990a-ddd8c704e683\" (UID: \"9eb20738-52aa-4cc7-990a-ddd8c704e683\") " Oct 31 04:32:51.220946 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:51.220918 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb20738-52aa-4cc7-990a-ddd8c704e683-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:32:51.221054 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:51.220949 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eb20738-52aa-4cc7-990a-ddd8c704e683-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:32:51.221054 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:51.220963 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb20738-52aa-4cc7-990a-ddd8c704e683-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:32:51.221054 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:51.220976 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eb20738-52aa-4cc7-990a-ddd8c704e683-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:32:51.221054 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:51.220991 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eb20738-52aa-4cc7-990a-ddd8c704e683-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:32:51.223648 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:51.223625 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eb20738-52aa-4cc7-990a-ddd8c704e683-istio-token" pod "9eb20738-52aa-4cc7-990a-ddd8c704e683" (UID: "9eb20738-52aa-4cc7-990a-ddd8c704e683"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:32:51.227634 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:51.227610 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb20738-52aa-4cc7-990a-ddd8c704e683-istio-envoy" pod "9eb20738-52aa-4cc7-990a-ddd8c704e683" (UID: "9eb20738-52aa-4cc7-990a-ddd8c704e683"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:32:51.227718 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:51.227646 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eb20738-52aa-4cc7-990a-ddd8c704e683-kube-api-access-bj8bz" pod "9eb20738-52aa-4cc7-990a-ddd8c704e683" (UID: "9eb20738-52aa-4cc7-990a-ddd8c704e683"). InnerVolumeSpecName "kube-api-access-bj8bz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:32:51.229049 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:51.229025 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9eb20738-52aa-4cc7-990a-ddd8c704e683-istio-podinfo" pod "9eb20738-52aa-4cc7-990a-ddd8c704e683" (UID: "9eb20738-52aa-4cc7-990a-ddd8c704e683"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:32:51.322445 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:51.322412 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eb20738-52aa-4cc7-990a-ddd8c704e683-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:32:51.322445 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:51.322442 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj8bz\" (UniqueName: \"kubernetes.io/projected/9eb20738-52aa-4cc7-990a-ddd8c704e683-kube-api-access-bj8bz\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:32:51.322445 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:51.322452 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eb20738-52aa-4cc7-990a-ddd8c704e683-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:32:51.322669 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:51.322460 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eb20738-52aa-4cc7-990a-ddd8c704e683-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:32:51.606699 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:51.606661 5019 generic.go:356] "Generic (PLEG): container finished" podID="9eb20738-52aa-4cc7-990a-ddd8c704e683" containerID="2fd2b448c613cc7efef4751099580e39ad6c7484d89578a86c30a80cf22ba576" exitCode=0 Oct 31 04:32:51.606873 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:51.606714 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s" Oct 31 04:32:51.606873 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:51.606736 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s" event={"ID":"9eb20738-52aa-4cc7-990a-ddd8c704e683","Type":"ContainerDied","Data":"2fd2b448c613cc7efef4751099580e39ad6c7484d89578a86c30a80cf22ba576"} Oct 31 04:32:51.606873 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:51.606812 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s" event={"ID":"9eb20738-52aa-4cc7-990a-ddd8c704e683","Type":"ContainerDied","Data":"a245874a62a57850ef1d2f1b7a663271d42a68128d44df4ecc87355e267c6d97"} Oct 31 04:32:51.606873 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:51.606846 5019 scope.go:122] "RemoveContainer" containerID="2fd2b448c613cc7efef4751099580e39ad6c7484d89578a86c30a80cf22ba576" Oct 31 04:32:51.617820 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:51.617802 5019 scope.go:122] "RemoveContainer" containerID="0e312c3b42b8ad05f96fb8246c9830331d8048aae53047546848409bf98da983" Oct 31 04:32:51.627632 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:51.627607 5019 scope.go:122] "RemoveContainer" containerID="2fd2b448c613cc7efef4751099580e39ad6c7484d89578a86c30a80cf22ba576" Oct 31 04:32:51.627917 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:32:51.627897 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fd2b448c613cc7efef4751099580e39ad6c7484d89578a86c30a80cf22ba576\": container with ID starting with 2fd2b448c613cc7efef4751099580e39ad6c7484d89578a86c30a80cf22ba576 not found: ID does not exist" containerID="2fd2b448c613cc7efef4751099580e39ad6c7484d89578a86c30a80cf22ba576" Oct 31 04:32:51.627981 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:51.627923 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fd2b448c613cc7efef4751099580e39ad6c7484d89578a86c30a80cf22ba576"} err="failed to get container status \"2fd2b448c613cc7efef4751099580e39ad6c7484d89578a86c30a80cf22ba576\": rpc error: code = NotFound desc = could not find container \"2fd2b448c613cc7efef4751099580e39ad6c7484d89578a86c30a80cf22ba576\": container with ID starting with 2fd2b448c613cc7efef4751099580e39ad6c7484d89578a86c30a80cf22ba576 not found: ID does not exist" Oct 31 04:32:51.627981 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:51.627941 5019 scope.go:122] "RemoveContainer" containerID="0e312c3b42b8ad05f96fb8246c9830331d8048aae53047546848409bf98da983" Oct 31 04:32:51.628193 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:32:51.628172 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e312c3b42b8ad05f96fb8246c9830331d8048aae53047546848409bf98da983\": container with ID starting with 0e312c3b42b8ad05f96fb8246c9830331d8048aae53047546848409bf98da983 not found: ID does not exist" containerID="0e312c3b42b8ad05f96fb8246c9830331d8048aae53047546848409bf98da983" Oct 31 04:32:51.628272 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:51.628198 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e312c3b42b8ad05f96fb8246c9830331d8048aae53047546848409bf98da983"} err="failed to get container status \"0e312c3b42b8ad05f96fb8246c9830331d8048aae53047546848409bf98da983\": rpc error: code = NotFound desc = could not find container \"0e312c3b42b8ad05f96fb8246c9830331d8048aae53047546848409bf98da983\": container with ID starting with 0e312c3b42b8ad05f96fb8246c9830331d8048aae53047546848409bf98da983 not found: ID does not exist" Oct 31 04:32:51.663459 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:51.663432 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s"] Oct 31 04:32:51.673028 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:51.672979 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-e6a0de-openshift-default-856c98d7d5-9pq9s"] Oct 31 04:32:52.398363 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:32:52.395507 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:32:52.986023 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:52.985989 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e299b1ef-str2b"] Oct 31 04:32:52.986704 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:52.986683 5019 cpu_manager.go:414] "containerMap: removing container" podUID="9eb20738-52aa-4cc7-990a-ddd8c704e683" containerName="istio-proxy" Oct 31 04:32:52.986828 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:52.986704 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eb20738-52aa-4cc7-990a-ddd8c704e683" containerName="istio-proxy" Oct 31 04:32:52.986828 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:52.986743 5019 cpu_manager.go:414] "containerMap: removing container" podUID="c62cf027-5252-4913-933e-31d98e74eb01" containerName="revoke-keys" Oct 31 04:32:52.986828 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:52.986752 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c62cf027-5252-4913-933e-31d98e74eb01" containerName="revoke-keys" Oct 31 04:32:52.987002 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:52.986905 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="9eb20738-52aa-4cc7-990a-ddd8c704e683" containerName="istio-proxy" Oct 31 04:32:52.987002 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:52.986938 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="c62cf027-5252-4913-933e-31d98e74eb01" containerName="revoke-keys" Oct 31 04:32:52.992403 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:52.992381 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e299b1ef-str2b" Oct 31 04:32:53.007101 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:53.007076 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:32:53.007310 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:53.007255 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:32:53.011410 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:53.011391 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e299b1ef-str2b"] Oct 31 04:32:53.018369 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:53.017668 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:32:53.018369 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:53.017732 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:32:53.051839 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:53.051812 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fc8ddf7-db96-4b79-a18d-afd173264640-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e299b1ef-str2b\" (UID: \"8fc8ddf7-db96-4b79-a18d-afd173264640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e299b1ef-str2b" Oct 31 04:32:53.051987 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:53.051888 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fc8ddf7-db96-4b79-a18d-afd173264640-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e299b1ef-str2b\" (UID: \"8fc8ddf7-db96-4b79-a18d-afd173264640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e299b1ef-str2b" Oct 31 04:32:53.052345 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:53.052051 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw55c\" (UniqueName: \"kubernetes.io/projected/8fc8ddf7-db96-4b79-a18d-afd173264640-kube-api-access-fw55c\") pod \"maas-api-revoke-sub-e299b1ef-str2b\" (UID: \"8fc8ddf7-db96-4b79-a18d-afd173264640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e299b1ef-str2b" Oct 31 04:32:53.153234 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:53.153204 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fc8ddf7-db96-4b79-a18d-afd173264640-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e299b1ef-str2b\" (UID: \"8fc8ddf7-db96-4b79-a18d-afd173264640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e299b1ef-str2b" Oct 31 04:32:53.153451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:53.153267 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fc8ddf7-db96-4b79-a18d-afd173264640-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e299b1ef-str2b\" (UID: \"8fc8ddf7-db96-4b79-a18d-afd173264640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e299b1ef-str2b" Oct 31 04:32:53.153526 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:53.153448 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw55c\" (UniqueName: \"kubernetes.io/projected/8fc8ddf7-db96-4b79-a18d-afd173264640-kube-api-access-fw55c\") pod \"maas-api-revoke-sub-e299b1ef-str2b\" (UID: \"8fc8ddf7-db96-4b79-a18d-afd173264640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e299b1ef-str2b" Oct 31 04:32:53.153954 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:53.153934 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fc8ddf7-db96-4b79-a18d-afd173264640-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e299b1ef-str2b\" (UID: \"8fc8ddf7-db96-4b79-a18d-afd173264640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e299b1ef-str2b" Oct 31 04:32:53.155816 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:53.155795 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fc8ddf7-db96-4b79-a18d-afd173264640-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e299b1ef-str2b\" (UID: \"8fc8ddf7-db96-4b79-a18d-afd173264640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e299b1ef-str2b" Oct 31 04:32:53.176273 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:53.176253 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw55c\" (UniqueName: \"kubernetes.io/projected/8fc8ddf7-db96-4b79-a18d-afd173264640-kube-api-access-fw55c\") pod \"maas-api-revoke-sub-e299b1ef-str2b\" (UID: \"8fc8ddf7-db96-4b79-a18d-afd173264640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e299b1ef-str2b" Oct 31 04:32:53.308727 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:53.308651 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e299b1ef-str2b" Oct 31 04:32:53.409753 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:53.409707 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9eb20738-52aa-4cc7-990a-ddd8c704e683" path="/var/lib/kubelet/pods/9eb20738-52aa-4cc7-990a-ddd8c704e683/volumes" Oct 31 04:32:53.611749 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:32:53.611715 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fc8ddf7_db96_4b79_a18d_afd173264640.slice/crio-c7547f26d271003c36e88c3576c25fd1d181a3207572486062bf725e5d3eaffd WatchSource:0}: Error finding container c7547f26d271003c36e88c3576c25fd1d181a3207572486062bf725e5d3eaffd: Status 404 returned error can't find the container with id c7547f26d271003c36e88c3576c25fd1d181a3207572486062bf725e5d3eaffd Oct 31 04:32:53.612763 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:53.612738 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e299b1ef-str2b"] Oct 31 04:32:53.625990 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:53.625964 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e299b1ef-str2b" event={"ID":"8fc8ddf7-db96-4b79-a18d-afd173264640","Type":"ContainerStarted","Data":"c7547f26d271003c36e88c3576c25fd1d181a3207572486062bf725e5d3eaffd"} Oct 31 04:32:54.635342 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:54.635293 5019 generic.go:356] "Generic (PLEG): container finished" podID="8fc8ddf7-db96-4b79-a18d-afd173264640" containerID="1ffe1b04522ca5bea64a747fab2f0df947208ba88bfc2024c72a7188216e64b7" exitCode=0 Oct 31 04:32:54.635734 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:54.635379 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e299b1ef-str2b" event={"ID":"8fc8ddf7-db96-4b79-a18d-afd173264640","Type":"ContainerDied","Data":"1ffe1b04522ca5bea64a747fab2f0df947208ba88bfc2024c72a7188216e64b7"} Oct 31 04:32:55.032182 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:55.032148 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7"] Oct 31 04:32:55.033036 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:55.033011 5019 cpu_manager.go:414] "containerMap: removing container" podUID="9eb20738-52aa-4cc7-990a-ddd8c704e683" containerName="istio-proxy" Oct 31 04:32:55.033036 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:55.033034 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eb20738-52aa-4cc7-990a-ddd8c704e683" containerName="istio-proxy" Oct 31 04:32:55.033222 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:55.033201 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="9eb20738-52aa-4cc7-990a-ddd8c704e683" containerName="istio-proxy" Oct 31 04:32:55.037189 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:55.037167 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7" Oct 31 04:32:55.064989 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:55.064963 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7"] Oct 31 04:32:55.065610 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:55.065587 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-031745-openshift-default-dockercfg-q5sr5\"" Oct 31 04:32:55.182317 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:55.182277 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5581a822-03f0-4f8c-bd5e-444274a4bd9b-credential-socket\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7\" (UID: \"5581a822-03f0-4f8c-bd5e-444274a4bd9b\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7" Oct 31 04:32:55.182531 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:55.182409 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5581a822-03f0-4f8c-bd5e-444274a4bd9b-istio-data\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7\" (UID: \"5581a822-03f0-4f8c-bd5e-444274a4bd9b\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7" Oct 31 04:32:55.182531 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:55.182473 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5581a822-03f0-4f8c-bd5e-444274a4bd9b-workload-socket\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7\" (UID: \"5581a822-03f0-4f8c-bd5e-444274a4bd9b\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7" Oct 31 04:32:55.182531 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:55.182492 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5581a822-03f0-4f8c-bd5e-444274a4bd9b-istio-envoy\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7\" (UID: \"5581a822-03f0-4f8c-bd5e-444274a4bd9b\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7" Oct 31 04:32:55.182704 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:55.182530 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5581a822-03f0-4f8c-bd5e-444274a4bd9b-istiod-ca-cert\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7\" (UID: \"5581a822-03f0-4f8c-bd5e-444274a4bd9b\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7" Oct 31 04:32:55.182704 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:55.182642 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5581a822-03f0-4f8c-bd5e-444274a4bd9b-istio-podinfo\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7\" (UID: \"5581a822-03f0-4f8c-bd5e-444274a4bd9b\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7" Oct 31 04:32:55.182704 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:55.182664 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqcqz\" (UniqueName: \"kubernetes.io/projected/5581a822-03f0-4f8c-bd5e-444274a4bd9b-kube-api-access-fqcqz\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7\" (UID: \"5581a822-03f0-4f8c-bd5e-444274a4bd9b\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7" Oct 31 04:32:55.182879 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:55.182767 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5581a822-03f0-4f8c-bd5e-444274a4bd9b-istio-token\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7\" (UID: \"5581a822-03f0-4f8c-bd5e-444274a4bd9b\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7" Oct 31 04:32:55.182879 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:55.182812 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5581a822-03f0-4f8c-bd5e-444274a4bd9b-workload-certs\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7\" (UID: \"5581a822-03f0-4f8c-bd5e-444274a4bd9b\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7" Oct 31 04:32:55.284407 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:55.284291 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5581a822-03f0-4f8c-bd5e-444274a4bd9b-workload-socket\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7\" (UID: \"5581a822-03f0-4f8c-bd5e-444274a4bd9b\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7" Oct 31 04:32:55.284407 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:55.284384 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5581a822-03f0-4f8c-bd5e-444274a4bd9b-istio-envoy\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7\" (UID: \"5581a822-03f0-4f8c-bd5e-444274a4bd9b\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7" Oct 31 04:32:55.284654 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:55.284430 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5581a822-03f0-4f8c-bd5e-444274a4bd9b-istiod-ca-cert\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7\" (UID: \"5581a822-03f0-4f8c-bd5e-444274a4bd9b\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7" Oct 31 04:32:55.284654 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:55.284543 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5581a822-03f0-4f8c-bd5e-444274a4bd9b-istio-podinfo\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7\" (UID: \"5581a822-03f0-4f8c-bd5e-444274a4bd9b\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7" Oct 31 04:32:55.284654 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:55.284573 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqcqz\" (UniqueName: \"kubernetes.io/projected/5581a822-03f0-4f8c-bd5e-444274a4bd9b-kube-api-access-fqcqz\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7\" (UID: \"5581a822-03f0-4f8c-bd5e-444274a4bd9b\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7" Oct 31 04:32:55.284816 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:55.284662 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5581a822-03f0-4f8c-bd5e-444274a4bd9b-istio-token\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7\" (UID: \"5581a822-03f0-4f8c-bd5e-444274a4bd9b\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7" Oct 31 04:32:55.284816 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:55.284697 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5581a822-03f0-4f8c-bd5e-444274a4bd9b-workload-certs\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7\" (UID: \"5581a822-03f0-4f8c-bd5e-444274a4bd9b\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7" Oct 31 04:32:55.284816 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:55.284790 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5581a822-03f0-4f8c-bd5e-444274a4bd9b-credential-socket\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7\" (UID: \"5581a822-03f0-4f8c-bd5e-444274a4bd9b\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7" Oct 31 04:32:55.284971 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:55.284854 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5581a822-03f0-4f8c-bd5e-444274a4bd9b-istio-data\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7\" (UID: \"5581a822-03f0-4f8c-bd5e-444274a4bd9b\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7" Oct 31 04:32:55.288363 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:55.285344 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5581a822-03f0-4f8c-bd5e-444274a4bd9b-istio-data\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7\" (UID: \"5581a822-03f0-4f8c-bd5e-444274a4bd9b\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7" Oct 31 04:32:55.288363 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:55.285611 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5581a822-03f0-4f8c-bd5e-444274a4bd9b-workload-socket\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7\" (UID: \"5581a822-03f0-4f8c-bd5e-444274a4bd9b\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7" Oct 31 04:32:55.288363 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:55.286866 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5581a822-03f0-4f8c-bd5e-444274a4bd9b-istiod-ca-cert\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7\" (UID: \"5581a822-03f0-4f8c-bd5e-444274a4bd9b\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7" Oct 31 04:32:55.288363 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:55.287644 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5581a822-03f0-4f8c-bd5e-444274a4bd9b-credential-socket\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7\" (UID: \"5581a822-03f0-4f8c-bd5e-444274a4bd9b\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7" Oct 31 04:32:55.288363 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:55.288069 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5581a822-03f0-4f8c-bd5e-444274a4bd9b-workload-certs\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7\" (UID: \"5581a822-03f0-4f8c-bd5e-444274a4bd9b\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7" Oct 31 04:32:55.295349 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:55.291683 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5581a822-03f0-4f8c-bd5e-444274a4bd9b-istio-envoy\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7\" (UID: \"5581a822-03f0-4f8c-bd5e-444274a4bd9b\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7" Oct 31 04:32:55.295349 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:55.293895 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5581a822-03f0-4f8c-bd5e-444274a4bd9b-istio-podinfo\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7\" (UID: \"5581a822-03f0-4f8c-bd5e-444274a4bd9b\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7" Oct 31 04:32:55.314285 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:55.314256 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5581a822-03f0-4f8c-bd5e-444274a4bd9b-istio-token\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7\" (UID: \"5581a822-03f0-4f8c-bd5e-444274a4bd9b\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7" Oct 31 04:32:55.316895 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:55.316872 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqcqz\" (UniqueName: \"kubernetes.io/projected/5581a822-03f0-4f8c-bd5e-444274a4bd9b-kube-api-access-fqcqz\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7\" (UID: \"5581a822-03f0-4f8c-bd5e-444274a4bd9b\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7" Oct 31 04:32:55.367688 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:55.367390 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7" Oct 31 04:32:55.708640 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:55.708610 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7"] Oct 31 04:32:55.713115 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:32:55.713046 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5581a822_03f0_4f8c_bd5e_444274a4bd9b.slice/crio-dc76064aee5acdefb29532e4002aaf2cebf2c5480b67f4ac51d144b5904e7ba1 WatchSource:0}: Error finding container dc76064aee5acdefb29532e4002aaf2cebf2c5480b67f4ac51d144b5904e7ba1: Status 404 returned error can't find the container with id dc76064aee5acdefb29532e4002aaf2cebf2c5480b67f4ac51d144b5904e7ba1 Oct 31 04:32:55.716852 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:55.716796 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:32:55.716942 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:55.716900 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:32:55.716999 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:55.716942 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:32:55.804640 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:55.804613 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e299b1ef-str2b" Oct 31 04:32:55.892702 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:55.892660 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fc8ddf7-db96-4b79-a18d-afd173264640-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fc8ddf7-db96-4b79-a18d-afd173264640-maas-api-xks-ca\") pod \"8fc8ddf7-db96-4b79-a18d-afd173264640\" (UID: \"8fc8ddf7-db96-4b79-a18d-afd173264640\") " Oct 31 04:32:55.892915 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:55.892745 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fc8ddf7-db96-4b79-a18d-afd173264640-kube-api-access-fw55c\" (UniqueName: \"kubernetes.io/projected/8fc8ddf7-db96-4b79-a18d-afd173264640-kube-api-access-fw55c\") pod \"8fc8ddf7-db96-4b79-a18d-afd173264640\" (UID: \"8fc8ddf7-db96-4b79-a18d-afd173264640\") " Oct 31 04:32:55.892915 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:55.892813 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fc8ddf7-db96-4b79-a18d-afd173264640-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fc8ddf7-db96-4b79-a18d-afd173264640-maas-api-openshift-ca\") pod \"8fc8ddf7-db96-4b79-a18d-afd173264640\" (UID: \"8fc8ddf7-db96-4b79-a18d-afd173264640\") " Oct 31 04:32:55.894194 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:55.894107 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fc8ddf7-db96-4b79-a18d-afd173264640-maas-api-openshift-ca" pod "8fc8ddf7-db96-4b79-a18d-afd173264640" (UID: "8fc8ddf7-db96-4b79-a18d-afd173264640"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:32:55.898059 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:55.898022 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fc8ddf7-db96-4b79-a18d-afd173264640-maas-api-xks-ca" pod "8fc8ddf7-db96-4b79-a18d-afd173264640" (UID: "8fc8ddf7-db96-4b79-a18d-afd173264640"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:32:55.908308 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:55.907960 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fc8ddf7-db96-4b79-a18d-afd173264640-kube-api-access-fw55c" pod "8fc8ddf7-db96-4b79-a18d-afd173264640" (UID: "8fc8ddf7-db96-4b79-a18d-afd173264640"). InnerVolumeSpecName "kube-api-access-fw55c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:32:55.995033 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:55.995001 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fc8ddf7-db96-4b79-a18d-afd173264640-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:32:55.995033 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:55.995027 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fc8ddf7-db96-4b79-a18d-afd173264640-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:32:55.995033 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:55.995039 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw55c\" (UniqueName: \"kubernetes.io/projected/8fc8ddf7-db96-4b79-a18d-afd173264640-kube-api-access-fw55c\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:32:56.651462 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:56.651387 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e299b1ef-str2b" Oct 31 04:32:56.651462 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:56.651403 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e299b1ef-str2b" event={"ID":"8fc8ddf7-db96-4b79-a18d-afd173264640","Type":"ContainerDied","Data":"c7547f26d271003c36e88c3576c25fd1d181a3207572486062bf725e5d3eaffd"} Oct 31 04:32:56.651670 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:56.651461 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7547f26d271003c36e88c3576c25fd1d181a3207572486062bf725e5d3eaffd" Oct 31 04:32:56.655989 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:56.655963 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7" event={"ID":"5581a822-03f0-4f8c-bd5e-444274a4bd9b","Type":"ContainerStarted","Data":"7a46c6e9ba6fea80c750c3d47dcad8df0c6a3dfc624839b3d723fdf3fb7d9e2c"} Oct 31 04:32:56.656120 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:56.656003 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7" event={"ID":"5581a822-03f0-4f8c-bd5e-444274a4bd9b","Type":"ContainerStarted","Data":"dc76064aee5acdefb29532e4002aaf2cebf2c5480b67f4ac51d144b5904e7ba1"} Oct 31 04:32:56.714594 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:56.714550 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7" podStartSLOduration=1.714537847 podStartE2EDuration="1.714537847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:32:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:32:56.711627233 +0000 UTC m=+1731.467965267" watchObservedRunningTime="2026-10-31 04:32:56.714537847 +0000 UTC m=+1731.470875881" Oct 31 04:32:57.368786 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:57.368756 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7" Oct 31 04:32:57.374758 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:57.374731 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7" Oct 31 04:32:57.663752 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:57.663682 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7" Oct 31 04:32:57.664695 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:57.664673 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7" Oct 31 04:32:58.375725 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:58.373227 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz"] Oct 31 04:32:58.375725 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:58.374177 5019 cpu_manager.go:414] "containerMap: removing container" podUID="8fc8ddf7-db96-4b79-a18d-afd173264640" containerName="revoke-keys" Oct 31 04:32:58.375725 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:58.374198 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fc8ddf7-db96-4b79-a18d-afd173264640" containerName="revoke-keys" Oct 31 04:32:58.375725 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:58.374414 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fc8ddf7-db96-4b79-a18d-afd173264640" containerName="revoke-keys" Oct 31 04:32:58.401390 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:58.401100 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz" Oct 31 04:32:58.414322 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:58.412711 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz"] Oct 31 04:32:58.418917 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:58.418624 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-77a153-openshift-default-dockercfg-wr9c4\"" Oct 31 04:32:58.539352 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:58.539166 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d771867-36b5-4b55-afb4-f889d1d91784-credential-socket\") pod \"e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz\" (UID: \"2d771867-36b5-4b55-afb4-f889d1d91784\") " pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz" Oct 31 04:32:58.539352 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:58.539228 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts69w\" (UniqueName: \"kubernetes.io/projected/2d771867-36b5-4b55-afb4-f889d1d91784-kube-api-access-ts69w\") pod \"e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz\" (UID: \"2d771867-36b5-4b55-afb4-f889d1d91784\") " pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz" Oct 31 04:32:58.539352 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:58.539277 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d771867-36b5-4b55-afb4-f889d1d91784-istiod-ca-cert\") pod \"e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz\" (UID: \"2d771867-36b5-4b55-afb4-f889d1d91784\") " pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz" Oct 31 04:32:58.539352 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:58.539314 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d771867-36b5-4b55-afb4-f889d1d91784-istio-token\") pod \"e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz\" (UID: \"2d771867-36b5-4b55-afb4-f889d1d91784\") " pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz" Oct 31 04:32:58.539352 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:58.539363 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d771867-36b5-4b55-afb4-f889d1d91784-workload-certs\") pod \"e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz\" (UID: \"2d771867-36b5-4b55-afb4-f889d1d91784\") " pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz" Oct 31 04:32:58.539818 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:58.539425 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d771867-36b5-4b55-afb4-f889d1d91784-istio-envoy\") pod \"e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz\" (UID: \"2d771867-36b5-4b55-afb4-f889d1d91784\") " pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz" Oct 31 04:32:58.539818 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:58.539492 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d771867-36b5-4b55-afb4-f889d1d91784-istio-data\") pod \"e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz\" (UID: \"2d771867-36b5-4b55-afb4-f889d1d91784\") " pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz" Oct 31 04:32:58.539818 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:58.539593 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d771867-36b5-4b55-afb4-f889d1d91784-istio-podinfo\") pod \"e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz\" (UID: \"2d771867-36b5-4b55-afb4-f889d1d91784\") " pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz" Oct 31 04:32:58.539818 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:58.539659 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d771867-36b5-4b55-afb4-f889d1d91784-workload-socket\") pod \"e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz\" (UID: \"2d771867-36b5-4b55-afb4-f889d1d91784\") " pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz" Oct 31 04:32:58.671599 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:58.669542 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d771867-36b5-4b55-afb4-f889d1d91784-istio-podinfo\") pod \"e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz\" (UID: \"2d771867-36b5-4b55-afb4-f889d1d91784\") " pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz" Oct 31 04:32:58.671599 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:58.669626 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d771867-36b5-4b55-afb4-f889d1d91784-workload-socket\") pod \"e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz\" (UID: \"2d771867-36b5-4b55-afb4-f889d1d91784\") " pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz" Oct 31 04:32:58.671599 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:58.669754 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d771867-36b5-4b55-afb4-f889d1d91784-credential-socket\") pod \"e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz\" (UID: \"2d771867-36b5-4b55-afb4-f889d1d91784\") " pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz" Oct 31 04:32:58.671599 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:58.669787 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts69w\" (UniqueName: \"kubernetes.io/projected/2d771867-36b5-4b55-afb4-f889d1d91784-kube-api-access-ts69w\") pod \"e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz\" (UID: \"2d771867-36b5-4b55-afb4-f889d1d91784\") " pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz" Oct 31 04:32:58.671599 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:58.669834 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d771867-36b5-4b55-afb4-f889d1d91784-istiod-ca-cert\") pod \"e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz\" (UID: \"2d771867-36b5-4b55-afb4-f889d1d91784\") " pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz" Oct 31 04:32:58.671599 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:58.669870 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d771867-36b5-4b55-afb4-f889d1d91784-istio-token\") pod \"e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz\" (UID: \"2d771867-36b5-4b55-afb4-f889d1d91784\") " pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz" Oct 31 04:32:58.671599 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:58.669902 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d771867-36b5-4b55-afb4-f889d1d91784-workload-certs\") pod \"e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz\" (UID: \"2d771867-36b5-4b55-afb4-f889d1d91784\") " pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz" Oct 31 04:32:58.671599 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:58.669959 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d771867-36b5-4b55-afb4-f889d1d91784-istio-envoy\") pod \"e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz\" (UID: \"2d771867-36b5-4b55-afb4-f889d1d91784\") " pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz" Oct 31 04:32:58.671599 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:58.670023 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d771867-36b5-4b55-afb4-f889d1d91784-istio-data\") pod \"e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz\" (UID: \"2d771867-36b5-4b55-afb4-f889d1d91784\") " pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz" Oct 31 04:32:58.671599 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:58.671072 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d771867-36b5-4b55-afb4-f889d1d91784-istio-data\") pod \"e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz\" (UID: \"2d771867-36b5-4b55-afb4-f889d1d91784\") " pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz" Oct 31 04:32:58.681051 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:58.681020 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d771867-36b5-4b55-afb4-f889d1d91784-istiod-ca-cert\") pod \"e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz\" (UID: \"2d771867-36b5-4b55-afb4-f889d1d91784\") " pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz" Oct 31 04:32:58.681539 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:58.681516 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d771867-36b5-4b55-afb4-f889d1d91784-workload-socket\") pod \"e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz\" (UID: \"2d771867-36b5-4b55-afb4-f889d1d91784\") " pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz" Oct 31 04:32:58.682693 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:58.682671 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d771867-36b5-4b55-afb4-f889d1d91784-credential-socket\") pod \"e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz\" (UID: \"2d771867-36b5-4b55-afb4-f889d1d91784\") " pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz" Oct 31 04:32:58.685938 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:58.685914 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d771867-36b5-4b55-afb4-f889d1d91784-workload-certs\") pod \"e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz\" (UID: \"2d771867-36b5-4b55-afb4-f889d1d91784\") " pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz" Oct 31 04:32:58.709437 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:58.708032 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts69w\" (UniqueName: \"kubernetes.io/projected/2d771867-36b5-4b55-afb4-f889d1d91784-kube-api-access-ts69w\") pod \"e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz\" (UID: \"2d771867-36b5-4b55-afb4-f889d1d91784\") " pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz" Oct 31 04:32:58.709437 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:58.708964 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d771867-36b5-4b55-afb4-f889d1d91784-istio-podinfo\") pod \"e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz\" (UID: \"2d771867-36b5-4b55-afb4-f889d1d91784\") " pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz" Oct 31 04:32:58.709437 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:58.709199 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d771867-36b5-4b55-afb4-f889d1d91784-istio-envoy\") pod \"e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz\" (UID: \"2d771867-36b5-4b55-afb4-f889d1d91784\") " pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz" Oct 31 04:32:58.711734 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:58.710780 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d771867-36b5-4b55-afb4-f889d1d91784-istio-token\") pod \"e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz\" (UID: \"2d771867-36b5-4b55-afb4-f889d1d91784\") " pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz" Oct 31 04:32:58.724000 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:58.723966 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz" Oct 31 04:32:59.044720 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:32:59.044680 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d771867_36b5_4b55_afb4_f889d1d91784.slice/crio-bfa6b5be2d47068f00e3bb2b7f3bf24be0d46d3eaf7502c387e9ac7d92a68a13 WatchSource:0}: Error finding container bfa6b5be2d47068f00e3bb2b7f3bf24be0d46d3eaf7502c387e9ac7d92a68a13: Status 404 returned error can't find the container with id bfa6b5be2d47068f00e3bb2b7f3bf24be0d46d3eaf7502c387e9ac7d92a68a13 Oct 31 04:32:59.045513 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:59.045487 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz"] Oct 31 04:32:59.047414 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:59.047381 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:32:59.047504 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:59.047443 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:32:59.047504 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:59.047474 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:32:59.700255 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:59.700218 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz" event={"ID":"2d771867-36b5-4b55-afb4-f889d1d91784","Type":"ContainerStarted","Data":"85a2b9dd4c1a3477ce87dd382e1a0c28fde2e96c96b8d364cc1769bfa3a2ae46"} Oct 31 04:32:59.700754 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:59.700289 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz" event={"ID":"2d771867-36b5-4b55-afb4-f889d1d91784","Type":"ContainerStarted","Data":"bfa6b5be2d47068f00e3bb2b7f3bf24be0d46d3eaf7502c387e9ac7d92a68a13"} Oct 31 04:32:59.724915 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:59.724889 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz" Oct 31 04:32:59.763792 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:32:59.763742 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz" podStartSLOduration=1.763728128 podStartE2EDuration="1.763728128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:32:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:32:59.758239127 +0000 UTC m=+1734.514577167" watchObservedRunningTime="2026-10-31 04:32:59.763728128 +0000 UTC m=+1734.520066168" Oct 31 04:33:00.730302 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:00.730264 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz" Oct 31 04:33:00.856372 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:00.856321 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s"] Oct 31 04:33:00.862476 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:00.862448 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s" Oct 31 04:33:00.882064 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:00.882036 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s"] Oct 31 04:33:00.888728 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:00.888702 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-b2a372-openshift-default-dockercfg-44djb\"" Oct 31 04:33:01.011409 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:01.004492 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f2a6e7-c1d8-4834-80d3-805cf819e158-workload-certs\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s\" (UID: \"22f2a6e7-c1d8-4834-80d3-805cf819e158\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s" Oct 31 04:33:01.011409 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:01.004603 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f2a6e7-c1d8-4834-80d3-805cf819e158-istio-data\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s\" (UID: \"22f2a6e7-c1d8-4834-80d3-805cf819e158\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s" Oct 31 04:33:01.011409 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:01.004637 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8s8j\" (UniqueName: \"kubernetes.io/projected/22f2a6e7-c1d8-4834-80d3-805cf819e158-kube-api-access-x8s8j\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s\" (UID: \"22f2a6e7-c1d8-4834-80d3-805cf819e158\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s" Oct 31 04:33:01.011409 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:01.004750 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22f2a6e7-c1d8-4834-80d3-805cf819e158-istio-token\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s\" (UID: \"22f2a6e7-c1d8-4834-80d3-805cf819e158\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s" Oct 31 04:33:01.011409 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:01.004779 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f2a6e7-c1d8-4834-80d3-805cf819e158-istio-envoy\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s\" (UID: \"22f2a6e7-c1d8-4834-80d3-805cf819e158\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s" Oct 31 04:33:01.011409 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:01.004901 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f2a6e7-c1d8-4834-80d3-805cf819e158-credential-socket\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s\" (UID: \"22f2a6e7-c1d8-4834-80d3-805cf819e158\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s" Oct 31 04:33:01.011409 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:01.004953 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f2a6e7-c1d8-4834-80d3-805cf819e158-istiod-ca-cert\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s\" (UID: \"22f2a6e7-c1d8-4834-80d3-805cf819e158\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s" Oct 31 04:33:01.011409 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:01.005041 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f2a6e7-c1d8-4834-80d3-805cf819e158-istio-podinfo\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s\" (UID: \"22f2a6e7-c1d8-4834-80d3-805cf819e158\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s" Oct 31 04:33:01.011409 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:01.005067 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f2a6e7-c1d8-4834-80d3-805cf819e158-workload-socket\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s\" (UID: \"22f2a6e7-c1d8-4834-80d3-805cf819e158\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s" Oct 31 04:33:01.110359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:01.109732 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f2a6e7-c1d8-4834-80d3-805cf819e158-istio-podinfo\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s\" (UID: \"22f2a6e7-c1d8-4834-80d3-805cf819e158\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s" Oct 31 04:33:01.110359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:01.109787 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f2a6e7-c1d8-4834-80d3-805cf819e158-workload-socket\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s\" (UID: \"22f2a6e7-c1d8-4834-80d3-805cf819e158\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s" Oct 31 04:33:01.110359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:01.109850 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f2a6e7-c1d8-4834-80d3-805cf819e158-workload-certs\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s\" (UID: \"22f2a6e7-c1d8-4834-80d3-805cf819e158\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s" Oct 31 04:33:01.110359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:01.109888 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f2a6e7-c1d8-4834-80d3-805cf819e158-istio-data\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s\" (UID: \"22f2a6e7-c1d8-4834-80d3-805cf819e158\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s" Oct 31 04:33:01.110359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:01.109927 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8s8j\" (UniqueName: \"kubernetes.io/projected/22f2a6e7-c1d8-4834-80d3-805cf819e158-kube-api-access-x8s8j\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s\" (UID: \"22f2a6e7-c1d8-4834-80d3-805cf819e158\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s" Oct 31 04:33:01.110359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:01.110058 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22f2a6e7-c1d8-4834-80d3-805cf819e158-istio-token\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s\" (UID: \"22f2a6e7-c1d8-4834-80d3-805cf819e158\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s" Oct 31 04:33:01.110359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:01.110088 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f2a6e7-c1d8-4834-80d3-805cf819e158-istio-envoy\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s\" (UID: \"22f2a6e7-c1d8-4834-80d3-805cf819e158\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s" Oct 31 04:33:01.110359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:01.110228 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f2a6e7-c1d8-4834-80d3-805cf819e158-credential-socket\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s\" (UID: \"22f2a6e7-c1d8-4834-80d3-805cf819e158\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s" Oct 31 04:33:01.110359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:01.110287 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f2a6e7-c1d8-4834-80d3-805cf819e158-istiod-ca-cert\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s\" (UID: \"22f2a6e7-c1d8-4834-80d3-805cf819e158\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s" Oct 31 04:33:01.132355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:01.111223 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f2a6e7-c1d8-4834-80d3-805cf819e158-istiod-ca-cert\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s\" (UID: \"22f2a6e7-c1d8-4834-80d3-805cf819e158\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s" Oct 31 04:33:01.132355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:01.112272 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f2a6e7-c1d8-4834-80d3-805cf819e158-workload-socket\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s\" (UID: \"22f2a6e7-c1d8-4834-80d3-805cf819e158\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s" Oct 31 04:33:01.132355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:01.112587 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f2a6e7-c1d8-4834-80d3-805cf819e158-istio-data\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s\" (UID: \"22f2a6e7-c1d8-4834-80d3-805cf819e158\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s" Oct 31 04:33:01.132355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:01.112678 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f2a6e7-c1d8-4834-80d3-805cf819e158-credential-socket\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s\" (UID: \"22f2a6e7-c1d8-4834-80d3-805cf819e158\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s" Oct 31 04:33:01.132355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:01.122727 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f2a6e7-c1d8-4834-80d3-805cf819e158-workload-certs\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s\" (UID: \"22f2a6e7-c1d8-4834-80d3-805cf819e158\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s" Oct 31 04:33:01.132355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:01.125094 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f2a6e7-c1d8-4834-80d3-805cf819e158-istio-envoy\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s\" (UID: \"22f2a6e7-c1d8-4834-80d3-805cf819e158\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s" Oct 31 04:33:01.146360 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:01.142218 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f2a6e7-c1d8-4834-80d3-805cf819e158-istio-podinfo\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s\" (UID: \"22f2a6e7-c1d8-4834-80d3-805cf819e158\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s" Oct 31 04:33:01.150350 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:01.149810 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8s8j\" (UniqueName: \"kubernetes.io/projected/22f2a6e7-c1d8-4834-80d3-805cf819e158-kube-api-access-x8s8j\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s\" (UID: \"22f2a6e7-c1d8-4834-80d3-805cf819e158\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s" Oct 31 04:33:01.154351 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:01.153830 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22f2a6e7-c1d8-4834-80d3-805cf819e158-istio-token\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s\" (UID: \"22f2a6e7-c1d8-4834-80d3-805cf819e158\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s" Oct 31 04:33:01.199572 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:01.198184 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s" Oct 31 04:33:01.394538 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:01.394499 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-031745-74cd69fbc5-s7bs2"] Oct 31 04:33:01.400401 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:01.400371 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-031745-74cd69fbc5-s7bs2" Oct 31 04:33:01.417115 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:01.415850 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-031745-dockercfg-pffjz\"" Oct 31 04:33:01.417115 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:01.416454 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-031745\"" Oct 31 04:33:01.423219 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:01.423186 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-031745-74cd69fbc5-s7bs2"] Oct 31 04:33:01.583795 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:01.583703 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc856b70-497c-4c46-9db5-f41282c6b91d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-031745-74cd69fbc5-s7bs2\" (UID: \"bc856b70-497c-4c46-9db5-f41282c6b91d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-031745-74cd69fbc5-s7bs2" Oct 31 04:33:01.583975 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:01.583927 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzd76\" (UniqueName: \"kubernetes.io/projected/bc856b70-497c-4c46-9db5-f41282c6b91d-kube-api-access-kzd76\") pod \"payload-pre-processing-e2e-shared-b-w2-031745-74cd69fbc5-s7bs2\" (UID: \"bc856b70-497c-4c46-9db5-f41282c6b91d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-031745-74cd69fbc5-s7bs2" Oct 31 04:33:01.617718 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:01.617602 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-031745-6d7b5f78dc-6k5z9"] Oct 31 04:33:01.655731 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:01.655496 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:33:01.656002 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:01.655969 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:33:01.656163 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:01.656141 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:33:01.659807 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:01.659782 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-031745-6d7b5f78dc-6k5z9"] Oct 31 04:33:01.659914 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:01.659892 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-031745-6d7b5f78dc-6k5z9" Oct 31 04:33:01.667954 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:01.667928 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s"] Oct 31 04:33:01.685395 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:01.685368 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzd76\" (UniqueName: \"kubernetes.io/projected/bc856b70-497c-4c46-9db5-f41282c6b91d-kube-api-access-kzd76\") pod \"payload-pre-processing-e2e-shared-b-w2-031745-74cd69fbc5-s7bs2\" (UID: \"bc856b70-497c-4c46-9db5-f41282c6b91d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-031745-74cd69fbc5-s7bs2" Oct 31 04:33:01.685622 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:01.685602 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc856b70-497c-4c46-9db5-f41282c6b91d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-031745-74cd69fbc5-s7bs2\" (UID: \"bc856b70-497c-4c46-9db5-f41282c6b91d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-031745-74cd69fbc5-s7bs2" Oct 31 04:33:01.686460 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:01.686434 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc856b70-497c-4c46-9db5-f41282c6b91d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-031745-74cd69fbc5-s7bs2\" (UID: \"bc856b70-497c-4c46-9db5-f41282c6b91d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-031745-74cd69fbc5-s7bs2" Oct 31 04:33:01.709471 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:01.709443 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzd76\" (UniqueName: \"kubernetes.io/projected/bc856b70-497c-4c46-9db5-f41282c6b91d-kube-api-access-kzd76\") pod \"payload-pre-processing-e2e-shared-b-w2-031745-74cd69fbc5-s7bs2\" (UID: \"bc856b70-497c-4c46-9db5-f41282c6b91d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-031745-74cd69fbc5-s7bs2" Oct 31 04:33:01.730731 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:01.730709 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-031745-74cd69fbc5-s7bs2" Oct 31 04:33:01.736495 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:01.736471 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s" event={"ID":"22f2a6e7-c1d8-4834-80d3-805cf819e158","Type":"ContainerStarted","Data":"57791878e1def00efad041a7f6da21b57070384e404ca1874a40d00729e877f5"} Oct 31 04:33:01.736829 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:01.736814 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz" Oct 31 04:33:01.737640 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:01.737623 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz" Oct 31 04:33:01.789885 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:01.789666 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9m9m\" (UniqueName: \"kubernetes.io/projected/3456208b-2c04-4e78-a654-fafe097e110f-kube-api-access-x9m9m\") pod \"payload-processing-e2e-shared-b-w2-031745-6d7b5f78dc-6k5z9\" (UID: \"3456208b-2c04-4e78-a654-fafe097e110f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-031745-6d7b5f78dc-6k5z9" Oct 31 04:33:01.789885 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:01.789740 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3456208b-2c04-4e78-a654-fafe097e110f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-031745-6d7b5f78dc-6k5z9\" (UID: \"3456208b-2c04-4e78-a654-fafe097e110f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-031745-6d7b5f78dc-6k5z9" Oct 31 04:33:01.905157 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:01.902501 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9m9m\" (UniqueName: \"kubernetes.io/projected/3456208b-2c04-4e78-a654-fafe097e110f-kube-api-access-x9m9m\") pod \"payload-processing-e2e-shared-b-w2-031745-6d7b5f78dc-6k5z9\" (UID: \"3456208b-2c04-4e78-a654-fafe097e110f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-031745-6d7b5f78dc-6k5z9" Oct 31 04:33:01.905157 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:01.902608 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3456208b-2c04-4e78-a654-fafe097e110f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-031745-6d7b5f78dc-6k5z9\" (UID: \"3456208b-2c04-4e78-a654-fafe097e110f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-031745-6d7b5f78dc-6k5z9" Oct 31 04:33:01.905157 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:01.904542 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3456208b-2c04-4e78-a654-fafe097e110f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-031745-6d7b5f78dc-6k5z9\" (UID: \"3456208b-2c04-4e78-a654-fafe097e110f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-031745-6d7b5f78dc-6k5z9" Oct 31 04:33:01.929888 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:01.929855 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9m9m\" (UniqueName: \"kubernetes.io/projected/3456208b-2c04-4e78-a654-fafe097e110f-kube-api-access-x9m9m\") pod \"payload-processing-e2e-shared-b-w2-031745-6d7b5f78dc-6k5z9\" (UID: \"3456208b-2c04-4e78-a654-fafe097e110f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-031745-6d7b5f78dc-6k5z9" Oct 31 04:33:01.993917 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:01.993889 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-031745-6d7b5f78dc-6k5z9" Oct 31 04:33:02.065727 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:33:02.065686 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc856b70_497c_4c46_9db5_f41282c6b91d.slice/crio-0d7ca1742a48f5cf43f9f1f21d17ddf970a1537a736458761dfbcd17aed4489e WatchSource:0}: Error finding container 0d7ca1742a48f5cf43f9f1f21d17ddf970a1537a736458761dfbcd17aed4489e: Status 404 returned error can't find the container with id 0d7ca1742a48f5cf43f9f1f21d17ddf970a1537a736458761dfbcd17aed4489e Oct 31 04:33:02.069917 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:02.069894 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-031745-74cd69fbc5-s7bs2"] Oct 31 04:33:02.304558 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:33:02.304527 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3456208b_2c04_4e78_a654_fafe097e110f.slice/crio-09c718c218f97218aebe4274411ee26e1b15ef9d06f98e133c5896a495c43c2b WatchSource:0}: Error finding container 09c718c218f97218aebe4274411ee26e1b15ef9d06f98e133c5896a495c43c2b: Status 404 returned error can't find the container with id 09c718c218f97218aebe4274411ee26e1b15ef9d06f98e133c5896a495c43c2b Oct 31 04:33:02.307212 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:02.307184 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-031745-6d7b5f78dc-6k5z9"] Oct 31 04:33:02.746194 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:02.746152 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-031745-74cd69fbc5-s7bs2" event={"ID":"bc856b70-497c-4c46-9db5-f41282c6b91d","Type":"ContainerStarted","Data":"a83e70acf36eb980d48e39d998c744f45ee371cc4dadebd59c9c4e9b800dcb2b"} Oct 31 04:33:02.746631 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:02.746230 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-031745-74cd69fbc5-s7bs2" Oct 31 04:33:02.746631 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:02.746257 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-031745-74cd69fbc5-s7bs2" event={"ID":"bc856b70-497c-4c46-9db5-f41282c6b91d","Type":"ContainerStarted","Data":"0d7ca1742a48f5cf43f9f1f21d17ddf970a1537a736458761dfbcd17aed4489e"} Oct 31 04:33:02.747783 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:02.747761 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-031745-6d7b5f78dc-6k5z9" event={"ID":"3456208b-2c04-4e78-a654-fafe097e110f","Type":"ContainerStarted","Data":"692485e1b18974950fc2c514d1d936c2595a8d004b1a7c4d108c2c7005c86ae7"} Oct 31 04:33:02.747929 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:02.747806 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-031745-6d7b5f78dc-6k5z9" event={"ID":"3456208b-2c04-4e78-a654-fafe097e110f","Type":"ContainerStarted","Data":"09c718c218f97218aebe4274411ee26e1b15ef9d06f98e133c5896a495c43c2b"} Oct 31 04:33:02.747929 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:02.747855 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-031745-6d7b5f78dc-6k5z9" Oct 31 04:33:02.749368 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:02.749345 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s" event={"ID":"22f2a6e7-c1d8-4834-80d3-805cf819e158","Type":"ContainerStarted","Data":"55b10d00a9cbae70610d3dddfad1eaa18550d082df6f4faa78043ac9e03bbf31"} Oct 31 04:33:02.796068 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:02.796008 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-031745-74cd69fbc5-s7bs2" podStartSLOduration=1.795988782 podStartE2EDuration="1.795988782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:33:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:33:02.786123881 +0000 UTC m=+1737.542461922" watchObservedRunningTime="2026-10-31 04:33:02.795988782 +0000 UTC m=+1737.552326825" Oct 31 04:33:02.836280 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:02.836232 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s" podStartSLOduration=2.836219023 podStartE2EDuration="2.836219023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:33:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:33:02.831521699 +0000 UTC m=+1737.587859738" watchObservedRunningTime="2026-10-31 04:33:02.836219023 +0000 UTC m=+1737.592557063" Oct 31 04:33:02.874293 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:02.874251 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-031745-6d7b5f78dc-6k5z9" podStartSLOduration=1.874236657 podStartE2EDuration="1.874236657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:33:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:33:02.870998483 +0000 UTC m=+1737.627336553" watchObservedRunningTime="2026-10-31 04:33:02.874236657 +0000 UTC m=+1737.630574698" Oct 31 04:33:03.202744 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:03.202661 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s" Oct 31 04:33:03.298354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:03.297951 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s" Oct 31 04:33:03.499811 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:03.499776 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70075897-46qrk"] Oct 31 04:33:03.504323 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:03.504303 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70075897-46qrk" Oct 31 04:33:03.520914 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:03.520887 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:33:03.520914 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:03.520910 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:33:03.521099 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:03.521009 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:33:03.521158 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:03.521101 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:33:03.523765 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:03.523739 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70075897-46qrk"] Oct 31 04:33:03.550511 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:03.550479 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqwg2\" (UniqueName: \"kubernetes.io/projected/99d9c372-faba-46f3-9229-a20a40177209-kube-api-access-cqwg2\") pod \"maas-api-revoke-sub-70075897-46qrk\" (UID: \"99d9c372-faba-46f3-9229-a20a40177209\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70075897-46qrk" Oct 31 04:33:03.550681 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:03.550573 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99d9c372-faba-46f3-9229-a20a40177209-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70075897-46qrk\" (UID: \"99d9c372-faba-46f3-9229-a20a40177209\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70075897-46qrk" Oct 31 04:33:03.550681 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:03.550598 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99d9c372-faba-46f3-9229-a20a40177209-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70075897-46qrk\" (UID: \"99d9c372-faba-46f3-9229-a20a40177209\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70075897-46qrk" Oct 31 04:33:03.651687 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:03.651648 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99d9c372-faba-46f3-9229-a20a40177209-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70075897-46qrk\" (UID: \"99d9c372-faba-46f3-9229-a20a40177209\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70075897-46qrk" Oct 31 04:33:03.651687 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:03.651692 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99d9c372-faba-46f3-9229-a20a40177209-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70075897-46qrk\" (UID: \"99d9c372-faba-46f3-9229-a20a40177209\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70075897-46qrk" Oct 31 04:33:03.652024 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:03.652004 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqwg2\" (UniqueName: \"kubernetes.io/projected/99d9c372-faba-46f3-9229-a20a40177209-kube-api-access-cqwg2\") pod \"maas-api-revoke-sub-70075897-46qrk\" (UID: \"99d9c372-faba-46f3-9229-a20a40177209\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70075897-46qrk" Oct 31 04:33:03.652319 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:03.652298 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99d9c372-faba-46f3-9229-a20a40177209-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70075897-46qrk\" (UID: \"99d9c372-faba-46f3-9229-a20a40177209\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70075897-46qrk" Oct 31 04:33:03.654134 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:03.654112 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99d9c372-faba-46f3-9229-a20a40177209-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70075897-46qrk\" (UID: \"99d9c372-faba-46f3-9229-a20a40177209\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70075897-46qrk" Oct 31 04:33:03.675912 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:03.675879 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqwg2\" (UniqueName: \"kubernetes.io/projected/99d9c372-faba-46f3-9229-a20a40177209-kube-api-access-cqwg2\") pod \"maas-api-revoke-sub-70075897-46qrk\" (UID: \"99d9c372-faba-46f3-9229-a20a40177209\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70075897-46qrk" Oct 31 04:33:03.757970 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:03.757889 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s" Oct 31 04:33:03.758935 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:03.758915 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s" Oct 31 04:33:03.822667 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:03.822626 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70075897-46qrk" Oct 31 04:33:04.141025 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:33:04.140988 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99d9c372_faba_46f3_9229_a20a40177209.slice/crio-9fcffc4e0014ea8286abe9db8892fd233722e351955cc793ff7f692e6a392b16 WatchSource:0}: Error finding container 9fcffc4e0014ea8286abe9db8892fd233722e351955cc793ff7f692e6a392b16: Status 404 returned error can't find the container with id 9fcffc4e0014ea8286abe9db8892fd233722e351955cc793ff7f692e6a392b16 Oct 31 04:33:04.141900 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:04.141873 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70075897-46qrk"] Oct 31 04:33:04.767993 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:04.767908 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70075897-46qrk" event={"ID":"99d9c372-faba-46f3-9229-a20a40177209","Type":"ContainerStarted","Data":"dc57f9b7fbadc5b8b5b4c9623ba18760c6125a51c5c94e52989017899c1b1f11"} Oct 31 04:33:04.767993 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:04.767964 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70075897-46qrk" event={"ID":"99d9c372-faba-46f3-9229-a20a40177209","Type":"ContainerStarted","Data":"9fcffc4e0014ea8286abe9db8892fd233722e351955cc793ff7f692e6a392b16"} Oct 31 04:33:04.814346 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:04.814273 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70075897-46qrk" podStartSLOduration=1.8142539850000001 podStartE2EDuration="1.814253985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:33:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:33:04.806991609 +0000 UTC m=+1739.563329650" watchObservedRunningTime="2026-10-31 04:33:04.814253985 +0000 UTC m=+1739.570592026" Oct 31 04:33:04.846105 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:04.846074 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-77a153-6d7d74f579-mfw72"] Oct 31 04:33:04.850554 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:04.850538 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-77a153-6d7d74f579-mfw72" Oct 31 04:33:04.867273 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:04.867241 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b27570f7-6a81-4a8c-aad3-404126c2b7de-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-77a153-6d7d74f579-mfw72\" (UID: \"b27570f7-6a81-4a8c-aad3-404126c2b7de\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-77a153-6d7d74f579-mfw72" Oct 31 04:33:04.867780 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:04.867748 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnlpx\" (UniqueName: \"kubernetes.io/projected/b27570f7-6a81-4a8c-aad3-404126c2b7de-kube-api-access-wnlpx\") pod \"payload-pre-processing-e2e-shared-a-w1-77a153-6d7d74f579-mfw72\" (UID: \"b27570f7-6a81-4a8c-aad3-404126c2b7de\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-77a153-6d7d74f579-mfw72" Oct 31 04:33:04.869049 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:04.867876 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-77a153-dockercfg-glhvp\"" Oct 31 04:33:04.869049 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:04.868105 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-77a153\"" Oct 31 04:33:04.870264 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:04.870241 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-77a153-6d7d74f579-mfw72"] Oct 31 04:33:05.050156 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:04.992410 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnlpx\" (UniqueName: \"kubernetes.io/projected/b27570f7-6a81-4a8c-aad3-404126c2b7de-kube-api-access-wnlpx\") pod \"payload-pre-processing-e2e-shared-a-w1-77a153-6d7d74f579-mfw72\" (UID: \"b27570f7-6a81-4a8c-aad3-404126c2b7de\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-77a153-6d7d74f579-mfw72" Oct 31 04:33:05.050156 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:04.992560 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b27570f7-6a81-4a8c-aad3-404126c2b7de-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-77a153-6d7d74f579-mfw72\" (UID: \"b27570f7-6a81-4a8c-aad3-404126c2b7de\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-77a153-6d7d74f579-mfw72" Oct 31 04:33:05.050156 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:04.993908 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b27570f7-6a81-4a8c-aad3-404126c2b7de-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-77a153-6d7d74f579-mfw72\" (UID: \"b27570f7-6a81-4a8c-aad3-404126c2b7de\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-77a153-6d7d74f579-mfw72" Oct 31 04:33:05.074355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:05.063476 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-77a153-6c975c99f5-mp8vx"] Oct 31 04:33:05.077590 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:05.077528 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnlpx\" (UniqueName: \"kubernetes.io/projected/b27570f7-6a81-4a8c-aad3-404126c2b7de-kube-api-access-wnlpx\") pod \"payload-pre-processing-e2e-shared-a-w1-77a153-6d7d74f579-mfw72\" (UID: \"b27570f7-6a81-4a8c-aad3-404126c2b7de\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-77a153-6d7d74f579-mfw72" Oct 31 04:33:05.122395 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:05.115814 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-77a153-6c975c99f5-mp8vx" Oct 31 04:33:05.150381 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:05.134150 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-77a153-6c975c99f5-mp8vx"] Oct 31 04:33:05.202396 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:05.168615 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-77a153-6d7d74f579-mfw72" Oct 31 04:33:05.206376 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:05.205415 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7139f8fd-4b2e-4c1e-9111-92e9b1bd1f62-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-77a153-6c975c99f5-mp8vx\" (UID: \"7139f8fd-4b2e-4c1e-9111-92e9b1bd1f62\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-77a153-6c975c99f5-mp8vx" Oct 31 04:33:05.206376 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:05.205478 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scvd9\" (UniqueName: \"kubernetes.io/projected/7139f8fd-4b2e-4c1e-9111-92e9b1bd1f62-kube-api-access-scvd9\") pod \"payload-processing-e2e-shared-a-w1-77a153-6c975c99f5-mp8vx\" (UID: \"7139f8fd-4b2e-4c1e-9111-92e9b1bd1f62\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-77a153-6c975c99f5-mp8vx" Oct 31 04:33:05.307582 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:05.307488 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7139f8fd-4b2e-4c1e-9111-92e9b1bd1f62-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-77a153-6c975c99f5-mp8vx\" (UID: \"7139f8fd-4b2e-4c1e-9111-92e9b1bd1f62\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-77a153-6c975c99f5-mp8vx" Oct 31 04:33:05.307582 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:05.307557 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scvd9\" (UniqueName: \"kubernetes.io/projected/7139f8fd-4b2e-4c1e-9111-92e9b1bd1f62-kube-api-access-scvd9\") pod \"payload-processing-e2e-shared-a-w1-77a153-6c975c99f5-mp8vx\" (UID: \"7139f8fd-4b2e-4c1e-9111-92e9b1bd1f62\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-77a153-6c975c99f5-mp8vx" Oct 31 04:33:05.308774 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:05.308723 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7139f8fd-4b2e-4c1e-9111-92e9b1bd1f62-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-77a153-6c975c99f5-mp8vx\" (UID: \"7139f8fd-4b2e-4c1e-9111-92e9b1bd1f62\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-77a153-6c975c99f5-mp8vx" Oct 31 04:33:05.333384 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:05.333272 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scvd9\" (UniqueName: \"kubernetes.io/projected/7139f8fd-4b2e-4c1e-9111-92e9b1bd1f62-kube-api-access-scvd9\") pod \"payload-processing-e2e-shared-a-w1-77a153-6c975c99f5-mp8vx\" (UID: \"7139f8fd-4b2e-4c1e-9111-92e9b1bd1f62\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-77a153-6c975c99f5-mp8vx" Oct 31 04:33:05.459207 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:05.459170 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-77a153-6c975c99f5-mp8vx" Oct 31 04:33:05.567813 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:05.567782 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-77a153-6d7d74f579-mfw72"] Oct 31 04:33:05.579844 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:33:05.571649 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb27570f7_6a81_4a8c_aad3_404126c2b7de.slice/crio-0649e8706ee0dbc33f7a43bd92a91f5923c41ae4d69a86ad67d7c5815566562c WatchSource:0}: Error finding container 0649e8706ee0dbc33f7a43bd92a91f5923c41ae4d69a86ad67d7c5815566562c: Status 404 returned error can't find the container with id 0649e8706ee0dbc33f7a43bd92a91f5923c41ae4d69a86ad67d7c5815566562c Oct 31 04:33:05.765597 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:33:05.765562 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7139f8fd_4b2e_4c1e_9111_92e9b1bd1f62.slice/crio-6232203bc2fdb0cabab3b96b42aa67c41f6426f4bc0dcd45af988c4f3dc6801c WatchSource:0}: Error finding container 6232203bc2fdb0cabab3b96b42aa67c41f6426f4bc0dcd45af988c4f3dc6801c: Status 404 returned error can't find the container with id 6232203bc2fdb0cabab3b96b42aa67c41f6426f4bc0dcd45af988c4f3dc6801c Oct 31 04:33:05.767394 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:05.767377 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-77a153-6c975c99f5-mp8vx"] Oct 31 04:33:05.780143 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:05.780111 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-77a153-6c975c99f5-mp8vx" event={"ID":"7139f8fd-4b2e-4c1e-9111-92e9b1bd1f62","Type":"ContainerStarted","Data":"6232203bc2fdb0cabab3b96b42aa67c41f6426f4bc0dcd45af988c4f3dc6801c"} Oct 31 04:33:05.782125 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:05.782097 5019 generic.go:356] "Generic (PLEG): container finished" podID="99d9c372-faba-46f3-9229-a20a40177209" containerID="dc57f9b7fbadc5b8b5b4c9623ba18760c6125a51c5c94e52989017899c1b1f11" exitCode=0 Oct 31 04:33:05.782210 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:05.782184 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70075897-46qrk" event={"ID":"99d9c372-faba-46f3-9229-a20a40177209","Type":"ContainerDied","Data":"dc57f9b7fbadc5b8b5b4c9623ba18760c6125a51c5c94e52989017899c1b1f11"} Oct 31 04:33:05.783977 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:05.783949 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-77a153-6d7d74f579-mfw72" event={"ID":"b27570f7-6a81-4a8c-aad3-404126c2b7de","Type":"ContainerStarted","Data":"055909b4eaf8584cd37ac55ea33ad929c6f418ec6d7fedc9fe43d41caccfa244"} Oct 31 04:33:05.784088 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:05.783999 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-77a153-6d7d74f579-mfw72" event={"ID":"b27570f7-6a81-4a8c-aad3-404126c2b7de","Type":"ContainerStarted","Data":"0649e8706ee0dbc33f7a43bd92a91f5923c41ae4d69a86ad67d7c5815566562c"} Oct 31 04:33:05.784149 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:05.784113 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-77a153-6d7d74f579-mfw72" Oct 31 04:33:05.881246 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:05.881186 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-77a153-6d7d74f579-mfw72" podStartSLOduration=1.881166331 podStartE2EDuration="1.881166331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:33:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:33:05.876872464 +0000 UTC m=+1740.633210504" watchObservedRunningTime="2026-10-31 04:33:05.881166331 +0000 UTC m=+1740.637504373" Oct 31 04:33:06.803238 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:06.803198 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-77a153-6c975c99f5-mp8vx" event={"ID":"7139f8fd-4b2e-4c1e-9111-92e9b1bd1f62","Type":"ContainerStarted","Data":"6e803cd33fa32f8e539c094464e09cbca4efeb86359db641670212855553a5f6"} Oct 31 04:33:06.803752 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:06.803460 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-77a153-6c975c99f5-mp8vx" Oct 31 04:33:06.851021 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:06.850883 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-77a153-6c975c99f5-mp8vx" podStartSLOduration=2.850865524 podStartE2EDuration="2.850865524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:33:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:33:06.850306665 +0000 UTC m=+1741.606644707" watchObservedRunningTime="2026-10-31 04:33:06.850865524 +0000 UTC m=+1741.607203565" Oct 31 04:33:07.098670 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:07.098522 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70075897-46qrk" Oct 31 04:33:07.157608 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:07.157573 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99d9c372-faba-46f3-9229-a20a40177209-kube-api-access-cqwg2\" (UniqueName: \"kubernetes.io/projected/99d9c372-faba-46f3-9229-a20a40177209-kube-api-access-cqwg2\") pod \"99d9c372-faba-46f3-9229-a20a40177209\" (UID: \"99d9c372-faba-46f3-9229-a20a40177209\") " Oct 31 04:33:07.157803 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:07.157786 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99d9c372-faba-46f3-9229-a20a40177209-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99d9c372-faba-46f3-9229-a20a40177209-maas-api-xks-ca\") pod \"99d9c372-faba-46f3-9229-a20a40177209\" (UID: \"99d9c372-faba-46f3-9229-a20a40177209\") " Oct 31 04:33:07.158005 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:07.157985 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99d9c372-faba-46f3-9229-a20a40177209-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99d9c372-faba-46f3-9229-a20a40177209-maas-api-openshift-ca\") pod \"99d9c372-faba-46f3-9229-a20a40177209\" (UID: \"99d9c372-faba-46f3-9229-a20a40177209\") " Oct 31 04:33:07.158964 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:07.158937 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99d9c372-faba-46f3-9229-a20a40177209-maas-api-openshift-ca" pod "99d9c372-faba-46f3-9229-a20a40177209" (UID: "99d9c372-faba-46f3-9229-a20a40177209"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:33:07.168587 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:07.168552 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99d9c372-faba-46f3-9229-a20a40177209-maas-api-xks-ca" pod "99d9c372-faba-46f3-9229-a20a40177209" (UID: "99d9c372-faba-46f3-9229-a20a40177209"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:33:07.169846 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:07.169817 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99d9c372-faba-46f3-9229-a20a40177209-kube-api-access-cqwg2" pod "99d9c372-faba-46f3-9229-a20a40177209" (UID: "99d9c372-faba-46f3-9229-a20a40177209"). InnerVolumeSpecName "kube-api-access-cqwg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:33:07.308358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:07.269154 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99d9c372-faba-46f3-9229-a20a40177209-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:33:07.308358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:07.269208 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99d9c372-faba-46f3-9229-a20a40177209-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:33:07.308358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:07.269226 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqwg2\" (UniqueName: \"kubernetes.io/projected/99d9c372-faba-46f3-9229-a20a40177209-kube-api-access-cqwg2\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:33:07.821209 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:07.821161 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70075897-46qrk" event={"ID":"99d9c372-faba-46f3-9229-a20a40177209","Type":"ContainerDied","Data":"9fcffc4e0014ea8286abe9db8892fd233722e351955cc793ff7f692e6a392b16"} Oct 31 04:33:07.821917 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:07.821231 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fcffc4e0014ea8286abe9db8892fd233722e351955cc793ff7f692e6a392b16" Oct 31 04:33:07.821917 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:07.821310 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70075897-46qrk" Oct 31 04:33:08.373547 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:08.373507 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-b2a372-dfc6bb5c9-2t4p7"] Oct 31 04:33:08.399355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:08.374813 5019 cpu_manager.go:414] "containerMap: removing container" podUID="99d9c372-faba-46f3-9229-a20a40177209" containerName="revoke-keys" Oct 31 04:33:08.401564 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:08.401535 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99d9c372-faba-46f3-9229-a20a40177209" containerName="revoke-keys" Oct 31 04:33:08.405814 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:08.405788 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="99d9c372-faba-46f3-9229-a20a40177209" containerName="revoke-keys" Oct 31 04:33:08.535355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:08.489703 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-b2a372-dfc6bb5c9-2t4p7"] Oct 31 04:33:08.535355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:08.490772 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b2a372-dfc6bb5c9-2t4p7" Oct 31 04:33:08.615359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:08.578373 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-b2a372\"" Oct 31 04:33:08.623428 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:08.583413 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-b2a372-dockercfg-p26ld\"" Oct 31 04:33:08.765990 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:08.765948 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bc5984a-022f-462f-aaf3-a421eb9ee119-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-b2a372-dfc6bb5c9-2t4p7\" (UID: \"3bc5984a-022f-462f-aaf3-a421eb9ee119\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b2a372-dfc6bb5c9-2t4p7" Oct 31 04:33:08.777498 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:08.766993 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6dwt\" (UniqueName: \"kubernetes.io/projected/3bc5984a-022f-462f-aaf3-a421eb9ee119-kube-api-access-x6dwt\") pod \"payload-processing-e2e-shared-b-w3-b2a372-dfc6bb5c9-2t4p7\" (UID: \"3bc5984a-022f-462f-aaf3-a421eb9ee119\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b2a372-dfc6bb5c9-2t4p7" Oct 31 04:33:08.868182 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:08.868151 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bc5984a-022f-462f-aaf3-a421eb9ee119-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-b2a372-dfc6bb5c9-2t4p7\" (UID: \"3bc5984a-022f-462f-aaf3-a421eb9ee119\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b2a372-dfc6bb5c9-2t4p7" Oct 31 04:33:08.868595 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:08.868287 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6dwt\" (UniqueName: \"kubernetes.io/projected/3bc5984a-022f-462f-aaf3-a421eb9ee119-kube-api-access-x6dwt\") pod \"payload-processing-e2e-shared-b-w3-b2a372-dfc6bb5c9-2t4p7\" (UID: \"3bc5984a-022f-462f-aaf3-a421eb9ee119\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b2a372-dfc6bb5c9-2t4p7" Oct 31 04:33:08.868879 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:08.868860 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bc5984a-022f-462f-aaf3-a421eb9ee119-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-b2a372-dfc6bb5c9-2t4p7\" (UID: \"3bc5984a-022f-462f-aaf3-a421eb9ee119\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b2a372-dfc6bb5c9-2t4p7" Oct 31 04:33:08.939042 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:08.939009 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6dwt\" (UniqueName: \"kubernetes.io/projected/3bc5984a-022f-462f-aaf3-a421eb9ee119-kube-api-access-x6dwt\") pod \"payload-processing-e2e-shared-b-w3-b2a372-dfc6bb5c9-2t4p7\" (UID: \"3bc5984a-022f-462f-aaf3-a421eb9ee119\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b2a372-dfc6bb5c9-2t4p7" Oct 31 04:33:09.143511 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:09.143423 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b2a372-dfc6bb5c9-2t4p7" Oct 31 04:33:09.234354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:09.226168 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6"] Oct 31 04:33:09.410501 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:09.407585 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6"] Oct 31 04:33:09.463443 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:09.416471 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6" Oct 31 04:33:09.490355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:09.464700 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-9633f0-openshift-default-dockercfg-5qfkr\"" Oct 31 04:33:09.662708 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:09.662622 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68b1965c-0483-419e-9e73-aacf0f6a5048-istio-podinfo\") pod \"e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6\" (UID: \"68b1965c-0483-419e-9e73-aacf0f6a5048\") " pod="openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6" Oct 31 04:33:09.662900 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:09.662883 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b1965c-0483-419e-9e73-aacf0f6a5048-workload-socket\") pod \"e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6\" (UID: \"68b1965c-0483-419e-9e73-aacf0f6a5048\") " pod="openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6" Oct 31 04:33:09.662972 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:09.662960 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwjsw\" (UniqueName: \"kubernetes.io/projected/68b1965c-0483-419e-9e73-aacf0f6a5048-kube-api-access-dwjsw\") pod \"e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6\" (UID: \"68b1965c-0483-419e-9e73-aacf0f6a5048\") " pod="openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6" Oct 31 04:33:09.663175 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:09.663165 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68b1965c-0483-419e-9e73-aacf0f6a5048-istiod-ca-cert\") pod \"e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6\" (UID: \"68b1965c-0483-419e-9e73-aacf0f6a5048\") " pod="openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6" Oct 31 04:33:09.663258 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:09.663247 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68b1965c-0483-419e-9e73-aacf0f6a5048-istio-envoy\") pod \"e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6\" (UID: \"68b1965c-0483-419e-9e73-aacf0f6a5048\") " pod="openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6" Oct 31 04:33:09.663380 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:09.663369 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68b1965c-0483-419e-9e73-aacf0f6a5048-workload-certs\") pod \"e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6\" (UID: \"68b1965c-0483-419e-9e73-aacf0f6a5048\") " pod="openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6" Oct 31 04:33:09.663487 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:09.663478 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68b1965c-0483-419e-9e73-aacf0f6a5048-istio-data\") pod \"e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6\" (UID: \"68b1965c-0483-419e-9e73-aacf0f6a5048\") " pod="openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6" Oct 31 04:33:09.663585 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:09.663574 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68b1965c-0483-419e-9e73-aacf0f6a5048-istio-token\") pod \"e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6\" (UID: \"68b1965c-0483-419e-9e73-aacf0f6a5048\") " pod="openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6" Oct 31 04:33:09.663662 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:09.663653 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b1965c-0483-419e-9e73-aacf0f6a5048-credential-socket\") pod \"e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6\" (UID: \"68b1965c-0483-419e-9e73-aacf0f6a5048\") " pod="openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6" Oct 31 04:33:09.764819 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:09.764785 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68b1965c-0483-419e-9e73-aacf0f6a5048-istiod-ca-cert\") pod \"e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6\" (UID: \"68b1965c-0483-419e-9e73-aacf0f6a5048\") " pod="openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6" Oct 31 04:33:09.765034 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:09.764843 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68b1965c-0483-419e-9e73-aacf0f6a5048-istio-envoy\") pod \"e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6\" (UID: \"68b1965c-0483-419e-9e73-aacf0f6a5048\") " pod="openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6" Oct 31 04:33:09.765034 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:09.764886 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68b1965c-0483-419e-9e73-aacf0f6a5048-workload-certs\") pod \"e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6\" (UID: \"68b1965c-0483-419e-9e73-aacf0f6a5048\") " pod="openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6" Oct 31 04:33:09.765034 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:09.764932 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68b1965c-0483-419e-9e73-aacf0f6a5048-istio-data\") pod \"e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6\" (UID: \"68b1965c-0483-419e-9e73-aacf0f6a5048\") " pod="openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6" Oct 31 04:33:09.765034 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:09.764981 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68b1965c-0483-419e-9e73-aacf0f6a5048-istio-token\") pod \"e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6\" (UID: \"68b1965c-0483-419e-9e73-aacf0f6a5048\") " pod="openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6" Oct 31 04:33:09.765034 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:09.765009 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b1965c-0483-419e-9e73-aacf0f6a5048-credential-socket\") pod \"e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6\" (UID: \"68b1965c-0483-419e-9e73-aacf0f6a5048\") " pod="openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6" Oct 31 04:33:09.765501 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:09.765472 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b1965c-0483-419e-9e73-aacf0f6a5048-credential-socket\") pod \"e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6\" (UID: \"68b1965c-0483-419e-9e73-aacf0f6a5048\") " pod="openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6" Oct 31 04:33:09.765648 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:09.765608 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68b1965c-0483-419e-9e73-aacf0f6a5048-workload-certs\") pod \"e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6\" (UID: \"68b1965c-0483-419e-9e73-aacf0f6a5048\") " pod="openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6" Oct 31 04:33:09.765895 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:09.765876 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68b1965c-0483-419e-9e73-aacf0f6a5048-istiod-ca-cert\") pod \"e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6\" (UID: \"68b1965c-0483-419e-9e73-aacf0f6a5048\") " pod="openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6" Oct 31 04:33:09.766255 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:09.766231 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68b1965c-0483-419e-9e73-aacf0f6a5048-istio-podinfo\") pod \"e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6\" (UID: \"68b1965c-0483-419e-9e73-aacf0f6a5048\") " pod="openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6" Oct 31 04:33:09.766307 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:09.766272 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b1965c-0483-419e-9e73-aacf0f6a5048-workload-socket\") pod \"e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6\" (UID: \"68b1965c-0483-419e-9e73-aacf0f6a5048\") " pod="openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6" Oct 31 04:33:09.766307 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:09.766297 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwjsw\" (UniqueName: \"kubernetes.io/projected/68b1965c-0483-419e-9e73-aacf0f6a5048-kube-api-access-dwjsw\") pod \"e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6\" (UID: \"68b1965c-0483-419e-9e73-aacf0f6a5048\") " pod="openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6" Oct 31 04:33:09.766553 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:09.766539 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68b1965c-0483-419e-9e73-aacf0f6a5048-istio-data\") pod \"e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6\" (UID: \"68b1965c-0483-419e-9e73-aacf0f6a5048\") " pod="openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6" Oct 31 04:33:09.766796 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:09.766782 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b1965c-0483-419e-9e73-aacf0f6a5048-workload-socket\") pod \"e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6\" (UID: \"68b1965c-0483-419e-9e73-aacf0f6a5048\") " pod="openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6" Oct 31 04:33:09.768130 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:09.768111 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68b1965c-0483-419e-9e73-aacf0f6a5048-istio-envoy\") pod \"e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6\" (UID: \"68b1965c-0483-419e-9e73-aacf0f6a5048\") " pod="openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6" Oct 31 04:33:09.793361 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:09.769611 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68b1965c-0483-419e-9e73-aacf0f6a5048-istio-podinfo\") pod \"e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6\" (UID: \"68b1965c-0483-419e-9e73-aacf0f6a5048\") " pod="openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6" Oct 31 04:33:09.841357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:09.808051 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwjsw\" (UniqueName: \"kubernetes.io/projected/68b1965c-0483-419e-9e73-aacf0f6a5048-kube-api-access-dwjsw\") pod \"e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6\" (UID: \"68b1965c-0483-419e-9e73-aacf0f6a5048\") " pod="openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6" Oct 31 04:33:09.841357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:09.808449 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68b1965c-0483-419e-9e73-aacf0f6a5048-istio-token\") pod \"e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6\" (UID: \"68b1965c-0483-419e-9e73-aacf0f6a5048\") " pod="openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6" Oct 31 04:33:09.856422 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:09.856001 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-b2a372-dfc6bb5c9-2t4p7"] Oct 31 04:33:10.017656 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:10.017614 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6" Oct 31 04:33:10.537168 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:33:10.537131 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68b1965c_0483_419e_9e73_aacf0f6a5048.slice/crio-ba1d39d29fed0d268ba2c7078ecc65bf5fe0c519fd2bf1810f58d0c6e890d1ee WatchSource:0}: Error finding container ba1d39d29fed0d268ba2c7078ecc65bf5fe0c519fd2bf1810f58d0c6e890d1ee: Status 404 returned error can't find the container with id ba1d39d29fed0d268ba2c7078ecc65bf5fe0c519fd2bf1810f58d0c6e890d1ee Oct 31 04:33:10.540395 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:10.540349 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:33:10.540609 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:10.540586 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:33:10.540748 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:10.540730 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:33:10.543385 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:10.543358 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6"] Oct 31 04:33:10.909040 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:10.908933 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6" event={"ID":"68b1965c-0483-419e-9e73-aacf0f6a5048","Type":"ContainerStarted","Data":"2928eb4559d6aa4257028602a5358f1f1b47cccf6cbb91de2759b7b36e0b94c1"} Oct 31 04:33:10.909298 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:10.909274 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6" event={"ID":"68b1965c-0483-419e-9e73-aacf0f6a5048","Type":"ContainerStarted","Data":"ba1d39d29fed0d268ba2c7078ecc65bf5fe0c519fd2bf1810f58d0c6e890d1ee"} Oct 31 04:33:10.923550 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:10.923505 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b2a372-dfc6bb5c9-2t4p7" event={"ID":"3bc5984a-022f-462f-aaf3-a421eb9ee119","Type":"ContainerStarted","Data":"0e33685bac1afa3700ed3f474486c1b55908d5ae440022eecb5e280f68a6c7e5"} Oct 31 04:33:10.923848 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:10.923822 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b2a372-dfc6bb5c9-2t4p7" event={"ID":"3bc5984a-022f-462f-aaf3-a421eb9ee119","Type":"ContainerStarted","Data":"6b7a945be9e734a1f1448e25e51e33ee8579caad9c06e8212c0907cab0319ebf"} Oct 31 04:33:10.924909 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:10.924887 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b2a372-dfc6bb5c9-2t4p7" Oct 31 04:33:10.982693 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:10.982536 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6" podStartSLOduration=1.9825148970000002 podStartE2EDuration="1.982514897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:33:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:33:10.982105022 +0000 UTC m=+1745.738443061" watchObservedRunningTime="2026-10-31 04:33:10.982514897 +0000 UTC m=+1745.738852938" Oct 31 04:33:11.019671 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:11.019632 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6" Oct 31 04:33:11.027881 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:11.027848 5019 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Oct 31 04:33:11.028170 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:11.028122 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6" podUID="68b1965c-0483-419e-9e73-aacf0f6a5048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Oct 31 04:33:11.043235 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:11.043173 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b2a372-dfc6bb5c9-2t4p7" podStartSLOduration=3.043153784 podStartE2EDuration="3.043153784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:33:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:33:11.042695226 +0000 UTC m=+1745.799033266" watchObservedRunningTime="2026-10-31 04:33:11.043153784 +0000 UTC m=+1745.799491823" Oct 31 04:33:12.042736 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:12.042704 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6" Oct 31 04:33:12.953481 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:12.953451 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6" Oct 31 04:33:12.954824 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:12.954804 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6" Oct 31 04:33:13.805578 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:13.805544 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-031745-6d7b5f78dc-6k5z9" Oct 31 04:33:13.820723 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:13.819606 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-031745-74cd69fbc5-s7bs2" Oct 31 04:33:15.329994 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:15.329959 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9633f0-54d5b879f5-qg5lc"] Oct 31 04:33:15.336099 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:15.336074 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9633f0-54d5b879f5-qg5lc" Oct 31 04:33:15.358033 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:15.357186 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-9633f0\"" Oct 31 04:33:15.361919 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:15.361875 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-9633f0-dockercfg-l6g8v\"" Oct 31 04:33:15.362237 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:15.362217 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9633f0-54d5b879f5-qg5lc"] Oct 31 04:33:15.445489 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:15.445446 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1803197-fd95-4f13-9cdb-4a9386cf9610-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-9633f0-54d5b879f5-qg5lc\" (UID: \"e1803197-fd95-4f13-9cdb-4a9386cf9610\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9633f0-54d5b879f5-qg5lc" Oct 31 04:33:15.571127 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:15.571085 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc82s\" (UniqueName: \"kubernetes.io/projected/e1803197-fd95-4f13-9cdb-4a9386cf9610-kube-api-access-xc82s\") pod \"payload-pre-processing-e2e-shared-b-w4-9633f0-54d5b879f5-qg5lc\" (UID: \"e1803197-fd95-4f13-9cdb-4a9386cf9610\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9633f0-54d5b879f5-qg5lc" Oct 31 04:33:15.673125 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:15.672865 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1803197-fd95-4f13-9cdb-4a9386cf9610-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-9633f0-54d5b879f5-qg5lc\" (UID: \"e1803197-fd95-4f13-9cdb-4a9386cf9610\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9633f0-54d5b879f5-qg5lc" Oct 31 04:33:15.673125 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:15.672989 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc82s\" (UniqueName: \"kubernetes.io/projected/e1803197-fd95-4f13-9cdb-4a9386cf9610-kube-api-access-xc82s\") pod \"payload-pre-processing-e2e-shared-b-w4-9633f0-54d5b879f5-qg5lc\" (UID: \"e1803197-fd95-4f13-9cdb-4a9386cf9610\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9633f0-54d5b879f5-qg5lc" Oct 31 04:33:15.673779 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:15.673751 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1803197-fd95-4f13-9cdb-4a9386cf9610-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-9633f0-54d5b879f5-qg5lc\" (UID: \"e1803197-fd95-4f13-9cdb-4a9386cf9610\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9633f0-54d5b879f5-qg5lc" Oct 31 04:33:15.730202 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:15.730109 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc82s\" (UniqueName: \"kubernetes.io/projected/e1803197-fd95-4f13-9cdb-4a9386cf9610-kube-api-access-xc82s\") pod \"payload-pre-processing-e2e-shared-b-w4-9633f0-54d5b879f5-qg5lc\" (UID: \"e1803197-fd95-4f13-9cdb-4a9386cf9610\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9633f0-54d5b879f5-qg5lc" Oct 31 04:33:15.788243 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:15.788189 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9633f0-54d5b879f5-qg5lc" Oct 31 04:33:16.239957 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:16.233319 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9633f0-54d5b879f5-qg5lc"] Oct 31 04:33:16.807600 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:16.807509 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-77a153-6d7d74f579-mfw72" Oct 31 04:33:17.057396 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:17.057348 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9633f0-54d5b879f5-qg5lc" event={"ID":"e1803197-fd95-4f13-9cdb-4a9386cf9610","Type":"ContainerStarted","Data":"40c212a00c030b2dc932f08ec578be766e64d3566dde30b251214d9f0e7961f0"} Oct 31 04:33:17.057692 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:17.057424 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9633f0-54d5b879f5-qg5lc" event={"ID":"e1803197-fd95-4f13-9cdb-4a9386cf9610","Type":"ContainerStarted","Data":"5bc2d5c2709f590083ec3f93d03f38f9920624f79bf38a021d3333e0ec96a15c"} Oct 31 04:33:17.057924 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:17.057893 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9633f0-54d5b879f5-qg5lc" Oct 31 04:33:17.847366 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:17.846925 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-77a153-6c975c99f5-mp8vx" Oct 31 04:33:17.890384 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:17.890292 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9633f0-54d5b879f5-qg5lc" podStartSLOduration=2.890256483 podStartE2EDuration="2.890256483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:33:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:33:17.101593676 +0000 UTC m=+1751.857931716" watchObservedRunningTime="2026-10-31 04:33:17.890256483 +0000 UTC m=+1752.646594523" Oct 31 04:33:18.990493 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:18.990453 5019 patch_prober.go:29] interesting pod/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:33:19.006646 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:18.990516 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9" podUID="b547bb0b-cae2-453b-aa56-4f37434a1ce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:33:20.084946 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:20.084908 5019 generic.go:356] "Generic (PLEG): container finished" podID="1132f111-a44a-4cc9-97bf-540c4f0dc0a5" containerID="f3f42daf2ba8bd5b75932b0e38fac44b2c54c9ea6b230874a15f71a6d01cb65c" exitCode=0 Oct 31 04:33:20.085435 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:20.084976 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z" event={"ID":"1132f111-a44a-4cc9-97bf-540c4f0dc0a5","Type":"ContainerDied","Data":"f3f42daf2ba8bd5b75932b0e38fac44b2c54c9ea6b230874a15f71a6d01cb65c"} Oct 31 04:33:20.085523 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:20.085501 5019 scope.go:122] "RemoveContainer" containerID="f3f42daf2ba8bd5b75932b0e38fac44b2c54c9ea6b230874a15f71a6d01cb65c" Oct 31 04:33:20.087046 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:20.087017 5019 generic.go:356] "Generic (PLEG): container finished" podID="b547bb0b-cae2-453b-aa56-4f37434a1ce8" containerID="07fe73b7b2ad40bd8c4007ec62b8acdccaf0a951bd6e972b1f4c02148cb75cd1" exitCode=0 Oct 31 04:33:20.087174 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:20.087091 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9" event={"ID":"b547bb0b-cae2-453b-aa56-4f37434a1ce8","Type":"ContainerDied","Data":"07fe73b7b2ad40bd8c4007ec62b8acdccaf0a951bd6e972b1f4c02148cb75cd1"} Oct 31 04:33:20.087288 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:20.087258 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:33:20.087401 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:20.087359 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:33:20.087477 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:20.087414 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:33:20.087477 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:20.087459 5019 scope.go:122] "RemoveContainer" containerID="07fe73b7b2ad40bd8c4007ec62b8acdccaf0a951bd6e972b1f4c02148cb75cd1" Oct 31 04:33:20.088897 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:20.088866 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:33:20.089480 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:20.088945 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:33:20.089480 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:20.088995 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:33:21.129393 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:21.129358 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z" event={"ID":"1132f111-a44a-4cc9-97bf-540c4f0dc0a5","Type":"ContainerStarted","Data":"43852a114edbd03c7597f3ad6140cb97401df73d73265964f13d5e6d8e0ac247"} Oct 31 04:33:21.132451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:21.132412 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9" event={"ID":"b547bb0b-cae2-453b-aa56-4f37434a1ce8","Type":"ContainerStarted","Data":"2e652653ef68a8b7a63a7ed2034dca31b8832bc21c7f3e04f7b3f517dc962883"} Oct 31 04:33:21.760179 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:21.759835 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9" Oct 31 04:33:21.760179 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:21.760001 5019 patch_prober.go:29] interesting pod/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 31 04:33:21.760179 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:21.760047 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9" podUID="b547bb0b-cae2-453b-aa56-4f37434a1ce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 31 04:33:22.053358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:22.052663 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-7229f2-798cdf44cf-4wm7j"] Oct 31 04:33:22.058414 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:22.058384 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z" Oct 31 04:33:22.058580 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:22.058503 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-7229f2-798cdf44cf-4wm7j" Oct 31 04:33:22.059918 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:22.059635 5019 patch_prober.go:29] interesting pod/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 31 04:33:22.059918 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:22.059748 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z" podUID="1132f111-a44a-4cc9-97bf-540c4f0dc0a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 31 04:33:22.063037 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:22.061165 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-7229f2-798cdf44cf-4wm7j"] Oct 31 04:33:22.122546 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:22.097298 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-7229f2\"" Oct 31 04:33:22.122546 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:22.097491 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-7229f2-dockercfg-dj94c\"" Oct 31 04:33:22.276302 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:22.274933 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69471f9f-03f8-433e-8793-26529badb542-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-7229f2-798cdf44cf-4wm7j\" (UID: \"69471f9f-03f8-433e-8793-26529badb542\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-7229f2-798cdf44cf-4wm7j" Oct 31 04:33:22.276302 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:22.275080 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fssq7\" (UniqueName: \"kubernetes.io/projected/69471f9f-03f8-433e-8793-26529badb542-kube-api-access-fssq7\") pod \"payload-pre-processing-e2e-autores-7229f2-798cdf44cf-4wm7j\" (UID: \"69471f9f-03f8-433e-8793-26529badb542\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-7229f2-798cdf44cf-4wm7j" Oct 31 04:33:22.418440 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:22.391219 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69471f9f-03f8-433e-8793-26529badb542-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-7229f2-798cdf44cf-4wm7j\" (UID: \"69471f9f-03f8-433e-8793-26529badb542\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-7229f2-798cdf44cf-4wm7j" Oct 31 04:33:22.418440 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:22.391410 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fssq7\" (UniqueName: \"kubernetes.io/projected/69471f9f-03f8-433e-8793-26529badb542-kube-api-access-fssq7\") pod \"payload-pre-processing-e2e-autores-7229f2-798cdf44cf-4wm7j\" (UID: \"69471f9f-03f8-433e-8793-26529badb542\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-7229f2-798cdf44cf-4wm7j" Oct 31 04:33:22.441903 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:22.441868 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69471f9f-03f8-433e-8793-26529badb542-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-7229f2-798cdf44cf-4wm7j\" (UID: \"69471f9f-03f8-433e-8793-26529badb542\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-7229f2-798cdf44cf-4wm7j" Oct 31 04:33:22.539741 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:22.539698 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fssq7\" (UniqueName: \"kubernetes.io/projected/69471f9f-03f8-433e-8793-26529badb542-kube-api-access-fssq7\") pod \"payload-pre-processing-e2e-autores-7229f2-798cdf44cf-4wm7j\" (UID: \"69471f9f-03f8-433e-8793-26529badb542\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-7229f2-798cdf44cf-4wm7j" Oct 31 04:33:22.546609 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:33:22.546562 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:33:22.698409 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:22.697617 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-7229f2-798cdf44cf-4wm7j" Oct 31 04:33:22.760845 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:22.760813 5019 patch_prober.go:29] interesting pod/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 31 04:33:22.761029 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:22.760872 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9" podUID="b547bb0b-cae2-453b-aa56-4f37434a1ce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 31 04:33:22.955284 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:22.955201 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b2a372-dfc6bb5c9-2t4p7" Oct 31 04:33:23.057439 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:23.057394 5019 patch_prober.go:29] interesting pod/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 31 04:33:23.057646 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:23.057452 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z" podUID="1132f111-a44a-4cc9-97bf-540c4f0dc0a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 31 04:33:23.153358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:23.138796 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-7229f2-798cdf44cf-4wm7j"] Oct 31 04:33:23.153768 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:33:23.153625 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69471f9f_03f8_433e_8793_26529badb542.slice/crio-0edd57895c385d281e3de88134f0e67ccc5ffd375d75cb8dc6715617dff062b6 WatchSource:0}: Error finding container 0edd57895c385d281e3de88134f0e67ccc5ffd375d75cb8dc6715617dff062b6: Status 404 returned error can't find the container with id 0edd57895c385d281e3de88134f0e67ccc5ffd375d75cb8dc6715617dff062b6 Oct 31 04:33:23.335359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:23.334711 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-7229f2-798cdf44cf-4wm7j" event={"ID":"69471f9f-03f8-433e-8793-26529badb542","Type":"ContainerStarted","Data":"0edd57895c385d281e3de88134f0e67ccc5ffd375d75cb8dc6715617dff062b6"} Oct 31 04:33:23.759983 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:23.759949 5019 patch_prober.go:29] interesting pod/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 31 04:33:23.760251 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:23.760220 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9" podUID="b547bb0b-cae2-453b-aa56-4f37434a1ce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 31 04:33:24.057201 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:24.057120 5019 patch_prober.go:29] interesting pod/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 31 04:33:24.057201 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:24.057172 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z" podUID="1132f111-a44a-4cc9-97bf-540c4f0dc0a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 31 04:33:24.515354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:24.512906 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-7229f2-798cdf44cf-4wm7j" event={"ID":"69471f9f-03f8-433e-8793-26529badb542","Type":"ContainerStarted","Data":"e85ecb3b9713e99a1b19838df01a61082cd9bb46d7a2e9eec91a7270db0f3728"} Oct 31 04:33:24.515354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:24.514542 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-7229f2-798cdf44cf-4wm7j" Oct 31 04:33:24.561066 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:24.561011 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-7229f2-798cdf44cf-4wm7j" podStartSLOduration=3.560995875 podStartE2EDuration="3.560995875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:33:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:33:24.555321905 +0000 UTC m=+1759.311659944" watchObservedRunningTime="2026-10-31 04:33:24.560995875 +0000 UTC m=+1759.317333914" Oct 31 04:33:24.766409 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:24.765844 5019 patch_prober.go:29] interesting pod/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 31 04:33:24.766409 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:24.765921 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9" podUID="b547bb0b-cae2-453b-aa56-4f37434a1ce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 31 04:33:25.057707 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:25.057626 5019 patch_prober.go:29] interesting pod/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 31 04:33:25.057707 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:25.057680 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z" podUID="1132f111-a44a-4cc9-97bf-540c4f0dc0a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 31 04:33:25.759933 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:25.759900 5019 patch_prober.go:29] interesting pod/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 31 04:33:25.760312 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:25.759954 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9" podUID="b547bb0b-cae2-453b-aa56-4f37434a1ce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 31 04:33:26.057696 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:26.057609 5019 patch_prober.go:29] interesting pod/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 31 04:33:26.057696 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:26.057659 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z" podUID="1132f111-a44a-4cc9-97bf-540c4f0dc0a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 31 04:33:26.119493 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:26.119450 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z" Oct 31 04:33:26.759688 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:26.759642 5019 patch_prober.go:29] interesting pod/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 31 04:33:26.759897 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:26.759713 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9" podUID="b547bb0b-cae2-453b-aa56-4f37434a1ce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 31 04:33:27.061915 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:27.061832 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z" Oct 31 04:33:27.063081 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:27.063061 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z" Oct 31 04:33:27.764833 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:27.764800 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9" Oct 31 04:33:27.765293 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:27.765273 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9" Oct 31 04:33:27.766027 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:27.766005 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9" Oct 31 04:33:28.069840 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:28.069765 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9633f0-54d5b879f5-qg5lc" Oct 31 04:33:31.038781 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.038748 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h"] Oct 31 04:33:31.044158 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.044134 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h" Oct 31 04:33:31.057452 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.057425 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-x829c"] Oct 31 04:33:31.060746 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.060724 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-965f61-openshift-default-dockercfg-ggnnv\"" Oct 31 04:33:31.062557 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.062540 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-x829c" Oct 31 04:33:31.072645 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.072622 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h"] Oct 31 04:33:31.079378 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.079355 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc"] Oct 31 04:33:31.089664 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.089637 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc" Oct 31 04:33:31.128495 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.128432 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc"] Oct 31 04:33:31.133983 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.133950 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22bd2484-d363-4287-a424-29a99c3153ac-istiod-ca-cert\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-x829c\" (UID: \"22bd2484-d363-4287-a424-29a99c3153ac\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-x829c" Oct 31 04:33:31.134263 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.134244 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22bd2484-d363-4287-a424-29a99c3153ac-credential-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-x829c\" (UID: \"22bd2484-d363-4287-a424-29a99c3153ac\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-x829c" Oct 31 04:33:31.134461 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.134443 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3743e35-69e7-4544-979d-7596c44b59a0-workload-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc\" (UID: \"b3743e35-69e7-4544-979d-7596c44b59a0\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc" Oct 31 04:33:31.134650 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.134634 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3743e35-69e7-4544-979d-7596c44b59a0-istiod-ca-cert\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc\" (UID: \"b3743e35-69e7-4544-979d-7596c44b59a0\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc" Oct 31 04:33:31.134817 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.134801 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kxpn\" (UniqueName: \"kubernetes.io/projected/686f7dd0-02cd-457a-b7cd-ef83520d5341-kube-api-access-8kxpn\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h\" (UID: \"686f7dd0-02cd-457a-b7cd-ef83520d5341\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h" Oct 31 04:33:31.135003 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.134972 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22bd2484-d363-4287-a424-29a99c3153ac-istio-envoy\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-x829c\" (UID: \"22bd2484-d363-4287-a424-29a99c3153ac\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-x829c" Oct 31 04:33:31.135130 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.134895 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-x829c"] Oct 31 04:33:31.135130 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.135008 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/686f7dd0-02cd-457a-b7cd-ef83520d5341-istio-envoy\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h\" (UID: \"686f7dd0-02cd-457a-b7cd-ef83520d5341\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h" Oct 31 04:33:31.135130 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.135113 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3743e35-69e7-4544-979d-7596c44b59a0-istio-podinfo\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc\" (UID: \"b3743e35-69e7-4544-979d-7596c44b59a0\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc" Oct 31 04:33:31.135278 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.135164 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrpjb\" (UniqueName: \"kubernetes.io/projected/22bd2484-d363-4287-a424-29a99c3153ac-kube-api-access-xrpjb\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-x829c\" (UID: \"22bd2484-d363-4287-a424-29a99c3153ac\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-x829c" Oct 31 04:33:31.135278 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.135217 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22bd2484-d363-4287-a424-29a99c3153ac-istio-podinfo\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-x829c\" (UID: \"22bd2484-d363-4287-a424-29a99c3153ac\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-x829c" Oct 31 04:33:31.135391 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.135276 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/686f7dd0-02cd-457a-b7cd-ef83520d5341-credential-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h\" (UID: \"686f7dd0-02cd-457a-b7cd-ef83520d5341\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h" Oct 31 04:33:31.135391 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.135372 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/686f7dd0-02cd-457a-b7cd-ef83520d5341-workload-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h\" (UID: \"686f7dd0-02cd-457a-b7cd-ef83520d5341\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h" Oct 31 04:33:31.135476 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.135451 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22bd2484-d363-4287-a424-29a99c3153ac-workload-certs\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-x829c\" (UID: \"22bd2484-d363-4287-a424-29a99c3153ac\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-x829c" Oct 31 04:33:31.135512 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.135482 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3743e35-69e7-4544-979d-7596c44b59a0-credential-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc\" (UID: \"b3743e35-69e7-4544-979d-7596c44b59a0\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc" Oct 31 04:33:31.135562 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.135528 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22bd2484-d363-4287-a424-29a99c3153ac-istio-token\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-x829c\" (UID: \"22bd2484-d363-4287-a424-29a99c3153ac\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-x829c" Oct 31 04:33:31.135600 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.135559 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/686f7dd0-02cd-457a-b7cd-ef83520d5341-istio-token\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h\" (UID: \"686f7dd0-02cd-457a-b7cd-ef83520d5341\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h" Oct 31 04:33:31.135654 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.135629 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3743e35-69e7-4544-979d-7596c44b59a0-istio-token\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc\" (UID: \"b3743e35-69e7-4544-979d-7596c44b59a0\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc" Oct 31 04:33:31.135733 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.135710 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/686f7dd0-02cd-457a-b7cd-ef83520d5341-workload-certs\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h\" (UID: \"686f7dd0-02cd-457a-b7cd-ef83520d5341\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h" Oct 31 04:33:31.135799 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.135785 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22bd2484-d363-4287-a424-29a99c3153ac-workload-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-x829c\" (UID: \"22bd2484-d363-4287-a424-29a99c3153ac\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-x829c" Oct 31 04:33:31.135876 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.135860 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/686f7dd0-02cd-457a-b7cd-ef83520d5341-istio-data\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h\" (UID: \"686f7dd0-02cd-457a-b7cd-ef83520d5341\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h" Oct 31 04:33:31.135975 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.135960 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3743e35-69e7-4544-979d-7596c44b59a0-workload-certs\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc\" (UID: \"b3743e35-69e7-4544-979d-7596c44b59a0\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc" Oct 31 04:33:31.136047 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.135987 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5bx8\" (UniqueName: \"kubernetes.io/projected/b3743e35-69e7-4544-979d-7596c44b59a0-kube-api-access-v5bx8\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc\" (UID: \"b3743e35-69e7-4544-979d-7596c44b59a0\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc" Oct 31 04:33:31.136047 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.136025 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/686f7dd0-02cd-457a-b7cd-ef83520d5341-istio-podinfo\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h\" (UID: \"686f7dd0-02cd-457a-b7cd-ef83520d5341\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h" Oct 31 04:33:31.136162 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.136054 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22bd2484-d363-4287-a424-29a99c3153ac-istio-data\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-x829c\" (UID: \"22bd2484-d363-4287-a424-29a99c3153ac\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-x829c" Oct 31 04:33:31.136162 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.136082 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/686f7dd0-02cd-457a-b7cd-ef83520d5341-istiod-ca-cert\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h\" (UID: \"686f7dd0-02cd-457a-b7cd-ef83520d5341\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h" Oct 31 04:33:31.136162 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.136107 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3743e35-69e7-4544-979d-7596c44b59a0-istio-data\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc\" (UID: \"b3743e35-69e7-4544-979d-7596c44b59a0\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc" Oct 31 04:33:31.136162 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.136139 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3743e35-69e7-4544-979d-7596c44b59a0-istio-envoy\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc\" (UID: \"b3743e35-69e7-4544-979d-7596c44b59a0\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc" Oct 31 04:33:31.249353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.238416 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3743e35-69e7-4544-979d-7596c44b59a0-workload-certs\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc\" (UID: \"b3743e35-69e7-4544-979d-7596c44b59a0\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc" Oct 31 04:33:31.249353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.238475 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5bx8\" (UniqueName: \"kubernetes.io/projected/b3743e35-69e7-4544-979d-7596c44b59a0-kube-api-access-v5bx8\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc\" (UID: \"b3743e35-69e7-4544-979d-7596c44b59a0\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc" Oct 31 04:33:31.249353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.238523 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/686f7dd0-02cd-457a-b7cd-ef83520d5341-istio-podinfo\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h\" (UID: \"686f7dd0-02cd-457a-b7cd-ef83520d5341\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h" Oct 31 04:33:31.249353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.238551 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22bd2484-d363-4287-a424-29a99c3153ac-istio-data\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-x829c\" (UID: \"22bd2484-d363-4287-a424-29a99c3153ac\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-x829c" Oct 31 04:33:31.249353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.238582 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/686f7dd0-02cd-457a-b7cd-ef83520d5341-istiod-ca-cert\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h\" (UID: \"686f7dd0-02cd-457a-b7cd-ef83520d5341\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h" Oct 31 04:33:31.249353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.238609 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3743e35-69e7-4544-979d-7596c44b59a0-istio-data\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc\" (UID: \"b3743e35-69e7-4544-979d-7596c44b59a0\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc" Oct 31 04:33:31.249353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.238643 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3743e35-69e7-4544-979d-7596c44b59a0-istio-envoy\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc\" (UID: \"b3743e35-69e7-4544-979d-7596c44b59a0\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc" Oct 31 04:33:31.249353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.238712 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22bd2484-d363-4287-a424-29a99c3153ac-istiod-ca-cert\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-x829c\" (UID: \"22bd2484-d363-4287-a424-29a99c3153ac\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-x829c" Oct 31 04:33:31.249353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.238759 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22bd2484-d363-4287-a424-29a99c3153ac-credential-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-x829c\" (UID: \"22bd2484-d363-4287-a424-29a99c3153ac\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-x829c" Oct 31 04:33:31.249353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.238807 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3743e35-69e7-4544-979d-7596c44b59a0-workload-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc\" (UID: \"b3743e35-69e7-4544-979d-7596c44b59a0\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc" Oct 31 04:33:31.249353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.238862 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3743e35-69e7-4544-979d-7596c44b59a0-istiod-ca-cert\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc\" (UID: \"b3743e35-69e7-4544-979d-7596c44b59a0\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc" Oct 31 04:33:31.249353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.238913 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kxpn\" (UniqueName: \"kubernetes.io/projected/686f7dd0-02cd-457a-b7cd-ef83520d5341-kube-api-access-8kxpn\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h\" (UID: \"686f7dd0-02cd-457a-b7cd-ef83520d5341\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h" Oct 31 04:33:31.249353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.238915 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3743e35-69e7-4544-979d-7596c44b59a0-workload-certs\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc\" (UID: \"b3743e35-69e7-4544-979d-7596c44b59a0\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc" Oct 31 04:33:31.249353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.238982 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22bd2484-d363-4287-a424-29a99c3153ac-istio-envoy\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-x829c\" (UID: \"22bd2484-d363-4287-a424-29a99c3153ac\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-x829c" Oct 31 04:33:31.249353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.239012 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/686f7dd0-02cd-457a-b7cd-ef83520d5341-istio-envoy\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h\" (UID: \"686f7dd0-02cd-457a-b7cd-ef83520d5341\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h" Oct 31 04:33:31.249353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.239048 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3743e35-69e7-4544-979d-7596c44b59a0-istio-podinfo\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc\" (UID: \"b3743e35-69e7-4544-979d-7596c44b59a0\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc" Oct 31 04:33:31.249353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.239100 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrpjb\" (UniqueName: \"kubernetes.io/projected/22bd2484-d363-4287-a424-29a99c3153ac-kube-api-access-xrpjb\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-x829c\" (UID: \"22bd2484-d363-4287-a424-29a99c3153ac\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-x829c" Oct 31 04:33:31.249353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.239125 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22bd2484-d363-4287-a424-29a99c3153ac-istio-podinfo\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-x829c\" (UID: \"22bd2484-d363-4287-a424-29a99c3153ac\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-x829c" Oct 31 04:33:31.249353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.239163 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/686f7dd0-02cd-457a-b7cd-ef83520d5341-credential-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h\" (UID: \"686f7dd0-02cd-457a-b7cd-ef83520d5341\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h" Oct 31 04:33:31.249353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.239223 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/686f7dd0-02cd-457a-b7cd-ef83520d5341-workload-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h\" (UID: \"686f7dd0-02cd-457a-b7cd-ef83520d5341\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h" Oct 31 04:33:31.249353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.239309 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22bd2484-d363-4287-a424-29a99c3153ac-workload-certs\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-x829c\" (UID: \"22bd2484-d363-4287-a424-29a99c3153ac\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-x829c" Oct 31 04:33:31.249353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.239361 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3743e35-69e7-4544-979d-7596c44b59a0-credential-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc\" (UID: \"b3743e35-69e7-4544-979d-7596c44b59a0\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc" Oct 31 04:33:31.249353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.239411 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22bd2484-d363-4287-a424-29a99c3153ac-istio-token\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-x829c\" (UID: \"22bd2484-d363-4287-a424-29a99c3153ac\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-x829c" Oct 31 04:33:31.249353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.239438 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/686f7dd0-02cd-457a-b7cd-ef83520d5341-istio-token\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h\" (UID: \"686f7dd0-02cd-457a-b7cd-ef83520d5341\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h" Oct 31 04:33:31.249353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.239528 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3743e35-69e7-4544-979d-7596c44b59a0-istio-token\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc\" (UID: \"b3743e35-69e7-4544-979d-7596c44b59a0\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc" Oct 31 04:33:31.249353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.239580 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/686f7dd0-02cd-457a-b7cd-ef83520d5341-workload-certs\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h\" (UID: \"686f7dd0-02cd-457a-b7cd-ef83520d5341\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h" Oct 31 04:33:31.249353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.239612 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22bd2484-d363-4287-a424-29a99c3153ac-workload-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-x829c\" (UID: \"22bd2484-d363-4287-a424-29a99c3153ac\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-x829c" Oct 31 04:33:31.249353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.239645 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/686f7dd0-02cd-457a-b7cd-ef83520d5341-istio-data\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h\" (UID: \"686f7dd0-02cd-457a-b7cd-ef83520d5341\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h" Oct 31 04:33:31.249353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.239789 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3743e35-69e7-4544-979d-7596c44b59a0-istiod-ca-cert\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc\" (UID: \"b3743e35-69e7-4544-979d-7596c44b59a0\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc" Oct 31 04:33:31.249353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.240085 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/686f7dd0-02cd-457a-b7cd-ef83520d5341-istio-data\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h\" (UID: \"686f7dd0-02cd-457a-b7cd-ef83520d5341\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h" Oct 31 04:33:31.249353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.240434 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22bd2484-d363-4287-a424-29a99c3153ac-istiod-ca-cert\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-x829c\" (UID: \"22bd2484-d363-4287-a424-29a99c3153ac\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-x829c" Oct 31 04:33:31.272356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.264733 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22bd2484-d363-4287-a424-29a99c3153ac-istio-envoy\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-x829c\" (UID: \"22bd2484-d363-4287-a424-29a99c3153ac\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-x829c" Oct 31 04:33:31.272356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.271743 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22bd2484-d363-4287-a424-29a99c3153ac-credential-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-x829c\" (UID: \"22bd2484-d363-4287-a424-29a99c3153ac\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-x829c" Oct 31 04:33:31.272356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.272040 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3743e35-69e7-4544-979d-7596c44b59a0-workload-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc\" (UID: \"b3743e35-69e7-4544-979d-7596c44b59a0\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc" Oct 31 04:33:31.272669 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.272581 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22bd2484-d363-4287-a424-29a99c3153ac-istio-data\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-x829c\" (UID: \"22bd2484-d363-4287-a424-29a99c3153ac\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-x829c" Oct 31 04:33:31.295622 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.273170 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22bd2484-d363-4287-a424-29a99c3153ac-workload-certs\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-x829c\" (UID: \"22bd2484-d363-4287-a424-29a99c3153ac\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-x829c" Oct 31 04:33:31.295622 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.273899 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/686f7dd0-02cd-457a-b7cd-ef83520d5341-istiod-ca-cert\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h\" (UID: \"686f7dd0-02cd-457a-b7cd-ef83520d5341\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h" Oct 31 04:33:31.295622 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.274420 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3743e35-69e7-4544-979d-7596c44b59a0-credential-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc\" (UID: \"b3743e35-69e7-4544-979d-7596c44b59a0\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc" Oct 31 04:33:31.295622 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.274885 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/686f7dd0-02cd-457a-b7cd-ef83520d5341-workload-certs\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h\" (UID: \"686f7dd0-02cd-457a-b7cd-ef83520d5341\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h" Oct 31 04:33:31.295622 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.275355 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22bd2484-d363-4287-a424-29a99c3153ac-workload-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-x829c\" (UID: \"22bd2484-d363-4287-a424-29a99c3153ac\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-x829c" Oct 31 04:33:31.310467 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.297060 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3743e35-69e7-4544-979d-7596c44b59a0-istio-data\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc\" (UID: \"b3743e35-69e7-4544-979d-7596c44b59a0\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc" Oct 31 04:33:31.310467 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.300548 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/686f7dd0-02cd-457a-b7cd-ef83520d5341-credential-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h\" (UID: \"686f7dd0-02cd-457a-b7cd-ef83520d5341\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h" Oct 31 04:33:31.310467 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.309881 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/686f7dd0-02cd-457a-b7cd-ef83520d5341-workload-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h\" (UID: \"686f7dd0-02cd-457a-b7cd-ef83520d5341\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h" Oct 31 04:33:31.322586 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.322522 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/686f7dd0-02cd-457a-b7cd-ef83520d5341-istio-podinfo\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h\" (UID: \"686f7dd0-02cd-457a-b7cd-ef83520d5341\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h" Oct 31 04:33:31.336358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.323306 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3743e35-69e7-4544-979d-7596c44b59a0-istio-token\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc\" (UID: \"b3743e35-69e7-4544-979d-7596c44b59a0\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc" Oct 31 04:33:31.337447 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.337374 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrpjb\" (UniqueName: \"kubernetes.io/projected/22bd2484-d363-4287-a424-29a99c3153ac-kube-api-access-xrpjb\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-x829c\" (UID: \"22bd2484-d363-4287-a424-29a99c3153ac\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-x829c" Oct 31 04:33:31.346967 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.346892 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22bd2484-d363-4287-a424-29a99c3153ac-istio-token\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-x829c\" (UID: \"22bd2484-d363-4287-a424-29a99c3153ac\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-x829c" Oct 31 04:33:31.378716 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.373987 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22bd2484-d363-4287-a424-29a99c3153ac-istio-podinfo\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-x829c\" (UID: \"22bd2484-d363-4287-a424-29a99c3153ac\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-x829c" Oct 31 04:33:31.378716 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.374767 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5bx8\" (UniqueName: \"kubernetes.io/projected/b3743e35-69e7-4544-979d-7596c44b59a0-kube-api-access-v5bx8\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc\" (UID: \"b3743e35-69e7-4544-979d-7596c44b59a0\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc" Oct 31 04:33:31.401149 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.398901 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-x829c" Oct 31 04:33:31.415012 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.414037 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3743e35-69e7-4544-979d-7596c44b59a0-istio-podinfo\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc\" (UID: \"b3743e35-69e7-4544-979d-7596c44b59a0\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc" Oct 31 04:33:31.420542 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.420475 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/686f7dd0-02cd-457a-b7cd-ef83520d5341-istio-envoy\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h\" (UID: \"686f7dd0-02cd-457a-b7cd-ef83520d5341\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h" Oct 31 04:33:31.421476 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.421443 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kxpn\" (UniqueName: \"kubernetes.io/projected/686f7dd0-02cd-457a-b7cd-ef83520d5341-kube-api-access-8kxpn\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h\" (UID: \"686f7dd0-02cd-457a-b7cd-ef83520d5341\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h" Oct 31 04:33:31.421764 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.421716 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3743e35-69e7-4544-979d-7596c44b59a0-istio-envoy\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc\" (UID: \"b3743e35-69e7-4544-979d-7596c44b59a0\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc" Oct 31 04:33:31.424681 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.424503 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/686f7dd0-02cd-457a-b7cd-ef83520d5341-istio-token\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h\" (UID: \"686f7dd0-02cd-457a-b7cd-ef83520d5341\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h" Oct 31 04:33:31.440985 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.440958 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc" Oct 31 04:33:31.662595 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.662515 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h" Oct 31 04:33:31.818411 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.818323 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:33:31.818731 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.818696 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:33:31.818931 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.818909 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:33:31.822710 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.822605 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-x829c"] Oct 31 04:33:31.832951 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.832904 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:33:31.833076 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.833054 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:33:31.833174 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.833154 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:33:31.838686 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:31.838640 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc"] Oct 31 04:33:32.052776 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:32.052536 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h"] Oct 31 04:33:32.054784 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:32.054124 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:33:32.054784 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:32.054214 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:33:32.054784 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:32.054259 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:33:32.656393 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:32.656323 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-x829c" event={"ID":"22bd2484-d363-4287-a424-29a99c3153ac","Type":"ContainerStarted","Data":"cc47c4219876b046a1e1f07e9ba418fccb8269f89357a67c4ae0288b24ae6b85"} Oct 31 04:33:32.656611 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:32.656407 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-x829c" event={"ID":"22bd2484-d363-4287-a424-29a99c3153ac","Type":"ContainerStarted","Data":"a28e4ebe40f3403bc5f54aa5afef6172080e9b78150c6571a8d351ddc8836624"} Oct 31 04:33:32.660081 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:32.660052 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc" event={"ID":"b3743e35-69e7-4544-979d-7596c44b59a0","Type":"ContainerStarted","Data":"f5e626561f82ca23306e02890938d51ca0ae716bbbac98b43f6aad8114a67338"} Oct 31 04:33:32.660253 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:32.660239 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc" event={"ID":"b3743e35-69e7-4544-979d-7596c44b59a0","Type":"ContainerStarted","Data":"0ef939667ceafeea31b3ffe57c86d5d5c506ee07fb1d871de9fa7ca40195cbf9"} Oct 31 04:33:32.664204 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:32.663573 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h" event={"ID":"686f7dd0-02cd-457a-b7cd-ef83520d5341","Type":"ContainerStarted","Data":"bcea24c66148803cee1eba69713d84e38d9b2d233d5a8dabc9a5958f7c962a3d"} Oct 31 04:33:32.664204 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:32.663626 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h" event={"ID":"686f7dd0-02cd-457a-b7cd-ef83520d5341","Type":"ContainerStarted","Data":"4051220cd229960ae5eea2a7463f82f6baa0ea7757f88ed1f6ad48485d4be6c4"} Oct 31 04:33:32.737399 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:32.735615 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-x829c" podStartSLOduration=1.735591976 podStartE2EDuration="1.735591976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:33:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:33:32.714807544 +0000 UTC m=+1767.471145584" watchObservedRunningTime="2026-10-31 04:33:32.735591976 +0000 UTC m=+1767.491930017" Oct 31 04:33:32.817599 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:32.817525 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc" podStartSLOduration=1.817505487 podStartE2EDuration="1.817505487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:33:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:33:32.796773482 +0000 UTC m=+1767.553111523" watchObservedRunningTime="2026-10-31 04:33:32.817505487 +0000 UTC m=+1767.573843528" Oct 31 04:33:33.654367 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:33.651870 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc" Oct 31 04:33:33.654367 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:33.651953 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-x829c" Oct 31 04:33:33.670210 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:33.670167 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-x829c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Oct 31 04:33:33.673635 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:33.673582 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-x829c" podUID="22bd2484-d363-4287-a424-29a99c3153ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Oct 31 04:33:33.674887 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:33.674869 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h" Oct 31 04:33:33.685486 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:33.685454 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Oct 31 04:33:33.685684 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:33.685659 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc" podUID="b3743e35-69e7-4544-979d-7596c44b59a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Oct 31 04:33:33.717302 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:33.717265 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body= Oct 31 04:33:33.720466 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:33.720412 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h" podUID="686f7dd0-02cd-457a-b7cd-ef83520d5341" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" Oct 31 04:33:33.951619 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:33.951497 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" podUID="e0f4b8c2-14f4-4052-bf54-dd65bf59e413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:33:33.987486 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:33.987434 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:33:34.056526 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:34.056470 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:33:34.062958 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:34.062918 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" podUID="643433d8-418e-4102-8bc3-cd544932614a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 04:33:34.100357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:34.099473 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" podUID="8e2c7041-ad52-49f4-8b55-d16691f4938f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:33:34.371521 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:34.371466 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" podUID="dccd15a9-47f5-4bd1-b21b-9d5ca8b05845" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:33:34.405064 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:34.405025 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-x829c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Oct 31 04:33:34.405398 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:34.405364 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-x829c" podUID="22bd2484-d363-4287-a424-29a99c3153ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Oct 31 04:33:34.565362 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:34.549933 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Oct 31 04:33:34.573452 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:34.573404 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc" podUID="b3743e35-69e7-4544-979d-7596c44b59a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Oct 31 04:33:34.621356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:34.600678 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" podUID="b4ced3ab-41e4-4c54-93eb-95eddaae992d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Oct 31 04:33:34.695414 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:34.675200 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:33:34.695414 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:34.675775 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body= Oct 31 04:33:34.695414 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:34.675815 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h" podUID="686f7dd0-02cd-457a-b7cd-ef83520d5341" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" Oct 31 04:33:34.710182 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:34.709408 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:33:34.981444 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:34.981402 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" podUID="e0f4b8c2-14f4-4052-bf54-dd65bf59e413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:33:34.982096 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:34.982071 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" Oct 31 04:33:34.991431 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:34.991398 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 31 04:33:34.992047 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:34.992025 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" Oct 31 04:33:35.041998 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:35.041938 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h" podStartSLOduration=4.041919276 podStartE2EDuration="4.041919276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:33:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:33:32.866674543 +0000 UTC m=+1767.623012583" watchObservedRunningTime="2026-10-31 04:33:35.041919276 +0000 UTC m=+1769.798257319" Oct 31 04:33:35.064484 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:35.064406 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" podUID="643433d8-418e-4102-8bc3-cd544932614a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 04:33:35.069401 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:35.069374 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" Oct 31 04:33:35.204401 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:35.111620 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" podUID="8e2c7041-ad52-49f4-8b55-d16691f4938f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:33:35.204401 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:35.165008 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" Oct 31 04:33:35.364436 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:35.343896 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" podUID="b4ced3ab-41e4-4c54-93eb-95eddaae992d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:33:35.376364 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:35.374675 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" podUID="dccd15a9-47f5-4bd1-b21b-9d5ca8b05845" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 31 04:33:35.376364 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:35.375208 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" Oct 31 04:33:35.455359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:35.430910 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-x829c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Oct 31 04:33:35.460413 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:35.455666 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-x829c" podUID="22bd2484-d363-4287-a424-29a99c3153ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Oct 31 04:33:36.123355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:36.119686 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" podUID="e0f4b8c2-14f4-4052-bf54-dd65bf59e413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 04:33:36.123355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:36.120742 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" podUID="643433d8-418e-4102-8bc3-cd544932614a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 04:33:36.174386 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:35.760396 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:33:36.245873 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:36.237718 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 31 04:33:36.246796 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:35.473577 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Oct 31 04:33:36.247081 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:36.247053 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc" podUID="b3743e35-69e7-4544-979d-7596c44b59a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Oct 31 04:33:36.247232 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:36.176906 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body= Oct 31 04:33:36.247418 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:36.247389 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h" podUID="686f7dd0-02cd-457a-b7cd-ef83520d5341" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" Oct 31 04:33:36.326546 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:36.325699 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 31 04:33:36.356255 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:36.356217 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" podUID="b4ced3ab-41e4-4c54-93eb-95eddaae992d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Oct 31 04:33:36.384697 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:36.384594 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" podUID="dccd15a9-47f5-4bd1-b21b-9d5ca8b05845" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:33:36.445832 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:36.445795 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Oct 31 04:33:36.446648 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:36.446612 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc" podUID="b3743e35-69e7-4544-979d-7596c44b59a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Oct 31 04:33:36.537231 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:36.537185 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-x829c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Oct 31 04:33:36.537566 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:36.537514 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-x829c" podUID="22bd2484-d363-4287-a424-29a99c3153ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Oct 31 04:33:36.564093 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:33:36.564044 5019 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.164s" Oct 31 04:33:36.565655 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:36.565634 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" Oct 31 04:33:36.565925 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:36.565873 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" Oct 31 04:33:36.566762 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:36.566747 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" Oct 31 04:33:36.568579 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:36.568564 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" Oct 31 04:33:36.569240 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:36.569225 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" Oct 31 04:33:36.639531 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:36.639445 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-7229f2-798cdf44cf-4wm7j" Oct 31 04:33:36.666008 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:36.665636 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body= Oct 31 04:33:36.666008 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:36.665686 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h" podUID="686f7dd0-02cd-457a-b7cd-ef83520d5341" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" Oct 31 04:33:37.136370 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:37.134385 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" podUID="643433d8-418e-4102-8bc3-cd544932614a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 04:33:37.199393 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:37.174767 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" podUID="e0f4b8c2-14f4-4052-bf54-dd65bf59e413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 04:33:37.352354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:37.328466 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 31 04:33:37.352354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:37.329549 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 31 04:33:37.369528 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:37.367050 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" podUID="b4ced3ab-41e4-4c54-93eb-95eddaae992d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:33:37.406413 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:37.392018 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" podUID="dccd15a9-47f5-4bd1-b21b-9d5ca8b05845" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 31 04:33:37.504126 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:37.451304 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Oct 31 04:33:37.504126 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:37.451424 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc" podUID="b3743e35-69e7-4544-979d-7596c44b59a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Oct 31 04:33:37.534357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:37.528826 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-x829c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Oct 31 04:33:37.534357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:37.528890 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-x829c" podUID="22bd2484-d363-4287-a424-29a99c3153ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Oct 31 04:33:37.703021 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:37.671051 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body= Oct 31 04:33:37.703021 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:37.671123 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h" podUID="686f7dd0-02cd-457a-b7cd-ef83520d5341" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" Oct 31 04:33:37.784617 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:37.784573 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 04:33:37.840122 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:37.837388 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:33:38.095374 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:38.091828 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" podUID="643433d8-418e-4102-8bc3-cd544932614a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:33:38.095374 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:38.092066 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" podUID="8e2c7041-ad52-49f4-8b55-d16691f4938f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": context deadline exceeded" Oct 31 04:33:38.135678 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:38.135630 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" podUID="643433d8-418e-4102-8bc3-cd544932614a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 04:33:38.236452 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:38.236402 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" podUID="e0f4b8c2-14f4-4052-bf54-dd65bf59e413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:33:38.331650 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:38.329425 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 31 04:33:38.331650 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:38.330458 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:33:38.356463 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:38.334532 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" podUID="8e2c7041-ad52-49f4-8b55-d16691f4938f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:33:38.389354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:38.371466 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" podUID="b4ced3ab-41e4-4c54-93eb-95eddaae992d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:33:38.403461 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:38.403411 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" podUID="dccd15a9-47f5-4bd1-b21b-9d5ca8b05845" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 31 04:33:38.416983 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:38.416953 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-x829c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Oct 31 04:33:38.417233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:38.417205 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-x829c" podUID="22bd2484-d363-4287-a424-29a99c3153ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Oct 31 04:33:38.510652 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:38.510617 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Oct 31 04:33:38.510921 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:38.510894 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc" podUID="b3743e35-69e7-4544-979d-7596c44b59a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Oct 31 04:33:38.686412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:38.685456 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body= Oct 31 04:33:38.686412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:38.685517 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h" podUID="686f7dd0-02cd-457a-b7cd-ef83520d5341" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" Oct 31 04:33:38.801371 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:38.785322 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:33:38.801371 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:38.786451 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" Oct 31 04:33:39.173415 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:39.173376 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" podUID="643433d8-418e-4102-8bc3-cd544932614a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 04:33:39.245664 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:39.242419 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" podUID="e0f4b8c2-14f4-4052-bf54-dd65bf59e413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 04:33:39.366426 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:39.346434 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" podUID="8e2c7041-ad52-49f4-8b55-d16691f4938f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:33:39.366426 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:39.346869 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" Oct 31 04:33:39.366426 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:39.354409 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 31 04:33:39.366426 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:39.355711 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 31 04:33:39.398785 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:39.397444 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" podUID="b4ced3ab-41e4-4c54-93eb-95eddaae992d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Oct 31 04:33:39.496357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:39.467428 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" podUID="dccd15a9-47f5-4bd1-b21b-9d5ca8b05845" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 31 04:33:39.822251 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:39.821844 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:33:39.831563 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:39.828060 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body= Oct 31 04:33:39.831563 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:39.828127 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h" podUID="686f7dd0-02cd-457a-b7cd-ef83520d5341" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" Oct 31 04:33:39.831563 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:39.828606 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Oct 31 04:33:39.831563 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:39.828653 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc" podUID="b3743e35-69e7-4544-979d-7596c44b59a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Oct 31 04:33:39.831563 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:39.828738 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-x829c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Oct 31 04:33:39.831563 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:39.828770 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-x829c" podUID="22bd2484-d363-4287-a424-29a99c3153ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Oct 31 04:33:40.154411 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:40.153994 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" Oct 31 04:33:40.195738 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:40.194746 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" podUID="643433d8-418e-4102-8bc3-cd544932614a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:33:40.259476 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:40.258242 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" podUID="e0f4b8c2-14f4-4052-bf54-dd65bf59e413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:33:40.374386 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:40.365870 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" Oct 31 04:33:40.374386 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:40.368061 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:33:40.374386 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:40.368372 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 31 04:33:40.395795 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:40.393589 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" Oct 31 04:33:40.418004 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:40.417916 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg"] Oct 31 04:33:40.420511 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:40.420460 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-x829c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Oct 31 04:33:40.420657 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:40.420517 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-x829c" podUID="22bd2484-d363-4287-a424-29a99c3153ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Oct 31 04:33:40.439543 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:40.439519 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg"] Oct 31 04:33:40.445971 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:40.441658 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" podUID="b4ced3ab-41e4-4c54-93eb-95eddaae992d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:33:40.447574 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:40.445012 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Oct 31 04:33:40.447574 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:40.446436 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc" podUID="b3743e35-69e7-4544-979d-7596c44b59a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Oct 31 04:33:40.447574 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:40.445088 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg" Oct 31 04:33:40.470845 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:40.468628 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" Oct 31 04:33:40.470845 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:40.469443 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" podUID="dccd15a9-47f5-4bd1-b21b-9d5ca8b05845" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:33:40.488382 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:40.481484 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d12de313-f7d1-4d6e-b2d7-0727073b032c-credential-socket\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg\" (UID: \"d12de313-f7d1-4d6e-b2d7-0727073b032c\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg" Oct 31 04:33:40.488382 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:40.481538 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d12de313-f7d1-4d6e-b2d7-0727073b032c-istio-token\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg\" (UID: \"d12de313-f7d1-4d6e-b2d7-0727073b032c\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg" Oct 31 04:33:40.488382 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:40.481583 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d12de313-f7d1-4d6e-b2d7-0727073b032c-istio-podinfo\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg\" (UID: \"d12de313-f7d1-4d6e-b2d7-0727073b032c\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg" Oct 31 04:33:40.488382 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:40.481624 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d12de313-f7d1-4d6e-b2d7-0727073b032c-workload-certs\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg\" (UID: \"d12de313-f7d1-4d6e-b2d7-0727073b032c\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg" Oct 31 04:33:40.488382 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:40.481648 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d12de313-f7d1-4d6e-b2d7-0727073b032c-istio-envoy\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg\" (UID: \"d12de313-f7d1-4d6e-b2d7-0727073b032c\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg" Oct 31 04:33:40.488382 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:40.481842 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcfnn\" (UniqueName: \"kubernetes.io/projected/d12de313-f7d1-4d6e-b2d7-0727073b032c-kube-api-access-jcfnn\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg\" (UID: \"d12de313-f7d1-4d6e-b2d7-0727073b032c\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg" Oct 31 04:33:40.488382 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:40.481937 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d12de313-f7d1-4d6e-b2d7-0727073b032c-workload-socket\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg\" (UID: \"d12de313-f7d1-4d6e-b2d7-0727073b032c\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg" Oct 31 04:33:40.488382 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:40.481969 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d12de313-f7d1-4d6e-b2d7-0727073b032c-istiod-ca-cert\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg\" (UID: \"d12de313-f7d1-4d6e-b2d7-0727073b032c\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg" Oct 31 04:33:40.501352 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:40.496913 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d12de313-f7d1-4d6e-b2d7-0727073b032c-istio-data\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg\" (UID: \"d12de313-f7d1-4d6e-b2d7-0727073b032c\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg" Oct 31 04:33:40.600680 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:40.600646 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" Oct 31 04:33:40.640118 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:40.604596 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d12de313-f7d1-4d6e-b2d7-0727073b032c-credential-socket\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg\" (UID: \"d12de313-f7d1-4d6e-b2d7-0727073b032c\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg" Oct 31 04:33:40.640118 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:40.604648 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d12de313-f7d1-4d6e-b2d7-0727073b032c-istio-token\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg\" (UID: \"d12de313-f7d1-4d6e-b2d7-0727073b032c\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg" Oct 31 04:33:40.640118 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:40.604693 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d12de313-f7d1-4d6e-b2d7-0727073b032c-istio-podinfo\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg\" (UID: \"d12de313-f7d1-4d6e-b2d7-0727073b032c\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg" Oct 31 04:33:40.640118 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:40.604734 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d12de313-f7d1-4d6e-b2d7-0727073b032c-workload-certs\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg\" (UID: \"d12de313-f7d1-4d6e-b2d7-0727073b032c\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg" Oct 31 04:33:40.640118 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:40.604759 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d12de313-f7d1-4d6e-b2d7-0727073b032c-istio-envoy\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg\" (UID: \"d12de313-f7d1-4d6e-b2d7-0727073b032c\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg" Oct 31 04:33:40.640118 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:40.604891 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcfnn\" (UniqueName: \"kubernetes.io/projected/d12de313-f7d1-4d6e-b2d7-0727073b032c-kube-api-access-jcfnn\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg\" (UID: \"d12de313-f7d1-4d6e-b2d7-0727073b032c\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg" Oct 31 04:33:40.640118 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:40.604984 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d12de313-f7d1-4d6e-b2d7-0727073b032c-workload-socket\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg\" (UID: \"d12de313-f7d1-4d6e-b2d7-0727073b032c\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg" Oct 31 04:33:40.640118 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:40.605017 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d12de313-f7d1-4d6e-b2d7-0727073b032c-istiod-ca-cert\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg\" (UID: \"d12de313-f7d1-4d6e-b2d7-0727073b032c\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg" Oct 31 04:33:40.640118 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:40.605108 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d12de313-f7d1-4d6e-b2d7-0727073b032c-istio-data\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg\" (UID: \"d12de313-f7d1-4d6e-b2d7-0727073b032c\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg" Oct 31 04:33:40.640118 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:40.605744 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d12de313-f7d1-4d6e-b2d7-0727073b032c-istio-data\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg\" (UID: \"d12de313-f7d1-4d6e-b2d7-0727073b032c\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg" Oct 31 04:33:40.663966 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:40.663940 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" Oct 31 04:33:40.665839 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:40.665791 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body= Oct 31 04:33:40.666952 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:40.665846 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h" podUID="686f7dd0-02cd-457a-b7cd-ef83520d5341" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" Oct 31 04:33:40.667523 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:40.667500 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" Oct 31 04:33:40.670981 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:40.670523 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" Oct 31 04:33:40.687682 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:40.687653 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d12de313-f7d1-4d6e-b2d7-0727073b032c-credential-socket\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg\" (UID: \"d12de313-f7d1-4d6e-b2d7-0727073b032c\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg" Oct 31 04:33:40.689137 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:40.689113 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d12de313-f7d1-4d6e-b2d7-0727073b032c-workload-certs\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg\" (UID: \"d12de313-f7d1-4d6e-b2d7-0727073b032c\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg" Oct 31 04:33:40.704381 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:40.691073 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d12de313-f7d1-4d6e-b2d7-0727073b032c-istiod-ca-cert\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg\" (UID: \"d12de313-f7d1-4d6e-b2d7-0727073b032c\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg" Oct 31 04:33:40.704563 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:40.691311 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d12de313-f7d1-4d6e-b2d7-0727073b032c-workload-socket\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg\" (UID: \"d12de313-f7d1-4d6e-b2d7-0727073b032c\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg" Oct 31 04:33:40.725791 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:40.725636 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d12de313-f7d1-4d6e-b2d7-0727073b032c-istio-envoy\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg\" (UID: \"d12de313-f7d1-4d6e-b2d7-0727073b032c\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg" Oct 31 04:33:40.738854 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:40.738829 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d12de313-f7d1-4d6e-b2d7-0727073b032c-istio-podinfo\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg\" (UID: \"d12de313-f7d1-4d6e-b2d7-0727073b032c\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg" Oct 31 04:33:40.745316 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:40.745274 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcfnn\" (UniqueName: \"kubernetes.io/projected/d12de313-f7d1-4d6e-b2d7-0727073b032c-kube-api-access-jcfnn\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg\" (UID: \"d12de313-f7d1-4d6e-b2d7-0727073b032c\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg" Oct 31 04:33:40.748243 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:40.748197 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d12de313-f7d1-4d6e-b2d7-0727073b032c-istio-token\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg\" (UID: \"d12de313-f7d1-4d6e-b2d7-0727073b032c\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg" Oct 31 04:33:40.860488 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:40.857808 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg" Oct 31 04:33:41.324738 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:41.324042 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg"] Oct 31 04:33:41.328660 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:33:41.328627 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd12de313_f7d1_4d6e_b2d7_0727073b032c.slice/crio-db4374bb5e4e0ddacb0295a966a4e6ebff6fd406886d24596e19f09b3c424509 WatchSource:0}: Error finding container db4374bb5e4e0ddacb0295a966a4e6ebff6fd406886d24596e19f09b3c424509: Status 404 returned error can't find the container with id db4374bb5e4e0ddacb0295a966a4e6ebff6fd406886d24596e19f09b3c424509 Oct 31 04:33:41.334836 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:41.333620 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:33:41.334836 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:41.333702 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:33:41.334836 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:41.333743 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:33:41.403390 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:41.402940 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-x829c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Oct 31 04:33:41.403390 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:41.402993 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-x829c" podUID="22bd2484-d363-4287-a424-29a99c3153ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Oct 31 04:33:41.442203 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:41.442174 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Oct 31 04:33:41.442432 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:41.442403 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc" podUID="b3743e35-69e7-4544-979d-7596c44b59a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Oct 31 04:33:41.457769 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:41.457734 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg" event={"ID":"d12de313-f7d1-4d6e-b2d7-0727073b032c","Type":"ContainerStarted","Data":"db4374bb5e4e0ddacb0295a966a4e6ebff6fd406886d24596e19f09b3c424509"} Oct 31 04:33:41.672407 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:41.669529 5019 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body= Oct 31 04:33:41.672407 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:41.669585 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h" podUID="686f7dd0-02cd-457a-b7cd-ef83520d5341" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" Oct 31 04:33:42.495907 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:42.494684 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg" event={"ID":"d12de313-f7d1-4d6e-b2d7-0727073b032c","Type":"ContainerStarted","Data":"4d2ad2b39c00ac5b93bac98d010d32ed1103e0db1d05e46753c724c707f4fc0f"} Oct 31 04:33:42.559302 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:42.556966 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc" Oct 31 04:33:42.559302 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:42.558559 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc" Oct 31 04:33:42.574344 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:42.571286 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc" Oct 31 04:33:42.577868 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:42.574843 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg" podStartSLOduration=2.574825025 podStartE2EDuration="2.574825025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:33:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:33:42.552262287 +0000 UTC m=+1777.308600329" watchObservedRunningTime="2026-10-31 04:33:42.574825025 +0000 UTC m=+1777.331163091" Oct 31 04:33:42.646817 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:42.644309 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-x829c" Oct 31 04:33:42.646817 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:42.645695 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-x829c" Oct 31 04:33:42.650635 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:42.650568 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-x829c" Oct 31 04:33:42.862380 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:42.862278 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg" Oct 31 04:33:42.862674 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:42.862468 5019 patch_prober.go:29] interesting pod/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Oct 31 04:33:42.862831 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:42.862808 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg" podUID="d12de313-f7d1-4d6e-b2d7-0727073b032c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Oct 31 04:33:42.931436 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:42.931403 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h" Oct 31 04:33:42.933795 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:42.933667 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h" Oct 31 04:33:42.944411 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:42.938983 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h" Oct 31 04:33:43.709080 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:43.709036 5019 patch_prober.go:29] interesting pod/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:33:43.709688 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:43.709650 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7" podUID="5581a822-03f0-4f8c-bd5e-444274a4bd9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:33:43.861123 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:43.861085 5019 patch_prober.go:29] interesting pod/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Oct 31 04:33:43.861322 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:43.861152 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg" podUID="d12de313-f7d1-4d6e-b2d7-0727073b032c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Oct 31 04:33:44.861611 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:44.861572 5019 patch_prober.go:29] interesting pod/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Oct 31 04:33:44.862011 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:44.861638 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg" podUID="d12de313-f7d1-4d6e-b2d7-0727073b032c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Oct 31 04:33:45.865412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:45.864393 5019 patch_prober.go:29] interesting pod/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Oct 31 04:33:45.865412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:45.864453 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg" podUID="d12de313-f7d1-4d6e-b2d7-0727073b032c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Oct 31 04:33:46.862093 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:46.862056 5019 patch_prober.go:29] interesting pod/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Oct 31 04:33:46.862252 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:46.862121 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg" podUID="d12de313-f7d1-4d6e-b2d7-0727073b032c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Oct 31 04:33:47.861454 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:47.861377 5019 patch_prober.go:29] interesting pod/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Oct 31 04:33:47.861454 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:47.861428 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg" podUID="d12de313-f7d1-4d6e-b2d7-0727073b032c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Oct 31 04:33:48.554196 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:48.554162 5019 generic.go:356] "Generic (PLEG): container finished" podID="5581a822-03f0-4f8c-bd5e-444274a4bd9b" containerID="7a46c6e9ba6fea80c750c3d47dcad8df0c6a3dfc624839b3d723fdf3fb7d9e2c" exitCode=0 Oct 31 04:33:48.554415 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:48.554243 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7" event={"ID":"5581a822-03f0-4f8c-bd5e-444274a4bd9b","Type":"ContainerDied","Data":"7a46c6e9ba6fea80c750c3d47dcad8df0c6a3dfc624839b3d723fdf3fb7d9e2c"} Oct 31 04:33:48.554783 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:48.554768 5019 scope.go:122] "RemoveContainer" containerID="7a46c6e9ba6fea80c750c3d47dcad8df0c6a3dfc624839b3d723fdf3fb7d9e2c" Oct 31 04:33:48.556476 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:48.556444 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:33:48.556591 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:48.556512 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:33:48.556591 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:48.556544 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:33:48.867116 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:48.867031 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg" Oct 31 04:33:48.867640 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:48.867567 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg" Oct 31 04:33:48.868288 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:48.868269 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg" Oct 31 04:33:49.566566 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:49.566528 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7" event={"ID":"5581a822-03f0-4f8c-bd5e-444274a4bd9b","Type":"ContainerStarted","Data":"af4ee7a37af498ec39a99b06bccebfb4238df4bba896e77312020e27b4555f17"} Oct 31 04:33:50.368932 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:50.368887 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7" Oct 31 04:33:50.369504 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:50.369035 5019 patch_prober.go:29] interesting pod/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Oct 31 04:33:50.369504 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:50.369089 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7" podUID="5581a822-03f0-4f8c-bd5e-444274a4bd9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Oct 31 04:33:51.369249 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:51.369213 5019 patch_prober.go:29] interesting pod/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Oct 31 04:33:51.369856 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:51.369826 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7" podUID="5581a822-03f0-4f8c-bd5e-444274a4bd9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Oct 31 04:33:52.306794 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:52.306758 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-c9707f/auth-test-model-c9707f-kserve-f5d44dd69-kqt6t"] Oct 31 04:33:52.314150 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:52.314122 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-c9707f/auth-test-model-c9707f-kserve-f5d44dd69-kqt6t" Oct 31 04:33:52.316140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:52.316105 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-c9707f/auth-test-model-c9707f-kserve-f5d44dd69-kqt6t"] Oct 31 04:33:52.331315 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:52.331250 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-c9707f\"/\"openshift-service-ca.crt\"" Oct 31 04:33:52.342681 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:52.342636 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-c9707f\"/\"auth-test-model-c9707f-kserve-self-signed-certs\"" Oct 31 04:33:52.342809 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:52.342763 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-c9707f\"/\"kube-root-ca.crt\"" Oct 31 04:33:52.342936 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:52.342923 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-c9707f\"/\"default-dockercfg-j6c7m\"" Oct 31 04:33:52.394354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:52.369148 5019 patch_prober.go:29] interesting pod/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Oct 31 04:33:52.394354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:52.369205 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7" podUID="5581a822-03f0-4f8c-bd5e-444274a4bd9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Oct 31 04:33:52.414386 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:52.411465 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91ef8aa6-c2a0-4a20-8131-89dd0b63514d-dshm\") pod \"auth-test-model-c9707f-kserve-f5d44dd69-kqt6t\" (UID: \"91ef8aa6-c2a0-4a20-8131-89dd0b63514d\") " pod="ai-tenant-e2e-shared-a-w2-c9707f/auth-test-model-c9707f-kserve-f5d44dd69-kqt6t" Oct 31 04:33:52.421208 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:52.419489 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91ef8aa6-c2a0-4a20-8131-89dd0b63514d-home\") pod \"auth-test-model-c9707f-kserve-f5d44dd69-kqt6t\" (UID: \"91ef8aa6-c2a0-4a20-8131-89dd0b63514d\") " pod="ai-tenant-e2e-shared-a-w2-c9707f/auth-test-model-c9707f-kserve-f5d44dd69-kqt6t" Oct 31 04:33:52.421208 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:52.419565 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91ef8aa6-c2a0-4a20-8131-89dd0b63514d-model-cache\") pod \"auth-test-model-c9707f-kserve-f5d44dd69-kqt6t\" (UID: \"91ef8aa6-c2a0-4a20-8131-89dd0b63514d\") " pod="ai-tenant-e2e-shared-a-w2-c9707f/auth-test-model-c9707f-kserve-f5d44dd69-kqt6t" Oct 31 04:33:52.421208 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:52.419726 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91ef8aa6-c2a0-4a20-8131-89dd0b63514d-tls-certs\") pod \"auth-test-model-c9707f-kserve-f5d44dd69-kqt6t\" (UID: \"91ef8aa6-c2a0-4a20-8131-89dd0b63514d\") " pod="ai-tenant-e2e-shared-a-w2-c9707f/auth-test-model-c9707f-kserve-f5d44dd69-kqt6t" Oct 31 04:33:52.421208 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:52.419788 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqvjn\" (UniqueName: \"kubernetes.io/projected/91ef8aa6-c2a0-4a20-8131-89dd0b63514d-kube-api-access-wqvjn\") pod \"auth-test-model-c9707f-kserve-f5d44dd69-kqt6t\" (UID: \"91ef8aa6-c2a0-4a20-8131-89dd0b63514d\") " pod="ai-tenant-e2e-shared-a-w2-c9707f/auth-test-model-c9707f-kserve-f5d44dd69-kqt6t" Oct 31 04:33:52.421208 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:52.419841 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91ef8aa6-c2a0-4a20-8131-89dd0b63514d-tmp-dir\") pod \"auth-test-model-c9707f-kserve-f5d44dd69-kqt6t\" (UID: \"91ef8aa6-c2a0-4a20-8131-89dd0b63514d\") " pod="ai-tenant-e2e-shared-a-w2-c9707f/auth-test-model-c9707f-kserve-f5d44dd69-kqt6t" Oct 31 04:33:52.442261 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:33:52.442223 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:33:52.522561 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:52.522531 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91ef8aa6-c2a0-4a20-8131-89dd0b63514d-home\") pod \"auth-test-model-c9707f-kserve-f5d44dd69-kqt6t\" (UID: \"91ef8aa6-c2a0-4a20-8131-89dd0b63514d\") " pod="ai-tenant-e2e-shared-a-w2-c9707f/auth-test-model-c9707f-kserve-f5d44dd69-kqt6t" Oct 31 04:33:52.522792 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:52.522604 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91ef8aa6-c2a0-4a20-8131-89dd0b63514d-model-cache\") pod \"auth-test-model-c9707f-kserve-f5d44dd69-kqt6t\" (UID: \"91ef8aa6-c2a0-4a20-8131-89dd0b63514d\") " pod="ai-tenant-e2e-shared-a-w2-c9707f/auth-test-model-c9707f-kserve-f5d44dd69-kqt6t" Oct 31 04:33:52.522903 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:52.522799 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91ef8aa6-c2a0-4a20-8131-89dd0b63514d-tls-certs\") pod \"auth-test-model-c9707f-kserve-f5d44dd69-kqt6t\" (UID: \"91ef8aa6-c2a0-4a20-8131-89dd0b63514d\") " pod="ai-tenant-e2e-shared-a-w2-c9707f/auth-test-model-c9707f-kserve-f5d44dd69-kqt6t" Oct 31 04:33:52.522903 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:52.522864 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqvjn\" (UniqueName: \"kubernetes.io/projected/91ef8aa6-c2a0-4a20-8131-89dd0b63514d-kube-api-access-wqvjn\") pod \"auth-test-model-c9707f-kserve-f5d44dd69-kqt6t\" (UID: \"91ef8aa6-c2a0-4a20-8131-89dd0b63514d\") " pod="ai-tenant-e2e-shared-a-w2-c9707f/auth-test-model-c9707f-kserve-f5d44dd69-kqt6t" Oct 31 04:33:52.523070 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:52.522941 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91ef8aa6-c2a0-4a20-8131-89dd0b63514d-tmp-dir\") pod \"auth-test-model-c9707f-kserve-f5d44dd69-kqt6t\" (UID: \"91ef8aa6-c2a0-4a20-8131-89dd0b63514d\") " pod="ai-tenant-e2e-shared-a-w2-c9707f/auth-test-model-c9707f-kserve-f5d44dd69-kqt6t" Oct 31 04:33:52.523070 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:52.522997 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91ef8aa6-c2a0-4a20-8131-89dd0b63514d-dshm\") pod \"auth-test-model-c9707f-kserve-f5d44dd69-kqt6t\" (UID: \"91ef8aa6-c2a0-4a20-8131-89dd0b63514d\") " pod="ai-tenant-e2e-shared-a-w2-c9707f/auth-test-model-c9707f-kserve-f5d44dd69-kqt6t" Oct 31 04:33:52.540399 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:52.537673 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91ef8aa6-c2a0-4a20-8131-89dd0b63514d-dshm\") pod \"auth-test-model-c9707f-kserve-f5d44dd69-kqt6t\" (UID: \"91ef8aa6-c2a0-4a20-8131-89dd0b63514d\") " pod="ai-tenant-e2e-shared-a-w2-c9707f/auth-test-model-c9707f-kserve-f5d44dd69-kqt6t" Oct 31 04:33:52.540399 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:52.538574 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91ef8aa6-c2a0-4a20-8131-89dd0b63514d-tmp-dir\") pod \"auth-test-model-c9707f-kserve-f5d44dd69-kqt6t\" (UID: \"91ef8aa6-c2a0-4a20-8131-89dd0b63514d\") " pod="ai-tenant-e2e-shared-a-w2-c9707f/auth-test-model-c9707f-kserve-f5d44dd69-kqt6t" Oct 31 04:33:52.540399 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:52.539155 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91ef8aa6-c2a0-4a20-8131-89dd0b63514d-home\") pod \"auth-test-model-c9707f-kserve-f5d44dd69-kqt6t\" (UID: \"91ef8aa6-c2a0-4a20-8131-89dd0b63514d\") " pod="ai-tenant-e2e-shared-a-w2-c9707f/auth-test-model-c9707f-kserve-f5d44dd69-kqt6t" Oct 31 04:33:52.546491 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:52.545754 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91ef8aa6-c2a0-4a20-8131-89dd0b63514d-model-cache\") pod \"auth-test-model-c9707f-kserve-f5d44dd69-kqt6t\" (UID: \"91ef8aa6-c2a0-4a20-8131-89dd0b63514d\") " pod="ai-tenant-e2e-shared-a-w2-c9707f/auth-test-model-c9707f-kserve-f5d44dd69-kqt6t" Oct 31 04:33:52.547271 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:52.547213 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91ef8aa6-c2a0-4a20-8131-89dd0b63514d-tls-certs\") pod \"auth-test-model-c9707f-kserve-f5d44dd69-kqt6t\" (UID: \"91ef8aa6-c2a0-4a20-8131-89dd0b63514d\") " pod="ai-tenant-e2e-shared-a-w2-c9707f/auth-test-model-c9707f-kserve-f5d44dd69-kqt6t" Oct 31 04:33:52.601399 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:52.595084 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqvjn\" (UniqueName: \"kubernetes.io/projected/91ef8aa6-c2a0-4a20-8131-89dd0b63514d-kube-api-access-wqvjn\") pod \"auth-test-model-c9707f-kserve-f5d44dd69-kqt6t\" (UID: \"91ef8aa6-c2a0-4a20-8131-89dd0b63514d\") " pod="ai-tenant-e2e-shared-a-w2-c9707f/auth-test-model-c9707f-kserve-f5d44dd69-kqt6t" Oct 31 04:33:52.650357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:52.643422 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-c9707f/auth-test-model-c9707f-kserve-f5d44dd69-kqt6t" Oct 31 04:33:53.076353 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:33:53.069318 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91ef8aa6_c2a0_4a20_8131_89dd0b63514d.slice/crio-74ae7be518e52c58738d03ea85f61d3f85e10a6793cebd680ec307b736397bf8 WatchSource:0}: Error finding container 74ae7be518e52c58738d03ea85f61d3f85e10a6793cebd680ec307b736397bf8: Status 404 returned error can't find the container with id 74ae7be518e52c58738d03ea85f61d3f85e10a6793cebd680ec307b736397bf8 Oct 31 04:33:53.076562 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:53.070326 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-c9707f/auth-test-model-c9707f-kserve-f5d44dd69-kqt6t"] Oct 31 04:33:53.369090 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:53.368993 5019 patch_prober.go:29] interesting pod/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Oct 31 04:33:53.369300 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:53.369276 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7" podUID="5581a822-03f0-4f8c-bd5e-444274a4bd9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Oct 31 04:33:53.804357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:53.801091 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-c9707f/auth-test-model-c9707f-kserve-f5d44dd69-kqt6t" event={"ID":"91ef8aa6-c2a0-4a20-8131-89dd0b63514d","Type":"ContainerStarted","Data":"5a24e8c3a3ba411b2248e297e9eca0a97ba2ea208923ebdb1a8e847b1dfd7d10"} Oct 31 04:33:53.804357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:53.801144 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-c9707f/auth-test-model-c9707f-kserve-f5d44dd69-kqt6t" event={"ID":"91ef8aa6-c2a0-4a20-8131-89dd0b63514d","Type":"ContainerStarted","Data":"74ae7be518e52c58738d03ea85f61d3f85e10a6793cebd680ec307b736397bf8"} Oct 31 04:33:53.873697 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:53.873631 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-c9707f/auth-test-model-c9707f-kserve-f5d44dd69-kqt6t" podStartSLOduration=1.873610652 podStartE2EDuration="1.873610652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:33:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:33:53.852093625 +0000 UTC m=+1788.608431666" watchObservedRunningTime="2026-10-31 04:33:53.873610652 +0000 UTC m=+1788.629948711" Oct 31 04:33:54.368662 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:54.368630 5019 patch_prober.go:29] interesting pod/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Oct 31 04:33:54.368867 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:54.368712 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7" podUID="5581a822-03f0-4f8c-bd5e-444274a4bd9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Oct 31 04:33:54.644889 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:54.644795 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-c9707f/auth-test-model-c9707f-kserve-f5d44dd69-kqt6t" Oct 31 04:33:55.373248 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:55.373216 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7" Oct 31 04:33:55.373817 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:55.373798 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7" Oct 31 04:33:55.374685 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:55.374662 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7" Oct 31 04:33:59.140481 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:59.140446 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq"] Oct 31 04:33:59.145519 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:59.145493 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq" Oct 31 04:33:59.175898 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:59.175870 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq"] Oct 31 04:33:59.179568 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:59.179537 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-d6ac82-openshift-default-dockercfg-nkwnj\"" Oct 31 04:33:59.293370 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:59.293318 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-istio-data\") pod \"e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq\" (UID: \"3f1a76d9-21b3-4831-a4b8-fd07b96cec87\") " pod="openshift-ingress/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq" Oct 31 04:33:59.293709 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:59.293681 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-istio-podinfo\") pod \"e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq\" (UID: \"3f1a76d9-21b3-4831-a4b8-fd07b96cec87\") " pod="openshift-ingress/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq" Oct 31 04:33:59.293913 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:59.293893 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-istio-envoy\") pod \"e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq\" (UID: \"3f1a76d9-21b3-4831-a4b8-fd07b96cec87\") " pod="openshift-ingress/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq" Oct 31 04:33:59.297459 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:59.297434 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-workload-certs\") pod \"e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq\" (UID: \"3f1a76d9-21b3-4831-a4b8-fd07b96cec87\") " pod="openshift-ingress/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq" Oct 31 04:33:59.297621 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:59.297602 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxzfh\" (UniqueName: \"kubernetes.io/projected/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-kube-api-access-xxzfh\") pod \"e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq\" (UID: \"3f1a76d9-21b3-4831-a4b8-fd07b96cec87\") " pod="openshift-ingress/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq" Oct 31 04:33:59.297818 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:59.297802 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-istiod-ca-cert\") pod \"e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq\" (UID: \"3f1a76d9-21b3-4831-a4b8-fd07b96cec87\") " pod="openshift-ingress/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq" Oct 31 04:33:59.310583 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:59.310552 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-workload-socket\") pod \"e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq\" (UID: \"3f1a76d9-21b3-4831-a4b8-fd07b96cec87\") " pod="openshift-ingress/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq" Oct 31 04:33:59.310833 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:59.310811 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-istio-token\") pod \"e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq\" (UID: \"3f1a76d9-21b3-4831-a4b8-fd07b96cec87\") " pod="openshift-ingress/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq" Oct 31 04:33:59.311006 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:59.310987 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-credential-socket\") pod \"e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq\" (UID: \"3f1a76d9-21b3-4831-a4b8-fd07b96cec87\") " pod="openshift-ingress/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq" Oct 31 04:33:59.428653 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:59.428562 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-istio-podinfo\") pod \"e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq\" (UID: \"3f1a76d9-21b3-4831-a4b8-fd07b96cec87\") " pod="openshift-ingress/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq" Oct 31 04:33:59.439501 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:59.439461 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-istio-envoy\") pod \"e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq\" (UID: \"3f1a76d9-21b3-4831-a4b8-fd07b96cec87\") " pod="openshift-ingress/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq" Oct 31 04:33:59.439834 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:59.439813 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-workload-certs\") pod \"e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq\" (UID: \"3f1a76d9-21b3-4831-a4b8-fd07b96cec87\") " pod="openshift-ingress/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq" Oct 31 04:33:59.439969 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:59.439952 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxzfh\" (UniqueName: \"kubernetes.io/projected/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-kube-api-access-xxzfh\") pod \"e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq\" (UID: \"3f1a76d9-21b3-4831-a4b8-fd07b96cec87\") " pod="openshift-ingress/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq" Oct 31 04:33:59.440179 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:59.440163 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-istiod-ca-cert\") pod \"e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq\" (UID: \"3f1a76d9-21b3-4831-a4b8-fd07b96cec87\") " pod="openshift-ingress/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq" Oct 31 04:33:59.440399 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:59.440379 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-workload-socket\") pod \"e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq\" (UID: \"3f1a76d9-21b3-4831-a4b8-fd07b96cec87\") " pod="openshift-ingress/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq" Oct 31 04:33:59.440549 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:59.440534 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-istio-token\") pod \"e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq\" (UID: \"3f1a76d9-21b3-4831-a4b8-fd07b96cec87\") " pod="openshift-ingress/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq" Oct 31 04:33:59.440658 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:59.440645 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-credential-socket\") pod \"e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq\" (UID: \"3f1a76d9-21b3-4831-a4b8-fd07b96cec87\") " pod="openshift-ingress/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq" Oct 31 04:33:59.440861 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:59.440849 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-istio-data\") pod \"e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq\" (UID: \"3f1a76d9-21b3-4831-a4b8-fd07b96cec87\") " pod="openshift-ingress/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq" Oct 31 04:33:59.441426 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:59.441404 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-istio-data\") pod \"e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq\" (UID: \"3f1a76d9-21b3-4831-a4b8-fd07b96cec87\") " pod="openshift-ingress/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq" Oct 31 04:33:59.449234 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:59.449210 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-istiod-ca-cert\") pod \"e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq\" (UID: \"3f1a76d9-21b3-4831-a4b8-fd07b96cec87\") " pod="openshift-ingress/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq" Oct 31 04:33:59.451947 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:59.451080 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-workload-certs\") pod \"e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq\" (UID: \"3f1a76d9-21b3-4831-a4b8-fd07b96cec87\") " pod="openshift-ingress/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq" Oct 31 04:33:59.451947 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:59.451360 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-workload-socket\") pod \"e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq\" (UID: \"3f1a76d9-21b3-4831-a4b8-fd07b96cec87\") " pod="openshift-ingress/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq" Oct 31 04:33:59.451947 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:59.451866 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-credential-socket\") pod \"e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq\" (UID: \"3f1a76d9-21b3-4831-a4b8-fd07b96cec87\") " pod="openshift-ingress/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq" Oct 31 04:33:59.458963 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:59.458942 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-istio-envoy\") pod \"e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq\" (UID: \"3f1a76d9-21b3-4831-a4b8-fd07b96cec87\") " pod="openshift-ingress/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq" Oct 31 04:33:59.502567 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:59.498090 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-istio-podinfo\") pod \"e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq\" (UID: \"3f1a76d9-21b3-4831-a4b8-fd07b96cec87\") " pod="openshift-ingress/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq" Oct 31 04:33:59.523144 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:59.510265 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-istio-token\") pod \"e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq\" (UID: \"3f1a76d9-21b3-4831-a4b8-fd07b96cec87\") " pod="openshift-ingress/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq" Oct 31 04:33:59.571612 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:59.567804 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxzfh\" (UniqueName: \"kubernetes.io/projected/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-kube-api-access-xxzfh\") pod \"e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq\" (UID: \"3f1a76d9-21b3-4831-a4b8-fd07b96cec87\") " pod="openshift-ingress/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq" Oct 31 04:33:59.775192 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:33:59.775162 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq" Oct 31 04:34:00.095538 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:34:00.095508 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f1a76d9_21b3_4831_a4b8_fd07b96cec87.slice/crio-ea45e52808a22c0864c1e2bdbea37593682e35308026dc329947e0f78d573f6b WatchSource:0}: Error finding container ea45e52808a22c0864c1e2bdbea37593682e35308026dc329947e0f78d573f6b: Status 404 returned error can't find the container with id ea45e52808a22c0864c1e2bdbea37593682e35308026dc329947e0f78d573f6b Oct 31 04:34:00.095924 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:00.095902 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq"] Oct 31 04:34:00.098694 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:00.098657 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:34:00.098823 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:00.098739 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:34:00.098823 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:00.098783 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:34:00.919638 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:00.919596 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq" event={"ID":"3f1a76d9-21b3-4831-a4b8-fd07b96cec87","Type":"ContainerStarted","Data":"90b1550534a1e97e636d4c271604b6e7b7b77a2751d5228485f696b2255c5b17"} Oct 31 04:34:00.920048 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:00.919659 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq" event={"ID":"3f1a76d9-21b3-4831-a4b8-fd07b96cec87","Type":"ContainerStarted","Data":"ea45e52808a22c0864c1e2bdbea37593682e35308026dc329947e0f78d573f6b"} Oct 31 04:34:00.982733 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:00.982680 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq" podStartSLOduration=1.982665299 podStartE2EDuration="1.982665299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:33:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:34:00.97705152 +0000 UTC m=+1795.733389583" watchObservedRunningTime="2026-10-31 04:34:00.982665299 +0000 UTC m=+1795.739003338" Oct 31 04:34:01.775444 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:01.775404 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq" Oct 31 04:34:01.781175 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:01.781147 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq" Oct 31 04:34:01.930878 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:01.930840 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq" Oct 31 04:34:01.931963 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:01.931940 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq" Oct 31 04:34:02.653364 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:02.651515 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-c9707f/auth-test-model-c9707f-kserve-f5d44dd69-kqt6t" Oct 31 04:34:02.761357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:02.760871 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-c9707f/auth-test-model-c9707f-kserve-f5d44dd69-kqt6t" Oct 31 04:34:03.310357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:03.309687 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-c9707f/auth-test-model-c9707f-kserve-f5d44dd69-kqt6t" Oct 31 04:34:04.071392 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:04.069462 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 04:34:04.340411 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:04.339678 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:34:06.970362 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:06.969414 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-c9707f/auth-test-model-c9707f-kserve-f5d44dd69-kqt6t" podUID="91ef8aa6-c2a0-4a20-8131-89dd0b63514d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.250:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:34:07.169029 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:07.168962 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 31 04:34:07.348415 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:07.281233 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" podUID="643433d8-418e-4102-8bc3-cd544932614a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 04:34:07.348415 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:07.298431 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" podUID="e0f4b8c2-14f4-4052-bf54-dd65bf59e413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 04:34:07.348415 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:07.335413 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:34:07.408832 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:07.408773 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" podUID="b4ced3ab-41e4-4c54-93eb-95eddaae992d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:34:07.431763 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:07.431711 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:34:07.462024 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:07.461982 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" podUID="8e2c7041-ad52-49f4-8b55-d16691f4938f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:34:07.489224 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:07.489174 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" podUID="dccd15a9-47f5-4bd1-b21b-9d5ca8b05845" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 04:34:07.823941 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:07.823911 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s88kp_3945ff65-e877-4372-9334-3968c68b7295/console-operator/2.log" Oct 31 04:34:07.887423 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:07.886892 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:34:07.909356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:07.895578 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s88kp_3945ff65-e877-4372-9334-3968c68b7295/console-operator/2.log" Oct 31 04:34:07.946404 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:07.943555 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" podUID="8e2c7041-ad52-49f4-8b55-d16691f4938f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:34:07.994600 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:07.994556 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-c9707f/auth-test-model-c9707f-kserve-f5d44dd69-kqt6t" podUID="91ef8aa6-c2a0-4a20-8131-89dd0b63514d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.250:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:34:07.996051 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:07.996027 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-c9707f/auth-test-model-c9707f-kserve-f5d44dd69-kqt6t" Oct 31 04:34:08.086414 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:08.085459 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" podUID="643433d8-418e-4102-8bc3-cd544932614a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:34:08.184355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:08.178381 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:34:08.229424 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:08.190649 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" Oct 31 04:34:08.334351 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:08.319325 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" podUID="643433d8-418e-4102-8bc3-cd544932614a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 04:34:08.334351 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:08.319876 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" Oct 31 04:34:08.343605 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:08.343497 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:34:08.365353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:08.362667 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" Oct 31 04:34:08.365353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:08.364788 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-c9707f/auth-test-model-c9707f-kserve-f5d44dd69-kqt6t" Oct 31 04:34:08.394854 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:08.369251 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" Oct 31 04:34:08.438020 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:08.437844 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" Oct 31 04:34:08.465772 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:08.465223 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" Oct 31 04:34:09.150816 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:09.146149 5019 scope.go:122] "RemoveContainer" containerID="c8070e1d0199dde9e4637f7bcc0c68218eaa11e1f91dfbc6eee8b82dd4913fb7" Oct 31 04:34:13.996572 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:13.996523 5019 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:34:13.997251 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:13.996601 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6" podUID="68b1965c-0483-419e-9e73-aacf0f6a5048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:34:16.759571 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:16.750107 5019 patch_prober.go:29] interesting pod/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body= Oct 31 04:34:16.759571 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:16.750179 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz" podUID="2d771867-36b5-4b55-afb4-f889d1d91784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" Oct 31 04:34:16.802956 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:16.802851 5019 generic.go:356] "Generic (PLEG): container finished" podID="2d771867-36b5-4b55-afb4-f889d1d91784" containerID="85a2b9dd4c1a3477ce87dd382e1a0c28fde2e96c96b8d364cc1769bfa3a2ae46" exitCode=0 Oct 31 04:34:16.802956 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:16.802891 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz" event={"ID":"2d771867-36b5-4b55-afb4-f889d1d91784","Type":"ContainerDied","Data":"85a2b9dd4c1a3477ce87dd382e1a0c28fde2e96c96b8d364cc1769bfa3a2ae46"} Oct 31 04:34:16.803504 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:16.803476 5019 scope.go:122] "RemoveContainer" containerID="85a2b9dd4c1a3477ce87dd382e1a0c28fde2e96c96b8d364cc1769bfa3a2ae46" Oct 31 04:34:16.805651 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:16.805571 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:34:16.805783 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:16.805665 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:34:16.805783 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:16.805705 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:34:17.815490 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:17.815454 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz" event={"ID":"2d771867-36b5-4b55-afb4-f889d1d91784","Type":"ContainerStarted","Data":"a8c165f1cf4f8c49f2f4ed1ad38f945ec38654b0460f84dd3abe49d97607631f"} Oct 31 04:34:17.817708 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:17.817678 5019 generic.go:356] "Generic (PLEG): container finished" podID="22f2a6e7-c1d8-4834-80d3-805cf819e158" containerID="55b10d00a9cbae70610d3dddfad1eaa18550d082df6f4faa78043ac9e03bbf31" exitCode=0 Oct 31 04:34:17.817876 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:17.817746 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s" event={"ID":"22f2a6e7-c1d8-4834-80d3-805cf819e158","Type":"ContainerDied","Data":"55b10d00a9cbae70610d3dddfad1eaa18550d082df6f4faa78043ac9e03bbf31"} Oct 31 04:34:17.818248 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:17.818231 5019 scope.go:122] "RemoveContainer" containerID="55b10d00a9cbae70610d3dddfad1eaa18550d082df6f4faa78043ac9e03bbf31" Oct 31 04:34:17.819905 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:17.819880 5019 generic.go:356] "Generic (PLEG): container finished" podID="68b1965c-0483-419e-9e73-aacf0f6a5048" containerID="2928eb4559d6aa4257028602a5358f1f1b47cccf6cbb91de2759b7b36e0b94c1" exitCode=0 Oct 31 04:34:17.820051 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:17.819963 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6" event={"ID":"68b1965c-0483-419e-9e73-aacf0f6a5048","Type":"ContainerDied","Data":"2928eb4559d6aa4257028602a5358f1f1b47cccf6cbb91de2759b7b36e0b94c1"} Oct 31 04:34:17.820306 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:17.820274 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:34:17.820416 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:17.820354 5019 scope.go:122] "RemoveContainer" containerID="2928eb4559d6aa4257028602a5358f1f1b47cccf6cbb91de2759b7b36e0b94c1" Oct 31 04:34:17.820416 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:17.820384 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:34:17.820535 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:17.820438 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:34:17.822271 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:17.822236 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:34:17.822388 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:17.822324 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:34:17.822452 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:17.822397 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:34:18.734305 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:18.734268 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz" Oct 31 04:34:18.736452 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:18.736421 5019 patch_prober.go:29] interesting pod/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body= Oct 31 04:34:18.736719 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:18.736692 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz" podUID="2d771867-36b5-4b55-afb4-f889d1d91784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" Oct 31 04:34:18.760261 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:18.759472 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s" Oct 31 04:34:18.857969 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:18.857920 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s" event={"ID":"22f2a6e7-c1d8-4834-80d3-805cf819e158","Type":"ContainerStarted","Data":"9b639dda4a875f7a915fd3f30eae68a6793d1056cc8a334a0e15f440fac2de45"} Oct 31 04:34:18.860998 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:18.860962 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6" event={"ID":"68b1965c-0483-419e-9e73-aacf0f6a5048","Type":"ContainerStarted","Data":"c7aee1c0f539e53fdd244f018e7f9224350d313b492072495922872fb4aa6e97"} Oct 31 04:34:19.021840 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:19.018633 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6" Oct 31 04:34:19.021840 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:19.018760 5019 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Oct 31 04:34:19.021840 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:19.018805 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6" podUID="68b1965c-0483-419e-9e73-aacf0f6a5048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Oct 31 04:34:19.053391 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:19.049010 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-338f2e/auth-test-model-338f2e-kserve-79676f7df-lwb9d"] Oct 31 04:34:19.061385 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:19.057388 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-338f2e/auth-test-model-338f2e-kserve-79676f7df-lwb9d" Oct 31 04:34:19.155359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:19.133142 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-338f2e/auth-test-model-338f2e-kserve-79676f7df-lwb9d"] Oct 31 04:34:19.163641 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:19.151704 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-338f2e\"/\"openshift-service-ca.crt\"" Oct 31 04:34:19.163641 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:19.151998 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-338f2e\"/\"default-dockercfg-7vg4l\"" Oct 31 04:34:19.163963 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:19.152101 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-338f2e\"/\"auth-test-model-338f2e-kserve-self-signed-certs\"" Oct 31 04:34:19.163963 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:19.160839 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-338f2e\"/\"kube-root-ca.crt\"" Oct 31 04:34:19.253239 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:19.253204 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s" Oct 31 04:34:19.285609 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:19.260074 5019 patch_prober.go:29] interesting pod/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Oct 31 04:34:19.285609 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:19.260161 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s" podUID="22f2a6e7-c1d8-4834-80d3-805cf819e158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Oct 31 04:34:19.485407 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:19.484111 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d03e187b-a841-486c-8a3d-bce5d3adee70-tmp-dir\") pod \"auth-test-model-338f2e-kserve-79676f7df-lwb9d\" (UID: \"d03e187b-a841-486c-8a3d-bce5d3adee70\") " pod="ai-tenant-e2e-shared-a-w3-338f2e/auth-test-model-338f2e-kserve-79676f7df-lwb9d" Oct 31 04:34:19.485407 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:19.484179 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d03e187b-a841-486c-8a3d-bce5d3adee70-dshm\") pod \"auth-test-model-338f2e-kserve-79676f7df-lwb9d\" (UID: \"d03e187b-a841-486c-8a3d-bce5d3adee70\") " pod="ai-tenant-e2e-shared-a-w3-338f2e/auth-test-model-338f2e-kserve-79676f7df-lwb9d" Oct 31 04:34:19.485407 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:19.484297 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d03e187b-a841-486c-8a3d-bce5d3adee70-tls-certs\") pod \"auth-test-model-338f2e-kserve-79676f7df-lwb9d\" (UID: \"d03e187b-a841-486c-8a3d-bce5d3adee70\") " pod="ai-tenant-e2e-shared-a-w3-338f2e/auth-test-model-338f2e-kserve-79676f7df-lwb9d" Oct 31 04:34:19.485407 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:19.484396 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5d2g\" (UniqueName: \"kubernetes.io/projected/d03e187b-a841-486c-8a3d-bce5d3adee70-kube-api-access-w5d2g\") pod \"auth-test-model-338f2e-kserve-79676f7df-lwb9d\" (UID: \"d03e187b-a841-486c-8a3d-bce5d3adee70\") " pod="ai-tenant-e2e-shared-a-w3-338f2e/auth-test-model-338f2e-kserve-79676f7df-lwb9d" Oct 31 04:34:19.485407 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:19.484493 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d03e187b-a841-486c-8a3d-bce5d3adee70-model-cache\") pod \"auth-test-model-338f2e-kserve-79676f7df-lwb9d\" (UID: \"d03e187b-a841-486c-8a3d-bce5d3adee70\") " pod="ai-tenant-e2e-shared-a-w3-338f2e/auth-test-model-338f2e-kserve-79676f7df-lwb9d" Oct 31 04:34:19.485407 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:19.484545 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d03e187b-a841-486c-8a3d-bce5d3adee70-home\") pod \"auth-test-model-338f2e-kserve-79676f7df-lwb9d\" (UID: \"d03e187b-a841-486c-8a3d-bce5d3adee70\") " pod="ai-tenant-e2e-shared-a-w3-338f2e/auth-test-model-338f2e-kserve-79676f7df-lwb9d" Oct 31 04:34:19.751426 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:19.675195 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5d2g\" (UniqueName: \"kubernetes.io/projected/d03e187b-a841-486c-8a3d-bce5d3adee70-kube-api-access-w5d2g\") pod \"auth-test-model-338f2e-kserve-79676f7df-lwb9d\" (UID: \"d03e187b-a841-486c-8a3d-bce5d3adee70\") " pod="ai-tenant-e2e-shared-a-w3-338f2e/auth-test-model-338f2e-kserve-79676f7df-lwb9d" Oct 31 04:34:19.788561 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:19.788511 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d03e187b-a841-486c-8a3d-bce5d3adee70-model-cache\") pod \"auth-test-model-338f2e-kserve-79676f7df-lwb9d\" (UID: \"d03e187b-a841-486c-8a3d-bce5d3adee70\") " pod="ai-tenant-e2e-shared-a-w3-338f2e/auth-test-model-338f2e-kserve-79676f7df-lwb9d" Oct 31 04:34:19.788872 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:19.788851 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d03e187b-a841-486c-8a3d-bce5d3adee70-home\") pod \"auth-test-model-338f2e-kserve-79676f7df-lwb9d\" (UID: \"d03e187b-a841-486c-8a3d-bce5d3adee70\") " pod="ai-tenant-e2e-shared-a-w3-338f2e/auth-test-model-338f2e-kserve-79676f7df-lwb9d" Oct 31 04:34:19.789348 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:19.789301 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d03e187b-a841-486c-8a3d-bce5d3adee70-tmp-dir\") pod \"auth-test-model-338f2e-kserve-79676f7df-lwb9d\" (UID: \"d03e187b-a841-486c-8a3d-bce5d3adee70\") " pod="ai-tenant-e2e-shared-a-w3-338f2e/auth-test-model-338f2e-kserve-79676f7df-lwb9d" Oct 31 04:34:19.796360 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:19.789523 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d03e187b-a841-486c-8a3d-bce5d3adee70-dshm\") pod \"auth-test-model-338f2e-kserve-79676f7df-lwb9d\" (UID: \"d03e187b-a841-486c-8a3d-bce5d3adee70\") " pod="ai-tenant-e2e-shared-a-w3-338f2e/auth-test-model-338f2e-kserve-79676f7df-lwb9d" Oct 31 04:34:19.822361 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:19.796850 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d03e187b-a841-486c-8a3d-bce5d3adee70-tls-certs\") pod \"auth-test-model-338f2e-kserve-79676f7df-lwb9d\" (UID: \"d03e187b-a841-486c-8a3d-bce5d3adee70\") " pod="ai-tenant-e2e-shared-a-w3-338f2e/auth-test-model-338f2e-kserve-79676f7df-lwb9d" Oct 31 04:34:19.837374 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:19.727555 5019 patch_prober.go:29] interesting pod/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body= Oct 31 04:34:19.837374 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:19.833289 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz" podUID="2d771867-36b5-4b55-afb4-f889d1d91784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" Oct 31 04:34:19.837374 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:19.809879 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d03e187b-a841-486c-8a3d-bce5d3adee70-home\") pod \"auth-test-model-338f2e-kserve-79676f7df-lwb9d\" (UID: \"d03e187b-a841-486c-8a3d-bce5d3adee70\") " pod="ai-tenant-e2e-shared-a-w3-338f2e/auth-test-model-338f2e-kserve-79676f7df-lwb9d" Oct 31 04:34:19.837374 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:19.810530 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d03e187b-a841-486c-8a3d-bce5d3adee70-tmp-dir\") pod \"auth-test-model-338f2e-kserve-79676f7df-lwb9d\" (UID: \"d03e187b-a841-486c-8a3d-bce5d3adee70\") " pod="ai-tenant-e2e-shared-a-w3-338f2e/auth-test-model-338f2e-kserve-79676f7df-lwb9d" Oct 31 04:34:19.837374 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:19.809391 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d03e187b-a841-486c-8a3d-bce5d3adee70-model-cache\") pod \"auth-test-model-338f2e-kserve-79676f7df-lwb9d\" (UID: \"d03e187b-a841-486c-8a3d-bce5d3adee70\") " pod="ai-tenant-e2e-shared-a-w3-338f2e/auth-test-model-338f2e-kserve-79676f7df-lwb9d" Oct 31 04:34:19.890372 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:19.889795 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5d2g\" (UniqueName: \"kubernetes.io/projected/d03e187b-a841-486c-8a3d-bce5d3adee70-kube-api-access-w5d2g\") pod \"auth-test-model-338f2e-kserve-79676f7df-lwb9d\" (UID: \"d03e187b-a841-486c-8a3d-bce5d3adee70\") " pod="ai-tenant-e2e-shared-a-w3-338f2e/auth-test-model-338f2e-kserve-79676f7df-lwb9d" Oct 31 04:34:19.920358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:19.919725 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d03e187b-a841-486c-8a3d-bce5d3adee70-dshm\") pod \"auth-test-model-338f2e-kserve-79676f7df-lwb9d\" (UID: \"d03e187b-a841-486c-8a3d-bce5d3adee70\") " pod="ai-tenant-e2e-shared-a-w3-338f2e/auth-test-model-338f2e-kserve-79676f7df-lwb9d" Oct 31 04:34:19.945358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:19.934060 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d03e187b-a841-486c-8a3d-bce5d3adee70-tls-certs\") pod \"auth-test-model-338f2e-kserve-79676f7df-lwb9d\" (UID: \"d03e187b-a841-486c-8a3d-bce5d3adee70\") " pod="ai-tenant-e2e-shared-a-w3-338f2e/auth-test-model-338f2e-kserve-79676f7df-lwb9d" Oct 31 04:34:20.044418 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:20.044058 5019 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Oct 31 04:34:20.044418 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:20.044121 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6" podUID="68b1965c-0483-419e-9e73-aacf0f6a5048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Oct 31 04:34:20.169381 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:20.166666 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-338f2e/auth-test-model-338f2e-kserve-79676f7df-lwb9d" Oct 31 04:34:20.229389 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:20.201743 5019 patch_prober.go:29] interesting pod/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Oct 31 04:34:20.229389 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:20.201811 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s" podUID="22f2a6e7-c1d8-4834-80d3-805cf819e158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Oct 31 04:34:20.338324 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:20.337741 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:34:20.452554 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:20.452489 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" podUID="8e2c7041-ad52-49f4-8b55-d16691f4938f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:34:20.453251 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:20.452834 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" podUID="643433d8-418e-4102-8bc3-cd544932614a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:34:20.475360 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:20.468448 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" podUID="b4ced3ab-41e4-4c54-93eb-95eddaae992d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:34:20.475360 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:20.468874 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:34:20.513397 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:20.488477 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 04:34:20.746360 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:20.736228 5019 patch_prober.go:29] interesting pod/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body= Oct 31 04:34:20.746360 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:20.736297 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz" podUID="2d771867-36b5-4b55-afb4-f889d1d91784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" Oct 31 04:34:20.888728 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:20.888430 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:34:21.042426 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:21.030785 5019 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Oct 31 04:34:21.042426 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:21.030853 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6" podUID="68b1965c-0483-419e-9e73-aacf0f6a5048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Oct 31 04:34:21.220315 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:21.219947 5019 patch_prober.go:29] interesting pod/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Oct 31 04:34:21.220315 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:21.220013 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s" podUID="22f2a6e7-c1d8-4834-80d3-805cf819e158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Oct 31 04:34:21.297423 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:21.269808 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" podUID="e0f4b8c2-14f4-4052-bf54-dd65bf59e413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:34:21.458457 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:21.432436 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-c9707f/auth-test-model-c9707f-kserve-f5d44dd69-kqt6t" podUID="91ef8aa6-c2a0-4a20-8131-89dd0b63514d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.250:8000/ready\": context deadline exceeded" Oct 31 04:34:21.458457 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:21.433202 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" podUID="dccd15a9-47f5-4bd1-b21b-9d5ca8b05845" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 31 04:34:21.462361 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:21.459758 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" podUID="643433d8-418e-4102-8bc3-cd544932614a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 04:34:21.462361 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:21.460739 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" podUID="8e2c7041-ad52-49f4-8b55-d16691f4938f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 31 04:34:21.480368 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:21.474395 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:34:21.504785 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:21.497843 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:34:21.554757 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:21.554640 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 31 04:34:21.575085 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:21.562231 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" podUID="b4ced3ab-41e4-4c54-93eb-95eddaae992d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:34:21.944359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:21.942824 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" Oct 31 04:34:21.944359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:21.942906 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-338f2e/auth-test-model-338f2e-kserve-79676f7df-lwb9d"] Oct 31 04:34:21.944359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:21.943001 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" Oct 31 04:34:21.944359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:21.943081 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" Oct 31 04:34:21.944359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:21.943156 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" Oct 31 04:34:21.944359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:21.943227 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" Oct 31 04:34:21.944359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:21.943310 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" Oct 31 04:34:22.003358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:22.002925 5019 patch_prober.go:29] interesting pod/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body= Oct 31 04:34:22.003358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:22.003003 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz" podUID="2d771867-36b5-4b55-afb4-f889d1d91784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" Oct 31 04:34:22.065134 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:22.065088 5019 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Oct 31 04:34:22.065815 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:22.065780 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6" podUID="68b1965c-0483-419e-9e73-aacf0f6a5048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Oct 31 04:34:22.269644 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:22.258168 5019 patch_prober.go:29] interesting pod/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Oct 31 04:34:22.368392 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:22.368298 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s" podUID="22f2a6e7-c1d8-4834-80d3-805cf819e158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Oct 31 04:34:22.489817 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:22.488119 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" podUID="643433d8-418e-4102-8bc3-cd544932614a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:34:22.490119 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:22.490068 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" podUID="8e2c7041-ad52-49f4-8b55-d16691f4938f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:34:22.508999 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:22.508473 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:34:22.508999 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:22.508809 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 31 04:34:22.577560 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:22.566600 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:34:22.577560 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:22.568383 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" podUID="b4ced3ab-41e4-4c54-93eb-95eddaae992d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:34:22.789013 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:22.788954 5019 patch_prober.go:29] interesting pod/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body= Oct 31 04:34:23.130462 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:23.130401 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz" podUID="2d771867-36b5-4b55-afb4-f889d1d91784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" Oct 31 04:34:23.154979 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:23.027463 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" podUID="e0f4b8c2-14f4-4052-bf54-dd65bf59e413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:34:23.155418 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:34:23.117018 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:34:23.155630 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:22.867292 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" Oct 31 04:34:23.155824 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:23.116293 5019 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Oct 31 04:34:23.196437 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:23.155985 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6" podUID="68b1965c-0483-419e-9e73-aacf0f6a5048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Oct 31 04:34:23.197317 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:23.197288 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" Oct 31 04:34:23.315601 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:23.313538 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" Oct 31 04:34:23.315601 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:23.313590 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-338f2e/auth-test-model-338f2e-kserve-79676f7df-lwb9d" event={"ID":"d03e187b-a841-486c-8a3d-bce5d3adee70","Type":"ContainerStarted","Data":"2ba7ddba50271c3122c6af1bc49cff973d8a3626191c2f24cacbca90a14bbb5b"} Oct 31 04:34:23.315601 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:23.313640 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" Oct 31 04:34:23.315601 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:23.297069 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" podUID="e0f4b8c2-14f4-4052-bf54-dd65bf59e413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:34:23.408543 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:23.326899 5019 patch_prober.go:29] interesting pod/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Oct 31 04:34:23.408543 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:23.326980 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s" podUID="22f2a6e7-c1d8-4834-80d3-805cf819e158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Oct 31 04:34:23.500661 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:23.500611 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-c9707f/auth-test-model-c9707f-kserve-f5d44dd69-kqt6t" podUID="91ef8aa6-c2a0-4a20-8131-89dd0b63514d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.250:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:34:23.545438 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:23.519464 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" podUID="dccd15a9-47f5-4bd1-b21b-9d5ca8b05845" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 04:34:23.600482 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:23.600421 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 31 04:34:23.658322 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:23.658272 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-a-w2-c9707f/auth-test-model-c9707f-kserve-f5d44dd69-kqt6t" podUID="91ef8aa6-c2a0-4a20-8131-89dd0b63514d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.250:8000/health\": context deadline exceeded" Oct 31 04:34:23.658789 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:23.658719 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" podUID="8e2c7041-ad52-49f4-8b55-d16691f4938f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 31 04:34:23.850368 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:23.846457 5019 patch_prober.go:29] interesting pod/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body= Oct 31 04:34:23.878374 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:23.870437 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz" podUID="2d771867-36b5-4b55-afb4-f889d1d91784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" Oct 31 04:34:24.024708 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:24.024661 5019 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Oct 31 04:34:24.025035 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:24.025000 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6" podUID="68b1965c-0483-419e-9e73-aacf0f6a5048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Oct 31 04:34:24.055985 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:24.055934 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:34:24.203359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:24.203283 5019 patch_prober.go:29] interesting pod/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Oct 31 04:34:24.203910 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:24.203375 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s" podUID="22f2a6e7-c1d8-4834-80d3-805cf819e158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Oct 31 04:34:24.263443 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:24.261961 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" Oct 31 04:34:24.285002 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:24.284900 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" Oct 31 04:34:24.734418 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:24.734381 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz" Oct 31 04:34:24.734752 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:24.734730 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz" Oct 31 04:34:24.735829 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:24.735810 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz" Oct 31 04:34:25.018911 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:25.018822 5019 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Oct 31 04:34:25.019093 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:25.018909 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6" podUID="68b1965c-0483-419e-9e73-aacf0f6a5048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Oct 31 04:34:25.226004 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:25.225954 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-338f2e/auth-test-model-338f2e-kserve-79676f7df-lwb9d" event={"ID":"d03e187b-a841-486c-8a3d-bce5d3adee70","Type":"ContainerStarted","Data":"03ad844a32c33a419fcbc42eb42e91eb7e0560f7ce57684e0e70eeab7d2a31b0"} Oct 31 04:34:25.269403 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:25.269200 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-338f2e/auth-test-model-338f2e-kserve-79676f7df-lwb9d" podStartSLOduration=7.269180859 podStartE2EDuration="7.269180859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:34:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:34:25.267798717 +0000 UTC m=+1820.024136758" watchObservedRunningTime="2026-10-31 04:34:25.269180859 +0000 UTC m=+1820.025518904" Oct 31 04:34:25.295662 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:25.295624 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s" Oct 31 04:34:25.296287 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:25.296252 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s" Oct 31 04:34:25.298309 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:25.297537 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s" Oct 31 04:34:26.024563 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:26.024529 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6" Oct 31 04:34:26.025024 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:26.024998 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6" Oct 31 04:34:26.025730 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:26.025708 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6" Oct 31 04:34:26.167346 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:26.167304 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-338f2e/auth-test-model-338f2e-kserve-79676f7df-lwb9d" Oct 31 04:34:27.166764 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:27.166719 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd"] Oct 31 04:34:27.167268 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:27.166945 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd" podUID="d4062db4-f8d8-4cbb-8f2d-428c06d30db7" containerName="istio-proxy" containerID="cri-o://16def1431858b071d0663d1233f8aa1864e13c85002f304693ecb8a306b6ad09" gracePeriod=30 Oct 31 04:34:30.173627 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:30.173585 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-338f2e/auth-test-model-338f2e-kserve-79676f7df-lwb9d" Oct 31 04:34:30.195871 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:30.195834 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-338f2e/auth-test-model-338f2e-kserve-79676f7df-lwb9d" Oct 31 04:34:30.304104 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:30.304068 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-338f2e/auth-test-model-338f2e-kserve-79676f7df-lwb9d" Oct 31 04:34:32.312067 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:32.312028 5019 generic.go:356] "Generic (PLEG): container finished" podID="d4062db4-f8d8-4cbb-8f2d-428c06d30db7" containerID="16def1431858b071d0663d1233f8aa1864e13c85002f304693ecb8a306b6ad09" exitCode=0 Oct 31 04:34:32.312557 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:32.312079 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd" event={"ID":"d4062db4-f8d8-4cbb-8f2d-428c06d30db7","Type":"ContainerDied","Data":"16def1431858b071d0663d1233f8aa1864e13c85002f304693ecb8a306b6ad09"} Oct 31 04:34:32.946318 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:32.946220 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd" Oct 31 04:34:33.083345 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:33.083272 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-credential-socket\") pod \"d4062db4-f8d8-4cbb-8f2d-428c06d30db7\" (UID: \"d4062db4-f8d8-4cbb-8f2d-428c06d30db7\") " Oct 31 04:34:33.086535 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:33.083427 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-istio-data\") pod \"d4062db4-f8d8-4cbb-8f2d-428c06d30db7\" (UID: \"d4062db4-f8d8-4cbb-8f2d-428c06d30db7\") " Oct 31 04:34:33.086535 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:33.083506 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-workload-certs\") pod \"d4062db4-f8d8-4cbb-8f2d-428c06d30db7\" (UID: \"d4062db4-f8d8-4cbb-8f2d-428c06d30db7\") " Oct 31 04:34:33.086535 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:33.083564 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-kube-api-access-8qrr6\" (UniqueName: \"kubernetes.io/projected/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-kube-api-access-8qrr6\") pod \"d4062db4-f8d8-4cbb-8f2d-428c06d30db7\" (UID: \"d4062db4-f8d8-4cbb-8f2d-428c06d30db7\") " Oct 31 04:34:33.086535 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:33.083736 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-istio-envoy\") pod \"d4062db4-f8d8-4cbb-8f2d-428c06d30db7\" (UID: \"d4062db4-f8d8-4cbb-8f2d-428c06d30db7\") " Oct 31 04:34:33.086535 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:33.083789 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-istio-token\" (UniqueName: \"kubernetes.io/projected/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-istio-token\") pod \"d4062db4-f8d8-4cbb-8f2d-428c06d30db7\" (UID: \"d4062db4-f8d8-4cbb-8f2d-428c06d30db7\") " Oct 31 04:34:33.086535 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:33.083875 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-workload-socket\") pod \"d4062db4-f8d8-4cbb-8f2d-428c06d30db7\" (UID: \"d4062db4-f8d8-4cbb-8f2d-428c06d30db7\") " Oct 31 04:34:33.086535 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:33.083905 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-istio-podinfo\") pod \"d4062db4-f8d8-4cbb-8f2d-428c06d30db7\" (UID: \"d4062db4-f8d8-4cbb-8f2d-428c06d30db7\") " Oct 31 04:34:33.086535 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:33.084000 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-istiod-ca-cert\") pod \"d4062db4-f8d8-4cbb-8f2d-428c06d30db7\" (UID: \"d4062db4-f8d8-4cbb-8f2d-428c06d30db7\") " Oct 31 04:34:33.086535 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:33.085315 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-istiod-ca-cert" pod "d4062db4-f8d8-4cbb-8f2d-428c06d30db7" (UID: "d4062db4-f8d8-4cbb-8f2d-428c06d30db7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:34:33.086535 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:33.085648 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-credential-socket" pod "d4062db4-f8d8-4cbb-8f2d-428c06d30db7" (UID: "d4062db4-f8d8-4cbb-8f2d-428c06d30db7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:34:33.086535 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:33.085835 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-istio-data" pod "d4062db4-f8d8-4cbb-8f2d-428c06d30db7" (UID: "d4062db4-f8d8-4cbb-8f2d-428c06d30db7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:34:33.086535 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:33.086020 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-workload-certs" pod "d4062db4-f8d8-4cbb-8f2d-428c06d30db7" (UID: "d4062db4-f8d8-4cbb-8f2d-428c06d30db7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:34:33.089008 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:33.088932 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-workload-socket" pod "d4062db4-f8d8-4cbb-8f2d-428c06d30db7" (UID: "d4062db4-f8d8-4cbb-8f2d-428c06d30db7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:34:33.096680 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:33.096638 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-kube-api-access-8qrr6" pod "d4062db4-f8d8-4cbb-8f2d-428c06d30db7" (UID: "d4062db4-f8d8-4cbb-8f2d-428c06d30db7"). InnerVolumeSpecName "kube-api-access-8qrr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:34:33.100805 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:33.100749 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-istio-envoy" pod "d4062db4-f8d8-4cbb-8f2d-428c06d30db7" (UID: "d4062db4-f8d8-4cbb-8f2d-428c06d30db7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:34:33.100805 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:33.100770 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-istio-podinfo" pod "d4062db4-f8d8-4cbb-8f2d-428c06d30db7" (UID: "d4062db4-f8d8-4cbb-8f2d-428c06d30db7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:34:33.102518 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:33.102478 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-istio-token" pod "d4062db4-f8d8-4cbb-8f2d-428c06d30db7" (UID: "d4062db4-f8d8-4cbb-8f2d-428c06d30db7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:34:33.190461 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:33.189986 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:34:33.190461 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:33.190027 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:34:33.190461 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:33.190043 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:34:33.190461 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:33.190057 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qrr6\" (UniqueName: \"kubernetes.io/projected/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-kube-api-access-8qrr6\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:34:33.190461 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:33.190070 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:34:33.190461 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:33.190083 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:34:33.190461 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:33.190095 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:34:33.190461 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:33.190108 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:34:33.190461 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:33.190121 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4062db4-f8d8-4cbb-8f2d-428c06d30db7-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:34:33.331670 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:33.331623 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd" event={"ID":"d4062db4-f8d8-4cbb-8f2d-428c06d30db7","Type":"ContainerDied","Data":"4a1ca431a8623371f11770acb01da89be78c911740acd76355df09a172bf8232"} Oct 31 04:34:33.332211 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:33.331755 5019 scope.go:122] "RemoveContainer" containerID="16def1431858b071d0663d1233f8aa1864e13c85002f304693ecb8a306b6ad09" Oct 31 04:34:33.332211 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:33.332003 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd" Oct 31 04:34:33.415149 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:33.415108 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd"] Oct 31 04:34:33.415350 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:33.415164 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-b4nmd"] Oct 31 04:34:34.081366 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:34.080148 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b11c770-bfxfp"] Oct 31 04:34:34.101414 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:34.098208 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b11c770-bfxfp"] Oct 31 04:34:35.412822 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:35.412780 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="692297c2-7276-498d-9b3b-5941673eacd1" path="/var/lib/kubelet/pods/692297c2-7276-498d-9b3b-5941673eacd1/volumes" Oct 31 04:34:35.413528 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:35.413503 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4062db4-f8d8-4cbb-8f2d-428c06d30db7" path="/var/lib/kubelet/pods/d4062db4-f8d8-4cbb-8f2d-428c06d30db7/volumes" Oct 31 04:34:36.909373 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:36.909309 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45"] Oct 31 04:34:36.909927 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:36.909607 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45" podUID="eed10564-909d-42d8-b5c6-e6a97da5ceff" containerName="istio-proxy" containerID="cri-o://8ad9ce25a3e2fbdce74bddd57b2cd734928aa7a12ab8143c09f2f92478baa058" gracePeriod=30 Oct 31 04:34:42.278866 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:42.278831 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45" Oct 31 04:34:42.403024 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:42.402974 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eed10564-909d-42d8-b5c6-e6a97da5ceff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eed10564-909d-42d8-b5c6-e6a97da5ceff-workload-socket\") pod \"eed10564-909d-42d8-b5c6-e6a97da5ceff\" (UID: \"eed10564-909d-42d8-b5c6-e6a97da5ceff\") " Oct 31 04:34:42.403238 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:42.403107 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eed10564-909d-42d8-b5c6-e6a97da5ceff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eed10564-909d-42d8-b5c6-e6a97da5ceff-workload-certs\") pod \"eed10564-909d-42d8-b5c6-e6a97da5ceff\" (UID: \"eed10564-909d-42d8-b5c6-e6a97da5ceff\") " Oct 31 04:34:42.403238 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:42.403177 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eed10564-909d-42d8-b5c6-e6a97da5ceff-istio-token\" (UniqueName: \"kubernetes.io/projected/eed10564-909d-42d8-b5c6-e6a97da5ceff-istio-token\") pod \"eed10564-909d-42d8-b5c6-e6a97da5ceff\" (UID: \"eed10564-909d-42d8-b5c6-e6a97da5ceff\") " Oct 31 04:34:42.403432 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:42.403255 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eed10564-909d-42d8-b5c6-e6a97da5ceff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eed10564-909d-42d8-b5c6-e6a97da5ceff-istio-envoy\") pod \"eed10564-909d-42d8-b5c6-e6a97da5ceff\" (UID: \"eed10564-909d-42d8-b5c6-e6a97da5ceff\") " Oct 31 04:34:42.403432 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:42.403302 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eed10564-909d-42d8-b5c6-e6a97da5ceff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eed10564-909d-42d8-b5c6-e6a97da5ceff-istiod-ca-cert\") pod \"eed10564-909d-42d8-b5c6-e6a97da5ceff\" (UID: \"eed10564-909d-42d8-b5c6-e6a97da5ceff\") " Oct 31 04:34:42.403432 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:42.403295 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eed10564-909d-42d8-b5c6-e6a97da5ceff-workload-socket" pod "eed10564-909d-42d8-b5c6-e6a97da5ceff" (UID: "eed10564-909d-42d8-b5c6-e6a97da5ceff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:34:42.403432 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:42.403344 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eed10564-909d-42d8-b5c6-e6a97da5ceff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eed10564-909d-42d8-b5c6-e6a97da5ceff-credential-socket\") pod \"eed10564-909d-42d8-b5c6-e6a97da5ceff\" (UID: \"eed10564-909d-42d8-b5c6-e6a97da5ceff\") " Oct 31 04:34:42.403432 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:42.403384 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eed10564-909d-42d8-b5c6-e6a97da5ceff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eed10564-909d-42d8-b5c6-e6a97da5ceff-istio-data\") pod \"eed10564-909d-42d8-b5c6-e6a97da5ceff\" (UID: \"eed10564-909d-42d8-b5c6-e6a97da5ceff\") " Oct 31 04:34:42.403740 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:42.403589 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eed10564-909d-42d8-b5c6-e6a97da5ceff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eed10564-909d-42d8-b5c6-e6a97da5ceff-istio-podinfo\") pod \"eed10564-909d-42d8-b5c6-e6a97da5ceff\" (UID: \"eed10564-909d-42d8-b5c6-e6a97da5ceff\") " Oct 31 04:34:42.403740 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:42.403656 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eed10564-909d-42d8-b5c6-e6a97da5ceff-kube-api-access-gz2nf\" (UniqueName: \"kubernetes.io/projected/eed10564-909d-42d8-b5c6-e6a97da5ceff-kube-api-access-gz2nf\") pod \"eed10564-909d-42d8-b5c6-e6a97da5ceff\" (UID: \"eed10564-909d-42d8-b5c6-e6a97da5ceff\") " Oct 31 04:34:42.403891 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:42.403845 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eed10564-909d-42d8-b5c6-e6a97da5ceff-workload-certs" pod "eed10564-909d-42d8-b5c6-e6a97da5ceff" (UID: "eed10564-909d-42d8-b5c6-e6a97da5ceff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:34:42.404088 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:42.404056 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eed10564-909d-42d8-b5c6-e6a97da5ceff-credential-socket" pod "eed10564-909d-42d8-b5c6-e6a97da5ceff" (UID: "eed10564-909d-42d8-b5c6-e6a97da5ceff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:34:42.404575 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:42.404549 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eed10564-909d-42d8-b5c6-e6a97da5ceff-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:34:42.404682 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:42.404580 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eed10564-909d-42d8-b5c6-e6a97da5ceff-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:34:42.404682 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:42.404595 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eed10564-909d-42d8-b5c6-e6a97da5ceff-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:34:42.406882 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:42.406825 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eed10564-909d-42d8-b5c6-e6a97da5ceff-istio-data" pod "eed10564-909d-42d8-b5c6-e6a97da5ceff" (UID: "eed10564-909d-42d8-b5c6-e6a97da5ceff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:34:42.407012 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:42.406974 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eed10564-909d-42d8-b5c6-e6a97da5ceff-istiod-ca-cert" pod "eed10564-909d-42d8-b5c6-e6a97da5ceff" (UID: "eed10564-909d-42d8-b5c6-e6a97da5ceff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:34:42.407263 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:42.407240 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eed10564-909d-42d8-b5c6-e6a97da5ceff-istio-token" pod "eed10564-909d-42d8-b5c6-e6a97da5ceff" (UID: "eed10564-909d-42d8-b5c6-e6a97da5ceff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:34:42.407501 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:42.407481 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eed10564-909d-42d8-b5c6-e6a97da5ceff-istio-envoy" pod "eed10564-909d-42d8-b5c6-e6a97da5ceff" (UID: "eed10564-909d-42d8-b5c6-e6a97da5ceff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:34:42.407710 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:42.407688 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eed10564-909d-42d8-b5c6-e6a97da5ceff-kube-api-access-gz2nf" pod "eed10564-909d-42d8-b5c6-e6a97da5ceff" (UID: "eed10564-909d-42d8-b5c6-e6a97da5ceff"). InnerVolumeSpecName "kube-api-access-gz2nf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:34:42.409214 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:42.409187 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eed10564-909d-42d8-b5c6-e6a97da5ceff-istio-podinfo" pod "eed10564-909d-42d8-b5c6-e6a97da5ceff" (UID: "eed10564-909d-42d8-b5c6-e6a97da5ceff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:34:42.430247 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:42.430212 5019 generic.go:356] "Generic (PLEG): container finished" podID="eed10564-909d-42d8-b5c6-e6a97da5ceff" containerID="8ad9ce25a3e2fbdce74bddd57b2cd734928aa7a12ab8143c09f2f92478baa058" exitCode=0 Oct 31 04:34:42.430439 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:42.430264 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45" Oct 31 04:34:42.430439 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:42.430294 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45" event={"ID":"eed10564-909d-42d8-b5c6-e6a97da5ceff","Type":"ContainerDied","Data":"8ad9ce25a3e2fbdce74bddd57b2cd734928aa7a12ab8143c09f2f92478baa058"} Oct 31 04:34:42.430439 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:42.430405 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45" event={"ID":"eed10564-909d-42d8-b5c6-e6a97da5ceff","Type":"ContainerDied","Data":"e75c941a7ca3ac38e628e17af52c0ff6195fb31e038c1b387307b7636ee88c60"} Oct 31 04:34:42.430439 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:42.430434 5019 scope.go:122] "RemoveContainer" containerID="8ad9ce25a3e2fbdce74bddd57b2cd734928aa7a12ab8143c09f2f92478baa058" Oct 31 04:34:42.457256 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:42.457230 5019 scope.go:122] "RemoveContainer" containerID="8ad9ce25a3e2fbdce74bddd57b2cd734928aa7a12ab8143c09f2f92478baa058" Oct 31 04:34:42.457970 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:34:42.457947 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ad9ce25a3e2fbdce74bddd57b2cd734928aa7a12ab8143c09f2f92478baa058\": container with ID starting with 8ad9ce25a3e2fbdce74bddd57b2cd734928aa7a12ab8143c09f2f92478baa058 not found: ID does not exist" containerID="8ad9ce25a3e2fbdce74bddd57b2cd734928aa7a12ab8143c09f2f92478baa058" Oct 31 04:34:42.458083 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:42.457980 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ad9ce25a3e2fbdce74bddd57b2cd734928aa7a12ab8143c09f2f92478baa058"} err="failed to get container status \"8ad9ce25a3e2fbdce74bddd57b2cd734928aa7a12ab8143c09f2f92478baa058\": rpc error: code = NotFound desc = could not find container \"8ad9ce25a3e2fbdce74bddd57b2cd734928aa7a12ab8143c09f2f92478baa058\": container with ID starting with 8ad9ce25a3e2fbdce74bddd57b2cd734928aa7a12ab8143c09f2f92478baa058 not found: ID does not exist" Oct 31 04:34:42.499397 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:42.499282 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45"] Oct 31 04:34:42.506760 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:42.506721 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q2f45"] Oct 31 04:34:42.507180 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:42.507156 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eed10564-909d-42d8-b5c6-e6a97da5ceff-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:34:42.507281 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:42.507183 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz2nf\" (UniqueName: \"kubernetes.io/projected/eed10564-909d-42d8-b5c6-e6a97da5ceff-kube-api-access-gz2nf\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:34:42.507281 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:42.507199 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eed10564-909d-42d8-b5c6-e6a97da5ceff-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:34:42.507281 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:42.507225 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eed10564-909d-42d8-b5c6-e6a97da5ceff-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:34:42.507281 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:42.507238 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eed10564-909d-42d8-b5c6-e6a97da5ceff-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:34:42.507281 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:42.507251 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eed10564-909d-42d8-b5c6-e6a97da5ceff-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:34:43.309855 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:43.309816 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-7229f2/autores-model-7229f2-kserve-84cff56d8f-f8lqw"] Oct 31 04:34:43.311058 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:43.311030 5019 cpu_manager.go:414] "containerMap: removing container" podUID="d4062db4-f8d8-4cbb-8f2d-428c06d30db7" containerName="istio-proxy" Oct 31 04:34:43.311185 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:43.311061 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4062db4-f8d8-4cbb-8f2d-428c06d30db7" containerName="istio-proxy" Oct 31 04:34:43.311185 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:43.311097 5019 cpu_manager.go:414] "containerMap: removing container" podUID="eed10564-909d-42d8-b5c6-e6a97da5ceff" containerName="istio-proxy" Oct 31 04:34:43.311185 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:43.311107 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eed10564-909d-42d8-b5c6-e6a97da5ceff" containerName="istio-proxy" Oct 31 04:34:43.311407 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:43.311296 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="eed10564-909d-42d8-b5c6-e6a97da5ceff" containerName="istio-proxy" Oct 31 04:34:43.311407 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:43.311325 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4062db4-f8d8-4cbb-8f2d-428c06d30db7" containerName="istio-proxy" Oct 31 04:34:43.317598 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:43.317569 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-7229f2/autores-model-7229f2-kserve-84cff56d8f-f8lqw" Oct 31 04:34:43.334938 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:43.334904 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-7229f2/autores-model-7229f2-kserve-84cff56d8f-f8lqw"] Oct 31 04:34:43.338315 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:43.338284 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-7229f2\"/\"openshift-service-ca.crt\"" Oct 31 04:34:43.338315 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:43.338295 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-7229f2\"/\"default-dockercfg-48v5f\"" Oct 31 04:34:43.338617 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:43.338294 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-7229f2\"/\"autores-model-7229f2-kserve-self-signed-certs\"" Oct 31 04:34:43.338811 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:43.338789 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-7229f2\"/\"kube-root-ca.crt\"" Oct 31 04:34:43.412050 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:43.412006 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eed10564-909d-42d8-b5c6-e6a97da5ceff" path="/var/lib/kubelet/pods/eed10564-909d-42d8-b5c6-e6a97da5ceff/volumes" Oct 31 04:34:43.423276 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:43.423235 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/549207bf-db66-4f89-b541-edbb6ddd4f2c-model-cache\") pod \"autores-model-7229f2-kserve-84cff56d8f-f8lqw\" (UID: \"549207bf-db66-4f89-b541-edbb6ddd4f2c\") " pod="ai-tenant-e2e-autores-7229f2/autores-model-7229f2-kserve-84cff56d8f-f8lqw" Oct 31 04:34:43.423492 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:43.423304 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/549207bf-db66-4f89-b541-edbb6ddd4f2c-tmp-dir\") pod \"autores-model-7229f2-kserve-84cff56d8f-f8lqw\" (UID: \"549207bf-db66-4f89-b541-edbb6ddd4f2c\") " pod="ai-tenant-e2e-autores-7229f2/autores-model-7229f2-kserve-84cff56d8f-f8lqw" Oct 31 04:34:43.423536 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:43.423490 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szpk6\" (UniqueName: \"kubernetes.io/projected/549207bf-db66-4f89-b541-edbb6ddd4f2c-kube-api-access-szpk6\") pod \"autores-model-7229f2-kserve-84cff56d8f-f8lqw\" (UID: \"549207bf-db66-4f89-b541-edbb6ddd4f2c\") " pod="ai-tenant-e2e-autores-7229f2/autores-model-7229f2-kserve-84cff56d8f-f8lqw" Oct 31 04:34:43.423579 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:43.423564 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/549207bf-db66-4f89-b541-edbb6ddd4f2c-home\") pod \"autores-model-7229f2-kserve-84cff56d8f-f8lqw\" (UID: \"549207bf-db66-4f89-b541-edbb6ddd4f2c\") " pod="ai-tenant-e2e-autores-7229f2/autores-model-7229f2-kserve-84cff56d8f-f8lqw" Oct 31 04:34:43.423688 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:43.423672 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/549207bf-db66-4f89-b541-edbb6ddd4f2c-dshm\") pod \"autores-model-7229f2-kserve-84cff56d8f-f8lqw\" (UID: \"549207bf-db66-4f89-b541-edbb6ddd4f2c\") " pod="ai-tenant-e2e-autores-7229f2/autores-model-7229f2-kserve-84cff56d8f-f8lqw" Oct 31 04:34:43.423735 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:43.423719 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/549207bf-db66-4f89-b541-edbb6ddd4f2c-tls-certs\") pod \"autores-model-7229f2-kserve-84cff56d8f-f8lqw\" (UID: \"549207bf-db66-4f89-b541-edbb6ddd4f2c\") " pod="ai-tenant-e2e-autores-7229f2/autores-model-7229f2-kserve-84cff56d8f-f8lqw" Oct 31 04:34:43.525713 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:43.525671 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/549207bf-db66-4f89-b541-edbb6ddd4f2c-model-cache\") pod \"autores-model-7229f2-kserve-84cff56d8f-f8lqw\" (UID: \"549207bf-db66-4f89-b541-edbb6ddd4f2c\") " pod="ai-tenant-e2e-autores-7229f2/autores-model-7229f2-kserve-84cff56d8f-f8lqw" Oct 31 04:34:43.525713 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:43.525726 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/549207bf-db66-4f89-b541-edbb6ddd4f2c-tmp-dir\") pod \"autores-model-7229f2-kserve-84cff56d8f-f8lqw\" (UID: \"549207bf-db66-4f89-b541-edbb6ddd4f2c\") " pod="ai-tenant-e2e-autores-7229f2/autores-model-7229f2-kserve-84cff56d8f-f8lqw" Oct 31 04:34:43.526014 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:43.525758 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szpk6\" (UniqueName: \"kubernetes.io/projected/549207bf-db66-4f89-b541-edbb6ddd4f2c-kube-api-access-szpk6\") pod \"autores-model-7229f2-kserve-84cff56d8f-f8lqw\" (UID: \"549207bf-db66-4f89-b541-edbb6ddd4f2c\") " pod="ai-tenant-e2e-autores-7229f2/autores-model-7229f2-kserve-84cff56d8f-f8lqw" Oct 31 04:34:43.526014 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:43.525822 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/549207bf-db66-4f89-b541-edbb6ddd4f2c-home\") pod \"autores-model-7229f2-kserve-84cff56d8f-f8lqw\" (UID: \"549207bf-db66-4f89-b541-edbb6ddd4f2c\") " pod="ai-tenant-e2e-autores-7229f2/autores-model-7229f2-kserve-84cff56d8f-f8lqw" Oct 31 04:34:43.526014 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:43.525910 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/549207bf-db66-4f89-b541-edbb6ddd4f2c-dshm\") pod \"autores-model-7229f2-kserve-84cff56d8f-f8lqw\" (UID: \"549207bf-db66-4f89-b541-edbb6ddd4f2c\") " pod="ai-tenant-e2e-autores-7229f2/autores-model-7229f2-kserve-84cff56d8f-f8lqw" Oct 31 04:34:43.526014 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:43.525962 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/549207bf-db66-4f89-b541-edbb6ddd4f2c-tls-certs\") pod \"autores-model-7229f2-kserve-84cff56d8f-f8lqw\" (UID: \"549207bf-db66-4f89-b541-edbb6ddd4f2c\") " pod="ai-tenant-e2e-autores-7229f2/autores-model-7229f2-kserve-84cff56d8f-f8lqw" Oct 31 04:34:43.526278 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:43.526172 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/549207bf-db66-4f89-b541-edbb6ddd4f2c-model-cache\") pod \"autores-model-7229f2-kserve-84cff56d8f-f8lqw\" (UID: \"549207bf-db66-4f89-b541-edbb6ddd4f2c\") " pod="ai-tenant-e2e-autores-7229f2/autores-model-7229f2-kserve-84cff56d8f-f8lqw" Oct 31 04:34:43.526359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:43.526305 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/549207bf-db66-4f89-b541-edbb6ddd4f2c-home\") pod \"autores-model-7229f2-kserve-84cff56d8f-f8lqw\" (UID: \"549207bf-db66-4f89-b541-edbb6ddd4f2c\") " pod="ai-tenant-e2e-autores-7229f2/autores-model-7229f2-kserve-84cff56d8f-f8lqw" Oct 31 04:34:43.526546 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:43.526525 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/549207bf-db66-4f89-b541-edbb6ddd4f2c-tmp-dir\") pod \"autores-model-7229f2-kserve-84cff56d8f-f8lqw\" (UID: \"549207bf-db66-4f89-b541-edbb6ddd4f2c\") " pod="ai-tenant-e2e-autores-7229f2/autores-model-7229f2-kserve-84cff56d8f-f8lqw" Oct 31 04:34:43.528633 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:43.528609 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/549207bf-db66-4f89-b541-edbb6ddd4f2c-dshm\") pod \"autores-model-7229f2-kserve-84cff56d8f-f8lqw\" (UID: \"549207bf-db66-4f89-b541-edbb6ddd4f2c\") " pod="ai-tenant-e2e-autores-7229f2/autores-model-7229f2-kserve-84cff56d8f-f8lqw" Oct 31 04:34:43.528990 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:43.528964 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/549207bf-db66-4f89-b541-edbb6ddd4f2c-tls-certs\") pod \"autores-model-7229f2-kserve-84cff56d8f-f8lqw\" (UID: \"549207bf-db66-4f89-b541-edbb6ddd4f2c\") " pod="ai-tenant-e2e-autores-7229f2/autores-model-7229f2-kserve-84cff56d8f-f8lqw" Oct 31 04:34:43.568940 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:43.568860 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szpk6\" (UniqueName: \"kubernetes.io/projected/549207bf-db66-4f89-b541-edbb6ddd4f2c-kube-api-access-szpk6\") pod \"autores-model-7229f2-kserve-84cff56d8f-f8lqw\" (UID: \"549207bf-db66-4f89-b541-edbb6ddd4f2c\") " pod="ai-tenant-e2e-autores-7229f2/autores-model-7229f2-kserve-84cff56d8f-f8lqw" Oct 31 04:34:43.639424 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:43.638758 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-7229f2/autores-model-7229f2-kserve-84cff56d8f-f8lqw" Oct 31 04:34:44.007442 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:44.007412 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-7229f2/autores-model-7229f2-kserve-84cff56d8f-f8lqw"] Oct 31 04:34:44.454017 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:44.453918 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-7229f2/autores-model-7229f2-kserve-84cff56d8f-f8lqw" event={"ID":"549207bf-db66-4f89-b541-edbb6ddd4f2c","Type":"ContainerStarted","Data":"085a061f041c7fbd48c3c55504145b57ab3ddca3ebfd057767af85cd417b74ba"} Oct 31 04:34:44.454017 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:44.453968 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-7229f2/autores-model-7229f2-kserve-84cff56d8f-f8lqw" event={"ID":"549207bf-db66-4f89-b541-edbb6ddd4f2c","Type":"ContainerStarted","Data":"5a107874dd2dfb8e55bb1e6b5974b557dd757e5c51ae207b8560a3205f4e07f3"} Oct 31 04:34:44.496120 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:44.496062 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-7229f2/autores-model-7229f2-kserve-84cff56d8f-f8lqw" podStartSLOduration=1.496049792 podStartE2EDuration="1.496049792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:34:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:34:44.495424502 +0000 UTC m=+1839.251762545" watchObservedRunningTime="2026-10-31 04:34:44.496049792 +0000 UTC m=+1839.252387832" Oct 31 04:34:44.642358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:44.639600 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-7229f2/autores-model-7229f2-kserve-84cff56d8f-f8lqw" Oct 31 04:34:48.031243 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:48.031150 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a2e7e0a-zf829"] Oct 31 04:34:48.043238 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:48.043199 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a2e7e0a-zf829"] Oct 31 04:34:48.676565 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:48.676525 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-031745/auth-test-model-031745-kserve-7656c77f57-l86zb"] Oct 31 04:34:48.683515 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:48.683467 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-031745/auth-test-model-031745-kserve-7656c77f57-l86zb" Oct 31 04:34:48.705143 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:48.704558 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-031745/auth-test-model-031745-kserve-7656c77f57-l86zb"] Oct 31 04:34:48.705143 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:48.704815 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-031745\"/\"default-dockercfg-jpj9n\"" Oct 31 04:34:48.705143 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:48.704815 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-031745\"/\"openshift-service-ca.crt\"" Oct 31 04:34:48.710224 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:48.710154 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-031745\"/\"auth-test-model-031745-kserve-self-signed-certs\"" Oct 31 04:34:48.710224 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:48.710155 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-031745\"/\"kube-root-ca.crt\"" Oct 31 04:34:48.825203 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:48.825160 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4tmm\" (UniqueName: \"kubernetes.io/projected/3a265311-7ffc-488e-a829-8f43ee2e9a77-kube-api-access-p4tmm\") pod \"auth-test-model-031745-kserve-7656c77f57-l86zb\" (UID: \"3a265311-7ffc-488e-a829-8f43ee2e9a77\") " pod="ai-tenant-e2e-shared-b-w2-031745/auth-test-model-031745-kserve-7656c77f57-l86zb" Oct 31 04:34:48.825467 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:48.825316 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a265311-7ffc-488e-a829-8f43ee2e9a77-home\") pod \"auth-test-model-031745-kserve-7656c77f57-l86zb\" (UID: \"3a265311-7ffc-488e-a829-8f43ee2e9a77\") " pod="ai-tenant-e2e-shared-b-w2-031745/auth-test-model-031745-kserve-7656c77f57-l86zb" Oct 31 04:34:48.825467 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:48.825399 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a265311-7ffc-488e-a829-8f43ee2e9a77-tls-certs\") pod \"auth-test-model-031745-kserve-7656c77f57-l86zb\" (UID: \"3a265311-7ffc-488e-a829-8f43ee2e9a77\") " pod="ai-tenant-e2e-shared-b-w2-031745/auth-test-model-031745-kserve-7656c77f57-l86zb" Oct 31 04:34:48.825562 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:48.825467 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a265311-7ffc-488e-a829-8f43ee2e9a77-dshm\") pod \"auth-test-model-031745-kserve-7656c77f57-l86zb\" (UID: \"3a265311-7ffc-488e-a829-8f43ee2e9a77\") " pod="ai-tenant-e2e-shared-b-w2-031745/auth-test-model-031745-kserve-7656c77f57-l86zb" Oct 31 04:34:48.825562 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:48.825497 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a265311-7ffc-488e-a829-8f43ee2e9a77-tmp-dir\") pod \"auth-test-model-031745-kserve-7656c77f57-l86zb\" (UID: \"3a265311-7ffc-488e-a829-8f43ee2e9a77\") " pod="ai-tenant-e2e-shared-b-w2-031745/auth-test-model-031745-kserve-7656c77f57-l86zb" Oct 31 04:34:48.825658 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:48.825644 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a265311-7ffc-488e-a829-8f43ee2e9a77-model-cache\") pod \"auth-test-model-031745-kserve-7656c77f57-l86zb\" (UID: \"3a265311-7ffc-488e-a829-8f43ee2e9a77\") " pod="ai-tenant-e2e-shared-b-w2-031745/auth-test-model-031745-kserve-7656c77f57-l86zb" Oct 31 04:34:48.957418 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:48.947856 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a265311-7ffc-488e-a829-8f43ee2e9a77-home\") pod \"auth-test-model-031745-kserve-7656c77f57-l86zb\" (UID: \"3a265311-7ffc-488e-a829-8f43ee2e9a77\") " pod="ai-tenant-e2e-shared-b-w2-031745/auth-test-model-031745-kserve-7656c77f57-l86zb" Oct 31 04:34:48.957418 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:48.947918 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a265311-7ffc-488e-a829-8f43ee2e9a77-tls-certs\") pod \"auth-test-model-031745-kserve-7656c77f57-l86zb\" (UID: \"3a265311-7ffc-488e-a829-8f43ee2e9a77\") " pod="ai-tenant-e2e-shared-b-w2-031745/auth-test-model-031745-kserve-7656c77f57-l86zb" Oct 31 04:34:48.957418 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:48.947983 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a265311-7ffc-488e-a829-8f43ee2e9a77-dshm\") pod \"auth-test-model-031745-kserve-7656c77f57-l86zb\" (UID: \"3a265311-7ffc-488e-a829-8f43ee2e9a77\") " pod="ai-tenant-e2e-shared-b-w2-031745/auth-test-model-031745-kserve-7656c77f57-l86zb" Oct 31 04:34:48.957418 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:48.948026 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a265311-7ffc-488e-a829-8f43ee2e9a77-tmp-dir\") pod \"auth-test-model-031745-kserve-7656c77f57-l86zb\" (UID: \"3a265311-7ffc-488e-a829-8f43ee2e9a77\") " pod="ai-tenant-e2e-shared-b-w2-031745/auth-test-model-031745-kserve-7656c77f57-l86zb" Oct 31 04:34:48.957418 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:48.948201 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a265311-7ffc-488e-a829-8f43ee2e9a77-model-cache\") pod \"auth-test-model-031745-kserve-7656c77f57-l86zb\" (UID: \"3a265311-7ffc-488e-a829-8f43ee2e9a77\") " pod="ai-tenant-e2e-shared-b-w2-031745/auth-test-model-031745-kserve-7656c77f57-l86zb" Oct 31 04:34:48.957418 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:48.948294 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4tmm\" (UniqueName: \"kubernetes.io/projected/3a265311-7ffc-488e-a829-8f43ee2e9a77-kube-api-access-p4tmm\") pod \"auth-test-model-031745-kserve-7656c77f57-l86zb\" (UID: \"3a265311-7ffc-488e-a829-8f43ee2e9a77\") " pod="ai-tenant-e2e-shared-b-w2-031745/auth-test-model-031745-kserve-7656c77f57-l86zb" Oct 31 04:34:48.957418 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:48.949044 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a265311-7ffc-488e-a829-8f43ee2e9a77-tmp-dir\") pod \"auth-test-model-031745-kserve-7656c77f57-l86zb\" (UID: \"3a265311-7ffc-488e-a829-8f43ee2e9a77\") " pod="ai-tenant-e2e-shared-b-w2-031745/auth-test-model-031745-kserve-7656c77f57-l86zb" Oct 31 04:34:48.957418 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:48.949326 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a265311-7ffc-488e-a829-8f43ee2e9a77-model-cache\") pod \"auth-test-model-031745-kserve-7656c77f57-l86zb\" (UID: \"3a265311-7ffc-488e-a829-8f43ee2e9a77\") " pod="ai-tenant-e2e-shared-b-w2-031745/auth-test-model-031745-kserve-7656c77f57-l86zb" Oct 31 04:34:48.957418 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:48.949676 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a265311-7ffc-488e-a829-8f43ee2e9a77-home\") pod \"auth-test-model-031745-kserve-7656c77f57-l86zb\" (UID: \"3a265311-7ffc-488e-a829-8f43ee2e9a77\") " pod="ai-tenant-e2e-shared-b-w2-031745/auth-test-model-031745-kserve-7656c77f57-l86zb" Oct 31 04:34:48.991489 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:48.991454 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4tmm\" (UniqueName: \"kubernetes.io/projected/3a265311-7ffc-488e-a829-8f43ee2e9a77-kube-api-access-p4tmm\") pod \"auth-test-model-031745-kserve-7656c77f57-l86zb\" (UID: \"3a265311-7ffc-488e-a829-8f43ee2e9a77\") " pod="ai-tenant-e2e-shared-b-w2-031745/auth-test-model-031745-kserve-7656c77f57-l86zb" Oct 31 04:34:49.017365 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:49.009447 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a265311-7ffc-488e-a829-8f43ee2e9a77-dshm\") pod \"auth-test-model-031745-kserve-7656c77f57-l86zb\" (UID: \"3a265311-7ffc-488e-a829-8f43ee2e9a77\") " pod="ai-tenant-e2e-shared-b-w2-031745/auth-test-model-031745-kserve-7656c77f57-l86zb" Oct 31 04:34:49.031425 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:49.020882 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a265311-7ffc-488e-a829-8f43ee2e9a77-tls-certs\") pod \"auth-test-model-031745-kserve-7656c77f57-l86zb\" (UID: \"3a265311-7ffc-488e-a829-8f43ee2e9a77\") " pod="ai-tenant-e2e-shared-b-w2-031745/auth-test-model-031745-kserve-7656c77f57-l86zb" Oct 31 04:34:49.311170 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:49.311127 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-031745/auth-test-model-031745-kserve-7656c77f57-l86zb" Oct 31 04:34:49.428665 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:49.428627 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34ec763e-a473-4d95-a094-6868c8e8f95d" path="/var/lib/kubelet/pods/34ec763e-a473-4d95-a094-6868c8e8f95d/volumes" Oct 31 04:34:49.632139 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:34:49.632104 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a265311_7ffc_488e_a829_8f43ee2e9a77.slice/crio-494ca30dd794cfff4419515f1e4be6a158dacc9a19e84b6504af9942381619bc WatchSource:0}: Error finding container 494ca30dd794cfff4419515f1e4be6a158dacc9a19e84b6504af9942381619bc: Status 404 returned error can't find the container with id 494ca30dd794cfff4419515f1e4be6a158dacc9a19e84b6504af9942381619bc Oct 31 04:34:49.632998 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:49.632969 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-031745/auth-test-model-031745-kserve-7656c77f57-l86zb"] Oct 31 04:34:50.516632 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:50.516590 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-031745/auth-test-model-031745-kserve-7656c77f57-l86zb" event={"ID":"3a265311-7ffc-488e-a829-8f43ee2e9a77","Type":"ContainerStarted","Data":"29ed1fe2333e5e8733f869f3152d7b50e90e1309fec309ac55ddabdd229a316f"} Oct 31 04:34:50.517159 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:50.516641 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-031745/auth-test-model-031745-kserve-7656c77f57-l86zb" event={"ID":"3a265311-7ffc-488e-a829-8f43ee2e9a77","Type":"ContainerStarted","Data":"494ca30dd794cfff4419515f1e4be6a158dacc9a19e84b6504af9942381619bc"} Oct 31 04:34:50.563570 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:50.563432 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-031745/auth-test-model-031745-kserve-7656c77f57-l86zb" podStartSLOduration=2.5634090929999997 podStartE2EDuration="2.563409093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:34:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:34:50.557904397 +0000 UTC m=+1845.314242437" watchObservedRunningTime="2026-10-31 04:34:50.563409093 +0000 UTC m=+1845.319747132" Oct 31 04:34:51.311578 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:51.311529 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-031745/auth-test-model-031745-kserve-7656c77f57-l86zb" Oct 31 04:34:52.394354 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:34:52.393845 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:34:53.640559 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:53.640512 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-7229f2/autores-model-7229f2-kserve-84cff56d8f-f8lqw" Oct 31 04:34:53.655306 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:53.655270 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-7229f2/autores-model-7229f2-kserve-84cff56d8f-f8lqw" Oct 31 04:34:54.567870 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:54.567836 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-7229f2/autores-model-7229f2-kserve-84cff56d8f-f8lqw" Oct 31 04:34:55.335835 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:55.335791 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-b2a372/auth-test-model-b2a372-kserve-85c4657f6f-7gst5"] Oct 31 04:34:55.341774 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:55.341744 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b2a372/auth-test-model-b2a372-kserve-85c4657f6f-7gst5" Oct 31 04:34:55.366782 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:55.366752 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-b2a372\"/\"kube-root-ca.crt\"" Oct 31 04:34:55.367066 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:55.367048 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-b2a372\"/\"openshift-service-ca.crt\"" Oct 31 04:34:55.367503 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:55.367477 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b2a372/auth-test-model-b2a372-kserve-85c4657f6f-7gst5"] Oct 31 04:34:55.369428 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:55.369404 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-b2a372\"/\"auth-test-model-b2a372-kserve-self-signed-certs\"" Oct 31 04:34:55.369549 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:55.369495 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-b2a372\"/\"default-dockercfg-dqlrt\"" Oct 31 04:34:55.458055 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:55.458017 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwccn\" (UniqueName: \"kubernetes.io/projected/38968c8c-ec31-40df-9e63-6324e59ae601-kube-api-access-wwccn\") pod \"auth-test-model-b2a372-kserve-85c4657f6f-7gst5\" (UID: \"38968c8c-ec31-40df-9e63-6324e59ae601\") " pod="ai-tenant-e2e-shared-b-w3-b2a372/auth-test-model-b2a372-kserve-85c4657f6f-7gst5" Oct 31 04:34:55.458224 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:55.458062 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38968c8c-ec31-40df-9e63-6324e59ae601-home\") pod \"auth-test-model-b2a372-kserve-85c4657f6f-7gst5\" (UID: \"38968c8c-ec31-40df-9e63-6324e59ae601\") " pod="ai-tenant-e2e-shared-b-w3-b2a372/auth-test-model-b2a372-kserve-85c4657f6f-7gst5" Oct 31 04:34:55.458224 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:55.458219 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38968c8c-ec31-40df-9e63-6324e59ae601-model-cache\") pod \"auth-test-model-b2a372-kserve-85c4657f6f-7gst5\" (UID: \"38968c8c-ec31-40df-9e63-6324e59ae601\") " pod="ai-tenant-e2e-shared-b-w3-b2a372/auth-test-model-b2a372-kserve-85c4657f6f-7gst5" Oct 31 04:34:55.458368 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:55.458266 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38968c8c-ec31-40df-9e63-6324e59ae601-dshm\") pod \"auth-test-model-b2a372-kserve-85c4657f6f-7gst5\" (UID: \"38968c8c-ec31-40df-9e63-6324e59ae601\") " pod="ai-tenant-e2e-shared-b-w3-b2a372/auth-test-model-b2a372-kserve-85c4657f6f-7gst5" Oct 31 04:34:55.458418 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:55.458364 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38968c8c-ec31-40df-9e63-6324e59ae601-tmp-dir\") pod \"auth-test-model-b2a372-kserve-85c4657f6f-7gst5\" (UID: \"38968c8c-ec31-40df-9e63-6324e59ae601\") " pod="ai-tenant-e2e-shared-b-w3-b2a372/auth-test-model-b2a372-kserve-85c4657f6f-7gst5" Oct 31 04:34:55.458465 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:55.458438 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38968c8c-ec31-40df-9e63-6324e59ae601-tls-certs\") pod \"auth-test-model-b2a372-kserve-85c4657f6f-7gst5\" (UID: \"38968c8c-ec31-40df-9e63-6324e59ae601\") " pod="ai-tenant-e2e-shared-b-w3-b2a372/auth-test-model-b2a372-kserve-85c4657f6f-7gst5" Oct 31 04:34:55.563634 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:55.563593 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38968c8c-ec31-40df-9e63-6324e59ae601-tls-certs\") pod \"auth-test-model-b2a372-kserve-85c4657f6f-7gst5\" (UID: \"38968c8c-ec31-40df-9e63-6324e59ae601\") " pod="ai-tenant-e2e-shared-b-w3-b2a372/auth-test-model-b2a372-kserve-85c4657f6f-7gst5" Oct 31 04:34:55.615411 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:55.615227 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38968c8c-ec31-40df-9e63-6324e59ae601-tls-certs\") pod \"auth-test-model-b2a372-kserve-85c4657f6f-7gst5\" (UID: \"38968c8c-ec31-40df-9e63-6324e59ae601\") " pod="ai-tenant-e2e-shared-b-w3-b2a372/auth-test-model-b2a372-kserve-85c4657f6f-7gst5" Oct 31 04:34:55.636355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:55.626999 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwccn\" (UniqueName: \"kubernetes.io/projected/38968c8c-ec31-40df-9e63-6324e59ae601-kube-api-access-wwccn\") pod \"auth-test-model-b2a372-kserve-85c4657f6f-7gst5\" (UID: \"38968c8c-ec31-40df-9e63-6324e59ae601\") " pod="ai-tenant-e2e-shared-b-w3-b2a372/auth-test-model-b2a372-kserve-85c4657f6f-7gst5" Oct 31 04:34:55.636355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:55.627088 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38968c8c-ec31-40df-9e63-6324e59ae601-home\") pod \"auth-test-model-b2a372-kserve-85c4657f6f-7gst5\" (UID: \"38968c8c-ec31-40df-9e63-6324e59ae601\") " pod="ai-tenant-e2e-shared-b-w3-b2a372/auth-test-model-b2a372-kserve-85c4657f6f-7gst5" Oct 31 04:34:55.636355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:55.627272 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38968c8c-ec31-40df-9e63-6324e59ae601-model-cache\") pod \"auth-test-model-b2a372-kserve-85c4657f6f-7gst5\" (UID: \"38968c8c-ec31-40df-9e63-6324e59ae601\") " pod="ai-tenant-e2e-shared-b-w3-b2a372/auth-test-model-b2a372-kserve-85c4657f6f-7gst5" Oct 31 04:34:55.636355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:55.627379 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38968c8c-ec31-40df-9e63-6324e59ae601-dshm\") pod \"auth-test-model-b2a372-kserve-85c4657f6f-7gst5\" (UID: \"38968c8c-ec31-40df-9e63-6324e59ae601\") " pod="ai-tenant-e2e-shared-b-w3-b2a372/auth-test-model-b2a372-kserve-85c4657f6f-7gst5" Oct 31 04:34:55.636355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:55.627447 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38968c8c-ec31-40df-9e63-6324e59ae601-tmp-dir\") pod \"auth-test-model-b2a372-kserve-85c4657f6f-7gst5\" (UID: \"38968c8c-ec31-40df-9e63-6324e59ae601\") " pod="ai-tenant-e2e-shared-b-w3-b2a372/auth-test-model-b2a372-kserve-85c4657f6f-7gst5" Oct 31 04:34:55.636355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:55.628028 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38968c8c-ec31-40df-9e63-6324e59ae601-tmp-dir\") pod \"auth-test-model-b2a372-kserve-85c4657f6f-7gst5\" (UID: \"38968c8c-ec31-40df-9e63-6324e59ae601\") " pod="ai-tenant-e2e-shared-b-w3-b2a372/auth-test-model-b2a372-kserve-85c4657f6f-7gst5" Oct 31 04:34:55.636355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:55.628296 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38968c8c-ec31-40df-9e63-6324e59ae601-home\") pod \"auth-test-model-b2a372-kserve-85c4657f6f-7gst5\" (UID: \"38968c8c-ec31-40df-9e63-6324e59ae601\") " pod="ai-tenant-e2e-shared-b-w3-b2a372/auth-test-model-b2a372-kserve-85c4657f6f-7gst5" Oct 31 04:34:55.647213 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:55.642820 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38968c8c-ec31-40df-9e63-6324e59ae601-model-cache\") pod \"auth-test-model-b2a372-kserve-85c4657f6f-7gst5\" (UID: \"38968c8c-ec31-40df-9e63-6324e59ae601\") " pod="ai-tenant-e2e-shared-b-w3-b2a372/auth-test-model-b2a372-kserve-85c4657f6f-7gst5" Oct 31 04:34:55.669366 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:55.663231 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwccn\" (UniqueName: \"kubernetes.io/projected/38968c8c-ec31-40df-9e63-6324e59ae601-kube-api-access-wwccn\") pod \"auth-test-model-b2a372-kserve-85c4657f6f-7gst5\" (UID: \"38968c8c-ec31-40df-9e63-6324e59ae601\") " pod="ai-tenant-e2e-shared-b-w3-b2a372/auth-test-model-b2a372-kserve-85c4657f6f-7gst5" Oct 31 04:34:55.699348 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:55.699254 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38968c8c-ec31-40df-9e63-6324e59ae601-dshm\") pod \"auth-test-model-b2a372-kserve-85c4657f6f-7gst5\" (UID: \"38968c8c-ec31-40df-9e63-6324e59ae601\") " pod="ai-tenant-e2e-shared-b-w3-b2a372/auth-test-model-b2a372-kserve-85c4657f6f-7gst5" Oct 31 04:34:55.970802 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:55.970753 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b2a372/auth-test-model-b2a372-kserve-85c4657f6f-7gst5" Oct 31 04:34:56.297417 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:56.297373 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b2a372/auth-test-model-b2a372-kserve-85c4657f6f-7gst5"] Oct 31 04:34:56.297602 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:34:56.297549 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38968c8c_ec31_40df_9e63_6324e59ae601.slice/crio-26c81db2c06c05f1dfacb34c84d32dc252c7037522a1117d9fd186c78c9468a3 WatchSource:0}: Error finding container 26c81db2c06c05f1dfacb34c84d32dc252c7037522a1117d9fd186c78c9468a3: Status 404 returned error can't find the container with id 26c81db2c06c05f1dfacb34c84d32dc252c7037522a1117d9fd186c78c9468a3 Oct 31 04:34:56.639246 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:56.639126 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b2a372/auth-test-model-b2a372-kserve-85c4657f6f-7gst5" event={"ID":"38968c8c-ec31-40df-9e63-6324e59ae601","Type":"ContainerStarted","Data":"9bf2324ab82fd6d829cc6054ce9e619df2ebf1999f2c496495ca3178c7d112db"} Oct 31 04:34:56.639246 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:56.639184 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b2a372/auth-test-model-b2a372-kserve-85c4657f6f-7gst5" event={"ID":"38968c8c-ec31-40df-9e63-6324e59ae601","Type":"ContainerStarted","Data":"26c81db2c06c05f1dfacb34c84d32dc252c7037522a1117d9fd186c78c9468a3"} Oct 31 04:34:56.683575 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:56.683511 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-b2a372/auth-test-model-b2a372-kserve-85c4657f6f-7gst5" podStartSLOduration=1.683493392 podStartE2EDuration="1.683493392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:34:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:34:56.680090942 +0000 UTC m=+1851.436428986" watchObservedRunningTime="2026-10-31 04:34:56.683493392 +0000 UTC m=+1851.439831432" Oct 31 04:34:56.971040 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:56.970984 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-b2a372/auth-test-model-b2a372-kserve-85c4657f6f-7gst5" Oct 31 04:34:59.312277 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:59.312236 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-031745/auth-test-model-031745-kserve-7656c77f57-l86zb" Oct 31 04:34:59.326557 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:59.326520 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-031745/auth-test-model-031745-kserve-7656c77f57-l86zb" Oct 31 04:34:59.687401 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:34:59.687240 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-031745/auth-test-model-031745-kserve-7656c77f57-l86zb" Oct 31 04:35:05.985673 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:05.985634 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-b2a372/auth-test-model-b2a372-kserve-85c4657f6f-7gst5" Oct 31 04:35:06.014002 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:06.013964 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-b2a372/auth-test-model-b2a372-kserve-85c4657f6f-7gst5" Oct 31 04:35:06.921925 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:06.921886 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-b2a372/auth-test-model-b2a372-kserve-85c4657f6f-7gst5" Oct 31 04:35:14.029029 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:14.028990 5019 generic.go:356] "Generic (PLEG): container finished" podID="3f1a76d9-21b3-4831-a4b8-fd07b96cec87" containerID="90b1550534a1e97e636d4c271604b6e7b7b77a2751d5228485f696b2255c5b17" exitCode=0 Oct 31 04:35:14.029547 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:14.029065 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq" event={"ID":"3f1a76d9-21b3-4831-a4b8-fd07b96cec87","Type":"ContainerDied","Data":"90b1550534a1e97e636d4c271604b6e7b7b77a2751d5228485f696b2255c5b17"} Oct 31 04:35:14.029608 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:14.029593 5019 scope.go:122] "RemoveContainer" containerID="90b1550534a1e97e636d4c271604b6e7b7b77a2751d5228485f696b2255c5b17" Oct 31 04:35:14.031524 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:14.031484 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:35:14.031680 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:14.031595 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:35:14.031680 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:14.031651 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:35:15.041679 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:15.041638 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq" event={"ID":"3f1a76d9-21b3-4831-a4b8-fd07b96cec87","Type":"ContainerStarted","Data":"3751a15cfb38f44c5aa11802fcac35be000fc885279b6cb58dcfe76c81b49a5f"} Oct 31 04:35:15.782767 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:15.782729 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq" Oct 31 04:35:15.791522 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:15.791484 5019 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 31 04:35:15.792185 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:15.792143 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq" podUID="3f1a76d9-21b3-4831-a4b8-fd07b96cec87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 31 04:35:16.780357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:16.778804 5019 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 31 04:35:16.780357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:16.778932 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq" podUID="3f1a76d9-21b3-4831-a4b8-fd07b96cec87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 31 04:35:16.941724 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:16.941650 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq" Oct 31 04:35:17.781360 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:17.778871 5019 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 31 04:35:17.781360 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:17.778961 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq" podUID="3f1a76d9-21b3-4831-a4b8-fd07b96cec87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 31 04:35:18.776899 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:18.776849 5019 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 31 04:35:18.777252 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:18.777214 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq" podUID="3f1a76d9-21b3-4831-a4b8-fd07b96cec87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 31 04:35:19.777437 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:19.776855 5019 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 31 04:35:19.777437 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:19.776932 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq" podUID="3f1a76d9-21b3-4831-a4b8-fd07b96cec87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 31 04:35:20.781223 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:20.781188 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq" Oct 31 04:35:20.782454 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:20.782430 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq" Oct 31 04:35:22.397230 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:35:22.397186 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:35:23.096531 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:23.096491 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-77a153/auth-test-model-77a153-kserve-6fcd554cc-c5d8v"] Oct 31 04:35:23.102080 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:23.102050 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-77a153/auth-test-model-77a153-kserve-6fcd554cc-c5d8v" Oct 31 04:35:23.126784 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:23.126742 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-77a153\"/\"auth-test-model-77a153-kserve-self-signed-certs\"" Oct 31 04:35:23.127087 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:23.126696 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-77a153\"/\"default-dockercfg-5c749\"" Oct 31 04:35:23.129158 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:23.129099 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-77a153/auth-test-model-77a153-kserve-6fcd554cc-c5d8v"] Oct 31 04:35:23.130686 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:23.130654 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-77a153\"/\"openshift-service-ca.crt\"" Oct 31 04:35:23.130830 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:23.130697 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-77a153\"/\"kube-root-ca.crt\"" Oct 31 04:35:23.189754 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:23.189710 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/655f7ad5-8f26-4d0c-88ac-3507229392c3-model-cache\") pod \"auth-test-model-77a153-kserve-6fcd554cc-c5d8v\" (UID: \"655f7ad5-8f26-4d0c-88ac-3507229392c3\") " pod="ai-tenant-e2e-shared-a-w1-77a153/auth-test-model-77a153-kserve-6fcd554cc-c5d8v" Oct 31 04:35:23.189962 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:23.189926 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkszn\" (UniqueName: \"kubernetes.io/projected/655f7ad5-8f26-4d0c-88ac-3507229392c3-kube-api-access-fkszn\") pod \"auth-test-model-77a153-kserve-6fcd554cc-c5d8v\" (UID: \"655f7ad5-8f26-4d0c-88ac-3507229392c3\") " pod="ai-tenant-e2e-shared-a-w1-77a153/auth-test-model-77a153-kserve-6fcd554cc-c5d8v" Oct 31 04:35:23.190006 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:23.189988 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/655f7ad5-8f26-4d0c-88ac-3507229392c3-tmp-dir\") pod \"auth-test-model-77a153-kserve-6fcd554cc-c5d8v\" (UID: \"655f7ad5-8f26-4d0c-88ac-3507229392c3\") " pod="ai-tenant-e2e-shared-a-w1-77a153/auth-test-model-77a153-kserve-6fcd554cc-c5d8v" Oct 31 04:35:23.190139 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:23.190111 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/655f7ad5-8f26-4d0c-88ac-3507229392c3-tls-certs\") pod \"auth-test-model-77a153-kserve-6fcd554cc-c5d8v\" (UID: \"655f7ad5-8f26-4d0c-88ac-3507229392c3\") " pod="ai-tenant-e2e-shared-a-w1-77a153/auth-test-model-77a153-kserve-6fcd554cc-c5d8v" Oct 31 04:35:23.190187 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:23.190160 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/655f7ad5-8f26-4d0c-88ac-3507229392c3-dshm\") pod \"auth-test-model-77a153-kserve-6fcd554cc-c5d8v\" (UID: \"655f7ad5-8f26-4d0c-88ac-3507229392c3\") " pod="ai-tenant-e2e-shared-a-w1-77a153/auth-test-model-77a153-kserve-6fcd554cc-c5d8v" Oct 31 04:35:23.190295 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:23.190274 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/655f7ad5-8f26-4d0c-88ac-3507229392c3-home\") pod \"auth-test-model-77a153-kserve-6fcd554cc-c5d8v\" (UID: \"655f7ad5-8f26-4d0c-88ac-3507229392c3\") " pod="ai-tenant-e2e-shared-a-w1-77a153/auth-test-model-77a153-kserve-6fcd554cc-c5d8v" Oct 31 04:35:23.291470 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:23.291400 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/655f7ad5-8f26-4d0c-88ac-3507229392c3-model-cache\") pod \"auth-test-model-77a153-kserve-6fcd554cc-c5d8v\" (UID: \"655f7ad5-8f26-4d0c-88ac-3507229392c3\") " pod="ai-tenant-e2e-shared-a-w1-77a153/auth-test-model-77a153-kserve-6fcd554cc-c5d8v" Oct 31 04:35:23.291680 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:23.291623 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkszn\" (UniqueName: \"kubernetes.io/projected/655f7ad5-8f26-4d0c-88ac-3507229392c3-kube-api-access-fkszn\") pod \"auth-test-model-77a153-kserve-6fcd554cc-c5d8v\" (UID: \"655f7ad5-8f26-4d0c-88ac-3507229392c3\") " pod="ai-tenant-e2e-shared-a-w1-77a153/auth-test-model-77a153-kserve-6fcd554cc-c5d8v" Oct 31 04:35:23.291746 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:23.291703 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/655f7ad5-8f26-4d0c-88ac-3507229392c3-tmp-dir\") pod \"auth-test-model-77a153-kserve-6fcd554cc-c5d8v\" (UID: \"655f7ad5-8f26-4d0c-88ac-3507229392c3\") " pod="ai-tenant-e2e-shared-a-w1-77a153/auth-test-model-77a153-kserve-6fcd554cc-c5d8v" Oct 31 04:35:23.291810 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:23.291758 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/655f7ad5-8f26-4d0c-88ac-3507229392c3-tls-certs\") pod \"auth-test-model-77a153-kserve-6fcd554cc-c5d8v\" (UID: \"655f7ad5-8f26-4d0c-88ac-3507229392c3\") " pod="ai-tenant-e2e-shared-a-w1-77a153/auth-test-model-77a153-kserve-6fcd554cc-c5d8v" Oct 31 04:35:23.291810 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:23.291788 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/655f7ad5-8f26-4d0c-88ac-3507229392c3-dshm\") pod \"auth-test-model-77a153-kserve-6fcd554cc-c5d8v\" (UID: \"655f7ad5-8f26-4d0c-88ac-3507229392c3\") " pod="ai-tenant-e2e-shared-a-w1-77a153/auth-test-model-77a153-kserve-6fcd554cc-c5d8v" Oct 31 04:35:23.291916 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:23.291887 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/655f7ad5-8f26-4d0c-88ac-3507229392c3-home\") pod \"auth-test-model-77a153-kserve-6fcd554cc-c5d8v\" (UID: \"655f7ad5-8f26-4d0c-88ac-3507229392c3\") " pod="ai-tenant-e2e-shared-a-w1-77a153/auth-test-model-77a153-kserve-6fcd554cc-c5d8v" Oct 31 04:35:23.292110 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:23.292077 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/655f7ad5-8f26-4d0c-88ac-3507229392c3-tmp-dir\") pod \"auth-test-model-77a153-kserve-6fcd554cc-c5d8v\" (UID: \"655f7ad5-8f26-4d0c-88ac-3507229392c3\") " pod="ai-tenant-e2e-shared-a-w1-77a153/auth-test-model-77a153-kserve-6fcd554cc-c5d8v" Oct 31 04:35:23.292204 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:23.292177 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/655f7ad5-8f26-4d0c-88ac-3507229392c3-model-cache\") pod \"auth-test-model-77a153-kserve-6fcd554cc-c5d8v\" (UID: \"655f7ad5-8f26-4d0c-88ac-3507229392c3\") " pod="ai-tenant-e2e-shared-a-w1-77a153/auth-test-model-77a153-kserve-6fcd554cc-c5d8v" Oct 31 04:35:23.292459 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:23.292437 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/655f7ad5-8f26-4d0c-88ac-3507229392c3-home\") pod \"auth-test-model-77a153-kserve-6fcd554cc-c5d8v\" (UID: \"655f7ad5-8f26-4d0c-88ac-3507229392c3\") " pod="ai-tenant-e2e-shared-a-w1-77a153/auth-test-model-77a153-kserve-6fcd554cc-c5d8v" Oct 31 04:35:23.294493 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:23.294459 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/655f7ad5-8f26-4d0c-88ac-3507229392c3-dshm\") pod \"auth-test-model-77a153-kserve-6fcd554cc-c5d8v\" (UID: \"655f7ad5-8f26-4d0c-88ac-3507229392c3\") " pod="ai-tenant-e2e-shared-a-w1-77a153/auth-test-model-77a153-kserve-6fcd554cc-c5d8v" Oct 31 04:35:23.294733 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:23.294716 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/655f7ad5-8f26-4d0c-88ac-3507229392c3-tls-certs\") pod \"auth-test-model-77a153-kserve-6fcd554cc-c5d8v\" (UID: \"655f7ad5-8f26-4d0c-88ac-3507229392c3\") " pod="ai-tenant-e2e-shared-a-w1-77a153/auth-test-model-77a153-kserve-6fcd554cc-c5d8v" Oct 31 04:35:23.315814 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:23.315775 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkszn\" (UniqueName: \"kubernetes.io/projected/655f7ad5-8f26-4d0c-88ac-3507229392c3-kube-api-access-fkszn\") pod \"auth-test-model-77a153-kserve-6fcd554cc-c5d8v\" (UID: \"655f7ad5-8f26-4d0c-88ac-3507229392c3\") " pod="ai-tenant-e2e-shared-a-w1-77a153/auth-test-model-77a153-kserve-6fcd554cc-c5d8v" Oct 31 04:35:23.428796 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:23.428690 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-77a153/auth-test-model-77a153-kserve-6fcd554cc-c5d8v" Oct 31 04:35:23.808765 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:35:23.808725 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod655f7ad5_8f26_4d0c_88ac_3507229392c3.slice/crio-e6c416121ec3b5aeed1f7b7dd2619db1c185c3324cb3c3f0b35e63e331232370 WatchSource:0}: Error finding container e6c416121ec3b5aeed1f7b7dd2619db1c185c3324cb3c3f0b35e63e331232370: Status 404 returned error can't find the container with id e6c416121ec3b5aeed1f7b7dd2619db1c185c3324cb3c3f0b35e63e331232370 Oct 31 04:35:23.809407 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:23.809377 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-77a153/auth-test-model-77a153-kserve-6fcd554cc-c5d8v"] Oct 31 04:35:24.157176 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:24.157077 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-77a153/auth-test-model-77a153-kserve-6fcd554cc-c5d8v" event={"ID":"655f7ad5-8f26-4d0c-88ac-3507229392c3","Type":"ContainerStarted","Data":"f27c8a6c77f65b5ca0ea21ad5ab171fa5d715f99695bdbe612003bd7d95c4cbc"} Oct 31 04:35:24.157176 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:24.157134 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-77a153/auth-test-model-77a153-kserve-6fcd554cc-c5d8v" event={"ID":"655f7ad5-8f26-4d0c-88ac-3507229392c3","Type":"ContainerStarted","Data":"e6c416121ec3b5aeed1f7b7dd2619db1c185c3324cb3c3f0b35e63e331232370"} Oct 31 04:35:24.205371 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:24.205291 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-77a153/auth-test-model-77a153-kserve-6fcd554cc-c5d8v" podStartSLOduration=1.205272584 podStartE2EDuration="1.205272584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:35:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:35:24.199850742 +0000 UTC m=+1878.956188785" watchObservedRunningTime="2026-10-31 04:35:24.205272584 +0000 UTC m=+1878.961610645" Oct 31 04:35:24.430589 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:24.430495 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-77a153/auth-test-model-77a153-kserve-6fcd554cc-c5d8v" Oct 31 04:35:29.035414 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:29.035352 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f6277de-fkjtd"] Oct 31 04:35:29.046956 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:29.046909 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f6277de-fkjtd"] Oct 31 04:35:29.411814 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:29.411719 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da002f29-2b99-451b-ada8-5c6d455341b8" path="/var/lib/kubelet/pods/da002f29-2b99-451b-ada8-5c6d455341b8/volumes" Oct 31 04:35:33.430413 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:33.430367 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-77a153/auth-test-model-77a153-kserve-6fcd554cc-c5d8v" Oct 31 04:35:33.443701 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:33.443670 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-77a153/auth-test-model-77a153-kserve-6fcd554cc-c5d8v" Oct 31 04:35:34.031813 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:34.031765 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8d161cf-8d8z4"] Oct 31 04:35:34.037469 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:34.037432 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8d161cf-8d8z4"] Oct 31 04:35:34.303866 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:34.303768 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-77a153/auth-test-model-77a153-kserve-6fcd554cc-c5d8v" Oct 31 04:35:35.412012 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:35.411975 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8d1a346-e7a0-4207-9764-6a429a868977" path="/var/lib/kubelet/pods/d8d1a346-e7a0-4207-9764-6a429a868977/volumes" Oct 31 04:35:36.255856 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:36.255817 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-7229f2/autores-model-7229f2-kserve-84cff56d8f-f8lqw"] Oct 31 04:35:39.048635 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:39.048585 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57d2fa8f-mzcpp"] Oct 31 04:35:39.051274 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:39.051244 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57d2fa8f-mzcpp"] Oct 31 04:35:39.414871 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:39.414783 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7c75409-79c8-4343-8cb6-117d0ae8f7e6" path="/var/lib/kubelet/pods/b7c75409-79c8-4343-8cb6-117d0ae8f7e6/volumes" Oct 31 04:35:51.032968 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:51.032919 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92d74b5e-q7pb7"] Oct 31 04:35:51.045191 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:51.045142 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92d74b5e-q7pb7"] Oct 31 04:35:51.291643 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:51.291472 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-7229f2/autores-model-7229f2-kserve-84cff56d8f-f8lqw" podUID="549207bf-db66-4f89-b541-edbb6ddd4f2c" containerName="main" containerID="cri-o://085a061f041c7fbd48c3c55504145b57ab3ddca3ebfd057767af85cd417b74ba" gracePeriod=45 Oct 31 04:35:51.414770 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:51.414703 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ea1146c-92d8-42f1-b7e2-5b03b24e0dae" path="/var/lib/kubelet/pods/2ea1146c-92d8-42f1-b7e2-5b03b24e0dae/volumes" Oct 31 04:35:51.640067 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:51.640024 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-7229f2/autores-model-7229f2-kserve-84cff56d8f-f8lqw" Oct 31 04:35:51.762640 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:51.762598 5019 generic.go:356] "Generic (PLEG): container finished" podID="549207bf-db66-4f89-b541-edbb6ddd4f2c" containerID="085a061f041c7fbd48c3c55504145b57ab3ddca3ebfd057767af85cd417b74ba" exitCode=0 Oct 31 04:35:51.762870 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:51.762657 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-7229f2/autores-model-7229f2-kserve-84cff56d8f-f8lqw" Oct 31 04:35:51.762870 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:51.762695 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-7229f2/autores-model-7229f2-kserve-84cff56d8f-f8lqw" event={"ID":"549207bf-db66-4f89-b541-edbb6ddd4f2c","Type":"ContainerDied","Data":"085a061f041c7fbd48c3c55504145b57ab3ddca3ebfd057767af85cd417b74ba"} Oct 31 04:35:51.762870 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:51.762745 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-7229f2/autores-model-7229f2-kserve-84cff56d8f-f8lqw" event={"ID":"549207bf-db66-4f89-b541-edbb6ddd4f2c","Type":"ContainerDied","Data":"5a107874dd2dfb8e55bb1e6b5974b557dd757e5c51ae207b8560a3205f4e07f3"} Oct 31 04:35:51.762870 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:51.762768 5019 scope.go:122] "RemoveContainer" containerID="085a061f041c7fbd48c3c55504145b57ab3ddca3ebfd057767af85cd417b74ba" Oct 31 04:35:51.782475 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:51.782447 5019 scope.go:122] "RemoveContainer" containerID="085a061f041c7fbd48c3c55504145b57ab3ddca3ebfd057767af85cd417b74ba" Oct 31 04:35:51.782888 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:35:51.782863 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"085a061f041c7fbd48c3c55504145b57ab3ddca3ebfd057767af85cd417b74ba\": container with ID starting with 085a061f041c7fbd48c3c55504145b57ab3ddca3ebfd057767af85cd417b74ba not found: ID does not exist" containerID="085a061f041c7fbd48c3c55504145b57ab3ddca3ebfd057767af85cd417b74ba" Oct 31 04:35:51.782946 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:51.782903 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"085a061f041c7fbd48c3c55504145b57ab3ddca3ebfd057767af85cd417b74ba"} err="failed to get container status \"085a061f041c7fbd48c3c55504145b57ab3ddca3ebfd057767af85cd417b74ba\": rpc error: code = NotFound desc = could not find container \"085a061f041c7fbd48c3c55504145b57ab3ddca3ebfd057767af85cd417b74ba\": container with ID starting with 085a061f041c7fbd48c3c55504145b57ab3ddca3ebfd057767af85cd417b74ba not found: ID does not exist" Oct 31 04:35:51.791482 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:51.791438 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/549207bf-db66-4f89-b541-edbb6ddd4f2c-kube-api-access-szpk6\" (UniqueName: \"kubernetes.io/projected/549207bf-db66-4f89-b541-edbb6ddd4f2c-kube-api-access-szpk6\") pod \"549207bf-db66-4f89-b541-edbb6ddd4f2c\" (UID: \"549207bf-db66-4f89-b541-edbb6ddd4f2c\") " Oct 31 04:35:51.791705 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:51.791558 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/549207bf-db66-4f89-b541-edbb6ddd4f2c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/549207bf-db66-4f89-b541-edbb6ddd4f2c-model-cache\") pod \"549207bf-db66-4f89-b541-edbb6ddd4f2c\" (UID: \"549207bf-db66-4f89-b541-edbb6ddd4f2c\") " Oct 31 04:35:51.791705 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:51.791615 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/549207bf-db66-4f89-b541-edbb6ddd4f2c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/549207bf-db66-4f89-b541-edbb6ddd4f2c-tmp-dir\") pod \"549207bf-db66-4f89-b541-edbb6ddd4f2c\" (UID: \"549207bf-db66-4f89-b541-edbb6ddd4f2c\") " Oct 31 04:35:51.791705 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:51.791664 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/549207bf-db66-4f89-b541-edbb6ddd4f2c-home\" (UniqueName: \"kubernetes.io/empty-dir/549207bf-db66-4f89-b541-edbb6ddd4f2c-home\") pod \"549207bf-db66-4f89-b541-edbb6ddd4f2c\" (UID: \"549207bf-db66-4f89-b541-edbb6ddd4f2c\") " Oct 31 04:35:51.791705 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:51.791687 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/549207bf-db66-4f89-b541-edbb6ddd4f2c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/549207bf-db66-4f89-b541-edbb6ddd4f2c-dshm\") pod \"549207bf-db66-4f89-b541-edbb6ddd4f2c\" (UID: \"549207bf-db66-4f89-b541-edbb6ddd4f2c\") " Oct 31 04:35:51.791927 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:51.791808 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/549207bf-db66-4f89-b541-edbb6ddd4f2c-tls-certs\" (UniqueName: \"kubernetes.io/secret/549207bf-db66-4f89-b541-edbb6ddd4f2c-tls-certs\") pod \"549207bf-db66-4f89-b541-edbb6ddd4f2c\" (UID: \"549207bf-db66-4f89-b541-edbb6ddd4f2c\") " Oct 31 04:35:51.791927 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:51.791885 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/549207bf-db66-4f89-b541-edbb6ddd4f2c-model-cache" pod "549207bf-db66-4f89-b541-edbb6ddd4f2c" (UID: "549207bf-db66-4f89-b541-edbb6ddd4f2c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:35:51.792037 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:51.791960 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/549207bf-db66-4f89-b541-edbb6ddd4f2c-home" pod "549207bf-db66-4f89-b541-edbb6ddd4f2c" (UID: "549207bf-db66-4f89-b541-edbb6ddd4f2c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:35:51.792111 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:51.792065 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/549207bf-db66-4f89-b541-edbb6ddd4f2c-tmp-dir" pod "549207bf-db66-4f89-b541-edbb6ddd4f2c" (UID: "549207bf-db66-4f89-b541-edbb6ddd4f2c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:35:51.792789 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:51.792747 5019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/549207bf-db66-4f89-b541-edbb6ddd4f2c-model-cache\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:35:51.792789 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:51.792773 5019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/549207bf-db66-4f89-b541-edbb6ddd4f2c-tmp-dir\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:35:51.792789 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:51.792785 5019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/549207bf-db66-4f89-b541-edbb6ddd4f2c-home\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:35:51.794261 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:51.794220 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/549207bf-db66-4f89-b541-edbb6ddd4f2c-dshm" pod "549207bf-db66-4f89-b541-edbb6ddd4f2c" (UID: "549207bf-db66-4f89-b541-edbb6ddd4f2c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:35:51.794261 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:51.794255 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/549207bf-db66-4f89-b541-edbb6ddd4f2c-kube-api-access-szpk6" pod "549207bf-db66-4f89-b541-edbb6ddd4f2c" (UID: "549207bf-db66-4f89-b541-edbb6ddd4f2c"). InnerVolumeSpecName "kube-api-access-szpk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:35:51.794489 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:51.794287 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/549207bf-db66-4f89-b541-edbb6ddd4f2c-tls-certs" pod "549207bf-db66-4f89-b541-edbb6ddd4f2c" (UID: "549207bf-db66-4f89-b541-edbb6ddd4f2c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:35:51.894648 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:51.894540 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szpk6\" (UniqueName: \"kubernetes.io/projected/549207bf-db66-4f89-b541-edbb6ddd4f2c-kube-api-access-szpk6\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:35:51.894648 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:51.894584 5019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/549207bf-db66-4f89-b541-edbb6ddd4f2c-dshm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:35:51.894648 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:51.894595 5019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/549207bf-db66-4f89-b541-edbb6ddd4f2c-tls-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:35:52.114821 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:52.114763 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-7229f2/autores-model-7229f2-kserve-84cff56d8f-f8lqw"] Oct 31 04:35:52.126000 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:52.125941 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-7229f2/autores-model-7229f2-kserve-84cff56d8f-f8lqw"] Oct 31 04:35:52.395392 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:35:52.395346 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:35:52.557166 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:52.557111 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-autores-7229f2/autores-model-7229f2-kserve-84cff56d8f-f8lqw" podUID="549207bf-db66-4f89-b541-edbb6ddd4f2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.253:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:35:53.030186 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:53.030147 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f393145-c6hh2"] Oct 31 04:35:53.041959 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:53.041917 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f393145-c6hh2"] Oct 31 04:35:53.412130 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:53.412041 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="549207bf-db66-4f89-b541-edbb6ddd4f2c" path="/var/lib/kubelet/pods/549207bf-db66-4f89-b541-edbb6ddd4f2c/volumes" Oct 31 04:35:53.412663 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:35:53.412644 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8da94f34-3152-486f-9da7-63ffb4186238" path="/var/lib/kubelet/pods/8da94f34-3152-486f-9da7-63ffb4186238/volumes" Oct 31 04:36:02.948603 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:02.948552 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-d6ac82/auth-test-model-d6ac82-kserve-5cc8848985-xm5bx"] Oct 31 04:36:02.951696 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:02.951657 5019 cpu_manager.go:414] "containerMap: removing container" podUID="549207bf-db66-4f89-b541-edbb6ddd4f2c" containerName="main" Oct 31 04:36:02.951964 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:02.951942 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="549207bf-db66-4f89-b541-edbb6ddd4f2c" containerName="main" Oct 31 04:36:02.952835 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:02.952808 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="549207bf-db66-4f89-b541-edbb6ddd4f2c" containerName="main" Oct 31 04:36:02.959905 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:02.959869 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-d6ac82/auth-test-model-d6ac82-kserve-5cc8848985-xm5bx" Oct 31 04:36:02.992917 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:02.992877 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-d6ac82\"/\"kube-root-ca.crt\"" Oct 31 04:36:02.993134 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:02.992994 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-d6ac82\"/\"openshift-service-ca.crt\"" Oct 31 04:36:02.993134 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:02.993045 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-d6ac82\"/\"default-dockercfg-jq9mz\"" Oct 31 04:36:02.993134 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:02.993105 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-d6ac82\"/\"auth-test-model-d6ac82-kserve-self-signed-certs\"" Oct 31 04:36:02.998272 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:02.998002 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-d6ac82/auth-test-model-d6ac82-kserve-5cc8848985-xm5bx"] Oct 31 04:36:03.099701 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:03.099653 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b71880a-8282-45bf-97f1-dc8edc6f4a9b-tmp-dir\") pod \"auth-test-model-d6ac82-kserve-5cc8848985-xm5bx\" (UID: \"7b71880a-8282-45bf-97f1-dc8edc6f4a9b\") " pod="ai-tenant-e2e-shared-b-w1-d6ac82/auth-test-model-d6ac82-kserve-5cc8848985-xm5bx" Oct 31 04:36:03.099939 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:03.099733 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b71880a-8282-45bf-97f1-dc8edc6f4a9b-home\") pod \"auth-test-model-d6ac82-kserve-5cc8848985-xm5bx\" (UID: \"7b71880a-8282-45bf-97f1-dc8edc6f4a9b\") " pod="ai-tenant-e2e-shared-b-w1-d6ac82/auth-test-model-d6ac82-kserve-5cc8848985-xm5bx" Oct 31 04:36:03.099939 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:03.099772 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbwv9\" (UniqueName: \"kubernetes.io/projected/7b71880a-8282-45bf-97f1-dc8edc6f4a9b-kube-api-access-lbwv9\") pod \"auth-test-model-d6ac82-kserve-5cc8848985-xm5bx\" (UID: \"7b71880a-8282-45bf-97f1-dc8edc6f4a9b\") " pod="ai-tenant-e2e-shared-b-w1-d6ac82/auth-test-model-d6ac82-kserve-5cc8848985-xm5bx" Oct 31 04:36:03.099939 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:03.099899 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b71880a-8282-45bf-97f1-dc8edc6f4a9b-dshm\") pod \"auth-test-model-d6ac82-kserve-5cc8848985-xm5bx\" (UID: \"7b71880a-8282-45bf-97f1-dc8edc6f4a9b\") " pod="ai-tenant-e2e-shared-b-w1-d6ac82/auth-test-model-d6ac82-kserve-5cc8848985-xm5bx" Oct 31 04:36:03.100141 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:03.099964 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b71880a-8282-45bf-97f1-dc8edc6f4a9b-model-cache\") pod \"auth-test-model-d6ac82-kserve-5cc8848985-xm5bx\" (UID: \"7b71880a-8282-45bf-97f1-dc8edc6f4a9b\") " pod="ai-tenant-e2e-shared-b-w1-d6ac82/auth-test-model-d6ac82-kserve-5cc8848985-xm5bx" Oct 31 04:36:03.100141 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:03.100031 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b71880a-8282-45bf-97f1-dc8edc6f4a9b-tls-certs\") pod \"auth-test-model-d6ac82-kserve-5cc8848985-xm5bx\" (UID: \"7b71880a-8282-45bf-97f1-dc8edc6f4a9b\") " pod="ai-tenant-e2e-shared-b-w1-d6ac82/auth-test-model-d6ac82-kserve-5cc8848985-xm5bx" Oct 31 04:36:03.201648 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:03.201531 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b71880a-8282-45bf-97f1-dc8edc6f4a9b-dshm\") pod \"auth-test-model-d6ac82-kserve-5cc8848985-xm5bx\" (UID: \"7b71880a-8282-45bf-97f1-dc8edc6f4a9b\") " pod="ai-tenant-e2e-shared-b-w1-d6ac82/auth-test-model-d6ac82-kserve-5cc8848985-xm5bx" Oct 31 04:36:03.201970 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:03.201945 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b71880a-8282-45bf-97f1-dc8edc6f4a9b-model-cache\") pod \"auth-test-model-d6ac82-kserve-5cc8848985-xm5bx\" (UID: \"7b71880a-8282-45bf-97f1-dc8edc6f4a9b\") " pod="ai-tenant-e2e-shared-b-w1-d6ac82/auth-test-model-d6ac82-kserve-5cc8848985-xm5bx" Oct 31 04:36:03.202168 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:03.202152 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b71880a-8282-45bf-97f1-dc8edc6f4a9b-tls-certs\") pod \"auth-test-model-d6ac82-kserve-5cc8848985-xm5bx\" (UID: \"7b71880a-8282-45bf-97f1-dc8edc6f4a9b\") " pod="ai-tenant-e2e-shared-b-w1-d6ac82/auth-test-model-d6ac82-kserve-5cc8848985-xm5bx" Oct 31 04:36:03.202603 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:03.202578 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b71880a-8282-45bf-97f1-dc8edc6f4a9b-tmp-dir\") pod \"auth-test-model-d6ac82-kserve-5cc8848985-xm5bx\" (UID: \"7b71880a-8282-45bf-97f1-dc8edc6f4a9b\") " pod="ai-tenant-e2e-shared-b-w1-d6ac82/auth-test-model-d6ac82-kserve-5cc8848985-xm5bx" Oct 31 04:36:03.202778 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:03.202760 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b71880a-8282-45bf-97f1-dc8edc6f4a9b-home\") pod \"auth-test-model-d6ac82-kserve-5cc8848985-xm5bx\" (UID: \"7b71880a-8282-45bf-97f1-dc8edc6f4a9b\") " pod="ai-tenant-e2e-shared-b-w1-d6ac82/auth-test-model-d6ac82-kserve-5cc8848985-xm5bx" Oct 31 04:36:03.202893 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:03.202875 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbwv9\" (UniqueName: \"kubernetes.io/projected/7b71880a-8282-45bf-97f1-dc8edc6f4a9b-kube-api-access-lbwv9\") pod \"auth-test-model-d6ac82-kserve-5cc8848985-xm5bx\" (UID: \"7b71880a-8282-45bf-97f1-dc8edc6f4a9b\") " pod="ai-tenant-e2e-shared-b-w1-d6ac82/auth-test-model-d6ac82-kserve-5cc8848985-xm5bx" Oct 31 04:36:03.213360 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:03.209379 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b71880a-8282-45bf-97f1-dc8edc6f4a9b-tmp-dir\") pod \"auth-test-model-d6ac82-kserve-5cc8848985-xm5bx\" (UID: \"7b71880a-8282-45bf-97f1-dc8edc6f4a9b\") " pod="ai-tenant-e2e-shared-b-w1-d6ac82/auth-test-model-d6ac82-kserve-5cc8848985-xm5bx" Oct 31 04:36:03.266791 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:03.214021 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b71880a-8282-45bf-97f1-dc8edc6f4a9b-home\") pod \"auth-test-model-d6ac82-kserve-5cc8848985-xm5bx\" (UID: \"7b71880a-8282-45bf-97f1-dc8edc6f4a9b\") " pod="ai-tenant-e2e-shared-b-w1-d6ac82/auth-test-model-d6ac82-kserve-5cc8848985-xm5bx" Oct 31 04:36:03.267061 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:03.214268 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b71880a-8282-45bf-97f1-dc8edc6f4a9b-model-cache\") pod \"auth-test-model-d6ac82-kserve-5cc8848985-xm5bx\" (UID: \"7b71880a-8282-45bf-97f1-dc8edc6f4a9b\") " pod="ai-tenant-e2e-shared-b-w1-d6ac82/auth-test-model-d6ac82-kserve-5cc8848985-xm5bx" Oct 31 04:36:03.267147 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:03.227142 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b71880a-8282-45bf-97f1-dc8edc6f4a9b-tls-certs\") pod \"auth-test-model-d6ac82-kserve-5cc8848985-xm5bx\" (UID: \"7b71880a-8282-45bf-97f1-dc8edc6f4a9b\") " pod="ai-tenant-e2e-shared-b-w1-d6ac82/auth-test-model-d6ac82-kserve-5cc8848985-xm5bx" Oct 31 04:36:03.281396 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:03.275714 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b71880a-8282-45bf-97f1-dc8edc6f4a9b-dshm\") pod \"auth-test-model-d6ac82-kserve-5cc8848985-xm5bx\" (UID: \"7b71880a-8282-45bf-97f1-dc8edc6f4a9b\") " pod="ai-tenant-e2e-shared-b-w1-d6ac82/auth-test-model-d6ac82-kserve-5cc8848985-xm5bx" Oct 31 04:36:03.321376 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:03.317554 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbwv9\" (UniqueName: \"kubernetes.io/projected/7b71880a-8282-45bf-97f1-dc8edc6f4a9b-kube-api-access-lbwv9\") pod \"auth-test-model-d6ac82-kserve-5cc8848985-xm5bx\" (UID: \"7b71880a-8282-45bf-97f1-dc8edc6f4a9b\") " pod="ai-tenant-e2e-shared-b-w1-d6ac82/auth-test-model-d6ac82-kserve-5cc8848985-xm5bx" Oct 31 04:36:03.606672 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:03.606625 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-d6ac82/auth-test-model-d6ac82-kserve-5cc8848985-xm5bx" Oct 31 04:36:03.925587 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:36:03.925551 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b71880a_8282_45bf_97f1_dc8edc6f4a9b.slice/crio-2c79920d1448181ab8128b46db4195550c34882a6946b5a34c72469358562f89 WatchSource:0}: Error finding container 2c79920d1448181ab8128b46db4195550c34882a6946b5a34c72469358562f89: Status 404 returned error can't find the container with id 2c79920d1448181ab8128b46db4195550c34882a6946b5a34c72469358562f89 Oct 31 04:36:03.926296 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:03.926260 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-d6ac82/auth-test-model-d6ac82-kserve-5cc8848985-xm5bx"] Oct 31 04:36:04.904836 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:04.904790 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-d6ac82/auth-test-model-d6ac82-kserve-5cc8848985-xm5bx" event={"ID":"7b71880a-8282-45bf-97f1-dc8edc6f4a9b","Type":"ContainerStarted","Data":"094d2a4d3defd6bfc581e3e2266e0df07797f292fb77e9b18a8c5034ed3e71a4"} Oct 31 04:36:04.905410 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:04.904850 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-d6ac82/auth-test-model-d6ac82-kserve-5cc8848985-xm5bx" event={"ID":"7b71880a-8282-45bf-97f1-dc8edc6f4a9b","Type":"ContainerStarted","Data":"2c79920d1448181ab8128b46db4195550c34882a6946b5a34c72469358562f89"} Oct 31 04:36:04.947352 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:04.947286 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-d6ac82/auth-test-model-d6ac82-kserve-5cc8848985-xm5bx" podStartSLOduration=2.947271533 podStartE2EDuration="2.947271533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:36:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:36:04.946162703 +0000 UTC m=+1919.702500757" watchObservedRunningTime="2026-10-31 04:36:04.947271533 +0000 UTC m=+1919.703609572" Oct 31 04:36:05.619477 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:05.619433 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-d6ac82/auth-test-model-d6ac82-kserve-5cc8848985-xm5bx" Oct 31 04:36:08.034841 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:08.034788 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-549ca28e-zhcw4"] Oct 31 04:36:08.046163 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:08.046120 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-549ca28e-zhcw4"] Oct 31 04:36:09.236179 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:09.236139 5019 scope.go:122] "RemoveContainer" containerID="09757164f5be9a87f68e6ab824bb3db2b7896effd161947f9522309365a19dff" Oct 31 04:36:09.253327 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:09.253292 5019 scope.go:122] "RemoveContainer" containerID="24efb5a8f7884be7d98fafa7151321897a5538ccf8c8694f68b698444b60553b" Oct 31 04:36:09.411308 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:09.411254 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9098da5-5571-4cfb-8c10-a1541763c969" path="/var/lib/kubelet/pods/d9098da5-5571-4cfb-8c10-a1541763c969/volumes" Oct 31 04:36:13.607060 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:13.607018 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-d6ac82/auth-test-model-d6ac82-kserve-5cc8848985-xm5bx" Oct 31 04:36:13.620060 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:13.620021 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-d6ac82/auth-test-model-d6ac82-kserve-5cc8848985-xm5bx" Oct 31 04:36:14.014767 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:14.014732 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-d6ac82/auth-test-model-d6ac82-kserve-5cc8848985-xm5bx" Oct 31 04:36:15.030774 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:15.030730 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f20a4797-csz4v"] Oct 31 04:36:15.042385 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:15.042323 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f20a4797-csz4v"] Oct 31 04:36:15.416291 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:15.416160 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c0927ea-ba50-457d-9cdc-c289169d44f7" path="/var/lib/kubelet/pods/9c0927ea-ba50-457d-9cdc-c289169d44f7/volumes" Oct 31 04:36:18.029483 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:18.029421 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-7229f2-798cdf44cf-4wm7j"] Oct 31 04:36:22.394746 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:36:22.394703 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:36:23.030453 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:23.030394 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-7229f2-798cdf44cf-4wm7j" podUID="69471f9f-03f8-433e-8793-26529badb542" containerName="payload-pre-processing" containerID="cri-o://e85ecb3b9713e99a1b19838df01a61082cd9bb46d7a2e9eec91a7270db0f3728" gracePeriod=25 Oct 31 04:36:23.364320 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:23.364289 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-7229f2-798cdf44cf-4wm7j" Oct 31 04:36:23.442664 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:23.442617 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69471f9f-03f8-433e-8793-26529badb542-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69471f9f-03f8-433e-8793-26529badb542-plugins-config-volume\") pod \"69471f9f-03f8-433e-8793-26529badb542\" (UID: \"69471f9f-03f8-433e-8793-26529badb542\") " Oct 31 04:36:23.443185 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:23.442786 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69471f9f-03f8-433e-8793-26529badb542-kube-api-access-fssq7\" (UniqueName: \"kubernetes.io/projected/69471f9f-03f8-433e-8793-26529badb542-kube-api-access-fssq7\") pod \"69471f9f-03f8-433e-8793-26529badb542\" (UID: \"69471f9f-03f8-433e-8793-26529badb542\") " Oct 31 04:36:23.443185 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:23.443090 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69471f9f-03f8-433e-8793-26529badb542-plugins-config-volume" pod "69471f9f-03f8-433e-8793-26529badb542" (UID: "69471f9f-03f8-433e-8793-26529badb542"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:36:23.443720 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:23.443696 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69471f9f-03f8-433e-8793-26529badb542-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:36:23.445125 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:23.445096 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69471f9f-03f8-433e-8793-26529badb542-kube-api-access-fssq7" pod "69471f9f-03f8-433e-8793-26529badb542" (UID: "69471f9f-03f8-433e-8793-26529badb542"). InnerVolumeSpecName "kube-api-access-fssq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:36:23.545861 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:23.545812 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fssq7\" (UniqueName: \"kubernetes.io/projected/69471f9f-03f8-433e-8793-26529badb542-kube-api-access-fssq7\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:36:24.030897 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:24.030850 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d033c6a-bgnt7"] Oct 31 04:36:24.042205 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:24.042163 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d033c6a-bgnt7"] Oct 31 04:36:24.110456 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:24.110416 5019 generic.go:356] "Generic (PLEG): container finished" podID="69471f9f-03f8-433e-8793-26529badb542" containerID="e85ecb3b9713e99a1b19838df01a61082cd9bb46d7a2e9eec91a7270db0f3728" exitCode=0 Oct 31 04:36:24.110670 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:24.110481 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-7229f2-798cdf44cf-4wm7j" Oct 31 04:36:24.110670 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:24.110516 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-7229f2-798cdf44cf-4wm7j" event={"ID":"69471f9f-03f8-433e-8793-26529badb542","Type":"ContainerDied","Data":"e85ecb3b9713e99a1b19838df01a61082cd9bb46d7a2e9eec91a7270db0f3728"} Oct 31 04:36:24.110670 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:24.110580 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-7229f2-798cdf44cf-4wm7j" event={"ID":"69471f9f-03f8-433e-8793-26529badb542","Type":"ContainerDied","Data":"0edd57895c385d281e3de88134f0e67ccc5ffd375d75cb8dc6715617dff062b6"} Oct 31 04:36:24.110670 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:24.110606 5019 scope.go:122] "RemoveContainer" containerID="e85ecb3b9713e99a1b19838df01a61082cd9bb46d7a2e9eec91a7270db0f3728" Oct 31 04:36:24.124928 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:24.124898 5019 scope.go:122] "RemoveContainer" containerID="e85ecb3b9713e99a1b19838df01a61082cd9bb46d7a2e9eec91a7270db0f3728" Oct 31 04:36:24.125283 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:36:24.125262 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e85ecb3b9713e99a1b19838df01a61082cd9bb46d7a2e9eec91a7270db0f3728\": container with ID starting with e85ecb3b9713e99a1b19838df01a61082cd9bb46d7a2e9eec91a7270db0f3728 not found: ID does not exist" containerID="e85ecb3b9713e99a1b19838df01a61082cd9bb46d7a2e9eec91a7270db0f3728" Oct 31 04:36:24.125429 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:24.125291 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e85ecb3b9713e99a1b19838df01a61082cd9bb46d7a2e9eec91a7270db0f3728"} err="failed to get container status \"e85ecb3b9713e99a1b19838df01a61082cd9bb46d7a2e9eec91a7270db0f3728\": rpc error: code = NotFound desc = could not find container \"e85ecb3b9713e99a1b19838df01a61082cd9bb46d7a2e9eec91a7270db0f3728\": container with ID starting with e85ecb3b9713e99a1b19838df01a61082cd9bb46d7a2e9eec91a7270db0f3728 not found: ID does not exist" Oct 31 04:36:24.165295 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:24.165250 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-7229f2-798cdf44cf-4wm7j"] Oct 31 04:36:24.170445 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:24.170406 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-7229f2-798cdf44cf-4wm7j"] Oct 31 04:36:25.427317 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:25.427268 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ace7b49-3ea3-4594-925a-09e1c3ebeb01" path="/var/lib/kubelet/pods/3ace7b49-3ea3-4594-925a-09e1c3ebeb01/volumes" Oct 31 04:36:25.427979 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:25.427956 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69471f9f-03f8-433e-8793-26529badb542" path="/var/lib/kubelet/pods/69471f9f-03f8-433e-8793-26529badb542/volumes" Oct 31 04:36:29.031700 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:29.031660 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ce95e7f-gkxj9"] Oct 31 04:36:29.043499 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:29.043456 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ce95e7f-gkxj9"] Oct 31 04:36:29.411045 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:29.410951 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69624fdb-1b77-4d58-a5bc-70daa1dd48e5" path="/var/lib/kubelet/pods/69624fdb-1b77-4d58-a5bc-70daa1dd48e5/volumes" Oct 31 04:36:37.030766 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:37.030723 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09a10af3-kzrw6"] Oct 31 04:36:37.042927 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:37.042891 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09a10af3-kzrw6"] Oct 31 04:36:37.411721 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:37.411595 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33e5de72-d4a8-4cc1-ae49-7877abe669ce" path="/var/lib/kubelet/pods/33e5de72-d4a8-4cc1-ae49-7877abe669ce/volumes" Oct 31 04:36:42.329567 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:42.329520 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb"] Oct 31 04:36:42.330840 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:42.330810 5019 cpu_manager.go:414] "containerMap: removing container" podUID="69471f9f-03f8-433e-8793-26529badb542" containerName="payload-pre-processing" Oct 31 04:36:42.330840 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:42.330841 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69471f9f-03f8-433e-8793-26529badb542" containerName="payload-pre-processing" Oct 31 04:36:42.331093 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:42.331074 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="69471f9f-03f8-433e-8793-26529badb542" containerName="payload-pre-processing" Oct 31 04:36:42.337312 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:42.337282 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb" Oct 31 04:36:42.355950 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:42.355913 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb"] Oct 31 04:36:42.396675 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:42.396612 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-575fe8-openshift-default-dockercfg-c295r\"" Oct 31 04:36:42.449365 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:42.445922 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-istiod-ca-cert\") pod \"e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb\" (UID: \"6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d\") " pod="openshift-ingress/e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb" Oct 31 04:36:42.449365 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:42.446101 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-workload-certs\") pod \"e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb\" (UID: \"6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d\") " pod="openshift-ingress/e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb" Oct 31 04:36:42.449365 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:42.446140 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-workload-socket\") pod \"e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb\" (UID: \"6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d\") " pod="openshift-ingress/e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb" Oct 31 04:36:42.449365 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:42.446187 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-istio-envoy\") pod \"e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb\" (UID: \"6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d\") " pod="openshift-ingress/e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb" Oct 31 04:36:42.449365 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:42.446356 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-credential-socket\") pod \"e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb\" (UID: \"6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d\") " pod="openshift-ingress/e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb" Oct 31 04:36:42.449365 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:42.446402 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-istio-podinfo\") pod \"e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb\" (UID: \"6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d\") " pod="openshift-ingress/e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb" Oct 31 04:36:42.449365 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:42.446475 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-istio-token\") pod \"e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb\" (UID: \"6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d\") " pod="openshift-ingress/e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb" Oct 31 04:36:42.449365 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:42.446554 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8br47\" (UniqueName: \"kubernetes.io/projected/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-kube-api-access-8br47\") pod \"e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb\" (UID: \"6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d\") " pod="openshift-ingress/e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb" Oct 31 04:36:42.449365 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:42.446596 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-istio-data\") pod \"e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb\" (UID: \"6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d\") " pod="openshift-ingress/e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb" Oct 31 04:36:42.557444 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:42.557401 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-credential-socket\") pod \"e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb\" (UID: \"6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d\") " pod="openshift-ingress/e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb" Oct 31 04:36:42.577357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:42.558029 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-credential-socket\") pod \"e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb\" (UID: \"6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d\") " pod="openshift-ingress/e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb" Oct 31 04:36:42.577718 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:42.577687 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-istio-podinfo\") pod \"e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb\" (UID: \"6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d\") " pod="openshift-ingress/e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb" Oct 31 04:36:42.577937 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:42.577918 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-istio-token\") pod \"e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb\" (UID: \"6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d\") " pod="openshift-ingress/e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb" Oct 31 04:36:42.578117 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:42.578100 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8br47\" (UniqueName: \"kubernetes.io/projected/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-kube-api-access-8br47\") pod \"e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb\" (UID: \"6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d\") " pod="openshift-ingress/e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb" Oct 31 04:36:42.578246 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:42.578231 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-istio-data\") pod \"e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb\" (UID: \"6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d\") " pod="openshift-ingress/e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb" Oct 31 04:36:42.578426 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:42.578409 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-istiod-ca-cert\") pod \"e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb\" (UID: \"6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d\") " pod="openshift-ingress/e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb" Oct 31 04:36:42.578675 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:42.578657 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-workload-certs\") pod \"e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb\" (UID: \"6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d\") " pod="openshift-ingress/e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb" Oct 31 04:36:42.578805 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:42.578788 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-workload-socket\") pod \"e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb\" (UID: \"6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d\") " pod="openshift-ingress/e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb" Oct 31 04:36:42.578941 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:42.578925 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-istio-envoy\") pod \"e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb\" (UID: \"6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d\") " pod="openshift-ingress/e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb" Oct 31 04:36:42.579985 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:42.579930 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-istio-data\") pod \"e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb\" (UID: \"6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d\") " pod="openshift-ingress/e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb" Oct 31 04:36:42.586153 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:42.586125 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-workload-certs\") pod \"e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb\" (UID: \"6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d\") " pod="openshift-ingress/e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb" Oct 31 04:36:42.586956 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:42.586934 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-istiod-ca-cert\") pod \"e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb\" (UID: \"6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d\") " pod="openshift-ingress/e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb" Oct 31 04:36:42.588427 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:42.587366 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-workload-socket\") pod \"e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb\" (UID: \"6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d\") " pod="openshift-ingress/e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb" Oct 31 04:36:42.604892 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:42.596234 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-istio-podinfo\") pod \"e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb\" (UID: \"6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d\") " pod="openshift-ingress/e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb" Oct 31 04:36:42.605911 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:42.605871 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-istio-envoy\") pod \"e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb\" (UID: \"6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d\") " pod="openshift-ingress/e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb" Oct 31 04:36:42.615867 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:42.615807 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8br47\" (UniqueName: \"kubernetes.io/projected/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-kube-api-access-8br47\") pod \"e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb\" (UID: \"6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d\") " pod="openshift-ingress/e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb" Oct 31 04:36:42.616513 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:42.616467 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-istio-token\") pod \"e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb\" (UID: \"6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d\") " pod="openshift-ingress/e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb" Oct 31 04:36:42.660829 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:42.660611 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb" Oct 31 04:36:43.089192 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:36:43.089158 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c2bc7c4_aca8_4c2f_a8df_7ad805f6ad2d.slice/crio-15ff27eaf06ecbf4da19b01a674ad53408809a09cd364f6c40f7d3bed19db46c WatchSource:0}: Error finding container 15ff27eaf06ecbf4da19b01a674ad53408809a09cd364f6c40f7d3bed19db46c: Status 404 returned error can't find the container with id 15ff27eaf06ecbf4da19b01a674ad53408809a09cd364f6c40f7d3bed19db46c Oct 31 04:36:43.091993 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:43.091964 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb"] Oct 31 04:36:43.092493 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:43.092453 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:36:43.092559 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:43.092543 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:36:43.092596 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:43.092584 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:36:43.419190 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:43.419101 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb" event={"ID":"6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d","Type":"ContainerStarted","Data":"cf82479d2cfcd12cadf0c83199b3e07b15f702f14b7b2ccdc0e524f2f36bd039"} Oct 31 04:36:43.419190 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:43.419154 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb" event={"ID":"6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d","Type":"ContainerStarted","Data":"15ff27eaf06ecbf4da19b01a674ad53408809a09cd364f6c40f7d3bed19db46c"} Oct 31 04:36:43.467763 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:43.467704 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb" podStartSLOduration=1.467676621 podStartE2EDuration="1.467676621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:36:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:36:43.463735611 +0000 UTC m=+1958.220073665" watchObservedRunningTime="2026-10-31 04:36:43.467676621 +0000 UTC m=+1958.224014661" Oct 31 04:36:43.661393 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:43.661321 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb" Oct 31 04:36:44.667729 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:44.667696 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb" Oct 31 04:36:45.437499 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:45.437456 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb" Oct 31 04:36:45.438490 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:45.438468 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb" Oct 31 04:36:46.031646 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:46.031606 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f8cb9ed-fb8mh"] Oct 31 04:36:46.036489 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:46.036452 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f8cb9ed-fb8mh"] Oct 31 04:36:47.514962 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:47.508651 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b49996e2-beb8-4dcf-b709-ddc13df6976c" path="/var/lib/kubelet/pods/b49996e2-beb8-4dcf-b709-ddc13df6976c/volumes" Oct 31 04:36:48.081039 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:48.080994 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ab66a90-tl5cg"] Oct 31 04:36:48.112293 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:48.112248 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ab66a90-tl5cg"] Oct 31 04:36:49.412950 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:49.412903 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26ef4705-4fef-4182-ad47-d3d7c232e0a0" path="/var/lib/kubelet/pods/26ef4705-4fef-4182-ad47-d3d7c232e0a0/volumes" Oct 31 04:36:50.033963 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:50.033930 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85913ce3-mjwbv"] Oct 31 04:36:50.045983 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:50.045952 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85913ce3-mjwbv"] Oct 31 04:36:51.413254 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:51.413202 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8041aee9-d4f6-41e3-8309-b298131dbf49" path="/var/lib/kubelet/pods/8041aee9-d4f6-41e3-8309-b298131dbf49/volumes" Oct 31 04:36:52.030308 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:52.030271 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81af36e3-qnf9f"] Oct 31 04:36:52.042736 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:52.042702 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81af36e3-qnf9f"] Oct 31 04:36:52.393879 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:36:52.393772 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:36:53.413679 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:36:53.413632 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80707894-d35d-43c3-8bbb-c443f7e9a4e2" path="/var/lib/kubelet/pods/80707894-d35d-43c3-8bbb-c443f7e9a4e2/volumes" Oct 31 04:37:01.030876 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:01.030831 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97790921-ljcjc"] Oct 31 04:37:01.042236 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:01.042196 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97790921-ljcjc"] Oct 31 04:37:01.411951 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:01.411854 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a912a29d-b513-4355-93b3-fee0059c902d" path="/var/lib/kubelet/pods/a912a29d-b513-4355-93b3-fee0059c902d/volumes" Oct 31 04:37:09.029683 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:09.029646 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2023e453-9kxs6"] Oct 31 04:37:09.034359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:09.034309 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2023e453-9kxs6"] Oct 31 04:37:09.311234 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:09.311136 5019 scope.go:122] "RemoveContainer" containerID="7473b80a497d8c89a2afcc061869f56df510bd4abe0df63458d50c6d5657f5e2" Oct 31 04:37:09.324178 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:09.324150 5019 scope.go:122] "RemoveContainer" containerID="de6b3d01e5343f1bc53229b51db6da07c35751ce45800f098ccb87b2e3176698" Oct 31 04:37:09.335942 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:09.335918 5019 scope.go:122] "RemoveContainer" containerID="5874438da227c29d36da52f00273f685a5d85d8d149ff5bdb4f0647db508be3c" Oct 31 04:37:09.347708 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:09.347683 5019 scope.go:122] "RemoveContainer" containerID="3d91f2721a05811c1ba9501dcdcf30b980d2428f82fa7ef716ba3b40cb487519" Oct 31 04:37:09.359486 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:09.359463 5019 scope.go:122] "RemoveContainer" containerID="0a20045ea6fb6b32f66a62b9e335ba3249e8592a5f4db52ee2ce4f3ef9b37c12" Oct 31 04:37:09.371964 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:09.371938 5019 scope.go:122] "RemoveContainer" containerID="7f030fb95c24d49d9c02c6bee389ee146f4c1e1d95f6e13490c1023d0ff0f963" Oct 31 04:37:09.411715 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:09.411679 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d1e5631-5416-4a0a-ab66-716b75268f1b" path="/var/lib/kubelet/pods/4d1e5631-5416-4a0a-ab66-716b75268f1b/volumes" Oct 31 04:37:11.407356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:11.404270 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb"] Oct 31 04:37:11.407356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:11.404596 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb" podUID="6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d" containerName="istio-proxy" containerID="cri-o://cf82479d2cfcd12cadf0c83199b3e07b15f702f14b7b2ccdc0e524f2f36bd039" gracePeriod=30 Oct 31 04:37:14.030135 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:14.030091 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-552fab68-mhm82"] Oct 31 04:37:14.041649 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:14.041614 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-552fab68-mhm82"] Oct 31 04:37:15.410927 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:15.410884 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32940c15-666b-42f4-ab4e-67a371895cfd" path="/var/lib/kubelet/pods/32940c15-666b-42f4-ab4e-67a371895cfd/volumes" Oct 31 04:37:15.439727 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:15.439679 5019 patch_prober.go:29] interesting pod/e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Oct 31 04:37:15.439907 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:15.439766 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb" podUID="6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 04:37:16.647425 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:16.647382 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb" Oct 31 04:37:16.705541 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:16.705419 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-istio-podinfo\") pod \"6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d\" (UID: \"6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d\") " Oct 31 04:37:16.705541 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:16.705516 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-kube-api-access-8br47\" (UniqueName: \"kubernetes.io/projected/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-kube-api-access-8br47\") pod \"6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d\" (UID: \"6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d\") " Oct 31 04:37:16.710786 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:16.705611 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-workload-socket\") pod \"6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d\" (UID: \"6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d\") " Oct 31 04:37:16.710786 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:16.705682 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-istio-envoy\") pod \"6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d\" (UID: \"6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d\") " Oct 31 04:37:16.710786 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:16.705793 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-credential-socket\") pod \"6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d\" (UID: \"6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d\") " Oct 31 04:37:16.710786 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:16.705856 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-istio-data\") pod \"6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d\" (UID: \"6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d\") " Oct 31 04:37:16.710786 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:16.706002 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-workload-certs\") pod \"6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d\" (UID: \"6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d\") " Oct 31 04:37:16.710786 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:16.706196 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-istiod-ca-cert\") pod \"6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d\" (UID: \"6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d\") " Oct 31 04:37:16.710786 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:16.706298 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-istio-token\" (UniqueName: \"kubernetes.io/projected/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-istio-token\") pod \"6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d\" (UID: \"6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d\") " Oct 31 04:37:16.710786 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:16.706601 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-credential-socket" pod "6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d" (UID: "6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:37:16.710786 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:16.706748 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-workload-certs" pod "6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d" (UID: "6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:37:16.710786 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:16.706830 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-istiod-ca-cert" pod "6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d" (UID: "6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:37:16.710786 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:16.707030 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-workload-socket" pod "6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d" (UID: "6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:37:16.710786 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:16.707225 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-istio-data" pod "6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d" (UID: "6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:37:16.710786 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:16.707521 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:37:16.710786 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:16.707545 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:37:16.710786 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:16.707559 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:37:16.710786 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:16.707574 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:37:16.712892 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:16.712849 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-istio-envoy" pod "6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d" (UID: "6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:37:16.713130 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:16.713028 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-kube-api-access-8br47" pod "6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d" (UID: "6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d"). InnerVolumeSpecName "kube-api-access-8br47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:37:16.720418 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:16.717710 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-istio-token" pod "6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d" (UID: "6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:37:16.720418 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:16.719524 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-istio-podinfo" pod "6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d" (UID: "6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:37:16.809284 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:16.809234 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:37:16.809284 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:16.809271 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:37:16.809284 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:16.809280 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:37:16.809284 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:16.809289 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:37:16.809284 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:16.809298 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8br47\" (UniqueName: \"kubernetes.io/projected/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d-kube-api-access-8br47\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:37:16.900485 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:16.900435 5019 generic.go:356] "Generic (PLEG): container finished" podID="6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d" containerID="cf82479d2cfcd12cadf0c83199b3e07b15f702f14b7b2ccdc0e524f2f36bd039" exitCode=0 Oct 31 04:37:16.900704 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:16.900489 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb" event={"ID":"6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d","Type":"ContainerDied","Data":"cf82479d2cfcd12cadf0c83199b3e07b15f702f14b7b2ccdc0e524f2f36bd039"} Oct 31 04:37:16.900704 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:16.900527 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb" Oct 31 04:37:16.900704 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:16.900558 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb" event={"ID":"6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d","Type":"ContainerDied","Data":"15ff27eaf06ecbf4da19b01a674ad53408809a09cd364f6c40f7d3bed19db46c"} Oct 31 04:37:16.900704 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:16.900588 5019 scope.go:122] "RemoveContainer" containerID="cf82479d2cfcd12cadf0c83199b3e07b15f702f14b7b2ccdc0e524f2f36bd039" Oct 31 04:37:16.916317 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:16.916284 5019 scope.go:122] "RemoveContainer" containerID="cf82479d2cfcd12cadf0c83199b3e07b15f702f14b7b2ccdc0e524f2f36bd039" Oct 31 04:37:16.916714 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:37:16.916687 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf82479d2cfcd12cadf0c83199b3e07b15f702f14b7b2ccdc0e524f2f36bd039\": container with ID starting with cf82479d2cfcd12cadf0c83199b3e07b15f702f14b7b2ccdc0e524f2f36bd039 not found: ID does not exist" containerID="cf82479d2cfcd12cadf0c83199b3e07b15f702f14b7b2ccdc0e524f2f36bd039" Oct 31 04:37:16.916799 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:16.916728 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf82479d2cfcd12cadf0c83199b3e07b15f702f14b7b2ccdc0e524f2f36bd039"} err="failed to get container status \"cf82479d2cfcd12cadf0c83199b3e07b15f702f14b7b2ccdc0e524f2f36bd039\": rpc error: code = NotFound desc = could not find container \"cf82479d2cfcd12cadf0c83199b3e07b15f702f14b7b2ccdc0e524f2f36bd039\": container with ID starting with cf82479d2cfcd12cadf0c83199b3e07b15f702f14b7b2ccdc0e524f2f36bd039 not found: ID does not exist" Oct 31 04:37:16.956246 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:16.956156 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb"] Oct 31 04:37:16.960826 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:16.960789 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-575fe8-openshift-default-85c4b685b5-sznbb"] Oct 31 04:37:17.412512 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:17.412470 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d" path="/var/lib/kubelet/pods/6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d/volumes" Oct 31 04:37:22.397459 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:37:22.397416 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:37:26.603072 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:26.603036 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp"] Oct 31 04:37:26.603549 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:26.603228 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp" podUID="6d85d51e-dcd7-4e26-b793-70db760b09a8" containerName="istio-proxy" containerID="cri-o://a70d56685a0531cce0c8dd173dca0076ffe53830bc1eb3293caeb17e71a7ede8" gracePeriod=30 Oct 31 04:37:26.617850 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:26.617700 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8"] Oct 31 04:37:26.618047 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:26.617951 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8" podUID="75bc7ee5-eb66-43cf-8bca-5322159afbb0" containerName="istio-proxy" containerID="cri-o://d82b084680c042ef089fa6846c20e46586cb28301fdb17e6e67d6ee7fcb37da2" gracePeriod=30 Oct 31 04:37:31.942277 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:31.941832 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp" Oct 31 04:37:31.984422 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:31.980610 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d85d51e-dcd7-4e26-b793-70db760b09a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d85d51e-dcd7-4e26-b793-70db760b09a8-istio-data\") pod \"6d85d51e-dcd7-4e26-b793-70db760b09a8\" (UID: \"6d85d51e-dcd7-4e26-b793-70db760b09a8\") " Oct 31 04:37:31.984422 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:31.980684 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d85d51e-dcd7-4e26-b793-70db760b09a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d85d51e-dcd7-4e26-b793-70db760b09a8-workload-socket\") pod \"6d85d51e-dcd7-4e26-b793-70db760b09a8\" (UID: \"6d85d51e-dcd7-4e26-b793-70db760b09a8\") " Oct 31 04:37:31.984422 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:31.980727 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d85d51e-dcd7-4e26-b793-70db760b09a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d85d51e-dcd7-4e26-b793-70db760b09a8-istio-podinfo\") pod \"6d85d51e-dcd7-4e26-b793-70db760b09a8\" (UID: \"6d85d51e-dcd7-4e26-b793-70db760b09a8\") " Oct 31 04:37:31.984422 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:31.980763 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d85d51e-dcd7-4e26-b793-70db760b09a8-istio-token\" (UniqueName: \"kubernetes.io/projected/6d85d51e-dcd7-4e26-b793-70db760b09a8-istio-token\") pod \"6d85d51e-dcd7-4e26-b793-70db760b09a8\" (UID: \"6d85d51e-dcd7-4e26-b793-70db760b09a8\") " Oct 31 04:37:31.984422 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:31.980844 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d85d51e-dcd7-4e26-b793-70db760b09a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d85d51e-dcd7-4e26-b793-70db760b09a8-istio-envoy\") pod \"6d85d51e-dcd7-4e26-b793-70db760b09a8\" (UID: \"6d85d51e-dcd7-4e26-b793-70db760b09a8\") " Oct 31 04:37:31.984422 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:31.980899 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d85d51e-dcd7-4e26-b793-70db760b09a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d85d51e-dcd7-4e26-b793-70db760b09a8-credential-socket\") pod \"6d85d51e-dcd7-4e26-b793-70db760b09a8\" (UID: \"6d85d51e-dcd7-4e26-b793-70db760b09a8\") " Oct 31 04:37:31.984422 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:31.981245 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d85d51e-dcd7-4e26-b793-70db760b09a8-kube-api-access-bjmvz\" (UniqueName: \"kubernetes.io/projected/6d85d51e-dcd7-4e26-b793-70db760b09a8-kube-api-access-bjmvz\") pod \"6d85d51e-dcd7-4e26-b793-70db760b09a8\" (UID: \"6d85d51e-dcd7-4e26-b793-70db760b09a8\") " Oct 31 04:37:31.984422 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:31.981353 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d85d51e-dcd7-4e26-b793-70db760b09a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d85d51e-dcd7-4e26-b793-70db760b09a8-istiod-ca-cert\") pod \"6d85d51e-dcd7-4e26-b793-70db760b09a8\" (UID: \"6d85d51e-dcd7-4e26-b793-70db760b09a8\") " Oct 31 04:37:31.984422 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:31.981408 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d85d51e-dcd7-4e26-b793-70db760b09a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d85d51e-dcd7-4e26-b793-70db760b09a8-workload-certs\") pod \"6d85d51e-dcd7-4e26-b793-70db760b09a8\" (UID: \"6d85d51e-dcd7-4e26-b793-70db760b09a8\") " Oct 31 04:37:31.984422 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:31.982974 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d85d51e-dcd7-4e26-b793-70db760b09a8-credential-socket" pod "6d85d51e-dcd7-4e26-b793-70db760b09a8" (UID: "6d85d51e-dcd7-4e26-b793-70db760b09a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:37:31.984422 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:31.983760 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d85d51e-dcd7-4e26-b793-70db760b09a8-workload-certs" pod "6d85d51e-dcd7-4e26-b793-70db760b09a8" (UID: "6d85d51e-dcd7-4e26-b793-70db760b09a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:37:31.984422 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:31.983899 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d85d51e-dcd7-4e26-b793-70db760b09a8-workload-socket" pod "6d85d51e-dcd7-4e26-b793-70db760b09a8" (UID: "6d85d51e-dcd7-4e26-b793-70db760b09a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:37:31.984422 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:31.984160 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d85d51e-dcd7-4e26-b793-70db760b09a8-istio-data" pod "6d85d51e-dcd7-4e26-b793-70db760b09a8" (UID: "6d85d51e-dcd7-4e26-b793-70db760b09a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:37:31.985366 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:31.984548 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d85d51e-dcd7-4e26-b793-70db760b09a8-istiod-ca-cert" pod "6d85d51e-dcd7-4e26-b793-70db760b09a8" (UID: "6d85d51e-dcd7-4e26-b793-70db760b09a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:37:31.987194 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:31.987146 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d85d51e-dcd7-4e26-b793-70db760b09a8-istio-envoy" pod "6d85d51e-dcd7-4e26-b793-70db760b09a8" (UID: "6d85d51e-dcd7-4e26-b793-70db760b09a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:37:31.987585 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:31.987533 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d85d51e-dcd7-4e26-b793-70db760b09a8-istio-podinfo" pod "6d85d51e-dcd7-4e26-b793-70db760b09a8" (UID: "6d85d51e-dcd7-4e26-b793-70db760b09a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:37:31.989576 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:31.989520 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d85d51e-dcd7-4e26-b793-70db760b09a8-istio-token" pod "6d85d51e-dcd7-4e26-b793-70db760b09a8" (UID: "6d85d51e-dcd7-4e26-b793-70db760b09a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:37:31.990730 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:31.990652 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d85d51e-dcd7-4e26-b793-70db760b09a8-kube-api-access-bjmvz" pod "6d85d51e-dcd7-4e26-b793-70db760b09a8" (UID: "6d85d51e-dcd7-4e26-b793-70db760b09a8"). InnerVolumeSpecName "kube-api-access-bjmvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:37:32.054316 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:32.054280 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8" Oct 31 04:37:32.061863 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:32.061826 5019 generic.go:356] "Generic (PLEG): container finished" podID="75bc7ee5-eb66-43cf-8bca-5322159afbb0" containerID="d82b084680c042ef089fa6846c20e46586cb28301fdb17e6e67d6ee7fcb37da2" exitCode=0 Oct 31 04:37:32.062059 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:32.061892 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8" event={"ID":"75bc7ee5-eb66-43cf-8bca-5322159afbb0","Type":"ContainerDied","Data":"d82b084680c042ef089fa6846c20e46586cb28301fdb17e6e67d6ee7fcb37da2"} Oct 31 04:37:32.062059 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:32.061974 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8" event={"ID":"75bc7ee5-eb66-43cf-8bca-5322159afbb0","Type":"ContainerDied","Data":"64dce2f6a76177c2cacf300c29d87a39c6993417fa435ad058147ee410781371"} Oct 31 04:37:32.062059 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:32.062011 5019 scope.go:122] "RemoveContainer" containerID="d82b084680c042ef089fa6846c20e46586cb28301fdb17e6e67d6ee7fcb37da2" Oct 31 04:37:32.067078 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:32.067042 5019 generic.go:356] "Generic (PLEG): container finished" podID="6d85d51e-dcd7-4e26-b793-70db760b09a8" containerID="a70d56685a0531cce0c8dd173dca0076ffe53830bc1eb3293caeb17e71a7ede8" exitCode=0 Oct 31 04:37:32.067406 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:32.067107 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp" Oct 31 04:37:32.067406 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:32.067145 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp" event={"ID":"6d85d51e-dcd7-4e26-b793-70db760b09a8","Type":"ContainerDied","Data":"a70d56685a0531cce0c8dd173dca0076ffe53830bc1eb3293caeb17e71a7ede8"} Oct 31 04:37:32.067406 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:32.067223 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp" event={"ID":"6d85d51e-dcd7-4e26-b793-70db760b09a8","Type":"ContainerDied","Data":"57f400a0aca34b4ba068444b1d07aa1ae583252140a88b5172f4163fb2118263"} Oct 31 04:37:32.082956 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:32.082923 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75bc7ee5-eb66-43cf-8bca-5322159afbb0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75bc7ee5-eb66-43cf-8bca-5322159afbb0-credential-socket\") pod \"75bc7ee5-eb66-43cf-8bca-5322159afbb0\" (UID: \"75bc7ee5-eb66-43cf-8bca-5322159afbb0\") " Oct 31 04:37:32.083051 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:32.083040 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75bc7ee5-eb66-43cf-8bca-5322159afbb0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75bc7ee5-eb66-43cf-8bca-5322159afbb0-istio-envoy\") pod \"75bc7ee5-eb66-43cf-8bca-5322159afbb0\" (UID: \"75bc7ee5-eb66-43cf-8bca-5322159afbb0\") " Oct 31 04:37:32.083109 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:32.083097 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/75bc7ee5-eb66-43cf-8bca-5322159afbb0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75bc7ee5-eb66-43cf-8bca-5322159afbb0-istio-podinfo\") pod \"75bc7ee5-eb66-43cf-8bca-5322159afbb0\" (UID: \"75bc7ee5-eb66-43cf-8bca-5322159afbb0\") " Oct 31 04:37:32.083186 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:32.083171 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75bc7ee5-eb66-43cf-8bca-5322159afbb0-kube-api-access-n5pgf\" (UniqueName: \"kubernetes.io/projected/75bc7ee5-eb66-43cf-8bca-5322159afbb0-kube-api-access-n5pgf\") pod \"75bc7ee5-eb66-43cf-8bca-5322159afbb0\" (UID: \"75bc7ee5-eb66-43cf-8bca-5322159afbb0\") " Oct 31 04:37:32.083224 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:32.083206 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75bc7ee5-eb66-43cf-8bca-5322159afbb0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75bc7ee5-eb66-43cf-8bca-5322159afbb0-workload-certs\") pod \"75bc7ee5-eb66-43cf-8bca-5322159afbb0\" (UID: \"75bc7ee5-eb66-43cf-8bca-5322159afbb0\") " Oct 31 04:37:32.083266 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:32.083215 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75bc7ee5-eb66-43cf-8bca-5322159afbb0-credential-socket" pod "75bc7ee5-eb66-43cf-8bca-5322159afbb0" (UID: "75bc7ee5-eb66-43cf-8bca-5322159afbb0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:37:32.083305 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:32.083264 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75bc7ee5-eb66-43cf-8bca-5322159afbb0-istio-token\" (UniqueName: \"kubernetes.io/projected/75bc7ee5-eb66-43cf-8bca-5322159afbb0-istio-token\") pod \"75bc7ee5-eb66-43cf-8bca-5322159afbb0\" (UID: \"75bc7ee5-eb66-43cf-8bca-5322159afbb0\") " Oct 31 04:37:32.083369 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:32.083323 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75bc7ee5-eb66-43cf-8bca-5322159afbb0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75bc7ee5-eb66-43cf-8bca-5322159afbb0-istio-data\") pod \"75bc7ee5-eb66-43cf-8bca-5322159afbb0\" (UID: \"75bc7ee5-eb66-43cf-8bca-5322159afbb0\") " Oct 31 04:37:32.083429 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:32.083412 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75bc7ee5-eb66-43cf-8bca-5322159afbb0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75bc7ee5-eb66-43cf-8bca-5322159afbb0-istiod-ca-cert\") pod \"75bc7ee5-eb66-43cf-8bca-5322159afbb0\" (UID: \"75bc7ee5-eb66-43cf-8bca-5322159afbb0\") " Oct 31 04:37:32.083658 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:32.083641 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75bc7ee5-eb66-43cf-8bca-5322159afbb0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75bc7ee5-eb66-43cf-8bca-5322159afbb0-workload-socket\") pod \"75bc7ee5-eb66-43cf-8bca-5322159afbb0\" (UID: \"75bc7ee5-eb66-43cf-8bca-5322159afbb0\") " Oct 31 04:37:32.084365 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:32.084345 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d85d51e-dcd7-4e26-b793-70db760b09a8-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:37:32.084458 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:32.084372 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d85d51e-dcd7-4e26-b793-70db760b09a8-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:37:32.084458 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:32.084387 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d85d51e-dcd7-4e26-b793-70db760b09a8-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:37:32.084458 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:32.084401 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d85d51e-dcd7-4e26-b793-70db760b09a8-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:37:32.084458 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:32.084414 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d85d51e-dcd7-4e26-b793-70db760b09a8-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:37:32.084458 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:32.084426 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75bc7ee5-eb66-43cf-8bca-5322159afbb0-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:37:32.084458 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:32.084439 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d85d51e-dcd7-4e26-b793-70db760b09a8-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:37:32.084458 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:32.084453 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjmvz\" (UniqueName: \"kubernetes.io/projected/6d85d51e-dcd7-4e26-b793-70db760b09a8-kube-api-access-bjmvz\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:37:32.084682 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:32.084467 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d85d51e-dcd7-4e26-b793-70db760b09a8-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:37:32.084682 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:32.084480 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d85d51e-dcd7-4e26-b793-70db760b09a8-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:37:32.084982 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:32.084958 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75bc7ee5-eb66-43cf-8bca-5322159afbb0-workload-socket" pod "75bc7ee5-eb66-43cf-8bca-5322159afbb0" (UID: "75bc7ee5-eb66-43cf-8bca-5322159afbb0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:37:32.085220 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:32.085200 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75bc7ee5-eb66-43cf-8bca-5322159afbb0-istio-data" pod "75bc7ee5-eb66-43cf-8bca-5322159afbb0" (UID: "75bc7ee5-eb66-43cf-8bca-5322159afbb0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:37:32.085417 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:32.085391 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75bc7ee5-eb66-43cf-8bca-5322159afbb0-istiod-ca-cert" pod "75bc7ee5-eb66-43cf-8bca-5322159afbb0" (UID: "75bc7ee5-eb66-43cf-8bca-5322159afbb0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:37:32.085699 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:32.085678 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75bc7ee5-eb66-43cf-8bca-5322159afbb0-workload-certs" pod "75bc7ee5-eb66-43cf-8bca-5322159afbb0" (UID: "75bc7ee5-eb66-43cf-8bca-5322159afbb0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:37:32.086295 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:32.086270 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75bc7ee5-eb66-43cf-8bca-5322159afbb0-istio-podinfo" pod "75bc7ee5-eb66-43cf-8bca-5322159afbb0" (UID: "75bc7ee5-eb66-43cf-8bca-5322159afbb0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:37:32.098398 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:32.098327 5019 scope.go:122] "RemoveContainer" containerID="d82b084680c042ef089fa6846c20e46586cb28301fdb17e6e67d6ee7fcb37da2" Oct 31 04:37:32.098810 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:37:32.098784 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d82b084680c042ef089fa6846c20e46586cb28301fdb17e6e67d6ee7fcb37da2\": container with ID starting with d82b084680c042ef089fa6846c20e46586cb28301fdb17e6e67d6ee7fcb37da2 not found: ID does not exist" containerID="d82b084680c042ef089fa6846c20e46586cb28301fdb17e6e67d6ee7fcb37da2" Oct 31 04:37:32.098920 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:32.098823 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d82b084680c042ef089fa6846c20e46586cb28301fdb17e6e67d6ee7fcb37da2"} err="failed to get container status \"d82b084680c042ef089fa6846c20e46586cb28301fdb17e6e67d6ee7fcb37da2\": rpc error: code = NotFound desc = could not find container \"d82b084680c042ef089fa6846c20e46586cb28301fdb17e6e67d6ee7fcb37da2\": container with ID starting with d82b084680c042ef089fa6846c20e46586cb28301fdb17e6e67d6ee7fcb37da2 not found: ID does not exist" Oct 31 04:37:32.098920 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:32.098851 5019 scope.go:122] "RemoveContainer" containerID="a70d56685a0531cce0c8dd173dca0076ffe53830bc1eb3293caeb17e71a7ede8" Oct 31 04:37:32.109778 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:32.109747 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75bc7ee5-eb66-43cf-8bca-5322159afbb0-istio-token" pod "75bc7ee5-eb66-43cf-8bca-5322159afbb0" (UID: "75bc7ee5-eb66-43cf-8bca-5322159afbb0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:37:32.112539 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:32.112488 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75bc7ee5-eb66-43cf-8bca-5322159afbb0-istio-envoy" pod "75bc7ee5-eb66-43cf-8bca-5322159afbb0" (UID: "75bc7ee5-eb66-43cf-8bca-5322159afbb0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:37:32.113558 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:32.113527 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75bc7ee5-eb66-43cf-8bca-5322159afbb0-kube-api-access-n5pgf" pod "75bc7ee5-eb66-43cf-8bca-5322159afbb0" (UID: "75bc7ee5-eb66-43cf-8bca-5322159afbb0"). InnerVolumeSpecName "kube-api-access-n5pgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:37:32.125345 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:32.125309 5019 scope.go:122] "RemoveContainer" containerID="a70d56685a0531cce0c8dd173dca0076ffe53830bc1eb3293caeb17e71a7ede8" Oct 31 04:37:32.125901 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:37:32.125868 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a70d56685a0531cce0c8dd173dca0076ffe53830bc1eb3293caeb17e71a7ede8\": container with ID starting with a70d56685a0531cce0c8dd173dca0076ffe53830bc1eb3293caeb17e71a7ede8 not found: ID does not exist" containerID="a70d56685a0531cce0c8dd173dca0076ffe53830bc1eb3293caeb17e71a7ede8" Oct 31 04:37:32.126028 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:32.125910 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a70d56685a0531cce0c8dd173dca0076ffe53830bc1eb3293caeb17e71a7ede8"} err="failed to get container status \"a70d56685a0531cce0c8dd173dca0076ffe53830bc1eb3293caeb17e71a7ede8\": rpc error: code = NotFound desc = could not find container \"a70d56685a0531cce0c8dd173dca0076ffe53830bc1eb3293caeb17e71a7ede8\": container with ID starting with a70d56685a0531cce0c8dd173dca0076ffe53830bc1eb3293caeb17e71a7ede8 not found: ID does not exist" Oct 31 04:37:32.138101 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:37:32.136209 5019 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d85d51e_dcd7_4e26_b793_70db760b09a8.slice/crio-57f400a0aca34b4ba068444b1d07aa1ae583252140a88b5172f4163fb2118263\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d85d51e_dcd7_4e26_b793_70db760b09a8.slice\": RecentStats: unable to find data in memory cache]" Oct 31 04:37:32.145424 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:32.145384 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp"] Oct 31 04:37:32.149892 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:32.149860 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-xmpnp"] Oct 31 04:37:32.186207 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:32.186126 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75bc7ee5-eb66-43cf-8bca-5322159afbb0-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:37:32.186207 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:32.186164 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75bc7ee5-eb66-43cf-8bca-5322159afbb0-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:37:32.186207 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:32.186178 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75bc7ee5-eb66-43cf-8bca-5322159afbb0-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:37:32.186207 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:32.186195 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75bc7ee5-eb66-43cf-8bca-5322159afbb0-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:37:32.186207 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:32.186208 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5pgf\" (UniqueName: \"kubernetes.io/projected/75bc7ee5-eb66-43cf-8bca-5322159afbb0-kube-api-access-n5pgf\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:37:32.186526 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:32.186221 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75bc7ee5-eb66-43cf-8bca-5322159afbb0-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:37:32.186526 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:32.186233 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75bc7ee5-eb66-43cf-8bca-5322159afbb0-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:37:32.186526 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:32.186245 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75bc7ee5-eb66-43cf-8bca-5322159afbb0-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:37:33.080177 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:33.080138 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8" Oct 31 04:37:33.153752 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:33.153706 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8"] Oct 31 04:37:33.165731 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:33.165692 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-mxtf8"] Oct 31 04:37:33.412144 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:33.412055 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d85d51e-dcd7-4e26-b793-70db760b09a8" path="/var/lib/kubelet/pods/6d85d51e-dcd7-4e26-b793-70db760b09a8/volumes" Oct 31 04:37:33.412620 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:33.412604 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75bc7ee5-eb66-43cf-8bca-5322159afbb0" path="/var/lib/kubelet/pods/75bc7ee5-eb66-43cf-8bca-5322159afbb0/volumes" Oct 31 04:37:50.030558 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:50.030468 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f55a7cc5-6vpwm"] Oct 31 04:37:50.042327 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:50.042293 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f55a7cc5-6vpwm"] Oct 31 04:37:51.410798 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:51.410761 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c62cf027-5252-4913-933e-31d98e74eb01" path="/var/lib/kubelet/pods/c62cf027-5252-4913-933e-31d98e74eb01/volumes" Oct 31 04:37:52.397713 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:37:52.397665 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:37:56.031379 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:56.031323 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e299b1ef-str2b"] Oct 31 04:37:56.042823 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:56.042792 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e299b1ef-str2b"] Oct 31 04:37:57.442297 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:57.441992 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fc8ddf7-db96-4b79-a18d-afd173264640" path="/var/lib/kubelet/pods/8fc8ddf7-db96-4b79-a18d-afd173264640/volumes" Oct 31 04:37:59.834704 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:59.834664 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz"] Oct 31 04:37:59.835851 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:59.835824 5019 cpu_manager.go:414] "containerMap: removing container" podUID="6d85d51e-dcd7-4e26-b793-70db760b09a8" containerName="istio-proxy" Oct 31 04:37:59.835851 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:59.835850 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d85d51e-dcd7-4e26-b793-70db760b09a8" containerName="istio-proxy" Oct 31 04:37:59.836029 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:59.835866 5019 cpu_manager.go:414] "containerMap: removing container" podUID="6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d" containerName="istio-proxy" Oct 31 04:37:59.836029 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:59.835875 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d" containerName="istio-proxy" Oct 31 04:37:59.836029 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:59.835925 5019 cpu_manager.go:414] "containerMap: removing container" podUID="75bc7ee5-eb66-43cf-8bca-5322159afbb0" containerName="istio-proxy" Oct 31 04:37:59.836029 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:59.835934 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75bc7ee5-eb66-43cf-8bca-5322159afbb0" containerName="istio-proxy" Oct 31 04:37:59.836223 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:59.836116 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d85d51e-dcd7-4e26-b793-70db760b09a8" containerName="istio-proxy" Oct 31 04:37:59.836223 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:59.836126 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c2bc7c4-aca8-4c2f-a8df-7ad805f6ad2d" containerName="istio-proxy" Oct 31 04:37:59.836223 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:59.836144 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="75bc7ee5-eb66-43cf-8bca-5322159afbb0" containerName="istio-proxy" Oct 31 04:37:59.840905 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:59.840876 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz" Oct 31 04:37:59.858123 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:59.858094 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-766728-openshift-default-dockercfg-chcqx\"" Oct 31 04:37:59.873030 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:59.872998 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de1f94d9-8507-47a5-aec8-88c546112c7f-istio-podinfo\") pod \"e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz\" (UID: \"de1f94d9-8507-47a5-aec8-88c546112c7f\") " pod="openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz" Oct 31 04:37:59.873287 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:59.873270 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de1f94d9-8507-47a5-aec8-88c546112c7f-istiod-ca-cert\") pod \"e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz\" (UID: \"de1f94d9-8507-47a5-aec8-88c546112c7f\") " pod="openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz" Oct 31 04:37:59.873487 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:59.873468 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de1f94d9-8507-47a5-aec8-88c546112c7f-istio-token\") pod \"e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz\" (UID: \"de1f94d9-8507-47a5-aec8-88c546112c7f\") " pod="openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz" Oct 31 04:37:59.873652 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:59.873630 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de1f94d9-8507-47a5-aec8-88c546112c7f-workload-certs\") pod \"e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz\" (UID: \"de1f94d9-8507-47a5-aec8-88c546112c7f\") " pod="openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz" Oct 31 04:37:59.873828 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:59.873662 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de1f94d9-8507-47a5-aec8-88c546112c7f-istio-envoy\") pod \"e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz\" (UID: \"de1f94d9-8507-47a5-aec8-88c546112c7f\") " pod="openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz" Oct 31 04:37:59.873828 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:59.873800 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de1f94d9-8507-47a5-aec8-88c546112c7f-istio-data\") pod \"e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz\" (UID: \"de1f94d9-8507-47a5-aec8-88c546112c7f\") " pod="openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz" Oct 31 04:37:59.874005 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:59.873828 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de1f94d9-8507-47a5-aec8-88c546112c7f-workload-socket\") pod \"e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz\" (UID: \"de1f94d9-8507-47a5-aec8-88c546112c7f\") " pod="openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz" Oct 31 04:37:59.874005 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:59.873858 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs5q2\" (UniqueName: \"kubernetes.io/projected/de1f94d9-8507-47a5-aec8-88c546112c7f-kube-api-access-xs5q2\") pod \"e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz\" (UID: \"de1f94d9-8507-47a5-aec8-88c546112c7f\") " pod="openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz" Oct 31 04:37:59.874005 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:59.873967 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de1f94d9-8507-47a5-aec8-88c546112c7f-credential-socket\") pod \"e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz\" (UID: \"de1f94d9-8507-47a5-aec8-88c546112c7f\") " pod="openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz" Oct 31 04:37:59.880625 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:59.880575 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz"] Oct 31 04:37:59.975938 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:59.975901 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de1f94d9-8507-47a5-aec8-88c546112c7f-credential-socket\") pod \"e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz\" (UID: \"de1f94d9-8507-47a5-aec8-88c546112c7f\") " pod="openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz" Oct 31 04:37:59.976120 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:59.976022 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de1f94d9-8507-47a5-aec8-88c546112c7f-istio-podinfo\") pod \"e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz\" (UID: \"de1f94d9-8507-47a5-aec8-88c546112c7f\") " pod="openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz" Oct 31 04:37:59.976120 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:59.976075 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de1f94d9-8507-47a5-aec8-88c546112c7f-istiod-ca-cert\") pod \"e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz\" (UID: \"de1f94d9-8507-47a5-aec8-88c546112c7f\") " pod="openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz" Oct 31 04:37:59.976120 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:59.976101 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de1f94d9-8507-47a5-aec8-88c546112c7f-istio-token\") pod \"e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz\" (UID: \"de1f94d9-8507-47a5-aec8-88c546112c7f\") " pod="openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz" Oct 31 04:37:59.976313 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:59.976143 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de1f94d9-8507-47a5-aec8-88c546112c7f-workload-certs\") pod \"e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz\" (UID: \"de1f94d9-8507-47a5-aec8-88c546112c7f\") " pod="openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz" Oct 31 04:37:59.976313 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:59.976159 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de1f94d9-8507-47a5-aec8-88c546112c7f-istio-envoy\") pod \"e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz\" (UID: \"de1f94d9-8507-47a5-aec8-88c546112c7f\") " pod="openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz" Oct 31 04:37:59.976313 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:59.976285 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de1f94d9-8507-47a5-aec8-88c546112c7f-istio-data\") pod \"e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz\" (UID: \"de1f94d9-8507-47a5-aec8-88c546112c7f\") " pod="openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz" Oct 31 04:37:59.976508 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:59.976317 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de1f94d9-8507-47a5-aec8-88c546112c7f-workload-socket\") pod \"e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz\" (UID: \"de1f94d9-8507-47a5-aec8-88c546112c7f\") " pod="openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz" Oct 31 04:37:59.976508 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:59.976360 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de1f94d9-8507-47a5-aec8-88c546112c7f-credential-socket\") pod \"e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz\" (UID: \"de1f94d9-8507-47a5-aec8-88c546112c7f\") " pod="openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz" Oct 31 04:37:59.976508 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:59.976366 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs5q2\" (UniqueName: \"kubernetes.io/projected/de1f94d9-8507-47a5-aec8-88c546112c7f-kube-api-access-xs5q2\") pod \"e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz\" (UID: \"de1f94d9-8507-47a5-aec8-88c546112c7f\") " pod="openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz" Oct 31 04:37:59.976805 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:59.976780 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de1f94d9-8507-47a5-aec8-88c546112c7f-workload-certs\") pod \"e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz\" (UID: \"de1f94d9-8507-47a5-aec8-88c546112c7f\") " pod="openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz" Oct 31 04:37:59.977046 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:59.976812 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de1f94d9-8507-47a5-aec8-88c546112c7f-istio-data\") pod \"e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz\" (UID: \"de1f94d9-8507-47a5-aec8-88c546112c7f\") " pod="openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz" Oct 31 04:37:59.977148 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:59.976813 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de1f94d9-8507-47a5-aec8-88c546112c7f-istiod-ca-cert\") pod \"e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz\" (UID: \"de1f94d9-8507-47a5-aec8-88c546112c7f\") " pod="openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz" Oct 31 04:37:59.977148 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:59.977052 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de1f94d9-8507-47a5-aec8-88c546112c7f-workload-socket\") pod \"e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz\" (UID: \"de1f94d9-8507-47a5-aec8-88c546112c7f\") " pod="openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz" Oct 31 04:37:59.978712 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:59.978689 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de1f94d9-8507-47a5-aec8-88c546112c7f-istio-envoy\") pod \"e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz\" (UID: \"de1f94d9-8507-47a5-aec8-88c546112c7f\") " pod="openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz" Oct 31 04:37:59.978833 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:59.978773 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de1f94d9-8507-47a5-aec8-88c546112c7f-istio-podinfo\") pod \"e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz\" (UID: \"de1f94d9-8507-47a5-aec8-88c546112c7f\") " pod="openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz" Oct 31 04:37:59.999385 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:59.999359 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de1f94d9-8507-47a5-aec8-88c546112c7f-istio-token\") pod \"e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz\" (UID: \"de1f94d9-8507-47a5-aec8-88c546112c7f\") " pod="openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz" Oct 31 04:37:59.999496 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:37:59.999473 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs5q2\" (UniqueName: \"kubernetes.io/projected/de1f94d9-8507-47a5-aec8-88c546112c7f-kube-api-access-xs5q2\") pod \"e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz\" (UID: \"de1f94d9-8507-47a5-aec8-88c546112c7f\") " pod="openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz" Oct 31 04:38:00.182405 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:00.181653 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz" Oct 31 04:38:00.614082 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:00.614052 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz"] Oct 31 04:38:00.617594 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:38:00.617542 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde1f94d9_8507_47a5_aec8_88c546112c7f.slice/crio-610c4119df441c58e59f0350da349afecbd924c14e598d62777928cedbd4eb1c WatchSource:0}: Error finding container 610c4119df441c58e59f0350da349afecbd924c14e598d62777928cedbd4eb1c: Status 404 returned error can't find the container with id 610c4119df441c58e59f0350da349afecbd924c14e598d62777928cedbd4eb1c Oct 31 04:38:00.620808 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:00.620764 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:38:00.620912 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:00.620839 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:38:00.620912 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:00.620867 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:38:00.954209 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:00.954125 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51d08ef9-fqnzh"] Oct 31 04:38:00.960513 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:00.960488 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51d08ef9-fqnzh" Oct 31 04:38:00.976580 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:00.976548 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:38:00.976768 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:00.976621 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:38:00.976888 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:00.976725 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:38:00.977036 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:00.977022 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:38:00.985262 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:00.985237 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51d08ef9-fqnzh"] Oct 31 04:38:01.075159 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:01.075117 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0814967-8fa3-4141-8fd2-3b4c3ff630b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51d08ef9-fqnzh\" (UID: \"e0814967-8fa3-4141-8fd2-3b4c3ff630b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51d08ef9-fqnzh" Oct 31 04:38:01.075665 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:01.075642 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfb8t\" (UniqueName: \"kubernetes.io/projected/e0814967-8fa3-4141-8fd2-3b4c3ff630b7-kube-api-access-rfb8t\") pod \"maas-api-revoke-sub-51d08ef9-fqnzh\" (UID: \"e0814967-8fa3-4141-8fd2-3b4c3ff630b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51d08ef9-fqnzh" Oct 31 04:38:01.075824 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:01.075736 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0814967-8fa3-4141-8fd2-3b4c3ff630b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51d08ef9-fqnzh\" (UID: \"e0814967-8fa3-4141-8fd2-3b4c3ff630b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51d08ef9-fqnzh" Oct 31 04:38:01.177095 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:01.177057 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfb8t\" (UniqueName: \"kubernetes.io/projected/e0814967-8fa3-4141-8fd2-3b4c3ff630b7-kube-api-access-rfb8t\") pod \"maas-api-revoke-sub-51d08ef9-fqnzh\" (UID: \"e0814967-8fa3-4141-8fd2-3b4c3ff630b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51d08ef9-fqnzh" Oct 31 04:38:01.177298 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:01.177104 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0814967-8fa3-4141-8fd2-3b4c3ff630b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51d08ef9-fqnzh\" (UID: \"e0814967-8fa3-4141-8fd2-3b4c3ff630b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51d08ef9-fqnzh" Oct 31 04:38:01.177298 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:01.177276 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0814967-8fa3-4141-8fd2-3b4c3ff630b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51d08ef9-fqnzh\" (UID: \"e0814967-8fa3-4141-8fd2-3b4c3ff630b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51d08ef9-fqnzh" Oct 31 04:38:01.177976 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:01.177952 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0814967-8fa3-4141-8fd2-3b4c3ff630b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51d08ef9-fqnzh\" (UID: \"e0814967-8fa3-4141-8fd2-3b4c3ff630b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51d08ef9-fqnzh" Oct 31 04:38:01.179753 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:01.179726 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0814967-8fa3-4141-8fd2-3b4c3ff630b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51d08ef9-fqnzh\" (UID: \"e0814967-8fa3-4141-8fd2-3b4c3ff630b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51d08ef9-fqnzh" Oct 31 04:38:01.201176 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:01.201138 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfb8t\" (UniqueName: \"kubernetes.io/projected/e0814967-8fa3-4141-8fd2-3b4c3ff630b7-kube-api-access-rfb8t\") pod \"maas-api-revoke-sub-51d08ef9-fqnzh\" (UID: \"e0814967-8fa3-4141-8fd2-3b4c3ff630b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51d08ef9-fqnzh" Oct 31 04:38:01.289294 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:01.289251 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51d08ef9-fqnzh" Oct 31 04:38:01.433788 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:01.433745 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz" event={"ID":"de1f94d9-8507-47a5-aec8-88c546112c7f","Type":"ContainerStarted","Data":"a3128cd2d5d93597ea6ac8b493b037d64093bbcf3f8da977dd3db858ff57b612"} Oct 31 04:38:01.433984 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:01.433808 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz" event={"ID":"de1f94d9-8507-47a5-aec8-88c546112c7f","Type":"ContainerStarted","Data":"610c4119df441c58e59f0350da349afecbd924c14e598d62777928cedbd4eb1c"} Oct 31 04:38:01.621918 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:01.621797 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz" podStartSLOduration=2.621773101 podStartE2EDuration="2.621773101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:37:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:38:01.503202192 +0000 UTC m=+2036.259540234" watchObservedRunningTime="2026-10-31 04:38:01.621773101 +0000 UTC m=+2036.378111141" Oct 31 04:38:01.648396 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:01.643940 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-691c771f-44gp5"] Oct 31 04:38:01.648396 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:38:01.646839 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0814967_8fa3_4141_8fd2_3b4c3ff630b7.slice/crio-f8f34fbe8c40f32876236d1bcd41e043e27e4f0a6d3d067bbd8f641633f36272 WatchSource:0}: Error finding container f8f34fbe8c40f32876236d1bcd41e043e27e4f0a6d3d067bbd8f641633f36272: Status 404 returned error can't find the container with id f8f34fbe8c40f32876236d1bcd41e043e27e4f0a6d3d067bbd8f641633f36272 Oct 31 04:38:01.715110 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:01.715067 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51d08ef9-fqnzh"] Oct 31 04:38:01.715311 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:01.715129 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-691c771f-44gp5"] Oct 31 04:38:01.715311 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:01.715209 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-691c771f-44gp5" Oct 31 04:38:01.718273 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:01.718187 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49d411b5-f78zv"] Oct 31 04:38:01.724389 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:01.723319 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d411b5-f78zv" Oct 31 04:38:01.752280 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:01.752169 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49d411b5-f78zv"] Oct 31 04:38:01.912137 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:01.912028 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db80704e-ed92-4de7-a6fb-d1189af34b16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-691c771f-44gp5\" (UID: \"db80704e-ed92-4de7-a6fb-d1189af34b16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-691c771f-44gp5" Oct 31 04:38:01.912317 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:01.912256 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db80704e-ed92-4de7-a6fb-d1189af34b16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-691c771f-44gp5\" (UID: \"db80704e-ed92-4de7-a6fb-d1189af34b16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-691c771f-44gp5" Oct 31 04:38:01.912413 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:01.912396 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dd71c11-2201-4923-8ba1-d6b14db03087-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49d411b5-f78zv\" (UID: \"4dd71c11-2201-4923-8ba1-d6b14db03087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d411b5-f78zv" Oct 31 04:38:01.912488 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:01.912469 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dd71c11-2201-4923-8ba1-d6b14db03087-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49d411b5-f78zv\" (UID: \"4dd71c11-2201-4923-8ba1-d6b14db03087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d411b5-f78zv" Oct 31 04:38:01.912534 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:01.912514 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf7d8\" (UniqueName: \"kubernetes.io/projected/4dd71c11-2201-4923-8ba1-d6b14db03087-kube-api-access-gf7d8\") pod \"maas-api-revoke-sub-49d411b5-f78zv\" (UID: \"4dd71c11-2201-4923-8ba1-d6b14db03087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d411b5-f78zv" Oct 31 04:38:01.912587 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:01.912543 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj8dc\" (UniqueName: \"kubernetes.io/projected/db80704e-ed92-4de7-a6fb-d1189af34b16-kube-api-access-cj8dc\") pod \"maas-api-revoke-sub-691c771f-44gp5\" (UID: \"db80704e-ed92-4de7-a6fb-d1189af34b16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-691c771f-44gp5" Oct 31 04:38:02.018106 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:02.018063 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf7d8\" (UniqueName: \"kubernetes.io/projected/4dd71c11-2201-4923-8ba1-d6b14db03087-kube-api-access-gf7d8\") pod \"maas-api-revoke-sub-49d411b5-f78zv\" (UID: \"4dd71c11-2201-4923-8ba1-d6b14db03087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d411b5-f78zv" Oct 31 04:38:02.018106 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:02.018127 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj8dc\" (UniqueName: \"kubernetes.io/projected/db80704e-ed92-4de7-a6fb-d1189af34b16-kube-api-access-cj8dc\") pod \"maas-api-revoke-sub-691c771f-44gp5\" (UID: \"db80704e-ed92-4de7-a6fb-d1189af34b16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-691c771f-44gp5" Oct 31 04:38:02.018750 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:02.018325 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db80704e-ed92-4de7-a6fb-d1189af34b16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-691c771f-44gp5\" (UID: \"db80704e-ed92-4de7-a6fb-d1189af34b16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-691c771f-44gp5" Oct 31 04:38:02.018750 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:02.018507 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db80704e-ed92-4de7-a6fb-d1189af34b16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-691c771f-44gp5\" (UID: \"db80704e-ed92-4de7-a6fb-d1189af34b16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-691c771f-44gp5" Oct 31 04:38:02.018750 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:02.018612 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dd71c11-2201-4923-8ba1-d6b14db03087-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49d411b5-f78zv\" (UID: \"4dd71c11-2201-4923-8ba1-d6b14db03087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d411b5-f78zv" Oct 31 04:38:02.018750 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:02.018662 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dd71c11-2201-4923-8ba1-d6b14db03087-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49d411b5-f78zv\" (UID: \"4dd71c11-2201-4923-8ba1-d6b14db03087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d411b5-f78zv" Oct 31 04:38:02.019545 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:02.019517 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dd71c11-2201-4923-8ba1-d6b14db03087-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49d411b5-f78zv\" (UID: \"4dd71c11-2201-4923-8ba1-d6b14db03087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d411b5-f78zv" Oct 31 04:38:02.020012 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:02.019988 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db80704e-ed92-4de7-a6fb-d1189af34b16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-691c771f-44gp5\" (UID: \"db80704e-ed92-4de7-a6fb-d1189af34b16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-691c771f-44gp5" Oct 31 04:38:02.022386 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:02.021671 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db80704e-ed92-4de7-a6fb-d1189af34b16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-691c771f-44gp5\" (UID: \"db80704e-ed92-4de7-a6fb-d1189af34b16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-691c771f-44gp5" Oct 31 04:38:02.024791 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:02.024766 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dd71c11-2201-4923-8ba1-d6b14db03087-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49d411b5-f78zv\" (UID: \"4dd71c11-2201-4923-8ba1-d6b14db03087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d411b5-f78zv" Oct 31 04:38:02.042437 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:02.042399 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf7d8\" (UniqueName: \"kubernetes.io/projected/4dd71c11-2201-4923-8ba1-d6b14db03087-kube-api-access-gf7d8\") pod \"maas-api-revoke-sub-49d411b5-f78zv\" (UID: \"4dd71c11-2201-4923-8ba1-d6b14db03087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d411b5-f78zv" Oct 31 04:38:02.044382 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:02.044324 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj8dc\" (UniqueName: \"kubernetes.io/projected/db80704e-ed92-4de7-a6fb-d1189af34b16-kube-api-access-cj8dc\") pod \"maas-api-revoke-sub-691c771f-44gp5\" (UID: \"db80704e-ed92-4de7-a6fb-d1189af34b16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-691c771f-44gp5" Oct 31 04:38:02.049227 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:02.049192 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-691c771f-44gp5" Oct 31 04:38:02.069221 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:02.069185 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d411b5-f78zv" Oct 31 04:38:02.182529 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:02.182448 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz" Oct 31 04:38:02.193223 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:02.193194 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz" Oct 31 04:38:02.389424 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:38:02.389388 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb80704e_ed92_4de7_a6fb_d1189af34b16.slice/crio-08c28448ac1957ad1640d77646e92b6a28c0b752d0d85560b8065ccbdf2cbfa2 WatchSource:0}: Error finding container 08c28448ac1957ad1640d77646e92b6a28c0b752d0d85560b8065ccbdf2cbfa2: Status 404 returned error can't find the container with id 08c28448ac1957ad1640d77646e92b6a28c0b752d0d85560b8065ccbdf2cbfa2 Oct 31 04:38:02.390863 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:02.390831 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-691c771f-44gp5"] Oct 31 04:38:02.414105 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:02.414037 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49d411b5-f78zv"] Oct 31 04:38:02.468615 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:02.468580 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d411b5-f78zv" event={"ID":"4dd71c11-2201-4923-8ba1-d6b14db03087","Type":"ContainerStarted","Data":"0ce4117410b952e375820772391ea256633513dc5c33124072daf60e0d55f489"} Oct 31 04:38:02.474151 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:02.472861 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51d08ef9-fqnzh" event={"ID":"e0814967-8fa3-4141-8fd2-3b4c3ff630b7","Type":"ContainerStarted","Data":"bdc8427a1673f544911c23bb6f2ae6b4153a78963c18c1ce417d3e93ae48c65e"} Oct 31 04:38:02.474151 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:02.472922 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51d08ef9-fqnzh" event={"ID":"e0814967-8fa3-4141-8fd2-3b4c3ff630b7","Type":"ContainerStarted","Data":"f8f34fbe8c40f32876236d1bcd41e043e27e4f0a6d3d067bbd8f641633f36272"} Oct 31 04:38:02.478875 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:02.478099 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-691c771f-44gp5" event={"ID":"db80704e-ed92-4de7-a6fb-d1189af34b16","Type":"ContainerStarted","Data":"08c28448ac1957ad1640d77646e92b6a28c0b752d0d85560b8065ccbdf2cbfa2"} Oct 31 04:38:02.478875 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:02.478179 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz" Oct 31 04:38:02.479513 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:02.479483 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz" Oct 31 04:38:02.517957 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:02.517564 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51d08ef9-fqnzh" podStartSLOduration=2.5175489779999998 podStartE2EDuration="2.517548978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:38:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:38:02.515123103 +0000 UTC m=+2037.271461141" watchObservedRunningTime="2026-10-31 04:38:02.517548978 +0000 UTC m=+2037.273887017" Oct 31 04:38:03.486660 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:03.486619 5019 generic.go:356] "Generic (PLEG): container finished" podID="db80704e-ed92-4de7-a6fb-d1189af34b16" containerID="7f981f358b1931cfb8c57126da1d63beeca57a7a92de0f93e60529dab68ffc7a" exitCode=0 Oct 31 04:38:03.487097 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:03.486702 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-691c771f-44gp5" event={"ID":"db80704e-ed92-4de7-a6fb-d1189af34b16","Type":"ContainerDied","Data":"7f981f358b1931cfb8c57126da1d63beeca57a7a92de0f93e60529dab68ffc7a"} Oct 31 04:38:03.488730 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:03.488704 5019 generic.go:356] "Generic (PLEG): container finished" podID="4dd71c11-2201-4923-8ba1-d6b14db03087" containerID="a6cf2ea41017d37df9669b3578bb2ffbe50ec9e51033132fc582dcd5bcebbced" exitCode=0 Oct 31 04:38:03.488879 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:03.488782 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d411b5-f78zv" event={"ID":"4dd71c11-2201-4923-8ba1-d6b14db03087","Type":"ContainerDied","Data":"a6cf2ea41017d37df9669b3578bb2ffbe50ec9e51033132fc582dcd5bcebbced"} Oct 31 04:38:03.491289 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:03.491260 5019 generic.go:356] "Generic (PLEG): container finished" podID="e0814967-8fa3-4141-8fd2-3b4c3ff630b7" containerID="bdc8427a1673f544911c23bb6f2ae6b4153a78963c18c1ce417d3e93ae48c65e" exitCode=0 Oct 31 04:38:03.491474 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:03.491380 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51d08ef9-fqnzh" event={"ID":"e0814967-8fa3-4141-8fd2-3b4c3ff630b7","Type":"ContainerDied","Data":"bdc8427a1673f544911c23bb6f2ae6b4153a78963c18c1ce417d3e93ae48c65e"} Oct 31 04:38:04.745623 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:04.745587 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-691c771f-44gp5" Oct 31 04:38:04.791176 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:04.791135 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db80704e-ed92-4de7-a6fb-d1189af34b16-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db80704e-ed92-4de7-a6fb-d1189af34b16-maas-api-openshift-ca\") pod \"db80704e-ed92-4de7-a6fb-d1189af34b16\" (UID: \"db80704e-ed92-4de7-a6fb-d1189af34b16\") " Oct 31 04:38:04.801827 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:04.801689 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db80704e-ed92-4de7-a6fb-d1189af34b16-maas-api-openshift-ca" pod "db80704e-ed92-4de7-a6fb-d1189af34b16" (UID: "db80704e-ed92-4de7-a6fb-d1189af34b16"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:38:04.803543 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:04.802109 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db80704e-ed92-4de7-a6fb-d1189af34b16-kube-api-access-cj8dc\" (UniqueName: \"kubernetes.io/projected/db80704e-ed92-4de7-a6fb-d1189af34b16-kube-api-access-cj8dc\") pod \"db80704e-ed92-4de7-a6fb-d1189af34b16\" (UID: \"db80704e-ed92-4de7-a6fb-d1189af34b16\") " Oct 31 04:38:04.803543 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:04.802195 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db80704e-ed92-4de7-a6fb-d1189af34b16-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db80704e-ed92-4de7-a6fb-d1189af34b16-maas-api-xks-ca\") pod \"db80704e-ed92-4de7-a6fb-d1189af34b16\" (UID: \"db80704e-ed92-4de7-a6fb-d1189af34b16\") " Oct 31 04:38:04.803543 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:04.803174 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db80704e-ed92-4de7-a6fb-d1189af34b16-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:38:04.806436 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:04.806148 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db80704e-ed92-4de7-a6fb-d1189af34b16-maas-api-xks-ca" pod "db80704e-ed92-4de7-a6fb-d1189af34b16" (UID: "db80704e-ed92-4de7-a6fb-d1189af34b16"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:38:04.809620 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:04.809226 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d411b5-f78zv" Oct 31 04:38:04.816555 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:04.816523 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db80704e-ed92-4de7-a6fb-d1189af34b16-kube-api-access-cj8dc" pod "db80704e-ed92-4de7-a6fb-d1189af34b16" (UID: "db80704e-ed92-4de7-a6fb-d1189af34b16"). InnerVolumeSpecName "kube-api-access-cj8dc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:38:04.832233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:04.832188 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51d08ef9-fqnzh" Oct 31 04:38:04.908921 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:04.908881 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dd71c11-2201-4923-8ba1-d6b14db03087-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dd71c11-2201-4923-8ba1-d6b14db03087-maas-api-openshift-ca\") pod \"4dd71c11-2201-4923-8ba1-d6b14db03087\" (UID: \"4dd71c11-2201-4923-8ba1-d6b14db03087\") " Oct 31 04:38:04.909132 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:04.909050 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0814967-8fa3-4141-8fd2-3b4c3ff630b7-kube-api-access-rfb8t\" (UniqueName: \"kubernetes.io/projected/e0814967-8fa3-4141-8fd2-3b4c3ff630b7-kube-api-access-rfb8t\") pod \"e0814967-8fa3-4141-8fd2-3b4c3ff630b7\" (UID: \"e0814967-8fa3-4141-8fd2-3b4c3ff630b7\") " Oct 31 04:38:04.909132 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:04.909114 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4dd71c11-2201-4923-8ba1-d6b14db03087-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dd71c11-2201-4923-8ba1-d6b14db03087-maas-api-xks-ca\") pod \"4dd71c11-2201-4923-8ba1-d6b14db03087\" (UID: \"4dd71c11-2201-4923-8ba1-d6b14db03087\") " Oct 31 04:38:04.909251 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:04.909213 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0814967-8fa3-4141-8fd2-3b4c3ff630b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0814967-8fa3-4141-8fd2-3b4c3ff630b7-maas-api-xks-ca\") pod \"e0814967-8fa3-4141-8fd2-3b4c3ff630b7\" (UID: \"e0814967-8fa3-4141-8fd2-3b4c3ff630b7\") " Oct 31 04:38:04.909413 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:04.909391 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0814967-8fa3-4141-8fd2-3b4c3ff630b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0814967-8fa3-4141-8fd2-3b4c3ff630b7-maas-api-openshift-ca\") pod \"e0814967-8fa3-4141-8fd2-3b4c3ff630b7\" (UID: \"e0814967-8fa3-4141-8fd2-3b4c3ff630b7\") " Oct 31 04:38:04.909476 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:04.909463 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dd71c11-2201-4923-8ba1-d6b14db03087-kube-api-access-gf7d8\" (UniqueName: \"kubernetes.io/projected/4dd71c11-2201-4923-8ba1-d6b14db03087-kube-api-access-gf7d8\") pod \"4dd71c11-2201-4923-8ba1-d6b14db03087\" (UID: \"4dd71c11-2201-4923-8ba1-d6b14db03087\") " Oct 31 04:38:04.910294 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:04.910270 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj8dc\" (UniqueName: \"kubernetes.io/projected/db80704e-ed92-4de7-a6fb-d1189af34b16-kube-api-access-cj8dc\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:38:04.910383 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:04.910303 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db80704e-ed92-4de7-a6fb-d1189af34b16-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:38:04.911774 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:04.911727 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dd71c11-2201-4923-8ba1-d6b14db03087-maas-api-openshift-ca" pod "4dd71c11-2201-4923-8ba1-d6b14db03087" (UID: "4dd71c11-2201-4923-8ba1-d6b14db03087"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:38:04.912080 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:04.912048 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0814967-8fa3-4141-8fd2-3b4c3ff630b7-maas-api-openshift-ca" pod "e0814967-8fa3-4141-8fd2-3b4c3ff630b7" (UID: "e0814967-8fa3-4141-8fd2-3b4c3ff630b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:38:04.913033 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:04.913007 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dd71c11-2201-4923-8ba1-d6b14db03087-maas-api-xks-ca" pod "4dd71c11-2201-4923-8ba1-d6b14db03087" (UID: "4dd71c11-2201-4923-8ba1-d6b14db03087"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:38:04.913433 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:04.913404 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dd71c11-2201-4923-8ba1-d6b14db03087-kube-api-access-gf7d8" pod "4dd71c11-2201-4923-8ba1-d6b14db03087" (UID: "4dd71c11-2201-4923-8ba1-d6b14db03087"). InnerVolumeSpecName "kube-api-access-gf7d8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:38:04.914145 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:04.914122 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0814967-8fa3-4141-8fd2-3b4c3ff630b7-maas-api-xks-ca" pod "e0814967-8fa3-4141-8fd2-3b4c3ff630b7" (UID: "e0814967-8fa3-4141-8fd2-3b4c3ff630b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:38:04.917983 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:04.917957 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0814967-8fa3-4141-8fd2-3b4c3ff630b7-kube-api-access-rfb8t" pod "e0814967-8fa3-4141-8fd2-3b4c3ff630b7" (UID: "e0814967-8fa3-4141-8fd2-3b4c3ff630b7"). InnerVolumeSpecName "kube-api-access-rfb8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:38:05.012164 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:05.012066 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfb8t\" (UniqueName: \"kubernetes.io/projected/e0814967-8fa3-4141-8fd2-3b4c3ff630b7-kube-api-access-rfb8t\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:38:05.012164 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:05.012109 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dd71c11-2201-4923-8ba1-d6b14db03087-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:38:05.012164 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:05.012126 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0814967-8fa3-4141-8fd2-3b4c3ff630b7-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:38:05.012164 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:05.012139 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0814967-8fa3-4141-8fd2-3b4c3ff630b7-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:38:05.012164 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:05.012153 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf7d8\" (UniqueName: \"kubernetes.io/projected/4dd71c11-2201-4923-8ba1-d6b14db03087-kube-api-access-gf7d8\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:38:05.012164 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:05.012169 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dd71c11-2201-4923-8ba1-d6b14db03087-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:38:05.529419 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:05.529386 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-691c771f-44gp5" Oct 31 04:38:05.529670 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:05.529419 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-691c771f-44gp5" event={"ID":"db80704e-ed92-4de7-a6fb-d1189af34b16","Type":"ContainerDied","Data":"08c28448ac1957ad1640d77646e92b6a28c0b752d0d85560b8065ccbdf2cbfa2"} Oct 31 04:38:05.529670 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:05.529472 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08c28448ac1957ad1640d77646e92b6a28c0b752d0d85560b8065ccbdf2cbfa2" Oct 31 04:38:05.530928 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:05.530907 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d411b5-f78zv" Oct 31 04:38:05.531078 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:05.530939 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d411b5-f78zv" event={"ID":"4dd71c11-2201-4923-8ba1-d6b14db03087","Type":"ContainerDied","Data":"0ce4117410b952e375820772391ea256633513dc5c33124072daf60e0d55f489"} Oct 31 04:38:05.531078 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:05.530988 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ce4117410b952e375820772391ea256633513dc5c33124072daf60e0d55f489" Oct 31 04:38:05.532600 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:05.532580 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51d08ef9-fqnzh" Oct 31 04:38:05.532723 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:05.532581 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51d08ef9-fqnzh" event={"ID":"e0814967-8fa3-4141-8fd2-3b4c3ff630b7","Type":"ContainerDied","Data":"f8f34fbe8c40f32876236d1bcd41e043e27e4f0a6d3d067bbd8f641633f36272"} Oct 31 04:38:05.532723 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:05.532696 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8f34fbe8c40f32876236d1bcd41e043e27e4f0a6d3d067bbd8f641633f36272" Oct 31 04:38:07.041154 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:07.041118 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70075897-46qrk"] Oct 31 04:38:07.045969 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:07.045939 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70075897-46qrk"] Oct 31 04:38:07.418228 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:07.418135 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99d9c372-faba-46f3-9229-a20a40177209" path="/var/lib/kubelet/pods/99d9c372-faba-46f3-9229-a20a40177209/volumes" Oct 31 04:38:09.460344 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:09.460303 5019 scope.go:122] "RemoveContainer" containerID="d55cace45d7e120bb4087a067f3dcf6d4ebed6cc1ff277845640a2c67520c19f" Oct 31 04:38:09.476645 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:09.476614 5019 scope.go:122] "RemoveContainer" containerID="a951b2503c42e6c502fdb4fea75fb4f5b37c1a3d7e5997c9c79f6bfbb496f6fc" Oct 31 04:38:09.488000 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:09.487977 5019 scope.go:122] "RemoveContainer" containerID="6c1572ef8da7e96145bca7460707d576f3bb5d5a7d8f856ff34acc06b6321ec0" Oct 31 04:38:09.500869 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:09.500840 5019 scope.go:122] "RemoveContainer" containerID="2a5379848e94cf984747f3d18aeaa5990a6186e519c125b63c7256f4b87726ac" Oct 31 04:38:09.512647 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:09.512619 5019 scope.go:122] "RemoveContainer" containerID="0e87eb6578ed42f510ce260bc026853df54829d892a91be664777009bfc0599f" Oct 31 04:38:09.525495 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:09.525472 5019 scope.go:122] "RemoveContainer" containerID="0683db2c97930678af2fd272c0b753a5b83a8111f342bd7b1dff8d4d27a49d03" Oct 31 04:38:09.535767 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:09.535745 5019 scope.go:122] "RemoveContainer" containerID="22dc606b9ac4e9b48264ff634e5565db4d8c614dcf75bfb02ca2a6aa3bd366d8" Oct 31 04:38:09.546711 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:09.546643 5019 scope.go:122] "RemoveContainer" containerID="c756020a4ac7c027dd90aea145781e56963fd24c14de885e0490798d6553e27b" Oct 31 04:38:09.557487 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:09.557465 5019 scope.go:122] "RemoveContainer" containerID="92f839d89e8307063c82a3778a3792cf73070a8b574d42cf0804e4c4449404f6" Oct 31 04:38:09.570144 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:09.570120 5019 scope.go:122] "RemoveContainer" containerID="5a858458189206a917c8c333f4ffed762f4cb44fc15f2164c02c051ad0404c69" Oct 31 04:38:11.331128 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:11.331091 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9937f7d0-l66sk"] Oct 31 04:38:11.331879 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:11.331862 5019 cpu_manager.go:414] "containerMap: removing container" podUID="e0814967-8fa3-4141-8fd2-3b4c3ff630b7" containerName="revoke-keys" Oct 31 04:38:11.331935 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:11.331881 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0814967-8fa3-4141-8fd2-3b4c3ff630b7" containerName="revoke-keys" Oct 31 04:38:11.331935 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:11.331902 5019 cpu_manager.go:414] "containerMap: removing container" podUID="4dd71c11-2201-4923-8ba1-d6b14db03087" containerName="revoke-keys" Oct 31 04:38:11.331935 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:11.331907 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dd71c11-2201-4923-8ba1-d6b14db03087" containerName="revoke-keys" Oct 31 04:38:11.331935 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:11.331929 5019 cpu_manager.go:414] "containerMap: removing container" podUID="db80704e-ed92-4de7-a6fb-d1189af34b16" containerName="revoke-keys" Oct 31 04:38:11.331935 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:11.331934 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db80704e-ed92-4de7-a6fb-d1189af34b16" containerName="revoke-keys" Oct 31 04:38:11.332101 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:11.332068 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dd71c11-2201-4923-8ba1-d6b14db03087" containerName="revoke-keys" Oct 31 04:38:11.332101 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:11.332080 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="db80704e-ed92-4de7-a6fb-d1189af34b16" containerName="revoke-keys" Oct 31 04:38:11.332101 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:11.332094 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0814967-8fa3-4141-8fd2-3b4c3ff630b7" containerName="revoke-keys" Oct 31 04:38:11.338824 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:11.338799 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9937f7d0-l66sk" Oct 31 04:38:11.354726 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:11.354691 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9937f7d0-l66sk"] Oct 31 04:38:11.361476 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:11.361372 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:38:11.361667 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:11.361521 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:38:11.361667 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:11.361520 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:38:11.361667 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:11.361523 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:38:11.383352 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:11.383300 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8aaf1cba-a101-45a1-b87b-afe4e7cf00ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9937f7d0-l66sk\" (UID: \"8aaf1cba-a101-45a1-b87b-afe4e7cf00ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9937f7d0-l66sk" Oct 31 04:38:11.383607 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:11.383496 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8aaf1cba-a101-45a1-b87b-afe4e7cf00ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9937f7d0-l66sk\" (UID: \"8aaf1cba-a101-45a1-b87b-afe4e7cf00ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9937f7d0-l66sk" Oct 31 04:38:11.383764 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:11.383741 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbvxd\" (UniqueName: \"kubernetes.io/projected/8aaf1cba-a101-45a1-b87b-afe4e7cf00ab-kube-api-access-tbvxd\") pod \"maas-api-revoke-sub-9937f7d0-l66sk\" (UID: \"8aaf1cba-a101-45a1-b87b-afe4e7cf00ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9937f7d0-l66sk" Oct 31 04:38:11.485345 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:11.485293 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbvxd\" (UniqueName: \"kubernetes.io/projected/8aaf1cba-a101-45a1-b87b-afe4e7cf00ab-kube-api-access-tbvxd\") pod \"maas-api-revoke-sub-9937f7d0-l66sk\" (UID: \"8aaf1cba-a101-45a1-b87b-afe4e7cf00ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9937f7d0-l66sk" Oct 31 04:38:11.485543 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:11.485521 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8aaf1cba-a101-45a1-b87b-afe4e7cf00ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9937f7d0-l66sk\" (UID: \"8aaf1cba-a101-45a1-b87b-afe4e7cf00ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9937f7d0-l66sk" Oct 31 04:38:11.485697 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:11.485677 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8aaf1cba-a101-45a1-b87b-afe4e7cf00ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9937f7d0-l66sk\" (UID: \"8aaf1cba-a101-45a1-b87b-afe4e7cf00ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9937f7d0-l66sk" Oct 31 04:38:11.486277 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:11.486248 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8aaf1cba-a101-45a1-b87b-afe4e7cf00ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9937f7d0-l66sk\" (UID: \"8aaf1cba-a101-45a1-b87b-afe4e7cf00ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9937f7d0-l66sk" Oct 31 04:38:11.488113 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:11.488089 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8aaf1cba-a101-45a1-b87b-afe4e7cf00ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9937f7d0-l66sk\" (UID: \"8aaf1cba-a101-45a1-b87b-afe4e7cf00ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9937f7d0-l66sk" Oct 31 04:38:11.508802 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:11.508767 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbvxd\" (UniqueName: \"kubernetes.io/projected/8aaf1cba-a101-45a1-b87b-afe4e7cf00ab-kube-api-access-tbvxd\") pod \"maas-api-revoke-sub-9937f7d0-l66sk\" (UID: \"8aaf1cba-a101-45a1-b87b-afe4e7cf00ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9937f7d0-l66sk" Oct 31 04:38:11.655459 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:11.655364 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9937f7d0-l66sk" Oct 31 04:38:11.674124 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:11.674092 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc"] Oct 31 04:38:11.674320 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:11.674291 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc" podUID="0c071116-bc3c-4d68-9e80-ec887ee96be7" containerName="istio-proxy" containerID="cri-o://134b6545aacd863720d0a6b9fcb2299399d73d2db685948a9ecf3c05102c3e3e" gracePeriod=30 Oct 31 04:38:11.978173 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:38:11.978136 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aaf1cba_a101_45a1_b87b_afe4e7cf00ab.slice/crio-d52fc927c2716772f80799e35ae54eeb27938935e51b2df57c58d4b23b3bc11c WatchSource:0}: Error finding container d52fc927c2716772f80799e35ae54eeb27938935e51b2df57c58d4b23b3bc11c: Status 404 returned error can't find the container with id d52fc927c2716772f80799e35ae54eeb27938935e51b2df57c58d4b23b3bc11c Oct 31 04:38:11.980618 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:11.980583 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9937f7d0-l66sk"] Oct 31 04:38:12.048413 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:12.048360 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf5ed941-8gm7j"] Oct 31 04:38:12.054379 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:12.054316 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf5ed941-8gm7j" Oct 31 04:38:12.074622 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:12.074584 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf5ed941-8gm7j"] Oct 31 04:38:12.100474 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:12.100423 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f521bb4-2d60-445e-9a83-96492311abd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf5ed941-8gm7j\" (UID: \"4f521bb4-2d60-445e-9a83-96492311abd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf5ed941-8gm7j" Oct 31 04:38:12.100674 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:12.100482 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x27h6\" (UniqueName: \"kubernetes.io/projected/4f521bb4-2d60-445e-9a83-96492311abd1-kube-api-access-x27h6\") pod \"maas-api-revoke-sub-bf5ed941-8gm7j\" (UID: \"4f521bb4-2d60-445e-9a83-96492311abd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf5ed941-8gm7j" Oct 31 04:38:12.100674 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:12.100625 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f521bb4-2d60-445e-9a83-96492311abd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf5ed941-8gm7j\" (UID: \"4f521bb4-2d60-445e-9a83-96492311abd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf5ed941-8gm7j" Oct 31 04:38:12.142909 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:12.142859 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b29c188-jdtv2"] Oct 31 04:38:12.147637 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:12.147610 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b29c188-jdtv2" Oct 31 04:38:12.164799 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:12.164772 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b29c188-jdtv2"] Oct 31 04:38:12.203039 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:12.203003 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f521bb4-2d60-445e-9a83-96492311abd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf5ed941-8gm7j\" (UID: \"4f521bb4-2d60-445e-9a83-96492311abd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf5ed941-8gm7j" Oct 31 04:38:12.203243 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:12.203050 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x27h6\" (UniqueName: \"kubernetes.io/projected/4f521bb4-2d60-445e-9a83-96492311abd1-kube-api-access-x27h6\") pod \"maas-api-revoke-sub-bf5ed941-8gm7j\" (UID: \"4f521bb4-2d60-445e-9a83-96492311abd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf5ed941-8gm7j" Oct 31 04:38:12.203243 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:12.203144 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gz68\" (UniqueName: \"kubernetes.io/projected/d25c55c3-f558-495e-b197-e0f5231c53f2-kube-api-access-7gz68\") pod \"maas-api-revoke-sub-0b29c188-jdtv2\" (UID: \"d25c55c3-f558-495e-b197-e0f5231c53f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b29c188-jdtv2" Oct 31 04:38:12.203243 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:12.203177 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d25c55c3-f558-495e-b197-e0f5231c53f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b29c188-jdtv2\" (UID: \"d25c55c3-f558-495e-b197-e0f5231c53f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b29c188-jdtv2" Oct 31 04:38:12.203495 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:12.203410 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f521bb4-2d60-445e-9a83-96492311abd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf5ed941-8gm7j\" (UID: \"4f521bb4-2d60-445e-9a83-96492311abd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf5ed941-8gm7j" Oct 31 04:38:12.204203 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:12.204172 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f521bb4-2d60-445e-9a83-96492311abd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf5ed941-8gm7j\" (UID: \"4f521bb4-2d60-445e-9a83-96492311abd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf5ed941-8gm7j" Oct 31 04:38:12.204482 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:12.204459 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d25c55c3-f558-495e-b197-e0f5231c53f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b29c188-jdtv2\" (UID: \"d25c55c3-f558-495e-b197-e0f5231c53f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b29c188-jdtv2" Oct 31 04:38:12.205802 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:12.205775 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f521bb4-2d60-445e-9a83-96492311abd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf5ed941-8gm7j\" (UID: \"4f521bb4-2d60-445e-9a83-96492311abd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf5ed941-8gm7j" Oct 31 04:38:12.226424 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:12.226350 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x27h6\" (UniqueName: \"kubernetes.io/projected/4f521bb4-2d60-445e-9a83-96492311abd1-kube-api-access-x27h6\") pod \"maas-api-revoke-sub-bf5ed941-8gm7j\" (UID: \"4f521bb4-2d60-445e-9a83-96492311abd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf5ed941-8gm7j" Oct 31 04:38:12.306510 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:12.306436 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gz68\" (UniqueName: \"kubernetes.io/projected/d25c55c3-f558-495e-b197-e0f5231c53f2-kube-api-access-7gz68\") pod \"maas-api-revoke-sub-0b29c188-jdtv2\" (UID: \"d25c55c3-f558-495e-b197-e0f5231c53f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b29c188-jdtv2" Oct 31 04:38:12.306707 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:12.306526 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d25c55c3-f558-495e-b197-e0f5231c53f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b29c188-jdtv2\" (UID: \"d25c55c3-f558-495e-b197-e0f5231c53f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b29c188-jdtv2" Oct 31 04:38:12.307392 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:12.307361 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d25c55c3-f558-495e-b197-e0f5231c53f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b29c188-jdtv2\" (UID: \"d25c55c3-f558-495e-b197-e0f5231c53f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b29c188-jdtv2" Oct 31 04:38:12.309481 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:12.309451 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d25c55c3-f558-495e-b197-e0f5231c53f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b29c188-jdtv2\" (UID: \"d25c55c3-f558-495e-b197-e0f5231c53f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b29c188-jdtv2" Oct 31 04:38:12.311441 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:12.311414 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d25c55c3-f558-495e-b197-e0f5231c53f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b29c188-jdtv2\" (UID: \"d25c55c3-f558-495e-b197-e0f5231c53f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b29c188-jdtv2" Oct 31 04:38:12.332948 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:12.332913 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gz68\" (UniqueName: \"kubernetes.io/projected/d25c55c3-f558-495e-b197-e0f5231c53f2-kube-api-access-7gz68\") pod \"maas-api-revoke-sub-0b29c188-jdtv2\" (UID: \"d25c55c3-f558-495e-b197-e0f5231c53f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b29c188-jdtv2" Oct 31 04:38:12.371501 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:12.371460 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf5ed941-8gm7j" Oct 31 04:38:12.463654 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:12.463616 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b29c188-jdtv2" Oct 31 04:38:12.607293 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:12.606406 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9937f7d0-l66sk" event={"ID":"8aaf1cba-a101-45a1-b87b-afe4e7cf00ab","Type":"ContainerStarted","Data":"1a8dc0e789b7ae0b0b196ab2c423d802c390013bad5ff3ec106cda10ea274256"} Oct 31 04:38:12.607293 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:12.606478 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9937f7d0-l66sk" event={"ID":"8aaf1cba-a101-45a1-b87b-afe4e7cf00ab","Type":"ContainerStarted","Data":"d52fc927c2716772f80799e35ae54eeb27938935e51b2df57c58d4b23b3bc11c"} Oct 31 04:38:12.650367 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:12.650298 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9937f7d0-l66sk" podStartSLOduration=1.6502836749999998 podStartE2EDuration="1.650283675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:38:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:38:12.648766949 +0000 UTC m=+2047.405104990" watchObservedRunningTime="2026-10-31 04:38:12.650283675 +0000 UTC m=+2047.406621715" Oct 31 04:38:12.721910 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:38:12.721871 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f521bb4_2d60_445e_9a83_96492311abd1.slice/crio-0466449c9bb8abb1f4be56066703ac3bac6b978e81352ba74b25a814736a9d17 WatchSource:0}: Error finding container 0466449c9bb8abb1f4be56066703ac3bac6b978e81352ba74b25a814736a9d17: Status 404 returned error can't find the container with id 0466449c9bb8abb1f4be56066703ac3bac6b978e81352ba74b25a814736a9d17 Oct 31 04:38:12.722613 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:12.722582 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf5ed941-8gm7j"] Oct 31 04:38:12.818171 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:12.818129 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b29c188-jdtv2"] Oct 31 04:38:12.824811 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:38:12.824778 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd25c55c3_f558_495e_b197_e0f5231c53f2.slice/crio-3ff62256dca661cecfb3c210ad6e5fc43407131f696ff7199f2d329c99e72037 WatchSource:0}: Error finding container 3ff62256dca661cecfb3c210ad6e5fc43407131f696ff7199f2d329c99e72037: Status 404 returned error can't find the container with id 3ff62256dca661cecfb3c210ad6e5fc43407131f696ff7199f2d329c99e72037 Oct 31 04:38:13.617505 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:13.617466 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf5ed941-8gm7j" event={"ID":"4f521bb4-2d60-445e-9a83-96492311abd1","Type":"ContainerStarted","Data":"b919fff27e7be0d2985b6d05fd01e36182cc793d60ecefbc3e7fbbe4345ee008"} Oct 31 04:38:13.617979 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:13.617533 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf5ed941-8gm7j" event={"ID":"4f521bb4-2d60-445e-9a83-96492311abd1","Type":"ContainerStarted","Data":"0466449c9bb8abb1f4be56066703ac3bac6b978e81352ba74b25a814736a9d17"} Oct 31 04:38:13.619182 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:13.619150 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b29c188-jdtv2" event={"ID":"d25c55c3-f558-495e-b197-e0f5231c53f2","Type":"ContainerStarted","Data":"feae27ab3c10cdda177858c606935bbd8c9294a78b529421394049c0504d3569"} Oct 31 04:38:13.619347 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:13.619209 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b29c188-jdtv2" event={"ID":"d25c55c3-f558-495e-b197-e0f5231c53f2","Type":"ContainerStarted","Data":"3ff62256dca661cecfb3c210ad6e5fc43407131f696ff7199f2d329c99e72037"} Oct 31 04:38:13.620846 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:13.620819 5019 generic.go:356] "Generic (PLEG): container finished" podID="8aaf1cba-a101-45a1-b87b-afe4e7cf00ab" containerID="1a8dc0e789b7ae0b0b196ab2c423d802c390013bad5ff3ec106cda10ea274256" exitCode=0 Oct 31 04:38:13.620949 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:13.620902 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9937f7d0-l66sk" event={"ID":"8aaf1cba-a101-45a1-b87b-afe4e7cf00ab","Type":"ContainerDied","Data":"1a8dc0e789b7ae0b0b196ab2c423d802c390013bad5ff3ec106cda10ea274256"} Oct 31 04:38:13.658983 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:13.658917 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf5ed941-8gm7j" podStartSLOduration=1.6589015740000002 podStartE2EDuration="1.658901574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:38:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:38:13.657756574 +0000 UTC m=+2048.414094617" watchObservedRunningTime="2026-10-31 04:38:13.658901574 +0000 UTC m=+2048.415239616" Oct 31 04:38:13.698281 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:13.698233 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b29c188-jdtv2" podStartSLOduration=1.698217266 podStartE2EDuration="1.698217266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:38:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:38:13.698042884 +0000 UTC m=+2048.454380926" watchObservedRunningTime="2026-10-31 04:38:13.698217266 +0000 UTC m=+2048.454555307" Oct 31 04:38:14.633097 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:14.633060 5019 generic.go:356] "Generic (PLEG): container finished" podID="4f521bb4-2d60-445e-9a83-96492311abd1" containerID="b919fff27e7be0d2985b6d05fd01e36182cc793d60ecefbc3e7fbbe4345ee008" exitCode=0 Oct 31 04:38:14.633588 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:14.633152 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf5ed941-8gm7j" event={"ID":"4f521bb4-2d60-445e-9a83-96492311abd1","Type":"ContainerDied","Data":"b919fff27e7be0d2985b6d05fd01e36182cc793d60ecefbc3e7fbbe4345ee008"} Oct 31 04:38:14.634934 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:14.634904 5019 generic.go:356] "Generic (PLEG): container finished" podID="d25c55c3-f558-495e-b197-e0f5231c53f2" containerID="feae27ab3c10cdda177858c606935bbd8c9294a78b529421394049c0504d3569" exitCode=0 Oct 31 04:38:14.635074 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:14.634931 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b29c188-jdtv2" event={"ID":"d25c55c3-f558-495e-b197-e0f5231c53f2","Type":"ContainerDied","Data":"feae27ab3c10cdda177858c606935bbd8c9294a78b529421394049c0504d3569"} Oct 31 04:38:14.871572 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:14.871546 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9937f7d0-l66sk" Oct 31 04:38:14.898886 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:14.898805 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8aaf1cba-a101-45a1-b87b-afe4e7cf00ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8aaf1cba-a101-45a1-b87b-afe4e7cf00ab-maas-api-openshift-ca\") pod \"8aaf1cba-a101-45a1-b87b-afe4e7cf00ab\" (UID: \"8aaf1cba-a101-45a1-b87b-afe4e7cf00ab\") " Oct 31 04:38:14.899031 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:14.898989 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aaf1cba-a101-45a1-b87b-afe4e7cf00ab-kube-api-access-tbvxd\" (UniqueName: \"kubernetes.io/projected/8aaf1cba-a101-45a1-b87b-afe4e7cf00ab-kube-api-access-tbvxd\") pod \"8aaf1cba-a101-45a1-b87b-afe4e7cf00ab\" (UID: \"8aaf1cba-a101-45a1-b87b-afe4e7cf00ab\") " Oct 31 04:38:14.899254 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:14.899225 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aaf1cba-a101-45a1-b87b-afe4e7cf00ab-maas-api-openshift-ca" pod "8aaf1cba-a101-45a1-b87b-afe4e7cf00ab" (UID: "8aaf1cba-a101-45a1-b87b-afe4e7cf00ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:38:14.899604 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:14.899583 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8aaf1cba-a101-45a1-b87b-afe4e7cf00ab-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:38:14.901249 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:14.901226 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aaf1cba-a101-45a1-b87b-afe4e7cf00ab-kube-api-access-tbvxd" pod "8aaf1cba-a101-45a1-b87b-afe4e7cf00ab" (UID: "8aaf1cba-a101-45a1-b87b-afe4e7cf00ab"). InnerVolumeSpecName "kube-api-access-tbvxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:38:15.000470 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:15.000436 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8aaf1cba-a101-45a1-b87b-afe4e7cf00ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8aaf1cba-a101-45a1-b87b-afe4e7cf00ab-maas-api-xks-ca\") pod \"8aaf1cba-a101-45a1-b87b-afe4e7cf00ab\" (UID: \"8aaf1cba-a101-45a1-b87b-afe4e7cf00ab\") " Oct 31 04:38:15.001015 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:15.000994 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbvxd\" (UniqueName: \"kubernetes.io/projected/8aaf1cba-a101-45a1-b87b-afe4e7cf00ab-kube-api-access-tbvxd\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:38:15.002586 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:15.002559 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aaf1cba-a101-45a1-b87b-afe4e7cf00ab-maas-api-xks-ca" pod "8aaf1cba-a101-45a1-b87b-afe4e7cf00ab" (UID: "8aaf1cba-a101-45a1-b87b-afe4e7cf00ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:38:15.102814 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:15.102781 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8aaf1cba-a101-45a1-b87b-afe4e7cf00ab-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:38:15.647492 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:15.647457 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9937f7d0-l66sk" event={"ID":"8aaf1cba-a101-45a1-b87b-afe4e7cf00ab","Type":"ContainerDied","Data":"d52fc927c2716772f80799e35ae54eeb27938935e51b2df57c58d4b23b3bc11c"} Oct 31 04:38:15.647990 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:15.647499 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9937f7d0-l66sk" Oct 31 04:38:15.647990 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:15.647532 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d52fc927c2716772f80799e35ae54eeb27938935e51b2df57c58d4b23b3bc11c" Oct 31 04:38:15.891318 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:15.891281 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b29c188-jdtv2" Oct 31 04:38:15.894954 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:15.894923 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf5ed941-8gm7j" Oct 31 04:38:15.916472 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:15.916414 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f521bb4-2d60-445e-9a83-96492311abd1-kube-api-access-x27h6\" (UniqueName: \"kubernetes.io/projected/4f521bb4-2d60-445e-9a83-96492311abd1-kube-api-access-x27h6\") pod \"4f521bb4-2d60-445e-9a83-96492311abd1\" (UID: \"4f521bb4-2d60-445e-9a83-96492311abd1\") " Oct 31 04:38:15.916691 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:15.916565 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d25c55c3-f558-495e-b197-e0f5231c53f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d25c55c3-f558-495e-b197-e0f5231c53f2-maas-api-xks-ca\") pod \"d25c55c3-f558-495e-b197-e0f5231c53f2\" (UID: \"d25c55c3-f558-495e-b197-e0f5231c53f2\") " Oct 31 04:38:15.916781 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:15.916764 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f521bb4-2d60-445e-9a83-96492311abd1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f521bb4-2d60-445e-9a83-96492311abd1-maas-api-xks-ca\") pod \"4f521bb4-2d60-445e-9a83-96492311abd1\" (UID: \"4f521bb4-2d60-445e-9a83-96492311abd1\") " Oct 31 04:38:15.916902 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:15.916869 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d25c55c3-f558-495e-b197-e0f5231c53f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d25c55c3-f558-495e-b197-e0f5231c53f2-maas-api-openshift-ca\") pod \"d25c55c3-f558-495e-b197-e0f5231c53f2\" (UID: \"d25c55c3-f558-495e-b197-e0f5231c53f2\") " Oct 31 04:38:15.917068 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:15.917051 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f521bb4-2d60-445e-9a83-96492311abd1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f521bb4-2d60-445e-9a83-96492311abd1-maas-api-openshift-ca\") pod \"4f521bb4-2d60-445e-9a83-96492311abd1\" (UID: \"4f521bb4-2d60-445e-9a83-96492311abd1\") " Oct 31 04:38:15.917129 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:15.917087 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d25c55c3-f558-495e-b197-e0f5231c53f2-kube-api-access-7gz68\" (UniqueName: \"kubernetes.io/projected/d25c55c3-f558-495e-b197-e0f5231c53f2-kube-api-access-7gz68\") pod \"d25c55c3-f558-495e-b197-e0f5231c53f2\" (UID: \"d25c55c3-f558-495e-b197-e0f5231c53f2\") " Oct 31 04:38:15.917290 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:15.917264 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d25c55c3-f558-495e-b197-e0f5231c53f2-maas-api-openshift-ca" pod "d25c55c3-f558-495e-b197-e0f5231c53f2" (UID: "d25c55c3-f558-495e-b197-e0f5231c53f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:38:15.917687 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:15.917662 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f521bb4-2d60-445e-9a83-96492311abd1-maas-api-openshift-ca" pod "4f521bb4-2d60-445e-9a83-96492311abd1" (UID: "4f521bb4-2d60-445e-9a83-96492311abd1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:38:15.917872 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:15.917848 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d25c55c3-f558-495e-b197-e0f5231c53f2-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:38:15.917872 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:15.917871 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f521bb4-2d60-445e-9a83-96492311abd1-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:38:15.919411 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:15.919299 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f521bb4-2d60-445e-9a83-96492311abd1-maas-api-xks-ca" pod "4f521bb4-2d60-445e-9a83-96492311abd1" (UID: "4f521bb4-2d60-445e-9a83-96492311abd1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:38:15.919411 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:15.919376 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f521bb4-2d60-445e-9a83-96492311abd1-kube-api-access-x27h6" pod "4f521bb4-2d60-445e-9a83-96492311abd1" (UID: "4f521bb4-2d60-445e-9a83-96492311abd1"). InnerVolumeSpecName "kube-api-access-x27h6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:38:15.919966 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:15.919940 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d25c55c3-f558-495e-b197-e0f5231c53f2-maas-api-xks-ca" pod "d25c55c3-f558-495e-b197-e0f5231c53f2" (UID: "d25c55c3-f558-495e-b197-e0f5231c53f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:38:15.920074 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:15.920052 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d25c55c3-f558-495e-b197-e0f5231c53f2-kube-api-access-7gz68" pod "d25c55c3-f558-495e-b197-e0f5231c53f2" (UID: "d25c55c3-f558-495e-b197-e0f5231c53f2"). InnerVolumeSpecName "kube-api-access-7gz68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:38:16.019230 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:16.019141 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d25c55c3-f558-495e-b197-e0f5231c53f2-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:38:16.019230 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:16.019181 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f521bb4-2d60-445e-9a83-96492311abd1-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:38:16.019230 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:16.019192 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gz68\" (UniqueName: \"kubernetes.io/projected/d25c55c3-f558-495e-b197-e0f5231c53f2-kube-api-access-7gz68\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:38:16.019230 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:16.019205 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x27h6\" (UniqueName: \"kubernetes.io/projected/4f521bb4-2d60-445e-9a83-96492311abd1-kube-api-access-x27h6\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:38:16.656941 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:16.656908 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf5ed941-8gm7j" Oct 31 04:38:16.657454 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:16.656942 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf5ed941-8gm7j" event={"ID":"4f521bb4-2d60-445e-9a83-96492311abd1","Type":"ContainerDied","Data":"0466449c9bb8abb1f4be56066703ac3bac6b978e81352ba74b25a814736a9d17"} Oct 31 04:38:16.657454 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:16.656992 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0466449c9bb8abb1f4be56066703ac3bac6b978e81352ba74b25a814736a9d17" Oct 31 04:38:16.658649 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:16.658628 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b29c188-jdtv2" Oct 31 04:38:16.658649 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:16.658634 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b29c188-jdtv2" event={"ID":"d25c55c3-f558-495e-b197-e0f5231c53f2","Type":"ContainerDied","Data":"3ff62256dca661cecfb3c210ad6e5fc43407131f696ff7199f2d329c99e72037"} Oct 31 04:38:16.658861 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:16.658666 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ff62256dca661cecfb3c210ad6e5fc43407131f696ff7199f2d329c99e72037" Oct 31 04:38:16.982552 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:16.982524 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc" Oct 31 04:38:17.038950 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:17.038913 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c071116-bc3c-4d68-9e80-ec887ee96be7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c071116-bc3c-4d68-9e80-ec887ee96be7-istio-envoy\") pod \"0c071116-bc3c-4d68-9e80-ec887ee96be7\" (UID: \"0c071116-bc3c-4d68-9e80-ec887ee96be7\") " Oct 31 04:38:17.039112 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:17.038989 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c071116-bc3c-4d68-9e80-ec887ee96be7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c071116-bc3c-4d68-9e80-ec887ee96be7-workload-certs\") pod \"0c071116-bc3c-4d68-9e80-ec887ee96be7\" (UID: \"0c071116-bc3c-4d68-9e80-ec887ee96be7\") " Oct 31 04:38:17.039112 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:17.039092 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c071116-bc3c-4d68-9e80-ec887ee96be7-kube-api-access-bkdhh\" (UniqueName: \"kubernetes.io/projected/0c071116-bc3c-4d68-9e80-ec887ee96be7-kube-api-access-bkdhh\") pod \"0c071116-bc3c-4d68-9e80-ec887ee96be7\" (UID: \"0c071116-bc3c-4d68-9e80-ec887ee96be7\") " Oct 31 04:38:17.039241 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:17.039136 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c071116-bc3c-4d68-9e80-ec887ee96be7-istio-token\" (UniqueName: \"kubernetes.io/projected/0c071116-bc3c-4d68-9e80-ec887ee96be7-istio-token\") pod \"0c071116-bc3c-4d68-9e80-ec887ee96be7\" (UID: \"0c071116-bc3c-4d68-9e80-ec887ee96be7\") " Oct 31 04:38:17.039241 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:17.039229 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c071116-bc3c-4d68-9e80-ec887ee96be7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c071116-bc3c-4d68-9e80-ec887ee96be7-istio-podinfo\") pod \"0c071116-bc3c-4d68-9e80-ec887ee96be7\" (UID: \"0c071116-bc3c-4d68-9e80-ec887ee96be7\") " Oct 31 04:38:17.039407 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:17.039274 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c071116-bc3c-4d68-9e80-ec887ee96be7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c071116-bc3c-4d68-9e80-ec887ee96be7-workload-socket\") pod \"0c071116-bc3c-4d68-9e80-ec887ee96be7\" (UID: \"0c071116-bc3c-4d68-9e80-ec887ee96be7\") " Oct 31 04:38:17.039829 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:17.039497 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c071116-bc3c-4d68-9e80-ec887ee96be7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c071116-bc3c-4d68-9e80-ec887ee96be7-istiod-ca-cert\") pod \"0c071116-bc3c-4d68-9e80-ec887ee96be7\" (UID: \"0c071116-bc3c-4d68-9e80-ec887ee96be7\") " Oct 31 04:38:17.039829 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:17.039580 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c071116-bc3c-4d68-9e80-ec887ee96be7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c071116-bc3c-4d68-9e80-ec887ee96be7-istio-data\") pod \"0c071116-bc3c-4d68-9e80-ec887ee96be7\" (UID: \"0c071116-bc3c-4d68-9e80-ec887ee96be7\") " Oct 31 04:38:17.039829 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:17.039635 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c071116-bc3c-4d68-9e80-ec887ee96be7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c071116-bc3c-4d68-9e80-ec887ee96be7-credential-socket\") pod \"0c071116-bc3c-4d68-9e80-ec887ee96be7\" (UID: \"0c071116-bc3c-4d68-9e80-ec887ee96be7\") " Oct 31 04:38:17.040404 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:17.040378 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c071116-bc3c-4d68-9e80-ec887ee96be7-workload-socket" pod "0c071116-bc3c-4d68-9e80-ec887ee96be7" (UID: "0c071116-bc3c-4d68-9e80-ec887ee96be7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:17.040683 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:17.040626 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c071116-bc3c-4d68-9e80-ec887ee96be7-workload-certs" pod "0c071116-bc3c-4d68-9e80-ec887ee96be7" (UID: "0c071116-bc3c-4d68-9e80-ec887ee96be7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:17.040764 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:17.040696 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c071116-bc3c-4d68-9e80-ec887ee96be7-credential-socket" pod "0c071116-bc3c-4d68-9e80-ec887ee96be7" (UID: "0c071116-bc3c-4d68-9e80-ec887ee96be7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:17.041040 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:17.041018 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c071116-bc3c-4d68-9e80-ec887ee96be7-istiod-ca-cert" pod "0c071116-bc3c-4d68-9e80-ec887ee96be7" (UID: "0c071116-bc3c-4d68-9e80-ec887ee96be7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:38:17.041296 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:17.041273 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c071116-bc3c-4d68-9e80-ec887ee96be7-istio-data" pod "0c071116-bc3c-4d68-9e80-ec887ee96be7" (UID: "0c071116-bc3c-4d68-9e80-ec887ee96be7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:17.043439 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:17.043413 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c071116-bc3c-4d68-9e80-ec887ee96be7-istio-token" pod "0c071116-bc3c-4d68-9e80-ec887ee96be7" (UID: "0c071116-bc3c-4d68-9e80-ec887ee96be7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:38:17.043784 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:17.043746 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c071116-bc3c-4d68-9e80-ec887ee96be7-istio-podinfo" pod "0c071116-bc3c-4d68-9e80-ec887ee96be7" (UID: "0c071116-bc3c-4d68-9e80-ec887ee96be7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:38:17.045690 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:17.045656 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c071116-bc3c-4d68-9e80-ec887ee96be7-kube-api-access-bkdhh" pod "0c071116-bc3c-4d68-9e80-ec887ee96be7" (UID: "0c071116-bc3c-4d68-9e80-ec887ee96be7"). InnerVolumeSpecName "kube-api-access-bkdhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:38:17.047018 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:17.046992 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c071116-bc3c-4d68-9e80-ec887ee96be7-istio-envoy" pod "0c071116-bc3c-4d68-9e80-ec887ee96be7" (UID: "0c071116-bc3c-4d68-9e80-ec887ee96be7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:17.142483 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:17.142443 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c071116-bc3c-4d68-9e80-ec887ee96be7-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:38:17.142483 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:17.142483 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkdhh\" (UniqueName: \"kubernetes.io/projected/0c071116-bc3c-4d68-9e80-ec887ee96be7-kube-api-access-bkdhh\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:38:17.142696 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:17.142498 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c071116-bc3c-4d68-9e80-ec887ee96be7-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:38:17.142696 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:17.142512 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c071116-bc3c-4d68-9e80-ec887ee96be7-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:38:17.142696 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:17.142525 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c071116-bc3c-4d68-9e80-ec887ee96be7-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:38:17.142696 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:17.142538 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c071116-bc3c-4d68-9e80-ec887ee96be7-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:38:17.142696 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:17.142550 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c071116-bc3c-4d68-9e80-ec887ee96be7-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:38:17.142696 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:17.142563 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c071116-bc3c-4d68-9e80-ec887ee96be7-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:38:17.142696 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:17.142575 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c071116-bc3c-4d68-9e80-ec887ee96be7-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:38:17.670879 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:17.670818 5019 generic.go:356] "Generic (PLEG): container finished" podID="0c071116-bc3c-4d68-9e80-ec887ee96be7" containerID="134b6545aacd863720d0a6b9fcb2299399d73d2db685948a9ecf3c05102c3e3e" exitCode=0 Oct 31 04:38:17.671425 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:17.670881 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc" Oct 31 04:38:17.671425 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:17.671023 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc" event={"ID":"0c071116-bc3c-4d68-9e80-ec887ee96be7","Type":"ContainerDied","Data":"134b6545aacd863720d0a6b9fcb2299399d73d2db685948a9ecf3c05102c3e3e"} Oct 31 04:38:17.671425 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:17.671081 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc" event={"ID":"0c071116-bc3c-4d68-9e80-ec887ee96be7","Type":"ContainerDied","Data":"429cb2d026da0878527670dec00aaef2fa89cae98201ecb03767c1c5bcd2e56d"} Oct 31 04:38:17.671425 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:17.671112 5019 scope.go:122] "RemoveContainer" containerID="134b6545aacd863720d0a6b9fcb2299399d73d2db685948a9ecf3c05102c3e3e" Oct 31 04:38:17.686936 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:17.686911 5019 scope.go:122] "RemoveContainer" containerID="134b6545aacd863720d0a6b9fcb2299399d73d2db685948a9ecf3c05102c3e3e" Oct 31 04:38:17.687278 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:38:17.687257 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"134b6545aacd863720d0a6b9fcb2299399d73d2db685948a9ecf3c05102c3e3e\": container with ID starting with 134b6545aacd863720d0a6b9fcb2299399d73d2db685948a9ecf3c05102c3e3e not found: ID does not exist" containerID="134b6545aacd863720d0a6b9fcb2299399d73d2db685948a9ecf3c05102c3e3e" Oct 31 04:38:17.687405 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:17.687289 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"134b6545aacd863720d0a6b9fcb2299399d73d2db685948a9ecf3c05102c3e3e"} err="failed to get container status \"134b6545aacd863720d0a6b9fcb2299399d73d2db685948a9ecf3c05102c3e3e\": rpc error: code = NotFound desc = could not find container \"134b6545aacd863720d0a6b9fcb2299399d73d2db685948a9ecf3c05102c3e3e\": container with ID starting with 134b6545aacd863720d0a6b9fcb2299399d73d2db685948a9ecf3c05102c3e3e not found: ID does not exist" Oct 31 04:38:17.730676 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:17.730641 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc"] Oct 31 04:38:17.735682 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:17.735648 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-bpdkc"] Oct 31 04:38:19.411764 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:19.411723 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c071116-bc3c-4d68-9e80-ec887ee96be7" path="/var/lib/kubelet/pods/0c071116-bc3c-4d68-9e80-ec887ee96be7/volumes" Oct 31 04:38:22.394288 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:38:22.394253 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:38:35.450346 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:35.450289 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-338f2e/auth-test-model-338f2e-kserve-79676f7df-lwb9d"] Oct 31 04:38:36.927567 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:36.927529 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n"] Oct 31 04:38:36.927963 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:36.927728 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n" podUID="88b7c3d5-09b5-4eda-a752-aac0c1f0a11c" containerName="istio-proxy" containerID="cri-o://e11593db6c87ea30fb2cc41c61641792a4ef7fb1d40533ee40fb5a297b6bb4b6" gracePeriod=30 Oct 31 04:38:36.940148 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:36.940122 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv"] Oct 31 04:38:36.940367 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:36.940302 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv" podUID="e308d744-11e3-4fef-bc86-eb0a7d9730b2" containerName="istio-proxy" containerID="cri-o://6f39ce4b9eca7f0e45ef638219c03e5402afb389d2d0bf222fdbf078e51b73d8" gracePeriod=30 Oct 31 04:38:42.317915 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:42.317881 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv" Oct 31 04:38:42.355756 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:42.355721 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n" Oct 31 04:38:42.409542 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:42.409503 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e308d744-11e3-4fef-bc86-eb0a7d9730b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e308d744-11e3-4fef-bc86-eb0a7d9730b2-istio-envoy\") pod \"e308d744-11e3-4fef-bc86-eb0a7d9730b2\" (UID: \"e308d744-11e3-4fef-bc86-eb0a7d9730b2\") " Oct 31 04:38:42.409719 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:42.409567 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e308d744-11e3-4fef-bc86-eb0a7d9730b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e308d744-11e3-4fef-bc86-eb0a7d9730b2-workload-certs\") pod \"e308d744-11e3-4fef-bc86-eb0a7d9730b2\" (UID: \"e308d744-11e3-4fef-bc86-eb0a7d9730b2\") " Oct 31 04:38:42.409719 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:42.409602 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-workload-socket\") pod \"88b7c3d5-09b5-4eda-a752-aac0c1f0a11c\" (UID: \"88b7c3d5-09b5-4eda-a752-aac0c1f0a11c\") " Oct 31 04:38:42.409719 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:42.409638 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e308d744-11e3-4fef-bc86-eb0a7d9730b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e308d744-11e3-4fef-bc86-eb0a7d9730b2-istio-data\") pod \"e308d744-11e3-4fef-bc86-eb0a7d9730b2\" (UID: \"e308d744-11e3-4fef-bc86-eb0a7d9730b2\") " Oct 31 04:38:42.409719 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:42.409682 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e308d744-11e3-4fef-bc86-eb0a7d9730b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e308d744-11e3-4fef-bc86-eb0a7d9730b2-workload-socket\") pod \"e308d744-11e3-4fef-bc86-eb0a7d9730b2\" (UID: \"e308d744-11e3-4fef-bc86-eb0a7d9730b2\") " Oct 31 04:38:42.409719 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:42.409710 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-istio-token\" (UniqueName: \"kubernetes.io/projected/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-istio-token\") pod \"88b7c3d5-09b5-4eda-a752-aac0c1f0a11c\" (UID: \"88b7c3d5-09b5-4eda-a752-aac0c1f0a11c\") " Oct 31 04:38:42.410017 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:42.409739 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-workload-certs\") pod \"88b7c3d5-09b5-4eda-a752-aac0c1f0a11c\" (UID: \"88b7c3d5-09b5-4eda-a752-aac0c1f0a11c\") " Oct 31 04:38:42.410017 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:42.409822 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e308d744-11e3-4fef-bc86-eb0a7d9730b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e308d744-11e3-4fef-bc86-eb0a7d9730b2-credential-socket\") pod \"e308d744-11e3-4fef-bc86-eb0a7d9730b2\" (UID: \"e308d744-11e3-4fef-bc86-eb0a7d9730b2\") " Oct 31 04:38:42.410017 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:42.409890 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-credential-socket\") pod \"88b7c3d5-09b5-4eda-a752-aac0c1f0a11c\" (UID: \"88b7c3d5-09b5-4eda-a752-aac0c1f0a11c\") " Oct 31 04:38:42.410017 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:42.409927 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-istiod-ca-cert\") pod \"88b7c3d5-09b5-4eda-a752-aac0c1f0a11c\" (UID: \"88b7c3d5-09b5-4eda-a752-aac0c1f0a11c\") " Oct 31 04:38:42.410017 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:42.409991 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e308d744-11e3-4fef-bc86-eb0a7d9730b2-istio-token\" (UniqueName: \"kubernetes.io/projected/e308d744-11e3-4fef-bc86-eb0a7d9730b2-istio-token\") pod \"e308d744-11e3-4fef-bc86-eb0a7d9730b2\" (UID: \"e308d744-11e3-4fef-bc86-eb0a7d9730b2\") " Oct 31 04:38:42.410290 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:42.410072 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-kube-api-access-p799z\" (UniqueName: \"kubernetes.io/projected/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-kube-api-access-p799z\") pod \"88b7c3d5-09b5-4eda-a752-aac0c1f0a11c\" (UID: \"88b7c3d5-09b5-4eda-a752-aac0c1f0a11c\") " Oct 31 04:38:42.410290 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:42.410102 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-istio-envoy\") pod \"88b7c3d5-09b5-4eda-a752-aac0c1f0a11c\" (UID: \"88b7c3d5-09b5-4eda-a752-aac0c1f0a11c\") " Oct 31 04:38:42.410290 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:42.410144 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-istio-podinfo\") pod \"88b7c3d5-09b5-4eda-a752-aac0c1f0a11c\" (UID: \"88b7c3d5-09b5-4eda-a752-aac0c1f0a11c\") " Oct 31 04:38:42.410290 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:42.410226 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e308d744-11e3-4fef-bc86-eb0a7d9730b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e308d744-11e3-4fef-bc86-eb0a7d9730b2-istiod-ca-cert\") pod \"e308d744-11e3-4fef-bc86-eb0a7d9730b2\" (UID: \"e308d744-11e3-4fef-bc86-eb0a7d9730b2\") " Oct 31 04:38:42.410574 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:42.410299 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e308d744-11e3-4fef-bc86-eb0a7d9730b2-kube-api-access-8ktg7\" (UniqueName: \"kubernetes.io/projected/e308d744-11e3-4fef-bc86-eb0a7d9730b2-kube-api-access-8ktg7\") pod \"e308d744-11e3-4fef-bc86-eb0a7d9730b2\" (UID: \"e308d744-11e3-4fef-bc86-eb0a7d9730b2\") " Oct 31 04:38:42.410574 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:42.410367 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e308d744-11e3-4fef-bc86-eb0a7d9730b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e308d744-11e3-4fef-bc86-eb0a7d9730b2-istio-podinfo\") pod \"e308d744-11e3-4fef-bc86-eb0a7d9730b2\" (UID: \"e308d744-11e3-4fef-bc86-eb0a7d9730b2\") " Oct 31 04:38:42.410574 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:42.410446 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-istio-data\") pod \"88b7c3d5-09b5-4eda-a752-aac0c1f0a11c\" (UID: \"88b7c3d5-09b5-4eda-a752-aac0c1f0a11c\") " Oct 31 04:38:42.410745 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:42.410584 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e308d744-11e3-4fef-bc86-eb0a7d9730b2-credential-socket" pod "e308d744-11e3-4fef-bc86-eb0a7d9730b2" (UID: "e308d744-11e3-4fef-bc86-eb0a7d9730b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:42.411623 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:42.411361 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-credential-socket" pod "88b7c3d5-09b5-4eda-a752-aac0c1f0a11c" (UID: "88b7c3d5-09b5-4eda-a752-aac0c1f0a11c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:42.411623 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:42.411544 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-istiod-ca-cert" pod "88b7c3d5-09b5-4eda-a752-aac0c1f0a11c" (UID: "88b7c3d5-09b5-4eda-a752-aac0c1f0a11c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:38:42.411874 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:42.411785 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-istio-data" pod "88b7c3d5-09b5-4eda-a752-aac0c1f0a11c" (UID: "88b7c3d5-09b5-4eda-a752-aac0c1f0a11c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:42.411874 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:42.411824 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:38:42.411874 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:42.411844 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:38:42.411874 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:42.411860 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e308d744-11e3-4fef-bc86-eb0a7d9730b2-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:38:42.412122 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:42.411994 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e308d744-11e3-4fef-bc86-eb0a7d9730b2-istio-data" pod "e308d744-11e3-4fef-bc86-eb0a7d9730b2" (UID: "e308d744-11e3-4fef-bc86-eb0a7d9730b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:42.412122 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:42.412069 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e308d744-11e3-4fef-bc86-eb0a7d9730b2-workload-socket" pod "e308d744-11e3-4fef-bc86-eb0a7d9730b2" (UID: "e308d744-11e3-4fef-bc86-eb0a7d9730b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:42.412237 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:42.412200 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e308d744-11e3-4fef-bc86-eb0a7d9730b2-workload-certs" pod "e308d744-11e3-4fef-bc86-eb0a7d9730b2" (UID: "e308d744-11e3-4fef-bc86-eb0a7d9730b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:42.412306 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:42.412257 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-workload-socket" pod "88b7c3d5-09b5-4eda-a752-aac0c1f0a11c" (UID: "88b7c3d5-09b5-4eda-a752-aac0c1f0a11c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:42.412856 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:42.412440 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-workload-certs" pod "88b7c3d5-09b5-4eda-a752-aac0c1f0a11c" (UID: "88b7c3d5-09b5-4eda-a752-aac0c1f0a11c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:42.413049 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:42.413016 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e308d744-11e3-4fef-bc86-eb0a7d9730b2-istiod-ca-cert" pod "e308d744-11e3-4fef-bc86-eb0a7d9730b2" (UID: "e308d744-11e3-4fef-bc86-eb0a7d9730b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:38:42.415438 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:42.415411 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-istio-token" pod "88b7c3d5-09b5-4eda-a752-aac0c1f0a11c" (UID: "88b7c3d5-09b5-4eda-a752-aac0c1f0a11c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:38:42.415713 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:42.415656 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e308d744-11e3-4fef-bc86-eb0a7d9730b2-kube-api-access-8ktg7" pod "e308d744-11e3-4fef-bc86-eb0a7d9730b2" (UID: "e308d744-11e3-4fef-bc86-eb0a7d9730b2"). InnerVolumeSpecName "kube-api-access-8ktg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:38:42.416683 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:42.416662 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-istio-envoy" pod "88b7c3d5-09b5-4eda-a752-aac0c1f0a11c" (UID: "88b7c3d5-09b5-4eda-a752-aac0c1f0a11c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:42.416813 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:42.416713 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e308d744-11e3-4fef-bc86-eb0a7d9730b2-istio-token" pod "e308d744-11e3-4fef-bc86-eb0a7d9730b2" (UID: "e308d744-11e3-4fef-bc86-eb0a7d9730b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:38:42.416813 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:42.416762 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e308d744-11e3-4fef-bc86-eb0a7d9730b2-istio-envoy" pod "e308d744-11e3-4fef-bc86-eb0a7d9730b2" (UID: "e308d744-11e3-4fef-bc86-eb0a7d9730b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:42.417292 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:42.417266 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-kube-api-access-p799z" pod "88b7c3d5-09b5-4eda-a752-aac0c1f0a11c" (UID: "88b7c3d5-09b5-4eda-a752-aac0c1f0a11c"). InnerVolumeSpecName "kube-api-access-p799z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:38:42.418249 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:42.418221 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e308d744-11e3-4fef-bc86-eb0a7d9730b2-istio-podinfo" pod "e308d744-11e3-4fef-bc86-eb0a7d9730b2" (UID: "e308d744-11e3-4fef-bc86-eb0a7d9730b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:38:42.418596 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:42.418576 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-istio-podinfo" pod "88b7c3d5-09b5-4eda-a752-aac0c1f0a11c" (UID: "88b7c3d5-09b5-4eda-a752-aac0c1f0a11c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:38:42.513512 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:42.513481 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e308d744-11e3-4fef-bc86-eb0a7d9730b2-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:38:42.513692 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:42.513514 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ktg7\" (UniqueName: \"kubernetes.io/projected/e308d744-11e3-4fef-bc86-eb0a7d9730b2-kube-api-access-8ktg7\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:38:42.513692 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:42.513531 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e308d744-11e3-4fef-bc86-eb0a7d9730b2-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:38:42.513692 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:42.513545 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:38:42.513692 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:42.513559 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e308d744-11e3-4fef-bc86-eb0a7d9730b2-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:38:42.513692 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:42.513572 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e308d744-11e3-4fef-bc86-eb0a7d9730b2-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:38:42.513692 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:42.513585 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:38:42.513692 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:42.513596 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e308d744-11e3-4fef-bc86-eb0a7d9730b2-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:38:42.513692 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:42.513610 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e308d744-11e3-4fef-bc86-eb0a7d9730b2-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:38:42.513692 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:42.513622 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:38:42.513692 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:42.513634 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:38:42.513692 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:42.513646 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e308d744-11e3-4fef-bc86-eb0a7d9730b2-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:38:42.513692 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:42.513659 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p799z\" (UniqueName: \"kubernetes.io/projected/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-kube-api-access-p799z\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:38:42.513692 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:42.513671 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:38:42.513692 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:42.513685 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:38:43.045124 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:43.045084 5019 generic.go:356] "Generic (PLEG): container finished" podID="88b7c3d5-09b5-4eda-a752-aac0c1f0a11c" containerID="e11593db6c87ea30fb2cc41c61641792a4ef7fb1d40533ee40fb5a297b6bb4b6" exitCode=0 Oct 31 04:38:43.045276 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:43.045119 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n" event={"ID":"88b7c3d5-09b5-4eda-a752-aac0c1f0a11c","Type":"ContainerDied","Data":"e11593db6c87ea30fb2cc41c61641792a4ef7fb1d40533ee40fb5a297b6bb4b6"} Oct 31 04:38:43.045276 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:43.045182 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n" event={"ID":"88b7c3d5-09b5-4eda-a752-aac0c1f0a11c","Type":"ContainerDied","Data":"a6c5d4441a902498dc0e1f70d0a54d9c2122a399b7eeef2c8714fc524d4d1f1f"} Oct 31 04:38:43.045276 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:43.045204 5019 scope.go:122] "RemoveContainer" containerID="e11593db6c87ea30fb2cc41c61641792a4ef7fb1d40533ee40fb5a297b6bb4b6" Oct 31 04:38:43.045276 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:43.045129 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n" Oct 31 04:38:43.048325 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:43.048292 5019 generic.go:356] "Generic (PLEG): container finished" podID="e308d744-11e3-4fef-bc86-eb0a7d9730b2" containerID="6f39ce4b9eca7f0e45ef638219c03e5402afb389d2d0bf222fdbf078e51b73d8" exitCode=0 Oct 31 04:38:43.048542 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:43.048515 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv" event={"ID":"e308d744-11e3-4fef-bc86-eb0a7d9730b2","Type":"ContainerDied","Data":"6f39ce4b9eca7f0e45ef638219c03e5402afb389d2d0bf222fdbf078e51b73d8"} Oct 31 04:38:43.048776 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:43.048746 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv" event={"ID":"e308d744-11e3-4fef-bc86-eb0a7d9730b2","Type":"ContainerDied","Data":"fda7b873f1a4ec5d1f5a13cbc11a117d3ab4ecff8252544ff3f7ef5805c30f53"} Oct 31 04:38:43.048902 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:43.048582 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv" Oct 31 04:38:43.062752 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:43.062521 5019 scope.go:122] "RemoveContainer" containerID="e11593db6c87ea30fb2cc41c61641792a4ef7fb1d40533ee40fb5a297b6bb4b6" Oct 31 04:38:43.062835 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:38:43.062785 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e11593db6c87ea30fb2cc41c61641792a4ef7fb1d40533ee40fb5a297b6bb4b6\": container with ID starting with e11593db6c87ea30fb2cc41c61641792a4ef7fb1d40533ee40fb5a297b6bb4b6 not found: ID does not exist" containerID="e11593db6c87ea30fb2cc41c61641792a4ef7fb1d40533ee40fb5a297b6bb4b6" Oct 31 04:38:43.062835 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:43.062827 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e11593db6c87ea30fb2cc41c61641792a4ef7fb1d40533ee40fb5a297b6bb4b6"} err="failed to get container status \"e11593db6c87ea30fb2cc41c61641792a4ef7fb1d40533ee40fb5a297b6bb4b6\": rpc error: code = NotFound desc = could not find container \"e11593db6c87ea30fb2cc41c61641792a4ef7fb1d40533ee40fb5a297b6bb4b6\": container with ID starting with e11593db6c87ea30fb2cc41c61641792a4ef7fb1d40533ee40fb5a297b6bb4b6 not found: ID does not exist" Oct 31 04:38:43.062960 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:43.062844 5019 scope.go:122] "RemoveContainer" containerID="6f39ce4b9eca7f0e45ef638219c03e5402afb389d2d0bf222fdbf078e51b73d8" Oct 31 04:38:43.075069 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:43.075048 5019 scope.go:122] "RemoveContainer" containerID="6f39ce4b9eca7f0e45ef638219c03e5402afb389d2d0bf222fdbf078e51b73d8" Oct 31 04:38:43.075393 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:38:43.075322 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f39ce4b9eca7f0e45ef638219c03e5402afb389d2d0bf222fdbf078e51b73d8\": container with ID starting with 6f39ce4b9eca7f0e45ef638219c03e5402afb389d2d0bf222fdbf078e51b73d8 not found: ID does not exist" containerID="6f39ce4b9eca7f0e45ef638219c03e5402afb389d2d0bf222fdbf078e51b73d8" Oct 31 04:38:43.075486 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:43.075403 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f39ce4b9eca7f0e45ef638219c03e5402afb389d2d0bf222fdbf078e51b73d8"} err="failed to get container status \"6f39ce4b9eca7f0e45ef638219c03e5402afb389d2d0bf222fdbf078e51b73d8\": rpc error: code = NotFound desc = could not find container \"6f39ce4b9eca7f0e45ef638219c03e5402afb389d2d0bf222fdbf078e51b73d8\": container with ID starting with 6f39ce4b9eca7f0e45ef638219c03e5402afb389d2d0bf222fdbf078e51b73d8 not found: ID does not exist" Oct 31 04:38:43.112596 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:43.112563 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n"] Oct 31 04:38:43.124353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:43.124318 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-m8x2n"] Oct 31 04:38:43.164900 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:43.164866 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv"] Oct 31 04:38:43.168951 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:43.168922 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-crdbv"] Oct 31 04:38:43.412256 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:43.412221 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88b7c3d5-09b5-4eda-a752-aac0c1f0a11c" path="/var/lib/kubelet/pods/88b7c3d5-09b5-4eda-a752-aac0c1f0a11c/volumes" Oct 31 04:38:43.413008 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:43.412988 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e308d744-11e3-4fef-bc86-eb0a7d9730b2" path="/var/lib/kubelet/pods/e308d744-11e3-4fef-bc86-eb0a7d9730b2/volumes" Oct 31 04:38:45.448611 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:45.448579 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-c9707f/auth-test-model-c9707f-kserve-f5d44dd69-kqt6t"] Oct 31 04:38:50.502580 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:50.502505 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-338f2e/auth-test-model-338f2e-kserve-79676f7df-lwb9d" podUID="d03e187b-a841-486c-8a3d-bce5d3adee70" containerName="main" containerID="cri-o://03ad844a32c33a419fcbc42eb42e91eb7e0560f7ce57684e0e70eeab7d2a31b0" gracePeriod=45 Oct 31 04:38:50.627694 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:50.626070 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b2a372/auth-test-model-b2a372-kserve-85c4657f6f-7gst5"] Oct 31 04:38:51.015875 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:51.015847 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-338f2e/auth-test-model-338f2e-kserve-79676f7df-lwb9d" Oct 31 04:38:51.055101 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:51.055074 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d03e187b-a841-486c-8a3d-bce5d3adee70-tls-certs\" (UniqueName: \"kubernetes.io/secret/d03e187b-a841-486c-8a3d-bce5d3adee70-tls-certs\") pod \"d03e187b-a841-486c-8a3d-bce5d3adee70\" (UID: \"d03e187b-a841-486c-8a3d-bce5d3adee70\") " Oct 31 04:38:51.055262 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:51.055211 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d03e187b-a841-486c-8a3d-bce5d3adee70-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d03e187b-a841-486c-8a3d-bce5d3adee70-tmp-dir\") pod \"d03e187b-a841-486c-8a3d-bce5d3adee70\" (UID: \"d03e187b-a841-486c-8a3d-bce5d3adee70\") " Oct 31 04:38:51.055352 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:51.055315 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d03e187b-a841-486c-8a3d-bce5d3adee70-kube-api-access-w5d2g\" (UniqueName: \"kubernetes.io/projected/d03e187b-a841-486c-8a3d-bce5d3adee70-kube-api-access-w5d2g\") pod \"d03e187b-a841-486c-8a3d-bce5d3adee70\" (UID: \"d03e187b-a841-486c-8a3d-bce5d3adee70\") " Oct 31 04:38:51.055473 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:51.055446 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d03e187b-a841-486c-8a3d-bce5d3adee70-tmp-dir" pod "d03e187b-a841-486c-8a3d-bce5d3adee70" (UID: "d03e187b-a841-486c-8a3d-bce5d3adee70"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:51.055473 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:51.055466 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d03e187b-a841-486c-8a3d-bce5d3adee70-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d03e187b-a841-486c-8a3d-bce5d3adee70-dshm\") pod \"d03e187b-a841-486c-8a3d-bce5d3adee70\" (UID: \"d03e187b-a841-486c-8a3d-bce5d3adee70\") " Oct 31 04:38:51.055606 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:51.055520 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d03e187b-a841-486c-8a3d-bce5d3adee70-home\" (UniqueName: \"kubernetes.io/empty-dir/d03e187b-a841-486c-8a3d-bce5d3adee70-home\") pod \"d03e187b-a841-486c-8a3d-bce5d3adee70\" (UID: \"d03e187b-a841-486c-8a3d-bce5d3adee70\") " Oct 31 04:38:51.055606 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:51.055553 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d03e187b-a841-486c-8a3d-bce5d3adee70-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d03e187b-a841-486c-8a3d-bce5d3adee70-model-cache\") pod \"d03e187b-a841-486c-8a3d-bce5d3adee70\" (UID: \"d03e187b-a841-486c-8a3d-bce5d3adee70\") " Oct 31 04:38:51.055956 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:51.055770 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d03e187b-a841-486c-8a3d-bce5d3adee70-home" pod "d03e187b-a841-486c-8a3d-bce5d3adee70" (UID: "d03e187b-a841-486c-8a3d-bce5d3adee70"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:51.055956 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:51.055929 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d03e187b-a841-486c-8a3d-bce5d3adee70-model-cache" pod "d03e187b-a841-486c-8a3d-bce5d3adee70" (UID: "d03e187b-a841-486c-8a3d-bce5d3adee70"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:51.057499 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:51.056145 5019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d03e187b-a841-486c-8a3d-bce5d3adee70-home\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:38:51.057499 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:51.056167 5019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d03e187b-a841-486c-8a3d-bce5d3adee70-model-cache\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:38:51.057499 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:51.056181 5019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d03e187b-a841-486c-8a3d-bce5d3adee70-tmp-dir\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:38:51.058279 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:51.058236 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d03e187b-a841-486c-8a3d-bce5d3adee70-kube-api-access-w5d2g" pod "d03e187b-a841-486c-8a3d-bce5d3adee70" (UID: "d03e187b-a841-486c-8a3d-bce5d3adee70"). InnerVolumeSpecName "kube-api-access-w5d2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:38:51.058826 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:51.058801 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d03e187b-a841-486c-8a3d-bce5d3adee70-tls-certs" pod "d03e187b-a841-486c-8a3d-bce5d3adee70" (UID: "d03e187b-a841-486c-8a3d-bce5d3adee70"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:38:51.062145 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:51.062123 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d03e187b-a841-486c-8a3d-bce5d3adee70-dshm" pod "d03e187b-a841-486c-8a3d-bce5d3adee70" (UID: "d03e187b-a841-486c-8a3d-bce5d3adee70"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:38:51.157906 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:51.157832 5019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d03e187b-a841-486c-8a3d-bce5d3adee70-tls-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:38:51.157906 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:51.157862 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5d2g\" (UniqueName: \"kubernetes.io/projected/d03e187b-a841-486c-8a3d-bce5d3adee70-kube-api-access-w5d2g\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:38:51.157906 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:51.157871 5019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d03e187b-a841-486c-8a3d-bce5d3adee70-dshm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:38:51.398242 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:51.398197 5019 generic.go:356] "Generic (PLEG): container finished" podID="d03e187b-a841-486c-8a3d-bce5d3adee70" containerID="03ad844a32c33a419fcbc42eb42e91eb7e0560f7ce57684e0e70eeab7d2a31b0" exitCode=0 Oct 31 04:38:51.398452 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:51.398264 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-338f2e/auth-test-model-338f2e-kserve-79676f7df-lwb9d" event={"ID":"d03e187b-a841-486c-8a3d-bce5d3adee70","Type":"ContainerDied","Data":"03ad844a32c33a419fcbc42eb42e91eb7e0560f7ce57684e0e70eeab7d2a31b0"} Oct 31 04:38:51.398452 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:51.398299 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-338f2e/auth-test-model-338f2e-kserve-79676f7df-lwb9d" event={"ID":"d03e187b-a841-486c-8a3d-bce5d3adee70","Type":"ContainerDied","Data":"2ba7ddba50271c3122c6af1bc49cff973d8a3626191c2f24cacbca90a14bbb5b"} Oct 31 04:38:51.398452 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:51.398321 5019 scope.go:122] "RemoveContainer" containerID="03ad844a32c33a419fcbc42eb42e91eb7e0560f7ce57684e0e70eeab7d2a31b0" Oct 31 04:38:51.398452 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:51.398379 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-338f2e/auth-test-model-338f2e-kserve-79676f7df-lwb9d" Oct 31 04:38:51.417047 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:51.417023 5019 scope.go:122] "RemoveContainer" containerID="03ad844a32c33a419fcbc42eb42e91eb7e0560f7ce57684e0e70eeab7d2a31b0" Oct 31 04:38:51.417404 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:38:51.417377 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03ad844a32c33a419fcbc42eb42e91eb7e0560f7ce57684e0e70eeab7d2a31b0\": container with ID starting with 03ad844a32c33a419fcbc42eb42e91eb7e0560f7ce57684e0e70eeab7d2a31b0 not found: ID does not exist" containerID="03ad844a32c33a419fcbc42eb42e91eb7e0560f7ce57684e0e70eeab7d2a31b0" Oct 31 04:38:51.417501 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:51.417415 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03ad844a32c33a419fcbc42eb42e91eb7e0560f7ce57684e0e70eeab7d2a31b0"} err="failed to get container status \"03ad844a32c33a419fcbc42eb42e91eb7e0560f7ce57684e0e70eeab7d2a31b0\": rpc error: code = NotFound desc = could not find container \"03ad844a32c33a419fcbc42eb42e91eb7e0560f7ce57684e0e70eeab7d2a31b0\": container with ID starting with 03ad844a32c33a419fcbc42eb42e91eb7e0560f7ce57684e0e70eeab7d2a31b0 not found: ID does not exist" Oct 31 04:38:51.459445 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:51.459412 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-338f2e/auth-test-model-338f2e-kserve-79676f7df-lwb9d"] Oct 31 04:38:51.463283 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:51.463258 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-338f2e/auth-test-model-338f2e-kserve-79676f7df-lwb9d"] Oct 31 04:38:52.411379 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:38:52.411291 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:38:53.414823 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:53.414781 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d03e187b-a841-486c-8a3d-bce5d3adee70" path="/var/lib/kubelet/pods/d03e187b-a841-486c-8a3d-bce5d3adee70/volumes" Oct 31 04:38:56.447776 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:56.447741 5019 generic.go:356] "Generic (PLEG): container finished" podID="de1f94d9-8507-47a5-aec8-88c546112c7f" containerID="a3128cd2d5d93597ea6ac8b493b037d64093bbcf3f8da977dd3db858ff57b612" exitCode=0 Oct 31 04:38:56.448378 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:56.447818 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz" event={"ID":"de1f94d9-8507-47a5-aec8-88c546112c7f","Type":"ContainerDied","Data":"a3128cd2d5d93597ea6ac8b493b037d64093bbcf3f8da977dd3db858ff57b612"} Oct 31 04:38:56.448491 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:56.448447 5019 scope.go:122] "RemoveContainer" containerID="a3128cd2d5d93597ea6ac8b493b037d64093bbcf3f8da977dd3db858ff57b612" Oct 31 04:38:56.450226 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:56.450196 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:38:56.450297 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:56.450267 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:38:56.450425 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:56.450298 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:38:57.457556 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:57.457518 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz" event={"ID":"de1f94d9-8507-47a5-aec8-88c546112c7f","Type":"ContainerStarted","Data":"f031d4562c563b9e35704ce4c384ddf3aed932bda02221db54e339d5cd5714b7"} Oct 31 04:38:58.191358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:58.186024 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz" Oct 31 04:38:58.194522 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:58.194482 5019 patch_prober.go:29] interesting pod/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" start-of-body= Oct 31 04:38:58.194823 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:58.194788 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz" podUID="de1f94d9-8507-47a5-aec8-88c546112c7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" Oct 31 04:38:59.191041 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:59.191004 5019 patch_prober.go:29] interesting pod/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" start-of-body= Oct 31 04:38:59.191694 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:38:59.191658 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz" podUID="de1f94d9-8507-47a5-aec8-88c546112c7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" Oct 31 04:39:00.187811 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:00.187768 5019 patch_prober.go:29] interesting pod/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" start-of-body= Oct 31 04:39:00.188118 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:00.188082 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz" podUID="de1f94d9-8507-47a5-aec8-88c546112c7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" Oct 31 04:39:00.503930 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:00.500539 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-c9707f/auth-test-model-c9707f-kserve-f5d44dd69-kqt6t" podUID="91ef8aa6-c2a0-4a20-8131-89dd0b63514d" containerName="main" containerID="cri-o://5a24e8c3a3ba411b2248e297e9eca0a97ba2ea208923ebdb1a8e847b1dfd7d10" gracePeriod=45 Oct 31 04:39:00.942817 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:00.942782 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-c9707f/auth-test-model-c9707f-kserve-f5d44dd69-kqt6t" Oct 31 04:39:00.956100 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:00.956049 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91ef8aa6-c2a0-4a20-8131-89dd0b63514d-kube-api-access-wqvjn\" (UniqueName: \"kubernetes.io/projected/91ef8aa6-c2a0-4a20-8131-89dd0b63514d-kube-api-access-wqvjn\") pod \"91ef8aa6-c2a0-4a20-8131-89dd0b63514d\" (UID: \"91ef8aa6-c2a0-4a20-8131-89dd0b63514d\") " Oct 31 04:39:00.956361 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:00.956202 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91ef8aa6-c2a0-4a20-8131-89dd0b63514d-tls-certs\" (UniqueName: \"kubernetes.io/secret/91ef8aa6-c2a0-4a20-8131-89dd0b63514d-tls-certs\") pod \"91ef8aa6-c2a0-4a20-8131-89dd0b63514d\" (UID: \"91ef8aa6-c2a0-4a20-8131-89dd0b63514d\") " Oct 31 04:39:00.956361 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:00.956293 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91ef8aa6-c2a0-4a20-8131-89dd0b63514d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/91ef8aa6-c2a0-4a20-8131-89dd0b63514d-dshm\") pod \"91ef8aa6-c2a0-4a20-8131-89dd0b63514d\" (UID: \"91ef8aa6-c2a0-4a20-8131-89dd0b63514d\") " Oct 31 04:39:00.956498 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:00.956391 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91ef8aa6-c2a0-4a20-8131-89dd0b63514d-home\" (UniqueName: \"kubernetes.io/empty-dir/91ef8aa6-c2a0-4a20-8131-89dd0b63514d-home\") pod \"91ef8aa6-c2a0-4a20-8131-89dd0b63514d\" (UID: \"91ef8aa6-c2a0-4a20-8131-89dd0b63514d\") " Oct 31 04:39:00.956498 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:00.956482 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91ef8aa6-c2a0-4a20-8131-89dd0b63514d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91ef8aa6-c2a0-4a20-8131-89dd0b63514d-model-cache\") pod \"91ef8aa6-c2a0-4a20-8131-89dd0b63514d\" (UID: \"91ef8aa6-c2a0-4a20-8131-89dd0b63514d\") " Oct 31 04:39:00.956647 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:00.956629 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91ef8aa6-c2a0-4a20-8131-89dd0b63514d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91ef8aa6-c2a0-4a20-8131-89dd0b63514d-tmp-dir\") pod \"91ef8aa6-c2a0-4a20-8131-89dd0b63514d\" (UID: \"91ef8aa6-c2a0-4a20-8131-89dd0b63514d\") " Oct 31 04:39:00.956900 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:00.956684 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ef8aa6-c2a0-4a20-8131-89dd0b63514d-home" pod "91ef8aa6-c2a0-4a20-8131-89dd0b63514d" (UID: "91ef8aa6-c2a0-4a20-8131-89dd0b63514d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:39:00.956996 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:00.956902 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ef8aa6-c2a0-4a20-8131-89dd0b63514d-tmp-dir" pod "91ef8aa6-c2a0-4a20-8131-89dd0b63514d" (UID: "91ef8aa6-c2a0-4a20-8131-89dd0b63514d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:39:00.956996 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:00.956801 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ef8aa6-c2a0-4a20-8131-89dd0b63514d-model-cache" pod "91ef8aa6-c2a0-4a20-8131-89dd0b63514d" (UID: "91ef8aa6-c2a0-4a20-8131-89dd0b63514d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:39:00.957529 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:00.957508 5019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91ef8aa6-c2a0-4a20-8131-89dd0b63514d-tmp-dir\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:39:00.957611 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:00.957531 5019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91ef8aa6-c2a0-4a20-8131-89dd0b63514d-home\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:39:00.957611 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:00.957544 5019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91ef8aa6-c2a0-4a20-8131-89dd0b63514d-model-cache\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:39:00.958812 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:00.958786 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91ef8aa6-c2a0-4a20-8131-89dd0b63514d-tls-certs" pod "91ef8aa6-c2a0-4a20-8131-89dd0b63514d" (UID: "91ef8aa6-c2a0-4a20-8131-89dd0b63514d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:39:00.958983 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:00.958959 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ef8aa6-c2a0-4a20-8131-89dd0b63514d-dshm" pod "91ef8aa6-c2a0-4a20-8131-89dd0b63514d" (UID: "91ef8aa6-c2a0-4a20-8131-89dd0b63514d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:39:00.959029 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:00.958973 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91ef8aa6-c2a0-4a20-8131-89dd0b63514d-kube-api-access-wqvjn" pod "91ef8aa6-c2a0-4a20-8131-89dd0b63514d" (UID: "91ef8aa6-c2a0-4a20-8131-89dd0b63514d"). InnerVolumeSpecName "kube-api-access-wqvjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:39:01.059278 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:01.059235 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqvjn\" (UniqueName: \"kubernetes.io/projected/91ef8aa6-c2a0-4a20-8131-89dd0b63514d-kube-api-access-wqvjn\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:39:01.059508 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:01.059428 5019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91ef8aa6-c2a0-4a20-8131-89dd0b63514d-tls-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:39:01.059508 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:01.059458 5019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91ef8aa6-c2a0-4a20-8131-89dd0b63514d-dshm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:39:01.187421 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:01.185167 5019 patch_prober.go:29] interesting pod/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" start-of-body= Oct 31 04:39:01.187421 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:01.185233 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz" podUID="de1f94d9-8507-47a5-aec8-88c546112c7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" Oct 31 04:39:01.544316 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:01.544271 5019 generic.go:356] "Generic (PLEG): container finished" podID="91ef8aa6-c2a0-4a20-8131-89dd0b63514d" containerID="5a24e8c3a3ba411b2248e297e9eca0a97ba2ea208923ebdb1a8e847b1dfd7d10" exitCode=0 Oct 31 04:39:01.544862 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:01.544319 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-c9707f/auth-test-model-c9707f-kserve-f5d44dd69-kqt6t" Oct 31 04:39:01.544862 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:01.544354 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-c9707f/auth-test-model-c9707f-kserve-f5d44dd69-kqt6t" event={"ID":"91ef8aa6-c2a0-4a20-8131-89dd0b63514d","Type":"ContainerDied","Data":"5a24e8c3a3ba411b2248e297e9eca0a97ba2ea208923ebdb1a8e847b1dfd7d10"} Oct 31 04:39:01.544862 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:01.544415 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-c9707f/auth-test-model-c9707f-kserve-f5d44dd69-kqt6t" event={"ID":"91ef8aa6-c2a0-4a20-8131-89dd0b63514d","Type":"ContainerDied","Data":"74ae7be518e52c58738d03ea85f61d3f85e10a6793cebd680ec307b736397bf8"} Oct 31 04:39:01.544862 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:01.544440 5019 scope.go:122] "RemoveContainer" containerID="5a24e8c3a3ba411b2248e297e9eca0a97ba2ea208923ebdb1a8e847b1dfd7d10" Oct 31 04:39:01.555728 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:01.555702 5019 scope.go:122] "RemoveContainer" containerID="5a24e8c3a3ba411b2248e297e9eca0a97ba2ea208923ebdb1a8e847b1dfd7d10" Oct 31 04:39:01.556130 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:39:01.556106 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a24e8c3a3ba411b2248e297e9eca0a97ba2ea208923ebdb1a8e847b1dfd7d10\": container with ID starting with 5a24e8c3a3ba411b2248e297e9eca0a97ba2ea208923ebdb1a8e847b1dfd7d10 not found: ID does not exist" containerID="5a24e8c3a3ba411b2248e297e9eca0a97ba2ea208923ebdb1a8e847b1dfd7d10" Oct 31 04:39:01.556203 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:01.556143 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a24e8c3a3ba411b2248e297e9eca0a97ba2ea208923ebdb1a8e847b1dfd7d10"} err="failed to get container status \"5a24e8c3a3ba411b2248e297e9eca0a97ba2ea208923ebdb1a8e847b1dfd7d10\": rpc error: code = NotFound desc = could not find container \"5a24e8c3a3ba411b2248e297e9eca0a97ba2ea208923ebdb1a8e847b1dfd7d10\": container with ID starting with 5a24e8c3a3ba411b2248e297e9eca0a97ba2ea208923ebdb1a8e847b1dfd7d10 not found: ID does not exist" Oct 31 04:39:01.611777 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:01.611723 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-c9707f/auth-test-model-c9707f-kserve-f5d44dd69-kqt6t"] Oct 31 04:39:01.616650 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:01.616612 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-c9707f/auth-test-model-c9707f-kserve-f5d44dd69-kqt6t"] Oct 31 04:39:02.018598 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:02.018540 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h"] Oct 31 04:39:02.033411 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:02.032715 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h" podUID="686f7dd0-02cd-457a-b7cd-ef83520d5341" containerName="istio-proxy" containerID="cri-o://bcea24c66148803cee1eba69713d84e38d9b2d233d5a8dabc9a5958f7c962a3d" gracePeriod=30 Oct 31 04:39:02.238358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:02.186549 5019 patch_prober.go:29] interesting pod/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" start-of-body= Oct 31 04:39:02.238725 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:02.238677 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz" podUID="de1f94d9-8507-47a5-aec8-88c546112c7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" Oct 31 04:39:02.502959 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:02.502912 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz" Oct 31 04:39:02.535748 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:02.535704 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-031745/auth-test-model-031745-kserve-7656c77f57-l86zb"] Oct 31 04:39:03.386741 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:03.386690 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz" Oct 31 04:39:03.390292 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:03.390250 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz" Oct 31 04:39:03.419788 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:03.419739 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91ef8aa6-c2a0-4a20-8131-89dd0b63514d" path="/var/lib/kubelet/pods/91ef8aa6-c2a0-4a20-8131-89dd0b63514d/volumes" Oct 31 04:39:04.544907 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:04.544853 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc"] Oct 31 04:39:04.545999 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:04.545975 5019 cpu_manager.go:414] "containerMap: removing container" podUID="8aaf1cba-a101-45a1-b87b-afe4e7cf00ab" containerName="revoke-keys" Oct 31 04:39:04.546086 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:04.546002 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aaf1cba-a101-45a1-b87b-afe4e7cf00ab" containerName="revoke-keys" Oct 31 04:39:04.546086 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:04.546059 5019 cpu_manager.go:414] "containerMap: removing container" podUID="88b7c3d5-09b5-4eda-a752-aac0c1f0a11c" containerName="istio-proxy" Oct 31 04:39:04.546086 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:04.546068 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88b7c3d5-09b5-4eda-a752-aac0c1f0a11c" containerName="istio-proxy" Oct 31 04:39:04.546187 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:04.546089 5019 cpu_manager.go:414] "containerMap: removing container" podUID="91ef8aa6-c2a0-4a20-8131-89dd0b63514d" containerName="main" Oct 31 04:39:04.546187 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:04.546098 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91ef8aa6-c2a0-4a20-8131-89dd0b63514d" containerName="main" Oct 31 04:39:04.546187 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:04.546117 5019 cpu_manager.go:414] "containerMap: removing container" podUID="e308d744-11e3-4fef-bc86-eb0a7d9730b2" containerName="istio-proxy" Oct 31 04:39:04.546187 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:04.546126 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e308d744-11e3-4fef-bc86-eb0a7d9730b2" containerName="istio-proxy" Oct 31 04:39:04.546187 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:04.546144 5019 cpu_manager.go:414] "containerMap: removing container" podUID="4f521bb4-2d60-445e-9a83-96492311abd1" containerName="revoke-keys" Oct 31 04:39:04.546187 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:04.546152 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f521bb4-2d60-445e-9a83-96492311abd1" containerName="revoke-keys" Oct 31 04:39:04.546187 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:04.546171 5019 cpu_manager.go:414] "containerMap: removing container" podUID="d03e187b-a841-486c-8a3d-bce5d3adee70" containerName="main" Oct 31 04:39:04.546187 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:04.546179 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d03e187b-a841-486c-8a3d-bce5d3adee70" containerName="main" Oct 31 04:39:04.546569 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:04.546193 5019 cpu_manager.go:414] "containerMap: removing container" podUID="0c071116-bc3c-4d68-9e80-ec887ee96be7" containerName="istio-proxy" Oct 31 04:39:04.546569 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:04.546201 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c071116-bc3c-4d68-9e80-ec887ee96be7" containerName="istio-proxy" Oct 31 04:39:04.546569 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:04.546215 5019 cpu_manager.go:414] "containerMap: removing container" podUID="d25c55c3-f558-495e-b197-e0f5231c53f2" containerName="revoke-keys" Oct 31 04:39:04.546569 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:04.546223 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d25c55c3-f558-495e-b197-e0f5231c53f2" containerName="revoke-keys" Oct 31 04:39:04.546569 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:04.546469 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="88b7c3d5-09b5-4eda-a752-aac0c1f0a11c" containerName="istio-proxy" Oct 31 04:39:04.546569 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:04.546502 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="91ef8aa6-c2a0-4a20-8131-89dd0b63514d" containerName="main" Oct 31 04:39:04.546569 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:04.546516 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e308d744-11e3-4fef-bc86-eb0a7d9730b2" containerName="istio-proxy" Oct 31 04:39:04.546569 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:04.546526 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f521bb4-2d60-445e-9a83-96492311abd1" containerName="revoke-keys" Oct 31 04:39:04.546569 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:04.546537 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="d03e187b-a841-486c-8a3d-bce5d3adee70" containerName="main" Oct 31 04:39:04.546569 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:04.546548 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c071116-bc3c-4d68-9e80-ec887ee96be7" containerName="istio-proxy" Oct 31 04:39:04.546569 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:04.546557 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="d25c55c3-f558-495e-b197-e0f5231c53f2" containerName="revoke-keys" Oct 31 04:39:04.546569 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:04.546572 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aaf1cba-a101-45a1-b87b-afe4e7cf00ab" containerName="revoke-keys" Oct 31 04:39:04.552705 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:04.552672 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc" Oct 31 04:39:04.579045 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:04.578764 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-401761-openshift-default-dockercfg-79l4h\"" Oct 31 04:39:04.580736 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:04.580694 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4a5e443-a1d7-4882-b86c-e6022b980073-istio-envoy\") pod \"e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc\" (UID: \"c4a5e443-a1d7-4882-b86c-e6022b980073\") " pod="openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc" Oct 31 04:39:04.581033 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:04.581013 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4a5e443-a1d7-4882-b86c-e6022b980073-workload-certs\") pod \"e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc\" (UID: \"c4a5e443-a1d7-4882-b86c-e6022b980073\") " pod="openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc" Oct 31 04:39:04.581147 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:04.581131 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djpnm\" (UniqueName: \"kubernetes.io/projected/c4a5e443-a1d7-4882-b86c-e6022b980073-kube-api-access-djpnm\") pod \"e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc\" (UID: \"c4a5e443-a1d7-4882-b86c-e6022b980073\") " pod="openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc" Oct 31 04:39:04.581340 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:04.581276 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4a5e443-a1d7-4882-b86c-e6022b980073-istio-podinfo\") pod \"e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc\" (UID: \"c4a5e443-a1d7-4882-b86c-e6022b980073\") " pod="openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc" Oct 31 04:39:04.585311 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:04.585277 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4a5e443-a1d7-4882-b86c-e6022b980073-istio-data\") pod \"e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc\" (UID: \"c4a5e443-a1d7-4882-b86c-e6022b980073\") " pod="openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc" Oct 31 04:39:04.585553 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:04.585374 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4a5e443-a1d7-4882-b86c-e6022b980073-istio-token\") pod \"e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc\" (UID: \"c4a5e443-a1d7-4882-b86c-e6022b980073\") " pod="openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc" Oct 31 04:39:04.585553 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:04.585507 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4a5e443-a1d7-4882-b86c-e6022b980073-istiod-ca-cert\") pod \"e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc\" (UID: \"c4a5e443-a1d7-4882-b86c-e6022b980073\") " pod="openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc" Oct 31 04:39:04.585675 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:04.585628 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4a5e443-a1d7-4882-b86c-e6022b980073-workload-socket\") pod \"e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc\" (UID: \"c4a5e443-a1d7-4882-b86c-e6022b980073\") " pod="openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc" Oct 31 04:39:04.585741 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:04.585681 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4a5e443-a1d7-4882-b86c-e6022b980073-credential-socket\") pod \"e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc\" (UID: \"c4a5e443-a1d7-4882-b86c-e6022b980073\") " pod="openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc" Oct 31 04:39:04.586861 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:04.586827 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc"] Oct 31 04:39:04.695684 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:04.695606 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4a5e443-a1d7-4882-b86c-e6022b980073-istio-data\") pod \"e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc\" (UID: \"c4a5e443-a1d7-4882-b86c-e6022b980073\") " pod="openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc" Oct 31 04:39:04.696007 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:04.695983 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4a5e443-a1d7-4882-b86c-e6022b980073-istio-token\") pod \"e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc\" (UID: \"c4a5e443-a1d7-4882-b86c-e6022b980073\") " pod="openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc" Oct 31 04:39:04.696172 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:04.696155 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4a5e443-a1d7-4882-b86c-e6022b980073-istiod-ca-cert\") pod \"e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc\" (UID: \"c4a5e443-a1d7-4882-b86c-e6022b980073\") " pod="openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc" Oct 31 04:39:04.696363 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:04.696347 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4a5e443-a1d7-4882-b86c-e6022b980073-workload-socket\") pod \"e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc\" (UID: \"c4a5e443-a1d7-4882-b86c-e6022b980073\") " pod="openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc" Oct 31 04:39:04.696569 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:04.696550 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4a5e443-a1d7-4882-b86c-e6022b980073-credential-socket\") pod \"e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc\" (UID: \"c4a5e443-a1d7-4882-b86c-e6022b980073\") " pod="openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc" Oct 31 04:39:04.696942 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:04.696921 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4a5e443-a1d7-4882-b86c-e6022b980073-istio-envoy\") pod \"e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc\" (UID: \"c4a5e443-a1d7-4882-b86c-e6022b980073\") " pod="openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc" Oct 31 04:39:04.698356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:04.698303 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4a5e443-a1d7-4882-b86c-e6022b980073-workload-certs\") pod \"e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc\" (UID: \"c4a5e443-a1d7-4882-b86c-e6022b980073\") " pod="openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc" Oct 31 04:39:04.698609 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:04.698565 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djpnm\" (UniqueName: \"kubernetes.io/projected/c4a5e443-a1d7-4882-b86c-e6022b980073-kube-api-access-djpnm\") pod \"e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc\" (UID: \"c4a5e443-a1d7-4882-b86c-e6022b980073\") " pod="openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc" Oct 31 04:39:04.698835 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:04.698802 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4a5e443-a1d7-4882-b86c-e6022b980073-istio-podinfo\") pod \"e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc\" (UID: \"c4a5e443-a1d7-4882-b86c-e6022b980073\") " pod="openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc" Oct 31 04:39:04.701720 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:04.699990 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4a5e443-a1d7-4882-b86c-e6022b980073-workload-socket\") pod \"e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc\" (UID: \"c4a5e443-a1d7-4882-b86c-e6022b980073\") " pod="openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc" Oct 31 04:39:04.701720 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:04.700262 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4a5e443-a1d7-4882-b86c-e6022b980073-istio-data\") pod \"e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc\" (UID: \"c4a5e443-a1d7-4882-b86c-e6022b980073\") " pod="openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc" Oct 31 04:39:04.701720 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:04.701311 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4a5e443-a1d7-4882-b86c-e6022b980073-istiod-ca-cert\") pod \"e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc\" (UID: \"c4a5e443-a1d7-4882-b86c-e6022b980073\") " pod="openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc" Oct 31 04:39:04.722395 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:04.712431 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4a5e443-a1d7-4882-b86c-e6022b980073-credential-socket\") pod \"e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc\" (UID: \"c4a5e443-a1d7-4882-b86c-e6022b980073\") " pod="openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc" Oct 31 04:39:04.722395 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:04.712773 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4a5e443-a1d7-4882-b86c-e6022b980073-workload-certs\") pod \"e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc\" (UID: \"c4a5e443-a1d7-4882-b86c-e6022b980073\") " pod="openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc" Oct 31 04:39:04.738841 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:04.738743 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4a5e443-a1d7-4882-b86c-e6022b980073-istio-envoy\") pod \"e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc\" (UID: \"c4a5e443-a1d7-4882-b86c-e6022b980073\") " pod="openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc" Oct 31 04:39:04.748377 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:04.747910 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4a5e443-a1d7-4882-b86c-e6022b980073-istio-podinfo\") pod \"e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc\" (UID: \"c4a5e443-a1d7-4882-b86c-e6022b980073\") " pod="openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc" Oct 31 04:39:04.762378 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:04.762020 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4a5e443-a1d7-4882-b86c-e6022b980073-istio-token\") pod \"e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc\" (UID: \"c4a5e443-a1d7-4882-b86c-e6022b980073\") " pod="openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc" Oct 31 04:39:04.820286 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:04.799092 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djpnm\" (UniqueName: \"kubernetes.io/projected/c4a5e443-a1d7-4882-b86c-e6022b980073-kube-api-access-djpnm\") pod \"e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc\" (UID: \"c4a5e443-a1d7-4882-b86c-e6022b980073\") " pod="openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc" Oct 31 04:39:04.890816 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:04.890369 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc" Oct 31 04:39:05.235267 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:39:05.235227 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4a5e443_a1d7_4882_b86c_e6022b980073.slice/crio-c592e5d5dcd3fbde67f0437356cdadee112347f0961d69568542d6661e05000a WatchSource:0}: Error finding container c592e5d5dcd3fbde67f0437356cdadee112347f0961d69568542d6661e05000a: Status 404 returned error can't find the container with id c592e5d5dcd3fbde67f0437356cdadee112347f0961d69568542d6661e05000a Oct 31 04:39:05.236668 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:05.236635 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc"] Oct 31 04:39:05.238737 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:05.238690 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:39:05.238841 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:05.238792 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:39:05.238887 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:05.238840 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:39:05.602145 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:05.601978 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc" event={"ID":"c4a5e443-a1d7-4882-b86c-e6022b980073","Type":"ContainerStarted","Data":"19b5d18539ac54474e37bed439fc85ae4f3767304b0d9d34948706c493b7a752"} Oct 31 04:39:05.602761 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:05.602733 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc" event={"ID":"c4a5e443-a1d7-4882-b86c-e6022b980073","Type":"ContainerStarted","Data":"c592e5d5dcd3fbde67f0437356cdadee112347f0961d69568542d6661e05000a"} Oct 31 04:39:05.654165 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:05.654077 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-b2a372/auth-test-model-b2a372-kserve-85c4657f6f-7gst5" podUID="38968c8c-ec31-40df-9e63-6324e59ae601" containerName="main" containerID="cri-o://9bf2324ab82fd6d829cc6054ce9e619df2ebf1999f2c496495ca3178c7d112db" gracePeriod=45 Oct 31 04:39:05.695517 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:05.695463 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc" podStartSLOduration=1.6954472329999999 podStartE2EDuration="1.695447233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:39:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:39:05.69082493 +0000 UTC m=+2100.447162962" watchObservedRunningTime="2026-10-31 04:39:05.695447233 +0000 UTC m=+2100.451785273" Oct 31 04:39:05.893356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:05.890736 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc" Oct 31 04:39:06.035875 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:06.035827 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b2a372/auth-test-model-b2a372-kserve-85c4657f6f-7gst5" Oct 31 04:39:06.126682 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:06.126572 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38968c8c-ec31-40df-9e63-6324e59ae601-kube-api-access-wwccn\" (UniqueName: \"kubernetes.io/projected/38968c8c-ec31-40df-9e63-6324e59ae601-kube-api-access-wwccn\") pod \"38968c8c-ec31-40df-9e63-6324e59ae601\" (UID: \"38968c8c-ec31-40df-9e63-6324e59ae601\") " Oct 31 04:39:06.126892 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:06.126683 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38968c8c-ec31-40df-9e63-6324e59ae601-dshm\" (UniqueName: \"kubernetes.io/empty-dir/38968c8c-ec31-40df-9e63-6324e59ae601-dshm\") pod \"38968c8c-ec31-40df-9e63-6324e59ae601\" (UID: \"38968c8c-ec31-40df-9e63-6324e59ae601\") " Oct 31 04:39:06.126892 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:06.126725 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38968c8c-ec31-40df-9e63-6324e59ae601-home\" (UniqueName: \"kubernetes.io/empty-dir/38968c8c-ec31-40df-9e63-6324e59ae601-home\") pod \"38968c8c-ec31-40df-9e63-6324e59ae601\" (UID: \"38968c8c-ec31-40df-9e63-6324e59ae601\") " Oct 31 04:39:06.126892 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:06.126774 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38968c8c-ec31-40df-9e63-6324e59ae601-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38968c8c-ec31-40df-9e63-6324e59ae601-tmp-dir\") pod \"38968c8c-ec31-40df-9e63-6324e59ae601\" (UID: \"38968c8c-ec31-40df-9e63-6324e59ae601\") " Oct 31 04:39:06.127105 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:06.126930 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38968c8c-ec31-40df-9e63-6324e59ae601-tls-certs\" (UniqueName: \"kubernetes.io/secret/38968c8c-ec31-40df-9e63-6324e59ae601-tls-certs\") pod \"38968c8c-ec31-40df-9e63-6324e59ae601\" (UID: \"38968c8c-ec31-40df-9e63-6324e59ae601\") " Oct 31 04:39:06.127105 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:06.127038 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38968c8c-ec31-40df-9e63-6324e59ae601-home" pod "38968c8c-ec31-40df-9e63-6324e59ae601" (UID: "38968c8c-ec31-40df-9e63-6324e59ae601"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:39:06.127105 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:06.127064 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38968c8c-ec31-40df-9e63-6324e59ae601-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38968c8c-ec31-40df-9e63-6324e59ae601-model-cache\") pod \"38968c8c-ec31-40df-9e63-6324e59ae601\" (UID: \"38968c8c-ec31-40df-9e63-6324e59ae601\") " Oct 31 04:39:06.127369 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:06.127277 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38968c8c-ec31-40df-9e63-6324e59ae601-model-cache" pod "38968c8c-ec31-40df-9e63-6324e59ae601" (UID: "38968c8c-ec31-40df-9e63-6324e59ae601"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:39:06.127489 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:06.127454 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38968c8c-ec31-40df-9e63-6324e59ae601-tmp-dir" pod "38968c8c-ec31-40df-9e63-6324e59ae601" (UID: "38968c8c-ec31-40df-9e63-6324e59ae601"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:39:06.127890 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:06.127869 5019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38968c8c-ec31-40df-9e63-6324e59ae601-model-cache\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:39:06.127979 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:06.127893 5019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38968c8c-ec31-40df-9e63-6324e59ae601-home\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:39:06.127979 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:06.127906 5019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38968c8c-ec31-40df-9e63-6324e59ae601-tmp-dir\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:39:06.129194 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:06.129169 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38968c8c-ec31-40df-9e63-6324e59ae601-dshm" pod "38968c8c-ec31-40df-9e63-6324e59ae601" (UID: "38968c8c-ec31-40df-9e63-6324e59ae601"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:39:06.129527 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:06.129495 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38968c8c-ec31-40df-9e63-6324e59ae601-tls-certs" pod "38968c8c-ec31-40df-9e63-6324e59ae601" (UID: "38968c8c-ec31-40df-9e63-6324e59ae601"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:39:06.129622 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:06.129547 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38968c8c-ec31-40df-9e63-6324e59ae601-kube-api-access-wwccn" pod "38968c8c-ec31-40df-9e63-6324e59ae601" (UID: "38968c8c-ec31-40df-9e63-6324e59ae601"). InnerVolumeSpecName "kube-api-access-wwccn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:39:06.241357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:06.240694 5019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38968c8c-ec31-40df-9e63-6324e59ae601-tls-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:39:06.241357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:06.240734 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwccn\" (UniqueName: \"kubernetes.io/projected/38968c8c-ec31-40df-9e63-6324e59ae601-kube-api-access-wwccn\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:39:06.241357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:06.240752 5019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38968c8c-ec31-40df-9e63-6324e59ae601-dshm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:39:06.612088 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:06.612039 5019 generic.go:356] "Generic (PLEG): container finished" podID="38968c8c-ec31-40df-9e63-6324e59ae601" containerID="9bf2324ab82fd6d829cc6054ce9e619df2ebf1999f2c496495ca3178c7d112db" exitCode=0 Oct 31 04:39:06.612667 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:06.612144 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b2a372/auth-test-model-b2a372-kserve-85c4657f6f-7gst5" Oct 31 04:39:06.612667 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:06.612179 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b2a372/auth-test-model-b2a372-kserve-85c4657f6f-7gst5" event={"ID":"38968c8c-ec31-40df-9e63-6324e59ae601","Type":"ContainerDied","Data":"9bf2324ab82fd6d829cc6054ce9e619df2ebf1999f2c496495ca3178c7d112db"} Oct 31 04:39:06.612667 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:06.612236 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b2a372/auth-test-model-b2a372-kserve-85c4657f6f-7gst5" event={"ID":"38968c8c-ec31-40df-9e63-6324e59ae601","Type":"ContainerDied","Data":"26c81db2c06c05f1dfacb34c84d32dc252c7037522a1117d9fd186c78c9468a3"} Oct 31 04:39:06.612667 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:06.612260 5019 scope.go:122] "RemoveContainer" containerID="9bf2324ab82fd6d829cc6054ce9e619df2ebf1999f2c496495ca3178c7d112db" Oct 31 04:39:06.627041 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:06.627007 5019 scope.go:122] "RemoveContainer" containerID="9bf2324ab82fd6d829cc6054ce9e619df2ebf1999f2c496495ca3178c7d112db" Oct 31 04:39:06.627506 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:39:06.627473 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bf2324ab82fd6d829cc6054ce9e619df2ebf1999f2c496495ca3178c7d112db\": container with ID starting with 9bf2324ab82fd6d829cc6054ce9e619df2ebf1999f2c496495ca3178c7d112db not found: ID does not exist" containerID="9bf2324ab82fd6d829cc6054ce9e619df2ebf1999f2c496495ca3178c7d112db" Oct 31 04:39:06.627633 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:06.627516 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bf2324ab82fd6d829cc6054ce9e619df2ebf1999f2c496495ca3178c7d112db"} err="failed to get container status \"9bf2324ab82fd6d829cc6054ce9e619df2ebf1999f2c496495ca3178c7d112db\": rpc error: code = NotFound desc = could not find container \"9bf2324ab82fd6d829cc6054ce9e619df2ebf1999f2c496495ca3178c7d112db\": container with ID starting with 9bf2324ab82fd6d829cc6054ce9e619df2ebf1999f2c496495ca3178c7d112db not found: ID does not exist" Oct 31 04:39:06.665997 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:06.665958 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b2a372/auth-test-model-b2a372-kserve-85c4657f6f-7gst5"] Oct 31 04:39:06.680177 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:06.680128 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b2a372/auth-test-model-b2a372-kserve-85c4657f6f-7gst5"] Oct 31 04:39:06.899220 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:06.898732 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc" Oct 31 04:39:06.909948 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:06.909894 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-b2a372/auth-test-model-b2a372-kserve-85c4657f6f-7gst5" podUID="38968c8c-ec31-40df-9e63-6324e59ae601" containerName="main" probeResult="failure" output="Get \"https://10.132.0.255:8000/ready\": context deadline exceeded" Oct 31 04:39:07.368894 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:07.368856 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h" Oct 31 04:39:07.410308 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:07.410270 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38968c8c-ec31-40df-9e63-6324e59ae601" path="/var/lib/kubelet/pods/38968c8c-ec31-40df-9e63-6324e59ae601/volumes" Oct 31 04:39:07.460184 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:07.460073 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/686f7dd0-02cd-457a-b7cd-ef83520d5341-kube-api-access-8kxpn\" (UniqueName: \"kubernetes.io/projected/686f7dd0-02cd-457a-b7cd-ef83520d5341-kube-api-access-8kxpn\") pod \"686f7dd0-02cd-457a-b7cd-ef83520d5341\" (UID: \"686f7dd0-02cd-457a-b7cd-ef83520d5341\") " Oct 31 04:39:07.460184 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:07.460176 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/686f7dd0-02cd-457a-b7cd-ef83520d5341-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/686f7dd0-02cd-457a-b7cd-ef83520d5341-credential-socket\") pod \"686f7dd0-02cd-457a-b7cd-ef83520d5341\" (UID: \"686f7dd0-02cd-457a-b7cd-ef83520d5341\") " Oct 31 04:39:07.460480 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:07.460280 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/686f7dd0-02cd-457a-b7cd-ef83520d5341-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/686f7dd0-02cd-457a-b7cd-ef83520d5341-istiod-ca-cert\") pod \"686f7dd0-02cd-457a-b7cd-ef83520d5341\" (UID: \"686f7dd0-02cd-457a-b7cd-ef83520d5341\") " Oct 31 04:39:07.460480 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:07.460385 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/686f7dd0-02cd-457a-b7cd-ef83520d5341-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/686f7dd0-02cd-457a-b7cd-ef83520d5341-istio-podinfo\") pod \"686f7dd0-02cd-457a-b7cd-ef83520d5341\" (UID: \"686f7dd0-02cd-457a-b7cd-ef83520d5341\") " Oct 31 04:39:07.460480 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:07.460429 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/686f7dd0-02cd-457a-b7cd-ef83520d5341-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/686f7dd0-02cd-457a-b7cd-ef83520d5341-workload-certs\") pod \"686f7dd0-02cd-457a-b7cd-ef83520d5341\" (UID: \"686f7dd0-02cd-457a-b7cd-ef83520d5341\") " Oct 31 04:39:07.460480 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:07.460462 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/686f7dd0-02cd-457a-b7cd-ef83520d5341-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/686f7dd0-02cd-457a-b7cd-ef83520d5341-istio-data\") pod \"686f7dd0-02cd-457a-b7cd-ef83520d5341\" (UID: \"686f7dd0-02cd-457a-b7cd-ef83520d5341\") " Oct 31 04:39:07.461914 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:07.460700 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/686f7dd0-02cd-457a-b7cd-ef83520d5341-istio-token\" (UniqueName: \"kubernetes.io/projected/686f7dd0-02cd-457a-b7cd-ef83520d5341-istio-token\") pod \"686f7dd0-02cd-457a-b7cd-ef83520d5341\" (UID: \"686f7dd0-02cd-457a-b7cd-ef83520d5341\") " Oct 31 04:39:07.461914 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:07.460799 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/686f7dd0-02cd-457a-b7cd-ef83520d5341-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/686f7dd0-02cd-457a-b7cd-ef83520d5341-workload-socket\") pod \"686f7dd0-02cd-457a-b7cd-ef83520d5341\" (UID: \"686f7dd0-02cd-457a-b7cd-ef83520d5341\") " Oct 31 04:39:07.461914 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:07.460800 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/686f7dd0-02cd-457a-b7cd-ef83520d5341-istiod-ca-cert" pod "686f7dd0-02cd-457a-b7cd-ef83520d5341" (UID: "686f7dd0-02cd-457a-b7cd-ef83520d5341"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:39:07.461914 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:07.460832 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/686f7dd0-02cd-457a-b7cd-ef83520d5341-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/686f7dd0-02cd-457a-b7cd-ef83520d5341-istio-envoy\") pod \"686f7dd0-02cd-457a-b7cd-ef83520d5341\" (UID: \"686f7dd0-02cd-457a-b7cd-ef83520d5341\") " Oct 31 04:39:07.461914 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:07.461091 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/686f7dd0-02cd-457a-b7cd-ef83520d5341-credential-socket" pod "686f7dd0-02cd-457a-b7cd-ef83520d5341" (UID: "686f7dd0-02cd-457a-b7cd-ef83520d5341"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:39:07.461914 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:07.461732 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/686f7dd0-02cd-457a-b7cd-ef83520d5341-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:39:07.461914 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:07.461758 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/686f7dd0-02cd-457a-b7cd-ef83520d5341-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:39:07.462300 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:07.462000 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/686f7dd0-02cd-457a-b7cd-ef83520d5341-istio-data" pod "686f7dd0-02cd-457a-b7cd-ef83520d5341" (UID: "686f7dd0-02cd-457a-b7cd-ef83520d5341"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:39:07.462848 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:07.462811 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/686f7dd0-02cd-457a-b7cd-ef83520d5341-workload-socket" pod "686f7dd0-02cd-457a-b7cd-ef83520d5341" (UID: "686f7dd0-02cd-457a-b7cd-ef83520d5341"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:39:07.463255 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:07.463208 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/686f7dd0-02cd-457a-b7cd-ef83520d5341-workload-certs" pod "686f7dd0-02cd-457a-b7cd-ef83520d5341" (UID: "686f7dd0-02cd-457a-b7cd-ef83520d5341"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:39:07.463964 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:07.463939 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/686f7dd0-02cd-457a-b7cd-ef83520d5341-kube-api-access-8kxpn" pod "686f7dd0-02cd-457a-b7cd-ef83520d5341" (UID: "686f7dd0-02cd-457a-b7cd-ef83520d5341"). InnerVolumeSpecName "kube-api-access-8kxpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:39:07.464449 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:07.464423 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/686f7dd0-02cd-457a-b7cd-ef83520d5341-istio-envoy" pod "686f7dd0-02cd-457a-b7cd-ef83520d5341" (UID: "686f7dd0-02cd-457a-b7cd-ef83520d5341"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:39:07.465211 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:07.465184 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/686f7dd0-02cd-457a-b7cd-ef83520d5341-istio-podinfo" pod "686f7dd0-02cd-457a-b7cd-ef83520d5341" (UID: "686f7dd0-02cd-457a-b7cd-ef83520d5341"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:39:07.466516 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:07.466484 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/686f7dd0-02cd-457a-b7cd-ef83520d5341-istio-token" pod "686f7dd0-02cd-457a-b7cd-ef83520d5341" (UID: "686f7dd0-02cd-457a-b7cd-ef83520d5341"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:39:07.563412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:07.563373 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kxpn\" (UniqueName: \"kubernetes.io/projected/686f7dd0-02cd-457a-b7cd-ef83520d5341-kube-api-access-8kxpn\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:39:07.563412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:07.563413 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/686f7dd0-02cd-457a-b7cd-ef83520d5341-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:39:07.563677 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:07.563427 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/686f7dd0-02cd-457a-b7cd-ef83520d5341-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:39:07.563677 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:07.563439 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/686f7dd0-02cd-457a-b7cd-ef83520d5341-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:39:07.563677 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:07.563449 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/686f7dd0-02cd-457a-b7cd-ef83520d5341-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:39:07.563677 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:07.563461 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/686f7dd0-02cd-457a-b7cd-ef83520d5341-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:39:07.563677 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:07.563471 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/686f7dd0-02cd-457a-b7cd-ef83520d5341-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:39:07.627989 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:07.627944 5019 generic.go:356] "Generic (PLEG): container finished" podID="686f7dd0-02cd-457a-b7cd-ef83520d5341" containerID="bcea24c66148803cee1eba69713d84e38d9b2d233d5a8dabc9a5958f7c962a3d" exitCode=0 Oct 31 04:39:07.628513 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:07.628016 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h" Oct 31 04:39:07.628513 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:07.628032 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h" event={"ID":"686f7dd0-02cd-457a-b7cd-ef83520d5341","Type":"ContainerDied","Data":"bcea24c66148803cee1eba69713d84e38d9b2d233d5a8dabc9a5958f7c962a3d"} Oct 31 04:39:07.628513 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:07.628126 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h" event={"ID":"686f7dd0-02cd-457a-b7cd-ef83520d5341","Type":"ContainerDied","Data":"4051220cd229960ae5eea2a7463f82f6baa0ea7757f88ed1f6ad48485d4be6c4"} Oct 31 04:39:07.628513 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:07.628181 5019 scope.go:122] "RemoveContainer" containerID="bcea24c66148803cee1eba69713d84e38d9b2d233d5a8dabc9a5958f7c962a3d" Oct 31 04:39:07.629255 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:07.629125 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc" Oct 31 04:39:07.630671 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:07.630631 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc" Oct 31 04:39:07.652584 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:07.651790 5019 scope.go:122] "RemoveContainer" containerID="bcea24c66148803cee1eba69713d84e38d9b2d233d5a8dabc9a5958f7c962a3d" Oct 31 04:39:07.653363 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:39:07.652740 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcea24c66148803cee1eba69713d84e38d9b2d233d5a8dabc9a5958f7c962a3d\": container with ID starting with bcea24c66148803cee1eba69713d84e38d9b2d233d5a8dabc9a5958f7c962a3d not found: ID does not exist" containerID="bcea24c66148803cee1eba69713d84e38d9b2d233d5a8dabc9a5958f7c962a3d" Oct 31 04:39:07.653363 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:07.652787 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcea24c66148803cee1eba69713d84e38d9b2d233d5a8dabc9a5958f7c962a3d"} err="failed to get container status \"bcea24c66148803cee1eba69713d84e38d9b2d233d5a8dabc9a5958f7c962a3d\": rpc error: code = NotFound desc = could not find container \"bcea24c66148803cee1eba69713d84e38d9b2d233d5a8dabc9a5958f7c962a3d\": container with ID starting with bcea24c66148803cee1eba69713d84e38d9b2d233d5a8dabc9a5958f7c962a3d not found: ID does not exist" Oct 31 04:39:07.736839 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:07.736789 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h"] Oct 31 04:39:07.744123 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:07.744076 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-qvp6h"] Oct 31 04:39:08.020147 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:08.020058 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s88kp_3945ff65-e877-4372-9334-3968c68b7295/console-operator/2.log" Oct 31 04:39:08.085896 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:08.085866 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s88kp_3945ff65-e877-4372-9334-3968c68b7295/console-operator/2.log" Oct 31 04:39:09.409700 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:09.409657 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="686f7dd0-02cd-457a-b7cd-ef83520d5341" path="/var/lib/kubelet/pods/686f7dd0-02cd-457a-b7cd-ef83520d5341/volumes" Oct 31 04:39:09.680111 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:09.680023 5019 scope.go:122] "RemoveContainer" containerID="cc00f52184f4fbccd1e6c56314d9e32013e6aabfe3a65d44ca2c670f7efbf0d8" Oct 31 04:39:09.691652 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:09.691616 5019 scope.go:122] "RemoveContainer" containerID="637c16d6733b797aeb8e045bb9452c904d29084b0c3c9baa6f61f09add36a310" Oct 31 04:39:09.703039 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:09.703010 5019 scope.go:122] "RemoveContainer" containerID="dc57f9b7fbadc5b8b5b4c9623ba18760c6125a51c5c94e52989017899c1b1f11" Oct 31 04:39:09.714132 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:09.714100 5019 scope.go:122] "RemoveContainer" containerID="1ffe1b04522ca5bea64a747fab2f0df947208ba88bfc2024c72a7188216e64b7" Oct 31 04:39:11.872589 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:11.872540 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj"] Oct 31 04:39:11.873186 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:11.872864 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj" podUID="ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77" containerName="istio-proxy" containerID="cri-o://7eb4cea754b9697f55742b13bf2a141ed71dd6ee7b2237ba73c698a9f0452a45" gracePeriod=30 Oct 31 04:39:11.937058 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:11.937006 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm"] Oct 31 04:39:11.937447 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:11.937408 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm" podUID="441c6ad5-5141-4b8e-b9db-9497f594f479" containerName="istio-proxy" containerID="cri-o://7a3961cea39f27cda42d781e29232cfe4c3ae6f605bd35ae90df10b6d3d39bc3" gracePeriod=30 Oct 31 04:39:17.272596 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.272555 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj" Oct 31 04:39:17.314699 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.314667 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm" Oct 31 04:39:17.398081 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.398039 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-istio-token\" (UniqueName: \"kubernetes.io/projected/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-istio-token\") pod \"ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77\" (UID: \"ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77\") " Oct 31 04:39:17.398300 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.398096 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/441c6ad5-5141-4b8e-b9db-9497f594f479-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/441c6ad5-5141-4b8e-b9db-9497f594f479-workload-certs\") pod \"441c6ad5-5141-4b8e-b9db-9497f594f479\" (UID: \"441c6ad5-5141-4b8e-b9db-9497f594f479\") " Oct 31 04:39:17.398300 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.398148 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-istio-data\") pod \"ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77\" (UID: \"ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77\") " Oct 31 04:39:17.398300 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.398207 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-istiod-ca-cert\") pod \"ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77\" (UID: \"ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77\") " Oct 31 04:39:17.398300 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.398239 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-istio-envoy\") pod \"ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77\" (UID: \"ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77\") " Oct 31 04:39:17.398300 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.398277 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/441c6ad5-5141-4b8e-b9db-9497f594f479-istio-token\" (UniqueName: \"kubernetes.io/projected/441c6ad5-5141-4b8e-b9db-9497f594f479-istio-token\") pod \"441c6ad5-5141-4b8e-b9db-9497f594f479\" (UID: \"441c6ad5-5141-4b8e-b9db-9497f594f479\") " Oct 31 04:39:17.398640 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.398321 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-workload-socket\") pod \"ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77\" (UID: \"ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77\") " Oct 31 04:39:17.398640 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.398378 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/441c6ad5-5141-4b8e-b9db-9497f594f479-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/441c6ad5-5141-4b8e-b9db-9497f594f479-istio-data\") pod \"441c6ad5-5141-4b8e-b9db-9497f594f479\" (UID: \"441c6ad5-5141-4b8e-b9db-9497f594f479\") " Oct 31 04:39:17.398640 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.398451 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/441c6ad5-5141-4b8e-b9db-9497f594f479-workload-certs" pod "441c6ad5-5141-4b8e-b9db-9497f594f479" (UID: "441c6ad5-5141-4b8e-b9db-9497f594f479"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:39:17.398640 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.398471 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/441c6ad5-5141-4b8e-b9db-9497f594f479-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/441c6ad5-5141-4b8e-b9db-9497f594f479-istio-podinfo\") pod \"441c6ad5-5141-4b8e-b9db-9497f594f479\" (UID: \"441c6ad5-5141-4b8e-b9db-9497f594f479\") " Oct 31 04:39:17.398640 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.398509 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/441c6ad5-5141-4b8e-b9db-9497f594f479-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/441c6ad5-5141-4b8e-b9db-9497f594f479-istiod-ca-cert\") pod \"441c6ad5-5141-4b8e-b9db-9497f594f479\" (UID: \"441c6ad5-5141-4b8e-b9db-9497f594f479\") " Oct 31 04:39:17.398640 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.398636 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/441c6ad5-5141-4b8e-b9db-9497f594f479-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/441c6ad5-5141-4b8e-b9db-9497f594f479-workload-socket\") pod \"441c6ad5-5141-4b8e-b9db-9497f594f479\" (UID: \"441c6ad5-5141-4b8e-b9db-9497f594f479\") " Oct 31 04:39:17.399032 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.398684 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/441c6ad5-5141-4b8e-b9db-9497f594f479-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/441c6ad5-5141-4b8e-b9db-9497f594f479-istio-envoy\") pod \"441c6ad5-5141-4b8e-b9db-9497f594f479\" (UID: \"441c6ad5-5141-4b8e-b9db-9497f594f479\") " Oct 31 04:39:17.399032 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.398738 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-credential-socket\") pod \"ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77\" (UID: \"ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77\") " Oct 31 04:39:17.399032 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.398766 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/441c6ad5-5141-4b8e-b9db-9497f594f479-kube-api-access-67j6w\" (UniqueName: \"kubernetes.io/projected/441c6ad5-5141-4b8e-b9db-9497f594f479-kube-api-access-67j6w\") pod \"441c6ad5-5141-4b8e-b9db-9497f594f479\" (UID: \"441c6ad5-5141-4b8e-b9db-9497f594f479\") " Oct 31 04:39:17.399032 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.398799 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-workload-certs\") pod \"ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77\" (UID: \"ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77\") " Oct 31 04:39:17.399032 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.398839 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/441c6ad5-5141-4b8e-b9db-9497f594f479-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/441c6ad5-5141-4b8e-b9db-9497f594f479-credential-socket\") pod \"441c6ad5-5141-4b8e-b9db-9497f594f479\" (UID: \"441c6ad5-5141-4b8e-b9db-9497f594f479\") " Oct 31 04:39:17.399032 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.398976 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-kube-api-access-xvnq2\" (UniqueName: \"kubernetes.io/projected/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-kube-api-access-xvnq2\") pod \"ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77\" (UID: \"ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77\") " Oct 31 04:39:17.399032 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.399028 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-istio-podinfo\") pod \"ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77\" (UID: \"ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77\") " Oct 31 04:39:17.399500 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.399475 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/441c6ad5-5141-4b8e-b9db-9497f594f479-workload-socket" pod "441c6ad5-5141-4b8e-b9db-9497f594f479" (UID: "441c6ad5-5141-4b8e-b9db-9497f594f479"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:39:17.399749 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.399730 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/441c6ad5-5141-4b8e-b9db-9497f594f479-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:39:17.399835 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.399753 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/441c6ad5-5141-4b8e-b9db-9497f594f479-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:39:17.399906 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.399870 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-istio-data" pod "ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77" (UID: "ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:39:17.400049 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.400028 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-workload-certs" pod "ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77" (UID: "ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:39:17.400229 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.400212 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/441c6ad5-5141-4b8e-b9db-9497f594f479-credential-socket" pod "441c6ad5-5141-4b8e-b9db-9497f594f479" (UID: "441c6ad5-5141-4b8e-b9db-9497f594f479"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:39:17.400292 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.400265 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-istiod-ca-cert" pod "ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77" (UID: "ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:39:17.400697 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.400541 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-credential-socket" pod "ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77" (UID: "ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:39:17.401007 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.400851 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/441c6ad5-5141-4b8e-b9db-9497f594f479-istio-data" pod "441c6ad5-5141-4b8e-b9db-9497f594f479" (UID: "441c6ad5-5141-4b8e-b9db-9497f594f479"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:39:17.401398 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.401116 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-workload-socket" pod "ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77" (UID: "ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:39:17.403137 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.403078 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-kube-api-access-xvnq2" pod "ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77" (UID: "ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77"). InnerVolumeSpecName "kube-api-access-xvnq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:39:17.405846 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.405813 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-istio-podinfo" pod "ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77" (UID: "ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:39:17.406210 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.406178 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/441c6ad5-5141-4b8e-b9db-9497f594f479-istiod-ca-cert" pod "441c6ad5-5141-4b8e-b9db-9497f594f479" (UID: "441c6ad5-5141-4b8e-b9db-9497f594f479"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:39:17.409846 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.409754 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/441c6ad5-5141-4b8e-b9db-9497f594f479-istio-podinfo" pod "441c6ad5-5141-4b8e-b9db-9497f594f479" (UID: "441c6ad5-5141-4b8e-b9db-9497f594f479"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:39:17.410858 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.410774 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/441c6ad5-5141-4b8e-b9db-9497f594f479-istio-token" pod "441c6ad5-5141-4b8e-b9db-9497f594f479" (UID: "441c6ad5-5141-4b8e-b9db-9497f594f479"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:39:17.410984 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.410817 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-istio-envoy" pod "ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77" (UID: "ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:39:17.411141 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.410834 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/441c6ad5-5141-4b8e-b9db-9497f594f479-kube-api-access-67j6w" pod "441c6ad5-5141-4b8e-b9db-9497f594f479" (UID: "441c6ad5-5141-4b8e-b9db-9497f594f479"). InnerVolumeSpecName "kube-api-access-67j6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:39:17.411544 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.411487 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-istio-token" pod "ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77" (UID: "ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:39:17.414553 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.414522 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/441c6ad5-5141-4b8e-b9db-9497f594f479-istio-envoy" pod "441c6ad5-5141-4b8e-b9db-9497f594f479" (UID: "441c6ad5-5141-4b8e-b9db-9497f594f479"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:39:17.501897 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.501851 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:39:17.501897 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.501897 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:39:17.502119 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.501910 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:39:17.502119 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.501923 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:39:17.502119 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.501936 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:39:17.502119 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.501950 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/441c6ad5-5141-4b8e-b9db-9497f594f479-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:39:17.502119 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.501962 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:39:17.502119 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.501977 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/441c6ad5-5141-4b8e-b9db-9497f594f479-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:39:17.502119 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.501991 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/441c6ad5-5141-4b8e-b9db-9497f594f479-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:39:17.502119 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.502004 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/441c6ad5-5141-4b8e-b9db-9497f594f479-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:39:17.502119 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.502017 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/441c6ad5-5141-4b8e-b9db-9497f594f479-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:39:17.502119 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.502029 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:39:17.502119 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.502044 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67j6w\" (UniqueName: \"kubernetes.io/projected/441c6ad5-5141-4b8e-b9db-9497f594f479-kube-api-access-67j6w\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:39:17.502119 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.502062 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:39:17.502119 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.502075 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/441c6ad5-5141-4b8e-b9db-9497f594f479-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:39:17.502119 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.502089 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvnq2\" (UniqueName: \"kubernetes.io/projected/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77-kube-api-access-xvnq2\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:39:17.566650 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.566581 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-031745/auth-test-model-031745-kserve-7656c77f57-l86zb" podUID="3a265311-7ffc-488e-a829-8f43ee2e9a77" containerName="main" containerID="cri-o://29ed1fe2333e5e8733f869f3152d7b50e90e1309fec309ac55ddabdd229a316f" gracePeriod=45 Oct 31 04:39:17.671435 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.671392 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-031745/auth-test-model-031745-kserve-7656c77f57-l86zb" podUID="3a265311-7ffc-488e-a829-8f43ee2e9a77" containerName="main" probeResult="failure" output="Get \"https://10.132.0.254:8000/ready\": dial tcp 10.132.0.254:8000: connect: connection refused" Oct 31 04:39:17.735375 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.731381 5019 generic.go:356] "Generic (PLEG): container finished" podID="ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77" containerID="7eb4cea754b9697f55742b13bf2a141ed71dd6ee7b2237ba73c698a9f0452a45" exitCode=0 Oct 31 04:39:17.735375 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.731452 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj" Oct 31 04:39:17.735375 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.731459 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj" event={"ID":"ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77","Type":"ContainerDied","Data":"7eb4cea754b9697f55742b13bf2a141ed71dd6ee7b2237ba73c698a9f0452a45"} Oct 31 04:39:17.735375 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.731516 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj" event={"ID":"ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77","Type":"ContainerDied","Data":"77c11ebe6bde388cf63dd955d18b08c9c5ba158fe741db3a8a6e1dd4a3fdf04b"} Oct 31 04:39:17.735375 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.731540 5019 scope.go:122] "RemoveContainer" containerID="7eb4cea754b9697f55742b13bf2a141ed71dd6ee7b2237ba73c698a9f0452a45" Oct 31 04:39:17.735375 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.733595 5019 generic.go:356] "Generic (PLEG): container finished" podID="3a265311-7ffc-488e-a829-8f43ee2e9a77" containerID="29ed1fe2333e5e8733f869f3152d7b50e90e1309fec309ac55ddabdd229a316f" exitCode=0 Oct 31 04:39:17.735375 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.733625 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-031745/auth-test-model-031745-kserve-7656c77f57-l86zb" event={"ID":"3a265311-7ffc-488e-a829-8f43ee2e9a77","Type":"ContainerDied","Data":"29ed1fe2333e5e8733f869f3152d7b50e90e1309fec309ac55ddabdd229a316f"} Oct 31 04:39:17.735926 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.735524 5019 generic.go:356] "Generic (PLEG): container finished" podID="441c6ad5-5141-4b8e-b9db-9497f594f479" containerID="7a3961cea39f27cda42d781e29232cfe4c3ae6f605bd35ae90df10b6d3d39bc3" exitCode=0 Oct 31 04:39:17.735926 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.735570 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm" Oct 31 04:39:17.735926 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.735582 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm" event={"ID":"441c6ad5-5141-4b8e-b9db-9497f594f479","Type":"ContainerDied","Data":"7a3961cea39f27cda42d781e29232cfe4c3ae6f605bd35ae90df10b6d3d39bc3"} Oct 31 04:39:17.735926 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.735681 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm" event={"ID":"441c6ad5-5141-4b8e-b9db-9497f594f479","Type":"ContainerDied","Data":"8abb84d49d7a0e6158e4b84cc653845f22bd7caa98ecfa650a96aeba5efa7fee"} Oct 31 04:39:17.745671 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.745635 5019 scope.go:122] "RemoveContainer" containerID="7eb4cea754b9697f55742b13bf2a141ed71dd6ee7b2237ba73c698a9f0452a45" Oct 31 04:39:17.748752 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:39:17.748718 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7eb4cea754b9697f55742b13bf2a141ed71dd6ee7b2237ba73c698a9f0452a45\": container with ID starting with 7eb4cea754b9697f55742b13bf2a141ed71dd6ee7b2237ba73c698a9f0452a45 not found: ID does not exist" containerID="7eb4cea754b9697f55742b13bf2a141ed71dd6ee7b2237ba73c698a9f0452a45" Oct 31 04:39:17.748934 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.748759 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7eb4cea754b9697f55742b13bf2a141ed71dd6ee7b2237ba73c698a9f0452a45"} err="failed to get container status \"7eb4cea754b9697f55742b13bf2a141ed71dd6ee7b2237ba73c698a9f0452a45\": rpc error: code = NotFound desc = could not find container \"7eb4cea754b9697f55742b13bf2a141ed71dd6ee7b2237ba73c698a9f0452a45\": container with ID starting with 7eb4cea754b9697f55742b13bf2a141ed71dd6ee7b2237ba73c698a9f0452a45 not found: ID does not exist" Oct 31 04:39:17.748934 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.748785 5019 scope.go:122] "RemoveContainer" containerID="7a3961cea39f27cda42d781e29232cfe4c3ae6f605bd35ae90df10b6d3d39bc3" Oct 31 04:39:17.767807 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.767750 5019 scope.go:122] "RemoveContainer" containerID="7a3961cea39f27cda42d781e29232cfe4c3ae6f605bd35ae90df10b6d3d39bc3" Oct 31 04:39:17.768206 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:39:17.768180 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a3961cea39f27cda42d781e29232cfe4c3ae6f605bd35ae90df10b6d3d39bc3\": container with ID starting with 7a3961cea39f27cda42d781e29232cfe4c3ae6f605bd35ae90df10b6d3d39bc3 not found: ID does not exist" containerID="7a3961cea39f27cda42d781e29232cfe4c3ae6f605bd35ae90df10b6d3d39bc3" Oct 31 04:39:17.768409 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.768215 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a3961cea39f27cda42d781e29232cfe4c3ae6f605bd35ae90df10b6d3d39bc3"} err="failed to get container status \"7a3961cea39f27cda42d781e29232cfe4c3ae6f605bd35ae90df10b6d3d39bc3\": rpc error: code = NotFound desc = could not find container \"7a3961cea39f27cda42d781e29232cfe4c3ae6f605bd35ae90df10b6d3d39bc3\": container with ID starting with 7a3961cea39f27cda42d781e29232cfe4c3ae6f605bd35ae90df10b6d3d39bc3 not found: ID does not exist" Oct 31 04:39:17.800143 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.800103 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj"] Oct 31 04:39:17.806729 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.806686 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-f29tj"] Oct 31 04:39:17.830117 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.830079 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm"] Oct 31 04:39:17.835876 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.835837 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-tlqzm"] Oct 31 04:39:17.867681 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:17.867648 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-031745/auth-test-model-031745-kserve-7656c77f57-l86zb" Oct 31 04:39:18.009753 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:18.009698 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a265311-7ffc-488e-a829-8f43ee2e9a77-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a265311-7ffc-488e-a829-8f43ee2e9a77-model-cache\") pod \"3a265311-7ffc-488e-a829-8f43ee2e9a77\" (UID: \"3a265311-7ffc-488e-a829-8f43ee2e9a77\") " Oct 31 04:39:18.009753 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:18.009749 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a265311-7ffc-488e-a829-8f43ee2e9a77-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a265311-7ffc-488e-a829-8f43ee2e9a77-tmp-dir\") pod \"3a265311-7ffc-488e-a829-8f43ee2e9a77\" (UID: \"3a265311-7ffc-488e-a829-8f43ee2e9a77\") " Oct 31 04:39:18.010033 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:18.009806 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a265311-7ffc-488e-a829-8f43ee2e9a77-home\" (UniqueName: \"kubernetes.io/empty-dir/3a265311-7ffc-488e-a829-8f43ee2e9a77-home\") pod \"3a265311-7ffc-488e-a829-8f43ee2e9a77\" (UID: \"3a265311-7ffc-488e-a829-8f43ee2e9a77\") " Oct 31 04:39:18.010033 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:18.009914 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a265311-7ffc-488e-a829-8f43ee2e9a77-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a265311-7ffc-488e-a829-8f43ee2e9a77-dshm\") pod \"3a265311-7ffc-488e-a829-8f43ee2e9a77\" (UID: \"3a265311-7ffc-488e-a829-8f43ee2e9a77\") " Oct 31 04:39:18.010033 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:18.010015 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a265311-7ffc-488e-a829-8f43ee2e9a77-kube-api-access-p4tmm\" (UniqueName: \"kubernetes.io/projected/3a265311-7ffc-488e-a829-8f43ee2e9a77-kube-api-access-p4tmm\") pod \"3a265311-7ffc-488e-a829-8f43ee2e9a77\" (UID: \"3a265311-7ffc-488e-a829-8f43ee2e9a77\") " Oct 31 04:39:18.010205 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:18.010066 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a265311-7ffc-488e-a829-8f43ee2e9a77-model-cache" pod "3a265311-7ffc-488e-a829-8f43ee2e9a77" (UID: "3a265311-7ffc-488e-a829-8f43ee2e9a77"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:39:18.010205 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:18.010086 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a265311-7ffc-488e-a829-8f43ee2e9a77-tmp-dir" pod "3a265311-7ffc-488e-a829-8f43ee2e9a77" (UID: "3a265311-7ffc-488e-a829-8f43ee2e9a77"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:39:18.010205 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:18.010162 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a265311-7ffc-488e-a829-8f43ee2e9a77-tls-certs\" (UniqueName: \"kubernetes.io/secret/3a265311-7ffc-488e-a829-8f43ee2e9a77-tls-certs\") pod \"3a265311-7ffc-488e-a829-8f43ee2e9a77\" (UID: \"3a265311-7ffc-488e-a829-8f43ee2e9a77\") " Oct 31 04:39:18.010488 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:18.010464 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a265311-7ffc-488e-a829-8f43ee2e9a77-home" pod "3a265311-7ffc-488e-a829-8f43ee2e9a77" (UID: "3a265311-7ffc-488e-a829-8f43ee2e9a77"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:39:18.010915 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:18.010890 5019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a265311-7ffc-488e-a829-8f43ee2e9a77-model-cache\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:39:18.010915 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:18.010915 5019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a265311-7ffc-488e-a829-8f43ee2e9a77-tmp-dir\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:39:18.011111 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:18.010927 5019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a265311-7ffc-488e-a829-8f43ee2e9a77-home\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:39:18.013197 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:18.013167 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a265311-7ffc-488e-a829-8f43ee2e9a77-tls-certs" pod "3a265311-7ffc-488e-a829-8f43ee2e9a77" (UID: "3a265311-7ffc-488e-a829-8f43ee2e9a77"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:39:18.013402 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:18.013295 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a265311-7ffc-488e-a829-8f43ee2e9a77-kube-api-access-p4tmm" pod "3a265311-7ffc-488e-a829-8f43ee2e9a77" (UID: "3a265311-7ffc-488e-a829-8f43ee2e9a77"). InnerVolumeSpecName "kube-api-access-p4tmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:39:18.013402 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:18.013395 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a265311-7ffc-488e-a829-8f43ee2e9a77-dshm" pod "3a265311-7ffc-488e-a829-8f43ee2e9a77" (UID: "3a265311-7ffc-488e-a829-8f43ee2e9a77"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:39:18.112760 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:18.112720 5019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a265311-7ffc-488e-a829-8f43ee2e9a77-dshm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:39:18.112760 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:18.112759 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4tmm\" (UniqueName: \"kubernetes.io/projected/3a265311-7ffc-488e-a829-8f43ee2e9a77-kube-api-access-p4tmm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:39:18.112962 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:18.112771 5019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a265311-7ffc-488e-a829-8f43ee2e9a77-tls-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:39:18.744941 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:18.744905 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-031745/auth-test-model-031745-kserve-7656c77f57-l86zb" Oct 31 04:39:18.744941 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:18.744932 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-031745/auth-test-model-031745-kserve-7656c77f57-l86zb" event={"ID":"3a265311-7ffc-488e-a829-8f43ee2e9a77","Type":"ContainerDied","Data":"494ca30dd794cfff4419515f1e4be6a158dacc9a19e84b6504af9942381619bc"} Oct 31 04:39:18.745532 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:18.744984 5019 scope.go:122] "RemoveContainer" containerID="29ed1fe2333e5e8733f869f3152d7b50e90e1309fec309ac55ddabdd229a316f" Oct 31 04:39:18.799038 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:18.798994 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-031745/auth-test-model-031745-kserve-7656c77f57-l86zb"] Oct 31 04:39:18.811406 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:18.811342 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-031745/auth-test-model-031745-kserve-7656c77f57-l86zb"] Oct 31 04:39:19.411855 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:19.411809 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a265311-7ffc-488e-a829-8f43ee2e9a77" path="/var/lib/kubelet/pods/3a265311-7ffc-488e-a829-8f43ee2e9a77/volumes" Oct 31 04:39:19.412652 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:19.412630 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="441c6ad5-5141-4b8e-b9db-9497f594f479" path="/var/lib/kubelet/pods/441c6ad5-5141-4b8e-b9db-9497f594f479/volumes" Oct 31 04:39:19.413319 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:19.413301 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77" path="/var/lib/kubelet/pods/ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77/volumes" Oct 31 04:39:22.399063 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:39:22.399020 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:39:26.228795 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:26.228750 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7099-0a9981"] Oct 31 04:39:26.229521 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:26.229502 5019 cpu_manager.go:414] "containerMap: removing container" podUID="686f7dd0-02cd-457a-b7cd-ef83520d5341" containerName="istio-proxy" Oct 31 04:39:26.229521 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:26.229521 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="686f7dd0-02cd-457a-b7cd-ef83520d5341" containerName="istio-proxy" Oct 31 04:39:26.229648 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:26.229535 5019 cpu_manager.go:414] "containerMap: removing container" podUID="3a265311-7ffc-488e-a829-8f43ee2e9a77" containerName="main" Oct 31 04:39:26.229648 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:26.229540 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a265311-7ffc-488e-a829-8f43ee2e9a77" containerName="main" Oct 31 04:39:26.229648 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:26.229560 5019 cpu_manager.go:414] "containerMap: removing container" podUID="441c6ad5-5141-4b8e-b9db-9497f594f479" containerName="istio-proxy" Oct 31 04:39:26.229648 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:26.229567 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="441c6ad5-5141-4b8e-b9db-9497f594f479" containerName="istio-proxy" Oct 31 04:39:26.229648 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:26.229574 5019 cpu_manager.go:414] "containerMap: removing container" podUID="ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77" containerName="istio-proxy" Oct 31 04:39:26.229648 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:26.229579 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77" containerName="istio-proxy" Oct 31 04:39:26.229648 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:26.229594 5019 cpu_manager.go:414] "containerMap: removing container" podUID="38968c8c-ec31-40df-9e63-6324e59ae601" containerName="main" Oct 31 04:39:26.229648 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:26.229598 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38968c8c-ec31-40df-9e63-6324e59ae601" containerName="main" Oct 31 04:39:26.230111 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:26.229719 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="686f7dd0-02cd-457a-b7cd-ef83520d5341" containerName="istio-proxy" Oct 31 04:39:26.230111 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:26.229726 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a265311-7ffc-488e-a829-8f43ee2e9a77" containerName="main" Oct 31 04:39:26.230111 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:26.229737 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac7b7b2b-b9c0-4b2d-8c26-0dd7ce7e4e77" containerName="istio-proxy" Oct 31 04:39:26.230111 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:26.229743 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="441c6ad5-5141-4b8e-b9db-9497f594f479" containerName="istio-proxy" Oct 31 04:39:26.230111 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:26.229751 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="38968c8c-ec31-40df-9e63-6324e59ae601" containerName="main" Oct 31 04:39:26.235646 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:26.235618 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7099-0a9981" Oct 31 04:39:26.249635 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:26.249602 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7099-0a9981"] Oct 31 04:39:26.254219 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:26.254185 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-prswc\"" Oct 31 04:39:26.309345 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:26.309285 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rxqn\" (UniqueName: \"kubernetes.io/projected/18833047-2116-4e0d-9262-d4f6bc305d0a-kube-api-access-8rxqn\") pod \"test-curl-7099-0a9981\" (UID: \"18833047-2116-4e0d-9262-d4f6bc305d0a\") " pod="opendatahub/test-curl-7099-0a9981" Oct 31 04:39:26.411050 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:26.410980 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rxqn\" (UniqueName: \"kubernetes.io/projected/18833047-2116-4e0d-9262-d4f6bc305d0a-kube-api-access-8rxqn\") pod \"test-curl-7099-0a9981\" (UID: \"18833047-2116-4e0d-9262-d4f6bc305d0a\") " pod="opendatahub/test-curl-7099-0a9981" Oct 31 04:39:26.434923 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:26.434882 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rxqn\" (UniqueName: \"kubernetes.io/projected/18833047-2116-4e0d-9262-d4f6bc305d0a-kube-api-access-8rxqn\") pod \"test-curl-7099-0a9981\" (UID: \"18833047-2116-4e0d-9262-d4f6bc305d0a\") " pod="opendatahub/test-curl-7099-0a9981" Oct 31 04:39:26.553593 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:26.553503 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7099-0a9981" Oct 31 04:39:26.878255 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:26.878207 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7099-0a9981"] Oct 31 04:39:26.878474 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:39:26.878385 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18833047_2116_4e0d_9262_d4f6bc305d0a.slice/crio-3ab5790d5ee29e29036edec59adade02a48e6b784755354f2608ae4bb06be765 WatchSource:0}: Error finding container 3ab5790d5ee29e29036edec59adade02a48e6b784755354f2608ae4bb06be765: Status 404 returned error can't find the container with id 3ab5790d5ee29e29036edec59adade02a48e6b784755354f2608ae4bb06be765 Oct 31 04:39:27.029601 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:27.029559 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7099-0a9981" event={"ID":"18833047-2116-4e0d-9262-d4f6bc305d0a","Type":"ContainerStarted","Data":"76fdb6a92b538a3784b90f6fff4ebcaf80fe9b5e9cbbc986ce0980db5f33b123"} Oct 31 04:39:27.029601 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:27.029606 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7099-0a9981" event={"ID":"18833047-2116-4e0d-9262-d4f6bc305d0a","Type":"ContainerStarted","Data":"3ab5790d5ee29e29036edec59adade02a48e6b784755354f2608ae4bb06be765"} Oct 31 04:39:27.073620 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:27.073548 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7099-0a9981" podStartSLOduration=1.073531361 podStartE2EDuration="1.073531361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:39:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:39:27.070308013 +0000 UTC m=+2121.826646053" watchObservedRunningTime="2026-10-31 04:39:27.073531361 +0000 UTC m=+2121.829869402" Oct 31 04:39:28.004435 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:28.004400 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7099-0a9981"] Oct 31 04:39:28.009888 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:28.009842 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7099-0a9981"] Oct 31 04:39:28.226878 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:28.226839 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee57b3ee-4mlbl"] Oct 31 04:39:28.227703 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:28.227677 5019 cpu_manager.go:414] "containerMap: removing container" podUID="18833047-2116-4e0d-9262-d4f6bc305d0a" containerName="test-curl-7099-0a9981" Oct 31 04:39:28.227703 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:28.227702 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18833047-2116-4e0d-9262-d4f6bc305d0a" containerName="test-curl-7099-0a9981" Oct 31 04:39:28.227960 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:28.227941 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="18833047-2116-4e0d-9262-d4f6bc305d0a" containerName="test-curl-7099-0a9981" Oct 31 04:39:28.231963 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:28.231933 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee57b3ee-4mlbl" Oct 31 04:39:28.250943 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:28.250902 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:39:28.252321 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:28.252290 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee57b3ee-4mlbl"] Oct 31 04:39:28.257237 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:28.257124 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:39:28.257420 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:28.257308 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:39:28.262891 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:28.262857 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:39:28.344043 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:28.343985 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp4hj\" (UniqueName: \"kubernetes.io/projected/de3e39e4-9e92-4fb5-b867-1d19b2571dc4-kube-api-access-vp4hj\") pod \"maas-api-revoke-sub-ee57b3ee-4mlbl\" (UID: \"de3e39e4-9e92-4fb5-b867-1d19b2571dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee57b3ee-4mlbl" Oct 31 04:39:28.344243 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:28.344153 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de3e39e4-9e92-4fb5-b867-1d19b2571dc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee57b3ee-4mlbl\" (UID: \"de3e39e4-9e92-4fb5-b867-1d19b2571dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee57b3ee-4mlbl" Oct 31 04:39:28.344307 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:28.344285 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de3e39e4-9e92-4fb5-b867-1d19b2571dc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee57b3ee-4mlbl\" (UID: \"de3e39e4-9e92-4fb5-b867-1d19b2571dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee57b3ee-4mlbl" Oct 31 04:39:28.466360 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:28.462429 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp4hj\" (UniqueName: \"kubernetes.io/projected/de3e39e4-9e92-4fb5-b867-1d19b2571dc4-kube-api-access-vp4hj\") pod \"maas-api-revoke-sub-ee57b3ee-4mlbl\" (UID: \"de3e39e4-9e92-4fb5-b867-1d19b2571dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee57b3ee-4mlbl" Oct 31 04:39:28.466360 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:28.462624 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de3e39e4-9e92-4fb5-b867-1d19b2571dc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee57b3ee-4mlbl\" (UID: \"de3e39e4-9e92-4fb5-b867-1d19b2571dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee57b3ee-4mlbl" Oct 31 04:39:28.466360 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:28.462757 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de3e39e4-9e92-4fb5-b867-1d19b2571dc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee57b3ee-4mlbl\" (UID: \"de3e39e4-9e92-4fb5-b867-1d19b2571dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee57b3ee-4mlbl" Oct 31 04:39:28.488356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:28.467070 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de3e39e4-9e92-4fb5-b867-1d19b2571dc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee57b3ee-4mlbl\" (UID: \"de3e39e4-9e92-4fb5-b867-1d19b2571dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee57b3ee-4mlbl" Oct 31 04:39:28.488356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:28.473908 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de3e39e4-9e92-4fb5-b867-1d19b2571dc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee57b3ee-4mlbl\" (UID: \"de3e39e4-9e92-4fb5-b867-1d19b2571dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee57b3ee-4mlbl" Oct 31 04:39:28.506355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:28.496648 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp4hj\" (UniqueName: \"kubernetes.io/projected/de3e39e4-9e92-4fb5-b867-1d19b2571dc4-kube-api-access-vp4hj\") pod \"maas-api-revoke-sub-ee57b3ee-4mlbl\" (UID: \"de3e39e4-9e92-4fb5-b867-1d19b2571dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee57b3ee-4mlbl" Oct 31 04:39:28.554979 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:28.554884 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee57b3ee-4mlbl" Oct 31 04:39:28.873606 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:28.872796 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee57b3ee-4mlbl"] Oct 31 04:39:28.873606 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:39:28.872910 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde3e39e4_9e92_4fb5_b867_1d19b2571dc4.slice/crio-5d1e6bbb320e0a886842480ae4b6e8d1b23ac95a009aeada8662a7e36e92cdd4 WatchSource:0}: Error finding container 5d1e6bbb320e0a886842480ae4b6e8d1b23ac95a009aeada8662a7e36e92cdd4: Status 404 returned error can't find the container with id 5d1e6bbb320e0a886842480ae4b6e8d1b23ac95a009aeada8662a7e36e92cdd4 Oct 31 04:39:29.063055 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:29.063005 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee57b3ee-4mlbl" event={"ID":"de3e39e4-9e92-4fb5-b867-1d19b2571dc4","Type":"ContainerStarted","Data":"5d1e6bbb320e0a886842480ae4b6e8d1b23ac95a009aeada8662a7e36e92cdd4"} Oct 31 04:39:29.063686 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:29.063251 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7099-0a9981" podUID="18833047-2116-4e0d-9262-d4f6bc305d0a" containerName="test-curl-7099-0a9981" containerID="cri-o://76fdb6a92b538a3784b90f6fff4ebcaf80fe9b5e9cbbc986ce0980db5f33b123" gracePeriod=2 Oct 31 04:39:29.077166 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:39:29.077114 5019 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7099-0a9981\" is forbidden: User \"system:node:ip-10-0-130-57.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-57.ec2.internal' and this object" podUID="18833047-2116-4e0d-9262-d4f6bc305d0a" pod="opendatahub/test-curl-7099-0a9981" Oct 31 04:39:30.075094 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:30.075052 5019 generic.go:356] "Generic (PLEG): container finished" podID="de3e39e4-9e92-4fb5-b867-1d19b2571dc4" containerID="c5b3ca8424957f31359283a62b5fbd2af7f5d909ad44e8f2644b40b990b5569b" exitCode=0 Oct 31 04:39:30.075533 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:30.075135 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee57b3ee-4mlbl" event={"ID":"de3e39e4-9e92-4fb5-b867-1d19b2571dc4","Type":"ContainerDied","Data":"c5b3ca8424957f31359283a62b5fbd2af7f5d909ad44e8f2644b40b990b5569b"} Oct 31 04:39:31.332157 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:31.332120 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee57b3ee-4mlbl" Oct 31 04:39:31.386795 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:31.386755 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7099-0a9981" Oct 31 04:39:31.406913 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:31.406871 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de3e39e4-9e92-4fb5-b867-1d19b2571dc4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de3e39e4-9e92-4fb5-b867-1d19b2571dc4-maas-api-xks-ca\") pod \"de3e39e4-9e92-4fb5-b867-1d19b2571dc4\" (UID: \"de3e39e4-9e92-4fb5-b867-1d19b2571dc4\") " Oct 31 04:39:31.407152 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:31.407010 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de3e39e4-9e92-4fb5-b867-1d19b2571dc4-kube-api-access-vp4hj\" (UniqueName: \"kubernetes.io/projected/de3e39e4-9e92-4fb5-b867-1d19b2571dc4-kube-api-access-vp4hj\") pod \"de3e39e4-9e92-4fb5-b867-1d19b2571dc4\" (UID: \"de3e39e4-9e92-4fb5-b867-1d19b2571dc4\") " Oct 31 04:39:31.407152 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:31.407137 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de3e39e4-9e92-4fb5-b867-1d19b2571dc4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de3e39e4-9e92-4fb5-b867-1d19b2571dc4-maas-api-openshift-ca\") pod \"de3e39e4-9e92-4fb5-b867-1d19b2571dc4\" (UID: \"de3e39e4-9e92-4fb5-b867-1d19b2571dc4\") " Oct 31 04:39:31.408315 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:31.408283 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de3e39e4-9e92-4fb5-b867-1d19b2571dc4-maas-api-openshift-ca" pod "de3e39e4-9e92-4fb5-b867-1d19b2571dc4" (UID: "de3e39e4-9e92-4fb5-b867-1d19b2571dc4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:39:31.411129 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:31.411088 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de3e39e4-9e92-4fb5-b867-1d19b2571dc4-maas-api-xks-ca" pod "de3e39e4-9e92-4fb5-b867-1d19b2571dc4" (UID: "de3e39e4-9e92-4fb5-b867-1d19b2571dc4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:39:31.412639 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:31.412582 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de3e39e4-9e92-4fb5-b867-1d19b2571dc4-kube-api-access-vp4hj" pod "de3e39e4-9e92-4fb5-b867-1d19b2571dc4" (UID: "de3e39e4-9e92-4fb5-b867-1d19b2571dc4"). InnerVolumeSpecName "kube-api-access-vp4hj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:39:31.508714 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:31.508664 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18833047-2116-4e0d-9262-d4f6bc305d0a-kube-api-access-8rxqn\" (UniqueName: \"kubernetes.io/projected/18833047-2116-4e0d-9262-d4f6bc305d0a-kube-api-access-8rxqn\") pod \"18833047-2116-4e0d-9262-d4f6bc305d0a\" (UID: \"18833047-2116-4e0d-9262-d4f6bc305d0a\") " Oct 31 04:39:31.509459 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:31.509433 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de3e39e4-9e92-4fb5-b867-1d19b2571dc4-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:39:31.509459 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:31.509460 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vp4hj\" (UniqueName: \"kubernetes.io/projected/de3e39e4-9e92-4fb5-b867-1d19b2571dc4-kube-api-access-vp4hj\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:39:31.509676 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:31.509474 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de3e39e4-9e92-4fb5-b867-1d19b2571dc4-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:39:31.511638 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:31.511604 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18833047-2116-4e0d-9262-d4f6bc305d0a-kube-api-access-8rxqn" pod "18833047-2116-4e0d-9262-d4f6bc305d0a" (UID: "18833047-2116-4e0d-9262-d4f6bc305d0a"). InnerVolumeSpecName "kube-api-access-8rxqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:39:31.611442 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:31.611308 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rxqn\" (UniqueName: \"kubernetes.io/projected/18833047-2116-4e0d-9262-d4f6bc305d0a-kube-api-access-8rxqn\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:39:32.099164 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:32.099121 5019 generic.go:356] "Generic (PLEG): container finished" podID="18833047-2116-4e0d-9262-d4f6bc305d0a" containerID="76fdb6a92b538a3784b90f6fff4ebcaf80fe9b5e9cbbc986ce0980db5f33b123" exitCode=137 Oct 31 04:39:32.099412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:32.099180 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7099-0a9981" Oct 31 04:39:32.099412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:32.099258 5019 scope.go:122] "RemoveContainer" containerID="76fdb6a92b538a3784b90f6fff4ebcaf80fe9b5e9cbbc986ce0980db5f33b123" Oct 31 04:39:32.101736 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:32.101711 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee57b3ee-4mlbl" Oct 31 04:39:32.101912 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:32.101703 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee57b3ee-4mlbl" event={"ID":"de3e39e4-9e92-4fb5-b867-1d19b2571dc4","Type":"ContainerDied","Data":"5d1e6bbb320e0a886842480ae4b6e8d1b23ac95a009aeada8662a7e36e92cdd4"} Oct 31 04:39:32.101912 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:32.101837 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d1e6bbb320e0a886842480ae4b6e8d1b23ac95a009aeada8662a7e36e92cdd4" Oct 31 04:39:32.113803 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:32.113769 5019 scope.go:122] "RemoveContainer" containerID="76fdb6a92b538a3784b90f6fff4ebcaf80fe9b5e9cbbc986ce0980db5f33b123" Oct 31 04:39:32.114221 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:39:32.114183 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76fdb6a92b538a3784b90f6fff4ebcaf80fe9b5e9cbbc986ce0980db5f33b123\": container with ID starting with 76fdb6a92b538a3784b90f6fff4ebcaf80fe9b5e9cbbc986ce0980db5f33b123 not found: ID does not exist" containerID="76fdb6a92b538a3784b90f6fff4ebcaf80fe9b5e9cbbc986ce0980db5f33b123" Oct 31 04:39:32.114379 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:32.114235 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76fdb6a92b538a3784b90f6fff4ebcaf80fe9b5e9cbbc986ce0980db5f33b123"} err="failed to get container status \"76fdb6a92b538a3784b90f6fff4ebcaf80fe9b5e9cbbc986ce0980db5f33b123\": rpc error: code = NotFound desc = could not find container \"76fdb6a92b538a3784b90f6fff4ebcaf80fe9b5e9cbbc986ce0980db5f33b123\": container with ID starting with 76fdb6a92b538a3784b90f6fff4ebcaf80fe9b5e9cbbc986ce0980db5f33b123 not found: ID does not exist" Oct 31 04:39:33.418527 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:33.418491 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18833047-2116-4e0d-9262-d4f6bc305d0a" path="/var/lib/kubelet/pods/18833047-2116-4e0d-9262-d4f6bc305d0a/volumes" Oct 31 04:39:38.594202 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:38.594163 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79257c5d-mrcfj"] Oct 31 04:39:38.594911 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:38.594894 5019 cpu_manager.go:414] "containerMap: removing container" podUID="de3e39e4-9e92-4fb5-b867-1d19b2571dc4" containerName="revoke-keys" Oct 31 04:39:38.594961 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:38.594912 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de3e39e4-9e92-4fb5-b867-1d19b2571dc4" containerName="revoke-keys" Oct 31 04:39:38.595079 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:38.595068 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="de3e39e4-9e92-4fb5-b867-1d19b2571dc4" containerName="revoke-keys" Oct 31 04:39:38.600213 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:38.600191 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79257c5d-mrcfj" Oct 31 04:39:38.618742 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:38.618704 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79257c5d-mrcfj"] Oct 31 04:39:38.625240 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:38.624986 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:39:38.625240 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:38.625043 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:39:38.625240 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:38.625083 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:39:38.640279 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:38.626962 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:39:38.724214 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:38.724172 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c0cd54f-f479-4e04-a535-5fc3b14344f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79257c5d-mrcfj\" (UID: \"9c0cd54f-f479-4e04-a535-5fc3b14344f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79257c5d-mrcfj" Oct 31 04:39:38.724442 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:38.724227 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b92hn\" (UniqueName: \"kubernetes.io/projected/9c0cd54f-f479-4e04-a535-5fc3b14344f2-kube-api-access-b92hn\") pod \"maas-api-revoke-sub-79257c5d-mrcfj\" (UID: \"9c0cd54f-f479-4e04-a535-5fc3b14344f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79257c5d-mrcfj" Oct 31 04:39:38.724442 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:38.724413 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c0cd54f-f479-4e04-a535-5fc3b14344f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79257c5d-mrcfj\" (UID: \"9c0cd54f-f479-4e04-a535-5fc3b14344f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79257c5d-mrcfj" Oct 31 04:39:38.825454 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:38.825407 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c0cd54f-f479-4e04-a535-5fc3b14344f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79257c5d-mrcfj\" (UID: \"9c0cd54f-f479-4e04-a535-5fc3b14344f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79257c5d-mrcfj" Oct 31 04:39:38.825680 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:38.825486 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b92hn\" (UniqueName: \"kubernetes.io/projected/9c0cd54f-f479-4e04-a535-5fc3b14344f2-kube-api-access-b92hn\") pod \"maas-api-revoke-sub-79257c5d-mrcfj\" (UID: \"9c0cd54f-f479-4e04-a535-5fc3b14344f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79257c5d-mrcfj" Oct 31 04:39:38.825680 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:38.825563 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c0cd54f-f479-4e04-a535-5fc3b14344f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79257c5d-mrcfj\" (UID: \"9c0cd54f-f479-4e04-a535-5fc3b14344f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79257c5d-mrcfj" Oct 31 04:39:38.826192 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:38.826160 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c0cd54f-f479-4e04-a535-5fc3b14344f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79257c5d-mrcfj\" (UID: \"9c0cd54f-f479-4e04-a535-5fc3b14344f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79257c5d-mrcfj" Oct 31 04:39:38.828117 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:38.828088 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c0cd54f-f479-4e04-a535-5fc3b14344f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79257c5d-mrcfj\" (UID: \"9c0cd54f-f479-4e04-a535-5fc3b14344f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79257c5d-mrcfj" Oct 31 04:39:38.849789 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:38.849705 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b92hn\" (UniqueName: \"kubernetes.io/projected/9c0cd54f-f479-4e04-a535-5fc3b14344f2-kube-api-access-b92hn\") pod \"maas-api-revoke-sub-79257c5d-mrcfj\" (UID: \"9c0cd54f-f479-4e04-a535-5fc3b14344f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79257c5d-mrcfj" Oct 31 04:39:38.915877 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:38.915828 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79257c5d-mrcfj" Oct 31 04:39:39.223348 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:39.223301 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79257c5d-mrcfj"] Oct 31 04:39:39.224930 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:39:39.224897 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c0cd54f_f479_4e04_a535_5fc3b14344f2.slice/crio-c2cd25ee475aefa732374a916ef717b638f4dc0d985b478d56ff2b4d6f8af77e WatchSource:0}: Error finding container c2cd25ee475aefa732374a916ef717b638f4dc0d985b478d56ff2b4d6f8af77e: Status 404 returned error can't find the container with id c2cd25ee475aefa732374a916ef717b638f4dc0d985b478d56ff2b4d6f8af77e Oct 31 04:39:40.214401 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:40.214054 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79257c5d-mrcfj" event={"ID":"9c0cd54f-f479-4e04-a535-5fc3b14344f2","Type":"ContainerStarted","Data":"0b5f28179129910967c4c516705085acc21af895f10e2e3a175d03a8589b2ca4"} Oct 31 04:39:40.214401 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:40.214149 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79257c5d-mrcfj" event={"ID":"9c0cd54f-f479-4e04-a535-5fc3b14344f2","Type":"ContainerStarted","Data":"c2cd25ee475aefa732374a916ef717b638f4dc0d985b478d56ff2b4d6f8af77e"} Oct 31 04:39:40.257404 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:40.257307 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79257c5d-mrcfj" podStartSLOduration=2.257286492 podStartE2EDuration="2.257286492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:39:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:39:40.256132773 +0000 UTC m=+2135.012470838" watchObservedRunningTime="2026-10-31 04:39:40.257286492 +0000 UTC m=+2135.013624533" Oct 31 04:39:41.239393 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:41.235925 5019 generic.go:356] "Generic (PLEG): container finished" podID="9c0cd54f-f479-4e04-a535-5fc3b14344f2" containerID="0b5f28179129910967c4c516705085acc21af895f10e2e3a175d03a8589b2ca4" exitCode=0 Oct 31 04:39:41.239393 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:41.235986 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79257c5d-mrcfj" event={"ID":"9c0cd54f-f479-4e04-a535-5fc3b14344f2","Type":"ContainerDied","Data":"0b5f28179129910967c4c516705085acc21af895f10e2e3a175d03a8589b2ca4"} Oct 31 04:39:42.437079 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:42.437044 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79257c5d-mrcfj" Oct 31 04:39:42.587602 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:42.587488 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c0cd54f-f479-4e04-a535-5fc3b14344f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c0cd54f-f479-4e04-a535-5fc3b14344f2-maas-api-xks-ca\") pod \"9c0cd54f-f479-4e04-a535-5fc3b14344f2\" (UID: \"9c0cd54f-f479-4e04-a535-5fc3b14344f2\") " Oct 31 04:39:42.587784 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:42.587659 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c0cd54f-f479-4e04-a535-5fc3b14344f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c0cd54f-f479-4e04-a535-5fc3b14344f2-maas-api-openshift-ca\") pod \"9c0cd54f-f479-4e04-a535-5fc3b14344f2\" (UID: \"9c0cd54f-f479-4e04-a535-5fc3b14344f2\") " Oct 31 04:39:42.587784 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:42.587726 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c0cd54f-f479-4e04-a535-5fc3b14344f2-kube-api-access-b92hn\" (UniqueName: \"kubernetes.io/projected/9c0cd54f-f479-4e04-a535-5fc3b14344f2-kube-api-access-b92hn\") pod \"9c0cd54f-f479-4e04-a535-5fc3b14344f2\" (UID: \"9c0cd54f-f479-4e04-a535-5fc3b14344f2\") " Oct 31 04:39:42.588070 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:42.588038 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c0cd54f-f479-4e04-a535-5fc3b14344f2-maas-api-openshift-ca" pod "9c0cd54f-f479-4e04-a535-5fc3b14344f2" (UID: "9c0cd54f-f479-4e04-a535-5fc3b14344f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:39:42.588978 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:42.588949 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c0cd54f-f479-4e04-a535-5fc3b14344f2-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:39:42.590146 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:42.590119 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c0cd54f-f479-4e04-a535-5fc3b14344f2-maas-api-xks-ca" pod "9c0cd54f-f479-4e04-a535-5fc3b14344f2" (UID: "9c0cd54f-f479-4e04-a535-5fc3b14344f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:39:42.590293 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:42.590207 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c0cd54f-f479-4e04-a535-5fc3b14344f2-kube-api-access-b92hn" pod "9c0cd54f-f479-4e04-a535-5fc3b14344f2" (UID: "9c0cd54f-f479-4e04-a535-5fc3b14344f2"). InnerVolumeSpecName "kube-api-access-b92hn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:39:42.690082 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:42.690036 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c0cd54f-f479-4e04-a535-5fc3b14344f2-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:39:42.690082 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:42.690075 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b92hn\" (UniqueName: \"kubernetes.io/projected/9c0cd54f-f479-4e04-a535-5fc3b14344f2-kube-api-access-b92hn\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:39:43.256195 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:43.256159 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79257c5d-mrcfj" Oct 31 04:39:43.256195 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:43.256178 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79257c5d-mrcfj" event={"ID":"9c0cd54f-f479-4e04-a535-5fc3b14344f2","Type":"ContainerDied","Data":"c2cd25ee475aefa732374a916ef717b638f4dc0d985b478d56ff2b4d6f8af77e"} Oct 31 04:39:43.256466 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:43.256237 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2cd25ee475aefa732374a916ef717b638f4dc0d985b478d56ff2b4d6f8af77e" Oct 31 04:39:50.831756 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:50.831710 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-77a153/auth-test-model-77a153-kserve-6fcd554cc-c5d8v"] Oct 31 04:39:52.398910 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:39:52.398867 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:39:57.281640 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:57.281599 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-d6ac82/auth-test-model-d6ac82-kserve-5cc8848985-xm5bx"] Oct 31 04:39:57.655286 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:57.655189 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt"] Oct 31 04:39:57.656237 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:57.656208 5019 cpu_manager.go:414] "containerMap: removing container" podUID="9c0cd54f-f479-4e04-a535-5fc3b14344f2" containerName="revoke-keys" Oct 31 04:39:57.656237 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:57.656239 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c0cd54f-f479-4e04-a535-5fc3b14344f2" containerName="revoke-keys" Oct 31 04:39:57.656501 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:57.656484 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c0cd54f-f479-4e04-a535-5fc3b14344f2" containerName="revoke-keys" Oct 31 04:39:57.661003 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:57.660970 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt" Oct 31 04:39:57.689656 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:57.689616 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-ded435-openshift-default-dockercfg-x8xsx\"" Oct 31 04:39:57.694917 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:57.694878 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt"] Oct 31 04:39:57.779431 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:57.777615 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw574\" (UniqueName: \"kubernetes.io/projected/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-kube-api-access-kw574\") pod \"e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt\" (UID: \"2764ba4c-dff3-4c51-a8ca-0121dd9f5b29\") " pod="openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt" Oct 31 04:39:57.779431 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:57.777751 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-istio-token\") pod \"e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt\" (UID: \"2764ba4c-dff3-4c51-a8ca-0121dd9f5b29\") " pod="openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt" Oct 31 04:39:57.779431 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:57.777813 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-workload-certs\") pod \"e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt\" (UID: \"2764ba4c-dff3-4c51-a8ca-0121dd9f5b29\") " pod="openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt" Oct 31 04:39:57.779431 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:57.777877 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-istio-envoy\") pod \"e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt\" (UID: \"2764ba4c-dff3-4c51-a8ca-0121dd9f5b29\") " pod="openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt" Oct 31 04:39:57.779431 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:57.778045 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-istio-data\") pod \"e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt\" (UID: \"2764ba4c-dff3-4c51-a8ca-0121dd9f5b29\") " pod="openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt" Oct 31 04:39:57.779431 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:57.778090 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-workload-socket\") pod \"e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt\" (UID: \"2764ba4c-dff3-4c51-a8ca-0121dd9f5b29\") " pod="openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt" Oct 31 04:39:57.779431 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:57.778177 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-istiod-ca-cert\") pod \"e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt\" (UID: \"2764ba4c-dff3-4c51-a8ca-0121dd9f5b29\") " pod="openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt" Oct 31 04:39:57.779431 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:57.778205 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-istio-podinfo\") pod \"e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt\" (UID: \"2764ba4c-dff3-4c51-a8ca-0121dd9f5b29\") " pod="openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt" Oct 31 04:39:57.779431 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:57.778232 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-credential-socket\") pod \"e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt\" (UID: \"2764ba4c-dff3-4c51-a8ca-0121dd9f5b29\") " pod="openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt" Oct 31 04:39:57.891353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:57.889965 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-istiod-ca-cert\") pod \"e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt\" (UID: \"2764ba4c-dff3-4c51-a8ca-0121dd9f5b29\") " pod="openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt" Oct 31 04:39:57.891353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:57.890039 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-istio-podinfo\") pod \"e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt\" (UID: \"2764ba4c-dff3-4c51-a8ca-0121dd9f5b29\") " pod="openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt" Oct 31 04:39:57.891353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:57.890082 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-credential-socket\") pod \"e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt\" (UID: \"2764ba4c-dff3-4c51-a8ca-0121dd9f5b29\") " pod="openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt" Oct 31 04:39:57.891353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:57.890192 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw574\" (UniqueName: \"kubernetes.io/projected/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-kube-api-access-kw574\") pod \"e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt\" (UID: \"2764ba4c-dff3-4c51-a8ca-0121dd9f5b29\") " pod="openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt" Oct 31 04:39:57.891353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:57.890280 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-istio-token\") pod \"e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt\" (UID: \"2764ba4c-dff3-4c51-a8ca-0121dd9f5b29\") " pod="openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt" Oct 31 04:39:57.910424 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:57.893653 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-workload-certs\") pod \"e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt\" (UID: \"2764ba4c-dff3-4c51-a8ca-0121dd9f5b29\") " pod="openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt" Oct 31 04:39:57.910424 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:57.893764 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-istio-envoy\") pod \"e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt\" (UID: \"2764ba4c-dff3-4c51-a8ca-0121dd9f5b29\") " pod="openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt" Oct 31 04:39:57.910424 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:57.893966 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-istio-data\") pod \"e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt\" (UID: \"2764ba4c-dff3-4c51-a8ca-0121dd9f5b29\") " pod="openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt" Oct 31 04:39:57.910424 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:57.894020 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-workload-socket\") pod \"e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt\" (UID: \"2764ba4c-dff3-4c51-a8ca-0121dd9f5b29\") " pod="openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt" Oct 31 04:39:57.910424 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:57.905527 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-istiod-ca-cert\") pod \"e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt\" (UID: \"2764ba4c-dff3-4c51-a8ca-0121dd9f5b29\") " pod="openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt" Oct 31 04:39:57.912708 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:57.912670 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-workload-certs\") pod \"e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt\" (UID: \"2764ba4c-dff3-4c51-a8ca-0121dd9f5b29\") " pod="openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt" Oct 31 04:39:57.965126 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:57.919210 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-workload-socket\") pod \"e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt\" (UID: \"2764ba4c-dff3-4c51-a8ca-0121dd9f5b29\") " pod="openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt" Oct 31 04:39:57.965445 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:57.920053 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-credential-socket\") pod \"e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt\" (UID: \"2764ba4c-dff3-4c51-a8ca-0121dd9f5b29\") " pod="openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt" Oct 31 04:39:57.965682 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:57.941734 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-istio-data\") pod \"e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt\" (UID: \"2764ba4c-dff3-4c51-a8ca-0121dd9f5b29\") " pod="openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt" Oct 31 04:39:57.965897 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:57.942268 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-istio-envoy\") pod \"e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt\" (UID: \"2764ba4c-dff3-4c51-a8ca-0121dd9f5b29\") " pod="openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt" Oct 31 04:39:57.966021 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:57.959657 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-istio-token\") pod \"e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt\" (UID: \"2764ba4c-dff3-4c51-a8ca-0121dd9f5b29\") " pod="openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt" Oct 31 04:39:57.966208 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:57.960502 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw574\" (UniqueName: \"kubernetes.io/projected/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-kube-api-access-kw574\") pod \"e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt\" (UID: \"2764ba4c-dff3-4c51-a8ca-0121dd9f5b29\") " pod="openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt" Oct 31 04:39:57.981656 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:57.981619 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-istio-podinfo\") pod \"e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt\" (UID: \"2764ba4c-dff3-4c51-a8ca-0121dd9f5b29\") " pod="openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt" Oct 31 04:39:57.997609 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:57.997504 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt" Oct 31 04:39:58.435426 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:58.435371 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:39:58.436292 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:58.436249 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:39:58.437223 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:58.437151 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:39:58.495163 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:58.489954 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp"] Oct 31 04:39:58.559025 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:58.558984 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp" Oct 31 04:39:58.560741 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:58.560695 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt" event={"ID":"2764ba4c-dff3-4c51-a8ca-0121dd9f5b29","Type":"ContainerStarted","Data":"2ad9671df018a064944518fa26e0f21c22b6fe2770e7ce0493c63a4fbdb76433"} Oct 31 04:39:58.615205 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:58.615174 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-99ca0c-openshift-default-dockercfg-vnbzk\"" Oct 31 04:39:58.660132 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:58.656286 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/468817c7-e1ca-41c0-8a91-cfce50aa71ac-workload-certs\") pod \"e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp\" (UID: \"468817c7-e1ca-41c0-8a91-cfce50aa71ac\") " pod="openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp" Oct 31 04:39:58.660132 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:58.656378 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/468817c7-e1ca-41c0-8a91-cfce50aa71ac-istio-envoy\") pod \"e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp\" (UID: \"468817c7-e1ca-41c0-8a91-cfce50aa71ac\") " pod="openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp" Oct 31 04:39:58.660132 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:58.656447 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/468817c7-e1ca-41c0-8a91-cfce50aa71ac-workload-socket\") pod \"e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp\" (UID: \"468817c7-e1ca-41c0-8a91-cfce50aa71ac\") " pod="openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp" Oct 31 04:39:58.660132 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:58.656586 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/468817c7-e1ca-41c0-8a91-cfce50aa71ac-credential-socket\") pod \"e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp\" (UID: \"468817c7-e1ca-41c0-8a91-cfce50aa71ac\") " pod="openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp" Oct 31 04:39:58.660132 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:58.656649 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/468817c7-e1ca-41c0-8a91-cfce50aa71ac-istio-data\") pod \"e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp\" (UID: \"468817c7-e1ca-41c0-8a91-cfce50aa71ac\") " pod="openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp" Oct 31 04:39:58.660132 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:58.656676 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/468817c7-e1ca-41c0-8a91-cfce50aa71ac-istio-token\") pod \"e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp\" (UID: \"468817c7-e1ca-41c0-8a91-cfce50aa71ac\") " pod="openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp" Oct 31 04:39:58.660132 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:58.656723 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/468817c7-e1ca-41c0-8a91-cfce50aa71ac-istiod-ca-cert\") pod \"e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp\" (UID: \"468817c7-e1ca-41c0-8a91-cfce50aa71ac\") " pod="openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp" Oct 31 04:39:58.660132 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:58.656790 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/468817c7-e1ca-41c0-8a91-cfce50aa71ac-istio-podinfo\") pod \"e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp\" (UID: \"468817c7-e1ca-41c0-8a91-cfce50aa71ac\") " pod="openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp" Oct 31 04:39:58.660132 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:58.656817 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtzth\" (UniqueName: \"kubernetes.io/projected/468817c7-e1ca-41c0-8a91-cfce50aa71ac-kube-api-access-rtzth\") pod \"e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp\" (UID: \"468817c7-e1ca-41c0-8a91-cfce50aa71ac\") " pod="openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp" Oct 31 04:39:58.678877 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:58.677814 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt"] Oct 31 04:39:58.687203 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:58.687169 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp"] Oct 31 04:39:58.758715 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:58.758658 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/468817c7-e1ca-41c0-8a91-cfce50aa71ac-workload-certs\") pod \"e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp\" (UID: \"468817c7-e1ca-41c0-8a91-cfce50aa71ac\") " pod="openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp" Oct 31 04:39:58.758715 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:58.758719 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/468817c7-e1ca-41c0-8a91-cfce50aa71ac-istio-envoy\") pod \"e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp\" (UID: \"468817c7-e1ca-41c0-8a91-cfce50aa71ac\") " pod="openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp" Oct 31 04:39:58.759384 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:58.758759 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/468817c7-e1ca-41c0-8a91-cfce50aa71ac-workload-socket\") pod \"e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp\" (UID: \"468817c7-e1ca-41c0-8a91-cfce50aa71ac\") " pod="openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp" Oct 31 04:39:58.759384 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:58.758873 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/468817c7-e1ca-41c0-8a91-cfce50aa71ac-credential-socket\") pod \"e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp\" (UID: \"468817c7-e1ca-41c0-8a91-cfce50aa71ac\") " pod="openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp" Oct 31 04:39:58.759384 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:58.758927 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/468817c7-e1ca-41c0-8a91-cfce50aa71ac-istio-data\") pod \"e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp\" (UID: \"468817c7-e1ca-41c0-8a91-cfce50aa71ac\") " pod="openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp" Oct 31 04:39:58.759384 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:58.758953 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/468817c7-e1ca-41c0-8a91-cfce50aa71ac-istio-token\") pod \"e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp\" (UID: \"468817c7-e1ca-41c0-8a91-cfce50aa71ac\") " pod="openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp" Oct 31 04:39:58.759384 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:58.758996 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/468817c7-e1ca-41c0-8a91-cfce50aa71ac-istiod-ca-cert\") pod \"e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp\" (UID: \"468817c7-e1ca-41c0-8a91-cfce50aa71ac\") " pod="openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp" Oct 31 04:39:58.759384 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:58.759057 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/468817c7-e1ca-41c0-8a91-cfce50aa71ac-istio-podinfo\") pod \"e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp\" (UID: \"468817c7-e1ca-41c0-8a91-cfce50aa71ac\") " pod="openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp" Oct 31 04:39:58.759384 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:58.759176 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/468817c7-e1ca-41c0-8a91-cfce50aa71ac-workload-certs\") pod \"e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp\" (UID: \"468817c7-e1ca-41c0-8a91-cfce50aa71ac\") " pod="openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp" Oct 31 04:39:58.759384 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:58.759226 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/468817c7-e1ca-41c0-8a91-cfce50aa71ac-workload-socket\") pod \"e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp\" (UID: \"468817c7-e1ca-41c0-8a91-cfce50aa71ac\") " pod="openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp" Oct 31 04:39:58.759384 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:58.759366 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtzth\" (UniqueName: \"kubernetes.io/projected/468817c7-e1ca-41c0-8a91-cfce50aa71ac-kube-api-access-rtzth\") pod \"e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp\" (UID: \"468817c7-e1ca-41c0-8a91-cfce50aa71ac\") " pod="openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp" Oct 31 04:39:58.759960 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:58.759936 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/468817c7-e1ca-41c0-8a91-cfce50aa71ac-istiod-ca-cert\") pod \"e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp\" (UID: \"468817c7-e1ca-41c0-8a91-cfce50aa71ac\") " pod="openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp" Oct 31 04:39:58.760047 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:58.759959 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/468817c7-e1ca-41c0-8a91-cfce50aa71ac-istio-data\") pod \"e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp\" (UID: \"468817c7-e1ca-41c0-8a91-cfce50aa71ac\") " pod="openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp" Oct 31 04:39:58.760047 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:58.760032 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/468817c7-e1ca-41c0-8a91-cfce50aa71ac-credential-socket\") pod \"e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp\" (UID: \"468817c7-e1ca-41c0-8a91-cfce50aa71ac\") " pod="openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp" Oct 31 04:39:58.761986 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:58.761966 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/468817c7-e1ca-41c0-8a91-cfce50aa71ac-istio-envoy\") pod \"e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp\" (UID: \"468817c7-e1ca-41c0-8a91-cfce50aa71ac\") " pod="openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp" Oct 31 04:39:58.762370 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:58.762322 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/468817c7-e1ca-41c0-8a91-cfce50aa71ac-istio-podinfo\") pod \"e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp\" (UID: \"468817c7-e1ca-41c0-8a91-cfce50aa71ac\") " pod="openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp" Oct 31 04:39:58.784119 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:58.784041 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/468817c7-e1ca-41c0-8a91-cfce50aa71ac-istio-token\") pod \"e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp\" (UID: \"468817c7-e1ca-41c0-8a91-cfce50aa71ac\") " pod="openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp" Oct 31 04:39:58.784322 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:58.784241 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtzth\" (UniqueName: \"kubernetes.io/projected/468817c7-e1ca-41c0-8a91-cfce50aa71ac-kube-api-access-rtzth\") pod \"e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp\" (UID: \"468817c7-e1ca-41c0-8a91-cfce50aa71ac\") " pod="openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp" Oct 31 04:39:58.931160 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:58.931065 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp" Oct 31 04:39:59.258767 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:59.258717 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp"] Oct 31 04:39:59.258953 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:39:59.258803 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod468817c7_e1ca_41c0_8a91_cfce50aa71ac.slice/crio-0e217789b8d0d774a4ffec3c0c9e9db926e3b2e3ecc6f1d4eb9813dcd7ff7e5c WatchSource:0}: Error finding container 0e217789b8d0d774a4ffec3c0c9e9db926e3b2e3ecc6f1d4eb9813dcd7ff7e5c: Status 404 returned error can't find the container with id 0e217789b8d0d774a4ffec3c0c9e9db926e3b2e3ecc6f1d4eb9813dcd7ff7e5c Oct 31 04:39:59.262020 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:59.261979 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:39:59.262165 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:59.262067 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:39:59.262165 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:59.262117 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:39:59.593439 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:59.577596 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt" event={"ID":"2764ba4c-dff3-4c51-a8ca-0121dd9f5b29","Type":"ContainerStarted","Data":"ae3722b96fc47f0c6468e33e2f94c81d370fd8ec6ad49c0f6464a7e744156dca"} Oct 31 04:39:59.593439 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:59.588905 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp" event={"ID":"468817c7-e1ca-41c0-8a91-cfce50aa71ac","Type":"ContainerStarted","Data":"5590a7d6cdfe15f49a0d13ca1f3456ca2da18f063702b9e3e0cc1dc83aae0e8a"} Oct 31 04:39:59.593439 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:59.588971 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp" event={"ID":"468817c7-e1ca-41c0-8a91-cfce50aa71ac","Type":"ContainerStarted","Data":"0e217789b8d0d774a4ffec3c0c9e9db926e3b2e3ecc6f1d4eb9813dcd7ff7e5c"} Oct 31 04:39:59.624646 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:59.624587 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt" podStartSLOduration=2.624571177 podStartE2EDuration="2.624571177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:39:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:39:59.622042876 +0000 UTC m=+2154.378380917" watchObservedRunningTime="2026-10-31 04:39:59.624571177 +0000 UTC m=+2154.380909216" Oct 31 04:39:59.670372 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:59.670276 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp" podStartSLOduration=1.670260458 podStartE2EDuration="1.670260458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:39:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:39:59.666171763 +0000 UTC m=+2154.422509805" watchObservedRunningTime="2026-10-31 04:39:59.670260458 +0000 UTC m=+2154.426598498" Oct 31 04:39:59.931911 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:59.931823 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp" Oct 31 04:39:59.998641 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:39:59.998597 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt" Oct 31 04:40:00.006669 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:00.006631 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt" Oct 31 04:40:00.598673 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:00.598636 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt" Oct 31 04:40:00.599764 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:00.599732 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt" Oct 31 04:40:00.938951 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:00.938862 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp" Oct 31 04:40:01.609070 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:01.609028 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp" Oct 31 04:40:01.610090 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:01.610066 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp" Oct 31 04:40:03.962767 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:03.962730 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-99ca0c-568c687f79-tqpf6"] Oct 31 04:40:03.967787 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:03.967761 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-99ca0c-568c687f79-tqpf6" Oct 31 04:40:03.990032 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:03.989988 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-99ca0c-568c687f79-tqpf6"] Oct 31 04:40:04.005921 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:04.005880 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-99ca0c\"" Oct 31 04:40:04.006135 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:04.006113 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-99ca0c-dockercfg-n68m4\"" Oct 31 04:40:04.097359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:04.089700 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d42130a2-25c9-4a81-ae93-1d6219e5794f-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-99ca0c-568c687f79-tqpf6\" (UID: \"d42130a2-25c9-4a81-ae93-1d6219e5794f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-99ca0c-568c687f79-tqpf6" Oct 31 04:40:04.097359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:04.089853 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6zf2\" (UniqueName: \"kubernetes.io/projected/d42130a2-25c9-4a81-ae93-1d6219e5794f-kube-api-access-r6zf2\") pod \"payload-pre-processing-e2e-explicit-99ca0c-568c687f79-tqpf6\" (UID: \"d42130a2-25c9-4a81-ae93-1d6219e5794f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-99ca0c-568c687f79-tqpf6" Oct 31 04:40:04.196095 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:04.177074 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-99ca0c-787d86f4c5-rhkjg"] Oct 31 04:40:04.219635 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:04.219163 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d42130a2-25c9-4a81-ae93-1d6219e5794f-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-99ca0c-568c687f79-tqpf6\" (UID: \"d42130a2-25c9-4a81-ae93-1d6219e5794f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-99ca0c-568c687f79-tqpf6" Oct 31 04:40:04.219635 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:04.219400 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6zf2\" (UniqueName: \"kubernetes.io/projected/d42130a2-25c9-4a81-ae93-1d6219e5794f-kube-api-access-r6zf2\") pod \"payload-pre-processing-e2e-explicit-99ca0c-568c687f79-tqpf6\" (UID: \"d42130a2-25c9-4a81-ae93-1d6219e5794f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-99ca0c-568c687f79-tqpf6" Oct 31 04:40:04.225502 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:04.224992 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d42130a2-25c9-4a81-ae93-1d6219e5794f-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-99ca0c-568c687f79-tqpf6\" (UID: \"d42130a2-25c9-4a81-ae93-1d6219e5794f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-99ca0c-568c687f79-tqpf6" Oct 31 04:40:04.226442 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:04.225952 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-99ca0c-787d86f4c5-rhkjg" Oct 31 04:40:04.233854 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:04.233473 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-99ca0c-787d86f4c5-rhkjg"] Oct 31 04:40:04.258645 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:04.258607 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6zf2\" (UniqueName: \"kubernetes.io/projected/d42130a2-25c9-4a81-ae93-1d6219e5794f-kube-api-access-r6zf2\") pod \"payload-pre-processing-e2e-explicit-99ca0c-568c687f79-tqpf6\" (UID: \"d42130a2-25c9-4a81-ae93-1d6219e5794f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-99ca0c-568c687f79-tqpf6" Oct 31 04:40:04.287824 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:04.287794 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-99ca0c-568c687f79-tqpf6" Oct 31 04:40:04.322680 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:04.322639 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq5hl\" (UniqueName: \"kubernetes.io/projected/13506896-9897-4d94-91df-2d3c6eb87f68-kube-api-access-hq5hl\") pod \"payload-processing-e2e-explicit-99ca0c-787d86f4c5-rhkjg\" (UID: \"13506896-9897-4d94-91df-2d3c6eb87f68\") " pod="openshift-ingress/payload-processing-e2e-explicit-99ca0c-787d86f4c5-rhkjg" Oct 31 04:40:04.322876 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:04.322692 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13506896-9897-4d94-91df-2d3c6eb87f68-plugins-config-volume\") pod \"payload-processing-e2e-explicit-99ca0c-787d86f4c5-rhkjg\" (UID: \"13506896-9897-4d94-91df-2d3c6eb87f68\") " pod="openshift-ingress/payload-processing-e2e-explicit-99ca0c-787d86f4c5-rhkjg" Oct 31 04:40:04.425018 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:04.424975 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq5hl\" (UniqueName: \"kubernetes.io/projected/13506896-9897-4d94-91df-2d3c6eb87f68-kube-api-access-hq5hl\") pod \"payload-processing-e2e-explicit-99ca0c-787d86f4c5-rhkjg\" (UID: \"13506896-9897-4d94-91df-2d3c6eb87f68\") " pod="openshift-ingress/payload-processing-e2e-explicit-99ca0c-787d86f4c5-rhkjg" Oct 31 04:40:04.425018 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:04.425020 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13506896-9897-4d94-91df-2d3c6eb87f68-plugins-config-volume\") pod \"payload-processing-e2e-explicit-99ca0c-787d86f4c5-rhkjg\" (UID: \"13506896-9897-4d94-91df-2d3c6eb87f68\") " pod="openshift-ingress/payload-processing-e2e-explicit-99ca0c-787d86f4c5-rhkjg" Oct 31 04:40:04.426010 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:04.425983 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13506896-9897-4d94-91df-2d3c6eb87f68-plugins-config-volume\") pod \"payload-processing-e2e-explicit-99ca0c-787d86f4c5-rhkjg\" (UID: \"13506896-9897-4d94-91df-2d3c6eb87f68\") " pod="openshift-ingress/payload-processing-e2e-explicit-99ca0c-787d86f4c5-rhkjg" Oct 31 04:40:04.458866 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:04.458820 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq5hl\" (UniqueName: \"kubernetes.io/projected/13506896-9897-4d94-91df-2d3c6eb87f68-kube-api-access-hq5hl\") pod \"payload-processing-e2e-explicit-99ca0c-787d86f4c5-rhkjg\" (UID: \"13506896-9897-4d94-91df-2d3c6eb87f68\") " pod="openshift-ingress/payload-processing-e2e-explicit-99ca0c-787d86f4c5-rhkjg" Oct 31 04:40:04.606022 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:04.605910 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-99ca0c-787d86f4c5-rhkjg" Oct 31 04:40:04.615715 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:04.615679 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-99ca0c-568c687f79-tqpf6"] Oct 31 04:40:04.622819 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:40:04.622781 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd42130a2_25c9_4a81_ae93_1d6219e5794f.slice/crio-0daf34be838f6eef7528437593a4a56462040c96c91696e9e3f321ad237bc18d WatchSource:0}: Error finding container 0daf34be838f6eef7528437593a4a56462040c96c91696e9e3f321ad237bc18d: Status 404 returned error can't find the container with id 0daf34be838f6eef7528437593a4a56462040c96c91696e9e3f321ad237bc18d Oct 31 04:40:04.647388 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:04.647341 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-99ca0c-568c687f79-tqpf6" event={"ID":"d42130a2-25c9-4a81-ae93-1d6219e5794f","Type":"ContainerStarted","Data":"0daf34be838f6eef7528437593a4a56462040c96c91696e9e3f321ad237bc18d"} Oct 31 04:40:04.953076 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:40:04.953040 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13506896_9897_4d94_91df_2d3c6eb87f68.slice/crio-642591e55de8b75b4dfb59022d3b6c0c390b4514db4bed0bcff91332cbaa6506 WatchSource:0}: Error finding container 642591e55de8b75b4dfb59022d3b6c0c390b4514db4bed0bcff91332cbaa6506: Status 404 returned error can't find the container with id 642591e55de8b75b4dfb59022d3b6c0c390b4514db4bed0bcff91332cbaa6506 Oct 31 04:40:04.953269 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:04.953237 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-99ca0c-787d86f4c5-rhkjg"] Oct 31 04:40:05.617352 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:05.617302 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-ded435-5cbbf45d4c-j9btt"] Oct 31 04:40:05.622220 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:05.622192 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-ded435-5cbbf45d4c-j9btt" Oct 31 04:40:05.637085 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:05.637049 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-ded435\"" Oct 31 04:40:05.641104 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:05.641074 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-ded435-5cbbf45d4c-j9btt"] Oct 31 04:40:05.664789 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:05.664746 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-99ca0c-568c687f79-tqpf6" event={"ID":"d42130a2-25c9-4a81-ae93-1d6219e5794f","Type":"ContainerStarted","Data":"927a73ebfe1ecdc9afb0f5c29284d07d828a0da65b9a3cb567f6ac8553d2decf"} Oct 31 04:40:05.667546 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:05.667505 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-99ca0c-568c687f79-tqpf6" Oct 31 04:40:05.670584 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:05.670389 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-99ca0c-787d86f4c5-rhkjg" event={"ID":"13506896-9897-4d94-91df-2d3c6eb87f68","Type":"ContainerStarted","Data":"1117fbfa053037e7c206c57e891f51a6770e1dc6cbfffb6cb1ece22860e27b2f"} Oct 31 04:40:05.670584 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:05.670461 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-99ca0c-787d86f4c5-rhkjg" event={"ID":"13506896-9897-4d94-91df-2d3c6eb87f68","Type":"ContainerStarted","Data":"642591e55de8b75b4dfb59022d3b6c0c390b4514db4bed0bcff91332cbaa6506"} Oct 31 04:40:05.670584 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:05.670511 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-99ca0c-787d86f4c5-rhkjg" Oct 31 04:40:05.684839 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:05.684803 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-ded435-5787fc7f9-c9dqc"] Oct 31 04:40:05.689880 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:05.689850 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ded435-5787fc7f9-c9dqc" Oct 31 04:40:05.710530 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:05.709738 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-ded435-dockercfg-jt9kp\"" Oct 31 04:40:05.711259 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:05.711226 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-ded435-5787fc7f9-c9dqc"] Oct 31 04:40:05.717044 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:05.716755 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-99ca0c-568c687f79-tqpf6" podStartSLOduration=2.716736075 podStartE2EDuration="2.716736075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:40:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:40:05.709860455 +0000 UTC m=+2160.466198495" watchObservedRunningTime="2026-10-31 04:40:05.716736075 +0000 UTC m=+2160.473074116" Oct 31 04:40:05.895984 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:05.895088 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-77a153/auth-test-model-77a153-kserve-6fcd554cc-c5d8v" podUID="655f7ad5-8f26-4d0c-88ac-3507229392c3" containerName="main" containerID="cri-o://f27c8a6c77f65b5ca0ea21ad5ab171fa5d715f99695bdbe612003bd7d95c4cbc" gracePeriod=45 Oct 31 04:40:05.920098 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:05.920047 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74e04a61-5277-4180-bc46-68af7640b25e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-ded435-5787fc7f9-c9dqc\" (UID: \"74e04a61-5277-4180-bc46-68af7640b25e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ded435-5787fc7f9-c9dqc" Oct 31 04:40:05.948536 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:05.948390 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91ac1809-b6e5-4c0e-a324-cfdc8c088e74-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-ded435-5cbbf45d4c-j9btt\" (UID: \"91ac1809-b6e5-4c0e-a324-cfdc8c088e74\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-ded435-5cbbf45d4c-j9btt" Oct 31 04:40:05.958921 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:05.958874 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md48j\" (UniqueName: \"kubernetes.io/projected/91ac1809-b6e5-4c0e-a324-cfdc8c088e74-kube-api-access-md48j\") pod \"payload-processing-e2e-extmod-b-ded435-5cbbf45d4c-j9btt\" (UID: \"91ac1809-b6e5-4c0e-a324-cfdc8c088e74\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-ded435-5cbbf45d4c-j9btt" Oct 31 04:40:05.975891 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:05.964641 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxzzz\" (UniqueName: \"kubernetes.io/projected/74e04a61-5277-4180-bc46-68af7640b25e-kube-api-access-wxzzz\") pod \"payload-pre-processing-e2e-extmod-b-ded435-5787fc7f9-c9dqc\" (UID: \"74e04a61-5277-4180-bc46-68af7640b25e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ded435-5787fc7f9-c9dqc" Oct 31 04:40:05.990123 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:05.988225 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-99ca0c-787d86f4c5-rhkjg" podStartSLOduration=1.988023884 podStartE2EDuration="1.988023884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:40:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:40:05.768775569 +0000 UTC m=+2160.525113610" watchObservedRunningTime="2026-10-31 04:40:05.988023884 +0000 UTC m=+2160.744361923" Oct 31 04:40:06.083763 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:06.083700 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74e04a61-5277-4180-bc46-68af7640b25e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-ded435-5787fc7f9-c9dqc\" (UID: \"74e04a61-5277-4180-bc46-68af7640b25e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ded435-5787fc7f9-c9dqc" Oct 31 04:40:06.083989 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:06.083783 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91ac1809-b6e5-4c0e-a324-cfdc8c088e74-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-ded435-5cbbf45d4c-j9btt\" (UID: \"91ac1809-b6e5-4c0e-a324-cfdc8c088e74\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-ded435-5cbbf45d4c-j9btt" Oct 31 04:40:06.083989 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:06.083949 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md48j\" (UniqueName: \"kubernetes.io/projected/91ac1809-b6e5-4c0e-a324-cfdc8c088e74-kube-api-access-md48j\") pod \"payload-processing-e2e-extmod-b-ded435-5cbbf45d4c-j9btt\" (UID: \"91ac1809-b6e5-4c0e-a324-cfdc8c088e74\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-ded435-5cbbf45d4c-j9btt" Oct 31 04:40:06.084105 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:06.083997 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxzzz\" (UniqueName: \"kubernetes.io/projected/74e04a61-5277-4180-bc46-68af7640b25e-kube-api-access-wxzzz\") pod \"payload-pre-processing-e2e-extmod-b-ded435-5787fc7f9-c9dqc\" (UID: \"74e04a61-5277-4180-bc46-68af7640b25e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ded435-5787fc7f9-c9dqc" Oct 31 04:40:06.084641 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:06.084612 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91ac1809-b6e5-4c0e-a324-cfdc8c088e74-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-ded435-5cbbf45d4c-j9btt\" (UID: \"91ac1809-b6e5-4c0e-a324-cfdc8c088e74\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-ded435-5cbbf45d4c-j9btt" Oct 31 04:40:06.084791 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:06.084766 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74e04a61-5277-4180-bc46-68af7640b25e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-ded435-5787fc7f9-c9dqc\" (UID: \"74e04a61-5277-4180-bc46-68af7640b25e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ded435-5787fc7f9-c9dqc" Oct 31 04:40:06.108702 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:06.108665 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxzzz\" (UniqueName: \"kubernetes.io/projected/74e04a61-5277-4180-bc46-68af7640b25e-kube-api-access-wxzzz\") pod \"payload-pre-processing-e2e-extmod-b-ded435-5787fc7f9-c9dqc\" (UID: \"74e04a61-5277-4180-bc46-68af7640b25e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ded435-5787fc7f9-c9dqc" Oct 31 04:40:06.108873 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:06.108664 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-md48j\" (UniqueName: \"kubernetes.io/projected/91ac1809-b6e5-4c0e-a324-cfdc8c088e74-kube-api-access-md48j\") pod \"payload-processing-e2e-extmod-b-ded435-5cbbf45d4c-j9btt\" (UID: \"91ac1809-b6e5-4c0e-a324-cfdc8c088e74\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-ded435-5cbbf45d4c-j9btt" Oct 31 04:40:06.238874 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:06.238614 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-ded435-5cbbf45d4c-j9btt" Oct 31 04:40:06.314881 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:06.314264 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ded435-5787fc7f9-c9dqc" Oct 31 04:40:06.493370 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:06.474574 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-77a153/auth-test-model-77a153-kserve-6fcd554cc-c5d8v" Oct 31 04:40:06.549558 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:06.549518 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/655f7ad5-8f26-4d0c-88ac-3507229392c3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/655f7ad5-8f26-4d0c-88ac-3507229392c3-tmp-dir\") pod \"655f7ad5-8f26-4d0c-88ac-3507229392c3\" (UID: \"655f7ad5-8f26-4d0c-88ac-3507229392c3\") " Oct 31 04:40:06.549878 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:06.549851 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/655f7ad5-8f26-4d0c-88ac-3507229392c3-kube-api-access-fkszn\" (UniqueName: \"kubernetes.io/projected/655f7ad5-8f26-4d0c-88ac-3507229392c3-kube-api-access-fkszn\") pod \"655f7ad5-8f26-4d0c-88ac-3507229392c3\" (UID: \"655f7ad5-8f26-4d0c-88ac-3507229392c3\") " Oct 31 04:40:06.550357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:06.550314 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/655f7ad5-8f26-4d0c-88ac-3507229392c3-tls-certs\" (UniqueName: \"kubernetes.io/secret/655f7ad5-8f26-4d0c-88ac-3507229392c3-tls-certs\") pod \"655f7ad5-8f26-4d0c-88ac-3507229392c3\" (UID: \"655f7ad5-8f26-4d0c-88ac-3507229392c3\") " Oct 31 04:40:06.551112 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:06.551047 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/655f7ad5-8f26-4d0c-88ac-3507229392c3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/655f7ad5-8f26-4d0c-88ac-3507229392c3-model-cache\") pod \"655f7ad5-8f26-4d0c-88ac-3507229392c3\" (UID: \"655f7ad5-8f26-4d0c-88ac-3507229392c3\") " Oct 31 04:40:06.551214 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:06.551202 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/655f7ad5-8f26-4d0c-88ac-3507229392c3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/655f7ad5-8f26-4d0c-88ac-3507229392c3-dshm\") pod \"655f7ad5-8f26-4d0c-88ac-3507229392c3\" (UID: \"655f7ad5-8f26-4d0c-88ac-3507229392c3\") " Oct 31 04:40:06.551383 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:06.551321 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/655f7ad5-8f26-4d0c-88ac-3507229392c3-model-cache" pod "655f7ad5-8f26-4d0c-88ac-3507229392c3" (UID: "655f7ad5-8f26-4d0c-88ac-3507229392c3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:40:06.551489 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:06.551467 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/655f7ad5-8f26-4d0c-88ac-3507229392c3-home\" (UniqueName: \"kubernetes.io/empty-dir/655f7ad5-8f26-4d0c-88ac-3507229392c3-home\") pod \"655f7ad5-8f26-4d0c-88ac-3507229392c3\" (UID: \"655f7ad5-8f26-4d0c-88ac-3507229392c3\") " Oct 31 04:40:06.552977 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:06.552060 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/655f7ad5-8f26-4d0c-88ac-3507229392c3-home" pod "655f7ad5-8f26-4d0c-88ac-3507229392c3" (UID: "655f7ad5-8f26-4d0c-88ac-3507229392c3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:40:06.552977 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:06.552518 5019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/655f7ad5-8f26-4d0c-88ac-3507229392c3-home\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:40:06.552977 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:06.552541 5019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/655f7ad5-8f26-4d0c-88ac-3507229392c3-model-cache\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:40:06.553261 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:06.553224 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/655f7ad5-8f26-4d0c-88ac-3507229392c3-tmp-dir" pod "655f7ad5-8f26-4d0c-88ac-3507229392c3" (UID: "655f7ad5-8f26-4d0c-88ac-3507229392c3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:40:06.568151 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:06.568107 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/655f7ad5-8f26-4d0c-88ac-3507229392c3-kube-api-access-fkszn" pod "655f7ad5-8f26-4d0c-88ac-3507229392c3" (UID: "655f7ad5-8f26-4d0c-88ac-3507229392c3"). InnerVolumeSpecName "kube-api-access-fkszn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:40:06.568380 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:06.568271 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/655f7ad5-8f26-4d0c-88ac-3507229392c3-tls-certs" pod "655f7ad5-8f26-4d0c-88ac-3507229392c3" (UID: "655f7ad5-8f26-4d0c-88ac-3507229392c3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:40:06.572917 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:06.572877 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/655f7ad5-8f26-4d0c-88ac-3507229392c3-dshm" pod "655f7ad5-8f26-4d0c-88ac-3507229392c3" (UID: "655f7ad5-8f26-4d0c-88ac-3507229392c3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:40:06.654208 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:06.654083 5019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/655f7ad5-8f26-4d0c-88ac-3507229392c3-tls-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:40:06.654208 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:06.654137 5019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/655f7ad5-8f26-4d0c-88ac-3507229392c3-dshm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:40:06.654208 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:06.654151 5019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/655f7ad5-8f26-4d0c-88ac-3507229392c3-tmp-dir\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:40:06.654208 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:06.654165 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkszn\" (UniqueName: \"kubernetes.io/projected/655f7ad5-8f26-4d0c-88ac-3507229392c3-kube-api-access-fkszn\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:40:06.685197 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:06.685144 5019 generic.go:356] "Generic (PLEG): container finished" podID="655f7ad5-8f26-4d0c-88ac-3507229392c3" containerID="f27c8a6c77f65b5ca0ea21ad5ab171fa5d715f99695bdbe612003bd7d95c4cbc" exitCode=0 Oct 31 04:40:06.687417 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:06.686324 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-77a153/auth-test-model-77a153-kserve-6fcd554cc-c5d8v" Oct 31 04:40:06.687615 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:06.687541 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-77a153/auth-test-model-77a153-kserve-6fcd554cc-c5d8v" event={"ID":"655f7ad5-8f26-4d0c-88ac-3507229392c3","Type":"ContainerDied","Data":"f27c8a6c77f65b5ca0ea21ad5ab171fa5d715f99695bdbe612003bd7d95c4cbc"} Oct 31 04:40:06.687615 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:06.687604 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-77a153/auth-test-model-77a153-kserve-6fcd554cc-c5d8v" event={"ID":"655f7ad5-8f26-4d0c-88ac-3507229392c3","Type":"ContainerDied","Data":"e6c416121ec3b5aeed1f7b7dd2619db1c185c3324cb3c3f0b35e63e331232370"} Oct 31 04:40:06.687743 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:06.687631 5019 scope.go:122] "RemoveContainer" containerID="f27c8a6c77f65b5ca0ea21ad5ab171fa5d715f99695bdbe612003bd7d95c4cbc" Oct 31 04:40:06.707709 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:06.707677 5019 scope.go:122] "RemoveContainer" containerID="f27c8a6c77f65b5ca0ea21ad5ab171fa5d715f99695bdbe612003bd7d95c4cbc" Oct 31 04:40:06.708303 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:40:06.708280 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f27c8a6c77f65b5ca0ea21ad5ab171fa5d715f99695bdbe612003bd7d95c4cbc\": container with ID starting with f27c8a6c77f65b5ca0ea21ad5ab171fa5d715f99695bdbe612003bd7d95c4cbc not found: ID does not exist" containerID="f27c8a6c77f65b5ca0ea21ad5ab171fa5d715f99695bdbe612003bd7d95c4cbc" Oct 31 04:40:06.708851 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:06.708822 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f27c8a6c77f65b5ca0ea21ad5ab171fa5d715f99695bdbe612003bd7d95c4cbc"} err="failed to get container status \"f27c8a6c77f65b5ca0ea21ad5ab171fa5d715f99695bdbe612003bd7d95c4cbc\": rpc error: code = NotFound desc = could not find container \"f27c8a6c77f65b5ca0ea21ad5ab171fa5d715f99695bdbe612003bd7d95c4cbc\": container with ID starting with f27c8a6c77f65b5ca0ea21ad5ab171fa5d715f99695bdbe612003bd7d95c4cbc not found: ID does not exist" Oct 31 04:40:06.745413 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:06.745310 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-77a153/auth-test-model-77a153-kserve-6fcd554cc-c5d8v"] Oct 31 04:40:06.752610 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:06.752560 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-77a153/auth-test-model-77a153-kserve-6fcd554cc-c5d8v"] Oct 31 04:40:06.776571 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:40:06.776532 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91ac1809_b6e5_4c0e_a324_cfdc8c088e74.slice/crio-c53bdadb908f29cd2f26efa882ed52b321760a89a9af6a17a1e5064093d762e0 WatchSource:0}: Error finding container c53bdadb908f29cd2f26efa882ed52b321760a89a9af6a17a1e5064093d762e0: Status 404 returned error can't find the container with id c53bdadb908f29cd2f26efa882ed52b321760a89a9af6a17a1e5064093d762e0 Oct 31 04:40:06.776830 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:06.776812 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-ded435-5cbbf45d4c-j9btt"] Oct 31 04:40:06.881012 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:06.880974 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-ded435-5787fc7f9-c9dqc"] Oct 31 04:40:06.883285 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:40:06.883252 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74e04a61_5277_4180_bc46_68af7640b25e.slice/crio-da7e56fa98d953fac990d82d820bd1f840e83df0f95901f56d5148d4332805cf WatchSource:0}: Error finding container da7e56fa98d953fac990d82d820bd1f840e83df0f95901f56d5148d4332805cf: Status 404 returned error can't find the container with id da7e56fa98d953fac990d82d820bd1f840e83df0f95901f56d5148d4332805cf Oct 31 04:40:07.293157 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:07.292872 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-77a153/auth-test-model-77a153-kserve-6fcd554cc-c5d8v" podUID="655f7ad5-8f26-4d0c-88ac-3507229392c3" containerName="main" probeResult="failure" output="Get \"https://10.132.1.0:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:40:07.410466 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:07.410416 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="655f7ad5-8f26-4d0c-88ac-3507229392c3" path="/var/lib/kubelet/pods/655f7ad5-8f26-4d0c-88ac-3507229392c3/volumes" Oct 31 04:40:07.695609 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:07.695502 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-ded435-5cbbf45d4c-j9btt" event={"ID":"91ac1809-b6e5-4c0e-a324-cfdc8c088e74","Type":"ContainerStarted","Data":"91cb972e9136e8af3bb377ede3771c827c383ec6692bae6165ed723119b92755"} Oct 31 04:40:07.695609 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:07.695564 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-ded435-5cbbf45d4c-j9btt" event={"ID":"91ac1809-b6e5-4c0e-a324-cfdc8c088e74","Type":"ContainerStarted","Data":"c53bdadb908f29cd2f26efa882ed52b321760a89a9af6a17a1e5064093d762e0"} Oct 31 04:40:07.695609 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:07.695595 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-ded435-5cbbf45d4c-j9btt" Oct 31 04:40:07.699136 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:07.699101 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ded435-5787fc7f9-c9dqc" event={"ID":"74e04a61-5277-4180-bc46-68af7640b25e","Type":"ContainerStarted","Data":"d664a380cf9fdd7c09c421f5e2960a6b2db4cc531109ee9946434ab68acc60f0"} Oct 31 04:40:07.699287 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:07.699156 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ded435-5787fc7f9-c9dqc" event={"ID":"74e04a61-5277-4180-bc46-68af7640b25e","Type":"ContainerStarted","Data":"da7e56fa98d953fac990d82d820bd1f840e83df0f95901f56d5148d4332805cf"} Oct 31 04:40:07.699287 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:07.699185 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ded435-5787fc7f9-c9dqc" Oct 31 04:40:07.739024 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:07.738963 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-ded435-5cbbf45d4c-j9btt" podStartSLOduration=2.738946453 podStartE2EDuration="2.738946453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:40:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:40:07.734778873 +0000 UTC m=+2162.491116930" watchObservedRunningTime="2026-10-31 04:40:07.738946453 +0000 UTC m=+2162.495284539" Oct 31 04:40:07.776746 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:07.776680 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ded435-5787fc7f9-c9dqc" podStartSLOduration=2.7766606879999998 podStartE2EDuration="2.776660688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:40:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:40:07.774242052 +0000 UTC m=+2162.530580093" watchObservedRunningTime="2026-10-31 04:40:07.776660688 +0000 UTC m=+2162.532998731" Oct 31 04:40:09.866605 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:09.866562 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk"] Oct 31 04:40:09.867608 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:09.867587 5019 cpu_manager.go:414] "containerMap: removing container" podUID="655f7ad5-8f26-4d0c-88ac-3507229392c3" containerName="main" Oct 31 04:40:09.867608 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:09.867608 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="655f7ad5-8f26-4d0c-88ac-3507229392c3" containerName="main" Oct 31 04:40:09.867755 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:09.867746 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="655f7ad5-8f26-4d0c-88ac-3507229392c3" containerName="main" Oct 31 04:40:09.874916 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:09.873557 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk" Oct 31 04:40:09.892688 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:09.892646 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-a99f2a-openshift-default-dockercfg-xn9hq\"" Oct 31 04:40:09.920506 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:09.920461 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk"] Oct 31 04:40:10.080423 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:10.077770 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17f31b2c-2988-45e1-92ae-e43ff492444a-istio-podinfo\") pod \"e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk\" (UID: \"17f31b2c-2988-45e1-92ae-e43ff492444a\") " pod="openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk" Oct 31 04:40:10.080423 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:10.077849 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17f31b2c-2988-45e1-92ae-e43ff492444a-istio-token\") pod \"e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk\" (UID: \"17f31b2c-2988-45e1-92ae-e43ff492444a\") " pod="openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk" Oct 31 04:40:10.080423 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:10.077943 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f31b2c-2988-45e1-92ae-e43ff492444a-workload-socket\") pod \"e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk\" (UID: \"17f31b2c-2988-45e1-92ae-e43ff492444a\") " pod="openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk" Oct 31 04:40:10.080423 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:10.078104 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17f31b2c-2988-45e1-92ae-e43ff492444a-istio-envoy\") pod \"e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk\" (UID: \"17f31b2c-2988-45e1-92ae-e43ff492444a\") " pod="openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk" Oct 31 04:40:10.080423 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:10.078130 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17f31b2c-2988-45e1-92ae-e43ff492444a-istiod-ca-cert\") pod \"e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk\" (UID: \"17f31b2c-2988-45e1-92ae-e43ff492444a\") " pod="openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk" Oct 31 04:40:10.080423 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:10.078174 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17f31b2c-2988-45e1-92ae-e43ff492444a-workload-certs\") pod \"e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk\" (UID: \"17f31b2c-2988-45e1-92ae-e43ff492444a\") " pod="openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk" Oct 31 04:40:10.080423 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:10.078272 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17f31b2c-2988-45e1-92ae-e43ff492444a-istio-data\") pod \"e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk\" (UID: \"17f31b2c-2988-45e1-92ae-e43ff492444a\") " pod="openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk" Oct 31 04:40:10.080423 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:10.078295 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f31b2c-2988-45e1-92ae-e43ff492444a-credential-socket\") pod \"e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk\" (UID: \"17f31b2c-2988-45e1-92ae-e43ff492444a\") " pod="openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk" Oct 31 04:40:10.080423 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:10.078430 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7h78\" (UniqueName: \"kubernetes.io/projected/17f31b2c-2988-45e1-92ae-e43ff492444a-kube-api-access-v7h78\") pod \"e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk\" (UID: \"17f31b2c-2988-45e1-92ae-e43ff492444a\") " pod="openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk" Oct 31 04:40:10.182361 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:10.181500 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17f31b2c-2988-45e1-92ae-e43ff492444a-istio-envoy\") pod \"e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk\" (UID: \"17f31b2c-2988-45e1-92ae-e43ff492444a\") " pod="openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk" Oct 31 04:40:10.182361 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:10.181561 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17f31b2c-2988-45e1-92ae-e43ff492444a-istiod-ca-cert\") pod \"e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk\" (UID: \"17f31b2c-2988-45e1-92ae-e43ff492444a\") " pod="openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk" Oct 31 04:40:10.182361 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:10.181614 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17f31b2c-2988-45e1-92ae-e43ff492444a-workload-certs\") pod \"e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk\" (UID: \"17f31b2c-2988-45e1-92ae-e43ff492444a\") " pod="openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk" Oct 31 04:40:10.182361 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:10.181709 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17f31b2c-2988-45e1-92ae-e43ff492444a-istio-data\") pod \"e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk\" (UID: \"17f31b2c-2988-45e1-92ae-e43ff492444a\") " pod="openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk" Oct 31 04:40:10.182361 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:10.181735 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f31b2c-2988-45e1-92ae-e43ff492444a-credential-socket\") pod \"e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk\" (UID: \"17f31b2c-2988-45e1-92ae-e43ff492444a\") " pod="openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk" Oct 31 04:40:10.182361 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:10.181804 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7h78\" (UniqueName: \"kubernetes.io/projected/17f31b2c-2988-45e1-92ae-e43ff492444a-kube-api-access-v7h78\") pod \"e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk\" (UID: \"17f31b2c-2988-45e1-92ae-e43ff492444a\") " pod="openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk" Oct 31 04:40:10.182361 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:10.181937 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17f31b2c-2988-45e1-92ae-e43ff492444a-istio-podinfo\") pod \"e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk\" (UID: \"17f31b2c-2988-45e1-92ae-e43ff492444a\") " pod="openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk" Oct 31 04:40:10.182361 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:10.181973 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17f31b2c-2988-45e1-92ae-e43ff492444a-istio-token\") pod \"e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk\" (UID: \"17f31b2c-2988-45e1-92ae-e43ff492444a\") " pod="openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk" Oct 31 04:40:10.182361 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:10.182062 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f31b2c-2988-45e1-92ae-e43ff492444a-workload-socket\") pod \"e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk\" (UID: \"17f31b2c-2988-45e1-92ae-e43ff492444a\") " pod="openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk" Oct 31 04:40:10.183007 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:10.182678 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f31b2c-2988-45e1-92ae-e43ff492444a-workload-socket\") pod \"e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk\" (UID: \"17f31b2c-2988-45e1-92ae-e43ff492444a\") " pod="openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk" Oct 31 04:40:10.189357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:10.183421 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f31b2c-2988-45e1-92ae-e43ff492444a-credential-socket\") pod \"e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk\" (UID: \"17f31b2c-2988-45e1-92ae-e43ff492444a\") " pod="openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk" Oct 31 04:40:10.189357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:10.184201 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17f31b2c-2988-45e1-92ae-e43ff492444a-istiod-ca-cert\") pod \"e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk\" (UID: \"17f31b2c-2988-45e1-92ae-e43ff492444a\") " pod="openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk" Oct 31 04:40:10.189357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:10.184484 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17f31b2c-2988-45e1-92ae-e43ff492444a-workload-certs\") pod \"e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk\" (UID: \"17f31b2c-2988-45e1-92ae-e43ff492444a\") " pod="openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk" Oct 31 04:40:10.189357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:10.184740 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17f31b2c-2988-45e1-92ae-e43ff492444a-istio-data\") pod \"e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk\" (UID: \"17f31b2c-2988-45e1-92ae-e43ff492444a\") " pod="openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk" Oct 31 04:40:10.213367 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:10.211224 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17f31b2c-2988-45e1-92ae-e43ff492444a-istio-podinfo\") pod \"e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk\" (UID: \"17f31b2c-2988-45e1-92ae-e43ff492444a\") " pod="openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk" Oct 31 04:40:10.226315 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:10.223015 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17f31b2c-2988-45e1-92ae-e43ff492444a-istio-envoy\") pod \"e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk\" (UID: \"17f31b2c-2988-45e1-92ae-e43ff492444a\") " pod="openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk" Oct 31 04:40:10.227133 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:10.226878 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7h78\" (UniqueName: \"kubernetes.io/projected/17f31b2c-2988-45e1-92ae-e43ff492444a-kube-api-access-v7h78\") pod \"e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk\" (UID: \"17f31b2c-2988-45e1-92ae-e43ff492444a\") " pod="openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk" Oct 31 04:40:10.227133 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:10.227102 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17f31b2c-2988-45e1-92ae-e43ff492444a-istio-token\") pod \"e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk\" (UID: \"17f31b2c-2988-45e1-92ae-e43ff492444a\") " pod="openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk" Oct 31 04:40:10.516768 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:10.516725 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk" Oct 31 04:40:10.866297 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:40:10.866255 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17f31b2c_2988_45e1_92ae_e43ff492444a.slice/crio-2333c1006ca0e0be3207a87a376cf26f332f84b79178a890a949132db589a3a6 WatchSource:0}: Error finding container 2333c1006ca0e0be3207a87a376cf26f332f84b79178a890a949132db589a3a6: Status 404 returned error can't find the container with id 2333c1006ca0e0be3207a87a376cf26f332f84b79178a890a949132db589a3a6 Oct 31 04:40:10.867035 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:10.867005 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk"] Oct 31 04:40:10.869766 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:10.869727 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:40:10.869845 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:10.869813 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:40:10.869937 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:10.869847 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:40:11.738727 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:11.738689 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk" event={"ID":"17f31b2c-2988-45e1-92ae-e43ff492444a","Type":"ContainerStarted","Data":"c6af17490a3d2c22c621122f5463b2021aa83b76ff7e6dc4b6dfd89f99ac1689"} Oct 31 04:40:11.738912 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:11.738758 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk" event={"ID":"17f31b2c-2988-45e1-92ae-e43ff492444a","Type":"ContainerStarted","Data":"2333c1006ca0e0be3207a87a376cf26f332f84b79178a890a949132db589a3a6"} Oct 31 04:40:11.801198 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:11.798481 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk" podStartSLOduration=2.7984599660000002 podStartE2EDuration="2.798459966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:40:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:40:11.795217554 +0000 UTC m=+2166.551555594" watchObservedRunningTime="2026-10-31 04:40:11.798459966 +0000 UTC m=+2166.554798007" Oct 31 04:40:12.307571 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:12.307513 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-d6ac82/auth-test-model-d6ac82-kserve-5cc8848985-xm5bx" podUID="7b71880a-8282-45bf-97f1-dc8edc6f4a9b" containerName="main" containerID="cri-o://094d2a4d3defd6bfc581e3e2266e0df07797f292fb77e9b18a8c5034ed3e71a4" gracePeriod=45 Oct 31 04:40:12.522622 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:12.522582 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk" Oct 31 04:40:12.537365 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:12.532052 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk" Oct 31 04:40:12.629022 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:12.628990 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-d6ac82/auth-test-model-d6ac82-kserve-5cc8848985-xm5bx" Oct 31 04:40:12.726776 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:12.726734 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b71880a-8282-45bf-97f1-dc8edc6f4a9b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b71880a-8282-45bf-97f1-dc8edc6f4a9b-dshm\") pod \"7b71880a-8282-45bf-97f1-dc8edc6f4a9b\" (UID: \"7b71880a-8282-45bf-97f1-dc8edc6f4a9b\") " Oct 31 04:40:12.727004 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:12.726891 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b71880a-8282-45bf-97f1-dc8edc6f4a9b-tls-certs\" (UniqueName: \"kubernetes.io/secret/7b71880a-8282-45bf-97f1-dc8edc6f4a9b-tls-certs\") pod \"7b71880a-8282-45bf-97f1-dc8edc6f4a9b\" (UID: \"7b71880a-8282-45bf-97f1-dc8edc6f4a9b\") " Oct 31 04:40:12.727004 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:12.726929 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b71880a-8282-45bf-97f1-dc8edc6f4a9b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b71880a-8282-45bf-97f1-dc8edc6f4a9b-tmp-dir\") pod \"7b71880a-8282-45bf-97f1-dc8edc6f4a9b\" (UID: \"7b71880a-8282-45bf-97f1-dc8edc6f4a9b\") " Oct 31 04:40:12.727004 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:12.726997 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b71880a-8282-45bf-97f1-dc8edc6f4a9b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b71880a-8282-45bf-97f1-dc8edc6f4a9b-model-cache\") pod \"7b71880a-8282-45bf-97f1-dc8edc6f4a9b\" (UID: \"7b71880a-8282-45bf-97f1-dc8edc6f4a9b\") " Oct 31 04:40:12.727204 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:12.727065 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b71880a-8282-45bf-97f1-dc8edc6f4a9b-kube-api-access-lbwv9\" (UniqueName: \"kubernetes.io/projected/7b71880a-8282-45bf-97f1-dc8edc6f4a9b-kube-api-access-lbwv9\") pod \"7b71880a-8282-45bf-97f1-dc8edc6f4a9b\" (UID: \"7b71880a-8282-45bf-97f1-dc8edc6f4a9b\") " Oct 31 04:40:12.727270 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:12.727198 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b71880a-8282-45bf-97f1-dc8edc6f4a9b-home\" (UniqueName: \"kubernetes.io/empty-dir/7b71880a-8282-45bf-97f1-dc8edc6f4a9b-home\") pod \"7b71880a-8282-45bf-97f1-dc8edc6f4a9b\" (UID: \"7b71880a-8282-45bf-97f1-dc8edc6f4a9b\") " Oct 31 04:40:12.727419 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:12.727385 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b71880a-8282-45bf-97f1-dc8edc6f4a9b-tmp-dir" pod "7b71880a-8282-45bf-97f1-dc8edc6f4a9b" (UID: "7b71880a-8282-45bf-97f1-dc8edc6f4a9b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:40:12.727719 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:12.727695 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b71880a-8282-45bf-97f1-dc8edc6f4a9b-home" pod "7b71880a-8282-45bf-97f1-dc8edc6f4a9b" (UID: "7b71880a-8282-45bf-97f1-dc8edc6f4a9b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:40:12.727835 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:12.727810 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b71880a-8282-45bf-97f1-dc8edc6f4a9b-model-cache" pod "7b71880a-8282-45bf-97f1-dc8edc6f4a9b" (UID: "7b71880a-8282-45bf-97f1-dc8edc6f4a9b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:40:12.728061 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:12.728036 5019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b71880a-8282-45bf-97f1-dc8edc6f4a9b-tmp-dir\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:40:12.728061 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:12.728059 5019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b71880a-8282-45bf-97f1-dc8edc6f4a9b-model-cache\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:40:12.728234 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:12.728073 5019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b71880a-8282-45bf-97f1-dc8edc6f4a9b-home\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:40:12.730023 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:12.729987 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b71880a-8282-45bf-97f1-dc8edc6f4a9b-dshm" pod "7b71880a-8282-45bf-97f1-dc8edc6f4a9b" (UID: "7b71880a-8282-45bf-97f1-dc8edc6f4a9b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:40:12.730239 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:12.730203 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b71880a-8282-45bf-97f1-dc8edc6f4a9b-tls-certs" pod "7b71880a-8282-45bf-97f1-dc8edc6f4a9b" (UID: "7b71880a-8282-45bf-97f1-dc8edc6f4a9b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:40:12.730531 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:12.730263 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b71880a-8282-45bf-97f1-dc8edc6f4a9b-kube-api-access-lbwv9" pod "7b71880a-8282-45bf-97f1-dc8edc6f4a9b" (UID: "7b71880a-8282-45bf-97f1-dc8edc6f4a9b"). InnerVolumeSpecName "kube-api-access-lbwv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:40:12.751172 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:12.751133 5019 generic.go:356] "Generic (PLEG): container finished" podID="7b71880a-8282-45bf-97f1-dc8edc6f4a9b" containerID="094d2a4d3defd6bfc581e3e2266e0df07797f292fb77e9b18a8c5034ed3e71a4" exitCode=0 Oct 31 04:40:12.751428 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:12.751180 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-d6ac82/auth-test-model-d6ac82-kserve-5cc8848985-xm5bx" Oct 31 04:40:12.751428 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:12.751225 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-d6ac82/auth-test-model-d6ac82-kserve-5cc8848985-xm5bx" event={"ID":"7b71880a-8282-45bf-97f1-dc8edc6f4a9b","Type":"ContainerDied","Data":"094d2a4d3defd6bfc581e3e2266e0df07797f292fb77e9b18a8c5034ed3e71a4"} Oct 31 04:40:12.751428 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:12.751278 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-d6ac82/auth-test-model-d6ac82-kserve-5cc8848985-xm5bx" event={"ID":"7b71880a-8282-45bf-97f1-dc8edc6f4a9b","Type":"ContainerDied","Data":"2c79920d1448181ab8128b46db4195550c34882a6946b5a34c72469358562f89"} Oct 31 04:40:12.751428 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:12.751302 5019 scope.go:122] "RemoveContainer" containerID="094d2a4d3defd6bfc581e3e2266e0df07797f292fb77e9b18a8c5034ed3e71a4" Oct 31 04:40:12.752266 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:12.751896 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk" Oct 31 04:40:12.753223 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:12.753191 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk" Oct 31 04:40:12.766868 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:12.766843 5019 scope.go:122] "RemoveContainer" containerID="094d2a4d3defd6bfc581e3e2266e0df07797f292fb77e9b18a8c5034ed3e71a4" Oct 31 04:40:12.767257 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:40:12.767232 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"094d2a4d3defd6bfc581e3e2266e0df07797f292fb77e9b18a8c5034ed3e71a4\": container with ID starting with 094d2a4d3defd6bfc581e3e2266e0df07797f292fb77e9b18a8c5034ed3e71a4 not found: ID does not exist" containerID="094d2a4d3defd6bfc581e3e2266e0df07797f292fb77e9b18a8c5034ed3e71a4" Oct 31 04:40:12.767368 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:12.767270 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"094d2a4d3defd6bfc581e3e2266e0df07797f292fb77e9b18a8c5034ed3e71a4"} err="failed to get container status \"094d2a4d3defd6bfc581e3e2266e0df07797f292fb77e9b18a8c5034ed3e71a4\": rpc error: code = NotFound desc = could not find container \"094d2a4d3defd6bfc581e3e2266e0df07797f292fb77e9b18a8c5034ed3e71a4\": container with ID starting with 094d2a4d3defd6bfc581e3e2266e0df07797f292fb77e9b18a8c5034ed3e71a4 not found: ID does not exist" Oct 31 04:40:12.807759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:12.807713 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-d6ac82/auth-test-model-d6ac82-kserve-5cc8848985-xm5bx"] Oct 31 04:40:12.814991 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:12.814948 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-d6ac82/auth-test-model-d6ac82-kserve-5cc8848985-xm5bx"] Oct 31 04:40:12.830165 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:12.830114 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbwv9\" (UniqueName: \"kubernetes.io/projected/7b71880a-8282-45bf-97f1-dc8edc6f4a9b-kube-api-access-lbwv9\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:40:12.830165 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:12.830157 5019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b71880a-8282-45bf-97f1-dc8edc6f4a9b-dshm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:40:12.830381 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:12.830189 5019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b71880a-8282-45bf-97f1-dc8edc6f4a9b-tls-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:40:13.410924 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:13.410883 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b71880a-8282-45bf-97f1-dc8edc6f4a9b" path="/var/lib/kubelet/pods/7b71880a-8282-45bf-97f1-dc8edc6f4a9b/volumes" Oct 31 04:40:16.085919 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:16.085879 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a99f2a-74bfdc65dc-9j694"] Oct 31 04:40:16.087053 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:16.087031 5019 cpu_manager.go:414] "containerMap: removing container" podUID="7b71880a-8282-45bf-97f1-dc8edc6f4a9b" containerName="main" Oct 31 04:40:16.087141 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:16.087055 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b71880a-8282-45bf-97f1-dc8edc6f4a9b" containerName="main" Oct 31 04:40:16.087270 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:16.087257 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b71880a-8282-45bf-97f1-dc8edc6f4a9b" containerName="main" Oct 31 04:40:16.093191 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:16.093167 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a99f2a-74bfdc65dc-9j694" Oct 31 04:40:16.107839 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:16.107807 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a99f2a-74bfdc65dc-9j694"] Oct 31 04:40:16.113873 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:16.113842 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-a99f2a\"" Oct 31 04:40:16.114026 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:16.113934 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-a99f2a-dockercfg-cjgwf\"" Oct 31 04:40:16.192353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:16.192041 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91300bb2-7008-42f1-ba58-f7ab87e1d748-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-a99f2a-74bfdc65dc-9j694\" (UID: \"91300bb2-7008-42f1-ba58-f7ab87e1d748\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a99f2a-74bfdc65dc-9j694" Oct 31 04:40:16.192353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:16.192185 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdjzw\" (UniqueName: \"kubernetes.io/projected/91300bb2-7008-42f1-ba58-f7ab87e1d748-kube-api-access-vdjzw\") pod \"payload-pre-processing-e2e-extmod-b-a99f2a-74bfdc65dc-9j694\" (UID: \"91300bb2-7008-42f1-ba58-f7ab87e1d748\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a99f2a-74bfdc65dc-9j694" Oct 31 04:40:16.409084 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:16.406157 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91300bb2-7008-42f1-ba58-f7ab87e1d748-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-a99f2a-74bfdc65dc-9j694\" (UID: \"91300bb2-7008-42f1-ba58-f7ab87e1d748\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a99f2a-74bfdc65dc-9j694" Oct 31 04:40:16.409839 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:16.409807 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdjzw\" (UniqueName: \"kubernetes.io/projected/91300bb2-7008-42f1-ba58-f7ab87e1d748-kube-api-access-vdjzw\") pod \"payload-pre-processing-e2e-extmod-b-a99f2a-74bfdc65dc-9j694\" (UID: \"91300bb2-7008-42f1-ba58-f7ab87e1d748\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a99f2a-74bfdc65dc-9j694" Oct 31 04:40:16.410798 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:16.407070 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91300bb2-7008-42f1-ba58-f7ab87e1d748-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-a99f2a-74bfdc65dc-9j694\" (UID: \"91300bb2-7008-42f1-ba58-f7ab87e1d748\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a99f2a-74bfdc65dc-9j694" Oct 31 04:40:16.434242 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:16.434063 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a99f2a-5bb987db5-2m54d"] Oct 31 04:40:16.441258 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:16.441218 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-a99f2a-5bb987db5-2m54d" Oct 31 04:40:16.441485 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:16.441368 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a99f2a-5bb987db5-2m54d"] Oct 31 04:40:16.460375 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:16.457962 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdjzw\" (UniqueName: \"kubernetes.io/projected/91300bb2-7008-42f1-ba58-f7ab87e1d748-kube-api-access-vdjzw\") pod \"payload-pre-processing-e2e-extmod-b-a99f2a-74bfdc65dc-9j694\" (UID: \"91300bb2-7008-42f1-ba58-f7ab87e1d748\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a99f2a-74bfdc65dc-9j694" Oct 31 04:40:16.512353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:16.512146 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsjdn\" (UniqueName: \"kubernetes.io/projected/46aae214-de7a-4736-9aad-3b5499aabd6f-kube-api-access-tsjdn\") pod \"payload-processing-e2e-extmod-b-a99f2a-5bb987db5-2m54d\" (UID: \"46aae214-de7a-4736-9aad-3b5499aabd6f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a99f2a-5bb987db5-2m54d" Oct 31 04:40:16.512575 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:16.512365 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46aae214-de7a-4736-9aad-3b5499aabd6f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-a99f2a-5bb987db5-2m54d\" (UID: \"46aae214-de7a-4736-9aad-3b5499aabd6f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a99f2a-5bb987db5-2m54d" Oct 31 04:40:16.614058 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:16.614022 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsjdn\" (UniqueName: \"kubernetes.io/projected/46aae214-de7a-4736-9aad-3b5499aabd6f-kube-api-access-tsjdn\") pod \"payload-processing-e2e-extmod-b-a99f2a-5bb987db5-2m54d\" (UID: \"46aae214-de7a-4736-9aad-3b5499aabd6f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a99f2a-5bb987db5-2m54d" Oct 31 04:40:16.614279 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:16.614123 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46aae214-de7a-4736-9aad-3b5499aabd6f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-a99f2a-5bb987db5-2m54d\" (UID: \"46aae214-de7a-4736-9aad-3b5499aabd6f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a99f2a-5bb987db5-2m54d" Oct 31 04:40:16.614956 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:16.614930 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46aae214-de7a-4736-9aad-3b5499aabd6f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-a99f2a-5bb987db5-2m54d\" (UID: \"46aae214-de7a-4736-9aad-3b5499aabd6f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a99f2a-5bb987db5-2m54d" Oct 31 04:40:16.640273 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:16.640238 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsjdn\" (UniqueName: \"kubernetes.io/projected/46aae214-de7a-4736-9aad-3b5499aabd6f-kube-api-access-tsjdn\") pod \"payload-processing-e2e-extmod-b-a99f2a-5bb987db5-2m54d\" (UID: \"46aae214-de7a-4736-9aad-3b5499aabd6f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a99f2a-5bb987db5-2m54d" Oct 31 04:40:16.689814 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:16.689725 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-99ca0c-787d86f4c5-rhkjg" Oct 31 04:40:16.711904 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:16.711858 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a99f2a-74bfdc65dc-9j694" Oct 31 04:40:16.762089 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:16.761762 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-a99f2a-5bb987db5-2m54d" Oct 31 04:40:17.039480 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:40:17.039448 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91300bb2_7008_42f1_ba58_f7ab87e1d748.slice/crio-9800ce632df4fb6b98c4e3b881491ff12f1217cee7f648b89381c65567b89eb6 WatchSource:0}: Error finding container 9800ce632df4fb6b98c4e3b881491ff12f1217cee7f648b89381c65567b89eb6: Status 404 returned error can't find the container with id 9800ce632df4fb6b98c4e3b881491ff12f1217cee7f648b89381c65567b89eb6 Oct 31 04:40:17.039879 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:17.039859 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a99f2a-74bfdc65dc-9j694"] Oct 31 04:40:17.171572 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:40:17.171523 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46aae214_de7a_4736_9aad_3b5499aabd6f.slice/crio-e97d069493cf40db3489d0abd3c18a088b20c8b8725f2d0c406856bc80bf27e4 WatchSource:0}: Error finding container e97d069493cf40db3489d0abd3c18a088b20c8b8725f2d0c406856bc80bf27e4: Status 404 returned error can't find the container with id e97d069493cf40db3489d0abd3c18a088b20c8b8725f2d0c406856bc80bf27e4 Oct 31 04:40:17.172036 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:17.171616 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a99f2a-5bb987db5-2m54d"] Oct 31 04:40:17.703631 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:17.703601 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-99ca0c-568c687f79-tqpf6" Oct 31 04:40:17.805562 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:17.805522 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a99f2a-74bfdc65dc-9j694" event={"ID":"91300bb2-7008-42f1-ba58-f7ab87e1d748","Type":"ContainerStarted","Data":"c00c2a8ba6fe92f2f072929ff59f31195b42edc1837e0ea9aa38b84cb72b2fe1"} Oct 31 04:40:17.805771 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:17.805588 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a99f2a-74bfdc65dc-9j694" event={"ID":"91300bb2-7008-42f1-ba58-f7ab87e1d748","Type":"ContainerStarted","Data":"9800ce632df4fb6b98c4e3b881491ff12f1217cee7f648b89381c65567b89eb6"} Oct 31 04:40:17.805771 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:17.805758 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a99f2a-74bfdc65dc-9j694" Oct 31 04:40:17.807780 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:17.807753 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-a99f2a-5bb987db5-2m54d" event={"ID":"46aae214-de7a-4736-9aad-3b5499aabd6f","Type":"ContainerStarted","Data":"06f3675d63e93c1f1fd74cf95d265d78d62afe73fd6fc65469c319162540ec87"} Oct 31 04:40:17.807932 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:17.807791 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-a99f2a-5bb987db5-2m54d" event={"ID":"46aae214-de7a-4736-9aad-3b5499aabd6f","Type":"ContainerStarted","Data":"e97d069493cf40db3489d0abd3c18a088b20c8b8725f2d0c406856bc80bf27e4"} Oct 31 04:40:17.807932 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:17.807868 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-a99f2a-5bb987db5-2m54d" Oct 31 04:40:17.847038 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:17.846986 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a99f2a-74bfdc65dc-9j694" podStartSLOduration=1.846971807 podStartE2EDuration="1.846971807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:40:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:40:17.846787509 +0000 UTC m=+2172.603125550" watchObservedRunningTime="2026-10-31 04:40:17.846971807 +0000 UTC m=+2172.603309847" Oct 31 04:40:17.888876 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:17.888814 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-a99f2a-5bb987db5-2m54d" podStartSLOduration=1.888792716 podStartE2EDuration="1.888792716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:40:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:40:17.887477588 +0000 UTC m=+2172.643815629" watchObservedRunningTime="2026-10-31 04:40:17.888792716 +0000 UTC m=+2172.645130818" Oct 31 04:40:18.712551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:18.712518 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-ded435-5cbbf45d4c-j9btt" Oct 31 04:40:18.713033 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:18.712608 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ded435-5787fc7f9-c9dqc" Oct 31 04:40:22.397484 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:40:22.397421 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:40:26.450296 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:26.450259 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv"] Oct 31 04:40:26.455537 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:26.455514 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv" Oct 31 04:40:26.497355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:26.492003 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-567ca6-openshift-default-dockercfg-nclf8\"" Oct 31 04:40:26.530713 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:26.530670 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcr9b\" (UniqueName: \"kubernetes.io/projected/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-kube-api-access-bcr9b\") pod \"e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv\" (UID: \"2f84895b-57a1-4c93-b20c-dc3f2ca754b4\") " pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv" Oct 31 04:40:26.530899 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:26.530731 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-workload-certs\") pod \"e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv\" (UID: \"2f84895b-57a1-4c93-b20c-dc3f2ca754b4\") " pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv" Oct 31 04:40:26.530899 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:26.530760 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-istio-token\") pod \"e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv\" (UID: \"2f84895b-57a1-4c93-b20c-dc3f2ca754b4\") " pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv" Oct 31 04:40:26.530899 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:26.530805 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-istio-data\") pod \"e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv\" (UID: \"2f84895b-57a1-4c93-b20c-dc3f2ca754b4\") " pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv" Oct 31 04:40:26.530899 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:26.530829 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-istio-podinfo\") pod \"e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv\" (UID: \"2f84895b-57a1-4c93-b20c-dc3f2ca754b4\") " pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv" Oct 31 04:40:26.531054 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:26.530927 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-credential-socket\") pod \"e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv\" (UID: \"2f84895b-57a1-4c93-b20c-dc3f2ca754b4\") " pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv" Oct 31 04:40:26.531093 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:26.531068 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-istio-envoy\") pod \"e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv\" (UID: \"2f84895b-57a1-4c93-b20c-dc3f2ca754b4\") " pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv" Oct 31 04:40:26.531387 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:26.531368 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-workload-socket\") pod \"e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv\" (UID: \"2f84895b-57a1-4c93-b20c-dc3f2ca754b4\") " pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv" Oct 31 04:40:26.531443 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:26.531412 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-istiod-ca-cert\") pod \"e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv\" (UID: \"2f84895b-57a1-4c93-b20c-dc3f2ca754b4\") " pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv" Oct 31 04:40:26.533758 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:26.533722 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv"] Oct 31 04:40:26.638385 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:26.637518 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-istio-envoy\") pod \"e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv\" (UID: \"2f84895b-57a1-4c93-b20c-dc3f2ca754b4\") " pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv" Oct 31 04:40:26.638385 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:26.637664 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-workload-socket\") pod \"e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv\" (UID: \"2f84895b-57a1-4c93-b20c-dc3f2ca754b4\") " pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv" Oct 31 04:40:26.638385 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:26.637698 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-istiod-ca-cert\") pod \"e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv\" (UID: \"2f84895b-57a1-4c93-b20c-dc3f2ca754b4\") " pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv" Oct 31 04:40:26.638385 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:26.637782 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcr9b\" (UniqueName: \"kubernetes.io/projected/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-kube-api-access-bcr9b\") pod \"e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv\" (UID: \"2f84895b-57a1-4c93-b20c-dc3f2ca754b4\") " pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv" Oct 31 04:40:26.638385 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:26.637810 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-workload-certs\") pod \"e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv\" (UID: \"2f84895b-57a1-4c93-b20c-dc3f2ca754b4\") " pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv" Oct 31 04:40:26.638385 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:26.637830 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-istio-token\") pod \"e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv\" (UID: \"2f84895b-57a1-4c93-b20c-dc3f2ca754b4\") " pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv" Oct 31 04:40:26.638385 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:26.637871 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-istio-data\") pod \"e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv\" (UID: \"2f84895b-57a1-4c93-b20c-dc3f2ca754b4\") " pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv" Oct 31 04:40:26.638385 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:26.637892 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-istio-podinfo\") pod \"e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv\" (UID: \"2f84895b-57a1-4c93-b20c-dc3f2ca754b4\") " pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv" Oct 31 04:40:26.638385 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:26.637992 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-credential-socket\") pod \"e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv\" (UID: \"2f84895b-57a1-4c93-b20c-dc3f2ca754b4\") " pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv" Oct 31 04:40:26.639092 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:26.638576 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-credential-socket\") pod \"e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv\" (UID: \"2f84895b-57a1-4c93-b20c-dc3f2ca754b4\") " pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv" Oct 31 04:40:26.645355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:26.639213 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-workload-certs\") pod \"e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv\" (UID: \"2f84895b-57a1-4c93-b20c-dc3f2ca754b4\") " pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv" Oct 31 04:40:26.645355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:26.639487 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-workload-socket\") pod \"e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv\" (UID: \"2f84895b-57a1-4c93-b20c-dc3f2ca754b4\") " pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv" Oct 31 04:40:26.645355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:26.640129 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-istiod-ca-cert\") pod \"e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv\" (UID: \"2f84895b-57a1-4c93-b20c-dc3f2ca754b4\") " pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv" Oct 31 04:40:26.663366 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:26.647653 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-istio-data\") pod \"e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv\" (UID: \"2f84895b-57a1-4c93-b20c-dc3f2ca754b4\") " pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv" Oct 31 04:40:26.690426 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:26.688879 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-istio-envoy\") pod \"e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv\" (UID: \"2f84895b-57a1-4c93-b20c-dc3f2ca754b4\") " pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv" Oct 31 04:40:26.691712 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:26.691678 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-istio-podinfo\") pod \"e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv\" (UID: \"2f84895b-57a1-4c93-b20c-dc3f2ca754b4\") " pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv" Oct 31 04:40:26.714525 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:26.714444 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcr9b\" (UniqueName: \"kubernetes.io/projected/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-kube-api-access-bcr9b\") pod \"e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv\" (UID: \"2f84895b-57a1-4c93-b20c-dc3f2ca754b4\") " pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv" Oct 31 04:40:26.721584 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:26.721551 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-istio-token\") pod \"e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv\" (UID: \"2f84895b-57a1-4c93-b20c-dc3f2ca754b4\") " pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv" Oct 31 04:40:26.842483 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:26.839721 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv" Oct 31 04:40:27.250230 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:27.250182 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv"] Oct 31 04:40:27.253036 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:40:27.252965 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f84895b_57a1_4c93_b20c_dc3f2ca754b4.slice/crio-69837732349f44555008f45574b782cddff9e7179fa631e03d57ab3d1ae77e25 WatchSource:0}: Error finding container 69837732349f44555008f45574b782cddff9e7179fa631e03d57ab3d1ae77e25: Status 404 returned error can't find the container with id 69837732349f44555008f45574b782cddff9e7179fa631e03d57ab3d1ae77e25 Oct 31 04:40:27.256899 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:27.256838 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:40:27.257054 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:27.256953 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:40:27.257054 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:27.257002 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:40:27.967491 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:27.967456 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv" event={"ID":"2f84895b-57a1-4c93-b20c-dc3f2ca754b4","Type":"ContainerStarted","Data":"a341aa061b790b2673aeedde095428a96501b3818ee27f7127958763081c82a2"} Oct 31 04:40:27.967915 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:27.967512 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv" event={"ID":"2f84895b-57a1-4c93-b20c-dc3f2ca754b4","Type":"ContainerStarted","Data":"69837732349f44555008f45574b782cddff9e7179fa631e03d57ab3d1ae77e25"} Oct 31 04:40:28.016518 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:28.016453 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv" podStartSLOduration=2.016436358 podStartE2EDuration="2.016436358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:40:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:40:28.012135893 +0000 UTC m=+2182.768473933" watchObservedRunningTime="2026-10-31 04:40:28.016436358 +0000 UTC m=+2182.772774398" Oct 31 04:40:28.822699 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:28.822663 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a99f2a-74bfdc65dc-9j694" Oct 31 04:40:28.822922 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:28.822739 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-a99f2a-5bb987db5-2m54d" Oct 31 04:40:28.840603 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:28.840566 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv" Oct 31 04:40:28.847258 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:28.847217 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv" Oct 31 04:40:29.021860 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:29.021827 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv" Oct 31 04:40:29.023589 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:29.023563 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv" Oct 31 04:40:29.419152 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:29.419086 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn"] Oct 31 04:40:29.444363 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:29.441657 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn" Oct 31 04:40:29.454663 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:29.453970 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn"] Oct 31 04:40:29.457179 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:29.457145 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-8973e2-openshift-default-dockercfg-lwl8l\"" Oct 31 04:40:29.576359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:29.551371 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-istiod-ca-cert\") pod \"e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn\" (UID: \"2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09\") " pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn" Oct 31 04:40:29.601568 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:29.601519 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-workload-socket\") pod \"e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn\" (UID: \"2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09\") " pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn" Oct 31 04:40:29.601853 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:29.601830 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-istio-token\") pod \"e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn\" (UID: \"2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09\") " pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn" Oct 31 04:40:29.733410 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:29.733370 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-credential-socket\") pod \"e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn\" (UID: \"2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09\") " pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn" Oct 31 04:40:29.733782 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:29.733758 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-workload-certs\") pod \"e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn\" (UID: \"2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09\") " pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn" Oct 31 04:40:29.734038 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:29.734020 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-istio-envoy\") pod \"e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn\" (UID: \"2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09\") " pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn" Oct 31 04:40:29.734191 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:29.734176 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv6xp\" (UniqueName: \"kubernetes.io/projected/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-kube-api-access-nv6xp\") pod \"e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn\" (UID: \"2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09\") " pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn" Oct 31 04:40:29.734555 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:29.734503 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-istio-data\") pod \"e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn\" (UID: \"2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09\") " pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn" Oct 31 04:40:29.734762 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:29.734744 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-istio-podinfo\") pod \"e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn\" (UID: \"2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09\") " pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn" Oct 31 04:40:29.838896 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:29.837814 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-credential-socket\") pod \"e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn\" (UID: \"2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09\") " pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn" Oct 31 04:40:29.838896 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:29.837864 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-workload-certs\") pod \"e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn\" (UID: \"2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09\") " pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn" Oct 31 04:40:29.838896 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:29.837937 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-istio-envoy\") pod \"e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn\" (UID: \"2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09\") " pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn" Oct 31 04:40:29.838896 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:29.837962 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv6xp\" (UniqueName: \"kubernetes.io/projected/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-kube-api-access-nv6xp\") pod \"e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn\" (UID: \"2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09\") " pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn" Oct 31 04:40:29.838896 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:29.838085 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-istio-data\") pod \"e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn\" (UID: \"2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09\") " pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn" Oct 31 04:40:29.838896 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:29.838112 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-istio-podinfo\") pod \"e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn\" (UID: \"2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09\") " pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn" Oct 31 04:40:29.838896 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:29.838235 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-istiod-ca-cert\") pod \"e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn\" (UID: \"2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09\") " pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn" Oct 31 04:40:29.838896 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:29.838354 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-workload-socket\") pod \"e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn\" (UID: \"2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09\") " pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn" Oct 31 04:40:29.838896 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:29.838381 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-istio-token\") pod \"e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn\" (UID: \"2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09\") " pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn" Oct 31 04:40:29.839705 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:29.839314 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-workload-socket\") pod \"e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn\" (UID: \"2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09\") " pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn" Oct 31 04:40:29.839705 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:29.839445 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-istio-data\") pod \"e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn\" (UID: \"2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09\") " pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn" Oct 31 04:40:29.839821 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:29.839788 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-workload-certs\") pod \"e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn\" (UID: \"2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09\") " pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn" Oct 31 04:40:29.840000 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:29.839962 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-credential-socket\") pod \"e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn\" (UID: \"2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09\") " pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn" Oct 31 04:40:29.840440 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:29.840416 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-istiod-ca-cert\") pod \"e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn\" (UID: \"2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09\") " pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn" Oct 31 04:40:29.842471 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:29.842435 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-istio-envoy\") pod \"e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn\" (UID: \"2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09\") " pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn" Oct 31 04:40:29.843114 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:29.843089 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-istio-podinfo\") pod \"e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn\" (UID: \"2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09\") " pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn" Oct 31 04:40:29.862547 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:29.862517 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-istio-token\") pod \"e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn\" (UID: \"2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09\") " pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn" Oct 31 04:40:29.862676 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:29.862601 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv6xp\" (UniqueName: \"kubernetes.io/projected/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-kube-api-access-nv6xp\") pod \"e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn\" (UID: \"2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09\") " pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn" Oct 31 04:40:30.062236 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:30.062133 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn" Oct 31 04:40:30.412480 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:40:30.412437 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e2d0bd6_c4ff_4e2f_8b2c_379fde919d09.slice/crio-4673d8306c5aa5307116b09c77d7bde416c6ead5efba6828afe24913c2f5587b WatchSource:0}: Error finding container 4673d8306c5aa5307116b09c77d7bde416c6ead5efba6828afe24913c2f5587b: Status 404 returned error can't find the container with id 4673d8306c5aa5307116b09c77d7bde416c6ead5efba6828afe24913c2f5587b Oct 31 04:40:30.413531 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:30.413388 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn"] Oct 31 04:40:30.416123 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:30.416082 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:40:30.416229 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:30.416156 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:40:30.416229 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:30.416202 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:40:31.042889 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:31.042847 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn" event={"ID":"2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09","Type":"ContainerStarted","Data":"cb00f57e539761d7c6459db710bfee12b40bb84e993b14d408ad19ebb35ab74b"} Oct 31 04:40:31.043081 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:31.042909 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn" event={"ID":"2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09","Type":"ContainerStarted","Data":"4673d8306c5aa5307116b09c77d7bde416c6ead5efba6828afe24913c2f5587b"} Oct 31 04:40:31.062862 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:31.062820 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn" Oct 31 04:40:31.070295 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:31.070264 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn" Oct 31 04:40:31.105783 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:31.100598 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn" podStartSLOduration=2.100582836 podStartE2EDuration="2.100582836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:40:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:40:31.100142556 +0000 UTC m=+2185.856480596" watchObservedRunningTime="2026-10-31 04:40:31.100582836 +0000 UTC m=+2185.856920884" Oct 31 04:40:32.057382 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:32.054311 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn" Oct 31 04:40:32.057382 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:32.055922 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn" Oct 31 04:40:32.283750 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:32.283708 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-567ca6-7fdf68b675-mfgzr"] Oct 31 04:40:32.288909 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:32.288883 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-567ca6-7fdf68b675-mfgzr" Oct 31 04:40:32.309314 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:32.309237 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-567ca6-7fdf68b675-mfgzr"] Oct 31 04:40:32.311254 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:32.311226 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-567ca6-dockercfg-7vsl7\"" Oct 31 04:40:32.320630 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:32.320601 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-567ca6\"" Oct 31 04:40:32.452354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:32.448461 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99145b1d-f102-4e18-942d-aee106ab1bdd-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-567ca6-7fdf68b675-mfgzr\" (UID: \"99145b1d-f102-4e18-942d-aee106ab1bdd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-567ca6-7fdf68b675-mfgzr" Oct 31 04:40:32.452354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:32.448598 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjgns\" (UniqueName: \"kubernetes.io/projected/99145b1d-f102-4e18-942d-aee106ab1bdd-kube-api-access-sjgns\") pod \"payload-pre-processing-e2e-extmod-b-567ca6-7fdf68b675-mfgzr\" (UID: \"99145b1d-f102-4e18-942d-aee106ab1bdd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-567ca6-7fdf68b675-mfgzr" Oct 31 04:40:32.555354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:32.552241 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99145b1d-f102-4e18-942d-aee106ab1bdd-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-567ca6-7fdf68b675-mfgzr\" (UID: \"99145b1d-f102-4e18-942d-aee106ab1bdd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-567ca6-7fdf68b675-mfgzr" Oct 31 04:40:32.555354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:32.552465 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjgns\" (UniqueName: \"kubernetes.io/projected/99145b1d-f102-4e18-942d-aee106ab1bdd-kube-api-access-sjgns\") pod \"payload-pre-processing-e2e-extmod-b-567ca6-7fdf68b675-mfgzr\" (UID: \"99145b1d-f102-4e18-942d-aee106ab1bdd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-567ca6-7fdf68b675-mfgzr" Oct 31 04:40:32.559393 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:32.557994 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99145b1d-f102-4e18-942d-aee106ab1bdd-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-567ca6-7fdf68b675-mfgzr\" (UID: \"99145b1d-f102-4e18-942d-aee106ab1bdd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-567ca6-7fdf68b675-mfgzr" Oct 31 04:40:32.616164 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:32.616126 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjgns\" (UniqueName: \"kubernetes.io/projected/99145b1d-f102-4e18-942d-aee106ab1bdd-kube-api-access-sjgns\") pod \"payload-pre-processing-e2e-extmod-b-567ca6-7fdf68b675-mfgzr\" (UID: \"99145b1d-f102-4e18-942d-aee106ab1bdd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-567ca6-7fdf68b675-mfgzr" Oct 31 04:40:32.622750 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:32.622715 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-567ca6-7fdf68b675-mfgzr" Oct 31 04:40:33.005000 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:33.004944 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-567ca6-7fdf68b675-mfgzr"] Oct 31 04:40:33.005297 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:40:33.005264 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99145b1d_f102_4e18_942d_aee106ab1bdd.slice/crio-4ef3b1ceb0b13e04477a248a02327b483a95c811b28d8c2bf517bdf829c1c351 WatchSource:0}: Error finding container 4ef3b1ceb0b13e04477a248a02327b483a95c811b28d8c2bf517bdf829c1c351: Status 404 returned error can't find the container with id 4ef3b1ceb0b13e04477a248a02327b483a95c811b28d8c2bf517bdf829c1c351 Oct 31 04:40:33.066762 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:33.066714 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-567ca6-7fdf68b675-mfgzr" event={"ID":"99145b1d-f102-4e18-942d-aee106ab1bdd","Type":"ContainerStarted","Data":"4ef3b1ceb0b13e04477a248a02327b483a95c811b28d8c2bf517bdf829c1c351"} Oct 31 04:40:34.080739 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:34.080677 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-567ca6-7fdf68b675-mfgzr" event={"ID":"99145b1d-f102-4e18-942d-aee106ab1bdd","Type":"ContainerStarted","Data":"5673d8ab5c1a17f1fa9b6a0e8a7372d6fe2e9c486de26632c98167ab54f3cc69"} Oct 31 04:40:34.081429 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:34.080789 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-567ca6-7fdf68b675-mfgzr" Oct 31 04:40:34.127433 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:34.127378 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-567ca6-7fdf68b675-mfgzr" podStartSLOduration=2.127362415 podStartE2EDuration="2.127362415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:40:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:40:34.123481366 +0000 UTC m=+2188.879819408" watchObservedRunningTime="2026-10-31 04:40:34.127362415 +0000 UTC m=+2188.883700454" Oct 31 04:40:35.092185 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:35.092146 5019 generic.go:356] "Generic (PLEG): container finished" podID="c4a5e443-a1d7-4882-b86c-e6022b980073" containerID="19b5d18539ac54474e37bed439fc85ae4f3767304b0d9d34948706c493b7a752" exitCode=0 Oct 31 04:40:35.092787 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:35.092221 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc" event={"ID":"c4a5e443-a1d7-4882-b86c-e6022b980073","Type":"ContainerDied","Data":"19b5d18539ac54474e37bed439fc85ae4f3767304b0d9d34948706c493b7a752"} Oct 31 04:40:35.092787 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:35.092753 5019 scope.go:122] "RemoveContainer" containerID="19b5d18539ac54474e37bed439fc85ae4f3767304b0d9d34948706c493b7a752" Oct 31 04:40:35.094600 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:35.094570 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:40:35.094713 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:35.094644 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:40:35.094713 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:35.094678 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:40:35.438137 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:35.438028 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8973e2-85c9f98f95-rgkrh"] Oct 31 04:40:35.444379 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:35.444348 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8973e2-85c9f98f95-rgkrh" Oct 31 04:40:35.460244 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:35.460198 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-8973e2\"" Oct 31 04:40:35.460627 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:35.460603 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-8973e2-dockercfg-j2r8t\"" Oct 31 04:40:35.463694 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:35.463665 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8973e2-85c9f98f95-rgkrh"] Oct 31 04:40:35.525253 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:35.525196 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns9tq\" (UniqueName: \"kubernetes.io/projected/a4a19a46-de23-4984-894c-b14e0d223fab-kube-api-access-ns9tq\") pod \"payload-pre-processing-e2e-extmod-a-8973e2-85c9f98f95-rgkrh\" (UID: \"a4a19a46-de23-4984-894c-b14e0d223fab\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8973e2-85c9f98f95-rgkrh" Oct 31 04:40:35.525585 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:35.525283 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4a19a46-de23-4984-894c-b14e0d223fab-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-8973e2-85c9f98f95-rgkrh\" (UID: \"a4a19a46-de23-4984-894c-b14e0d223fab\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8973e2-85c9f98f95-rgkrh" Oct 31 04:40:35.639388 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:35.639312 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns9tq\" (UniqueName: \"kubernetes.io/projected/a4a19a46-de23-4984-894c-b14e0d223fab-kube-api-access-ns9tq\") pod \"payload-pre-processing-e2e-extmod-a-8973e2-85c9f98f95-rgkrh\" (UID: \"a4a19a46-de23-4984-894c-b14e0d223fab\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8973e2-85c9f98f95-rgkrh" Oct 31 04:40:35.639650 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:35.639433 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4a19a46-de23-4984-894c-b14e0d223fab-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-8973e2-85c9f98f95-rgkrh\" (UID: \"a4a19a46-de23-4984-894c-b14e0d223fab\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8973e2-85c9f98f95-rgkrh" Oct 31 04:40:35.640705 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:35.640677 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4a19a46-de23-4984-894c-b14e0d223fab-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-8973e2-85c9f98f95-rgkrh\" (UID: \"a4a19a46-de23-4984-894c-b14e0d223fab\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8973e2-85c9f98f95-rgkrh" Oct 31 04:40:35.743533 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:35.735052 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns9tq\" (UniqueName: \"kubernetes.io/projected/a4a19a46-de23-4984-894c-b14e0d223fab-kube-api-access-ns9tq\") pod \"payload-pre-processing-e2e-extmod-a-8973e2-85c9f98f95-rgkrh\" (UID: \"a4a19a46-de23-4984-894c-b14e0d223fab\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8973e2-85c9f98f95-rgkrh" Oct 31 04:40:35.761993 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:35.761954 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8973e2-85c9f98f95-rgkrh" Oct 31 04:40:36.109321 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:36.109220 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc" event={"ID":"c4a5e443-a1d7-4882-b86c-e6022b980073","Type":"ContainerStarted","Data":"8f2cda9389a568536a9adeb8d2cca0b34bd87243df394127ec7c24e6d353d837"} Oct 31 04:40:36.191499 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:36.172936 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8973e2-85c9f98f95-rgkrh"] Oct 31 04:40:36.894851 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:36.890698 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc" Oct 31 04:40:36.895537 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:36.895501 5019 patch_prober.go:29] interesting pod/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 31 04:40:36.895703 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:36.895577 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc" podUID="c4a5e443-a1d7-4882-b86c-e6022b980073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 31 04:40:37.121795 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:37.121753 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8973e2-85c9f98f95-rgkrh" event={"ID":"a4a19a46-de23-4984-894c-b14e0d223fab","Type":"ContainerStarted","Data":"af140097b359d0fca97e3965becf2a8cb63c13d3bbebadbf832c621e1398f23f"} Oct 31 04:40:37.122280 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:37.121822 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8973e2-85c9f98f95-rgkrh" event={"ID":"a4a19a46-de23-4984-894c-b14e0d223fab","Type":"ContainerStarted","Data":"38806fdc81eab273824c83fc7d552d4a0395aab807bf86bf5cce80cb25526b50"} Oct 31 04:40:37.122280 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:37.121883 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8973e2-85c9f98f95-rgkrh" Oct 31 04:40:37.166284 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:37.166159 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8973e2-85c9f98f95-rgkrh" podStartSLOduration=2.166138852 podStartE2EDuration="2.166138852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:40:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:40:37.161530656 +0000 UTC m=+2191.917868696" watchObservedRunningTime="2026-10-31 04:40:37.166138852 +0000 UTC m=+2191.922476891" Oct 31 04:40:37.628872 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:37.628833 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc" Oct 31 04:40:37.893416 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:37.893198 5019 patch_prober.go:29] interesting pod/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 31 04:40:37.893416 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:37.893269 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc" podUID="c4a5e443-a1d7-4882-b86c-e6022b980073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 31 04:40:38.897358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:38.891836 5019 patch_prober.go:29] interesting pod/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 31 04:40:38.897358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:38.891906 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc" podUID="c4a5e443-a1d7-4882-b86c-e6022b980073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 31 04:40:39.894560 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:39.894513 5019 patch_prober.go:29] interesting pod/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 31 04:40:39.894919 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:39.894581 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc" podUID="c4a5e443-a1d7-4882-b86c-e6022b980073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 31 04:40:40.893358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:40.890850 5019 patch_prober.go:29] interesting pod/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 31 04:40:40.893358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:40.890922 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc" podUID="c4a5e443-a1d7-4882-b86c-e6022b980073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 31 04:40:41.896075 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:41.896037 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc" Oct 31 04:40:41.897131 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:41.897100 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc" Oct 31 04:40:45.095499 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:45.095469 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-567ca6-7fdf68b675-mfgzr" Oct 31 04:40:48.134391 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:40:48.134357 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8973e2-85c9f98f95-rgkrh" Oct 31 04:40:52.394464 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:40:52.394422 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:41:09.563239 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:09.563204 5019 generic.go:356] "Generic (PLEG): container finished" podID="2764ba4c-dff3-4c51-a8ca-0121dd9f5b29" containerID="ae3722b96fc47f0c6468e33e2f94c81d370fd8ec6ad49c0f6464a7e744156dca" exitCode=0 Oct 31 04:41:09.563822 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:09.563276 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt" event={"ID":"2764ba4c-dff3-4c51-a8ca-0121dd9f5b29","Type":"ContainerDied","Data":"ae3722b96fc47f0c6468e33e2f94c81d370fd8ec6ad49c0f6464a7e744156dca"} Oct 31 04:41:09.563902 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:09.563883 5019 scope.go:122] "RemoveContainer" containerID="ae3722b96fc47f0c6468e33e2f94c81d370fd8ec6ad49c0f6464a7e744156dca" Oct 31 04:41:09.565355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:09.565311 5019 generic.go:356] "Generic (PLEG): container finished" podID="468817c7-e1ca-41c0-8a91-cfce50aa71ac" containerID="5590a7d6cdfe15f49a0d13ca1f3456ca2da18f063702b9e3e0cc1dc83aae0e8a" exitCode=0 Oct 31 04:41:09.565483 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:09.565363 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp" event={"ID":"468817c7-e1ca-41c0-8a91-cfce50aa71ac","Type":"ContainerDied","Data":"5590a7d6cdfe15f49a0d13ca1f3456ca2da18f063702b9e3e0cc1dc83aae0e8a"} Oct 31 04:41:09.565785 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:09.565771 5019 scope.go:122] "RemoveContainer" containerID="5590a7d6cdfe15f49a0d13ca1f3456ca2da18f063702b9e3e0cc1dc83aae0e8a" Oct 31 04:41:09.565916 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:09.565887 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:41:09.565973 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:09.565960 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:41:09.566032 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:09.565996 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:41:09.567672 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:09.567632 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:41:09.567790 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:09.567703 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:41:09.567790 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:09.567744 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:41:10.615120 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:10.615079 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt" event={"ID":"2764ba4c-dff3-4c51-a8ca-0121dd9f5b29","Type":"ContainerStarted","Data":"ba5bf75249bd21f67e87b4770c90f3f88d061af1f1f287a2250a8dda09f6d977"} Oct 31 04:41:10.621352 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:10.620862 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp" event={"ID":"468817c7-e1ca-41c0-8a91-cfce50aa71ac","Type":"ContainerStarted","Data":"c5ffa648dd5facb6a15713079808dcd094cda1cb89634456282a04f7f27c4180"} Oct 31 04:41:10.932507 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:10.932403 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp" Oct 31 04:41:10.932675 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:10.932499 5019 patch_prober.go:29] interesting pod/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" start-of-body= Oct 31 04:41:10.932675 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:10.932559 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp" podUID="468817c7-e1ca-41c0-8a91-cfce50aa71ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" Oct 31 04:41:11.002356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:11.001732 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt" Oct 31 04:41:11.008520 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:11.008459 5019 patch_prober.go:29] interesting pod/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body= Oct 31 04:41:11.008733 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:11.008531 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt" podUID="2764ba4c-dff3-4c51-a8ca-0121dd9f5b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" Oct 31 04:41:11.931757 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:11.931715 5019 patch_prober.go:29] interesting pod/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" start-of-body= Oct 31 04:41:11.932188 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:11.931773 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp" podUID="468817c7-e1ca-41c0-8a91-cfce50aa71ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" Oct 31 04:41:12.001781 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:12.001736 5019 patch_prober.go:29] interesting pod/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body= Oct 31 04:41:12.002093 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:12.002057 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt" podUID="2764ba4c-dff3-4c51-a8ca-0121dd9f5b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" Oct 31 04:41:12.932397 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:12.932130 5019 patch_prober.go:29] interesting pod/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" start-of-body= Oct 31 04:41:12.932397 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:12.932200 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp" podUID="468817c7-e1ca-41c0-8a91-cfce50aa71ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" Oct 31 04:41:13.002361 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:13.001858 5019 patch_prober.go:29] interesting pod/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body= Oct 31 04:41:13.002361 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:13.001940 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt" podUID="2764ba4c-dff3-4c51-a8ca-0121dd9f5b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" Oct 31 04:41:13.931987 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:13.931939 5019 patch_prober.go:29] interesting pod/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" start-of-body= Oct 31 04:41:13.932298 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:13.932011 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp" podUID="468817c7-e1ca-41c0-8a91-cfce50aa71ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" Oct 31 04:41:13.998969 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:13.998914 5019 patch_prober.go:29] interesting pod/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body= Oct 31 04:41:13.999661 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:13.999608 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt" podUID="2764ba4c-dff3-4c51-a8ca-0121dd9f5b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" Oct 31 04:41:14.932059 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:14.932007 5019 patch_prober.go:29] interesting pod/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" start-of-body= Oct 31 04:41:14.932264 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:14.932078 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp" podUID="468817c7-e1ca-41c0-8a91-cfce50aa71ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" Oct 31 04:41:14.999355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:14.998797 5019 patch_prober.go:29] interesting pod/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body= Oct 31 04:41:14.999355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:14.998869 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt" podUID="2764ba4c-dff3-4c51-a8ca-0121dd9f5b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" Oct 31 04:41:15.607501 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:15.607452 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt" Oct 31 04:41:15.932100 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:15.931975 5019 patch_prober.go:29] interesting pod/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" start-of-body= Oct 31 04:41:15.932100 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:15.932052 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp" podUID="468817c7-e1ca-41c0-8a91-cfce50aa71ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" Oct 31 04:41:16.055000 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:16.054959 5019 patch_prober.go:29] interesting pod/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body= Oct 31 04:41:16.055560 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:16.055101 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt" podUID="2764ba4c-dff3-4c51-a8ca-0121dd9f5b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" Oct 31 04:41:16.609117 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:16.609070 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp" Oct 31 04:41:16.938044 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:16.937953 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp" Oct 31 04:41:16.939136 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:16.939112 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp" Oct 31 04:41:17.004136 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:17.004103 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt" Oct 31 04:41:17.005241 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:17.005218 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt" Oct 31 04:41:22.405849 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:41:22.405806 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:41:31.338080 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:31.338048 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-99ca0c/explicit-model-99ca0c-kserve-c975b5fdf-lsvd6"] Oct 31 04:41:31.344390 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:31.344320 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-99ca0c/explicit-model-99ca0c-kserve-c975b5fdf-lsvd6" Oct 31 04:41:31.359313 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:31.359287 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-99ca0c/explicit-model-99ca0c-kserve-c975b5fdf-lsvd6"] Oct 31 04:41:31.369147 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:31.368974 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-99ca0c\"/\"default-dockercfg-428qx\"" Oct 31 04:41:31.369147 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:31.369082 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-99ca0c\"/\"openshift-service-ca.crt\"" Oct 31 04:41:31.372593 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:31.372568 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-99ca0c\"/\"explicit-model-99ca0c-kserve-self-signed-certs\"" Oct 31 04:41:31.372746 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:31.372579 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-99ca0c\"/\"kube-root-ca.crt\"" Oct 31 04:41:31.383421 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:31.383395 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c64c7635-8fa4-4d84-93aa-95e846139f7f-tls-certs\") pod \"explicit-model-99ca0c-kserve-c975b5fdf-lsvd6\" (UID: \"c64c7635-8fa4-4d84-93aa-95e846139f7f\") " pod="ai-tenant-e2e-explicit-99ca0c/explicit-model-99ca0c-kserve-c975b5fdf-lsvd6" Oct 31 04:41:31.383782 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:31.383763 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9lrc\" (UniqueName: \"kubernetes.io/projected/c64c7635-8fa4-4d84-93aa-95e846139f7f-kube-api-access-p9lrc\") pod \"explicit-model-99ca0c-kserve-c975b5fdf-lsvd6\" (UID: \"c64c7635-8fa4-4d84-93aa-95e846139f7f\") " pod="ai-tenant-e2e-explicit-99ca0c/explicit-model-99ca0c-kserve-c975b5fdf-lsvd6" Oct 31 04:41:31.383927 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:31.383908 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c64c7635-8fa4-4d84-93aa-95e846139f7f-home\") pod \"explicit-model-99ca0c-kserve-c975b5fdf-lsvd6\" (UID: \"c64c7635-8fa4-4d84-93aa-95e846139f7f\") " pod="ai-tenant-e2e-explicit-99ca0c/explicit-model-99ca0c-kserve-c975b5fdf-lsvd6" Oct 31 04:41:31.384067 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:31.384051 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c64c7635-8fa4-4d84-93aa-95e846139f7f-dshm\") pod \"explicit-model-99ca0c-kserve-c975b5fdf-lsvd6\" (UID: \"c64c7635-8fa4-4d84-93aa-95e846139f7f\") " pod="ai-tenant-e2e-explicit-99ca0c/explicit-model-99ca0c-kserve-c975b5fdf-lsvd6" Oct 31 04:41:31.384318 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:31.384298 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c64c7635-8fa4-4d84-93aa-95e846139f7f-model-cache\") pod \"explicit-model-99ca0c-kserve-c975b5fdf-lsvd6\" (UID: \"c64c7635-8fa4-4d84-93aa-95e846139f7f\") " pod="ai-tenant-e2e-explicit-99ca0c/explicit-model-99ca0c-kserve-c975b5fdf-lsvd6" Oct 31 04:41:31.384666 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:31.384644 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c64c7635-8fa4-4d84-93aa-95e846139f7f-tmp-dir\") pod \"explicit-model-99ca0c-kserve-c975b5fdf-lsvd6\" (UID: \"c64c7635-8fa4-4d84-93aa-95e846139f7f\") " pod="ai-tenant-e2e-explicit-99ca0c/explicit-model-99ca0c-kserve-c975b5fdf-lsvd6" Oct 31 04:41:31.486474 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:31.486437 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c64c7635-8fa4-4d84-93aa-95e846139f7f-tls-certs\") pod \"explicit-model-99ca0c-kserve-c975b5fdf-lsvd6\" (UID: \"c64c7635-8fa4-4d84-93aa-95e846139f7f\") " pod="ai-tenant-e2e-explicit-99ca0c/explicit-model-99ca0c-kserve-c975b5fdf-lsvd6" Oct 31 04:41:31.486691 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:31.486548 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9lrc\" (UniqueName: \"kubernetes.io/projected/c64c7635-8fa4-4d84-93aa-95e846139f7f-kube-api-access-p9lrc\") pod \"explicit-model-99ca0c-kserve-c975b5fdf-lsvd6\" (UID: \"c64c7635-8fa4-4d84-93aa-95e846139f7f\") " pod="ai-tenant-e2e-explicit-99ca0c/explicit-model-99ca0c-kserve-c975b5fdf-lsvd6" Oct 31 04:41:31.486691 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:31.486573 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c64c7635-8fa4-4d84-93aa-95e846139f7f-home\") pod \"explicit-model-99ca0c-kserve-c975b5fdf-lsvd6\" (UID: \"c64c7635-8fa4-4d84-93aa-95e846139f7f\") " pod="ai-tenant-e2e-explicit-99ca0c/explicit-model-99ca0c-kserve-c975b5fdf-lsvd6" Oct 31 04:41:31.486691 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:31.486588 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c64c7635-8fa4-4d84-93aa-95e846139f7f-dshm\") pod \"explicit-model-99ca0c-kserve-c975b5fdf-lsvd6\" (UID: \"c64c7635-8fa4-4d84-93aa-95e846139f7f\") " pod="ai-tenant-e2e-explicit-99ca0c/explicit-model-99ca0c-kserve-c975b5fdf-lsvd6" Oct 31 04:41:31.486691 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:31.486690 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c64c7635-8fa4-4d84-93aa-95e846139f7f-model-cache\") pod \"explicit-model-99ca0c-kserve-c975b5fdf-lsvd6\" (UID: \"c64c7635-8fa4-4d84-93aa-95e846139f7f\") " pod="ai-tenant-e2e-explicit-99ca0c/explicit-model-99ca0c-kserve-c975b5fdf-lsvd6" Oct 31 04:41:31.486984 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:31.486840 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c64c7635-8fa4-4d84-93aa-95e846139f7f-tmp-dir\") pod \"explicit-model-99ca0c-kserve-c975b5fdf-lsvd6\" (UID: \"c64c7635-8fa4-4d84-93aa-95e846139f7f\") " pod="ai-tenant-e2e-explicit-99ca0c/explicit-model-99ca0c-kserve-c975b5fdf-lsvd6" Oct 31 04:41:31.487360 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:31.487321 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c64c7635-8fa4-4d84-93aa-95e846139f7f-tmp-dir\") pod \"explicit-model-99ca0c-kserve-c975b5fdf-lsvd6\" (UID: \"c64c7635-8fa4-4d84-93aa-95e846139f7f\") " pod="ai-tenant-e2e-explicit-99ca0c/explicit-model-99ca0c-kserve-c975b5fdf-lsvd6" Oct 31 04:41:31.488616 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:31.488027 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c64c7635-8fa4-4d84-93aa-95e846139f7f-model-cache\") pod \"explicit-model-99ca0c-kserve-c975b5fdf-lsvd6\" (UID: \"c64c7635-8fa4-4d84-93aa-95e846139f7f\") " pod="ai-tenant-e2e-explicit-99ca0c/explicit-model-99ca0c-kserve-c975b5fdf-lsvd6" Oct 31 04:41:31.488616 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:31.488299 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c64c7635-8fa4-4d84-93aa-95e846139f7f-home\") pod \"explicit-model-99ca0c-kserve-c975b5fdf-lsvd6\" (UID: \"c64c7635-8fa4-4d84-93aa-95e846139f7f\") " pod="ai-tenant-e2e-explicit-99ca0c/explicit-model-99ca0c-kserve-c975b5fdf-lsvd6" Oct 31 04:41:31.490402 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:31.490374 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c64c7635-8fa4-4d84-93aa-95e846139f7f-dshm\") pod \"explicit-model-99ca0c-kserve-c975b5fdf-lsvd6\" (UID: \"c64c7635-8fa4-4d84-93aa-95e846139f7f\") " pod="ai-tenant-e2e-explicit-99ca0c/explicit-model-99ca0c-kserve-c975b5fdf-lsvd6" Oct 31 04:41:31.490565 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:31.490545 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c64c7635-8fa4-4d84-93aa-95e846139f7f-tls-certs\") pod \"explicit-model-99ca0c-kserve-c975b5fdf-lsvd6\" (UID: \"c64c7635-8fa4-4d84-93aa-95e846139f7f\") " pod="ai-tenant-e2e-explicit-99ca0c/explicit-model-99ca0c-kserve-c975b5fdf-lsvd6" Oct 31 04:41:31.511650 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:31.511614 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9lrc\" (UniqueName: \"kubernetes.io/projected/c64c7635-8fa4-4d84-93aa-95e846139f7f-kube-api-access-p9lrc\") pod \"explicit-model-99ca0c-kserve-c975b5fdf-lsvd6\" (UID: \"c64c7635-8fa4-4d84-93aa-95e846139f7f\") " pod="ai-tenant-e2e-explicit-99ca0c/explicit-model-99ca0c-kserve-c975b5fdf-lsvd6" Oct 31 04:41:31.672410 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:31.667887 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-99ca0c/explicit-model-99ca0c-kserve-c975b5fdf-lsvd6" Oct 31 04:41:32.368984 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:32.368947 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-99ca0c/explicit-model-99ca0c-kserve-c975b5fdf-lsvd6"] Oct 31 04:41:32.369307 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:41:32.368995 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc64c7635_8fa4_4d84_93aa_95e846139f7f.slice/crio-be1129aa8ea2cfe2ab456222580d68ba9ff3f8d079311107dcdc73157901bcdd WatchSource:0}: Error finding container be1129aa8ea2cfe2ab456222580d68ba9ff3f8d079311107dcdc73157901bcdd: Status 404 returned error can't find the container with id be1129aa8ea2cfe2ab456222580d68ba9ff3f8d079311107dcdc73157901bcdd Oct 31 04:41:33.030215 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:33.030164 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-99ca0c/explicit-model-99ca0c-kserve-c975b5fdf-lsvd6" event={"ID":"c64c7635-8fa4-4d84-93aa-95e846139f7f","Type":"ContainerStarted","Data":"21770dbe63c989aee1f5d787ec23167a7e0560622087420b8c281b6ac139a942"} Oct 31 04:41:33.030215 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:33.030218 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-99ca0c/explicit-model-99ca0c-kserve-c975b5fdf-lsvd6" event={"ID":"c64c7635-8fa4-4d84-93aa-95e846139f7f","Type":"ContainerStarted","Data":"be1129aa8ea2cfe2ab456222580d68ba9ff3f8d079311107dcdc73157901bcdd"} Oct 31 04:41:33.075357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:33.074499 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-99ca0c/explicit-model-99ca0c-kserve-c975b5fdf-lsvd6" podStartSLOduration=2.074476978 podStartE2EDuration="2.074476978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:41:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:41:33.073779786 +0000 UTC m=+2247.830117837" watchObservedRunningTime="2026-10-31 04:41:33.074476978 +0000 UTC m=+2247.830815019" Oct 31 04:41:33.668424 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:33.668387 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-99ca0c/explicit-model-99ca0c-kserve-c975b5fdf-lsvd6" Oct 31 04:41:38.835884 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:38.831886 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cf8wdr4"] Oct 31 04:41:38.838653 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:38.838624 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cf8wdr4" Oct 31 04:41:38.849892 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:38.849857 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cf8wdr4"] Oct 31 04:41:38.855567 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:38.855479 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ded435\"/\"payload-processing-external-model-e2e--b0123f89-dockercfg-hvjs2\"" Oct 31 04:41:38.855800 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:38.855626 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ded435\"/\"openshift-service-ca.crt\"" Oct 31 04:41:38.855920 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:38.855799 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ded435\"/\"kube-root-ca.crt\"" Oct 31 04:41:38.856027 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:38.855875 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ded435\"/\"agc-cred-3637b2c5d50f\"" Oct 31 04:41:38.866603 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:38.866550 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ded435\"/\"payload-processing-external-model-plugins-e2e-extmod-b-ded435\"" Oct 31 04:41:38.866603 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:38.866579 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ded435\"/\"routing-overlay\"" Oct 31 04:41:38.961399 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:38.961356 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b4a05f11-cbea-4b1f-8769-5487bbb89add-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-55cf8wdr4\" (UID: \"b4a05f11-cbea-4b1f-8769-5487bbb89add\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cf8wdr4" Oct 31 04:41:38.961591 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:38.961565 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b4a05f11-cbea-4b1f-8769-5487bbb89add-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-55cf8wdr4\" (UID: \"b4a05f11-cbea-4b1f-8769-5487bbb89add\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cf8wdr4" Oct 31 04:41:38.961652 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:38.961635 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4a05f11-cbea-4b1f-8769-5487bbb89add-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-55cf8wdr4\" (UID: \"b4a05f11-cbea-4b1f-8769-5487bbb89add\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cf8wdr4" Oct 31 04:41:38.961792 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:38.961753 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw7xl\" (UniqueName: \"kubernetes.io/projected/b4a05f11-cbea-4b1f-8769-5487bbb89add-kube-api-access-bw7xl\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-55cf8wdr4\" (UID: \"b4a05f11-cbea-4b1f-8769-5487bbb89add\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cf8wdr4" Oct 31 04:41:39.078019 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:39.068416 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b4a05f11-cbea-4b1f-8769-5487bbb89add-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-55cf8wdr4\" (UID: \"b4a05f11-cbea-4b1f-8769-5487bbb89add\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cf8wdr4" Oct 31 04:41:39.078019 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:39.068601 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b4a05f11-cbea-4b1f-8769-5487bbb89add-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-55cf8wdr4\" (UID: \"b4a05f11-cbea-4b1f-8769-5487bbb89add\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cf8wdr4" Oct 31 04:41:39.078019 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:39.068677 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4a05f11-cbea-4b1f-8769-5487bbb89add-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-55cf8wdr4\" (UID: \"b4a05f11-cbea-4b1f-8769-5487bbb89add\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cf8wdr4" Oct 31 04:41:39.078019 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:39.068734 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw7xl\" (UniqueName: \"kubernetes.io/projected/b4a05f11-cbea-4b1f-8769-5487bbb89add-kube-api-access-bw7xl\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-55cf8wdr4\" (UID: \"b4a05f11-cbea-4b1f-8769-5487bbb89add\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cf8wdr4" Oct 31 04:41:39.078019 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:39.070085 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b4a05f11-cbea-4b1f-8769-5487bbb89add-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-55cf8wdr4\" (UID: \"b4a05f11-cbea-4b1f-8769-5487bbb89add\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cf8wdr4" Oct 31 04:41:39.078019 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:39.072202 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4a05f11-cbea-4b1f-8769-5487bbb89add-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-55cf8wdr4\" (UID: \"b4a05f11-cbea-4b1f-8769-5487bbb89add\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cf8wdr4" Oct 31 04:41:39.089358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:39.085996 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b4a05f11-cbea-4b1f-8769-5487bbb89add-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-55cf8wdr4\" (UID: \"b4a05f11-cbea-4b1f-8769-5487bbb89add\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cf8wdr4" Oct 31 04:41:39.107754 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:39.102211 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw7xl\" (UniqueName: \"kubernetes.io/projected/b4a05f11-cbea-4b1f-8769-5487bbb89add-kube-api-access-bw7xl\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-55cf8wdr4\" (UID: \"b4a05f11-cbea-4b1f-8769-5487bbb89add\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cf8wdr4" Oct 31 04:41:39.194356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:39.185641 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cf8wdr4" Oct 31 04:41:39.618620 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:39.614651 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cf8wdr4"] Oct 31 04:41:39.622009 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:41:39.621974 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4a05f11_cbea_4b1f_8769_5487bbb89add.slice/crio-9ec4078304889611ec94842cbc5b78d05de33dec114f149270836d11d33d7a71 WatchSource:0}: Error finding container 9ec4078304889611ec94842cbc5b78d05de33dec114f149270836d11d33d7a71: Status 404 returned error can't find the container with id 9ec4078304889611ec94842cbc5b78d05de33dec114f149270836d11d33d7a71 Oct 31 04:41:40.299187 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:40.299106 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cf8wdr4" event={"ID":"b4a05f11-cbea-4b1f-8769-5487bbb89add","Type":"ContainerStarted","Data":"792f5865dcef9ac87b3d0599cda9d1f8099db54cce2b15347d2697ef7308e272"} Oct 31 04:41:40.299933 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:40.299905 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cf8wdr4" event={"ID":"b4a05f11-cbea-4b1f-8769-5487bbb89add","Type":"ContainerStarted","Data":"9ec4078304889611ec94842cbc5b78d05de33dec114f149270836d11d33d7a71"} Oct 31 04:41:40.300107 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:40.300091 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cf8wdr4" Oct 31 04:41:40.342174 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:40.342108 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cf8wdr4" podStartSLOduration=2.342089925 podStartE2EDuration="2.342089925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:41:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:41:40.341270806 +0000 UTC m=+2255.097608847" watchObservedRunningTime="2026-10-31 04:41:40.342089925 +0000 UTC m=+2255.098427967" Oct 31 04:41:41.669288 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:41.669256 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-99ca0c/explicit-model-99ca0c-kserve-c975b5fdf-lsvd6" Oct 31 04:41:41.688870 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:41.688837 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-99ca0c/explicit-model-99ca0c-kserve-c975b5fdf-lsvd6" Oct 31 04:41:42.335048 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:42.335015 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-99ca0c/explicit-model-99ca0c-kserve-c975b5fdf-lsvd6" Oct 31 04:41:45.043655 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:45.042404 5019 patch_prober.go:29] interesting pod/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.23:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:41:45.043655 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:45.042488 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv" podUID="2f84895b-57a1-4c93-b20c-dc3f2ca754b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.23:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:41:45.360803 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:45.360709 5019 generic.go:356] "Generic (PLEG): container finished" podID="2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09" containerID="cb00f57e539761d7c6459db710bfee12b40bb84e993b14d408ad19ebb35ab74b" exitCode=0 Oct 31 04:41:45.360803 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:45.360779 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn" event={"ID":"2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09","Type":"ContainerDied","Data":"cb00f57e539761d7c6459db710bfee12b40bb84e993b14d408ad19ebb35ab74b"} Oct 31 04:41:45.361262 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:45.361238 5019 scope.go:122] "RemoveContainer" containerID="cb00f57e539761d7c6459db710bfee12b40bb84e993b14d408ad19ebb35ab74b" Oct 31 04:41:45.362908 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:45.362880 5019 generic.go:356] "Generic (PLEG): container finished" podID="2f84895b-57a1-4c93-b20c-dc3f2ca754b4" containerID="a341aa061b790b2673aeedde095428a96501b3818ee27f7127958763081c82a2" exitCode=0 Oct 31 04:41:45.363013 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:45.362958 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv" event={"ID":"2f84895b-57a1-4c93-b20c-dc3f2ca754b4","Type":"ContainerDied","Data":"a341aa061b790b2673aeedde095428a96501b3818ee27f7127958763081c82a2"} Oct 31 04:41:45.363087 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:45.363058 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:41:45.363145 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:45.363132 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:41:45.363183 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:45.363165 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:41:45.363363 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:45.363313 5019 scope.go:122] "RemoveContainer" containerID="a341aa061b790b2673aeedde095428a96501b3818ee27f7127958763081c82a2" Oct 31 04:41:45.364880 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:45.364854 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:41:45.364963 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:45.364918 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:41:45.365002 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:45.364962 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:41:46.398357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:46.397691 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv" event={"ID":"2f84895b-57a1-4c93-b20c-dc3f2ca754b4","Type":"ContainerStarted","Data":"7d8b013ace00c2653492da4e9a3e0d8942408ffbba4f3cdbfcffea5fff43b0f6"} Oct 31 04:41:46.404853 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:46.404795 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn" event={"ID":"2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09","Type":"ContainerStarted","Data":"1d0ca6465c7367d099b5546a74d634b5f427886ba40e83c236374ce2e874bb71"} Oct 31 04:41:46.841358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:46.840845 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv" Oct 31 04:41:46.841358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:46.841054 5019 patch_prober.go:29] interesting pod/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" start-of-body= Oct 31 04:41:46.841358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:46.841136 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv" podUID="2f84895b-57a1-4c93-b20c-dc3f2ca754b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" Oct 31 04:41:47.054492 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:47.054445 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn" Oct 31 04:41:47.063534 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:47.063407 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn" Oct 31 04:41:47.064232 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:47.064204 5019 patch_prober.go:29] interesting pod/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" start-of-body= Oct 31 04:41:47.064303 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:47.064283 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn" podUID="2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" Oct 31 04:41:47.849357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:47.848812 5019 patch_prober.go:29] interesting pod/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" start-of-body= Oct 31 04:41:47.849357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:47.848886 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv" podUID="2f84895b-57a1-4c93-b20c-dc3f2ca754b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" Oct 31 04:41:48.109449 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:48.087428 5019 patch_prober.go:29] interesting pod/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" start-of-body= Oct 31 04:41:48.109449 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:48.087497 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn" podUID="2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" Oct 31 04:41:48.887357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:48.861680 5019 patch_prober.go:29] interesting pod/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" start-of-body= Oct 31 04:41:48.887357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:48.861748 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv" podUID="2f84895b-57a1-4c93-b20c-dc3f2ca754b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" Oct 31 04:41:49.083847 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:49.083729 5019 patch_prober.go:29] interesting pod/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" start-of-body= Oct 31 04:41:49.084603 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:49.084538 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn" podUID="2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" Oct 31 04:41:49.841207 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:49.841170 5019 patch_prober.go:29] interesting pod/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" start-of-body= Oct 31 04:41:49.841407 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:49.841222 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv" podUID="2f84895b-57a1-4c93-b20c-dc3f2ca754b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" Oct 31 04:41:50.063233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:50.063192 5019 patch_prober.go:29] interesting pod/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" start-of-body= Oct 31 04:41:50.063688 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:50.063253 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn" podUID="2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" Oct 31 04:41:50.704992 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:50.704954 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc5675cq"] Oct 31 04:41:50.713034 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:50.713005 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc5675cq" Oct 31 04:41:50.751369 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:50.732133 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a99f2a\"/\"agc-cred-7194275e28ab\"" Oct 31 04:41:50.751369 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:50.749476 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc5675cq"] Oct 31 04:41:50.767056 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:50.756094 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a99f2a\"/\"routing-overlay\"" Oct 31 04:41:50.767056 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:50.756168 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a99f2a\"/\"payload-processing-external-model-e2e--c07b99ee-dockercfg-gzb75\"" Oct 31 04:41:50.767056 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:50.756177 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a99f2a\"/\"payload-processing-external-model-plugins-e2e-extmod-b-a99f2a\"" Oct 31 04:41:50.767056 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:50.756094 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a99f2a\"/\"openshift-service-ca.crt\"" Oct 31 04:41:50.767056 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:50.756256 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a99f2a\"/\"kube-root-ca.crt\"" Oct 31 04:41:50.855798 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:50.852551 5019 patch_prober.go:29] interesting pod/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" start-of-body= Oct 31 04:41:50.855798 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:50.852610 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv" podUID="2f84895b-57a1-4c93-b20c-dc3f2ca754b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" Oct 31 04:41:50.867352 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:50.862912 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d841141c-29f5-48bf-9807-92267e6373ee-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-7bc5675cq\" (UID: \"d841141c-29f5-48bf-9807-92267e6373ee\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc5675cq" Oct 31 04:41:50.867352 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:50.863131 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jpwt\" (UniqueName: \"kubernetes.io/projected/d841141c-29f5-48bf-9807-92267e6373ee-kube-api-access-2jpwt\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-7bc5675cq\" (UID: \"d841141c-29f5-48bf-9807-92267e6373ee\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc5675cq" Oct 31 04:41:50.867352 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:50.863236 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d841141c-29f5-48bf-9807-92267e6373ee-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-7bc5675cq\" (UID: \"d841141c-29f5-48bf-9807-92267e6373ee\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc5675cq" Oct 31 04:41:50.867352 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:50.863299 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d841141c-29f5-48bf-9807-92267e6373ee-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-7bc5675cq\" (UID: \"d841141c-29f5-48bf-9807-92267e6373ee\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc5675cq" Oct 31 04:41:50.976483 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:50.966257 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jpwt\" (UniqueName: \"kubernetes.io/projected/d841141c-29f5-48bf-9807-92267e6373ee-kube-api-access-2jpwt\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-7bc5675cq\" (UID: \"d841141c-29f5-48bf-9807-92267e6373ee\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc5675cq" Oct 31 04:41:50.976483 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:50.966399 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d841141c-29f5-48bf-9807-92267e6373ee-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-7bc5675cq\" (UID: \"d841141c-29f5-48bf-9807-92267e6373ee\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc5675cq" Oct 31 04:41:50.976483 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:50.966465 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d841141c-29f5-48bf-9807-92267e6373ee-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-7bc5675cq\" (UID: \"d841141c-29f5-48bf-9807-92267e6373ee\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc5675cq" Oct 31 04:41:50.976483 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:50.966665 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d841141c-29f5-48bf-9807-92267e6373ee-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-7bc5675cq\" (UID: \"d841141c-29f5-48bf-9807-92267e6373ee\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc5675cq" Oct 31 04:41:51.121446 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:51.121404 5019 patch_prober.go:29] interesting pod/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" start-of-body= Oct 31 04:41:51.130511 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:51.130469 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn" podUID="2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" Oct 31 04:41:51.130739 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:51.126062 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d841141c-29f5-48bf-9807-92267e6373ee-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-7bc5675cq\" (UID: \"d841141c-29f5-48bf-9807-92267e6373ee\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc5675cq" Oct 31 04:41:51.149848 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:51.125884 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d841141c-29f5-48bf-9807-92267e6373ee-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-7bc5675cq\" (UID: \"d841141c-29f5-48bf-9807-92267e6373ee\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc5675cq" Oct 31 04:41:51.169027 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:51.168993 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d841141c-29f5-48bf-9807-92267e6373ee-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-7bc5675cq\" (UID: \"d841141c-29f5-48bf-9807-92267e6373ee\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc5675cq" Oct 31 04:41:51.235410 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:51.211688 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jpwt\" (UniqueName: \"kubernetes.io/projected/d841141c-29f5-48bf-9807-92267e6373ee-kube-api-access-2jpwt\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-7bc5675cq\" (UID: \"d841141c-29f5-48bf-9807-92267e6373ee\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc5675cq" Oct 31 04:41:51.490410 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:51.483196 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc5675cq" Oct 31 04:41:51.497532 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:51.497499 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cf8wdr4" Oct 31 04:41:51.499171 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:51.499144 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-99ca0c/explicit-model-99ca0c-kserve-c975b5fdf-lsvd6"] Oct 31 04:41:51.841063 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:51.840964 5019 patch_prober.go:29] interesting pod/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" start-of-body= Oct 31 04:41:51.841063 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:51.841028 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv" podUID="2f84895b-57a1-4c93-b20c-dc3f2ca754b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" Oct 31 04:41:52.138431 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:52.115954 5019 patch_prober.go:29] interesting pod/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" start-of-body= Oct 31 04:41:52.138431 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:52.116021 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn" podUID="2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" Oct 31 04:41:52.160386 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:52.153301 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc5675cq"] Oct 31 04:41:52.452931 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:41:52.452892 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:41:52.768898 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:52.768859 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc5675cq" event={"ID":"d841141c-29f5-48bf-9807-92267e6373ee","Type":"ContainerStarted","Data":"15b88dc61ceee242b03cdbb2fc14912f05d949d9bf1c32bc6b9f08f68e214b6d"} Oct 31 04:41:52.769112 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:52.768927 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc5675cq" event={"ID":"d841141c-29f5-48bf-9807-92267e6373ee","Type":"ContainerStarted","Data":"ee640fb8ce89fb19691fb476cfb9b3d28bfd7789cf83e9623888eab56bfb6160"} Oct 31 04:41:52.769112 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:52.768958 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc5675cq" Oct 31 04:41:52.825357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:52.817947 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc5675cq" podStartSLOduration=2.817922458 podStartE2EDuration="2.817922458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:41:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:41:52.810177223 +0000 UTC m=+2267.566515264" watchObservedRunningTime="2026-10-31 04:41:52.817922458 +0000 UTC m=+2267.574260498" Oct 31 04:41:52.842540 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:52.840848 5019 patch_prober.go:29] interesting pod/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" start-of-body= Oct 31 04:41:52.842540 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:52.840898 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv" podUID="2f84895b-57a1-4c93-b20c-dc3f2ca754b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" Oct 31 04:41:53.213277 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:53.213183 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn" Oct 31 04:41:53.216311 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:53.216229 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn" Oct 31 04:41:53.871729 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:53.867051 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv" Oct 31 04:41:53.871729 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:53.868644 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv" Oct 31 04:41:53.876021 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:53.875988 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv" Oct 31 04:41:56.809749 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:56.809710 5019 generic.go:356] "Generic (PLEG): container finished" podID="17f31b2c-2988-45e1-92ae-e43ff492444a" containerID="c6af17490a3d2c22c621122f5463b2021aa83b76ff7e6dc4b6dfd89f99ac1689" exitCode=0 Oct 31 04:41:56.810292 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:56.809783 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk" event={"ID":"17f31b2c-2988-45e1-92ae-e43ff492444a","Type":"ContainerDied","Data":"c6af17490a3d2c22c621122f5463b2021aa83b76ff7e6dc4b6dfd89f99ac1689"} Oct 31 04:41:56.810292 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:56.810287 5019 scope.go:122] "RemoveContainer" containerID="c6af17490a3d2c22c621122f5463b2021aa83b76ff7e6dc4b6dfd89f99ac1689" Oct 31 04:41:56.812279 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:56.812244 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:41:56.812446 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:56.812370 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:41:56.812446 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:56.812422 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:41:57.752290 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:57.752249 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk" Oct 31 04:41:57.837033 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:57.836986 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk" event={"ID":"17f31b2c-2988-45e1-92ae-e43ff492444a","Type":"ContainerStarted","Data":"542b8882dcaccae106c7375802a129b8f749f6d1a38c3394460d4a81cc4a7ba9"} Oct 31 04:41:58.517885 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:58.517851 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk" Oct 31 04:41:58.519208 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:58.519178 5019 patch_prober.go:29] interesting pod/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body= Oct 31 04:41:58.519419 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:58.519237 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk" podUID="17f31b2c-2988-45e1-92ae-e43ff492444a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" Oct 31 04:41:59.518690 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:59.518617 5019 patch_prober.go:29] interesting pod/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body= Oct 31 04:41:59.519364 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:41:59.519300 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk" podUID="17f31b2c-2988-45e1-92ae-e43ff492444a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" Oct 31 04:42:00.517231 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:00.517188 5019 patch_prober.go:29] interesting pod/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body= Oct 31 04:42:00.517476 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:00.517250 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk" podUID="17f31b2c-2988-45e1-92ae-e43ff492444a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" Oct 31 04:42:01.517557 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:01.517514 5019 patch_prober.go:29] interesting pod/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body= Oct 31 04:42:01.518102 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:01.517596 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk" podUID="17f31b2c-2988-45e1-92ae-e43ff492444a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" Oct 31 04:42:02.517989 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:02.517938 5019 patch_prober.go:29] interesting pod/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body= Oct 31 04:42:02.518644 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:02.518604 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk" podUID="17f31b2c-2988-45e1-92ae-e43ff492444a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" Oct 31 04:42:03.523885 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:03.523853 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk" Oct 31 04:42:03.524453 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:03.524373 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk" Oct 31 04:42:03.525192 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:03.525167 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk" Oct 31 04:42:03.782316 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:03.782228 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc5675cq" Oct 31 04:42:06.562889 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:06.562852 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8m5q9c"] Oct 31 04:42:06.567949 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:06.567919 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8m5q9c" Oct 31 04:42:06.590031 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:06.589997 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8m5q9c"] Oct 31 04:42:06.594166 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:06.594129 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-567ca6\"/\"agc-cred-34c4b3c7af7f\"" Oct 31 04:42:06.594166 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:06.594145 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-567ca6\"/\"kube-root-ca.crt\"" Oct 31 04:42:06.594166 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:06.594178 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-567ca6\"/\"payload-processing-external-model-e2e--a8f34224-dockercfg-s86rw\"" Oct 31 04:42:06.599803 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:06.599775 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-567ca6\"/\"payload-processing-external-model-plugins-e2e-extmod-b-567ca6\"" Oct 31 04:42:06.599983 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:06.599798 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-567ca6\"/\"openshift-service-ca.crt\"" Oct 31 04:42:06.599983 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:06.599887 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-567ca6\"/\"routing-overlay\"" Oct 31 04:42:06.611688 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:06.611620 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-99ca0c/explicit-model-99ca0c-kserve-c975b5fdf-lsvd6" podUID="c64c7635-8fa4-4d84-93aa-95e846139f7f" containerName="main" containerID="cri-o://21770dbe63c989aee1f5d787ec23167a7e0560622087420b8c281b6ac139a942" gracePeriod=45 Oct 31 04:42:06.763439 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:06.746688 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-67f8m5q9c\" (UID: \"54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8m5q9c" Oct 31 04:42:06.770365 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:06.763817 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-67f8m5q9c\" (UID: \"54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8m5q9c" Oct 31 04:42:06.770365 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:06.763973 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-67f8m5q9c\" (UID: \"54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8m5q9c" Oct 31 04:42:06.770365 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:06.764039 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpjzg\" (UniqueName: \"kubernetes.io/projected/54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f-kube-api-access-mpjzg\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-67f8m5q9c\" (UID: \"54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8m5q9c" Oct 31 04:42:06.880413 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:06.879939 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-67f8m5q9c\" (UID: \"54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8m5q9c" Oct 31 04:42:06.880413 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:06.880069 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-67f8m5q9c\" (UID: \"54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8m5q9c" Oct 31 04:42:06.880413 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:06.880150 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-67f8m5q9c\" (UID: \"54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8m5q9c" Oct 31 04:42:06.880413 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:06.880204 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpjzg\" (UniqueName: \"kubernetes.io/projected/54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f-kube-api-access-mpjzg\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-67f8m5q9c\" (UID: \"54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8m5q9c" Oct 31 04:42:06.902360 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:06.881634 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-67f8m5q9c\" (UID: \"54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8m5q9c" Oct 31 04:42:06.906266 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:06.906227 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-67f8m5q9c\" (UID: \"54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8m5q9c" Oct 31 04:42:06.942211 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:06.927204 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpjzg\" (UniqueName: \"kubernetes.io/projected/54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f-kube-api-access-mpjzg\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-67f8m5q9c\" (UID: \"54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8m5q9c" Oct 31 04:42:06.943029 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:06.942989 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-67f8m5q9c\" (UID: \"54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8m5q9c" Oct 31 04:42:06.990724 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:06.990683 5019 generic.go:356] "Generic (PLEG): container finished" podID="c64c7635-8fa4-4d84-93aa-95e846139f7f" containerID="21770dbe63c989aee1f5d787ec23167a7e0560622087420b8c281b6ac139a942" exitCode=0 Oct 31 04:42:06.991035 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:06.990757 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-99ca0c/explicit-model-99ca0c-kserve-c975b5fdf-lsvd6" event={"ID":"c64c7635-8fa4-4d84-93aa-95e846139f7f","Type":"ContainerDied","Data":"21770dbe63c989aee1f5d787ec23167a7e0560622087420b8c281b6ac139a942"} Oct 31 04:42:07.157984 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:07.157953 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-99ca0c/explicit-model-99ca0c-kserve-c975b5fdf-lsvd6" Oct 31 04:42:07.191143 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:07.191104 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8m5q9c" Oct 31 04:42:07.191379 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:07.191342 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c64c7635-8fa4-4d84-93aa-95e846139f7f-kube-api-access-p9lrc\" (UniqueName: \"kubernetes.io/projected/c64c7635-8fa4-4d84-93aa-95e846139f7f-kube-api-access-p9lrc\") pod \"c64c7635-8fa4-4d84-93aa-95e846139f7f\" (UID: \"c64c7635-8fa4-4d84-93aa-95e846139f7f\") " Oct 31 04:42:07.191720 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:07.191690 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c64c7635-8fa4-4d84-93aa-95e846139f7f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c64c7635-8fa4-4d84-93aa-95e846139f7f-dshm\") pod \"c64c7635-8fa4-4d84-93aa-95e846139f7f\" (UID: \"c64c7635-8fa4-4d84-93aa-95e846139f7f\") " Oct 31 04:42:07.191929 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:07.191904 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c64c7635-8fa4-4d84-93aa-95e846139f7f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c64c7635-8fa4-4d84-93aa-95e846139f7f-model-cache\") pod \"c64c7635-8fa4-4d84-93aa-95e846139f7f\" (UID: \"c64c7635-8fa4-4d84-93aa-95e846139f7f\") " Oct 31 04:42:07.192043 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:07.191994 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c64c7635-8fa4-4d84-93aa-95e846139f7f-home\" (UniqueName: \"kubernetes.io/empty-dir/c64c7635-8fa4-4d84-93aa-95e846139f7f-home\") pod \"c64c7635-8fa4-4d84-93aa-95e846139f7f\" (UID: \"c64c7635-8fa4-4d84-93aa-95e846139f7f\") " Oct 31 04:42:07.192043 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:07.192032 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c64c7635-8fa4-4d84-93aa-95e846139f7f-tls-certs\" (UniqueName: \"kubernetes.io/secret/c64c7635-8fa4-4d84-93aa-95e846139f7f-tls-certs\") pod \"c64c7635-8fa4-4d84-93aa-95e846139f7f\" (UID: \"c64c7635-8fa4-4d84-93aa-95e846139f7f\") " Oct 31 04:42:07.192225 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:07.192064 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c64c7635-8fa4-4d84-93aa-95e846139f7f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c64c7635-8fa4-4d84-93aa-95e846139f7f-tmp-dir\") pod \"c64c7635-8fa4-4d84-93aa-95e846139f7f\" (UID: \"c64c7635-8fa4-4d84-93aa-95e846139f7f\") " Oct 31 04:42:07.193042 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:07.192671 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c64c7635-8fa4-4d84-93aa-95e846139f7f-home" pod "c64c7635-8fa4-4d84-93aa-95e846139f7f" (UID: "c64c7635-8fa4-4d84-93aa-95e846139f7f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:42:07.193042 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:07.192910 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c64c7635-8fa4-4d84-93aa-95e846139f7f-tmp-dir" pod "c64c7635-8fa4-4d84-93aa-95e846139f7f" (UID: "c64c7635-8fa4-4d84-93aa-95e846139f7f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:42:07.193042 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:07.192957 5019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c64c7635-8fa4-4d84-93aa-95e846139f7f-home\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:42:07.193042 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:07.193020 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c64c7635-8fa4-4d84-93aa-95e846139f7f-model-cache" pod "c64c7635-8fa4-4d84-93aa-95e846139f7f" (UID: "c64c7635-8fa4-4d84-93aa-95e846139f7f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:42:07.195527 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:07.195503 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c64c7635-8fa4-4d84-93aa-95e846139f7f-dshm" pod "c64c7635-8fa4-4d84-93aa-95e846139f7f" (UID: "c64c7635-8fa4-4d84-93aa-95e846139f7f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:42:07.195527 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:07.195521 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c64c7635-8fa4-4d84-93aa-95e846139f7f-tls-certs" pod "c64c7635-8fa4-4d84-93aa-95e846139f7f" (UID: "c64c7635-8fa4-4d84-93aa-95e846139f7f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:42:07.195711 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:07.195568 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c64c7635-8fa4-4d84-93aa-95e846139f7f-kube-api-access-p9lrc" pod "c64c7635-8fa4-4d84-93aa-95e846139f7f" (UID: "c64c7635-8fa4-4d84-93aa-95e846139f7f"). InnerVolumeSpecName "kube-api-access-p9lrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:42:07.294780 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:07.294744 5019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c64c7635-8fa4-4d84-93aa-95e846139f7f-tls-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:42:07.294780 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:07.294781 5019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c64c7635-8fa4-4d84-93aa-95e846139f7f-tmp-dir\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:42:07.295016 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:07.294797 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9lrc\" (UniqueName: \"kubernetes.io/projected/c64c7635-8fa4-4d84-93aa-95e846139f7f-kube-api-access-p9lrc\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:42:07.295016 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:07.294812 5019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c64c7635-8fa4-4d84-93aa-95e846139f7f-dshm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:42:07.295016 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:07.294829 5019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c64c7635-8fa4-4d84-93aa-95e846139f7f-model-cache\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:42:07.329476 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:07.329427 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4"] Oct 31 04:42:07.330461 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:07.330440 5019 cpu_manager.go:414] "containerMap: removing container" podUID="c64c7635-8fa4-4d84-93aa-95e846139f7f" containerName="main" Oct 31 04:42:07.330461 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:07.330461 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c64c7635-8fa4-4d84-93aa-95e846139f7f" containerName="main" Oct 31 04:42:07.330610 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:07.330593 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="c64c7635-8fa4-4d84-93aa-95e846139f7f" containerName="main" Oct 31 04:42:07.335703 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:07.335681 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4" Oct 31 04:42:07.351136 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:07.351088 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4"] Oct 31 04:42:07.352780 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:07.352752 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-227c93-openshift-default-dockercfg-g2wrb\"" Oct 31 04:42:07.396494 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:07.396323 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-istio-token\") pod \"e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4\" (UID: \"39eb6eb3-a2ae-4931-bab8-1e3e07e7d764\") " pod="openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4" Oct 31 04:42:07.396904 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:07.396876 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-workload-socket\") pod \"e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4\" (UID: \"39eb6eb3-a2ae-4931-bab8-1e3e07e7d764\") " pod="openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4" Oct 31 04:42:07.397135 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:07.397117 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r7vq\" (UniqueName: \"kubernetes.io/projected/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-kube-api-access-8r7vq\") pod \"e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4\" (UID: \"39eb6eb3-a2ae-4931-bab8-1e3e07e7d764\") " pod="openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4" Oct 31 04:42:07.397414 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:07.397397 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-istio-podinfo\") pod \"e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4\" (UID: \"39eb6eb3-a2ae-4931-bab8-1e3e07e7d764\") " pod="openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4" Oct 31 04:42:07.397649 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:07.397632 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-credential-socket\") pod \"e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4\" (UID: \"39eb6eb3-a2ae-4931-bab8-1e3e07e7d764\") " pod="openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4" Oct 31 04:42:07.397805 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:07.397790 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-istiod-ca-cert\") pod \"e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4\" (UID: \"39eb6eb3-a2ae-4931-bab8-1e3e07e7d764\") " pod="openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4" Oct 31 04:42:07.397940 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:07.397927 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-istio-envoy\") pod \"e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4\" (UID: \"39eb6eb3-a2ae-4931-bab8-1e3e07e7d764\") " pod="openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4" Oct 31 04:42:07.398103 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:07.398085 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-istio-data\") pod \"e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4\" (UID: \"39eb6eb3-a2ae-4931-bab8-1e3e07e7d764\") " pod="openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4" Oct 31 04:42:07.398447 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:07.398427 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-workload-certs\") pod \"e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4\" (UID: \"39eb6eb3-a2ae-4931-bab8-1e3e07e7d764\") " pod="openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4" Oct 31 04:42:07.534361 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:07.510750 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-workload-socket\") pod \"e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4\" (UID: \"39eb6eb3-a2ae-4931-bab8-1e3e07e7d764\") " pod="openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4" Oct 31 04:42:07.534361 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:07.510869 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r7vq\" (UniqueName: \"kubernetes.io/projected/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-kube-api-access-8r7vq\") pod \"e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4\" (UID: \"39eb6eb3-a2ae-4931-bab8-1e3e07e7d764\") " pod="openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4" Oct 31 04:42:07.534361 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:07.511011 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-istio-podinfo\") pod \"e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4\" (UID: \"39eb6eb3-a2ae-4931-bab8-1e3e07e7d764\") " pod="openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4" Oct 31 04:42:07.534361 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:07.511044 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-credential-socket\") pod \"e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4\" (UID: \"39eb6eb3-a2ae-4931-bab8-1e3e07e7d764\") " pod="openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4" Oct 31 04:42:07.534361 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:07.511086 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-istiod-ca-cert\") pod \"e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4\" (UID: \"39eb6eb3-a2ae-4931-bab8-1e3e07e7d764\") " pod="openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4" Oct 31 04:42:07.534361 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:07.511117 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-istio-envoy\") pod \"e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4\" (UID: \"39eb6eb3-a2ae-4931-bab8-1e3e07e7d764\") " pod="openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4" Oct 31 04:42:07.534361 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:07.511183 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-istio-data\") pod \"e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4\" (UID: \"39eb6eb3-a2ae-4931-bab8-1e3e07e7d764\") " pod="openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4" Oct 31 04:42:07.534971 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:07.511373 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-workload-certs\") pod \"e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4\" (UID: \"39eb6eb3-a2ae-4931-bab8-1e3e07e7d764\") " pod="openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4" Oct 31 04:42:07.535546 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:07.535513 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-istio-token\") pod \"e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4\" (UID: \"39eb6eb3-a2ae-4931-bab8-1e3e07e7d764\") " pod="openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4" Oct 31 04:42:07.544357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:07.512505 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-credential-socket\") pod \"e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4\" (UID: \"39eb6eb3-a2ae-4931-bab8-1e3e07e7d764\") " pod="openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4" Oct 31 04:42:07.546601 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:07.518743 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-workload-certs\") pod \"e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4\" (UID: \"39eb6eb3-a2ae-4931-bab8-1e3e07e7d764\") " pod="openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4" Oct 31 04:42:07.561151 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:07.519541 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-istio-data\") pod \"e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4\" (UID: \"39eb6eb3-a2ae-4931-bab8-1e3e07e7d764\") " pod="openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4" Oct 31 04:42:07.561413 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:07.534773 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-istiod-ca-cert\") pod \"e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4\" (UID: \"39eb6eb3-a2ae-4931-bab8-1e3e07e7d764\") " pod="openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4" Oct 31 04:42:07.561533 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:07.511593 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-workload-socket\") pod \"e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4\" (UID: \"39eb6eb3-a2ae-4931-bab8-1e3e07e7d764\") " pod="openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4" Oct 31 04:42:07.600357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:07.574883 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8m5q9c"] Oct 31 04:42:07.619838 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:07.609779 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-istio-token\") pod \"e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4\" (UID: \"39eb6eb3-a2ae-4931-bab8-1e3e07e7d764\") " pod="openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4" Oct 31 04:42:07.633180 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:07.633102 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r7vq\" (UniqueName: \"kubernetes.io/projected/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-kube-api-access-8r7vq\") pod \"e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4\" (UID: \"39eb6eb3-a2ae-4931-bab8-1e3e07e7d764\") " pod="openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4" Oct 31 04:42:07.653430 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:07.641775 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-istio-envoy\") pod \"e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4\" (UID: \"39eb6eb3-a2ae-4931-bab8-1e3e07e7d764\") " pod="openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4" Oct 31 04:42:07.709648 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:07.709587 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-istio-podinfo\") pod \"e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4\" (UID: \"39eb6eb3-a2ae-4931-bab8-1e3e07e7d764\") " pod="openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4" Oct 31 04:42:07.709943 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:07.709923 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4" Oct 31 04:42:08.004982 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:08.004930 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8m5q9c" event={"ID":"54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f","Type":"ContainerStarted","Data":"7db5592b443ab1cccb9f109b4ce310ec6829b30d5a58308ec74e419eab88c177"} Oct 31 04:42:08.005196 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:08.005041 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8m5q9c" Oct 31 04:42:08.005196 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:08.005068 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8m5q9c" event={"ID":"54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f","Type":"ContainerStarted","Data":"7d0d002f5c97c114add1689b1200e7ebd34fb7bf6f965873f6433540e6d67812"} Oct 31 04:42:08.007088 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:08.007058 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-99ca0c/explicit-model-99ca0c-kserve-c975b5fdf-lsvd6" event={"ID":"c64c7635-8fa4-4d84-93aa-95e846139f7f","Type":"ContainerDied","Data":"be1129aa8ea2cfe2ab456222580d68ba9ff3f8d079311107dcdc73157901bcdd"} Oct 31 04:42:08.007259 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:08.007103 5019 scope.go:122] "RemoveContainer" containerID="21770dbe63c989aee1f5d787ec23167a7e0560622087420b8c281b6ac139a942" Oct 31 04:42:08.007259 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:08.007064 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-99ca0c/explicit-model-99ca0c-kserve-c975b5fdf-lsvd6" Oct 31 04:42:08.055024 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:08.054960 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8m5q9c" podStartSLOduration=2.054939273 podStartE2EDuration="2.054939273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:42:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:42:08.050475128 +0000 UTC m=+2282.806813167" watchObservedRunningTime="2026-10-31 04:42:08.054939273 +0000 UTC m=+2282.811277317" Oct 31 04:42:08.093841 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:08.093798 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-99ca0c/explicit-model-99ca0c-kserve-c975b5fdf-lsvd6"] Oct 31 04:42:08.099453 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:08.099417 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-99ca0c/explicit-model-99ca0c-kserve-c975b5fdf-lsvd6"] Oct 31 04:42:08.107598 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:42:08.106481 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39eb6eb3_a2ae_4931_bab8_1e3e07e7d764.slice/crio-9e63a6c28088564bf014675e9e3c6b864aea7fb3d89a93dd0903060e00579115 WatchSource:0}: Error finding container 9e63a6c28088564bf014675e9e3c6b864aea7fb3d89a93dd0903060e00579115: Status 404 returned error can't find the container with id 9e63a6c28088564bf014675e9e3c6b864aea7fb3d89a93dd0903060e00579115 Oct 31 04:42:08.115502 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:08.115440 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:42:08.115688 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:08.115624 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:42:08.115748 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:08.115716 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:42:08.118749 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:08.118711 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4"] Oct 31 04:42:09.018137 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:09.018096 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4" event={"ID":"39eb6eb3-a2ae-4931-bab8-1e3e07e7d764","Type":"ContainerStarted","Data":"ceaa0513f8cbeb4411b336dc868532a62efa55a1e389ef109e97c3eec5ade4ba"} Oct 31 04:42:09.018676 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:09.018168 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4" event={"ID":"39eb6eb3-a2ae-4931-bab8-1e3e07e7d764","Type":"ContainerStarted","Data":"9e63a6c28088564bf014675e9e3c6b864aea7fb3d89a93dd0903060e00579115"} Oct 31 04:42:09.079566 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:09.079455 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4" podStartSLOduration=2.079430956 podStartE2EDuration="2.079430956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:42:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:42:09.074168298 +0000 UTC m=+2283.830506338" watchObservedRunningTime="2026-10-31 04:42:09.079430956 +0000 UTC m=+2283.835768997" Oct 31 04:42:09.416523 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:09.416425 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c64c7635-8fa4-4d84-93aa-95e846139f7f" path="/var/lib/kubelet/pods/c64c7635-8fa4-4d84-93aa-95e846139f7f/volumes" Oct 31 04:42:09.712444 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:09.711776 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4" Oct 31 04:42:09.719431 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:09.719391 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4" Oct 31 04:42:10.030133 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:10.030091 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4" Oct 31 04:42:10.031502 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:10.031476 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4" Oct 31 04:42:13.919359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:13.910667 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-227c93-5c594ccb75-k8gp2"] Oct 31 04:42:13.967037 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:13.966922 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-227c93-5c594ccb75-k8gp2"] Oct 31 04:42:13.967996 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:13.967895 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-227c93-5c594ccb75-k8gp2" Oct 31 04:42:13.983844 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:13.983692 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-227c93-dockercfg-9jfts\"" Oct 31 04:42:13.983844 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:13.983780 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-227c93\"" Oct 31 04:42:14.056493 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:14.056459 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16674b5b-1b4d-40d5-aa1d-704a9c56b54b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-227c93-5c594ccb75-k8gp2\" (UID: \"16674b5b-1b4d-40d5-aa1d-704a9c56b54b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-227c93-5c594ccb75-k8gp2" Oct 31 04:42:14.056737 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:14.056713 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nc6v\" (UniqueName: \"kubernetes.io/projected/16674b5b-1b4d-40d5-aa1d-704a9c56b54b-kube-api-access-2nc6v\") pod \"payload-pre-processing-e2e-extmod-b-227c93-5c594ccb75-k8gp2\" (UID: \"16674b5b-1b4d-40d5-aa1d-704a9c56b54b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-227c93-5c594ccb75-k8gp2" Oct 31 04:42:14.160365 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:14.159115 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nc6v\" (UniqueName: \"kubernetes.io/projected/16674b5b-1b4d-40d5-aa1d-704a9c56b54b-kube-api-access-2nc6v\") pod \"payload-pre-processing-e2e-extmod-b-227c93-5c594ccb75-k8gp2\" (UID: \"16674b5b-1b4d-40d5-aa1d-704a9c56b54b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-227c93-5c594ccb75-k8gp2" Oct 31 04:42:14.160365 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:14.159259 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16674b5b-1b4d-40d5-aa1d-704a9c56b54b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-227c93-5c594ccb75-k8gp2\" (UID: \"16674b5b-1b4d-40d5-aa1d-704a9c56b54b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-227c93-5c594ccb75-k8gp2" Oct 31 04:42:14.160877 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:14.160838 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16674b5b-1b4d-40d5-aa1d-704a9c56b54b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-227c93-5c594ccb75-k8gp2\" (UID: \"16674b5b-1b4d-40d5-aa1d-704a9c56b54b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-227c93-5c594ccb75-k8gp2" Oct 31 04:42:14.183791 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:14.183727 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nc6v\" (UniqueName: \"kubernetes.io/projected/16674b5b-1b4d-40d5-aa1d-704a9c56b54b-kube-api-access-2nc6v\") pod \"payload-pre-processing-e2e-extmod-b-227c93-5c594ccb75-k8gp2\" (UID: \"16674b5b-1b4d-40d5-aa1d-704a9c56b54b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-227c93-5c594ccb75-k8gp2" Oct 31 04:42:14.294188 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:14.294144 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-227c93-5c594ccb75-k8gp2" Oct 31 04:42:14.611665 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:14.611602 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-227c93-5c594ccb75-k8gp2"] Oct 31 04:42:14.614119 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:42:14.614086 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16674b5b_1b4d_40d5_aa1d_704a9c56b54b.slice/crio-a07872063ed96ab7e6ee6ae5a9e8de94a2424d5f83600e3cc725d1398945639a WatchSource:0}: Error finding container a07872063ed96ab7e6ee6ae5a9e8de94a2424d5f83600e3cc725d1398945639a: Status 404 returned error can't find the container with id a07872063ed96ab7e6ee6ae5a9e8de94a2424d5f83600e3cc725d1398945639a Oct 31 04:42:15.100715 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:15.100672 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-227c93-5c594ccb75-k8gp2" event={"ID":"16674b5b-1b4d-40d5-aa1d-704a9c56b54b","Type":"ContainerStarted","Data":"af8420201364c24f0ba46fd4cc30670ae1f1c76da091787aa63374dd9638bd24"} Oct 31 04:42:15.101162 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:15.100748 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-227c93-5c594ccb75-k8gp2" event={"ID":"16674b5b-1b4d-40d5-aa1d-704a9c56b54b","Type":"ContainerStarted","Data":"a07872063ed96ab7e6ee6ae5a9e8de94a2424d5f83600e3cc725d1398945639a"} Oct 31 04:42:15.101162 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:15.100791 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-227c93-5c594ccb75-k8gp2" Oct 31 04:42:15.175676 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:15.173042 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-227c93-5c594ccb75-k8gp2" podStartSLOduration=2.173016961 podStartE2EDuration="2.173016961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:42:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:42:15.143545853 +0000 UTC m=+2289.899883893" watchObservedRunningTime="2026-10-31 04:42:15.173016961 +0000 UTC m=+2289.929355008" Oct 31 04:42:19.024511 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:19.024475 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8m5q9c" Oct 31 04:42:22.393613 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:42:22.393571 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:42:26.114401 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:26.114326 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-227c93-5c594ccb75-k8gp2" Oct 31 04:42:33.080663 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:33.080614 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-99ca0c-787d86f4c5-rhkjg"] Oct 31 04:42:33.148309 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:33.148271 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-99ca0c-568c687f79-tqpf6"] Oct 31 04:42:38.081246 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:38.081197 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-99ca0c-787d86f4c5-rhkjg" podUID="13506896-9897-4d94-91df-2d3c6eb87f68" containerName="payload-processing" containerID="cri-o://1117fbfa053037e7c206c57e891f51a6770e1dc6cbfffb6cb1ece22860e27b2f" gracePeriod=25 Oct 31 04:42:38.149261 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:38.149215 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-99ca0c-568c687f79-tqpf6" podUID="d42130a2-25c9-4a81-ae93-1d6219e5794f" containerName="payload-pre-processing" containerID="cri-o://927a73ebfe1ecdc9afb0f5c29284d07d828a0da65b9a3cb567f6ac8553d2decf" gracePeriod=25 Oct 31 04:42:38.528303 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:38.528269 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-99ca0c-787d86f4c5-rhkjg" Oct 31 04:42:38.534891 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:38.534861 5019 generic.go:356] "Generic (PLEG): container finished" podID="d42130a2-25c9-4a81-ae93-1d6219e5794f" containerID="927a73ebfe1ecdc9afb0f5c29284d07d828a0da65b9a3cb567f6ac8553d2decf" exitCode=0 Oct 31 04:42:38.535046 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:38.534930 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-99ca0c-568c687f79-tqpf6" event={"ID":"d42130a2-25c9-4a81-ae93-1d6219e5794f","Type":"ContainerDied","Data":"927a73ebfe1ecdc9afb0f5c29284d07d828a0da65b9a3cb567f6ac8553d2decf"} Oct 31 04:42:38.536704 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:38.536678 5019 generic.go:356] "Generic (PLEG): container finished" podID="13506896-9897-4d94-91df-2d3c6eb87f68" containerID="1117fbfa053037e7c206c57e891f51a6770e1dc6cbfffb6cb1ece22860e27b2f" exitCode=0 Oct 31 04:42:38.536836 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:38.536718 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-99ca0c-787d86f4c5-rhkjg" event={"ID":"13506896-9897-4d94-91df-2d3c6eb87f68","Type":"ContainerDied","Data":"1117fbfa053037e7c206c57e891f51a6770e1dc6cbfffb6cb1ece22860e27b2f"} Oct 31 04:42:38.536836 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:38.536731 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-99ca0c-787d86f4c5-rhkjg" Oct 31 04:42:38.536836 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:38.536755 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-99ca0c-787d86f4c5-rhkjg" event={"ID":"13506896-9897-4d94-91df-2d3c6eb87f68","Type":"ContainerDied","Data":"642591e55de8b75b4dfb59022d3b6c0c390b4514db4bed0bcff91332cbaa6506"} Oct 31 04:42:38.536836 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:38.536775 5019 scope.go:122] "RemoveContainer" containerID="1117fbfa053037e7c206c57e891f51a6770e1dc6cbfffb6cb1ece22860e27b2f" Oct 31 04:42:38.560455 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:38.560430 5019 scope.go:122] "RemoveContainer" containerID="1117fbfa053037e7c206c57e891f51a6770e1dc6cbfffb6cb1ece22860e27b2f" Oct 31 04:42:38.560785 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:42:38.560737 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1117fbfa053037e7c206c57e891f51a6770e1dc6cbfffb6cb1ece22860e27b2f\": container with ID starting with 1117fbfa053037e7c206c57e891f51a6770e1dc6cbfffb6cb1ece22860e27b2f not found: ID does not exist" containerID="1117fbfa053037e7c206c57e891f51a6770e1dc6cbfffb6cb1ece22860e27b2f" Oct 31 04:42:38.560844 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:38.560803 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1117fbfa053037e7c206c57e891f51a6770e1dc6cbfffb6cb1ece22860e27b2f"} err="failed to get container status \"1117fbfa053037e7c206c57e891f51a6770e1dc6cbfffb6cb1ece22860e27b2f\": rpc error: code = NotFound desc = could not find container \"1117fbfa053037e7c206c57e891f51a6770e1dc6cbfffb6cb1ece22860e27b2f\": container with ID starting with 1117fbfa053037e7c206c57e891f51a6770e1dc6cbfffb6cb1ece22860e27b2f not found: ID does not exist" Oct 31 04:42:38.561798 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:38.561776 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-99ca0c-568c687f79-tqpf6" Oct 31 04:42:38.626128 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:38.626029 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d42130a2-25c9-4a81-ae93-1d6219e5794f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d42130a2-25c9-4a81-ae93-1d6219e5794f-plugins-config-volume\") pod \"d42130a2-25c9-4a81-ae93-1d6219e5794f\" (UID: \"d42130a2-25c9-4a81-ae93-1d6219e5794f\") " Oct 31 04:42:38.626398 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:38.626378 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13506896-9897-4d94-91df-2d3c6eb87f68-kube-api-access-hq5hl\" (UniqueName: \"kubernetes.io/projected/13506896-9897-4d94-91df-2d3c6eb87f68-kube-api-access-hq5hl\") pod \"13506896-9897-4d94-91df-2d3c6eb87f68\" (UID: \"13506896-9897-4d94-91df-2d3c6eb87f68\") " Oct 31 04:42:38.626492 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:38.626464 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d42130a2-25c9-4a81-ae93-1d6219e5794f-kube-api-access-r6zf2\" (UniqueName: \"kubernetes.io/projected/d42130a2-25c9-4a81-ae93-1d6219e5794f-kube-api-access-r6zf2\") pod \"d42130a2-25c9-4a81-ae93-1d6219e5794f\" (UID: \"d42130a2-25c9-4a81-ae93-1d6219e5794f\") " Oct 31 04:42:38.626553 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:38.626535 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13506896-9897-4d94-91df-2d3c6eb87f68-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13506896-9897-4d94-91df-2d3c6eb87f68-plugins-config-volume\") pod \"13506896-9897-4d94-91df-2d3c6eb87f68\" (UID: \"13506896-9897-4d94-91df-2d3c6eb87f68\") " Oct 31 04:42:38.628429 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:38.628030 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13506896-9897-4d94-91df-2d3c6eb87f68-plugins-config-volume" pod "13506896-9897-4d94-91df-2d3c6eb87f68" (UID: "13506896-9897-4d94-91df-2d3c6eb87f68"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:42:38.628588 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:38.628535 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d42130a2-25c9-4a81-ae93-1d6219e5794f-plugins-config-volume" pod "d42130a2-25c9-4a81-ae93-1d6219e5794f" (UID: "d42130a2-25c9-4a81-ae93-1d6219e5794f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:42:38.630152 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:38.630124 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13506896-9897-4d94-91df-2d3c6eb87f68-kube-api-access-hq5hl" pod "13506896-9897-4d94-91df-2d3c6eb87f68" (UID: "13506896-9897-4d94-91df-2d3c6eb87f68"). InnerVolumeSpecName "kube-api-access-hq5hl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:42:38.632274 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:38.632249 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d42130a2-25c9-4a81-ae93-1d6219e5794f-kube-api-access-r6zf2" pod "d42130a2-25c9-4a81-ae93-1d6219e5794f" (UID: "d42130a2-25c9-4a81-ae93-1d6219e5794f"). InnerVolumeSpecName "kube-api-access-r6zf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:42:38.728713 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:38.728672 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hq5hl\" (UniqueName: \"kubernetes.io/projected/13506896-9897-4d94-91df-2d3c6eb87f68-kube-api-access-hq5hl\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:42:38.728713 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:38.728710 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6zf2\" (UniqueName: \"kubernetes.io/projected/d42130a2-25c9-4a81-ae93-1d6219e5794f-kube-api-access-r6zf2\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:42:38.728713 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:38.728719 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13506896-9897-4d94-91df-2d3c6eb87f68-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:42:38.728974 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:38.728730 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d42130a2-25c9-4a81-ae93-1d6219e5794f-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:42:38.885875 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:38.885781 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-99ca0c-787d86f4c5-rhkjg"] Oct 31 04:42:38.898136 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:38.898097 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-99ca0c-787d86f4c5-rhkjg"] Oct 31 04:42:39.411498 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:39.411458 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13506896-9897-4d94-91df-2d3c6eb87f68" path="/var/lib/kubelet/pods/13506896-9897-4d94-91df-2d3c6eb87f68/volumes" Oct 31 04:42:39.551304 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:39.551267 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-99ca0c-568c687f79-tqpf6" event={"ID":"d42130a2-25c9-4a81-ae93-1d6219e5794f","Type":"ContainerDied","Data":"0daf34be838f6eef7528437593a4a56462040c96c91696e9e3f321ad237bc18d"} Oct 31 04:42:39.551500 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:39.551327 5019 scope.go:122] "RemoveContainer" containerID="927a73ebfe1ecdc9afb0f5c29284d07d828a0da65b9a3cb567f6ac8553d2decf" Oct 31 04:42:39.551500 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:39.551280 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-99ca0c-568c687f79-tqpf6" Oct 31 04:42:39.608022 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:39.607979 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-99ca0c-568c687f79-tqpf6"] Oct 31 04:42:39.619851 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:39.619816 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-99ca0c-568c687f79-tqpf6"] Oct 31 04:42:41.420180 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:41.420133 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d42130a2-25c9-4a81-ae93-1d6219e5794f" path="/var/lib/kubelet/pods/d42130a2-25c9-4a81-ae93-1d6219e5794f/volumes" Oct 31 04:42:52.393089 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:42:52.393049 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:42:59.506353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:59.504860 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b6zwwdr"] Oct 31 04:42:59.506353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:59.506275 5019 cpu_manager.go:414] "containerMap: removing container" podUID="13506896-9897-4d94-91df-2d3c6eb87f68" containerName="payload-processing" Oct 31 04:42:59.506353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:59.506298 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13506896-9897-4d94-91df-2d3c6eb87f68" containerName="payload-processing" Oct 31 04:42:59.506972 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:59.506370 5019 cpu_manager.go:414] "containerMap: removing container" podUID="d42130a2-25c9-4a81-ae93-1d6219e5794f" containerName="payload-pre-processing" Oct 31 04:42:59.506972 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:59.506382 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d42130a2-25c9-4a81-ae93-1d6219e5794f" containerName="payload-pre-processing" Oct 31 04:42:59.506972 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:59.506782 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="13506896-9897-4d94-91df-2d3c6eb87f68" containerName="payload-processing" Oct 31 04:42:59.506972 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:59.506816 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="d42130a2-25c9-4a81-ae93-1d6219e5794f" containerName="payload-pre-processing" Oct 31 04:42:59.515480 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:59.515452 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b6zwwdr" Oct 31 04:42:59.536046 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:59.536004 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ded435\"/\"agc-cred-7fd1d61db48f\"" Oct 31 04:42:59.536534 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:59.536509 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b6zwwdr"] Oct 31 04:42:59.649471 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:59.649433 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjfxb\" (UniqueName: \"kubernetes.io/projected/c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88-kube-api-access-bjfxb\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-75b6zwwdr\" (UID: \"c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b6zwwdr" Oct 31 04:42:59.649656 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:59.649561 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-75b6zwwdr\" (UID: \"c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b6zwwdr" Oct 31 04:42:59.649745 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:59.649715 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-75b6zwwdr\" (UID: \"c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b6zwwdr" Oct 31 04:42:59.649903 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:59.649888 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-75b6zwwdr\" (UID: \"c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b6zwwdr" Oct 31 04:42:59.751454 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:59.751419 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-75b6zwwdr\" (UID: \"c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b6zwwdr" Oct 31 04:42:59.751694 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:59.751493 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-75b6zwwdr\" (UID: \"c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b6zwwdr" Oct 31 04:42:59.751694 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:59.751577 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-75b6zwwdr\" (UID: \"c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b6zwwdr" Oct 31 04:42:59.751694 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:59.751656 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjfxb\" (UniqueName: \"kubernetes.io/projected/c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88-kube-api-access-bjfxb\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-75b6zwwdr\" (UID: \"c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b6zwwdr" Oct 31 04:42:59.752172 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:59.752145 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-75b6zwwdr\" (UID: \"c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b6zwwdr" Oct 31 04:42:59.752268 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:59.752220 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-75b6zwwdr\" (UID: \"c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b6zwwdr" Oct 31 04:42:59.754259 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:59.754239 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-75b6zwwdr\" (UID: \"c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b6zwwdr" Oct 31 04:42:59.775546 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:59.775463 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjfxb\" (UniqueName: \"kubernetes.io/projected/c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88-kube-api-access-bjfxb\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-75b6zwwdr\" (UID: \"c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b6zwwdr" Oct 31 04:42:59.840856 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:42:59.840821 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b6zwwdr" Oct 31 04:43:00.167558 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:43:00.167520 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5d8a723_1bca_463c_9d7b_3a2e7b9d2a88.slice/crio-3cdb5bc6070ff7aac6407f19ea2e2f7944dd816d8ffbf2b78723361c0d7e888b WatchSource:0}: Error finding container 3cdb5bc6070ff7aac6407f19ea2e2f7944dd816d8ffbf2b78723361c0d7e888b: Status 404 returned error can't find the container with id 3cdb5bc6070ff7aac6407f19ea2e2f7944dd816d8ffbf2b78723361c0d7e888b Oct 31 04:43:00.168723 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:00.168694 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b6zwwdr"] Oct 31 04:43:00.297594 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:00.297561 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-79742xx4f"] Oct 31 04:43:00.302385 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:00.302366 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-79742xx4f" Oct 31 04:43:00.317248 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:00.317223 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a99f2a\"/\"agc-cred-add5bfec2790\"" Oct 31 04:43:00.324070 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:00.324046 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-79742xx4f"] Oct 31 04:43:00.461600 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:00.461517 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5jrb\" (UniqueName: \"kubernetes.io/projected/835f4c9d-588d-4fe1-8ec3-d916abc72af1-kube-api-access-m5jrb\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-79742xx4f\" (UID: \"835f4c9d-588d-4fe1-8ec3-d916abc72af1\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-79742xx4f" Oct 31 04:43:00.461768 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:00.461618 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/835f4c9d-588d-4fe1-8ec3-d916abc72af1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-79742xx4f\" (UID: \"835f4c9d-588d-4fe1-8ec3-d916abc72af1\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-79742xx4f" Oct 31 04:43:00.461768 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:00.461656 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/835f4c9d-588d-4fe1-8ec3-d916abc72af1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-79742xx4f\" (UID: \"835f4c9d-588d-4fe1-8ec3-d916abc72af1\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-79742xx4f" Oct 31 04:43:00.461881 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:00.461793 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/835f4c9d-588d-4fe1-8ec3-d916abc72af1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-79742xx4f\" (UID: \"835f4c9d-588d-4fe1-8ec3-d916abc72af1\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-79742xx4f" Oct 31 04:43:00.563829 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:00.563788 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5jrb\" (UniqueName: \"kubernetes.io/projected/835f4c9d-588d-4fe1-8ec3-d916abc72af1-kube-api-access-m5jrb\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-79742xx4f\" (UID: \"835f4c9d-588d-4fe1-8ec3-d916abc72af1\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-79742xx4f" Oct 31 04:43:00.564429 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:00.563853 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/835f4c9d-588d-4fe1-8ec3-d916abc72af1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-79742xx4f\" (UID: \"835f4c9d-588d-4fe1-8ec3-d916abc72af1\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-79742xx4f" Oct 31 04:43:00.564429 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:00.563886 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/835f4c9d-588d-4fe1-8ec3-d916abc72af1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-79742xx4f\" (UID: \"835f4c9d-588d-4fe1-8ec3-d916abc72af1\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-79742xx4f" Oct 31 04:43:00.564429 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:00.564015 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/835f4c9d-588d-4fe1-8ec3-d916abc72af1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-79742xx4f\" (UID: \"835f4c9d-588d-4fe1-8ec3-d916abc72af1\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-79742xx4f" Oct 31 04:43:00.564958 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:00.564928 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/835f4c9d-588d-4fe1-8ec3-d916abc72af1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-79742xx4f\" (UID: \"835f4c9d-588d-4fe1-8ec3-d916abc72af1\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-79742xx4f" Oct 31 04:43:00.565082 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:00.565053 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/835f4c9d-588d-4fe1-8ec3-d916abc72af1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-79742xx4f\" (UID: \"835f4c9d-588d-4fe1-8ec3-d916abc72af1\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-79742xx4f" Oct 31 04:43:00.567441 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:00.567413 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/835f4c9d-588d-4fe1-8ec3-d916abc72af1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-79742xx4f\" (UID: \"835f4c9d-588d-4fe1-8ec3-d916abc72af1\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-79742xx4f" Oct 31 04:43:00.591047 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:00.587805 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5jrb\" (UniqueName: \"kubernetes.io/projected/835f4c9d-588d-4fe1-8ec3-d916abc72af1-kube-api-access-m5jrb\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-79742xx4f\" (UID: \"835f4c9d-588d-4fe1-8ec3-d916abc72af1\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-79742xx4f" Oct 31 04:43:00.620291 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:00.620250 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-79742xx4f" Oct 31 04:43:00.929034 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:00.928999 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-79742xx4f"] Oct 31 04:43:00.929171 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:43:00.928996 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod835f4c9d_588d_4fe1_8ec3_d916abc72af1.slice/crio-84d62f5d32a77194f11d99a9f2b3eec75b75744842c981aeb5d6d8519784251a WatchSource:0}: Error finding container 84d62f5d32a77194f11d99a9f2b3eec75b75744842c981aeb5d6d8519784251a: Status 404 returned error can't find the container with id 84d62f5d32a77194f11d99a9f2b3eec75b75744842c981aeb5d6d8519784251a Oct 31 04:43:00.989570 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:00.989521 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-79742xx4f" event={"ID":"835f4c9d-588d-4fe1-8ec3-d916abc72af1","Type":"ContainerStarted","Data":"84d62f5d32a77194f11d99a9f2b3eec75b75744842c981aeb5d6d8519784251a"} Oct 31 04:43:00.991976 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:00.991932 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b6zwwdr" event={"ID":"c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88","Type":"ContainerStarted","Data":"40f55460b5abe5128bdb5da2ae93bdc891024ff271aad9c7c381e855430bf021"} Oct 31 04:43:00.992138 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:00.992002 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b6zwwdr" event={"ID":"c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88","Type":"ContainerStarted","Data":"3cdb5bc6070ff7aac6407f19ea2e2f7944dd816d8ffbf2b78723361c0d7e888b"} Oct 31 04:43:00.992211 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:00.992153 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b6zwwdr" Oct 31 04:43:01.034251 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:01.034192 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b6zwwdr" podStartSLOduration=2.034172773 podStartE2EDuration="2.034172773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:42:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:43:01.032854025 +0000 UTC m=+2335.789192065" watchObservedRunningTime="2026-10-31 04:43:01.034172773 +0000 UTC m=+2335.790510827" Oct 31 04:43:02.005243 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:02.005188 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-79742xx4f" event={"ID":"835f4c9d-588d-4fe1-8ec3-d916abc72af1","Type":"ContainerStarted","Data":"53cd8f3025f102574acaf5e5b280da8c6222f5d2619c3f9a33ba43f6a873f56a"} Oct 31 04:43:02.005854 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:02.005461 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-79742xx4f" Oct 31 04:43:02.052035 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:02.051974 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-79742xx4f" podStartSLOduration=2.051957798 podStartE2EDuration="2.051957798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:43:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:43:02.047271557 +0000 UTC m=+2336.803609598" watchObservedRunningTime="2026-10-31 04:43:02.051957798 +0000 UTC m=+2336.808295886" Oct 31 04:43:05.031592 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:05.031543 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51d08ef9-fqnzh"] Oct 31 04:43:05.043936 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:05.043887 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-691c771f-44gp5"] Oct 31 04:43:05.048912 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:05.048876 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49d411b5-f78zv"] Oct 31 04:43:05.053753 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:05.053718 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51d08ef9-fqnzh"] Oct 31 04:43:05.058245 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:05.058219 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-691c771f-44gp5"] Oct 31 04:43:05.062713 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:05.062685 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49d411b5-f78zv"] Oct 31 04:43:05.420572 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:05.420482 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dd71c11-2201-4923-8ba1-d6b14db03087" path="/var/lib/kubelet/pods/4dd71c11-2201-4923-8ba1-d6b14db03087/volumes" Oct 31 04:43:05.421489 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:05.421460 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db80704e-ed92-4de7-a6fb-d1189af34b16" path="/var/lib/kubelet/pods/db80704e-ed92-4de7-a6fb-d1189af34b16/volumes" Oct 31 04:43:05.422124 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:05.422103 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0814967-8fa3-4141-8fd2-3b4c3ff630b7" path="/var/lib/kubelet/pods/e0814967-8fa3-4141-8fd2-3b4c3ff630b7/volumes" Oct 31 04:43:06.993566 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:06.993527 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp"] Oct 31 04:43:06.994075 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:06.993787 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp" podUID="468817c7-e1ca-41c0-8a91-cfce50aa71ac" containerName="istio-proxy" containerID="cri-o://c5ffa648dd5facb6a15713079808dcd094cda1cb89634456282a04f7f27c4180" gracePeriod=30 Oct 31 04:43:09.337678 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:09.337641 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf"] Oct 31 04:43:09.345164 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:09.345135 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf" Oct 31 04:43:09.363446 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:09.363420 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-47c409-openshift-default-dockercfg-thhmx\"" Oct 31 04:43:09.383250 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:09.383215 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf"] Oct 31 04:43:09.394822 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:09.394787 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ce5abc5-55a0-4b15-80a6-7e21d360e389-istio-podinfo\") pod \"e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf\" (UID: \"7ce5abc5-55a0-4b15-80a6-7e21d360e389\") " pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf" Oct 31 04:43:09.395008 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:09.394846 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ce5abc5-55a0-4b15-80a6-7e21d360e389-istio-data\") pod \"e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf\" (UID: \"7ce5abc5-55a0-4b15-80a6-7e21d360e389\") " pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf" Oct 31 04:43:09.395008 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:09.394873 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ce5abc5-55a0-4b15-80a6-7e21d360e389-istiod-ca-cert\") pod \"e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf\" (UID: \"7ce5abc5-55a0-4b15-80a6-7e21d360e389\") " pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf" Oct 31 04:43:09.395008 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:09.394956 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ce5abc5-55a0-4b15-80a6-7e21d360e389-istio-envoy\") pod \"e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf\" (UID: \"7ce5abc5-55a0-4b15-80a6-7e21d360e389\") " pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf" Oct 31 04:43:09.395008 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:09.395002 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce5abc5-55a0-4b15-80a6-7e21d360e389-credential-socket\") pod \"e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf\" (UID: \"7ce5abc5-55a0-4b15-80a6-7e21d360e389\") " pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf" Oct 31 04:43:09.395345 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:09.395069 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ce5abc5-55a0-4b15-80a6-7e21d360e389-workload-certs\") pod \"e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf\" (UID: \"7ce5abc5-55a0-4b15-80a6-7e21d360e389\") " pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf" Oct 31 04:43:09.395345 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:09.395141 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce5abc5-55a0-4b15-80a6-7e21d360e389-workload-socket\") pod \"e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf\" (UID: \"7ce5abc5-55a0-4b15-80a6-7e21d360e389\") " pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf" Oct 31 04:43:09.395462 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:09.395385 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr9q9\" (UniqueName: \"kubernetes.io/projected/7ce5abc5-55a0-4b15-80a6-7e21d360e389-kube-api-access-xr9q9\") pod \"e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf\" (UID: \"7ce5abc5-55a0-4b15-80a6-7e21d360e389\") " pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf" Oct 31 04:43:09.395462 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:09.395441 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ce5abc5-55a0-4b15-80a6-7e21d360e389-istio-token\") pod \"e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf\" (UID: \"7ce5abc5-55a0-4b15-80a6-7e21d360e389\") " pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf" Oct 31 04:43:09.533354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:09.531858 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce5abc5-55a0-4b15-80a6-7e21d360e389-workload-socket\") pod \"e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf\" (UID: \"7ce5abc5-55a0-4b15-80a6-7e21d360e389\") " pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf" Oct 31 04:43:09.533354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:09.532672 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce5abc5-55a0-4b15-80a6-7e21d360e389-workload-socket\") pod \"e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf\" (UID: \"7ce5abc5-55a0-4b15-80a6-7e21d360e389\") " pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf" Oct 31 04:43:09.533635 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:09.533265 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr9q9\" (UniqueName: \"kubernetes.io/projected/7ce5abc5-55a0-4b15-80a6-7e21d360e389-kube-api-access-xr9q9\") pod \"e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf\" (UID: \"7ce5abc5-55a0-4b15-80a6-7e21d360e389\") " pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf" Oct 31 04:43:09.533635 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:09.533614 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ce5abc5-55a0-4b15-80a6-7e21d360e389-istio-token\") pod \"e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf\" (UID: \"7ce5abc5-55a0-4b15-80a6-7e21d360e389\") " pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf" Oct 31 04:43:09.549353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:09.547721 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ce5abc5-55a0-4b15-80a6-7e21d360e389-istio-podinfo\") pod \"e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf\" (UID: \"7ce5abc5-55a0-4b15-80a6-7e21d360e389\") " pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf" Oct 31 04:43:09.549353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:09.548001 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ce5abc5-55a0-4b15-80a6-7e21d360e389-istio-data\") pod \"e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf\" (UID: \"7ce5abc5-55a0-4b15-80a6-7e21d360e389\") " pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf" Oct 31 04:43:09.549353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:09.548035 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ce5abc5-55a0-4b15-80a6-7e21d360e389-istiod-ca-cert\") pod \"e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf\" (UID: \"7ce5abc5-55a0-4b15-80a6-7e21d360e389\") " pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf" Oct 31 04:43:09.549353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:09.548543 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ce5abc5-55a0-4b15-80a6-7e21d360e389-istio-envoy\") pod \"e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf\" (UID: \"7ce5abc5-55a0-4b15-80a6-7e21d360e389\") " pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf" Oct 31 04:43:09.549353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:09.548770 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce5abc5-55a0-4b15-80a6-7e21d360e389-credential-socket\") pod \"e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf\" (UID: \"7ce5abc5-55a0-4b15-80a6-7e21d360e389\") " pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf" Oct 31 04:43:09.549353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:09.549036 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ce5abc5-55a0-4b15-80a6-7e21d360e389-workload-certs\") pod \"e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf\" (UID: \"7ce5abc5-55a0-4b15-80a6-7e21d360e389\") " pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf" Oct 31 04:43:09.553353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:09.552706 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ce5abc5-55a0-4b15-80a6-7e21d360e389-workload-certs\") pod \"e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf\" (UID: \"7ce5abc5-55a0-4b15-80a6-7e21d360e389\") " pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf" Oct 31 04:43:09.553353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:09.552996 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce5abc5-55a0-4b15-80a6-7e21d360e389-credential-socket\") pod \"e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf\" (UID: \"7ce5abc5-55a0-4b15-80a6-7e21d360e389\") " pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf" Oct 31 04:43:09.553730 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:09.553701 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ce5abc5-55a0-4b15-80a6-7e21d360e389-istio-data\") pod \"e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf\" (UID: \"7ce5abc5-55a0-4b15-80a6-7e21d360e389\") " pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf" Oct 31 04:43:09.579836 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:09.578720 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ce5abc5-55a0-4b15-80a6-7e21d360e389-istiod-ca-cert\") pod \"e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf\" (UID: \"7ce5abc5-55a0-4b15-80a6-7e21d360e389\") " pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf" Oct 31 04:43:09.616991 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:09.601141 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr9q9\" (UniqueName: \"kubernetes.io/projected/7ce5abc5-55a0-4b15-80a6-7e21d360e389-kube-api-access-xr9q9\") pod \"e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf\" (UID: \"7ce5abc5-55a0-4b15-80a6-7e21d360e389\") " pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf" Oct 31 04:43:09.621810 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:09.617515 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ce5abc5-55a0-4b15-80a6-7e21d360e389-istio-envoy\") pod \"e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf\" (UID: \"7ce5abc5-55a0-4b15-80a6-7e21d360e389\") " pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf" Oct 31 04:43:09.621810 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:09.620872 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ce5abc5-55a0-4b15-80a6-7e21d360e389-istio-token\") pod \"e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf\" (UID: \"7ce5abc5-55a0-4b15-80a6-7e21d360e389\") " pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf" Oct 31 04:43:09.629356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:09.626874 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ce5abc5-55a0-4b15-80a6-7e21d360e389-istio-podinfo\") pod \"e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf\" (UID: \"7ce5abc5-55a0-4b15-80a6-7e21d360e389\") " pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf" Oct 31 04:43:09.671356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:09.670438 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf" Oct 31 04:43:10.091568 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:43:10.091525 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ce5abc5_55a0_4b15_80a6_7e21d360e389.slice/crio-c100eed766336bded3bba90fcd6304a45f691e1e1d94f41defcd0fe7e27084f1 WatchSource:0}: Error finding container c100eed766336bded3bba90fcd6304a45f691e1e1d94f41defcd0fe7e27084f1: Status 404 returned error can't find the container with id c100eed766336bded3bba90fcd6304a45f691e1e1d94f41defcd0fe7e27084f1 Oct 31 04:43:10.092081 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:10.092053 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf"] Oct 31 04:43:10.094743 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:10.094707 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:43:10.094829 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:10.094781 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:43:10.094829 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:10.094808 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:43:10.179815 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:10.179782 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf" event={"ID":"7ce5abc5-55a0-4b15-80a6-7e21d360e389","Type":"ContainerStarted","Data":"c100eed766336bded3bba90fcd6304a45f691e1e1d94f41defcd0fe7e27084f1"} Oct 31 04:43:11.190407 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:11.190365 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf" event={"ID":"7ce5abc5-55a0-4b15-80a6-7e21d360e389","Type":"ContainerStarted","Data":"0e29406cdd0f4aeca185f400307c928d1f7c7040da088704a7b1e5c45dca56ac"} Oct 31 04:43:11.248562 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:11.248499 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf" podStartSLOduration=2.248471584 podStartE2EDuration="2.248471584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:43:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:43:11.247036803 +0000 UTC m=+2346.003374846" watchObservedRunningTime="2026-10-31 04:43:11.248471584 +0000 UTC m=+2346.004809624" Oct 31 04:43:11.670842 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:11.670794 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf" Oct 31 04:43:11.677539 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:11.677506 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf" Oct 31 04:43:12.008865 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:12.008842 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b6zwwdr" Oct 31 04:43:12.095130 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:12.095085 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cf8wdr4"] Oct 31 04:43:12.220277 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:12.220241 5019 generic.go:356] "Generic (PLEG): container finished" podID="468817c7-e1ca-41c0-8a91-cfce50aa71ac" containerID="c5ffa648dd5facb6a15713079808dcd094cda1cb89634456282a04f7f27c4180" exitCode=0 Oct 31 04:43:12.221985 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:12.221637 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp" event={"ID":"468817c7-e1ca-41c0-8a91-cfce50aa71ac","Type":"ContainerDied","Data":"c5ffa648dd5facb6a15713079808dcd094cda1cb89634456282a04f7f27c4180"} Oct 31 04:43:12.221985 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:12.221708 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf" Oct 31 04:43:12.221985 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:12.221741 5019 scope.go:122] "RemoveContainer" containerID="5590a7d6cdfe15f49a0d13ca1f3456ca2da18f063702b9e3e0cc1dc83aae0e8a" Oct 31 04:43:12.223130 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:12.223107 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf" Oct 31 04:43:12.366352 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:12.366314 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp" Oct 31 04:43:12.523052 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:12.523013 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/468817c7-e1ca-41c0-8a91-cfce50aa71ac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/468817c7-e1ca-41c0-8a91-cfce50aa71ac-istio-envoy\") pod \"468817c7-e1ca-41c0-8a91-cfce50aa71ac\" (UID: \"468817c7-e1ca-41c0-8a91-cfce50aa71ac\") " Oct 31 04:43:12.523236 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:12.523084 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/468817c7-e1ca-41c0-8a91-cfce50aa71ac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/468817c7-e1ca-41c0-8a91-cfce50aa71ac-istio-podinfo\") pod \"468817c7-e1ca-41c0-8a91-cfce50aa71ac\" (UID: \"468817c7-e1ca-41c0-8a91-cfce50aa71ac\") " Oct 31 04:43:12.523236 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:12.523143 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/468817c7-e1ca-41c0-8a91-cfce50aa71ac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/468817c7-e1ca-41c0-8a91-cfce50aa71ac-workload-certs\") pod \"468817c7-e1ca-41c0-8a91-cfce50aa71ac\" (UID: \"468817c7-e1ca-41c0-8a91-cfce50aa71ac\") " Oct 31 04:43:12.523236 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:12.523203 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/468817c7-e1ca-41c0-8a91-cfce50aa71ac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/468817c7-e1ca-41c0-8a91-cfce50aa71ac-workload-socket\") pod \"468817c7-e1ca-41c0-8a91-cfce50aa71ac\" (UID: \"468817c7-e1ca-41c0-8a91-cfce50aa71ac\") " Oct 31 04:43:12.523524 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:12.523251 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/468817c7-e1ca-41c0-8a91-cfce50aa71ac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/468817c7-e1ca-41c0-8a91-cfce50aa71ac-istio-data\") pod \"468817c7-e1ca-41c0-8a91-cfce50aa71ac\" (UID: \"468817c7-e1ca-41c0-8a91-cfce50aa71ac\") " Oct 31 04:43:12.523524 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:12.523316 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/468817c7-e1ca-41c0-8a91-cfce50aa71ac-istio-token\" (UniqueName: \"kubernetes.io/projected/468817c7-e1ca-41c0-8a91-cfce50aa71ac-istio-token\") pod \"468817c7-e1ca-41c0-8a91-cfce50aa71ac\" (UID: \"468817c7-e1ca-41c0-8a91-cfce50aa71ac\") " Oct 31 04:43:12.523524 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:12.523412 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/468817c7-e1ca-41c0-8a91-cfce50aa71ac-kube-api-access-rtzth\" (UniqueName: \"kubernetes.io/projected/468817c7-e1ca-41c0-8a91-cfce50aa71ac-kube-api-access-rtzth\") pod \"468817c7-e1ca-41c0-8a91-cfce50aa71ac\" (UID: \"468817c7-e1ca-41c0-8a91-cfce50aa71ac\") " Oct 31 04:43:12.523524 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:12.523459 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/468817c7-e1ca-41c0-8a91-cfce50aa71ac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/468817c7-e1ca-41c0-8a91-cfce50aa71ac-istiod-ca-cert\") pod \"468817c7-e1ca-41c0-8a91-cfce50aa71ac\" (UID: \"468817c7-e1ca-41c0-8a91-cfce50aa71ac\") " Oct 31 04:43:12.523524 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:12.523471 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/468817c7-e1ca-41c0-8a91-cfce50aa71ac-workload-certs" pod "468817c7-e1ca-41c0-8a91-cfce50aa71ac" (UID: "468817c7-e1ca-41c0-8a91-cfce50aa71ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:43:12.523785 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:12.523525 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/468817c7-e1ca-41c0-8a91-cfce50aa71ac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/468817c7-e1ca-41c0-8a91-cfce50aa71ac-credential-socket\") pod \"468817c7-e1ca-41c0-8a91-cfce50aa71ac\" (UID: \"468817c7-e1ca-41c0-8a91-cfce50aa71ac\") " Oct 31 04:43:12.523785 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:12.523586 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/468817c7-e1ca-41c0-8a91-cfce50aa71ac-workload-socket" pod "468817c7-e1ca-41c0-8a91-cfce50aa71ac" (UID: "468817c7-e1ca-41c0-8a91-cfce50aa71ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:43:12.523913 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:12.523891 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/468817c7-e1ca-41c0-8a91-cfce50aa71ac-credential-socket" pod "468817c7-e1ca-41c0-8a91-cfce50aa71ac" (UID: "468817c7-e1ca-41c0-8a91-cfce50aa71ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:43:12.524075 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:12.523913 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/468817c7-e1ca-41c0-8a91-cfce50aa71ac-istio-data" pod "468817c7-e1ca-41c0-8a91-cfce50aa71ac" (UID: "468817c7-e1ca-41c0-8a91-cfce50aa71ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:43:12.524075 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:12.523934 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/468817c7-e1ca-41c0-8a91-cfce50aa71ac-istiod-ca-cert" pod "468817c7-e1ca-41c0-8a91-cfce50aa71ac" (UID: "468817c7-e1ca-41c0-8a91-cfce50aa71ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:43:12.524414 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:12.524394 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/468817c7-e1ca-41c0-8a91-cfce50aa71ac-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:43:12.524519 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:12.524419 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/468817c7-e1ca-41c0-8a91-cfce50aa71ac-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:43:12.524519 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:12.524435 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/468817c7-e1ca-41c0-8a91-cfce50aa71ac-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:43:12.524519 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:12.524448 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/468817c7-e1ca-41c0-8a91-cfce50aa71ac-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:43:12.524519 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:12.524462 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/468817c7-e1ca-41c0-8a91-cfce50aa71ac-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:43:12.526467 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:12.526433 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/468817c7-e1ca-41c0-8a91-cfce50aa71ac-istio-podinfo" pod "468817c7-e1ca-41c0-8a91-cfce50aa71ac" (UID: "468817c7-e1ca-41c0-8a91-cfce50aa71ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:43:12.526640 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:12.526623 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/468817c7-e1ca-41c0-8a91-cfce50aa71ac-kube-api-access-rtzth" pod "468817c7-e1ca-41c0-8a91-cfce50aa71ac" (UID: "468817c7-e1ca-41c0-8a91-cfce50aa71ac"). InnerVolumeSpecName "kube-api-access-rtzth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:43:12.526941 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:12.526917 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/468817c7-e1ca-41c0-8a91-cfce50aa71ac-istio-token" pod "468817c7-e1ca-41c0-8a91-cfce50aa71ac" (UID: "468817c7-e1ca-41c0-8a91-cfce50aa71ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:43:12.527043 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:12.527027 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/468817c7-e1ca-41c0-8a91-cfce50aa71ac-istio-envoy" pod "468817c7-e1ca-41c0-8a91-cfce50aa71ac" (UID: "468817c7-e1ca-41c0-8a91-cfce50aa71ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:43:12.626523 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:12.626423 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/468817c7-e1ca-41c0-8a91-cfce50aa71ac-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:43:12.626523 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:12.626469 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtzth\" (UniqueName: \"kubernetes.io/projected/468817c7-e1ca-41c0-8a91-cfce50aa71ac-kube-api-access-rtzth\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:43:12.626523 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:12.626484 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/468817c7-e1ca-41c0-8a91-cfce50aa71ac-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:43:12.626523 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:12.626499 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/468817c7-e1ca-41c0-8a91-cfce50aa71ac-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:43:13.060621 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:13.060586 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-79742xx4f" Oct 31 04:43:13.138363 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:13.138304 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc5675cq"] Oct 31 04:43:13.233673 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:13.233572 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp" Oct 31 04:43:13.233673 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:13.233608 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp" event={"ID":"468817c7-e1ca-41c0-8a91-cfce50aa71ac","Type":"ContainerDied","Data":"0e217789b8d0d774a4ffec3c0c9e9db926e3b2e3ecc6f1d4eb9813dcd7ff7e5c"} Oct 31 04:43:13.233673 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:13.233673 5019 scope.go:122] "RemoveContainer" containerID="c5ffa648dd5facb6a15713079808dcd094cda1cb89634456282a04f7f27c4180" Oct 31 04:43:13.296561 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:13.296522 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp"] Oct 31 04:43:13.308643 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:13.308605 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-99ca0c-openshift-default-6d8f9b55fc-4cpsp"] Oct 31 04:43:13.413154 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:13.413071 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="468817c7-e1ca-41c0-8a91-cfce50aa71ac" path="/var/lib/kubelet/pods/468817c7-e1ca-41c0-8a91-cfce50aa71ac/volumes" Oct 31 04:43:15.035147 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:15.035111 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9937f7d0-l66sk"] Oct 31 04:43:15.046953 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:15.046926 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9937f7d0-l66sk"] Oct 31 04:43:15.437437 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:15.437310 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8aaf1cba-a101-45a1-b87b-afe4e7cf00ab" path="/var/lib/kubelet/pods/8aaf1cba-a101-45a1-b87b-afe4e7cf00ab/volumes" Oct 31 04:43:15.587832 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:15.587786 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-47c409-d86754547-hbqjk"] Oct 31 04:43:15.588864 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:15.588836 5019 cpu_manager.go:414] "containerMap: removing container" podUID="468817c7-e1ca-41c0-8a91-cfce50aa71ac" containerName="istio-proxy" Oct 31 04:43:15.588864 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:15.588860 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="468817c7-e1ca-41c0-8a91-cfce50aa71ac" containerName="istio-proxy" Oct 31 04:43:15.589078 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:15.589067 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="468817c7-e1ca-41c0-8a91-cfce50aa71ac" containerName="istio-proxy" Oct 31 04:43:15.589139 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:15.589089 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="468817c7-e1ca-41c0-8a91-cfce50aa71ac" containerName="istio-proxy" Oct 31 04:43:15.597465 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:15.597430 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-47c409-d86754547-hbqjk" Oct 31 04:43:15.617601 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:15.617564 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-47c409-d86754547-hbqjk"] Oct 31 04:43:15.680358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:15.649854 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-47c409\"" Oct 31 04:43:15.725430 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:15.704637 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-47c409-dockercfg-vbmzr\"" Oct 31 04:43:15.763376 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:15.763001 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm8hq\" (UniqueName: \"kubernetes.io/projected/7d2f4246-7ce7-475e-87ba-a8afdfec5b00-kube-api-access-vm8hq\") pod \"payload-pre-processing-e2e-extmod-a-47c409-d86754547-hbqjk\" (UID: \"7d2f4246-7ce7-475e-87ba-a8afdfec5b00\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-47c409-d86754547-hbqjk" Oct 31 04:43:15.806358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:15.805876 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d2f4246-7ce7-475e-87ba-a8afdfec5b00-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-47c409-d86754547-hbqjk\" (UID: \"7d2f4246-7ce7-475e-87ba-a8afdfec5b00\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-47c409-d86754547-hbqjk" Oct 31 04:43:15.908356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:15.908219 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm8hq\" (UniqueName: \"kubernetes.io/projected/7d2f4246-7ce7-475e-87ba-a8afdfec5b00-kube-api-access-vm8hq\") pod \"payload-pre-processing-e2e-extmod-a-47c409-d86754547-hbqjk\" (UID: \"7d2f4246-7ce7-475e-87ba-a8afdfec5b00\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-47c409-d86754547-hbqjk" Oct 31 04:43:15.908577 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:15.908513 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d2f4246-7ce7-475e-87ba-a8afdfec5b00-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-47c409-d86754547-hbqjk\" (UID: \"7d2f4246-7ce7-475e-87ba-a8afdfec5b00\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-47c409-d86754547-hbqjk" Oct 31 04:43:15.928513 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:15.909650 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d2f4246-7ce7-475e-87ba-a8afdfec5b00-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-47c409-d86754547-hbqjk\" (UID: \"7d2f4246-7ce7-475e-87ba-a8afdfec5b00\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-47c409-d86754547-hbqjk" Oct 31 04:43:15.944555 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:15.944519 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm8hq\" (UniqueName: \"kubernetes.io/projected/7d2f4246-7ce7-475e-87ba-a8afdfec5b00-kube-api-access-vm8hq\") pod \"payload-pre-processing-e2e-extmod-a-47c409-d86754547-hbqjk\" (UID: \"7d2f4246-7ce7-475e-87ba-a8afdfec5b00\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-47c409-d86754547-hbqjk" Oct 31 04:43:16.062188 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:16.062076 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf5ed941-8gm7j"] Oct 31 04:43:16.068550 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:16.068514 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b29c188-jdtv2"] Oct 31 04:43:16.073306 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:16.073267 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf5ed941-8gm7j"] Oct 31 04:43:16.077833 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:16.077807 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b29c188-jdtv2"] Oct 31 04:43:16.223890 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:16.223853 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-47c409-d86754547-hbqjk" Oct 31 04:43:16.541079 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:16.541040 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-47c409-d86754547-hbqjk"] Oct 31 04:43:16.541509 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:43:16.541479 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d2f4246_7ce7_475e_87ba_a8afdfec5b00.slice/crio-7eb351c52050821d66c86f8ba8bc2ac7b5b2fbbc753bb1d461c5c25ea9b85381 WatchSource:0}: Error finding container 7eb351c52050821d66c86f8ba8bc2ac7b5b2fbbc753bb1d461c5c25ea9b85381: Status 404 returned error can't find the container with id 7eb351c52050821d66c86f8ba8bc2ac7b5b2fbbc753bb1d461c5c25ea9b85381 Oct 31 04:43:17.096914 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:17.096868 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cf8wdr4" podUID="b4a05f11-cbea-4b1f-8769-5487bbb89add" containerName="payload-processing" containerID="cri-o://792f5865dcef9ac87b3d0599cda9d1f8099db54cce2b15347d2697ef7308e272" gracePeriod=25 Oct 31 04:43:17.285608 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:17.285565 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-47c409-d86754547-hbqjk" event={"ID":"7d2f4246-7ce7-475e-87ba-a8afdfec5b00","Type":"ContainerStarted","Data":"daca217026a488d54600774e0a4804f22e83927d017cd6ed263aa6bbe7178644"} Oct 31 04:43:17.285827 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:17.285640 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-47c409-d86754547-hbqjk" event={"ID":"7d2f4246-7ce7-475e-87ba-a8afdfec5b00","Type":"ContainerStarted","Data":"7eb351c52050821d66c86f8ba8bc2ac7b5b2fbbc753bb1d461c5c25ea9b85381"} Oct 31 04:43:17.285997 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:17.285963 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-47c409-d86754547-hbqjk" Oct 31 04:43:17.288152 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:17.288117 5019 generic.go:356] "Generic (PLEG): container finished" podID="b4a05f11-cbea-4b1f-8769-5487bbb89add" containerID="792f5865dcef9ac87b3d0599cda9d1f8099db54cce2b15347d2697ef7308e272" exitCode=0 Oct 31 04:43:17.288531 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:17.288170 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cf8wdr4" event={"ID":"b4a05f11-cbea-4b1f-8769-5487bbb89add","Type":"ContainerDied","Data":"792f5865dcef9ac87b3d0599cda9d1f8099db54cce2b15347d2697ef7308e272"} Oct 31 04:43:17.332215 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:17.332150 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-47c409-d86754547-hbqjk" podStartSLOduration=2.332131842 podStartE2EDuration="2.332131842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:43:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:43:17.327728284 +0000 UTC m=+2352.084066318" watchObservedRunningTime="2026-10-31 04:43:17.332131842 +0000 UTC m=+2352.088469881" Oct 31 04:43:17.418138 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:17.418050 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f521bb4-2d60-445e-9a83-96492311abd1" path="/var/lib/kubelet/pods/4f521bb4-2d60-445e-9a83-96492311abd1/volumes" Oct 31 04:43:17.418944 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:17.418781 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d25c55c3-f558-495e-b197-e0f5231c53f2" path="/var/lib/kubelet/pods/d25c55c3-f558-495e-b197-e0f5231c53f2/volumes" Oct 31 04:43:17.500266 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:17.498363 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cf8wdr4" Oct 31 04:43:17.567759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:17.567133 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4a05f11-cbea-4b1f-8769-5487bbb89add-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b4a05f11-cbea-4b1f-8769-5487bbb89add-provider-credentials\") pod \"b4a05f11-cbea-4b1f-8769-5487bbb89add\" (UID: \"b4a05f11-cbea-4b1f-8769-5487bbb89add\") " Oct 31 04:43:17.567759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:17.567256 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4a05f11-cbea-4b1f-8769-5487bbb89add-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b4a05f11-cbea-4b1f-8769-5487bbb89add-routing-overlay-volume\") pod \"b4a05f11-cbea-4b1f-8769-5487bbb89add\" (UID: \"b4a05f11-cbea-4b1f-8769-5487bbb89add\") " Oct 31 04:43:17.567759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:17.567391 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4a05f11-cbea-4b1f-8769-5487bbb89add-kube-api-access-bw7xl\" (UniqueName: \"kubernetes.io/projected/b4a05f11-cbea-4b1f-8769-5487bbb89add-kube-api-access-bw7xl\") pod \"b4a05f11-cbea-4b1f-8769-5487bbb89add\" (UID: \"b4a05f11-cbea-4b1f-8769-5487bbb89add\") " Oct 31 04:43:17.567759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:17.567517 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4a05f11-cbea-4b1f-8769-5487bbb89add-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4a05f11-cbea-4b1f-8769-5487bbb89add-plugins-config-volume\") pod \"b4a05f11-cbea-4b1f-8769-5487bbb89add\" (UID: \"b4a05f11-cbea-4b1f-8769-5487bbb89add\") " Oct 31 04:43:17.567759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:17.567733 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4a05f11-cbea-4b1f-8769-5487bbb89add-routing-overlay-volume" pod "b4a05f11-cbea-4b1f-8769-5487bbb89add" (UID: "b4a05f11-cbea-4b1f-8769-5487bbb89add"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:43:17.568206 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:17.568180 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4a05f11-cbea-4b1f-8769-5487bbb89add-plugins-config-volume" pod "b4a05f11-cbea-4b1f-8769-5487bbb89add" (UID: "b4a05f11-cbea-4b1f-8769-5487bbb89add"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:43:17.568449 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:17.568421 5019 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b4a05f11-cbea-4b1f-8769-5487bbb89add-routing-overlay-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:43:17.568565 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:17.568450 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4a05f11-cbea-4b1f-8769-5487bbb89add-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:43:17.571412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:17.571226 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4a05f11-cbea-4b1f-8769-5487bbb89add-kube-api-access-bw7xl" pod "b4a05f11-cbea-4b1f-8769-5487bbb89add" (UID: "b4a05f11-cbea-4b1f-8769-5487bbb89add"). InnerVolumeSpecName "kube-api-access-bw7xl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:43:17.572033 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:17.571996 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4a05f11-cbea-4b1f-8769-5487bbb89add-provider-credentials" pod "b4a05f11-cbea-4b1f-8769-5487bbb89add" (UID: "b4a05f11-cbea-4b1f-8769-5487bbb89add"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:43:17.670190 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:17.670088 5019 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b4a05f11-cbea-4b1f-8769-5487bbb89add-provider-credentials\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:43:17.670190 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:17.670138 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw7xl\" (UniqueName: \"kubernetes.io/projected/b4a05f11-cbea-4b1f-8769-5487bbb89add-kube-api-access-bw7xl\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:43:18.138716 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:18.138663 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc5675cq" podUID="d841141c-29f5-48bf-9807-92267e6373ee" containerName="payload-processing" containerID="cri-o://15b88dc61ceee242b03cdbb2fc14912f05d949d9bf1c32bc6b9f08f68e214b6d" gracePeriod=25 Oct 31 04:43:18.306665 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:18.306611 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cf8wdr4" event={"ID":"b4a05f11-cbea-4b1f-8769-5487bbb89add","Type":"ContainerDied","Data":"9ec4078304889611ec94842cbc5b78d05de33dec114f149270836d11d33d7a71"} Oct 31 04:43:18.306860 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:18.306742 5019 scope.go:122] "RemoveContainer" containerID="792f5865dcef9ac87b3d0599cda9d1f8099db54cce2b15347d2697ef7308e272" Oct 31 04:43:18.306906 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:18.306636 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cf8wdr4" Oct 31 04:43:18.315503 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:18.315462 5019 generic.go:356] "Generic (PLEG): container finished" podID="d841141c-29f5-48bf-9807-92267e6373ee" containerID="15b88dc61ceee242b03cdbb2fc14912f05d949d9bf1c32bc6b9f08f68e214b6d" exitCode=0 Oct 31 04:43:18.315714 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:18.315543 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc5675cq" event={"ID":"d841141c-29f5-48bf-9807-92267e6373ee","Type":"ContainerDied","Data":"15b88dc61ceee242b03cdbb2fc14912f05d949d9bf1c32bc6b9f08f68e214b6d"} Oct 31 04:43:18.367712 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:18.367672 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cf8wdr4"] Oct 31 04:43:18.424130 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:18.424022 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-55cf8wdr4"] Oct 31 04:43:18.477734 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:18.477697 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc5675cq" Oct 31 04:43:18.597733 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:18.597694 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d841141c-29f5-48bf-9807-92267e6373ee-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d841141c-29f5-48bf-9807-92267e6373ee-provider-credentials\") pod \"d841141c-29f5-48bf-9807-92267e6373ee\" (UID: \"d841141c-29f5-48bf-9807-92267e6373ee\") " Oct 31 04:43:18.597953 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:18.597802 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d841141c-29f5-48bf-9807-92267e6373ee-kube-api-access-2jpwt\" (UniqueName: \"kubernetes.io/projected/d841141c-29f5-48bf-9807-92267e6373ee-kube-api-access-2jpwt\") pod \"d841141c-29f5-48bf-9807-92267e6373ee\" (UID: \"d841141c-29f5-48bf-9807-92267e6373ee\") " Oct 31 04:43:18.598020 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:18.598006 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d841141c-29f5-48bf-9807-92267e6373ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d841141c-29f5-48bf-9807-92267e6373ee-plugins-config-volume\") pod \"d841141c-29f5-48bf-9807-92267e6373ee\" (UID: \"d841141c-29f5-48bf-9807-92267e6373ee\") " Oct 31 04:43:18.598139 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:18.598120 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d841141c-29f5-48bf-9807-92267e6373ee-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d841141c-29f5-48bf-9807-92267e6373ee-routing-overlay-volume\") pod \"d841141c-29f5-48bf-9807-92267e6373ee\" (UID: \"d841141c-29f5-48bf-9807-92267e6373ee\") " Oct 31 04:43:18.599071 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:18.599035 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d841141c-29f5-48bf-9807-92267e6373ee-plugins-config-volume" pod "d841141c-29f5-48bf-9807-92267e6373ee" (UID: "d841141c-29f5-48bf-9807-92267e6373ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:43:18.599239 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:18.599205 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d841141c-29f5-48bf-9807-92267e6373ee-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:43:18.600595 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:18.600559 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d841141c-29f5-48bf-9807-92267e6373ee-provider-credentials" pod "d841141c-29f5-48bf-9807-92267e6373ee" (UID: "d841141c-29f5-48bf-9807-92267e6373ee"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:43:18.600700 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:18.600635 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d841141c-29f5-48bf-9807-92267e6373ee-kube-api-access-2jpwt" pod "d841141c-29f5-48bf-9807-92267e6373ee" (UID: "d841141c-29f5-48bf-9807-92267e6373ee"). InnerVolumeSpecName "kube-api-access-2jpwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:43:18.604615 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:18.604581 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d841141c-29f5-48bf-9807-92267e6373ee-routing-overlay-volume" pod "d841141c-29f5-48bf-9807-92267e6373ee" (UID: "d841141c-29f5-48bf-9807-92267e6373ee"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:43:18.700869 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:18.700764 5019 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d841141c-29f5-48bf-9807-92267e6373ee-routing-overlay-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:43:18.700869 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:18.700803 5019 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d841141c-29f5-48bf-9807-92267e6373ee-provider-credentials\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:43:18.700869 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:18.700813 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jpwt\" (UniqueName: \"kubernetes.io/projected/d841141c-29f5-48bf-9807-92267e6373ee-kube-api-access-2jpwt\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:43:19.328303 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:19.328266 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc5675cq" Oct 31 04:43:19.328303 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:19.328295 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc5675cq" event={"ID":"d841141c-29f5-48bf-9807-92267e6373ee","Type":"ContainerDied","Data":"ee640fb8ce89fb19691fb476cfb9b3d28bfd7789cf83e9623888eab56bfb6160"} Oct 31 04:43:19.328853 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:19.328373 5019 scope.go:122] "RemoveContainer" containerID="15b88dc61ceee242b03cdbb2fc14912f05d949d9bf1c32bc6b9f08f68e214b6d" Oct 31 04:43:19.382938 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:19.382894 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc5675cq"] Oct 31 04:43:19.394798 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:19.394752 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc5675cq"] Oct 31 04:43:19.415022 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:19.414980 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4a05f11-cbea-4b1f-8769-5487bbb89add" path="/var/lib/kubelet/pods/b4a05f11-cbea-4b1f-8769-5487bbb89add/volumes" Oct 31 04:43:19.415563 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:19.415535 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d841141c-29f5-48bf-9807-92267e6373ee" path="/var/lib/kubelet/pods/d841141c-29f5-48bf-9807-92267e6373ee/volumes" Oct 31 04:43:22.399764 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:43:22.399716 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:43:28.334744 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:28.334692 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-47c409-d86754547-hbqjk" Oct 31 04:43:33.780647 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:33.780600 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bntvwv"] Oct 31 04:43:33.781459 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:33.781439 5019 cpu_manager.go:414] "containerMap: removing container" podUID="d841141c-29f5-48bf-9807-92267e6373ee" containerName="payload-processing" Oct 31 04:43:33.781506 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:33.781459 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d841141c-29f5-48bf-9807-92267e6373ee" containerName="payload-processing" Oct 31 04:43:33.781506 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:33.781504 5019 cpu_manager.go:414] "containerMap: removing container" podUID="b4a05f11-cbea-4b1f-8769-5487bbb89add" containerName="payload-processing" Oct 31 04:43:33.781586 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:33.781513 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4a05f11-cbea-4b1f-8769-5487bbb89add" containerName="payload-processing" Oct 31 04:43:33.781586 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:33.781523 5019 cpu_manager.go:414] "containerMap: removing container" podUID="468817c7-e1ca-41c0-8a91-cfce50aa71ac" containerName="istio-proxy" Oct 31 04:43:33.781586 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:33.781529 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="468817c7-e1ca-41c0-8a91-cfce50aa71ac" containerName="istio-proxy" Oct 31 04:43:33.781725 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:33.781714 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="d841141c-29f5-48bf-9807-92267e6373ee" containerName="payload-processing" Oct 31 04:43:33.781758 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:33.781729 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4a05f11-cbea-4b1f-8769-5487bbb89add" containerName="payload-processing" Oct 31 04:43:33.785834 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:33.785808 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bntvwv" Oct 31 04:43:33.806306 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:33.806270 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bntvwv"] Oct 31 04:43:33.811582 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:33.811532 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-567ca6\"/\"agc-cred-4d916acdf91b\"" Oct 31 04:43:33.939921 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:33.939873 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq8r6\" (UniqueName: \"kubernetes.io/projected/8a727ed8-e1e8-4728-af11-eb6c7a57bc72-kube-api-access-nq8r6\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-866bntvwv\" (UID: \"8a727ed8-e1e8-4728-af11-eb6c7a57bc72\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bntvwv" Oct 31 04:43:33.940154 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:33.940010 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8a727ed8-e1e8-4728-af11-eb6c7a57bc72-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-866bntvwv\" (UID: \"8a727ed8-e1e8-4728-af11-eb6c7a57bc72\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bntvwv" Oct 31 04:43:33.940260 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:33.940240 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8a727ed8-e1e8-4728-af11-eb6c7a57bc72-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-866bntvwv\" (UID: \"8a727ed8-e1e8-4728-af11-eb6c7a57bc72\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bntvwv" Oct 31 04:43:33.940327 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:33.940295 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a727ed8-e1e8-4728-af11-eb6c7a57bc72-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-866bntvwv\" (UID: \"8a727ed8-e1e8-4728-af11-eb6c7a57bc72\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bntvwv" Oct 31 04:43:34.041779 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:34.041682 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq8r6\" (UniqueName: \"kubernetes.io/projected/8a727ed8-e1e8-4728-af11-eb6c7a57bc72-kube-api-access-nq8r6\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-866bntvwv\" (UID: \"8a727ed8-e1e8-4728-af11-eb6c7a57bc72\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bntvwv" Oct 31 04:43:34.041779 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:34.041752 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8a727ed8-e1e8-4728-af11-eb6c7a57bc72-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-866bntvwv\" (UID: \"8a727ed8-e1e8-4728-af11-eb6c7a57bc72\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bntvwv" Oct 31 04:43:34.042044 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:34.041881 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8a727ed8-e1e8-4728-af11-eb6c7a57bc72-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-866bntvwv\" (UID: \"8a727ed8-e1e8-4728-af11-eb6c7a57bc72\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bntvwv" Oct 31 04:43:34.042044 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:34.041904 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a727ed8-e1e8-4728-af11-eb6c7a57bc72-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-866bntvwv\" (UID: \"8a727ed8-e1e8-4728-af11-eb6c7a57bc72\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bntvwv" Oct 31 04:43:34.042740 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:34.042715 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8a727ed8-e1e8-4728-af11-eb6c7a57bc72-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-866bntvwv\" (UID: \"8a727ed8-e1e8-4728-af11-eb6c7a57bc72\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bntvwv" Oct 31 04:43:34.042740 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:34.042737 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a727ed8-e1e8-4728-af11-eb6c7a57bc72-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-866bntvwv\" (UID: \"8a727ed8-e1e8-4728-af11-eb6c7a57bc72\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bntvwv" Oct 31 04:43:34.044541 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:34.044513 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8a727ed8-e1e8-4728-af11-eb6c7a57bc72-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-866bntvwv\" (UID: \"8a727ed8-e1e8-4728-af11-eb6c7a57bc72\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bntvwv" Oct 31 04:43:34.065305 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:34.065266 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq8r6\" (UniqueName: \"kubernetes.io/projected/8a727ed8-e1e8-4728-af11-eb6c7a57bc72-kube-api-access-nq8r6\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-866bntvwv\" (UID: \"8a727ed8-e1e8-4728-af11-eb6c7a57bc72\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bntvwv" Oct 31 04:43:34.108352 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:34.108298 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bntvwv" Oct 31 04:43:34.422877 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:43:34.422826 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a727ed8_e1e8_4728_af11_eb6c7a57bc72.slice/crio-b44b8aa8c86e64579f1c766a8915273cc8e1167623a2ea6f115fdc72f7dde920 WatchSource:0}: Error finding container b44b8aa8c86e64579f1c766a8915273cc8e1167623a2ea6f115fdc72f7dde920: Status 404 returned error can't find the container with id b44b8aa8c86e64579f1c766a8915273cc8e1167623a2ea6f115fdc72f7dde920 Oct 31 04:43:34.423122 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:34.423095 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bntvwv"] Oct 31 04:43:34.660673 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:34.660633 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-b794w26n5"] Oct 31 04:43:34.665897 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:34.665863 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-b794w26n5" Oct 31 04:43:34.683060 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:34.683020 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ded435\"/\"agc-cred-43acb978be07\"" Oct 31 04:43:34.686136 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:34.686088 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bntvwv" event={"ID":"8a727ed8-e1e8-4728-af11-eb6c7a57bc72","Type":"ContainerStarted","Data":"f543bb92b1ce072d4c0ade472f07b675671481555d023cdf8e1ac105ef933851"} Oct 31 04:43:34.686301 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:34.686217 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bntvwv" event={"ID":"8a727ed8-e1e8-4728-af11-eb6c7a57bc72","Type":"ContainerStarted","Data":"b44b8aa8c86e64579f1c766a8915273cc8e1167623a2ea6f115fdc72f7dde920"} Oct 31 04:43:34.702318 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:34.702267 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-b794w26n5"] Oct 31 04:43:34.751239 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:34.751178 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bntvwv" podStartSLOduration=1.751161841 podStartE2EDuration="1.751161841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:43:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:43:34.750154207 +0000 UTC m=+2369.506492246" watchObservedRunningTime="2026-10-31 04:43:34.751161841 +0000 UTC m=+2369.507499913" Oct 31 04:43:34.753243 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:34.753203 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5255293a-4f6d-48d0-b162-d5270e2d4b52-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-b794w26n5\" (UID: \"5255293a-4f6d-48d0-b162-d5270e2d4b52\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-b794w26n5" Oct 31 04:43:34.753482 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:34.753252 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5255293a-4f6d-48d0-b162-d5270e2d4b52-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-b794w26n5\" (UID: \"5255293a-4f6d-48d0-b162-d5270e2d4b52\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-b794w26n5" Oct 31 04:43:34.753482 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:34.753441 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqrs5\" (UniqueName: \"kubernetes.io/projected/5255293a-4f6d-48d0-b162-d5270e2d4b52-kube-api-access-pqrs5\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-b794w26n5\" (UID: \"5255293a-4f6d-48d0-b162-d5270e2d4b52\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-b794w26n5" Oct 31 04:43:34.753675 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:34.753660 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5255293a-4f6d-48d0-b162-d5270e2d4b52-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-b794w26n5\" (UID: \"5255293a-4f6d-48d0-b162-d5270e2d4b52\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-b794w26n5" Oct 31 04:43:34.854835 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:34.854787 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqrs5\" (UniqueName: \"kubernetes.io/projected/5255293a-4f6d-48d0-b162-d5270e2d4b52-kube-api-access-pqrs5\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-b794w26n5\" (UID: \"5255293a-4f6d-48d0-b162-d5270e2d4b52\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-b794w26n5" Oct 31 04:43:34.855997 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:34.854994 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5255293a-4f6d-48d0-b162-d5270e2d4b52-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-b794w26n5\" (UID: \"5255293a-4f6d-48d0-b162-d5270e2d4b52\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-b794w26n5" Oct 31 04:43:34.855997 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:34.855079 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5255293a-4f6d-48d0-b162-d5270e2d4b52-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-b794w26n5\" (UID: \"5255293a-4f6d-48d0-b162-d5270e2d4b52\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-b794w26n5" Oct 31 04:43:34.855997 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:34.855106 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5255293a-4f6d-48d0-b162-d5270e2d4b52-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-b794w26n5\" (UID: \"5255293a-4f6d-48d0-b162-d5270e2d4b52\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-b794w26n5" Oct 31 04:43:34.856127 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:34.856011 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5255293a-4f6d-48d0-b162-d5270e2d4b52-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-b794w26n5\" (UID: \"5255293a-4f6d-48d0-b162-d5270e2d4b52\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-b794w26n5" Oct 31 04:43:34.856208 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:34.856186 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5255293a-4f6d-48d0-b162-d5270e2d4b52-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-b794w26n5\" (UID: \"5255293a-4f6d-48d0-b162-d5270e2d4b52\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-b794w26n5" Oct 31 04:43:34.858132 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:34.858107 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5255293a-4f6d-48d0-b162-d5270e2d4b52-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-b794w26n5\" (UID: \"5255293a-4f6d-48d0-b162-d5270e2d4b52\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-b794w26n5" Oct 31 04:43:34.878220 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:34.878184 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqrs5\" (UniqueName: \"kubernetes.io/projected/5255293a-4f6d-48d0-b162-d5270e2d4b52-kube-api-access-pqrs5\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-b794w26n5\" (UID: \"5255293a-4f6d-48d0-b162-d5270e2d4b52\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-b794w26n5" Oct 31 04:43:34.993820 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:34.993764 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-b794w26n5" Oct 31 04:43:35.321442 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:43:35.321400 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5255293a_4f6d_48d0_b162_d5270e2d4b52.slice/crio-8672862a35d73a51de524226dc3056b67421c321fc34411f3f93c3613a6813f8 WatchSource:0}: Error finding container 8672862a35d73a51de524226dc3056b67421c321fc34411f3f93c3613a6813f8: Status 404 returned error can't find the container with id 8672862a35d73a51de524226dc3056b67421c321fc34411f3f93c3613a6813f8 Oct 31 04:43:35.321600 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:35.321478 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-b794w26n5"] Oct 31 04:43:35.698581 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:35.698476 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-b794w26n5" event={"ID":"5255293a-4f6d-48d0-b162-d5270e2d4b52","Type":"ContainerStarted","Data":"b067ea72ce0b76cfff18ec7ddbbe72d75a5e003dcc75d79ff3ed1d7341864866"} Oct 31 04:43:35.698581 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:35.698552 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-b794w26n5" event={"ID":"5255293a-4f6d-48d0-b162-d5270e2d4b52","Type":"ContainerStarted","Data":"8672862a35d73a51de524226dc3056b67421c321fc34411f3f93c3613a6813f8"} Oct 31 04:43:35.698804 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:35.698628 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-b794w26n5" Oct 31 04:43:35.698899 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:35.698881 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bntvwv" Oct 31 04:43:35.741541 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:35.741481 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-b794w26n5" podStartSLOduration=1.741466388 podStartE2EDuration="1.741466388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:43:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:43:35.738984414 +0000 UTC m=+2370.495322454" watchObservedRunningTime="2026-10-31 04:43:35.741466388 +0000 UTC m=+2370.497804428" Oct 31 04:43:44.415044 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:44.414893 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-79742xx4f"] Oct 31 04:43:46.716367 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:46.716266 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-b794w26n5" Oct 31 04:43:46.717088 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:46.717063 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bntvwv" Oct 31 04:43:46.815624 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:46.815584 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b6zwwdr"] Oct 31 04:43:46.840890 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:46.840847 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8m5q9c"] Oct 31 04:43:49.415990 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:49.415917 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-79742xx4f" podUID="835f4c9d-588d-4fe1-8ec3-d916abc72af1" containerName="payload-processing" containerID="cri-o://53cd8f3025f102574acaf5e5b280da8c6222f5d2619c3f9a33ba43f6a873f56a" gracePeriod=25 Oct 31 04:43:49.785687 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:49.785644 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-79742xx4f" Oct 31 04:43:49.792657 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:49.792615 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/835f4c9d-588d-4fe1-8ec3-d916abc72af1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/835f4c9d-588d-4fe1-8ec3-d916abc72af1-routing-overlay-volume\") pod \"835f4c9d-588d-4fe1-8ec3-d916abc72af1\" (UID: \"835f4c9d-588d-4fe1-8ec3-d916abc72af1\") " Oct 31 04:43:49.792885 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:49.792685 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/835f4c9d-588d-4fe1-8ec3-d916abc72af1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/835f4c9d-588d-4fe1-8ec3-d916abc72af1-plugins-config-volume\") pod \"835f4c9d-588d-4fe1-8ec3-d916abc72af1\" (UID: \"835f4c9d-588d-4fe1-8ec3-d916abc72af1\") " Oct 31 04:43:49.792885 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:49.792840 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/835f4c9d-588d-4fe1-8ec3-d916abc72af1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/835f4c9d-588d-4fe1-8ec3-d916abc72af1-provider-credentials\") pod \"835f4c9d-588d-4fe1-8ec3-d916abc72af1\" (UID: \"835f4c9d-588d-4fe1-8ec3-d916abc72af1\") " Oct 31 04:43:49.793029 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:49.792968 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/835f4c9d-588d-4fe1-8ec3-d916abc72af1-kube-api-access-m5jrb\" (UniqueName: \"kubernetes.io/projected/835f4c9d-588d-4fe1-8ec3-d916abc72af1-kube-api-access-m5jrb\") pod \"835f4c9d-588d-4fe1-8ec3-d916abc72af1\" (UID: \"835f4c9d-588d-4fe1-8ec3-d916abc72af1\") " Oct 31 04:43:49.793125 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:49.793093 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/835f4c9d-588d-4fe1-8ec3-d916abc72af1-routing-overlay-volume" pod "835f4c9d-588d-4fe1-8ec3-d916abc72af1" (UID: "835f4c9d-588d-4fe1-8ec3-d916abc72af1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:43:49.793596 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:49.793550 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/835f4c9d-588d-4fe1-8ec3-d916abc72af1-plugins-config-volume" pod "835f4c9d-588d-4fe1-8ec3-d916abc72af1" (UID: "835f4c9d-588d-4fe1-8ec3-d916abc72af1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:43:49.793779 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:49.793756 5019 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/835f4c9d-588d-4fe1-8ec3-d916abc72af1-routing-overlay-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:43:49.793779 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:49.793779 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/835f4c9d-588d-4fe1-8ec3-d916abc72af1-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:43:49.795814 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:49.795779 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/835f4c9d-588d-4fe1-8ec3-d916abc72af1-provider-credentials" pod "835f4c9d-588d-4fe1-8ec3-d916abc72af1" (UID: "835f4c9d-588d-4fe1-8ec3-d916abc72af1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:43:49.796290 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:49.796265 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/835f4c9d-588d-4fe1-8ec3-d916abc72af1-kube-api-access-m5jrb" pod "835f4c9d-588d-4fe1-8ec3-d916abc72af1" (UID: "835f4c9d-588d-4fe1-8ec3-d916abc72af1"). InnerVolumeSpecName "kube-api-access-m5jrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:43:49.896893 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:49.896843 5019 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/835f4c9d-588d-4fe1-8ec3-d916abc72af1-provider-credentials\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:43:49.896893 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:49.896896 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5jrb\" (UniqueName: \"kubernetes.io/projected/835f4c9d-588d-4fe1-8ec3-d916abc72af1-kube-api-access-m5jrb\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:43:49.956302 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:49.956261 5019 generic.go:356] "Generic (PLEG): container finished" podID="835f4c9d-588d-4fe1-8ec3-d916abc72af1" containerID="53cd8f3025f102574acaf5e5b280da8c6222f5d2619c3f9a33ba43f6a873f56a" exitCode=0 Oct 31 04:43:49.956590 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:49.956319 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-79742xx4f" Oct 31 04:43:49.956590 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:49.956411 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-79742xx4f" event={"ID":"835f4c9d-588d-4fe1-8ec3-d916abc72af1","Type":"ContainerDied","Data":"53cd8f3025f102574acaf5e5b280da8c6222f5d2619c3f9a33ba43f6a873f56a"} Oct 31 04:43:49.956590 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:49.956480 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-79742xx4f" event={"ID":"835f4c9d-588d-4fe1-8ec3-d916abc72af1","Type":"ContainerDied","Data":"84d62f5d32a77194f11d99a9f2b3eec75b75744842c981aeb5d6d8519784251a"} Oct 31 04:43:49.956590 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:49.956513 5019 scope.go:122] "RemoveContainer" containerID="53cd8f3025f102574acaf5e5b280da8c6222f5d2619c3f9a33ba43f6a873f56a" Oct 31 04:43:49.972321 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:49.972294 5019 scope.go:122] "RemoveContainer" containerID="53cd8f3025f102574acaf5e5b280da8c6222f5d2619c3f9a33ba43f6a873f56a" Oct 31 04:43:49.972829 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:43:49.972805 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53cd8f3025f102574acaf5e5b280da8c6222f5d2619c3f9a33ba43f6a873f56a\": container with ID starting with 53cd8f3025f102574acaf5e5b280da8c6222f5d2619c3f9a33ba43f6a873f56a not found: ID does not exist" containerID="53cd8f3025f102574acaf5e5b280da8c6222f5d2619c3f9a33ba43f6a873f56a" Oct 31 04:43:49.972908 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:49.972840 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53cd8f3025f102574acaf5e5b280da8c6222f5d2619c3f9a33ba43f6a873f56a"} err="failed to get container status \"53cd8f3025f102574acaf5e5b280da8c6222f5d2619c3f9a33ba43f6a873f56a\": rpc error: code = NotFound desc = could not find container \"53cd8f3025f102574acaf5e5b280da8c6222f5d2619c3f9a33ba43f6a873f56a\": container with ID starting with 53cd8f3025f102574acaf5e5b280da8c6222f5d2619c3f9a33ba43f6a873f56a not found: ID does not exist" Oct 31 04:43:50.016603 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:50.016507 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-79742xx4f"] Oct 31 04:43:50.024654 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:50.024612 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-79742xx4f"] Oct 31 04:43:51.420004 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:51.419951 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="835f4c9d-588d-4fe1-8ec3-d916abc72af1" path="/var/lib/kubelet/pods/835f4c9d-588d-4fe1-8ec3-d916abc72af1/volumes" Oct 31 04:43:51.816984 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:51.816925 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b6zwwdr" podUID="c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88" containerName="payload-processing" containerID="cri-o://40f55460b5abe5128bdb5da2ae93bdc891024ff271aad9c7c381e855430bf021" gracePeriod=25 Oct 31 04:43:51.841292 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:51.841241 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8m5q9c" podUID="54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f" containerName="payload-processing" containerID="cri-o://7db5592b443ab1cccb9f109b4ce310ec6829b30d5a58308ec74e419eab88c177" gracePeriod=25 Oct 31 04:43:51.985322 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:51.985110 5019 generic.go:356] "Generic (PLEG): container finished" podID="54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f" containerID="7db5592b443ab1cccb9f109b4ce310ec6829b30d5a58308ec74e419eab88c177" exitCode=0 Oct 31 04:43:51.985322 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:51.985235 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8m5q9c" event={"ID":"54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f","Type":"ContainerDied","Data":"7db5592b443ab1cccb9f109b4ce310ec6829b30d5a58308ec74e419eab88c177"} Oct 31 04:43:51.988998 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:51.988956 5019 generic.go:356] "Generic (PLEG): container finished" podID="39eb6eb3-a2ae-4931-bab8-1e3e07e7d764" containerID="ceaa0513f8cbeb4411b336dc868532a62efa55a1e389ef109e97c3eec5ade4ba" exitCode=0 Oct 31 04:43:51.989203 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:51.989055 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4" event={"ID":"39eb6eb3-a2ae-4931-bab8-1e3e07e7d764","Type":"ContainerDied","Data":"ceaa0513f8cbeb4411b336dc868532a62efa55a1e389ef109e97c3eec5ade4ba"} Oct 31 04:43:51.989651 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:51.989623 5019 scope.go:122] "RemoveContainer" containerID="ceaa0513f8cbeb4411b336dc868532a62efa55a1e389ef109e97c3eec5ade4ba" Oct 31 04:43:51.994163 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:51.992380 5019 generic.go:356] "Generic (PLEG): container finished" podID="c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88" containerID="40f55460b5abe5128bdb5da2ae93bdc891024ff271aad9c7c381e855430bf021" exitCode=0 Oct 31 04:43:51.994163 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:51.992435 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b6zwwdr" event={"ID":"c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88","Type":"ContainerDied","Data":"40f55460b5abe5128bdb5da2ae93bdc891024ff271aad9c7c381e855430bf021"} Oct 31 04:43:51.994163 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:51.993437 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:43:51.994163 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:51.993604 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:43:51.994163 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:51.993661 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:43:52.006234 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:43:52.005967 5019 logging.go:55] [core] [Channel #5828 SubChannel #5829]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.1.33:50052", ServerName: "10.132.1.33:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.1.33:50052: connect: connection refused" Oct 31 04:43:52.227604 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:52.227250 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b6zwwdr" Oct 31 04:43:52.279623 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:52.276284 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8m5q9c" Oct 31 04:43:52.352771 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:52.352652 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88-plugins-config-volume\") pod \"c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88\" (UID: \"c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88\") " Oct 31 04:43:52.352771 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:52.352740 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f-plugins-config-volume\") pod \"54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f\" (UID: \"54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f\") " Oct 31 04:43:52.353053 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:52.352809 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88-provider-credentials\") pod \"c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88\" (UID: \"c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88\") " Oct 31 04:43:52.353053 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:52.352945 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f-routing-overlay-volume\") pod \"54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f\" (UID: \"54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f\") " Oct 31 04:43:52.353173 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:52.353072 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88-routing-overlay-volume\") pod \"c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88\" (UID: \"c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88\") " Oct 31 04:43:52.353248 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:52.353229 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88-kube-api-access-bjfxb\" (UniqueName: \"kubernetes.io/projected/c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88-kube-api-access-bjfxb\") pod \"c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88\" (UID: \"c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88\") " Oct 31 04:43:52.353323 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:52.353307 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f-kube-api-access-mpjzg\" (UniqueName: \"kubernetes.io/projected/54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f-kube-api-access-mpjzg\") pod \"54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f\" (UID: \"54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f\") " Oct 31 04:43:52.353507 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:52.353364 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f-provider-credentials\") pod \"54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f\" (UID: \"54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f\") " Oct 31 04:43:52.354291 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:52.354251 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88-plugins-config-volume" pod "c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88" (UID: "c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:43:52.354936 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:52.354889 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f-plugins-config-volume" pod "54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f" (UID: "54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:43:52.355154 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:52.355122 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88-routing-overlay-volume" pod "c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88" (UID: "c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:43:52.355641 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:52.355620 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f-routing-overlay-volume" pod "54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f" (UID: "54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:43:52.357439 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:52.356014 5019 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88-routing-overlay-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:43:52.357439 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:52.356042 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:43:52.357439 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:52.356059 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:43:52.357439 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:52.356075 5019 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f-routing-overlay-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:43:52.357439 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:52.357379 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88-provider-credentials" pod "c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88" (UID: "c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:43:52.357690 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:52.357525 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f-kube-api-access-mpjzg" pod "54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f" (UID: "54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f"). InnerVolumeSpecName "kube-api-access-mpjzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:43:52.357690 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:52.357650 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f-provider-credentials" pod "54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f" (UID: "54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:43:52.360359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:52.359154 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88-kube-api-access-bjfxb" pod "c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88" (UID: "c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88"). InnerVolumeSpecName "kube-api-access-bjfxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:43:52.413508 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:43:52.413438 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:43:52.458128 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:52.458092 5019 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88-provider-credentials\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:43:52.458128 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:52.458123 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjfxb\" (UniqueName: \"kubernetes.io/projected/c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88-kube-api-access-bjfxb\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:43:52.458128 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:52.458133 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpjzg\" (UniqueName: \"kubernetes.io/projected/54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f-kube-api-access-mpjzg\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:43:52.458650 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:52.458142 5019 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f-provider-credentials\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:43:53.003694 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:53.003658 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b6zwwdr" Oct 31 04:43:53.003931 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:53.003686 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b6zwwdr" event={"ID":"c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88","Type":"ContainerDied","Data":"3cdb5bc6070ff7aac6407f19ea2e2f7944dd816d8ffbf2b78723361c0d7e888b"} Oct 31 04:43:53.003931 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:53.003773 5019 scope.go:122] "RemoveContainer" containerID="40f55460b5abe5128bdb5da2ae93bdc891024ff271aad9c7c381e855430bf021" Oct 31 04:43:53.005556 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:53.005521 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b6zwwdr" podUID="c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.1.33:50052\" within 1s: context deadline exceeded" Oct 31 04:43:53.006122 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:53.006094 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8m5q9c" event={"ID":"54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f","Type":"ContainerDied","Data":"7d0d002f5c97c114add1689b1200e7ebd34fb7bf6f965873f6433540e6d67812"} Oct 31 04:43:53.006251 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:53.006217 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8m5q9c" Oct 31 04:43:53.020039 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:53.020018 5019 scope.go:122] "RemoveContainer" containerID="7db5592b443ab1cccb9f109b4ce310ec6829b30d5a58308ec74e419eab88c177" Oct 31 04:43:53.021273 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:53.021018 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4" event={"ID":"39eb6eb3-a2ae-4931-bab8-1e3e07e7d764","Type":"ContainerStarted","Data":"0f63c21083717c83ce5abebb963eb427f75e6bb26fe840ae77bcc21b23cfb4cb"} Oct 31 04:43:53.075296 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:53.075258 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b6zwwdr"] Oct 31 04:43:53.083027 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:53.082980 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b6zwwdr"] Oct 31 04:43:53.159247 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:53.159151 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8m5q9c"] Oct 31 04:43:53.163781 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:53.163753 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8m5q9c"] Oct 31 04:43:53.417815 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:53.417723 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f" path="/var/lib/kubelet/pods/54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f/volumes" Oct 31 04:43:53.418476 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:53.418456 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88" path="/var/lib/kubelet/pods/c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88/volumes" Oct 31 04:43:53.711669 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:53.711567 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4" Oct 31 04:43:53.713073 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:53.713038 5019 patch_prober.go:29] interesting pod/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body= Oct 31 04:43:53.713247 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:53.713105 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4" podUID="39eb6eb3-a2ae-4931-bab8-1e3e07e7d764" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" Oct 31 04:43:54.711942 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:54.711901 5019 patch_prober.go:29] interesting pod/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body= Oct 31 04:43:54.712404 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:54.711963 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4" podUID="39eb6eb3-a2ae-4931-bab8-1e3e07e7d764" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" Oct 31 04:43:55.030816 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:55.030771 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4" Oct 31 04:43:55.715986 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:55.715929 5019 patch_prober.go:29] interesting pod/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body= Oct 31 04:43:55.716703 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:55.716652 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4" podUID="39eb6eb3-a2ae-4931-bab8-1e3e07e7d764" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" Oct 31 04:43:56.711627 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:56.711517 5019 patch_prober.go:29] interesting pod/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body= Oct 31 04:43:56.711627 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:56.711601 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4" podUID="39eb6eb3-a2ae-4931-bab8-1e3e07e7d764" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" Oct 31 04:43:57.711410 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:57.711306 5019 patch_prober.go:29] interesting pod/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body= Oct 31 04:43:57.711882 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:57.711443 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4" podUID="39eb6eb3-a2ae-4931-bab8-1e3e07e7d764" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" Oct 31 04:43:58.711173 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:58.711134 5019 patch_prober.go:29] interesting pod/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body= Oct 31 04:43:58.711475 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:58.711199 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4" podUID="39eb6eb3-a2ae-4931-bab8-1e3e07e7d764" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" Oct 31 04:43:59.718054 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:59.718014 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4" Oct 31 04:43:59.719433 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:43:59.719405 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4" Oct 31 04:44:08.142010 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:08.141976 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s88kp_3945ff65-e877-4372-9334-3968c68b7295/console-operator/2.log" Oct 31 04:44:08.157728 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:08.157695 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s88kp_3945ff65-e877-4372-9334-3968c68b7295/console-operator/2.log" Oct 31 04:44:08.873230 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:08.873190 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-5d44jzg8t"] Oct 31 04:44:08.874048 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:08.874028 5019 cpu_manager.go:414] "containerMap: removing container" podUID="c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88" containerName="payload-processing" Oct 31 04:44:08.874048 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:08.874047 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88" containerName="payload-processing" Oct 31 04:44:08.874205 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:08.874060 5019 cpu_manager.go:414] "containerMap: removing container" podUID="54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f" containerName="payload-processing" Oct 31 04:44:08.874205 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:08.874066 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f" containerName="payload-processing" Oct 31 04:44:08.874205 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:08.874073 5019 cpu_manager.go:414] "containerMap: removing container" podUID="835f4c9d-588d-4fe1-8ec3-d916abc72af1" containerName="payload-processing" Oct 31 04:44:08.874205 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:08.874078 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="835f4c9d-588d-4fe1-8ec3-d916abc72af1" containerName="payload-processing" Oct 31 04:44:08.874472 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:08.874226 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5d8a723-1bca-463c-9d7b-3a2e7b9d2a88" containerName="payload-processing" Oct 31 04:44:08.874472 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:08.874233 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="54ad73b6-7fb3-43e3-a05c-67fe8b67ce8f" containerName="payload-processing" Oct 31 04:44:08.874472 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:08.874240 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="835f4c9d-588d-4fe1-8ec3-d916abc72af1" containerName="payload-processing" Oct 31 04:44:08.878659 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:08.878627 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-5d44jzg8t" Oct 31 04:44:08.900572 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:08.900527 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-5d44jzg8t"] Oct 31 04:44:08.903129 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:08.903092 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-567ca6\"/\"agc-cred-876b469dc150\"" Oct 31 04:44:08.949786 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:08.949746 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/096d1018-1c07-4bce-b449-08c34c8457fb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-5d44jzg8t\" (UID: \"096d1018-1c07-4bce-b449-08c34c8457fb\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-5d44jzg8t" Oct 31 04:44:08.949786 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:08.949798 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/096d1018-1c07-4bce-b449-08c34c8457fb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-5d44jzg8t\" (UID: \"096d1018-1c07-4bce-b449-08c34c8457fb\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-5d44jzg8t" Oct 31 04:44:08.950125 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:08.950089 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/096d1018-1c07-4bce-b449-08c34c8457fb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-5d44jzg8t\" (UID: \"096d1018-1c07-4bce-b449-08c34c8457fb\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-5d44jzg8t" Oct 31 04:44:08.950316 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:08.950298 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2nhj\" (UniqueName: \"kubernetes.io/projected/096d1018-1c07-4bce-b449-08c34c8457fb-kube-api-access-b2nhj\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-5d44jzg8t\" (UID: \"096d1018-1c07-4bce-b449-08c34c8457fb\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-5d44jzg8t" Oct 31 04:44:09.051398 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:09.051349 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2nhj\" (UniqueName: \"kubernetes.io/projected/096d1018-1c07-4bce-b449-08c34c8457fb-kube-api-access-b2nhj\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-5d44jzg8t\" (UID: \"096d1018-1c07-4bce-b449-08c34c8457fb\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-5d44jzg8t" Oct 31 04:44:09.051606 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:09.051470 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/096d1018-1c07-4bce-b449-08c34c8457fb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-5d44jzg8t\" (UID: \"096d1018-1c07-4bce-b449-08c34c8457fb\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-5d44jzg8t" Oct 31 04:44:09.051606 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:09.051499 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/096d1018-1c07-4bce-b449-08c34c8457fb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-5d44jzg8t\" (UID: \"096d1018-1c07-4bce-b449-08c34c8457fb\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-5d44jzg8t" Oct 31 04:44:09.051734 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:09.051620 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/096d1018-1c07-4bce-b449-08c34c8457fb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-5d44jzg8t\" (UID: \"096d1018-1c07-4bce-b449-08c34c8457fb\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-5d44jzg8t" Oct 31 04:44:09.052262 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:09.052234 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/096d1018-1c07-4bce-b449-08c34c8457fb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-5d44jzg8t\" (UID: \"096d1018-1c07-4bce-b449-08c34c8457fb\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-5d44jzg8t" Oct 31 04:44:09.052402 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:09.052370 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/096d1018-1c07-4bce-b449-08c34c8457fb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-5d44jzg8t\" (UID: \"096d1018-1c07-4bce-b449-08c34c8457fb\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-5d44jzg8t" Oct 31 04:44:09.054493 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:09.054467 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/096d1018-1c07-4bce-b449-08c34c8457fb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-5d44jzg8t\" (UID: \"096d1018-1c07-4bce-b449-08c34c8457fb\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-5d44jzg8t" Oct 31 04:44:09.075284 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:09.075244 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2nhj\" (UniqueName: \"kubernetes.io/projected/096d1018-1c07-4bce-b449-08c34c8457fb-kube-api-access-b2nhj\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-5d44jzg8t\" (UID: \"096d1018-1c07-4bce-b449-08c34c8457fb\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-5d44jzg8t" Oct 31 04:44:09.199770 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:09.199663 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-5d44jzg8t" Oct 31 04:44:09.576198 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:09.576155 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-5d44jzg8t"] Oct 31 04:44:09.581054 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:44:09.581009 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod096d1018_1c07_4bce_b449_08c34c8457fb.slice/crio-204cf78b7f01406ff21ffb223d788bfa437752d1ad04e05d3850e036dd177004 WatchSource:0}: Error finding container 204cf78b7f01406ff21ffb223d788bfa437752d1ad04e05d3850e036dd177004: Status 404 returned error can't find the container with id 204cf78b7f01406ff21ffb223d788bfa437752d1ad04e05d3850e036dd177004 Oct 31 04:44:10.050916 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:10.050707 5019 scope.go:122] "RemoveContainer" containerID="7f981f358b1931cfb8c57126da1d63beeca57a7a92de0f93e60529dab68ffc7a" Oct 31 04:44:10.087746 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:10.087717 5019 scope.go:122] "RemoveContainer" containerID="a6cf2ea41017d37df9669b3578bb2ffbe50ec9e51033132fc582dcd5bcebbced" Oct 31 04:44:10.100856 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:10.100826 5019 scope.go:122] "RemoveContainer" containerID="bdc8427a1673f544911c23bb6f2ae6b4153a78963c18c1ce417d3e93ae48c65e" Oct 31 04:44:10.102210 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:10.102184 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6c59wxpxq"] Oct 31 04:44:10.108934 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:10.108910 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6c59wxpxq" Oct 31 04:44:10.123713 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:10.123669 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-401761\"/\"agc-cred-c5bd7c69632c\"" Oct 31 04:44:10.124623 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:10.124527 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-401761\"/\"openshift-service-ca.crt\"" Oct 31 04:44:10.124623 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:10.124580 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-401761\"/\"kube-root-ca.crt\"" Oct 31 04:44:10.124830 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:10.124658 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-401761\"/\"routing-overlay\"" Oct 31 04:44:10.124830 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:10.124733 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-401761\"/\"payload-processing-external-model-e2e--201094b0-dockercfg-2l2zq\"" Oct 31 04:44:10.125087 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:10.125061 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-401761\"/\"payload-processing-external-model-plugins-e2e-extmod-a-401761\"" Oct 31 04:44:10.129358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:10.129306 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6c59wxpxq"] Oct 31 04:44:10.180738 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:10.180698 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4310f02c-7bfd-4e01-acaa-67b2357988bb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-401761-6c59wxpxq\" (UID: \"4310f02c-7bfd-4e01-acaa-67b2357988bb\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6c59wxpxq" Oct 31 04:44:10.180881 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:10.180775 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4310f02c-7bfd-4e01-acaa-67b2357988bb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-401761-6c59wxpxq\" (UID: \"4310f02c-7bfd-4e01-acaa-67b2357988bb\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6c59wxpxq" Oct 31 04:44:10.181149 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:10.181127 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4310f02c-7bfd-4e01-acaa-67b2357988bb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-401761-6c59wxpxq\" (UID: \"4310f02c-7bfd-4e01-acaa-67b2357988bb\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6c59wxpxq" Oct 31 04:44:10.181235 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:10.181179 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t98q\" (UniqueName: \"kubernetes.io/projected/4310f02c-7bfd-4e01-acaa-67b2357988bb-kube-api-access-2t98q\") pod \"payload-processing-external-model-e2e-extmod-a-401761-6c59wxpxq\" (UID: \"4310f02c-7bfd-4e01-acaa-67b2357988bb\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6c59wxpxq" Oct 31 04:44:10.303417 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:10.302645 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4310f02c-7bfd-4e01-acaa-67b2357988bb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-401761-6c59wxpxq\" (UID: \"4310f02c-7bfd-4e01-acaa-67b2357988bb\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6c59wxpxq" Oct 31 04:44:10.303417 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:10.302721 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t98q\" (UniqueName: \"kubernetes.io/projected/4310f02c-7bfd-4e01-acaa-67b2357988bb-kube-api-access-2t98q\") pod \"payload-processing-external-model-e2e-extmod-a-401761-6c59wxpxq\" (UID: \"4310f02c-7bfd-4e01-acaa-67b2357988bb\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6c59wxpxq" Oct 31 04:44:10.303417 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:10.302807 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4310f02c-7bfd-4e01-acaa-67b2357988bb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-401761-6c59wxpxq\" (UID: \"4310f02c-7bfd-4e01-acaa-67b2357988bb\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6c59wxpxq" Oct 31 04:44:10.303417 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:10.302858 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4310f02c-7bfd-4e01-acaa-67b2357988bb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-401761-6c59wxpxq\" (UID: \"4310f02c-7bfd-4e01-acaa-67b2357988bb\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6c59wxpxq" Oct 31 04:44:10.304182 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:10.304148 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4310f02c-7bfd-4e01-acaa-67b2357988bb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-401761-6c59wxpxq\" (UID: \"4310f02c-7bfd-4e01-acaa-67b2357988bb\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6c59wxpxq" Oct 31 04:44:10.343386 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:10.323885 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4310f02c-7bfd-4e01-acaa-67b2357988bb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-401761-6c59wxpxq\" (UID: \"4310f02c-7bfd-4e01-acaa-67b2357988bb\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6c59wxpxq" Oct 31 04:44:10.375357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:10.366951 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4310f02c-7bfd-4e01-acaa-67b2357988bb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-401761-6c59wxpxq\" (UID: \"4310f02c-7bfd-4e01-acaa-67b2357988bb\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6c59wxpxq" Oct 31 04:44:10.437358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:10.437085 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t98q\" (UniqueName: \"kubernetes.io/projected/4310f02c-7bfd-4e01-acaa-67b2357988bb-kube-api-access-2t98q\") pod \"payload-processing-external-model-e2e-extmod-a-401761-6c59wxpxq\" (UID: \"4310f02c-7bfd-4e01-acaa-67b2357988bb\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6c59wxpxq" Oct 31 04:44:10.477360 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:10.477073 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-5d44jzg8t" event={"ID":"096d1018-1c07-4bce-b449-08c34c8457fb","Type":"ContainerStarted","Data":"41d8db62847648e1c16a07ba0ae18c67b5614ffbb99804183b5378c6e0c6304f"} Oct 31 04:44:10.477360 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:10.477146 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-5d44jzg8t" event={"ID":"096d1018-1c07-4bce-b449-08c34c8457fb","Type":"ContainerStarted","Data":"204cf78b7f01406ff21ffb223d788bfa437752d1ad04e05d3850e036dd177004"} Oct 31 04:44:10.497388 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:10.495463 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-5d44jzg8t" Oct 31 04:44:10.545297 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:10.524008 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-5d44jzg8t" podStartSLOduration=2.523990948 podStartE2EDuration="2.523990948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:44:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:44:10.521830723 +0000 UTC m=+2405.278168763" watchObservedRunningTime="2026-10-31 04:44:10.523990948 +0000 UTC m=+2405.280329046" Oct 31 04:44:10.731569 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:10.731526 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6c59wxpxq" Oct 31 04:44:11.052836 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:11.052799 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6c59wxpxq"] Oct 31 04:44:11.052998 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:44:11.052836 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4310f02c_7bfd_4e01_acaa_67b2357988bb.slice/crio-f9c5975f934746d692d6d096870c6cb4e40d2a6309517b38628e25819bfd8b24 WatchSource:0}: Error finding container f9c5975f934746d692d6d096870c6cb4e40d2a6309517b38628e25819bfd8b24: Status 404 returned error can't find the container with id f9c5975f934746d692d6d096870c6cb4e40d2a6309517b38628e25819bfd8b24 Oct 31 04:44:11.492944 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:11.492902 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6c59wxpxq" event={"ID":"4310f02c-7bfd-4e01-acaa-67b2357988bb","Type":"ContainerStarted","Data":"9df4fc9d03f2da1a6e95c8dc120f4d8ce22889991e9dbb196bcd271913982deb"} Oct 31 04:44:11.493526 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:11.492970 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6c59wxpxq" event={"ID":"4310f02c-7bfd-4e01-acaa-67b2357988bb","Type":"ContainerStarted","Data":"f9c5975f934746d692d6d096870c6cb4e40d2a6309517b38628e25819bfd8b24"} Oct 31 04:44:11.535270 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:11.535202 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6c59wxpxq" podStartSLOduration=1.535186052 podStartE2EDuration="1.535186052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:44:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:44:11.533147439 +0000 UTC m=+2406.289485481" watchObservedRunningTime="2026-10-31 04:44:11.535186052 +0000 UTC m=+2406.291524092" Oct 31 04:44:12.030602 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:12.030561 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7"] Oct 31 04:44:12.030833 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:12.030765 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7" podUID="5581a822-03f0-4f8c-bd5e-444274a4bd9b" containerName="istio-proxy" containerID="cri-o://af4ee7a37af498ec39a99b06bccebfb4238df4bba896e77312020e27b4555f17" gracePeriod=30 Oct 31 04:44:12.502574 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:12.502522 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6c59wxpxq" Oct 31 04:44:15.893043 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:15.892999 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fkmdlx"] Oct 31 04:44:15.901161 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:15.901124 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fkmdlx" Oct 31 04:44:15.919876 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:15.919530 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fkmdlx"] Oct 31 04:44:15.920142 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:15.920064 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8973e2\"/\"payload-processing-external-model-plugins-e2e-extmod-a-8973e2\"" Oct 31 04:44:15.928768 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:15.928734 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8973e2\"/\"routing-overlay\"" Oct 31 04:44:15.929172 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:15.929108 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8973e2\"/\"kube-root-ca.crt\"" Oct 31 04:44:15.933299 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:15.933266 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8973e2\"/\"agc-cred-cc70fa128d1a\"" Oct 31 04:44:15.933508 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:15.933368 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8973e2\"/\"payload-processing-external-model-e2e--508e01fb-dockercfg-f7kv4\"" Oct 31 04:44:15.933668 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:15.933648 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8973e2\"/\"openshift-service-ca.crt\"" Oct 31 04:44:16.076363 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:16.068143 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5e889be1-d856-4ffc-86b9-f9fa9da208c0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-7b8fkmdlx\" (UID: \"5e889be1-d856-4ffc-86b9-f9fa9da208c0\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fkmdlx" Oct 31 04:44:16.076363 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:16.068277 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e889be1-d856-4ffc-86b9-f9fa9da208c0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-7b8fkmdlx\" (UID: \"5e889be1-d856-4ffc-86b9-f9fa9da208c0\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fkmdlx" Oct 31 04:44:16.076363 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:16.068308 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5e889be1-d856-4ffc-86b9-f9fa9da208c0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-7b8fkmdlx\" (UID: \"5e889be1-d856-4ffc-86b9-f9fa9da208c0\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fkmdlx" Oct 31 04:44:16.076363 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:16.068527 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2hrg\" (UniqueName: \"kubernetes.io/projected/5e889be1-d856-4ffc-86b9-f9fa9da208c0-kube-api-access-n2hrg\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-7b8fkmdlx\" (UID: \"5e889be1-d856-4ffc-86b9-f9fa9da208c0\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fkmdlx" Oct 31 04:44:16.219874 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:16.181029 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5e889be1-d856-4ffc-86b9-f9fa9da208c0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-7b8fkmdlx\" (UID: \"5e889be1-d856-4ffc-86b9-f9fa9da208c0\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fkmdlx" Oct 31 04:44:16.219874 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:16.181149 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e889be1-d856-4ffc-86b9-f9fa9da208c0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-7b8fkmdlx\" (UID: \"5e889be1-d856-4ffc-86b9-f9fa9da208c0\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fkmdlx" Oct 31 04:44:16.219874 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:16.181179 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5e889be1-d856-4ffc-86b9-f9fa9da208c0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-7b8fkmdlx\" (UID: \"5e889be1-d856-4ffc-86b9-f9fa9da208c0\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fkmdlx" Oct 31 04:44:16.219874 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:16.181360 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2hrg\" (UniqueName: \"kubernetes.io/projected/5e889be1-d856-4ffc-86b9-f9fa9da208c0-kube-api-access-n2hrg\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-7b8fkmdlx\" (UID: \"5e889be1-d856-4ffc-86b9-f9fa9da208c0\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fkmdlx" Oct 31 04:44:16.219874 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:16.182988 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5e889be1-d856-4ffc-86b9-f9fa9da208c0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-7b8fkmdlx\" (UID: \"5e889be1-d856-4ffc-86b9-f9fa9da208c0\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fkmdlx" Oct 31 04:44:16.219874 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:16.183349 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e889be1-d856-4ffc-86b9-f9fa9da208c0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-7b8fkmdlx\" (UID: \"5e889be1-d856-4ffc-86b9-f9fa9da208c0\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fkmdlx" Oct 31 04:44:16.248703 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:16.237529 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2hrg\" (UniqueName: \"kubernetes.io/projected/5e889be1-d856-4ffc-86b9-f9fa9da208c0-kube-api-access-n2hrg\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-7b8fkmdlx\" (UID: \"5e889be1-d856-4ffc-86b9-f9fa9da208c0\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fkmdlx" Oct 31 04:44:16.248703 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:16.239769 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5e889be1-d856-4ffc-86b9-f9fa9da208c0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-7b8fkmdlx\" (UID: \"5e889be1-d856-4ffc-86b9-f9fa9da208c0\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fkmdlx" Oct 31 04:44:16.252645 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:16.252567 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fkmdlx" Oct 31 04:44:16.305503 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:16.305456 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6c59wxpxq"] Oct 31 04:44:16.309736 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:16.309696 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6c59wxpxq" Oct 31 04:44:16.610552 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:16.610508 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fkmdlx"] Oct 31 04:44:16.611215 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:44:16.611187 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e889be1_d856_4ffc_86b9_f9fa9da208c0.slice/crio-32ae6cfc5574d427913d40cb1e62d94c5b92c71e5f5f01b51ce257bcd5065e67 WatchSource:0}: Error finding container 32ae6cfc5574d427913d40cb1e62d94c5b92c71e5f5f01b51ce257bcd5065e67: Status 404 returned error can't find the container with id 32ae6cfc5574d427913d40cb1e62d94c5b92c71e5f5f01b51ce257bcd5065e67 Oct 31 04:44:17.403831 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:17.403751 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7" Oct 31 04:44:17.448948 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:17.448907 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5581a822-03f0-4f8c-bd5e-444274a4bd9b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5581a822-03f0-4f8c-bd5e-444274a4bd9b-workload-socket\") pod \"5581a822-03f0-4f8c-bd5e-444274a4bd9b\" (UID: \"5581a822-03f0-4f8c-bd5e-444274a4bd9b\") " Oct 31 04:44:17.449180 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:17.449072 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5581a822-03f0-4f8c-bd5e-444274a4bd9b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5581a822-03f0-4f8c-bd5e-444274a4bd9b-istiod-ca-cert\") pod \"5581a822-03f0-4f8c-bd5e-444274a4bd9b\" (UID: \"5581a822-03f0-4f8c-bd5e-444274a4bd9b\") " Oct 31 04:44:17.449180 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:17.449119 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5581a822-03f0-4f8c-bd5e-444274a4bd9b-kube-api-access-fqcqz\" (UniqueName: \"kubernetes.io/projected/5581a822-03f0-4f8c-bd5e-444274a4bd9b-kube-api-access-fqcqz\") pod \"5581a822-03f0-4f8c-bd5e-444274a4bd9b\" (UID: \"5581a822-03f0-4f8c-bd5e-444274a4bd9b\") " Oct 31 04:44:17.449180 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:17.449173 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5581a822-03f0-4f8c-bd5e-444274a4bd9b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5581a822-03f0-4f8c-bd5e-444274a4bd9b-istio-data\") pod \"5581a822-03f0-4f8c-bd5e-444274a4bd9b\" (UID: \"5581a822-03f0-4f8c-bd5e-444274a4bd9b\") " Oct 31 04:44:17.449451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:17.449266 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5581a822-03f0-4f8c-bd5e-444274a4bd9b-workload-socket" pod "5581a822-03f0-4f8c-bd5e-444274a4bd9b" (UID: "5581a822-03f0-4f8c-bd5e-444274a4bd9b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:44:17.449451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:17.449277 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5581a822-03f0-4f8c-bd5e-444274a4bd9b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5581a822-03f0-4f8c-bd5e-444274a4bd9b-workload-certs\") pod \"5581a822-03f0-4f8c-bd5e-444274a4bd9b\" (UID: \"5581a822-03f0-4f8c-bd5e-444274a4bd9b\") " Oct 31 04:44:17.449451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:17.449435 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5581a822-03f0-4f8c-bd5e-444274a4bd9b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5581a822-03f0-4f8c-bd5e-444274a4bd9b-istio-podinfo\") pod \"5581a822-03f0-4f8c-bd5e-444274a4bd9b\" (UID: \"5581a822-03f0-4f8c-bd5e-444274a4bd9b\") " Oct 31 04:44:17.449628 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:17.449512 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5581a822-03f0-4f8c-bd5e-444274a4bd9b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5581a822-03f0-4f8c-bd5e-444274a4bd9b-istio-envoy\") pod \"5581a822-03f0-4f8c-bd5e-444274a4bd9b\" (UID: \"5581a822-03f0-4f8c-bd5e-444274a4bd9b\") " Oct 31 04:44:17.449628 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:17.449520 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5581a822-03f0-4f8c-bd5e-444274a4bd9b-workload-certs" pod "5581a822-03f0-4f8c-bd5e-444274a4bd9b" (UID: "5581a822-03f0-4f8c-bd5e-444274a4bd9b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:44:17.449628 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:17.449550 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5581a822-03f0-4f8c-bd5e-444274a4bd9b-istio-data" pod "5581a822-03f0-4f8c-bd5e-444274a4bd9b" (UID: "5581a822-03f0-4f8c-bd5e-444274a4bd9b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:44:17.449628 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:17.449572 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5581a822-03f0-4f8c-bd5e-444274a4bd9b-istiod-ca-cert" pod "5581a822-03f0-4f8c-bd5e-444274a4bd9b" (UID: "5581a822-03f0-4f8c-bd5e-444274a4bd9b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:44:17.449628 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:17.449595 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5581a822-03f0-4f8c-bd5e-444274a4bd9b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5581a822-03f0-4f8c-bd5e-444274a4bd9b-credential-socket\") pod \"5581a822-03f0-4f8c-bd5e-444274a4bd9b\" (UID: \"5581a822-03f0-4f8c-bd5e-444274a4bd9b\") " Oct 31 04:44:17.449967 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:17.449641 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5581a822-03f0-4f8c-bd5e-444274a4bd9b-istio-token\" (UniqueName: \"kubernetes.io/projected/5581a822-03f0-4f8c-bd5e-444274a4bd9b-istio-token\") pod \"5581a822-03f0-4f8c-bd5e-444274a4bd9b\" (UID: \"5581a822-03f0-4f8c-bd5e-444274a4bd9b\") " Oct 31 04:44:17.449967 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:17.449874 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5581a822-03f0-4f8c-bd5e-444274a4bd9b-credential-socket" pod "5581a822-03f0-4f8c-bd5e-444274a4bd9b" (UID: "5581a822-03f0-4f8c-bd5e-444274a4bd9b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:44:17.450762 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:17.450736 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5581a822-03f0-4f8c-bd5e-444274a4bd9b-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:44:17.450955 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:17.450920 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5581a822-03f0-4f8c-bd5e-444274a4bd9b-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:44:17.450955 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:17.450949 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5581a822-03f0-4f8c-bd5e-444274a4bd9b-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:44:17.451292 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:17.450966 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5581a822-03f0-4f8c-bd5e-444274a4bd9b-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:44:17.451292 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:17.450980 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5581a822-03f0-4f8c-bd5e-444274a4bd9b-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:44:17.452890 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:17.452861 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5581a822-03f0-4f8c-bd5e-444274a4bd9b-istio-envoy" pod "5581a822-03f0-4f8c-bd5e-444274a4bd9b" (UID: "5581a822-03f0-4f8c-bd5e-444274a4bd9b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:44:17.454107 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:17.454079 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5581a822-03f0-4f8c-bd5e-444274a4bd9b-istio-podinfo" pod "5581a822-03f0-4f8c-bd5e-444274a4bd9b" (UID: "5581a822-03f0-4f8c-bd5e-444274a4bd9b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:44:17.455925 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:17.455892 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5581a822-03f0-4f8c-bd5e-444274a4bd9b-kube-api-access-fqcqz" pod "5581a822-03f0-4f8c-bd5e-444274a4bd9b" (UID: "5581a822-03f0-4f8c-bd5e-444274a4bd9b"). InnerVolumeSpecName "kube-api-access-fqcqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:44:17.456059 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:17.456021 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5581a822-03f0-4f8c-bd5e-444274a4bd9b-istio-token" pod "5581a822-03f0-4f8c-bd5e-444274a4bd9b" (UID: "5581a822-03f0-4f8c-bd5e-444274a4bd9b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:44:17.553209 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:17.553167 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqcqz\" (UniqueName: \"kubernetes.io/projected/5581a822-03f0-4f8c-bd5e-444274a4bd9b-kube-api-access-fqcqz\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:44:17.553209 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:17.553206 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5581a822-03f0-4f8c-bd5e-444274a4bd9b-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:44:17.553209 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:17.553216 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5581a822-03f0-4f8c-bd5e-444274a4bd9b-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:44:17.554769 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:17.553228 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5581a822-03f0-4f8c-bd5e-444274a4bd9b-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:44:17.555840 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:17.555802 5019 generic.go:356] "Generic (PLEG): container finished" podID="5581a822-03f0-4f8c-bd5e-444274a4bd9b" containerID="af4ee7a37af498ec39a99b06bccebfb4238df4bba896e77312020e27b4555f17" exitCode=0 Oct 31 04:44:17.556017 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:17.555867 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7" Oct 31 04:44:17.556017 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:17.555887 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7" event={"ID":"5581a822-03f0-4f8c-bd5e-444274a4bd9b","Type":"ContainerDied","Data":"af4ee7a37af498ec39a99b06bccebfb4238df4bba896e77312020e27b4555f17"} Oct 31 04:44:17.556017 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:17.555961 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7" event={"ID":"5581a822-03f0-4f8c-bd5e-444274a4bd9b","Type":"ContainerDied","Data":"dc76064aee5acdefb29532e4002aaf2cebf2c5480b67f4ac51d144b5904e7ba1"} Oct 31 04:44:17.556017 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:17.555994 5019 scope.go:122] "RemoveContainer" containerID="af4ee7a37af498ec39a99b06bccebfb4238df4bba896e77312020e27b4555f17" Oct 31 04:44:17.558512 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:17.558485 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fkmdlx" event={"ID":"5e889be1-d856-4ffc-86b9-f9fa9da208c0","Type":"ContainerStarted","Data":"c59dbcf2549046b477ac5bb3edb4db72d3fd7125d98a9ec219defece7236b46f"} Oct 31 04:44:17.558669 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:17.558546 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fkmdlx" event={"ID":"5e889be1-d856-4ffc-86b9-f9fa9da208c0","Type":"ContainerStarted","Data":"32ae6cfc5574d427913d40cb1e62d94c5b92c71e5f5f01b51ce257bcd5065e67"} Oct 31 04:44:17.558669 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:17.558579 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fkmdlx" Oct 31 04:44:17.586573 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:17.586542 5019 scope.go:122] "RemoveContainer" containerID="7a46c6e9ba6fea80c750c3d47dcad8df0c6a3dfc624839b3d723fdf3fb7d9e2c" Oct 31 04:44:17.602029 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:17.602002 5019 scope.go:122] "RemoveContainer" containerID="af4ee7a37af498ec39a99b06bccebfb4238df4bba896e77312020e27b4555f17" Oct 31 04:44:17.602470 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:44:17.602435 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af4ee7a37af498ec39a99b06bccebfb4238df4bba896e77312020e27b4555f17\": container with ID starting with af4ee7a37af498ec39a99b06bccebfb4238df4bba896e77312020e27b4555f17 not found: ID does not exist" containerID="af4ee7a37af498ec39a99b06bccebfb4238df4bba896e77312020e27b4555f17" Oct 31 04:44:17.602641 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:17.602470 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af4ee7a37af498ec39a99b06bccebfb4238df4bba896e77312020e27b4555f17"} err="failed to get container status \"af4ee7a37af498ec39a99b06bccebfb4238df4bba896e77312020e27b4555f17\": rpc error: code = NotFound desc = could not find container \"af4ee7a37af498ec39a99b06bccebfb4238df4bba896e77312020e27b4555f17\": container with ID starting with af4ee7a37af498ec39a99b06bccebfb4238df4bba896e77312020e27b4555f17 not found: ID does not exist" Oct 31 04:44:17.602641 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:17.602488 5019 scope.go:122] "RemoveContainer" containerID="7a46c6e9ba6fea80c750c3d47dcad8df0c6a3dfc624839b3d723fdf3fb7d9e2c" Oct 31 04:44:17.602866 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:44:17.602826 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a46c6e9ba6fea80c750c3d47dcad8df0c6a3dfc624839b3d723fdf3fb7d9e2c\": container with ID starting with 7a46c6e9ba6fea80c750c3d47dcad8df0c6a3dfc624839b3d723fdf3fb7d9e2c not found: ID does not exist" containerID="7a46c6e9ba6fea80c750c3d47dcad8df0c6a3dfc624839b3d723fdf3fb7d9e2c" Oct 31 04:44:17.602938 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:17.602867 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a46c6e9ba6fea80c750c3d47dcad8df0c6a3dfc624839b3d723fdf3fb7d9e2c"} err="failed to get container status \"7a46c6e9ba6fea80c750c3d47dcad8df0c6a3dfc624839b3d723fdf3fb7d9e2c\": rpc error: code = NotFound desc = could not find container \"7a46c6e9ba6fea80c750c3d47dcad8df0c6a3dfc624839b3d723fdf3fb7d9e2c\": container with ID starting with 7a46c6e9ba6fea80c750c3d47dcad8df0c6a3dfc624839b3d723fdf3fb7d9e2c not found: ID does not exist" Oct 31 04:44:17.614841 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:17.614780 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fkmdlx" podStartSLOduration=2.614761325 podStartE2EDuration="2.614761325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:44:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:44:17.611967108 +0000 UTC m=+2412.368305149" watchObservedRunningTime="2026-10-31 04:44:17.614761325 +0000 UTC m=+2412.371099377" Oct 31 04:44:17.652561 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:17.652505 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7"] Oct 31 04:44:17.658061 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:17.657968 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-56nq7"] Oct 31 04:44:19.410231 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:19.410195 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5581a822-03f0-4f8c-bd5e-444274a4bd9b" path="/var/lib/kubelet/pods/5581a822-03f0-4f8c-bd5e-444274a4bd9b/volumes" Oct 31 04:44:21.306617 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:21.306548 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6c59wxpxq" podUID="4310f02c-7bfd-4e01-acaa-67b2357988bb" containerName="payload-processing" containerID="cri-o://9df4fc9d03f2da1a6e95c8dc120f4d8ce22889991e9dbb196bcd271913982deb" gracePeriod=25 Oct 31 04:44:21.601139 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:21.601102 5019 generic.go:356] "Generic (PLEG): container finished" podID="4310f02c-7bfd-4e01-acaa-67b2357988bb" containerID="9df4fc9d03f2da1a6e95c8dc120f4d8ce22889991e9dbb196bcd271913982deb" exitCode=0 Oct 31 04:44:21.601326 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:21.601145 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6c59wxpxq" event={"ID":"4310f02c-7bfd-4e01-acaa-67b2357988bb","Type":"ContainerDied","Data":"9df4fc9d03f2da1a6e95c8dc120f4d8ce22889991e9dbb196bcd271913982deb"} Oct 31 04:44:21.601326 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:21.601208 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6c59wxpxq" event={"ID":"4310f02c-7bfd-4e01-acaa-67b2357988bb","Type":"ContainerDied","Data":"f9c5975f934746d692d6d096870c6cb4e40d2a6309517b38628e25819bfd8b24"} Oct 31 04:44:21.601326 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:21.601225 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9c5975f934746d692d6d096870c6cb4e40d2a6309517b38628e25819bfd8b24" Oct 31 04:44:21.606846 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:21.606818 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6c59wxpxq" Oct 31 04:44:21.724582 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:21.724532 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4310f02c-7bfd-4e01-acaa-67b2357988bb-kube-api-access-2t98q\" (UniqueName: \"kubernetes.io/projected/4310f02c-7bfd-4e01-acaa-67b2357988bb-kube-api-access-2t98q\") pod \"4310f02c-7bfd-4e01-acaa-67b2357988bb\" (UID: \"4310f02c-7bfd-4e01-acaa-67b2357988bb\") " Oct 31 04:44:21.724775 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:21.724685 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4310f02c-7bfd-4e01-acaa-67b2357988bb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4310f02c-7bfd-4e01-acaa-67b2357988bb-routing-overlay-volume\") pod \"4310f02c-7bfd-4e01-acaa-67b2357988bb\" (UID: \"4310f02c-7bfd-4e01-acaa-67b2357988bb\") " Oct 31 04:44:21.724775 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:21.724724 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4310f02c-7bfd-4e01-acaa-67b2357988bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4310f02c-7bfd-4e01-acaa-67b2357988bb-plugins-config-volume\") pod \"4310f02c-7bfd-4e01-acaa-67b2357988bb\" (UID: \"4310f02c-7bfd-4e01-acaa-67b2357988bb\") " Oct 31 04:44:21.724951 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:21.724934 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4310f02c-7bfd-4e01-acaa-67b2357988bb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4310f02c-7bfd-4e01-acaa-67b2357988bb-provider-credentials\") pod \"4310f02c-7bfd-4e01-acaa-67b2357988bb\" (UID: \"4310f02c-7bfd-4e01-acaa-67b2357988bb\") " Oct 31 04:44:21.725109 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:21.725078 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4310f02c-7bfd-4e01-acaa-67b2357988bb-routing-overlay-volume" pod "4310f02c-7bfd-4e01-acaa-67b2357988bb" (UID: "4310f02c-7bfd-4e01-acaa-67b2357988bb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:44:21.725202 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:21.725177 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4310f02c-7bfd-4e01-acaa-67b2357988bb-plugins-config-volume" pod "4310f02c-7bfd-4e01-acaa-67b2357988bb" (UID: "4310f02c-7bfd-4e01-acaa-67b2357988bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:44:21.726210 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:21.726180 5019 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4310f02c-7bfd-4e01-acaa-67b2357988bb-routing-overlay-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:44:21.726469 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:21.726452 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4310f02c-7bfd-4e01-acaa-67b2357988bb-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:44:21.739536 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:21.739486 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4310f02c-7bfd-4e01-acaa-67b2357988bb-provider-credentials" pod "4310f02c-7bfd-4e01-acaa-67b2357988bb" (UID: "4310f02c-7bfd-4e01-acaa-67b2357988bb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:44:21.742920 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:21.742880 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4310f02c-7bfd-4e01-acaa-67b2357988bb-kube-api-access-2t98q" pod "4310f02c-7bfd-4e01-acaa-67b2357988bb" (UID: "4310f02c-7bfd-4e01-acaa-67b2357988bb"). InnerVolumeSpecName "kube-api-access-2t98q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:44:21.827991 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:21.827893 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t98q\" (UniqueName: \"kubernetes.io/projected/4310f02c-7bfd-4e01-acaa-67b2357988bb-kube-api-access-2t98q\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:44:21.827991 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:21.827933 5019 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4310f02c-7bfd-4e01-acaa-67b2357988bb-provider-credentials\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:44:22.394210 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:44:22.394167 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:44:22.540356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:22.533372 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-5d44jzg8t" Oct 31 04:44:22.630826 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:22.630784 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bntvwv"] Oct 31 04:44:22.633793 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:22.633760 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6c59wxpxq" Oct 31 04:44:22.713597 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:22.713549 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6c59wxpxq"] Oct 31 04:44:22.729254 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:22.729201 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6c59wxpxq"] Oct 31 04:44:23.412763 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:23.412723 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4310f02c-7bfd-4e01-acaa-67b2357988bb" path="/var/lib/kubelet/pods/4310f02c-7bfd-4e01-acaa-67b2357988bb/volumes" Oct 31 04:44:27.631886 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:27.631823 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bntvwv" podUID="8a727ed8-e1e8-4728-af11-eb6c7a57bc72" containerName="payload-processing" containerID="cri-o://f543bb92b1ce072d4c0ade472f07b675671481555d023cdf8e1ac105ef933851" gracePeriod=25 Oct 31 04:44:27.967630 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:27.967587 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bntvwv" Oct 31 04:44:28.047629 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:28.047594 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a727ed8-e1e8-4728-af11-eb6c7a57bc72-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8a727ed8-e1e8-4728-af11-eb6c7a57bc72-provider-credentials\") pod \"8a727ed8-e1e8-4728-af11-eb6c7a57bc72\" (UID: \"8a727ed8-e1e8-4728-af11-eb6c7a57bc72\") " Oct 31 04:44:28.047854 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:28.047677 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a727ed8-e1e8-4728-af11-eb6c7a57bc72-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a727ed8-e1e8-4728-af11-eb6c7a57bc72-plugins-config-volume\") pod \"8a727ed8-e1e8-4728-af11-eb6c7a57bc72\" (UID: \"8a727ed8-e1e8-4728-af11-eb6c7a57bc72\") " Oct 31 04:44:28.047854 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:28.047736 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a727ed8-e1e8-4728-af11-eb6c7a57bc72-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8a727ed8-e1e8-4728-af11-eb6c7a57bc72-routing-overlay-volume\") pod \"8a727ed8-e1e8-4728-af11-eb6c7a57bc72\" (UID: \"8a727ed8-e1e8-4728-af11-eb6c7a57bc72\") " Oct 31 04:44:28.047992 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:28.047926 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a727ed8-e1e8-4728-af11-eb6c7a57bc72-kube-api-access-nq8r6\" (UniqueName: \"kubernetes.io/projected/8a727ed8-e1e8-4728-af11-eb6c7a57bc72-kube-api-access-nq8r6\") pod \"8a727ed8-e1e8-4728-af11-eb6c7a57bc72\" (UID: \"8a727ed8-e1e8-4728-af11-eb6c7a57bc72\") " Oct 31 04:44:28.048273 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:28.048244 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a727ed8-e1e8-4728-af11-eb6c7a57bc72-plugins-config-volume" pod "8a727ed8-e1e8-4728-af11-eb6c7a57bc72" (UID: "8a727ed8-e1e8-4728-af11-eb6c7a57bc72"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:44:28.048407 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:28.048293 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a727ed8-e1e8-4728-af11-eb6c7a57bc72-routing-overlay-volume" pod "8a727ed8-e1e8-4728-af11-eb6c7a57bc72" (UID: "8a727ed8-e1e8-4728-af11-eb6c7a57bc72"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:44:28.048754 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:28.048732 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a727ed8-e1e8-4728-af11-eb6c7a57bc72-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:44:28.048754 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:28.048755 5019 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8a727ed8-e1e8-4728-af11-eb6c7a57bc72-routing-overlay-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:44:28.050321 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:28.050293 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a727ed8-e1e8-4728-af11-eb6c7a57bc72-kube-api-access-nq8r6" pod "8a727ed8-e1e8-4728-af11-eb6c7a57bc72" (UID: "8a727ed8-e1e8-4728-af11-eb6c7a57bc72"). InnerVolumeSpecName "kube-api-access-nq8r6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:44:28.050540 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:28.050507 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a727ed8-e1e8-4728-af11-eb6c7a57bc72-provider-credentials" pod "8a727ed8-e1e8-4728-af11-eb6c7a57bc72" (UID: "8a727ed8-e1e8-4728-af11-eb6c7a57bc72"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:44:28.150110 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:28.150057 5019 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8a727ed8-e1e8-4728-af11-eb6c7a57bc72-provider-credentials\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:44:28.150110 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:28.150104 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nq8r6\" (UniqueName: \"kubernetes.io/projected/8a727ed8-e1e8-4728-af11-eb6c7a57bc72-kube-api-access-nq8r6\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:44:28.573725 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:28.573689 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fkmdlx" Oct 31 04:44:28.690410 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:28.690315 5019 generic.go:356] "Generic (PLEG): container finished" podID="8a727ed8-e1e8-4728-af11-eb6c7a57bc72" containerID="f543bb92b1ce072d4c0ade472f07b675671481555d023cdf8e1ac105ef933851" exitCode=0 Oct 31 04:44:28.690410 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:28.690406 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bntvwv" Oct 31 04:44:28.690965 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:28.690415 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bntvwv" event={"ID":"8a727ed8-e1e8-4728-af11-eb6c7a57bc72","Type":"ContainerDied","Data":"f543bb92b1ce072d4c0ade472f07b675671481555d023cdf8e1ac105ef933851"} Oct 31 04:44:28.690965 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:28.690488 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bntvwv" event={"ID":"8a727ed8-e1e8-4728-af11-eb6c7a57bc72","Type":"ContainerDied","Data":"b44b8aa8c86e64579f1c766a8915273cc8e1167623a2ea6f115fdc72f7dde920"} Oct 31 04:44:28.690965 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:28.690521 5019 scope.go:122] "RemoveContainer" containerID="f543bb92b1ce072d4c0ade472f07b675671481555d023cdf8e1ac105ef933851" Oct 31 04:44:28.706161 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:28.705971 5019 scope.go:122] "RemoveContainer" containerID="f543bb92b1ce072d4c0ade472f07b675671481555d023cdf8e1ac105ef933851" Oct 31 04:44:28.706477 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:44:28.706445 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f543bb92b1ce072d4c0ade472f07b675671481555d023cdf8e1ac105ef933851\": container with ID starting with f543bb92b1ce072d4c0ade472f07b675671481555d023cdf8e1ac105ef933851 not found: ID does not exist" containerID="f543bb92b1ce072d4c0ade472f07b675671481555d023cdf8e1ac105ef933851" Oct 31 04:44:28.706607 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:28.706485 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f543bb92b1ce072d4c0ade472f07b675671481555d023cdf8e1ac105ef933851"} err="failed to get container status \"f543bb92b1ce072d4c0ade472f07b675671481555d023cdf8e1ac105ef933851\": rpc error: code = NotFound desc = could not find container \"f543bb92b1ce072d4c0ade472f07b675671481555d023cdf8e1ac105ef933851\": container with ID starting with f543bb92b1ce072d4c0ade472f07b675671481555d023cdf8e1ac105ef933851 not found: ID does not exist" Oct 31 04:44:28.744798 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:28.744760 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bntvwv"] Oct 31 04:44:28.749533 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:28.749493 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bntvwv"] Oct 31 04:44:29.414468 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:29.414432 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a727ed8-e1e8-4728-af11-eb6c7a57bc72" path="/var/lib/kubelet/pods/8a727ed8-e1e8-4728-af11-eb6c7a57bc72/volumes" Oct 31 04:44:32.037857 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:32.037814 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee57b3ee-4mlbl"] Oct 31 04:44:32.048550 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:32.048506 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee57b3ee-4mlbl"] Oct 31 04:44:33.411637 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:33.411593 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de3e39e4-9e92-4fb5-b867-1d19b2571dc4" path="/var/lib/kubelet/pods/de3e39e4-9e92-4fb5-b867-1d19b2571dc4/volumes" Oct 31 04:44:42.895236 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:42.895195 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-797454d2z"] Oct 31 04:44:42.896073 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:42.896053 5019 cpu_manager.go:414] "containerMap: removing container" podUID="8a727ed8-e1e8-4728-af11-eb6c7a57bc72" containerName="payload-processing" Oct 31 04:44:42.896125 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:42.896074 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a727ed8-e1e8-4728-af11-eb6c7a57bc72" containerName="payload-processing" Oct 31 04:44:42.896125 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:42.896092 5019 cpu_manager.go:414] "containerMap: removing container" podUID="5581a822-03f0-4f8c-bd5e-444274a4bd9b" containerName="istio-proxy" Oct 31 04:44:42.896125 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:42.896097 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5581a822-03f0-4f8c-bd5e-444274a4bd9b" containerName="istio-proxy" Oct 31 04:44:42.896228 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:42.896144 5019 cpu_manager.go:414] "containerMap: removing container" podUID="4310f02c-7bfd-4e01-acaa-67b2357988bb" containerName="payload-processing" Oct 31 04:44:42.896228 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:42.896150 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4310f02c-7bfd-4e01-acaa-67b2357988bb" containerName="payload-processing" Oct 31 04:44:42.896302 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:42.896277 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a727ed8-e1e8-4728-af11-eb6c7a57bc72" containerName="payload-processing" Oct 31 04:44:42.896302 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:42.896289 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="5581a822-03f0-4f8c-bd5e-444274a4bd9b" containerName="istio-proxy" Oct 31 04:44:42.896302 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:42.896301 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="5581a822-03f0-4f8c-bd5e-444274a4bd9b" containerName="istio-proxy" Oct 31 04:44:42.896495 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:42.896309 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="4310f02c-7bfd-4e01-acaa-67b2357988bb" containerName="payload-processing" Oct 31 04:44:42.900892 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:42.900862 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-797454d2z" Oct 31 04:44:42.936362 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:42.936282 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a99f2a\"/\"payload-processing-external-model-plugins-e2e-extmod-b-a99f2a\"" Oct 31 04:44:42.936362 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:42.936290 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a99f2a\"/\"openshift-service-ca.crt\"" Oct 31 04:44:42.936362 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:42.936349 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a99f2a\"/\"routing-overlay\"" Oct 31 04:44:42.936702 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:42.936483 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a99f2a\"/\"kube-root-ca.crt\"" Oct 31 04:44:42.942041 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:42.942004 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-797454d2z"] Oct 31 04:44:42.984722 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:42.984456 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a99f2a\"/\"agc-cred-7194275e28ab\"" Oct 31 04:44:42.984722 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:42.984544 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a99f2a\"/\"payload-processing-external-model-e2e--c07b99ee-dockercfg-gzb75\"" Oct 31 04:44:42.984722 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:42.984549 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a99f2a\"/\"agc-cred-add5bfec2790\"" Oct 31 04:44:42.999717 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:42.999675 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d8fa7e98-3021-4077-9e2b-14f31b6efd65-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-797454d2z\" (UID: \"d8fa7e98-3021-4077-9e2b-14f31b6efd65\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-797454d2z" Oct 31 04:44:42.999912 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:42.999749 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d8fa7e98-3021-4077-9e2b-14f31b6efd65-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-797454d2z\" (UID: \"d8fa7e98-3021-4077-9e2b-14f31b6efd65\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-797454d2z" Oct 31 04:44:42.999962 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:42.999924 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8hxh\" (UniqueName: \"kubernetes.io/projected/d8fa7e98-3021-4077-9e2b-14f31b6efd65-kube-api-access-l8hxh\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-797454d2z\" (UID: \"d8fa7e98-3021-4077-9e2b-14f31b6efd65\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-797454d2z" Oct 31 04:44:43.000109 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:43.000089 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8fa7e98-3021-4077-9e2b-14f31b6efd65-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-797454d2z\" (UID: \"d8fa7e98-3021-4077-9e2b-14f31b6efd65\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-797454d2z" Oct 31 04:44:43.031369 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:43.031314 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79257c5d-mrcfj"] Oct 31 04:44:43.036280 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:43.036240 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79257c5d-mrcfj"] Oct 31 04:44:43.101466 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:43.101420 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8fa7e98-3021-4077-9e2b-14f31b6efd65-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-797454d2z\" (UID: \"d8fa7e98-3021-4077-9e2b-14f31b6efd65\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-797454d2z" Oct 31 04:44:43.101690 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:43.101633 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d8fa7e98-3021-4077-9e2b-14f31b6efd65-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-797454d2z\" (UID: \"d8fa7e98-3021-4077-9e2b-14f31b6efd65\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-797454d2z" Oct 31 04:44:43.101773 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:43.101712 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d8fa7e98-3021-4077-9e2b-14f31b6efd65-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-797454d2z\" (UID: \"d8fa7e98-3021-4077-9e2b-14f31b6efd65\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-797454d2z" Oct 31 04:44:43.101839 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:43.101823 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8hxh\" (UniqueName: \"kubernetes.io/projected/d8fa7e98-3021-4077-9e2b-14f31b6efd65-kube-api-access-l8hxh\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-797454d2z\" (UID: \"d8fa7e98-3021-4077-9e2b-14f31b6efd65\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-797454d2z" Oct 31 04:44:43.102371 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:43.102272 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d8fa7e98-3021-4077-9e2b-14f31b6efd65-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-797454d2z\" (UID: \"d8fa7e98-3021-4077-9e2b-14f31b6efd65\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-797454d2z" Oct 31 04:44:43.102371 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:43.102301 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8fa7e98-3021-4077-9e2b-14f31b6efd65-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-797454d2z\" (UID: \"d8fa7e98-3021-4077-9e2b-14f31b6efd65\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-797454d2z" Oct 31 04:44:43.104473 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:43.104442 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d8fa7e98-3021-4077-9e2b-14f31b6efd65-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-797454d2z\" (UID: \"d8fa7e98-3021-4077-9e2b-14f31b6efd65\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-797454d2z" Oct 31 04:44:43.125479 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:43.125443 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8hxh\" (UniqueName: \"kubernetes.io/projected/d8fa7e98-3021-4077-9e2b-14f31b6efd65-kube-api-access-l8hxh\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-797454d2z\" (UID: \"d8fa7e98-3021-4077-9e2b-14f31b6efd65\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-797454d2z" Oct 31 04:44:43.217816 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:43.217781 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-797454d2z" Oct 31 04:44:43.412440 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:43.412400 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c0cd54f-f479-4e04-a535-5fc3b14344f2" path="/var/lib/kubelet/pods/9c0cd54f-f479-4e04-a535-5fc3b14344f2/volumes" Oct 31 04:44:43.538105 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:43.538067 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-797454d2z"] Oct 31 04:44:43.888238 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:43.888130 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-797454d2z" event={"ID":"d8fa7e98-3021-4077-9e2b-14f31b6efd65","Type":"ContainerStarted","Data":"94dc9c8d78568acfead4453ab8f571aabb096b70541ca875cf586a356be63eee"} Oct 31 04:44:43.888238 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:43.888192 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-797454d2z" event={"ID":"d8fa7e98-3021-4077-9e2b-14f31b6efd65","Type":"ContainerStarted","Data":"a0bec2d5c768a03354f96a716ed56ee1eed94944cb6902482be7fbdd03d9cd22"} Oct 31 04:44:43.888238 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:43.888225 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-797454d2z" Oct 31 04:44:43.955371 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:43.955264 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-797454d2z" podStartSLOduration=1.955240348 podStartE2EDuration="1.955240348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:44:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:44:43.950907467 +0000 UTC m=+2438.707245506" watchObservedRunningTime="2026-10-31 04:44:43.955240348 +0000 UTC m=+2438.711578393" Oct 31 04:44:49.264080 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:49.264041 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b66kbvh"] Oct 31 04:44:49.264976 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:49.264956 5019 cpu_manager.go:414] "containerMap: removing container" podUID="5581a822-03f0-4f8c-bd5e-444274a4bd9b" containerName="istio-proxy" Oct 31 04:44:49.264976 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:49.264976 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5581a822-03f0-4f8c-bd5e-444274a4bd9b" containerName="istio-proxy" Oct 31 04:44:49.277360 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:49.269319 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b66kbvh" Oct 31 04:44:49.313682 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:49.313433 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b66kbvh"] Oct 31 04:44:49.374368 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:49.371046 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/837f143a-072a-446d-9062-cb6f57eb4884-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-75b66kbvh\" (UID: \"837f143a-072a-446d-9062-cb6f57eb4884\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b66kbvh" Oct 31 04:44:49.374368 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:49.371218 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/837f143a-072a-446d-9062-cb6f57eb4884-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-75b66kbvh\" (UID: \"837f143a-072a-446d-9062-cb6f57eb4884\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b66kbvh" Oct 31 04:44:49.374368 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:49.371308 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zsj7\" (UniqueName: \"kubernetes.io/projected/837f143a-072a-446d-9062-cb6f57eb4884-kube-api-access-4zsj7\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-75b66kbvh\" (UID: \"837f143a-072a-446d-9062-cb6f57eb4884\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b66kbvh" Oct 31 04:44:49.374368 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:49.371452 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/837f143a-072a-446d-9062-cb6f57eb4884-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-75b66kbvh\" (UID: \"837f143a-072a-446d-9062-cb6f57eb4884\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b66kbvh" Oct 31 04:44:49.485355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:49.475525 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/837f143a-072a-446d-9062-cb6f57eb4884-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-75b66kbvh\" (UID: \"837f143a-072a-446d-9062-cb6f57eb4884\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b66kbvh" Oct 31 04:44:49.489836 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:49.489797 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zsj7\" (UniqueName: \"kubernetes.io/projected/837f143a-072a-446d-9062-cb6f57eb4884-kube-api-access-4zsj7\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-75b66kbvh\" (UID: \"837f143a-072a-446d-9062-cb6f57eb4884\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b66kbvh" Oct 31 04:44:49.506295 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:49.500131 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/837f143a-072a-446d-9062-cb6f57eb4884-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-75b66kbvh\" (UID: \"837f143a-072a-446d-9062-cb6f57eb4884\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b66kbvh" Oct 31 04:44:49.506599 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:49.506458 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/837f143a-072a-446d-9062-cb6f57eb4884-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-75b66kbvh\" (UID: \"837f143a-072a-446d-9062-cb6f57eb4884\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b66kbvh" Oct 31 04:44:49.506797 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:49.506773 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/837f143a-072a-446d-9062-cb6f57eb4884-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-75b66kbvh\" (UID: \"837f143a-072a-446d-9062-cb6f57eb4884\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b66kbvh" Oct 31 04:44:49.507612 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:49.507582 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/837f143a-072a-446d-9062-cb6f57eb4884-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-75b66kbvh\" (UID: \"837f143a-072a-446d-9062-cb6f57eb4884\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b66kbvh" Oct 31 04:44:49.508295 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:49.508270 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/837f143a-072a-446d-9062-cb6f57eb4884-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-75b66kbvh\" (UID: \"837f143a-072a-446d-9062-cb6f57eb4884\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b66kbvh" Oct 31 04:44:49.525229 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:49.525150 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zsj7\" (UniqueName: \"kubernetes.io/projected/837f143a-072a-446d-9062-cb6f57eb4884-kube-api-access-4zsj7\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-75b66kbvh\" (UID: \"837f143a-072a-446d-9062-cb6f57eb4884\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b66kbvh" Oct 31 04:44:49.626011 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:49.625958 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b66kbvh" Oct 31 04:44:50.011256 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:50.011209 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b66kbvh"] Oct 31 04:44:50.014043 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:44:50.014008 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod837f143a_072a_446d_9062_cb6f57eb4884.slice/crio-32f598492f61ed4d4046439e4abec5accbe1d4ca044352a919e2f8866754e0d1 WatchSource:0}: Error finding container 32f598492f61ed4d4046439e4abec5accbe1d4ca044352a919e2f8866754e0d1: Status 404 returned error can't find the container with id 32f598492f61ed4d4046439e4abec5accbe1d4ca044352a919e2f8866754e0d1 Oct 31 04:44:50.064521 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:50.064479 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b66kbvh" event={"ID":"837f143a-072a-446d-9062-cb6f57eb4884","Type":"ContainerStarted","Data":"32f598492f61ed4d4046439e4abec5accbe1d4ca044352a919e2f8866754e0d1"} Oct 31 04:44:51.102553 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:51.102507 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b66kbvh" event={"ID":"837f143a-072a-446d-9062-cb6f57eb4884","Type":"ContainerStarted","Data":"8fb9d282e707fe200d7bc379420e5e3be964dda612aad5ecec9894948f61f498"} Oct 31 04:44:51.103071 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:51.102668 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b66kbvh" Oct 31 04:44:51.145666 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:51.145601 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b66kbvh" podStartSLOduration=2.145581562 podStartE2EDuration="2.145581562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:44:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:44:51.143414009 +0000 UTC m=+2445.899752049" watchObservedRunningTime="2026-10-31 04:44:51.145581562 +0000 UTC m=+2445.901919600" Oct 31 04:44:52.402111 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:44:52.402061 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:44:52.581305 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:52.580321 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz"] Oct 31 04:44:52.587325 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:52.587284 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz" Oct 31 04:44:52.628045 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:52.627584 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-a04e4e-openshift-default-dockercfg-22x4n\"" Oct 31 04:44:52.654447 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:52.640480 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz"] Oct 31 04:44:52.705359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:52.701686 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b132bea6-493c-43f7-bd51-29f02a01ae9a-workload-certs\") pod \"e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz\" (UID: \"b132bea6-493c-43f7-bd51-29f02a01ae9a\") " pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz" Oct 31 04:44:52.705359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:52.701743 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b132bea6-493c-43f7-bd51-29f02a01ae9a-istio-data\") pod \"e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz\" (UID: \"b132bea6-493c-43f7-bd51-29f02a01ae9a\") " pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz" Oct 31 04:44:52.735355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:52.731664 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b132bea6-493c-43f7-bd51-29f02a01ae9a-istio-envoy\") pod \"e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz\" (UID: \"b132bea6-493c-43f7-bd51-29f02a01ae9a\") " pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz" Oct 31 04:44:52.735355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:52.731771 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b132bea6-493c-43f7-bd51-29f02a01ae9a-istio-token\") pod \"e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz\" (UID: \"b132bea6-493c-43f7-bd51-29f02a01ae9a\") " pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz" Oct 31 04:44:52.735355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:52.731852 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b132bea6-493c-43f7-bd51-29f02a01ae9a-istiod-ca-cert\") pod \"e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz\" (UID: \"b132bea6-493c-43f7-bd51-29f02a01ae9a\") " pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz" Oct 31 04:44:52.735355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:52.731910 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b132bea6-493c-43f7-bd51-29f02a01ae9a-workload-socket\") pod \"e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz\" (UID: \"b132bea6-493c-43f7-bd51-29f02a01ae9a\") " pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz" Oct 31 04:44:52.735355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:52.732018 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b132bea6-493c-43f7-bd51-29f02a01ae9a-istio-podinfo\") pod \"e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz\" (UID: \"b132bea6-493c-43f7-bd51-29f02a01ae9a\") " pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz" Oct 31 04:44:52.735355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:52.732150 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzvwl\" (UniqueName: \"kubernetes.io/projected/b132bea6-493c-43f7-bd51-29f02a01ae9a-kube-api-access-pzvwl\") pod \"e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz\" (UID: \"b132bea6-493c-43f7-bd51-29f02a01ae9a\") " pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz" Oct 31 04:44:52.735355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:52.732233 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b132bea6-493c-43f7-bd51-29f02a01ae9a-credential-socket\") pod \"e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz\" (UID: \"b132bea6-493c-43f7-bd51-29f02a01ae9a\") " pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz" Oct 31 04:44:52.834976 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:52.834933 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b132bea6-493c-43f7-bd51-29f02a01ae9a-istio-envoy\") pod \"e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz\" (UID: \"b132bea6-493c-43f7-bd51-29f02a01ae9a\") " pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz" Oct 31 04:44:52.835293 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:52.835270 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b132bea6-493c-43f7-bd51-29f02a01ae9a-istio-token\") pod \"e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz\" (UID: \"b132bea6-493c-43f7-bd51-29f02a01ae9a\") " pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz" Oct 31 04:44:52.835533 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:52.835513 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b132bea6-493c-43f7-bd51-29f02a01ae9a-istiod-ca-cert\") pod \"e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz\" (UID: \"b132bea6-493c-43f7-bd51-29f02a01ae9a\") " pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz" Oct 31 04:44:52.835681 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:52.835665 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b132bea6-493c-43f7-bd51-29f02a01ae9a-workload-socket\") pod \"e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz\" (UID: \"b132bea6-493c-43f7-bd51-29f02a01ae9a\") " pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz" Oct 31 04:44:52.835862 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:52.835846 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b132bea6-493c-43f7-bd51-29f02a01ae9a-istio-podinfo\") pod \"e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz\" (UID: \"b132bea6-493c-43f7-bd51-29f02a01ae9a\") " pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz" Oct 31 04:44:52.836066 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:52.836050 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzvwl\" (UniqueName: \"kubernetes.io/projected/b132bea6-493c-43f7-bd51-29f02a01ae9a-kube-api-access-pzvwl\") pod \"e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz\" (UID: \"b132bea6-493c-43f7-bd51-29f02a01ae9a\") " pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz" Oct 31 04:44:52.836223 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:52.836208 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b132bea6-493c-43f7-bd51-29f02a01ae9a-credential-socket\") pod \"e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz\" (UID: \"b132bea6-493c-43f7-bd51-29f02a01ae9a\") " pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz" Oct 31 04:44:52.836422 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:52.836400 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b132bea6-493c-43f7-bd51-29f02a01ae9a-workload-certs\") pod \"e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz\" (UID: \"b132bea6-493c-43f7-bd51-29f02a01ae9a\") " pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz" Oct 31 04:44:52.836562 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:52.836547 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b132bea6-493c-43f7-bd51-29f02a01ae9a-istio-data\") pod \"e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz\" (UID: \"b132bea6-493c-43f7-bd51-29f02a01ae9a\") " pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz" Oct 31 04:44:52.837414 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:52.837305 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b132bea6-493c-43f7-bd51-29f02a01ae9a-istio-data\") pod \"e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz\" (UID: \"b132bea6-493c-43f7-bd51-29f02a01ae9a\") " pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz" Oct 31 04:44:52.851695 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:52.837922 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b132bea6-493c-43f7-bd51-29f02a01ae9a-workload-socket\") pod \"e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz\" (UID: \"b132bea6-493c-43f7-bd51-29f02a01ae9a\") " pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz" Oct 31 04:44:52.851924 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:52.848400 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b132bea6-493c-43f7-bd51-29f02a01ae9a-istiod-ca-cert\") pod \"e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz\" (UID: \"b132bea6-493c-43f7-bd51-29f02a01ae9a\") " pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz" Oct 31 04:44:52.852050 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:52.850763 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b132bea6-493c-43f7-bd51-29f02a01ae9a-workload-certs\") pod \"e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz\" (UID: \"b132bea6-493c-43f7-bd51-29f02a01ae9a\") " pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz" Oct 31 04:44:52.852152 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:52.851169 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b132bea6-493c-43f7-bd51-29f02a01ae9a-credential-socket\") pod \"e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz\" (UID: \"b132bea6-493c-43f7-bd51-29f02a01ae9a\") " pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz" Oct 31 04:44:52.925166 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:52.905939 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b132bea6-493c-43f7-bd51-29f02a01ae9a-istio-token\") pod \"e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz\" (UID: \"b132bea6-493c-43f7-bd51-29f02a01ae9a\") " pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz" Oct 31 04:44:52.961229 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:52.957466 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzvwl\" (UniqueName: \"kubernetes.io/projected/b132bea6-493c-43f7-bd51-29f02a01ae9a-kube-api-access-pzvwl\") pod \"e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz\" (UID: \"b132bea6-493c-43f7-bd51-29f02a01ae9a\") " pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz" Oct 31 04:44:52.986352 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:52.986178 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b132bea6-493c-43f7-bd51-29f02a01ae9a-istio-podinfo\") pod \"e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz\" (UID: \"b132bea6-493c-43f7-bd51-29f02a01ae9a\") " pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz" Oct 31 04:44:52.989880 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:52.989791 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b132bea6-493c-43f7-bd51-29f02a01ae9a-istio-envoy\") pod \"e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz\" (UID: \"b132bea6-493c-43f7-bd51-29f02a01ae9a\") " pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz" Oct 31 04:44:53.256990 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:53.256956 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz" Oct 31 04:44:53.724365 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:53.724309 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz"] Oct 31 04:44:53.732686 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:44:53.732633 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb132bea6_493c_43f7_bd51_29f02a01ae9a.slice/crio-a8b9d8f6768538703a73e1008fc959d55a26d28a55ec6915912da38b8f51fd8e WatchSource:0}: Error finding container a8b9d8f6768538703a73e1008fc959d55a26d28a55ec6915912da38b8f51fd8e: Status 404 returned error can't find the container with id a8b9d8f6768538703a73e1008fc959d55a26d28a55ec6915912da38b8f51fd8e Oct 31 04:44:53.736385 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:53.736317 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:44:53.736584 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:53.736421 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:44:53.736584 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:53.736456 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:44:54.139862 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:54.139759 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz" event={"ID":"b132bea6-493c-43f7-bd51-29f02a01ae9a","Type":"ContainerStarted","Data":"d61e05c21f05c1503bdface0acfc18118fd9973b66cd14a67890c6020f1f59a6"} Oct 31 04:44:54.139862 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:54.139830 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz" event={"ID":"b132bea6-493c-43f7-bd51-29f02a01ae9a","Type":"ContainerStarted","Data":"a8b9d8f6768538703a73e1008fc959d55a26d28a55ec6915912da38b8f51fd8e"} Oct 31 04:44:54.192393 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:54.192306 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz" podStartSLOduration=2.192284953 podStartE2EDuration="2.192284953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:44:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:44:54.190310822 +0000 UTC m=+2448.946648863" watchObservedRunningTime="2026-10-31 04:44:54.192284953 +0000 UTC m=+2448.948622993" Oct 31 04:44:54.259806 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:54.258104 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz" Oct 31 04:44:54.259806 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:54.259462 5019 patch_prober.go:29] interesting pod/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" start-of-body= Oct 31 04:44:54.259806 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:54.259525 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz" podUID="b132bea6-493c-43f7-bd51-29f02a01ae9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" Oct 31 04:44:54.905286 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:54.905248 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-797454d2z" Oct 31 04:44:55.279871 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:55.279478 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz" Oct 31 04:44:56.161993 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:56.161943 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz" Oct 31 04:44:56.163264 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:56.163233 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz" Oct 31 04:44:57.330711 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:57.329232 5019 patch_prober.go:29] interesting pod/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": read tcp 10.132.0.2:38192->10.132.1.35:15021: read: connection reset by peer" start-of-body= Oct 31 04:44:57.330711 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:57.329352 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf" podUID="7ce5abc5-55a0-4b15-80a6-7e21d360e389" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": read tcp 10.132.0.2:38192->10.132.1.35:15021: read: connection reset by peer" Oct 31 04:44:58.221168 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:58.221131 5019 generic.go:356] "Generic (PLEG): container finished" podID="7ce5abc5-55a0-4b15-80a6-7e21d360e389" containerID="0e29406cdd0f4aeca185f400307c928d1f7c7040da088704a7b1e5c45dca56ac" exitCode=0 Oct 31 04:44:58.221377 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:58.221205 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf" event={"ID":"7ce5abc5-55a0-4b15-80a6-7e21d360e389","Type":"ContainerDied","Data":"0e29406cdd0f4aeca185f400307c928d1f7c7040da088704a7b1e5c45dca56ac"} Oct 31 04:44:58.221748 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:58.221725 5019 scope.go:122] "RemoveContainer" containerID="0e29406cdd0f4aeca185f400307c928d1f7c7040da088704a7b1e5c45dca56ac" Oct 31 04:44:58.223677 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:58.223642 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:44:58.223793 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:58.223718 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:44:58.223793 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:58.223753 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:44:59.235838 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:59.235791 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf" event={"ID":"7ce5abc5-55a0-4b15-80a6-7e21d360e389","Type":"ContainerStarted","Data":"7ac9c8e148e3b8eb2c8dd3ded1ee78d693cca6c7855b72d66c4a5b69328290d6"} Oct 31 04:44:59.676424 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:59.673724 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf" Oct 31 04:44:59.688530 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:59.688484 5019 patch_prober.go:29] interesting pod/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body= Oct 31 04:44:59.689772 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:44:59.689732 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf" podUID="7ce5abc5-55a0-4b15-80a6-7e21d360e389" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" Oct 31 04:45:00.153381 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.153225 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890365-t5r5d"] Oct 31 04:45:00.162929 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.162881 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890365-74cxv"] Oct 31 04:45:00.213390 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.204485 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890365-t5r5d" Oct 31 04:45:00.213390 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.211746 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890365-74cxv" Oct 31 04:45:00.222354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.222116 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e282b47a-b89e-48ae-88bf-347945705a1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890365-t5r5d\" (UID: \"e282b47a-b89e-48ae-88bf-347945705a1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890365-t5r5d" Oct 31 04:45:00.222354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.222168 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w2cw\" (UniqueName: \"kubernetes.io/projected/6171ff86-f98a-4575-834d-2f6bd135424d-kube-api-access-8w2cw\") pod \"maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890365-74cxv\" (UID: \"6171ff86-f98a-4575-834d-2f6bd135424d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890365-74cxv" Oct 31 04:45:00.222354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.222318 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6171ff86-f98a-4575-834d-2f6bd135424d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890365-74cxv\" (UID: \"6171ff86-f98a-4575-834d-2f6bd135424d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890365-74cxv" Oct 31 04:45:00.222710 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.222402 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4sdj\" (UniqueName: \"kubernetes.io/projected/e282b47a-b89e-48ae-88bf-347945705a1d-kube-api-access-r4sdj\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890365-t5r5d\" (UID: \"e282b47a-b89e-48ae-88bf-347945705a1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890365-t5r5d" Oct 31 04:45:00.227122 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.227092 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:45:00.227503 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.227487 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:45:00.227790 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.227766 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:45:00.253900 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.253865 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890365-jdjqt"] Oct 31 04:45:00.315549 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.315505 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890365-f7zkm"] Oct 31 04:45:00.315862 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.315652 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890365-jdjqt" Oct 31 04:45:00.321984 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.321949 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-031745-29890365-grgsg"] Oct 31 04:45:00.322730 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.322679 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890365-f7zkm" Oct 31 04:45:00.323852 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.323822 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e282b47a-b89e-48ae-88bf-347945705a1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890365-t5r5d\" (UID: \"e282b47a-b89e-48ae-88bf-347945705a1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890365-t5r5d" Oct 31 04:45:00.323990 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.323859 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w2cw\" (UniqueName: \"kubernetes.io/projected/6171ff86-f98a-4575-834d-2f6bd135424d-kube-api-access-8w2cw\") pod \"maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890365-74cxv\" (UID: \"6171ff86-f98a-4575-834d-2f6bd135424d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890365-74cxv" Oct 31 04:45:00.323990 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.323954 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6171ff86-f98a-4575-834d-2f6bd135424d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890365-74cxv\" (UID: \"6171ff86-f98a-4575-834d-2f6bd135424d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890365-74cxv" Oct 31 04:45:00.324113 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.323999 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4sdj\" (UniqueName: \"kubernetes.io/projected/e282b47a-b89e-48ae-88bf-347945705a1d-kube-api-access-r4sdj\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890365-t5r5d\" (UID: \"e282b47a-b89e-48ae-88bf-347945705a1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890365-t5r5d" Oct 31 04:45:00.325277 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.325219 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e282b47a-b89e-48ae-88bf-347945705a1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890365-t5r5d\" (UID: \"e282b47a-b89e-48ae-88bf-347945705a1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890365-t5r5d" Oct 31 04:45:00.327617 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.327583 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6171ff86-f98a-4575-834d-2f6bd135424d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890365-74cxv\" (UID: \"6171ff86-f98a-4575-834d-2f6bd135424d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890365-74cxv" Oct 31 04:45:00.329286 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.329211 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890365-x6d4s"] Oct 31 04:45:00.329523 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.329497 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-031745-29890365-grgsg" Oct 31 04:45:00.336103 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.336072 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890365-xf5cz"] Oct 31 04:45:00.342852 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.339611 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890365-x6d4s" Oct 31 04:45:00.432429 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.417838 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w2cw\" (UniqueName: \"kubernetes.io/projected/6171ff86-f98a-4575-834d-2f6bd135424d-kube-api-access-8w2cw\") pod \"maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890365-74cxv\" (UID: \"6171ff86-f98a-4575-834d-2f6bd135424d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890365-74cxv" Oct 31 04:45:00.432429 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.431793 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrw28\" (UniqueName: \"kubernetes.io/projected/19237217-929c-4b26-ac1f-fb27745d6dab-kube-api-access-lrw28\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890365-jdjqt\" (UID: \"19237217-929c-4b26-ac1f-fb27745d6dab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890365-jdjqt" Oct 31 04:45:00.447396 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.447347 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19237217-929c-4b26-ac1f-fb27745d6dab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890365-jdjqt\" (UID: \"19237217-929c-4b26-ac1f-fb27745d6dab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890365-jdjqt" Oct 31 04:45:00.447612 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.447590 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/681c73ad-b2a1-458f-b37a-cc4e9e198926-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-afc519-29890365-f7zkm\" (UID: \"681c73ad-b2a1-458f-b37a-cc4e9e198926\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890365-f7zkm" Oct 31 04:45:00.447693 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.447676 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc5fg\" (UniqueName: \"kubernetes.io/projected/681c73ad-b2a1-458f-b37a-cc4e9e198926-kube-api-access-hc5fg\") pod \"maas-api-key-cleanup-e2e-extmod-a-afc519-29890365-f7zkm\" (UID: \"681c73ad-b2a1-458f-b37a-cc4e9e198926\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890365-f7zkm" Oct 31 04:45:00.457421 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.457278 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890365-xf5cz" Oct 31 04:45:00.457624 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.457577 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-567ca6-29890365-pbxxk"] Oct 31 04:45:00.461652 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.461613 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4sdj\" (UniqueName: \"kubernetes.io/projected/e282b47a-b89e-48ae-88bf-347945705a1d-kube-api-access-r4sdj\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890365-t5r5d\" (UID: \"e282b47a-b89e-48ae-88bf-347945705a1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890365-t5r5d" Oct 31 04:45:00.463948 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.463908 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890365-74cxv"] Oct 31 04:45:00.464243 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.464227 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890365-jdjqt"] Oct 31 04:45:00.464398 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.464386 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-654741-29890365-fd9cp"] Oct 31 04:45:00.464713 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.464030 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-567ca6-29890365-pbxxk" Oct 31 04:45:00.484500 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.484457 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890365-t5r5d"] Oct 31 04:45:00.484793 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.484768 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890365-wwx6m"] Oct 31 04:45:00.508752 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.508709 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-654741-29890365-fd9cp" Oct 31 04:45:00.517546 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.517515 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890365-wwx6m" Oct 31 04:45:00.518047 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.518023 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890365-f7zkm"] Oct 31 04:45:00.526680 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.526230 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-031745-29890365-grgsg"] Oct 31 04:45:00.533805 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.533712 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890365-x6d4s"] Oct 31 04:45:00.540831 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.540788 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890365-5h7hv"] Oct 31 04:45:00.549794 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.549755 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890365-spg6x"] Oct 31 04:45:00.550055 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.549970 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc5fg\" (UniqueName: \"kubernetes.io/projected/681c73ad-b2a1-458f-b37a-cc4e9e198926-kube-api-access-hc5fg\") pod \"maas-api-key-cleanup-e2e-extmod-a-afc519-29890365-f7zkm\" (UID: \"681c73ad-b2a1-458f-b37a-cc4e9e198926\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890365-f7zkm" Oct 31 04:45:00.550055 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.550034 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9tv5\" (UniqueName: \"kubernetes.io/projected/a53b729a-b280-41d4-8661-44826b486246-kube-api-access-q9tv5\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890365-xf5cz\" (UID: \"a53b729a-b280-41d4-8661-44826b486246\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890365-xf5cz" Oct 31 04:45:00.550130 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.550068 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/668f2b2b-4d01-4054-8ca0-16a4b0b4dcdd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890365-x6d4s\" (UID: \"668f2b2b-4d01-4054-8ca0-16a4b0b4dcdd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890365-x6d4s" Oct 31 04:45:00.550130 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.550098 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrw28\" (UniqueName: \"kubernetes.io/projected/19237217-929c-4b26-ac1f-fb27745d6dab-kube-api-access-lrw28\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890365-jdjqt\" (UID: \"19237217-929c-4b26-ac1f-fb27745d6dab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890365-jdjqt" Oct 31 04:45:00.550201 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.550143 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a53b729a-b280-41d4-8661-44826b486246-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890365-xf5cz\" (UID: \"a53b729a-b280-41d4-8661-44826b486246\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890365-xf5cz" Oct 31 04:45:00.550238 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.550202 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a9a0d3a-eced-4fd8-85fa-7c08b317c39f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-031745-29890365-grgsg\" (UID: \"0a9a0d3a-eced-4fd8-85fa-7c08b317c39f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-031745-29890365-grgsg" Oct 31 04:45:00.550273 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.550238 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr22n\" (UniqueName: \"kubernetes.io/projected/0a9a0d3a-eced-4fd8-85fa-7c08b317c39f-kube-api-access-jr22n\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-031745-29890365-grgsg\" (UID: \"0a9a0d3a-eced-4fd8-85fa-7c08b317c39f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-031745-29890365-grgsg" Oct 31 04:45:00.550313 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.550280 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jrzs\" (UniqueName: \"kubernetes.io/projected/d992b6ec-1a57-4ca9-af09-370245d1ccfe-kube-api-access-4jrzs\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890365-wwx6m\" (UID: \"d992b6ec-1a57-4ca9-af09-370245d1ccfe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890365-wwx6m" Oct 31 04:45:00.550393 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.550371 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqp9r\" (UniqueName: \"kubernetes.io/projected/668f2b2b-4d01-4054-8ca0-16a4b0b4dcdd-kube-api-access-rqp9r\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890365-x6d4s\" (UID: \"668f2b2b-4d01-4054-8ca0-16a4b0b4dcdd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890365-x6d4s" Oct 31 04:45:00.550452 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.550436 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19237217-929c-4b26-ac1f-fb27745d6dab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890365-jdjqt\" (UID: \"19237217-929c-4b26-ac1f-fb27745d6dab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890365-jdjqt" Oct 31 04:45:00.550523 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.550504 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vl29\" (UniqueName: \"kubernetes.io/projected/2fc510cb-aae5-4d8c-a612-2270ccbbf432-kube-api-access-9vl29\") pod \"maas-api-key-cleanup-e2e-extmod-a-654741-29890365-fd9cp\" (UID: \"2fc510cb-aae5-4d8c-a612-2270ccbbf432\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-654741-29890365-fd9cp" Oct 31 04:45:00.550606 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.550588 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6sgb\" (UniqueName: \"kubernetes.io/projected/b5c95c35-2055-4d02-af07-fc3574c29ecf-kube-api-access-v6sgb\") pod \"maas-api-key-cleanup-e2e-extmod-b-567ca6-29890365-pbxxk\" (UID: \"b5c95c35-2055-4d02-af07-fc3574c29ecf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-567ca6-29890365-pbxxk" Oct 31 04:45:00.550657 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.550642 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5c95c35-2055-4d02-af07-fc3574c29ecf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-567ca6-29890365-pbxxk\" (UID: \"b5c95c35-2055-4d02-af07-fc3574c29ecf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-567ca6-29890365-pbxxk" Oct 31 04:45:00.550715 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.550700 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fc510cb-aae5-4d8c-a612-2270ccbbf432-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-654741-29890365-fd9cp\" (UID: \"2fc510cb-aae5-4d8c-a612-2270ccbbf432\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-654741-29890365-fd9cp" Oct 31 04:45:00.550814 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.550797 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d992b6ec-1a57-4ca9-af09-370245d1ccfe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890365-wwx6m\" (UID: \"d992b6ec-1a57-4ca9-af09-370245d1ccfe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890365-wwx6m" Oct 31 04:45:00.550895 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.550880 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/681c73ad-b2a1-458f-b37a-cc4e9e198926-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-afc519-29890365-f7zkm\" (UID: \"681c73ad-b2a1-458f-b37a-cc4e9e198926\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890365-f7zkm" Oct 31 04:45:00.551923 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.551900 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890365-5h7hv" Oct 31 04:45:00.552157 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.552135 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/681c73ad-b2a1-458f-b37a-cc4e9e198926-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-afc519-29890365-f7zkm\" (UID: \"681c73ad-b2a1-458f-b37a-cc4e9e198926\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890365-f7zkm" Oct 31 04:45:00.552303 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.552284 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19237217-929c-4b26-ac1f-fb27745d6dab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890365-jdjqt\" (UID: \"19237217-929c-4b26-ac1f-fb27745d6dab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890365-jdjqt" Oct 31 04:45:00.558144 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.558114 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890365-cnzxg"] Oct 31 04:45:00.558297 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.558278 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890365-spg6x" Oct 31 04:45:00.564621 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.564588 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890365-t5r5d" Oct 31 04:45:00.568127 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.568093 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-654741-29890365-fd9cp"] Oct 31 04:45:00.568306 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.568146 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890365-spg6x"] Oct 31 04:45:00.568444 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.568421 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890365-cnzxg" Oct 31 04:45:00.601369 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.596446 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890365-cnzxg"] Oct 31 04:45:00.601369 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.596549 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890365-f74kl"] Oct 31 04:45:00.614418 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.611738 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890365-f74kl" Oct 31 04:45:00.614418 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.612165 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890365-74cxv" Oct 31 04:45:00.620697 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.619985 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrw28\" (UniqueName: \"kubernetes.io/projected/19237217-929c-4b26-ac1f-fb27745d6dab-kube-api-access-lrw28\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890365-jdjqt\" (UID: \"19237217-929c-4b26-ac1f-fb27745d6dab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890365-jdjqt" Oct 31 04:45:00.620697 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.620055 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890365-wwx6m"] Oct 31 04:45:00.633450 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.631515 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890365-rhn85"] Oct 31 04:45:00.638973 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.638939 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc5fg\" (UniqueName: \"kubernetes.io/projected/681c73ad-b2a1-458f-b37a-cc4e9e198926-kube-api-access-hc5fg\") pod \"maas-api-key-cleanup-e2e-extmod-a-afc519-29890365-f7zkm\" (UID: \"681c73ad-b2a1-458f-b37a-cc4e9e198926\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890365-f7zkm" Oct 31 04:45:00.639372 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.639117 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890365-xf5cz"] Oct 31 04:45:00.639372 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.639245 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890365-rhn85" Oct 31 04:45:00.644378 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.644019 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-567ca6-29890365-pbxxk"] Oct 31 04:45:00.649231 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.649183 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890365-xj5rc"] Oct 31 04:45:00.652478 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.652434 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr22n\" (UniqueName: \"kubernetes.io/projected/0a9a0d3a-eced-4fd8-85fa-7c08b317c39f-kube-api-access-jr22n\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-031745-29890365-grgsg\" (UID: \"0a9a0d3a-eced-4fd8-85fa-7c08b317c39f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-031745-29890365-grgsg" Oct 31 04:45:00.652646 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.652501 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7abcb04-58ef-4efd-a350-c8f5286ebc14-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-09a20a-29890365-f74kl\" (UID: \"b7abcb04-58ef-4efd-a350-c8f5286ebc14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890365-f74kl" Oct 31 04:45:00.652646 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.652552 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jrzs\" (UniqueName: \"kubernetes.io/projected/d992b6ec-1a57-4ca9-af09-370245d1ccfe-kube-api-access-4jrzs\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890365-wwx6m\" (UID: \"d992b6ec-1a57-4ca9-af09-370245d1ccfe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890365-wwx6m" Oct 31 04:45:00.652646 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.652592 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5j7q\" (UniqueName: \"kubernetes.io/projected/b7abcb04-58ef-4efd-a350-c8f5286ebc14-kube-api-access-q5j7q\") pod \"maas-api-key-cleanup-e2e-worker-w2-09a20a-29890365-f74kl\" (UID: \"b7abcb04-58ef-4efd-a350-c8f5286ebc14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890365-f74kl" Oct 31 04:45:00.652646 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.652632 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mjls\" (UniqueName: \"kubernetes.io/projected/e12c1431-1aa4-4c72-94a9-d3b6646ee4b2-kube-api-access-7mjls\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890365-spg6x\" (UID: \"e12c1431-1aa4-4c72-94a9-d3b6646ee4b2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890365-spg6x" Oct 31 04:45:00.652938 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.652696 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq5cb\" (UniqueName: \"kubernetes.io/projected/bc26f88e-b019-4540-a826-63fb1d624a8d-kube-api-access-jq5cb\") pod \"maas-api-key-cleanup-e2e-extmod-a-8973e2-29890365-5h7hv\" (UID: \"bc26f88e-b019-4540-a826-63fb1d624a8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890365-5h7hv" Oct 31 04:45:00.652938 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.652729 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqp9r\" (UniqueName: \"kubernetes.io/projected/668f2b2b-4d01-4054-8ca0-16a4b0b4dcdd-kube-api-access-rqp9r\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890365-x6d4s\" (UID: \"668f2b2b-4d01-4054-8ca0-16a4b0b4dcdd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890365-x6d4s" Oct 31 04:45:00.652938 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.652836 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vl29\" (UniqueName: \"kubernetes.io/projected/2fc510cb-aae5-4d8c-a612-2270ccbbf432-kube-api-access-9vl29\") pod \"maas-api-key-cleanup-e2e-extmod-a-654741-29890365-fd9cp\" (UID: \"2fc510cb-aae5-4d8c-a612-2270ccbbf432\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-654741-29890365-fd9cp" Oct 31 04:45:00.652938 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.652871 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6sgb\" (UniqueName: \"kubernetes.io/projected/b5c95c35-2055-4d02-af07-fc3574c29ecf-kube-api-access-v6sgb\") pod \"maas-api-key-cleanup-e2e-extmod-b-567ca6-29890365-pbxxk\" (UID: \"b5c95c35-2055-4d02-af07-fc3574c29ecf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-567ca6-29890365-pbxxk" Oct 31 04:45:00.652938 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.652919 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5c95c35-2055-4d02-af07-fc3574c29ecf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-567ca6-29890365-pbxxk\" (UID: \"b5c95c35-2055-4d02-af07-fc3574c29ecf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-567ca6-29890365-pbxxk" Oct 31 04:45:00.653201 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.652962 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/089f7eeb-81d7-4562-b906-da33b0fb5fa1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-439989-29890365-cnzxg\" (UID: \"089f7eeb-81d7-4562-b906-da33b0fb5fa1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890365-cnzxg" Oct 31 04:45:00.653201 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.653001 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e12c1431-1aa4-4c72-94a9-d3b6646ee4b2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890365-spg6x\" (UID: \"e12c1431-1aa4-4c72-94a9-d3b6646ee4b2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890365-spg6x" Oct 31 04:45:00.653201 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.653033 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fc510cb-aae5-4d8c-a612-2270ccbbf432-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-654741-29890365-fd9cp\" (UID: \"2fc510cb-aae5-4d8c-a612-2270ccbbf432\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-654741-29890365-fd9cp" Oct 31 04:45:00.653201 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.653092 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hmzg\" (UniqueName: \"kubernetes.io/projected/089f7eeb-81d7-4562-b906-da33b0fb5fa1-kube-api-access-8hmzg\") pod \"maas-api-key-cleanup-e2e-worker-w1-439989-29890365-cnzxg\" (UID: \"089f7eeb-81d7-4562-b906-da33b0fb5fa1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890365-cnzxg" Oct 31 04:45:00.653201 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.653161 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d992b6ec-1a57-4ca9-af09-370245d1ccfe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890365-wwx6m\" (UID: \"d992b6ec-1a57-4ca9-af09-370245d1ccfe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890365-wwx6m" Oct 31 04:45:00.653590 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.653228 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01758cb6-b948-46f8-a607-210d75bb7716-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-766728-29890365-rhn85\" (UID: \"01758cb6-b948-46f8-a607-210d75bb7716\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890365-rhn85" Oct 31 04:45:00.653590 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.653274 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc26f88e-b019-4540-a826-63fb1d624a8d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-8973e2-29890365-5h7hv\" (UID: \"bc26f88e-b019-4540-a826-63fb1d624a8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890365-5h7hv" Oct 31 04:45:00.653590 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.653421 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9tv5\" (UniqueName: \"kubernetes.io/projected/a53b729a-b280-41d4-8661-44826b486246-kube-api-access-q9tv5\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890365-xf5cz\" (UID: \"a53b729a-b280-41d4-8661-44826b486246\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890365-xf5cz" Oct 31 04:45:00.653590 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.653453 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/668f2b2b-4d01-4054-8ca0-16a4b0b4dcdd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890365-x6d4s\" (UID: \"668f2b2b-4d01-4054-8ca0-16a4b0b4dcdd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890365-x6d4s" Oct 31 04:45:00.653590 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.653513 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljm75\" (UniqueName: \"kubernetes.io/projected/01758cb6-b948-46f8-a607-210d75bb7716-kube-api-access-ljm75\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-766728-29890365-rhn85\" (UID: \"01758cb6-b948-46f8-a607-210d75bb7716\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890365-rhn85" Oct 31 04:45:00.653590 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.653551 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a53b729a-b280-41d4-8661-44826b486246-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890365-xf5cz\" (UID: \"a53b729a-b280-41d4-8661-44826b486246\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890365-xf5cz" Oct 31 04:45:00.653948 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.653641 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a9a0d3a-eced-4fd8-85fa-7c08b317c39f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-031745-29890365-grgsg\" (UID: \"0a9a0d3a-eced-4fd8-85fa-7c08b317c39f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-031745-29890365-grgsg" Oct 31 04:45:00.655663 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.655629 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a53b729a-b280-41d4-8661-44826b486246-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890365-xf5cz\" (UID: \"a53b729a-b280-41d4-8661-44826b486246\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890365-xf5cz" Oct 31 04:45:00.656205 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.656176 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5c95c35-2055-4d02-af07-fc3574c29ecf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-567ca6-29890365-pbxxk\" (UID: \"b5c95c35-2055-4d02-af07-fc3574c29ecf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-567ca6-29890365-pbxxk" Oct 31 04:45:00.656288 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.656218 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fc510cb-aae5-4d8c-a612-2270ccbbf432-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-654741-29890365-fd9cp\" (UID: \"2fc510cb-aae5-4d8c-a612-2270ccbbf432\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-654741-29890365-fd9cp" Oct 31 04:45:00.657051 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.656662 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/668f2b2b-4d01-4054-8ca0-16a4b0b4dcdd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890365-x6d4s\" (UID: \"668f2b2b-4d01-4054-8ca0-16a4b0b4dcdd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890365-x6d4s" Oct 31 04:45:00.657051 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.656913 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890365-xj5rc" Oct 31 04:45:00.657051 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.657006 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d992b6ec-1a57-4ca9-af09-370245d1ccfe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890365-wwx6m\" (UID: \"d992b6ec-1a57-4ca9-af09-370245d1ccfe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890365-wwx6m" Oct 31 04:45:00.657300 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.657050 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a9a0d3a-eced-4fd8-85fa-7c08b317c39f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-031745-29890365-grgsg\" (UID: \"0a9a0d3a-eced-4fd8-85fa-7c08b317c39f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-031745-29890365-grgsg" Oct 31 04:45:00.658288 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.658263 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890365-5h7hv"] Oct 31 04:45:00.719698 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.717790 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9tv5\" (UniqueName: \"kubernetes.io/projected/a53b729a-b280-41d4-8661-44826b486246-kube-api-access-q9tv5\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890365-xf5cz\" (UID: \"a53b729a-b280-41d4-8661-44826b486246\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890365-xf5cz" Oct 31 04:45:00.720752 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.720715 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vl29\" (UniqueName: \"kubernetes.io/projected/2fc510cb-aae5-4d8c-a612-2270ccbbf432-kube-api-access-9vl29\") pod \"maas-api-key-cleanup-e2e-extmod-a-654741-29890365-fd9cp\" (UID: \"2fc510cb-aae5-4d8c-a612-2270ccbbf432\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-654741-29890365-fd9cp" Oct 31 04:45:00.722774 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.721347 5019 patch_prober.go:29] interesting pod/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body= Oct 31 04:45:00.722774 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.721411 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf" podUID="7ce5abc5-55a0-4b15-80a6-7e21d360e389" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" Oct 31 04:45:00.722774 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.722581 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890365-rhn85"] Oct 31 04:45:00.725225 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.725193 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6sgb\" (UniqueName: \"kubernetes.io/projected/b5c95c35-2055-4d02-af07-fc3574c29ecf-kube-api-access-v6sgb\") pod \"maas-api-key-cleanup-e2e-extmod-b-567ca6-29890365-pbxxk\" (UID: \"b5c95c35-2055-4d02-af07-fc3574c29ecf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-567ca6-29890365-pbxxk" Oct 31 04:45:00.725225 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.725213 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqp9r\" (UniqueName: \"kubernetes.io/projected/668f2b2b-4d01-4054-8ca0-16a4b0b4dcdd-kube-api-access-rqp9r\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890365-x6d4s\" (UID: \"668f2b2b-4d01-4054-8ca0-16a4b0b4dcdd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890365-x6d4s" Oct 31 04:45:00.725907 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.725884 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-654741-29890365-fd9cp" Oct 31 04:45:00.728164 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.728103 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr22n\" (UniqueName: \"kubernetes.io/projected/0a9a0d3a-eced-4fd8-85fa-7c08b317c39f-kube-api-access-jr22n\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-031745-29890365-grgsg\" (UID: \"0a9a0d3a-eced-4fd8-85fa-7c08b317c39f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-031745-29890365-grgsg" Oct 31 04:45:00.729232 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.729155 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890365-f74kl"] Oct 31 04:45:00.730151 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.730068 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jrzs\" (UniqueName: \"kubernetes.io/projected/d992b6ec-1a57-4ca9-af09-370245d1ccfe-kube-api-access-4jrzs\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890365-wwx6m\" (UID: \"d992b6ec-1a57-4ca9-af09-370245d1ccfe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890365-wwx6m" Oct 31 04:45:00.744753 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.744710 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890365-xj5rc"] Oct 31 04:45:00.745696 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.745404 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890365-jdjqt" Oct 31 04:45:00.767670 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.767631 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890365-f7zkm" Oct 31 04:45:00.771586 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.770322 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5j7q\" (UniqueName: \"kubernetes.io/projected/b7abcb04-58ef-4efd-a350-c8f5286ebc14-kube-api-access-q5j7q\") pod \"maas-api-key-cleanup-e2e-worker-w2-09a20a-29890365-f74kl\" (UID: \"b7abcb04-58ef-4efd-a350-c8f5286ebc14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890365-f74kl" Oct 31 04:45:00.771763 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.771654 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mjls\" (UniqueName: \"kubernetes.io/projected/e12c1431-1aa4-4c72-94a9-d3b6646ee4b2-kube-api-access-7mjls\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890365-spg6x\" (UID: \"e12c1431-1aa4-4c72-94a9-d3b6646ee4b2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890365-spg6x" Oct 31 04:45:00.771763 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.771743 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq5cb\" (UniqueName: \"kubernetes.io/projected/bc26f88e-b019-4540-a826-63fb1d624a8d-kube-api-access-jq5cb\") pod \"maas-api-key-cleanup-e2e-extmod-a-8973e2-29890365-5h7hv\" (UID: \"bc26f88e-b019-4540-a826-63fb1d624a8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890365-5h7hv" Oct 31 04:45:00.771945 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.771925 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99be4628-816c-46f6-abcf-9de2a8452895-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-4c2387-29890365-xj5rc\" (UID: \"99be4628-816c-46f6-abcf-9de2a8452895\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890365-xj5rc" Oct 31 04:45:00.772011 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.771994 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/089f7eeb-81d7-4562-b906-da33b0fb5fa1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-439989-29890365-cnzxg\" (UID: \"089f7eeb-81d7-4562-b906-da33b0fb5fa1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890365-cnzxg" Oct 31 04:45:00.772063 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.772044 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e12c1431-1aa4-4c72-94a9-d3b6646ee4b2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890365-spg6x\" (UID: \"e12c1431-1aa4-4c72-94a9-d3b6646ee4b2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890365-spg6x" Oct 31 04:45:00.772152 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.772133 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hmzg\" (UniqueName: \"kubernetes.io/projected/089f7eeb-81d7-4562-b906-da33b0fb5fa1-kube-api-access-8hmzg\") pod \"maas-api-key-cleanup-e2e-worker-w1-439989-29890365-cnzxg\" (UID: \"089f7eeb-81d7-4562-b906-da33b0fb5fa1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890365-cnzxg" Oct 31 04:45:00.772259 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.772241 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01758cb6-b948-46f8-a607-210d75bb7716-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-766728-29890365-rhn85\" (UID: \"01758cb6-b948-46f8-a607-210d75bb7716\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890365-rhn85" Oct 31 04:45:00.772319 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.772304 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc26f88e-b019-4540-a826-63fb1d624a8d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-8973e2-29890365-5h7hv\" (UID: \"bc26f88e-b019-4540-a826-63fb1d624a8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890365-5h7hv" Oct 31 04:45:00.795445 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.772478 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8hc9\" (UniqueName: \"kubernetes.io/projected/99be4628-816c-46f6-abcf-9de2a8452895-kube-api-access-s8hc9\") pod \"maas-api-key-cleanup-e2e-worker-w3-4c2387-29890365-xj5rc\" (UID: \"99be4628-816c-46f6-abcf-9de2a8452895\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890365-xj5rc" Oct 31 04:45:00.795445 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.772557 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljm75\" (UniqueName: \"kubernetes.io/projected/01758cb6-b948-46f8-a607-210d75bb7716-kube-api-access-ljm75\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-766728-29890365-rhn85\" (UID: \"01758cb6-b948-46f8-a607-210d75bb7716\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890365-rhn85" Oct 31 04:45:00.795445 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.772699 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7abcb04-58ef-4efd-a350-c8f5286ebc14-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-09a20a-29890365-f74kl\" (UID: \"b7abcb04-58ef-4efd-a350-c8f5286ebc14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890365-f74kl" Oct 31 04:45:00.795445 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.774815 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e12c1431-1aa4-4c72-94a9-d3b6646ee4b2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890365-spg6x\" (UID: \"e12c1431-1aa4-4c72-94a9-d3b6646ee4b2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890365-spg6x" Oct 31 04:45:00.795445 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.777044 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/089f7eeb-81d7-4562-b906-da33b0fb5fa1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-439989-29890365-cnzxg\" (UID: \"089f7eeb-81d7-4562-b906-da33b0fb5fa1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890365-cnzxg" Oct 31 04:45:00.802036 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.797987 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01758cb6-b948-46f8-a607-210d75bb7716-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-766728-29890365-rhn85\" (UID: \"01758cb6-b948-46f8-a607-210d75bb7716\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890365-rhn85" Oct 31 04:45:00.816907 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.803021 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7abcb04-58ef-4efd-a350-c8f5286ebc14-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-09a20a-29890365-f74kl\" (UID: \"b7abcb04-58ef-4efd-a350-c8f5286ebc14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890365-f74kl" Oct 31 04:45:00.817943 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.817844 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc26f88e-b019-4540-a826-63fb1d624a8d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-8973e2-29890365-5h7hv\" (UID: \"bc26f88e-b019-4540-a826-63fb1d624a8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890365-5h7hv" Oct 31 04:45:00.893428 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.834524 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-031745-29890365-grgsg" Oct 31 04:45:00.893428 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.848752 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890365-x6d4s" Oct 31 04:45:00.893428 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.850928 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljm75\" (UniqueName: \"kubernetes.io/projected/01758cb6-b948-46f8-a607-210d75bb7716-kube-api-access-ljm75\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-766728-29890365-rhn85\" (UID: \"01758cb6-b948-46f8-a607-210d75bb7716\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890365-rhn85" Oct 31 04:45:00.893428 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.853175 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq5cb\" (UniqueName: \"kubernetes.io/projected/bc26f88e-b019-4540-a826-63fb1d624a8d-kube-api-access-jq5cb\") pod \"maas-api-key-cleanup-e2e-extmod-a-8973e2-29890365-5h7hv\" (UID: \"bc26f88e-b019-4540-a826-63fb1d624a8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890365-5h7hv" Oct 31 04:45:00.893428 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.861925 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890365-wwx6m" Oct 31 04:45:00.893428 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.863680 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890365-xf5cz" Oct 31 04:45:00.893428 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.877819 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hmzg\" (UniqueName: \"kubernetes.io/projected/089f7eeb-81d7-4562-b906-da33b0fb5fa1-kube-api-access-8hmzg\") pod \"maas-api-key-cleanup-e2e-worker-w1-439989-29890365-cnzxg\" (UID: \"089f7eeb-81d7-4562-b906-da33b0fb5fa1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890365-cnzxg" Oct 31 04:45:00.893428 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.881553 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99be4628-816c-46f6-abcf-9de2a8452895-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-4c2387-29890365-xj5rc\" (UID: \"99be4628-816c-46f6-abcf-9de2a8452895\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890365-xj5rc" Oct 31 04:45:00.893428 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.881742 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8hc9\" (UniqueName: \"kubernetes.io/projected/99be4628-816c-46f6-abcf-9de2a8452895-kube-api-access-s8hc9\") pod \"maas-api-key-cleanup-e2e-worker-w3-4c2387-29890365-xj5rc\" (UID: \"99be4628-816c-46f6-abcf-9de2a8452895\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890365-xj5rc" Oct 31 04:45:00.893428 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.882922 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99be4628-816c-46f6-abcf-9de2a8452895-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-4c2387-29890365-xj5rc\" (UID: \"99be4628-816c-46f6-abcf-9de2a8452895\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890365-xj5rc" Oct 31 04:45:00.911665 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.911576 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mjls\" (UniqueName: \"kubernetes.io/projected/e12c1431-1aa4-4c72-94a9-d3b6646ee4b2-kube-api-access-7mjls\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890365-spg6x\" (UID: \"e12c1431-1aa4-4c72-94a9-d3b6646ee4b2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890365-spg6x" Oct 31 04:45:00.912989 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.912950 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5j7q\" (UniqueName: \"kubernetes.io/projected/b7abcb04-58ef-4efd-a350-c8f5286ebc14-kube-api-access-q5j7q\") pod \"maas-api-key-cleanup-e2e-worker-w2-09a20a-29890365-f74kl\" (UID: \"b7abcb04-58ef-4efd-a350-c8f5286ebc14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890365-f74kl" Oct 31 04:45:00.930915 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.930671 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890365-5h7hv" Oct 31 04:45:00.940829 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.938737 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-567ca6-29890365-pbxxk" Oct 31 04:45:00.948865 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.946666 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890365-spg6x" Oct 31 04:45:00.948865 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:00.948399 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8hc9\" (UniqueName: \"kubernetes.io/projected/99be4628-816c-46f6-abcf-9de2a8452895-kube-api-access-s8hc9\") pod \"maas-api-key-cleanup-e2e-worker-w3-4c2387-29890365-xj5rc\" (UID: \"99be4628-816c-46f6-abcf-9de2a8452895\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890365-xj5rc" Oct 31 04:45:01.148991 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:01.148953 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890365-cnzxg" Oct 31 04:45:01.216812 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:01.216005 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890365-f74kl" Oct 31 04:45:01.533813 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:01.324793 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890365-rhn85" Oct 31 04:45:01.634719 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:01.634686 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890365-xj5rc" Oct 31 04:45:01.740443 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:01.738595 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b66kbvh"] Oct 31 04:45:01.742371 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:01.742342 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b66kbvh" Oct 31 04:45:01.742640 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:01.742616 5019 patch_prober.go:29] interesting pod/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body= Oct 31 04:45:01.742797 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:01.742771 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf" podUID="7ce5abc5-55a0-4b15-80a6-7e21d360e389" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" Oct 31 04:45:01.862604 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:01.862551 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890365-t5r5d"] Oct 31 04:45:01.943090 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:45:01.942137 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode282b47a_b89e_48ae_88bf_347945705a1d.slice/crio-eefd7160bd6f80cca3e25e453383e3d1a1e134abd440d87424219edd605c239f WatchSource:0}: Error finding container eefd7160bd6f80cca3e25e453383e3d1a1e134abd440d87424219edd605c239f: Status 404 returned error can't find the container with id eefd7160bd6f80cca3e25e453383e3d1a1e134abd440d87424219edd605c239f Oct 31 04:45:01.945077 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:01.944999 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890365-74cxv"] Oct 31 04:45:01.954261 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:01.954213 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890365-f7zkm"] Oct 31 04:45:01.965025 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:45:01.961411 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod681c73ad_b2a1_458f_b37a_cc4e9e198926.slice/crio-6193130c65e7a6a5fe7ba135caceb90c92d165281e6786eadfb587fd2d0c303c WatchSource:0}: Error finding container 6193130c65e7a6a5fe7ba135caceb90c92d165281e6786eadfb587fd2d0c303c: Status 404 returned error can't find the container with id 6193130c65e7a6a5fe7ba135caceb90c92d165281e6786eadfb587fd2d0c303c Oct 31 04:45:02.109164 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:02.086910 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890365-wwx6m"] Oct 31 04:45:02.109164 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:02.095072 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890365-jdjqt"] Oct 31 04:45:02.265582 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:02.264736 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-654741-29890365-fd9cp"] Oct 31 04:45:02.542600 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:02.541451 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-031745-29890365-grgsg"] Oct 31 04:45:02.548786 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:02.548751 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890365-x6d4s"] Oct 31 04:45:02.627128 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:02.627080 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890365-74cxv" event={"ID":"6171ff86-f98a-4575-834d-2f6bd135424d","Type":"ContainerStarted","Data":"6b2de0ec0aafe7380dee186b5ed7b37d9c4a49e7a4449dd617cb86a643a021d7"} Oct 31 04:45:02.635360 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:02.632240 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890365-jdjqt" event={"ID":"19237217-929c-4b26-ac1f-fb27745d6dab","Type":"ContainerStarted","Data":"207b1bd41b0b17acf31881e10772678f6609ca6a17efc7119500147ce781d487"} Oct 31 04:45:02.635360 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:02.633992 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-031745-29890365-grgsg" event={"ID":"0a9a0d3a-eced-4fd8-85fa-7c08b317c39f","Type":"ContainerStarted","Data":"aacf4c26761fd7e45fe5990cfc3b82dfe3dfc2434cbeb3d6d4c435ae60dd0d7e"} Oct 31 04:45:02.639355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:02.635619 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-654741-29890365-fd9cp" event={"ID":"2fc510cb-aae5-4d8c-a612-2270ccbbf432","Type":"ContainerStarted","Data":"429b6c93d1beb9894d66acc8a1e46ac6e325172f095a9e377e8c14f7edeec0ea"} Oct 31 04:45:02.639355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:02.638865 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890365-f7zkm" event={"ID":"681c73ad-b2a1-458f-b37a-cc4e9e198926","Type":"ContainerStarted","Data":"6193130c65e7a6a5fe7ba135caceb90c92d165281e6786eadfb587fd2d0c303c"} Oct 31 04:45:02.654619 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:02.654573 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890365-t5r5d" event={"ID":"e282b47a-b89e-48ae-88bf-347945705a1d","Type":"ContainerStarted","Data":"eefd7160bd6f80cca3e25e453383e3d1a1e134abd440d87424219edd605c239f"} Oct 31 04:45:02.657661 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:02.657622 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890365-x6d4s" event={"ID":"668f2b2b-4d01-4054-8ca0-16a4b0b4dcdd","Type":"ContainerStarted","Data":"d242716c53eedbcf4cb962a5c7e0f339fa5aeda65b65d1bf630495a0335dcd58"} Oct 31 04:45:02.660111 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:02.660071 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890365-wwx6m" event={"ID":"d992b6ec-1a57-4ca9-af09-370245d1ccfe","Type":"ContainerStarted","Data":"1b8b3d4a30752a26c59ec536352859844016102395c49e4620992166908d732d"} Oct 31 04:45:02.696922 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:02.696881 5019 patch_prober.go:29] interesting pod/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body= Oct 31 04:45:02.701645 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:02.698197 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf" podUID="7ce5abc5-55a0-4b15-80a6-7e21d360e389" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" Oct 31 04:45:02.720438 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:02.720391 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-567ca6-29890365-pbxxk"] Oct 31 04:45:02.844968 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:02.844930 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890365-xf5cz"] Oct 31 04:45:02.851721 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:45:02.850585 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda53b729a_b280_41d4_8661_44826b486246.slice/crio-f232f751ee6a94eb689bf5071f8f6c72b9c2e248537674b56baa82ebcef3fdb5 WatchSource:0}: Error finding container f232f751ee6a94eb689bf5071f8f6c72b9c2e248537674b56baa82ebcef3fdb5: Status 404 returned error can't find the container with id f232f751ee6a94eb689bf5071f8f6c72b9c2e248537674b56baa82ebcef3fdb5 Oct 31 04:45:02.916191 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:02.916153 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890365-rhn85"] Oct 31 04:45:02.924591 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:02.924561 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890365-5h7hv"] Oct 31 04:45:02.945587 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:02.945554 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890365-spg6x"] Oct 31 04:45:03.015013 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:03.014908 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890365-xj5rc"] Oct 31 04:45:03.019695 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:03.019672 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890365-f74kl"] Oct 31 04:45:03.024473 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:03.024447 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890365-cnzxg"] Oct 31 04:45:03.115141 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:45:03.115035 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod089f7eeb_81d7_4562_b906_da33b0fb5fa1.slice/crio-6b600a1669134983f95f2d94a01870e70981fba49d4836ff9ba65ae9509ce77b WatchSource:0}: Error finding container 6b600a1669134983f95f2d94a01870e70981fba49d4836ff9ba65ae9509ce77b: Status 404 returned error can't find the container with id 6b600a1669134983f95f2d94a01870e70981fba49d4836ff9ba65ae9509ce77b Oct 31 04:45:03.116899 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:45:03.116867 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99be4628_816c_46f6_abcf_9de2a8452895.slice/crio-c01851506533c6030e601a61c08cc6a19fd0bb046042ca7f4e5d2449671347f7 WatchSource:0}: Error finding container c01851506533c6030e601a61c08cc6a19fd0bb046042ca7f4e5d2449671347f7: Status 404 returned error can't find the container with id c01851506533c6030e601a61c08cc6a19fd0bb046042ca7f4e5d2449671347f7 Oct 31 04:45:03.687234 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:03.686658 5019 patch_prober.go:29] interesting pod/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body= Oct 31 04:45:03.687234 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:03.686743 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf" podUID="7ce5abc5-55a0-4b15-80a6-7e21d360e389" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" Oct 31 04:45:03.719755 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:03.719716 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-654741-29890365-fd9cp" event={"ID":"2fc510cb-aae5-4d8c-a612-2270ccbbf432","Type":"ContainerStarted","Data":"46dd33effa75eacaebf0c28b3a7821f876a42e57304ea68f97b00ceecec7e31c"} Oct 31 04:45:03.727626 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:03.727287 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890365-f7zkm" event={"ID":"681c73ad-b2a1-458f-b37a-cc4e9e198926","Type":"ContainerStarted","Data":"e14613b52ef9112e9c900cc71923eb66877039bc4db7168a2639e0ea27d41b1f"} Oct 31 04:45:03.729496 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:03.729465 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-567ca6-29890365-pbxxk" event={"ID":"b5c95c35-2055-4d02-af07-fc3574c29ecf","Type":"ContainerStarted","Data":"8d4e54df4b39398ab7e23377317b6332b9fc2e0f6e3508d6e162381230970d22"} Oct 31 04:45:03.731554 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:03.731374 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890365-spg6x" event={"ID":"e12c1431-1aa4-4c72-94a9-d3b6646ee4b2","Type":"ContainerStarted","Data":"88cd50a8036975ca8bd0a4f62b3ec52457caa313421c4350f7e5761ff4611a9a"} Oct 31 04:45:03.732678 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:03.732647 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890365-t5r5d" event={"ID":"e282b47a-b89e-48ae-88bf-347945705a1d","Type":"ContainerStarted","Data":"29dad26c5456d090ff4d85fb8e5544ae6e40d79ec52e36601e2154f033badf33"} Oct 31 04:45:03.735216 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:03.735187 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890365-f74kl" event={"ID":"b7abcb04-58ef-4efd-a350-c8f5286ebc14","Type":"ContainerStarted","Data":"d57aa8ad8523ba955b03a2f29a105d27e01a3c75b4ae0c21a29c02afbc2c1cf8"} Oct 31 04:45:03.737264 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:03.737241 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890365-cnzxg" event={"ID":"089f7eeb-81d7-4562-b906-da33b0fb5fa1","Type":"ContainerStarted","Data":"6b600a1669134983f95f2d94a01870e70981fba49d4836ff9ba65ae9509ce77b"} Oct 31 04:45:03.739271 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:03.739244 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890365-5h7hv" event={"ID":"bc26f88e-b019-4540-a826-63fb1d624a8d","Type":"ContainerStarted","Data":"44b4aeac6e5030e002b2019bda1740d9448576bc7ea119af8e8166062b7a2731"} Oct 31 04:45:03.740693 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:03.740666 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890365-rhn85" event={"ID":"01758cb6-b948-46f8-a607-210d75bb7716","Type":"ContainerStarted","Data":"eddea1500a30031d941514f79da162d7c07432232915a722b3cd4240a8b27d23"} Oct 31 04:45:03.742669 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:03.742385 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890365-wwx6m" event={"ID":"d992b6ec-1a57-4ca9-af09-370245d1ccfe","Type":"ContainerStarted","Data":"e8779a53348e75d4857a839bbfeb0beea313466e097c2574a048ddf9c0af7e6b"} Oct 31 04:45:03.743719 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:03.743701 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890365-xj5rc" event={"ID":"99be4628-816c-46f6-abcf-9de2a8452895","Type":"ContainerStarted","Data":"c01851506533c6030e601a61c08cc6a19fd0bb046042ca7f4e5d2449671347f7"} Oct 31 04:45:03.746099 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:03.746078 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890365-74cxv" event={"ID":"6171ff86-f98a-4575-834d-2f6bd135424d","Type":"ContainerStarted","Data":"a34bb3052c949841523e2f707b861d65034fc30573070e3ffef7e8568e9b086d"} Oct 31 04:45:03.746638 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:03.746581 5019 scope.go:122] "RemoveContainer" containerID="a34bb3052c949841523e2f707b861d65034fc30573070e3ffef7e8568e9b086d" Oct 31 04:45:03.748151 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:03.748125 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890365-jdjqt" event={"ID":"19237217-929c-4b26-ac1f-fb27745d6dab","Type":"ContainerStarted","Data":"445c1608b0e271e773d364bee5474b9222baa719568207e3d8dbbadcad084ff4"} Oct 31 04:45:03.749748 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:03.749728 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890365-xf5cz" event={"ID":"a53b729a-b280-41d4-8661-44826b486246","Type":"ContainerStarted","Data":"f232f751ee6a94eb689bf5071f8f6c72b9c2e248537674b56baa82ebcef3fdb5"} Oct 31 04:45:03.763869 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:03.763816 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-654741-29890365-fd9cp" podStartSLOduration=3.763799661 podStartE2EDuration="3.763799661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:45:03.759580524 +0000 UTC m=+2458.515918567" watchObservedRunningTime="2026-10-31 04:45:03.763799661 +0000 UTC m=+2458.520137702" Oct 31 04:45:03.845731 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:03.845671 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890365-f7zkm" podStartSLOduration=3.845655039 podStartE2EDuration="3.845655039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:45:03.841867399 +0000 UTC m=+2458.598205439" watchObservedRunningTime="2026-10-31 04:45:03.845655039 +0000 UTC m=+2458.601993103" Oct 31 04:45:03.890544 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:03.888633 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890365-wwx6m" podStartSLOduration=3.8886122309999998 podStartE2EDuration="3.888612231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:45:03.887112737 +0000 UTC m=+2458.643450780" watchObservedRunningTime="2026-10-31 04:45:03.888612231 +0000 UTC m=+2458.644950273" Oct 31 04:45:03.931318 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:03.928728 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890365-t5r5d" podStartSLOduration=3.928712496 podStartE2EDuration="3.928712496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:45:03.927306655 +0000 UTC m=+2458.683644694" watchObservedRunningTime="2026-10-31 04:45:03.928712496 +0000 UTC m=+2458.685050535" Oct 31 04:45:04.005541 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:04.005484 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890365-jdjqt" podStartSLOduration=4.005466646 podStartE2EDuration="4.005466646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:45:03.965863528 +0000 UTC m=+2458.722201572" watchObservedRunningTime="2026-10-31 04:45:04.005466646 +0000 UTC m=+2458.761804722" Oct 31 04:45:04.679775 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:04.679656 5019 patch_prober.go:29] interesting pod/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body= Oct 31 04:45:04.679775 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:04.679716 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf" podUID="7ce5abc5-55a0-4b15-80a6-7e21d360e389" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" Oct 31 04:45:04.761638 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:04.761547 5019 generic.go:356] "Generic (PLEG): container finished" podID="19237217-929c-4b26-ac1f-fb27745d6dab" containerID="445c1608b0e271e773d364bee5474b9222baa719568207e3d8dbbadcad084ff4" exitCode=60 Oct 31 04:45:04.762035 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:04.761630 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890365-jdjqt" event={"ID":"19237217-929c-4b26-ac1f-fb27745d6dab","Type":"ContainerDied","Data":"445c1608b0e271e773d364bee5474b9222baa719568207e3d8dbbadcad084ff4"} Oct 31 04:45:04.762035 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:04.761960 5019 scope.go:122] "RemoveContainer" containerID="445c1608b0e271e773d364bee5474b9222baa719568207e3d8dbbadcad084ff4" Oct 31 04:45:04.764615 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:04.764586 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890365-xf5cz" event={"ID":"a53b729a-b280-41d4-8661-44826b486246","Type":"ContainerStarted","Data":"90842b342d746cdee675f7dab701a30f5088bfd7e45cb6fc67579f949452908e"} Oct 31 04:45:04.788358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:04.769003 5019 generic.go:356] "Generic (PLEG): container finished" podID="0a9a0d3a-eced-4fd8-85fa-7c08b317c39f" containerID="2fff6219a7a2997aeae1b5c3a08d0ef55b3d76d3b6733c72f59b18f695e0c209" exitCode=60 Oct 31 04:45:04.788358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:04.769088 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-031745-29890365-grgsg" event={"ID":"0a9a0d3a-eced-4fd8-85fa-7c08b317c39f","Type":"ContainerDied","Data":"2fff6219a7a2997aeae1b5c3a08d0ef55b3d76d3b6733c72f59b18f695e0c209"} Oct 31 04:45:04.788358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:04.784567 5019 scope.go:122] "RemoveContainer" containerID="2fff6219a7a2997aeae1b5c3a08d0ef55b3d76d3b6733c72f59b18f695e0c209" Oct 31 04:45:04.802841 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:04.802807 5019 generic.go:356] "Generic (PLEG): container finished" podID="2fc510cb-aae5-4d8c-a612-2270ccbbf432" containerID="46dd33effa75eacaebf0c28b3a7821f876a42e57304ea68f97b00ceecec7e31c" exitCode=60 Oct 31 04:45:04.802988 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:04.802897 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-654741-29890365-fd9cp" event={"ID":"2fc510cb-aae5-4d8c-a612-2270ccbbf432","Type":"ContainerDied","Data":"46dd33effa75eacaebf0c28b3a7821f876a42e57304ea68f97b00ceecec7e31c"} Oct 31 04:45:04.803371 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:04.803318 5019 scope.go:122] "RemoveContainer" containerID="46dd33effa75eacaebf0c28b3a7821f876a42e57304ea68f97b00ceecec7e31c" Oct 31 04:45:04.807753 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:04.807725 5019 generic.go:356] "Generic (PLEG): container finished" podID="681c73ad-b2a1-458f-b37a-cc4e9e198926" containerID="e14613b52ef9112e9c900cc71923eb66877039bc4db7168a2639e0ea27d41b1f" exitCode=60 Oct 31 04:45:04.807852 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:04.807807 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890365-f7zkm" event={"ID":"681c73ad-b2a1-458f-b37a-cc4e9e198926","Type":"ContainerDied","Data":"e14613b52ef9112e9c900cc71923eb66877039bc4db7168a2639e0ea27d41b1f"} Oct 31 04:45:04.808222 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:04.808202 5019 scope.go:122] "RemoveContainer" containerID="e14613b52ef9112e9c900cc71923eb66877039bc4db7168a2639e0ea27d41b1f" Oct 31 04:45:04.813581 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:04.813550 5019 generic.go:356] "Generic (PLEG): container finished" podID="b5c95c35-2055-4d02-af07-fc3574c29ecf" containerID="b50bb7f040e162636ba4bdb4fc1bf964c8911537017f76865216c76830620244" exitCode=60 Oct 31 04:45:04.813712 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:04.813651 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-567ca6-29890365-pbxxk" event={"ID":"b5c95c35-2055-4d02-af07-fc3574c29ecf","Type":"ContainerDied","Data":"b50bb7f040e162636ba4bdb4fc1bf964c8911537017f76865216c76830620244"} Oct 31 04:45:04.814623 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:04.814102 5019 scope.go:122] "RemoveContainer" containerID="b50bb7f040e162636ba4bdb4fc1bf964c8911537017f76865216c76830620244" Oct 31 04:45:04.819632 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:04.819556 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890365-spg6x" event={"ID":"e12c1431-1aa4-4c72-94a9-d3b6646ee4b2","Type":"ContainerStarted","Data":"a300bd76b5eea137c14bd913da24e8cca5c666a13d1ffc2a0d8ff9a7f254a871"} Oct 31 04:45:04.821871 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:04.821409 5019 generic.go:356] "Generic (PLEG): container finished" podID="e282b47a-b89e-48ae-88bf-347945705a1d" containerID="29dad26c5456d090ff4d85fb8e5544ae6e40d79ec52e36601e2154f033badf33" exitCode=60 Oct 31 04:45:04.821871 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:04.821493 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890365-t5r5d" event={"ID":"e282b47a-b89e-48ae-88bf-347945705a1d","Type":"ContainerDied","Data":"29dad26c5456d090ff4d85fb8e5544ae6e40d79ec52e36601e2154f033badf33"} Oct 31 04:45:04.821871 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:04.821829 5019 scope.go:122] "RemoveContainer" containerID="29dad26c5456d090ff4d85fb8e5544ae6e40d79ec52e36601e2154f033badf33" Oct 31 04:45:04.829220 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:04.829182 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890365-f74kl" event={"ID":"b7abcb04-58ef-4efd-a350-c8f5286ebc14","Type":"ContainerStarted","Data":"3c2505d98cae21bb48b61702985be8657e603aa908e049a36f11eeed52e7d1a7"} Oct 31 04:45:04.832345 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:04.832300 5019 generic.go:356] "Generic (PLEG): container finished" podID="668f2b2b-4d01-4054-8ca0-16a4b0b4dcdd" containerID="4a6dfb802a29ce3c6ef58ef6fa4849c69639adb8d4a1689b1d26361457da6b86" exitCode=60 Oct 31 04:45:04.832444 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:04.832404 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890365-x6d4s" event={"ID":"668f2b2b-4d01-4054-8ca0-16a4b0b4dcdd","Type":"ContainerDied","Data":"4a6dfb802a29ce3c6ef58ef6fa4849c69639adb8d4a1689b1d26361457da6b86"} Oct 31 04:45:04.832688 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:04.832671 5019 scope.go:122] "RemoveContainer" containerID="4a6dfb802a29ce3c6ef58ef6fa4849c69639adb8d4a1689b1d26361457da6b86" Oct 31 04:45:04.834984 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:04.834960 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890365-cnzxg" event={"ID":"089f7eeb-81d7-4562-b906-da33b0fb5fa1","Type":"ContainerStarted","Data":"03d2d6dbaeb701309f19958e780f9807f85f1dd8f1f661cbaed71c5aec72f4c1"} Oct 31 04:45:04.837481 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:04.837448 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890365-5h7hv" event={"ID":"bc26f88e-b019-4540-a826-63fb1d624a8d","Type":"ContainerStarted","Data":"51cac0cf5f16870608c609d75ed01737bf29b3c3ef903174f55c9b6d23d765bc"} Oct 31 04:45:04.844076 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:04.843963 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890365-rhn85" event={"ID":"01758cb6-b948-46f8-a607-210d75bb7716","Type":"ContainerStarted","Data":"8f926deb3ccf97b321ca815b8db5736cc2956a5b6c6113387df603f5b6f2345e"} Oct 31 04:45:04.845695 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:04.845672 5019 generic.go:356] "Generic (PLEG): container finished" podID="d992b6ec-1a57-4ca9-af09-370245d1ccfe" containerID="e8779a53348e75d4857a839bbfeb0beea313466e097c2574a048ddf9c0af7e6b" exitCode=60 Oct 31 04:45:04.845773 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:04.845731 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890365-wwx6m" event={"ID":"d992b6ec-1a57-4ca9-af09-370245d1ccfe","Type":"ContainerDied","Data":"e8779a53348e75d4857a839bbfeb0beea313466e097c2574a048ddf9c0af7e6b"} Oct 31 04:45:04.846042 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:04.846023 5019 scope.go:122] "RemoveContainer" containerID="e8779a53348e75d4857a839bbfeb0beea313466e097c2574a048ddf9c0af7e6b" Oct 31 04:45:04.848311 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:04.848285 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890365-xj5rc" event={"ID":"99be4628-816c-46f6-abcf-9de2a8452895","Type":"ContainerStarted","Data":"ea42f0ef09870f9ed8eda40f065500d3733ab490d0413120a8ed797f714e11f9"} Oct 31 04:45:04.850981 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:04.850955 5019 generic.go:356] "Generic (PLEG): container finished" podID="6171ff86-f98a-4575-834d-2f6bd135424d" containerID="a34bb3052c949841523e2f707b861d65034fc30573070e3ffef7e8568e9b086d" exitCode=60 Oct 31 04:45:04.851084 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:04.850996 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890365-74cxv" event={"ID":"6171ff86-f98a-4575-834d-2f6bd135424d","Type":"ContainerDied","Data":"a34bb3052c949841523e2f707b861d65034fc30573070e3ffef7e8568e9b086d"} Oct 31 04:45:04.851084 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:04.851035 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890365-74cxv" event={"ID":"6171ff86-f98a-4575-834d-2f6bd135424d","Type":"ContainerStarted","Data":"63cc09de94f00b6797af27c7c9c00c81205970dff40ce0ecf29257f810165454"} Oct 31 04:45:04.857007 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:04.856916 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890365-xf5cz" podStartSLOduration=4.856898376 podStartE2EDuration="4.856898376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:45:04.852660176 +0000 UTC m=+2459.608998216" watchObservedRunningTime="2026-10-31 04:45:04.856898376 +0000 UTC m=+2459.613236440" Oct 31 04:45:05.032382 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:05.032258 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890365-f74kl" podStartSLOduration=5.032239656 podStartE2EDuration="5.032239656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:45:05.026842584 +0000 UTC m=+2459.783180629" watchObservedRunningTime="2026-10-31 04:45:05.032239656 +0000 UTC m=+2459.788577764" Oct 31 04:45:05.176502 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:05.176428 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890365-cnzxg" podStartSLOduration=5.176407351 podStartE2EDuration="5.176407351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:45:05.176018142 +0000 UTC m=+2459.932356183" watchObservedRunningTime="2026-10-31 04:45:05.176407351 +0000 UTC m=+2459.932745393" Oct 31 04:45:05.224538 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:05.224449 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890365-xj5rc" podStartSLOduration=5.224432667 podStartE2EDuration="5.224432667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:45:05.218526356 +0000 UTC m=+2459.974864402" watchObservedRunningTime="2026-10-31 04:45:05.224432667 +0000 UTC m=+2459.980770700" Oct 31 04:45:05.267499 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:05.267426 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890365-spg6x" podStartSLOduration=5.267391892 podStartE2EDuration="5.267391892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:45:05.266362414 +0000 UTC m=+2460.022700471" watchObservedRunningTime="2026-10-31 04:45:05.267391892 +0000 UTC m=+2460.023730016" Oct 31 04:45:05.312315 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:05.312180 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890365-5h7hv" podStartSLOduration=5.312163098 podStartE2EDuration="5.312163098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:45:05.307421545 +0000 UTC m=+2460.063759606" watchObservedRunningTime="2026-10-31 04:45:05.312163098 +0000 UTC m=+2460.068501138" Oct 31 04:45:05.351978 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:05.351910 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890365-rhn85" podStartSLOduration=5.351893452 podStartE2EDuration="5.351893452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:45:05.346562862 +0000 UTC m=+2460.102900904" watchObservedRunningTime="2026-10-31 04:45:05.351893452 +0000 UTC m=+2460.108231491" Oct 31 04:45:05.390924 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:05.390851 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890365-74cxv" podStartSLOduration=5.390826097 podStartE2EDuration="5.390826097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:45:05.385877785 +0000 UTC m=+2460.142215832" watchObservedRunningTime="2026-10-31 04:45:05.390826097 +0000 UTC m=+2460.147164135" Oct 31 04:45:05.455552 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:05.455499 5019 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-ded435/agc-cred-43acb978be07: secret "agc-cred-43acb978be07" not found Oct 31 04:45:05.455552 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:05.455558 5019 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-ded435/agc-cred-7fd1d61db48f: secret "agc-cred-7fd1d61db48f" not found Oct 31 04:45:05.455913 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:05.455574 5019 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-b794w26n5: [secret "agc-cred-43acb978be07" not found, secret "agc-cred-7fd1d61db48f" not found] Oct 31 04:45:05.455913 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:05.455655 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5255293a-4f6d-48d0-b162-d5270e2d4b52-provider-credentials podName:5255293a-4f6d-48d0-b162-d5270e2d4b52 nodeName:}" failed. No retries permitted until 2026-10-31 04:45:05.955630581 +0000 UTC m=+2460.711968626 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5255293a-4f6d-48d0-b162-d5270e2d4b52-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-ded435-b794w26n5" (UID: "5255293a-4f6d-48d0-b162-d5270e2d4b52") : [secret "agc-cred-43acb978be07" not found, secret "agc-cred-7fd1d61db48f" not found] Oct 31 04:45:05.678588 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:05.678550 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf" Oct 31 04:45:05.679056 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:05.679027 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf" Oct 31 04:45:05.680019 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:05.679996 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf" Oct 31 04:45:05.867466 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:05.867366 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-031745-29890365-grgsg" event={"ID":"0a9a0d3a-eced-4fd8-85fa-7c08b317c39f","Type":"ContainerStarted","Data":"b89d781650a64de1497c734f2194c0d377a527170683ac44e41b55fc5fa1153c"} Oct 31 04:45:05.872692 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:05.872657 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-654741-29890365-fd9cp" event={"ID":"2fc510cb-aae5-4d8c-a612-2270ccbbf432","Type":"ContainerStarted","Data":"667d384355c86ef041aeb1f21583791296d4c936620f6b2176633c909f940c17"} Oct 31 04:45:05.876022 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:05.875984 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890365-f7zkm" event={"ID":"681c73ad-b2a1-458f-b37a-cc4e9e198926","Type":"ContainerStarted","Data":"f70b5cca24c60d6e0b85189092a7f43c1d21154486d89a497e314afdc41a423c"} Oct 31 04:45:05.895738 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:05.895705 5019 generic.go:356] "Generic (PLEG): container finished" podID="e12c1431-1aa4-4c72-94a9-d3b6646ee4b2" containerID="a300bd76b5eea137c14bd913da24e8cca5c666a13d1ffc2a0d8ff9a7f254a871" exitCode=60 Oct 31 04:45:05.896003 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:05.895986 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890365-spg6x" event={"ID":"e12c1431-1aa4-4c72-94a9-d3b6646ee4b2","Type":"ContainerDied","Data":"a300bd76b5eea137c14bd913da24e8cca5c666a13d1ffc2a0d8ff9a7f254a871"} Oct 31 04:45:05.898316 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:05.897388 5019 scope.go:122] "RemoveContainer" containerID="a300bd76b5eea137c14bd913da24e8cca5c666a13d1ffc2a0d8ff9a7f254a871" Oct 31 04:45:05.901108 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:05.901078 5019 generic.go:356] "Generic (PLEG): container finished" podID="b7abcb04-58ef-4efd-a350-c8f5286ebc14" containerID="3c2505d98cae21bb48b61702985be8657e603aa908e049a36f11eeed52e7d1a7" exitCode=60 Oct 31 04:45:05.901233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:05.901170 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890365-f74kl" event={"ID":"b7abcb04-58ef-4efd-a350-c8f5286ebc14","Type":"ContainerDied","Data":"3c2505d98cae21bb48b61702985be8657e603aa908e049a36f11eeed52e7d1a7"} Oct 31 04:45:05.901563 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:05.901543 5019 scope.go:122] "RemoveContainer" containerID="3c2505d98cae21bb48b61702985be8657e603aa908e049a36f11eeed52e7d1a7" Oct 31 04:45:05.910672 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:05.910640 5019 generic.go:356] "Generic (PLEG): container finished" podID="089f7eeb-81d7-4562-b906-da33b0fb5fa1" containerID="03d2d6dbaeb701309f19958e780f9807f85f1dd8f1f661cbaed71c5aec72f4c1" exitCode=60 Oct 31 04:45:05.910802 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:05.910790 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890365-cnzxg" event={"ID":"089f7eeb-81d7-4562-b906-da33b0fb5fa1","Type":"ContainerDied","Data":"03d2d6dbaeb701309f19958e780f9807f85f1dd8f1f661cbaed71c5aec72f4c1"} Oct 31 04:45:05.911273 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:05.911256 5019 scope.go:122] "RemoveContainer" containerID="03d2d6dbaeb701309f19958e780f9807f85f1dd8f1f661cbaed71c5aec72f4c1" Oct 31 04:45:05.914712 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:05.914660 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-031745-29890365-grgsg" podStartSLOduration=5.914642663 podStartE2EDuration="5.914642663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:45:05.908867669 +0000 UTC m=+2460.665205709" watchObservedRunningTime="2026-10-31 04:45:05.914642663 +0000 UTC m=+2460.670980704" Oct 31 04:45:05.914867 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:05.914825 5019 generic.go:356] "Generic (PLEG): container finished" podID="bc26f88e-b019-4540-a826-63fb1d624a8d" containerID="51cac0cf5f16870608c609d75ed01737bf29b3c3ef903174f55c9b6d23d765bc" exitCode=60 Oct 31 04:45:05.914930 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:05.914893 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890365-5h7hv" event={"ID":"bc26f88e-b019-4540-a826-63fb1d624a8d","Type":"ContainerDied","Data":"51cac0cf5f16870608c609d75ed01737bf29b3c3ef903174f55c9b6d23d765bc"} Oct 31 04:45:05.915279 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:05.915263 5019 scope.go:122] "RemoveContainer" containerID="51cac0cf5f16870608c609d75ed01737bf29b3c3ef903174f55c9b6d23d765bc" Oct 31 04:45:05.919050 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:05.919024 5019 generic.go:356] "Generic (PLEG): container finished" podID="01758cb6-b948-46f8-a607-210d75bb7716" containerID="8f926deb3ccf97b321ca815b8db5736cc2956a5b6c6113387df603f5b6f2345e" exitCode=60 Oct 31 04:45:05.919169 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:05.919124 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890365-rhn85" event={"ID":"01758cb6-b948-46f8-a607-210d75bb7716","Type":"ContainerDied","Data":"8f926deb3ccf97b321ca815b8db5736cc2956a5b6c6113387df603f5b6f2345e"} Oct 31 04:45:05.919873 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:05.919454 5019 scope.go:122] "RemoveContainer" containerID="8f926deb3ccf97b321ca815b8db5736cc2956a5b6c6113387df603f5b6f2345e" Oct 31 04:45:05.925827 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:05.923155 5019 generic.go:356] "Generic (PLEG): container finished" podID="99be4628-816c-46f6-abcf-9de2a8452895" containerID="ea42f0ef09870f9ed8eda40f065500d3733ab490d0413120a8ed797f714e11f9" exitCode=60 Oct 31 04:45:05.925827 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:05.923319 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890365-xj5rc" event={"ID":"99be4628-816c-46f6-abcf-9de2a8452895","Type":"ContainerDied","Data":"ea42f0ef09870f9ed8eda40f065500d3733ab490d0413120a8ed797f714e11f9"} Oct 31 04:45:05.925827 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:05.923552 5019 scope.go:122] "RemoveContainer" containerID="ea42f0ef09870f9ed8eda40f065500d3733ab490d0413120a8ed797f714e11f9" Oct 31 04:45:05.926870 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:05.926849 5019 generic.go:356] "Generic (PLEG): container finished" podID="6171ff86-f98a-4575-834d-2f6bd135424d" containerID="63cc09de94f00b6797af27c7c9c00c81205970dff40ce0ecf29257f810165454" exitCode=60 Oct 31 04:45:05.926962 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:05.926889 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890365-74cxv" event={"ID":"6171ff86-f98a-4575-834d-2f6bd135424d","Type":"ContainerDied","Data":"63cc09de94f00b6797af27c7c9c00c81205970dff40ce0ecf29257f810165454"} Oct 31 04:45:05.926962 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:05.926955 5019 scope.go:122] "RemoveContainer" containerID="a34bb3052c949841523e2f707b861d65034fc30573070e3ffef7e8568e9b086d" Oct 31 04:45:05.927386 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:05.927365 5019 scope.go:122] "RemoveContainer" containerID="63cc09de94f00b6797af27c7c9c00c81205970dff40ce0ecf29257f810165454" Oct 31 04:45:05.927847 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:05.927818 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890365-74cxv_odh-ai-gateway-infra(6171ff86-f98a-4575-834d-2f6bd135424d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890365-74cxv" podUID="6171ff86-f98a-4575-834d-2f6bd135424d" Oct 31 04:45:05.941755 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:05.941717 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890365-jdjqt" event={"ID":"19237217-929c-4b26-ac1f-fb27745d6dab","Type":"ContainerStarted","Data":"2965770faabe08de689fd9a5deafcb07784512bdec3e3c01afe60888690cd434"} Oct 31 04:45:05.944483 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:05.944451 5019 generic.go:356] "Generic (PLEG): container finished" podID="a53b729a-b280-41d4-8661-44826b486246" containerID="90842b342d746cdee675f7dab701a30f5088bfd7e45cb6fc67579f949452908e" exitCode=60 Oct 31 04:45:05.945940 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:05.945831 5019 scope.go:122] "RemoveContainer" containerID="90842b342d746cdee675f7dab701a30f5088bfd7e45cb6fc67579f949452908e" Oct 31 04:45:05.946280 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:05.946249 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890365-xf5cz" event={"ID":"a53b729a-b280-41d4-8661-44826b486246","Type":"ContainerDied","Data":"90842b342d746cdee675f7dab701a30f5088bfd7e45cb6fc67579f949452908e"} Oct 31 04:45:05.972485 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:05.972451 5019 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-ded435/agc-cred-43acb978be07: secret "agc-cred-43acb978be07" not found Oct 31 04:45:05.972710 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:05.972692 5019 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-ded435/agc-cred-7fd1d61db48f: secret "agc-cred-7fd1d61db48f" not found Oct 31 04:45:05.972824 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:05.972812 5019 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-b794w26n5: [secret "agc-cred-43acb978be07" not found, secret "agc-cred-7fd1d61db48f" not found] Oct 31 04:45:05.972982 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:05.972969 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5255293a-4f6d-48d0-b162-d5270e2d4b52-provider-credentials podName:5255293a-4f6d-48d0-b162-d5270e2d4b52 nodeName:}" failed. No retries permitted until 2026-10-31 04:45:06.972944337 +0000 UTC m=+2461.729282357 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5255293a-4f6d-48d0-b162-d5270e2d4b52-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-ded435-b794w26n5" (UID: "5255293a-4f6d-48d0-b162-d5270e2d4b52") : [secret "agc-cred-43acb978be07" not found, secret "agc-cred-7fd1d61db48f" not found] Oct 31 04:45:06.739185 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:06.739139 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b66kbvh" podUID="837f143a-072a-446d-9062-cb6f57eb4884" containerName="payload-processing" containerID="cri-o://8fb9d282e707fe200d7bc379420e5e3be964dda612aad5ecec9894948f61f498" gracePeriod=25 Oct 31 04:45:06.962430 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:06.962279 5019 generic.go:356] "Generic (PLEG): container finished" podID="837f143a-072a-446d-9062-cb6f57eb4884" containerID="8fb9d282e707fe200d7bc379420e5e3be964dda612aad5ecec9894948f61f498" exitCode=0 Oct 31 04:45:06.962430 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:06.962368 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b66kbvh" event={"ID":"837f143a-072a-446d-9062-cb6f57eb4884","Type":"ContainerDied","Data":"8fb9d282e707fe200d7bc379420e5e3be964dda612aad5ecec9894948f61f498"} Oct 31 04:45:06.965186 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:06.965154 5019 generic.go:356] "Generic (PLEG): container finished" podID="d992b6ec-1a57-4ca9-af09-370245d1ccfe" containerID="26ac7fafdc762d571b83685da74bb7f1bb57e0a81e6e8bf179afddf598c401e2" exitCode=60 Oct 31 04:45:06.965561 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:06.965533 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890365-wwx6m" event={"ID":"d992b6ec-1a57-4ca9-af09-370245d1ccfe","Type":"ContainerDied","Data":"26ac7fafdc762d571b83685da74bb7f1bb57e0a81e6e8bf179afddf598c401e2"} Oct 31 04:45:06.965694 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:06.965605 5019 scope.go:122] "RemoveContainer" containerID="e8779a53348e75d4857a839bbfeb0beea313466e097c2574a048ddf9c0af7e6b" Oct 31 04:45:06.965828 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:06.965806 5019 scope.go:122] "RemoveContainer" containerID="26ac7fafdc762d571b83685da74bb7f1bb57e0a81e6e8bf179afddf598c401e2" Oct 31 04:45:06.966644 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:06.966595 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890365-wwx6m_odh-ai-gateway-infra(d992b6ec-1a57-4ca9-af09-370245d1ccfe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890365-wwx6m" podUID="d992b6ec-1a57-4ca9-af09-370245d1ccfe" Oct 31 04:45:06.969515 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:06.969497 5019 scope.go:122] "RemoveContainer" containerID="63cc09de94f00b6797af27c7c9c00c81205970dff40ce0ecf29257f810165454" Oct 31 04:45:06.970088 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:06.970058 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890365-74cxv_odh-ai-gateway-infra(6171ff86-f98a-4575-834d-2f6bd135424d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890365-74cxv" podUID="6171ff86-f98a-4575-834d-2f6bd135424d" Oct 31 04:45:06.973359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:06.972162 5019 generic.go:356] "Generic (PLEG): container finished" podID="19237217-929c-4b26-ac1f-fb27745d6dab" containerID="2965770faabe08de689fd9a5deafcb07784512bdec3e3c01afe60888690cd434" exitCode=60 Oct 31 04:45:06.973359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:06.972284 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890365-jdjqt" event={"ID":"19237217-929c-4b26-ac1f-fb27745d6dab","Type":"ContainerDied","Data":"2965770faabe08de689fd9a5deafcb07784512bdec3e3c01afe60888690cd434"} Oct 31 04:45:06.973359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:06.972847 5019 scope.go:122] "RemoveContainer" containerID="2965770faabe08de689fd9a5deafcb07784512bdec3e3c01afe60888690cd434" Oct 31 04:45:06.973359 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:06.973287 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890365-jdjqt_odh-ai-gateway-infra(19237217-929c-4b26-ac1f-fb27745d6dab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890365-jdjqt" podUID="19237217-929c-4b26-ac1f-fb27745d6dab" Oct 31 04:45:07.064783 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:07.037663 5019 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-ded435/agc-cred-43acb978be07: secret "agc-cred-43acb978be07" not found Oct 31 04:45:07.064783 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:07.037690 5019 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-ded435/agc-cred-7fd1d61db48f: secret "agc-cred-7fd1d61db48f" not found Oct 31 04:45:07.064783 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:07.037701 5019 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-b794w26n5: [secret "agc-cred-43acb978be07" not found, secret "agc-cred-7fd1d61db48f" not found] Oct 31 04:45:07.064783 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:07.037815 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5255293a-4f6d-48d0-b162-d5270e2d4b52-provider-credentials podName:5255293a-4f6d-48d0-b162-d5270e2d4b52 nodeName:}" failed. No retries permitted until 2026-10-31 04:45:09.037790282 +0000 UTC m=+2463.794128322 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5255293a-4f6d-48d0-b162-d5270e2d4b52-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-ded435-b794w26n5" (UID: "5255293a-4f6d-48d0-b162-d5270e2d4b52") : [secret "agc-cred-43acb978be07" not found, secret "agc-cred-7fd1d61db48f" not found] Oct 31 04:45:07.064783 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:07.045792 5019 generic.go:356] "Generic (PLEG): container finished" podID="0a9a0d3a-eced-4fd8-85fa-7c08b317c39f" containerID="b89d781650a64de1497c734f2194c0d377a527170683ac44e41b55fc5fa1153c" exitCode=60 Oct 31 04:45:07.064783 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:07.046033 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-031745-29890365-grgsg" event={"ID":"0a9a0d3a-eced-4fd8-85fa-7c08b317c39f","Type":"ContainerDied","Data":"b89d781650a64de1497c734f2194c0d377a527170683ac44e41b55fc5fa1153c"} Oct 31 04:45:07.064783 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:07.046781 5019 scope.go:122] "RemoveContainer" containerID="b89d781650a64de1497c734f2194c0d377a527170683ac44e41b55fc5fa1153c" Oct 31 04:45:07.064783 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:07.047264 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-031745-29890365-grgsg_odh-ai-gateway-infra(0a9a0d3a-eced-4fd8-85fa-7c08b317c39f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-031745-29890365-grgsg" podUID="0a9a0d3a-eced-4fd8-85fa-7c08b317c39f" Oct 31 04:45:07.073713 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:07.067175 5019 generic.go:356] "Generic (PLEG): container finished" podID="2fc510cb-aae5-4d8c-a612-2270ccbbf432" containerID="667d384355c86ef041aeb1f21583791296d4c936620f6b2176633c909f940c17" exitCode=60 Oct 31 04:45:07.073713 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:07.067302 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-654741-29890365-fd9cp" event={"ID":"2fc510cb-aae5-4d8c-a612-2270ccbbf432","Type":"ContainerDied","Data":"667d384355c86ef041aeb1f21583791296d4c936620f6b2176633c909f940c17"} Oct 31 04:45:07.073713 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:07.067932 5019 scope.go:122] "RemoveContainer" containerID="667d384355c86ef041aeb1f21583791296d4c936620f6b2176633c909f940c17" Oct 31 04:45:07.073713 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:07.068438 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-654741-29890365-fd9cp_odh-ai-gateway-infra(2fc510cb-aae5-4d8c-a612-2270ccbbf432)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-654741-29890365-fd9cp" podUID="2fc510cb-aae5-4d8c-a612-2270ccbbf432" Oct 31 04:45:07.082632 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:07.082589 5019 generic.go:356] "Generic (PLEG): container finished" podID="681c73ad-b2a1-458f-b37a-cc4e9e198926" containerID="f70b5cca24c60d6e0b85189092a7f43c1d21154486d89a497e314afdc41a423c" exitCode=60 Oct 31 04:45:07.082941 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:07.082917 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890365-f7zkm" event={"ID":"681c73ad-b2a1-458f-b37a-cc4e9e198926","Type":"ContainerDied","Data":"f70b5cca24c60d6e0b85189092a7f43c1d21154486d89a497e314afdc41a423c"} Oct 31 04:45:07.086487 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:07.085594 5019 scope.go:122] "RemoveContainer" containerID="f70b5cca24c60d6e0b85189092a7f43c1d21154486d89a497e314afdc41a423c" Oct 31 04:45:07.086977 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:07.086944 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-afc519-29890365-f7zkm_odh-ai-gateway-infra(681c73ad-b2a1-458f-b37a-cc4e9e198926)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890365-f7zkm" podUID="681c73ad-b2a1-458f-b37a-cc4e9e198926" Oct 31 04:45:07.109422 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:07.107871 5019 generic.go:356] "Generic (PLEG): container finished" podID="b5c95c35-2055-4d02-af07-fc3574c29ecf" containerID="b2db622e7647f6ead204dd1b113aa0629a8c072eaa478afae61e4b0a025a10ea" exitCode=60 Oct 31 04:45:07.109422 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:07.108644 5019 scope.go:122] "RemoveContainer" containerID="b2db622e7647f6ead204dd1b113aa0629a8c072eaa478afae61e4b0a025a10ea" Oct 31 04:45:07.109422 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:07.109278 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-567ca6-29890365-pbxxk_odh-ai-gateway-infra(b5c95c35-2055-4d02-af07-fc3574c29ecf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-567ca6-29890365-pbxxk" podUID="b5c95c35-2055-4d02-af07-fc3574c29ecf" Oct 31 04:45:07.109422 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:07.109362 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-567ca6-29890365-pbxxk" event={"ID":"b5c95c35-2055-4d02-af07-fc3574c29ecf","Type":"ContainerDied","Data":"b2db622e7647f6ead204dd1b113aa0629a8c072eaa478afae61e4b0a025a10ea"} Oct 31 04:45:07.126306 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:07.126272 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890365-spg6x" event={"ID":"e12c1431-1aa4-4c72-94a9-d3b6646ee4b2","Type":"ContainerStarted","Data":"fef7cbf0d74a80fecc7705ed813dfe5c0a4e2d2bfcf8f105000cb58762243978"} Oct 31 04:45:07.128421 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:07.127865 5019 scope.go:122] "RemoveContainer" containerID="445c1608b0e271e773d364bee5474b9222baa719568207e3d8dbbadcad084ff4" Oct 31 04:45:07.141352 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:07.137652 5019 generic.go:356] "Generic (PLEG): container finished" podID="e282b47a-b89e-48ae-88bf-347945705a1d" containerID="ab2cc3789670dc2f1e4c409c77ef9a5d7d877e9ffc0d16521ebc00ee9b91f04d" exitCode=60 Oct 31 04:45:07.141352 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:07.137811 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890365-t5r5d" event={"ID":"e282b47a-b89e-48ae-88bf-347945705a1d","Type":"ContainerDied","Data":"ab2cc3789670dc2f1e4c409c77ef9a5d7d877e9ffc0d16521ebc00ee9b91f04d"} Oct 31 04:45:07.141352 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:07.138492 5019 scope.go:122] "RemoveContainer" containerID="ab2cc3789670dc2f1e4c409c77ef9a5d7d877e9ffc0d16521ebc00ee9b91f04d" Oct 31 04:45:07.141352 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:07.139201 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890365-t5r5d_odh-ai-gateway-infra(e282b47a-b89e-48ae-88bf-347945705a1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890365-t5r5d" podUID="e282b47a-b89e-48ae-88bf-347945705a1d" Oct 31 04:45:07.144595 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:07.144557 5019 generic.go:356] "Generic (PLEG): container finished" podID="668f2b2b-4d01-4054-8ca0-16a4b0b4dcdd" containerID="56f80212ba31fd736d68c6da62484c1aa5d4fce51d7558d09d25d1a51a04d8b0" exitCode=60 Oct 31 04:45:07.147120 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:07.145203 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890365-x6d4s" event={"ID":"668f2b2b-4d01-4054-8ca0-16a4b0b4dcdd","Type":"ContainerDied","Data":"56f80212ba31fd736d68c6da62484c1aa5d4fce51d7558d09d25d1a51a04d8b0"} Oct 31 04:45:07.227605 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:07.190794 5019 scope.go:122] "RemoveContainer" containerID="56f80212ba31fd736d68c6da62484c1aa5d4fce51d7558d09d25d1a51a04d8b0" Oct 31 04:45:07.270062 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:07.270023 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890365-x6d4s_odh-ai-gateway-infra(668f2b2b-4d01-4054-8ca0-16a4b0b4dcdd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890365-x6d4s" podUID="668f2b2b-4d01-4054-8ca0-16a4b0b4dcdd" Oct 31 04:45:07.434178 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:07.434150 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b66kbvh" Oct 31 04:45:07.462531 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:07.462503 5019 scope.go:122] "RemoveContainer" containerID="2fff6219a7a2997aeae1b5c3a08d0ef55b3d76d3b6733c72f59b18f695e0c209" Oct 31 04:45:07.491511 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:07.491478 5019 scope.go:122] "RemoveContainer" containerID="46dd33effa75eacaebf0c28b3a7821f876a42e57304ea68f97b00ceecec7e31c" Oct 31 04:45:07.508687 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:07.508660 5019 scope.go:122] "RemoveContainer" containerID="e14613b52ef9112e9c900cc71923eb66877039bc4db7168a2639e0ea27d41b1f" Oct 31 04:45:07.606385 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:07.604666 5019 scope.go:122] "RemoveContainer" containerID="b50bb7f040e162636ba4bdb4fc1bf964c8911537017f76865216c76830620244" Oct 31 04:45:07.635098 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:07.635052 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/837f143a-072a-446d-9062-cb6f57eb4884-provider-credentials\" (UniqueName: \"kubernetes.io/projected/837f143a-072a-446d-9062-cb6f57eb4884-provider-credentials\") pod \"837f143a-072a-446d-9062-cb6f57eb4884\" (UID: \"837f143a-072a-446d-9062-cb6f57eb4884\") " Oct 31 04:45:07.635289 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:07.635222 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/837f143a-072a-446d-9062-cb6f57eb4884-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/837f143a-072a-446d-9062-cb6f57eb4884-routing-overlay-volume\") pod \"837f143a-072a-446d-9062-cb6f57eb4884\" (UID: \"837f143a-072a-446d-9062-cb6f57eb4884\") " Oct 31 04:45:07.635429 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:07.635317 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/837f143a-072a-446d-9062-cb6f57eb4884-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/837f143a-072a-446d-9062-cb6f57eb4884-plugins-config-volume\") pod \"837f143a-072a-446d-9062-cb6f57eb4884\" (UID: \"837f143a-072a-446d-9062-cb6f57eb4884\") " Oct 31 04:45:07.635514 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:07.635446 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/837f143a-072a-446d-9062-cb6f57eb4884-kube-api-access-4zsj7\" (UniqueName: \"kubernetes.io/projected/837f143a-072a-446d-9062-cb6f57eb4884-kube-api-access-4zsj7\") pod \"837f143a-072a-446d-9062-cb6f57eb4884\" (UID: \"837f143a-072a-446d-9062-cb6f57eb4884\") " Oct 31 04:45:07.644397 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:07.643317 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/837f143a-072a-446d-9062-cb6f57eb4884-routing-overlay-volume" pod "837f143a-072a-446d-9062-cb6f57eb4884" (UID: "837f143a-072a-446d-9062-cb6f57eb4884"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:45:07.644397 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:07.643730 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/837f143a-072a-446d-9062-cb6f57eb4884-plugins-config-volume" pod "837f143a-072a-446d-9062-cb6f57eb4884" (UID: "837f143a-072a-446d-9062-cb6f57eb4884"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:45:07.667424 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:07.662004 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/837f143a-072a-446d-9062-cb6f57eb4884-provider-credentials" pod "837f143a-072a-446d-9062-cb6f57eb4884" (UID: "837f143a-072a-446d-9062-cb6f57eb4884"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:45:07.687959 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:07.687927 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/837f143a-072a-446d-9062-cb6f57eb4884-kube-api-access-4zsj7" pod "837f143a-072a-446d-9062-cb6f57eb4884" (UID: "837f143a-072a-446d-9062-cb6f57eb4884"). InnerVolumeSpecName "kube-api-access-4zsj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:45:07.697591 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:07.697432 5019 scope.go:122] "RemoveContainer" containerID="29dad26c5456d090ff4d85fb8e5544ae6e40d79ec52e36601e2154f033badf33" Oct 31 04:45:07.720188 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:07.720152 5019 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99be4628_816c_46f6_abcf_9de2a8452895.slice/crio-1e8a810e742b555a2f60ad8f1a8d9180f335898db3e3cc93e343b945dfaad3ff.scope\": RecentStats: unable to find data in memory cache]" Oct 31 04:45:07.738300 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:07.738272 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zsj7\" (UniqueName: \"kubernetes.io/projected/837f143a-072a-446d-9062-cb6f57eb4884-kube-api-access-4zsj7\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:07.738300 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:07.738299 5019 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/837f143a-072a-446d-9062-cb6f57eb4884-provider-credentials\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:07.738489 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:07.738311 5019 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/837f143a-072a-446d-9062-cb6f57eb4884-routing-overlay-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:07.738489 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:07.738320 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/837f143a-072a-446d-9062-cb6f57eb4884-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:07.791151 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:07.791129 5019 scope.go:122] "RemoveContainer" containerID="4a6dfb802a29ce3c6ef58ef6fa4849c69639adb8d4a1689b1d26361457da6b86" Oct 31 04:45:08.160756 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:08.160721 5019 scope.go:122] "RemoveContainer" containerID="26ac7fafdc762d571b83685da74bb7f1bb57e0a81e6e8bf179afddf598c401e2" Oct 31 04:45:08.161193 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:08.161168 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890365-wwx6m_odh-ai-gateway-infra(d992b6ec-1a57-4ca9-af09-370245d1ccfe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890365-wwx6m" podUID="d992b6ec-1a57-4ca9-af09-370245d1ccfe" Oct 31 04:45:08.162532 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:08.162507 5019 generic.go:356] "Generic (PLEG): container finished" podID="99be4628-816c-46f6-abcf-9de2a8452895" containerID="1e8a810e742b555a2f60ad8f1a8d9180f335898db3e3cc93e343b945dfaad3ff" exitCode=60 Oct 31 04:45:08.162642 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:08.162598 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890365-xj5rc" event={"ID":"99be4628-816c-46f6-abcf-9de2a8452895","Type":"ContainerDied","Data":"1e8a810e742b555a2f60ad8f1a8d9180f335898db3e3cc93e343b945dfaad3ff"} Oct 31 04:45:08.162699 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:08.162670 5019 scope.go:122] "RemoveContainer" containerID="ea42f0ef09870f9ed8eda40f065500d3733ab490d0413120a8ed797f714e11f9" Oct 31 04:45:08.162937 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:08.162913 5019 scope.go:122] "RemoveContainer" containerID="1e8a810e742b555a2f60ad8f1a8d9180f335898db3e3cc93e343b945dfaad3ff" Oct 31 04:45:08.163323 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:08.163295 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-4c2387-29890365-xj5rc_odh-ai-gateway-infra(99be4628-816c-46f6-abcf-9de2a8452895)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890365-xj5rc" podUID="99be4628-816c-46f6-abcf-9de2a8452895" Oct 31 04:45:08.164868 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:08.164838 5019 scope.go:122] "RemoveContainer" containerID="2965770faabe08de689fd9a5deafcb07784512bdec3e3c01afe60888690cd434" Oct 31 04:45:08.165255 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:08.165226 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890365-jdjqt_odh-ai-gateway-infra(19237217-929c-4b26-ac1f-fb27745d6dab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890365-jdjqt" podUID="19237217-929c-4b26-ac1f-fb27745d6dab" Oct 31 04:45:08.166797 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:08.166776 5019 generic.go:356] "Generic (PLEG): container finished" podID="a53b729a-b280-41d4-8661-44826b486246" containerID="88fa47d69e8da32266f83de06f0095bb431dce730b18aef1ff3794c63d0d21af" exitCode=60 Oct 31 04:45:08.166898 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:08.166861 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890365-xf5cz" event={"ID":"a53b729a-b280-41d4-8661-44826b486246","Type":"ContainerDied","Data":"88fa47d69e8da32266f83de06f0095bb431dce730b18aef1ff3794c63d0d21af"} Oct 31 04:45:08.167136 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:08.167122 5019 scope.go:122] "RemoveContainer" containerID="88fa47d69e8da32266f83de06f0095bb431dce730b18aef1ff3794c63d0d21af" Oct 31 04:45:08.167516 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:08.167489 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890365-xf5cz_odh-ai-gateway-infra(a53b729a-b280-41d4-8661-44826b486246)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890365-xf5cz" podUID="a53b729a-b280-41d4-8661-44826b486246" Oct 31 04:45:08.169034 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:08.169017 5019 scope.go:122] "RemoveContainer" containerID="b89d781650a64de1497c734f2194c0d377a527170683ac44e41b55fc5fa1153c" Oct 31 04:45:08.169306 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:08.169282 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-031745-29890365-grgsg_odh-ai-gateway-infra(0a9a0d3a-eced-4fd8-85fa-7c08b317c39f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-031745-29890365-grgsg" podUID="0a9a0d3a-eced-4fd8-85fa-7c08b317c39f" Oct 31 04:45:08.171219 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:08.171197 5019 scope.go:122] "RemoveContainer" containerID="667d384355c86ef041aeb1f21583791296d4c936620f6b2176633c909f940c17" Oct 31 04:45:08.171534 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:08.171508 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-654741-29890365-fd9cp_odh-ai-gateway-infra(2fc510cb-aae5-4d8c-a612-2270ccbbf432)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-654741-29890365-fd9cp" podUID="2fc510cb-aae5-4d8c-a612-2270ccbbf432" Oct 31 04:45:08.173023 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:08.173006 5019 scope.go:122] "RemoveContainer" containerID="f70b5cca24c60d6e0b85189092a7f43c1d21154486d89a497e314afdc41a423c" Oct 31 04:45:08.173254 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:08.173232 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-afc519-29890365-f7zkm_odh-ai-gateway-infra(681c73ad-b2a1-458f-b37a-cc4e9e198926)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890365-f7zkm" podUID="681c73ad-b2a1-458f-b37a-cc4e9e198926" Oct 31 04:45:08.174848 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:08.174825 5019 scope.go:122] "RemoveContainer" containerID="b2db622e7647f6ead204dd1b113aa0629a8c072eaa478afae61e4b0a025a10ea" Oct 31 04:45:08.175142 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:08.175117 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-567ca6-29890365-pbxxk_odh-ai-gateway-infra(b5c95c35-2055-4d02-af07-fc3574c29ecf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-567ca6-29890365-pbxxk" podUID="b5c95c35-2055-4d02-af07-fc3574c29ecf" Oct 31 04:45:08.176619 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:08.176596 5019 generic.go:356] "Generic (PLEG): container finished" podID="e12c1431-1aa4-4c72-94a9-d3b6646ee4b2" containerID="fef7cbf0d74a80fecc7705ed813dfe5c0a4e2d2bfcf8f105000cb58762243978" exitCode=60 Oct 31 04:45:08.176729 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:08.176678 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890365-spg6x" event={"ID":"e12c1431-1aa4-4c72-94a9-d3b6646ee4b2","Type":"ContainerDied","Data":"fef7cbf0d74a80fecc7705ed813dfe5c0a4e2d2bfcf8f105000cb58762243978"} Oct 31 04:45:08.176836 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:08.176822 5019 scope.go:122] "RemoveContainer" containerID="fef7cbf0d74a80fecc7705ed813dfe5c0a4e2d2bfcf8f105000cb58762243978" Oct 31 04:45:08.177050 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:08.177032 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890365-spg6x_odh-ai-gateway-infra(e12c1431-1aa4-4c72-94a9-d3b6646ee4b2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890365-spg6x" podUID="e12c1431-1aa4-4c72-94a9-d3b6646ee4b2" Oct 31 04:45:08.179484 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:08.179463 5019 scope.go:122] "RemoveContainer" containerID="ab2cc3789670dc2f1e4c409c77ef9a5d7d877e9ffc0d16521ebc00ee9b91f04d" Oct 31 04:45:08.179718 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:08.179695 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890365-t5r5d_odh-ai-gateway-infra(e282b47a-b89e-48ae-88bf-347945705a1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890365-t5r5d" podUID="e282b47a-b89e-48ae-88bf-347945705a1d" Oct 31 04:45:08.181400 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:08.181379 5019 generic.go:356] "Generic (PLEG): container finished" podID="b7abcb04-58ef-4efd-a350-c8f5286ebc14" containerID="fecbb7f9f57537ec516e66644012163fe52fc628357ee666513264c015ed8a71" exitCode=60 Oct 31 04:45:08.181497 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:08.181417 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890365-f74kl" event={"ID":"b7abcb04-58ef-4efd-a350-c8f5286ebc14","Type":"ContainerDied","Data":"fecbb7f9f57537ec516e66644012163fe52fc628357ee666513264c015ed8a71"} Oct 31 04:45:08.181727 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:08.181704 5019 scope.go:122] "RemoveContainer" containerID="fecbb7f9f57537ec516e66644012163fe52fc628357ee666513264c015ed8a71" Oct 31 04:45:08.181994 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:08.181966 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-09a20a-29890365-f74kl_odh-ai-gateway-infra(b7abcb04-58ef-4efd-a350-c8f5286ebc14)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890365-f74kl" podUID="b7abcb04-58ef-4efd-a350-c8f5286ebc14" Oct 31 04:45:08.183844 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:08.183823 5019 scope.go:122] "RemoveContainer" containerID="56f80212ba31fd736d68c6da62484c1aa5d4fce51d7558d09d25d1a51a04d8b0" Oct 31 04:45:08.184066 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:08.184042 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890365-x6d4s_odh-ai-gateway-infra(668f2b2b-4d01-4054-8ca0-16a4b0b4dcdd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890365-x6d4s" podUID="668f2b2b-4d01-4054-8ca0-16a4b0b4dcdd" Oct 31 04:45:08.185955 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:08.185934 5019 generic.go:356] "Generic (PLEG): container finished" podID="089f7eeb-81d7-4562-b906-da33b0fb5fa1" containerID="e0739fb7237fd34717fb9e1f95b1f9e6cda1d45d5910860608e806d33b17f942" exitCode=60 Oct 31 04:45:08.186073 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:08.186017 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890365-cnzxg" event={"ID":"089f7eeb-81d7-4562-b906-da33b0fb5fa1","Type":"ContainerDied","Data":"e0739fb7237fd34717fb9e1f95b1f9e6cda1d45d5910860608e806d33b17f942"} Oct 31 04:45:08.186243 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:08.186226 5019 scope.go:122] "RemoveContainer" containerID="e0739fb7237fd34717fb9e1f95b1f9e6cda1d45d5910860608e806d33b17f942" Oct 31 04:45:08.186501 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:08.186481 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-439989-29890365-cnzxg_odh-ai-gateway-infra(089f7eeb-81d7-4562-b906-da33b0fb5fa1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890365-cnzxg" podUID="089f7eeb-81d7-4562-b906-da33b0fb5fa1" Oct 31 04:45:08.187511 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:08.187489 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b66kbvh" Oct 31 04:45:08.187613 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:08.187509 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b66kbvh" event={"ID":"837f143a-072a-446d-9062-cb6f57eb4884","Type":"ContainerDied","Data":"32f598492f61ed4d4046439e4abec5accbe1d4ca044352a919e2f8866754e0d1"} Oct 31 04:45:08.188780 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:08.188757 5019 scope.go:122] "RemoveContainer" containerID="90842b342d746cdee675f7dab701a30f5088bfd7e45cb6fc67579f949452908e" Oct 31 04:45:08.189871 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:08.189847 5019 generic.go:356] "Generic (PLEG): container finished" podID="bc26f88e-b019-4540-a826-63fb1d624a8d" containerID="e223e7c544ed8d169986c16d73117d9d2a99c0ddbfaabd6d237d93efa4679583" exitCode=60 Oct 31 04:45:08.189955 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:08.189939 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890365-5h7hv" event={"ID":"bc26f88e-b019-4540-a826-63fb1d624a8d","Type":"ContainerDied","Data":"e223e7c544ed8d169986c16d73117d9d2a99c0ddbfaabd6d237d93efa4679583"} Oct 31 04:45:08.190313 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:08.190291 5019 scope.go:122] "RemoveContainer" containerID="e223e7c544ed8d169986c16d73117d9d2a99c0ddbfaabd6d237d93efa4679583" Oct 31 04:45:08.190660 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:08.190634 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-8973e2-29890365-5h7hv_odh-ai-gateway-infra(bc26f88e-b019-4540-a826-63fb1d624a8d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890365-5h7hv" podUID="bc26f88e-b019-4540-a826-63fb1d624a8d" Oct 31 04:45:08.192312 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:08.192290 5019 generic.go:356] "Generic (PLEG): container finished" podID="01758cb6-b948-46f8-a607-210d75bb7716" containerID="00c377295d4e367fda0adfca705f1d40cb9737d8d2c84cedcf2660e3df5393b2" exitCode=60 Oct 31 04:45:08.192451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:08.192364 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890365-rhn85" event={"ID":"01758cb6-b948-46f8-a607-210d75bb7716","Type":"ContainerDied","Data":"00c377295d4e367fda0adfca705f1d40cb9737d8d2c84cedcf2660e3df5393b2"} Oct 31 04:45:08.192628 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:08.192610 5019 scope.go:122] "RemoveContainer" containerID="00c377295d4e367fda0adfca705f1d40cb9737d8d2c84cedcf2660e3df5393b2" Oct 31 04:45:08.192863 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:08.192842 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-766728-29890365-rhn85_odh-ai-gateway-infra(01758cb6-b948-46f8-a607-210d75bb7716)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890365-rhn85" podUID="01758cb6-b948-46f8-a607-210d75bb7716" Oct 31 04:45:08.209144 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:08.209123 5019 scope.go:122] "RemoveContainer" containerID="a300bd76b5eea137c14bd913da24e8cca5c666a13d1ffc2a0d8ff9a7f254a871" Oct 31 04:45:08.229015 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:08.228991 5019 scope.go:122] "RemoveContainer" containerID="3c2505d98cae21bb48b61702985be8657e603aa908e049a36f11eeed52e7d1a7" Oct 31 04:45:08.253958 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:08.253934 5019 scope.go:122] "RemoveContainer" containerID="03d2d6dbaeb701309f19958e780f9807f85f1dd8f1f661cbaed71c5aec72f4c1" Oct 31 04:45:08.271906 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:08.271882 5019 scope.go:122] "RemoveContainer" containerID="8fb9d282e707fe200d7bc379420e5e3be964dda612aad5ecec9894948f61f498" Oct 31 04:45:08.283324 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:08.283303 5019 scope.go:122] "RemoveContainer" containerID="51cac0cf5f16870608c609d75ed01737bf29b3c3ef903174f55c9b6d23d765bc" Oct 31 04:45:08.298760 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:08.298737 5019 scope.go:122] "RemoveContainer" containerID="8f926deb3ccf97b321ca815b8db5736cc2956a5b6c6113387df603f5b6f2345e" Oct 31 04:45:08.479437 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:08.479387 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b66kbvh"] Oct 31 04:45:08.484533 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:08.484500 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-75b66kbvh"] Oct 31 04:45:09.062483 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:09.062448 5019 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-ded435/agc-cred-43acb978be07: secret "agc-cred-43acb978be07" not found Oct 31 04:45:09.062483 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:09.062483 5019 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-ded435/agc-cred-7fd1d61db48f: secret "agc-cred-7fd1d61db48f" not found Oct 31 04:45:09.062483 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:09.062494 5019 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-b794w26n5: [secret "agc-cred-43acb978be07" not found, secret "agc-cred-7fd1d61db48f" not found] Oct 31 04:45:09.062746 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:09.062556 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5255293a-4f6d-48d0-b162-d5270e2d4b52-provider-credentials podName:5255293a-4f6d-48d0-b162-d5270e2d4b52 nodeName:}" failed. No retries permitted until 2026-10-31 04:45:13.062536938 +0000 UTC m=+2467.818874974 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5255293a-4f6d-48d0-b162-d5270e2d4b52-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-ded435-b794w26n5" (UID: "5255293a-4f6d-48d0-b162-d5270e2d4b52") : [secret "agc-cred-43acb978be07" not found, secret "agc-cred-7fd1d61db48f" not found] Oct 31 04:45:09.206408 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:09.206371 5019 scope.go:122] "RemoveContainer" containerID="fef7cbf0d74a80fecc7705ed813dfe5c0a4e2d2bfcf8f105000cb58762243978" Oct 31 04:45:09.206889 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:09.206847 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890365-spg6x_odh-ai-gateway-infra(e12c1431-1aa4-4c72-94a9-d3b6646ee4b2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890365-spg6x" podUID="e12c1431-1aa4-4c72-94a9-d3b6646ee4b2" Oct 31 04:45:09.208315 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:09.208288 5019 scope.go:122] "RemoveContainer" containerID="fecbb7f9f57537ec516e66644012163fe52fc628357ee666513264c015ed8a71" Oct 31 04:45:09.208768 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:09.208739 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-09a20a-29890365-f74kl_odh-ai-gateway-infra(b7abcb04-58ef-4efd-a350-c8f5286ebc14)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890365-f74kl" podUID="b7abcb04-58ef-4efd-a350-c8f5286ebc14" Oct 31 04:45:09.210389 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:09.210360 5019 scope.go:122] "RemoveContainer" containerID="e0739fb7237fd34717fb9e1f95b1f9e6cda1d45d5910860608e806d33b17f942" Oct 31 04:45:09.210751 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:09.210727 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-439989-29890365-cnzxg_odh-ai-gateway-infra(089f7eeb-81d7-4562-b906-da33b0fb5fa1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890365-cnzxg" podUID="089f7eeb-81d7-4562-b906-da33b0fb5fa1" Oct 31 04:45:09.213012 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:09.212994 5019 scope.go:122] "RemoveContainer" containerID="e223e7c544ed8d169986c16d73117d9d2a99c0ddbfaabd6d237d93efa4679583" Oct 31 04:45:09.213377 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:09.213350 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-8973e2-29890365-5h7hv_odh-ai-gateway-infra(bc26f88e-b019-4540-a826-63fb1d624a8d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890365-5h7hv" podUID="bc26f88e-b019-4540-a826-63fb1d624a8d" Oct 31 04:45:09.214634 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:09.214607 5019 scope.go:122] "RemoveContainer" containerID="00c377295d4e367fda0adfca705f1d40cb9737d8d2c84cedcf2660e3df5393b2" Oct 31 04:45:09.214915 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:09.214894 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-766728-29890365-rhn85_odh-ai-gateway-infra(01758cb6-b948-46f8-a607-210d75bb7716)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890365-rhn85" podUID="01758cb6-b948-46f8-a607-210d75bb7716" Oct 31 04:45:09.216264 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:09.216246 5019 scope.go:122] "RemoveContainer" containerID="1e8a810e742b555a2f60ad8f1a8d9180f335898db3e3cc93e343b945dfaad3ff" Oct 31 04:45:09.216609 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:09.216579 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-4c2387-29890365-xj5rc_odh-ai-gateway-infra(99be4628-816c-46f6-abcf-9de2a8452895)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890365-xj5rc" podUID="99be4628-816c-46f6-abcf-9de2a8452895" Oct 31 04:45:09.218067 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:09.218051 5019 scope.go:122] "RemoveContainer" containerID="88fa47d69e8da32266f83de06f0095bb431dce730b18aef1ff3794c63d0d21af" Oct 31 04:45:09.218286 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:09.218269 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890365-xf5cz_odh-ai-gateway-infra(a53b729a-b280-41d4-8661-44826b486246)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890365-xf5cz" podUID="a53b729a-b280-41d4-8661-44826b486246" Oct 31 04:45:09.418568 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:09.418474 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="837f143a-072a-446d-9062-cb6f57eb4884" path="/var/lib/kubelet/pods/837f143a-072a-446d-9062-cb6f57eb4884/volumes" Oct 31 04:45:10.223875 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:10.223830 5019 scope.go:122] "RemoveContainer" containerID="1a8dc0e789b7ae0b0b196ab2c423d802c390013bad5ff3ec106cda10ea274256" Oct 31 04:45:10.236920 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:10.236893 5019 scope.go:122] "RemoveContainer" containerID="b919fff27e7be0d2985b6d05fd01e36182cc793d60ecefbc3e7fbbe4345ee008" Oct 31 04:45:10.250257 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:10.250227 5019 scope.go:122] "RemoveContainer" containerID="feae27ab3c10cdda177858c606935bbd8c9294a78b529421394049c0504d3569" Oct 31 04:45:13.146149 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:13.146104 5019 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-ded435/agc-cred-43acb978be07: secret "agc-cred-43acb978be07" not found Oct 31 04:45:13.146149 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:13.146143 5019 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-ded435/agc-cred-7fd1d61db48f: secret "agc-cred-7fd1d61db48f" not found Oct 31 04:45:13.146149 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:13.146155 5019 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-b794w26n5: [secret "agc-cred-43acb978be07" not found, secret "agc-cred-7fd1d61db48f" not found] Oct 31 04:45:13.146693 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:13.146217 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5255293a-4f6d-48d0-b162-d5270e2d4b52-provider-credentials podName:5255293a-4f6d-48d0-b162-d5270e2d4b52 nodeName:}" failed. No retries permitted until 2026-10-31 04:45:21.146197596 +0000 UTC m=+2475.902535631 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5255293a-4f6d-48d0-b162-d5270e2d4b52-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-ded435-b794w26n5" (UID: "5255293a-4f6d-48d0-b162-d5270e2d4b52") : [secret "agc-cred-43acb978be07" not found, secret "agc-cred-7fd1d61db48f" not found] Oct 31 04:45:14.283249 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:14.283201 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-b794w26n5"] Oct 31 04:45:16.403413 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:16.402351 5019 scope.go:122] "RemoveContainer" containerID="b89d781650a64de1497c734f2194c0d377a527170683ac44e41b55fc5fa1153c" Oct 31 04:45:17.345964 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:17.345927 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-031745-29890365-grgsg" event={"ID":"0a9a0d3a-eced-4fd8-85fa-7c08b317c39f","Type":"ContainerStarted","Data":"3d730d3f9c7b85d238cb3ac1a8377392601da407da7445e77f937adfe8c26942"} Oct 31 04:45:17.400138 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:17.400095 5019 scope.go:122] "RemoveContainer" containerID="2965770faabe08de689fd9a5deafcb07784512bdec3e3c01afe60888690cd434" Oct 31 04:45:17.400420 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:17.400246 5019 scope.go:122] "RemoveContainer" containerID="63cc09de94f00b6797af27c7c9c00c81205970dff40ce0ecf29257f810165454" Oct 31 04:45:18.356544 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:18.356452 5019 generic.go:356] "Generic (PLEG): container finished" podID="0a9a0d3a-eced-4fd8-85fa-7c08b317c39f" containerID="3d730d3f9c7b85d238cb3ac1a8377392601da407da7445e77f937adfe8c26942" exitCode=60 Oct 31 04:45:18.356544 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:18.356500 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-031745-29890365-grgsg" event={"ID":"0a9a0d3a-eced-4fd8-85fa-7c08b317c39f","Type":"ContainerDied","Data":"3d730d3f9c7b85d238cb3ac1a8377392601da407da7445e77f937adfe8c26942"} Oct 31 04:45:18.357154 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:18.356564 5019 scope.go:122] "RemoveContainer" containerID="b89d781650a64de1497c734f2194c0d377a527170683ac44e41b55fc5fa1153c" Oct 31 04:45:18.357154 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:18.357087 5019 scope.go:122] "RemoveContainer" containerID="3d730d3f9c7b85d238cb3ac1a8377392601da407da7445e77f937adfe8c26942" Oct 31 04:45:18.357663 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:18.357554 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-031745-29890365-grgsg_odh-ai-gateway-infra(0a9a0d3a-eced-4fd8-85fa-7c08b317c39f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-031745-29890365-grgsg" podUID="0a9a0d3a-eced-4fd8-85fa-7c08b317c39f" Oct 31 04:45:18.358918 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:18.358895 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890365-74cxv" event={"ID":"6171ff86-f98a-4575-834d-2f6bd135424d","Type":"ContainerStarted","Data":"74b15b58ba442dc7b7d20e608540f82c903e0620a4ab8fe4cc611a0888f2e0f3"} Oct 31 04:45:18.360968 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:18.360940 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890365-jdjqt" event={"ID":"19237217-929c-4b26-ac1f-fb27745d6dab","Type":"ContainerStarted","Data":"9b0f18a8e97e78410152c9fdc19e5b84b10de1ab8b598b20392a7eee0b559f19"} Oct 31 04:45:18.397627 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:18.397589 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-031745-29890365-grgsg"] Oct 31 04:45:18.400201 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:18.400164 5019 scope.go:122] "RemoveContainer" containerID="fef7cbf0d74a80fecc7705ed813dfe5c0a4e2d2bfcf8f105000cb58762243978" Oct 31 04:45:18.401212 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:18.401190 5019 scope.go:122] "RemoveContainer" containerID="b2db622e7647f6ead204dd1b113aa0629a8c072eaa478afae61e4b0a025a10ea" Oct 31 04:45:18.402938 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:18.401591 5019 scope.go:122] "RemoveContainer" containerID="56f80212ba31fd736d68c6da62484c1aa5d4fce51d7558d09d25d1a51a04d8b0" Oct 31 04:45:18.402938 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:18.401983 5019 scope.go:122] "RemoveContainer" containerID="f70b5cca24c60d6e0b85189092a7f43c1d21154486d89a497e314afdc41a423c" Oct 31 04:45:19.283907 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:19.283811 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-b794w26n5" podUID="5255293a-4f6d-48d0-b162-d5270e2d4b52" containerName="payload-processing" containerID="cri-o://b067ea72ce0b76cfff18ec7ddbbe72d75a5e003dcc75d79ff3ed1d7341864866" gracePeriod=25 Oct 31 04:45:19.376428 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:19.376388 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890365-f7zkm" event={"ID":"681c73ad-b2a1-458f-b37a-cc4e9e198926","Type":"ContainerStarted","Data":"e6565c263a8a79584ae51643ce0a44c19ee9ad5314ac8b314783d278b5525499"} Oct 31 04:45:19.379141 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:19.379095 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-567ca6-29890365-pbxxk" event={"ID":"b5c95c35-2055-4d02-af07-fc3574c29ecf","Type":"ContainerStarted","Data":"d11ae5ebf613a1e4f93bdc7bd7531271980df4bb8022c5151fa192d43f1bd784"} Oct 31 04:45:19.381552 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:19.381521 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890365-spg6x" event={"ID":"e12c1431-1aa4-4c72-94a9-d3b6646ee4b2","Type":"ContainerStarted","Data":"f9b76f019713eecf23e7722932c906b1b39b6e9df8d3582efda2383f7b304b67"} Oct 31 04:45:19.385105 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:19.385071 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890365-x6d4s" event={"ID":"668f2b2b-4d01-4054-8ca0-16a4b0b4dcdd","Type":"ContainerStarted","Data":"ce0bd0a8878cd93285b10091aa652321dd737a52f2039075e515eb5e0710a6ab"} Oct 31 04:45:19.388775 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:19.388734 5019 generic.go:356] "Generic (PLEG): container finished" podID="6171ff86-f98a-4575-834d-2f6bd135424d" containerID="74b15b58ba442dc7b7d20e608540f82c903e0620a4ab8fe4cc611a0888f2e0f3" exitCode=60 Oct 31 04:45:19.388949 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:19.388778 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890365-74cxv" event={"ID":"6171ff86-f98a-4575-834d-2f6bd135424d","Type":"ContainerDied","Data":"74b15b58ba442dc7b7d20e608540f82c903e0620a4ab8fe4cc611a0888f2e0f3"} Oct 31 04:45:19.388949 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:19.388839 5019 scope.go:122] "RemoveContainer" containerID="63cc09de94f00b6797af27c7c9c00c81205970dff40ce0ecf29257f810165454" Oct 31 04:45:19.389439 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:19.389418 5019 scope.go:122] "RemoveContainer" containerID="74b15b58ba442dc7b7d20e608540f82c903e0620a4ab8fe4cc611a0888f2e0f3" Oct 31 04:45:19.389952 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:19.389921 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890365-74cxv_odh-ai-gateway-infra(6171ff86-f98a-4575-834d-2f6bd135424d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890365-74cxv" podUID="6171ff86-f98a-4575-834d-2f6bd135424d" Oct 31 04:45:19.392384 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:19.391310 5019 generic.go:356] "Generic (PLEG): container finished" podID="19237217-929c-4b26-ac1f-fb27745d6dab" containerID="9b0f18a8e97e78410152c9fdc19e5b84b10de1ab8b598b20392a7eee0b559f19" exitCode=60 Oct 31 04:45:19.392384 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:19.391388 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890365-jdjqt" event={"ID":"19237217-929c-4b26-ac1f-fb27745d6dab","Type":"ContainerDied","Data":"9b0f18a8e97e78410152c9fdc19e5b84b10de1ab8b598b20392a7eee0b559f19"} Oct 31 04:45:19.392384 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:19.391861 5019 scope.go:122] "RemoveContainer" containerID="9b0f18a8e97e78410152c9fdc19e5b84b10de1ab8b598b20392a7eee0b559f19" Oct 31 04:45:19.392384 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:19.392311 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890365-jdjqt_odh-ai-gateway-infra(19237217-929c-4b26-ac1f-fb27745d6dab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890365-jdjqt" podUID="19237217-929c-4b26-ac1f-fb27745d6dab" Oct 31 04:45:19.407409 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:19.404201 5019 scope.go:122] "RemoveContainer" containerID="00c377295d4e367fda0adfca705f1d40cb9737d8d2c84cedcf2660e3df5393b2" Oct 31 04:45:19.407409 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:19.404856 5019 scope.go:122] "RemoveContainer" containerID="ab2cc3789670dc2f1e4c409c77ef9a5d7d877e9ffc0d16521ebc00ee9b91f04d" Oct 31 04:45:19.407409 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:19.405845 5019 scope.go:122] "RemoveContainer" containerID="667d384355c86ef041aeb1f21583791296d4c936620f6b2176633c909f940c17" Oct 31 04:45:19.407409 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:19.406703 5019 scope.go:122] "RemoveContainer" containerID="e0739fb7237fd34717fb9e1f95b1f9e6cda1d45d5910860608e806d33b17f942" Oct 31 04:45:19.409910 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:19.409884 5019 scope.go:122] "RemoveContainer" containerID="2965770faabe08de689fd9a5deafcb07784512bdec3e3c01afe60888690cd434" Oct 31 04:45:19.500380 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:19.500327 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890365-jdjqt"] Oct 31 04:45:19.553163 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:19.551634 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890365-74cxv"] Oct 31 04:45:19.566990 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:19.566923 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-567ca6-29890365-pbxxk" podStartSLOduration=19.566900656 podStartE2EDuration="19.566900656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:45:19.557133864 +0000 UTC m=+2474.313471908" watchObservedRunningTime="2026-10-31 04:45:19.566900656 +0000 UTC m=+2474.323238697" Oct 31 04:45:19.609404 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:19.608255 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890365-x6d4s" podStartSLOduration=19.608232103 podStartE2EDuration="19.608232103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:45:19.602136013 +0000 UTC m=+2474.358474054" watchObservedRunningTime="2026-10-31 04:45:19.608232103 +0000 UTC m=+2474.364570145" Oct 31 04:45:20.064302 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.064276 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-031745-29890365-grgsg" Oct 31 04:45:20.068634 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.068611 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-b794w26n5" Oct 31 04:45:20.233921 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.233891 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a9a0d3a-eced-4fd8-85fa-7c08b317c39f-kube-api-access-jr22n\" (UniqueName: \"kubernetes.io/projected/0a9a0d3a-eced-4fd8-85fa-7c08b317c39f-kube-api-access-jr22n\") pod \"0a9a0d3a-eced-4fd8-85fa-7c08b317c39f\" (UID: \"0a9a0d3a-eced-4fd8-85fa-7c08b317c39f\") " Oct 31 04:45:20.234099 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.234058 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5255293a-4f6d-48d0-b162-d5270e2d4b52-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5255293a-4f6d-48d0-b162-d5270e2d4b52-plugins-config-volume\") pod \"5255293a-4f6d-48d0-b162-d5270e2d4b52\" (UID: \"5255293a-4f6d-48d0-b162-d5270e2d4b52\") " Oct 31 04:45:20.234169 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.234109 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5255293a-4f6d-48d0-b162-d5270e2d4b52-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5255293a-4f6d-48d0-b162-d5270e2d4b52-provider-credentials\") pod \"5255293a-4f6d-48d0-b162-d5270e2d4b52\" (UID: \"5255293a-4f6d-48d0-b162-d5270e2d4b52\") " Oct 31 04:45:20.234289 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.234175 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5255293a-4f6d-48d0-b162-d5270e2d4b52-kube-api-access-pqrs5\" (UniqueName: \"kubernetes.io/projected/5255293a-4f6d-48d0-b162-d5270e2d4b52-kube-api-access-pqrs5\") pod \"5255293a-4f6d-48d0-b162-d5270e2d4b52\" (UID: \"5255293a-4f6d-48d0-b162-d5270e2d4b52\") " Oct 31 04:45:20.234365 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.234306 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a9a0d3a-eced-4fd8-85fa-7c08b317c39f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a9a0d3a-eced-4fd8-85fa-7c08b317c39f-maas-api-service-ca\") pod \"0a9a0d3a-eced-4fd8-85fa-7c08b317c39f\" (UID: \"0a9a0d3a-eced-4fd8-85fa-7c08b317c39f\") " Oct 31 04:45:20.234428 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.234397 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5255293a-4f6d-48d0-b162-d5270e2d4b52-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5255293a-4f6d-48d0-b162-d5270e2d4b52-routing-overlay-volume\") pod \"5255293a-4f6d-48d0-b162-d5270e2d4b52\" (UID: \"5255293a-4f6d-48d0-b162-d5270e2d4b52\") " Oct 31 04:45:20.234550 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.234522 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5255293a-4f6d-48d0-b162-d5270e2d4b52-plugins-config-volume" pod "5255293a-4f6d-48d0-b162-d5270e2d4b52" (UID: "5255293a-4f6d-48d0-b162-d5270e2d4b52"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:45:20.234926 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.234901 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a9a0d3a-eced-4fd8-85fa-7c08b317c39f-maas-api-service-ca" pod "0a9a0d3a-eced-4fd8-85fa-7c08b317c39f" (UID: "0a9a0d3a-eced-4fd8-85fa-7c08b317c39f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:45:20.235193 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.235155 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5255293a-4f6d-48d0-b162-d5270e2d4b52-routing-overlay-volume" pod "5255293a-4f6d-48d0-b162-d5270e2d4b52" (UID: "5255293a-4f6d-48d0-b162-d5270e2d4b52"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:45:20.235324 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.235289 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a9a0d3a-eced-4fd8-85fa-7c08b317c39f-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:20.235324 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.235311 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5255293a-4f6d-48d0-b162-d5270e2d4b52-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:20.237086 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.237064 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a9a0d3a-eced-4fd8-85fa-7c08b317c39f-kube-api-access-jr22n" pod "0a9a0d3a-eced-4fd8-85fa-7c08b317c39f" (UID: "0a9a0d3a-eced-4fd8-85fa-7c08b317c39f"). InnerVolumeSpecName "kube-api-access-jr22n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:45:20.237178 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.237096 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5255293a-4f6d-48d0-b162-d5270e2d4b52-provider-credentials" pod "5255293a-4f6d-48d0-b162-d5270e2d4b52" (UID: "5255293a-4f6d-48d0-b162-d5270e2d4b52"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:45:20.241063 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.241041 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5255293a-4f6d-48d0-b162-d5270e2d4b52-kube-api-access-pqrs5" pod "5255293a-4f6d-48d0-b162-d5270e2d4b52" (UID: "5255293a-4f6d-48d0-b162-d5270e2d4b52"). InnerVolumeSpecName "kube-api-access-pqrs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:45:20.337171 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.337088 5019 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5255293a-4f6d-48d0-b162-d5270e2d4b52-routing-overlay-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:20.337171 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.337118 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr22n\" (UniqueName: \"kubernetes.io/projected/0a9a0d3a-eced-4fd8-85fa-7c08b317c39f-kube-api-access-jr22n\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:20.337171 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.337128 5019 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5255293a-4f6d-48d0-b162-d5270e2d4b52-provider-credentials\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:20.337171 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.337137 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqrs5\" (UniqueName: \"kubernetes.io/projected/5255293a-4f6d-48d0-b162-d5270e2d4b52-kube-api-access-pqrs5\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:20.400100 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.400066 5019 scope.go:122] "RemoveContainer" containerID="26ac7fafdc762d571b83685da74bb7f1bb57e0a81e6e8bf179afddf598c401e2" Oct 31 04:45:20.407853 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.407819 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890365-cnzxg" event={"ID":"089f7eeb-81d7-4562-b906-da33b0fb5fa1","Type":"ContainerStarted","Data":"a67017a010b2d5576a21a145df5fd6336193c9f672ac4dc4822bc84a41258c6e"} Oct 31 04:45:20.410682 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.410659 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890365-rhn85" event={"ID":"01758cb6-b948-46f8-a607-210d75bb7716","Type":"ContainerStarted","Data":"81717c974ab191a8d0af168d95187831a0fd4204a3ec2e2abe275da732d1b57e"} Oct 31 04:45:20.418411 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.418378 5019 generic.go:356] "Generic (PLEG): container finished" podID="5255293a-4f6d-48d0-b162-d5270e2d4b52" containerID="b067ea72ce0b76cfff18ec7ddbbe72d75a5e003dcc75d79ff3ed1d7341864866" exitCode=0 Oct 31 04:45:20.418515 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.418416 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-b794w26n5" Oct 31 04:45:20.418515 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.418447 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-b794w26n5" event={"ID":"5255293a-4f6d-48d0-b162-d5270e2d4b52","Type":"ContainerDied","Data":"b067ea72ce0b76cfff18ec7ddbbe72d75a5e003dcc75d79ff3ed1d7341864866"} Oct 31 04:45:20.418515 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.418493 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-b794w26n5" event={"ID":"5255293a-4f6d-48d0-b162-d5270e2d4b52","Type":"ContainerDied","Data":"8672862a35d73a51de524226dc3056b67421c321fc34411f3f93c3613a6813f8"} Oct 31 04:45:20.418680 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.418521 5019 scope.go:122] "RemoveContainer" containerID="b067ea72ce0b76cfff18ec7ddbbe72d75a5e003dcc75d79ff3ed1d7341864866" Oct 31 04:45:20.421139 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.421113 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-031745-29890365-grgsg" event={"ID":"0a9a0d3a-eced-4fd8-85fa-7c08b317c39f","Type":"ContainerDied","Data":"aacf4c26761fd7e45fe5990cfc3b82dfe3dfc2434cbeb3d6d4c435ae60dd0d7e"} Oct 31 04:45:20.421245 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.421193 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-031745-29890365-grgsg" Oct 31 04:45:20.426186 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.426152 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-654741-29890365-fd9cp" event={"ID":"2fc510cb-aae5-4d8c-a612-2270ccbbf432","Type":"ContainerStarted","Data":"afe12c7c58ffc9ed537bad7556c3b2eba6baaac037d74e5dd51f67ad96ccdf37"} Oct 31 04:45:20.431656 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.431629 5019 generic.go:356] "Generic (PLEG): container finished" podID="681c73ad-b2a1-458f-b37a-cc4e9e198926" containerID="e6565c263a8a79584ae51643ce0a44c19ee9ad5314ac8b314783d278b5525499" exitCode=60 Oct 31 04:45:20.431763 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.431721 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890365-f7zkm" event={"ID":"681c73ad-b2a1-458f-b37a-cc4e9e198926","Type":"ContainerDied","Data":"e6565c263a8a79584ae51643ce0a44c19ee9ad5314ac8b314783d278b5525499"} Oct 31 04:45:20.432128 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.432107 5019 scope.go:122] "RemoveContainer" containerID="e6565c263a8a79584ae51643ce0a44c19ee9ad5314ac8b314783d278b5525499" Oct 31 04:45:20.432483 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:20.432453 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-afc519-29890365-f7zkm_odh-ai-gateway-infra(681c73ad-b2a1-458f-b37a-cc4e9e198926)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890365-f7zkm" podUID="681c73ad-b2a1-458f-b37a-cc4e9e198926" Oct 31 04:45:20.436614 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.436586 5019 generic.go:356] "Generic (PLEG): container finished" podID="b5c95c35-2055-4d02-af07-fc3574c29ecf" containerID="d11ae5ebf613a1e4f93bdc7bd7531271980df4bb8022c5151fa192d43f1bd784" exitCode=60 Oct 31 04:45:20.436732 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.436684 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-567ca6-29890365-pbxxk" event={"ID":"b5c95c35-2055-4d02-af07-fc3574c29ecf","Type":"ContainerDied","Data":"d11ae5ebf613a1e4f93bdc7bd7531271980df4bb8022c5151fa192d43f1bd784"} Oct 31 04:45:20.437097 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.437077 5019 scope.go:122] "RemoveContainer" containerID="d11ae5ebf613a1e4f93bdc7bd7531271980df4bb8022c5151fa192d43f1bd784" Oct 31 04:45:20.437411 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:20.437385 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-567ca6-29890365-pbxxk_odh-ai-gateway-infra(b5c95c35-2055-4d02-af07-fc3574c29ecf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-567ca6-29890365-pbxxk" podUID="b5c95c35-2055-4d02-af07-fc3574c29ecf" Oct 31 04:45:20.442092 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.442022 5019 generic.go:356] "Generic (PLEG): container finished" podID="e12c1431-1aa4-4c72-94a9-d3b6646ee4b2" containerID="f9b76f019713eecf23e7722932c906b1b39b6e9df8d3582efda2383f7b304b67" exitCode=60 Oct 31 04:45:20.442188 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.442114 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890365-spg6x" event={"ID":"e12c1431-1aa4-4c72-94a9-d3b6646ee4b2","Type":"ContainerDied","Data":"f9b76f019713eecf23e7722932c906b1b39b6e9df8d3582efda2383f7b304b67"} Oct 31 04:45:20.442512 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.442479 5019 scope.go:122] "RemoveContainer" containerID="f9b76f019713eecf23e7722932c906b1b39b6e9df8d3582efda2383f7b304b67" Oct 31 04:45:20.445497 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:20.445235 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890365-spg6x_odh-ai-gateway-infra(e12c1431-1aa4-4c72-94a9-d3b6646ee4b2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890365-spg6x" podUID="e12c1431-1aa4-4c72-94a9-d3b6646ee4b2" Oct 31 04:45:20.455822 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.455739 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890365-t5r5d" event={"ID":"e282b47a-b89e-48ae-88bf-347945705a1d","Type":"ContainerStarted","Data":"1acb5e38e1402ac750fa5b8db706f884a2193978327a1b5d0bfc5b591fb8c894"} Oct 31 04:45:20.459116 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.459078 5019 generic.go:356] "Generic (PLEG): container finished" podID="668f2b2b-4d01-4054-8ca0-16a4b0b4dcdd" containerID="ce0bd0a8878cd93285b10091aa652321dd737a52f2039075e515eb5e0710a6ab" exitCode=60 Oct 31 04:45:20.459284 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.459139 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890365-x6d4s" event={"ID":"668f2b2b-4d01-4054-8ca0-16a4b0b4dcdd","Type":"ContainerDied","Data":"ce0bd0a8878cd93285b10091aa652321dd737a52f2039075e515eb5e0710a6ab"} Oct 31 04:45:20.459825 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.459798 5019 scope.go:122] "RemoveContainer" containerID="ce0bd0a8878cd93285b10091aa652321dd737a52f2039075e515eb5e0710a6ab" Oct 31 04:45:20.460342 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:20.460298 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890365-x6d4s_odh-ai-gateway-infra(668f2b2b-4d01-4054-8ca0-16a4b0b4dcdd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890365-x6d4s" podUID="668f2b2b-4d01-4054-8ca0-16a4b0b4dcdd" Oct 31 04:45:20.479681 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.471833 5019 scope.go:122] "RemoveContainer" containerID="b067ea72ce0b76cfff18ec7ddbbe72d75a5e003dcc75d79ff3ed1d7341864866" Oct 31 04:45:20.481959 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:20.481924 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b067ea72ce0b76cfff18ec7ddbbe72d75a5e003dcc75d79ff3ed1d7341864866\": container with ID starting with b067ea72ce0b76cfff18ec7ddbbe72d75a5e003dcc75d79ff3ed1d7341864866 not found: ID does not exist" containerID="b067ea72ce0b76cfff18ec7ddbbe72d75a5e003dcc75d79ff3ed1d7341864866" Oct 31 04:45:20.482913 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.482883 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b067ea72ce0b76cfff18ec7ddbbe72d75a5e003dcc75d79ff3ed1d7341864866"} err="failed to get container status \"b067ea72ce0b76cfff18ec7ddbbe72d75a5e003dcc75d79ff3ed1d7341864866\": rpc error: code = NotFound desc = could not find container \"b067ea72ce0b76cfff18ec7ddbbe72d75a5e003dcc75d79ff3ed1d7341864866\": container with ID starting with b067ea72ce0b76cfff18ec7ddbbe72d75a5e003dcc75d79ff3ed1d7341864866 not found: ID does not exist" Oct 31 04:45:20.483161 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.483146 5019 scope.go:122] "RemoveContainer" containerID="3d730d3f9c7b85d238cb3ac1a8377392601da407da7445e77f937adfe8c26942" Oct 31 04:45:20.508519 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.508453 5019 scope.go:122] "RemoveContainer" containerID="f70b5cca24c60d6e0b85189092a7f43c1d21154486d89a497e314afdc41a423c" Oct 31 04:45:20.533659 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.533621 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890365-f7zkm"] Oct 31 04:45:20.535469 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.535444 5019 scope.go:122] "RemoveContainer" containerID="b2db622e7647f6ead204dd1b113aa0629a8c072eaa478afae61e4b0a025a10ea" Oct 31 04:45:20.568663 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.568408 5019 scope.go:122] "RemoveContainer" containerID="fef7cbf0d74a80fecc7705ed813dfe5c0a4e2d2bfcf8f105000cb58762243978" Oct 31 04:45:20.583638 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.583124 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-567ca6-29890365-pbxxk"] Oct 31 04:45:20.615447 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.615311 5019 scope.go:122] "RemoveContainer" containerID="56f80212ba31fd736d68c6da62484c1aa5d4fce51d7558d09d25d1a51a04d8b0" Oct 31 04:45:20.625159 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.625125 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890365-x6d4s"] Oct 31 04:45:20.666294 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.666265 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890365-spg6x"] Oct 31 04:45:20.666561 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.666370 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890365-jdjqt" Oct 31 04:45:20.767000 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.766975 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890365-74cxv" Oct 31 04:45:20.861161 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.861077 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19237217-929c-4b26-ac1f-fb27745d6dab-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19237217-929c-4b26-ac1f-fb27745d6dab-maas-api-service-ca\") pod \"19237217-929c-4b26-ac1f-fb27745d6dab\" (UID: \"19237217-929c-4b26-ac1f-fb27745d6dab\") " Oct 31 04:45:20.861307 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.861234 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6171ff86-f98a-4575-834d-2f6bd135424d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6171ff86-f98a-4575-834d-2f6bd135424d-maas-api-service-ca\") pod \"6171ff86-f98a-4575-834d-2f6bd135424d\" (UID: \"6171ff86-f98a-4575-834d-2f6bd135424d\") " Oct 31 04:45:20.861393 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.861310 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19237217-929c-4b26-ac1f-fb27745d6dab-kube-api-access-lrw28\" (UniqueName: \"kubernetes.io/projected/19237217-929c-4b26-ac1f-fb27745d6dab-kube-api-access-lrw28\") pod \"19237217-929c-4b26-ac1f-fb27745d6dab\" (UID: \"19237217-929c-4b26-ac1f-fb27745d6dab\") " Oct 31 04:45:20.861603 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.861531 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19237217-929c-4b26-ac1f-fb27745d6dab-maas-api-service-ca" pod "19237217-929c-4b26-ac1f-fb27745d6dab" (UID: "19237217-929c-4b26-ac1f-fb27745d6dab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:45:20.861603 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.861599 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6171ff86-f98a-4575-834d-2f6bd135424d-maas-api-service-ca" pod "6171ff86-f98a-4575-834d-2f6bd135424d" (UID: "6171ff86-f98a-4575-834d-2f6bd135424d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:45:20.862316 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.862291 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19237217-929c-4b26-ac1f-fb27745d6dab-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:20.862444 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.862323 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6171ff86-f98a-4575-834d-2f6bd135424d-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:20.863836 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.863797 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19237217-929c-4b26-ac1f-fb27745d6dab-kube-api-access-lrw28" pod "19237217-929c-4b26-ac1f-fb27745d6dab" (UID: "19237217-929c-4b26-ac1f-fb27745d6dab"). InnerVolumeSpecName "kube-api-access-lrw28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:45:20.963846 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.963812 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6171ff86-f98a-4575-834d-2f6bd135424d-kube-api-access-8w2cw\" (UniqueName: \"kubernetes.io/projected/6171ff86-f98a-4575-834d-2f6bd135424d-kube-api-access-8w2cw\") pod \"6171ff86-f98a-4575-834d-2f6bd135424d\" (UID: \"6171ff86-f98a-4575-834d-2f6bd135424d\") " Oct 31 04:45:20.964858 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.964825 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrw28\" (UniqueName: \"kubernetes.io/projected/19237217-929c-4b26-ac1f-fb27745d6dab-kube-api-access-lrw28\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:20.966202 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.966172 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6171ff86-f98a-4575-834d-2f6bd135424d-kube-api-access-8w2cw" pod "6171ff86-f98a-4575-834d-2f6bd135424d" (UID: "6171ff86-f98a-4575-834d-2f6bd135424d"). InnerVolumeSpecName "kube-api-access-8w2cw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:45:20.988640 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.988615 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-031745-29890365-grgsg"] Oct 31 04:45:20.995040 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:20.995014 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-031745-29890365-grgsg"] Oct 31 04:45:21.061173 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.061135 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e7fa7b6-nlmpq"] Oct 31 04:45:21.062633 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.062603 5019 cpu_manager.go:414] "containerMap: removing container" podUID="0a9a0d3a-eced-4fd8-85fa-7c08b317c39f" containerName="cleanup" Oct 31 04:45:21.062633 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.062634 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a9a0d3a-eced-4fd8-85fa-7c08b317c39f" containerName="cleanup" Oct 31 04:45:21.062835 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.062659 5019 cpu_manager.go:414] "containerMap: removing container" podUID="19237217-929c-4b26-ac1f-fb27745d6dab" containerName="cleanup" Oct 31 04:45:21.062835 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.062668 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19237217-929c-4b26-ac1f-fb27745d6dab" containerName="cleanup" Oct 31 04:45:21.062835 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.062709 5019 cpu_manager.go:414] "containerMap: removing container" podUID="6171ff86-f98a-4575-834d-2f6bd135424d" containerName="cleanup" Oct 31 04:45:21.062835 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.062717 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6171ff86-f98a-4575-834d-2f6bd135424d" containerName="cleanup" Oct 31 04:45:21.062835 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.062745 5019 cpu_manager.go:414] "containerMap: removing container" podUID="19237217-929c-4b26-ac1f-fb27745d6dab" containerName="cleanup" Oct 31 04:45:21.062835 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.062753 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19237217-929c-4b26-ac1f-fb27745d6dab" containerName="cleanup" Oct 31 04:45:21.062835 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.062781 5019 cpu_manager.go:414] "containerMap: removing container" podUID="19237217-929c-4b26-ac1f-fb27745d6dab" containerName="cleanup" Oct 31 04:45:21.062835 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.062789 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19237217-929c-4b26-ac1f-fb27745d6dab" containerName="cleanup" Oct 31 04:45:21.062835 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.062813 5019 cpu_manager.go:414] "containerMap: removing container" podUID="6171ff86-f98a-4575-834d-2f6bd135424d" containerName="cleanup" Oct 31 04:45:21.062835 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.062820 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6171ff86-f98a-4575-834d-2f6bd135424d" containerName="cleanup" Oct 31 04:45:21.062835 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.062838 5019 cpu_manager.go:414] "containerMap: removing container" podUID="837f143a-072a-446d-9062-cb6f57eb4884" containerName="payload-processing" Oct 31 04:45:21.063412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.062845 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="837f143a-072a-446d-9062-cb6f57eb4884" containerName="payload-processing" Oct 31 04:45:21.063412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.062863 5019 cpu_manager.go:414] "containerMap: removing container" podUID="5255293a-4f6d-48d0-b162-d5270e2d4b52" containerName="payload-processing" Oct 31 04:45:21.063412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.062870 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5255293a-4f6d-48d0-b162-d5270e2d4b52" containerName="payload-processing" Oct 31 04:45:21.063412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.062893 5019 cpu_manager.go:414] "containerMap: removing container" podUID="6171ff86-f98a-4575-834d-2f6bd135424d" containerName="cleanup" Oct 31 04:45:21.063412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.062900 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6171ff86-f98a-4575-834d-2f6bd135424d" containerName="cleanup" Oct 31 04:45:21.063412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.062925 5019 cpu_manager.go:414] "containerMap: removing container" podUID="0a9a0d3a-eced-4fd8-85fa-7c08b317c39f" containerName="cleanup" Oct 31 04:45:21.063412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.062932 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a9a0d3a-eced-4fd8-85fa-7c08b317c39f" containerName="cleanup" Oct 31 04:45:21.063412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.063171 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="6171ff86-f98a-4575-834d-2f6bd135424d" containerName="cleanup" Oct 31 04:45:21.063412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.063194 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a9a0d3a-eced-4fd8-85fa-7c08b317c39f" containerName="cleanup" Oct 31 04:45:21.063412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.063208 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="19237217-929c-4b26-ac1f-fb27745d6dab" containerName="cleanup" Oct 31 04:45:21.063412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.063225 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="6171ff86-f98a-4575-834d-2f6bd135424d" containerName="cleanup" Oct 31 04:45:21.063412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.063243 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="19237217-929c-4b26-ac1f-fb27745d6dab" containerName="cleanup" Oct 31 04:45:21.063412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.063254 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a9a0d3a-eced-4fd8-85fa-7c08b317c39f" containerName="cleanup" Oct 31 04:45:21.063412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.063264 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="837f143a-072a-446d-9062-cb6f57eb4884" containerName="payload-processing" Oct 31 04:45:21.063412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.063271 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="5255293a-4f6d-48d0-b162-d5270e2d4b52" containerName="payload-processing" Oct 31 04:45:21.066183 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.066117 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w2cw\" (UniqueName: \"kubernetes.io/projected/6171ff86-f98a-4575-834d-2f6bd135424d-kube-api-access-8w2cw\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:21.067424 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.067400 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-b794w26n5"] Oct 31 04:45:21.067831 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.067513 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e7fa7b6-nlmpq" Oct 31 04:45:21.073628 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.073603 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-b794w26n5"] Oct 31 04:45:21.083347 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.082245 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:45:21.087119 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.087017 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e7fa7b6-nlmpq"] Oct 31 04:45:21.167761 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.167669 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9192eff-e3eb-4bef-a0e7-447afe65cc78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e7fa7b6-nlmpq\" (UID: \"a9192eff-e3eb-4bef-a0e7-447afe65cc78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e7fa7b6-nlmpq" Oct 31 04:45:21.167761 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.167726 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9192eff-e3eb-4bef-a0e7-447afe65cc78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e7fa7b6-nlmpq\" (UID: \"a9192eff-e3eb-4bef-a0e7-447afe65cc78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e7fa7b6-nlmpq" Oct 31 04:45:21.168047 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.168021 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rtgq\" (UniqueName: \"kubernetes.io/projected/a9192eff-e3eb-4bef-a0e7-447afe65cc78-kube-api-access-5rtgq\") pod \"maas-api-revoke-sub-1e7fa7b6-nlmpq\" (UID: \"a9192eff-e3eb-4bef-a0e7-447afe65cc78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e7fa7b6-nlmpq" Oct 31 04:45:21.269637 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.269595 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9192eff-e3eb-4bef-a0e7-447afe65cc78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e7fa7b6-nlmpq\" (UID: \"a9192eff-e3eb-4bef-a0e7-447afe65cc78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e7fa7b6-nlmpq" Oct 31 04:45:21.269868 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.269668 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9192eff-e3eb-4bef-a0e7-447afe65cc78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e7fa7b6-nlmpq\" (UID: \"a9192eff-e3eb-4bef-a0e7-447afe65cc78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e7fa7b6-nlmpq" Oct 31 04:45:21.270032 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.270014 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rtgq\" (UniqueName: \"kubernetes.io/projected/a9192eff-e3eb-4bef-a0e7-447afe65cc78-kube-api-access-5rtgq\") pod \"maas-api-revoke-sub-1e7fa7b6-nlmpq\" (UID: \"a9192eff-e3eb-4bef-a0e7-447afe65cc78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e7fa7b6-nlmpq" Oct 31 04:45:21.270591 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.270559 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9192eff-e3eb-4bef-a0e7-447afe65cc78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e7fa7b6-nlmpq\" (UID: \"a9192eff-e3eb-4bef-a0e7-447afe65cc78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e7fa7b6-nlmpq" Oct 31 04:45:21.272209 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.272185 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9192eff-e3eb-4bef-a0e7-447afe65cc78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e7fa7b6-nlmpq\" (UID: \"a9192eff-e3eb-4bef-a0e7-447afe65cc78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e7fa7b6-nlmpq" Oct 31 04:45:21.293302 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.293275 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rtgq\" (UniqueName: \"kubernetes.io/projected/a9192eff-e3eb-4bef-a0e7-447afe65cc78-kube-api-access-5rtgq\") pod \"maas-api-revoke-sub-1e7fa7b6-nlmpq\" (UID: \"a9192eff-e3eb-4bef-a0e7-447afe65cc78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e7fa7b6-nlmpq" Oct 31 04:45:21.388851 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.388816 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e7fa7b6-nlmpq" Oct 31 04:45:21.401356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.401099 5019 scope.go:122] "RemoveContainer" containerID="1e8a810e742b555a2f60ad8f1a8d9180f335898db3e3cc93e343b945dfaad3ff" Oct 31 04:45:21.401356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.401202 5019 scope.go:122] "RemoveContainer" containerID="e223e7c544ed8d169986c16d73117d9d2a99c0ddbfaabd6d237d93efa4679583" Oct 31 04:45:21.414616 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.414584 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a9a0d3a-eced-4fd8-85fa-7c08b317c39f" path="/var/lib/kubelet/pods/0a9a0d3a-eced-4fd8-85fa-7c08b317c39f/volumes" Oct 31 04:45:21.415107 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.415084 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5255293a-4f6d-48d0-b162-d5270e2d4b52" path="/var/lib/kubelet/pods/5255293a-4f6d-48d0-b162-d5270e2d4b52/volumes" Oct 31 04:45:21.489356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.482244 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890365-cnzxg"] Oct 31 04:45:21.489356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.486525 5019 generic.go:356] "Generic (PLEG): container finished" podID="e282b47a-b89e-48ae-88bf-347945705a1d" containerID="1acb5e38e1402ac750fa5b8db706f884a2193978327a1b5d0bfc5b591fb8c894" exitCode=60 Oct 31 04:45:21.489356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.486649 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890365-t5r5d" event={"ID":"e282b47a-b89e-48ae-88bf-347945705a1d","Type":"ContainerDied","Data":"1acb5e38e1402ac750fa5b8db706f884a2193978327a1b5d0bfc5b591fb8c894"} Oct 31 04:45:21.489356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.486706 5019 scope.go:122] "RemoveContainer" containerID="ab2cc3789670dc2f1e4c409c77ef9a5d7d877e9ffc0d16521ebc00ee9b91f04d" Oct 31 04:45:21.489356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.487169 5019 scope.go:122] "RemoveContainer" containerID="1acb5e38e1402ac750fa5b8db706f884a2193978327a1b5d0bfc5b591fb8c894" Oct 31 04:45:21.489356 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:21.487589 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890365-t5r5d_odh-ai-gateway-infra(e282b47a-b89e-48ae-88bf-347945705a1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890365-t5r5d" podUID="e282b47a-b89e-48ae-88bf-347945705a1d" Oct 31 04:45:21.519034 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.518665 5019 generic.go:356] "Generic (PLEG): container finished" podID="089f7eeb-81d7-4562-b906-da33b0fb5fa1" containerID="a67017a010b2d5576a21a145df5fd6336193c9f672ac4dc4822bc84a41258c6e" exitCode=60 Oct 31 04:45:21.519034 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.518832 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890365-cnzxg" event={"ID":"089f7eeb-81d7-4562-b906-da33b0fb5fa1","Type":"ContainerDied","Data":"a67017a010b2d5576a21a145df5fd6336193c9f672ac4dc4822bc84a41258c6e"} Oct 31 04:45:21.524993 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.523378 5019 generic.go:356] "Generic (PLEG): container finished" podID="01758cb6-b948-46f8-a607-210d75bb7716" containerID="81717c974ab191a8d0af168d95187831a0fd4204a3ec2e2abe275da732d1b57e" exitCode=60 Oct 31 04:45:21.524993 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.523503 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890365-rhn85" event={"ID":"01758cb6-b948-46f8-a607-210d75bb7716","Type":"ContainerDied","Data":"81717c974ab191a8d0af168d95187831a0fd4204a3ec2e2abe275da732d1b57e"} Oct 31 04:45:21.524993 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.524015 5019 scope.go:122] "RemoveContainer" containerID="81717c974ab191a8d0af168d95187831a0fd4204a3ec2e2abe275da732d1b57e" Oct 31 04:45:21.524993 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:21.524568 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-766728-29890365-rhn85_odh-ai-gateway-infra(01758cb6-b948-46f8-a607-210d75bb7716)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890365-rhn85" podUID="01758cb6-b948-46f8-a607-210d75bb7716" Oct 31 04:45:21.529034 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.529017 5019 scope.go:122] "RemoveContainer" containerID="e0739fb7237fd34717fb9e1f95b1f9e6cda1d45d5910860608e806d33b17f942" Oct 31 04:45:21.531896 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.531679 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890365-wwx6m" event={"ID":"d992b6ec-1a57-4ca9-af09-370245d1ccfe","Type":"ContainerStarted","Data":"80dbdf76c886b3ab707b1f19c50dc6624eb670ee672b008e5018c9fb28f8b4f2"} Oct 31 04:45:21.533472 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.533448 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890365-74cxv" Oct 31 04:45:21.533472 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.533466 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890365-74cxv" event={"ID":"6171ff86-f98a-4575-834d-2f6bd135424d","Type":"ContainerDied","Data":"6b2de0ec0aafe7380dee186b5ed7b37d9c4a49e7a4449dd617cb86a643a021d7"} Oct 31 04:45:21.535595 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.535561 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890365-jdjqt" Oct 31 04:45:21.535703 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.535601 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890365-jdjqt" event={"ID":"19237217-929c-4b26-ac1f-fb27745d6dab","Type":"ContainerDied","Data":"207b1bd41b0b17acf31881e10772678f6609ca6a17efc7119500147ce781d487"} Oct 31 04:45:21.551408 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.551380 5019 scope.go:122] "RemoveContainer" containerID="00c377295d4e367fda0adfca705f1d40cb9737d8d2c84cedcf2660e3df5393b2" Oct 31 04:45:21.556865 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.556815 5019 generic.go:356] "Generic (PLEG): container finished" podID="2fc510cb-aae5-4d8c-a612-2270ccbbf432" containerID="afe12c7c58ffc9ed537bad7556c3b2eba6baaac037d74e5dd51f67ad96ccdf37" exitCode=60 Oct 31 04:45:21.557002 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.556921 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-654741-29890365-fd9cp" event={"ID":"2fc510cb-aae5-4d8c-a612-2270ccbbf432","Type":"ContainerDied","Data":"afe12c7c58ffc9ed537bad7556c3b2eba6baaac037d74e5dd51f67ad96ccdf37"} Oct 31 04:45:21.557447 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.557424 5019 scope.go:122] "RemoveContainer" containerID="afe12c7c58ffc9ed537bad7556c3b2eba6baaac037d74e5dd51f67ad96ccdf37" Oct 31 04:45:21.557882 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:21.557853 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-654741-29890365-fd9cp_odh-ai-gateway-infra(2fc510cb-aae5-4d8c-a612-2270ccbbf432)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-654741-29890365-fd9cp" podUID="2fc510cb-aae5-4d8c-a612-2270ccbbf432" Oct 31 04:45:21.572236 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.572149 5019 scope.go:122] "RemoveContainer" containerID="74b15b58ba442dc7b7d20e608540f82c903e0620a4ab8fe4cc611a0888f2e0f3" Oct 31 04:45:21.633476 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.632318 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890365-rhn85"] Oct 31 04:45:21.639662 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.639620 5019 scope.go:122] "RemoveContainer" containerID="9b0f18a8e97e78410152c9fdc19e5b84b10de1ab8b598b20392a7eee0b559f19" Oct 31 04:45:21.741277 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.741193 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e7fa7b6-nlmpq"] Oct 31 04:45:21.752676 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.752645 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-654741-29890365-fd9cp"] Oct 31 04:45:21.803094 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:45:21.802717 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9192eff_e3eb_4bef_a0e7_447afe65cc78.slice/crio-49d7fdf6165996cf3c614db3d8943caeb50e14ea5dbfef76fefd835e41b88d2c WatchSource:0}: Error finding container 49d7fdf6165996cf3c614db3d8943caeb50e14ea5dbfef76fefd835e41b88d2c: Status 404 returned error can't find the container with id 49d7fdf6165996cf3c614db3d8943caeb50e14ea5dbfef76fefd835e41b88d2c Oct 31 04:45:21.830151 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.830038 5019 scope.go:122] "RemoveContainer" containerID="667d384355c86ef041aeb1f21583791296d4c936620f6b2176633c909f940c17" Oct 31 04:45:21.845022 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.844999 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890365-t5r5d"] Oct 31 04:45:21.854901 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.854852 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890365-jdjqt"] Oct 31 04:45:21.865250 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.865193 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890365-jdjqt"] Oct 31 04:45:21.919059 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.918689 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890365-74cxv"] Oct 31 04:45:21.921968 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:21.921942 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890365-74cxv"] Oct 31 04:45:22.058847 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:22.058749 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890365-spg6x" Oct 31 04:45:22.202503 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:22.202448 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890365-x6d4s" Oct 31 04:45:22.205775 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:22.205754 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890365-cnzxg" Oct 31 04:45:22.210449 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:22.210422 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890365-f7zkm" Oct 31 04:45:22.214235 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:22.214215 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-567ca6-29890365-pbxxk" Oct 31 04:45:22.238272 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:22.238249 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e12c1431-1aa4-4c72-94a9-d3b6646ee4b2-kube-api-access-7mjls\" (UniqueName: \"kubernetes.io/projected/e12c1431-1aa4-4c72-94a9-d3b6646ee4b2-kube-api-access-7mjls\") pod \"e12c1431-1aa4-4c72-94a9-d3b6646ee4b2\" (UID: \"e12c1431-1aa4-4c72-94a9-d3b6646ee4b2\") " Oct 31 04:45:22.238422 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:22.238373 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e12c1431-1aa4-4c72-94a9-d3b6646ee4b2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e12c1431-1aa4-4c72-94a9-d3b6646ee4b2-maas-api-service-ca\") pod \"e12c1431-1aa4-4c72-94a9-d3b6646ee4b2\" (UID: \"e12c1431-1aa4-4c72-94a9-d3b6646ee4b2\") " Oct 31 04:45:22.238795 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:22.238768 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e12c1431-1aa4-4c72-94a9-d3b6646ee4b2-maas-api-service-ca" pod "e12c1431-1aa4-4c72-94a9-d3b6646ee4b2" (UID: "e12c1431-1aa4-4c72-94a9-d3b6646ee4b2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:45:22.239261 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:22.239240 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e12c1431-1aa4-4c72-94a9-d3b6646ee4b2-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:22.240705 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:22.240682 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e12c1431-1aa4-4c72-94a9-d3b6646ee4b2-kube-api-access-7mjls" pod "e12c1431-1aa4-4c72-94a9-d3b6646ee4b2" (UID: "e12c1431-1aa4-4c72-94a9-d3b6646ee4b2"). InnerVolumeSpecName "kube-api-access-7mjls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:45:22.339833 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:22.339797 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/668f2b2b-4d01-4054-8ca0-16a4b0b4dcdd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/668f2b2b-4d01-4054-8ca0-16a4b0b4dcdd-maas-api-service-ca\") pod \"668f2b2b-4d01-4054-8ca0-16a4b0b4dcdd\" (UID: \"668f2b2b-4d01-4054-8ca0-16a4b0b4dcdd\") " Oct 31 04:45:22.340026 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:22.340006 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/681c73ad-b2a1-458f-b37a-cc4e9e198926-kube-api-access-hc5fg\" (UniqueName: \"kubernetes.io/projected/681c73ad-b2a1-458f-b37a-cc4e9e198926-kube-api-access-hc5fg\") pod \"681c73ad-b2a1-458f-b37a-cc4e9e198926\" (UID: \"681c73ad-b2a1-458f-b37a-cc4e9e198926\") " Oct 31 04:45:22.340100 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:22.340085 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/681c73ad-b2a1-458f-b37a-cc4e9e198926-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/681c73ad-b2a1-458f-b37a-cc4e9e198926-maas-api-service-ca\") pod \"681c73ad-b2a1-458f-b37a-cc4e9e198926\" (UID: \"681c73ad-b2a1-458f-b37a-cc4e9e198926\") " Oct 31 04:45:22.340155 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:22.340123 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/089f7eeb-81d7-4562-b906-da33b0fb5fa1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/089f7eeb-81d7-4562-b906-da33b0fb5fa1-maas-api-service-ca\") pod \"089f7eeb-81d7-4562-b906-da33b0fb5fa1\" (UID: \"089f7eeb-81d7-4562-b906-da33b0fb5fa1\") " Oct 31 04:45:22.340212 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:22.340175 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5c95c35-2055-4d02-af07-fc3574c29ecf-kube-api-access-v6sgb\" (UniqueName: \"kubernetes.io/projected/b5c95c35-2055-4d02-af07-fc3574c29ecf-kube-api-access-v6sgb\") pod \"b5c95c35-2055-4d02-af07-fc3574c29ecf\" (UID: \"b5c95c35-2055-4d02-af07-fc3574c29ecf\") " Oct 31 04:45:22.340313 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:22.340293 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/089f7eeb-81d7-4562-b906-da33b0fb5fa1-kube-api-access-8hmzg\" (UniqueName: \"kubernetes.io/projected/089f7eeb-81d7-4562-b906-da33b0fb5fa1-kube-api-access-8hmzg\") pod \"089f7eeb-81d7-4562-b906-da33b0fb5fa1\" (UID: \"089f7eeb-81d7-4562-b906-da33b0fb5fa1\") " Oct 31 04:45:22.340542 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:22.340512 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/681c73ad-b2a1-458f-b37a-cc4e9e198926-maas-api-service-ca" pod "681c73ad-b2a1-458f-b37a-cc4e9e198926" (UID: "681c73ad-b2a1-458f-b37a-cc4e9e198926"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:45:22.340828 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:22.340806 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5c95c35-2055-4d02-af07-fc3574c29ecf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5c95c35-2055-4d02-af07-fc3574c29ecf-maas-api-service-ca\") pod \"b5c95c35-2055-4d02-af07-fc3574c29ecf\" (UID: \"b5c95c35-2055-4d02-af07-fc3574c29ecf\") " Oct 31 04:45:22.341272 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:22.340992 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/668f2b2b-4d01-4054-8ca0-16a4b0b4dcdd-kube-api-access-rqp9r\" (UniqueName: \"kubernetes.io/projected/668f2b2b-4d01-4054-8ca0-16a4b0b4dcdd-kube-api-access-rqp9r\") pod \"668f2b2b-4d01-4054-8ca0-16a4b0b4dcdd\" (UID: \"668f2b2b-4d01-4054-8ca0-16a4b0b4dcdd\") " Oct 31 04:45:22.341430 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:22.340920 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/089f7eeb-81d7-4562-b906-da33b0fb5fa1-maas-api-service-ca" pod "089f7eeb-81d7-4562-b906-da33b0fb5fa1" (UID: "089f7eeb-81d7-4562-b906-da33b0fb5fa1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:45:22.341430 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:22.341033 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/668f2b2b-4d01-4054-8ca0-16a4b0b4dcdd-maas-api-service-ca" pod "668f2b2b-4d01-4054-8ca0-16a4b0b4dcdd" (UID: "668f2b2b-4d01-4054-8ca0-16a4b0b4dcdd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:45:22.341430 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:22.341248 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5c95c35-2055-4d02-af07-fc3574c29ecf-maas-api-service-ca" pod "b5c95c35-2055-4d02-af07-fc3574c29ecf" (UID: "b5c95c35-2055-4d02-af07-fc3574c29ecf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:45:22.342024 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:22.342004 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mjls\" (UniqueName: \"kubernetes.io/projected/e12c1431-1aa4-4c72-94a9-d3b6646ee4b2-kube-api-access-7mjls\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:22.342110 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:22.342028 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/681c73ad-b2a1-458f-b37a-cc4e9e198926-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:22.342110 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:22.342043 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/089f7eeb-81d7-4562-b906-da33b0fb5fa1-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:22.342110 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:22.342058 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5c95c35-2055-4d02-af07-fc3574c29ecf-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:22.342110 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:22.342071 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/668f2b2b-4d01-4054-8ca0-16a4b0b4dcdd-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:22.344222 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:22.344197 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/089f7eeb-81d7-4562-b906-da33b0fb5fa1-kube-api-access-8hmzg" pod "089f7eeb-81d7-4562-b906-da33b0fb5fa1" (UID: "089f7eeb-81d7-4562-b906-da33b0fb5fa1"). InnerVolumeSpecName "kube-api-access-8hmzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:45:22.344394 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:22.344239 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/681c73ad-b2a1-458f-b37a-cc4e9e198926-kube-api-access-hc5fg" pod "681c73ad-b2a1-458f-b37a-cc4e9e198926" (UID: "681c73ad-b2a1-458f-b37a-cc4e9e198926"). InnerVolumeSpecName "kube-api-access-hc5fg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:45:22.344545 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:22.344526 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5c95c35-2055-4d02-af07-fc3574c29ecf-kube-api-access-v6sgb" pod "b5c95c35-2055-4d02-af07-fc3574c29ecf" (UID: "b5c95c35-2055-4d02-af07-fc3574c29ecf"). InnerVolumeSpecName "kube-api-access-v6sgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:45:22.345248 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:22.345228 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/668f2b2b-4d01-4054-8ca0-16a4b0b4dcdd-kube-api-access-rqp9r" pod "668f2b2b-4d01-4054-8ca0-16a4b0b4dcdd" (UID: "668f2b2b-4d01-4054-8ca0-16a4b0b4dcdd"). InnerVolumeSpecName "kube-api-access-rqp9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:45:22.401746 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:22.401716 5019 scope.go:122] "RemoveContainer" containerID="fecbb7f9f57537ec516e66644012163fe52fc628357ee666513264c015ed8a71" Oct 31 04:45:22.402490 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:22.402469 5019 scope.go:122] "RemoveContainer" containerID="88fa47d69e8da32266f83de06f0095bb431dce730b18aef1ff3794c63d0d21af" Oct 31 04:45:22.411129 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:22.410781 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:45:22.443370 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:22.443302 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc5fg\" (UniqueName: \"kubernetes.io/projected/681c73ad-b2a1-458f-b37a-cc4e9e198926-kube-api-access-hc5fg\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:22.443370 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:22.443353 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6sgb\" (UniqueName: \"kubernetes.io/projected/b5c95c35-2055-4d02-af07-fc3574c29ecf-kube-api-access-v6sgb\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:22.443507 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:22.443371 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hmzg\" (UniqueName: \"kubernetes.io/projected/089f7eeb-81d7-4562-b906-da33b0fb5fa1-kube-api-access-8hmzg\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:22.443507 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:22.443389 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqp9r\" (UniqueName: \"kubernetes.io/projected/668f2b2b-4d01-4054-8ca0-16a4b0b4dcdd-kube-api-access-rqp9r\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:22.828918 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:22.827384 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890365-wwx6m"] Oct 31 04:45:22.832724 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:22.831607 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890365-f7zkm" event={"ID":"681c73ad-b2a1-458f-b37a-cc4e9e198926","Type":"ContainerDied","Data":"6193130c65e7a6a5fe7ba135caceb90c92d165281e6786eadfb587fd2d0c303c"} Oct 31 04:45:22.832724 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:22.831690 5019 scope.go:122] "RemoveContainer" containerID="e6565c263a8a79584ae51643ce0a44c19ee9ad5314ac8b314783d278b5525499" Oct 31 04:45:22.832724 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:22.831822 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890365-f7zkm" Oct 31 04:45:22.889699 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:22.870493 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-567ca6-29890365-pbxxk" event={"ID":"b5c95c35-2055-4d02-af07-fc3574c29ecf","Type":"ContainerDied","Data":"8d4e54df4b39398ab7e23377317b6332b9fc2e0f6e3508d6e162381230970d22"} Oct 31 04:45:22.889699 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:22.870632 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-567ca6-29890365-pbxxk" Oct 31 04:45:22.889699 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:22.876679 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890365-spg6x" event={"ID":"e12c1431-1aa4-4c72-94a9-d3b6646ee4b2","Type":"ContainerDied","Data":"88cd50a8036975ca8bd0a4f62b3ec52457caa313421c4350f7e5761ff4611a9a"} Oct 31 04:45:22.889699 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:22.876811 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890365-spg6x" Oct 31 04:45:23.028432 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:23.028381 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890365-x6d4s" event={"ID":"668f2b2b-4d01-4054-8ca0-16a4b0b4dcdd","Type":"ContainerDied","Data":"d242716c53eedbcf4cb962a5c7e0f339fa5aeda65b65d1bf630495a0335dcd58"} Oct 31 04:45:23.036693 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:23.036635 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890365-cnzxg" event={"ID":"089f7eeb-81d7-4562-b906-da33b0fb5fa1","Type":"ContainerDied","Data":"6b600a1669134983f95f2d94a01870e70981fba49d4836ff9ba65ae9509ce77b"} Oct 31 04:45:23.037047 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:23.031281 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890365-cnzxg" Oct 31 04:45:23.049877 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:23.049837 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890365-5h7hv" event={"ID":"bc26f88e-b019-4540-a826-63fb1d624a8d","Type":"ContainerStarted","Data":"a7bf9ddf2aad62cec0707561b935f8e1f3c6aca5478a810606cf252508f66ed1"} Oct 31 04:45:23.052413 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:23.051595 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890365-x6d4s" Oct 31 04:45:23.207649 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:23.198117 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-567ca6-29890365-pbxxk"] Oct 31 04:45:23.207649 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:23.203279 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-567ca6-29890365-pbxxk"] Oct 31 04:45:23.230625 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:23.226606 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e7fa7b6-nlmpq" event={"ID":"a9192eff-e3eb-4bef-a0e7-447afe65cc78","Type":"ContainerStarted","Data":"829a502f82b4af793a6be4060dbebd11449d3d775c10a53f82926e9ce7f98035"} Oct 31 04:45:23.230625 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:23.226669 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e7fa7b6-nlmpq" event={"ID":"a9192eff-e3eb-4bef-a0e7-447afe65cc78","Type":"ContainerStarted","Data":"49d7fdf6165996cf3c614db3d8943caeb50e14ea5dbfef76fefd835e41b88d2c"} Oct 31 04:45:23.230625 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:23.229924 5019 generic.go:356] "Generic (PLEG): container finished" podID="d992b6ec-1a57-4ca9-af09-370245d1ccfe" containerID="80dbdf76c886b3ab707b1f19c50dc6624eb670ee672b008e5018c9fb28f8b4f2" exitCode=60 Oct 31 04:45:23.230625 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:23.230006 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890365-wwx6m" event={"ID":"d992b6ec-1a57-4ca9-af09-370245d1ccfe","Type":"ContainerDied","Data":"80dbdf76c886b3ab707b1f19c50dc6624eb670ee672b008e5018c9fb28f8b4f2"} Oct 31 04:45:23.248364 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:23.244692 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890365-xj5rc" event={"ID":"99be4628-816c-46f6-abcf-9de2a8452895","Type":"ContainerStarted","Data":"c4b4f322482b92d21e2760d07cc028a74e07b5751d1bb02627ccb814978c536a"} Oct 31 04:45:23.248364 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:23.245216 5019 scope.go:122] "RemoveContainer" containerID="c4b4f322482b92d21e2760d07cc028a74e07b5751d1bb02627ccb814978c536a" Oct 31 04:45:23.248364 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:23.245662 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-4c2387-29890365-xj5rc_odh-ai-gateway-infra(99be4628-816c-46f6-abcf-9de2a8452895)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890365-xj5rc" podUID="99be4628-816c-46f6-abcf-9de2a8452895" Oct 31 04:45:23.270087 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:23.267268 5019 scope.go:122] "RemoveContainer" containerID="d11ae5ebf613a1e4f93bdc7bd7531271980df4bb8022c5151fa192d43f1bd784" Oct 31 04:45:23.360395 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:23.359179 5019 scope.go:122] "RemoveContainer" containerID="f9b76f019713eecf23e7722932c906b1b39b6e9df8d3582efda2383f7b304b67" Oct 31 04:45:23.364411 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:23.364274 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890365-f7zkm"] Oct 31 04:45:23.371393 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:23.371305 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890365-f7zkm"] Oct 31 04:45:23.448833 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:23.446750 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19237217-929c-4b26-ac1f-fb27745d6dab" path="/var/lib/kubelet/pods/19237217-929c-4b26-ac1f-fb27745d6dab/volumes" Oct 31 04:45:23.448833 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:23.447481 5019 scope.go:122] "RemoveContainer" containerID="ce0bd0a8878cd93285b10091aa652321dd737a52f2039075e515eb5e0710a6ab" Oct 31 04:45:23.448833 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:23.447526 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6171ff86-f98a-4575-834d-2f6bd135424d" path="/var/lib/kubelet/pods/6171ff86-f98a-4575-834d-2f6bd135424d/volumes" Oct 31 04:45:23.448833 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:23.448237 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="681c73ad-b2a1-458f-b37a-cc4e9e198926" path="/var/lib/kubelet/pods/681c73ad-b2a1-458f-b37a-cc4e9e198926/volumes" Oct 31 04:45:23.449920 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:23.449826 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5c95c35-2055-4d02-af07-fc3574c29ecf" path="/var/lib/kubelet/pods/b5c95c35-2055-4d02-af07-fc3574c29ecf/volumes" Oct 31 04:45:23.470741 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:23.469837 5019 scope.go:122] "RemoveContainer" containerID="a67017a010b2d5576a21a145df5fd6336193c9f672ac4dc4822bc84a41258c6e" Oct 31 04:45:23.474482 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:23.474446 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890365-x6d4s"] Oct 31 04:45:23.489858 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:23.489831 5019 scope.go:122] "RemoveContainer" containerID="26ac7fafdc762d571b83685da74bb7f1bb57e0a81e6e8bf179afddf598c401e2" Oct 31 04:45:23.518771 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:23.517617 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890365-x6d4s"] Oct 31 04:45:23.957954 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:23.956896 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890365-t5r5d" Oct 31 04:45:24.024425 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.006804 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e282b47a-b89e-48ae-88bf-347945705a1d-kube-api-access-r4sdj\" (UniqueName: \"kubernetes.io/projected/e282b47a-b89e-48ae-88bf-347945705a1d-kube-api-access-r4sdj\") pod \"e282b47a-b89e-48ae-88bf-347945705a1d\" (UID: \"e282b47a-b89e-48ae-88bf-347945705a1d\") " Oct 31 04:45:24.024425 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.006967 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e282b47a-b89e-48ae-88bf-347945705a1d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e282b47a-b89e-48ae-88bf-347945705a1d-maas-api-service-ca\") pod \"e282b47a-b89e-48ae-88bf-347945705a1d\" (UID: \"e282b47a-b89e-48ae-88bf-347945705a1d\") " Oct 31 04:45:24.024425 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.021947 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e282b47a-b89e-48ae-88bf-347945705a1d-kube-api-access-r4sdj" pod "e282b47a-b89e-48ae-88bf-347945705a1d" (UID: "e282b47a-b89e-48ae-88bf-347945705a1d"). InnerVolumeSpecName "kube-api-access-r4sdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:45:24.036650 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.036596 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e282b47a-b89e-48ae-88bf-347945705a1d-maas-api-service-ca" pod "e282b47a-b89e-48ae-88bf-347945705a1d" (UID: "e282b47a-b89e-48ae-88bf-347945705a1d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:45:24.066755 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.066694 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890365-cnzxg"] Oct 31 04:45:24.125277 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.112313 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4sdj\" (UniqueName: \"kubernetes.io/projected/e282b47a-b89e-48ae-88bf-347945705a1d-kube-api-access-r4sdj\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:24.125277 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.120442 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e282b47a-b89e-48ae-88bf-347945705a1d-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:24.144026 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.142891 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890365-rhn85" Oct 31 04:45:24.145172 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.145097 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890365-wwx6m" Oct 31 04:45:24.166145 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.165441 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890365-cnzxg"] Oct 31 04:45:24.180581 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.180498 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-654741-29890365-fd9cp" Oct 31 04:45:24.184071 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.183457 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890365-spg6x"] Oct 31 04:45:24.192288 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.192263 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890365-spg6x"] Oct 31 04:45:24.261246 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.261210 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890365-5h7hv"] Oct 31 04:45:24.331381 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:24.329919 5019 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7abcb04_58ef_4efd_a350_c8f5286ebc14.slice/crio-c83e84291d71b0272dfe2519562d4d495078d77f90565f38c43e0d1321445487.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7abcb04_58ef_4efd_a350_c8f5286ebc14.slice/crio-conmon-c83e84291d71b0272dfe2519562d4d495078d77f90565f38c43e0d1321445487.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda53b729a_b280_41d4_8661_44826b486246.slice/crio-e90d16e0b494de0a9d2f11baa512d3e048afa7de178595ecb6e44d549154ed8e.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda53b729a_b280_41d4_8661_44826b486246.slice/crio-conmon-e90d16e0b494de0a9d2f11baa512d3e048afa7de178595ecb6e44d549154ed8e.scope\": RecentStats: unable to find data in memory cache]" Oct 31 04:45:24.346889 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.346801 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fc510cb-aae5-4d8c-a612-2270ccbbf432-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fc510cb-aae5-4d8c-a612-2270ccbbf432-maas-api-service-ca\") pod \"2fc510cb-aae5-4d8c-a612-2270ccbbf432\" (UID: \"2fc510cb-aae5-4d8c-a612-2270ccbbf432\") " Oct 31 04:45:24.347025 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.346961 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d992b6ec-1a57-4ca9-af09-370245d1ccfe-kube-api-access-4jrzs\" (UniqueName: \"kubernetes.io/projected/d992b6ec-1a57-4ca9-af09-370245d1ccfe-kube-api-access-4jrzs\") pod \"d992b6ec-1a57-4ca9-af09-370245d1ccfe\" (UID: \"d992b6ec-1a57-4ca9-af09-370245d1ccfe\") " Oct 31 04:45:24.347096 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.347078 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01758cb6-b948-46f8-a607-210d75bb7716-kube-api-access-ljm75\" (UniqueName: \"kubernetes.io/projected/01758cb6-b948-46f8-a607-210d75bb7716-kube-api-access-ljm75\") pod \"01758cb6-b948-46f8-a607-210d75bb7716\" (UID: \"01758cb6-b948-46f8-a607-210d75bb7716\") " Oct 31 04:45:24.347142 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.347128 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01758cb6-b948-46f8-a607-210d75bb7716-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01758cb6-b948-46f8-a607-210d75bb7716-maas-api-service-ca\") pod \"01758cb6-b948-46f8-a607-210d75bb7716\" (UID: \"01758cb6-b948-46f8-a607-210d75bb7716\") " Oct 31 04:45:24.347311 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.347295 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fc510cb-aae5-4d8c-a612-2270ccbbf432-kube-api-access-9vl29\" (UniqueName: \"kubernetes.io/projected/2fc510cb-aae5-4d8c-a612-2270ccbbf432-kube-api-access-9vl29\") pod \"2fc510cb-aae5-4d8c-a612-2270ccbbf432\" (UID: \"2fc510cb-aae5-4d8c-a612-2270ccbbf432\") " Oct 31 04:45:24.347398 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.347365 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d992b6ec-1a57-4ca9-af09-370245d1ccfe-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d992b6ec-1a57-4ca9-af09-370245d1ccfe-maas-api-service-ca\") pod \"d992b6ec-1a57-4ca9-af09-370245d1ccfe\" (UID: \"d992b6ec-1a57-4ca9-af09-370245d1ccfe\") " Oct 31 04:45:24.349147 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.349118 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d992b6ec-1a57-4ca9-af09-370245d1ccfe-maas-api-service-ca" pod "d992b6ec-1a57-4ca9-af09-370245d1ccfe" (UID: "d992b6ec-1a57-4ca9-af09-370245d1ccfe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:45:24.350295 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.349946 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fc510cb-aae5-4d8c-a612-2270ccbbf432-maas-api-service-ca" pod "2fc510cb-aae5-4d8c-a612-2270ccbbf432" (UID: "2fc510cb-aae5-4d8c-a612-2270ccbbf432"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:45:24.354721 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.351512 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01758cb6-b948-46f8-a607-210d75bb7716-maas-api-service-ca" pod "01758cb6-b948-46f8-a607-210d75bb7716" (UID: "01758cb6-b948-46f8-a607-210d75bb7716"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:45:24.354721 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.352097 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890365-t5r5d" event={"ID":"e282b47a-b89e-48ae-88bf-347945705a1d","Type":"ContainerDied","Data":"eefd7160bd6f80cca3e25e453383e3d1a1e134abd440d87424219edd605c239f"} Oct 31 04:45:24.354721 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.352146 5019 scope.go:122] "RemoveContainer" containerID="1acb5e38e1402ac750fa5b8db706f884a2193978327a1b5d0bfc5b591fb8c894" Oct 31 04:45:24.354721 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.352210 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890365-t5r5d" Oct 31 04:45:24.355024 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.354980 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d992b6ec-1a57-4ca9-af09-370245d1ccfe-kube-api-access-4jrzs" pod "d992b6ec-1a57-4ca9-af09-370245d1ccfe" (UID: "d992b6ec-1a57-4ca9-af09-370245d1ccfe"). InnerVolumeSpecName "kube-api-access-4jrzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:45:24.355702 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.355451 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fc510cb-aae5-4d8c-a612-2270ccbbf432-kube-api-access-9vl29" pod "2fc510cb-aae5-4d8c-a612-2270ccbbf432" (UID: "2fc510cb-aae5-4d8c-a612-2270ccbbf432"). InnerVolumeSpecName "kube-api-access-9vl29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:45:24.357477 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.357013 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01758cb6-b948-46f8-a607-210d75bb7716-kube-api-access-ljm75" pod "01758cb6-b948-46f8-a607-210d75bb7716" (UID: "01758cb6-b948-46f8-a607-210d75bb7716"). InnerVolumeSpecName "kube-api-access-ljm75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:45:24.360180 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.358413 5019 generic.go:356] "Generic (PLEG): container finished" podID="b7abcb04-58ef-4efd-a350-c8f5286ebc14" containerID="c83e84291d71b0272dfe2519562d4d495078d77f90565f38c43e0d1321445487" exitCode=60 Oct 31 04:45:24.360180 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.358516 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890365-f74kl" event={"ID":"b7abcb04-58ef-4efd-a350-c8f5286ebc14","Type":"ContainerDied","Data":"c83e84291d71b0272dfe2519562d4d495078d77f90565f38c43e0d1321445487"} Oct 31 04:45:24.360180 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.358923 5019 scope.go:122] "RemoveContainer" containerID="c83e84291d71b0272dfe2519562d4d495078d77f90565f38c43e0d1321445487" Oct 31 04:45:24.360180 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:24.359230 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-09a20a-29890365-f74kl_odh-ai-gateway-infra(b7abcb04-58ef-4efd-a350-c8f5286ebc14)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890365-f74kl" podUID="b7abcb04-58ef-4efd-a350-c8f5286ebc14" Oct 31 04:45:24.368888 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.368849 5019 generic.go:356] "Generic (PLEG): container finished" podID="bc26f88e-b019-4540-a826-63fb1d624a8d" containerID="a7bf9ddf2aad62cec0707561b935f8e1f3c6aca5478a810606cf252508f66ed1" exitCode=60 Oct 31 04:45:24.369054 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.368955 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890365-5h7hv" event={"ID":"bc26f88e-b019-4540-a826-63fb1d624a8d","Type":"ContainerDied","Data":"a7bf9ddf2aad62cec0707561b935f8e1f3c6aca5478a810606cf252508f66ed1"} Oct 31 04:45:24.380004 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.379432 5019 scope.go:122] "RemoveContainer" containerID="fecbb7f9f57537ec516e66644012163fe52fc628357ee666513264c015ed8a71" Oct 31 04:45:24.381731 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.381675 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890365-rhn85" event={"ID":"01758cb6-b948-46f8-a607-210d75bb7716","Type":"ContainerDied","Data":"eddea1500a30031d941514f79da162d7c07432232915a722b3cd4240a8b27d23"} Oct 31 04:45:24.381948 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.381845 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890365-rhn85" Oct 31 04:45:24.392297 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.391698 5019 generic.go:356] "Generic (PLEG): container finished" podID="a9192eff-e3eb-4bef-a0e7-447afe65cc78" containerID="829a502f82b4af793a6be4060dbebd11449d3d775c10a53f82926e9ce7f98035" exitCode=0 Oct 31 04:45:24.392297 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.391809 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e7fa7b6-nlmpq" event={"ID":"a9192eff-e3eb-4bef-a0e7-447afe65cc78","Type":"ContainerDied","Data":"829a502f82b4af793a6be4060dbebd11449d3d775c10a53f82926e9ce7f98035"} Oct 31 04:45:24.397412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.394726 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890365-wwx6m" event={"ID":"d992b6ec-1a57-4ca9-af09-370245d1ccfe","Type":"ContainerDied","Data":"1b8b3d4a30752a26c59ec536352859844016102395c49e4620992166908d732d"} Oct 31 04:45:24.397412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.394849 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890365-wwx6m" Oct 31 04:45:24.402996 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.401360 5019 generic.go:356] "Generic (PLEG): container finished" podID="99be4628-816c-46f6-abcf-9de2a8452895" containerID="c4b4f322482b92d21e2760d07cc028a74e07b5751d1bb02627ccb814978c536a" exitCode=60 Oct 31 04:45:24.402996 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.401517 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890365-xj5rc" event={"ID":"99be4628-816c-46f6-abcf-9de2a8452895","Type":"ContainerDied","Data":"c4b4f322482b92d21e2760d07cc028a74e07b5751d1bb02627ccb814978c536a"} Oct 31 04:45:24.402996 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.402045 5019 scope.go:122] "RemoveContainer" containerID="c4b4f322482b92d21e2760d07cc028a74e07b5751d1bb02627ccb814978c536a" Oct 31 04:45:24.402996 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:24.402536 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-4c2387-29890365-xj5rc_odh-ai-gateway-infra(99be4628-816c-46f6-abcf-9de2a8452895)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890365-xj5rc" podUID="99be4628-816c-46f6-abcf-9de2a8452895" Oct 31 04:45:24.407056 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.405921 5019 generic.go:356] "Generic (PLEG): container finished" podID="a53b729a-b280-41d4-8661-44826b486246" containerID="e90d16e0b494de0a9d2f11baa512d3e048afa7de178595ecb6e44d549154ed8e" exitCode=60 Oct 31 04:45:24.407056 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.406157 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890365-xf5cz" event={"ID":"a53b729a-b280-41d4-8661-44826b486246","Type":"ContainerDied","Data":"e90d16e0b494de0a9d2f11baa512d3e048afa7de178595ecb6e44d549154ed8e"} Oct 31 04:45:24.407056 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.406629 5019 scope.go:122] "RemoveContainer" containerID="e90d16e0b494de0a9d2f11baa512d3e048afa7de178595ecb6e44d549154ed8e" Oct 31 04:45:24.407056 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:24.407014 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890365-xf5cz_odh-ai-gateway-infra(a53b729a-b280-41d4-8661-44826b486246)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890365-xf5cz" podUID="a53b729a-b280-41d4-8661-44826b486246" Oct 31 04:45:24.409156 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.409125 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-654741-29890365-fd9cp" event={"ID":"2fc510cb-aae5-4d8c-a612-2270ccbbf432","Type":"ContainerDied","Data":"429b6c93d1beb9894d66acc8a1e46ac6e325172f095a9e377e8c14f7edeec0ea"} Oct 31 04:45:24.409374 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.409361 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-654741-29890365-fd9cp" Oct 31 04:45:24.440308 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.440196 5019 scope.go:122] "RemoveContainer" containerID="e223e7c544ed8d169986c16d73117d9d2a99c0ddbfaabd6d237d93efa4679583" Oct 31 04:45:24.455094 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.452570 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljm75\" (UniqueName: \"kubernetes.io/projected/01758cb6-b948-46f8-a607-210d75bb7716-kube-api-access-ljm75\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:24.455094 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.452617 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01758cb6-b948-46f8-a607-210d75bb7716-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:24.455094 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.452629 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vl29\" (UniqueName: \"kubernetes.io/projected/2fc510cb-aae5-4d8c-a612-2270ccbbf432-kube-api-access-9vl29\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:24.455094 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.452643 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d992b6ec-1a57-4ca9-af09-370245d1ccfe-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:24.455094 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.452657 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fc510cb-aae5-4d8c-a612-2270ccbbf432-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:24.455094 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.452670 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jrzs\" (UniqueName: \"kubernetes.io/projected/d992b6ec-1a57-4ca9-af09-370245d1ccfe-kube-api-access-4jrzs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:24.457838 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.457758 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890365-t5r5d"] Oct 31 04:45:24.485402 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.474794 5019 scope.go:122] "RemoveContainer" containerID="81717c974ab191a8d0af168d95187831a0fd4204a3ec2e2abe275da732d1b57e" Oct 31 04:45:24.486517 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.485960 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-77a153-29890365-t5r5d"] Oct 31 04:45:24.523800 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.522321 5019 scope.go:122] "RemoveContainer" containerID="80dbdf76c886b3ab707b1f19c50dc6624eb670ee672b008e5018c9fb28f8b4f2" Oct 31 04:45:24.545127 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.545093 5019 scope.go:122] "RemoveContainer" containerID="1e8a810e742b555a2f60ad8f1a8d9180f335898db3e3cc93e343b945dfaad3ff" Oct 31 04:45:24.569999 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.569970 5019 scope.go:122] "RemoveContainer" containerID="88fa47d69e8da32266f83de06f0095bb431dce730b18aef1ff3794c63d0d21af" Oct 31 04:45:24.594319 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.594282 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890365-5h7hv" Oct 31 04:45:24.608073 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.608030 5019 scope.go:122] "RemoveContainer" containerID="afe12c7c58ffc9ed537bad7556c3b2eba6baaac037d74e5dd51f67ad96ccdf37" Oct 31 04:45:24.634921 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.634874 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890365-rhn85"] Oct 31 04:45:24.639706 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.639671 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890365-rhn85"] Oct 31 04:45:24.658952 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.658915 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc26f88e-b019-4540-a826-63fb1d624a8d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc26f88e-b019-4540-a826-63fb1d624a8d-maas-api-service-ca\") pod \"bc26f88e-b019-4540-a826-63fb1d624a8d\" (UID: \"bc26f88e-b019-4540-a826-63fb1d624a8d\") " Oct 31 04:45:24.659149 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.659092 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc26f88e-b019-4540-a826-63fb1d624a8d-kube-api-access-jq5cb\" (UniqueName: \"kubernetes.io/projected/bc26f88e-b019-4540-a826-63fb1d624a8d-kube-api-access-jq5cb\") pod \"bc26f88e-b019-4540-a826-63fb1d624a8d\" (UID: \"bc26f88e-b019-4540-a826-63fb1d624a8d\") " Oct 31 04:45:24.660111 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.660081 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc26f88e-b019-4540-a826-63fb1d624a8d-maas-api-service-ca" pod "bc26f88e-b019-4540-a826-63fb1d624a8d" (UID: "bc26f88e-b019-4540-a826-63fb1d624a8d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:45:24.662399 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.662316 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc26f88e-b019-4540-a826-63fb1d624a8d-kube-api-access-jq5cb" pod "bc26f88e-b019-4540-a826-63fb1d624a8d" (UID: "bc26f88e-b019-4540-a826-63fb1d624a8d"). InnerVolumeSpecName "kube-api-access-jq5cb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:45:24.665206 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.665186 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e7fa7b6-nlmpq" Oct 31 04:45:24.671885 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.671855 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-654741-29890365-fd9cp"] Oct 31 04:45:24.676959 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.676929 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-654741-29890365-fd9cp"] Oct 31 04:45:24.707799 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.707757 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890365-wwx6m"] Oct 31 04:45:24.712719 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.712686 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890365-wwx6m"] Oct 31 04:45:24.760680 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.760636 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9192eff-e3eb-4bef-a0e7-447afe65cc78-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9192eff-e3eb-4bef-a0e7-447afe65cc78-maas-api-xks-ca\") pod \"a9192eff-e3eb-4bef-a0e7-447afe65cc78\" (UID: \"a9192eff-e3eb-4bef-a0e7-447afe65cc78\") " Oct 31 04:45:24.760984 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.760697 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9192eff-e3eb-4bef-a0e7-447afe65cc78-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9192eff-e3eb-4bef-a0e7-447afe65cc78-maas-api-openshift-ca\") pod \"a9192eff-e3eb-4bef-a0e7-447afe65cc78\" (UID: \"a9192eff-e3eb-4bef-a0e7-447afe65cc78\") " Oct 31 04:45:24.760984 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.760887 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9192eff-e3eb-4bef-a0e7-447afe65cc78-kube-api-access-5rtgq\" (UniqueName: \"kubernetes.io/projected/a9192eff-e3eb-4bef-a0e7-447afe65cc78-kube-api-access-5rtgq\") pod \"a9192eff-e3eb-4bef-a0e7-447afe65cc78\" (UID: \"a9192eff-e3eb-4bef-a0e7-447afe65cc78\") " Oct 31 04:45:24.761490 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.761463 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc26f88e-b019-4540-a826-63fb1d624a8d-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:24.761602 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.761495 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq5cb\" (UniqueName: \"kubernetes.io/projected/bc26f88e-b019-4540-a826-63fb1d624a8d-kube-api-access-jq5cb\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:24.763674 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.763639 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9192eff-e3eb-4bef-a0e7-447afe65cc78-maas-api-openshift-ca" pod "a9192eff-e3eb-4bef-a0e7-447afe65cc78" (UID: "a9192eff-e3eb-4bef-a0e7-447afe65cc78"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:45:24.764915 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.764878 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9192eff-e3eb-4bef-a0e7-447afe65cc78-kube-api-access-5rtgq" pod "a9192eff-e3eb-4bef-a0e7-447afe65cc78" (UID: "a9192eff-e3eb-4bef-a0e7-447afe65cc78"). InnerVolumeSpecName "kube-api-access-5rtgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:45:24.765440 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.765413 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9192eff-e3eb-4bef-a0e7-447afe65cc78-maas-api-xks-ca" pod "a9192eff-e3eb-4bef-a0e7-447afe65cc78" (UID: "a9192eff-e3eb-4bef-a0e7-447afe65cc78"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:45:24.799623 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.799586 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890365-xj5rc"] Oct 31 04:45:24.863005 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.862912 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rtgq\" (UniqueName: \"kubernetes.io/projected/a9192eff-e3eb-4bef-a0e7-447afe65cc78-kube-api-access-5rtgq\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:24.863005 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.862946 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9192eff-e3eb-4bef-a0e7-447afe65cc78-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:24.863005 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:24.862962 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9192eff-e3eb-4bef-a0e7-447afe65cc78-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:25.412690 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:25.412656 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01758cb6-b948-46f8-a607-210d75bb7716" path="/var/lib/kubelet/pods/01758cb6-b948-46f8-a607-210d75bb7716/volumes" Oct 31 04:45:25.413205 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:25.413190 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="089f7eeb-81d7-4562-b906-da33b0fb5fa1" path="/var/lib/kubelet/pods/089f7eeb-81d7-4562-b906-da33b0fb5fa1/volumes" Oct 31 04:45:25.413663 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:25.413648 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fc510cb-aae5-4d8c-a612-2270ccbbf432" path="/var/lib/kubelet/pods/2fc510cb-aae5-4d8c-a612-2270ccbbf432/volumes" Oct 31 04:45:25.414345 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:25.414316 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="668f2b2b-4d01-4054-8ca0-16a4b0b4dcdd" path="/var/lib/kubelet/pods/668f2b2b-4d01-4054-8ca0-16a4b0b4dcdd/volumes" Oct 31 04:45:25.414839 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:25.414824 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d992b6ec-1a57-4ca9-af09-370245d1ccfe" path="/var/lib/kubelet/pods/d992b6ec-1a57-4ca9-af09-370245d1ccfe/volumes" Oct 31 04:45:25.415258 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:25.415245 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e12c1431-1aa4-4c72-94a9-d3b6646ee4b2" path="/var/lib/kubelet/pods/e12c1431-1aa4-4c72-94a9-d3b6646ee4b2/volumes" Oct 31 04:45:25.415988 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:25.415972 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e282b47a-b89e-48ae-88bf-347945705a1d" path="/var/lib/kubelet/pods/e282b47a-b89e-48ae-88bf-347945705a1d/volumes" Oct 31 04:45:25.429180 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:25.428803 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890365-f74kl"] Oct 31 04:45:25.446556 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:25.446522 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e7fa7b6-nlmpq" Oct 31 04:45:25.446758 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:25.446524 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e7fa7b6-nlmpq" event={"ID":"a9192eff-e3eb-4bef-a0e7-447afe65cc78","Type":"ContainerDied","Data":"49d7fdf6165996cf3c614db3d8943caeb50e14ea5dbfef76fefd835e41b88d2c"} Oct 31 04:45:25.446758 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:25.446669 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49d7fdf6165996cf3c614db3d8943caeb50e14ea5dbfef76fefd835e41b88d2c" Oct 31 04:45:25.458444 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:25.458401 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890365-5h7hv" event={"ID":"bc26f88e-b019-4540-a826-63fb1d624a8d","Type":"ContainerDied","Data":"44b4aeac6e5030e002b2019bda1740d9448576bc7ea119af8e8166062b7a2731"} Oct 31 04:45:25.458916 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:25.458481 5019 scope.go:122] "RemoveContainer" containerID="a7bf9ddf2aad62cec0707561b935f8e1f3c6aca5478a810606cf252508f66ed1" Oct 31 04:45:25.458916 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:25.458431 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890365-5h7hv" Oct 31 04:45:25.573294 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:25.571253 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890365-xf5cz"] Oct 31 04:45:25.605805 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:25.605705 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890365-5h7hv"] Oct 31 04:45:25.615910 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:25.615691 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890365-5h7hv"] Oct 31 04:45:25.745692 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:25.745665 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890365-xj5rc" Oct 31 04:45:25.748226 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:25.748201 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890365-f74kl" Oct 31 04:45:25.779649 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:25.779612 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7abcb04-58ef-4efd-a350-c8f5286ebc14-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7abcb04-58ef-4efd-a350-c8f5286ebc14-maas-api-service-ca\") pod \"b7abcb04-58ef-4efd-a350-c8f5286ebc14\" (UID: \"b7abcb04-58ef-4efd-a350-c8f5286ebc14\") " Oct 31 04:45:25.779649 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:25.779653 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99be4628-816c-46f6-abcf-9de2a8452895-kube-api-access-s8hc9\" (UniqueName: \"kubernetes.io/projected/99be4628-816c-46f6-abcf-9de2a8452895-kube-api-access-s8hc9\") pod \"99be4628-816c-46f6-abcf-9de2a8452895\" (UID: \"99be4628-816c-46f6-abcf-9de2a8452895\") " Oct 31 04:45:25.779972 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:25.779950 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7abcb04-58ef-4efd-a350-c8f5286ebc14-kube-api-access-q5j7q\" (UniqueName: \"kubernetes.io/projected/b7abcb04-58ef-4efd-a350-c8f5286ebc14-kube-api-access-q5j7q\") pod \"b7abcb04-58ef-4efd-a350-c8f5286ebc14\" (UID: \"b7abcb04-58ef-4efd-a350-c8f5286ebc14\") " Oct 31 04:45:25.780068 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:25.780048 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7abcb04-58ef-4efd-a350-c8f5286ebc14-maas-api-service-ca" pod "b7abcb04-58ef-4efd-a350-c8f5286ebc14" (UID: "b7abcb04-58ef-4efd-a350-c8f5286ebc14"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:45:25.780132 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:25.780062 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99be4628-816c-46f6-abcf-9de2a8452895-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99be4628-816c-46f6-abcf-9de2a8452895-maas-api-service-ca\") pod \"99be4628-816c-46f6-abcf-9de2a8452895\" (UID: \"99be4628-816c-46f6-abcf-9de2a8452895\") " Oct 31 04:45:25.780539 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:25.780514 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99be4628-816c-46f6-abcf-9de2a8452895-maas-api-service-ca" pod "99be4628-816c-46f6-abcf-9de2a8452895" (UID: "99be4628-816c-46f6-abcf-9de2a8452895"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:45:25.780861 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:25.780834 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99be4628-816c-46f6-abcf-9de2a8452895-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:25.780861 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:25.780863 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7abcb04-58ef-4efd-a350-c8f5286ebc14-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:25.782624 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:25.782600 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7abcb04-58ef-4efd-a350-c8f5286ebc14-kube-api-access-q5j7q" pod "b7abcb04-58ef-4efd-a350-c8f5286ebc14" (UID: "b7abcb04-58ef-4efd-a350-c8f5286ebc14"). InnerVolumeSpecName "kube-api-access-q5j7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:45:25.783501 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:25.783474 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99be4628-816c-46f6-abcf-9de2a8452895-kube-api-access-s8hc9" pod "99be4628-816c-46f6-abcf-9de2a8452895" (UID: "99be4628-816c-46f6-abcf-9de2a8452895"). InnerVolumeSpecName "kube-api-access-s8hc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:45:25.881261 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:25.881228 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890365-xf5cz" Oct 31 04:45:25.882795 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:25.882760 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5j7q\" (UniqueName: \"kubernetes.io/projected/b7abcb04-58ef-4efd-a350-c8f5286ebc14-kube-api-access-q5j7q\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:25.882941 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:25.882798 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8hc9\" (UniqueName: \"kubernetes.io/projected/99be4628-816c-46f6-abcf-9de2a8452895-kube-api-access-s8hc9\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:25.984076 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:25.984034 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a53b729a-b280-41d4-8661-44826b486246-kube-api-access-q9tv5\" (UniqueName: \"kubernetes.io/projected/a53b729a-b280-41d4-8661-44826b486246-kube-api-access-q9tv5\") pod \"a53b729a-b280-41d4-8661-44826b486246\" (UID: \"a53b729a-b280-41d4-8661-44826b486246\") " Oct 31 04:45:25.984400 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:25.984386 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a53b729a-b280-41d4-8661-44826b486246-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a53b729a-b280-41d4-8661-44826b486246-maas-api-service-ca\") pod \"a53b729a-b280-41d4-8661-44826b486246\" (UID: \"a53b729a-b280-41d4-8661-44826b486246\") " Oct 31 04:45:25.984794 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:25.984764 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a53b729a-b280-41d4-8661-44826b486246-maas-api-service-ca" pod "a53b729a-b280-41d4-8661-44826b486246" (UID: "a53b729a-b280-41d4-8661-44826b486246"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:45:25.985083 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:25.985064 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a53b729a-b280-41d4-8661-44826b486246-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:25.986559 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:25.986529 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a53b729a-b280-41d4-8661-44826b486246-kube-api-access-q9tv5" pod "a53b729a-b280-41d4-8661-44826b486246" (UID: "a53b729a-b280-41d4-8661-44826b486246"). InnerVolumeSpecName "kube-api-access-q9tv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:45:26.086836 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.086787 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9tv5\" (UniqueName: \"kubernetes.io/projected/a53b729a-b280-41d4-8661-44826b486246-kube-api-access-q9tv5\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:26.589358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.576166 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890365-xj5rc" event={"ID":"99be4628-816c-46f6-abcf-9de2a8452895","Type":"ContainerDied","Data":"c01851506533c6030e601a61c08cc6a19fd0bb046042ca7f4e5d2449671347f7"} Oct 31 04:45:26.589358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.576244 5019 scope.go:122] "RemoveContainer" containerID="c4b4f322482b92d21e2760d07cc028a74e07b5751d1bb02627ccb814978c536a" Oct 31 04:45:26.589358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.576368 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890365-xj5rc" Oct 31 04:45:26.650076 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.649229 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d812a3a9-zhkql"] Oct 31 04:45:26.655292 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.654201 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890365-xf5cz" Oct 31 04:45:26.660452 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.660404 5019 cpu_manager.go:414] "containerMap: removing container" podUID="01758cb6-b948-46f8-a607-210d75bb7716" containerName="cleanup" Oct 31 04:45:26.674165 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.674123 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01758cb6-b948-46f8-a607-210d75bb7716" containerName="cleanup" Oct 31 04:45:26.674416 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.674402 5019 cpu_manager.go:414] "containerMap: removing container" podUID="0a9a0d3a-eced-4fd8-85fa-7c08b317c39f" containerName="cleanup" Oct 31 04:45:26.674574 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.674561 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a9a0d3a-eced-4fd8-85fa-7c08b317c39f" containerName="cleanup" Oct 31 04:45:26.674698 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.674688 5019 cpu_manager.go:414] "containerMap: removing container" podUID="d992b6ec-1a57-4ca9-af09-370245d1ccfe" containerName="cleanup" Oct 31 04:45:26.674805 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.674795 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d992b6ec-1a57-4ca9-af09-370245d1ccfe" containerName="cleanup" Oct 31 04:45:26.674918 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.674902 5019 cpu_manager.go:414] "containerMap: removing container" podUID="01758cb6-b948-46f8-a607-210d75bb7716" containerName="cleanup" Oct 31 04:45:26.675017 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.675006 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01758cb6-b948-46f8-a607-210d75bb7716" containerName="cleanup" Oct 31 04:45:26.675130 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.675121 5019 cpu_manager.go:414] "containerMap: removing container" podUID="b7abcb04-58ef-4efd-a350-c8f5286ebc14" containerName="cleanup" Oct 31 04:45:26.675232 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.675222 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7abcb04-58ef-4efd-a350-c8f5286ebc14" containerName="cleanup" Oct 31 04:45:26.675397 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.675387 5019 cpu_manager.go:414] "containerMap: removing container" podUID="a53b729a-b280-41d4-8661-44826b486246" containerName="cleanup" Oct 31 04:45:26.675520 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.675508 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a53b729a-b280-41d4-8661-44826b486246" containerName="cleanup" Oct 31 04:45:26.675644 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.675634 5019 cpu_manager.go:414] "containerMap: removing container" podUID="e12c1431-1aa4-4c72-94a9-d3b6646ee4b2" containerName="cleanup" Oct 31 04:45:26.675745 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.675734 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e12c1431-1aa4-4c72-94a9-d3b6646ee4b2" containerName="cleanup" Oct 31 04:45:26.675863 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.675854 5019 cpu_manager.go:414] "containerMap: removing container" podUID="668f2b2b-4d01-4054-8ca0-16a4b0b4dcdd" containerName="cleanup" Oct 31 04:45:26.675963 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.675951 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="668f2b2b-4d01-4054-8ca0-16a4b0b4dcdd" containerName="cleanup" Oct 31 04:45:26.676095 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.676086 5019 cpu_manager.go:414] "containerMap: removing container" podUID="d992b6ec-1a57-4ca9-af09-370245d1ccfe" containerName="cleanup" Oct 31 04:45:26.676225 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.676212 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d992b6ec-1a57-4ca9-af09-370245d1ccfe" containerName="cleanup" Oct 31 04:45:26.676352 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.676324 5019 cpu_manager.go:414] "containerMap: removing container" podUID="b5c95c35-2055-4d02-af07-fc3574c29ecf" containerName="cleanup" Oct 31 04:45:26.676458 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.676447 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5c95c35-2055-4d02-af07-fc3574c29ecf" containerName="cleanup" Oct 31 04:45:26.676567 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.676557 5019 cpu_manager.go:414] "containerMap: removing container" podUID="e12c1431-1aa4-4c72-94a9-d3b6646ee4b2" containerName="cleanup" Oct 31 04:45:26.676662 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.676652 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e12c1431-1aa4-4c72-94a9-d3b6646ee4b2" containerName="cleanup" Oct 31 04:45:26.676773 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.676764 5019 cpu_manager.go:414] "containerMap: removing container" podUID="e12c1431-1aa4-4c72-94a9-d3b6646ee4b2" containerName="cleanup" Oct 31 04:45:26.676868 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.676858 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e12c1431-1aa4-4c72-94a9-d3b6646ee4b2" containerName="cleanup" Oct 31 04:45:26.676989 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.676979 5019 cpu_manager.go:414] "containerMap: removing container" podUID="681c73ad-b2a1-458f-b37a-cc4e9e198926" containerName="cleanup" Oct 31 04:45:26.677086 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.677075 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="681c73ad-b2a1-458f-b37a-cc4e9e198926" containerName="cleanup" Oct 31 04:45:26.677190 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.677180 5019 cpu_manager.go:414] "containerMap: removing container" podUID="2fc510cb-aae5-4d8c-a612-2270ccbbf432" containerName="cleanup" Oct 31 04:45:26.677279 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.677269 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fc510cb-aae5-4d8c-a612-2270ccbbf432" containerName="cleanup" Oct 31 04:45:26.677379 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.677369 5019 cpu_manager.go:414] "containerMap: removing container" podUID="b5c95c35-2055-4d02-af07-fc3574c29ecf" containerName="cleanup" Oct 31 04:45:26.677486 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.677474 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5c95c35-2055-4d02-af07-fc3574c29ecf" containerName="cleanup" Oct 31 04:45:26.677600 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.677589 5019 cpu_manager.go:414] "containerMap: removing container" podUID="bc26f88e-b019-4540-a826-63fb1d624a8d" containerName="cleanup" Oct 31 04:45:26.677688 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.677677 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc26f88e-b019-4540-a826-63fb1d624a8d" containerName="cleanup" Oct 31 04:45:26.677800 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.677790 5019 cpu_manager.go:414] "containerMap: removing container" podUID="b5c95c35-2055-4d02-af07-fc3574c29ecf" containerName="cleanup" Oct 31 04:45:26.677892 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.677882 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5c95c35-2055-4d02-af07-fc3574c29ecf" containerName="cleanup" Oct 31 04:45:26.677999 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.677989 5019 cpu_manager.go:414] "containerMap: removing container" podUID="b7abcb04-58ef-4efd-a350-c8f5286ebc14" containerName="cleanup" Oct 31 04:45:26.678220 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.678209 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7abcb04-58ef-4efd-a350-c8f5286ebc14" containerName="cleanup" Oct 31 04:45:26.678320 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.678311 5019 cpu_manager.go:414] "containerMap: removing container" podUID="bc26f88e-b019-4540-a826-63fb1d624a8d" containerName="cleanup" Oct 31 04:45:26.678432 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.678420 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc26f88e-b019-4540-a826-63fb1d624a8d" containerName="cleanup" Oct 31 04:45:26.678545 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.678535 5019 cpu_manager.go:414] "containerMap: removing container" podUID="a9192eff-e3eb-4bef-a0e7-447afe65cc78" containerName="revoke-keys" Oct 31 04:45:26.678678 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.678635 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9192eff-e3eb-4bef-a0e7-447afe65cc78" containerName="revoke-keys" Oct 31 04:45:26.678925 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.678893 5019 cpu_manager.go:414] "containerMap: removing container" podUID="e282b47a-b89e-48ae-88bf-347945705a1d" containerName="cleanup" Oct 31 04:45:26.679035 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.679023 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e282b47a-b89e-48ae-88bf-347945705a1d" containerName="cleanup" Oct 31 04:45:26.679154 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.679144 5019 cpu_manager.go:414] "containerMap: removing container" podUID="089f7eeb-81d7-4562-b906-da33b0fb5fa1" containerName="cleanup" Oct 31 04:45:26.679249 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.679238 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="089f7eeb-81d7-4562-b906-da33b0fb5fa1" containerName="cleanup" Oct 31 04:45:26.679364 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.679354 5019 cpu_manager.go:414] "containerMap: removing container" podUID="681c73ad-b2a1-458f-b37a-cc4e9e198926" containerName="cleanup" Oct 31 04:45:26.679464 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.679454 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="681c73ad-b2a1-458f-b37a-cc4e9e198926" containerName="cleanup" Oct 31 04:45:26.679592 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.679582 5019 cpu_manager.go:414] "containerMap: removing container" podUID="089f7eeb-81d7-4562-b906-da33b0fb5fa1" containerName="cleanup" Oct 31 04:45:26.679681 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.679671 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="089f7eeb-81d7-4562-b906-da33b0fb5fa1" containerName="cleanup" Oct 31 04:45:26.679787 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.679778 5019 cpu_manager.go:414] "containerMap: removing container" podUID="e282b47a-b89e-48ae-88bf-347945705a1d" containerName="cleanup" Oct 31 04:45:26.679889 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.679879 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e282b47a-b89e-48ae-88bf-347945705a1d" containerName="cleanup" Oct 31 04:45:26.680017 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.680007 5019 cpu_manager.go:414] "containerMap: removing container" podUID="99be4628-816c-46f6-abcf-9de2a8452895" containerName="cleanup" Oct 31 04:45:26.680122 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.680112 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99be4628-816c-46f6-abcf-9de2a8452895" containerName="cleanup" Oct 31 04:45:26.680223 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.680214 5019 cpu_manager.go:414] "containerMap: removing container" podUID="e282b47a-b89e-48ae-88bf-347945705a1d" containerName="cleanup" Oct 31 04:45:26.680323 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.680313 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e282b47a-b89e-48ae-88bf-347945705a1d" containerName="cleanup" Oct 31 04:45:26.680559 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.680544 5019 cpu_manager.go:414] "containerMap: removing container" podUID="a53b729a-b280-41d4-8661-44826b486246" containerName="cleanup" Oct 31 04:45:26.680681 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.680668 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a53b729a-b280-41d4-8661-44826b486246" containerName="cleanup" Oct 31 04:45:26.680799 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.680788 5019 cpu_manager.go:414] "containerMap: removing container" podUID="2fc510cb-aae5-4d8c-a612-2270ccbbf432" containerName="cleanup" Oct 31 04:45:26.680895 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.680885 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fc510cb-aae5-4d8c-a612-2270ccbbf432" containerName="cleanup" Oct 31 04:45:26.681027 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.681018 5019 cpu_manager.go:414] "containerMap: removing container" podUID="99be4628-816c-46f6-abcf-9de2a8452895" containerName="cleanup" Oct 31 04:45:26.681125 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.681115 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99be4628-816c-46f6-abcf-9de2a8452895" containerName="cleanup" Oct 31 04:45:26.681244 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.681234 5019 cpu_manager.go:414] "containerMap: removing container" podUID="99be4628-816c-46f6-abcf-9de2a8452895" containerName="cleanup" Oct 31 04:45:26.681355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.681344 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99be4628-816c-46f6-abcf-9de2a8452895" containerName="cleanup" Oct 31 04:45:26.681476 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.681465 5019 cpu_manager.go:414] "containerMap: removing container" podUID="668f2b2b-4d01-4054-8ca0-16a4b0b4dcdd" containerName="cleanup" Oct 31 04:45:26.681571 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.681560 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="668f2b2b-4d01-4054-8ca0-16a4b0b4dcdd" containerName="cleanup" Oct 31 04:45:26.682309 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.682290 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="99be4628-816c-46f6-abcf-9de2a8452895" containerName="cleanup" Oct 31 04:45:26.682500 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.682488 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e282b47a-b89e-48ae-88bf-347945705a1d" containerName="cleanup" Oct 31 04:45:26.682627 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.682616 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="a53b729a-b280-41d4-8661-44826b486246" containerName="cleanup" Oct 31 04:45:26.682733 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.682725 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fc510cb-aae5-4d8c-a612-2270ccbbf432" containerName="cleanup" Oct 31 04:45:26.682841 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.682832 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="d992b6ec-1a57-4ca9-af09-370245d1ccfe" containerName="cleanup" Oct 31 04:45:26.682937 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.682929 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc26f88e-b019-4540-a826-63fb1d624a8d" containerName="cleanup" Oct 31 04:45:26.683050 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.666361 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890365-f74kl" Oct 31 04:45:26.684527 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.683081 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="99be4628-816c-46f6-abcf-9de2a8452895" containerName="cleanup" Oct 31 04:45:26.684681 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.684668 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7abcb04-58ef-4efd-a350-c8f5286ebc14" containerName="cleanup" Oct 31 04:45:26.684778 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.684768 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="668f2b2b-4d01-4054-8ca0-16a4b0b4dcdd" containerName="cleanup" Oct 31 04:45:26.684883 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.684873 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="99be4628-816c-46f6-abcf-9de2a8452895" containerName="cleanup" Oct 31 04:45:26.684970 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.684961 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="089f7eeb-81d7-4562-b906-da33b0fb5fa1" containerName="cleanup" Oct 31 04:45:26.685067 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.685058 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="668f2b2b-4d01-4054-8ca0-16a4b0b4dcdd" containerName="cleanup" Oct 31 04:45:26.685161 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.685151 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a9a0d3a-eced-4fd8-85fa-7c08b317c39f" containerName="cleanup" Oct 31 04:45:26.685265 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.685256 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="681c73ad-b2a1-458f-b37a-cc4e9e198926" containerName="cleanup" Oct 31 04:45:26.685507 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.685357 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="01758cb6-b948-46f8-a607-210d75bb7716" containerName="cleanup" Oct 31 04:45:26.685625 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.685614 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="d992b6ec-1a57-4ca9-af09-370245d1ccfe" containerName="cleanup" Oct 31 04:45:26.685715 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.685706 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="01758cb6-b948-46f8-a607-210d75bb7716" containerName="cleanup" Oct 31 04:45:26.685817 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.685808 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="a53b729a-b280-41d4-8661-44826b486246" containerName="cleanup" Oct 31 04:45:26.685908 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.685899 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e12c1431-1aa4-4c72-94a9-d3b6646ee4b2" containerName="cleanup" Oct 31 04:45:26.686004 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.685995 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5c95c35-2055-4d02-af07-fc3574c29ecf" containerName="cleanup" Oct 31 04:45:26.686091 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.686083 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e12c1431-1aa4-4c72-94a9-d3b6646ee4b2" containerName="cleanup" Oct 31 04:45:26.686180 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.686169 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="a53b729a-b280-41d4-8661-44826b486246" containerName="cleanup" Oct 31 04:45:26.686273 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.686263 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e12c1431-1aa4-4c72-94a9-d3b6646ee4b2" containerName="cleanup" Oct 31 04:45:26.686406 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.686396 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="6171ff86-f98a-4575-834d-2f6bd135424d" containerName="cleanup" Oct 31 04:45:26.686511 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.686501 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="681c73ad-b2a1-458f-b37a-cc4e9e198926" containerName="cleanup" Oct 31 04:45:26.686602 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.686593 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fc510cb-aae5-4d8c-a612-2270ccbbf432" containerName="cleanup" Oct 31 04:45:26.686703 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.686694 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5c95c35-2055-4d02-af07-fc3574c29ecf" containerName="cleanup" Oct 31 04:45:26.686796 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.686786 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5c95c35-2055-4d02-af07-fc3574c29ecf" containerName="cleanup" Oct 31 04:45:26.686893 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.686884 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7abcb04-58ef-4efd-a350-c8f5286ebc14" containerName="cleanup" Oct 31 04:45:26.686987 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.686978 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc26f88e-b019-4540-a826-63fb1d624a8d" containerName="cleanup" Oct 31 04:45:26.687096 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.687074 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9192eff-e3eb-4bef-a0e7-447afe65cc78" containerName="revoke-keys" Oct 31 04:45:26.687196 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.687186 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="19237217-929c-4b26-ac1f-fb27745d6dab" containerName="cleanup" Oct 31 04:45:26.687296 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.687286 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="681c73ad-b2a1-458f-b37a-cc4e9e198926" containerName="cleanup" Oct 31 04:45:26.687421 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.687410 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="089f7eeb-81d7-4562-b906-da33b0fb5fa1" containerName="cleanup" Oct 31 04:45:26.687518 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.687509 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e282b47a-b89e-48ae-88bf-347945705a1d" containerName="cleanup" Oct 31 04:45:26.693825 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.693699 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890365-xf5cz" event={"ID":"a53b729a-b280-41d4-8661-44826b486246","Type":"ContainerDied","Data":"f232f751ee6a94eb689bf5071f8f6c72b9c2e248537674b56baa82ebcef3fdb5"} Oct 31 04:45:26.693825 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.693752 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d812a3a9-zhkql"] Oct 31 04:45:26.693825 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.693787 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890365-f74kl" event={"ID":"b7abcb04-58ef-4efd-a350-c8f5286ebc14","Type":"ContainerDied","Data":"d57aa8ad8523ba955b03a2f29a105d27e01a3c75b4ae0c21a29c02afbc2c1cf8"} Oct 31 04:45:26.693825 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.693822 5019 scope.go:122] "RemoveContainer" containerID="e90d16e0b494de0a9d2f11baa512d3e048afa7de178595ecb6e44d549154ed8e" Oct 31 04:45:26.696036 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.694009 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d812a3a9-zhkql" Oct 31 04:45:26.709097 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.708896 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:45:26.709097 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.708986 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:45:26.709097 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.709049 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:45:26.709400 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.709147 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:45:26.715658 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.715593 5019 scope.go:122] "RemoveContainer" containerID="c83e84291d71b0272dfe2519562d4d495078d77f90565f38c43e0d1321445487" Oct 31 04:45:26.723961 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.723921 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890365-xj5rc"] Oct 31 04:45:26.728652 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.728617 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890365-xj5rc"] Oct 31 04:45:26.755462 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.755415 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c68e7b82-71db-48e2-8d07-36ddb22e9ae7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d812a3a9-zhkql\" (UID: \"c68e7b82-71db-48e2-8d07-36ddb22e9ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d812a3a9-zhkql" Oct 31 04:45:26.755695 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.755537 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c68e7b82-71db-48e2-8d07-36ddb22e9ae7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d812a3a9-zhkql\" (UID: \"c68e7b82-71db-48e2-8d07-36ddb22e9ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d812a3a9-zhkql" Oct 31 04:45:26.755862 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.755842 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrc7r\" (UniqueName: \"kubernetes.io/projected/c68e7b82-71db-48e2-8d07-36ddb22e9ae7-kube-api-access-vrc7r\") pod \"maas-api-revoke-sub-d812a3a9-zhkql\" (UID: \"c68e7b82-71db-48e2-8d07-36ddb22e9ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d812a3a9-zhkql" Oct 31 04:45:26.761907 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.761865 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890365-xf5cz"] Oct 31 04:45:26.766452 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.766414 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890365-xf5cz"] Oct 31 04:45:26.797169 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.797129 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890365-f74kl"] Oct 31 04:45:26.801799 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.801757 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890365-f74kl"] Oct 31 04:45:26.857966 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.857859 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrc7r\" (UniqueName: \"kubernetes.io/projected/c68e7b82-71db-48e2-8d07-36ddb22e9ae7-kube-api-access-vrc7r\") pod \"maas-api-revoke-sub-d812a3a9-zhkql\" (UID: \"c68e7b82-71db-48e2-8d07-36ddb22e9ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d812a3a9-zhkql" Oct 31 04:45:26.858168 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.858056 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c68e7b82-71db-48e2-8d07-36ddb22e9ae7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d812a3a9-zhkql\" (UID: \"c68e7b82-71db-48e2-8d07-36ddb22e9ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d812a3a9-zhkql" Oct 31 04:45:26.858168 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.858135 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c68e7b82-71db-48e2-8d07-36ddb22e9ae7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d812a3a9-zhkql\" (UID: \"c68e7b82-71db-48e2-8d07-36ddb22e9ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d812a3a9-zhkql" Oct 31 04:45:26.858862 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.858840 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c68e7b82-71db-48e2-8d07-36ddb22e9ae7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d812a3a9-zhkql\" (UID: \"c68e7b82-71db-48e2-8d07-36ddb22e9ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d812a3a9-zhkql" Oct 31 04:45:26.860585 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.860550 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c68e7b82-71db-48e2-8d07-36ddb22e9ae7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d812a3a9-zhkql\" (UID: \"c68e7b82-71db-48e2-8d07-36ddb22e9ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d812a3a9-zhkql" Oct 31 04:45:26.882999 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:26.882966 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrc7r\" (UniqueName: \"kubernetes.io/projected/c68e7b82-71db-48e2-8d07-36ddb22e9ae7-kube-api-access-vrc7r\") pod \"maas-api-revoke-sub-d812a3a9-zhkql\" (UID: \"c68e7b82-71db-48e2-8d07-36ddb22e9ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d812a3a9-zhkql" Oct 31 04:45:27.025774 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:27.025728 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d812a3a9-zhkql" Oct 31 04:45:27.354009 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:27.353970 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d812a3a9-zhkql"] Oct 31 04:45:27.354236 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:45:27.354212 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc68e7b82_71db_48e2_8d07_36ddb22e9ae7.slice/crio-0ad4f5e8347d61ccd6d9623fb4cfeeb1be0ec4a091efbfedd81fcef6d96571db WatchSource:0}: Error finding container 0ad4f5e8347d61ccd6d9623fb4cfeeb1be0ec4a091efbfedd81fcef6d96571db: Status 404 returned error can't find the container with id 0ad4f5e8347d61ccd6d9623fb4cfeeb1be0ec4a091efbfedd81fcef6d96571db Oct 31 04:45:27.413533 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:27.413502 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99be4628-816c-46f6-abcf-9de2a8452895" path="/var/lib/kubelet/pods/99be4628-816c-46f6-abcf-9de2a8452895/volumes" Oct 31 04:45:27.414095 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:27.414079 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a53b729a-b280-41d4-8661-44826b486246" path="/var/lib/kubelet/pods/a53b729a-b280-41d4-8661-44826b486246/volumes" Oct 31 04:45:27.414574 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:27.414558 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7abcb04-58ef-4efd-a350-c8f5286ebc14" path="/var/lib/kubelet/pods/b7abcb04-58ef-4efd-a350-c8f5286ebc14/volumes" Oct 31 04:45:27.415510 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:27.415489 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc26f88e-b019-4540-a826-63fb1d624a8d" path="/var/lib/kubelet/pods/bc26f88e-b019-4540-a826-63fb1d624a8d/volumes" Oct 31 04:45:27.544421 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:27.544369 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbgnrql"] Oct 31 04:45:27.545203 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:27.545186 5019 cpu_manager.go:414] "containerMap: removing container" podUID="d992b6ec-1a57-4ca9-af09-370245d1ccfe" containerName="cleanup" Oct 31 04:45:27.545268 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:27.545204 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d992b6ec-1a57-4ca9-af09-370245d1ccfe" containerName="cleanup" Oct 31 04:45:27.545268 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:27.545230 5019 cpu_manager.go:414] "containerMap: removing container" podUID="b7abcb04-58ef-4efd-a350-c8f5286ebc14" containerName="cleanup" Oct 31 04:45:27.545268 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:27.545235 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7abcb04-58ef-4efd-a350-c8f5286ebc14" containerName="cleanup" Oct 31 04:45:27.545268 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:27.545244 5019 cpu_manager.go:414] "containerMap: removing container" podUID="bc26f88e-b019-4540-a826-63fb1d624a8d" containerName="cleanup" Oct 31 04:45:27.545268 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:27.545249 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc26f88e-b019-4540-a826-63fb1d624a8d" containerName="cleanup" Oct 31 04:45:27.545268 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:27.545268 5019 cpu_manager.go:414] "containerMap: removing container" podUID="681c73ad-b2a1-458f-b37a-cc4e9e198926" containerName="cleanup" Oct 31 04:45:27.545483 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:27.545274 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="681c73ad-b2a1-458f-b37a-cc4e9e198926" containerName="cleanup" Oct 31 04:45:27.545483 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:27.545284 5019 cpu_manager.go:414] "containerMap: removing container" podUID="089f7eeb-81d7-4562-b906-da33b0fb5fa1" containerName="cleanup" Oct 31 04:45:27.545483 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:27.545288 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="089f7eeb-81d7-4562-b906-da33b0fb5fa1" containerName="cleanup" Oct 31 04:45:27.545483 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:27.545300 5019 cpu_manager.go:414] "containerMap: removing container" podUID="a53b729a-b280-41d4-8661-44826b486246" containerName="cleanup" Oct 31 04:45:27.545483 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:27.545304 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a53b729a-b280-41d4-8661-44826b486246" containerName="cleanup" Oct 31 04:45:27.545483 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:27.545311 5019 cpu_manager.go:414] "containerMap: removing container" podUID="2fc510cb-aae5-4d8c-a612-2270ccbbf432" containerName="cleanup" Oct 31 04:45:27.545483 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:27.545315 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fc510cb-aae5-4d8c-a612-2270ccbbf432" containerName="cleanup" Oct 31 04:45:27.545483 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:27.545357 5019 cpu_manager.go:414] "containerMap: removing container" podUID="01758cb6-b948-46f8-a607-210d75bb7716" containerName="cleanup" Oct 31 04:45:27.545483 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:27.545363 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01758cb6-b948-46f8-a607-210d75bb7716" containerName="cleanup" Oct 31 04:45:27.545483 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:27.545378 5019 cpu_manager.go:414] "containerMap: removing container" podUID="668f2b2b-4d01-4054-8ca0-16a4b0b4dcdd" containerName="cleanup" Oct 31 04:45:27.545483 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:27.545382 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="668f2b2b-4d01-4054-8ca0-16a4b0b4dcdd" containerName="cleanup" Oct 31 04:45:27.545825 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:27.545491 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e282b47a-b89e-48ae-88bf-347945705a1d" containerName="cleanup" Oct 31 04:45:27.545825 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:27.545499 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fc510cb-aae5-4d8c-a612-2270ccbbf432" containerName="cleanup" Oct 31 04:45:27.545825 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:27.545509 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc26f88e-b019-4540-a826-63fb1d624a8d" containerName="cleanup" Oct 31 04:45:27.545825 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:27.545519 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7abcb04-58ef-4efd-a350-c8f5286ebc14" containerName="cleanup" Oct 31 04:45:27.545825 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:27.545530 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="668f2b2b-4d01-4054-8ca0-16a4b0b4dcdd" containerName="cleanup" Oct 31 04:45:27.545825 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:27.545540 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="d992b6ec-1a57-4ca9-af09-370245d1ccfe" containerName="cleanup" Oct 31 04:45:27.545825 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:27.545547 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="01758cb6-b948-46f8-a607-210d75bb7716" containerName="cleanup" Oct 31 04:45:27.545825 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:27.545561 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="089f7eeb-81d7-4562-b906-da33b0fb5fa1" containerName="cleanup" Oct 31 04:45:27.550895 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:27.550869 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbgnrql" Oct 31 04:45:27.602354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:27.595655 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8973e2\"/\"agc-cred-055b438a3215\"" Oct 31 04:45:27.709059 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:27.709023 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbgnrql"] Oct 31 04:45:27.762119 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:27.762059 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/16e63654-7991-44d7-addb-985c942af606-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-7bbbgnrql\" (UID: \"16e63654-7991-44d7-addb-985c942af606\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbgnrql" Oct 31 04:45:27.762450 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:27.762422 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/16e63654-7991-44d7-addb-985c942af606-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-7bbbgnrql\" (UID: \"16e63654-7991-44d7-addb-985c942af606\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbgnrql" Oct 31 04:45:27.762774 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:27.762750 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16e63654-7991-44d7-addb-985c942af606-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-7bbbgnrql\" (UID: \"16e63654-7991-44d7-addb-985c942af606\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbgnrql" Oct 31 04:45:27.762953 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:27.762933 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n885f\" (UniqueName: \"kubernetes.io/projected/16e63654-7991-44d7-addb-985c942af606-kube-api-access-n885f\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-7bbbgnrql\" (UID: \"16e63654-7991-44d7-addb-985c942af606\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbgnrql" Oct 31 04:45:27.885371 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:27.884113 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16e63654-7991-44d7-addb-985c942af606-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-7bbbgnrql\" (UID: \"16e63654-7991-44d7-addb-985c942af606\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbgnrql" Oct 31 04:45:27.885371 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:27.884191 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n885f\" (UniqueName: \"kubernetes.io/projected/16e63654-7991-44d7-addb-985c942af606-kube-api-access-n885f\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-7bbbgnrql\" (UID: \"16e63654-7991-44d7-addb-985c942af606\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbgnrql" Oct 31 04:45:27.885371 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:27.884470 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/16e63654-7991-44d7-addb-985c942af606-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-7bbbgnrql\" (UID: \"16e63654-7991-44d7-addb-985c942af606\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbgnrql" Oct 31 04:45:27.885371 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:27.884504 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/16e63654-7991-44d7-addb-985c942af606-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-7bbbgnrql\" (UID: \"16e63654-7991-44d7-addb-985c942af606\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbgnrql" Oct 31 04:45:27.903358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:27.892597 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16e63654-7991-44d7-addb-985c942af606-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-7bbbgnrql\" (UID: \"16e63654-7991-44d7-addb-985c942af606\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbgnrql" Oct 31 04:45:27.904652 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:27.904614 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/16e63654-7991-44d7-addb-985c942af606-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-7bbbgnrql\" (UID: \"16e63654-7991-44d7-addb-985c942af606\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbgnrql" Oct 31 04:45:27.944194 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:27.944142 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/16e63654-7991-44d7-addb-985c942af606-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-7bbbgnrql\" (UID: \"16e63654-7991-44d7-addb-985c942af606\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbgnrql" Oct 31 04:45:27.952403 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:27.950133 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n885f\" (UniqueName: \"kubernetes.io/projected/16e63654-7991-44d7-addb-985c942af606-kube-api-access-n885f\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-7bbbgnrql\" (UID: \"16e63654-7991-44d7-addb-985c942af606\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbgnrql" Oct 31 04:45:27.994064 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:27.993966 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d812a3a9-zhkql" event={"ID":"c68e7b82-71db-48e2-8d07-36ddb22e9ae7","Type":"ContainerStarted","Data":"0ad4f5e8347d61ccd6d9623fb4cfeeb1be0ec4a091efbfedd81fcef6d96571db"} Oct 31 04:45:28.214665 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:28.214564 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbgnrql" Oct 31 04:45:28.534386 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:45:28.534343 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16e63654_7991_44d7_addb_985c942af606.slice/crio-184fc403ac6f27c70610c56b6d7cb7fa0ff9f2f16c02cfae28242f0d4f26fef7 WatchSource:0}: Error finding container 184fc403ac6f27c70610c56b6d7cb7fa0ff9f2f16c02cfae28242f0d4f26fef7: Status 404 returned error can't find the container with id 184fc403ac6f27c70610c56b6d7cb7fa0ff9f2f16c02cfae28242f0d4f26fef7 Oct 31 04:45:28.534536 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:28.534406 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbgnrql"] Oct 31 04:45:28.607495 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:28.607459 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbgnrql"] Oct 31 04:45:28.631298 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:28.631265 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7qh6vn"] Oct 31 04:45:28.636158 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:28.636130 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7qh6vn" Oct 31 04:45:28.652665 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:28.652625 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8973e2\"/\"agc-cred-fcdfce9e22b3\"" Oct 31 04:45:28.656393 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:28.656361 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7qh6vn"] Oct 31 04:45:28.703471 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:28.703418 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c9f3184-0ead-44b0-bc51-fc4972f5a031-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-67b7qh6vn\" (UID: \"4c9f3184-0ead-44b0-bc51-fc4972f5a031\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7qh6vn" Oct 31 04:45:28.703683 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:28.703619 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4c9f3184-0ead-44b0-bc51-fc4972f5a031-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-67b7qh6vn\" (UID: \"4c9f3184-0ead-44b0-bc51-fc4972f5a031\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7qh6vn" Oct 31 04:45:28.703789 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:28.703700 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz6pr\" (UniqueName: \"kubernetes.io/projected/4c9f3184-0ead-44b0-bc51-fc4972f5a031-kube-api-access-fz6pr\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-67b7qh6vn\" (UID: \"4c9f3184-0ead-44b0-bc51-fc4972f5a031\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7qh6vn" Oct 31 04:45:28.703789 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:28.703739 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4c9f3184-0ead-44b0-bc51-fc4972f5a031-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-67b7qh6vn\" (UID: \"4c9f3184-0ead-44b0-bc51-fc4972f5a031\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7qh6vn" Oct 31 04:45:28.805090 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:28.804979 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c9f3184-0ead-44b0-bc51-fc4972f5a031-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-67b7qh6vn\" (UID: \"4c9f3184-0ead-44b0-bc51-fc4972f5a031\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7qh6vn" Oct 31 04:45:28.805281 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:28.805095 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4c9f3184-0ead-44b0-bc51-fc4972f5a031-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-67b7qh6vn\" (UID: \"4c9f3184-0ead-44b0-bc51-fc4972f5a031\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7qh6vn" Oct 31 04:45:28.805281 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:28.805158 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz6pr\" (UniqueName: \"kubernetes.io/projected/4c9f3184-0ead-44b0-bc51-fc4972f5a031-kube-api-access-fz6pr\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-67b7qh6vn\" (UID: \"4c9f3184-0ead-44b0-bc51-fc4972f5a031\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7qh6vn" Oct 31 04:45:28.805281 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:28.805191 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4c9f3184-0ead-44b0-bc51-fc4972f5a031-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-67b7qh6vn\" (UID: \"4c9f3184-0ead-44b0-bc51-fc4972f5a031\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7qh6vn" Oct 31 04:45:28.805885 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:28.805851 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c9f3184-0ead-44b0-bc51-fc4972f5a031-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-67b7qh6vn\" (UID: \"4c9f3184-0ead-44b0-bc51-fc4972f5a031\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7qh6vn" Oct 31 04:45:28.806022 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:28.805932 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4c9f3184-0ead-44b0-bc51-fc4972f5a031-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-67b7qh6vn\" (UID: \"4c9f3184-0ead-44b0-bc51-fc4972f5a031\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7qh6vn" Oct 31 04:45:28.808393 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:28.808363 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4c9f3184-0ead-44b0-bc51-fc4972f5a031-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-67b7qh6vn\" (UID: \"4c9f3184-0ead-44b0-bc51-fc4972f5a031\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7qh6vn" Oct 31 04:45:28.828475 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:28.828444 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz6pr\" (UniqueName: \"kubernetes.io/projected/4c9f3184-0ead-44b0-bc51-fc4972f5a031-kube-api-access-fz6pr\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-67b7qh6vn\" (UID: \"4c9f3184-0ead-44b0-bc51-fc4972f5a031\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7qh6vn" Oct 31 04:45:28.967830 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:28.967786 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7qh6vn" Oct 31 04:45:29.107780 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:29.103184 5019 generic.go:356] "Generic (PLEG): container finished" podID="c68e7b82-71db-48e2-8d07-36ddb22e9ae7" containerID="d64da45f766c20875f51615657f1a73606b775a3928f5b502426bb8552162c1d" exitCode=0 Oct 31 04:45:29.107780 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:29.103416 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d812a3a9-zhkql" event={"ID":"c68e7b82-71db-48e2-8d07-36ddb22e9ae7","Type":"ContainerDied","Data":"d64da45f766c20875f51615657f1a73606b775a3928f5b502426bb8552162c1d"} Oct 31 04:45:29.107780 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:29.106942 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbgnrql" event={"ID":"16e63654-7991-44d7-addb-985c942af606","Type":"ContainerStarted","Data":"e2df96ae3cafbad1609b4cdc22017be35b0a8ee95029acf2f21a69fd09cb72d4"} Oct 31 04:45:29.107780 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:29.107010 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbgnrql" event={"ID":"16e63654-7991-44d7-addb-985c942af606","Type":"ContainerStarted","Data":"184fc403ac6f27c70610c56b6d7cb7fa0ff9f2f16c02cfae28242f0d4f26fef7"} Oct 31 04:45:29.107780 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:29.107281 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbgnrql" Oct 31 04:45:29.190979 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:29.190915 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbgnrql" podStartSLOduration=2.1908997 podStartE2EDuration="2.1908997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:45:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:45:29.18723657 +0000 UTC m=+2483.943574611" watchObservedRunningTime="2026-10-31 04:45:29.1908997 +0000 UTC m=+2483.947237761" Oct 31 04:45:29.291266 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:45:29.291227 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c9f3184_0ead_44b0_bc51_fc4972f5a031.slice/crio-96bc3f71c389abdd1d356f0cac529f3bc3396af879ad94512e3f94569cb2df82 WatchSource:0}: Error finding container 96bc3f71c389abdd1d356f0cac529f3bc3396af879ad94512e3f94569cb2df82: Status 404 returned error can't find the container with id 96bc3f71c389abdd1d356f0cac529f3bc3396af879ad94512e3f94569cb2df82 Oct 31 04:45:29.292202 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:29.292170 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7qh6vn"] Oct 31 04:45:29.498412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:29.495247 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-797454d2z"] Oct 31 04:45:30.120880 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:30.120833 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7qh6vn" event={"ID":"4c9f3184-0ead-44b0-bc51-fc4972f5a031","Type":"ContainerStarted","Data":"03e920087f6a143280eaa6b64dff0aee6e85ade6c9c61e8806547e9e9ad61abd"} Oct 31 04:45:30.121379 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:30.120946 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7qh6vn" Oct 31 04:45:30.121379 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:30.120987 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7qh6vn" event={"ID":"4c9f3184-0ead-44b0-bc51-fc4972f5a031","Type":"ContainerStarted","Data":"96bc3f71c389abdd1d356f0cac529f3bc3396af879ad94512e3f94569cb2df82"} Oct 31 04:45:30.162314 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:30.162259 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7qh6vn" podStartSLOduration=2.162242824 podStartE2EDuration="2.162242824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:45:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:45:30.161468912 +0000 UTC m=+2484.917806953" watchObservedRunningTime="2026-10-31 04:45:30.162242824 +0000 UTC m=+2484.918580865" Oct 31 04:45:30.335406 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:30.335374 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d812a3a9-zhkql" Oct 31 04:45:30.436246 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:30.436147 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c68e7b82-71db-48e2-8d07-36ddb22e9ae7-kube-api-access-vrc7r\" (UniqueName: \"kubernetes.io/projected/c68e7b82-71db-48e2-8d07-36ddb22e9ae7-kube-api-access-vrc7r\") pod \"c68e7b82-71db-48e2-8d07-36ddb22e9ae7\" (UID: \"c68e7b82-71db-48e2-8d07-36ddb22e9ae7\") " Oct 31 04:45:30.436481 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:30.436462 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c68e7b82-71db-48e2-8d07-36ddb22e9ae7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c68e7b82-71db-48e2-8d07-36ddb22e9ae7-maas-api-openshift-ca\") pod \"c68e7b82-71db-48e2-8d07-36ddb22e9ae7\" (UID: \"c68e7b82-71db-48e2-8d07-36ddb22e9ae7\") " Oct 31 04:45:30.436605 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:30.436565 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c68e7b82-71db-48e2-8d07-36ddb22e9ae7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c68e7b82-71db-48e2-8d07-36ddb22e9ae7-maas-api-xks-ca\") pod \"c68e7b82-71db-48e2-8d07-36ddb22e9ae7\" (UID: \"c68e7b82-71db-48e2-8d07-36ddb22e9ae7\") " Oct 31 04:45:30.436813 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:30.436787 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c68e7b82-71db-48e2-8d07-36ddb22e9ae7-maas-api-openshift-ca" pod "c68e7b82-71db-48e2-8d07-36ddb22e9ae7" (UID: "c68e7b82-71db-48e2-8d07-36ddb22e9ae7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:45:30.437695 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:30.437516 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c68e7b82-71db-48e2-8d07-36ddb22e9ae7-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:30.439033 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:30.439001 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c68e7b82-71db-48e2-8d07-36ddb22e9ae7-maas-api-xks-ca" pod "c68e7b82-71db-48e2-8d07-36ddb22e9ae7" (UID: "c68e7b82-71db-48e2-8d07-36ddb22e9ae7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:45:30.439505 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:30.439476 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c68e7b82-71db-48e2-8d07-36ddb22e9ae7-kube-api-access-vrc7r" pod "c68e7b82-71db-48e2-8d07-36ddb22e9ae7" (UID: "c68e7b82-71db-48e2-8d07-36ddb22e9ae7"). InnerVolumeSpecName "kube-api-access-vrc7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:45:30.539362 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:30.539308 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c68e7b82-71db-48e2-8d07-36ddb22e9ae7-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:30.539362 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:30.539359 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrc7r\" (UniqueName: \"kubernetes.io/projected/c68e7b82-71db-48e2-8d07-36ddb22e9ae7-kube-api-access-vrc7r\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:31.132208 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:31.132171 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d812a3a9-zhkql" Oct 31 04:45:31.132669 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:31.132211 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d812a3a9-zhkql" event={"ID":"c68e7b82-71db-48e2-8d07-36ddb22e9ae7","Type":"ContainerDied","Data":"0ad4f5e8347d61ccd6d9623fb4cfeeb1be0ec4a091efbfedd81fcef6d96571db"} Oct 31 04:45:31.132669 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:31.132270 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ad4f5e8347d61ccd6d9623fb4cfeeb1be0ec4a091efbfedd81fcef6d96571db" Oct 31 04:45:34.108262 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:34.108179 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbgnrql" podUID="16e63654-7991-44d7-addb-985c942af606" containerName="payload-processing" containerID="cri-o://e2df96ae3cafbad1609b4cdc22017be35b0a8ee95029acf2f21a69fd09cb72d4" gracePeriod=25 Oct 31 04:45:34.429031 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:34.429002 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbgnrql" Oct 31 04:45:34.461074 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:34.461036 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93f03b01-5tzb7"] Oct 31 04:45:34.461898 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:34.461878 5019 cpu_manager.go:414] "containerMap: removing container" podUID="c68e7b82-71db-48e2-8d07-36ddb22e9ae7" containerName="revoke-keys" Oct 31 04:45:34.461974 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:34.461899 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c68e7b82-71db-48e2-8d07-36ddb22e9ae7" containerName="revoke-keys" Oct 31 04:45:34.461974 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:34.461926 5019 cpu_manager.go:414] "containerMap: removing container" podUID="16e63654-7991-44d7-addb-985c942af606" containerName="payload-processing" Oct 31 04:45:34.461974 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:34.461931 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16e63654-7991-44d7-addb-985c942af606" containerName="payload-processing" Oct 31 04:45:34.462168 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:34.462153 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="c68e7b82-71db-48e2-8d07-36ddb22e9ae7" containerName="revoke-keys" Oct 31 04:45:34.462221 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:34.462171 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="16e63654-7991-44d7-addb-985c942af606" containerName="payload-processing" Oct 31 04:45:34.465996 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:34.465979 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93f03b01-5tzb7" Oct 31 04:45:34.486378 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:34.474615 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93f03b01-5tzb7"] Oct 31 04:45:34.486378 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:34.482031 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:45:34.486378 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:34.482120 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:45:34.486378 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:34.482162 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:45:34.486378 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:34.482845 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:45:34.496193 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:34.496152 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-797454d2z" podUID="d8fa7e98-3021-4077-9e2b-14f31b6efd65" containerName="payload-processing" containerID="cri-o://94dc9c8d78568acfead4453ab8f571aabb096b70541ca875cf586a356be63eee" gracePeriod=25 Oct 31 04:45:34.520500 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:34.520467 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16e63654-7991-44d7-addb-985c942af606-kube-api-access-n885f\" (UniqueName: \"kubernetes.io/projected/16e63654-7991-44d7-addb-985c942af606-kube-api-access-n885f\") pod \"16e63654-7991-44d7-addb-985c942af606\" (UID: \"16e63654-7991-44d7-addb-985c942af606\") " Oct 31 04:45:34.520665 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:34.520512 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16e63654-7991-44d7-addb-985c942af606-provider-credentials\" (UniqueName: \"kubernetes.io/projected/16e63654-7991-44d7-addb-985c942af606-provider-credentials\") pod \"16e63654-7991-44d7-addb-985c942af606\" (UID: \"16e63654-7991-44d7-addb-985c942af606\") " Oct 31 04:45:34.520665 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:34.520553 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16e63654-7991-44d7-addb-985c942af606-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/16e63654-7991-44d7-addb-985c942af606-routing-overlay-volume\") pod \"16e63654-7991-44d7-addb-985c942af606\" (UID: \"16e63654-7991-44d7-addb-985c942af606\") " Oct 31 04:45:34.520777 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:34.520731 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16e63654-7991-44d7-addb-985c942af606-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16e63654-7991-44d7-addb-985c942af606-plugins-config-volume\") pod \"16e63654-7991-44d7-addb-985c942af606\" (UID: \"16e63654-7991-44d7-addb-985c942af606\") " Oct 31 04:45:34.521621 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:34.521598 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16e63654-7991-44d7-addb-985c942af606-plugins-config-volume" pod "16e63654-7991-44d7-addb-985c942af606" (UID: "16e63654-7991-44d7-addb-985c942af606"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:45:34.522177 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:34.522135 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16e63654-7991-44d7-addb-985c942af606-routing-overlay-volume" pod "16e63654-7991-44d7-addb-985c942af606" (UID: "16e63654-7991-44d7-addb-985c942af606"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:45:34.525143 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:34.524160 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16e63654-7991-44d7-addb-985c942af606-provider-credentials" pod "16e63654-7991-44d7-addb-985c942af606" (UID: "16e63654-7991-44d7-addb-985c942af606"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:45:34.525852 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:34.525830 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16e63654-7991-44d7-addb-985c942af606-kube-api-access-n885f" pod "16e63654-7991-44d7-addb-985c942af606" (UID: "16e63654-7991-44d7-addb-985c942af606"). InnerVolumeSpecName "kube-api-access-n885f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:45:34.622475 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:34.622439 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/609c6d46-2c07-412b-b44f-7b2908fb93d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93f03b01-5tzb7\" (UID: \"609c6d46-2c07-412b-b44f-7b2908fb93d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93f03b01-5tzb7" Oct 31 04:45:34.622620 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:34.622591 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-464pr\" (UniqueName: \"kubernetes.io/projected/609c6d46-2c07-412b-b44f-7b2908fb93d2-kube-api-access-464pr\") pod \"maas-api-revoke-sub-93f03b01-5tzb7\" (UID: \"609c6d46-2c07-412b-b44f-7b2908fb93d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93f03b01-5tzb7" Oct 31 04:45:34.622745 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:34.622728 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/609c6d46-2c07-412b-b44f-7b2908fb93d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93f03b01-5tzb7\" (UID: \"609c6d46-2c07-412b-b44f-7b2908fb93d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93f03b01-5tzb7" Oct 31 04:45:34.622915 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:34.622898 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n885f\" (UniqueName: \"kubernetes.io/projected/16e63654-7991-44d7-addb-985c942af606-kube-api-access-n885f\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:34.622966 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:34.622918 5019 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/16e63654-7991-44d7-addb-985c942af606-provider-credentials\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:34.622966 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:34.622935 5019 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/16e63654-7991-44d7-addb-985c942af606-routing-overlay-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:34.622966 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:34.622945 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16e63654-7991-44d7-addb-985c942af606-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:34.724946 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:34.724900 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-464pr\" (UniqueName: \"kubernetes.io/projected/609c6d46-2c07-412b-b44f-7b2908fb93d2-kube-api-access-464pr\") pod \"maas-api-revoke-sub-93f03b01-5tzb7\" (UID: \"609c6d46-2c07-412b-b44f-7b2908fb93d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93f03b01-5tzb7" Oct 31 04:45:34.725160 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:34.725053 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/609c6d46-2c07-412b-b44f-7b2908fb93d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93f03b01-5tzb7\" (UID: \"609c6d46-2c07-412b-b44f-7b2908fb93d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93f03b01-5tzb7" Oct 31 04:45:34.725345 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:34.725311 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/609c6d46-2c07-412b-b44f-7b2908fb93d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93f03b01-5tzb7\" (UID: \"609c6d46-2c07-412b-b44f-7b2908fb93d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93f03b01-5tzb7" Oct 31 04:45:34.726234 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:34.726204 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/609c6d46-2c07-412b-b44f-7b2908fb93d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93f03b01-5tzb7\" (UID: \"609c6d46-2c07-412b-b44f-7b2908fb93d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93f03b01-5tzb7" Oct 31 04:45:34.729439 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:34.729410 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/609c6d46-2c07-412b-b44f-7b2908fb93d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93f03b01-5tzb7\" (UID: \"609c6d46-2c07-412b-b44f-7b2908fb93d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93f03b01-5tzb7" Oct 31 04:45:34.750109 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:34.750059 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-464pr\" (UniqueName: \"kubernetes.io/projected/609c6d46-2c07-412b-b44f-7b2908fb93d2-kube-api-access-464pr\") pod \"maas-api-revoke-sub-93f03b01-5tzb7\" (UID: \"609c6d46-2c07-412b-b44f-7b2908fb93d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93f03b01-5tzb7" Oct 31 04:45:34.768912 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:34.768887 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-797454d2z" Oct 31 04:45:34.792807 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:34.792773 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93f03b01-5tzb7" Oct 31 04:45:34.930570 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:34.930319 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8fa7e98-3021-4077-9e2b-14f31b6efd65-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8fa7e98-3021-4077-9e2b-14f31b6efd65-plugins-config-volume\") pod \"d8fa7e98-3021-4077-9e2b-14f31b6efd65\" (UID: \"d8fa7e98-3021-4077-9e2b-14f31b6efd65\") " Oct 31 04:45:34.930833 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:34.930799 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8fa7e98-3021-4077-9e2b-14f31b6efd65-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d8fa7e98-3021-4077-9e2b-14f31b6efd65-provider-credentials\") pod \"d8fa7e98-3021-4077-9e2b-14f31b6efd65\" (UID: \"d8fa7e98-3021-4077-9e2b-14f31b6efd65\") " Oct 31 04:45:34.931038 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:34.930898 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8fa7e98-3021-4077-9e2b-14f31b6efd65-plugins-config-volume" pod "d8fa7e98-3021-4077-9e2b-14f31b6efd65" (UID: "d8fa7e98-3021-4077-9e2b-14f31b6efd65"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:45:34.931281 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:34.931234 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8fa7e98-3021-4077-9e2b-14f31b6efd65-kube-api-access-l8hxh\" (UniqueName: \"kubernetes.io/projected/d8fa7e98-3021-4077-9e2b-14f31b6efd65-kube-api-access-l8hxh\") pod \"d8fa7e98-3021-4077-9e2b-14f31b6efd65\" (UID: \"d8fa7e98-3021-4077-9e2b-14f31b6efd65\") " Oct 31 04:45:34.931426 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:34.931406 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8fa7e98-3021-4077-9e2b-14f31b6efd65-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d8fa7e98-3021-4077-9e2b-14f31b6efd65-routing-overlay-volume\") pod \"d8fa7e98-3021-4077-9e2b-14f31b6efd65\" (UID: \"d8fa7e98-3021-4077-9e2b-14f31b6efd65\") " Oct 31 04:45:34.931954 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:34.931923 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8fa7e98-3021-4077-9e2b-14f31b6efd65-routing-overlay-volume" pod "d8fa7e98-3021-4077-9e2b-14f31b6efd65" (UID: "d8fa7e98-3021-4077-9e2b-14f31b6efd65"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:45:34.932271 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:34.932243 5019 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d8fa7e98-3021-4077-9e2b-14f31b6efd65-routing-overlay-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:34.932440 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:34.932275 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8fa7e98-3021-4077-9e2b-14f31b6efd65-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:34.933499 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:34.933475 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8fa7e98-3021-4077-9e2b-14f31b6efd65-provider-credentials" pod "d8fa7e98-3021-4077-9e2b-14f31b6efd65" (UID: "d8fa7e98-3021-4077-9e2b-14f31b6efd65"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:45:34.934017 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:34.933996 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8fa7e98-3021-4077-9e2b-14f31b6efd65-kube-api-access-l8hxh" pod "d8fa7e98-3021-4077-9e2b-14f31b6efd65" (UID: "d8fa7e98-3021-4077-9e2b-14f31b6efd65"). InnerVolumeSpecName "kube-api-access-l8hxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:45:35.034001 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:35.033949 5019 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d8fa7e98-3021-4077-9e2b-14f31b6efd65-provider-credentials\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:35.034001 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:35.034008 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8hxh\" (UniqueName: \"kubernetes.io/projected/d8fa7e98-3021-4077-9e2b-14f31b6efd65-kube-api-access-l8hxh\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:35.105970 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:35.105937 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93f03b01-5tzb7"] Oct 31 04:45:35.106814 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:45:35.106786 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod609c6d46_2c07_412b_b44f_7b2908fb93d2.slice/crio-f5155ee66bc74de2e36548c066919f0c33b07c5fd591b244e1fb8f83f0ba37ed WatchSource:0}: Error finding container f5155ee66bc74de2e36548c066919f0c33b07c5fd591b244e1fb8f83f0ba37ed: Status 404 returned error can't find the container with id f5155ee66bc74de2e36548c066919f0c33b07c5fd591b244e1fb8f83f0ba37ed Oct 31 04:45:35.172030 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:35.171990 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93f03b01-5tzb7" event={"ID":"609c6d46-2c07-412b-b44f-7b2908fb93d2","Type":"ContainerStarted","Data":"f5155ee66bc74de2e36548c066919f0c33b07c5fd591b244e1fb8f83f0ba37ed"} Oct 31 04:45:35.173796 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:35.173764 5019 generic.go:356] "Generic (PLEG): container finished" podID="16e63654-7991-44d7-addb-985c942af606" containerID="e2df96ae3cafbad1609b4cdc22017be35b0a8ee95029acf2f21a69fd09cb72d4" exitCode=0 Oct 31 04:45:35.174066 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:35.174050 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbgnrql" Oct 31 04:45:35.174784 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:35.174751 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbgnrql" event={"ID":"16e63654-7991-44d7-addb-985c942af606","Type":"ContainerDied","Data":"e2df96ae3cafbad1609b4cdc22017be35b0a8ee95029acf2f21a69fd09cb72d4"} Oct 31 04:45:35.174896 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:35.174816 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbgnrql" event={"ID":"16e63654-7991-44d7-addb-985c942af606","Type":"ContainerDied","Data":"184fc403ac6f27c70610c56b6d7cb7fa0ff9f2f16c02cfae28242f0d4f26fef7"} Oct 31 04:45:35.174896 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:35.174850 5019 scope.go:122] "RemoveContainer" containerID="e2df96ae3cafbad1609b4cdc22017be35b0a8ee95029acf2f21a69fd09cb72d4" Oct 31 04:45:35.176510 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:35.176482 5019 generic.go:356] "Generic (PLEG): container finished" podID="d8fa7e98-3021-4077-9e2b-14f31b6efd65" containerID="94dc9c8d78568acfead4453ab8f571aabb096b70541ca875cf586a356be63eee" exitCode=0 Oct 31 04:45:35.176591 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:35.176511 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-797454d2z" event={"ID":"d8fa7e98-3021-4077-9e2b-14f31b6efd65","Type":"ContainerDied","Data":"94dc9c8d78568acfead4453ab8f571aabb096b70541ca875cf586a356be63eee"} Oct 31 04:45:35.176591 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:35.176541 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-797454d2z" event={"ID":"d8fa7e98-3021-4077-9e2b-14f31b6efd65","Type":"ContainerDied","Data":"a0bec2d5c768a03354f96a716ed56ee1eed94944cb6902482be7fbdd03d9cd22"} Oct 31 04:45:35.176714 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:35.176592 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-797454d2z" Oct 31 04:45:35.194565 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:35.187609 5019 scope.go:122] "RemoveContainer" containerID="e2df96ae3cafbad1609b4cdc22017be35b0a8ee95029acf2f21a69fd09cb72d4" Oct 31 04:45:35.194565 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:35.187982 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2df96ae3cafbad1609b4cdc22017be35b0a8ee95029acf2f21a69fd09cb72d4\": container with ID starting with e2df96ae3cafbad1609b4cdc22017be35b0a8ee95029acf2f21a69fd09cb72d4 not found: ID does not exist" containerID="e2df96ae3cafbad1609b4cdc22017be35b0a8ee95029acf2f21a69fd09cb72d4" Oct 31 04:45:35.194565 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:35.188017 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2df96ae3cafbad1609b4cdc22017be35b0a8ee95029acf2f21a69fd09cb72d4"} err="failed to get container status \"e2df96ae3cafbad1609b4cdc22017be35b0a8ee95029acf2f21a69fd09cb72d4\": rpc error: code = NotFound desc = could not find container \"e2df96ae3cafbad1609b4cdc22017be35b0a8ee95029acf2f21a69fd09cb72d4\": container with ID starting with e2df96ae3cafbad1609b4cdc22017be35b0a8ee95029acf2f21a69fd09cb72d4 not found: ID does not exist" Oct 31 04:45:35.194565 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:35.188042 5019 scope.go:122] "RemoveContainer" containerID="94dc9c8d78568acfead4453ab8f571aabb096b70541ca875cf586a356be63eee" Oct 31 04:45:35.201824 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:35.201803 5019 scope.go:122] "RemoveContainer" containerID="94dc9c8d78568acfead4453ab8f571aabb096b70541ca875cf586a356be63eee" Oct 31 04:45:35.202158 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:35.202132 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94dc9c8d78568acfead4453ab8f571aabb096b70541ca875cf586a356be63eee\": container with ID starting with 94dc9c8d78568acfead4453ab8f571aabb096b70541ca875cf586a356be63eee not found: ID does not exist" containerID="94dc9c8d78568acfead4453ab8f571aabb096b70541ca875cf586a356be63eee" Oct 31 04:45:35.202236 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:35.202171 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94dc9c8d78568acfead4453ab8f571aabb096b70541ca875cf586a356be63eee"} err="failed to get container status \"94dc9c8d78568acfead4453ab8f571aabb096b70541ca875cf586a356be63eee\": rpc error: code = NotFound desc = could not find container \"94dc9c8d78568acfead4453ab8f571aabb096b70541ca875cf586a356be63eee\": container with ID starting with 94dc9c8d78568acfead4453ab8f571aabb096b70541ca875cf586a356be63eee not found: ID does not exist" Oct 31 04:45:35.245177 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:35.245145 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbgnrql"] Oct 31 04:45:35.249943 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:35.249912 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbgnrql"] Oct 31 04:45:35.284124 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:35.284087 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-797454d2z"] Oct 31 04:45:35.288865 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:35.288835 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-797454d2z"] Oct 31 04:45:35.428458 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:35.428418 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16e63654-7991-44d7-addb-985c942af606" path="/var/lib/kubelet/pods/16e63654-7991-44d7-addb-985c942af606/volumes" Oct 31 04:45:35.429100 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:35.429083 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8fa7e98-3021-4077-9e2b-14f31b6efd65" path="/var/lib/kubelet/pods/d8fa7e98-3021-4077-9e2b-14f31b6efd65/volumes" Oct 31 04:45:36.193870 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:36.193833 5019 generic.go:356] "Generic (PLEG): container finished" podID="609c6d46-2c07-412b-b44f-7b2908fb93d2" containerID="9d909ffe2911dfd54ab8908d9cd6f1969bfa23a897766ba59cd62e90547ce37c" exitCode=0 Oct 31 04:45:36.194317 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:36.193914 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93f03b01-5tzb7" event={"ID":"609c6d46-2c07-412b-b44f-7b2908fb93d2","Type":"ContainerDied","Data":"9d909ffe2911dfd54ab8908d9cd6f1969bfa23a897766ba59cd62e90547ce37c"} Oct 31 04:45:37.449141 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:37.449103 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93f03b01-5tzb7" Oct 31 04:45:37.578606 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:37.578568 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/609c6d46-2c07-412b-b44f-7b2908fb93d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/609c6d46-2c07-412b-b44f-7b2908fb93d2-maas-api-openshift-ca\") pod \"609c6d46-2c07-412b-b44f-7b2908fb93d2\" (UID: \"609c6d46-2c07-412b-b44f-7b2908fb93d2\") " Oct 31 04:45:37.578841 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:37.578731 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/609c6d46-2c07-412b-b44f-7b2908fb93d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/609c6d46-2c07-412b-b44f-7b2908fb93d2-maas-api-xks-ca\") pod \"609c6d46-2c07-412b-b44f-7b2908fb93d2\" (UID: \"609c6d46-2c07-412b-b44f-7b2908fb93d2\") " Oct 31 04:45:37.578841 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:37.578827 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/609c6d46-2c07-412b-b44f-7b2908fb93d2-kube-api-access-464pr\" (UniqueName: \"kubernetes.io/projected/609c6d46-2c07-412b-b44f-7b2908fb93d2-kube-api-access-464pr\") pod \"609c6d46-2c07-412b-b44f-7b2908fb93d2\" (UID: \"609c6d46-2c07-412b-b44f-7b2908fb93d2\") " Oct 31 04:45:37.579477 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:37.579328 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/609c6d46-2c07-412b-b44f-7b2908fb93d2-maas-api-openshift-ca" pod "609c6d46-2c07-412b-b44f-7b2908fb93d2" (UID: "609c6d46-2c07-412b-b44f-7b2908fb93d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:45:37.579720 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:37.579671 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/609c6d46-2c07-412b-b44f-7b2908fb93d2-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:37.581295 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:37.581263 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/609c6d46-2c07-412b-b44f-7b2908fb93d2-kube-api-access-464pr" pod "609c6d46-2c07-412b-b44f-7b2908fb93d2" (UID: "609c6d46-2c07-412b-b44f-7b2908fb93d2"). InnerVolumeSpecName "kube-api-access-464pr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:45:37.582163 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:37.582141 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/609c6d46-2c07-412b-b44f-7b2908fb93d2-maas-api-xks-ca" pod "609c6d46-2c07-412b-b44f-7b2908fb93d2" (UID: "609c6d46-2c07-412b-b44f-7b2908fb93d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:45:37.681733 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:37.681691 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/609c6d46-2c07-412b-b44f-7b2908fb93d2-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:37.681733 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:37.681728 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-464pr\" (UniqueName: \"kubernetes.io/projected/609c6d46-2c07-412b-b44f-7b2908fb93d2-kube-api-access-464pr\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:38.057896 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:38.057856 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bmtf9q"] Oct 31 04:45:38.058746 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:38.058725 5019 cpu_manager.go:414] "containerMap: removing container" podUID="d8fa7e98-3021-4077-9e2b-14f31b6efd65" containerName="payload-processing" Oct 31 04:45:38.058746 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:38.058747 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8fa7e98-3021-4077-9e2b-14f31b6efd65" containerName="payload-processing" Oct 31 04:45:38.058855 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:38.058772 5019 cpu_manager.go:414] "containerMap: removing container" podUID="609c6d46-2c07-412b-b44f-7b2908fb93d2" containerName="revoke-keys" Oct 31 04:45:38.058855 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:38.058777 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="609c6d46-2c07-412b-b44f-7b2908fb93d2" containerName="revoke-keys" Oct 31 04:45:38.058927 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:38.058922 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8fa7e98-3021-4077-9e2b-14f31b6efd65" containerName="payload-processing" Oct 31 04:45:38.058962 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:38.058933 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="609c6d46-2c07-412b-b44f-7b2908fb93d2" containerName="revoke-keys" Oct 31 04:45:38.063021 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:38.062997 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bmtf9q" Oct 31 04:45:38.084548 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:38.083308 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bmtf9q"] Oct 31 04:45:38.192739 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:38.192699 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f10d9686-b9c2-40ad-97df-b358573a707c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-866bmtf9q\" (UID: \"f10d9686-b9c2-40ad-97df-b358573a707c\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bmtf9q" Oct 31 04:45:38.192922 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:38.192777 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc6hv\" (UniqueName: \"kubernetes.io/projected/f10d9686-b9c2-40ad-97df-b358573a707c-kube-api-access-sc6hv\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-866bmtf9q\" (UID: \"f10d9686-b9c2-40ad-97df-b358573a707c\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bmtf9q" Oct 31 04:45:38.192966 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:38.192920 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f10d9686-b9c2-40ad-97df-b358573a707c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-866bmtf9q\" (UID: \"f10d9686-b9c2-40ad-97df-b358573a707c\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bmtf9q" Oct 31 04:45:38.193034 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:38.193018 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f10d9686-b9c2-40ad-97df-b358573a707c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-866bmtf9q\" (UID: \"f10d9686-b9c2-40ad-97df-b358573a707c\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bmtf9q" Oct 31 04:45:38.408430 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:38.396513 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f10d9686-b9c2-40ad-97df-b358573a707c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-866bmtf9q\" (UID: \"f10d9686-b9c2-40ad-97df-b358573a707c\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bmtf9q" Oct 31 04:45:38.408430 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:38.396660 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc6hv\" (UniqueName: \"kubernetes.io/projected/f10d9686-b9c2-40ad-97df-b358573a707c-kube-api-access-sc6hv\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-866bmtf9q\" (UID: \"f10d9686-b9c2-40ad-97df-b358573a707c\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bmtf9q" Oct 31 04:45:38.408430 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:38.396705 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f10d9686-b9c2-40ad-97df-b358573a707c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-866bmtf9q\" (UID: \"f10d9686-b9c2-40ad-97df-b358573a707c\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bmtf9q" Oct 31 04:45:38.408430 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:38.396767 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f10d9686-b9c2-40ad-97df-b358573a707c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-866bmtf9q\" (UID: \"f10d9686-b9c2-40ad-97df-b358573a707c\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bmtf9q" Oct 31 04:45:38.408430 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:38.403082 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f10d9686-b9c2-40ad-97df-b358573a707c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-866bmtf9q\" (UID: \"f10d9686-b9c2-40ad-97df-b358573a707c\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bmtf9q" Oct 31 04:45:38.408430 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:38.403509 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f10d9686-b9c2-40ad-97df-b358573a707c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-866bmtf9q\" (UID: \"f10d9686-b9c2-40ad-97df-b358573a707c\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bmtf9q" Oct 31 04:45:38.471385 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:38.466775 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f10d9686-b9c2-40ad-97df-b358573a707c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-866bmtf9q\" (UID: \"f10d9686-b9c2-40ad-97df-b358573a707c\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bmtf9q" Oct 31 04:45:38.497275 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:38.497087 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93f03b01-5tzb7" Oct 31 04:45:38.497641 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:38.497395 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93f03b01-5tzb7" event={"ID":"609c6d46-2c07-412b-b44f-7b2908fb93d2","Type":"ContainerDied","Data":"f5155ee66bc74de2e36548c066919f0c33b07c5fd591b244e1fb8f83f0ba37ed"} Oct 31 04:45:38.497747 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:38.497694 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5155ee66bc74de2e36548c066919f0c33b07c5fd591b244e1fb8f83f0ba37ed" Oct 31 04:45:38.507937 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:38.506485 5019 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-567ca6/agc-cred-876b469dc150: secret "agc-cred-876b469dc150" not found Oct 31 04:45:38.507937 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:38.506521 5019 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-5d44jzg8t: secret "agc-cred-876b469dc150" not found Oct 31 04:45:38.507937 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:38.506603 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/096d1018-1c07-4bce-b449-08c34c8457fb-provider-credentials podName:096d1018-1c07-4bce-b449-08c34c8457fb nodeName:}" failed. No retries permitted until 2026-10-31 04:45:39.006577708 +0000 UTC m=+2493.762915743 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/096d1018-1c07-4bce-b449-08c34c8457fb-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-567ca6-5d44jzg8t" (UID: "096d1018-1c07-4bce-b449-08c34c8457fb") : secret "agc-cred-876b469dc150" not found Oct 31 04:45:38.536704 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:38.536665 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b181387a-z7q2c"] Oct 31 04:45:38.554400 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:38.549251 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc6hv\" (UniqueName: \"kubernetes.io/projected/f10d9686-b9c2-40ad-97df-b358573a707c-kube-api-access-sc6hv\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-866bmtf9q\" (UID: \"f10d9686-b9c2-40ad-97df-b358573a707c\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bmtf9q" Oct 31 04:45:38.554400 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:38.551424 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b181387a-z7q2c"] Oct 31 04:45:38.554400 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:38.551530 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b181387a-z7q2c" Oct 31 04:45:38.570451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:38.566559 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:45:38.570451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:38.566701 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:45:38.570451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:38.566715 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:45:38.570451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:38.566789 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:45:38.681392 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:38.681253 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bmtf9q" Oct 31 04:45:38.710732 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:38.710691 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/846eb567-29ee-430d-9f65-007a72737fcd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b181387a-z7q2c\" (UID: \"846eb567-29ee-430d-9f65-007a72737fcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b181387a-z7q2c" Oct 31 04:45:38.710936 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:38.710815 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stpg2\" (UniqueName: \"kubernetes.io/projected/846eb567-29ee-430d-9f65-007a72737fcd-kube-api-access-stpg2\") pod \"maas-api-revoke-sub-b181387a-z7q2c\" (UID: \"846eb567-29ee-430d-9f65-007a72737fcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b181387a-z7q2c" Oct 31 04:45:38.710936 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:38.710882 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/846eb567-29ee-430d-9f65-007a72737fcd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b181387a-z7q2c\" (UID: \"846eb567-29ee-430d-9f65-007a72737fcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b181387a-z7q2c" Oct 31 04:45:38.812656 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:38.812611 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/846eb567-29ee-430d-9f65-007a72737fcd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b181387a-z7q2c\" (UID: \"846eb567-29ee-430d-9f65-007a72737fcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b181387a-z7q2c" Oct 31 04:45:38.812862 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:38.812746 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stpg2\" (UniqueName: \"kubernetes.io/projected/846eb567-29ee-430d-9f65-007a72737fcd-kube-api-access-stpg2\") pod \"maas-api-revoke-sub-b181387a-z7q2c\" (UID: \"846eb567-29ee-430d-9f65-007a72737fcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b181387a-z7q2c" Oct 31 04:45:38.812862 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:38.812813 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/846eb567-29ee-430d-9f65-007a72737fcd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b181387a-z7q2c\" (UID: \"846eb567-29ee-430d-9f65-007a72737fcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b181387a-z7q2c" Oct 31 04:45:38.813584 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:38.813551 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/846eb567-29ee-430d-9f65-007a72737fcd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b181387a-z7q2c\" (UID: \"846eb567-29ee-430d-9f65-007a72737fcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b181387a-z7q2c" Oct 31 04:45:38.816277 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:38.816237 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/846eb567-29ee-430d-9f65-007a72737fcd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b181387a-z7q2c\" (UID: \"846eb567-29ee-430d-9f65-007a72737fcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b181387a-z7q2c" Oct 31 04:45:38.837110 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:38.837065 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stpg2\" (UniqueName: \"kubernetes.io/projected/846eb567-29ee-430d-9f65-007a72737fcd-kube-api-access-stpg2\") pod \"maas-api-revoke-sub-b181387a-z7q2c\" (UID: \"846eb567-29ee-430d-9f65-007a72737fcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b181387a-z7q2c" Oct 31 04:45:38.882637 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:38.882590 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b181387a-z7q2c" Oct 31 04:45:38.995880 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:38.995842 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bmtf9q"] Oct 31 04:45:38.998620 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:45:38.998588 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf10d9686_b9c2_40ad_97df_b358573a707c.slice/crio-254adc3e04f61cd38c8a4b8ae9b46f9300651cf7cb8be9cce7bfac5ba417ce77 WatchSource:0}: Error finding container 254adc3e04f61cd38c8a4b8ae9b46f9300651cf7cb8be9cce7bfac5ba417ce77: Status 404 returned error can't find the container with id 254adc3e04f61cd38c8a4b8ae9b46f9300651cf7cb8be9cce7bfac5ba417ce77 Oct 31 04:45:39.016462 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:39.016320 5019 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-567ca6/agc-cred-876b469dc150: secret "agc-cred-876b469dc150" not found Oct 31 04:45:39.016462 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:39.016366 5019 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-5d44jzg8t: secret "agc-cred-876b469dc150" not found Oct 31 04:45:39.016462 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:39.016432 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/096d1018-1c07-4bce-b449-08c34c8457fb-provider-credentials podName:096d1018-1c07-4bce-b449-08c34c8457fb nodeName:}" failed. No retries permitted until 2026-10-31 04:45:40.016412587 +0000 UTC m=+2494.772750618 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/096d1018-1c07-4bce-b449-08c34c8457fb-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-567ca6-5d44jzg8t" (UID: "096d1018-1c07-4bce-b449-08c34c8457fb") : secret "agc-cred-876b469dc150" not found Oct 31 04:45:39.197410 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:45:39.196906 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod846eb567_29ee_430d_9f65_007a72737fcd.slice/crio-034e67971ad08b346dc0597f40183a7178cf6fb71d27281f20e7efd6ffcb215b WatchSource:0}: Error finding container 034e67971ad08b346dc0597f40183a7178cf6fb71d27281f20e7efd6ffcb215b: Status 404 returned error can't find the container with id 034e67971ad08b346dc0597f40183a7178cf6fb71d27281f20e7efd6ffcb215b Oct 31 04:45:39.197956 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:39.197919 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b181387a-z7q2c"] Oct 31 04:45:39.510210 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:39.510165 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b181387a-z7q2c" event={"ID":"846eb567-29ee-430d-9f65-007a72737fcd","Type":"ContainerStarted","Data":"034e67971ad08b346dc0597f40183a7178cf6fb71d27281f20e7efd6ffcb215b"} Oct 31 04:45:39.512387 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:39.512353 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bmtf9q" event={"ID":"f10d9686-b9c2-40ad-97df-b358573a707c","Type":"ContainerStarted","Data":"aa95430dafae41d63c5a42250992db2de97219ebb33957f5850abed01188546e"} Oct 31 04:45:39.512586 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:39.512568 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bmtf9q" Oct 31 04:45:39.512663 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:39.512603 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bmtf9q" event={"ID":"f10d9686-b9c2-40ad-97df-b358573a707c","Type":"ContainerStarted","Data":"254adc3e04f61cd38c8a4b8ae9b46f9300651cf7cb8be9cce7bfac5ba417ce77"} Oct 31 04:45:39.557692 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:39.557626 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bmtf9q" podStartSLOduration=1.557605297 podStartE2EDuration="1.557605297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:45:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:45:39.553537884 +0000 UTC m=+2494.309875924" watchObservedRunningTime="2026-10-31 04:45:39.557605297 +0000 UTC m=+2494.313943339" Oct 31 04:45:40.034031 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:40.033988 5019 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-567ca6/agc-cred-876b469dc150: secret "agc-cred-876b469dc150" not found Oct 31 04:45:40.034031 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:40.034024 5019 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-5d44jzg8t: secret "agc-cred-876b469dc150" not found Oct 31 04:45:40.034286 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:40.034082 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/096d1018-1c07-4bce-b449-08c34c8457fb-provider-credentials podName:096d1018-1c07-4bce-b449-08c34c8457fb nodeName:}" failed. No retries permitted until 2026-10-31 04:45:42.034067437 +0000 UTC m=+2496.790405455 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/096d1018-1c07-4bce-b449-08c34c8457fb-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-567ca6-5d44jzg8t" (UID: "096d1018-1c07-4bce-b449-08c34c8457fb") : secret "agc-cred-876b469dc150" not found Oct 31 04:45:40.526587 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:40.526550 5019 generic.go:356] "Generic (PLEG): container finished" podID="846eb567-29ee-430d-9f65-007a72737fcd" containerID="fb077ff201abe5a21e7859f651d82292b0fa227358abb785efec341b4cd31152" exitCode=0 Oct 31 04:45:40.527074 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:40.526629 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b181387a-z7q2c" event={"ID":"846eb567-29ee-430d-9f65-007a72737fcd","Type":"ContainerDied","Data":"fb077ff201abe5a21e7859f651d82292b0fa227358abb785efec341b4cd31152"} Oct 31 04:45:41.136894 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:41.136860 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7qh6vn" Oct 31 04:45:41.230843 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:41.230807 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fkmdlx"] Oct 31 04:45:41.745775 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:41.745744 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b181387a-z7q2c" Oct 31 04:45:41.866839 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:41.866739 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/846eb567-29ee-430d-9f65-007a72737fcd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/846eb567-29ee-430d-9f65-007a72737fcd-maas-api-xks-ca\") pod \"846eb567-29ee-430d-9f65-007a72737fcd\" (UID: \"846eb567-29ee-430d-9f65-007a72737fcd\") " Oct 31 04:45:41.867035 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:41.866922 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/846eb567-29ee-430d-9f65-007a72737fcd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/846eb567-29ee-430d-9f65-007a72737fcd-maas-api-openshift-ca\") pod \"846eb567-29ee-430d-9f65-007a72737fcd\" (UID: \"846eb567-29ee-430d-9f65-007a72737fcd\") " Oct 31 04:45:41.867113 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:41.867032 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/846eb567-29ee-430d-9f65-007a72737fcd-kube-api-access-stpg2\" (UniqueName: \"kubernetes.io/projected/846eb567-29ee-430d-9f65-007a72737fcd-kube-api-access-stpg2\") pod \"846eb567-29ee-430d-9f65-007a72737fcd\" (UID: \"846eb567-29ee-430d-9f65-007a72737fcd\") " Oct 31 04:45:41.867305 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:41.867280 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/846eb567-29ee-430d-9f65-007a72737fcd-maas-api-openshift-ca" pod "846eb567-29ee-430d-9f65-007a72737fcd" (UID: "846eb567-29ee-430d-9f65-007a72737fcd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:45:41.867771 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:41.867751 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/846eb567-29ee-430d-9f65-007a72737fcd-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:41.869161 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:41.869134 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/846eb567-29ee-430d-9f65-007a72737fcd-maas-api-xks-ca" pod "846eb567-29ee-430d-9f65-007a72737fcd" (UID: "846eb567-29ee-430d-9f65-007a72737fcd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:45:41.869262 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:41.869173 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/846eb567-29ee-430d-9f65-007a72737fcd-kube-api-access-stpg2" pod "846eb567-29ee-430d-9f65-007a72737fcd" (UID: "846eb567-29ee-430d-9f65-007a72737fcd"). InnerVolumeSpecName "kube-api-access-stpg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:45:41.969524 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:41.969484 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stpg2\" (UniqueName: \"kubernetes.io/projected/846eb567-29ee-430d-9f65-007a72737fcd-kube-api-access-stpg2\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:41.969524 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:41.969521 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/846eb567-29ee-430d-9f65-007a72737fcd-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:42.071778 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:42.071732 5019 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-567ca6/agc-cred-876b469dc150: secret "agc-cred-876b469dc150" not found Oct 31 04:45:42.071778 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:42.071770 5019 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-5d44jzg8t: secret "agc-cred-876b469dc150" not found Oct 31 04:45:42.072020 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:42.071836 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/096d1018-1c07-4bce-b449-08c34c8457fb-provider-credentials podName:096d1018-1c07-4bce-b449-08c34c8457fb nodeName:}" failed. No retries permitted until 2026-10-31 04:45:46.071813419 +0000 UTC m=+2500.828151447 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/096d1018-1c07-4bce-b449-08c34c8457fb-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-567ca6-5d44jzg8t" (UID: "096d1018-1c07-4bce-b449-08c34c8457fb") : secret "agc-cred-876b469dc150" not found Oct 31 04:45:42.547443 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:42.547399 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b181387a-z7q2c" event={"ID":"846eb567-29ee-430d-9f65-007a72737fcd","Type":"ContainerDied","Data":"034e67971ad08b346dc0597f40183a7178cf6fb71d27281f20e7efd6ffcb215b"} Oct 31 04:45:42.547649 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:42.547466 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="034e67971ad08b346dc0597f40183a7178cf6fb71d27281f20e7efd6ffcb215b" Oct 31 04:45:42.547649 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:42.547428 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b181387a-z7q2c" Oct 31 04:45:46.174426 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:46.174378 5019 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-567ca6/agc-cred-876b469dc150: secret "agc-cred-876b469dc150" not found Oct 31 04:45:46.175164 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:46.175141 5019 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-5d44jzg8t: secret "agc-cred-876b469dc150" not found Oct 31 04:45:46.175376 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:46.175360 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/096d1018-1c07-4bce-b449-08c34c8457fb-provider-credentials podName:096d1018-1c07-4bce-b449-08c34c8457fb nodeName:}" failed. No retries permitted until 2026-10-31 04:45:54.175315679 +0000 UTC m=+2508.931653717 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/096d1018-1c07-4bce-b449-08c34c8457fb-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-567ca6-5d44jzg8t" (UID: "096d1018-1c07-4bce-b449-08c34c8457fb") : secret "agc-cred-876b469dc150" not found Oct 31 04:45:46.236490 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:46.236407 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fkmdlx" podUID="5e889be1-d856-4ffc-86b9-f9fa9da208c0" containerName="payload-processing" containerID="cri-o://c59dbcf2549046b477ac5bb3edb4db72d3fd7125d98a9ec219defece7236b46f" gracePeriod=25 Oct 31 04:45:46.599470 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:46.599431 5019 generic.go:356] "Generic (PLEG): container finished" podID="5e889be1-d856-4ffc-86b9-f9fa9da208c0" containerID="c59dbcf2549046b477ac5bb3edb4db72d3fd7125d98a9ec219defece7236b46f" exitCode=0 Oct 31 04:45:46.599662 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:46.599500 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fkmdlx" event={"ID":"5e889be1-d856-4ffc-86b9-f9fa9da208c0","Type":"ContainerDied","Data":"c59dbcf2549046b477ac5bb3edb4db72d3fd7125d98a9ec219defece7236b46f"} Oct 31 04:45:46.762324 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:46.762294 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fkmdlx" Oct 31 04:45:46.851380 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:46.851296 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e889be1-d856-4ffc-86b9-f9fa9da208c0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e889be1-d856-4ffc-86b9-f9fa9da208c0-plugins-config-volume\") pod \"5e889be1-d856-4ffc-86b9-f9fa9da208c0\" (UID: \"5e889be1-d856-4ffc-86b9-f9fa9da208c0\") " Oct 31 04:45:46.851600 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:46.851472 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e889be1-d856-4ffc-86b9-f9fa9da208c0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5e889be1-d856-4ffc-86b9-f9fa9da208c0-provider-credentials\") pod \"5e889be1-d856-4ffc-86b9-f9fa9da208c0\" (UID: \"5e889be1-d856-4ffc-86b9-f9fa9da208c0\") " Oct 31 04:45:46.851600 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:46.851532 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e889be1-d856-4ffc-86b9-f9fa9da208c0-kube-api-access-n2hrg\" (UniqueName: \"kubernetes.io/projected/5e889be1-d856-4ffc-86b9-f9fa9da208c0-kube-api-access-n2hrg\") pod \"5e889be1-d856-4ffc-86b9-f9fa9da208c0\" (UID: \"5e889be1-d856-4ffc-86b9-f9fa9da208c0\") " Oct 31 04:45:46.851777 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:46.851739 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e889be1-d856-4ffc-86b9-f9fa9da208c0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5e889be1-d856-4ffc-86b9-f9fa9da208c0-routing-overlay-volume\") pod \"5e889be1-d856-4ffc-86b9-f9fa9da208c0\" (UID: \"5e889be1-d856-4ffc-86b9-f9fa9da208c0\") " Oct 31 04:45:46.851942 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:46.851912 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e889be1-d856-4ffc-86b9-f9fa9da208c0-plugins-config-volume" pod "5e889be1-d856-4ffc-86b9-f9fa9da208c0" (UID: "5e889be1-d856-4ffc-86b9-f9fa9da208c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:45:46.852494 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:46.852464 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e889be1-d856-4ffc-86b9-f9fa9da208c0-routing-overlay-volume" pod "5e889be1-d856-4ffc-86b9-f9fa9da208c0" (UID: "5e889be1-d856-4ffc-86b9-f9fa9da208c0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:45:46.852677 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:46.852656 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e889be1-d856-4ffc-86b9-f9fa9da208c0-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:46.852677 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:46.852681 5019 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5e889be1-d856-4ffc-86b9-f9fa9da208c0-routing-overlay-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:46.854918 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:46.854892 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e889be1-d856-4ffc-86b9-f9fa9da208c0-provider-credentials" pod "5e889be1-d856-4ffc-86b9-f9fa9da208c0" (UID: "5e889be1-d856-4ffc-86b9-f9fa9da208c0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:45:46.855220 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:46.855192 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e889be1-d856-4ffc-86b9-f9fa9da208c0-kube-api-access-n2hrg" pod "5e889be1-d856-4ffc-86b9-f9fa9da208c0" (UID: "5e889be1-d856-4ffc-86b9-f9fa9da208c0"). InnerVolumeSpecName "kube-api-access-n2hrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:45:46.954682 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:46.954640 5019 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5e889be1-d856-4ffc-86b9-f9fa9da208c0-provider-credentials\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:46.954682 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:46.954671 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2hrg\" (UniqueName: \"kubernetes.io/projected/5e889be1-d856-4ffc-86b9-f9fa9da208c0-kube-api-access-n2hrg\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:47.610527 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:47.610495 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fkmdlx" Oct 31 04:45:47.610985 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:47.610502 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fkmdlx" event={"ID":"5e889be1-d856-4ffc-86b9-f9fa9da208c0","Type":"ContainerDied","Data":"32ae6cfc5574d427913d40cb1e62d94c5b92c71e5f5f01b51ce257bcd5065e67"} Oct 31 04:45:47.610985 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:47.610634 5019 scope.go:122] "RemoveContainer" containerID="c59dbcf2549046b477ac5bb3edb4db72d3fd7125d98a9ec219defece7236b46f" Oct 31 04:45:47.666279 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:47.666247 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fkmdlx"] Oct 31 04:45:47.678533 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:47.678461 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fkmdlx"] Oct 31 04:45:49.413580 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:49.413541 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e889be1-d856-4ffc-86b9-f9fa9da208c0" path="/var/lib/kubelet/pods/5e889be1-d856-4ffc-86b9-f9fa9da208c0/volumes" Oct 31 04:45:50.012435 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:50.012393 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bmtf9q"] Oct 31 04:45:50.015599 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:50.015573 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bmtf9q" Oct 31 04:45:50.209433 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:50.209389 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f4d9e2c-plnkf"] Oct 31 04:45:50.210516 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:50.210490 5019 cpu_manager.go:414] "containerMap: removing container" podUID="846eb567-29ee-430d-9f65-007a72737fcd" containerName="revoke-keys" Oct 31 04:45:50.210516 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:50.210517 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="846eb567-29ee-430d-9f65-007a72737fcd" containerName="revoke-keys" Oct 31 04:45:50.210646 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:50.210534 5019 cpu_manager.go:414] "containerMap: removing container" podUID="5e889be1-d856-4ffc-86b9-f9fa9da208c0" containerName="payload-processing" Oct 31 04:45:50.210646 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:50.210539 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e889be1-d856-4ffc-86b9-f9fa9da208c0" containerName="payload-processing" Oct 31 04:45:50.210723 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:50.210701 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="846eb567-29ee-430d-9f65-007a72737fcd" containerName="revoke-keys" Oct 31 04:45:50.210723 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:50.210711 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e889be1-d856-4ffc-86b9-f9fa9da208c0" containerName="payload-processing" Oct 31 04:45:50.216015 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:50.215987 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f4d9e2c-plnkf" Oct 31 04:45:50.263936 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:50.263840 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:45:50.300176 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:50.264714 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:45:50.300441 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:50.264971 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:45:50.300524 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:50.265644 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:45:50.357496 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:50.357440 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f4d9e2c-plnkf"] Oct 31 04:45:50.400360 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:50.383533 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed0a8acb-f490-41fb-83e6-d9cd440e269c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f4d9e2c-plnkf\" (UID: \"ed0a8acb-f490-41fb-83e6-d9cd440e269c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f4d9e2c-plnkf" Oct 31 04:45:50.412371 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:50.404609 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9b4c\" (UniqueName: \"kubernetes.io/projected/ed0a8acb-f490-41fb-83e6-d9cd440e269c-kube-api-access-s9b4c\") pod \"maas-api-revoke-sub-3f4d9e2c-plnkf\" (UID: \"ed0a8acb-f490-41fb-83e6-d9cd440e269c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f4d9e2c-plnkf" Oct 31 04:45:50.426354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:50.420646 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed0a8acb-f490-41fb-83e6-d9cd440e269c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f4d9e2c-plnkf\" (UID: \"ed0a8acb-f490-41fb-83e6-d9cd440e269c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f4d9e2c-plnkf" Oct 31 04:45:50.536569 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:50.536466 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9b4c\" (UniqueName: \"kubernetes.io/projected/ed0a8acb-f490-41fb-83e6-d9cd440e269c-kube-api-access-s9b4c\") pod \"maas-api-revoke-sub-3f4d9e2c-plnkf\" (UID: \"ed0a8acb-f490-41fb-83e6-d9cd440e269c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f4d9e2c-plnkf" Oct 31 04:45:50.536755 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:50.536581 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed0a8acb-f490-41fb-83e6-d9cd440e269c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f4d9e2c-plnkf\" (UID: \"ed0a8acb-f490-41fb-83e6-d9cd440e269c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f4d9e2c-plnkf" Oct 31 04:45:50.536971 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:50.536950 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed0a8acb-f490-41fb-83e6-d9cd440e269c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f4d9e2c-plnkf\" (UID: \"ed0a8acb-f490-41fb-83e6-d9cd440e269c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f4d9e2c-plnkf" Oct 31 04:45:50.539319 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:50.539282 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed0a8acb-f490-41fb-83e6-d9cd440e269c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f4d9e2c-plnkf\" (UID: \"ed0a8acb-f490-41fb-83e6-d9cd440e269c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f4d9e2c-plnkf" Oct 31 04:45:50.543545 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:50.543440 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed0a8acb-f490-41fb-83e6-d9cd440e269c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f4d9e2c-plnkf\" (UID: \"ed0a8acb-f490-41fb-83e6-d9cd440e269c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f4d9e2c-plnkf" Oct 31 04:45:50.562044 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:50.562011 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9b4c\" (UniqueName: \"kubernetes.io/projected/ed0a8acb-f490-41fb-83e6-d9cd440e269c-kube-api-access-s9b4c\") pod \"maas-api-revoke-sub-3f4d9e2c-plnkf\" (UID: \"ed0a8acb-f490-41fb-83e6-d9cd440e269c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f4d9e2c-plnkf" Oct 31 04:45:50.597490 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:50.597449 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f4d9e2c-plnkf" Oct 31 04:45:50.932184 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:45:50.932144 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded0a8acb_f490_41fb_83e6_d9cd440e269c.slice/crio-54ca9661117073542c1f41037bd3905b758e62a43973fe6e34bc0715fc2a5843 WatchSource:0}: Error finding container 54ca9661117073542c1f41037bd3905b758e62a43973fe6e34bc0715fc2a5843: Status 404 returned error can't find the container with id 54ca9661117073542c1f41037bd3905b758e62a43973fe6e34bc0715fc2a5843 Oct 31 04:45:50.933225 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:50.933106 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f4d9e2c-plnkf"] Oct 31 04:45:51.655199 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:51.655161 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f4d9e2c-plnkf" event={"ID":"ed0a8acb-f490-41fb-83e6-d9cd440e269c","Type":"ContainerStarted","Data":"b2ab5c53d147b351b7d23f60d10b7c7c61421f099cc286faac3b2a07b3c28743"} Oct 31 04:45:51.655712 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:51.655217 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f4d9e2c-plnkf" event={"ID":"ed0a8acb-f490-41fb-83e6-d9cd440e269c","Type":"ContainerStarted","Data":"54ca9661117073542c1f41037bd3905b758e62a43973fe6e34bc0715fc2a5843"} Oct 31 04:45:51.699716 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:51.699663 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f4d9e2c-plnkf" podStartSLOduration=1.699649216 podStartE2EDuration="1.699649216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:45:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:45:51.694655938 +0000 UTC m=+2506.450993980" watchObservedRunningTime="2026-10-31 04:45:51.699649216 +0000 UTC m=+2506.455987283" Oct 31 04:45:52.392257 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:52.392150 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:45:52.667864 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:52.667764 5019 generic.go:356] "Generic (PLEG): container finished" podID="ed0a8acb-f490-41fb-83e6-d9cd440e269c" containerID="b2ab5c53d147b351b7d23f60d10b7c7c61421f099cc286faac3b2a07b3c28743" exitCode=0 Oct 31 04:45:52.667864 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:52.667848 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f4d9e2c-plnkf" event={"ID":"ed0a8acb-f490-41fb-83e6-d9cd440e269c","Type":"ContainerDied","Data":"b2ab5c53d147b351b7d23f60d10b7c7c61421f099cc286faac3b2a07b3c28743"} Oct 31 04:45:53.886665 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:53.886628 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f4d9e2c-plnkf" Oct 31 04:45:53.993619 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:53.993575 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed0a8acb-f490-41fb-83e6-d9cd440e269c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed0a8acb-f490-41fb-83e6-d9cd440e269c-maas-api-openshift-ca\") pod \"ed0a8acb-f490-41fb-83e6-d9cd440e269c\" (UID: \"ed0a8acb-f490-41fb-83e6-d9cd440e269c\") " Oct 31 04:45:53.993845 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:53.993803 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed0a8acb-f490-41fb-83e6-d9cd440e269c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed0a8acb-f490-41fb-83e6-d9cd440e269c-maas-api-xks-ca\") pod \"ed0a8acb-f490-41fb-83e6-d9cd440e269c\" (UID: \"ed0a8acb-f490-41fb-83e6-d9cd440e269c\") " Oct 31 04:45:53.993970 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:53.993935 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed0a8acb-f490-41fb-83e6-d9cd440e269c-kube-api-access-s9b4c\" (UniqueName: \"kubernetes.io/projected/ed0a8acb-f490-41fb-83e6-d9cd440e269c-kube-api-access-s9b4c\") pod \"ed0a8acb-f490-41fb-83e6-d9cd440e269c\" (UID: \"ed0a8acb-f490-41fb-83e6-d9cd440e269c\") " Oct 31 04:45:53.994039 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:53.994001 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed0a8acb-f490-41fb-83e6-d9cd440e269c-maas-api-openshift-ca" pod "ed0a8acb-f490-41fb-83e6-d9cd440e269c" (UID: "ed0a8acb-f490-41fb-83e6-d9cd440e269c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:45:53.994866 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:53.994809 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed0a8acb-f490-41fb-83e6-d9cd440e269c-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:53.996174 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:53.996144 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed0a8acb-f490-41fb-83e6-d9cd440e269c-maas-api-xks-ca" pod "ed0a8acb-f490-41fb-83e6-d9cd440e269c" (UID: "ed0a8acb-f490-41fb-83e6-d9cd440e269c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:45:53.996688 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:53.996666 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed0a8acb-f490-41fb-83e6-d9cd440e269c-kube-api-access-s9b4c" pod "ed0a8acb-f490-41fb-83e6-d9cd440e269c" (UID: "ed0a8acb-f490-41fb-83e6-d9cd440e269c"). InnerVolumeSpecName "kube-api-access-s9b4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:45:54.096105 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:54.096064 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed0a8acb-f490-41fb-83e6-d9cd440e269c-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:54.096105 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:54.096102 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9b4c\" (UniqueName: \"kubernetes.io/projected/ed0a8acb-f490-41fb-83e6-d9cd440e269c-kube-api-access-s9b4c\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:54.198378 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:54.198304 5019 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-567ca6/agc-cred-4d916acdf91b: secret "agc-cred-4d916acdf91b" not found Oct 31 04:45:54.198378 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:54.198366 5019 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-567ca6/agc-cred-876b469dc150: secret "agc-cred-876b469dc150" not found Oct 31 04:45:54.198378 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:54.198377 5019 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-5d44jzg8t: [secret "agc-cred-4d916acdf91b" not found, secret "agc-cred-876b469dc150" not found] Oct 31 04:45:54.198701 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:54.198456 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/096d1018-1c07-4bce-b449-08c34c8457fb-provider-credentials podName:096d1018-1c07-4bce-b449-08c34c8457fb nodeName:}" failed. No retries permitted until 2026-10-31 04:46:10.198437982 +0000 UTC m=+2524.954776013 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/096d1018-1c07-4bce-b449-08c34c8457fb-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-567ca6-5d44jzg8t" (UID: "096d1018-1c07-4bce-b449-08c34c8457fb") : [secret "agc-cred-4d916acdf91b" not found, secret "agc-cred-876b469dc150" not found] Oct 31 04:45:54.690114 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:54.690082 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f4d9e2c-plnkf" Oct 31 04:45:54.690297 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:54.690081 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f4d9e2c-plnkf" event={"ID":"ed0a8acb-f490-41fb-83e6-d9cd440e269c","Type":"ContainerDied","Data":"54ca9661117073542c1f41037bd3905b758e62a43973fe6e34bc0715fc2a5843"} Oct 31 04:45:54.690297 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:54.690230 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54ca9661117073542c1f41037bd3905b758e62a43973fe6e34bc0715fc2a5843" Oct 31 04:45:55.013989 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:55.013915 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bmtf9q" podUID="f10d9686-b9c2-40ad-97df-b358573a707c" containerName="payload-processing" containerID="cri-o://aa95430dafae41d63c5a42250992db2de97219ebb33957f5850abed01188546e" gracePeriod=25 Oct 31 04:45:55.355956 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:55.355923 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bmtf9q" Oct 31 04:45:55.529393 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:55.529325 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f10d9686-b9c2-40ad-97df-b358573a707c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f10d9686-b9c2-40ad-97df-b358573a707c-routing-overlay-volume\") pod \"f10d9686-b9c2-40ad-97df-b358573a707c\" (UID: \"f10d9686-b9c2-40ad-97df-b358573a707c\") " Oct 31 04:45:55.529616 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:55.529463 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f10d9686-b9c2-40ad-97df-b358573a707c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f10d9686-b9c2-40ad-97df-b358573a707c-plugins-config-volume\") pod \"f10d9686-b9c2-40ad-97df-b358573a707c\" (UID: \"f10d9686-b9c2-40ad-97df-b358573a707c\") " Oct 31 04:45:55.529700 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:55.529630 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f10d9686-b9c2-40ad-97df-b358573a707c-routing-overlay-volume" pod "f10d9686-b9c2-40ad-97df-b358573a707c" (UID: "f10d9686-b9c2-40ad-97df-b358573a707c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:45:55.529700 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:55.529644 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f10d9686-b9c2-40ad-97df-b358573a707c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f10d9686-b9c2-40ad-97df-b358573a707c-provider-credentials\") pod \"f10d9686-b9c2-40ad-97df-b358573a707c\" (UID: \"f10d9686-b9c2-40ad-97df-b358573a707c\") " Oct 31 04:45:55.529826 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:55.529787 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f10d9686-b9c2-40ad-97df-b358573a707c-kube-api-access-sc6hv\" (UniqueName: \"kubernetes.io/projected/f10d9686-b9c2-40ad-97df-b358573a707c-kube-api-access-sc6hv\") pod \"f10d9686-b9c2-40ad-97df-b358573a707c\" (UID: \"f10d9686-b9c2-40ad-97df-b358573a707c\") " Oct 31 04:45:55.529929 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:55.529902 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f10d9686-b9c2-40ad-97df-b358573a707c-plugins-config-volume" pod "f10d9686-b9c2-40ad-97df-b358573a707c" (UID: "f10d9686-b9c2-40ad-97df-b358573a707c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:45:55.530523 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:55.530493 5019 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f10d9686-b9c2-40ad-97df-b358573a707c-routing-overlay-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:55.530523 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:55.530517 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f10d9686-b9c2-40ad-97df-b358573a707c-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:55.532577 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:55.532548 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f10d9686-b9c2-40ad-97df-b358573a707c-provider-credentials" pod "f10d9686-b9c2-40ad-97df-b358573a707c" (UID: "f10d9686-b9c2-40ad-97df-b358573a707c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:45:55.532921 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:55.532887 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f10d9686-b9c2-40ad-97df-b358573a707c-kube-api-access-sc6hv" pod "f10d9686-b9c2-40ad-97df-b358573a707c" (UID: "f10d9686-b9c2-40ad-97df-b358573a707c"). InnerVolumeSpecName "kube-api-access-sc6hv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:45:55.632737 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:55.632640 5019 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f10d9686-b9c2-40ad-97df-b358573a707c-provider-credentials\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:55.632737 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:55.632683 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc6hv\" (UniqueName: \"kubernetes.io/projected/f10d9686-b9c2-40ad-97df-b358573a707c-kube-api-access-sc6hv\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:45:55.706177 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:55.706134 5019 generic.go:356] "Generic (PLEG): container finished" podID="f10d9686-b9c2-40ad-97df-b358573a707c" containerID="aa95430dafae41d63c5a42250992db2de97219ebb33957f5850abed01188546e" exitCode=0 Oct 31 04:45:55.706400 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:55.706216 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bmtf9q" event={"ID":"f10d9686-b9c2-40ad-97df-b358573a707c","Type":"ContainerDied","Data":"aa95430dafae41d63c5a42250992db2de97219ebb33957f5850abed01188546e"} Oct 31 04:45:55.706400 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:55.706323 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bmtf9q" event={"ID":"f10d9686-b9c2-40ad-97df-b358573a707c","Type":"ContainerDied","Data":"254adc3e04f61cd38c8a4b8ae9b46f9300651cf7cb8be9cce7bfac5ba417ce77"} Oct 31 04:45:55.706400 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:55.706395 5019 scope.go:122] "RemoveContainer" containerID="aa95430dafae41d63c5a42250992db2de97219ebb33957f5850abed01188546e" Oct 31 04:45:55.706774 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:55.706745 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bmtf9q" Oct 31 04:45:55.722667 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:55.722637 5019 scope.go:122] "RemoveContainer" containerID="aa95430dafae41d63c5a42250992db2de97219ebb33957f5850abed01188546e" Oct 31 04:45:55.722986 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:45:55.722963 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa95430dafae41d63c5a42250992db2de97219ebb33957f5850abed01188546e\": container with ID starting with aa95430dafae41d63c5a42250992db2de97219ebb33957f5850abed01188546e not found: ID does not exist" containerID="aa95430dafae41d63c5a42250992db2de97219ebb33957f5850abed01188546e" Oct 31 04:45:55.723048 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:55.722998 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa95430dafae41d63c5a42250992db2de97219ebb33957f5850abed01188546e"} err="failed to get container status \"aa95430dafae41d63c5a42250992db2de97219ebb33957f5850abed01188546e\": rpc error: code = NotFound desc = could not find container \"aa95430dafae41d63c5a42250992db2de97219ebb33957f5850abed01188546e\": container with ID starting with aa95430dafae41d63c5a42250992db2de97219ebb33957f5850abed01188546e not found: ID does not exist" Oct 31 04:45:55.760691 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:55.760618 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bmtf9q"] Oct 31 04:45:55.765406 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:55.765374 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-866bmtf9q"] Oct 31 04:45:57.412117 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:45:57.412069 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f10d9686-b9c2-40ad-97df-b358573a707c" path="/var/lib/kubelet/pods/f10d9686-b9c2-40ad-97df-b358573a707c/volumes" Oct 31 04:46:01.004245 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:01.004201 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-648dnwfg6"] Oct 31 04:46:01.012110 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:01.006030 5019 cpu_manager.go:414] "containerMap: removing container" podUID="f10d9686-b9c2-40ad-97df-b358573a707c" containerName="payload-processing" Oct 31 04:46:01.012110 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:01.006060 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f10d9686-b9c2-40ad-97df-b358573a707c" containerName="payload-processing" Oct 31 04:46:01.012110 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:01.006220 5019 cpu_manager.go:414] "containerMap: removing container" podUID="ed0a8acb-f490-41fb-83e6-d9cd440e269c" containerName="revoke-keys" Oct 31 04:46:01.012110 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:01.006230 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed0a8acb-f490-41fb-83e6-d9cd440e269c" containerName="revoke-keys" Oct 31 04:46:01.012110 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:01.006591 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed0a8acb-f490-41fb-83e6-d9cd440e269c" containerName="revoke-keys" Oct 31 04:46:01.012110 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:01.006622 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="f10d9686-b9c2-40ad-97df-b358573a707c" containerName="payload-processing" Oct 31 04:46:01.014863 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:01.014834 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-648dnwfg6" Oct 31 04:46:01.025596 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:01.025561 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-648dnwfg6"] Oct 31 04:46:01.031762 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:01.031701 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-654741\"/\"e2e-cmp-legacy-27b6d5-provider-cred\"" Oct 31 04:46:01.050705 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:01.050670 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-654741\"/\"kube-root-ca.crt\"" Oct 31 04:46:01.051025 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:01.050692 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-654741\"/\"routing-overlay\"" Oct 31 04:46:01.051261 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:01.051240 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-654741\"/\"payload-processing-external-model-e2e--dfe12206-dockercfg-754kc\"" Oct 31 04:46:01.051497 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:01.051479 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-654741\"/\"payload-processing-external-model-plugins-e2e-extmod-a-654741\"" Oct 31 04:46:01.051707 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:01.051691 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-654741\"/\"openshift-service-ca.crt\"" Oct 31 04:46:01.156713 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:01.156670 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5589c835-ca5c-4dd9-938d-3cc2e50f295b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-654741-648dnwfg6\" (UID: \"5589c835-ca5c-4dd9-938d-3cc2e50f295b\") " pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-648dnwfg6" Oct 31 04:46:01.156958 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:01.156756 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5589c835-ca5c-4dd9-938d-3cc2e50f295b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-654741-648dnwfg6\" (UID: \"5589c835-ca5c-4dd9-938d-3cc2e50f295b\") " pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-648dnwfg6" Oct 31 04:46:01.156958 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:01.156885 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzr5f\" (UniqueName: \"kubernetes.io/projected/5589c835-ca5c-4dd9-938d-3cc2e50f295b-kube-api-access-gzr5f\") pod \"payload-processing-external-model-e2e-extmod-a-654741-648dnwfg6\" (UID: \"5589c835-ca5c-4dd9-938d-3cc2e50f295b\") " pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-648dnwfg6" Oct 31 04:46:01.157244 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:01.157220 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5589c835-ca5c-4dd9-938d-3cc2e50f295b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-654741-648dnwfg6\" (UID: \"5589c835-ca5c-4dd9-938d-3cc2e50f295b\") " pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-648dnwfg6" Oct 31 04:46:01.262409 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:01.260195 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5589c835-ca5c-4dd9-938d-3cc2e50f295b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-654741-648dnwfg6\" (UID: \"5589c835-ca5c-4dd9-938d-3cc2e50f295b\") " pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-648dnwfg6" Oct 31 04:46:01.262409 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:01.260297 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5589c835-ca5c-4dd9-938d-3cc2e50f295b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-654741-648dnwfg6\" (UID: \"5589c835-ca5c-4dd9-938d-3cc2e50f295b\") " pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-648dnwfg6" Oct 31 04:46:01.262409 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:01.260357 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzr5f\" (UniqueName: \"kubernetes.io/projected/5589c835-ca5c-4dd9-938d-3cc2e50f295b-kube-api-access-gzr5f\") pod \"payload-processing-external-model-e2e-extmod-a-654741-648dnwfg6\" (UID: \"5589c835-ca5c-4dd9-938d-3cc2e50f295b\") " pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-648dnwfg6" Oct 31 04:46:01.262409 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:01.260646 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5589c835-ca5c-4dd9-938d-3cc2e50f295b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-654741-648dnwfg6\" (UID: \"5589c835-ca5c-4dd9-938d-3cc2e50f295b\") " pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-648dnwfg6" Oct 31 04:46:01.262794 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:01.262745 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5589c835-ca5c-4dd9-938d-3cc2e50f295b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-654741-648dnwfg6\" (UID: \"5589c835-ca5c-4dd9-938d-3cc2e50f295b\") " pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-648dnwfg6" Oct 31 04:46:01.275355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:01.275232 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5589c835-ca5c-4dd9-938d-3cc2e50f295b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-654741-648dnwfg6\" (UID: \"5589c835-ca5c-4dd9-938d-3cc2e50f295b\") " pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-648dnwfg6" Oct 31 04:46:01.300358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:01.291021 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzr5f\" (UniqueName: \"kubernetes.io/projected/5589c835-ca5c-4dd9-938d-3cc2e50f295b-kube-api-access-gzr5f\") pod \"payload-processing-external-model-e2e-extmod-a-654741-648dnwfg6\" (UID: \"5589c835-ca5c-4dd9-938d-3cc2e50f295b\") " pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-648dnwfg6" Oct 31 04:46:01.300358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:01.298037 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5589c835-ca5c-4dd9-938d-3cc2e50f295b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-654741-648dnwfg6\" (UID: \"5589c835-ca5c-4dd9-938d-3cc2e50f295b\") " pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-648dnwfg6" Oct 31 04:46:01.355364 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:01.354659 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-648dnwfg6" Oct 31 04:46:02.227472 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:02.227430 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-648dnwfg6"] Oct 31 04:46:02.715358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:02.714091 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-5d44jzg8t"] Oct 31 04:46:03.295357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:03.295158 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-648dnwfg6" event={"ID":"5589c835-ca5c-4dd9-938d-3cc2e50f295b","Type":"ContainerStarted","Data":"9fc85b04047eb4c68798defede14b453e32a2f5dae1c76c269b9241757ea29ad"} Oct 31 04:46:03.295357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:03.295229 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-648dnwfg6" event={"ID":"5589c835-ca5c-4dd9-938d-3cc2e50f295b","Type":"ContainerStarted","Data":"693fdc49b09089d08d1c5339dc7166be635a6db5b3512a57532b36e2c5af358a"} Oct 31 04:46:03.303476 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:03.296453 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-648dnwfg6" Oct 31 04:46:03.342166 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:03.342110 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-648dnwfg6" podStartSLOduration=3.342095359 podStartE2EDuration="3.342095359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:46:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:46:03.339664403 +0000 UTC m=+2518.096002443" watchObservedRunningTime="2026-10-31 04:46:03.342095359 +0000 UTC m=+2518.098433418" Oct 31 04:46:07.714753 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:07.714700 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-5d44jzg8t" podUID="096d1018-1c07-4bce-b449-08c34c8457fb" containerName="payload-processing" containerID="cri-o://41d8db62847648e1c16a07ba0ae18c67b5614ffbb99804183b5378c6e0c6304f" gracePeriod=25 Oct 31 04:46:08.335797 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:08.335757 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-5d44jzg8t" Oct 31 04:46:08.353789 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:08.353748 5019 generic.go:356] "Generic (PLEG): container finished" podID="096d1018-1c07-4bce-b449-08c34c8457fb" containerID="41d8db62847648e1c16a07ba0ae18c67b5614ffbb99804183b5378c6e0c6304f" exitCode=0 Oct 31 04:46:08.353975 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:08.353791 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-5d44jzg8t" Oct 31 04:46:08.353975 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:08.353876 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-5d44jzg8t" event={"ID":"096d1018-1c07-4bce-b449-08c34c8457fb","Type":"ContainerDied","Data":"41d8db62847648e1c16a07ba0ae18c67b5614ffbb99804183b5378c6e0c6304f"} Oct 31 04:46:08.353975 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:08.353931 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-5d44jzg8t" event={"ID":"096d1018-1c07-4bce-b449-08c34c8457fb","Type":"ContainerDied","Data":"204cf78b7f01406ff21ffb223d788bfa437752d1ad04e05d3850e036dd177004"} Oct 31 04:46:08.353975 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:08.353947 5019 scope.go:122] "RemoveContainer" containerID="41d8db62847648e1c16a07ba0ae18c67b5614ffbb99804183b5378c6e0c6304f" Oct 31 04:46:08.368987 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:08.368958 5019 scope.go:122] "RemoveContainer" containerID="41d8db62847648e1c16a07ba0ae18c67b5614ffbb99804183b5378c6e0c6304f" Oct 31 04:46:08.369309 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:46:08.369285 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41d8db62847648e1c16a07ba0ae18c67b5614ffbb99804183b5378c6e0c6304f\": container with ID starting with 41d8db62847648e1c16a07ba0ae18c67b5614ffbb99804183b5378c6e0c6304f not found: ID does not exist" containerID="41d8db62847648e1c16a07ba0ae18c67b5614ffbb99804183b5378c6e0c6304f" Oct 31 04:46:08.369388 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:08.369318 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41d8db62847648e1c16a07ba0ae18c67b5614ffbb99804183b5378c6e0c6304f"} err="failed to get container status \"41d8db62847648e1c16a07ba0ae18c67b5614ffbb99804183b5378c6e0c6304f\": rpc error: code = NotFound desc = could not find container \"41d8db62847648e1c16a07ba0ae18c67b5614ffbb99804183b5378c6e0c6304f\": container with ID starting with 41d8db62847648e1c16a07ba0ae18c67b5614ffbb99804183b5378c6e0c6304f not found: ID does not exist" Oct 31 04:46:08.431519 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:08.431489 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/096d1018-1c07-4bce-b449-08c34c8457fb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/096d1018-1c07-4bce-b449-08c34c8457fb-provider-credentials\") pod \"096d1018-1c07-4bce-b449-08c34c8457fb\" (UID: \"096d1018-1c07-4bce-b449-08c34c8457fb\") " Oct 31 04:46:08.431700 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:08.431687 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/096d1018-1c07-4bce-b449-08c34c8457fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/096d1018-1c07-4bce-b449-08c34c8457fb-plugins-config-volume\") pod \"096d1018-1c07-4bce-b449-08c34c8457fb\" (UID: \"096d1018-1c07-4bce-b449-08c34c8457fb\") " Oct 31 04:46:08.431825 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:08.431812 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/096d1018-1c07-4bce-b449-08c34c8457fb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/096d1018-1c07-4bce-b449-08c34c8457fb-routing-overlay-volume\") pod \"096d1018-1c07-4bce-b449-08c34c8457fb\" (UID: \"096d1018-1c07-4bce-b449-08c34c8457fb\") " Oct 31 04:46:08.431954 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:08.431937 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/096d1018-1c07-4bce-b449-08c34c8457fb-kube-api-access-b2nhj\" (UniqueName: \"kubernetes.io/projected/096d1018-1c07-4bce-b449-08c34c8457fb-kube-api-access-b2nhj\") pod \"096d1018-1c07-4bce-b449-08c34c8457fb\" (UID: \"096d1018-1c07-4bce-b449-08c34c8457fb\") " Oct 31 04:46:08.432135 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:08.432105 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/096d1018-1c07-4bce-b449-08c34c8457fb-plugins-config-volume" pod "096d1018-1c07-4bce-b449-08c34c8457fb" (UID: "096d1018-1c07-4bce-b449-08c34c8457fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:46:08.432391 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:08.432194 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/096d1018-1c07-4bce-b449-08c34c8457fb-routing-overlay-volume" pod "096d1018-1c07-4bce-b449-08c34c8457fb" (UID: "096d1018-1c07-4bce-b449-08c34c8457fb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:46:08.432712 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:08.432695 5019 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/096d1018-1c07-4bce-b449-08c34c8457fb-routing-overlay-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:46:08.432797 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:08.432714 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/096d1018-1c07-4bce-b449-08c34c8457fb-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:46:08.434182 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:08.434152 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/096d1018-1c07-4bce-b449-08c34c8457fb-provider-credentials" pod "096d1018-1c07-4bce-b449-08c34c8457fb" (UID: "096d1018-1c07-4bce-b449-08c34c8457fb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:46:08.434458 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:08.434437 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/096d1018-1c07-4bce-b449-08c34c8457fb-kube-api-access-b2nhj" pod "096d1018-1c07-4bce-b449-08c34c8457fb" (UID: "096d1018-1c07-4bce-b449-08c34c8457fb"). InnerVolumeSpecName "kube-api-access-b2nhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:46:08.534032 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:08.533987 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2nhj\" (UniqueName: \"kubernetes.io/projected/096d1018-1c07-4bce-b449-08c34c8457fb-kube-api-access-b2nhj\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:46:08.534032 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:08.534026 5019 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/096d1018-1c07-4bce-b449-08c34c8457fb-provider-credentials\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:46:08.703270 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:08.703233 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-5d44jzg8t"] Oct 31 04:46:08.708020 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:08.707988 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-5d44jzg8t"] Oct 31 04:46:09.415208 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:09.415172 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="096d1018-1c07-4bce-b449-08c34c8457fb" path="/var/lib/kubelet/pods/096d1018-1c07-4bce-b449-08c34c8457fb/volumes" Oct 31 04:46:10.347085 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:10.347050 5019 scope.go:122] "RemoveContainer" containerID="c5b3ca8424957f31359283a62b5fbd2af7f5d909ad44e8f2644b40b990b5569b" Oct 31 04:46:10.360054 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:10.360029 5019 scope.go:122] "RemoveContainer" containerID="0b5f28179129910967c4c516705085acc21af895f10e2e3a175d03a8589b2ca4" Oct 31 04:46:12.203512 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:12.203465 5019 patch_prober.go:29] interesting pod/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:46:12.204168 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:12.203563 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz" podUID="b132bea6-493c-43f7-bd51-29f02a01ae9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:46:13.462750 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:13.462715 5019 generic.go:356] "Generic (PLEG): container finished" podID="b132bea6-493c-43f7-bd51-29f02a01ae9a" containerID="d61e05c21f05c1503bdface0acfc18118fd9973b66cd14a67890c6020f1f59a6" exitCode=0 Oct 31 04:46:13.463384 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:13.462793 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz" event={"ID":"b132bea6-493c-43f7-bd51-29f02a01ae9a","Type":"ContainerDied","Data":"d61e05c21f05c1503bdface0acfc18118fd9973b66cd14a67890c6020f1f59a6"} Oct 31 04:46:13.463482 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:13.463402 5019 scope.go:122] "RemoveContainer" containerID="d61e05c21f05c1503bdface0acfc18118fd9973b66cd14a67890c6020f1f59a6" Oct 31 04:46:13.465432 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:13.465399 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:46:13.465523 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:13.465477 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:46:13.465523 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:13.465510 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:46:14.504354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:14.503625 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz" event={"ID":"b132bea6-493c-43f7-bd51-29f02a01ae9a","Type":"ContainerStarted","Data":"0db71e0ec6717ba59c028b1f566d455a81ff4c2e5ac9854dedfc32a723851199"} Oct 31 04:46:15.258024 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:15.257994 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz" Oct 31 04:46:15.258293 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:15.258195 5019 patch_prober.go:29] interesting pod/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" start-of-body= Oct 31 04:46:15.258433 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:15.258321 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz" podUID="b132bea6-493c-43f7-bd51-29f02a01ae9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" Oct 31 04:46:15.328881 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:15.328845 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-648dnwfg6" Oct 31 04:46:16.258212 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:16.258170 5019 patch_prober.go:29] interesting pod/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" start-of-body= Oct 31 04:46:16.258687 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:16.258231 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz" podUID="b132bea6-493c-43f7-bd51-29f02a01ae9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" Oct 31 04:46:17.263322 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:17.263263 5019 patch_prober.go:29] interesting pod/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" start-of-body= Oct 31 04:46:17.263926 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:17.263347 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz" podUID="b132bea6-493c-43f7-bd51-29f02a01ae9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" Oct 31 04:46:18.258778 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:18.258738 5019 patch_prober.go:29] interesting pod/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" start-of-body= Oct 31 04:46:18.259171 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:18.259131 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz" podUID="b132bea6-493c-43f7-bd51-29f02a01ae9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" Oct 31 04:46:18.487199 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:18.487143 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78992cw88"] Oct 31 04:46:18.489843 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:18.489810 5019 cpu_manager.go:414] "containerMap: removing container" podUID="096d1018-1c07-4bce-b449-08c34c8457fb" containerName="payload-processing" Oct 31 04:46:18.491641 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:18.491612 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="096d1018-1c07-4bce-b449-08c34c8457fb" containerName="payload-processing" Oct 31 04:46:18.492683 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:18.492659 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="096d1018-1c07-4bce-b449-08c34c8457fb" containerName="payload-processing" Oct 31 04:46:18.522736 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:18.522646 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78992cw88" Oct 31 04:46:18.523477 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:18.523445 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78992cw88"] Oct 31 04:46:18.576356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:18.575414 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-401761\"/\"payload-processing-external-model-e2e--201094b0-dockercfg-fx78q\"" Oct 31 04:46:18.576356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:18.575570 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-401761\"/\"routing-overlay\"" Oct 31 04:46:18.576356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:18.575618 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-401761\"/\"agc-cred-512fbb167122\"" Oct 31 04:46:18.576356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:18.575708 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-401761\"/\"kube-root-ca.crt\"" Oct 31 04:46:18.576356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:18.575772 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-401761\"/\"openshift-service-ca.crt\"" Oct 31 04:46:18.576356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:18.575827 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-401761\"/\"payload-processing-external-model-plugins-e2e-extmod-a-401761\"" Oct 31 04:46:18.870688 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:18.870167 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tps5b\" (UniqueName: \"kubernetes.io/projected/3b01dc27-d7fd-4b67-81ec-b4be83260db1-kube-api-access-tps5b\") pod \"payload-processing-external-model-e2e-extmod-a-401761-78992cw88\" (UID: \"3b01dc27-d7fd-4b67-81ec-b4be83260db1\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78992cw88" Oct 31 04:46:18.870688 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:18.870382 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b01dc27-d7fd-4b67-81ec-b4be83260db1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-401761-78992cw88\" (UID: \"3b01dc27-d7fd-4b67-81ec-b4be83260db1\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78992cw88" Oct 31 04:46:18.870688 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:18.870438 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3b01dc27-d7fd-4b67-81ec-b4be83260db1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-401761-78992cw88\" (UID: \"3b01dc27-d7fd-4b67-81ec-b4be83260db1\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78992cw88" Oct 31 04:46:18.870688 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:18.870491 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3b01dc27-d7fd-4b67-81ec-b4be83260db1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-401761-78992cw88\" (UID: \"3b01dc27-d7fd-4b67-81ec-b4be83260db1\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78992cw88" Oct 31 04:46:18.988356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:18.976425 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b01dc27-d7fd-4b67-81ec-b4be83260db1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-401761-78992cw88\" (UID: \"3b01dc27-d7fd-4b67-81ec-b4be83260db1\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78992cw88" Oct 31 04:46:18.988356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:18.976496 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3b01dc27-d7fd-4b67-81ec-b4be83260db1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-401761-78992cw88\" (UID: \"3b01dc27-d7fd-4b67-81ec-b4be83260db1\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78992cw88" Oct 31 04:46:18.988356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:18.976546 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3b01dc27-d7fd-4b67-81ec-b4be83260db1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-401761-78992cw88\" (UID: \"3b01dc27-d7fd-4b67-81ec-b4be83260db1\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78992cw88" Oct 31 04:46:18.988356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:18.976799 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tps5b\" (UniqueName: \"kubernetes.io/projected/3b01dc27-d7fd-4b67-81ec-b4be83260db1-kube-api-access-tps5b\") pod \"payload-processing-external-model-e2e-extmod-a-401761-78992cw88\" (UID: \"3b01dc27-d7fd-4b67-81ec-b4be83260db1\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78992cw88" Oct 31 04:46:18.988356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:18.978185 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b01dc27-d7fd-4b67-81ec-b4be83260db1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-401761-78992cw88\" (UID: \"3b01dc27-d7fd-4b67-81ec-b4be83260db1\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78992cw88" Oct 31 04:46:19.005439 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:18.999505 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3b01dc27-d7fd-4b67-81ec-b4be83260db1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-401761-78992cw88\" (UID: \"3b01dc27-d7fd-4b67-81ec-b4be83260db1\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78992cw88" Oct 31 04:46:19.006514 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:19.006456 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3b01dc27-d7fd-4b67-81ec-b4be83260db1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-401761-78992cw88\" (UID: \"3b01dc27-d7fd-4b67-81ec-b4be83260db1\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78992cw88" Oct 31 04:46:19.008592 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:19.008558 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tps5b\" (UniqueName: \"kubernetes.io/projected/3b01dc27-d7fd-4b67-81ec-b4be83260db1-kube-api-access-tps5b\") pod \"payload-processing-external-model-e2e-extmod-a-401761-78992cw88\" (UID: \"3b01dc27-d7fd-4b67-81ec-b4be83260db1\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78992cw88" Oct 31 04:46:19.147658 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:19.147574 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78992cw88" Oct 31 04:46:19.264242 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:19.263766 5019 patch_prober.go:29] interesting pod/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" start-of-body= Oct 31 04:46:19.264242 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:19.263846 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz" podUID="b132bea6-493c-43f7-bd51-29f02a01ae9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" Oct 31 04:46:20.102114 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:20.101864 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78992cw88"] Oct 31 04:46:20.106892 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:46:20.106858 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b01dc27_d7fd_4b67_81ec_b4be83260db1.slice/crio-be833a9d5b7378a230a7ab48adc8c8f1d753c738c3ae81cd8a6c536551b9ac70 WatchSource:0}: Error finding container be833a9d5b7378a230a7ab48adc8c8f1d753c738c3ae81cd8a6c536551b9ac70: Status 404 returned error can't find the container with id be833a9d5b7378a230a7ab48adc8c8f1d753c738c3ae81cd8a6c536551b9ac70 Oct 31 04:46:20.258409 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:20.258376 5019 patch_prober.go:29] interesting pod/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" start-of-body= Oct 31 04:46:20.258619 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:20.258428 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz" podUID="b132bea6-493c-43f7-bd51-29f02a01ae9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" Oct 31 04:46:21.049479 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:21.049437 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78992cw88" event={"ID":"3b01dc27-d7fd-4b67-81ec-b4be83260db1","Type":"ContainerStarted","Data":"cd51a263ba9c04cd5b9c38c9a4a878253b948f23034fa47975b950f825280186"} Oct 31 04:46:21.049671 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:21.049504 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78992cw88" event={"ID":"3b01dc27-d7fd-4b67-81ec-b4be83260db1","Type":"ContainerStarted","Data":"be833a9d5b7378a230a7ab48adc8c8f1d753c738c3ae81cd8a6c536551b9ac70"} Oct 31 04:46:21.049671 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:21.049532 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78992cw88" Oct 31 04:46:21.095020 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:21.094958 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78992cw88" podStartSLOduration=3.094943692 podStartE2EDuration="3.094943692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:46:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:46:21.090532589 +0000 UTC m=+2535.846870628" watchObservedRunningTime="2026-10-31 04:46:21.094943692 +0000 UTC m=+2535.851281731" Oct 31 04:46:21.155430 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:46:21.155392 5019 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-401761/agc-cred-512fbb167122: secret "agc-cred-512fbb167122" not found Oct 31 04:46:21.155977 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:46:21.155537 5019 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78992cw88: secret "agc-cred-512fbb167122" not found Oct 31 04:46:21.155977 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:46:21.155647 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b01dc27-d7fd-4b67-81ec-b4be83260db1-provider-credentials podName:3b01dc27-d7fd-4b67-81ec-b4be83260db1 nodeName:}" failed. No retries permitted until 2026-10-31 04:46:21.655615356 +0000 UTC m=+2536.411953374 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3b01dc27-d7fd-4b67-81ec-b4be83260db1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-401761-78992cw88" (UID: "3b01dc27-d7fd-4b67-81ec-b4be83260db1") : secret "agc-cred-512fbb167122" not found Oct 31 04:46:21.262766 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:21.262738 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz" Oct 31 04:46:21.263249 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:21.263227 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz" Oct 31 04:46:21.264076 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:21.264056 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz" Oct 31 04:46:21.663412 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:46:21.663378 5019 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-401761/agc-cred-512fbb167122: secret "agc-cred-512fbb167122" not found Oct 31 04:46:21.663412 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:46:21.663408 5019 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78992cw88: secret "agc-cred-512fbb167122" not found Oct 31 04:46:21.663640 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:46:21.663479 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b01dc27-d7fd-4b67-81ec-b4be83260db1-provider-credentials podName:3b01dc27-d7fd-4b67-81ec-b4be83260db1 nodeName:}" failed. No retries permitted until 2026-10-31 04:46:22.663461231 +0000 UTC m=+2537.419799256 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3b01dc27-d7fd-4b67-81ec-b4be83260db1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-401761-78992cw88" (UID: "3b01dc27-d7fd-4b67-81ec-b4be83260db1") : secret "agc-cred-512fbb167122" not found Oct 31 04:46:22.397437 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:46:22.397399 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:46:22.689069 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:46:22.688965 5019 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-401761/agc-cred-512fbb167122: secret "agc-cred-512fbb167122" not found Oct 31 04:46:22.689069 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:46:22.688997 5019 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78992cw88: secret "agc-cred-512fbb167122" not found Oct 31 04:46:22.689378 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:46:22.689083 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b01dc27-d7fd-4b67-81ec-b4be83260db1-provider-credentials podName:3b01dc27-d7fd-4b67-81ec-b4be83260db1 nodeName:}" failed. No retries permitted until 2026-10-31 04:46:24.689060938 +0000 UTC m=+2539.445398956 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3b01dc27-d7fd-4b67-81ec-b4be83260db1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-401761-78992cw88" (UID: "3b01dc27-d7fd-4b67-81ec-b4be83260db1") : secret "agc-cred-512fbb167122" not found Oct 31 04:46:24.730872 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:46:24.730829 5019 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-401761/agc-cred-512fbb167122: secret "agc-cred-512fbb167122" not found Oct 31 04:46:24.730872 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:46:24.730861 5019 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78992cw88: secret "agc-cred-512fbb167122" not found Oct 31 04:46:24.731402 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:46:24.730922 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b01dc27-d7fd-4b67-81ec-b4be83260db1-provider-credentials podName:3b01dc27-d7fd-4b67-81ec-b4be83260db1 nodeName:}" failed. No retries permitted until 2026-10-31 04:46:28.730902445 +0000 UTC m=+2543.487240478 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3b01dc27-d7fd-4b67-81ec-b4be83260db1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-401761-78992cw88" (UID: "3b01dc27-d7fd-4b67-81ec-b4be83260db1") : secret "agc-cred-512fbb167122" not found Oct 31 04:46:28.801109 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:46:28.801062 5019 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-401761/agc-cred-512fbb167122: secret "agc-cred-512fbb167122" not found Oct 31 04:46:28.801109 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:46:28.801099 5019 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78992cw88: secret "agc-cred-512fbb167122" not found Oct 31 04:46:28.801829 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:46:28.801170 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b01dc27-d7fd-4b67-81ec-b4be83260db1-provider-credentials podName:3b01dc27-d7fd-4b67-81ec-b4be83260db1 nodeName:}" failed. No retries permitted until 2026-10-31 04:46:36.801154256 +0000 UTC m=+2551.557492274 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3b01dc27-d7fd-4b67-81ec-b4be83260db1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-401761-78992cw88" (UID: "3b01dc27-d7fd-4b67-81ec-b4be83260db1") : secret "agc-cred-512fbb167122" not found Oct 31 04:46:32.062850 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:32.062815 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78992cw88" Oct 31 04:46:32.890838 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:32.890790 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78992cw88"] Oct 31 04:46:36.864294 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:46:36.864248 5019 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-401761/agc-cred-512fbb167122: secret "agc-cred-512fbb167122" not found Oct 31 04:46:36.864294 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:46:36.864288 5019 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78992cw88: secret "agc-cred-512fbb167122" not found Oct 31 04:46:36.864772 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:46:36.864412 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b01dc27-d7fd-4b67-81ec-b4be83260db1-provider-credentials podName:3b01dc27-d7fd-4b67-81ec-b4be83260db1 nodeName:}" failed. No retries permitted until 2026-10-31 04:46:52.864391326 +0000 UTC m=+2567.620729360 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3b01dc27-d7fd-4b67-81ec-b4be83260db1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-401761-78992cw88" (UID: "3b01dc27-d7fd-4b67-81ec-b4be83260db1") : secret "agc-cred-512fbb167122" not found Oct 31 04:46:37.891463 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:37.891419 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78992cw88" podUID="3b01dc27-d7fd-4b67-81ec-b4be83260db1" containerName="payload-processing" containerID="cri-o://cd51a263ba9c04cd5b9c38c9a4a878253b948f23034fa47975b950f825280186" gracePeriod=25 Oct 31 04:46:38.195278 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:38.195241 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78992cw88" Oct 31 04:46:38.221887 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:38.221852 5019 generic.go:356] "Generic (PLEG): container finished" podID="3b01dc27-d7fd-4b67-81ec-b4be83260db1" containerID="cd51a263ba9c04cd5b9c38c9a4a878253b948f23034fa47975b950f825280186" exitCode=0 Oct 31 04:46:38.222106 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:38.221891 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78992cw88" event={"ID":"3b01dc27-d7fd-4b67-81ec-b4be83260db1","Type":"ContainerDied","Data":"cd51a263ba9c04cd5b9c38c9a4a878253b948f23034fa47975b950f825280186"} Oct 31 04:46:38.222106 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:38.221927 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78992cw88" Oct 31 04:46:38.222106 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:38.221967 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78992cw88" event={"ID":"3b01dc27-d7fd-4b67-81ec-b4be83260db1","Type":"ContainerDied","Data":"be833a9d5b7378a230a7ab48adc8c8f1d753c738c3ae81cd8a6c536551b9ac70"} Oct 31 04:46:38.222106 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:38.222000 5019 scope.go:122] "RemoveContainer" containerID="cd51a263ba9c04cd5b9c38c9a4a878253b948f23034fa47975b950f825280186" Oct 31 04:46:38.235465 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:38.235439 5019 scope.go:122] "RemoveContainer" containerID="cd51a263ba9c04cd5b9c38c9a4a878253b948f23034fa47975b950f825280186" Oct 31 04:46:38.235800 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:46:38.235763 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd51a263ba9c04cd5b9c38c9a4a878253b948f23034fa47975b950f825280186\": container with ID starting with cd51a263ba9c04cd5b9c38c9a4a878253b948f23034fa47975b950f825280186 not found: ID does not exist" containerID="cd51a263ba9c04cd5b9c38c9a4a878253b948f23034fa47975b950f825280186" Oct 31 04:46:38.235883 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:38.235812 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd51a263ba9c04cd5b9c38c9a4a878253b948f23034fa47975b950f825280186"} err="failed to get container status \"cd51a263ba9c04cd5b9c38c9a4a878253b948f23034fa47975b950f825280186\": rpc error: code = NotFound desc = could not find container \"cd51a263ba9c04cd5b9c38c9a4a878253b948f23034fa47975b950f825280186\": container with ID starting with cd51a263ba9c04cd5b9c38c9a4a878253b948f23034fa47975b950f825280186 not found: ID does not exist" Oct 31 04:46:38.295912 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:38.295451 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b01dc27-d7fd-4b67-81ec-b4be83260db1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3b01dc27-d7fd-4b67-81ec-b4be83260db1-provider-credentials\") pod \"3b01dc27-d7fd-4b67-81ec-b4be83260db1\" (UID: \"3b01dc27-d7fd-4b67-81ec-b4be83260db1\") " Oct 31 04:46:38.295912 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:38.295511 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b01dc27-d7fd-4b67-81ec-b4be83260db1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3b01dc27-d7fd-4b67-81ec-b4be83260db1-routing-overlay-volume\") pod \"3b01dc27-d7fd-4b67-81ec-b4be83260db1\" (UID: \"3b01dc27-d7fd-4b67-81ec-b4be83260db1\") " Oct 31 04:46:38.295912 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:38.295773 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b01dc27-d7fd-4b67-81ec-b4be83260db1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b01dc27-d7fd-4b67-81ec-b4be83260db1-plugins-config-volume\") pod \"3b01dc27-d7fd-4b67-81ec-b4be83260db1\" (UID: \"3b01dc27-d7fd-4b67-81ec-b4be83260db1\") " Oct 31 04:46:38.295912 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:38.295842 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b01dc27-d7fd-4b67-81ec-b4be83260db1-routing-overlay-volume" pod "3b01dc27-d7fd-4b67-81ec-b4be83260db1" (UID: "3b01dc27-d7fd-4b67-81ec-b4be83260db1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:46:38.295912 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:38.295886 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b01dc27-d7fd-4b67-81ec-b4be83260db1-kube-api-access-tps5b\" (UniqueName: \"kubernetes.io/projected/3b01dc27-d7fd-4b67-81ec-b4be83260db1-kube-api-access-tps5b\") pod \"3b01dc27-d7fd-4b67-81ec-b4be83260db1\" (UID: \"3b01dc27-d7fd-4b67-81ec-b4be83260db1\") " Oct 31 04:46:38.296482 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:38.296313 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b01dc27-d7fd-4b67-81ec-b4be83260db1-plugins-config-volume" pod "3b01dc27-d7fd-4b67-81ec-b4be83260db1" (UID: "3b01dc27-d7fd-4b67-81ec-b4be83260db1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:46:38.296921 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:38.296892 5019 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3b01dc27-d7fd-4b67-81ec-b4be83260db1-routing-overlay-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:46:38.297071 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:38.296925 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b01dc27-d7fd-4b67-81ec-b4be83260db1-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:46:38.299420 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:38.299392 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b01dc27-d7fd-4b67-81ec-b4be83260db1-kube-api-access-tps5b" pod "3b01dc27-d7fd-4b67-81ec-b4be83260db1" (UID: "3b01dc27-d7fd-4b67-81ec-b4be83260db1"). InnerVolumeSpecName "kube-api-access-tps5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:46:38.299833 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:38.299809 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b01dc27-d7fd-4b67-81ec-b4be83260db1-provider-credentials" pod "3b01dc27-d7fd-4b67-81ec-b4be83260db1" (UID: "3b01dc27-d7fd-4b67-81ec-b4be83260db1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:46:38.398576 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:38.398539 5019 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3b01dc27-d7fd-4b67-81ec-b4be83260db1-provider-credentials\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:46:38.398576 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:38.398573 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tps5b\" (UniqueName: \"kubernetes.io/projected/3b01dc27-d7fd-4b67-81ec-b4be83260db1-kube-api-access-tps5b\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:46:38.573218 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:38.573178 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78992cw88"] Oct 31 04:46:38.580604 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:38.580573 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-78992cw88"] Oct 31 04:46:39.412992 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:39.412943 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b01dc27-d7fd-4b67-81ec-b4be83260db1" path="/var/lib/kubelet/pods/3b01dc27-d7fd-4b67-81ec-b4be83260db1/volumes" Oct 31 04:46:43.727547 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:43.727508 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-5986pclf5"] Oct 31 04:46:43.728441 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:43.728420 5019 cpu_manager.go:414] "containerMap: removing container" podUID="3b01dc27-d7fd-4b67-81ec-b4be83260db1" containerName="payload-processing" Oct 31 04:46:43.728514 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:43.728442 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b01dc27-d7fd-4b67-81ec-b4be83260db1" containerName="payload-processing" Oct 31 04:46:43.728608 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:43.728599 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b01dc27-d7fd-4b67-81ec-b4be83260db1" containerName="payload-processing" Oct 31 04:46:43.733926 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:43.733895 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-5986pclf5" Oct 31 04:46:43.751327 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:43.751280 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-5986pclf5"] Oct 31 04:46:43.757239 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:43.757158 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a04e4e\"/\"agc-cred-59f7f64fabf6\"" Oct 31 04:46:43.757239 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:43.757155 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a04e4e\"/\"kube-root-ca.crt\"" Oct 31 04:46:43.760030 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:43.759990 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a04e4e\"/\"payload-processing-external-model-plugins-e2e-extmod-b-a04e4e\"" Oct 31 04:46:43.763140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:43.763103 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a04e4e\"/\"routing-overlay\"" Oct 31 04:46:43.763317 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:43.763174 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a04e4e\"/\"openshift-service-ca.crt\"" Oct 31 04:46:43.763317 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:43.763220 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a04e4e\"/\"payload-processing-external-model-e2e--ae39f93f-dockercfg-ncvq5\"" Oct 31 04:46:43.854282 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:43.854229 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/838a111f-a040-487f-be53-9bdae5e3b36c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-5986pclf5\" (UID: \"838a111f-a040-487f-be53-9bdae5e3b36c\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-5986pclf5" Oct 31 04:46:43.854493 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:43.854408 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/838a111f-a040-487f-be53-9bdae5e3b36c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-5986pclf5\" (UID: \"838a111f-a040-487f-be53-9bdae5e3b36c\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-5986pclf5" Oct 31 04:46:43.854572 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:43.854551 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m9j9\" (UniqueName: \"kubernetes.io/projected/838a111f-a040-487f-be53-9bdae5e3b36c-kube-api-access-4m9j9\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-5986pclf5\" (UID: \"838a111f-a040-487f-be53-9bdae5e3b36c\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-5986pclf5" Oct 31 04:46:43.854812 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:43.854788 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/838a111f-a040-487f-be53-9bdae5e3b36c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-5986pclf5\" (UID: \"838a111f-a040-487f-be53-9bdae5e3b36c\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-5986pclf5" Oct 31 04:46:43.958356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:43.957866 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/838a111f-a040-487f-be53-9bdae5e3b36c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-5986pclf5\" (UID: \"838a111f-a040-487f-be53-9bdae5e3b36c\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-5986pclf5" Oct 31 04:46:43.958356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:43.957970 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/838a111f-a040-487f-be53-9bdae5e3b36c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-5986pclf5\" (UID: \"838a111f-a040-487f-be53-9bdae5e3b36c\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-5986pclf5" Oct 31 04:46:43.958356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:43.958076 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m9j9\" (UniqueName: \"kubernetes.io/projected/838a111f-a040-487f-be53-9bdae5e3b36c-kube-api-access-4m9j9\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-5986pclf5\" (UID: \"838a111f-a040-487f-be53-9bdae5e3b36c\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-5986pclf5" Oct 31 04:46:43.958356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:43.958262 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/838a111f-a040-487f-be53-9bdae5e3b36c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-5986pclf5\" (UID: \"838a111f-a040-487f-be53-9bdae5e3b36c\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-5986pclf5" Oct 31 04:46:43.981406 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:43.980686 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/838a111f-a040-487f-be53-9bdae5e3b36c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-5986pclf5\" (UID: \"838a111f-a040-487f-be53-9bdae5e3b36c\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-5986pclf5" Oct 31 04:46:43.981406 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:43.981303 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/838a111f-a040-487f-be53-9bdae5e3b36c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-5986pclf5\" (UID: \"838a111f-a040-487f-be53-9bdae5e3b36c\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-5986pclf5" Oct 31 04:46:44.033356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:44.033153 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/838a111f-a040-487f-be53-9bdae5e3b36c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-5986pclf5\" (UID: \"838a111f-a040-487f-be53-9bdae5e3b36c\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-5986pclf5" Oct 31 04:46:44.046358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:44.039802 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m9j9\" (UniqueName: \"kubernetes.io/projected/838a111f-a040-487f-be53-9bdae5e3b36c-kube-api-access-4m9j9\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-5986pclf5\" (UID: \"838a111f-a040-487f-be53-9bdae5e3b36c\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-5986pclf5" Oct 31 04:46:44.052356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:44.050729 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-5986pclf5" Oct 31 04:46:44.479953 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:46:44.479921 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod838a111f_a040_487f_be53_9bdae5e3b36c.slice/crio-0260329fbd4f44a5c99ebce1c0222c3a5b9b02f916be2047cc0c320f39dc7981 WatchSource:0}: Error finding container 0260329fbd4f44a5c99ebce1c0222c3a5b9b02f916be2047cc0c320f39dc7981: Status 404 returned error can't find the container with id 0260329fbd4f44a5c99ebce1c0222c3a5b9b02f916be2047cc0c320f39dc7981 Oct 31 04:46:44.480134 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:44.480105 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-5986pclf5"] Oct 31 04:46:45.300822 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:45.300776 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-5986pclf5" event={"ID":"838a111f-a040-487f-be53-9bdae5e3b36c","Type":"ContainerStarted","Data":"88ab75cff536a14eb4fe8c15faf9e227ccc2c6c47533b95ff8e3c48dfc5b305e"} Oct 31 04:46:45.301376 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:45.300847 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-5986pclf5" event={"ID":"838a111f-a040-487f-be53-9bdae5e3b36c","Type":"ContainerStarted","Data":"0260329fbd4f44a5c99ebce1c0222c3a5b9b02f916be2047cc0c320f39dc7981"} Oct 31 04:46:45.301376 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:45.300879 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-5986pclf5" Oct 31 04:46:45.347054 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:45.346982 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-5986pclf5" podStartSLOduration=2.3469649439999998 podStartE2EDuration="2.346964944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:46:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:46:45.34175615 +0000 UTC m=+2560.098094192" watchObservedRunningTime="2026-10-31 04:46:45.346964944 +0000 UTC m=+2560.103303004" Oct 31 04:46:52.406376 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:46:52.405900 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:46:56.313441 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:46:56.313404 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-5986pclf5" Oct 31 04:47:12.400370 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:12.391664 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af60c299-x6r7g"] Oct 31 04:47:12.403301 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:12.401547 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af60c299-x6r7g" Oct 31 04:47:12.410467 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:12.410433 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af60c299-x6r7g"] Oct 31 04:47:12.416680 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:12.416646 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:47:12.428934 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:12.428897 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:47:12.428934 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:12.428897 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:47:12.429171 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:12.429007 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:47:12.502877 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:12.502825 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/763a4426-74f9-4458-981b-c7df977f3f5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af60c299-x6r7g\" (UID: \"763a4426-74f9-4458-981b-c7df977f3f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af60c299-x6r7g" Oct 31 04:47:12.503107 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:12.502908 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/763a4426-74f9-4458-981b-c7df977f3f5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af60c299-x6r7g\" (UID: \"763a4426-74f9-4458-981b-c7df977f3f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af60c299-x6r7g" Oct 31 04:47:12.503211 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:12.503187 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghd8z\" (UniqueName: \"kubernetes.io/projected/763a4426-74f9-4458-981b-c7df977f3f5d-kube-api-access-ghd8z\") pod \"maas-api-revoke-sub-af60c299-x6r7g\" (UID: \"763a4426-74f9-4458-981b-c7df977f3f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af60c299-x6r7g" Oct 31 04:47:12.605116 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:12.605062 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghd8z\" (UniqueName: \"kubernetes.io/projected/763a4426-74f9-4458-981b-c7df977f3f5d-kube-api-access-ghd8z\") pod \"maas-api-revoke-sub-af60c299-x6r7g\" (UID: \"763a4426-74f9-4458-981b-c7df977f3f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af60c299-x6r7g" Oct 31 04:47:12.605324 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:12.605295 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/763a4426-74f9-4458-981b-c7df977f3f5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af60c299-x6r7g\" (UID: \"763a4426-74f9-4458-981b-c7df977f3f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af60c299-x6r7g" Oct 31 04:47:12.605415 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:12.605347 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/763a4426-74f9-4458-981b-c7df977f3f5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af60c299-x6r7g\" (UID: \"763a4426-74f9-4458-981b-c7df977f3f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af60c299-x6r7g" Oct 31 04:47:12.606097 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:12.606066 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/763a4426-74f9-4458-981b-c7df977f3f5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af60c299-x6r7g\" (UID: \"763a4426-74f9-4458-981b-c7df977f3f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af60c299-x6r7g" Oct 31 04:47:12.608388 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:12.608327 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/763a4426-74f9-4458-981b-c7df977f3f5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af60c299-x6r7g\" (UID: \"763a4426-74f9-4458-981b-c7df977f3f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af60c299-x6r7g" Oct 31 04:47:12.645552 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:12.645507 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghd8z\" (UniqueName: \"kubernetes.io/projected/763a4426-74f9-4458-981b-c7df977f3f5d-kube-api-access-ghd8z\") pod \"maas-api-revoke-sub-af60c299-x6r7g\" (UID: \"763a4426-74f9-4458-981b-c7df977f3f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af60c299-x6r7g" Oct 31 04:47:12.731488 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:12.731443 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af60c299-x6r7g" Oct 31 04:47:13.050971 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:13.050931 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af60c299-x6r7g"] Oct 31 04:47:13.051464 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:47:13.051432 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod763a4426_74f9_4458_981b_c7df977f3f5d.slice/crio-bb32bc7212d7d50882c2164fdec356fe19393e7bc8cd65c6fc8b267f561e1dc1 WatchSource:0}: Error finding container bb32bc7212d7d50882c2164fdec356fe19393e7bc8cd65c6fc8b267f561e1dc1: Status 404 returned error can't find the container with id bb32bc7212d7d50882c2164fdec356fe19393e7bc8cd65c6fc8b267f561e1dc1 Oct 31 04:47:13.641216 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:13.641111 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af60c299-x6r7g" event={"ID":"763a4426-74f9-4458-981b-c7df977f3f5d","Type":"ContainerStarted","Data":"642ac8ae05c2d248bf950a6d69458a7ec71848118be2a82f47c4b66faa134613"} Oct 31 04:47:13.641216 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:13.641179 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af60c299-x6r7g" event={"ID":"763a4426-74f9-4458-981b-c7df977f3f5d","Type":"ContainerStarted","Data":"bb32bc7212d7d50882c2164fdec356fe19393e7bc8cd65c6fc8b267f561e1dc1"} Oct 31 04:47:13.684446 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:13.684369 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af60c299-x6r7g" podStartSLOduration=1.6843465279999998 podStartE2EDuration="1.684346528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:47:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:47:13.680624115 +0000 UTC m=+2588.436962158" watchObservedRunningTime="2026-10-31 04:47:13.684346528 +0000 UTC m=+2588.440684560" Oct 31 04:47:14.653489 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:14.653441 5019 generic.go:356] "Generic (PLEG): container finished" podID="763a4426-74f9-4458-981b-c7df977f3f5d" containerID="642ac8ae05c2d248bf950a6d69458a7ec71848118be2a82f47c4b66faa134613" exitCode=0 Oct 31 04:47:14.653943 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:14.653521 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af60c299-x6r7g" event={"ID":"763a4426-74f9-4458-981b-c7df977f3f5d","Type":"ContainerDied","Data":"642ac8ae05c2d248bf950a6d69458a7ec71848118be2a82f47c4b66faa134613"} Oct 31 04:47:15.870568 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:15.870532 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af60c299-x6r7g" Oct 31 04:47:15.989988 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:15.989937 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/763a4426-74f9-4458-981b-c7df977f3f5d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/763a4426-74f9-4458-981b-c7df977f3f5d-maas-api-openshift-ca\") pod \"763a4426-74f9-4458-981b-c7df977f3f5d\" (UID: \"763a4426-74f9-4458-981b-c7df977f3f5d\") " Oct 31 04:47:15.990231 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:15.990200 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/763a4426-74f9-4458-981b-c7df977f3f5d-kube-api-access-ghd8z\" (UniqueName: \"kubernetes.io/projected/763a4426-74f9-4458-981b-c7df977f3f5d-kube-api-access-ghd8z\") pod \"763a4426-74f9-4458-981b-c7df977f3f5d\" (UID: \"763a4426-74f9-4458-981b-c7df977f3f5d\") " Oct 31 04:47:15.990382 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:15.990361 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/763a4426-74f9-4458-981b-c7df977f3f5d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/763a4426-74f9-4458-981b-c7df977f3f5d-maas-api-xks-ca\") pod \"763a4426-74f9-4458-981b-c7df977f3f5d\" (UID: \"763a4426-74f9-4458-981b-c7df977f3f5d\") " Oct 31 04:47:15.991489 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:15.991401 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/763a4426-74f9-4458-981b-c7df977f3f5d-maas-api-openshift-ca" pod "763a4426-74f9-4458-981b-c7df977f3f5d" (UID: "763a4426-74f9-4458-981b-c7df977f3f5d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:47:15.993391 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:15.993305 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/763a4426-74f9-4458-981b-c7df977f3f5d-maas-api-xks-ca" pod "763a4426-74f9-4458-981b-c7df977f3f5d" (UID: "763a4426-74f9-4458-981b-c7df977f3f5d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:47:15.994031 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:15.994008 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/763a4426-74f9-4458-981b-c7df977f3f5d-kube-api-access-ghd8z" pod "763a4426-74f9-4458-981b-c7df977f3f5d" (UID: "763a4426-74f9-4458-981b-c7df977f3f5d"). InnerVolumeSpecName "kube-api-access-ghd8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:47:16.092872 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:16.092830 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghd8z\" (UniqueName: \"kubernetes.io/projected/763a4426-74f9-4458-981b-c7df977f3f5d-kube-api-access-ghd8z\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:47:16.092872 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:16.092870 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/763a4426-74f9-4458-981b-c7df977f3f5d-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:47:16.092872 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:16.092881 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/763a4426-74f9-4458-981b-c7df977f3f5d-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:47:16.672248 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:16.672209 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af60c299-x6r7g" Oct 31 04:47:16.672248 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:16.672223 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af60c299-x6r7g" event={"ID":"763a4426-74f9-4458-981b-c7df977f3f5d","Type":"ContainerDied","Data":"bb32bc7212d7d50882c2164fdec356fe19393e7bc8cd65c6fc8b267f561e1dc1"} Oct 31 04:47:16.672515 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:16.672280 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb32bc7212d7d50882c2164fdec356fe19393e7bc8cd65c6fc8b267f561e1dc1" Oct 31 04:47:22.445472 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:47:22.445415 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:47:24.065317 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:24.065276 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bf4a51a-rhtx7"] Oct 31 04:47:24.066146 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:24.066127 5019 cpu_manager.go:414] "containerMap: removing container" podUID="763a4426-74f9-4458-981b-c7df977f3f5d" containerName="revoke-keys" Oct 31 04:47:24.066193 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:24.066147 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="763a4426-74f9-4458-981b-c7df977f3f5d" containerName="revoke-keys" Oct 31 04:47:24.066327 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:24.066317 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="763a4426-74f9-4458-981b-c7df977f3f5d" containerName="revoke-keys" Oct 31 04:47:24.070462 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:24.070435 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf4a51a-rhtx7" Oct 31 04:47:24.090702 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:24.090663 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bf4a51a-rhtx7"] Oct 31 04:47:24.095767 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:24.095731 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:47:24.095941 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:24.095739 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:47:24.095941 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:24.095748 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:47:24.097267 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:24.097238 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:47:24.201268 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:24.201229 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d8d41c0-35f8-4b9b-a0c9-78bb458296e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bf4a51a-rhtx7\" (UID: \"9d8d41c0-35f8-4b9b-a0c9-78bb458296e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf4a51a-rhtx7" Oct 31 04:47:24.201538 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:24.201493 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d8d41c0-35f8-4b9b-a0c9-78bb458296e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bf4a51a-rhtx7\" (UID: \"9d8d41c0-35f8-4b9b-a0c9-78bb458296e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf4a51a-rhtx7" Oct 31 04:47:24.201678 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:24.201647 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqmt6\" (UniqueName: \"kubernetes.io/projected/9d8d41c0-35f8-4b9b-a0c9-78bb458296e5-kube-api-access-xqmt6\") pod \"maas-api-revoke-sub-4bf4a51a-rhtx7\" (UID: \"9d8d41c0-35f8-4b9b-a0c9-78bb458296e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf4a51a-rhtx7" Oct 31 04:47:24.302973 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:24.302932 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d8d41c0-35f8-4b9b-a0c9-78bb458296e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bf4a51a-rhtx7\" (UID: \"9d8d41c0-35f8-4b9b-a0c9-78bb458296e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf4a51a-rhtx7" Oct 31 04:47:24.303182 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:24.303068 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqmt6\" (UniqueName: \"kubernetes.io/projected/9d8d41c0-35f8-4b9b-a0c9-78bb458296e5-kube-api-access-xqmt6\") pod \"maas-api-revoke-sub-4bf4a51a-rhtx7\" (UID: \"9d8d41c0-35f8-4b9b-a0c9-78bb458296e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf4a51a-rhtx7" Oct 31 04:47:24.303235 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:24.303194 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d8d41c0-35f8-4b9b-a0c9-78bb458296e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bf4a51a-rhtx7\" (UID: \"9d8d41c0-35f8-4b9b-a0c9-78bb458296e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf4a51a-rhtx7" Oct 31 04:47:24.303934 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:24.303910 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d8d41c0-35f8-4b9b-a0c9-78bb458296e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bf4a51a-rhtx7\" (UID: \"9d8d41c0-35f8-4b9b-a0c9-78bb458296e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf4a51a-rhtx7" Oct 31 04:47:24.305628 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:24.305597 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d8d41c0-35f8-4b9b-a0c9-78bb458296e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bf4a51a-rhtx7\" (UID: \"9d8d41c0-35f8-4b9b-a0c9-78bb458296e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf4a51a-rhtx7" Oct 31 04:47:24.326954 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:24.326876 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqmt6\" (UniqueName: \"kubernetes.io/projected/9d8d41c0-35f8-4b9b-a0c9-78bb458296e5-kube-api-access-xqmt6\") pod \"maas-api-revoke-sub-4bf4a51a-rhtx7\" (UID: \"9d8d41c0-35f8-4b9b-a0c9-78bb458296e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf4a51a-rhtx7" Oct 31 04:47:24.388737 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:24.388637 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf4a51a-rhtx7" Oct 31 04:47:24.710393 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:24.710350 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bf4a51a-rhtx7"] Oct 31 04:47:24.710727 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:47:24.710700 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d8d41c0_35f8_4b9b_a0c9_78bb458296e5.slice/crio-4d98c352d4856976021239265cfb567b44171688a5d4758af07f269c85aacd9d WatchSource:0}: Error finding container 4d98c352d4856976021239265cfb567b44171688a5d4758af07f269c85aacd9d: Status 404 returned error can't find the container with id 4d98c352d4856976021239265cfb567b44171688a5d4758af07f269c85aacd9d Oct 31 04:47:24.814062 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:24.814012 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf4a51a-rhtx7" event={"ID":"9d8d41c0-35f8-4b9b-a0c9-78bb458296e5","Type":"ContainerStarted","Data":"4d98c352d4856976021239265cfb567b44171688a5d4758af07f269c85aacd9d"} Oct 31 04:47:25.824878 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:25.824840 5019 generic.go:356] "Generic (PLEG): container finished" podID="9d8d41c0-35f8-4b9b-a0c9-78bb458296e5" containerID="a12f033cee8fac7725a3064944db0adf96febccc80a81b0e5fd2658f7b29bceb" exitCode=0 Oct 31 04:47:25.825414 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:25.824903 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf4a51a-rhtx7" event={"ID":"9d8d41c0-35f8-4b9b-a0c9-78bb458296e5","Type":"ContainerDied","Data":"a12f033cee8fac7725a3064944db0adf96febccc80a81b0e5fd2658f7b29bceb"} Oct 31 04:47:25.922021 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:25.921970 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8de338de-dhrs2"] Oct 31 04:47:25.927436 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:25.927406 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de338de-dhrs2" Oct 31 04:47:25.950500 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:25.950458 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8de338de-dhrs2"] Oct 31 04:47:26.051668 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:26.051561 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/606c64ec-b8db-43e3-9d99-2f8092b031ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8de338de-dhrs2\" (UID: \"606c64ec-b8db-43e3-9d99-2f8092b031ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de338de-dhrs2" Oct 31 04:47:26.051668 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:26.051632 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqstl\" (UniqueName: \"kubernetes.io/projected/606c64ec-b8db-43e3-9d99-2f8092b031ad-kube-api-access-rqstl\") pod \"maas-api-revoke-sub-8de338de-dhrs2\" (UID: \"606c64ec-b8db-43e3-9d99-2f8092b031ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de338de-dhrs2" Oct 31 04:47:26.051905 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:26.051880 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/606c64ec-b8db-43e3-9d99-2f8092b031ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8de338de-dhrs2\" (UID: \"606c64ec-b8db-43e3-9d99-2f8092b031ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de338de-dhrs2" Oct 31 04:47:26.153413 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:26.153306 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/606c64ec-b8db-43e3-9d99-2f8092b031ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8de338de-dhrs2\" (UID: \"606c64ec-b8db-43e3-9d99-2f8092b031ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de338de-dhrs2" Oct 31 04:47:26.153613 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:26.153430 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqstl\" (UniqueName: \"kubernetes.io/projected/606c64ec-b8db-43e3-9d99-2f8092b031ad-kube-api-access-rqstl\") pod \"maas-api-revoke-sub-8de338de-dhrs2\" (UID: \"606c64ec-b8db-43e3-9d99-2f8092b031ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de338de-dhrs2" Oct 31 04:47:26.153613 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:26.153572 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/606c64ec-b8db-43e3-9d99-2f8092b031ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8de338de-dhrs2\" (UID: \"606c64ec-b8db-43e3-9d99-2f8092b031ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de338de-dhrs2" Oct 31 04:47:26.154244 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:26.154216 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/606c64ec-b8db-43e3-9d99-2f8092b031ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8de338de-dhrs2\" (UID: \"606c64ec-b8db-43e3-9d99-2f8092b031ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de338de-dhrs2" Oct 31 04:47:26.156022 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:26.155988 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/606c64ec-b8db-43e3-9d99-2f8092b031ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8de338de-dhrs2\" (UID: \"606c64ec-b8db-43e3-9d99-2f8092b031ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de338de-dhrs2" Oct 31 04:47:26.177326 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:26.177292 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqstl\" (UniqueName: \"kubernetes.io/projected/606c64ec-b8db-43e3-9d99-2f8092b031ad-kube-api-access-rqstl\") pod \"maas-api-revoke-sub-8de338de-dhrs2\" (UID: \"606c64ec-b8db-43e3-9d99-2f8092b031ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de338de-dhrs2" Oct 31 04:47:26.245689 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:26.245655 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de338de-dhrs2" Oct 31 04:47:26.781739 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:47:26.781694 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod606c64ec_b8db_43e3_9d99_2f8092b031ad.slice/crio-22305cd57c1849a7ee2ff0202fe565099d23b593d3ce495e2a7371dd708ca330 WatchSource:0}: Error finding container 22305cd57c1849a7ee2ff0202fe565099d23b593d3ce495e2a7371dd708ca330: Status 404 returned error can't find the container with id 22305cd57c1849a7ee2ff0202fe565099d23b593d3ce495e2a7371dd708ca330 Oct 31 04:47:26.813832 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:26.813787 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8de338de-dhrs2"] Oct 31 04:47:26.849440 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:26.849355 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de338de-dhrs2" event={"ID":"606c64ec-b8db-43e3-9d99-2f8092b031ad","Type":"ContainerStarted","Data":"22305cd57c1849a7ee2ff0202fe565099d23b593d3ce495e2a7371dd708ca330"} Oct 31 04:47:27.116103 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:27.116070 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf4a51a-rhtx7" Oct 31 04:47:27.179106 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:27.179065 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d8d41c0-35f8-4b9b-a0c9-78bb458296e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d8d41c0-35f8-4b9b-a0c9-78bb458296e5-maas-api-openshift-ca\") pod \"9d8d41c0-35f8-4b9b-a0c9-78bb458296e5\" (UID: \"9d8d41c0-35f8-4b9b-a0c9-78bb458296e5\") " Oct 31 04:47:27.179301 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:27.179266 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d8d41c0-35f8-4b9b-a0c9-78bb458296e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d8d41c0-35f8-4b9b-a0c9-78bb458296e5-maas-api-xks-ca\") pod \"9d8d41c0-35f8-4b9b-a0c9-78bb458296e5\" (UID: \"9d8d41c0-35f8-4b9b-a0c9-78bb458296e5\") " Oct 31 04:47:27.179509 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:27.179471 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d8d41c0-35f8-4b9b-a0c9-78bb458296e5-kube-api-access-xqmt6\" (UniqueName: \"kubernetes.io/projected/9d8d41c0-35f8-4b9b-a0c9-78bb458296e5-kube-api-access-xqmt6\") pod \"9d8d41c0-35f8-4b9b-a0c9-78bb458296e5\" (UID: \"9d8d41c0-35f8-4b9b-a0c9-78bb458296e5\") " Oct 31 04:47:27.183191 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:27.180441 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d8d41c0-35f8-4b9b-a0c9-78bb458296e5-maas-api-openshift-ca" pod "9d8d41c0-35f8-4b9b-a0c9-78bb458296e5" (UID: "9d8d41c0-35f8-4b9b-a0c9-78bb458296e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:47:27.183796 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:27.183575 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d8d41c0-35f8-4b9b-a0c9-78bb458296e5-maas-api-xks-ca" pod "9d8d41c0-35f8-4b9b-a0c9-78bb458296e5" (UID: "9d8d41c0-35f8-4b9b-a0c9-78bb458296e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:47:27.183796 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:27.183716 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d8d41c0-35f8-4b9b-a0c9-78bb458296e5-kube-api-access-xqmt6" pod "9d8d41c0-35f8-4b9b-a0c9-78bb458296e5" (UID: "9d8d41c0-35f8-4b9b-a0c9-78bb458296e5"). InnerVolumeSpecName "kube-api-access-xqmt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:47:27.282722 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:27.282683 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d8d41c0-35f8-4b9b-a0c9-78bb458296e5-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:47:27.282722 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:27.282722 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d8d41c0-35f8-4b9b-a0c9-78bb458296e5-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:47:27.282903 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:27.282734 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqmt6\" (UniqueName: \"kubernetes.io/projected/9d8d41c0-35f8-4b9b-a0c9-78bb458296e5-kube-api-access-xqmt6\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:47:27.860920 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:27.860877 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de338de-dhrs2" event={"ID":"606c64ec-b8db-43e3-9d99-2f8092b031ad","Type":"ContainerStarted","Data":"14243c76407f35cfa75c8b2509a61e8106d323de7946353d6ae89461636a8b06"} Oct 31 04:47:27.862582 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:27.862555 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf4a51a-rhtx7" Oct 31 04:47:27.862582 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:27.862567 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf4a51a-rhtx7" event={"ID":"9d8d41c0-35f8-4b9b-a0c9-78bb458296e5","Type":"ContainerDied","Data":"4d98c352d4856976021239265cfb567b44171688a5d4758af07f269c85aacd9d"} Oct 31 04:47:27.862792 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:27.862610 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d98c352d4856976021239265cfb567b44171688a5d4758af07f269c85aacd9d" Oct 31 04:47:27.904542 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:27.904484 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de338de-dhrs2" podStartSLOduration=2.904469022 podStartE2EDuration="2.904469022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:47:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:47:27.901540627 +0000 UTC m=+2602.657878670" watchObservedRunningTime="2026-10-31 04:47:27.904469022 +0000 UTC m=+2602.660807062" Oct 31 04:47:28.873902 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:28.873861 5019 generic.go:356] "Generic (PLEG): container finished" podID="606c64ec-b8db-43e3-9d99-2f8092b031ad" containerID="14243c76407f35cfa75c8b2509a61e8106d323de7946353d6ae89461636a8b06" exitCode=0 Oct 31 04:47:28.874380 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:28.873942 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de338de-dhrs2" event={"ID":"606c64ec-b8db-43e3-9d99-2f8092b031ad","Type":"ContainerDied","Data":"14243c76407f35cfa75c8b2509a61e8106d323de7946353d6ae89461636a8b06"} Oct 31 04:47:30.090951 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:30.090918 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de338de-dhrs2" Oct 31 04:47:30.232073 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:30.232033 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/606c64ec-b8db-43e3-9d99-2f8092b031ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/606c64ec-b8db-43e3-9d99-2f8092b031ad-maas-api-openshift-ca\") pod \"606c64ec-b8db-43e3-9d99-2f8092b031ad\" (UID: \"606c64ec-b8db-43e3-9d99-2f8092b031ad\") " Oct 31 04:47:30.232266 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:30.232168 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/606c64ec-b8db-43e3-9d99-2f8092b031ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/606c64ec-b8db-43e3-9d99-2f8092b031ad-maas-api-xks-ca\") pod \"606c64ec-b8db-43e3-9d99-2f8092b031ad\" (UID: \"606c64ec-b8db-43e3-9d99-2f8092b031ad\") " Oct 31 04:47:30.232546 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:30.232517 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/606c64ec-b8db-43e3-9d99-2f8092b031ad-kube-api-access-rqstl\" (UniqueName: \"kubernetes.io/projected/606c64ec-b8db-43e3-9d99-2f8092b031ad-kube-api-access-rqstl\") pod \"606c64ec-b8db-43e3-9d99-2f8092b031ad\" (UID: \"606c64ec-b8db-43e3-9d99-2f8092b031ad\") " Oct 31 04:47:30.232645 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:30.232510 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/606c64ec-b8db-43e3-9d99-2f8092b031ad-maas-api-openshift-ca" pod "606c64ec-b8db-43e3-9d99-2f8092b031ad" (UID: "606c64ec-b8db-43e3-9d99-2f8092b031ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:47:30.233106 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:30.233083 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/606c64ec-b8db-43e3-9d99-2f8092b031ad-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:47:30.234840 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:30.234810 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/606c64ec-b8db-43e3-9d99-2f8092b031ad-maas-api-xks-ca" pod "606c64ec-b8db-43e3-9d99-2f8092b031ad" (UID: "606c64ec-b8db-43e3-9d99-2f8092b031ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:47:30.234961 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:30.234843 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/606c64ec-b8db-43e3-9d99-2f8092b031ad-kube-api-access-rqstl" pod "606c64ec-b8db-43e3-9d99-2f8092b031ad" (UID: "606c64ec-b8db-43e3-9d99-2f8092b031ad"). InnerVolumeSpecName "kube-api-access-rqstl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:47:30.334704 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:30.334660 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqstl\" (UniqueName: \"kubernetes.io/projected/606c64ec-b8db-43e3-9d99-2f8092b031ad-kube-api-access-rqstl\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:47:30.334704 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:30.334696 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/606c64ec-b8db-43e3-9d99-2f8092b031ad-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:47:30.895570 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:30.895527 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de338de-dhrs2" Oct 31 04:47:30.895782 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:30.895573 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de338de-dhrs2" event={"ID":"606c64ec-b8db-43e3-9d99-2f8092b031ad","Type":"ContainerDied","Data":"22305cd57c1849a7ee2ff0202fe565099d23b593d3ce495e2a7371dd708ca330"} Oct 31 04:47:30.895782 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:30.895640 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22305cd57c1849a7ee2ff0202fe565099d23b593d3ce495e2a7371dd708ca330" Oct 31 04:47:34.648570 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:34.648519 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-445cfbac-vw5hf"] Oct 31 04:47:34.649791 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:34.649769 5019 cpu_manager.go:414] "containerMap: removing container" podUID="606c64ec-b8db-43e3-9d99-2f8092b031ad" containerName="revoke-keys" Oct 31 04:47:34.649851 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:34.649793 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="606c64ec-b8db-43e3-9d99-2f8092b031ad" containerName="revoke-keys" Oct 31 04:47:34.649896 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:34.649879 5019 cpu_manager.go:414] "containerMap: removing container" podUID="9d8d41c0-35f8-4b9b-a0c9-78bb458296e5" containerName="revoke-keys" Oct 31 04:47:34.649896 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:34.649888 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d8d41c0-35f8-4b9b-a0c9-78bb458296e5" containerName="revoke-keys" Oct 31 04:47:34.650092 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:34.650080 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="606c64ec-b8db-43e3-9d99-2f8092b031ad" containerName="revoke-keys" Oct 31 04:47:34.650130 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:34.650107 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d8d41c0-35f8-4b9b-a0c9-78bb458296e5" containerName="revoke-keys" Oct 31 04:47:34.654880 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:34.654855 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-445cfbac-vw5hf" Oct 31 04:47:34.676432 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:34.676393 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-445cfbac-vw5hf"] Oct 31 04:47:34.677519 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:34.677451 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:47:34.677519 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:34.677478 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:47:34.677519 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:34.677506 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:47:34.677788 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:34.677583 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:47:34.822085 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:34.822040 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b7643e1-621e-4610-a378-8732b4b33d70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-445cfbac-vw5hf\" (UID: \"6b7643e1-621e-4610-a378-8732b4b33d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-445cfbac-vw5hf" Oct 31 04:47:34.822299 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:34.822183 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b7643e1-621e-4610-a378-8732b4b33d70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-445cfbac-vw5hf\" (UID: \"6b7643e1-621e-4610-a378-8732b4b33d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-445cfbac-vw5hf" Oct 31 04:47:34.822460 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:34.822441 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-829tz\" (UniqueName: \"kubernetes.io/projected/6b7643e1-621e-4610-a378-8732b4b33d70-kube-api-access-829tz\") pod \"maas-api-revoke-sub-445cfbac-vw5hf\" (UID: \"6b7643e1-621e-4610-a378-8732b4b33d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-445cfbac-vw5hf" Oct 31 04:47:34.924101 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:34.923901 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-829tz\" (UniqueName: \"kubernetes.io/projected/6b7643e1-621e-4610-a378-8732b4b33d70-kube-api-access-829tz\") pod \"maas-api-revoke-sub-445cfbac-vw5hf\" (UID: \"6b7643e1-621e-4610-a378-8732b4b33d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-445cfbac-vw5hf" Oct 31 04:47:34.924356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:34.924106 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b7643e1-621e-4610-a378-8732b4b33d70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-445cfbac-vw5hf\" (UID: \"6b7643e1-621e-4610-a378-8732b4b33d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-445cfbac-vw5hf" Oct 31 04:47:34.924356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:34.924249 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b7643e1-621e-4610-a378-8732b4b33d70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-445cfbac-vw5hf\" (UID: \"6b7643e1-621e-4610-a378-8732b4b33d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-445cfbac-vw5hf" Oct 31 04:47:34.925190 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:34.925158 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b7643e1-621e-4610-a378-8732b4b33d70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-445cfbac-vw5hf\" (UID: \"6b7643e1-621e-4610-a378-8732b4b33d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-445cfbac-vw5hf" Oct 31 04:47:34.927093 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:34.927069 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b7643e1-621e-4610-a378-8732b4b33d70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-445cfbac-vw5hf\" (UID: \"6b7643e1-621e-4610-a378-8732b4b33d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-445cfbac-vw5hf" Oct 31 04:47:34.947534 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:34.947493 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-829tz\" (UniqueName: \"kubernetes.io/projected/6b7643e1-621e-4610-a378-8732b4b33d70-kube-api-access-829tz\") pod \"maas-api-revoke-sub-445cfbac-vw5hf\" (UID: \"6b7643e1-621e-4610-a378-8732b4b33d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-445cfbac-vw5hf" Oct 31 04:47:34.973261 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:34.973215 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-445cfbac-vw5hf" Oct 31 04:47:35.288227 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:35.288176 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-445cfbac-vw5hf"] Oct 31 04:47:35.943167 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:35.943075 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-445cfbac-vw5hf" event={"ID":"6b7643e1-621e-4610-a378-8732b4b33d70","Type":"ContainerStarted","Data":"a2f934ca7ab02202e9c1290f1ce40075d7e960f2274cdadafd4a9d82c8a7a2ee"} Oct 31 04:47:35.943167 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:35.943136 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-445cfbac-vw5hf" event={"ID":"6b7643e1-621e-4610-a378-8732b4b33d70","Type":"ContainerStarted","Data":"2c92d9da710391b2febc4d545c795b0280bfad39eb24880b96ae7c3eaef11bfd"} Oct 31 04:47:35.989945 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:35.989877 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-445cfbac-vw5hf" podStartSLOduration=1.989858838 podStartE2EDuration="1.989858838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:47:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:47:35.984075014 +0000 UTC m=+2610.740413056" watchObservedRunningTime="2026-10-31 04:47:35.989858838 +0000 UTC m=+2610.746196891" Oct 31 04:47:36.315026 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:36.314985 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4df7c260-rdsmc"] Oct 31 04:47:36.320020 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:36.319991 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4df7c260-rdsmc" Oct 31 04:47:36.339482 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:36.339445 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4df7c260-rdsmc"] Oct 31 04:47:36.450974 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:36.450928 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/184444ef-c8f0-43a1-8223-82d2f99d8e29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4df7c260-rdsmc\" (UID: \"184444ef-c8f0-43a1-8223-82d2f99d8e29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4df7c260-rdsmc" Oct 31 04:47:36.451209 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:36.451024 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfnps\" (UniqueName: \"kubernetes.io/projected/184444ef-c8f0-43a1-8223-82d2f99d8e29-kube-api-access-lfnps\") pod \"maas-api-revoke-sub-4df7c260-rdsmc\" (UID: \"184444ef-c8f0-43a1-8223-82d2f99d8e29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4df7c260-rdsmc" Oct 31 04:47:36.451307 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:36.451281 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/184444ef-c8f0-43a1-8223-82d2f99d8e29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4df7c260-rdsmc\" (UID: \"184444ef-c8f0-43a1-8223-82d2f99d8e29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4df7c260-rdsmc" Oct 31 04:47:36.552710 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:36.552659 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfnps\" (UniqueName: \"kubernetes.io/projected/184444ef-c8f0-43a1-8223-82d2f99d8e29-kube-api-access-lfnps\") pod \"maas-api-revoke-sub-4df7c260-rdsmc\" (UID: \"184444ef-c8f0-43a1-8223-82d2f99d8e29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4df7c260-rdsmc" Oct 31 04:47:36.552948 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:36.552745 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/184444ef-c8f0-43a1-8223-82d2f99d8e29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4df7c260-rdsmc\" (UID: \"184444ef-c8f0-43a1-8223-82d2f99d8e29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4df7c260-rdsmc" Oct 31 04:47:36.552948 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:36.552937 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/184444ef-c8f0-43a1-8223-82d2f99d8e29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4df7c260-rdsmc\" (UID: \"184444ef-c8f0-43a1-8223-82d2f99d8e29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4df7c260-rdsmc" Oct 31 04:47:36.553676 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:36.553651 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/184444ef-c8f0-43a1-8223-82d2f99d8e29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4df7c260-rdsmc\" (UID: \"184444ef-c8f0-43a1-8223-82d2f99d8e29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4df7c260-rdsmc" Oct 31 04:47:36.555492 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:36.555465 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/184444ef-c8f0-43a1-8223-82d2f99d8e29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4df7c260-rdsmc\" (UID: \"184444ef-c8f0-43a1-8223-82d2f99d8e29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4df7c260-rdsmc" Oct 31 04:47:36.576882 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:36.576786 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfnps\" (UniqueName: \"kubernetes.io/projected/184444ef-c8f0-43a1-8223-82d2f99d8e29-kube-api-access-lfnps\") pod \"maas-api-revoke-sub-4df7c260-rdsmc\" (UID: \"184444ef-c8f0-43a1-8223-82d2f99d8e29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4df7c260-rdsmc" Oct 31 04:47:36.642358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:36.640227 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4df7c260-rdsmc" Oct 31 04:47:36.954472 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:36.954376 5019 generic.go:356] "Generic (PLEG): container finished" podID="6b7643e1-621e-4610-a378-8732b4b33d70" containerID="a2f934ca7ab02202e9c1290f1ce40075d7e960f2274cdadafd4a9d82c8a7a2ee" exitCode=0 Oct 31 04:47:36.954874 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:36.954465 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-445cfbac-vw5hf" event={"ID":"6b7643e1-621e-4610-a378-8732b4b33d70","Type":"ContainerDied","Data":"a2f934ca7ab02202e9c1290f1ce40075d7e960f2274cdadafd4a9d82c8a7a2ee"} Oct 31 04:47:36.971633 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:47:36.971589 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod184444ef_c8f0_43a1_8223_82d2f99d8e29.slice/crio-9f0944053da64277baac8d60248f638bd397d004db066ed989cb312681446968 WatchSource:0}: Error finding container 9f0944053da64277baac8d60248f638bd397d004db066ed989cb312681446968: Status 404 returned error can't find the container with id 9f0944053da64277baac8d60248f638bd397d004db066ed989cb312681446968 Oct 31 04:47:36.972697 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:36.972557 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4df7c260-rdsmc"] Oct 31 04:47:38.095357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:38.093259 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4df7c260-rdsmc" event={"ID":"184444ef-c8f0-43a1-8223-82d2f99d8e29","Type":"ContainerStarted","Data":"d20d0c6edf652f3baf63b7cfa9a21fbb02d8f79ba4350dc6b38da4cf0a186a32"} Oct 31 04:47:38.095357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:38.093324 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4df7c260-rdsmc" event={"ID":"184444ef-c8f0-43a1-8223-82d2f99d8e29","Type":"ContainerStarted","Data":"9f0944053da64277baac8d60248f638bd397d004db066ed989cb312681446968"} Oct 31 04:47:38.519860 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:38.519824 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-445cfbac-vw5hf" Oct 31 04:47:38.563298 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:38.563229 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4df7c260-rdsmc" podStartSLOduration=2.563208748 podStartE2EDuration="2.563208748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:47:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:47:38.140418689 +0000 UTC m=+2612.896756730" watchObservedRunningTime="2026-10-31 04:47:38.563208748 +0000 UTC m=+2613.319546789" Oct 31 04:47:38.645635 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:38.645595 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b7643e1-621e-4610-a378-8732b4b33d70-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b7643e1-621e-4610-a378-8732b4b33d70-maas-api-xks-ca\") pod \"6b7643e1-621e-4610-a378-8732b4b33d70\" (UID: \"6b7643e1-621e-4610-a378-8732b4b33d70\") " Oct 31 04:47:38.645966 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:38.645667 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b7643e1-621e-4610-a378-8732b4b33d70-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b7643e1-621e-4610-a378-8732b4b33d70-maas-api-openshift-ca\") pod \"6b7643e1-621e-4610-a378-8732b4b33d70\" (UID: \"6b7643e1-621e-4610-a378-8732b4b33d70\") " Oct 31 04:47:38.645966 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:38.645714 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b7643e1-621e-4610-a378-8732b4b33d70-kube-api-access-829tz\" (UniqueName: \"kubernetes.io/projected/6b7643e1-621e-4610-a378-8732b4b33d70-kube-api-access-829tz\") pod \"6b7643e1-621e-4610-a378-8732b4b33d70\" (UID: \"6b7643e1-621e-4610-a378-8732b4b33d70\") " Oct 31 04:47:38.646118 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:38.646037 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b7643e1-621e-4610-a378-8732b4b33d70-maas-api-openshift-ca" pod "6b7643e1-621e-4610-a378-8732b4b33d70" (UID: "6b7643e1-621e-4610-a378-8732b4b33d70"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:47:38.646762 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:38.646731 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b7643e1-621e-4610-a378-8732b4b33d70-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:47:38.648233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:38.648206 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b7643e1-621e-4610-a378-8732b4b33d70-kube-api-access-829tz" pod "6b7643e1-621e-4610-a378-8732b4b33d70" (UID: "6b7643e1-621e-4610-a378-8732b4b33d70"). InnerVolumeSpecName "kube-api-access-829tz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:47:38.648488 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:38.648461 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b7643e1-621e-4610-a378-8732b4b33d70-maas-api-xks-ca" pod "6b7643e1-621e-4610-a378-8732b4b33d70" (UID: "6b7643e1-621e-4610-a378-8732b4b33d70"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:47:38.749106 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:38.749055 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b7643e1-621e-4610-a378-8732b4b33d70-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:47:38.749106 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:38.749104 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-829tz\" (UniqueName: \"kubernetes.io/projected/6b7643e1-621e-4610-a378-8732b4b33d70-kube-api-access-829tz\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:47:39.106631 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:39.106585 5019 generic.go:356] "Generic (PLEG): container finished" podID="184444ef-c8f0-43a1-8223-82d2f99d8e29" containerID="d20d0c6edf652f3baf63b7cfa9a21fbb02d8f79ba4350dc6b38da4cf0a186a32" exitCode=0 Oct 31 04:47:39.107177 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:39.106920 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4df7c260-rdsmc" event={"ID":"184444ef-c8f0-43a1-8223-82d2f99d8e29","Type":"ContainerDied","Data":"d20d0c6edf652f3baf63b7cfa9a21fbb02d8f79ba4350dc6b38da4cf0a186a32"} Oct 31 04:47:39.109222 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:39.109191 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-445cfbac-vw5hf" Oct 31 04:47:39.109222 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:39.109207 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-445cfbac-vw5hf" event={"ID":"6b7643e1-621e-4610-a378-8732b4b33d70","Type":"ContainerDied","Data":"2c92d9da710391b2febc4d545c795b0280bfad39eb24880b96ae7c3eaef11bfd"} Oct 31 04:47:39.109527 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:39.109270 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c92d9da710391b2febc4d545c795b0280bfad39eb24880b96ae7c3eaef11bfd" Oct 31 04:47:40.328573 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:40.328548 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4df7c260-rdsmc" Oct 31 04:47:40.417419 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:40.417266 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/184444ef-c8f0-43a1-8223-82d2f99d8e29-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/184444ef-c8f0-43a1-8223-82d2f99d8e29-maas-api-openshift-ca\") pod \"184444ef-c8f0-43a1-8223-82d2f99d8e29\" (UID: \"184444ef-c8f0-43a1-8223-82d2f99d8e29\") " Oct 31 04:47:40.417603 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:40.417511 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/184444ef-c8f0-43a1-8223-82d2f99d8e29-kube-api-access-lfnps\" (UniqueName: \"kubernetes.io/projected/184444ef-c8f0-43a1-8223-82d2f99d8e29-kube-api-access-lfnps\") pod \"184444ef-c8f0-43a1-8223-82d2f99d8e29\" (UID: \"184444ef-c8f0-43a1-8223-82d2f99d8e29\") " Oct 31 04:47:40.417603 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:40.417555 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/184444ef-c8f0-43a1-8223-82d2f99d8e29-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/184444ef-c8f0-43a1-8223-82d2f99d8e29-maas-api-xks-ca\") pod \"184444ef-c8f0-43a1-8223-82d2f99d8e29\" (UID: \"184444ef-c8f0-43a1-8223-82d2f99d8e29\") " Oct 31 04:47:40.419420 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:40.419387 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/184444ef-c8f0-43a1-8223-82d2f99d8e29-maas-api-openshift-ca" pod "184444ef-c8f0-43a1-8223-82d2f99d8e29" (UID: "184444ef-c8f0-43a1-8223-82d2f99d8e29"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:47:40.422288 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:40.422257 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/184444ef-c8f0-43a1-8223-82d2f99d8e29-maas-api-xks-ca" pod "184444ef-c8f0-43a1-8223-82d2f99d8e29" (UID: "184444ef-c8f0-43a1-8223-82d2f99d8e29"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:47:40.426385 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:40.425317 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7qh6vn"] Oct 31 04:47:40.434079 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:40.434043 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/184444ef-c8f0-43a1-8223-82d2f99d8e29-kube-api-access-lfnps" pod "184444ef-c8f0-43a1-8223-82d2f99d8e29" (UID: "184444ef-c8f0-43a1-8223-82d2f99d8e29"). InnerVolumeSpecName "kube-api-access-lfnps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:47:40.521324 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:40.521288 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/184444ef-c8f0-43a1-8223-82d2f99d8e29-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:47:40.521324 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:40.521322 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfnps\" (UniqueName: \"kubernetes.io/projected/184444ef-c8f0-43a1-8223-82d2f99d8e29-kube-api-access-lfnps\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:47:40.521324 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:40.521350 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/184444ef-c8f0-43a1-8223-82d2f99d8e29-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:47:41.134728 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:41.134694 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4df7c260-rdsmc" event={"ID":"184444ef-c8f0-43a1-8223-82d2f99d8e29","Type":"ContainerDied","Data":"9f0944053da64277baac8d60248f638bd397d004db066ed989cb312681446968"} Oct 31 04:47:41.134927 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:41.134744 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f0944053da64277baac8d60248f638bd397d004db066ed989cb312681446968" Oct 31 04:47:41.134927 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:41.134708 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4df7c260-rdsmc" Oct 31 04:47:45.426172 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:45.426129 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7qh6vn" podUID="4c9f3184-0ead-44b0-bc51-fc4972f5a031" containerName="payload-processing" containerID="cri-o://03e920087f6a143280eaa6b64dff0aee6e85ade6c9c61e8806547e9e9ad61abd" gracePeriod=25 Oct 31 04:47:46.024031 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:46.023992 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7qh6vn" Oct 31 04:47:46.087441 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:46.087387 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c9f3184-0ead-44b0-bc51-fc4972f5a031-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4c9f3184-0ead-44b0-bc51-fc4972f5a031-provider-credentials\") pod \"4c9f3184-0ead-44b0-bc51-fc4972f5a031\" (UID: \"4c9f3184-0ead-44b0-bc51-fc4972f5a031\") " Oct 31 04:47:46.088621 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:46.088549 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c9f3184-0ead-44b0-bc51-fc4972f5a031-kube-api-access-fz6pr\" (UniqueName: \"kubernetes.io/projected/4c9f3184-0ead-44b0-bc51-fc4972f5a031-kube-api-access-fz6pr\") pod \"4c9f3184-0ead-44b0-bc51-fc4972f5a031\" (UID: \"4c9f3184-0ead-44b0-bc51-fc4972f5a031\") " Oct 31 04:47:46.088621 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:46.088608 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c9f3184-0ead-44b0-bc51-fc4972f5a031-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4c9f3184-0ead-44b0-bc51-fc4972f5a031-routing-overlay-volume\") pod \"4c9f3184-0ead-44b0-bc51-fc4972f5a031\" (UID: \"4c9f3184-0ead-44b0-bc51-fc4972f5a031\") " Oct 31 04:47:46.088828 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:46.088706 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c9f3184-0ead-44b0-bc51-fc4972f5a031-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c9f3184-0ead-44b0-bc51-fc4972f5a031-plugins-config-volume\") pod \"4c9f3184-0ead-44b0-bc51-fc4972f5a031\" (UID: \"4c9f3184-0ead-44b0-bc51-fc4972f5a031\") " Oct 31 04:47:46.089514 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:46.089247 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c9f3184-0ead-44b0-bc51-fc4972f5a031-routing-overlay-volume" pod "4c9f3184-0ead-44b0-bc51-fc4972f5a031" (UID: "4c9f3184-0ead-44b0-bc51-fc4972f5a031"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:47:46.089514 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:46.089480 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c9f3184-0ead-44b0-bc51-fc4972f5a031-plugins-config-volume" pod "4c9f3184-0ead-44b0-bc51-fc4972f5a031" (UID: "4c9f3184-0ead-44b0-bc51-fc4972f5a031"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:47:46.089767 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:46.089657 5019 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4c9f3184-0ead-44b0-bc51-fc4972f5a031-routing-overlay-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:47:46.089767 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:46.089682 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c9f3184-0ead-44b0-bc51-fc4972f5a031-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:47:46.108505 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:46.108309 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c9f3184-0ead-44b0-bc51-fc4972f5a031-kube-api-access-fz6pr" pod "4c9f3184-0ead-44b0-bc51-fc4972f5a031" (UID: "4c9f3184-0ead-44b0-bc51-fc4972f5a031"). InnerVolumeSpecName "kube-api-access-fz6pr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:47:46.108823 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:46.108790 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c9f3184-0ead-44b0-bc51-fc4972f5a031-provider-credentials" pod "4c9f3184-0ead-44b0-bc51-fc4972f5a031" (UID: "4c9f3184-0ead-44b0-bc51-fc4972f5a031"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:47:46.201471 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:46.194750 5019 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4c9f3184-0ead-44b0-bc51-fc4972f5a031-provider-credentials\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:47:46.201471 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:46.194796 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz6pr\" (UniqueName: \"kubernetes.io/projected/4c9f3184-0ead-44b0-bc51-fc4972f5a031-kube-api-access-fz6pr\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:47:46.258354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:46.257376 5019 generic.go:356] "Generic (PLEG): container finished" podID="4c9f3184-0ead-44b0-bc51-fc4972f5a031" containerID="03e920087f6a143280eaa6b64dff0aee6e85ade6c9c61e8806547e9e9ad61abd" exitCode=0 Oct 31 04:47:46.258354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:46.257667 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7qh6vn" event={"ID":"4c9f3184-0ead-44b0-bc51-fc4972f5a031","Type":"ContainerDied","Data":"03e920087f6a143280eaa6b64dff0aee6e85ade6c9c61e8806547e9e9ad61abd"} Oct 31 04:47:46.258354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:46.257755 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7qh6vn" event={"ID":"4c9f3184-0ead-44b0-bc51-fc4972f5a031","Type":"ContainerDied","Data":"96bc3f71c389abdd1d356f0cac529f3bc3396af879ad94512e3f94569cb2df82"} Oct 31 04:47:46.258354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:46.257783 5019 scope.go:122] "RemoveContainer" containerID="03e920087f6a143280eaa6b64dff0aee6e85ade6c9c61e8806547e9e9ad61abd" Oct 31 04:47:46.258354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:46.257968 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7qh6vn" Oct 31 04:47:46.287466 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:46.287434 5019 scope.go:122] "RemoveContainer" containerID="03e920087f6a143280eaa6b64dff0aee6e85ade6c9c61e8806547e9e9ad61abd" Oct 31 04:47:46.287972 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:47:46.287932 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03e920087f6a143280eaa6b64dff0aee6e85ade6c9c61e8806547e9e9ad61abd\": container with ID starting with 03e920087f6a143280eaa6b64dff0aee6e85ade6c9c61e8806547e9e9ad61abd not found: ID does not exist" containerID="03e920087f6a143280eaa6b64dff0aee6e85ade6c9c61e8806547e9e9ad61abd" Oct 31 04:47:46.288075 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:46.287989 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03e920087f6a143280eaa6b64dff0aee6e85ade6c9c61e8806547e9e9ad61abd"} err="failed to get container status \"03e920087f6a143280eaa6b64dff0aee6e85ade6c9c61e8806547e9e9ad61abd\": rpc error: code = NotFound desc = could not find container \"03e920087f6a143280eaa6b64dff0aee6e85ade6c9c61e8806547e9e9ad61abd\": container with ID starting with 03e920087f6a143280eaa6b64dff0aee6e85ade6c9c61e8806547e9e9ad61abd not found: ID does not exist" Oct 31 04:47:46.324735 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:46.324702 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7qh6vn"] Oct 31 04:47:46.329575 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:46.329548 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7qh6vn"] Oct 31 04:47:47.137520 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:47.137477 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b236824d-k5j5w"] Oct 31 04:47:47.138508 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:47.138483 5019 cpu_manager.go:414] "containerMap: removing container" podUID="184444ef-c8f0-43a1-8223-82d2f99d8e29" containerName="revoke-keys" Oct 31 04:47:47.138508 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:47.138509 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="184444ef-c8f0-43a1-8223-82d2f99d8e29" containerName="revoke-keys" Oct 31 04:47:47.138655 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:47.138534 5019 cpu_manager.go:414] "containerMap: removing container" podUID="4c9f3184-0ead-44b0-bc51-fc4972f5a031" containerName="payload-processing" Oct 31 04:47:47.138655 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:47.138542 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c9f3184-0ead-44b0-bc51-fc4972f5a031" containerName="payload-processing" Oct 31 04:47:47.138655 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:47.138578 5019 cpu_manager.go:414] "containerMap: removing container" podUID="6b7643e1-621e-4610-a378-8732b4b33d70" containerName="revoke-keys" Oct 31 04:47:47.138655 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:47.138586 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b7643e1-621e-4610-a378-8732b4b33d70" containerName="revoke-keys" Oct 31 04:47:47.138804 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:47.138797 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="184444ef-c8f0-43a1-8223-82d2f99d8e29" containerName="revoke-keys" Oct 31 04:47:47.138844 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:47.138809 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c9f3184-0ead-44b0-bc51-fc4972f5a031" containerName="payload-processing" Oct 31 04:47:47.138844 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:47.138825 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b7643e1-621e-4610-a378-8732b4b33d70" containerName="revoke-keys" Oct 31 04:47:47.144721 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:47.144700 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b236824d-k5j5w" Oct 31 04:47:47.162098 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:47.162074 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b236824d-k5j5w"] Oct 31 04:47:47.167569 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:47.167538 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:47:47.167723 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:47.167642 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:47:47.167723 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:47.167676 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:47:47.167852 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:47.167744 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:47:47.168498 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:47.168474 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cq5m\" (UniqueName: \"kubernetes.io/projected/b578bb3a-b726-4057-a46b-b25e1cb0d767-kube-api-access-7cq5m\") pod \"maas-api-revoke-sub-b236824d-k5j5w\" (UID: \"b578bb3a-b726-4057-a46b-b25e1cb0d767\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b236824d-k5j5w" Oct 31 04:47:47.168676 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:47.168612 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b578bb3a-b726-4057-a46b-b25e1cb0d767-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b236824d-k5j5w\" (UID: \"b578bb3a-b726-4057-a46b-b25e1cb0d767\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b236824d-k5j5w" Oct 31 04:47:47.168754 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:47.168673 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b578bb3a-b726-4057-a46b-b25e1cb0d767-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b236824d-k5j5w\" (UID: \"b578bb3a-b726-4057-a46b-b25e1cb0d767\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b236824d-k5j5w" Oct 31 04:47:47.269819 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:47.269776 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cq5m\" (UniqueName: \"kubernetes.io/projected/b578bb3a-b726-4057-a46b-b25e1cb0d767-kube-api-access-7cq5m\") pod \"maas-api-revoke-sub-b236824d-k5j5w\" (UID: \"b578bb3a-b726-4057-a46b-b25e1cb0d767\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b236824d-k5j5w" Oct 31 04:47:47.270041 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:47.269870 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b578bb3a-b726-4057-a46b-b25e1cb0d767-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b236824d-k5j5w\" (UID: \"b578bb3a-b726-4057-a46b-b25e1cb0d767\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b236824d-k5j5w" Oct 31 04:47:47.270041 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:47.269906 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b578bb3a-b726-4057-a46b-b25e1cb0d767-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b236824d-k5j5w\" (UID: \"b578bb3a-b726-4057-a46b-b25e1cb0d767\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b236824d-k5j5w" Oct 31 04:47:47.271449 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:47.271411 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b578bb3a-b726-4057-a46b-b25e1cb0d767-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b236824d-k5j5w\" (UID: \"b578bb3a-b726-4057-a46b-b25e1cb0d767\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b236824d-k5j5w" Oct 31 04:47:47.272627 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:47.272596 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b578bb3a-b726-4057-a46b-b25e1cb0d767-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b236824d-k5j5w\" (UID: \"b578bb3a-b726-4057-a46b-b25e1cb0d767\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b236824d-k5j5w" Oct 31 04:47:47.294979 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:47.294944 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cq5m\" (UniqueName: \"kubernetes.io/projected/b578bb3a-b726-4057-a46b-b25e1cb0d767-kube-api-access-7cq5m\") pod \"maas-api-revoke-sub-b236824d-k5j5w\" (UID: \"b578bb3a-b726-4057-a46b-b25e1cb0d767\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b236824d-k5j5w" Oct 31 04:47:47.411876 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:47.411789 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c9f3184-0ead-44b0-bc51-fc4972f5a031" path="/var/lib/kubelet/pods/4c9f3184-0ead-44b0-bc51-fc4972f5a031/volumes" Oct 31 04:47:47.462574 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:47.462527 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b236824d-k5j5w" Oct 31 04:47:47.839432 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:47.839394 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b236824d-k5j5w"] Oct 31 04:47:47.844524 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:47:47.844481 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb578bb3a_b726_4057_a46b_b25e1cb0d767.slice/crio-ac16efb854a13af2e24ee252ad1a0288b62302e3a666d5e3e5d8fd285b446668 WatchSource:0}: Error finding container ac16efb854a13af2e24ee252ad1a0288b62302e3a666d5e3e5d8fd285b446668: Status 404 returned error can't find the container with id ac16efb854a13af2e24ee252ad1a0288b62302e3a666d5e3e5d8fd285b446668 Oct 31 04:47:47.898036 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:47.897989 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-648dnwfg6"] Oct 31 04:47:48.295915 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:48.295871 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b236824d-k5j5w" event={"ID":"b578bb3a-b726-4057-a46b-b25e1cb0d767","Type":"ContainerStarted","Data":"ac16efb854a13af2e24ee252ad1a0288b62302e3a666d5e3e5d8fd285b446668"} Oct 31 04:47:49.308970 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:49.308869 5019 generic.go:356] "Generic (PLEG): container finished" podID="b578bb3a-b726-4057-a46b-b25e1cb0d767" containerID="71dc248c40833079a458d00ae8b39197d140db07289b8d0888a74521e012d10c" exitCode=0 Oct 31 04:47:49.308970 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:49.308954 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b236824d-k5j5w" event={"ID":"b578bb3a-b726-4057-a46b-b25e1cb0d767","Type":"ContainerDied","Data":"71dc248c40833079a458d00ae8b39197d140db07289b8d0888a74521e012d10c"} Oct 31 04:47:50.726356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:50.724378 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b236824d-k5j5w" Oct 31 04:47:50.888659 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:50.886884 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b578bb3a-b726-4057-a46b-b25e1cb0d767-kube-api-access-7cq5m\" (UniqueName: \"kubernetes.io/projected/b578bb3a-b726-4057-a46b-b25e1cb0d767-kube-api-access-7cq5m\") pod \"b578bb3a-b726-4057-a46b-b25e1cb0d767\" (UID: \"b578bb3a-b726-4057-a46b-b25e1cb0d767\") " Oct 31 04:47:50.888659 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:50.887009 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b578bb3a-b726-4057-a46b-b25e1cb0d767-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b578bb3a-b726-4057-a46b-b25e1cb0d767-maas-api-openshift-ca\") pod \"b578bb3a-b726-4057-a46b-b25e1cb0d767\" (UID: \"b578bb3a-b726-4057-a46b-b25e1cb0d767\") " Oct 31 04:47:50.888659 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:50.887072 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b578bb3a-b726-4057-a46b-b25e1cb0d767-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b578bb3a-b726-4057-a46b-b25e1cb0d767-maas-api-xks-ca\") pod \"b578bb3a-b726-4057-a46b-b25e1cb0d767\" (UID: \"b578bb3a-b726-4057-a46b-b25e1cb0d767\") " Oct 31 04:47:50.932733 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:50.927851 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b578bb3a-b726-4057-a46b-b25e1cb0d767-maas-api-openshift-ca" pod "b578bb3a-b726-4057-a46b-b25e1cb0d767" (UID: "b578bb3a-b726-4057-a46b-b25e1cb0d767"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:47:50.964484 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:50.964447 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b578bb3a-b726-4057-a46b-b25e1cb0d767-maas-api-xks-ca" pod "b578bb3a-b726-4057-a46b-b25e1cb0d767" (UID: "b578bb3a-b726-4057-a46b-b25e1cb0d767"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:47:50.994414 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:50.994297 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b578bb3a-b726-4057-a46b-b25e1cb0d767-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:47:50.994863 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:50.994838 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b578bb3a-b726-4057-a46b-b25e1cb0d767-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:47:51.001390 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:51.001351 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b578bb3a-b726-4057-a46b-b25e1cb0d767-kube-api-access-7cq5m" pod "b578bb3a-b726-4057-a46b-b25e1cb0d767" (UID: "b578bb3a-b726-4057-a46b-b25e1cb0d767"). InnerVolumeSpecName "kube-api-access-7cq5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:47:51.098677 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:51.098630 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cq5m\" (UniqueName: \"kubernetes.io/projected/b578bb3a-b726-4057-a46b-b25e1cb0d767-kube-api-access-7cq5m\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:47:51.399410 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:51.397072 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b236824d-k5j5w" event={"ID":"b578bb3a-b726-4057-a46b-b25e1cb0d767","Type":"ContainerDied","Data":"ac16efb854a13af2e24ee252ad1a0288b62302e3a666d5e3e5d8fd285b446668"} Oct 31 04:47:51.399410 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:51.397142 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac16efb854a13af2e24ee252ad1a0288b62302e3a666d5e3e5d8fd285b446668" Oct 31 04:47:51.399410 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:51.397220 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b236824d-k5j5w" Oct 31 04:47:52.515864 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:47:52.510874 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:47:52.544499 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:52.544459 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f4752pxg"] Oct 31 04:47:52.545309 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:52.545292 5019 cpu_manager.go:414] "containerMap: removing container" podUID="b578bb3a-b726-4057-a46b-b25e1cb0d767" containerName="revoke-keys" Oct 31 04:47:52.545378 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:52.545311 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b578bb3a-b726-4057-a46b-b25e1cb0d767" containerName="revoke-keys" Oct 31 04:47:52.545478 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:52.545466 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="b578bb3a-b726-4057-a46b-b25e1cb0d767" containerName="revoke-keys" Oct 31 04:47:52.549599 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:52.549573 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f4752pxg" Oct 31 04:47:52.565722 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:52.565680 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-47c409\"/\"agc-cred-f9929e0cf976\"" Oct 31 04:47:52.569588 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:52.569552 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f4752pxg"] Oct 31 04:47:52.582856 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:52.575363 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-47c409\"/\"payload-processing-external-model-e2e--6f807998-dockercfg-9lmtn\"" Oct 31 04:47:52.582856 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:52.575692 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-47c409\"/\"routing-overlay\"" Oct 31 04:47:52.582856 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:52.575743 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-47c409\"/\"kube-root-ca.crt\"" Oct 31 04:47:52.582856 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:52.575791 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-47c409\"/\"payload-processing-external-model-plugins-e2e-extmod-a-47c409\"" Oct 31 04:47:52.582856 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:52.575828 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-47c409\"/\"openshift-service-ca.crt\"" Oct 31 04:47:52.696354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:52.694881 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94055ff8-c29d-4d2b-a22c-917999b46454-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-5f4752pxg\" (UID: \"94055ff8-c29d-4d2b-a22c-917999b46454\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f4752pxg" Oct 31 04:47:52.696354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:52.694992 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/94055ff8-c29d-4d2b-a22c-917999b46454-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-5f4752pxg\" (UID: \"94055ff8-c29d-4d2b-a22c-917999b46454\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f4752pxg" Oct 31 04:47:52.696354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:52.695124 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztdkl\" (UniqueName: \"kubernetes.io/projected/94055ff8-c29d-4d2b-a22c-917999b46454-kube-api-access-ztdkl\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-5f4752pxg\" (UID: \"94055ff8-c29d-4d2b-a22c-917999b46454\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f4752pxg" Oct 31 04:47:52.696354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:52.695204 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/94055ff8-c29d-4d2b-a22c-917999b46454-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-5f4752pxg\" (UID: \"94055ff8-c29d-4d2b-a22c-917999b46454\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f4752pxg" Oct 31 04:47:52.801408 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:52.800470 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94055ff8-c29d-4d2b-a22c-917999b46454-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-5f4752pxg\" (UID: \"94055ff8-c29d-4d2b-a22c-917999b46454\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f4752pxg" Oct 31 04:47:52.801408 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:52.800576 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/94055ff8-c29d-4d2b-a22c-917999b46454-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-5f4752pxg\" (UID: \"94055ff8-c29d-4d2b-a22c-917999b46454\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f4752pxg" Oct 31 04:47:52.801408 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:52.800697 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztdkl\" (UniqueName: \"kubernetes.io/projected/94055ff8-c29d-4d2b-a22c-917999b46454-kube-api-access-ztdkl\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-5f4752pxg\" (UID: \"94055ff8-c29d-4d2b-a22c-917999b46454\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f4752pxg" Oct 31 04:47:52.801408 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:52.800769 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/94055ff8-c29d-4d2b-a22c-917999b46454-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-5f4752pxg\" (UID: \"94055ff8-c29d-4d2b-a22c-917999b46454\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f4752pxg" Oct 31 04:47:52.808499 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:52.802287 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/94055ff8-c29d-4d2b-a22c-917999b46454-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-5f4752pxg\" (UID: \"94055ff8-c29d-4d2b-a22c-917999b46454\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f4752pxg" Oct 31 04:47:52.808499 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:52.803044 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94055ff8-c29d-4d2b-a22c-917999b46454-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-5f4752pxg\" (UID: \"94055ff8-c29d-4d2b-a22c-917999b46454\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f4752pxg" Oct 31 04:47:52.833718 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:52.833681 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/94055ff8-c29d-4d2b-a22c-917999b46454-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-5f4752pxg\" (UID: \"94055ff8-c29d-4d2b-a22c-917999b46454\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f4752pxg" Oct 31 04:47:52.876175 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:52.875664 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztdkl\" (UniqueName: \"kubernetes.io/projected/94055ff8-c29d-4d2b-a22c-917999b46454-kube-api-access-ztdkl\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-5f4752pxg\" (UID: \"94055ff8-c29d-4d2b-a22c-917999b46454\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f4752pxg" Oct 31 04:47:52.899065 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:52.898846 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-648dnwfg6" podUID="5589c835-ca5c-4dd9-938d-3cc2e50f295b" containerName="payload-processing" containerID="cri-o://9fc85b04047eb4c68798defede14b453e32a2f5dae1c76c269b9241757ea29ad" gracePeriod=25 Oct 31 04:47:53.166073 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:53.166038 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f4752pxg" Oct 31 04:47:53.262722 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:53.262284 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-648dnwfg6" Oct 31 04:47:53.330224 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:53.327686 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5589c835-ca5c-4dd9-938d-3cc2e50f295b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5589c835-ca5c-4dd9-938d-3cc2e50f295b-plugins-config-volume\") pod \"5589c835-ca5c-4dd9-938d-3cc2e50f295b\" (UID: \"5589c835-ca5c-4dd9-938d-3cc2e50f295b\") " Oct 31 04:47:53.330224 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:53.327844 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5589c835-ca5c-4dd9-938d-3cc2e50f295b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5589c835-ca5c-4dd9-938d-3cc2e50f295b-provider-credentials\") pod \"5589c835-ca5c-4dd9-938d-3cc2e50f295b\" (UID: \"5589c835-ca5c-4dd9-938d-3cc2e50f295b\") " Oct 31 04:47:53.330224 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:53.327943 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5589c835-ca5c-4dd9-938d-3cc2e50f295b-kube-api-access-gzr5f\" (UniqueName: \"kubernetes.io/projected/5589c835-ca5c-4dd9-938d-3cc2e50f295b-kube-api-access-gzr5f\") pod \"5589c835-ca5c-4dd9-938d-3cc2e50f295b\" (UID: \"5589c835-ca5c-4dd9-938d-3cc2e50f295b\") " Oct 31 04:47:53.330224 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:53.328016 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5589c835-ca5c-4dd9-938d-3cc2e50f295b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5589c835-ca5c-4dd9-938d-3cc2e50f295b-routing-overlay-volume\") pod \"5589c835-ca5c-4dd9-938d-3cc2e50f295b\" (UID: \"5589c835-ca5c-4dd9-938d-3cc2e50f295b\") " Oct 31 04:47:53.333195 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:53.333160 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5589c835-ca5c-4dd9-938d-3cc2e50f295b-routing-overlay-volume" pod "5589c835-ca5c-4dd9-938d-3cc2e50f295b" (UID: "5589c835-ca5c-4dd9-938d-3cc2e50f295b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:47:53.336514 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:53.336471 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5589c835-ca5c-4dd9-938d-3cc2e50f295b-plugins-config-volume" pod "5589c835-ca5c-4dd9-938d-3cc2e50f295b" (UID: "5589c835-ca5c-4dd9-938d-3cc2e50f295b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:47:53.338759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:53.338722 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5589c835-ca5c-4dd9-938d-3cc2e50f295b-kube-api-access-gzr5f" pod "5589c835-ca5c-4dd9-938d-3cc2e50f295b" (UID: "5589c835-ca5c-4dd9-938d-3cc2e50f295b"). InnerVolumeSpecName "kube-api-access-gzr5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:47:53.338877 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:53.338762 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5589c835-ca5c-4dd9-938d-3cc2e50f295b-provider-credentials" pod "5589c835-ca5c-4dd9-938d-3cc2e50f295b" (UID: "5589c835-ca5c-4dd9-938d-3cc2e50f295b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:47:53.431148 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:53.431111 5019 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5589c835-ca5c-4dd9-938d-3cc2e50f295b-provider-credentials\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:47:53.431148 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:53.431147 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzr5f\" (UniqueName: \"kubernetes.io/projected/5589c835-ca5c-4dd9-938d-3cc2e50f295b-kube-api-access-gzr5f\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:47:53.431364 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:53.431157 5019 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5589c835-ca5c-4dd9-938d-3cc2e50f295b-routing-overlay-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:47:53.431364 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:53.431168 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5589c835-ca5c-4dd9-938d-3cc2e50f295b-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:47:53.450797 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:53.450756 5019 generic.go:356] "Generic (PLEG): container finished" podID="5589c835-ca5c-4dd9-938d-3cc2e50f295b" containerID="9fc85b04047eb4c68798defede14b453e32a2f5dae1c76c269b9241757ea29ad" exitCode=0 Oct 31 04:47:53.450944 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:53.450820 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-648dnwfg6" Oct 31 04:47:53.450944 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:53.450842 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-648dnwfg6" event={"ID":"5589c835-ca5c-4dd9-938d-3cc2e50f295b","Type":"ContainerDied","Data":"9fc85b04047eb4c68798defede14b453e32a2f5dae1c76c269b9241757ea29ad"} Oct 31 04:47:53.450944 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:53.450902 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-648dnwfg6" event={"ID":"5589c835-ca5c-4dd9-938d-3cc2e50f295b","Type":"ContainerDied","Data":"693fdc49b09089d08d1c5339dc7166be635a6db5b3512a57532b36e2c5af358a"} Oct 31 04:47:53.450944 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:53.450929 5019 scope.go:122] "RemoveContainer" containerID="9fc85b04047eb4c68798defede14b453e32a2f5dae1c76c269b9241757ea29ad" Oct 31 04:47:53.466780 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:53.466741 5019 scope.go:122] "RemoveContainer" containerID="9fc85b04047eb4c68798defede14b453e32a2f5dae1c76c269b9241757ea29ad" Oct 31 04:47:53.467285 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:47:53.467250 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fc85b04047eb4c68798defede14b453e32a2f5dae1c76c269b9241757ea29ad\": container with ID starting with 9fc85b04047eb4c68798defede14b453e32a2f5dae1c76c269b9241757ea29ad not found: ID does not exist" containerID="9fc85b04047eb4c68798defede14b453e32a2f5dae1c76c269b9241757ea29ad" Oct 31 04:47:53.467506 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:53.467296 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fc85b04047eb4c68798defede14b453e32a2f5dae1c76c269b9241757ea29ad"} err="failed to get container status \"9fc85b04047eb4c68798defede14b453e32a2f5dae1c76c269b9241757ea29ad\": rpc error: code = NotFound desc = could not find container \"9fc85b04047eb4c68798defede14b453e32a2f5dae1c76c269b9241757ea29ad\": container with ID starting with 9fc85b04047eb4c68798defede14b453e32a2f5dae1c76c269b9241757ea29ad not found: ID does not exist" Oct 31 04:47:53.498311 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:53.498276 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f4752pxg"] Oct 31 04:47:53.499415 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:47:53.499316 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94055ff8_c29d_4d2b_a22c_917999b46454.slice/crio-67da0229a284da1f913e8cde9ed4b847f6d3f7f13512baaaa14d3653a1e736d0 WatchSource:0}: Error finding container 67da0229a284da1f913e8cde9ed4b847f6d3f7f13512baaaa14d3653a1e736d0: Status 404 returned error can't find the container with id 67da0229a284da1f913e8cde9ed4b847f6d3f7f13512baaaa14d3653a1e736d0 Oct 31 04:47:53.508352 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:53.508311 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-648dnwfg6"] Oct 31 04:47:53.516633 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:53.516596 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-648dnwfg6"] Oct 31 04:47:54.685357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:54.681648 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f4752pxg" event={"ID":"94055ff8-c29d-4d2b-a22c-917999b46454","Type":"ContainerStarted","Data":"2f4e67e5e1b3afe4df993548ca46198f621d01a37306ee400c4ba4a873be5a58"} Oct 31 04:47:54.685357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:54.681717 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f4752pxg" event={"ID":"94055ff8-c29d-4d2b-a22c-917999b46454","Type":"ContainerStarted","Data":"67da0229a284da1f913e8cde9ed4b847f6d3f7f13512baaaa14d3653a1e736d0"} Oct 31 04:47:54.685357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:54.681780 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f4752pxg" Oct 31 04:47:54.728021 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:54.727951 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f4752pxg" podStartSLOduration=2.72793051 podStartE2EDuration="2.72793051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:47:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:47:54.727455305 +0000 UTC m=+2629.483793346" watchObservedRunningTime="2026-10-31 04:47:54.72793051 +0000 UTC m=+2629.484268550" Oct 31 04:47:54.822702 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:54.822665 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-55776gtzl"] Oct 31 04:47:54.823531 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:54.823514 5019 cpu_manager.go:414] "containerMap: removing container" podUID="5589c835-ca5c-4dd9-938d-3cc2e50f295b" containerName="payload-processing" Oct 31 04:47:54.823607 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:54.823532 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5589c835-ca5c-4dd9-938d-3cc2e50f295b" containerName="payload-processing" Oct 31 04:47:54.823658 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:54.823648 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="5589c835-ca5c-4dd9-938d-3cc2e50f295b" containerName="payload-processing" Oct 31 04:47:54.827730 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:54.827707 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-55776gtzl" Oct 31 04:47:54.860521 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:54.846801 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-654741\"/\"agc-cred-097ec564cb5e\"" Oct 31 04:47:54.860521 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:54.846972 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-654741\"/\"payload-processing-external-model-e2e--dfe12206-dockercfg-mtr6q\"" Oct 31 04:47:54.860521 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:54.847173 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-654741\"/\"payload-processing-external-model-plugins-e2e-extmod-a-654741\"" Oct 31 04:47:54.860521 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:54.847750 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-654741\"/\"kube-root-ca.crt\"" Oct 31 04:47:54.860521 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:54.856463 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-654741\"/\"routing-overlay\"" Oct 31 04:47:54.860521 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:54.856672 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-654741\"/\"openshift-service-ca.crt\"" Oct 31 04:47:54.926968 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:54.926928 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-55776gtzl"] Oct 31 04:47:55.056418 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:55.049922 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/66059d76-7532-4d81-80c6-7581d9af4bbc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-654741-55776gtzl\" (UID: \"66059d76-7532-4d81-80c6-7581d9af4bbc\") " pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-55776gtzl" Oct 31 04:47:55.056862 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:55.056831 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66059d76-7532-4d81-80c6-7581d9af4bbc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-654741-55776gtzl\" (UID: \"66059d76-7532-4d81-80c6-7581d9af4bbc\") " pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-55776gtzl" Oct 31 04:47:55.057098 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:55.057080 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzvbc\" (UniqueName: \"kubernetes.io/projected/66059d76-7532-4d81-80c6-7581d9af4bbc-kube-api-access-fzvbc\") pod \"payload-processing-external-model-e2e-extmod-a-654741-55776gtzl\" (UID: \"66059d76-7532-4d81-80c6-7581d9af4bbc\") " pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-55776gtzl" Oct 31 04:47:55.063379 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:55.057308 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/66059d76-7532-4d81-80c6-7581d9af4bbc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-654741-55776gtzl\" (UID: \"66059d76-7532-4d81-80c6-7581d9af4bbc\") " pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-55776gtzl" Oct 31 04:47:55.165306 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:55.165244 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/66059d76-7532-4d81-80c6-7581d9af4bbc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-654741-55776gtzl\" (UID: \"66059d76-7532-4d81-80c6-7581d9af4bbc\") " pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-55776gtzl" Oct 31 04:47:55.165519 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:55.165314 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66059d76-7532-4d81-80c6-7581d9af4bbc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-654741-55776gtzl\" (UID: \"66059d76-7532-4d81-80c6-7581d9af4bbc\") " pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-55776gtzl" Oct 31 04:47:55.165519 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:55.165414 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzvbc\" (UniqueName: \"kubernetes.io/projected/66059d76-7532-4d81-80c6-7581d9af4bbc-kube-api-access-fzvbc\") pod \"payload-processing-external-model-e2e-extmod-a-654741-55776gtzl\" (UID: \"66059d76-7532-4d81-80c6-7581d9af4bbc\") " pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-55776gtzl" Oct 31 04:47:55.165658 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:55.165514 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/66059d76-7532-4d81-80c6-7581d9af4bbc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-654741-55776gtzl\" (UID: \"66059d76-7532-4d81-80c6-7581d9af4bbc\") " pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-55776gtzl" Oct 31 04:47:55.166891 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:55.166865 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/66059d76-7532-4d81-80c6-7581d9af4bbc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-654741-55776gtzl\" (UID: \"66059d76-7532-4d81-80c6-7581d9af4bbc\") " pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-55776gtzl" Oct 31 04:47:55.167012 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:55.166940 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66059d76-7532-4d81-80c6-7581d9af4bbc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-654741-55776gtzl\" (UID: \"66059d76-7532-4d81-80c6-7581d9af4bbc\") " pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-55776gtzl" Oct 31 04:47:55.168518 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:55.168495 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/66059d76-7532-4d81-80c6-7581d9af4bbc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-654741-55776gtzl\" (UID: \"66059d76-7532-4d81-80c6-7581d9af4bbc\") " pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-55776gtzl" Oct 31 04:47:55.190806 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:55.190770 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzvbc\" (UniqueName: \"kubernetes.io/projected/66059d76-7532-4d81-80c6-7581d9af4bbc-kube-api-access-fzvbc\") pod \"payload-processing-external-model-e2e-extmod-a-654741-55776gtzl\" (UID: \"66059d76-7532-4d81-80c6-7581d9af4bbc\") " pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-55776gtzl" Oct 31 04:47:55.414228 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:55.414127 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5589c835-ca5c-4dd9-938d-3cc2e50f295b" path="/var/lib/kubelet/pods/5589c835-ca5c-4dd9-938d-3cc2e50f295b/volumes" Oct 31 04:47:55.488358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:55.473014 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-55776gtzl" Oct 31 04:47:55.642725 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:55.640221 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80ec86b6-lmk2g"] Oct 31 04:47:55.659674 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:55.659639 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80ec86b6-lmk2g"] Oct 31 04:47:55.659978 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:55.659957 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ec86b6-lmk2g" Oct 31 04:47:55.676276 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:55.674653 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:47:55.676276 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:55.675307 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:47:55.676276 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:55.676103 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:47:55.676276 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:55.676183 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:47:55.779711 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:55.779664 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhwsn\" (UniqueName: \"kubernetes.io/projected/b2b416ac-8e8c-48d6-bf4d-7a70947bdfa0-kube-api-access-qhwsn\") pod \"maas-api-revoke-sub-80ec86b6-lmk2g\" (UID: \"b2b416ac-8e8c-48d6-bf4d-7a70947bdfa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ec86b6-lmk2g" Oct 31 04:47:55.780981 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:55.780822 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2b416ac-8e8c-48d6-bf4d-7a70947bdfa0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80ec86b6-lmk2g\" (UID: \"b2b416ac-8e8c-48d6-bf4d-7a70947bdfa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ec86b6-lmk2g" Oct 31 04:47:55.782571 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:55.782544 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2b416ac-8e8c-48d6-bf4d-7a70947bdfa0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80ec86b6-lmk2g\" (UID: \"b2b416ac-8e8c-48d6-bf4d-7a70947bdfa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ec86b6-lmk2g" Oct 31 04:47:55.902356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:55.896442 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhwsn\" (UniqueName: \"kubernetes.io/projected/b2b416ac-8e8c-48d6-bf4d-7a70947bdfa0-kube-api-access-qhwsn\") pod \"maas-api-revoke-sub-80ec86b6-lmk2g\" (UID: \"b2b416ac-8e8c-48d6-bf4d-7a70947bdfa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ec86b6-lmk2g" Oct 31 04:47:55.902356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:55.896555 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2b416ac-8e8c-48d6-bf4d-7a70947bdfa0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80ec86b6-lmk2g\" (UID: \"b2b416ac-8e8c-48d6-bf4d-7a70947bdfa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ec86b6-lmk2g" Oct 31 04:47:55.902356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:55.900541 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2b416ac-8e8c-48d6-bf4d-7a70947bdfa0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80ec86b6-lmk2g\" (UID: \"b2b416ac-8e8c-48d6-bf4d-7a70947bdfa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ec86b6-lmk2g" Oct 31 04:47:55.915718 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:55.915679 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2b416ac-8e8c-48d6-bf4d-7a70947bdfa0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80ec86b6-lmk2g\" (UID: \"b2b416ac-8e8c-48d6-bf4d-7a70947bdfa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ec86b6-lmk2g" Oct 31 04:47:56.054127 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:56.054088 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2b416ac-8e8c-48d6-bf4d-7a70947bdfa0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80ec86b6-lmk2g\" (UID: \"b2b416ac-8e8c-48d6-bf4d-7a70947bdfa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ec86b6-lmk2g" Oct 31 04:47:56.077153 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:56.077114 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhwsn\" (UniqueName: \"kubernetes.io/projected/b2b416ac-8e8c-48d6-bf4d-7a70947bdfa0-kube-api-access-qhwsn\") pod \"maas-api-revoke-sub-80ec86b6-lmk2g\" (UID: \"b2b416ac-8e8c-48d6-bf4d-7a70947bdfa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ec86b6-lmk2g" Oct 31 04:47:56.099012 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:56.098945 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-55776gtzl"] Oct 31 04:47:56.304385 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:56.303920 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ec86b6-lmk2g" Oct 31 04:47:56.638633 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:56.638573 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80ec86b6-lmk2g"] Oct 31 04:47:56.638787 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:47:56.638628 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2b416ac_8e8c_48d6_bf4d_7a70947bdfa0.slice/crio-2879d123172c1235c2f814e003197a9439c68951e4d17c4683149e63ffaddd35 WatchSource:0}: Error finding container 2879d123172c1235c2f814e003197a9439c68951e4d17c4683149e63ffaddd35: Status 404 returned error can't find the container with id 2879d123172c1235c2f814e003197a9439c68951e4d17c4683149e63ffaddd35 Oct 31 04:47:56.755552 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:56.755497 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-55776gtzl" event={"ID":"66059d76-7532-4d81-80c6-7581d9af4bbc","Type":"ContainerStarted","Data":"1a22ec0034a0ac00f9860a4f5725625e0439899456a4a368aacd13703490e8d2"} Oct 31 04:47:56.755769 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:56.755588 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-55776gtzl" Oct 31 04:47:56.755769 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:56.755613 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-55776gtzl" event={"ID":"66059d76-7532-4d81-80c6-7581d9af4bbc","Type":"ContainerStarted","Data":"b42c07de78243a02340d39b112a5d2edeee0c6ae9ef8926bae385ebb687555e1"} Oct 31 04:47:56.757454 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:56.757421 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ec86b6-lmk2g" event={"ID":"b2b416ac-8e8c-48d6-bf4d-7a70947bdfa0","Type":"ContainerStarted","Data":"2879d123172c1235c2f814e003197a9439c68951e4d17c4683149e63ffaddd35"} Oct 31 04:47:56.799771 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:56.799705 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-55776gtzl" podStartSLOduration=2.7996854669999998 podStartE2EDuration="2.799685467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:47:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:47:56.797592887 +0000 UTC m=+2631.553930940" watchObservedRunningTime="2026-10-31 04:47:56.799685467 +0000 UTC m=+2631.556023507" Oct 31 04:47:57.776709 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:57.776670 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ec86b6-lmk2g" event={"ID":"b2b416ac-8e8c-48d6-bf4d-7a70947bdfa0","Type":"ContainerStarted","Data":"f0e60eca91549acd179bc444543229deb6b81dbae6b68247a4e5bc307d150fd4"} Oct 31 04:47:57.817226 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:57.817167 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ec86b6-lmk2g" podStartSLOduration=2.817153822 podStartE2EDuration="2.817153822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:47:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:47:57.815604238 +0000 UTC m=+2632.571942282" watchObservedRunningTime="2026-10-31 04:47:57.817153822 +0000 UTC m=+2632.573491861" Oct 31 04:47:58.788859 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:58.788820 5019 generic.go:356] "Generic (PLEG): container finished" podID="b2b416ac-8e8c-48d6-bf4d-7a70947bdfa0" containerID="f0e60eca91549acd179bc444543229deb6b81dbae6b68247a4e5bc307d150fd4" exitCode=0 Oct 31 04:47:58.789088 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:47:58.788877 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ec86b6-lmk2g" event={"ID":"b2b416ac-8e8c-48d6-bf4d-7a70947bdfa0","Type":"ContainerDied","Data":"f0e60eca91549acd179bc444543229deb6b81dbae6b68247a4e5bc307d150fd4"} Oct 31 04:48:00.014850 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:00.014815 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ec86b6-lmk2g" Oct 31 04:48:00.121295 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:00.121186 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2b416ac-8e8c-48d6-bf4d-7a70947bdfa0-kube-api-access-qhwsn\" (UniqueName: \"kubernetes.io/projected/b2b416ac-8e8c-48d6-bf4d-7a70947bdfa0-kube-api-access-qhwsn\") pod \"b2b416ac-8e8c-48d6-bf4d-7a70947bdfa0\" (UID: \"b2b416ac-8e8c-48d6-bf4d-7a70947bdfa0\") " Oct 31 04:48:00.121295 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:00.121259 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2b416ac-8e8c-48d6-bf4d-7a70947bdfa0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2b416ac-8e8c-48d6-bf4d-7a70947bdfa0-maas-api-openshift-ca\") pod \"b2b416ac-8e8c-48d6-bf4d-7a70947bdfa0\" (UID: \"b2b416ac-8e8c-48d6-bf4d-7a70947bdfa0\") " Oct 31 04:48:00.121295 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:00.121290 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2b416ac-8e8c-48d6-bf4d-7a70947bdfa0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2b416ac-8e8c-48d6-bf4d-7a70947bdfa0-maas-api-xks-ca\") pod \"b2b416ac-8e8c-48d6-bf4d-7a70947bdfa0\" (UID: \"b2b416ac-8e8c-48d6-bf4d-7a70947bdfa0\") " Oct 31 04:48:00.121711 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:00.121685 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2b416ac-8e8c-48d6-bf4d-7a70947bdfa0-maas-api-openshift-ca" pod "b2b416ac-8e8c-48d6-bf4d-7a70947bdfa0" (UID: "b2b416ac-8e8c-48d6-bf4d-7a70947bdfa0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:48:00.122211 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:00.122186 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2b416ac-8e8c-48d6-bf4d-7a70947bdfa0-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:48:00.123670 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:00.123640 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2b416ac-8e8c-48d6-bf4d-7a70947bdfa0-maas-api-xks-ca" pod "b2b416ac-8e8c-48d6-bf4d-7a70947bdfa0" (UID: "b2b416ac-8e8c-48d6-bf4d-7a70947bdfa0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:48:00.123754 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:00.123721 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2b416ac-8e8c-48d6-bf4d-7a70947bdfa0-kube-api-access-qhwsn" pod "b2b416ac-8e8c-48d6-bf4d-7a70947bdfa0" (UID: "b2b416ac-8e8c-48d6-bf4d-7a70947bdfa0"). InnerVolumeSpecName "kube-api-access-qhwsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:48:00.223580 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:00.223538 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhwsn\" (UniqueName: \"kubernetes.io/projected/b2b416ac-8e8c-48d6-bf4d-7a70947bdfa0-kube-api-access-qhwsn\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:48:00.223580 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:00.223577 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2b416ac-8e8c-48d6-bf4d-7a70947bdfa0-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:48:00.598259 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:00.598218 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-697d4911-tk4xj"] Oct 31 04:48:00.599544 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:00.599512 5019 cpu_manager.go:414] "containerMap: removing container" podUID="b2b416ac-8e8c-48d6-bf4d-7a70947bdfa0" containerName="revoke-keys" Oct 31 04:48:00.599544 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:00.599542 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2b416ac-8e8c-48d6-bf4d-7a70947bdfa0" containerName="revoke-keys" Oct 31 04:48:00.599836 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:00.599818 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2b416ac-8e8c-48d6-bf4d-7a70947bdfa0" containerName="revoke-keys" Oct 31 04:48:00.604824 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:00.604801 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-697d4911-tk4xj" Oct 31 04:48:00.623555 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:00.623524 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-697d4911-tk4xj"] Oct 31 04:48:00.639844 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:00.639798 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/022f2eaf-4d8a-4d8f-82d0-3906e02a7b5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-697d4911-tk4xj\" (UID: \"022f2eaf-4d8a-4d8f-82d0-3906e02a7b5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-697d4911-tk4xj" Oct 31 04:48:00.640270 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:00.640242 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfnxz\" (UniqueName: \"kubernetes.io/projected/022f2eaf-4d8a-4d8f-82d0-3906e02a7b5a-kube-api-access-gfnxz\") pod \"maas-api-revoke-sub-697d4911-tk4xj\" (UID: \"022f2eaf-4d8a-4d8f-82d0-3906e02a7b5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-697d4911-tk4xj" Oct 31 04:48:00.640409 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:00.640291 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/022f2eaf-4d8a-4d8f-82d0-3906e02a7b5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-697d4911-tk4xj\" (UID: \"022f2eaf-4d8a-4d8f-82d0-3906e02a7b5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-697d4911-tk4xj" Oct 31 04:48:00.742068 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:00.742018 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfnxz\" (UniqueName: \"kubernetes.io/projected/022f2eaf-4d8a-4d8f-82d0-3906e02a7b5a-kube-api-access-gfnxz\") pod \"maas-api-revoke-sub-697d4911-tk4xj\" (UID: \"022f2eaf-4d8a-4d8f-82d0-3906e02a7b5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-697d4911-tk4xj" Oct 31 04:48:00.742288 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:00.742076 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/022f2eaf-4d8a-4d8f-82d0-3906e02a7b5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-697d4911-tk4xj\" (UID: \"022f2eaf-4d8a-4d8f-82d0-3906e02a7b5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-697d4911-tk4xj" Oct 31 04:48:00.742288 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:00.742249 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/022f2eaf-4d8a-4d8f-82d0-3906e02a7b5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-697d4911-tk4xj\" (UID: \"022f2eaf-4d8a-4d8f-82d0-3906e02a7b5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-697d4911-tk4xj" Oct 31 04:48:00.742915 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:00.742893 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/022f2eaf-4d8a-4d8f-82d0-3906e02a7b5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-697d4911-tk4xj\" (UID: \"022f2eaf-4d8a-4d8f-82d0-3906e02a7b5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-697d4911-tk4xj" Oct 31 04:48:00.744597 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:00.744572 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/022f2eaf-4d8a-4d8f-82d0-3906e02a7b5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-697d4911-tk4xj\" (UID: \"022f2eaf-4d8a-4d8f-82d0-3906e02a7b5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-697d4911-tk4xj" Oct 31 04:48:00.765402 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:00.765370 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfnxz\" (UniqueName: \"kubernetes.io/projected/022f2eaf-4d8a-4d8f-82d0-3906e02a7b5a-kube-api-access-gfnxz\") pod \"maas-api-revoke-sub-697d4911-tk4xj\" (UID: \"022f2eaf-4d8a-4d8f-82d0-3906e02a7b5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-697d4911-tk4xj" Oct 31 04:48:00.792974 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:00.792931 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f4752pxg"] Oct 31 04:48:00.796151 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:00.796122 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f4752pxg" Oct 31 04:48:00.815916 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:00.815875 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ec86b6-lmk2g" Oct 31 04:48:00.816778 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:00.815878 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ec86b6-lmk2g" event={"ID":"b2b416ac-8e8c-48d6-bf4d-7a70947bdfa0","Type":"ContainerDied","Data":"2879d123172c1235c2f814e003197a9439c68951e4d17c4683149e63ffaddd35"} Oct 31 04:48:00.816778 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:00.816749 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2879d123172c1235c2f814e003197a9439c68951e4d17c4683149e63ffaddd35" Oct 31 04:48:00.926548 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:00.926466 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-697d4911-tk4xj" Oct 31 04:48:01.400129 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:48:01.400093 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod022f2eaf_4d8a_4d8f_82d0_3906e02a7b5a.slice/crio-a32494f76217180a856951f67e7d9f4a28edc8a8418293194ab38d92ffcd352b WatchSource:0}: Error finding container a32494f76217180a856951f67e7d9f4a28edc8a8418293194ab38d92ffcd352b: Status 404 returned error can't find the container with id a32494f76217180a856951f67e7d9f4a28edc8a8418293194ab38d92ffcd352b Oct 31 04:48:01.412854 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:01.412803 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-697d4911-tk4xj"] Oct 31 04:48:01.827790 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:01.827742 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-697d4911-tk4xj" event={"ID":"022f2eaf-4d8a-4d8f-82d0-3906e02a7b5a","Type":"ContainerStarted","Data":"bec1244edceaed40eda78054330cac8bf6a2bb0f14d2b8b05b36759f2a73b94f"} Oct 31 04:48:01.827984 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:01.827809 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-697d4911-tk4xj" event={"ID":"022f2eaf-4d8a-4d8f-82d0-3906e02a7b5a","Type":"ContainerStarted","Data":"a32494f76217180a856951f67e7d9f4a28edc8a8418293194ab38d92ffcd352b"} Oct 31 04:48:01.870076 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:01.870009 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-697d4911-tk4xj" podStartSLOduration=1.869993731 podStartE2EDuration="1.869993731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:48:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:48:01.867157772 +0000 UTC m=+2636.623495813" watchObservedRunningTime="2026-10-31 04:48:01.869993731 +0000 UTC m=+2636.626331770" Oct 31 04:48:02.841665 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:02.841618 5019 generic.go:356] "Generic (PLEG): container finished" podID="022f2eaf-4d8a-4d8f-82d0-3906e02a7b5a" containerID="bec1244edceaed40eda78054330cac8bf6a2bb0f14d2b8b05b36759f2a73b94f" exitCode=0 Oct 31 04:48:02.842146 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:02.841695 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-697d4911-tk4xj" event={"ID":"022f2eaf-4d8a-4d8f-82d0-3906e02a7b5a","Type":"ContainerDied","Data":"bec1244edceaed40eda78054330cac8bf6a2bb0f14d2b8b05b36759f2a73b94f"} Oct 31 04:48:04.080812 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:04.080784 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-697d4911-tk4xj" Oct 31 04:48:04.128576 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:04.128510 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/022f2eaf-4d8a-4d8f-82d0-3906e02a7b5a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/022f2eaf-4d8a-4d8f-82d0-3906e02a7b5a-maas-api-xks-ca\") pod \"022f2eaf-4d8a-4d8f-82d0-3906e02a7b5a\" (UID: \"022f2eaf-4d8a-4d8f-82d0-3906e02a7b5a\") " Oct 31 04:48:04.128751 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:04.128612 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/022f2eaf-4d8a-4d8f-82d0-3906e02a7b5a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/022f2eaf-4d8a-4d8f-82d0-3906e02a7b5a-maas-api-openshift-ca\") pod \"022f2eaf-4d8a-4d8f-82d0-3906e02a7b5a\" (UID: \"022f2eaf-4d8a-4d8f-82d0-3906e02a7b5a\") " Oct 31 04:48:04.128917 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:04.128896 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/022f2eaf-4d8a-4d8f-82d0-3906e02a7b5a-kube-api-access-gfnxz\" (UniqueName: \"kubernetes.io/projected/022f2eaf-4d8a-4d8f-82d0-3906e02a7b5a-kube-api-access-gfnxz\") pod \"022f2eaf-4d8a-4d8f-82d0-3906e02a7b5a\" (UID: \"022f2eaf-4d8a-4d8f-82d0-3906e02a7b5a\") " Oct 31 04:48:04.129019 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:04.128994 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/022f2eaf-4d8a-4d8f-82d0-3906e02a7b5a-maas-api-openshift-ca" pod "022f2eaf-4d8a-4d8f-82d0-3906e02a7b5a" (UID: "022f2eaf-4d8a-4d8f-82d0-3906e02a7b5a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:48:04.129829 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:04.129803 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/022f2eaf-4d8a-4d8f-82d0-3906e02a7b5a-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:48:04.130950 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:04.130926 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/022f2eaf-4d8a-4d8f-82d0-3906e02a7b5a-maas-api-xks-ca" pod "022f2eaf-4d8a-4d8f-82d0-3906e02a7b5a" (UID: "022f2eaf-4d8a-4d8f-82d0-3906e02a7b5a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:48:04.131032 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:04.130994 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/022f2eaf-4d8a-4d8f-82d0-3906e02a7b5a-kube-api-access-gfnxz" pod "022f2eaf-4d8a-4d8f-82d0-3906e02a7b5a" (UID: "022f2eaf-4d8a-4d8f-82d0-3906e02a7b5a"). InnerVolumeSpecName "kube-api-access-gfnxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:48:04.231109 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:04.231055 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfnxz\" (UniqueName: \"kubernetes.io/projected/022f2eaf-4d8a-4d8f-82d0-3906e02a7b5a-kube-api-access-gfnxz\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:48:04.231109 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:04.231099 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/022f2eaf-4d8a-4d8f-82d0-3906e02a7b5a-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:48:04.899485 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:04.899449 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-697d4911-tk4xj" Oct 31 04:48:04.899690 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:04.899487 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-697d4911-tk4xj" event={"ID":"022f2eaf-4d8a-4d8f-82d0-3906e02a7b5a","Type":"ContainerDied","Data":"a32494f76217180a856951f67e7d9f4a28edc8a8418293194ab38d92ffcd352b"} Oct 31 04:48:04.899690 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:04.899551 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a32494f76217180a856951f67e7d9f4a28edc8a8418293194ab38d92ffcd352b" Oct 31 04:48:05.793493 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:05.793445 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f4752pxg" podUID="94055ff8-c29d-4d2b-a22c-917999b46454" containerName="payload-processing" containerID="cri-o://2f4e67e5e1b3afe4df993548ca46198f621d01a37306ee400c4ba4a873be5a58" gracePeriod=25 Oct 31 04:48:05.912999 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:05.912968 5019 generic.go:356] "Generic (PLEG): container finished" podID="94055ff8-c29d-4d2b-a22c-917999b46454" containerID="2f4e67e5e1b3afe4df993548ca46198f621d01a37306ee400c4ba4a873be5a58" exitCode=0 Oct 31 04:48:05.913157 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:05.913040 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f4752pxg" event={"ID":"94055ff8-c29d-4d2b-a22c-917999b46454","Type":"ContainerDied","Data":"2f4e67e5e1b3afe4df993548ca46198f621d01a37306ee400c4ba4a873be5a58"} Oct 31 04:48:06.097011 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:06.096982 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f4752pxg" Oct 31 04:48:06.162531 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:06.162495 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94055ff8-c29d-4d2b-a22c-917999b46454-kube-api-access-ztdkl\" (UniqueName: \"kubernetes.io/projected/94055ff8-c29d-4d2b-a22c-917999b46454-kube-api-access-ztdkl\") pod \"94055ff8-c29d-4d2b-a22c-917999b46454\" (UID: \"94055ff8-c29d-4d2b-a22c-917999b46454\") " Oct 31 04:48:06.162740 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:06.162589 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94055ff8-c29d-4d2b-a22c-917999b46454-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94055ff8-c29d-4d2b-a22c-917999b46454-plugins-config-volume\") pod \"94055ff8-c29d-4d2b-a22c-917999b46454\" (UID: \"94055ff8-c29d-4d2b-a22c-917999b46454\") " Oct 31 04:48:06.162740 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:06.162651 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94055ff8-c29d-4d2b-a22c-917999b46454-provider-credentials\" (UniqueName: \"kubernetes.io/projected/94055ff8-c29d-4d2b-a22c-917999b46454-provider-credentials\") pod \"94055ff8-c29d-4d2b-a22c-917999b46454\" (UID: \"94055ff8-c29d-4d2b-a22c-917999b46454\") " Oct 31 04:48:06.162881 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:06.162776 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94055ff8-c29d-4d2b-a22c-917999b46454-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/94055ff8-c29d-4d2b-a22c-917999b46454-routing-overlay-volume\") pod \"94055ff8-c29d-4d2b-a22c-917999b46454\" (UID: \"94055ff8-c29d-4d2b-a22c-917999b46454\") " Oct 31 04:48:06.163545 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:06.163510 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94055ff8-c29d-4d2b-a22c-917999b46454-plugins-config-volume" pod "94055ff8-c29d-4d2b-a22c-917999b46454" (UID: "94055ff8-c29d-4d2b-a22c-917999b46454"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:48:06.163545 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:06.163523 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94055ff8-c29d-4d2b-a22c-917999b46454-routing-overlay-volume" pod "94055ff8-c29d-4d2b-a22c-917999b46454" (UID: "94055ff8-c29d-4d2b-a22c-917999b46454"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:48:06.165954 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:06.165929 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94055ff8-c29d-4d2b-a22c-917999b46454-provider-credentials" pod "94055ff8-c29d-4d2b-a22c-917999b46454" (UID: "94055ff8-c29d-4d2b-a22c-917999b46454"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:48:06.168845 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:06.168820 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94055ff8-c29d-4d2b-a22c-917999b46454-kube-api-access-ztdkl" pod "94055ff8-c29d-4d2b-a22c-917999b46454" (UID: "94055ff8-c29d-4d2b-a22c-917999b46454"). InnerVolumeSpecName "kube-api-access-ztdkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:48:06.264943 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:06.264833 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94055ff8-c29d-4d2b-a22c-917999b46454-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:48:06.264943 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:06.264879 5019 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/94055ff8-c29d-4d2b-a22c-917999b46454-provider-credentials\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:48:06.264943 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:06.264895 5019 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/94055ff8-c29d-4d2b-a22c-917999b46454-routing-overlay-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:48:06.264943 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:06.264904 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztdkl\" (UniqueName: \"kubernetes.io/projected/94055ff8-c29d-4d2b-a22c-917999b46454-kube-api-access-ztdkl\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:48:06.927242 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:06.927191 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f4752pxg" event={"ID":"94055ff8-c29d-4d2b-a22c-917999b46454","Type":"ContainerDied","Data":"67da0229a284da1f913e8cde9ed4b847f6d3f7f13512baaaa14d3653a1e736d0"} Oct 31 04:48:06.927781 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:06.927276 5019 scope.go:122] "RemoveContainer" containerID="2f4e67e5e1b3afe4df993548ca46198f621d01a37306ee400c4ba4a873be5a58" Oct 31 04:48:06.927781 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:06.927475 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f4752pxg" Oct 31 04:48:06.991891 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:06.991853 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f4752pxg"] Oct 31 04:48:07.002948 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:07.002914 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f4752pxg"] Oct 31 04:48:07.411842 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:07.411810 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94055ff8-c29d-4d2b-a22c-917999b46454" path="/var/lib/kubelet/pods/94055ff8-c29d-4d2b-a22c-917999b46454/volumes" Oct 31 04:48:07.500856 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:07.500820 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7ttlj5"] Oct 31 04:48:07.501677 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:07.501650 5019 cpu_manager.go:414] "containerMap: removing container" podUID="022f2eaf-4d8a-4d8f-82d0-3906e02a7b5a" containerName="revoke-keys" Oct 31 04:48:07.501677 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:07.501674 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="022f2eaf-4d8a-4d8f-82d0-3906e02a7b5a" containerName="revoke-keys" Oct 31 04:48:07.501870 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:07.501713 5019 cpu_manager.go:414] "containerMap: removing container" podUID="94055ff8-c29d-4d2b-a22c-917999b46454" containerName="payload-processing" Oct 31 04:48:07.501870 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:07.501723 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94055ff8-c29d-4d2b-a22c-917999b46454" containerName="payload-processing" Oct 31 04:48:07.501870 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:07.501860 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="022f2eaf-4d8a-4d8f-82d0-3906e02a7b5a" containerName="revoke-keys" Oct 31 04:48:07.501982 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:07.501873 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="94055ff8-c29d-4d2b-a22c-917999b46454" containerName="payload-processing" Oct 31 04:48:07.505920 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:07.505899 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7ttlj5" Oct 31 04:48:07.521986 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:07.520630 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8973e2\"/\"agc-cred-fcdfce9e22b3\"" Oct 31 04:48:07.531030 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:07.530997 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8973e2\"/\"payload-processing-external-model-e2e--508e01fb-dockercfg-f7kv4\"" Oct 31 04:48:07.531255 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:07.531240 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8973e2\"/\"agc-cred-055b438a3215\"" Oct 31 04:48:07.531515 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:07.531500 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8973e2\"/\"kube-root-ca.crt\"" Oct 31 04:48:07.532554 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:07.532528 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8973e2\"/\"openshift-service-ca.crt\"" Oct 31 04:48:07.532671 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:07.532590 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8973e2\"/\"agc-cred-cc70fa128d1a\"" Oct 31 04:48:07.532671 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:07.532662 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8973e2\"/\"payload-processing-external-model-plugins-e2e-extmod-a-8973e2\"" Oct 31 04:48:07.532789 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:07.532759 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8973e2\"/\"routing-overlay\"" Oct 31 04:48:07.545340 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:07.545308 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7ttlj5"] Oct 31 04:48:07.650354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:07.647550 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbttq\" (UniqueName: \"kubernetes.io/projected/a60e5db6-a9bf-40c1-9077-c3dd6b5272db-kube-api-access-lbttq\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-67b7ttlj5\" (UID: \"a60e5db6-a9bf-40c1-9077-c3dd6b5272db\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7ttlj5" Oct 31 04:48:07.650354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:07.647622 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a60e5db6-a9bf-40c1-9077-c3dd6b5272db-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-67b7ttlj5\" (UID: \"a60e5db6-a9bf-40c1-9077-c3dd6b5272db\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7ttlj5" Oct 31 04:48:07.650354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:07.647657 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a60e5db6-a9bf-40c1-9077-c3dd6b5272db-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-67b7ttlj5\" (UID: \"a60e5db6-a9bf-40c1-9077-c3dd6b5272db\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7ttlj5" Oct 31 04:48:07.650354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:07.647727 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a60e5db6-a9bf-40c1-9077-c3dd6b5272db-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-67b7ttlj5\" (UID: \"a60e5db6-a9bf-40c1-9077-c3dd6b5272db\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7ttlj5" Oct 31 04:48:07.751353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:07.746367 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c6e458a-jtds8"] Oct 31 04:48:07.784249 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:07.777990 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbttq\" (UniqueName: \"kubernetes.io/projected/a60e5db6-a9bf-40c1-9077-c3dd6b5272db-kube-api-access-lbttq\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-67b7ttlj5\" (UID: \"a60e5db6-a9bf-40c1-9077-c3dd6b5272db\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7ttlj5" Oct 31 04:48:07.795813 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:07.795769 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a60e5db6-a9bf-40c1-9077-c3dd6b5272db-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-67b7ttlj5\" (UID: \"a60e5db6-a9bf-40c1-9077-c3dd6b5272db\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7ttlj5" Oct 31 04:48:07.796902 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:07.796866 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a60e5db6-a9bf-40c1-9077-c3dd6b5272db-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-67b7ttlj5\" (UID: \"a60e5db6-a9bf-40c1-9077-c3dd6b5272db\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7ttlj5" Oct 31 04:48:07.797172 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:07.797153 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a60e5db6-a9bf-40c1-9077-c3dd6b5272db-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-67b7ttlj5\" (UID: \"a60e5db6-a9bf-40c1-9077-c3dd6b5272db\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7ttlj5" Oct 31 04:48:07.804414 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:07.798102 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a60e5db6-a9bf-40c1-9077-c3dd6b5272db-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-67b7ttlj5\" (UID: \"a60e5db6-a9bf-40c1-9077-c3dd6b5272db\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7ttlj5" Oct 31 04:48:07.805160 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:07.796798 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a60e5db6-a9bf-40c1-9077-c3dd6b5272db-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-67b7ttlj5\" (UID: \"a60e5db6-a9bf-40c1-9077-c3dd6b5272db\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7ttlj5" Oct 31 04:48:07.862509 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:07.862352 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a60e5db6-a9bf-40c1-9077-c3dd6b5272db-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-67b7ttlj5\" (UID: \"a60e5db6-a9bf-40c1-9077-c3dd6b5272db\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7ttlj5" Oct 31 04:48:07.863748 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:07.863676 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbttq\" (UniqueName: \"kubernetes.io/projected/a60e5db6-a9bf-40c1-9077-c3dd6b5272db-kube-api-access-lbttq\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-67b7ttlj5\" (UID: \"a60e5db6-a9bf-40c1-9077-c3dd6b5272db\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7ttlj5" Oct 31 04:48:07.871262 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:07.871234 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-55776gtzl" Oct 31 04:48:07.871826 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:07.871805 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c6e458a-jtds8"] Oct 31 04:48:07.872078 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:07.871313 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c6e458a-jtds8" Oct 31 04:48:07.892826 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:07.892792 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:48:07.894417 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:07.893121 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:48:07.894417 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:07.894139 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:48:07.894624 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:07.894237 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:48:07.907582 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:07.907544 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpv8m\" (UniqueName: \"kubernetes.io/projected/7f259ab0-453f-4c0b-97e2-2d511cced150-kube-api-access-hpv8m\") pod \"maas-api-revoke-sub-9c6e458a-jtds8\" (UID: \"7f259ab0-453f-4c0b-97e2-2d511cced150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c6e458a-jtds8" Oct 31 04:48:07.907915 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:07.907888 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f259ab0-453f-4c0b-97e2-2d511cced150-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c6e458a-jtds8\" (UID: \"7f259ab0-453f-4c0b-97e2-2d511cced150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c6e458a-jtds8" Oct 31 04:48:07.908079 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:07.908059 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f259ab0-453f-4c0b-97e2-2d511cced150-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c6e458a-jtds8\" (UID: \"7f259ab0-453f-4c0b-97e2-2d511cced150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c6e458a-jtds8" Oct 31 04:48:08.010101 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:08.010012 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f259ab0-453f-4c0b-97e2-2d511cced150-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c6e458a-jtds8\" (UID: \"7f259ab0-453f-4c0b-97e2-2d511cced150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c6e458a-jtds8" Oct 31 04:48:08.010597 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:08.010128 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpv8m\" (UniqueName: \"kubernetes.io/projected/7f259ab0-453f-4c0b-97e2-2d511cced150-kube-api-access-hpv8m\") pod \"maas-api-revoke-sub-9c6e458a-jtds8\" (UID: \"7f259ab0-453f-4c0b-97e2-2d511cced150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c6e458a-jtds8" Oct 31 04:48:08.010597 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:08.010276 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f259ab0-453f-4c0b-97e2-2d511cced150-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c6e458a-jtds8\" (UID: \"7f259ab0-453f-4c0b-97e2-2d511cced150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c6e458a-jtds8" Oct 31 04:48:08.010985 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:08.010964 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f259ab0-453f-4c0b-97e2-2d511cced150-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c6e458a-jtds8\" (UID: \"7f259ab0-453f-4c0b-97e2-2d511cced150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c6e458a-jtds8" Oct 31 04:48:08.012481 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:08.012462 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f259ab0-453f-4c0b-97e2-2d511cced150-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c6e458a-jtds8\" (UID: \"7f259ab0-453f-4c0b-97e2-2d511cced150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c6e458a-jtds8" Oct 31 04:48:08.048870 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:08.048841 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpv8m\" (UniqueName: \"kubernetes.io/projected/7f259ab0-453f-4c0b-97e2-2d511cced150-kube-api-access-hpv8m\") pod \"maas-api-revoke-sub-9c6e458a-jtds8\" (UID: \"7f259ab0-453f-4c0b-97e2-2d511cced150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c6e458a-jtds8" Oct 31 04:48:08.135843 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:08.135799 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7ttlj5" Oct 31 04:48:08.207354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:08.203459 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c6e458a-jtds8" Oct 31 04:48:08.449342 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:48:08.449307 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda60e5db6_a9bf_40c1_9077_c3dd6b5272db.slice/crio-fb4691e88e827956ccb537fddacfac2ea2b34c80905944700376a2e7d873753b WatchSource:0}: Error finding container fb4691e88e827956ccb537fddacfac2ea2b34c80905944700376a2e7d873753b: Status 404 returned error can't find the container with id fb4691e88e827956ccb537fddacfac2ea2b34c80905944700376a2e7d873753b Oct 31 04:48:08.451800 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:08.451771 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7ttlj5"] Oct 31 04:48:08.549525 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:48:08.549484 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f259ab0_453f_4c0b_97e2_2d511cced150.slice/crio-8f50cf9783dcd4b81ade71d290ba9aa5b86f59403572338465c5e8598d39b37e WatchSource:0}: Error finding container 8f50cf9783dcd4b81ade71d290ba9aa5b86f59403572338465c5e8598d39b37e: Status 404 returned error can't find the container with id 8f50cf9783dcd4b81ade71d290ba9aa5b86f59403572338465c5e8598d39b37e Oct 31 04:48:08.553542 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:08.553517 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c6e458a-jtds8"] Oct 31 04:48:08.952757 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:08.952717 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7ttlj5" event={"ID":"a60e5db6-a9bf-40c1-9077-c3dd6b5272db","Type":"ContainerStarted","Data":"633ac3376ef27c59c9f5c91c07d14d52d8030fcf427fbc907632a4badb9947b2"} Oct 31 04:48:08.952961 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:08.952781 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7ttlj5" event={"ID":"a60e5db6-a9bf-40c1-9077-c3dd6b5272db","Type":"ContainerStarted","Data":"fb4691e88e827956ccb537fddacfac2ea2b34c80905944700376a2e7d873753b"} Oct 31 04:48:08.952961 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:08.952809 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7ttlj5" Oct 31 04:48:08.954311 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:08.954283 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c6e458a-jtds8" event={"ID":"7f259ab0-453f-4c0b-97e2-2d511cced150","Type":"ContainerStarted","Data":"8f50cf9783dcd4b81ade71d290ba9aa5b86f59403572338465c5e8598d39b37e"} Oct 31 04:48:08.998416 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:08.998367 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7ttlj5" podStartSLOduration=1.998352369 podStartE2EDuration="1.998352369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:48:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:48:08.993075581 +0000 UTC m=+2643.749413632" watchObservedRunningTime="2026-10-31 04:48:08.998352369 +0000 UTC m=+2643.754690406" Oct 31 04:48:09.968467 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:09.968429 5019 generic.go:356] "Generic (PLEG): container finished" podID="7f259ab0-453f-4c0b-97e2-2d511cced150" containerID="20d8a7a15abeca2fac23bb5cf5e7c7a2f4ff001e9eae76e0a529a0f0d7491f36" exitCode=0 Oct 31 04:48:09.968891 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:09.968510 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c6e458a-jtds8" event={"ID":"7f259ab0-453f-4c0b-97e2-2d511cced150","Type":"ContainerDied","Data":"20d8a7a15abeca2fac23bb5cf5e7c7a2f4ff001e9eae76e0a529a0f0d7491f36"} Oct 31 04:48:11.172038 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:11.172005 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c6e458a-jtds8" Oct 31 04:48:11.262564 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:11.262523 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f259ab0-453f-4c0b-97e2-2d511cced150-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f259ab0-453f-4c0b-97e2-2d511cced150-maas-api-xks-ca\") pod \"7f259ab0-453f-4c0b-97e2-2d511cced150\" (UID: \"7f259ab0-453f-4c0b-97e2-2d511cced150\") " Oct 31 04:48:11.262809 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:11.262631 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f259ab0-453f-4c0b-97e2-2d511cced150-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f259ab0-453f-4c0b-97e2-2d511cced150-maas-api-openshift-ca\") pod \"7f259ab0-453f-4c0b-97e2-2d511cced150\" (UID: \"7f259ab0-453f-4c0b-97e2-2d511cced150\") " Oct 31 04:48:11.262809 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:11.262665 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f259ab0-453f-4c0b-97e2-2d511cced150-kube-api-access-hpv8m\" (UniqueName: \"kubernetes.io/projected/7f259ab0-453f-4c0b-97e2-2d511cced150-kube-api-access-hpv8m\") pod \"7f259ab0-453f-4c0b-97e2-2d511cced150\" (UID: \"7f259ab0-453f-4c0b-97e2-2d511cced150\") " Oct 31 04:48:11.263499 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:11.263465 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f259ab0-453f-4c0b-97e2-2d511cced150-maas-api-openshift-ca" pod "7f259ab0-453f-4c0b-97e2-2d511cced150" (UID: "7f259ab0-453f-4c0b-97e2-2d511cced150"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:48:11.263713 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:11.263669 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f259ab0-453f-4c0b-97e2-2d511cced150-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:48:11.265317 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:11.265286 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f259ab0-453f-4c0b-97e2-2d511cced150-maas-api-xks-ca" pod "7f259ab0-453f-4c0b-97e2-2d511cced150" (UID: "7f259ab0-453f-4c0b-97e2-2d511cced150"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:48:11.265512 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:11.265490 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f259ab0-453f-4c0b-97e2-2d511cced150-kube-api-access-hpv8m" pod "7f259ab0-453f-4c0b-97e2-2d511cced150" (UID: "7f259ab0-453f-4c0b-97e2-2d511cced150"). InnerVolumeSpecName "kube-api-access-hpv8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:48:11.364996 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:11.364893 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f259ab0-453f-4c0b-97e2-2d511cced150-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:48:11.364996 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:11.364940 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpv8m\" (UniqueName: \"kubernetes.io/projected/7f259ab0-453f-4c0b-97e2-2d511cced150-kube-api-access-hpv8m\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:48:11.989935 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:11.989895 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c6e458a-jtds8" event={"ID":"7f259ab0-453f-4c0b-97e2-2d511cced150","Type":"ContainerDied","Data":"8f50cf9783dcd4b81ade71d290ba9aa5b86f59403572338465c5e8598d39b37e"} Oct 31 04:48:11.990165 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:11.989906 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c6e458a-jtds8" Oct 31 04:48:11.990165 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:11.990045 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f50cf9783dcd4b81ade71d290ba9aa5b86f59403572338465c5e8598d39b37e" Oct 31 04:48:12.034640 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:12.034596 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-5986pclf5"] Oct 31 04:48:17.035620 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:17.035515 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-5986pclf5" podUID="838a111f-a040-487f-be53-9bdae5e3b36c" containerName="payload-processing" containerID="cri-o://88ab75cff536a14eb4fe8c15faf9e227ccc2c6c47533b95ff8e3c48dfc5b305e" gracePeriod=25 Oct 31 04:48:17.361155 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:17.361125 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-5986pclf5" Oct 31 04:48:17.535025 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:17.534992 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/838a111f-a040-487f-be53-9bdae5e3b36c-kube-api-access-4m9j9\" (UniqueName: \"kubernetes.io/projected/838a111f-a040-487f-be53-9bdae5e3b36c-kube-api-access-4m9j9\") pod \"838a111f-a040-487f-be53-9bdae5e3b36c\" (UID: \"838a111f-a040-487f-be53-9bdae5e3b36c\") " Oct 31 04:48:17.535229 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:17.535215 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/838a111f-a040-487f-be53-9bdae5e3b36c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/838a111f-a040-487f-be53-9bdae5e3b36c-provider-credentials\") pod \"838a111f-a040-487f-be53-9bdae5e3b36c\" (UID: \"838a111f-a040-487f-be53-9bdae5e3b36c\") " Oct 31 04:48:17.535271 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:17.535260 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/838a111f-a040-487f-be53-9bdae5e3b36c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/838a111f-a040-487f-be53-9bdae5e3b36c-plugins-config-volume\") pod \"838a111f-a040-487f-be53-9bdae5e3b36c\" (UID: \"838a111f-a040-487f-be53-9bdae5e3b36c\") " Oct 31 04:48:17.535451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:17.535434 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/838a111f-a040-487f-be53-9bdae5e3b36c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/838a111f-a040-487f-be53-9bdae5e3b36c-routing-overlay-volume\") pod \"838a111f-a040-487f-be53-9bdae5e3b36c\" (UID: \"838a111f-a040-487f-be53-9bdae5e3b36c\") " Oct 31 04:48:17.535818 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:17.535775 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/838a111f-a040-487f-be53-9bdae5e3b36c-plugins-config-volume" pod "838a111f-a040-487f-be53-9bdae5e3b36c" (UID: "838a111f-a040-487f-be53-9bdae5e3b36c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:48:17.535939 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:17.535863 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/838a111f-a040-487f-be53-9bdae5e3b36c-routing-overlay-volume" pod "838a111f-a040-487f-be53-9bdae5e3b36c" (UID: "838a111f-a040-487f-be53-9bdae5e3b36c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:48:17.536376 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:17.536325 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/838a111f-a040-487f-be53-9bdae5e3b36c-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:48:17.536443 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:17.536381 5019 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/838a111f-a040-487f-be53-9bdae5e3b36c-routing-overlay-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:48:17.537671 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:17.537648 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/838a111f-a040-487f-be53-9bdae5e3b36c-provider-credentials" pod "838a111f-a040-487f-be53-9bdae5e3b36c" (UID: "838a111f-a040-487f-be53-9bdae5e3b36c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:48:17.537738 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:17.537682 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/838a111f-a040-487f-be53-9bdae5e3b36c-kube-api-access-4m9j9" pod "838a111f-a040-487f-be53-9bdae5e3b36c" (UID: "838a111f-a040-487f-be53-9bdae5e3b36c"). InnerVolumeSpecName "kube-api-access-4m9j9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:48:17.638650 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:17.638552 5019 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/838a111f-a040-487f-be53-9bdae5e3b36c-provider-credentials\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:48:17.638650 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:17.638588 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4m9j9\" (UniqueName: \"kubernetes.io/projected/838a111f-a040-487f-be53-9bdae5e3b36c-kube-api-access-4m9j9\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:48:18.058856 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:18.058823 5019 generic.go:356] "Generic (PLEG): container finished" podID="838a111f-a040-487f-be53-9bdae5e3b36c" containerID="88ab75cff536a14eb4fe8c15faf9e227ccc2c6c47533b95ff8e3c48dfc5b305e" exitCode=0 Oct 31 04:48:18.059273 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:18.058876 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-5986pclf5" Oct 31 04:48:18.059273 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:18.058907 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-5986pclf5" event={"ID":"838a111f-a040-487f-be53-9bdae5e3b36c","Type":"ContainerDied","Data":"88ab75cff536a14eb4fe8c15faf9e227ccc2c6c47533b95ff8e3c48dfc5b305e"} Oct 31 04:48:18.059273 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:18.058961 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-5986pclf5" event={"ID":"838a111f-a040-487f-be53-9bdae5e3b36c","Type":"ContainerDied","Data":"0260329fbd4f44a5c99ebce1c0222c3a5b9b02f916be2047cc0c320f39dc7981"} Oct 31 04:48:18.059273 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:18.058988 5019 scope.go:122] "RemoveContainer" containerID="88ab75cff536a14eb4fe8c15faf9e227ccc2c6c47533b95ff8e3c48dfc5b305e" Oct 31 04:48:18.072172 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:18.072152 5019 scope.go:122] "RemoveContainer" containerID="88ab75cff536a14eb4fe8c15faf9e227ccc2c6c47533b95ff8e3c48dfc5b305e" Oct 31 04:48:18.072511 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:48:18.072489 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88ab75cff536a14eb4fe8c15faf9e227ccc2c6c47533b95ff8e3c48dfc5b305e\": container with ID starting with 88ab75cff536a14eb4fe8c15faf9e227ccc2c6c47533b95ff8e3c48dfc5b305e not found: ID does not exist" containerID="88ab75cff536a14eb4fe8c15faf9e227ccc2c6c47533b95ff8e3c48dfc5b305e" Oct 31 04:48:18.072617 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:18.072517 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88ab75cff536a14eb4fe8c15faf9e227ccc2c6c47533b95ff8e3c48dfc5b305e"} err="failed to get container status \"88ab75cff536a14eb4fe8c15faf9e227ccc2c6c47533b95ff8e3c48dfc5b305e\": rpc error: code = NotFound desc = could not find container \"88ab75cff536a14eb4fe8c15faf9e227ccc2c6c47533b95ff8e3c48dfc5b305e\": container with ID starting with 88ab75cff536a14eb4fe8c15faf9e227ccc2c6c47533b95ff8e3c48dfc5b305e not found: ID does not exist" Oct 31 04:48:18.115125 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:18.115091 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-5986pclf5"] Oct 31 04:48:18.120121 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:18.120095 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-5986pclf5"] Oct 31 04:48:18.507945 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:18.507908 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-758649b3-qbzbg"] Oct 31 04:48:18.508836 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:18.508811 5019 cpu_manager.go:414] "containerMap: removing container" podUID="838a111f-a040-487f-be53-9bdae5e3b36c" containerName="payload-processing" Oct 31 04:48:18.508836 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:18.508836 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="838a111f-a040-487f-be53-9bdae5e3b36c" containerName="payload-processing" Oct 31 04:48:18.509109 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:18.508860 5019 cpu_manager.go:414] "containerMap: removing container" podUID="7f259ab0-453f-4c0b-97e2-2d511cced150" containerName="revoke-keys" Oct 31 04:48:18.509109 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:18.508868 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f259ab0-453f-4c0b-97e2-2d511cced150" containerName="revoke-keys" Oct 31 04:48:18.509109 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:18.509097 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="838a111f-a040-487f-be53-9bdae5e3b36c" containerName="payload-processing" Oct 31 04:48:18.509277 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:18.509115 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f259ab0-453f-4c0b-97e2-2d511cced150" containerName="revoke-keys" Oct 31 04:48:18.525192 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:18.525065 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-758649b3-qbzbg" Oct 31 04:48:18.536626 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:18.535701 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-758649b3-qbzbg"] Oct 31 04:48:18.540390 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:18.540366 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:48:18.549477 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:18.540738 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:48:18.549477 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:18.540852 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:48:18.549477 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:18.547504 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:48:18.654007 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:18.653966 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr7rt\" (UniqueName: \"kubernetes.io/projected/acfc3ef9-fbeb-4d63-bab5-04e927417ccd-kube-api-access-jr7rt\") pod \"maas-api-revoke-sub-758649b3-qbzbg\" (UID: \"acfc3ef9-fbeb-4d63-bab5-04e927417ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-758649b3-qbzbg" Oct 31 04:48:18.654193 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:18.654166 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acfc3ef9-fbeb-4d63-bab5-04e927417ccd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-758649b3-qbzbg\" (UID: \"acfc3ef9-fbeb-4d63-bab5-04e927417ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-758649b3-qbzbg" Oct 31 04:48:18.654248 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:18.654230 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acfc3ef9-fbeb-4d63-bab5-04e927417ccd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-758649b3-qbzbg\" (UID: \"acfc3ef9-fbeb-4d63-bab5-04e927417ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-758649b3-qbzbg" Oct 31 04:48:18.755213 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:18.755176 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr7rt\" (UniqueName: \"kubernetes.io/projected/acfc3ef9-fbeb-4d63-bab5-04e927417ccd-kube-api-access-jr7rt\") pod \"maas-api-revoke-sub-758649b3-qbzbg\" (UID: \"acfc3ef9-fbeb-4d63-bab5-04e927417ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-758649b3-qbzbg" Oct 31 04:48:18.755425 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:18.755268 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acfc3ef9-fbeb-4d63-bab5-04e927417ccd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-758649b3-qbzbg\" (UID: \"acfc3ef9-fbeb-4d63-bab5-04e927417ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-758649b3-qbzbg" Oct 31 04:48:18.755425 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:18.755309 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acfc3ef9-fbeb-4d63-bab5-04e927417ccd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-758649b3-qbzbg\" (UID: \"acfc3ef9-fbeb-4d63-bab5-04e927417ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-758649b3-qbzbg" Oct 31 04:48:18.756067 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:18.756039 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acfc3ef9-fbeb-4d63-bab5-04e927417ccd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-758649b3-qbzbg\" (UID: \"acfc3ef9-fbeb-4d63-bab5-04e927417ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-758649b3-qbzbg" Oct 31 04:48:18.757738 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:18.757716 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acfc3ef9-fbeb-4d63-bab5-04e927417ccd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-758649b3-qbzbg\" (UID: \"acfc3ef9-fbeb-4d63-bab5-04e927417ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-758649b3-qbzbg" Oct 31 04:48:18.778202 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:18.778139 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr7rt\" (UniqueName: \"kubernetes.io/projected/acfc3ef9-fbeb-4d63-bab5-04e927417ccd-kube-api-access-jr7rt\") pod \"maas-api-revoke-sub-758649b3-qbzbg\" (UID: \"acfc3ef9-fbeb-4d63-bab5-04e927417ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-758649b3-qbzbg" Oct 31 04:48:18.850614 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:18.850568 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-758649b3-qbzbg" Oct 31 04:48:19.167034 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:48:19.167001 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacfc3ef9_fbeb_4d63_bab5_04e927417ccd.slice/crio-e4cff23c9e248b1683d776f96da26d2ececcdc6f2f7e7058c7ae56d3495f6bb9 WatchSource:0}: Error finding container e4cff23c9e248b1683d776f96da26d2ececcdc6f2f7e7058c7ae56d3495f6bb9: Status 404 returned error can't find the container with id e4cff23c9e248b1683d776f96da26d2ececcdc6f2f7e7058c7ae56d3495f6bb9 Oct 31 04:48:19.167758 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:19.167723 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-758649b3-qbzbg"] Oct 31 04:48:19.413057 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:19.413016 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="838a111f-a040-487f-be53-9bdae5e3b36c" path="/var/lib/kubelet/pods/838a111f-a040-487f-be53-9bdae5e3b36c/volumes" Oct 31 04:48:19.972229 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:19.972200 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7ttlj5" Oct 31 04:48:20.095110 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:20.095068 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-758649b3-qbzbg" event={"ID":"acfc3ef9-fbeb-4d63-bab5-04e927417ccd","Type":"ContainerStarted","Data":"e47ebc9f47bdbb91987d85f254cd74d3f8219e7a49151b0b60711a9000d1d103"} Oct 31 04:48:20.095317 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:20.095140 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-758649b3-qbzbg" event={"ID":"acfc3ef9-fbeb-4d63-bab5-04e927417ccd","Type":"ContainerStarted","Data":"e4cff23c9e248b1683d776f96da26d2ececcdc6f2f7e7058c7ae56d3495f6bb9"} Oct 31 04:48:20.137878 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:20.137817 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-758649b3-qbzbg" podStartSLOduration=2.1378038679999998 podStartE2EDuration="2.137803868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:48:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:48:20.13420564 +0000 UTC m=+2654.890543679" watchObservedRunningTime="2026-10-31 04:48:20.137803868 +0000 UTC m=+2654.894141908" Oct 31 04:48:21.106555 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:21.106516 5019 generic.go:356] "Generic (PLEG): container finished" podID="acfc3ef9-fbeb-4d63-bab5-04e927417ccd" containerID="e47ebc9f47bdbb91987d85f254cd74d3f8219e7a49151b0b60711a9000d1d103" exitCode=0 Oct 31 04:48:21.106966 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:21.106605 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-758649b3-qbzbg" event={"ID":"acfc3ef9-fbeb-4d63-bab5-04e927417ccd","Type":"ContainerDied","Data":"e47ebc9f47bdbb91987d85f254cd74d3f8219e7a49151b0b60711a9000d1d103"} Oct 31 04:48:22.344596 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:22.344563 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-758649b3-qbzbg" Oct 31 04:48:22.432642 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:48:22.432529 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:48:22.443561 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:22.443524 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acfc3ef9-fbeb-4d63-bab5-04e927417ccd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acfc3ef9-fbeb-4d63-bab5-04e927417ccd-maas-api-xks-ca\") pod \"acfc3ef9-fbeb-4d63-bab5-04e927417ccd\" (UID: \"acfc3ef9-fbeb-4d63-bab5-04e927417ccd\") " Oct 31 04:48:22.443779 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:22.443640 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acfc3ef9-fbeb-4d63-bab5-04e927417ccd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acfc3ef9-fbeb-4d63-bab5-04e927417ccd-maas-api-openshift-ca\") pod \"acfc3ef9-fbeb-4d63-bab5-04e927417ccd\" (UID: \"acfc3ef9-fbeb-4d63-bab5-04e927417ccd\") " Oct 31 04:48:22.443779 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:22.443674 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acfc3ef9-fbeb-4d63-bab5-04e927417ccd-kube-api-access-jr7rt\" (UniqueName: \"kubernetes.io/projected/acfc3ef9-fbeb-4d63-bab5-04e927417ccd-kube-api-access-jr7rt\") pod \"acfc3ef9-fbeb-4d63-bab5-04e927417ccd\" (UID: \"acfc3ef9-fbeb-4d63-bab5-04e927417ccd\") " Oct 31 04:48:22.444391 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:22.444351 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acfc3ef9-fbeb-4d63-bab5-04e927417ccd-maas-api-openshift-ca" pod "acfc3ef9-fbeb-4d63-bab5-04e927417ccd" (UID: "acfc3ef9-fbeb-4d63-bab5-04e927417ccd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:48:22.446899 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:22.446870 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acfc3ef9-fbeb-4d63-bab5-04e927417ccd-kube-api-access-jr7rt" pod "acfc3ef9-fbeb-4d63-bab5-04e927417ccd" (UID: "acfc3ef9-fbeb-4d63-bab5-04e927417ccd"). InnerVolumeSpecName "kube-api-access-jr7rt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:48:22.448634 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:22.448608 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acfc3ef9-fbeb-4d63-bab5-04e927417ccd-maas-api-xks-ca" pod "acfc3ef9-fbeb-4d63-bab5-04e927417ccd" (UID: "acfc3ef9-fbeb-4d63-bab5-04e927417ccd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:48:22.546507 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:22.546465 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acfc3ef9-fbeb-4d63-bab5-04e927417ccd-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:48:22.546507 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:22.546503 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acfc3ef9-fbeb-4d63-bab5-04e927417ccd-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:48:22.546507 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:22.546513 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr7rt\" (UniqueName: \"kubernetes.io/projected/acfc3ef9-fbeb-4d63-bab5-04e927417ccd-kube-api-access-jr7rt\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:48:23.166357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:23.160914 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-758649b3-qbzbg" event={"ID":"acfc3ef9-fbeb-4d63-bab5-04e927417ccd","Type":"ContainerDied","Data":"e4cff23c9e248b1683d776f96da26d2ececcdc6f2f7e7058c7ae56d3495f6bb9"} Oct 31 04:48:23.166357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:23.161014 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4cff23c9e248b1683d776f96da26d2ececcdc6f2f7e7058c7ae56d3495f6bb9" Oct 31 04:48:23.166357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:23.161082 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-758649b3-qbzbg" Oct 31 04:48:29.097354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:29.093390 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-d6d49sdpr"] Oct 31 04:48:29.109581 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:29.109542 5019 cpu_manager.go:414] "containerMap: removing container" podUID="acfc3ef9-fbeb-4d63-bab5-04e927417ccd" containerName="revoke-keys" Oct 31 04:48:29.110323 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:29.110295 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acfc3ef9-fbeb-4d63-bab5-04e927417ccd" containerName="revoke-keys" Oct 31 04:48:29.110868 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:29.110847 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="acfc3ef9-fbeb-4d63-bab5-04e927417ccd" containerName="revoke-keys" Oct 31 04:48:29.128494 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:29.128457 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-d6d49sdpr" Oct 31 04:48:29.153616 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:29.153581 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-47c409\"/\"e2e-cmp-legacy-19321b-provider-cred\"" Oct 31 04:48:29.168730 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:29.168690 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-47c409\"/\"openshift-service-ca.crt\"" Oct 31 04:48:29.169400 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:29.169089 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-47c409\"/\"payload-processing-external-model-e2e--6f807998-dockercfg-zv2c7\"" Oct 31 04:48:29.169554 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:29.169160 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-47c409\"/\"payload-processing-external-model-plugins-e2e-extmod-a-47c409\"" Oct 31 04:48:29.169672 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:29.169218 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-47c409\"/\"routing-overlay\"" Oct 31 04:48:29.169780 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:29.169305 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-47c409\"/\"kube-root-ca.crt\"" Oct 31 04:48:29.256051 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:29.255975 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/44d5e932-9425-4f90-bf9a-94ac9bfe52b2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-d6d49sdpr\" (UID: \"44d5e932-9425-4f90-bf9a-94ac9bfe52b2\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-d6d49sdpr" Oct 31 04:48:29.256268 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:29.256151 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzx2n\" (UniqueName: \"kubernetes.io/projected/44d5e932-9425-4f90-bf9a-94ac9bfe52b2-kube-api-access-mzx2n\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-d6d49sdpr\" (UID: \"44d5e932-9425-4f90-bf9a-94ac9bfe52b2\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-d6d49sdpr" Oct 31 04:48:29.256577 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:29.256506 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44d5e932-9425-4f90-bf9a-94ac9bfe52b2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-d6d49sdpr\" (UID: \"44d5e932-9425-4f90-bf9a-94ac9bfe52b2\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-d6d49sdpr" Oct 31 04:48:29.256679 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:29.256587 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/44d5e932-9425-4f90-bf9a-94ac9bfe52b2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-d6d49sdpr\" (UID: \"44d5e932-9425-4f90-bf9a-94ac9bfe52b2\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-d6d49sdpr" Oct 31 04:48:29.259913 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:29.259877 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-d6d49sdpr"] Oct 31 04:48:29.358712 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:29.358616 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44d5e932-9425-4f90-bf9a-94ac9bfe52b2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-d6d49sdpr\" (UID: \"44d5e932-9425-4f90-bf9a-94ac9bfe52b2\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-d6d49sdpr" Oct 31 04:48:29.358712 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:29.358685 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/44d5e932-9425-4f90-bf9a-94ac9bfe52b2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-d6d49sdpr\" (UID: \"44d5e932-9425-4f90-bf9a-94ac9bfe52b2\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-d6d49sdpr" Oct 31 04:48:29.358941 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:29.358833 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/44d5e932-9425-4f90-bf9a-94ac9bfe52b2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-d6d49sdpr\" (UID: \"44d5e932-9425-4f90-bf9a-94ac9bfe52b2\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-d6d49sdpr" Oct 31 04:48:29.359008 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:29.358946 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzx2n\" (UniqueName: \"kubernetes.io/projected/44d5e932-9425-4f90-bf9a-94ac9bfe52b2-kube-api-access-mzx2n\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-d6d49sdpr\" (UID: \"44d5e932-9425-4f90-bf9a-94ac9bfe52b2\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-d6d49sdpr" Oct 31 04:48:29.359654 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:29.359621 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44d5e932-9425-4f90-bf9a-94ac9bfe52b2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-d6d49sdpr\" (UID: \"44d5e932-9425-4f90-bf9a-94ac9bfe52b2\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-d6d49sdpr" Oct 31 04:48:29.359835 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:29.359638 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/44d5e932-9425-4f90-bf9a-94ac9bfe52b2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-d6d49sdpr\" (UID: \"44d5e932-9425-4f90-bf9a-94ac9bfe52b2\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-d6d49sdpr" Oct 31 04:48:29.365351 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:29.361995 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/44d5e932-9425-4f90-bf9a-94ac9bfe52b2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-d6d49sdpr\" (UID: \"44d5e932-9425-4f90-bf9a-94ac9bfe52b2\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-d6d49sdpr" Oct 31 04:48:29.396356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:29.392927 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzx2n\" (UniqueName: \"kubernetes.io/projected/44d5e932-9425-4f90-bf9a-94ac9bfe52b2-kube-api-access-mzx2n\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-d6d49sdpr\" (UID: \"44d5e932-9425-4f90-bf9a-94ac9bfe52b2\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-d6d49sdpr" Oct 31 04:48:29.514355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:29.513638 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-d6d49sdpr" Oct 31 04:48:30.462258 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:30.462221 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-d6d49sdpr"] Oct 31 04:48:30.469878 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:48:30.468271 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44d5e932_9425_4f90_bf9a_94ac9bfe52b2.slice/crio-4cd39f96054fbf9b291eac7ebeb54d08317ef6dce91f3cc5c1230f3ff4798375 WatchSource:0}: Error finding container 4cd39f96054fbf9b291eac7ebeb54d08317ef6dce91f3cc5c1230f3ff4798375: Status 404 returned error can't find the container with id 4cd39f96054fbf9b291eac7ebeb54d08317ef6dce91f3cc5c1230f3ff4798375 Oct 31 04:48:31.317360 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:31.317303 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-d6d49sdpr" event={"ID":"44d5e932-9425-4f90-bf9a-94ac9bfe52b2","Type":"ContainerStarted","Data":"55d75a8eab9bd1c58973c16f9f2be92a0fa4411b37c2d925faa98f760f6fa6cf"} Oct 31 04:48:31.317550 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:31.317388 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-d6d49sdpr" event={"ID":"44d5e932-9425-4f90-bf9a-94ac9bfe52b2","Type":"ContainerStarted","Data":"4cd39f96054fbf9b291eac7ebeb54d08317ef6dce91f3cc5c1230f3ff4798375"} Oct 31 04:48:31.317550 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:31.317419 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-d6d49sdpr" Oct 31 04:48:31.363970 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:31.363881 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-d6d49sdpr" podStartSLOduration=2.363859705 podStartE2EDuration="2.363859705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:48:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:48:31.357208892 +0000 UTC m=+2666.113546934" watchObservedRunningTime="2026-10-31 04:48:31.363859705 +0000 UTC m=+2666.120197746" Oct 31 04:48:32.063996 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:32.063955 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e38e6ad-n8vmc"] Oct 31 04:48:32.069543 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:32.069513 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e38e6ad-n8vmc" Oct 31 04:48:32.090434 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:32.090382 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e38e6ad-n8vmc"] Oct 31 04:48:32.093846 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:32.093815 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:48:32.093846 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:32.093834 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:48:32.094066 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:32.093813 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:48:32.094066 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:32.093954 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:48:32.129588 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:32.129541 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be76e744-5738-416c-8125-feb6a1cac37f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e38e6ad-n8vmc\" (UID: \"be76e744-5738-416c-8125-feb6a1cac37f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e38e6ad-n8vmc" Oct 31 04:48:32.129793 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:32.129769 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrchq\" (UniqueName: \"kubernetes.io/projected/be76e744-5738-416c-8125-feb6a1cac37f-kube-api-access-xrchq\") pod \"maas-api-revoke-sub-5e38e6ad-n8vmc\" (UID: \"be76e744-5738-416c-8125-feb6a1cac37f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e38e6ad-n8vmc" Oct 31 04:48:32.129954 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:32.129929 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be76e744-5738-416c-8125-feb6a1cac37f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e38e6ad-n8vmc\" (UID: \"be76e744-5738-416c-8125-feb6a1cac37f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e38e6ad-n8vmc" Oct 31 04:48:32.233350 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:32.232176 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be76e744-5738-416c-8125-feb6a1cac37f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e38e6ad-n8vmc\" (UID: \"be76e744-5738-416c-8125-feb6a1cac37f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e38e6ad-n8vmc" Oct 31 04:48:32.233350 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:32.232399 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrchq\" (UniqueName: \"kubernetes.io/projected/be76e744-5738-416c-8125-feb6a1cac37f-kube-api-access-xrchq\") pod \"maas-api-revoke-sub-5e38e6ad-n8vmc\" (UID: \"be76e744-5738-416c-8125-feb6a1cac37f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e38e6ad-n8vmc" Oct 31 04:48:32.233350 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:32.232515 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be76e744-5738-416c-8125-feb6a1cac37f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e38e6ad-n8vmc\" (UID: \"be76e744-5738-416c-8125-feb6a1cac37f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e38e6ad-n8vmc" Oct 31 04:48:32.252354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:32.236518 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be76e744-5738-416c-8125-feb6a1cac37f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e38e6ad-n8vmc\" (UID: \"be76e744-5738-416c-8125-feb6a1cac37f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e38e6ad-n8vmc" Oct 31 04:48:32.252354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:32.250791 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be76e744-5738-416c-8125-feb6a1cac37f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e38e6ad-n8vmc\" (UID: \"be76e744-5738-416c-8125-feb6a1cac37f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e38e6ad-n8vmc" Oct 31 04:48:32.291312 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:32.291278 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrchq\" (UniqueName: \"kubernetes.io/projected/be76e744-5738-416c-8125-feb6a1cac37f-kube-api-access-xrchq\") pod \"maas-api-revoke-sub-5e38e6ad-n8vmc\" (UID: \"be76e744-5738-416c-8125-feb6a1cac37f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e38e6ad-n8vmc" Oct 31 04:48:32.389949 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:32.389850 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e38e6ad-n8vmc" Oct 31 04:48:32.881053 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:32.881007 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e38e6ad-n8vmc"] Oct 31 04:48:32.933210 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:32.933162 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-236b5948-rpkxk"] Oct 31 04:48:32.941186 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:32.941151 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-236b5948-rpkxk" Oct 31 04:48:32.941652 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:32.941047 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-236b5948-rpkxk"] Oct 31 04:48:33.083023 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:33.082926 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f2e1623-8b13-486d-87b7-e1c206687932-maas-api-xks-ca\") pod \"maas-api-revoke-sub-236b5948-rpkxk\" (UID: \"1f2e1623-8b13-486d-87b7-e1c206687932\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-236b5948-rpkxk" Oct 31 04:48:33.083023 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:33.082996 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f2e1623-8b13-486d-87b7-e1c206687932-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-236b5948-rpkxk\" (UID: \"1f2e1623-8b13-486d-87b7-e1c206687932\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-236b5948-rpkxk" Oct 31 04:48:33.083596 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:33.083205 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrrn7\" (UniqueName: \"kubernetes.io/projected/1f2e1623-8b13-486d-87b7-e1c206687932-kube-api-access-lrrn7\") pod \"maas-api-revoke-sub-236b5948-rpkxk\" (UID: \"1f2e1623-8b13-486d-87b7-e1c206687932\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-236b5948-rpkxk" Oct 31 04:48:33.185369 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:33.185300 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrrn7\" (UniqueName: \"kubernetes.io/projected/1f2e1623-8b13-486d-87b7-e1c206687932-kube-api-access-lrrn7\") pod \"maas-api-revoke-sub-236b5948-rpkxk\" (UID: \"1f2e1623-8b13-486d-87b7-e1c206687932\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-236b5948-rpkxk" Oct 31 04:48:33.185636 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:33.185614 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f2e1623-8b13-486d-87b7-e1c206687932-maas-api-xks-ca\") pod \"maas-api-revoke-sub-236b5948-rpkxk\" (UID: \"1f2e1623-8b13-486d-87b7-e1c206687932\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-236b5948-rpkxk" Oct 31 04:48:33.185730 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:33.185688 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f2e1623-8b13-486d-87b7-e1c206687932-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-236b5948-rpkxk\" (UID: \"1f2e1623-8b13-486d-87b7-e1c206687932\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-236b5948-rpkxk" Oct 31 04:48:33.186703 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:33.186664 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f2e1623-8b13-486d-87b7-e1c206687932-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-236b5948-rpkxk\" (UID: \"1f2e1623-8b13-486d-87b7-e1c206687932\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-236b5948-rpkxk" Oct 31 04:48:33.188631 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:33.188604 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f2e1623-8b13-486d-87b7-e1c206687932-maas-api-xks-ca\") pod \"maas-api-revoke-sub-236b5948-rpkxk\" (UID: \"1f2e1623-8b13-486d-87b7-e1c206687932\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-236b5948-rpkxk" Oct 31 04:48:33.209388 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:33.209323 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrrn7\" (UniqueName: \"kubernetes.io/projected/1f2e1623-8b13-486d-87b7-e1c206687932-kube-api-access-lrrn7\") pod \"maas-api-revoke-sub-236b5948-rpkxk\" (UID: \"1f2e1623-8b13-486d-87b7-e1c206687932\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-236b5948-rpkxk" Oct 31 04:48:33.257602 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:33.257566 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-236b5948-rpkxk" Oct 31 04:48:33.345518 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:33.344486 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e38e6ad-n8vmc" event={"ID":"be76e744-5738-416c-8125-feb6a1cac37f","Type":"ContainerStarted","Data":"ea6d20035b6395b44d2a168b71eef4d48063b1eb33345e6da61888575cc7f9f6"} Oct 31 04:48:33.345518 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:33.344562 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e38e6ad-n8vmc" event={"ID":"be76e744-5738-416c-8125-feb6a1cac37f","Type":"ContainerStarted","Data":"7c4eb184178d7166239d51887dacf10a9d3fa83175b4f88374d2f724e6f447b6"} Oct 31 04:48:33.385128 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:33.385086 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fs8zdw"] Oct 31 04:48:33.385765 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:33.385711 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e38e6ad-n8vmc" podStartSLOduration=1.385693762 podStartE2EDuration="1.385693762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:48:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:48:33.384357177 +0000 UTC m=+2668.140695219" watchObservedRunningTime="2026-10-31 04:48:33.385693762 +0000 UTC m=+2668.142031803" Oct 31 04:48:33.410380 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:33.408903 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fs8zdw" Oct 31 04:48:33.432820 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:33.432788 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fs8zdw"] Oct 31 04:48:33.490722 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:33.490664 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbbbb\" (UniqueName: \"kubernetes.io/projected/c44e4f0b-b5ea-4570-9a69-bc52942f8a6a-kube-api-access-hbbbb\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-7b8fs8zdw\" (UID: \"c44e4f0b-b5ea-4570-9a69-bc52942f8a6a\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fs8zdw" Oct 31 04:48:33.490925 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:33.490777 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c44e4f0b-b5ea-4570-9a69-bc52942f8a6a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-7b8fs8zdw\" (UID: \"c44e4f0b-b5ea-4570-9a69-bc52942f8a6a\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fs8zdw" Oct 31 04:48:33.491049 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:33.491019 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c44e4f0b-b5ea-4570-9a69-bc52942f8a6a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-7b8fs8zdw\" (UID: \"c44e4f0b-b5ea-4570-9a69-bc52942f8a6a\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fs8zdw" Oct 31 04:48:33.491117 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:33.491076 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c44e4f0b-b5ea-4570-9a69-bc52942f8a6a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-7b8fs8zdw\" (UID: \"c44e4f0b-b5ea-4570-9a69-bc52942f8a6a\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fs8zdw" Oct 31 04:48:33.574587 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:33.574550 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-236b5948-rpkxk"] Oct 31 04:48:33.594496 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:33.593900 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c44e4f0b-b5ea-4570-9a69-bc52942f8a6a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-7b8fs8zdw\" (UID: \"c44e4f0b-b5ea-4570-9a69-bc52942f8a6a\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fs8zdw" Oct 31 04:48:33.594496 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:33.593956 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c44e4f0b-b5ea-4570-9a69-bc52942f8a6a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-7b8fs8zdw\" (UID: \"c44e4f0b-b5ea-4570-9a69-bc52942f8a6a\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fs8zdw" Oct 31 04:48:33.594496 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:33.594205 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbbbb\" (UniqueName: \"kubernetes.io/projected/c44e4f0b-b5ea-4570-9a69-bc52942f8a6a-kube-api-access-hbbbb\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-7b8fs8zdw\" (UID: \"c44e4f0b-b5ea-4570-9a69-bc52942f8a6a\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fs8zdw" Oct 31 04:48:33.594496 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:33.594275 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c44e4f0b-b5ea-4570-9a69-bc52942f8a6a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-7b8fs8zdw\" (UID: \"c44e4f0b-b5ea-4570-9a69-bc52942f8a6a\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fs8zdw" Oct 31 04:48:33.595472 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:33.595446 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c44e4f0b-b5ea-4570-9a69-bc52942f8a6a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-7b8fs8zdw\" (UID: \"c44e4f0b-b5ea-4570-9a69-bc52942f8a6a\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fs8zdw" Oct 31 04:48:33.596591 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:33.596162 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c44e4f0b-b5ea-4570-9a69-bc52942f8a6a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-7b8fs8zdw\" (UID: \"c44e4f0b-b5ea-4570-9a69-bc52942f8a6a\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fs8zdw" Oct 31 04:48:33.598063 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:33.598030 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c44e4f0b-b5ea-4570-9a69-bc52942f8a6a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-7b8fs8zdw\" (UID: \"c44e4f0b-b5ea-4570-9a69-bc52942f8a6a\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fs8zdw" Oct 31 04:48:33.618656 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:33.618623 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbbbb\" (UniqueName: \"kubernetes.io/projected/c44e4f0b-b5ea-4570-9a69-bc52942f8a6a-kube-api-access-hbbbb\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-7b8fs8zdw\" (UID: \"c44e4f0b-b5ea-4570-9a69-bc52942f8a6a\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fs8zdw" Oct 31 04:48:33.733361 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:33.733306 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fs8zdw" Oct 31 04:48:34.121723 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:48:34.121640 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc44e4f0b_b5ea_4570_9a69_bc52942f8a6a.slice/crio-70d62ee6a926da6810d64924efc4d333d3a085b2cb08be5fe91c6c229818847d WatchSource:0}: Error finding container 70d62ee6a926da6810d64924efc4d333d3a085b2cb08be5fe91c6c229818847d: Status 404 returned error can't find the container with id 70d62ee6a926da6810d64924efc4d333d3a085b2cb08be5fe91c6c229818847d Oct 31 04:48:34.124105 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:34.124078 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fs8zdw"] Oct 31 04:48:34.367926 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:34.367889 5019 generic.go:356] "Generic (PLEG): container finished" podID="be76e744-5738-416c-8125-feb6a1cac37f" containerID="ea6d20035b6395b44d2a168b71eef4d48063b1eb33345e6da61888575cc7f9f6" exitCode=0 Oct 31 04:48:34.368163 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:34.367965 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e38e6ad-n8vmc" event={"ID":"be76e744-5738-416c-8125-feb6a1cac37f","Type":"ContainerDied","Data":"ea6d20035b6395b44d2a168b71eef4d48063b1eb33345e6da61888575cc7f9f6"} Oct 31 04:48:34.370011 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:34.369979 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-236b5948-rpkxk" event={"ID":"1f2e1623-8b13-486d-87b7-e1c206687932","Type":"ContainerStarted","Data":"3cd5188fca2efcd76523196c962784dbdebae23e6a00f68efcc9c916c1ef93d7"} Oct 31 04:48:34.370151 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:34.370029 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-236b5948-rpkxk" event={"ID":"1f2e1623-8b13-486d-87b7-e1c206687932","Type":"ContainerStarted","Data":"ef9d28ab84763e469af905f0ae0ba3922b606d1437109b1700b5082fb1518051"} Oct 31 04:48:34.372084 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:34.372007 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fs8zdw" event={"ID":"c44e4f0b-b5ea-4570-9a69-bc52942f8a6a","Type":"ContainerStarted","Data":"57b1f7b3df376147e96c8f6ffe8a036fac3440adc5586502d7707b308b65f9b0"} Oct 31 04:48:34.372084 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:34.372067 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fs8zdw" event={"ID":"c44e4f0b-b5ea-4570-9a69-bc52942f8a6a","Type":"ContainerStarted","Data":"70d62ee6a926da6810d64924efc4d333d3a085b2cb08be5fe91c6c229818847d"} Oct 31 04:48:34.372212 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:34.372120 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fs8zdw" Oct 31 04:48:34.454173 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:34.454105 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-236b5948-rpkxk" podStartSLOduration=2.454083576 podStartE2EDuration="2.454083576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:48:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:48:34.450724523 +0000 UTC m=+2669.207062563" watchObservedRunningTime="2026-10-31 04:48:34.454083576 +0000 UTC m=+2669.210421643" Oct 31 04:48:34.492631 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:34.492563 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fs8zdw" podStartSLOduration=1.492542141 podStartE2EDuration="1.492542141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:48:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:48:34.49094625 +0000 UTC m=+2669.247284287" watchObservedRunningTime="2026-10-31 04:48:34.492542141 +0000 UTC m=+2669.248880184" Oct 31 04:48:35.385409 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:35.385359 5019 generic.go:356] "Generic (PLEG): container finished" podID="1f2e1623-8b13-486d-87b7-e1c206687932" containerID="3cd5188fca2efcd76523196c962784dbdebae23e6a00f68efcc9c916c1ef93d7" exitCode=0 Oct 31 04:48:35.385945 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:35.385478 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-236b5948-rpkxk" event={"ID":"1f2e1623-8b13-486d-87b7-e1c206687932","Type":"ContainerDied","Data":"3cd5188fca2efcd76523196c962784dbdebae23e6a00f68efcc9c916c1ef93d7"} Oct 31 04:48:35.590378 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:35.590346 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e38e6ad-n8vmc" Oct 31 04:48:35.735897 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:35.735852 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be76e744-5738-416c-8125-feb6a1cac37f-kube-api-access-xrchq\" (UniqueName: \"kubernetes.io/projected/be76e744-5738-416c-8125-feb6a1cac37f-kube-api-access-xrchq\") pod \"be76e744-5738-416c-8125-feb6a1cac37f\" (UID: \"be76e744-5738-416c-8125-feb6a1cac37f\") " Oct 31 04:48:35.736118 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:35.736068 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be76e744-5738-416c-8125-feb6a1cac37f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be76e744-5738-416c-8125-feb6a1cac37f-maas-api-openshift-ca\") pod \"be76e744-5738-416c-8125-feb6a1cac37f\" (UID: \"be76e744-5738-416c-8125-feb6a1cac37f\") " Oct 31 04:48:35.736192 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:35.736117 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be76e744-5738-416c-8125-feb6a1cac37f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be76e744-5738-416c-8125-feb6a1cac37f-maas-api-xks-ca\") pod \"be76e744-5738-416c-8125-feb6a1cac37f\" (UID: \"be76e744-5738-416c-8125-feb6a1cac37f\") " Oct 31 04:48:35.736492 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:35.736463 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be76e744-5738-416c-8125-feb6a1cac37f-maas-api-openshift-ca" pod "be76e744-5738-416c-8125-feb6a1cac37f" (UID: "be76e744-5738-416c-8125-feb6a1cac37f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:48:35.736901 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:35.736878 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be76e744-5738-416c-8125-feb6a1cac37f-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:48:35.738247 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:35.738220 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be76e744-5738-416c-8125-feb6a1cac37f-maas-api-xks-ca" pod "be76e744-5738-416c-8125-feb6a1cac37f" (UID: "be76e744-5738-416c-8125-feb6a1cac37f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:48:35.738313 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:35.738234 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be76e744-5738-416c-8125-feb6a1cac37f-kube-api-access-xrchq" pod "be76e744-5738-416c-8125-feb6a1cac37f" (UID: "be76e744-5738-416c-8125-feb6a1cac37f"). InnerVolumeSpecName "kube-api-access-xrchq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:48:35.838242 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:35.838198 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be76e744-5738-416c-8125-feb6a1cac37f-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:48:35.838242 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:35.838242 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrchq\" (UniqueName: \"kubernetes.io/projected/be76e744-5738-416c-8125-feb6a1cac37f-kube-api-access-xrchq\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:48:36.400448 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:36.400418 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e38e6ad-n8vmc" Oct 31 04:48:36.401473 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:36.400450 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e38e6ad-n8vmc" event={"ID":"be76e744-5738-416c-8125-feb6a1cac37f","Type":"ContainerDied","Data":"7c4eb184178d7166239d51887dacf10a9d3fa83175b4f88374d2f724e6f447b6"} Oct 31 04:48:36.401688 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:36.401669 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c4eb184178d7166239d51887dacf10a9d3fa83175b4f88374d2f724e6f447b6" Oct 31 04:48:36.663612 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:36.663524 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-236b5948-rpkxk" Oct 31 04:48:36.757271 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:36.757228 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f2e1623-8b13-486d-87b7-e1c206687932-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f2e1623-8b13-486d-87b7-e1c206687932-maas-api-openshift-ca\") pod \"1f2e1623-8b13-486d-87b7-e1c206687932\" (UID: \"1f2e1623-8b13-486d-87b7-e1c206687932\") " Oct 31 04:48:36.757602 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:36.757385 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f2e1623-8b13-486d-87b7-e1c206687932-kube-api-access-lrrn7\" (UniqueName: \"kubernetes.io/projected/1f2e1623-8b13-486d-87b7-e1c206687932-kube-api-access-lrrn7\") pod \"1f2e1623-8b13-486d-87b7-e1c206687932\" (UID: \"1f2e1623-8b13-486d-87b7-e1c206687932\") " Oct 31 04:48:36.757602 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:36.757521 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f2e1623-8b13-486d-87b7-e1c206687932-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f2e1623-8b13-486d-87b7-e1c206687932-maas-api-xks-ca\") pod \"1f2e1623-8b13-486d-87b7-e1c206687932\" (UID: \"1f2e1623-8b13-486d-87b7-e1c206687932\") " Oct 31 04:48:36.757749 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:36.757638 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f2e1623-8b13-486d-87b7-e1c206687932-maas-api-openshift-ca" pod "1f2e1623-8b13-486d-87b7-e1c206687932" (UID: "1f2e1623-8b13-486d-87b7-e1c206687932"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:48:36.758253 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:36.758229 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f2e1623-8b13-486d-87b7-e1c206687932-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:48:36.759773 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:36.759742 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f2e1623-8b13-486d-87b7-e1c206687932-kube-api-access-lrrn7" pod "1f2e1623-8b13-486d-87b7-e1c206687932" (UID: "1f2e1623-8b13-486d-87b7-e1c206687932"). InnerVolumeSpecName "kube-api-access-lrrn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:48:36.759908 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:36.759880 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f2e1623-8b13-486d-87b7-e1c206687932-maas-api-xks-ca" pod "1f2e1623-8b13-486d-87b7-e1c206687932" (UID: "1f2e1623-8b13-486d-87b7-e1c206687932"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:48:36.859694 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:36.859646 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrrn7\" (UniqueName: \"kubernetes.io/projected/1f2e1623-8b13-486d-87b7-e1c206687932-kube-api-access-lrrn7\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:48:36.859694 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:36.859687 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f2e1623-8b13-486d-87b7-e1c206687932-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:48:37.412868 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:37.412830 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-236b5948-rpkxk" Oct 31 04:48:37.415926 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:37.415881 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-236b5948-rpkxk" event={"ID":"1f2e1623-8b13-486d-87b7-e1c206687932","Type":"ContainerDied","Data":"ef9d28ab84763e469af905f0ae0ba3922b606d1437109b1700b5082fb1518051"} Oct 31 04:48:37.416130 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:37.415947 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef9d28ab84763e469af905f0ae0ba3922b606d1437109b1700b5082fb1518051" Oct 31 04:48:42.332070 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:42.332035 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-d6d49sdpr" Oct 31 04:48:44.781352 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:44.781273 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k"] Oct 31 04:48:44.782625 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:44.782595 5019 cpu_manager.go:414] "containerMap: removing container" podUID="be76e744-5738-416c-8125-feb6a1cac37f" containerName="revoke-keys" Oct 31 04:48:44.782625 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:44.782621 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be76e744-5738-416c-8125-feb6a1cac37f" containerName="revoke-keys" Oct 31 04:48:44.782802 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:44.782637 5019 cpu_manager.go:414] "containerMap: removing container" podUID="1f2e1623-8b13-486d-87b7-e1c206687932" containerName="revoke-keys" Oct 31 04:48:44.782802 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:44.782644 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f2e1623-8b13-486d-87b7-e1c206687932" containerName="revoke-keys" Oct 31 04:48:44.782923 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:44.782816 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="be76e744-5738-416c-8125-feb6a1cac37f" containerName="revoke-keys" Oct 31 04:48:44.782923 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:44.782828 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f2e1623-8b13-486d-87b7-e1c206687932" containerName="revoke-keys" Oct 31 04:48:44.788316 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:44.788282 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k" Oct 31 04:48:44.813790 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:44.813529 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k"] Oct 31 04:48:44.937717 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:44.937656 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-dlm7k\" (UID: \"29c8fd65-abc5-4bc2-8ea8-02d11e203e36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k" Oct 31 04:48:44.937927 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:44.937732 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-dlm7k\" (UID: \"29c8fd65-abc5-4bc2-8ea8-02d11e203e36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k" Oct 31 04:48:44.937927 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:44.937849 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-dlm7k\" (UID: \"29c8fd65-abc5-4bc2-8ea8-02d11e203e36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k" Oct 31 04:48:44.937927 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:44.937909 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqll5\" (UniqueName: \"kubernetes.io/projected/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-kube-api-access-mqll5\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-dlm7k\" (UID: \"29c8fd65-abc5-4bc2-8ea8-02d11e203e36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k" Oct 31 04:48:44.938059 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:44.938026 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-dlm7k\" (UID: \"29c8fd65-abc5-4bc2-8ea8-02d11e203e36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k" Oct 31 04:48:44.938116 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:44.938063 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-dlm7k\" (UID: \"29c8fd65-abc5-4bc2-8ea8-02d11e203e36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k" Oct 31 04:48:44.938116 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:44.938090 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-dlm7k\" (UID: \"29c8fd65-abc5-4bc2-8ea8-02d11e203e36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k" Oct 31 04:48:44.938236 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:44.938223 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-dlm7k\" (UID: \"29c8fd65-abc5-4bc2-8ea8-02d11e203e36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k" Oct 31 04:48:44.938284 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:44.938261 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-dlm7k\" (UID: \"29c8fd65-abc5-4bc2-8ea8-02d11e203e36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k" Oct 31 04:48:45.040225 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:45.040117 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-dlm7k\" (UID: \"29c8fd65-abc5-4bc2-8ea8-02d11e203e36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k" Oct 31 04:48:45.040225 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:45.040190 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-dlm7k\" (UID: \"29c8fd65-abc5-4bc2-8ea8-02d11e203e36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k" Oct 31 04:48:45.040510 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:45.040254 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-dlm7k\" (UID: \"29c8fd65-abc5-4bc2-8ea8-02d11e203e36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k" Oct 31 04:48:45.040510 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:45.040309 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqll5\" (UniqueName: \"kubernetes.io/projected/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-kube-api-access-mqll5\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-dlm7k\" (UID: \"29c8fd65-abc5-4bc2-8ea8-02d11e203e36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k" Oct 31 04:48:45.040510 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:45.040406 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-dlm7k\" (UID: \"29c8fd65-abc5-4bc2-8ea8-02d11e203e36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k" Oct 31 04:48:45.040510 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:45.040452 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-dlm7k\" (UID: \"29c8fd65-abc5-4bc2-8ea8-02d11e203e36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k" Oct 31 04:48:45.040510 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:45.040486 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-dlm7k\" (UID: \"29c8fd65-abc5-4bc2-8ea8-02d11e203e36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k" Oct 31 04:48:45.040774 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:45.040670 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-dlm7k\" (UID: \"29c8fd65-abc5-4bc2-8ea8-02d11e203e36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k" Oct 31 04:48:45.040774 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:45.040703 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-dlm7k\" (UID: \"29c8fd65-abc5-4bc2-8ea8-02d11e203e36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k" Oct 31 04:48:45.040774 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:45.040737 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-dlm7k\" (UID: \"29c8fd65-abc5-4bc2-8ea8-02d11e203e36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k" Oct 31 04:48:45.040991 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:45.040610 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-dlm7k\" (UID: \"29c8fd65-abc5-4bc2-8ea8-02d11e203e36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k" Oct 31 04:48:45.040991 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:45.040872 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-dlm7k\" (UID: \"29c8fd65-abc5-4bc2-8ea8-02d11e203e36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k" Oct 31 04:48:45.040991 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:45.040895 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-dlm7k\" (UID: \"29c8fd65-abc5-4bc2-8ea8-02d11e203e36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k" Oct 31 04:48:45.041521 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:45.041497 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-dlm7k\" (UID: \"29c8fd65-abc5-4bc2-8ea8-02d11e203e36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k" Oct 31 04:48:45.043157 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:45.043134 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-dlm7k\" (UID: \"29c8fd65-abc5-4bc2-8ea8-02d11e203e36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k" Oct 31 04:48:45.043305 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:45.043286 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-dlm7k\" (UID: \"29c8fd65-abc5-4bc2-8ea8-02d11e203e36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k" Oct 31 04:48:45.064125 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:45.064092 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-dlm7k\" (UID: \"29c8fd65-abc5-4bc2-8ea8-02d11e203e36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k" Oct 31 04:48:45.064372 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:45.064318 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqll5\" (UniqueName: \"kubernetes.io/projected/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-kube-api-access-mqll5\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-dlm7k\" (UID: \"29c8fd65-abc5-4bc2-8ea8-02d11e203e36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k" Oct 31 04:48:45.109659 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:45.109615 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k" Oct 31 04:48:45.389928 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:45.389836 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fs8zdw" Oct 31 04:48:45.630357 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:48:45.600044 5019 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-plugins-e2e-extmod-a-8973e2: configmap "payload-processing-external-model-plugins-e2e-extmod-a-8973e2" not found Oct 31 04:48:45.702939 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:48:45.702904 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c44e4f0b-b5ea-4570-9a69-bc52942f8a6a-plugins-config-volume podName:c44e4f0b-b5ea-4570-9a69-bc52942f8a6a nodeName:}" failed. No retries permitted until 2026-10-31 04:48:46.202873448 +0000 UTC m=+2680.959211479 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/c44e4f0b-b5ea-4570-9a69-bc52942f8a6a-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-8973e2-7b8fs8zdw" (UID: "c44e4f0b-b5ea-4570-9a69-bc52942f8a6a") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-8973e2" not found Oct 31 04:48:45.703185 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:45.668805 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:48:45.703396 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:45.703361 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:48:45.703577 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:45.703556 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:48:45.704266 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:45.612277 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fs8zdw"] Oct 31 04:48:45.704470 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:48:45.607812 5019 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8973e2/agc-cred-cc70fa128d1a: secret "agc-cred-cc70fa128d1a" not found Oct 31 04:48:45.704573 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:48:45.704563 5019 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fs8zdw: secret "agc-cred-cc70fa128d1a" not found Oct 31 04:48:45.704697 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:48:45.704689 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c44e4f0b-b5ea-4570-9a69-bc52942f8a6a-provider-credentials podName:c44e4f0b-b5ea-4570-9a69-bc52942f8a6a nodeName:}" failed. No retries permitted until 2026-10-31 04:48:46.204669323 +0000 UTC m=+2680.961007347 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c44e4f0b-b5ea-4570-9a69-bc52942f8a6a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-8973e2-7b8fs8zdw" (UID: "c44e4f0b-b5ea-4570-9a69-bc52942f8a6a") : secret "agc-cred-cc70fa128d1a" not found Oct 31 04:48:45.802570 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:45.798790 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7ttlj5"] Oct 31 04:48:45.924627 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:45.918951 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k"] Oct 31 04:48:46.205410 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:48:46.205305 5019 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8973e2/agc-cred-cc70fa128d1a: secret "agc-cred-cc70fa128d1a" not found Oct 31 04:48:46.205410 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:48:46.205350 5019 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fs8zdw: secret "agc-cred-cc70fa128d1a" not found Oct 31 04:48:46.205410 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:48:46.205389 5019 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-plugins-e2e-extmod-a-8973e2: configmap "payload-processing-external-model-plugins-e2e-extmod-a-8973e2" not found Oct 31 04:48:46.205736 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:48:46.205430 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c44e4f0b-b5ea-4570-9a69-bc52942f8a6a-provider-credentials podName:c44e4f0b-b5ea-4570-9a69-bc52942f8a6a nodeName:}" failed. No retries permitted until 2026-10-31 04:48:47.205408397 +0000 UTC m=+2681.961746416 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c44e4f0b-b5ea-4570-9a69-bc52942f8a6a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-8973e2-7b8fs8zdw" (UID: "c44e4f0b-b5ea-4570-9a69-bc52942f8a6a") : secret "agc-cred-cc70fa128d1a" not found Oct 31 04:48:46.205736 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:48:46.205449 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c44e4f0b-b5ea-4570-9a69-bc52942f8a6a-plugins-config-volume podName:c44e4f0b-b5ea-4570-9a69-bc52942f8a6a nodeName:}" failed. No retries permitted until 2026-10-31 04:48:47.205440618 +0000 UTC m=+2681.961778642 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/c44e4f0b-b5ea-4570-9a69-bc52942f8a6a-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-8973e2-7b8fs8zdw" (UID: "c44e4f0b-b5ea-4570-9a69-bc52942f8a6a") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-8973e2" not found Oct 31 04:48:46.863627 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:46.863579 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k" event={"ID":"29c8fd65-abc5-4bc2-8ea8-02d11e203e36","Type":"ContainerStarted","Data":"938a6ddb68d4afc5adbb6e31684b7c235d50c8777545539eb62215f6c6ac30a5"} Oct 31 04:48:46.864159 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:46.863648 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k" event={"ID":"29c8fd65-abc5-4bc2-8ea8-02d11e203e36","Type":"ContainerStarted","Data":"ad5f4c9f439a4b2b185cf732c558e1f686b4542815e645a8353c4b1733e3bf99"} Oct 31 04:48:46.916620 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:46.916549 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k" podStartSLOduration=2.91653054 podStartE2EDuration="2.91653054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:48:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:48:46.913381431 +0000 UTC m=+2681.669719473" watchObservedRunningTime="2026-10-31 04:48:46.91653054 +0000 UTC m=+2681.672868581" Oct 31 04:48:47.110834 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:47.110785 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k" Oct 31 04:48:47.113162 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:47.113120 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.89:15021/healthz/ready\": dial tcp 10.132.1.89:15021: connect: connection refused" start-of-body= Oct 31 04:48:47.113405 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:47.113193 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k" podUID="29c8fd65-abc5-4bc2-8ea8-02d11e203e36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.89:15021/healthz/ready\": dial tcp 10.132.1.89:15021: connect: connection refused" Oct 31 04:48:47.260280 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:48:47.260238 5019 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-plugins-e2e-extmod-a-8973e2: configmap "payload-processing-external-model-plugins-e2e-extmod-a-8973e2" not found Oct 31 04:48:47.260850 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:48:47.260812 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c44e4f0b-b5ea-4570-9a69-bc52942f8a6a-plugins-config-volume podName:c44e4f0b-b5ea-4570-9a69-bc52942f8a6a nodeName:}" failed. No retries permitted until 2026-10-31 04:48:49.260788137 +0000 UTC m=+2684.017126166 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/c44e4f0b-b5ea-4570-9a69-bc52942f8a6a-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-8973e2-7b8fs8zdw" (UID: "c44e4f0b-b5ea-4570-9a69-bc52942f8a6a") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-8973e2" not found Oct 31 04:48:47.261407 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:48:47.260715 5019 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8973e2/agc-cred-cc70fa128d1a: secret "agc-cred-cc70fa128d1a" not found Oct 31 04:48:47.261556 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:48:47.261540 5019 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fs8zdw: secret "agc-cred-cc70fa128d1a" not found Oct 31 04:48:47.261698 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:48:47.261685 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c44e4f0b-b5ea-4570-9a69-bc52942f8a6a-provider-credentials podName:c44e4f0b-b5ea-4570-9a69-bc52942f8a6a nodeName:}" failed. No retries permitted until 2026-10-31 04:48:49.261667616 +0000 UTC m=+2684.018005634 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c44e4f0b-b5ea-4570-9a69-bc52942f8a6a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-8973e2-7b8fs8zdw" (UID: "c44e4f0b-b5ea-4570-9a69-bc52942f8a6a") : secret "agc-cred-cc70fa128d1a" not found Oct 31 04:48:48.110834 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:48.110786 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.89:15021/healthz/ready\": dial tcp 10.132.1.89:15021: connect: connection refused" start-of-body= Oct 31 04:48:48.111398 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:48.110847 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k" podUID="29c8fd65-abc5-4bc2-8ea8-02d11e203e36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.89:15021/healthz/ready\": dial tcp 10.132.1.89:15021: connect: connection refused" Oct 31 04:48:49.110165 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:49.110124 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.89:15021/healthz/ready\": dial tcp 10.132.1.89:15021: connect: connection refused" start-of-body= Oct 31 04:48:49.110426 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:49.110194 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k" podUID="29c8fd65-abc5-4bc2-8ea8-02d11e203e36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.89:15021/healthz/ready\": dial tcp 10.132.1.89:15021: connect: connection refused" Oct 31 04:48:49.348289 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:48:49.348244 5019 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-plugins-e2e-extmod-a-8973e2: configmap "payload-processing-external-model-plugins-e2e-extmod-a-8973e2" not found Oct 31 04:48:49.348742 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:48:49.348326 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c44e4f0b-b5ea-4570-9a69-bc52942f8a6a-plugins-config-volume podName:c44e4f0b-b5ea-4570-9a69-bc52942f8a6a nodeName:}" failed. No retries permitted until 2026-10-31 04:48:53.348311073 +0000 UTC m=+2688.104649118 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/c44e4f0b-b5ea-4570-9a69-bc52942f8a6a-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-8973e2-7b8fs8zdw" (UID: "c44e4f0b-b5ea-4570-9a69-bc52942f8a6a") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-8973e2" not found Oct 31 04:48:49.348742 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:48:49.348456 5019 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8973e2/agc-cred-cc70fa128d1a: secret "agc-cred-cc70fa128d1a" not found Oct 31 04:48:49.348742 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:48:49.348486 5019 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fs8zdw: secret "agc-cred-cc70fa128d1a" not found Oct 31 04:48:49.348742 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:48:49.348544 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c44e4f0b-b5ea-4570-9a69-bc52942f8a6a-provider-credentials podName:c44e4f0b-b5ea-4570-9a69-bc52942f8a6a nodeName:}" failed. No retries permitted until 2026-10-31 04:48:53.348526937 +0000 UTC m=+2688.104864959 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c44e4f0b-b5ea-4570-9a69-bc52942f8a6a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-8973e2-7b8fs8zdw" (UID: "c44e4f0b-b5ea-4570-9a69-bc52942f8a6a") : secret "agc-cred-cc70fa128d1a" not found Oct 31 04:48:50.110832 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:50.110779 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.89:15021/healthz/ready\": dial tcp 10.132.1.89:15021: connect: connection refused" start-of-body= Oct 31 04:48:50.111084 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:50.110850 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k" podUID="29c8fd65-abc5-4bc2-8ea8-02d11e203e36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.89:15021/healthz/ready\": dial tcp 10.132.1.89:15021: connect: connection refused" Oct 31 04:48:50.852403 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:50.852352 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fs8zdw" podUID="c44e4f0b-b5ea-4570-9a69-bc52942f8a6a" containerName="payload-processing" containerID="cri-o://57b1f7b3df376147e96c8f6ffe8a036fac3440adc5586502d7707b308b65f9b0" gracePeriod=25 Oct 31 04:48:50.880561 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:50.880445 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7ttlj5" podUID="a60e5db6-a9bf-40c1-9077-c3dd6b5272db" containerName="payload-processing" containerID="cri-o://633ac3376ef27c59c9f5c91c07d14d52d8030fcf427fbc907632a4badb9947b2" gracePeriod=25 Oct 31 04:48:51.112090 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:51.111961 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.89:15021/healthz/ready\": dial tcp 10.132.1.89:15021: connect: connection refused" start-of-body= Oct 31 04:48:51.112290 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:51.112084 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k" podUID="29c8fd65-abc5-4bc2-8ea8-02d11e203e36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.89:15021/healthz/ready\": dial tcp 10.132.1.89:15021: connect: connection refused" Oct 31 04:48:51.315998 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:51.315964 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fs8zdw" Oct 31 04:48:51.352223 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:51.352132 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7ttlj5" Oct 31 04:48:51.414404 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:51.414115 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c44e4f0b-b5ea-4570-9a69-bc52942f8a6a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c44e4f0b-b5ea-4570-9a69-bc52942f8a6a-plugins-config-volume\") pod \"c44e4f0b-b5ea-4570-9a69-bc52942f8a6a\" (UID: \"c44e4f0b-b5ea-4570-9a69-bc52942f8a6a\") " Oct 31 04:48:51.414404 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:51.414245 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c44e4f0b-b5ea-4570-9a69-bc52942f8a6a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c44e4f0b-b5ea-4570-9a69-bc52942f8a6a-routing-overlay-volume\") pod \"c44e4f0b-b5ea-4570-9a69-bc52942f8a6a\" (UID: \"c44e4f0b-b5ea-4570-9a69-bc52942f8a6a\") " Oct 31 04:48:51.414404 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:51.414327 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c44e4f0b-b5ea-4570-9a69-bc52942f8a6a-kube-api-access-hbbbb\" (UniqueName: \"kubernetes.io/projected/c44e4f0b-b5ea-4570-9a69-bc52942f8a6a-kube-api-access-hbbbb\") pod \"c44e4f0b-b5ea-4570-9a69-bc52942f8a6a\" (UID: \"c44e4f0b-b5ea-4570-9a69-bc52942f8a6a\") " Oct 31 04:48:51.414404 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:51.414403 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c44e4f0b-b5ea-4570-9a69-bc52942f8a6a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c44e4f0b-b5ea-4570-9a69-bc52942f8a6a-provider-credentials\") pod \"c44e4f0b-b5ea-4570-9a69-bc52942f8a6a\" (UID: \"c44e4f0b-b5ea-4570-9a69-bc52942f8a6a\") " Oct 31 04:48:51.414818 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:51.414674 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c44e4f0b-b5ea-4570-9a69-bc52942f8a6a-plugins-config-volume" pod "c44e4f0b-b5ea-4570-9a69-bc52942f8a6a" (UID: "c44e4f0b-b5ea-4570-9a69-bc52942f8a6a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:48:51.418079 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:51.415059 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c44e4f0b-b5ea-4570-9a69-bc52942f8a6a-routing-overlay-volume" pod "c44e4f0b-b5ea-4570-9a69-bc52942f8a6a" (UID: "c44e4f0b-b5ea-4570-9a69-bc52942f8a6a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:48:51.418079 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:51.416083 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c44e4f0b-b5ea-4570-9a69-bc52942f8a6a-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:48:51.418079 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:51.416108 5019 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c44e4f0b-b5ea-4570-9a69-bc52942f8a6a-routing-overlay-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:48:51.418981 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:51.418948 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c44e4f0b-b5ea-4570-9a69-bc52942f8a6a-kube-api-access-hbbbb" pod "c44e4f0b-b5ea-4570-9a69-bc52942f8a6a" (UID: "c44e4f0b-b5ea-4570-9a69-bc52942f8a6a"). InnerVolumeSpecName "kube-api-access-hbbbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:48:51.421480 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:51.421450 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c44e4f0b-b5ea-4570-9a69-bc52942f8a6a-provider-credentials" pod "c44e4f0b-b5ea-4570-9a69-bc52942f8a6a" (UID: "c44e4f0b-b5ea-4570-9a69-bc52942f8a6a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:48:51.524358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:51.523717 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a60e5db6-a9bf-40c1-9077-c3dd6b5272db-routing-overlay-volume" pod "a60e5db6-a9bf-40c1-9077-c3dd6b5272db" (UID: "a60e5db6-a9bf-40c1-9077-c3dd6b5272db"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:48:51.524358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:51.523824 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a60e5db6-a9bf-40c1-9077-c3dd6b5272db-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a60e5db6-a9bf-40c1-9077-c3dd6b5272db-routing-overlay-volume\") pod \"a60e5db6-a9bf-40c1-9077-c3dd6b5272db\" (UID: \"a60e5db6-a9bf-40c1-9077-c3dd6b5272db\") " Oct 31 04:48:51.524726 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:51.524684 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a60e5db6-a9bf-40c1-9077-c3dd6b5272db-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a60e5db6-a9bf-40c1-9077-c3dd6b5272db-plugins-config-volume\") pod \"a60e5db6-a9bf-40c1-9077-c3dd6b5272db\" (UID: \"a60e5db6-a9bf-40c1-9077-c3dd6b5272db\") " Oct 31 04:48:51.525545 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:51.525519 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a60e5db6-a9bf-40c1-9077-c3dd6b5272db-kube-api-access-lbttq\" (UniqueName: \"kubernetes.io/projected/a60e5db6-a9bf-40c1-9077-c3dd6b5272db-kube-api-access-lbttq\") pod \"a60e5db6-a9bf-40c1-9077-c3dd6b5272db\" (UID: \"a60e5db6-a9bf-40c1-9077-c3dd6b5272db\") " Oct 31 04:48:51.525868 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:51.525847 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a60e5db6-a9bf-40c1-9077-c3dd6b5272db-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a60e5db6-a9bf-40c1-9077-c3dd6b5272db-provider-credentials\") pod \"a60e5db6-a9bf-40c1-9077-c3dd6b5272db\" (UID: \"a60e5db6-a9bf-40c1-9077-c3dd6b5272db\") " Oct 31 04:48:51.526092 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:51.525362 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a60e5db6-a9bf-40c1-9077-c3dd6b5272db-plugins-config-volume" pod "a60e5db6-a9bf-40c1-9077-c3dd6b5272db" (UID: "a60e5db6-a9bf-40c1-9077-c3dd6b5272db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:48:51.527043 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:51.527017 5019 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a60e5db6-a9bf-40c1-9077-c3dd6b5272db-routing-overlay-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:48:51.527205 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:51.527190 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbbbb\" (UniqueName: \"kubernetes.io/projected/c44e4f0b-b5ea-4570-9a69-bc52942f8a6a-kube-api-access-hbbbb\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:48:51.527295 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:51.527283 5019 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c44e4f0b-b5ea-4570-9a69-bc52942f8a6a-provider-credentials\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:48:51.527495 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:51.527481 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a60e5db6-a9bf-40c1-9077-c3dd6b5272db-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:48:51.529905 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:51.529877 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a60e5db6-a9bf-40c1-9077-c3dd6b5272db-provider-credentials" pod "a60e5db6-a9bf-40c1-9077-c3dd6b5272db" (UID: "a60e5db6-a9bf-40c1-9077-c3dd6b5272db"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:48:51.530381 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:51.530319 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a60e5db6-a9bf-40c1-9077-c3dd6b5272db-kube-api-access-lbttq" pod "a60e5db6-a9bf-40c1-9077-c3dd6b5272db" (UID: "a60e5db6-a9bf-40c1-9077-c3dd6b5272db"). InnerVolumeSpecName "kube-api-access-lbttq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:48:51.629323 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:51.629277 5019 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a60e5db6-a9bf-40c1-9077-c3dd6b5272db-provider-credentials\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:48:51.629323 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:51.629315 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbttq\" (UniqueName: \"kubernetes.io/projected/a60e5db6-a9bf-40c1-9077-c3dd6b5272db-kube-api-access-lbttq\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:48:51.936292 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:51.936246 5019 generic.go:356] "Generic (PLEG): container finished" podID="a60e5db6-a9bf-40c1-9077-c3dd6b5272db" containerID="633ac3376ef27c59c9f5c91c07d14d52d8030fcf427fbc907632a4badb9947b2" exitCode=0 Oct 31 04:48:51.936793 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:51.936295 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7ttlj5" Oct 31 04:48:51.936793 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:51.936316 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7ttlj5" event={"ID":"a60e5db6-a9bf-40c1-9077-c3dd6b5272db","Type":"ContainerDied","Data":"633ac3376ef27c59c9f5c91c07d14d52d8030fcf427fbc907632a4badb9947b2"} Oct 31 04:48:51.936793 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:51.936397 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7ttlj5" event={"ID":"a60e5db6-a9bf-40c1-9077-c3dd6b5272db","Type":"ContainerDied","Data":"fb4691e88e827956ccb537fddacfac2ea2b34c80905944700376a2e7d873753b"} Oct 31 04:48:51.936793 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:51.936426 5019 scope.go:122] "RemoveContainer" containerID="633ac3376ef27c59c9f5c91c07d14d52d8030fcf427fbc907632a4badb9947b2" Oct 31 04:48:51.938520 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:51.938491 5019 generic.go:356] "Generic (PLEG): container finished" podID="c44e4f0b-b5ea-4570-9a69-bc52942f8a6a" containerID="57b1f7b3df376147e96c8f6ffe8a036fac3440adc5586502d7707b308b65f9b0" exitCode=0 Oct 31 04:48:51.938714 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:51.938694 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fs8zdw" event={"ID":"c44e4f0b-b5ea-4570-9a69-bc52942f8a6a","Type":"ContainerDied","Data":"57b1f7b3df376147e96c8f6ffe8a036fac3440adc5586502d7707b308b65f9b0"} Oct 31 04:48:51.938790 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:51.938752 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fs8zdw" event={"ID":"c44e4f0b-b5ea-4570-9a69-bc52942f8a6a","Type":"ContainerDied","Data":"70d62ee6a926da6810d64924efc4d333d3a085b2cb08be5fe91c6c229818847d"} Oct 31 04:48:51.938846 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:51.938833 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fs8zdw" Oct 31 04:48:51.951821 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:51.951800 5019 scope.go:122] "RemoveContainer" containerID="633ac3376ef27c59c9f5c91c07d14d52d8030fcf427fbc907632a4badb9947b2" Oct 31 04:48:51.952198 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:48:51.952176 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"633ac3376ef27c59c9f5c91c07d14d52d8030fcf427fbc907632a4badb9947b2\": container with ID starting with 633ac3376ef27c59c9f5c91c07d14d52d8030fcf427fbc907632a4badb9947b2 not found: ID does not exist" containerID="633ac3376ef27c59c9f5c91c07d14d52d8030fcf427fbc907632a4badb9947b2" Oct 31 04:48:51.952250 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:51.952211 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"633ac3376ef27c59c9f5c91c07d14d52d8030fcf427fbc907632a4badb9947b2"} err="failed to get container status \"633ac3376ef27c59c9f5c91c07d14d52d8030fcf427fbc907632a4badb9947b2\": rpc error: code = NotFound desc = could not find container \"633ac3376ef27c59c9f5c91c07d14d52d8030fcf427fbc907632a4badb9947b2\": container with ID starting with 633ac3376ef27c59c9f5c91c07d14d52d8030fcf427fbc907632a4badb9947b2 not found: ID does not exist" Oct 31 04:48:51.952250 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:51.952230 5019 scope.go:122] "RemoveContainer" containerID="57b1f7b3df376147e96c8f6ffe8a036fac3440adc5586502d7707b308b65f9b0" Oct 31 04:48:51.966797 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:51.966767 5019 scope.go:122] "RemoveContainer" containerID="57b1f7b3df376147e96c8f6ffe8a036fac3440adc5586502d7707b308b65f9b0" Oct 31 04:48:51.967159 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:48:51.967136 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57b1f7b3df376147e96c8f6ffe8a036fac3440adc5586502d7707b308b65f9b0\": container with ID starting with 57b1f7b3df376147e96c8f6ffe8a036fac3440adc5586502d7707b308b65f9b0 not found: ID does not exist" containerID="57b1f7b3df376147e96c8f6ffe8a036fac3440adc5586502d7707b308b65f9b0" Oct 31 04:48:51.967286 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:51.967167 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57b1f7b3df376147e96c8f6ffe8a036fac3440adc5586502d7707b308b65f9b0"} err="failed to get container status \"57b1f7b3df376147e96c8f6ffe8a036fac3440adc5586502d7707b308b65f9b0\": rpc error: code = NotFound desc = could not find container \"57b1f7b3df376147e96c8f6ffe8a036fac3440adc5586502d7707b308b65f9b0\": container with ID starting with 57b1f7b3df376147e96c8f6ffe8a036fac3440adc5586502d7707b308b65f9b0 not found: ID does not exist" Oct 31 04:48:52.003092 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:52.003044 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fs8zdw"] Oct 31 04:48:52.014080 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:52.014031 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7b8fs8zdw"] Oct 31 04:48:52.050567 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:52.050531 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7ttlj5"] Oct 31 04:48:52.056063 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:52.056021 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-67b7ttlj5"] Oct 31 04:48:52.114355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:52.113793 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.89:15021/healthz/ready\": dial tcp 10.132.1.89:15021: connect: connection refused" start-of-body= Oct 31 04:48:52.114355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:52.113866 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k" podUID="29c8fd65-abc5-4bc2-8ea8-02d11e203e36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.89:15021/healthz/ready\": dial tcp 10.132.1.89:15021: connect: connection refused" Oct 31 04:48:52.618069 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:48:52.618023 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:48:53.110924 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:53.110879 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.89:15021/healthz/ready\": dial tcp 10.132.1.89:15021: connect: connection refused" start-of-body= Oct 31 04:48:53.111391 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:53.110963 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k" podUID="29c8fd65-abc5-4bc2-8ea8-02d11e203e36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.89:15021/healthz/ready\": dial tcp 10.132.1.89:15021: connect: connection refused" Oct 31 04:48:53.412902 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:53.412810 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a60e5db6-a9bf-40c1-9077-c3dd6b5272db" path="/var/lib/kubelet/pods/a60e5db6-a9bf-40c1-9077-c3dd6b5272db/volumes" Oct 31 04:48:53.413387 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:53.413367 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c44e4f0b-b5ea-4570-9a69-bc52942f8a6a" path="/var/lib/kubelet/pods/c44e4f0b-b5ea-4570-9a69-bc52942f8a6a/volumes" Oct 31 04:48:54.110791 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:54.110744 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.89:15021/healthz/ready\": dial tcp 10.132.1.89:15021: connect: connection refused" start-of-body= Oct 31 04:48:54.110992 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:54.110820 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k" podUID="29c8fd65-abc5-4bc2-8ea8-02d11e203e36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.89:15021/healthz/ready\": dial tcp 10.132.1.89:15021: connect: connection refused" Oct 31 04:48:55.687919 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:55.687881 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k" Oct 31 04:48:55.688531 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:55.688507 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k" Oct 31 04:48:55.689799 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:55.689775 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k" Oct 31 04:48:59.081852 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:59.081810 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-98755745d-wmjfq"] Oct 31 04:48:59.082725 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:59.082705 5019 cpu_manager.go:414] "containerMap: removing container" podUID="c44e4f0b-b5ea-4570-9a69-bc52942f8a6a" containerName="payload-processing" Oct 31 04:48:59.082827 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:59.082727 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c44e4f0b-b5ea-4570-9a69-bc52942f8a6a" containerName="payload-processing" Oct 31 04:48:59.082827 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:59.082759 5019 cpu_manager.go:414] "containerMap: removing container" podUID="a60e5db6-a9bf-40c1-9077-c3dd6b5272db" containerName="payload-processing" Oct 31 04:48:59.082827 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:59.082764 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a60e5db6-a9bf-40c1-9077-c3dd6b5272db" containerName="payload-processing" Oct 31 04:48:59.082939 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:59.082902 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="a60e5db6-a9bf-40c1-9077-c3dd6b5272db" containerName="payload-processing" Oct 31 04:48:59.082939 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:59.082920 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="c44e4f0b-b5ea-4570-9a69-bc52942f8a6a" containerName="payload-processing" Oct 31 04:48:59.087032 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:59.087010 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-98755745d-wmjfq" Oct 31 04:48:59.102631 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:59.101683 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-11e01939582a\"" Oct 31 04:48:59.110726 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:59.110683 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-98755745d-wmjfq"] Oct 31 04:48:59.113802 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:59.113764 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 31 04:48:59.113802 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:59.113773 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 31 04:48:59.114067 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:59.113928 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 31 04:48:59.114067 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:59.113960 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 31 04:48:59.114067 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:59.114010 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-cfpcf\"" Oct 31 04:48:59.127383 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:59.127342 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ddad8e95-046a-49bc-84a7-6877fdc30788-provider-credentials\") pod \"payload-processing-external-model-98755745d-wmjfq\" (UID: \"ddad8e95-046a-49bc-84a7-6877fdc30788\") " pod="models-as-a-service/payload-processing-external-model-98755745d-wmjfq" Oct 31 04:48:59.127637 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:59.127409 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddad8e95-046a-49bc-84a7-6877fdc30788-plugins-config-volume\") pod \"payload-processing-external-model-98755745d-wmjfq\" (UID: \"ddad8e95-046a-49bc-84a7-6877fdc30788\") " pod="models-as-a-service/payload-processing-external-model-98755745d-wmjfq" Oct 31 04:48:59.127637 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:59.127489 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ddad8e95-046a-49bc-84a7-6877fdc30788-routing-overlay-volume\") pod \"payload-processing-external-model-98755745d-wmjfq\" (UID: \"ddad8e95-046a-49bc-84a7-6877fdc30788\") " pod="models-as-a-service/payload-processing-external-model-98755745d-wmjfq" Oct 31 04:48:59.127776 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:59.127734 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zktf\" (UniqueName: \"kubernetes.io/projected/ddad8e95-046a-49bc-84a7-6877fdc30788-kube-api-access-9zktf\") pod \"payload-processing-external-model-98755745d-wmjfq\" (UID: \"ddad8e95-046a-49bc-84a7-6877fdc30788\") " pod="models-as-a-service/payload-processing-external-model-98755745d-wmjfq" Oct 31 04:48:59.229827 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:59.229778 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zktf\" (UniqueName: \"kubernetes.io/projected/ddad8e95-046a-49bc-84a7-6877fdc30788-kube-api-access-9zktf\") pod \"payload-processing-external-model-98755745d-wmjfq\" (UID: \"ddad8e95-046a-49bc-84a7-6877fdc30788\") " pod="models-as-a-service/payload-processing-external-model-98755745d-wmjfq" Oct 31 04:48:59.229999 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:59.229979 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ddad8e95-046a-49bc-84a7-6877fdc30788-provider-credentials\") pod \"payload-processing-external-model-98755745d-wmjfq\" (UID: \"ddad8e95-046a-49bc-84a7-6877fdc30788\") " pod="models-as-a-service/payload-processing-external-model-98755745d-wmjfq" Oct 31 04:48:59.230063 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:59.230035 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddad8e95-046a-49bc-84a7-6877fdc30788-plugins-config-volume\") pod \"payload-processing-external-model-98755745d-wmjfq\" (UID: \"ddad8e95-046a-49bc-84a7-6877fdc30788\") " pod="models-as-a-service/payload-processing-external-model-98755745d-wmjfq" Oct 31 04:48:59.230114 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:59.230069 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ddad8e95-046a-49bc-84a7-6877fdc30788-routing-overlay-volume\") pod \"payload-processing-external-model-98755745d-wmjfq\" (UID: \"ddad8e95-046a-49bc-84a7-6877fdc30788\") " pod="models-as-a-service/payload-processing-external-model-98755745d-wmjfq" Oct 31 04:48:59.230935 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:59.230908 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ddad8e95-046a-49bc-84a7-6877fdc30788-routing-overlay-volume\") pod \"payload-processing-external-model-98755745d-wmjfq\" (UID: \"ddad8e95-046a-49bc-84a7-6877fdc30788\") " pod="models-as-a-service/payload-processing-external-model-98755745d-wmjfq" Oct 31 04:48:59.231071 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:59.230911 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddad8e95-046a-49bc-84a7-6877fdc30788-plugins-config-volume\") pod \"payload-processing-external-model-98755745d-wmjfq\" (UID: \"ddad8e95-046a-49bc-84a7-6877fdc30788\") " pod="models-as-a-service/payload-processing-external-model-98755745d-wmjfq" Oct 31 04:48:59.232710 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:59.232685 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ddad8e95-046a-49bc-84a7-6877fdc30788-provider-credentials\") pod \"payload-processing-external-model-98755745d-wmjfq\" (UID: \"ddad8e95-046a-49bc-84a7-6877fdc30788\") " pod="models-as-a-service/payload-processing-external-model-98755745d-wmjfq" Oct 31 04:48:59.253418 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:59.253385 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zktf\" (UniqueName: \"kubernetes.io/projected/ddad8e95-046a-49bc-84a7-6877fdc30788-kube-api-access-9zktf\") pod \"payload-processing-external-model-98755745d-wmjfq\" (UID: \"ddad8e95-046a-49bc-84a7-6877fdc30788\") " pod="models-as-a-service/payload-processing-external-model-98755745d-wmjfq" Oct 31 04:48:59.409216 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:59.409115 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-98755745d-wmjfq" Oct 31 04:48:59.839384 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:59.839302 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4"] Oct 31 04:48:59.846736 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:59.846695 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4" Oct 31 04:48:59.870590 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:59.870541 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7"] Oct 31 04:48:59.878750 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:59.878712 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7" Oct 31 04:48:59.911410 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:59.911372 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4"] Oct 31 04:48:59.927496 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:59.923426 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7"] Oct 31 04:48:59.965608 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:59.965554 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qfbk4\" (UID: \"7de5a068-ea2b-4e78-8c43-3ae8dddabfd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4" Oct 31 04:48:59.965608 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:59.965613 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1daeedb6-a292-4bad-8d14-22a40dc22100-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rzxt7\" (UID: \"1daeedb6-a292-4bad-8d14-22a40dc22100\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7" Oct 31 04:48:59.965900 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:59.965703 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1daeedb6-a292-4bad-8d14-22a40dc22100-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rzxt7\" (UID: \"1daeedb6-a292-4bad-8d14-22a40dc22100\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7" Oct 31 04:48:59.965900 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:59.965745 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qfbk4\" (UID: \"7de5a068-ea2b-4e78-8c43-3ae8dddabfd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4" Oct 31 04:48:59.965900 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:59.965770 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1daeedb6-a292-4bad-8d14-22a40dc22100-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rzxt7\" (UID: \"1daeedb6-a292-4bad-8d14-22a40dc22100\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7" Oct 31 04:48:59.965900 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:59.965831 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qfbk4\" (UID: \"7de5a068-ea2b-4e78-8c43-3ae8dddabfd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4" Oct 31 04:48:59.965900 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:59.965866 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8b85\" (UniqueName: \"kubernetes.io/projected/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-kube-api-access-h8b85\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qfbk4\" (UID: \"7de5a068-ea2b-4e78-8c43-3ae8dddabfd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4" Oct 31 04:48:59.966112 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:59.965952 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1daeedb6-a292-4bad-8d14-22a40dc22100-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rzxt7\" (UID: \"1daeedb6-a292-4bad-8d14-22a40dc22100\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7" Oct 31 04:48:59.966112 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:59.965985 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1daeedb6-a292-4bad-8d14-22a40dc22100-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rzxt7\" (UID: \"1daeedb6-a292-4bad-8d14-22a40dc22100\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7" Oct 31 04:48:59.966112 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:59.966027 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qfbk4\" (UID: \"7de5a068-ea2b-4e78-8c43-3ae8dddabfd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4" Oct 31 04:48:59.966112 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:59.966090 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1daeedb6-a292-4bad-8d14-22a40dc22100-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rzxt7\" (UID: \"1daeedb6-a292-4bad-8d14-22a40dc22100\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7" Oct 31 04:48:59.966267 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:59.966151 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgtbq\" (UniqueName: \"kubernetes.io/projected/1daeedb6-a292-4bad-8d14-22a40dc22100-kube-api-access-vgtbq\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rzxt7\" (UID: \"1daeedb6-a292-4bad-8d14-22a40dc22100\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7" Oct 31 04:48:59.966267 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:59.966209 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qfbk4\" (UID: \"7de5a068-ea2b-4e78-8c43-3ae8dddabfd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4" Oct 31 04:48:59.966361 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:59.966280 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qfbk4\" (UID: \"7de5a068-ea2b-4e78-8c43-3ae8dddabfd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4" Oct 31 04:48:59.966361 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:59.966326 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qfbk4\" (UID: \"7de5a068-ea2b-4e78-8c43-3ae8dddabfd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4" Oct 31 04:48:59.966480 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:59.966454 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1daeedb6-a292-4bad-8d14-22a40dc22100-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rzxt7\" (UID: \"1daeedb6-a292-4bad-8d14-22a40dc22100\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7" Oct 31 04:48:59.966535 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:59.966491 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1daeedb6-a292-4bad-8d14-22a40dc22100-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rzxt7\" (UID: \"1daeedb6-a292-4bad-8d14-22a40dc22100\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7" Oct 31 04:48:59.966576 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:59.966551 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qfbk4\" (UID: \"7de5a068-ea2b-4e78-8c43-3ae8dddabfd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4" Oct 31 04:48:59.978194 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:48:59.978147 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddad8e95_046a_49bc_84a7_6877fdc30788.slice/crio-7b31c3306488dce3416b4c091d8a49b0e0f3ad2eb26d204b91f4180c03bf102d WatchSource:0}: Error finding container 7b31c3306488dce3416b4c091d8a49b0e0f3ad2eb26d204b91f4180c03bf102d: Status 404 returned error can't find the container with id 7b31c3306488dce3416b4c091d8a49b0e0f3ad2eb26d204b91f4180c03bf102d Oct 31 04:48:59.983272 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:48:59.983235 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-98755745d-wmjfq"] Oct 31 04:49:00.069354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:00.069300 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1daeedb6-a292-4bad-8d14-22a40dc22100-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rzxt7\" (UID: \"1daeedb6-a292-4bad-8d14-22a40dc22100\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7" Oct 31 04:49:00.069959 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:00.069931 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qfbk4\" (UID: \"7de5a068-ea2b-4e78-8c43-3ae8dddabfd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4" Oct 31 04:49:00.070182 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:00.070164 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1daeedb6-a292-4bad-8d14-22a40dc22100-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rzxt7\" (UID: \"1daeedb6-a292-4bad-8d14-22a40dc22100\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7" Oct 31 04:49:00.070403 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:00.070383 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgtbq\" (UniqueName: \"kubernetes.io/projected/1daeedb6-a292-4bad-8d14-22a40dc22100-kube-api-access-vgtbq\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rzxt7\" (UID: \"1daeedb6-a292-4bad-8d14-22a40dc22100\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7" Oct 31 04:49:00.070564 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:00.070549 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qfbk4\" (UID: \"7de5a068-ea2b-4e78-8c43-3ae8dddabfd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4" Oct 31 04:49:00.070785 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:00.070768 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qfbk4\" (UID: \"7de5a068-ea2b-4e78-8c43-3ae8dddabfd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4" Oct 31 04:49:00.070959 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:00.070889 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qfbk4\" (UID: \"7de5a068-ea2b-4e78-8c43-3ae8dddabfd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4" Oct 31 04:49:00.071202 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:00.071185 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1daeedb6-a292-4bad-8d14-22a40dc22100-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rzxt7\" (UID: \"1daeedb6-a292-4bad-8d14-22a40dc22100\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7" Oct 31 04:49:00.071318 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:00.071305 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1daeedb6-a292-4bad-8d14-22a40dc22100-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rzxt7\" (UID: \"1daeedb6-a292-4bad-8d14-22a40dc22100\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7" Oct 31 04:49:00.071482 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:00.071467 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qfbk4\" (UID: \"7de5a068-ea2b-4e78-8c43-3ae8dddabfd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4" Oct 31 04:49:00.081367 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:00.072945 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qfbk4\" (UID: \"7de5a068-ea2b-4e78-8c43-3ae8dddabfd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4" Oct 31 04:49:00.081367 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:00.073655 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qfbk4\" (UID: \"7de5a068-ea2b-4e78-8c43-3ae8dddabfd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4" Oct 31 04:49:00.081367 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:00.074025 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1daeedb6-a292-4bad-8d14-22a40dc22100-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rzxt7\" (UID: \"1daeedb6-a292-4bad-8d14-22a40dc22100\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7" Oct 31 04:49:00.081367 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:00.074854 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1daeedb6-a292-4bad-8d14-22a40dc22100-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rzxt7\" (UID: \"1daeedb6-a292-4bad-8d14-22a40dc22100\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7" Oct 31 04:49:00.081367 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:00.076459 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qfbk4\" (UID: \"7de5a068-ea2b-4e78-8c43-3ae8dddabfd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4" Oct 31 04:49:00.081367 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:00.076510 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1daeedb6-a292-4bad-8d14-22a40dc22100-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rzxt7\" (UID: \"1daeedb6-a292-4bad-8d14-22a40dc22100\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7" Oct 31 04:49:00.081367 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:00.076543 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1daeedb6-a292-4bad-8d14-22a40dc22100-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rzxt7\" (UID: \"1daeedb6-a292-4bad-8d14-22a40dc22100\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7" Oct 31 04:49:00.081367 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:00.076573 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qfbk4\" (UID: \"7de5a068-ea2b-4e78-8c43-3ae8dddabfd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4" Oct 31 04:49:00.081367 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:00.076600 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1daeedb6-a292-4bad-8d14-22a40dc22100-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rzxt7\" (UID: \"1daeedb6-a292-4bad-8d14-22a40dc22100\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7" Oct 31 04:49:00.081367 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:00.076629 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qfbk4\" (UID: \"7de5a068-ea2b-4e78-8c43-3ae8dddabfd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4" Oct 31 04:49:00.081367 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:00.076664 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8b85\" (UniqueName: \"kubernetes.io/projected/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-kube-api-access-h8b85\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qfbk4\" (UID: \"7de5a068-ea2b-4e78-8c43-3ae8dddabfd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4" Oct 31 04:49:00.081367 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:00.076734 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1daeedb6-a292-4bad-8d14-22a40dc22100-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rzxt7\" (UID: \"1daeedb6-a292-4bad-8d14-22a40dc22100\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7" Oct 31 04:49:00.081367 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:00.077805 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1daeedb6-a292-4bad-8d14-22a40dc22100-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rzxt7\" (UID: \"1daeedb6-a292-4bad-8d14-22a40dc22100\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7" Oct 31 04:49:00.081367 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:00.078607 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qfbk4\" (UID: \"7de5a068-ea2b-4e78-8c43-3ae8dddabfd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4" Oct 31 04:49:00.081367 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:00.079314 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1daeedb6-a292-4bad-8d14-22a40dc22100-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rzxt7\" (UID: \"1daeedb6-a292-4bad-8d14-22a40dc22100\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7" Oct 31 04:49:00.081367 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:00.079628 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qfbk4\" (UID: \"7de5a068-ea2b-4e78-8c43-3ae8dddabfd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4" Oct 31 04:49:00.081367 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:00.080380 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1daeedb6-a292-4bad-8d14-22a40dc22100-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rzxt7\" (UID: \"1daeedb6-a292-4bad-8d14-22a40dc22100\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7" Oct 31 04:49:00.081367 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:00.080708 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qfbk4\" (UID: \"7de5a068-ea2b-4e78-8c43-3ae8dddabfd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4" Oct 31 04:49:00.088976 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:00.088902 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1daeedb6-a292-4bad-8d14-22a40dc22100-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rzxt7\" (UID: \"1daeedb6-a292-4bad-8d14-22a40dc22100\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7" Oct 31 04:49:00.089540 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:00.089483 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qfbk4\" (UID: \"7de5a068-ea2b-4e78-8c43-3ae8dddabfd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4" Oct 31 04:49:00.090420 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:00.090392 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1daeedb6-a292-4bad-8d14-22a40dc22100-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rzxt7\" (UID: \"1daeedb6-a292-4bad-8d14-22a40dc22100\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7" Oct 31 04:49:00.096403 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:00.096285 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-98755745d-wmjfq" event={"ID":"ddad8e95-046a-49bc-84a7-6877fdc30788","Type":"ContainerStarted","Data":"7b31c3306488dce3416b4c091d8a49b0e0f3ad2eb26d204b91f4180c03bf102d"} Oct 31 04:49:00.097023 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:00.096961 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qfbk4\" (UID: \"7de5a068-ea2b-4e78-8c43-3ae8dddabfd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4" Oct 31 04:49:00.097765 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:00.097716 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1daeedb6-a292-4bad-8d14-22a40dc22100-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rzxt7\" (UID: \"1daeedb6-a292-4bad-8d14-22a40dc22100\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7" Oct 31 04:49:00.099153 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:00.099128 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgtbq\" (UniqueName: \"kubernetes.io/projected/1daeedb6-a292-4bad-8d14-22a40dc22100-kube-api-access-vgtbq\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rzxt7\" (UID: \"1daeedb6-a292-4bad-8d14-22a40dc22100\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7" Oct 31 04:49:00.104181 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:00.104145 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8b85\" (UniqueName: \"kubernetes.io/projected/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-kube-api-access-h8b85\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qfbk4\" (UID: \"7de5a068-ea2b-4e78-8c43-3ae8dddabfd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4" Oct 31 04:49:00.113298 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:00.113265 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qfbk4\" (UID: \"7de5a068-ea2b-4e78-8c43-3ae8dddabfd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4" Oct 31 04:49:00.166110 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:00.166066 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4" Oct 31 04:49:00.214139 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:00.214097 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7" Oct 31 04:49:00.511503 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:00.511460 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4"] Oct 31 04:49:00.517557 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:49:00.517513 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7de5a068_ea2b_4e78_8c43_3ae8dddabfd7.slice/crio-19a6c376e6ccc769905e368d91d7ba087f27ab51d20f611722f3eb205124b9c6 WatchSource:0}: Error finding container 19a6c376e6ccc769905e368d91d7ba087f27ab51d20f611722f3eb205124b9c6: Status 404 returned error can't find the container with id 19a6c376e6ccc769905e368d91d7ba087f27ab51d20f611722f3eb205124b9c6 Oct 31 04:49:00.521094 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:00.521044 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:49:00.521236 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:00.521138 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:49:00.521236 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:00.521174 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:49:00.617433 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:00.615190 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7"] Oct 31 04:49:00.618504 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:49:00.618460 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1daeedb6_a292_4bad_8d14_22a40dc22100.slice/crio-61f8b9b7aedf43c5278f568b6abade6e6a17a4c3c460c0bd764cf73d0b037dee WatchSource:0}: Error finding container 61f8b9b7aedf43c5278f568b6abade6e6a17a4c3c460c0bd764cf73d0b037dee: Status 404 returned error can't find the container with id 61f8b9b7aedf43c5278f568b6abade6e6a17a4c3c460c0bd764cf73d0b037dee Oct 31 04:49:00.622125 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:00.622065 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:49:00.622281 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:00.622191 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:49:00.622281 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:00.622239 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:49:01.153118 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:01.153067 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7" event={"ID":"1daeedb6-a292-4bad-8d14-22a40dc22100","Type":"ContainerStarted","Data":"9c1233731ce94edf4f073b3ea698c07838accc458169eab254675391fc1e64a4"} Oct 31 04:49:01.153787 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:01.153758 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7" event={"ID":"1daeedb6-a292-4bad-8d14-22a40dc22100","Type":"ContainerStarted","Data":"61f8b9b7aedf43c5278f568b6abade6e6a17a4c3c460c0bd764cf73d0b037dee"} Oct 31 04:49:01.182359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:01.173312 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-98755745d-wmjfq" event={"ID":"ddad8e95-046a-49bc-84a7-6877fdc30788","Type":"ContainerStarted","Data":"6724ba6734b29d5c8c06c831a6d4c71b49e8a83116d04de67c619e2069eb4763"} Oct 31 04:49:01.182721 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:01.182686 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-98755745d-wmjfq" Oct 31 04:49:01.261622 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:01.256976 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7" Oct 31 04:49:01.310347 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:01.304512 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4" event={"ID":"7de5a068-ea2b-4e78-8c43-3ae8dddabfd7","Type":"ContainerStarted","Data":"681977346a36ec8b3ad827764ccb563ce12fa653303fb275231a79c591325f46"} Oct 31 04:49:01.310347 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:01.304612 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4" event={"ID":"7de5a068-ea2b-4e78-8c43-3ae8dddabfd7","Type":"ContainerStarted","Data":"19a6c376e6ccc769905e368d91d7ba087f27ab51d20f611722f3eb205124b9c6"} Oct 31 04:49:01.379524 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:01.379481 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" start-of-body= Oct 31 04:49:01.379756 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:01.379547 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7" podUID="1daeedb6-a292-4bad-8d14-22a40dc22100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" Oct 31 04:49:01.395417 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:01.395348 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7" podStartSLOduration=2.395305598 podStartE2EDuration="2.395305598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:48:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:49:01.237372865 +0000 UTC m=+2695.993710923" watchObservedRunningTime="2026-10-31 04:49:01.395305598 +0000 UTC m=+2696.151643639" Oct 31 04:49:01.432929 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:01.432805 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-98755745d-wmjfq" podStartSLOduration=2.4327886 podStartE2EDuration="2.4327886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:48:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:49:01.334225239 +0000 UTC m=+2696.090563293" watchObservedRunningTime="2026-10-31 04:49:01.4327886 +0000 UTC m=+2696.189126641" Oct 31 04:49:01.509363 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:01.483831 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4" podStartSLOduration=2.4838110159999998 podStartE2EDuration="2.483811016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:48:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:49:01.408173272 +0000 UTC m=+2696.164511313" watchObservedRunningTime="2026-10-31 04:49:01.483811016 +0000 UTC m=+2696.240149057" Oct 31 04:49:02.166973 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:02.166940 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4" Oct 31 04:49:02.169616 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:02.169577 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" start-of-body= Oct 31 04:49:02.169887 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:02.169850 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4" podUID="7de5a068-ea2b-4e78-8c43-3ae8dddabfd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" Oct 31 04:49:02.224066 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:02.224022 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" start-of-body= Oct 31 04:49:02.224375 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:02.224344 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7" podUID="1daeedb6-a292-4bad-8d14-22a40dc22100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" Oct 31 04:49:03.204553 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:03.204508 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" start-of-body= Oct 31 04:49:03.205224 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:03.205184 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4" podUID="7de5a068-ea2b-4e78-8c43-3ae8dddabfd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" Oct 31 04:49:03.236363 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:03.233259 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" start-of-body= Oct 31 04:49:03.236363 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:03.233367 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7" podUID="1daeedb6-a292-4bad-8d14-22a40dc22100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" Oct 31 04:49:04.166959 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:04.166915 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" start-of-body= Oct 31 04:49:04.167177 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:04.166976 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4" podUID="7de5a068-ea2b-4e78-8c43-3ae8dddabfd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" Oct 31 04:49:04.229372 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:04.226053 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" start-of-body= Oct 31 04:49:04.229372 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:04.226124 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7" podUID="1daeedb6-a292-4bad-8d14-22a40dc22100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" Oct 31 04:49:04.958740 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:04.958697 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-98755745d-wmjfq"] Oct 31 04:49:05.167696 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:05.167654 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" start-of-body= Oct 31 04:49:05.167889 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:05.167713 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4" podUID="7de5a068-ea2b-4e78-8c43-3ae8dddabfd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" Oct 31 04:49:05.235269 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:05.235212 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" start-of-body= Oct 31 04:49:05.240247 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:05.239502 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7" podUID="1daeedb6-a292-4bad-8d14-22a40dc22100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" Oct 31 04:49:06.166977 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:06.166937 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" start-of-body= Oct 31 04:49:06.167189 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:06.167006 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4" podUID="7de5a068-ea2b-4e78-8c43-3ae8dddabfd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" Oct 31 04:49:06.216357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:06.216106 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" start-of-body= Oct 31 04:49:06.216357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:06.216173 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7" podUID="1daeedb6-a292-4bad-8d14-22a40dc22100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" Oct 31 04:49:07.168080 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:07.168033 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" start-of-body= Oct 31 04:49:07.168765 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:07.168724 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4" podUID="7de5a068-ea2b-4e78-8c43-3ae8dddabfd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" Oct 31 04:49:07.220357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:07.216787 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" start-of-body= Oct 31 04:49:07.220357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:07.216861 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7" podUID="1daeedb6-a292-4bad-8d14-22a40dc22100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" Oct 31 04:49:08.188917 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:08.188859 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" start-of-body= Oct 31 04:49:08.189643 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:08.188949 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4" podUID="7de5a068-ea2b-4e78-8c43-3ae8dddabfd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" Oct 31 04:49:08.271357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:08.269127 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" start-of-body= Oct 31 04:49:08.271357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:08.269198 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7" podUID="1daeedb6-a292-4bad-8d14-22a40dc22100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" Oct 31 04:49:09.064203 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:09.064174 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s88kp_3945ff65-e877-4372-9334-3968c68b7295/console-operator/2.log" Oct 31 04:49:09.142246 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:09.142208 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s88kp_3945ff65-e877-4372-9334-3968c68b7295/console-operator/2.log" Oct 31 04:49:09.167416 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:09.167324 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" start-of-body= Oct 31 04:49:09.167649 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:09.167494 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4" podUID="7de5a068-ea2b-4e78-8c43-3ae8dddabfd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" Oct 31 04:49:09.215026 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:09.214987 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" start-of-body= Oct 31 04:49:09.215675 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:09.215644 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7" podUID="1daeedb6-a292-4bad-8d14-22a40dc22100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" Oct 31 04:49:09.960364 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:09.960023 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-98755745d-wmjfq" podUID="ddad8e95-046a-49bc-84a7-6877fdc30788" containerName="payload-processing" containerID="cri-o://6724ba6734b29d5c8c06c831a6d4c71b49e8a83116d04de67c619e2069eb4763" gracePeriod=25 Oct 31 04:49:10.167493 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:10.167308 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" start-of-body= Oct 31 04:49:10.167493 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:10.167407 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4" podUID="7de5a068-ea2b-4e78-8c43-3ae8dddabfd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" Oct 31 04:49:10.215154 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:10.214977 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" start-of-body= Oct 31 04:49:10.215154 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:10.215041 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7" podUID="1daeedb6-a292-4bad-8d14-22a40dc22100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" Oct 31 04:49:10.314181 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:10.314146 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-98755745d-wmjfq" Oct 31 04:49:10.388758 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:10.387104 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddad8e95-046a-49bc-84a7-6877fdc30788-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ddad8e95-046a-49bc-84a7-6877fdc30788-routing-overlay-volume\") pod \"ddad8e95-046a-49bc-84a7-6877fdc30788\" (UID: \"ddad8e95-046a-49bc-84a7-6877fdc30788\") " Oct 31 04:49:10.388758 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:10.387213 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddad8e95-046a-49bc-84a7-6877fdc30788-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ddad8e95-046a-49bc-84a7-6877fdc30788-provider-credentials\") pod \"ddad8e95-046a-49bc-84a7-6877fdc30788\" (UID: \"ddad8e95-046a-49bc-84a7-6877fdc30788\") " Oct 31 04:49:10.388758 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:10.387269 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddad8e95-046a-49bc-84a7-6877fdc30788-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddad8e95-046a-49bc-84a7-6877fdc30788-plugins-config-volume\") pod \"ddad8e95-046a-49bc-84a7-6877fdc30788\" (UID: \"ddad8e95-046a-49bc-84a7-6877fdc30788\") " Oct 31 04:49:10.388758 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:10.387461 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddad8e95-046a-49bc-84a7-6877fdc30788-kube-api-access-9zktf\" (UniqueName: \"kubernetes.io/projected/ddad8e95-046a-49bc-84a7-6877fdc30788-kube-api-access-9zktf\") pod \"ddad8e95-046a-49bc-84a7-6877fdc30788\" (UID: \"ddad8e95-046a-49bc-84a7-6877fdc30788\") " Oct 31 04:49:10.390766 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:10.390706 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddad8e95-046a-49bc-84a7-6877fdc30788-plugins-config-volume" pod "ddad8e95-046a-49bc-84a7-6877fdc30788" (UID: "ddad8e95-046a-49bc-84a7-6877fdc30788"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:49:10.391720 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:10.391668 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddad8e95-046a-49bc-84a7-6877fdc30788-routing-overlay-volume" pod "ddad8e95-046a-49bc-84a7-6877fdc30788" (UID: "ddad8e95-046a-49bc-84a7-6877fdc30788"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:49:10.456702 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:10.430028 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddad8e95-046a-49bc-84a7-6877fdc30788-kube-api-access-9zktf" pod "ddad8e95-046a-49bc-84a7-6877fdc30788" (UID: "ddad8e95-046a-49bc-84a7-6877fdc30788"). InnerVolumeSpecName "kube-api-access-9zktf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:49:10.457257 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:10.457204 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddad8e95-046a-49bc-84a7-6877fdc30788-provider-credentials" pod "ddad8e95-046a-49bc-84a7-6877fdc30788" (UID: "ddad8e95-046a-49bc-84a7-6877fdc30788"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:49:10.495785 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:10.494034 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zktf\" (UniqueName: \"kubernetes.io/projected/ddad8e95-046a-49bc-84a7-6877fdc30788-kube-api-access-9zktf\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:49:10.495785 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:10.494076 5019 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ddad8e95-046a-49bc-84a7-6877fdc30788-routing-overlay-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:49:10.495785 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:10.494091 5019 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ddad8e95-046a-49bc-84a7-6877fdc30788-provider-credentials\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:49:10.495785 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:10.494105 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddad8e95-046a-49bc-84a7-6877fdc30788-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:49:10.942141 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:10.942100 5019 generic.go:356] "Generic (PLEG): container finished" podID="ddad8e95-046a-49bc-84a7-6877fdc30788" containerID="6724ba6734b29d5c8c06c831a6d4c71b49e8a83116d04de67c619e2069eb4763" exitCode=0 Oct 31 04:49:10.942381 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:10.942179 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-98755745d-wmjfq" Oct 31 04:49:10.942381 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:10.942186 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-98755745d-wmjfq" event={"ID":"ddad8e95-046a-49bc-84a7-6877fdc30788","Type":"ContainerDied","Data":"6724ba6734b29d5c8c06c831a6d4c71b49e8a83116d04de67c619e2069eb4763"} Oct 31 04:49:10.942381 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:10.942327 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-98755745d-wmjfq" event={"ID":"ddad8e95-046a-49bc-84a7-6877fdc30788","Type":"ContainerDied","Data":"7b31c3306488dce3416b4c091d8a49b0e0f3ad2eb26d204b91f4180c03bf102d"} Oct 31 04:49:10.942381 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:10.942379 5019 scope.go:122] "RemoveContainer" containerID="6724ba6734b29d5c8c06c831a6d4c71b49e8a83116d04de67c619e2069eb4763" Oct 31 04:49:10.958522 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:10.958496 5019 scope.go:122] "RemoveContainer" containerID="6724ba6734b29d5c8c06c831a6d4c71b49e8a83116d04de67c619e2069eb4763" Oct 31 04:49:10.958947 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:49:10.958922 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6724ba6734b29d5c8c06c831a6d4c71b49e8a83116d04de67c619e2069eb4763\": container with ID starting with 6724ba6734b29d5c8c06c831a6d4c71b49e8a83116d04de67c619e2069eb4763 not found: ID does not exist" containerID="6724ba6734b29d5c8c06c831a6d4c71b49e8a83116d04de67c619e2069eb4763" Oct 31 04:49:10.959029 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:10.958962 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6724ba6734b29d5c8c06c831a6d4c71b49e8a83116d04de67c619e2069eb4763"} err="failed to get container status \"6724ba6734b29d5c8c06c831a6d4c71b49e8a83116d04de67c619e2069eb4763\": rpc error: code = NotFound desc = could not find container \"6724ba6734b29d5c8c06c831a6d4c71b49e8a83116d04de67c619e2069eb4763\": container with ID starting with 6724ba6734b29d5c8c06c831a6d4c71b49e8a83116d04de67c619e2069eb4763 not found: ID does not exist" Oct 31 04:49:11.001480 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:11.001429 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-98755745d-wmjfq"] Oct 31 04:49:11.013236 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:11.013190 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-98755745d-wmjfq"] Oct 31 04:49:11.167452 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:11.167316 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" start-of-body= Oct 31 04:49:11.167452 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:11.167418 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4" podUID="7de5a068-ea2b-4e78-8c43-3ae8dddabfd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" Oct 31 04:49:11.215693 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:11.215648 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" start-of-body= Oct 31 04:49:11.216149 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:11.215724 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7" podUID="1daeedb6-a292-4bad-8d14-22a40dc22100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" Oct 31 04:49:11.421370 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:11.421270 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddad8e95-046a-49bc-84a7-6877fdc30788" path="/var/lib/kubelet/pods/ddad8e95-046a-49bc-84a7-6877fdc30788/volumes" Oct 31 04:49:12.547832 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:12.546639 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4" Oct 31 04:49:12.547832 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:12.547818 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4" Oct 31 04:49:12.549082 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:12.549056 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4" Oct 31 04:49:12.582292 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:12.582254 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7" Oct 31 04:49:12.582735 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:12.582710 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7" Oct 31 04:49:12.584704 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:12.584676 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7" Oct 31 04:49:17.604304 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:17.604236 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" podUID="b4ced3ab-41e4-4c54-93eb-95eddaae992d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:49:17.651357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:17.646715 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 31 04:49:17.651357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:17.647222 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:49:18.085762 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:18.085707 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" podUID="643433d8-418e-4102-8bc3-cd544932614a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:49:18.247493 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:18.247437 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" podUID="e0f4b8c2-14f4-4052-bf54-dd65bf59e413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 04:49:18.426646 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:18.426536 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" podUID="dccd15a9-47f5-4bd1-b21b-9d5ca8b05845" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 31 04:49:18.639573 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:18.639504 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" podUID="b4ced3ab-41e4-4c54-93eb-95eddaae992d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Oct 31 04:49:18.641030 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:18.641000 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" Oct 31 04:49:19.270149 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:19.270107 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" Oct 31 04:49:19.294957 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:19.286619 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" podUID="e0f4b8c2-14f4-4052-bf54-dd65bf59e413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:49:19.394463 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:19.367432 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" Oct 31 04:49:20.161818 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:20.161775 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" Oct 31 04:49:20.885148 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:20.885099 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9wg7c8"] Oct 31 04:49:20.886873 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:20.886761 5019 cpu_manager.go:414] "containerMap: removing container" podUID="ddad8e95-046a-49bc-84a7-6877fdc30788" containerName="payload-processing" Oct 31 04:49:20.887094 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:20.887074 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddad8e95-046a-49bc-84a7-6877fdc30788" containerName="payload-processing" Oct 31 04:49:20.887576 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:20.887541 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddad8e95-046a-49bc-84a7-6877fdc30788" containerName="payload-processing" Oct 31 04:49:20.894885 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:20.894850 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9wg7c8" Oct 31 04:49:20.918834 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:20.918801 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9wg7c8"] Oct 31 04:49:20.939791 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:20.939753 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-567ca6\"/\"agc-cred-fcc1f4ac9c30\"" Oct 31 04:49:20.940010 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:20.939850 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-567ca6\"/\"openshift-service-ca.crt\"" Oct 31 04:49:20.940010 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:20.939936 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-567ca6\"/\"payload-processing-external-model-e2e--a8f34224-dockercfg-gc8kx\"" Oct 31 04:49:20.940193 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:20.940005 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-567ca6\"/\"routing-overlay\"" Oct 31 04:49:20.940193 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:20.940086 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-567ca6\"/\"kube-root-ca.crt\"" Oct 31 04:49:20.940193 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:20.940140 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-567ca6\"/\"payload-processing-external-model-plugins-e2e-extmod-b-567ca6\"" Oct 31 04:49:20.985681 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:20.985643 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ltd6\" (UniqueName: \"kubernetes.io/projected/cf5c3cf2-fb82-44c8-89e2-0b280d3ed769-kube-api-access-7ltd6\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-64c9wg7c8\" (UID: \"cf5c3cf2-fb82-44c8-89e2-0b280d3ed769\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9wg7c8" Oct 31 04:49:20.985906 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:20.985748 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cf5c3cf2-fb82-44c8-89e2-0b280d3ed769-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-64c9wg7c8\" (UID: \"cf5c3cf2-fb82-44c8-89e2-0b280d3ed769\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9wg7c8" Oct 31 04:49:20.985906 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:20.985899 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf5c3cf2-fb82-44c8-89e2-0b280d3ed769-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-64c9wg7c8\" (UID: \"cf5c3cf2-fb82-44c8-89e2-0b280d3ed769\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9wg7c8" Oct 31 04:49:20.986054 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:20.986036 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cf5c3cf2-fb82-44c8-89e2-0b280d3ed769-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-64c9wg7c8\" (UID: \"cf5c3cf2-fb82-44c8-89e2-0b280d3ed769\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9wg7c8" Oct 31 04:49:21.106606 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:21.106557 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cf5c3cf2-fb82-44c8-89e2-0b280d3ed769-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-64c9wg7c8\" (UID: \"cf5c3cf2-fb82-44c8-89e2-0b280d3ed769\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9wg7c8" Oct 31 04:49:21.106990 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:21.106967 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ltd6\" (UniqueName: \"kubernetes.io/projected/cf5c3cf2-fb82-44c8-89e2-0b280d3ed769-kube-api-access-7ltd6\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-64c9wg7c8\" (UID: \"cf5c3cf2-fb82-44c8-89e2-0b280d3ed769\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9wg7c8" Oct 31 04:49:21.107200 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:21.107181 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cf5c3cf2-fb82-44c8-89e2-0b280d3ed769-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-64c9wg7c8\" (UID: \"cf5c3cf2-fb82-44c8-89e2-0b280d3ed769\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9wg7c8" Oct 31 04:49:21.111566 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:21.111531 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf5c3cf2-fb82-44c8-89e2-0b280d3ed769-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-64c9wg7c8\" (UID: \"cf5c3cf2-fb82-44c8-89e2-0b280d3ed769\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9wg7c8" Oct 31 04:49:21.129601 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:21.113083 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cf5c3cf2-fb82-44c8-89e2-0b280d3ed769-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-64c9wg7c8\" (UID: \"cf5c3cf2-fb82-44c8-89e2-0b280d3ed769\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9wg7c8" Oct 31 04:49:21.129882 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:21.114230 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf5c3cf2-fb82-44c8-89e2-0b280d3ed769-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-64c9wg7c8\" (UID: \"cf5c3cf2-fb82-44c8-89e2-0b280d3ed769\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9wg7c8" Oct 31 04:49:21.149058 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:21.134010 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cf5c3cf2-fb82-44c8-89e2-0b280d3ed769-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-64c9wg7c8\" (UID: \"cf5c3cf2-fb82-44c8-89e2-0b280d3ed769\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9wg7c8" Oct 31 04:49:21.234268 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:21.234232 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ltd6\" (UniqueName: \"kubernetes.io/projected/cf5c3cf2-fb82-44c8-89e2-0b280d3ed769-kube-api-access-7ltd6\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-64c9wg7c8\" (UID: \"cf5c3cf2-fb82-44c8-89e2-0b280d3ed769\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9wg7c8" Oct 31 04:49:21.535869 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:21.535828 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9wg7c8" Oct 31 04:49:21.924651 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:21.924501 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9wg7c8"] Oct 31 04:49:21.943574 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:49:21.943527 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf5c3cf2_fb82_44c8_89e2_0b280d3ed769.slice/crio-49fa71350f30a02ce4bb4e0c75b3a9ecaa332b7e52c0f21ac5b84cead6d692ac WatchSource:0}: Error finding container 49fa71350f30a02ce4bb4e0c75b3a9ecaa332b7e52c0f21ac5b84cead6d692ac: Status 404 returned error can't find the container with id 49fa71350f30a02ce4bb4e0c75b3a9ecaa332b7e52c0f21ac5b84cead6d692ac Oct 31 04:49:22.255464 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:22.255412 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9wg7c8" event={"ID":"cf5c3cf2-fb82-44c8-89e2-0b280d3ed769","Type":"ContainerStarted","Data":"562813c63698940a6b7f62ea03886f720ae3857f1e73de578e98fa81f1d55510"} Oct 31 04:49:22.256019 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:22.255478 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9wg7c8" event={"ID":"cf5c3cf2-fb82-44c8-89e2-0b280d3ed769","Type":"ContainerStarted","Data":"49fa71350f30a02ce4bb4e0c75b3a9ecaa332b7e52c0f21ac5b84cead6d692ac"} Oct 31 04:49:22.256101 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:22.256068 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9wg7c8" Oct 31 04:49:22.315358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:22.315069 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9wg7c8" podStartSLOduration=2.315046754 podStartE2EDuration="2.315046754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:49:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:49:22.307065021 +0000 UTC m=+2717.063403059" watchObservedRunningTime="2026-10-31 04:49:22.315046754 +0000 UTC m=+2717.071384795" Oct 31 04:49:22.493116 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:49:22.493074 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:49:33.288553 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:49:33.288515 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9wg7c8" Oct 31 04:49:52.394934 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:49:52.394839 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:50:00.497870 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:00.497833 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42907744-t2knt"] Oct 31 04:50:00.503675 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:00.503644 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42907744-t2knt" Oct 31 04:50:00.525228 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:00.525200 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42907744-t2knt"] Oct 31 04:50:00.527373 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:00.527304 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:50:00.527518 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:00.527420 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:50:00.527518 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:00.527491 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:50:00.527518 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:00.527507 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:50:00.542004 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:00.541977 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3d6fe36-f010-4322-95af-a7e419f76d8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42907744-t2knt\" (UID: \"c3d6fe36-f010-4322-95af-a7e419f76d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42907744-t2knt" Oct 31 04:50:00.542148 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:00.542116 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw7p4\" (UniqueName: \"kubernetes.io/projected/c3d6fe36-f010-4322-95af-a7e419f76d8d-kube-api-access-qw7p4\") pod \"maas-api-revoke-sub-42907744-t2knt\" (UID: \"c3d6fe36-f010-4322-95af-a7e419f76d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42907744-t2knt" Oct 31 04:50:00.542278 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:00.542256 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3d6fe36-f010-4322-95af-a7e419f76d8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42907744-t2knt\" (UID: \"c3d6fe36-f010-4322-95af-a7e419f76d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42907744-t2knt" Oct 31 04:50:00.644104 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:00.644069 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3d6fe36-f010-4322-95af-a7e419f76d8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42907744-t2knt\" (UID: \"c3d6fe36-f010-4322-95af-a7e419f76d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42907744-t2knt" Oct 31 04:50:00.644308 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:00.644266 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3d6fe36-f010-4322-95af-a7e419f76d8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42907744-t2knt\" (UID: \"c3d6fe36-f010-4322-95af-a7e419f76d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42907744-t2knt" Oct 31 04:50:00.644446 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:00.644393 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw7p4\" (UniqueName: \"kubernetes.io/projected/c3d6fe36-f010-4322-95af-a7e419f76d8d-kube-api-access-qw7p4\") pod \"maas-api-revoke-sub-42907744-t2knt\" (UID: \"c3d6fe36-f010-4322-95af-a7e419f76d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42907744-t2knt" Oct 31 04:50:00.645124 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:00.645096 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3d6fe36-f010-4322-95af-a7e419f76d8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42907744-t2knt\" (UID: \"c3d6fe36-f010-4322-95af-a7e419f76d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42907744-t2knt" Oct 31 04:50:00.646661 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:00.646640 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3d6fe36-f010-4322-95af-a7e419f76d8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42907744-t2knt\" (UID: \"c3d6fe36-f010-4322-95af-a7e419f76d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42907744-t2knt" Oct 31 04:50:00.667467 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:00.667442 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw7p4\" (UniqueName: \"kubernetes.io/projected/c3d6fe36-f010-4322-95af-a7e419f76d8d-kube-api-access-qw7p4\") pod \"maas-api-revoke-sub-42907744-t2knt\" (UID: \"c3d6fe36-f010-4322-95af-a7e419f76d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42907744-t2knt" Oct 31 04:50:00.826303 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:00.826223 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42907744-t2knt" Oct 31 04:50:01.138083 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:50:01.138040 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3d6fe36_f010_4322_95af_a7e419f76d8d.slice/crio-ea38d746e7fd64858933f40d78523e7416e1ee5e7af5177846e2cdbf627cc837 WatchSource:0}: Error finding container ea38d746e7fd64858933f40d78523e7416e1ee5e7af5177846e2cdbf627cc837: Status 404 returned error can't find the container with id ea38d746e7fd64858933f40d78523e7416e1ee5e7af5177846e2cdbf627cc837 Oct 31 04:50:01.141407 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:01.141382 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42907744-t2knt"] Oct 31 04:50:01.752878 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:01.752839 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42907744-t2knt" event={"ID":"c3d6fe36-f010-4322-95af-a7e419f76d8d","Type":"ContainerStarted","Data":"3af4c4e24bfd30d2da9b238c03217ca6e943486cf2f41200c79178e919e1584c"} Oct 31 04:50:01.753274 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:01.752903 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42907744-t2knt" event={"ID":"c3d6fe36-f010-4322-95af-a7e419f76d8d","Type":"ContainerStarted","Data":"ea38d746e7fd64858933f40d78523e7416e1ee5e7af5177846e2cdbf627cc837"} Oct 31 04:50:01.798140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:01.798092 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42907744-t2knt" podStartSLOduration=1.7980788300000001 podStartE2EDuration="1.79807883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:50:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:50:01.794795876 +0000 UTC m=+2756.551133917" watchObservedRunningTime="2026-10-31 04:50:01.79807883 +0000 UTC m=+2756.554416939" Oct 31 04:50:02.764937 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:02.764887 5019 generic.go:356] "Generic (PLEG): container finished" podID="c3d6fe36-f010-4322-95af-a7e419f76d8d" containerID="3af4c4e24bfd30d2da9b238c03217ca6e943486cf2f41200c79178e919e1584c" exitCode=0 Oct 31 04:50:02.764937 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:02.764923 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42907744-t2knt" event={"ID":"c3d6fe36-f010-4322-95af-a7e419f76d8d","Type":"ContainerDied","Data":"3af4c4e24bfd30d2da9b238c03217ca6e943486cf2f41200c79178e919e1584c"} Oct 31 04:50:03.984505 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:03.984472 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42907744-t2knt" Oct 31 04:50:04.116793 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:04.116676 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3d6fe36-f010-4322-95af-a7e419f76d8d-kube-api-access-qw7p4\" (UniqueName: \"kubernetes.io/projected/c3d6fe36-f010-4322-95af-a7e419f76d8d-kube-api-access-qw7p4\") pod \"c3d6fe36-f010-4322-95af-a7e419f76d8d\" (UID: \"c3d6fe36-f010-4322-95af-a7e419f76d8d\") " Oct 31 04:50:04.116995 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:04.116883 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3d6fe36-f010-4322-95af-a7e419f76d8d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3d6fe36-f010-4322-95af-a7e419f76d8d-maas-api-openshift-ca\") pod \"c3d6fe36-f010-4322-95af-a7e419f76d8d\" (UID: \"c3d6fe36-f010-4322-95af-a7e419f76d8d\") " Oct 31 04:50:04.116995 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:04.116949 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3d6fe36-f010-4322-95af-a7e419f76d8d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3d6fe36-f010-4322-95af-a7e419f76d8d-maas-api-xks-ca\") pod \"c3d6fe36-f010-4322-95af-a7e419f76d8d\" (UID: \"c3d6fe36-f010-4322-95af-a7e419f76d8d\") " Oct 31 04:50:04.117300 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:04.117250 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3d6fe36-f010-4322-95af-a7e419f76d8d-maas-api-openshift-ca" pod "c3d6fe36-f010-4322-95af-a7e419f76d8d" (UID: "c3d6fe36-f010-4322-95af-a7e419f76d8d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:50:04.117954 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:04.117925 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3d6fe36-f010-4322-95af-a7e419f76d8d-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:50:04.119177 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:04.119150 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3d6fe36-f010-4322-95af-a7e419f76d8d-maas-api-xks-ca" pod "c3d6fe36-f010-4322-95af-a7e419f76d8d" (UID: "c3d6fe36-f010-4322-95af-a7e419f76d8d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:50:04.119269 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:04.119199 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3d6fe36-f010-4322-95af-a7e419f76d8d-kube-api-access-qw7p4" pod "c3d6fe36-f010-4322-95af-a7e419f76d8d" (UID: "c3d6fe36-f010-4322-95af-a7e419f76d8d"). InnerVolumeSpecName "kube-api-access-qw7p4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:50:04.219472 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:04.219431 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qw7p4\" (UniqueName: \"kubernetes.io/projected/c3d6fe36-f010-4322-95af-a7e419f76d8d-kube-api-access-qw7p4\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:50:04.219472 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:04.219465 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3d6fe36-f010-4322-95af-a7e419f76d8d-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:50:04.785389 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:04.785308 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42907744-t2knt" event={"ID":"c3d6fe36-f010-4322-95af-a7e419f76d8d","Type":"ContainerDied","Data":"ea38d746e7fd64858933f40d78523e7416e1ee5e7af5177846e2cdbf627cc837"} Oct 31 04:50:04.785584 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:04.785400 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42907744-t2knt" Oct 31 04:50:04.785584 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:04.785450 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea38d746e7fd64858933f40d78523e7416e1ee5e7af5177846e2cdbf627cc837" Oct 31 04:50:11.785089 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:11.785034 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-55776gtzl"] Oct 31 04:50:16.785858 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:16.785800 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-55776gtzl" podUID="66059d76-7532-4d81-80c6-7581d9af4bbc" containerName="payload-processing" containerID="cri-o://1a22ec0034a0ac00f9860a4f5725625e0439899456a4a368aacd13703490e8d2" gracePeriod=25 Oct 31 04:50:16.912871 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:16.912824 5019 generic.go:356] "Generic (PLEG): container finished" podID="66059d76-7532-4d81-80c6-7581d9af4bbc" containerID="1a22ec0034a0ac00f9860a4f5725625e0439899456a4a368aacd13703490e8d2" exitCode=0 Oct 31 04:50:16.913081 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:16.912879 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-55776gtzl" event={"ID":"66059d76-7532-4d81-80c6-7581d9af4bbc","Type":"ContainerDied","Data":"1a22ec0034a0ac00f9860a4f5725625e0439899456a4a368aacd13703490e8d2"} Oct 31 04:50:17.079654 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:17.079621 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-55776gtzl" Oct 31 04:50:17.240557 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:17.240521 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66059d76-7532-4d81-80c6-7581d9af4bbc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66059d76-7532-4d81-80c6-7581d9af4bbc-plugins-config-volume\") pod \"66059d76-7532-4d81-80c6-7581d9af4bbc\" (UID: \"66059d76-7532-4d81-80c6-7581d9af4bbc\") " Oct 31 04:50:17.240771 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:17.240639 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66059d76-7532-4d81-80c6-7581d9af4bbc-kube-api-access-fzvbc\" (UniqueName: \"kubernetes.io/projected/66059d76-7532-4d81-80c6-7581d9af4bbc-kube-api-access-fzvbc\") pod \"66059d76-7532-4d81-80c6-7581d9af4bbc\" (UID: \"66059d76-7532-4d81-80c6-7581d9af4bbc\") " Oct 31 04:50:17.240860 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:17.240843 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66059d76-7532-4d81-80c6-7581d9af4bbc-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/66059d76-7532-4d81-80c6-7581d9af4bbc-routing-overlay-volume\") pod \"66059d76-7532-4d81-80c6-7581d9af4bbc\" (UID: \"66059d76-7532-4d81-80c6-7581d9af4bbc\") " Oct 31 04:50:17.240929 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:17.240905 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66059d76-7532-4d81-80c6-7581d9af4bbc-provider-credentials\" (UniqueName: \"kubernetes.io/projected/66059d76-7532-4d81-80c6-7581d9af4bbc-provider-credentials\") pod \"66059d76-7532-4d81-80c6-7581d9af4bbc\" (UID: \"66059d76-7532-4d81-80c6-7581d9af4bbc\") " Oct 31 04:50:17.240985 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:17.240911 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66059d76-7532-4d81-80c6-7581d9af4bbc-plugins-config-volume" pod "66059d76-7532-4d81-80c6-7581d9af4bbc" (UID: "66059d76-7532-4d81-80c6-7581d9af4bbc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:50:17.241235 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:17.241213 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66059d76-7532-4d81-80c6-7581d9af4bbc-routing-overlay-volume" pod "66059d76-7532-4d81-80c6-7581d9af4bbc" (UID: "66059d76-7532-4d81-80c6-7581d9af4bbc"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:50:17.241649 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:17.241630 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66059d76-7532-4d81-80c6-7581d9af4bbc-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:50:17.241742 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:17.241656 5019 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/66059d76-7532-4d81-80c6-7581d9af4bbc-routing-overlay-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:50:17.243056 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:17.243036 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66059d76-7532-4d81-80c6-7581d9af4bbc-provider-credentials" pod "66059d76-7532-4d81-80c6-7581d9af4bbc" (UID: "66059d76-7532-4d81-80c6-7581d9af4bbc"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:50:17.243243 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:17.243224 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66059d76-7532-4d81-80c6-7581d9af4bbc-kube-api-access-fzvbc" pod "66059d76-7532-4d81-80c6-7581d9af4bbc" (UID: "66059d76-7532-4d81-80c6-7581d9af4bbc"). InnerVolumeSpecName "kube-api-access-fzvbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:50:17.348163 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:17.348065 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzvbc\" (UniqueName: \"kubernetes.io/projected/66059d76-7532-4d81-80c6-7581d9af4bbc-kube-api-access-fzvbc\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:50:17.348423 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:17.348403 5019 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/66059d76-7532-4d81-80c6-7581d9af4bbc-provider-credentials\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:50:17.928221 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:17.928179 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-55776gtzl" Oct 31 04:50:17.928221 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:17.928201 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-55776gtzl" event={"ID":"66059d76-7532-4d81-80c6-7581d9af4bbc","Type":"ContainerDied","Data":"b42c07de78243a02340d39b112a5d2edeee0c6ae9ef8926bae385ebb687555e1"} Oct 31 04:50:17.928925 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:17.928270 5019 scope.go:122] "RemoveContainer" containerID="1a22ec0034a0ac00f9860a4f5725625e0439899456a4a368aacd13703490e8d2" Oct 31 04:50:18.008288 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:18.008239 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-55776gtzl"] Oct 31 04:50:18.024905 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:18.024863 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-55776gtzl"] Oct 31 04:50:19.410734 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:19.410695 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66059d76-7532-4d81-80c6-7581d9af4bbc" path="/var/lib/kubelet/pods/66059d76-7532-4d81-80c6-7581d9af4bbc/volumes" Oct 31 04:50:20.156102 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:20.156059 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b6cfebd-q5h6g"] Oct 31 04:50:20.156980 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:20.156960 5019 cpu_manager.go:414] "containerMap: removing container" podUID="c3d6fe36-f010-4322-95af-a7e419f76d8d" containerName="revoke-keys" Oct 31 04:50:20.157050 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:20.156982 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3d6fe36-f010-4322-95af-a7e419f76d8d" containerName="revoke-keys" Oct 31 04:50:20.157050 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:20.157004 5019 cpu_manager.go:414] "containerMap: removing container" podUID="66059d76-7532-4d81-80c6-7581d9af4bbc" containerName="payload-processing" Oct 31 04:50:20.157050 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:20.157010 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66059d76-7532-4d81-80c6-7581d9af4bbc" containerName="payload-processing" Oct 31 04:50:20.157188 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:20.157178 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3d6fe36-f010-4322-95af-a7e419f76d8d" containerName="revoke-keys" Oct 31 04:50:20.157230 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:20.157195 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="66059d76-7532-4d81-80c6-7581d9af4bbc" containerName="payload-processing" Oct 31 04:50:20.170135 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:20.170100 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b6cfebd-q5h6g" Oct 31 04:50:20.193697 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:20.192841 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:50:20.193697 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:20.192915 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:50:20.193697 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:20.192979 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:50:20.217598 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:20.216820 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:50:20.220808 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:20.220595 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a99d6b5-ee66-4b2a-b55c-1fcd5f0f6130-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b6cfebd-q5h6g\" (UID: \"9a99d6b5-ee66-4b2a-b55c-1fcd5f0f6130\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b6cfebd-q5h6g" Oct 31 04:50:20.220808 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:20.220735 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm8z2\" (UniqueName: \"kubernetes.io/projected/9a99d6b5-ee66-4b2a-b55c-1fcd5f0f6130-kube-api-access-cm8z2\") pod \"maas-api-revoke-sub-0b6cfebd-q5h6g\" (UID: \"9a99d6b5-ee66-4b2a-b55c-1fcd5f0f6130\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b6cfebd-q5h6g" Oct 31 04:50:20.221090 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:20.221060 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a99d6b5-ee66-4b2a-b55c-1fcd5f0f6130-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b6cfebd-q5h6g\" (UID: \"9a99d6b5-ee66-4b2a-b55c-1fcd5f0f6130\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b6cfebd-q5h6g" Oct 31 04:50:20.229405 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:20.229372 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b6cfebd-q5h6g"] Oct 31 04:50:20.323396 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:20.322424 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a99d6b5-ee66-4b2a-b55c-1fcd5f0f6130-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b6cfebd-q5h6g\" (UID: \"9a99d6b5-ee66-4b2a-b55c-1fcd5f0f6130\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b6cfebd-q5h6g" Oct 31 04:50:20.323396 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:20.322545 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a99d6b5-ee66-4b2a-b55c-1fcd5f0f6130-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b6cfebd-q5h6g\" (UID: \"9a99d6b5-ee66-4b2a-b55c-1fcd5f0f6130\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b6cfebd-q5h6g" Oct 31 04:50:20.323396 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:20.322663 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm8z2\" (UniqueName: \"kubernetes.io/projected/9a99d6b5-ee66-4b2a-b55c-1fcd5f0f6130-kube-api-access-cm8z2\") pod \"maas-api-revoke-sub-0b6cfebd-q5h6g\" (UID: \"9a99d6b5-ee66-4b2a-b55c-1fcd5f0f6130\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b6cfebd-q5h6g" Oct 31 04:50:20.331393 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:20.323882 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a99d6b5-ee66-4b2a-b55c-1fcd5f0f6130-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b6cfebd-q5h6g\" (UID: \"9a99d6b5-ee66-4b2a-b55c-1fcd5f0f6130\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b6cfebd-q5h6g" Oct 31 04:50:20.332771 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:20.332361 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a99d6b5-ee66-4b2a-b55c-1fcd5f0f6130-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b6cfebd-q5h6g\" (UID: \"9a99d6b5-ee66-4b2a-b55c-1fcd5f0f6130\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b6cfebd-q5h6g" Oct 31 04:50:20.347043 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:20.347011 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm8z2\" (UniqueName: \"kubernetes.io/projected/9a99d6b5-ee66-4b2a-b55c-1fcd5f0f6130-kube-api-access-cm8z2\") pod \"maas-api-revoke-sub-0b6cfebd-q5h6g\" (UID: \"9a99d6b5-ee66-4b2a-b55c-1fcd5f0f6130\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b6cfebd-q5h6g" Oct 31 04:50:20.531380 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:20.531341 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b6cfebd-q5h6g" Oct 31 04:50:20.848196 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:50:20.848161 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a99d6b5_ee66_4b2a_b55c_1fcd5f0f6130.slice/crio-9e97d0cfc87c96409324df123c8e8e42f2605a03ee7b9e057d3de6688b655770 WatchSource:0}: Error finding container 9e97d0cfc87c96409324df123c8e8e42f2605a03ee7b9e057d3de6688b655770: Status 404 returned error can't find the container with id 9e97d0cfc87c96409324df123c8e8e42f2605a03ee7b9e057d3de6688b655770 Oct 31 04:50:20.849895 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:20.849865 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b6cfebd-q5h6g"] Oct 31 04:50:20.979872 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:20.979836 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b6cfebd-q5h6g" event={"ID":"9a99d6b5-ee66-4b2a-b55c-1fcd5f0f6130","Type":"ContainerStarted","Data":"9e97d0cfc87c96409324df123c8e8e42f2605a03ee7b9e057d3de6688b655770"} Oct 31 04:50:21.991139 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:21.991098 5019 generic.go:356] "Generic (PLEG): container finished" podID="9a99d6b5-ee66-4b2a-b55c-1fcd5f0f6130" containerID="2f34943474931ccc257e133d9060b397941bb412a56708e245f6e09fd4b8beb0" exitCode=0 Oct 31 04:50:21.991638 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:21.991178 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b6cfebd-q5h6g" event={"ID":"9a99d6b5-ee66-4b2a-b55c-1fcd5f0f6130","Type":"ContainerDied","Data":"2f34943474931ccc257e133d9060b397941bb412a56708e245f6e09fd4b8beb0"} Oct 31 04:50:22.396905 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:50:22.396861 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:50:23.212644 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:23.212612 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b6cfebd-q5h6g" Oct 31 04:50:23.291617 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:23.291513 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a99d6b5-ee66-4b2a-b55c-1fcd5f0f6130-kube-api-access-cm8z2\" (UniqueName: \"kubernetes.io/projected/9a99d6b5-ee66-4b2a-b55c-1fcd5f0f6130-kube-api-access-cm8z2\") pod \"9a99d6b5-ee66-4b2a-b55c-1fcd5f0f6130\" (UID: \"9a99d6b5-ee66-4b2a-b55c-1fcd5f0f6130\") " Oct 31 04:50:23.292231 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:23.292173 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a99d6b5-ee66-4b2a-b55c-1fcd5f0f6130-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a99d6b5-ee66-4b2a-b55c-1fcd5f0f6130-maas-api-openshift-ca\") pod \"9a99d6b5-ee66-4b2a-b55c-1fcd5f0f6130\" (UID: \"9a99d6b5-ee66-4b2a-b55c-1fcd5f0f6130\") " Oct 31 04:50:23.292449 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:23.292429 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a99d6b5-ee66-4b2a-b55c-1fcd5f0f6130-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a99d6b5-ee66-4b2a-b55c-1fcd5f0f6130-maas-api-xks-ca\") pod \"9a99d6b5-ee66-4b2a-b55c-1fcd5f0f6130\" (UID: \"9a99d6b5-ee66-4b2a-b55c-1fcd5f0f6130\") " Oct 31 04:50:23.292727 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:23.292682 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a99d6b5-ee66-4b2a-b55c-1fcd5f0f6130-maas-api-openshift-ca" pod "9a99d6b5-ee66-4b2a-b55c-1fcd5f0f6130" (UID: "9a99d6b5-ee66-4b2a-b55c-1fcd5f0f6130"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:50:23.293689 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:23.293659 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a99d6b5-ee66-4b2a-b55c-1fcd5f0f6130-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:50:23.294784 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:23.294747 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a99d6b5-ee66-4b2a-b55c-1fcd5f0f6130-kube-api-access-cm8z2" pod "9a99d6b5-ee66-4b2a-b55c-1fcd5f0f6130" (UID: "9a99d6b5-ee66-4b2a-b55c-1fcd5f0f6130"). InnerVolumeSpecName "kube-api-access-cm8z2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:50:23.303837 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:23.303804 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a99d6b5-ee66-4b2a-b55c-1fcd5f0f6130-maas-api-xks-ca" pod "9a99d6b5-ee66-4b2a-b55c-1fcd5f0f6130" (UID: "9a99d6b5-ee66-4b2a-b55c-1fcd5f0f6130"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:50:23.395301 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:23.395262 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a99d6b5-ee66-4b2a-b55c-1fcd5f0f6130-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:50:23.395301 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:23.395298 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm8z2\" (UniqueName: \"kubernetes.io/projected/9a99d6b5-ee66-4b2a-b55c-1fcd5f0f6130-kube-api-access-cm8z2\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:50:24.017191 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:24.017158 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b6cfebd-q5h6g" Oct 31 04:50:24.017439 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:24.017185 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b6cfebd-q5h6g" event={"ID":"9a99d6b5-ee66-4b2a-b55c-1fcd5f0f6130","Type":"ContainerDied","Data":"9e97d0cfc87c96409324df123c8e8e42f2605a03ee7b9e057d3de6688b655770"} Oct 31 04:50:24.017439 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:24.017249 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e97d0cfc87c96409324df123c8e8e42f2605a03ee7b9e057d3de6688b655770" Oct 31 04:50:25.032628 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:25.032591 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e7fa7b6-nlmpq"] Oct 31 04:50:25.044521 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:25.044484 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e7fa7b6-nlmpq"] Oct 31 04:50:25.429135 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:25.429055 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9192eff-e3eb-4bef-a0e7-447afe65cc78" path="/var/lib/kubelet/pods/a9192eff-e3eb-4bef-a0e7-447afe65cc78/volumes" Oct 31 04:50:28.690626 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:28.690589 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bc4gchxx"] Oct 31 04:50:28.692485 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:28.692455 5019 cpu_manager.go:414] "containerMap: removing container" podUID="9a99d6b5-ee66-4b2a-b55c-1fcd5f0f6130" containerName="revoke-keys" Oct 31 04:50:28.692639 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:28.692626 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a99d6b5-ee66-4b2a-b55c-1fcd5f0f6130" containerName="revoke-keys" Oct 31 04:50:28.693051 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:28.693035 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a99d6b5-ee66-4b2a-b55c-1fcd5f0f6130" containerName="revoke-keys" Oct 31 04:50:28.702021 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:28.701999 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bc4gchxx"] Oct 31 04:50:28.702144 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:28.702131 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bc4gchxx" Oct 31 04:50:28.717394 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:28.717368 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ded435\"/\"payload-processing-external-model-e2e--b0123f89-dockercfg-54866\"" Oct 31 04:50:28.717637 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:28.717613 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ded435\"/\"agc-cred-425c31a80fac\"" Oct 31 04:50:28.729546 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:28.729520 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ded435\"/\"routing-overlay\"" Oct 31 04:50:28.729697 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:28.729532 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ded435\"/\"kube-root-ca.crt\"" Oct 31 04:50:28.729697 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:28.729533 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ded435\"/\"payload-processing-external-model-plugins-e2e-extmod-b-ded435\"" Oct 31 04:50:28.729697 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:28.729617 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ded435\"/\"openshift-service-ca.crt\"" Oct 31 04:50:28.854357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:28.849389 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/23409f1d-b628-4669-a2f8-3363bdc6b09b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-5bc4gchxx\" (UID: \"23409f1d-b628-4669-a2f8-3363bdc6b09b\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bc4gchxx" Oct 31 04:50:28.854357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:28.849555 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/23409f1d-b628-4669-a2f8-3363bdc6b09b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-5bc4gchxx\" (UID: \"23409f1d-b628-4669-a2f8-3363bdc6b09b\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bc4gchxx" Oct 31 04:50:28.854357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:28.849706 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23409f1d-b628-4669-a2f8-3363bdc6b09b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-5bc4gchxx\" (UID: \"23409f1d-b628-4669-a2f8-3363bdc6b09b\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bc4gchxx" Oct 31 04:50:28.854357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:28.849738 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdfx5\" (UniqueName: \"kubernetes.io/projected/23409f1d-b628-4669-a2f8-3363bdc6b09b-kube-api-access-qdfx5\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-5bc4gchxx\" (UID: \"23409f1d-b628-4669-a2f8-3363bdc6b09b\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bc4gchxx" Oct 31 04:50:28.953411 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:28.951532 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23409f1d-b628-4669-a2f8-3363bdc6b09b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-5bc4gchxx\" (UID: \"23409f1d-b628-4669-a2f8-3363bdc6b09b\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bc4gchxx" Oct 31 04:50:28.953411 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:28.951597 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdfx5\" (UniqueName: \"kubernetes.io/projected/23409f1d-b628-4669-a2f8-3363bdc6b09b-kube-api-access-qdfx5\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-5bc4gchxx\" (UID: \"23409f1d-b628-4669-a2f8-3363bdc6b09b\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bc4gchxx" Oct 31 04:50:28.953411 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:28.951908 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/23409f1d-b628-4669-a2f8-3363bdc6b09b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-5bc4gchxx\" (UID: \"23409f1d-b628-4669-a2f8-3363bdc6b09b\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bc4gchxx" Oct 31 04:50:28.953411 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:28.952046 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/23409f1d-b628-4669-a2f8-3363bdc6b09b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-5bc4gchxx\" (UID: \"23409f1d-b628-4669-a2f8-3363bdc6b09b\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bc4gchxx" Oct 31 04:50:28.953837 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:28.953534 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/23409f1d-b628-4669-a2f8-3363bdc6b09b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-5bc4gchxx\" (UID: \"23409f1d-b628-4669-a2f8-3363bdc6b09b\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bc4gchxx" Oct 31 04:50:28.965363 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:28.965316 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23409f1d-b628-4669-a2f8-3363bdc6b09b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-5bc4gchxx\" (UID: \"23409f1d-b628-4669-a2f8-3363bdc6b09b\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bc4gchxx" Oct 31 04:50:29.006846 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:29.006808 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdfx5\" (UniqueName: \"kubernetes.io/projected/23409f1d-b628-4669-a2f8-3363bdc6b09b-kube-api-access-qdfx5\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-5bc4gchxx\" (UID: \"23409f1d-b628-4669-a2f8-3363bdc6b09b\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bc4gchxx" Oct 31 04:50:29.017007 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:29.016973 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/23409f1d-b628-4669-a2f8-3363bdc6b09b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-5bc4gchxx\" (UID: \"23409f1d-b628-4669-a2f8-3363bdc6b09b\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bc4gchxx" Oct 31 04:50:29.019318 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:29.019089 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bc4gchxx" Oct 31 04:50:29.396074 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:50:29.396035 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23409f1d_b628_4669_a2f8_3363bdc6b09b.slice/crio-97606501aa74c573fa6a91e234182114c61844cc4e6bc2037ce73eecf4a9ce2a WatchSource:0}: Error finding container 97606501aa74c573fa6a91e234182114c61844cc4e6bc2037ce73eecf4a9ce2a: Status 404 returned error can't find the container with id 97606501aa74c573fa6a91e234182114c61844cc4e6bc2037ce73eecf4a9ce2a Oct 31 04:50:29.398858 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:29.398832 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bc4gchxx"] Oct 31 04:50:29.777683 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:29.777060 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-74fckxtnc"] Oct 31 04:50:29.783792 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:29.783766 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-74fckxtnc" Oct 31 04:50:29.791423 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:29.791299 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5606bd96-b466-42d3-9eb8-fffe43b65bff-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-654741-74fckxtnc\" (UID: \"5606bd96-b466-42d3-9eb8-fffe43b65bff\") " pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-74fckxtnc" Oct 31 04:50:29.791423 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:29.791373 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5606bd96-b466-42d3-9eb8-fffe43b65bff-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-654741-74fckxtnc\" (UID: \"5606bd96-b466-42d3-9eb8-fffe43b65bff\") " pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-74fckxtnc" Oct 31 04:50:29.791665 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:29.791640 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5606bd96-b466-42d3-9eb8-fffe43b65bff-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-654741-74fckxtnc\" (UID: \"5606bd96-b466-42d3-9eb8-fffe43b65bff\") " pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-74fckxtnc" Oct 31 04:50:29.791721 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:29.791689 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvx4c\" (UniqueName: \"kubernetes.io/projected/5606bd96-b466-42d3-9eb8-fffe43b65bff-kube-api-access-vvx4c\") pod \"payload-processing-external-model-e2e-extmod-a-654741-74fckxtnc\" (UID: \"5606bd96-b466-42d3-9eb8-fffe43b65bff\") " pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-74fckxtnc" Oct 31 04:50:29.799795 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:29.799769 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-74fckxtnc"] Oct 31 04:50:29.805489 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:29.805467 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-654741\"/\"agc-cred-8c9201ffc741\"" Oct 31 04:50:29.805643 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:29.805500 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-654741\"/\"openshift-service-ca.crt\"" Oct 31 04:50:29.805763 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:29.805702 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-654741\"/\"kube-root-ca.crt\"" Oct 31 04:50:29.810243 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:29.810203 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-654741\"/\"payload-processing-external-model-e2e--dfe12206-dockercfg-jwsgv\"" Oct 31 04:50:29.810383 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:29.810281 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-654741\"/\"routing-overlay\"" Oct 31 04:50:29.810463 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:29.810385 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-654741\"/\"payload-processing-external-model-plugins-e2e-extmod-a-654741\"" Oct 31 04:50:29.893239 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:29.893198 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5606bd96-b466-42d3-9eb8-fffe43b65bff-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-654741-74fckxtnc\" (UID: \"5606bd96-b466-42d3-9eb8-fffe43b65bff\") " pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-74fckxtnc" Oct 31 04:50:29.893489 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:29.893243 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvx4c\" (UniqueName: \"kubernetes.io/projected/5606bd96-b466-42d3-9eb8-fffe43b65bff-kube-api-access-vvx4c\") pod \"payload-processing-external-model-e2e-extmod-a-654741-74fckxtnc\" (UID: \"5606bd96-b466-42d3-9eb8-fffe43b65bff\") " pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-74fckxtnc" Oct 31 04:50:29.893489 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:29.893411 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5606bd96-b466-42d3-9eb8-fffe43b65bff-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-654741-74fckxtnc\" (UID: \"5606bd96-b466-42d3-9eb8-fffe43b65bff\") " pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-74fckxtnc" Oct 31 04:50:29.893489 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:29.893462 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5606bd96-b466-42d3-9eb8-fffe43b65bff-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-654741-74fckxtnc\" (UID: \"5606bd96-b466-42d3-9eb8-fffe43b65bff\") " pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-74fckxtnc" Oct 31 04:50:29.894120 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:29.894091 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5606bd96-b466-42d3-9eb8-fffe43b65bff-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-654741-74fckxtnc\" (UID: \"5606bd96-b466-42d3-9eb8-fffe43b65bff\") " pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-74fckxtnc" Oct 31 04:50:29.894263 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:29.894120 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5606bd96-b466-42d3-9eb8-fffe43b65bff-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-654741-74fckxtnc\" (UID: \"5606bd96-b466-42d3-9eb8-fffe43b65bff\") " pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-74fckxtnc" Oct 31 04:50:29.896052 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:29.896029 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5606bd96-b466-42d3-9eb8-fffe43b65bff-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-654741-74fckxtnc\" (UID: \"5606bd96-b466-42d3-9eb8-fffe43b65bff\") " pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-74fckxtnc" Oct 31 04:50:29.916126 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:29.916098 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvx4c\" (UniqueName: \"kubernetes.io/projected/5606bd96-b466-42d3-9eb8-fffe43b65bff-kube-api-access-vvx4c\") pod \"payload-processing-external-model-e2e-extmod-a-654741-74fckxtnc\" (UID: \"5606bd96-b466-42d3-9eb8-fffe43b65bff\") " pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-74fckxtnc" Oct 31 04:50:30.106176 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:30.101631 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-74fckxtnc" Oct 31 04:50:30.228671 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:30.227772 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bc4gchxx" event={"ID":"23409f1d-b628-4669-a2f8-3363bdc6b09b","Type":"ContainerStarted","Data":"d724386f7d82034ff29c6fa11b2b3b0c5e41f5dbbd843684c61565d45017690f"} Oct 31 04:50:30.228671 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:30.227839 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bc4gchxx" event={"ID":"23409f1d-b628-4669-a2f8-3363bdc6b09b","Type":"ContainerStarted","Data":"97606501aa74c573fa6a91e234182114c61844cc4e6bc2037ce73eecf4a9ce2a"} Oct 31 04:50:30.228671 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:30.228554 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bc4gchxx" Oct 31 04:50:30.294108 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:30.294046 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bc4gchxx" podStartSLOduration=2.294028808 podStartE2EDuration="2.294028808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:50:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:50:30.288260517 +0000 UTC m=+2785.044598557" watchObservedRunningTime="2026-10-31 04:50:30.294028808 +0000 UTC m=+2785.050366849" Oct 31 04:50:30.510499 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:50:30.510462 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5606bd96_b466_42d3_9eb8_fffe43b65bff.slice/crio-a6fdd51578a9d36d15be3a870ff35171ca77d83839ff2cea474ecd2e93d13e4a WatchSource:0}: Error finding container a6fdd51578a9d36d15be3a870ff35171ca77d83839ff2cea474ecd2e93d13e4a: Status 404 returned error can't find the container with id a6fdd51578a9d36d15be3a870ff35171ca77d83839ff2cea474ecd2e93d13e4a Oct 31 04:50:30.511702 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:30.511664 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-74fckxtnc"] Oct 31 04:50:31.030957 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:31.030919 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d812a3a9-zhkql"] Oct 31 04:50:31.036701 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:31.036668 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d812a3a9-zhkql"] Oct 31 04:50:31.242604 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:31.242551 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-74fckxtnc" event={"ID":"5606bd96-b466-42d3-9eb8-fffe43b65bff","Type":"ContainerStarted","Data":"4422794e18d1a9e77fc037eec3c2866e0e975344d3e6f9f14be54c894a1f0d1f"} Oct 31 04:50:31.242814 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:31.242621 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-74fckxtnc" event={"ID":"5606bd96-b466-42d3-9eb8-fffe43b65bff","Type":"ContainerStarted","Data":"a6fdd51578a9d36d15be3a870ff35171ca77d83839ff2cea474ecd2e93d13e4a"} Oct 31 04:50:31.287053 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:31.286925 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-74fckxtnc" podStartSLOduration=2.286909398 podStartE2EDuration="2.286909398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:50:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:50:31.283842412 +0000 UTC m=+2786.040180446" watchObservedRunningTime="2026-10-31 04:50:31.286909398 +0000 UTC m=+2786.043247437" Oct 31 04:50:31.415588 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:31.415543 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c68e7b82-71db-48e2-8d07-36ddb22e9ae7" path="/var/lib/kubelet/pods/c68e7b82-71db-48e2-8d07-36ddb22e9ae7/volumes" Oct 31 04:50:31.958149 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:31.958114 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-d6d49sdpr"] Oct 31 04:50:32.307366 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:32.305412 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-74fckxtnc" Oct 31 04:50:36.958671 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:36.958624 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-d6d49sdpr" podUID="44d5e932-9425-4f90-bf9a-94ac9bfe52b2" containerName="payload-processing" containerID="cri-o://55d75a8eab9bd1c58973c16f9f2be92a0fa4411b37c2d925faa98f760f6fa6cf" gracePeriod=25 Oct 31 04:50:37.288793 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:37.288761 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-d6d49sdpr" Oct 31 04:50:37.366130 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:37.366087 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44d5e932-9425-4f90-bf9a-94ac9bfe52b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44d5e932-9425-4f90-bf9a-94ac9bfe52b2-plugins-config-volume\") pod \"44d5e932-9425-4f90-bf9a-94ac9bfe52b2\" (UID: \"44d5e932-9425-4f90-bf9a-94ac9bfe52b2\") " Oct 31 04:50:37.366439 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:37.366416 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44d5e932-9425-4f90-bf9a-94ac9bfe52b2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/44d5e932-9425-4f90-bf9a-94ac9bfe52b2-routing-overlay-volume\") pod \"44d5e932-9425-4f90-bf9a-94ac9bfe52b2\" (UID: \"44d5e932-9425-4f90-bf9a-94ac9bfe52b2\") " Oct 31 04:50:37.366531 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:37.366495 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44d5e932-9425-4f90-bf9a-94ac9bfe52b2-kube-api-access-mzx2n\" (UniqueName: \"kubernetes.io/projected/44d5e932-9425-4f90-bf9a-94ac9bfe52b2-kube-api-access-mzx2n\") pod \"44d5e932-9425-4f90-bf9a-94ac9bfe52b2\" (UID: \"44d5e932-9425-4f90-bf9a-94ac9bfe52b2\") " Oct 31 04:50:37.366606 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:37.366584 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44d5e932-9425-4f90-bf9a-94ac9bfe52b2-plugins-config-volume" pod "44d5e932-9425-4f90-bf9a-94ac9bfe52b2" (UID: "44d5e932-9425-4f90-bf9a-94ac9bfe52b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:50:37.366680 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:37.366640 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44d5e932-9425-4f90-bf9a-94ac9bfe52b2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/44d5e932-9425-4f90-bf9a-94ac9bfe52b2-provider-credentials\") pod \"44d5e932-9425-4f90-bf9a-94ac9bfe52b2\" (UID: \"44d5e932-9425-4f90-bf9a-94ac9bfe52b2\") " Oct 31 04:50:37.366821 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:37.366795 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44d5e932-9425-4f90-bf9a-94ac9bfe52b2-routing-overlay-volume" pod "44d5e932-9425-4f90-bf9a-94ac9bfe52b2" (UID: "44d5e932-9425-4f90-bf9a-94ac9bfe52b2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:50:37.367252 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:37.367232 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44d5e932-9425-4f90-bf9a-94ac9bfe52b2-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:50:37.367252 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:37.367253 5019 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/44d5e932-9425-4f90-bf9a-94ac9bfe52b2-routing-overlay-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:50:37.368970 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:37.368940 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44d5e932-9425-4f90-bf9a-94ac9bfe52b2-kube-api-access-mzx2n" pod "44d5e932-9425-4f90-bf9a-94ac9bfe52b2" (UID: "44d5e932-9425-4f90-bf9a-94ac9bfe52b2"). InnerVolumeSpecName "kube-api-access-mzx2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:50:37.369091 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:37.368972 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44d5e932-9425-4f90-bf9a-94ac9bfe52b2-provider-credentials" pod "44d5e932-9425-4f90-bf9a-94ac9bfe52b2" (UID: "44d5e932-9425-4f90-bf9a-94ac9bfe52b2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:50:37.419393 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:37.419347 5019 generic.go:356] "Generic (PLEG): container finished" podID="44d5e932-9425-4f90-bf9a-94ac9bfe52b2" containerID="55d75a8eab9bd1c58973c16f9f2be92a0fa4411b37c2d925faa98f760f6fa6cf" exitCode=0 Oct 31 04:50:37.419623 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:37.419391 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-d6d49sdpr" event={"ID":"44d5e932-9425-4f90-bf9a-94ac9bfe52b2","Type":"ContainerDied","Data":"55d75a8eab9bd1c58973c16f9f2be92a0fa4411b37c2d925faa98f760f6fa6cf"} Oct 31 04:50:37.419623 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:37.419445 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-d6d49sdpr" Oct 31 04:50:37.419623 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:37.419460 5019 scope.go:122] "RemoveContainer" containerID="55d75a8eab9bd1c58973c16f9f2be92a0fa4411b37c2d925faa98f760f6fa6cf" Oct 31 04:50:37.419623 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:37.419450 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-d6d49sdpr" event={"ID":"44d5e932-9425-4f90-bf9a-94ac9bfe52b2","Type":"ContainerDied","Data":"4cd39f96054fbf9b291eac7ebeb54d08317ef6dce91f3cc5c1230f3ff4798375"} Oct 31 04:50:37.432688 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:37.432667 5019 scope.go:122] "RemoveContainer" containerID="55d75a8eab9bd1c58973c16f9f2be92a0fa4411b37c2d925faa98f760f6fa6cf" Oct 31 04:50:37.433001 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:50:37.432980 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55d75a8eab9bd1c58973c16f9f2be92a0fa4411b37c2d925faa98f760f6fa6cf\": container with ID starting with 55d75a8eab9bd1c58973c16f9f2be92a0fa4411b37c2d925faa98f760f6fa6cf not found: ID does not exist" containerID="55d75a8eab9bd1c58973c16f9f2be92a0fa4411b37c2d925faa98f760f6fa6cf" Oct 31 04:50:37.433052 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:37.433009 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55d75a8eab9bd1c58973c16f9f2be92a0fa4411b37c2d925faa98f760f6fa6cf"} err="failed to get container status \"55d75a8eab9bd1c58973c16f9f2be92a0fa4411b37c2d925faa98f760f6fa6cf\": rpc error: code = NotFound desc = could not find container \"55d75a8eab9bd1c58973c16f9f2be92a0fa4411b37c2d925faa98f760f6fa6cf\": container with ID starting with 55d75a8eab9bd1c58973c16f9f2be92a0fa4411b37c2d925faa98f760f6fa6cf not found: ID does not exist" Oct 31 04:50:37.469196 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:37.469162 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzx2n\" (UniqueName: \"kubernetes.io/projected/44d5e932-9425-4f90-bf9a-94ac9bfe52b2-kube-api-access-mzx2n\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:50:37.469196 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:37.469197 5019 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/44d5e932-9425-4f90-bf9a-94ac9bfe52b2-provider-credentials\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:50:37.475003 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:37.474969 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-d6d49sdpr"] Oct 31 04:50:37.480712 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:37.480683 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-d6d49sdpr"] Oct 31 04:50:38.035196 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:38.035156 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93f03b01-5tzb7"] Oct 31 04:50:38.040767 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:38.040733 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93f03b01-5tzb7"] Oct 31 04:50:39.415509 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:39.415470 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44d5e932-9425-4f90-bf9a-94ac9bfe52b2" path="/var/lib/kubelet/pods/44d5e932-9425-4f90-bf9a-94ac9bfe52b2/volumes" Oct 31 04:50:39.416256 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:39.416234 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="609c6d46-2c07-412b-b44f-7b2908fb93d2" path="/var/lib/kubelet/pods/609c6d46-2c07-412b-b44f-7b2908fb93d2/volumes" Oct 31 04:50:41.245891 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:41.245857 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bc4gchxx" Oct 31 04:50:42.045173 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:42.045137 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b181387a-z7q2c"] Oct 31 04:50:42.051223 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:42.051188 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b181387a-z7q2c"] Oct 31 04:50:43.344027 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:43.343990 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-74fckxtnc" Oct 31 04:50:43.415936 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:43.415896 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="846eb567-29ee-430d-9f65-007a72737fcd" path="/var/lib/kubelet/pods/846eb567-29ee-430d-9f65-007a72737fcd/volumes" Oct 31 04:50:52.401585 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:50:52.401545 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:50:54.059162 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:54.057388 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f4d9e2c-plnkf"] Oct 31 04:50:54.092210 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:54.092167 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f4d9e2c-plnkf"] Oct 31 04:50:55.419006 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:55.418968 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed0a8acb-f490-41fb-83e6-d9cd440e269c" path="/var/lib/kubelet/pods/ed0a8acb-f490-41fb-83e6-d9cd440e269c/volumes" Oct 31 04:50:57.246355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:57.237128 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" podUID="e0f4b8c2-14f4-4052-bf54-dd65bf59e413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 04:50:57.344290 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:57.344233 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" podUID="b4ced3ab-41e4-4c54-93eb-95eddaae992d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:50:57.527415 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:57.522442 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" podUID="643433d8-418e-4102-8bc3-cd544932614a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 04:50:57.527415 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:57.525041 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 31 04:50:57.527415 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:50:57.525367 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 31 04:51:00.651521 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:00.651486 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b803b38-8rvz9"] Oct 31 04:51:00.652496 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:00.652475 5019 cpu_manager.go:414] "containerMap: removing container" podUID="44d5e932-9425-4f90-bf9a-94ac9bfe52b2" containerName="payload-processing" Oct 31 04:51:00.652496 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:00.652495 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44d5e932-9425-4f90-bf9a-94ac9bfe52b2" containerName="payload-processing" Oct 31 04:51:00.652676 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:00.652665 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="44d5e932-9425-4f90-bf9a-94ac9bfe52b2" containerName="payload-processing" Oct 31 04:51:00.658485 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:00.658454 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b803b38-8rvz9" Oct 31 04:51:00.676461 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:00.676416 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b803b38-8rvz9"] Oct 31 04:51:00.679786 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:00.679757 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:51:00.680022 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:00.679862 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:51:00.680022 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:00.679911 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:51:00.680022 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:00.679966 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:51:00.738432 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:00.734234 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/980bcf39-71be-432a-b878-e5fbfdec5dba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b803b38-8rvz9\" (UID: \"980bcf39-71be-432a-b878-e5fbfdec5dba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b803b38-8rvz9" Oct 31 04:51:00.738432 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:00.734532 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/980bcf39-71be-432a-b878-e5fbfdec5dba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b803b38-8rvz9\" (UID: \"980bcf39-71be-432a-b878-e5fbfdec5dba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b803b38-8rvz9" Oct 31 04:51:00.738432 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:00.734615 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbd5n\" (UniqueName: \"kubernetes.io/projected/980bcf39-71be-432a-b878-e5fbfdec5dba-kube-api-access-sbd5n\") pod \"maas-api-revoke-sub-5b803b38-8rvz9\" (UID: \"980bcf39-71be-432a-b878-e5fbfdec5dba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b803b38-8rvz9" Oct 31 04:51:00.837359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:00.836659 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/980bcf39-71be-432a-b878-e5fbfdec5dba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b803b38-8rvz9\" (UID: \"980bcf39-71be-432a-b878-e5fbfdec5dba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b803b38-8rvz9" Oct 31 04:51:00.837359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:00.836764 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbd5n\" (UniqueName: \"kubernetes.io/projected/980bcf39-71be-432a-b878-e5fbfdec5dba-kube-api-access-sbd5n\") pod \"maas-api-revoke-sub-5b803b38-8rvz9\" (UID: \"980bcf39-71be-432a-b878-e5fbfdec5dba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b803b38-8rvz9" Oct 31 04:51:00.837359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:00.837080 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/980bcf39-71be-432a-b878-e5fbfdec5dba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b803b38-8rvz9\" (UID: \"980bcf39-71be-432a-b878-e5fbfdec5dba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b803b38-8rvz9" Oct 31 04:51:00.838129 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:00.838091 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/980bcf39-71be-432a-b878-e5fbfdec5dba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b803b38-8rvz9\" (UID: \"980bcf39-71be-432a-b878-e5fbfdec5dba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b803b38-8rvz9" Oct 31 04:51:00.844354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:00.841250 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/980bcf39-71be-432a-b878-e5fbfdec5dba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b803b38-8rvz9\" (UID: \"980bcf39-71be-432a-b878-e5fbfdec5dba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b803b38-8rvz9" Oct 31 04:51:00.902643 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:00.895137 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbd5n\" (UniqueName: \"kubernetes.io/projected/980bcf39-71be-432a-b878-e5fbfdec5dba-kube-api-access-sbd5n\") pod \"maas-api-revoke-sub-5b803b38-8rvz9\" (UID: \"980bcf39-71be-432a-b878-e5fbfdec5dba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b803b38-8rvz9" Oct 31 04:51:00.982199 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:00.982159 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b803b38-8rvz9" Oct 31 04:51:01.317757 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:01.317716 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b803b38-8rvz9"] Oct 31 04:51:01.318193 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:51:01.318162 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod980bcf39_71be_432a_b878_e5fbfdec5dba.slice/crio-55256560422c597370228bf6ef79555ebb59d034d7db375c83508aebe22a05bf WatchSource:0}: Error finding container 55256560422c597370228bf6ef79555ebb59d034d7db375c83508aebe22a05bf: Status 404 returned error can't find the container with id 55256560422c597370228bf6ef79555ebb59d034d7db375c83508aebe22a05bf Oct 31 04:51:01.555945 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:01.555910 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f18f16e8-hlq5c"] Oct 31 04:51:01.562134 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:01.562107 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18f16e8-hlq5c" Oct 31 04:51:01.602988 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:01.602942 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f18f16e8-hlq5c"] Oct 31 04:51:01.653274 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:01.653239 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvkl4\" (UniqueName: \"kubernetes.io/projected/6dec4623-e861-413c-8222-0c34801f3a79-kube-api-access-pvkl4\") pod \"maas-api-revoke-sub-f18f16e8-hlq5c\" (UID: \"6dec4623-e861-413c-8222-0c34801f3a79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18f16e8-hlq5c" Oct 31 04:51:01.653710 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:01.653404 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dec4623-e861-413c-8222-0c34801f3a79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f18f16e8-hlq5c\" (UID: \"6dec4623-e861-413c-8222-0c34801f3a79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18f16e8-hlq5c" Oct 31 04:51:01.653710 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:01.653495 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dec4623-e861-413c-8222-0c34801f3a79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f18f16e8-hlq5c\" (UID: \"6dec4623-e861-413c-8222-0c34801f3a79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18f16e8-hlq5c" Oct 31 04:51:01.755636 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:01.755597 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvkl4\" (UniqueName: \"kubernetes.io/projected/6dec4623-e861-413c-8222-0c34801f3a79-kube-api-access-pvkl4\") pod \"maas-api-revoke-sub-f18f16e8-hlq5c\" (UID: \"6dec4623-e861-413c-8222-0c34801f3a79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18f16e8-hlq5c" Oct 31 04:51:01.755801 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:01.755738 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dec4623-e861-413c-8222-0c34801f3a79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f18f16e8-hlq5c\" (UID: \"6dec4623-e861-413c-8222-0c34801f3a79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18f16e8-hlq5c" Oct 31 04:51:01.755881 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:01.755864 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dec4623-e861-413c-8222-0c34801f3a79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f18f16e8-hlq5c\" (UID: \"6dec4623-e861-413c-8222-0c34801f3a79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18f16e8-hlq5c" Oct 31 04:51:01.756554 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:01.756526 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dec4623-e861-413c-8222-0c34801f3a79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f18f16e8-hlq5c\" (UID: \"6dec4623-e861-413c-8222-0c34801f3a79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18f16e8-hlq5c" Oct 31 04:51:01.758326 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:01.758298 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dec4623-e861-413c-8222-0c34801f3a79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f18f16e8-hlq5c\" (UID: \"6dec4623-e861-413c-8222-0c34801f3a79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18f16e8-hlq5c" Oct 31 04:51:01.779069 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:01.779037 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvkl4\" (UniqueName: \"kubernetes.io/projected/6dec4623-e861-413c-8222-0c34801f3a79-kube-api-access-pvkl4\") pod \"maas-api-revoke-sub-f18f16e8-hlq5c\" (UID: \"6dec4623-e861-413c-8222-0c34801f3a79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18f16e8-hlq5c" Oct 31 04:51:01.883191 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:01.883105 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18f16e8-hlq5c" Oct 31 04:51:02.119044 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:02.118988 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b803b38-8rvz9" event={"ID":"980bcf39-71be-432a-b878-e5fbfdec5dba","Type":"ContainerStarted","Data":"181053a9118aee7f5dcc3aabdec008593874cfd3a139ac47df2568fa32ae2d89"} Oct 31 04:51:02.119255 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:02.119060 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b803b38-8rvz9" event={"ID":"980bcf39-71be-432a-b878-e5fbfdec5dba","Type":"ContainerStarted","Data":"55256560422c597370228bf6ef79555ebb59d034d7db375c83508aebe22a05bf"} Oct 31 04:51:02.163877 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:02.163761 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b803b38-8rvz9" podStartSLOduration=2.163741667 podStartE2EDuration="2.163741667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:51:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:51:02.15841938 +0000 UTC m=+2816.914757421" watchObservedRunningTime="2026-10-31 04:51:02.163741667 +0000 UTC m=+2816.920079709" Oct 31 04:51:02.195922 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:51:02.195877 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dec4623_e861_413c_8222_0c34801f3a79.slice/crio-676811bf34ed7a8061bae32786464f7bcc41ff21bb348c9905b26f7530cc66fe WatchSource:0}: Error finding container 676811bf34ed7a8061bae32786464f7bcc41ff21bb348c9905b26f7530cc66fe: Status 404 returned error can't find the container with id 676811bf34ed7a8061bae32786464f7bcc41ff21bb348c9905b26f7530cc66fe Oct 31 04:51:02.199042 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:02.199006 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f18f16e8-hlq5c"] Oct 31 04:51:03.137319 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:03.137214 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18f16e8-hlq5c" event={"ID":"6dec4623-e861-413c-8222-0c34801f3a79","Type":"ContainerStarted","Data":"59431a92e60a9677cbe9b1faab7dddefb49303af6fd19591bb45d57284cc79ef"} Oct 31 04:51:03.137319 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:03.137292 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18f16e8-hlq5c" event={"ID":"6dec4623-e861-413c-8222-0c34801f3a79","Type":"ContainerStarted","Data":"676811bf34ed7a8061bae32786464f7bcc41ff21bb348c9905b26f7530cc66fe"} Oct 31 04:51:03.139617 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:03.139589 5019 generic.go:356] "Generic (PLEG): container finished" podID="980bcf39-71be-432a-b878-e5fbfdec5dba" containerID="181053a9118aee7f5dcc3aabdec008593874cfd3a139ac47df2568fa32ae2d89" exitCode=0 Oct 31 04:51:03.139767 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:03.139632 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b803b38-8rvz9" event={"ID":"980bcf39-71be-432a-b878-e5fbfdec5dba","Type":"ContainerDied","Data":"181053a9118aee7f5dcc3aabdec008593874cfd3a139ac47df2568fa32ae2d89"} Oct 31 04:51:03.183368 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:03.183295 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18f16e8-hlq5c" podStartSLOduration=2.183279267 podStartE2EDuration="2.183279267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:51:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:51:03.178395559 +0000 UTC m=+2817.934733599" watchObservedRunningTime="2026-10-31 04:51:03.183279267 +0000 UTC m=+2817.939617306" Oct 31 04:51:04.216381 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:04.215473 5019 generic.go:356] "Generic (PLEG): container finished" podID="6dec4623-e861-413c-8222-0c34801f3a79" containerID="59431a92e60a9677cbe9b1faab7dddefb49303af6fd19591bb45d57284cc79ef" exitCode=0 Oct 31 04:51:04.216381 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:04.216287 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18f16e8-hlq5c" event={"ID":"6dec4623-e861-413c-8222-0c34801f3a79","Type":"ContainerDied","Data":"59431a92e60a9677cbe9b1faab7dddefb49303af6fd19591bb45d57284cc79ef"} Oct 31 04:51:04.533376 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:04.533302 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b803b38-8rvz9" Oct 31 04:51:04.666736 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:04.666703 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/980bcf39-71be-432a-b878-e5fbfdec5dba-kube-api-access-sbd5n\" (UniqueName: \"kubernetes.io/projected/980bcf39-71be-432a-b878-e5fbfdec5dba-kube-api-access-sbd5n\") pod \"980bcf39-71be-432a-b878-e5fbfdec5dba\" (UID: \"980bcf39-71be-432a-b878-e5fbfdec5dba\") " Oct 31 04:51:04.666932 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:04.666759 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/980bcf39-71be-432a-b878-e5fbfdec5dba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/980bcf39-71be-432a-b878-e5fbfdec5dba-maas-api-xks-ca\") pod \"980bcf39-71be-432a-b878-e5fbfdec5dba\" (UID: \"980bcf39-71be-432a-b878-e5fbfdec5dba\") " Oct 31 04:51:04.666932 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:04.666834 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/980bcf39-71be-432a-b878-e5fbfdec5dba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/980bcf39-71be-432a-b878-e5fbfdec5dba-maas-api-openshift-ca\") pod \"980bcf39-71be-432a-b878-e5fbfdec5dba\" (UID: \"980bcf39-71be-432a-b878-e5fbfdec5dba\") " Oct 31 04:51:04.667308 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:04.667274 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/980bcf39-71be-432a-b878-e5fbfdec5dba-maas-api-openshift-ca" pod "980bcf39-71be-432a-b878-e5fbfdec5dba" (UID: "980bcf39-71be-432a-b878-e5fbfdec5dba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:51:04.667614 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:04.667594 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/980bcf39-71be-432a-b878-e5fbfdec5dba-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:51:04.669054 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:04.669020 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/980bcf39-71be-432a-b878-e5fbfdec5dba-maas-api-xks-ca" pod "980bcf39-71be-432a-b878-e5fbfdec5dba" (UID: "980bcf39-71be-432a-b878-e5fbfdec5dba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:51:04.669182 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:04.669056 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/980bcf39-71be-432a-b878-e5fbfdec5dba-kube-api-access-sbd5n" pod "980bcf39-71be-432a-b878-e5fbfdec5dba" (UID: "980bcf39-71be-432a-b878-e5fbfdec5dba"). InnerVolumeSpecName "kube-api-access-sbd5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:51:04.781439 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:04.770074 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbd5n\" (UniqueName: \"kubernetes.io/projected/980bcf39-71be-432a-b878-e5fbfdec5dba-kube-api-access-sbd5n\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:51:04.781439 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:04.770121 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/980bcf39-71be-432a-b878-e5fbfdec5dba-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:51:05.227476 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:05.227443 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b803b38-8rvz9" Oct 31 04:51:05.227476 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:05.227465 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b803b38-8rvz9" event={"ID":"980bcf39-71be-432a-b878-e5fbfdec5dba","Type":"ContainerDied","Data":"55256560422c597370228bf6ef79555ebb59d034d7db375c83508aebe22a05bf"} Oct 31 04:51:05.227955 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:05.227518 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55256560422c597370228bf6ef79555ebb59d034d7db375c83508aebe22a05bf" Oct 31 04:51:05.473012 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:05.472972 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18f16e8-hlq5c" Oct 31 04:51:05.591429 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:05.591313 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6dec4623-e861-413c-8222-0c34801f3a79-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dec4623-e861-413c-8222-0c34801f3a79-maas-api-xks-ca\") pod \"6dec4623-e861-413c-8222-0c34801f3a79\" (UID: \"6dec4623-e861-413c-8222-0c34801f3a79\") " Oct 31 04:51:05.591623 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:05.591513 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dec4623-e861-413c-8222-0c34801f3a79-kube-api-access-pvkl4\" (UniqueName: \"kubernetes.io/projected/6dec4623-e861-413c-8222-0c34801f3a79-kube-api-access-pvkl4\") pod \"6dec4623-e861-413c-8222-0c34801f3a79\" (UID: \"6dec4623-e861-413c-8222-0c34801f3a79\") " Oct 31 04:51:05.591792 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:05.591775 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dec4623-e861-413c-8222-0c34801f3a79-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dec4623-e861-413c-8222-0c34801f3a79-maas-api-openshift-ca\") pod \"6dec4623-e861-413c-8222-0c34801f3a79\" (UID: \"6dec4623-e861-413c-8222-0c34801f3a79\") " Oct 31 04:51:05.592286 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:05.592257 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dec4623-e861-413c-8222-0c34801f3a79-maas-api-openshift-ca" pod "6dec4623-e861-413c-8222-0c34801f3a79" (UID: "6dec4623-e861-413c-8222-0c34801f3a79"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:51:05.592780 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:05.592756 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dec4623-e861-413c-8222-0c34801f3a79-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:51:05.594383 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:05.593787 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dec4623-e861-413c-8222-0c34801f3a79-maas-api-xks-ca" pod "6dec4623-e861-413c-8222-0c34801f3a79" (UID: "6dec4623-e861-413c-8222-0c34801f3a79"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:51:05.594383 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:05.593863 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dec4623-e861-413c-8222-0c34801f3a79-kube-api-access-pvkl4" pod "6dec4623-e861-413c-8222-0c34801f3a79" (UID: "6dec4623-e861-413c-8222-0c34801f3a79"). InnerVolumeSpecName "kube-api-access-pvkl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:51:05.694034 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:05.693990 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dec4623-e861-413c-8222-0c34801f3a79-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:51:05.694034 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:05.694022 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvkl4\" (UniqueName: \"kubernetes.io/projected/6dec4623-e861-413c-8222-0c34801f3a79-kube-api-access-pvkl4\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:51:06.240104 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:06.240067 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18f16e8-hlq5c" Oct 31 04:51:06.240520 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:06.240107 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18f16e8-hlq5c" event={"ID":"6dec4623-e861-413c-8222-0c34801f3a79","Type":"ContainerDied","Data":"676811bf34ed7a8061bae32786464f7bcc41ff21bb348c9905b26f7530cc66fe"} Oct 31 04:51:06.240520 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:06.240171 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="676811bf34ed7a8061bae32786464f7bcc41ff21bb348c9905b26f7530cc66fe" Oct 31 04:51:10.851059 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:10.851010 5019 scope.go:122] "RemoveContainer" containerID="9df4fc9d03f2da1a6e95c8dc120f4d8ce22889991e9dbb196bcd271913982deb" Oct 31 04:51:11.457013 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:11.456970 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8768315-fsv74"] Oct 31 04:51:11.457916 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:11.457896 5019 cpu_manager.go:414] "containerMap: removing container" podUID="980bcf39-71be-432a-b878-e5fbfdec5dba" containerName="revoke-keys" Oct 31 04:51:11.457973 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:11.457919 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="980bcf39-71be-432a-b878-e5fbfdec5dba" containerName="revoke-keys" Oct 31 04:51:11.457973 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:11.457960 5019 cpu_manager.go:414] "containerMap: removing container" podUID="6dec4623-e861-413c-8222-0c34801f3a79" containerName="revoke-keys" Oct 31 04:51:11.457973 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:11.457965 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dec4623-e861-413c-8222-0c34801f3a79" containerName="revoke-keys" Oct 31 04:51:11.458105 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:11.458095 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dec4623-e861-413c-8222-0c34801f3a79" containerName="revoke-keys" Oct 31 04:51:11.458140 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:11.458113 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="980bcf39-71be-432a-b878-e5fbfdec5dba" containerName="revoke-keys" Oct 31 04:51:11.463919 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:11.463898 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8768315-fsv74" Oct 31 04:51:11.481587 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:11.481557 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8768315-fsv74"] Oct 31 04:51:11.485812 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:11.485783 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:51:11.485994 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:11.485957 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:51:11.486070 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:11.486062 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:51:11.486132 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:11.486110 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:51:11.595081 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:11.595047 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kwfm\" (UniqueName: \"kubernetes.io/projected/506507c1-3d2b-413c-8f8f-0bcf84ce0ed8-kube-api-access-9kwfm\") pod \"maas-api-revoke-sub-b8768315-fsv74\" (UID: \"506507c1-3d2b-413c-8f8f-0bcf84ce0ed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8768315-fsv74" Oct 31 04:51:11.595303 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:11.595198 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/506507c1-3d2b-413c-8f8f-0bcf84ce0ed8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8768315-fsv74\" (UID: \"506507c1-3d2b-413c-8f8f-0bcf84ce0ed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8768315-fsv74" Oct 31 04:51:11.595303 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:11.595250 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/506507c1-3d2b-413c-8f8f-0bcf84ce0ed8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8768315-fsv74\" (UID: \"506507c1-3d2b-413c-8f8f-0bcf84ce0ed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8768315-fsv74" Oct 31 04:51:11.697419 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:11.697380 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kwfm\" (UniqueName: \"kubernetes.io/projected/506507c1-3d2b-413c-8f8f-0bcf84ce0ed8-kube-api-access-9kwfm\") pod \"maas-api-revoke-sub-b8768315-fsv74\" (UID: \"506507c1-3d2b-413c-8f8f-0bcf84ce0ed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8768315-fsv74" Oct 31 04:51:11.697595 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:11.697497 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/506507c1-3d2b-413c-8f8f-0bcf84ce0ed8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8768315-fsv74\" (UID: \"506507c1-3d2b-413c-8f8f-0bcf84ce0ed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8768315-fsv74" Oct 31 04:51:11.697595 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:11.697536 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/506507c1-3d2b-413c-8f8f-0bcf84ce0ed8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8768315-fsv74\" (UID: \"506507c1-3d2b-413c-8f8f-0bcf84ce0ed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8768315-fsv74" Oct 31 04:51:11.698221 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:11.698198 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/506507c1-3d2b-413c-8f8f-0bcf84ce0ed8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8768315-fsv74\" (UID: \"506507c1-3d2b-413c-8f8f-0bcf84ce0ed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8768315-fsv74" Oct 31 04:51:11.699902 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:11.699877 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/506507c1-3d2b-413c-8f8f-0bcf84ce0ed8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8768315-fsv74\" (UID: \"506507c1-3d2b-413c-8f8f-0bcf84ce0ed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8768315-fsv74" Oct 31 04:51:11.720993 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:11.720969 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kwfm\" (UniqueName: \"kubernetes.io/projected/506507c1-3d2b-413c-8f8f-0bcf84ce0ed8-kube-api-access-9kwfm\") pod \"maas-api-revoke-sub-b8768315-fsv74\" (UID: \"506507c1-3d2b-413c-8f8f-0bcf84ce0ed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8768315-fsv74" Oct 31 04:51:11.781572 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:11.781550 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8768315-fsv74" Oct 31 04:51:12.086891 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:51:12.086855 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod506507c1_3d2b_413c_8f8f_0bcf84ce0ed8.slice/crio-bfccb21279ffd431f31990eaf120f212f96ac519b66528b447850cbad802350f WatchSource:0}: Error finding container bfccb21279ffd431f31990eaf120f212f96ac519b66528b447850cbad802350f: Status 404 returned error can't find the container with id bfccb21279ffd431f31990eaf120f212f96ac519b66528b447850cbad802350f Oct 31 04:51:12.087212 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:12.086885 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8768315-fsv74"] Oct 31 04:51:12.299189 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:12.299153 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8768315-fsv74" event={"ID":"506507c1-3d2b-413c-8f8f-0bcf84ce0ed8","Type":"ContainerStarted","Data":"bfccb21279ffd431f31990eaf120f212f96ac519b66528b447850cbad802350f"} Oct 31 04:51:13.311066 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:13.311030 5019 generic.go:356] "Generic (PLEG): container finished" podID="506507c1-3d2b-413c-8f8f-0bcf84ce0ed8" containerID="6be82bc75b5ba7bd69d15554547b1bfd13bad2b5548a04ef8b620333c09f965e" exitCode=0 Oct 31 04:51:13.311608 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:13.311255 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8768315-fsv74" event={"ID":"506507c1-3d2b-413c-8f8f-0bcf84ce0ed8","Type":"ContainerDied","Data":"6be82bc75b5ba7bd69d15554547b1bfd13bad2b5548a04ef8b620333c09f965e"} Oct 31 04:51:14.519063 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:14.519027 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8768315-fsv74" Oct 31 04:51:14.691524 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:14.691419 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/506507c1-3d2b-413c-8f8f-0bcf84ce0ed8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/506507c1-3d2b-413c-8f8f-0bcf84ce0ed8-maas-api-xks-ca\") pod \"506507c1-3d2b-413c-8f8f-0bcf84ce0ed8\" (UID: \"506507c1-3d2b-413c-8f8f-0bcf84ce0ed8\") " Oct 31 04:51:14.691807 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:14.691787 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/506507c1-3d2b-413c-8f8f-0bcf84ce0ed8-kube-api-access-9kwfm\" (UniqueName: \"kubernetes.io/projected/506507c1-3d2b-413c-8f8f-0bcf84ce0ed8-kube-api-access-9kwfm\") pod \"506507c1-3d2b-413c-8f8f-0bcf84ce0ed8\" (UID: \"506507c1-3d2b-413c-8f8f-0bcf84ce0ed8\") " Oct 31 04:51:14.691878 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:14.691824 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/506507c1-3d2b-413c-8f8f-0bcf84ce0ed8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/506507c1-3d2b-413c-8f8f-0bcf84ce0ed8-maas-api-openshift-ca\") pod \"506507c1-3d2b-413c-8f8f-0bcf84ce0ed8\" (UID: \"506507c1-3d2b-413c-8f8f-0bcf84ce0ed8\") " Oct 31 04:51:14.692384 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:14.692313 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/506507c1-3d2b-413c-8f8f-0bcf84ce0ed8-maas-api-openshift-ca" pod "506507c1-3d2b-413c-8f8f-0bcf84ce0ed8" (UID: "506507c1-3d2b-413c-8f8f-0bcf84ce0ed8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:51:14.692614 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:14.692595 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/506507c1-3d2b-413c-8f8f-0bcf84ce0ed8-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:51:14.693796 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:14.693774 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/506507c1-3d2b-413c-8f8f-0bcf84ce0ed8-maas-api-xks-ca" pod "506507c1-3d2b-413c-8f8f-0bcf84ce0ed8" (UID: "506507c1-3d2b-413c-8f8f-0bcf84ce0ed8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:51:14.694520 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:14.694490 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/506507c1-3d2b-413c-8f8f-0bcf84ce0ed8-kube-api-access-9kwfm" pod "506507c1-3d2b-413c-8f8f-0bcf84ce0ed8" (UID: "506507c1-3d2b-413c-8f8f-0bcf84ce0ed8"). InnerVolumeSpecName "kube-api-access-9kwfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:51:14.794725 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:14.794681 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kwfm\" (UniqueName: \"kubernetes.io/projected/506507c1-3d2b-413c-8f8f-0bcf84ce0ed8-kube-api-access-9kwfm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:51:14.794725 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:14.794726 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/506507c1-3d2b-413c-8f8f-0bcf84ce0ed8-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:51:15.332942 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:15.332908 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8768315-fsv74" Oct 31 04:51:15.333135 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:15.332940 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8768315-fsv74" event={"ID":"506507c1-3d2b-413c-8f8f-0bcf84ce0ed8","Type":"ContainerDied","Data":"bfccb21279ffd431f31990eaf120f212f96ac519b66528b447850cbad802350f"} Oct 31 04:51:15.333135 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:15.333021 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfccb21279ffd431f31990eaf120f212f96ac519b66528b447850cbad802350f" Oct 31 04:51:17.384717 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:17.384681 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-796dfn2nl"] Oct 31 04:51:17.385937 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:17.385917 5019 cpu_manager.go:414] "containerMap: removing container" podUID="506507c1-3d2b-413c-8f8f-0bcf84ce0ed8" containerName="revoke-keys" Oct 31 04:51:17.386016 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:17.385939 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="506507c1-3d2b-413c-8f8f-0bcf84ce0ed8" containerName="revoke-keys" Oct 31 04:51:17.386199 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:17.386187 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="506507c1-3d2b-413c-8f8f-0bcf84ce0ed8" containerName="revoke-keys" Oct 31 04:51:17.391375 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:17.391350 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-796dfn2nl" Oct 31 04:51:17.412882 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:17.412703 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a04e4e\"/\"kube-root-ca.crt\"" Oct 31 04:51:17.412882 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:17.412709 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a04e4e\"/\"openshift-service-ca.crt\"" Oct 31 04:51:17.412882 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:17.412713 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a04e4e\"/\"routing-overlay\"" Oct 31 04:51:17.412882 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:17.412764 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a04e4e\"/\"payload-processing-external-model-plugins-e2e-extmod-b-a04e4e\"" Oct 31 04:51:17.412882 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:17.412772 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a04e4e\"/\"agc-cred-e77d307cbbfa\"" Oct 31 04:51:17.413921 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:17.413899 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-796dfn2nl"] Oct 31 04:51:17.418132 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:17.418092 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a04e4e\"/\"payload-processing-external-model-e2e--ae39f93f-dockercfg-5wzdp\"" Oct 31 04:51:17.686665 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:17.686561 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a15da393-ddf3-4681-bfc7-e09c01bb1cf4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-796dfn2nl\" (UID: \"a15da393-ddf3-4681-bfc7-e09c01bb1cf4\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-796dfn2nl" Oct 31 04:51:17.687668 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:17.687637 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a15da393-ddf3-4681-bfc7-e09c01bb1cf4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-796dfn2nl\" (UID: \"a15da393-ddf3-4681-bfc7-e09c01bb1cf4\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-796dfn2nl" Oct 31 04:51:17.687915 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:17.687897 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a15da393-ddf3-4681-bfc7-e09c01bb1cf4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-796dfn2nl\" (UID: \"a15da393-ddf3-4681-bfc7-e09c01bb1cf4\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-796dfn2nl" Oct 31 04:51:17.688089 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:17.688070 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r47k6\" (UniqueName: \"kubernetes.io/projected/a15da393-ddf3-4681-bfc7-e09c01bb1cf4-kube-api-access-r47k6\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-796dfn2nl\" (UID: \"a15da393-ddf3-4681-bfc7-e09c01bb1cf4\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-796dfn2nl" Oct 31 04:51:17.789662 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:17.789622 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a15da393-ddf3-4681-bfc7-e09c01bb1cf4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-796dfn2nl\" (UID: \"a15da393-ddf3-4681-bfc7-e09c01bb1cf4\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-796dfn2nl" Oct 31 04:51:17.789852 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:17.789778 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a15da393-ddf3-4681-bfc7-e09c01bb1cf4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-796dfn2nl\" (UID: \"a15da393-ddf3-4681-bfc7-e09c01bb1cf4\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-796dfn2nl" Oct 31 04:51:17.789896 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:17.789865 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r47k6\" (UniqueName: \"kubernetes.io/projected/a15da393-ddf3-4681-bfc7-e09c01bb1cf4-kube-api-access-r47k6\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-796dfn2nl\" (UID: \"a15da393-ddf3-4681-bfc7-e09c01bb1cf4\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-796dfn2nl" Oct 31 04:51:17.790365 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:17.790318 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a15da393-ddf3-4681-bfc7-e09c01bb1cf4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-796dfn2nl\" (UID: \"a15da393-ddf3-4681-bfc7-e09c01bb1cf4\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-796dfn2nl" Oct 31 04:51:17.790448 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:17.790426 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a15da393-ddf3-4681-bfc7-e09c01bb1cf4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-796dfn2nl\" (UID: \"a15da393-ddf3-4681-bfc7-e09c01bb1cf4\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-796dfn2nl" Oct 31 04:51:17.790691 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:17.790671 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a15da393-ddf3-4681-bfc7-e09c01bb1cf4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-796dfn2nl\" (UID: \"a15da393-ddf3-4681-bfc7-e09c01bb1cf4\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-796dfn2nl" Oct 31 04:51:17.792936 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:17.792912 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a15da393-ddf3-4681-bfc7-e09c01bb1cf4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-796dfn2nl\" (UID: \"a15da393-ddf3-4681-bfc7-e09c01bb1cf4\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-796dfn2nl" Oct 31 04:51:17.813025 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:17.812996 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r47k6\" (UniqueName: \"kubernetes.io/projected/a15da393-ddf3-4681-bfc7-e09c01bb1cf4-kube-api-access-r47k6\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-796dfn2nl\" (UID: \"a15da393-ddf3-4681-bfc7-e09c01bb1cf4\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-796dfn2nl" Oct 31 04:51:18.010228 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:18.010192 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-796dfn2nl" Oct 31 04:51:18.413438 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:18.410170 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-796dfn2nl"] Oct 31 04:51:19.504354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:19.504301 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-796dfn2nl" event={"ID":"a15da393-ddf3-4681-bfc7-e09c01bb1cf4","Type":"ContainerStarted","Data":"00cb394a4f7a953a645af9c125cc5ce767bc22fdeb6ea24fe03759a5228a08a2"} Oct 31 04:51:19.504788 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:19.504384 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-796dfn2nl" event={"ID":"a15da393-ddf3-4681-bfc7-e09c01bb1cf4","Type":"ContainerStarted","Data":"9bd6a77f0e4ab04b0ea61cbcd4851cda2873a8ea2c3c3b91b5e9589026d2f29b"} Oct 31 04:51:19.504788 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:19.504430 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-796dfn2nl" Oct 31 04:51:19.551966 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:19.551906 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-796dfn2nl" podStartSLOduration=2.551887729 podStartE2EDuration="2.551887729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:51:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:51:19.546710274 +0000 UTC m=+2834.303048313" watchObservedRunningTime="2026-10-31 04:51:19.551887729 +0000 UTC m=+2834.308225772" Oct 31 04:51:22.394812 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:51:22.394752 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:51:24.029012 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:24.028965 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9c665w6"] Oct 31 04:51:24.034681 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:24.034653 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9c665w6" Oct 31 04:51:24.053914 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:24.053876 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9c665w6"] Oct 31 04:51:24.058496 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:24.058456 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-afc519\"/\"agc-cred-603fbee8e306\"" Oct 31 04:51:24.058496 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:24.058494 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-afc519\"/\"payload-processing-external-model-e2e--49628f74-dockercfg-xg7x2\"" Oct 31 04:51:24.058730 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:24.058566 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-afc519\"/\"routing-overlay\"" Oct 31 04:51:24.061777 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:24.061744 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-afc519\"/\"kube-root-ca.crt\"" Oct 31 04:51:24.061949 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:24.061744 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-afc519\"/\"payload-processing-external-model-plugins-e2e-extmod-a-afc519\"" Oct 31 04:51:24.061949 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:24.061744 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-afc519\"/\"openshift-service-ca.crt\"" Oct 31 04:51:24.089947 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:24.089891 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9445887a-a8dd-4d27-8442-6f9b0759364e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-afc519-6c9c665w6\" (UID: \"9445887a-a8dd-4d27-8442-6f9b0759364e\") " pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9c665w6" Oct 31 04:51:24.090169 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:24.089962 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r6cm\" (UniqueName: \"kubernetes.io/projected/9445887a-a8dd-4d27-8442-6f9b0759364e-kube-api-access-6r6cm\") pod \"payload-processing-external-model-e2e-extmod-a-afc519-6c9c665w6\" (UID: \"9445887a-a8dd-4d27-8442-6f9b0759364e\") " pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9c665w6" Oct 31 04:51:24.090247 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:24.090224 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9445887a-a8dd-4d27-8442-6f9b0759364e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-afc519-6c9c665w6\" (UID: \"9445887a-a8dd-4d27-8442-6f9b0759364e\") " pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9c665w6" Oct 31 04:51:24.090317 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:24.090285 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9445887a-a8dd-4d27-8442-6f9b0759364e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-afc519-6c9c665w6\" (UID: \"9445887a-a8dd-4d27-8442-6f9b0759364e\") " pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9c665w6" Oct 31 04:51:24.192191 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:24.192145 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9445887a-a8dd-4d27-8442-6f9b0759364e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-afc519-6c9c665w6\" (UID: \"9445887a-a8dd-4d27-8442-6f9b0759364e\") " pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9c665w6" Oct 31 04:51:24.192191 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:24.192196 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r6cm\" (UniqueName: \"kubernetes.io/projected/9445887a-a8dd-4d27-8442-6f9b0759364e-kube-api-access-6r6cm\") pod \"payload-processing-external-model-e2e-extmod-a-afc519-6c9c665w6\" (UID: \"9445887a-a8dd-4d27-8442-6f9b0759364e\") " pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9c665w6" Oct 31 04:51:24.192502 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:24.192427 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9445887a-a8dd-4d27-8442-6f9b0759364e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-afc519-6c9c665w6\" (UID: \"9445887a-a8dd-4d27-8442-6f9b0759364e\") " pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9c665w6" Oct 31 04:51:24.192502 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:24.192476 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9445887a-a8dd-4d27-8442-6f9b0759364e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-afc519-6c9c665w6\" (UID: \"9445887a-a8dd-4d27-8442-6f9b0759364e\") " pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9c665w6" Oct 31 04:51:24.193226 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:24.193194 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9445887a-a8dd-4d27-8442-6f9b0759364e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-afc519-6c9c665w6\" (UID: \"9445887a-a8dd-4d27-8442-6f9b0759364e\") " pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9c665w6" Oct 31 04:51:24.193361 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:24.193244 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9445887a-a8dd-4d27-8442-6f9b0759364e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-afc519-6c9c665w6\" (UID: \"9445887a-a8dd-4d27-8442-6f9b0759364e\") " pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9c665w6" Oct 31 04:51:24.195047 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:24.195021 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9445887a-a8dd-4d27-8442-6f9b0759364e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-afc519-6c9c665w6\" (UID: \"9445887a-a8dd-4d27-8442-6f9b0759364e\") " pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9c665w6" Oct 31 04:51:24.219260 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:24.219188 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r6cm\" (UniqueName: \"kubernetes.io/projected/9445887a-a8dd-4d27-8442-6f9b0759364e-kube-api-access-6r6cm\") pod \"payload-processing-external-model-e2e-extmod-a-afc519-6c9c665w6\" (UID: \"9445887a-a8dd-4d27-8442-6f9b0759364e\") " pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9c665w6" Oct 31 04:51:24.358760 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:24.358673 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9c665w6" Oct 31 04:51:24.674179 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:24.674102 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9c665w6"] Oct 31 04:51:24.680741 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:51:24.680687 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9445887a_a8dd_4d27_8442_6f9b0759364e.slice/crio-0da21e7c158c8e78e7ad135a50b71c513a2513a1beb660ac1cbf9bc3eb6ff8ba WatchSource:0}: Error finding container 0da21e7c158c8e78e7ad135a50b71c513a2513a1beb660ac1cbf9bc3eb6ff8ba: Status 404 returned error can't find the container with id 0da21e7c158c8e78e7ad135a50b71c513a2513a1beb660ac1cbf9bc3eb6ff8ba Oct 31 04:51:24.697143 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:24.696696 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9c665w6"] Oct 31 04:51:25.571206 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:25.571169 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9c665w6" event={"ID":"9445887a-a8dd-4d27-8442-6f9b0759364e","Type":"ContainerStarted","Data":"a5c1ccec4752acb0f3fb596eea65fe8022f455e992d236c0cc533a3f5be46d5d"} Oct 31 04:51:25.571764 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:25.571231 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9c665w6" event={"ID":"9445887a-a8dd-4d27-8442-6f9b0759364e","Type":"ContainerStarted","Data":"0da21e7c158c8e78e7ad135a50b71c513a2513a1beb660ac1cbf9bc3eb6ff8ba"} Oct 31 04:51:25.571764 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:25.571272 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9c665w6" Oct 31 04:51:25.613826 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:25.613756 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9c665w6" podStartSLOduration=1.613737327 podStartE2EDuration="1.613737327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:51:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:51:25.611848904 +0000 UTC m=+2840.368186942" watchObservedRunningTime="2026-10-31 04:51:25.613737327 +0000 UTC m=+2840.370075368" Oct 31 04:51:30.518430 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:30.518397 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-796dfn2nl" Oct 31 04:51:30.572133 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:30.572072 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9c665w6" podUID="9445887a-a8dd-4d27-8442-6f9b0759364e" containerName="payload-processing" containerID="cri-o://a5c1ccec4752acb0f3fb596eea65fe8022f455e992d236c0cc533a3f5be46d5d" gracePeriod=25 Oct 31 04:51:30.889053 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:30.889021 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9c665w6" Oct 31 04:51:31.035496 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:31.035464 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9445887a-a8dd-4d27-8442-6f9b0759364e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9445887a-a8dd-4d27-8442-6f9b0759364e-routing-overlay-volume\") pod \"9445887a-a8dd-4d27-8442-6f9b0759364e\" (UID: \"9445887a-a8dd-4d27-8442-6f9b0759364e\") " Oct 31 04:51:31.035711 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:31.035517 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9445887a-a8dd-4d27-8442-6f9b0759364e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9445887a-a8dd-4d27-8442-6f9b0759364e-provider-credentials\") pod \"9445887a-a8dd-4d27-8442-6f9b0759364e\" (UID: \"9445887a-a8dd-4d27-8442-6f9b0759364e\") " Oct 31 04:51:31.035711 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:31.035558 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9445887a-a8dd-4d27-8442-6f9b0759364e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9445887a-a8dd-4d27-8442-6f9b0759364e-plugins-config-volume\") pod \"9445887a-a8dd-4d27-8442-6f9b0759364e\" (UID: \"9445887a-a8dd-4d27-8442-6f9b0759364e\") " Oct 31 04:51:31.035845 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:31.035827 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9445887a-a8dd-4d27-8442-6f9b0759364e-kube-api-access-6r6cm\" (UniqueName: \"kubernetes.io/projected/9445887a-a8dd-4d27-8442-6f9b0759364e-kube-api-access-6r6cm\") pod \"9445887a-a8dd-4d27-8442-6f9b0759364e\" (UID: \"9445887a-a8dd-4d27-8442-6f9b0759364e\") " Oct 31 04:51:31.035937 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:31.035832 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9445887a-a8dd-4d27-8442-6f9b0759364e-routing-overlay-volume" pod "9445887a-a8dd-4d27-8442-6f9b0759364e" (UID: "9445887a-a8dd-4d27-8442-6f9b0759364e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:51:31.036018 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:31.035998 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9445887a-a8dd-4d27-8442-6f9b0759364e-plugins-config-volume" pod "9445887a-a8dd-4d27-8442-6f9b0759364e" (UID: "9445887a-a8dd-4d27-8442-6f9b0759364e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:51:31.036479 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:31.036459 5019 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9445887a-a8dd-4d27-8442-6f9b0759364e-routing-overlay-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:51:31.036479 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:31.036479 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9445887a-a8dd-4d27-8442-6f9b0759364e-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:51:31.037907 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:31.037886 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9445887a-a8dd-4d27-8442-6f9b0759364e-kube-api-access-6r6cm" pod "9445887a-a8dd-4d27-8442-6f9b0759364e" (UID: "9445887a-a8dd-4d27-8442-6f9b0759364e"). InnerVolumeSpecName "kube-api-access-6r6cm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:51:31.037987 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:31.037912 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9445887a-a8dd-4d27-8442-6f9b0759364e-provider-credentials" pod "9445887a-a8dd-4d27-8442-6f9b0759364e" (UID: "9445887a-a8dd-4d27-8442-6f9b0759364e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:51:31.137420 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:31.137347 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r6cm\" (UniqueName: \"kubernetes.io/projected/9445887a-a8dd-4d27-8442-6f9b0759364e-kube-api-access-6r6cm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:51:31.137420 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:31.137373 5019 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9445887a-a8dd-4d27-8442-6f9b0759364e-provider-credentials\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:51:31.641508 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:31.641473 5019 generic.go:356] "Generic (PLEG): container finished" podID="9445887a-a8dd-4d27-8442-6f9b0759364e" containerID="a5c1ccec4752acb0f3fb596eea65fe8022f455e992d236c0cc533a3f5be46d5d" exitCode=0 Oct 31 04:51:31.641982 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:31.641518 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9c665w6" Oct 31 04:51:31.641982 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:31.641560 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9c665w6" event={"ID":"9445887a-a8dd-4d27-8442-6f9b0759364e","Type":"ContainerDied","Data":"a5c1ccec4752acb0f3fb596eea65fe8022f455e992d236c0cc533a3f5be46d5d"} Oct 31 04:51:31.641982 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:31.641622 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9c665w6" event={"ID":"9445887a-a8dd-4d27-8442-6f9b0759364e","Type":"ContainerDied","Data":"0da21e7c158c8e78e7ad135a50b71c513a2513a1beb660ac1cbf9bc3eb6ff8ba"} Oct 31 04:51:31.641982 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:31.641657 5019 scope.go:122] "RemoveContainer" containerID="a5c1ccec4752acb0f3fb596eea65fe8022f455e992d236c0cc533a3f5be46d5d" Oct 31 04:51:31.653744 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:31.653723 5019 scope.go:122] "RemoveContainer" containerID="a5c1ccec4752acb0f3fb596eea65fe8022f455e992d236c0cc533a3f5be46d5d" Oct 31 04:51:31.654012 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:51:31.653992 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5c1ccec4752acb0f3fb596eea65fe8022f455e992d236c0cc533a3f5be46d5d\": container with ID starting with a5c1ccec4752acb0f3fb596eea65fe8022f455e992d236c0cc533a3f5be46d5d not found: ID does not exist" containerID="a5c1ccec4752acb0f3fb596eea65fe8022f455e992d236c0cc533a3f5be46d5d" Oct 31 04:51:31.654070 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:31.654021 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5c1ccec4752acb0f3fb596eea65fe8022f455e992d236c0cc533a3f5be46d5d"} err="failed to get container status \"a5c1ccec4752acb0f3fb596eea65fe8022f455e992d236c0cc533a3f5be46d5d\": rpc error: code = NotFound desc = could not find container \"a5c1ccec4752acb0f3fb596eea65fe8022f455e992d236c0cc533a3f5be46d5d\": container with ID starting with a5c1ccec4752acb0f3fb596eea65fe8022f455e992d236c0cc533a3f5be46d5d not found: ID does not exist" Oct 31 04:51:31.723318 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:31.723281 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9c665w6"] Oct 31 04:51:31.734684 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:31.734656 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9c665w6"] Oct 31 04:51:33.414115 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:33.414072 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9445887a-a8dd-4d27-8442-6f9b0759364e" path="/var/lib/kubelet/pods/9445887a-a8dd-4d27-8442-6f9b0759364e/volumes" Oct 31 04:51:35.474090 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:35.474055 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-b86f7hh2j"] Oct 31 04:51:35.474977 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:35.474956 5019 cpu_manager.go:414] "containerMap: removing container" podUID="9445887a-a8dd-4d27-8442-6f9b0759364e" containerName="payload-processing" Oct 31 04:51:35.474977 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:35.474977 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9445887a-a8dd-4d27-8442-6f9b0759364e" containerName="payload-processing" Oct 31 04:51:35.475146 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:35.475134 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="9445887a-a8dd-4d27-8442-6f9b0759364e" containerName="payload-processing" Oct 31 04:51:35.479392 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:35.479370 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-b86f7hh2j" Oct 31 04:51:35.499688 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:35.499667 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-b86f7hh2j"] Oct 31 04:51:35.504385 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:35.504363 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a04e4e\"/\"agc-cred-3522d14a4618\"" Oct 31 04:51:35.559136 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:35.559112 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/24e66441-4418-4bb6-b24c-f3f3bd9f49fd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-b86f7hh2j\" (UID: \"24e66441-4418-4bb6-b24c-f3f3bd9f49fd\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-b86f7hh2j" Oct 31 04:51:35.559307 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:35.559292 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhbwb\" (UniqueName: \"kubernetes.io/projected/24e66441-4418-4bb6-b24c-f3f3bd9f49fd-kube-api-access-xhbwb\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-b86f7hh2j\" (UID: \"24e66441-4418-4bb6-b24c-f3f3bd9f49fd\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-b86f7hh2j" Oct 31 04:51:35.559394 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:35.559322 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/24e66441-4418-4bb6-b24c-f3f3bd9f49fd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-b86f7hh2j\" (UID: \"24e66441-4418-4bb6-b24c-f3f3bd9f49fd\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-b86f7hh2j" Oct 31 04:51:35.559394 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:35.559384 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24e66441-4418-4bb6-b24c-f3f3bd9f49fd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-b86f7hh2j\" (UID: \"24e66441-4418-4bb6-b24c-f3f3bd9f49fd\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-b86f7hh2j" Oct 31 04:51:35.661193 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:35.661160 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhbwb\" (UniqueName: \"kubernetes.io/projected/24e66441-4418-4bb6-b24c-f3f3bd9f49fd-kube-api-access-xhbwb\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-b86f7hh2j\" (UID: \"24e66441-4418-4bb6-b24c-f3f3bd9f49fd\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-b86f7hh2j" Oct 31 04:51:35.661385 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:35.661213 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/24e66441-4418-4bb6-b24c-f3f3bd9f49fd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-b86f7hh2j\" (UID: \"24e66441-4418-4bb6-b24c-f3f3bd9f49fd\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-b86f7hh2j" Oct 31 04:51:35.661385 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:35.661249 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24e66441-4418-4bb6-b24c-f3f3bd9f49fd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-b86f7hh2j\" (UID: \"24e66441-4418-4bb6-b24c-f3f3bd9f49fd\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-b86f7hh2j" Oct 31 04:51:35.661518 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:35.661489 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/24e66441-4418-4bb6-b24c-f3f3bd9f49fd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-b86f7hh2j\" (UID: \"24e66441-4418-4bb6-b24c-f3f3bd9f49fd\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-b86f7hh2j" Oct 31 04:51:35.662163 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:35.662139 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/24e66441-4418-4bb6-b24c-f3f3bd9f49fd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-b86f7hh2j\" (UID: \"24e66441-4418-4bb6-b24c-f3f3bd9f49fd\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-b86f7hh2j" Oct 31 04:51:35.662242 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:35.662222 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24e66441-4418-4bb6-b24c-f3f3bd9f49fd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-b86f7hh2j\" (UID: \"24e66441-4418-4bb6-b24c-f3f3bd9f49fd\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-b86f7hh2j" Oct 31 04:51:35.663906 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:35.663885 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/24e66441-4418-4bb6-b24c-f3f3bd9f49fd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-b86f7hh2j\" (UID: \"24e66441-4418-4bb6-b24c-f3f3bd9f49fd\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-b86f7hh2j" Oct 31 04:51:35.684434 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:35.684377 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhbwb\" (UniqueName: \"kubernetes.io/projected/24e66441-4418-4bb6-b24c-f3f3bd9f49fd-kube-api-access-xhbwb\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-b86f7hh2j\" (UID: \"24e66441-4418-4bb6-b24c-f3f3bd9f49fd\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-b86f7hh2j" Oct 31 04:51:35.800721 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:35.800369 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-b86f7hh2j" Oct 31 04:51:36.123564 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:36.123523 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-b86f7hh2j"] Oct 31 04:51:36.822223 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:36.822191 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-b86f7hh2j" event={"ID":"24e66441-4418-4bb6-b24c-f3f3bd9f49fd","Type":"ContainerStarted","Data":"f3f7bd429ba03731aeb62cb298d49f95a09eee5ba860ee71fb8e0c5a1a86a585"} Oct 31 04:51:36.822728 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:36.822249 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-b86f7hh2j" event={"ID":"24e66441-4418-4bb6-b24c-f3f3bd9f49fd","Type":"ContainerStarted","Data":"f31caea86f63cadd579d7aefeb75423c2f4db11b56c28c19b943134e6945e07c"} Oct 31 04:51:36.823385 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:36.823326 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-b86f7hh2j" Oct 31 04:51:36.865269 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:36.865218 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-b86f7hh2j" podStartSLOduration=1.865204677 podStartE2EDuration="1.865204677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:51:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:51:36.863138635 +0000 UTC m=+2851.619476689" watchObservedRunningTime="2026-10-31 04:51:36.865204677 +0000 UTC m=+2851.621542752" Oct 31 04:51:48.891708 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:48.891672 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-b86f7hh2j" Oct 31 04:51:48.972778 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:48.972741 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-796dfn2nl"] Oct 31 04:51:52.392181 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:51:52.392135 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:51:53.973609 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:53.973563 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-796dfn2nl" podUID="a15da393-ddf3-4681-bfc7-e09c01bb1cf4" containerName="payload-processing" containerID="cri-o://00cb394a4f7a953a645af9c125cc5ce767bc22fdeb6ea24fe03759a5228a08a2" gracePeriod=25 Oct 31 04:51:54.277093 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:54.277068 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-796dfn2nl" Oct 31 04:51:54.386342 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:54.386309 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a15da393-ddf3-4681-bfc7-e09c01bb1cf4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a15da393-ddf3-4681-bfc7-e09c01bb1cf4-routing-overlay-volume\") pod \"a15da393-ddf3-4681-bfc7-e09c01bb1cf4\" (UID: \"a15da393-ddf3-4681-bfc7-e09c01bb1cf4\") " Oct 31 04:51:54.386590 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:54.386570 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a15da393-ddf3-4681-bfc7-e09c01bb1cf4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a15da393-ddf3-4681-bfc7-e09c01bb1cf4-plugins-config-volume\") pod \"a15da393-ddf3-4681-bfc7-e09c01bb1cf4\" (UID: \"a15da393-ddf3-4681-bfc7-e09c01bb1cf4\") " Oct 31 04:51:54.386689 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:54.386645 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a15da393-ddf3-4681-bfc7-e09c01bb1cf4-routing-overlay-volume" pod "a15da393-ddf3-4681-bfc7-e09c01bb1cf4" (UID: "a15da393-ddf3-4681-bfc7-e09c01bb1cf4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:51:54.386805 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:54.386786 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a15da393-ddf3-4681-bfc7-e09c01bb1cf4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a15da393-ddf3-4681-bfc7-e09c01bb1cf4-provider-credentials\") pod \"a15da393-ddf3-4681-bfc7-e09c01bb1cf4\" (UID: \"a15da393-ddf3-4681-bfc7-e09c01bb1cf4\") " Oct 31 04:51:54.386907 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:54.386885 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a15da393-ddf3-4681-bfc7-e09c01bb1cf4-kube-api-access-r47k6\" (UniqueName: \"kubernetes.io/projected/a15da393-ddf3-4681-bfc7-e09c01bb1cf4-kube-api-access-r47k6\") pod \"a15da393-ddf3-4681-bfc7-e09c01bb1cf4\" (UID: \"a15da393-ddf3-4681-bfc7-e09c01bb1cf4\") " Oct 31 04:51:54.387033 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:54.386966 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a15da393-ddf3-4681-bfc7-e09c01bb1cf4-plugins-config-volume" pod "a15da393-ddf3-4681-bfc7-e09c01bb1cf4" (UID: "a15da393-ddf3-4681-bfc7-e09c01bb1cf4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:51:54.387688 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:54.387664 5019 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a15da393-ddf3-4681-bfc7-e09c01bb1cf4-routing-overlay-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:51:54.387807 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:54.387691 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a15da393-ddf3-4681-bfc7-e09c01bb1cf4-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:51:54.389172 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:54.389144 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a15da393-ddf3-4681-bfc7-e09c01bb1cf4-provider-credentials" pod "a15da393-ddf3-4681-bfc7-e09c01bb1cf4" (UID: "a15da393-ddf3-4681-bfc7-e09c01bb1cf4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:51:54.389272 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:54.389252 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a15da393-ddf3-4681-bfc7-e09c01bb1cf4-kube-api-access-r47k6" pod "a15da393-ddf3-4681-bfc7-e09c01bb1cf4" (UID: "a15da393-ddf3-4681-bfc7-e09c01bb1cf4"). InnerVolumeSpecName "kube-api-access-r47k6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:51:54.504611 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:54.504404 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r47k6\" (UniqueName: \"kubernetes.io/projected/a15da393-ddf3-4681-bfc7-e09c01bb1cf4-kube-api-access-r47k6\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:51:54.505457 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:54.505435 5019 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a15da393-ddf3-4681-bfc7-e09c01bb1cf4-provider-credentials\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:51:55.086564 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:55.086520 5019 generic.go:356] "Generic (PLEG): container finished" podID="a15da393-ddf3-4681-bfc7-e09c01bb1cf4" containerID="00cb394a4f7a953a645af9c125cc5ce767bc22fdeb6ea24fe03759a5228a08a2" exitCode=0 Oct 31 04:51:55.087025 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:55.086581 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-796dfn2nl" Oct 31 04:51:55.087025 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:55.086603 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-796dfn2nl" event={"ID":"a15da393-ddf3-4681-bfc7-e09c01bb1cf4","Type":"ContainerDied","Data":"00cb394a4f7a953a645af9c125cc5ce767bc22fdeb6ea24fe03759a5228a08a2"} Oct 31 04:51:55.087025 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:55.086666 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-796dfn2nl" event={"ID":"a15da393-ddf3-4681-bfc7-e09c01bb1cf4","Type":"ContainerDied","Data":"9bd6a77f0e4ab04b0ea61cbcd4851cda2873a8ea2c3c3b91b5e9589026d2f29b"} Oct 31 04:51:55.087025 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:55.086691 5019 scope.go:122] "RemoveContainer" containerID="00cb394a4f7a953a645af9c125cc5ce767bc22fdeb6ea24fe03759a5228a08a2" Oct 31 04:51:55.098611 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:55.098595 5019 scope.go:122] "RemoveContainer" containerID="00cb394a4f7a953a645af9c125cc5ce767bc22fdeb6ea24fe03759a5228a08a2" Oct 31 04:51:55.098832 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:51:55.098812 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00cb394a4f7a953a645af9c125cc5ce767bc22fdeb6ea24fe03759a5228a08a2\": container with ID starting with 00cb394a4f7a953a645af9c125cc5ce767bc22fdeb6ea24fe03759a5228a08a2 not found: ID does not exist" containerID="00cb394a4f7a953a645af9c125cc5ce767bc22fdeb6ea24fe03759a5228a08a2" Oct 31 04:51:55.098878 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:55.098845 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00cb394a4f7a953a645af9c125cc5ce767bc22fdeb6ea24fe03759a5228a08a2"} err="failed to get container status \"00cb394a4f7a953a645af9c125cc5ce767bc22fdeb6ea24fe03759a5228a08a2\": rpc error: code = NotFound desc = could not find container \"00cb394a4f7a953a645af9c125cc5ce767bc22fdeb6ea24fe03759a5228a08a2\": container with ID starting with 00cb394a4f7a953a645af9c125cc5ce767bc22fdeb6ea24fe03759a5228a08a2 not found: ID does not exist" Oct 31 04:51:55.164346 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:55.164306 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-796dfn2nl"] Oct 31 04:51:55.169934 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:55.169910 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-796dfn2nl"] Oct 31 04:51:55.411518 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:51:55.411442 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a15da393-ddf3-4681-bfc7-e09c01bb1cf4" path="/var/lib/kubelet/pods/a15da393-ddf3-4681-bfc7-e09c01bb1cf4/volumes" Oct 31 04:52:10.906437 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:52:10.906402 5019 scope.go:122] "RemoveContainer" containerID="fb077ff201abe5a21e7859f651d82292b0fa227358abb785efec341b4cd31152" Oct 31 04:52:10.918128 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:52:10.918096 5019 scope.go:122] "RemoveContainer" containerID="b2ab5c53d147b351b7d23f60d10b7c7c61421f099cc286faac3b2a07b3c28743" Oct 31 04:52:10.929121 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:52:10.929103 5019 scope.go:122] "RemoveContainer" containerID="9d909ffe2911dfd54ab8908d9cd6f1969bfa23a897766ba59cd62e90547ce37c" Oct 31 04:52:10.938886 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:52:10.938856 5019 scope.go:122] "RemoveContainer" containerID="d64da45f766c20875f51615657f1a73606b775a3928f5b502426bb8552162c1d" Oct 31 04:52:10.948907 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:52:10.948887 5019 scope.go:122] "RemoveContainer" containerID="829a502f82b4af793a6be4060dbebd11449d3d775c10a53f82926e9ce7f98035" Oct 31 04:52:16.034104 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:52:16.034067 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af60c299-x6r7g"] Oct 31 04:52:16.046377 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:52:16.046346 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af60c299-x6r7g"] Oct 31 04:52:17.414470 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:52:17.414438 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="763a4426-74f9-4458-981b-c7df977f3f5d" path="/var/lib/kubelet/pods/763a4426-74f9-4458-981b-c7df977f3f5d/volumes" Oct 31 04:52:20.517445 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:52:20.517402 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:52:22.421757 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:52:22.420904 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:52:28.033974 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:52:28.033935 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bf4a51a-rhtx7"] Oct 31 04:52:28.044182 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:52:28.044152 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bf4a51a-rhtx7"] Oct 31 04:52:29.411014 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:52:29.410975 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d8d41c0-35f8-4b9b-a0c9-78bb458296e5" path="/var/lib/kubelet/pods/9d8d41c0-35f8-4b9b-a0c9-78bb458296e5/volumes" Oct 31 04:52:30.028942 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:52:30.028908 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8de338de-dhrs2"] Oct 31 04:52:30.040647 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:52:30.040609 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8de338de-dhrs2"] Oct 31 04:52:31.413642 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:52:31.413599 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="606c64ec-b8db-43e3-9d99-2f8092b031ad" path="/var/lib/kubelet/pods/606c64ec-b8db-43e3-9d99-2f8092b031ad/volumes" Oct 31 04:52:39.031277 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:52:39.031241 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-445cfbac-vw5hf"] Oct 31 04:52:39.042376 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:52:39.042345 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-445cfbac-vw5hf"] Oct 31 04:52:39.414824 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:52:39.414744 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b7643e1-621e-4610-a378-8732b4b33d70" path="/var/lib/kubelet/pods/6b7643e1-621e-4610-a378-8732b4b33d70/volumes" Oct 31 04:52:41.032910 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:52:41.032875 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4df7c260-rdsmc"] Oct 31 04:52:41.046521 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:52:41.046479 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4df7c260-rdsmc"] Oct 31 04:52:41.582358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:52:41.557576 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="184444ef-c8f0-43a1-8223-82d2f99d8e29" path="/var/lib/kubelet/pods/184444ef-c8f0-43a1-8223-82d2f99d8e29/volumes" Oct 31 04:52:42.797615 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:52:42.797572 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-cb4fj785m"] Oct 31 04:52:42.798864 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:52:42.798838 5019 cpu_manager.go:414] "containerMap: removing container" podUID="a15da393-ddf3-4681-bfc7-e09c01bb1cf4" containerName="payload-processing" Oct 31 04:52:42.798988 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:52:42.798866 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a15da393-ddf3-4681-bfc7-e09c01bb1cf4" containerName="payload-processing" Oct 31 04:52:42.799080 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:52:42.799068 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="a15da393-ddf3-4681-bfc7-e09c01bb1cf4" containerName="payload-processing" Oct 31 04:52:42.804586 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:52:42.804563 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-cb4fj785m" Oct 31 04:52:42.822976 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:52:42.822940 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-cb4fj785m"] Oct 31 04:52:42.827258 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:52:42.827223 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-567ca6\"/\"agc-cred-1d6252858bc8\"" Oct 31 04:52:42.908936 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:52:42.908884 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5cc6ba4-689b-49dc-982a-a4812e64befb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-cb4fj785m\" (UID: \"e5cc6ba4-689b-49dc-982a-a4812e64befb\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-cb4fj785m" Oct 31 04:52:42.909149 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:52:42.908962 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e5cc6ba4-689b-49dc-982a-a4812e64befb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-cb4fj785m\" (UID: \"e5cc6ba4-689b-49dc-982a-a4812e64befb\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-cb4fj785m" Oct 31 04:52:42.911403 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:52:42.909344 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e5cc6ba4-689b-49dc-982a-a4812e64befb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-cb4fj785m\" (UID: \"e5cc6ba4-689b-49dc-982a-a4812e64befb\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-cb4fj785m" Oct 31 04:52:42.911403 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:52:42.909454 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfvcl\" (UniqueName: \"kubernetes.io/projected/e5cc6ba4-689b-49dc-982a-a4812e64befb-kube-api-access-vfvcl\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-cb4fj785m\" (UID: \"e5cc6ba4-689b-49dc-982a-a4812e64befb\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-cb4fj785m" Oct 31 04:52:43.010917 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:52:43.010858 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e5cc6ba4-689b-49dc-982a-a4812e64befb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-cb4fj785m\" (UID: \"e5cc6ba4-689b-49dc-982a-a4812e64befb\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-cb4fj785m" Oct 31 04:52:43.011133 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:52:43.010996 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfvcl\" (UniqueName: \"kubernetes.io/projected/e5cc6ba4-689b-49dc-982a-a4812e64befb-kube-api-access-vfvcl\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-cb4fj785m\" (UID: \"e5cc6ba4-689b-49dc-982a-a4812e64befb\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-cb4fj785m" Oct 31 04:52:43.011133 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:52:43.011070 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5cc6ba4-689b-49dc-982a-a4812e64befb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-cb4fj785m\" (UID: \"e5cc6ba4-689b-49dc-982a-a4812e64befb\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-cb4fj785m" Oct 31 04:52:43.011133 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:52:43.011103 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e5cc6ba4-689b-49dc-982a-a4812e64befb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-cb4fj785m\" (UID: \"e5cc6ba4-689b-49dc-982a-a4812e64befb\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-cb4fj785m" Oct 31 04:52:43.012107 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:52:43.012073 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e5cc6ba4-689b-49dc-982a-a4812e64befb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-cb4fj785m\" (UID: \"e5cc6ba4-689b-49dc-982a-a4812e64befb\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-cb4fj785m" Oct 31 04:52:43.012248 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:52:43.012117 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5cc6ba4-689b-49dc-982a-a4812e64befb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-cb4fj785m\" (UID: \"e5cc6ba4-689b-49dc-982a-a4812e64befb\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-cb4fj785m" Oct 31 04:52:43.013763 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:52:43.013739 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e5cc6ba4-689b-49dc-982a-a4812e64befb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-cb4fj785m\" (UID: \"e5cc6ba4-689b-49dc-982a-a4812e64befb\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-cb4fj785m" Oct 31 04:52:43.034368 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:52:43.034302 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfvcl\" (UniqueName: \"kubernetes.io/projected/e5cc6ba4-689b-49dc-982a-a4812e64befb-kube-api-access-vfvcl\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-cb4fj785m\" (UID: \"e5cc6ba4-689b-49dc-982a-a4812e64befb\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-cb4fj785m" Oct 31 04:52:43.123678 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:52:43.123578 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-cb4fj785m" Oct 31 04:52:43.434216 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:52:43.434182 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5cc6ba4_689b_49dc_982a_a4812e64befb.slice/crio-75ee894228c29acc9472a2d262a880e3b616a0045ceb182b134141131c947f1a WatchSource:0}: Error finding container 75ee894228c29acc9472a2d262a880e3b616a0045ceb182b134141131c947f1a: Status 404 returned error can't find the container with id 75ee894228c29acc9472a2d262a880e3b616a0045ceb182b134141131c947f1a Oct 31 04:52:43.434946 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:52:43.434920 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-cb4fj785m"] Oct 31 04:52:43.928691 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:52:43.928653 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-cb4fj785m" event={"ID":"e5cc6ba4-689b-49dc-982a-a4812e64befb","Type":"ContainerStarted","Data":"50a419b59df09321c9ec81fe8a38261098415cb8e568909f3d2230567e4150d9"} Oct 31 04:52:43.929098 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:52:43.928716 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-cb4fj785m" event={"ID":"e5cc6ba4-689b-49dc-982a-a4812e64befb","Type":"ContainerStarted","Data":"75ee894228c29acc9472a2d262a880e3b616a0045ceb182b134141131c947f1a"} Oct 31 04:52:43.929098 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:52:43.928747 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-cb4fj785m" Oct 31 04:52:43.970919 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:52:43.970857 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-cb4fj785m" podStartSLOduration=1.9708384730000001 podStartE2EDuration="1.970838473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:52:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:52:43.968707467 +0000 UTC m=+2918.725045507" watchObservedRunningTime="2026-10-31 04:52:43.970838473 +0000 UTC m=+2918.727176515" Oct 31 04:52:51.031313 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:52:51.031238 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b236824d-k5j5w"] Oct 31 04:52:51.043263 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:52:51.043232 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b236824d-k5j5w"] Oct 31 04:52:51.413549 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:52:51.413464 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b578bb3a-b726-4057-a46b-b25e1cb0d767" path="/var/lib/kubelet/pods/b578bb3a-b726-4057-a46b-b25e1cb0d767/volumes" Oct 31 04:52:52.393359 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:52:52.393301 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:52:54.940851 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:52:54.940820 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-cb4fj785m" Oct 31 04:52:55.022673 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:52:55.022629 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9wg7c8"] Oct 31 04:53:00.023840 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:00.023793 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9wg7c8" podUID="cf5c3cf2-fb82-44c8-89e2-0b280d3ed769" containerName="payload-processing" containerID="cri-o://562813c63698940a6b7f62ea03886f720ae3857f1e73de578e98fa81f1d55510" gracePeriod=25 Oct 31 04:53:00.030706 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:00.030678 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80ec86b6-lmk2g"] Oct 31 04:53:00.041867 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:00.041842 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80ec86b6-lmk2g"] Oct 31 04:53:00.345324 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:00.345303 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9wg7c8" Oct 31 04:53:00.401233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:00.401197 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf5c3cf2-fb82-44c8-89e2-0b280d3ed769-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf5c3cf2-fb82-44c8-89e2-0b280d3ed769-plugins-config-volume\") pod \"cf5c3cf2-fb82-44c8-89e2-0b280d3ed769\" (UID: \"cf5c3cf2-fb82-44c8-89e2-0b280d3ed769\") " Oct 31 04:53:00.401440 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:00.401314 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf5c3cf2-fb82-44c8-89e2-0b280d3ed769-provider-credentials\" (UniqueName: \"kubernetes.io/projected/cf5c3cf2-fb82-44c8-89e2-0b280d3ed769-provider-credentials\") pod \"cf5c3cf2-fb82-44c8-89e2-0b280d3ed769\" (UID: \"cf5c3cf2-fb82-44c8-89e2-0b280d3ed769\") " Oct 31 04:53:00.401517 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:00.401488 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf5c3cf2-fb82-44c8-89e2-0b280d3ed769-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cf5c3cf2-fb82-44c8-89e2-0b280d3ed769-routing-overlay-volume\") pod \"cf5c3cf2-fb82-44c8-89e2-0b280d3ed769\" (UID: \"cf5c3cf2-fb82-44c8-89e2-0b280d3ed769\") " Oct 31 04:53:00.401635 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:00.401616 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf5c3cf2-fb82-44c8-89e2-0b280d3ed769-kube-api-access-7ltd6\" (UniqueName: \"kubernetes.io/projected/cf5c3cf2-fb82-44c8-89e2-0b280d3ed769-kube-api-access-7ltd6\") pod \"cf5c3cf2-fb82-44c8-89e2-0b280d3ed769\" (UID: \"cf5c3cf2-fb82-44c8-89e2-0b280d3ed769\") " Oct 31 04:53:00.402020 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:00.401993 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf5c3cf2-fb82-44c8-89e2-0b280d3ed769-routing-overlay-volume" pod "cf5c3cf2-fb82-44c8-89e2-0b280d3ed769" (UID: "cf5c3cf2-fb82-44c8-89e2-0b280d3ed769"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:53:00.402101 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:00.402018 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf5c3cf2-fb82-44c8-89e2-0b280d3ed769-plugins-config-volume" pod "cf5c3cf2-fb82-44c8-89e2-0b280d3ed769" (UID: "cf5c3cf2-fb82-44c8-89e2-0b280d3ed769"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:53:00.402731 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:00.402683 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf5c3cf2-fb82-44c8-89e2-0b280d3ed769-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:53:00.402731 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:00.402710 5019 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cf5c3cf2-fb82-44c8-89e2-0b280d3ed769-routing-overlay-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:53:00.404775 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:00.404729 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf5c3cf2-fb82-44c8-89e2-0b280d3ed769-kube-api-access-7ltd6" pod "cf5c3cf2-fb82-44c8-89e2-0b280d3ed769" (UID: "cf5c3cf2-fb82-44c8-89e2-0b280d3ed769"). InnerVolumeSpecName "kube-api-access-7ltd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:53:00.407505 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:00.407479 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf5c3cf2-fb82-44c8-89e2-0b280d3ed769-provider-credentials" pod "cf5c3cf2-fb82-44c8-89e2-0b280d3ed769" (UID: "cf5c3cf2-fb82-44c8-89e2-0b280d3ed769"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:53:00.510722 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:00.509387 5019 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cf5c3cf2-fb82-44c8-89e2-0b280d3ed769-provider-credentials\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:53:00.511308 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:00.511277 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ltd6\" (UniqueName: \"kubernetes.io/projected/cf5c3cf2-fb82-44c8-89e2-0b280d3ed769-kube-api-access-7ltd6\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:53:01.126646 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:01.126609 5019 generic.go:356] "Generic (PLEG): container finished" podID="cf5c3cf2-fb82-44c8-89e2-0b280d3ed769" containerID="562813c63698940a6b7f62ea03886f720ae3857f1e73de578e98fa81f1d55510" exitCode=0 Oct 31 04:53:01.127098 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:01.126652 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9wg7c8" Oct 31 04:53:01.127098 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:01.126693 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9wg7c8" event={"ID":"cf5c3cf2-fb82-44c8-89e2-0b280d3ed769","Type":"ContainerDied","Data":"562813c63698940a6b7f62ea03886f720ae3857f1e73de578e98fa81f1d55510"} Oct 31 04:53:01.127098 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:01.126748 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9wg7c8" event={"ID":"cf5c3cf2-fb82-44c8-89e2-0b280d3ed769","Type":"ContainerDied","Data":"49fa71350f30a02ce4bb4e0c75b3a9ecaa332b7e52c0f21ac5b84cead6d692ac"} Oct 31 04:53:01.127098 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:01.126774 5019 scope.go:122] "RemoveContainer" containerID="562813c63698940a6b7f62ea03886f720ae3857f1e73de578e98fa81f1d55510" Oct 31 04:53:01.144283 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:01.144265 5019 scope.go:122] "RemoveContainer" containerID="562813c63698940a6b7f62ea03886f720ae3857f1e73de578e98fa81f1d55510" Oct 31 04:53:01.144610 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:53:01.144585 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"562813c63698940a6b7f62ea03886f720ae3857f1e73de578e98fa81f1d55510\": container with ID starting with 562813c63698940a6b7f62ea03886f720ae3857f1e73de578e98fa81f1d55510 not found: ID does not exist" containerID="562813c63698940a6b7f62ea03886f720ae3857f1e73de578e98fa81f1d55510" Oct 31 04:53:01.144685 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:01.144615 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"562813c63698940a6b7f62ea03886f720ae3857f1e73de578e98fa81f1d55510"} err="failed to get container status \"562813c63698940a6b7f62ea03886f720ae3857f1e73de578e98fa81f1d55510\": rpc error: code = NotFound desc = could not find container \"562813c63698940a6b7f62ea03886f720ae3857f1e73de578e98fa81f1d55510\": container with ID starting with 562813c63698940a6b7f62ea03886f720ae3857f1e73de578e98fa81f1d55510 not found: ID does not exist" Oct 31 04:53:01.182583 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:01.182556 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9wg7c8"] Oct 31 04:53:01.194101 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:01.194076 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-64c9wg7c8"] Oct 31 04:53:01.413686 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:01.413607 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2b416ac-8e8c-48d6-bf4d-7a70947bdfa0" path="/var/lib/kubelet/pods/b2b416ac-8e8c-48d6-bf4d-7a70947bdfa0/volumes" Oct 31 04:53:01.414240 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:01.414209 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf5c3cf2-fb82-44c8-89e2-0b280d3ed769" path="/var/lib/kubelet/pods/cf5c3cf2-fb82-44c8-89e2-0b280d3ed769/volumes" Oct 31 04:53:04.031842 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:04.031807 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-697d4911-tk4xj"] Oct 31 04:53:04.043297 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:04.043269 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-697d4911-tk4xj"] Oct 31 04:53:05.416771 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:05.416730 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="022f2eaf-4d8a-4d8f-82d0-3906e02a7b5a" path="/var/lib/kubelet/pods/022f2eaf-4d8a-4d8f-82d0-3906e02a7b5a/volumes" Oct 31 04:53:10.901476 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:10.901418 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": context deadline exceeded" Oct 31 04:53:12.067342 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:12.067268 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c6e458a-jtds8"] Oct 31 04:53:12.103355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:12.090250 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c6e458a-jtds8"] Oct 31 04:53:13.132356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:13.119450 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-pghv7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:53:13.132356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:13.119539 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7" podUID="e0ccc03e-0e5e-40b7-8a40-718f199dd747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:53:13.169207 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:13.169167 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" podUID="e0f4b8c2-14f4-4052-bf54-dd65bf59e413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:53:13.271355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:13.259172 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" podUID="b4ced3ab-41e4-4c54-93eb-95eddaae992d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:53:13.291360 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:13.290518 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" podUID="dccd15a9-47f5-4bd1-b21b-9d5ca8b05845" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:53:13.494151 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:13.492453 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" podUID="643433d8-418e-4102-8bc3-cd544932614a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:53:13.523478 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:13.523408 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:53:13.524039 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:13.523929 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f259ab0-453f-4c0b-97e2-2d511cced150" path="/var/lib/kubelet/pods/7f259ab0-453f-4c0b-97e2-2d511cced150/volumes" Oct 31 04:53:13.562507 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:13.562458 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 04:53:13.572971 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:13.572929 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 31 04:53:13.598710 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:13.594442 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:53:13.598710 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:13.594516 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4" podUID="7de5a068-ea2b-4e78-8c43-3ae8dddabfd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:53:17.515560 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:17.515512 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9893a9cc-jm4zm"] Oct 31 04:53:17.516500 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:17.516475 5019 cpu_manager.go:414] "containerMap: removing container" podUID="cf5c3cf2-fb82-44c8-89e2-0b280d3ed769" containerName="payload-processing" Oct 31 04:53:17.516500 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:17.516497 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf5c3cf2-fb82-44c8-89e2-0b280d3ed769" containerName="payload-processing" Oct 31 04:53:17.516728 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:17.516713 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf5c3cf2-fb82-44c8-89e2-0b280d3ed769" containerName="payload-processing" Oct 31 04:53:17.520774 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:17.520742 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9893a9cc-jm4zm" Oct 31 04:53:17.530608 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:17.530579 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94x8z\" (UniqueName: \"kubernetes.io/projected/3a334173-d81b-4b31-8a77-5ad4241215b4-kube-api-access-94x8z\") pod \"maas-api-revoke-sub-9893a9cc-jm4zm\" (UID: \"3a334173-d81b-4b31-8a77-5ad4241215b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9893a9cc-jm4zm" Oct 31 04:53:17.530700 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:17.530655 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a334173-d81b-4b31-8a77-5ad4241215b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9893a9cc-jm4zm\" (UID: \"3a334173-d81b-4b31-8a77-5ad4241215b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9893a9cc-jm4zm" Oct 31 04:53:17.530932 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:17.530898 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a334173-d81b-4b31-8a77-5ad4241215b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9893a9cc-jm4zm\" (UID: \"3a334173-d81b-4b31-8a77-5ad4241215b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9893a9cc-jm4zm" Oct 31 04:53:17.540050 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:17.540010 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9893a9cc-jm4zm"] Oct 31 04:53:17.544544 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:17.544496 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:53:17.544759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:17.544549 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:53:17.544759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:17.544596 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:53:17.544759 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:17.544618 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:53:17.633442 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:17.633389 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a334173-d81b-4b31-8a77-5ad4241215b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9893a9cc-jm4zm\" (UID: \"3a334173-d81b-4b31-8a77-5ad4241215b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9893a9cc-jm4zm" Oct 31 04:53:17.633686 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:17.633663 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a334173-d81b-4b31-8a77-5ad4241215b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9893a9cc-jm4zm\" (UID: \"3a334173-d81b-4b31-8a77-5ad4241215b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9893a9cc-jm4zm" Oct 31 04:53:17.633929 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:17.633909 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94x8z\" (UniqueName: \"kubernetes.io/projected/3a334173-d81b-4b31-8a77-5ad4241215b4-kube-api-access-94x8z\") pod \"maas-api-revoke-sub-9893a9cc-jm4zm\" (UID: \"3a334173-d81b-4b31-8a77-5ad4241215b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9893a9cc-jm4zm" Oct 31 04:53:17.634513 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:17.634489 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a334173-d81b-4b31-8a77-5ad4241215b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9893a9cc-jm4zm\" (UID: \"3a334173-d81b-4b31-8a77-5ad4241215b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9893a9cc-jm4zm" Oct 31 04:53:17.636656 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:17.636626 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a334173-d81b-4b31-8a77-5ad4241215b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9893a9cc-jm4zm\" (UID: \"3a334173-d81b-4b31-8a77-5ad4241215b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9893a9cc-jm4zm" Oct 31 04:53:17.658257 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:17.658222 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94x8z\" (UniqueName: \"kubernetes.io/projected/3a334173-d81b-4b31-8a77-5ad4241215b4-kube-api-access-94x8z\") pod \"maas-api-revoke-sub-9893a9cc-jm4zm\" (UID: \"3a334173-d81b-4b31-8a77-5ad4241215b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9893a9cc-jm4zm" Oct 31 04:53:17.839032 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:17.838932 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9893a9cc-jm4zm" Oct 31 04:53:18.338990 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:53:18.338952 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a334173_d81b_4b31_8a77_5ad4241215b4.slice/crio-97e6e9a2bc5ac6ee8f3a7a279798d5666391d162166b6eb939b8e54c6a82e31c WatchSource:0}: Error finding container 97e6e9a2bc5ac6ee8f3a7a279798d5666391d162166b6eb939b8e54c6a82e31c: Status 404 returned error can't find the container with id 97e6e9a2bc5ac6ee8f3a7a279798d5666391d162166b6eb939b8e54c6a82e31c Oct 31 04:53:18.340424 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:18.340386 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9893a9cc-jm4zm"] Oct 31 04:53:18.465948 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:18.465903 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9893a9cc-jm4zm" event={"ID":"3a334173-d81b-4b31-8a77-5ad4241215b4","Type":"ContainerStarted","Data":"97e6e9a2bc5ac6ee8f3a7a279798d5666391d162166b6eb939b8e54c6a82e31c"} Oct 31 04:53:19.478945 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:19.478907 5019 generic.go:356] "Generic (PLEG): container finished" podID="3a334173-d81b-4b31-8a77-5ad4241215b4" containerID="7d5e9fb0e58e3de2848acadea0f1aa8f96465b5dc337246addbb580d75a6e384" exitCode=0 Oct 31 04:53:19.479487 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:19.479006 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9893a9cc-jm4zm" event={"ID":"3a334173-d81b-4b31-8a77-5ad4241215b4","Type":"ContainerDied","Data":"7d5e9fb0e58e3de2848acadea0f1aa8f96465b5dc337246addbb580d75a6e384"} Oct 31 04:53:20.701003 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:20.700971 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9893a9cc-jm4zm" Oct 31 04:53:20.790742 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:20.790647 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a334173-d81b-4b31-8a77-5ad4241215b4-kube-api-access-94x8z\" (UniqueName: \"kubernetes.io/projected/3a334173-d81b-4b31-8a77-5ad4241215b4-kube-api-access-94x8z\") pod \"3a334173-d81b-4b31-8a77-5ad4241215b4\" (UID: \"3a334173-d81b-4b31-8a77-5ad4241215b4\") " Oct 31 04:53:20.790742 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:20.790719 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a334173-d81b-4b31-8a77-5ad4241215b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a334173-d81b-4b31-8a77-5ad4241215b4-maas-api-openshift-ca\") pod \"3a334173-d81b-4b31-8a77-5ad4241215b4\" (UID: \"3a334173-d81b-4b31-8a77-5ad4241215b4\") " Oct 31 04:53:20.791015 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:20.790931 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a334173-d81b-4b31-8a77-5ad4241215b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a334173-d81b-4b31-8a77-5ad4241215b4-maas-api-xks-ca\") pod \"3a334173-d81b-4b31-8a77-5ad4241215b4\" (UID: \"3a334173-d81b-4b31-8a77-5ad4241215b4\") " Oct 31 04:53:20.791157 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:20.791127 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a334173-d81b-4b31-8a77-5ad4241215b4-maas-api-openshift-ca" pod "3a334173-d81b-4b31-8a77-5ad4241215b4" (UID: "3a334173-d81b-4b31-8a77-5ad4241215b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:53:20.791533 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:20.791513 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a334173-d81b-4b31-8a77-5ad4241215b4-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:53:20.793863 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:20.793822 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a334173-d81b-4b31-8a77-5ad4241215b4-maas-api-xks-ca" pod "3a334173-d81b-4b31-8a77-5ad4241215b4" (UID: "3a334173-d81b-4b31-8a77-5ad4241215b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:53:20.793975 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:20.793889 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a334173-d81b-4b31-8a77-5ad4241215b4-kube-api-access-94x8z" pod "3a334173-d81b-4b31-8a77-5ad4241215b4" (UID: "3a334173-d81b-4b31-8a77-5ad4241215b4"). InnerVolumeSpecName "kube-api-access-94x8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:53:20.893693 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:20.893650 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a334173-d81b-4b31-8a77-5ad4241215b4-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:53:20.893693 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:20.893689 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94x8z\" (UniqueName: \"kubernetes.io/projected/3a334173-d81b-4b31-8a77-5ad4241215b4-kube-api-access-94x8z\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:53:21.412909 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:21.412876 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-826dd91f-rzcxg"] Oct 31 04:53:21.413926 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:21.413902 5019 cpu_manager.go:414] "containerMap: removing container" podUID="3a334173-d81b-4b31-8a77-5ad4241215b4" containerName="revoke-keys" Oct 31 04:53:21.413926 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:21.413924 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a334173-d81b-4b31-8a77-5ad4241215b4" containerName="revoke-keys" Oct 31 04:53:21.414134 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:21.414089 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a334173-d81b-4b31-8a77-5ad4241215b4" containerName="revoke-keys" Oct 31 04:53:21.418768 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:21.418739 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-826dd91f-rzcxg" Oct 31 04:53:21.432283 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:21.432232 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-826dd91f-rzcxg"] Oct 31 04:53:21.504056 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:21.504018 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2d931e9-f094-4771-8af5-a11bf84d8978-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-826dd91f-rzcxg\" (UID: \"d2d931e9-f094-4771-8af5-a11bf84d8978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-826dd91f-rzcxg" Oct 31 04:53:21.504056 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:21.504067 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbtp6\" (UniqueName: \"kubernetes.io/projected/d2d931e9-f094-4771-8af5-a11bf84d8978-kube-api-access-tbtp6\") pod \"maas-api-revoke-sub-826dd91f-rzcxg\" (UID: \"d2d931e9-f094-4771-8af5-a11bf84d8978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-826dd91f-rzcxg" Oct 31 04:53:21.504863 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:21.504555 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2d931e9-f094-4771-8af5-a11bf84d8978-maas-api-xks-ca\") pod \"maas-api-revoke-sub-826dd91f-rzcxg\" (UID: \"d2d931e9-f094-4771-8af5-a11bf84d8978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-826dd91f-rzcxg" Oct 31 04:53:21.505470 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:21.505435 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9893a9cc-jm4zm" event={"ID":"3a334173-d81b-4b31-8a77-5ad4241215b4","Type":"ContainerDied","Data":"97e6e9a2bc5ac6ee8f3a7a279798d5666391d162166b6eb939b8e54c6a82e31c"} Oct 31 04:53:21.505470 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:21.505462 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9893a9cc-jm4zm" Oct 31 04:53:21.505619 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:21.505500 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97e6e9a2bc5ac6ee8f3a7a279798d5666391d162166b6eb939b8e54c6a82e31c" Oct 31 04:53:21.610036 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:21.609183 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2d931e9-f094-4771-8af5-a11bf84d8978-maas-api-xks-ca\") pod \"maas-api-revoke-sub-826dd91f-rzcxg\" (UID: \"d2d931e9-f094-4771-8af5-a11bf84d8978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-826dd91f-rzcxg" Oct 31 04:53:21.610036 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:21.609435 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2d931e9-f094-4771-8af5-a11bf84d8978-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-826dd91f-rzcxg\" (UID: \"d2d931e9-f094-4771-8af5-a11bf84d8978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-826dd91f-rzcxg" Oct 31 04:53:21.610036 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:21.609481 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbtp6\" (UniqueName: \"kubernetes.io/projected/d2d931e9-f094-4771-8af5-a11bf84d8978-kube-api-access-tbtp6\") pod \"maas-api-revoke-sub-826dd91f-rzcxg\" (UID: \"d2d931e9-f094-4771-8af5-a11bf84d8978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-826dd91f-rzcxg" Oct 31 04:53:21.614610 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:21.611194 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2d931e9-f094-4771-8af5-a11bf84d8978-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-826dd91f-rzcxg\" (UID: \"d2d931e9-f094-4771-8af5-a11bf84d8978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-826dd91f-rzcxg" Oct 31 04:53:21.616173 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:21.616129 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2d931e9-f094-4771-8af5-a11bf84d8978-maas-api-xks-ca\") pod \"maas-api-revoke-sub-826dd91f-rzcxg\" (UID: \"d2d931e9-f094-4771-8af5-a11bf84d8978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-826dd91f-rzcxg" Oct 31 04:53:21.635500 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:21.635464 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbtp6\" (UniqueName: \"kubernetes.io/projected/d2d931e9-f094-4771-8af5-a11bf84d8978-kube-api-access-tbtp6\") pod \"maas-api-revoke-sub-826dd91f-rzcxg\" (UID: \"d2d931e9-f094-4771-8af5-a11bf84d8978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-826dd91f-rzcxg" Oct 31 04:53:21.749549 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:21.749508 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-826dd91f-rzcxg" Oct 31 04:53:22.066167 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:22.066132 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-826dd91f-rzcxg"] Oct 31 04:53:22.066377 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:53:22.066238 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2d931e9_f094_4771_8af5_a11bf84d8978.slice/crio-692862dfeba5da8bf1663f39558a7334c0cf44b4eb211cb52d824b7d565555b1 WatchSource:0}: Error finding container 692862dfeba5da8bf1663f39558a7334c0cf44b4eb211cb52d824b7d565555b1: Status 404 returned error can't find the container with id 692862dfeba5da8bf1663f39558a7334c0cf44b4eb211cb52d824b7d565555b1 Oct 31 04:53:22.401309 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:53:22.401197 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:53:22.518280 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:22.518236 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-826dd91f-rzcxg" event={"ID":"d2d931e9-f094-4771-8af5-a11bf84d8978","Type":"ContainerStarted","Data":"b175dcd2444cc6289ab7a2a9d7f0fc0168ac7d174d4c0d1bbb40fa487aeab5c1"} Oct 31 04:53:22.518537 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:22.518312 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-826dd91f-rzcxg" event={"ID":"d2d931e9-f094-4771-8af5-a11bf84d8978","Type":"ContainerStarted","Data":"692862dfeba5da8bf1663f39558a7334c0cf44b4eb211cb52d824b7d565555b1"} Oct 31 04:53:22.564929 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:22.564850 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-826dd91f-rzcxg" podStartSLOduration=1.564828784 podStartE2EDuration="1.564828784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:53:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:53:22.557836819 +0000 UTC m=+2957.314174860" watchObservedRunningTime="2026-10-31 04:53:22.564828784 +0000 UTC m=+2957.321166825" Oct 31 04:53:23.030752 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:23.030705 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-758649b3-qbzbg"] Oct 31 04:53:23.042472 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:23.042433 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-758649b3-qbzbg"] Oct 31 04:53:23.427630 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:23.427530 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acfc3ef9-fbeb-4d63-bab5-04e927417ccd" path="/var/lib/kubelet/pods/acfc3ef9-fbeb-4d63-bab5-04e927417ccd/volumes" Oct 31 04:53:23.531148 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:23.531110 5019 generic.go:356] "Generic (PLEG): container finished" podID="d2d931e9-f094-4771-8af5-a11bf84d8978" containerID="b175dcd2444cc6289ab7a2a9d7f0fc0168ac7d174d4c0d1bbb40fa487aeab5c1" exitCode=0 Oct 31 04:53:23.531349 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:23.531191 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-826dd91f-rzcxg" event={"ID":"d2d931e9-f094-4771-8af5-a11bf84d8978","Type":"ContainerDied","Data":"b175dcd2444cc6289ab7a2a9d7f0fc0168ac7d174d4c0d1bbb40fa487aeab5c1"} Oct 31 04:53:24.760467 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:24.760432 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-826dd91f-rzcxg" Oct 31 04:53:24.863468 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:24.863358 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2d931e9-f094-4771-8af5-a11bf84d8978-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2d931e9-f094-4771-8af5-a11bf84d8978-maas-api-openshift-ca\") pod \"d2d931e9-f094-4771-8af5-a11bf84d8978\" (UID: \"d2d931e9-f094-4771-8af5-a11bf84d8978\") " Oct 31 04:53:24.863675 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:24.863511 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2d931e9-f094-4771-8af5-a11bf84d8978-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2d931e9-f094-4771-8af5-a11bf84d8978-maas-api-xks-ca\") pod \"d2d931e9-f094-4771-8af5-a11bf84d8978\" (UID: \"d2d931e9-f094-4771-8af5-a11bf84d8978\") " Oct 31 04:53:24.863675 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:24.863647 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2d931e9-f094-4771-8af5-a11bf84d8978-kube-api-access-tbtp6\" (UniqueName: \"kubernetes.io/projected/d2d931e9-f094-4771-8af5-a11bf84d8978-kube-api-access-tbtp6\") pod \"d2d931e9-f094-4771-8af5-a11bf84d8978\" (UID: \"d2d931e9-f094-4771-8af5-a11bf84d8978\") " Oct 31 04:53:24.863882 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:24.863856 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2d931e9-f094-4771-8af5-a11bf84d8978-maas-api-openshift-ca" pod "d2d931e9-f094-4771-8af5-a11bf84d8978" (UID: "d2d931e9-f094-4771-8af5-a11bf84d8978"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:53:24.864546 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:24.864517 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2d931e9-f094-4771-8af5-a11bf84d8978-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:53:24.866077 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:24.866048 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2d931e9-f094-4771-8af5-a11bf84d8978-kube-api-access-tbtp6" pod "d2d931e9-f094-4771-8af5-a11bf84d8978" (UID: "d2d931e9-f094-4771-8af5-a11bf84d8978"). InnerVolumeSpecName "kube-api-access-tbtp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:53:24.866312 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:24.866276 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2d931e9-f094-4771-8af5-a11bf84d8978-maas-api-xks-ca" pod "d2d931e9-f094-4771-8af5-a11bf84d8978" (UID: "d2d931e9-f094-4771-8af5-a11bf84d8978"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:53:24.965852 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:24.965812 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2d931e9-f094-4771-8af5-a11bf84d8978-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:53:24.965852 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:24.965851 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbtp6\" (UniqueName: \"kubernetes.io/projected/d2d931e9-f094-4771-8af5-a11bf84d8978-kube-api-access-tbtp6\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:53:25.551743 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:25.551702 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-826dd91f-rzcxg" Oct 31 04:53:25.551743 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:25.551728 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-826dd91f-rzcxg" event={"ID":"d2d931e9-f094-4771-8af5-a11bf84d8978","Type":"ContainerDied","Data":"692862dfeba5da8bf1663f39558a7334c0cf44b4eb211cb52d824b7d565555b1"} Oct 31 04:53:25.552009 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:25.551792 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="692862dfeba5da8bf1663f39558a7334c0cf44b4eb211cb52d824b7d565555b1" Oct 31 04:53:29.865361 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:29.851122 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-340168f9-8mnh2"] Oct 31 04:53:29.865361 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:29.852562 5019 cpu_manager.go:414] "containerMap: removing container" podUID="d2d931e9-f094-4771-8af5-a11bf84d8978" containerName="revoke-keys" Oct 31 04:53:29.865361 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:29.852590 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2d931e9-f094-4771-8af5-a11bf84d8978" containerName="revoke-keys" Oct 31 04:53:29.865361 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:29.852864 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2d931e9-f094-4771-8af5-a11bf84d8978" containerName="revoke-keys" Oct 31 04:53:29.893750 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:29.893705 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-340168f9-8mnh2" Oct 31 04:53:29.940969 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:29.932471 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:53:29.944865 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:29.944808 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-340168f9-8mnh2"] Oct 31 04:53:29.946623 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:29.946575 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:53:29.947126 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:29.947078 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:53:29.947255 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:29.947126 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:53:30.004477 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:30.004414 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/481beb7c-221e-430f-800c-ad11adc67b4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-340168f9-8mnh2\" (UID: \"481beb7c-221e-430f-800c-ad11adc67b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-340168f9-8mnh2" Oct 31 04:53:30.004714 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:30.004656 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/481beb7c-221e-430f-800c-ad11adc67b4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-340168f9-8mnh2\" (UID: \"481beb7c-221e-430f-800c-ad11adc67b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-340168f9-8mnh2" Oct 31 04:53:30.004798 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:30.004734 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcxwb\" (UniqueName: \"kubernetes.io/projected/481beb7c-221e-430f-800c-ad11adc67b4f-kube-api-access-dcxwb\") pod \"maas-api-revoke-sub-340168f9-8mnh2\" (UID: \"481beb7c-221e-430f-800c-ad11adc67b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-340168f9-8mnh2" Oct 31 04:53:30.107016 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:30.106972 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/481beb7c-221e-430f-800c-ad11adc67b4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-340168f9-8mnh2\" (UID: \"481beb7c-221e-430f-800c-ad11adc67b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-340168f9-8mnh2" Oct 31 04:53:30.107227 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:30.107198 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/481beb7c-221e-430f-800c-ad11adc67b4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-340168f9-8mnh2\" (UID: \"481beb7c-221e-430f-800c-ad11adc67b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-340168f9-8mnh2" Oct 31 04:53:30.107295 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:30.107274 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcxwb\" (UniqueName: \"kubernetes.io/projected/481beb7c-221e-430f-800c-ad11adc67b4f-kube-api-access-dcxwb\") pod \"maas-api-revoke-sub-340168f9-8mnh2\" (UID: \"481beb7c-221e-430f-800c-ad11adc67b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-340168f9-8mnh2" Oct 31 04:53:30.107957 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:30.107928 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/481beb7c-221e-430f-800c-ad11adc67b4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-340168f9-8mnh2\" (UID: \"481beb7c-221e-430f-800c-ad11adc67b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-340168f9-8mnh2" Oct 31 04:53:30.109613 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:30.109589 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/481beb7c-221e-430f-800c-ad11adc67b4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-340168f9-8mnh2\" (UID: \"481beb7c-221e-430f-800c-ad11adc67b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-340168f9-8mnh2" Oct 31 04:53:30.130979 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:30.130896 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcxwb\" (UniqueName: \"kubernetes.io/projected/481beb7c-221e-430f-800c-ad11adc67b4f-kube-api-access-dcxwb\") pod \"maas-api-revoke-sub-340168f9-8mnh2\" (UID: \"481beb7c-221e-430f-800c-ad11adc67b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-340168f9-8mnh2" Oct 31 04:53:30.270584 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:30.270540 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-340168f9-8mnh2" Oct 31 04:53:30.589800 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:53:30.589757 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod481beb7c_221e_430f_800c_ad11adc67b4f.slice/crio-c00c007477100fe35a7d0d2a01081acf633153e05ecf77d19c000e37d913fe18 WatchSource:0}: Error finding container c00c007477100fe35a7d0d2a01081acf633153e05ecf77d19c000e37d913fe18: Status 404 returned error can't find the container with id c00c007477100fe35a7d0d2a01081acf633153e05ecf77d19c000e37d913fe18 Oct 31 04:53:30.592606 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:30.592570 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-340168f9-8mnh2"] Oct 31 04:53:30.647240 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:30.647186 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-340168f9-8mnh2" event={"ID":"481beb7c-221e-430f-800c-ad11adc67b4f","Type":"ContainerStarted","Data":"c00c007477100fe35a7d0d2a01081acf633153e05ecf77d19c000e37d913fe18"} Oct 31 04:53:31.658792 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:31.658749 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-340168f9-8mnh2" event={"ID":"481beb7c-221e-430f-800c-ad11adc67b4f","Type":"ContainerStarted","Data":"f57907e303a7dc599e3e49f8e946b8a8bcbc302aed9da2e3aac4a2e9979df2bb"} Oct 31 04:53:31.703814 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:31.703745 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-340168f9-8mnh2" podStartSLOduration=2.7037225510000003 podStartE2EDuration="2.703722551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:53:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:53:31.697874456 +0000 UTC m=+2966.454212496" watchObservedRunningTime="2026-10-31 04:53:31.703722551 +0000 UTC m=+2966.460060591" Oct 31 04:53:32.689429 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:32.686612 5019 generic.go:356] "Generic (PLEG): container finished" podID="481beb7c-221e-430f-800c-ad11adc67b4f" containerID="f57907e303a7dc599e3e49f8e946b8a8bcbc302aed9da2e3aac4a2e9979df2bb" exitCode=0 Oct 31 04:53:32.689429 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:32.687086 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-340168f9-8mnh2" event={"ID":"481beb7c-221e-430f-800c-ad11adc67b4f","Type":"ContainerDied","Data":"f57907e303a7dc599e3e49f8e946b8a8bcbc302aed9da2e3aac4a2e9979df2bb"} Oct 31 04:53:33.922018 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:33.921983 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-340168f9-8mnh2" Oct 31 04:53:33.984283 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:33.984224 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/481beb7c-221e-430f-800c-ad11adc67b4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/481beb7c-221e-430f-800c-ad11adc67b4f-maas-api-xks-ca\") pod \"481beb7c-221e-430f-800c-ad11adc67b4f\" (UID: \"481beb7c-221e-430f-800c-ad11adc67b4f\") " Oct 31 04:53:33.984525 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:33.984391 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/481beb7c-221e-430f-800c-ad11adc67b4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/481beb7c-221e-430f-800c-ad11adc67b4f-maas-api-openshift-ca\") pod \"481beb7c-221e-430f-800c-ad11adc67b4f\" (UID: \"481beb7c-221e-430f-800c-ad11adc67b4f\") " Oct 31 04:53:33.984525 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:33.984510 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/481beb7c-221e-430f-800c-ad11adc67b4f-kube-api-access-dcxwb\" (UniqueName: \"kubernetes.io/projected/481beb7c-221e-430f-800c-ad11adc67b4f-kube-api-access-dcxwb\") pod \"481beb7c-221e-430f-800c-ad11adc67b4f\" (UID: \"481beb7c-221e-430f-800c-ad11adc67b4f\") " Oct 31 04:53:33.984980 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:33.984948 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/481beb7c-221e-430f-800c-ad11adc67b4f-maas-api-openshift-ca" pod "481beb7c-221e-430f-800c-ad11adc67b4f" (UID: "481beb7c-221e-430f-800c-ad11adc67b4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:53:33.985567 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:33.985536 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/481beb7c-221e-430f-800c-ad11adc67b4f-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:53:34.001402 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:33.989773 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/481beb7c-221e-430f-800c-ad11adc67b4f-kube-api-access-dcxwb" pod "481beb7c-221e-430f-800c-ad11adc67b4f" (UID: "481beb7c-221e-430f-800c-ad11adc67b4f"). InnerVolumeSpecName "kube-api-access-dcxwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:53:34.001402 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:33.994318 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/481beb7c-221e-430f-800c-ad11adc67b4f-maas-api-xks-ca" pod "481beb7c-221e-430f-800c-ad11adc67b4f" (UID: "481beb7c-221e-430f-800c-ad11adc67b4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:53:34.086821 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:34.086764 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/481beb7c-221e-430f-800c-ad11adc67b4f-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:53:34.086821 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:34.086810 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcxwb\" (UniqueName: \"kubernetes.io/projected/481beb7c-221e-430f-800c-ad11adc67b4f-kube-api-access-dcxwb\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:53:34.742355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:34.741678 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-340168f9-8mnh2" event={"ID":"481beb7c-221e-430f-800c-ad11adc67b4f","Type":"ContainerDied","Data":"c00c007477100fe35a7d0d2a01081acf633153e05ecf77d19c000e37d913fe18"} Oct 31 04:53:34.742355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:34.741756 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c00c007477100fe35a7d0d2a01081acf633153e05ecf77d19c000e37d913fe18" Oct 31 04:53:34.742355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:34.741836 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-340168f9-8mnh2" Oct 31 04:53:35.219239 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:35.217565 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t"] Oct 31 04:53:35.220475 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:35.220448 5019 cpu_manager.go:414] "containerMap: removing container" podUID="481beb7c-221e-430f-800c-ad11adc67b4f" containerName="revoke-keys" Oct 31 04:53:35.220475 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:35.220472 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="481beb7c-221e-430f-800c-ad11adc67b4f" containerName="revoke-keys" Oct 31 04:53:35.251887 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:35.248453 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="481beb7c-221e-430f-800c-ad11adc67b4f" containerName="revoke-keys" Oct 31 04:53:35.253972 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:35.253929 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t" Oct 31 04:53:35.259667 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:35.257869 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t"] Oct 31 04:53:35.283633 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:35.283595 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-670795-openshift-default-dockercfg-r4799\"" Oct 31 04:53:35.309134 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:35.309071 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51ef5aae-6bf0-4340-9e57-51b388049f40-istio-token\") pod \"e2e-inf-a-670795-openshift-default-764f54c759-gtm6t\" (UID: \"51ef5aae-6bf0-4340-9e57-51b388049f40\") " pod="openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t" Oct 31 04:53:35.309716 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:35.309681 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5qbx\" (UniqueName: \"kubernetes.io/projected/51ef5aae-6bf0-4340-9e57-51b388049f40-kube-api-access-p5qbx\") pod \"e2e-inf-a-670795-openshift-default-764f54c759-gtm6t\" (UID: \"51ef5aae-6bf0-4340-9e57-51b388049f40\") " pod="openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t" Oct 31 04:53:35.310025 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:35.310001 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51ef5aae-6bf0-4340-9e57-51b388049f40-workload-certs\") pod \"e2e-inf-a-670795-openshift-default-764f54c759-gtm6t\" (UID: \"51ef5aae-6bf0-4340-9e57-51b388049f40\") " pod="openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t" Oct 31 04:53:35.310276 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:35.310256 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51ef5aae-6bf0-4340-9e57-51b388049f40-workload-socket\") pod \"e2e-inf-a-670795-openshift-default-764f54c759-gtm6t\" (UID: \"51ef5aae-6bf0-4340-9e57-51b388049f40\") " pod="openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t" Oct 31 04:53:35.310546 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:35.310514 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51ef5aae-6bf0-4340-9e57-51b388049f40-istiod-ca-cert\") pod \"e2e-inf-a-670795-openshift-default-764f54c759-gtm6t\" (UID: \"51ef5aae-6bf0-4340-9e57-51b388049f40\") " pod="openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t" Oct 31 04:53:35.310647 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:35.310620 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51ef5aae-6bf0-4340-9e57-51b388049f40-istio-data\") pod \"e2e-inf-a-670795-openshift-default-764f54c759-gtm6t\" (UID: \"51ef5aae-6bf0-4340-9e57-51b388049f40\") " pod="openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t" Oct 31 04:53:35.310724 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:35.310647 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51ef5aae-6bf0-4340-9e57-51b388049f40-istio-podinfo\") pod \"e2e-inf-a-670795-openshift-default-764f54c759-gtm6t\" (UID: \"51ef5aae-6bf0-4340-9e57-51b388049f40\") " pod="openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t" Oct 31 04:53:35.310724 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:35.310693 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51ef5aae-6bf0-4340-9e57-51b388049f40-istio-envoy\") pod \"e2e-inf-a-670795-openshift-default-764f54c759-gtm6t\" (UID: \"51ef5aae-6bf0-4340-9e57-51b388049f40\") " pod="openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t" Oct 31 04:53:35.310839 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:35.310752 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51ef5aae-6bf0-4340-9e57-51b388049f40-credential-socket\") pod \"e2e-inf-a-670795-openshift-default-764f54c759-gtm6t\" (UID: \"51ef5aae-6bf0-4340-9e57-51b388049f40\") " pod="openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t" Oct 31 04:53:35.702746 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:35.701858 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51ef5aae-6bf0-4340-9e57-51b388049f40-istio-token\") pod \"e2e-inf-a-670795-openshift-default-764f54c759-gtm6t\" (UID: \"51ef5aae-6bf0-4340-9e57-51b388049f40\") " pod="openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t" Oct 31 04:53:35.702746 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:35.702054 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5qbx\" (UniqueName: \"kubernetes.io/projected/51ef5aae-6bf0-4340-9e57-51b388049f40-kube-api-access-p5qbx\") pod \"e2e-inf-a-670795-openshift-default-764f54c759-gtm6t\" (UID: \"51ef5aae-6bf0-4340-9e57-51b388049f40\") " pod="openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t" Oct 31 04:53:35.702746 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:35.702116 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51ef5aae-6bf0-4340-9e57-51b388049f40-workload-certs\") pod \"e2e-inf-a-670795-openshift-default-764f54c759-gtm6t\" (UID: \"51ef5aae-6bf0-4340-9e57-51b388049f40\") " pod="openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t" Oct 31 04:53:35.702746 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:35.702166 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51ef5aae-6bf0-4340-9e57-51b388049f40-workload-socket\") pod \"e2e-inf-a-670795-openshift-default-764f54c759-gtm6t\" (UID: \"51ef5aae-6bf0-4340-9e57-51b388049f40\") " pod="openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t" Oct 31 04:53:35.702746 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:35.702207 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51ef5aae-6bf0-4340-9e57-51b388049f40-istiod-ca-cert\") pod \"e2e-inf-a-670795-openshift-default-764f54c759-gtm6t\" (UID: \"51ef5aae-6bf0-4340-9e57-51b388049f40\") " pod="openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t" Oct 31 04:53:35.702746 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:35.702260 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51ef5aae-6bf0-4340-9e57-51b388049f40-istio-data\") pod \"e2e-inf-a-670795-openshift-default-764f54c759-gtm6t\" (UID: \"51ef5aae-6bf0-4340-9e57-51b388049f40\") " pod="openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t" Oct 31 04:53:35.702746 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:35.702281 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51ef5aae-6bf0-4340-9e57-51b388049f40-istio-podinfo\") pod \"e2e-inf-a-670795-openshift-default-764f54c759-gtm6t\" (UID: \"51ef5aae-6bf0-4340-9e57-51b388049f40\") " pod="openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t" Oct 31 04:53:35.702746 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:35.702318 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51ef5aae-6bf0-4340-9e57-51b388049f40-istio-envoy\") pod \"e2e-inf-a-670795-openshift-default-764f54c759-gtm6t\" (UID: \"51ef5aae-6bf0-4340-9e57-51b388049f40\") " pod="openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t" Oct 31 04:53:35.702746 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:35.702452 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51ef5aae-6bf0-4340-9e57-51b388049f40-credential-socket\") pod \"e2e-inf-a-670795-openshift-default-764f54c759-gtm6t\" (UID: \"51ef5aae-6bf0-4340-9e57-51b388049f40\") " pod="openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t" Oct 31 04:53:35.703421 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:35.702850 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51ef5aae-6bf0-4340-9e57-51b388049f40-workload-socket\") pod \"e2e-inf-a-670795-openshift-default-764f54c759-gtm6t\" (UID: \"51ef5aae-6bf0-4340-9e57-51b388049f40\") " pod="openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t" Oct 31 04:53:35.703421 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:35.703286 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51ef5aae-6bf0-4340-9e57-51b388049f40-credential-socket\") pod \"e2e-inf-a-670795-openshift-default-764f54c759-gtm6t\" (UID: \"51ef5aae-6bf0-4340-9e57-51b388049f40\") " pod="openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t" Oct 31 04:53:35.703807 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:35.703768 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51ef5aae-6bf0-4340-9e57-51b388049f40-workload-certs\") pod \"e2e-inf-a-670795-openshift-default-764f54c759-gtm6t\" (UID: \"51ef5aae-6bf0-4340-9e57-51b388049f40\") " pod="openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t" Oct 31 04:53:35.704093 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:35.704069 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51ef5aae-6bf0-4340-9e57-51b388049f40-istio-data\") pod \"e2e-inf-a-670795-openshift-default-764f54c759-gtm6t\" (UID: \"51ef5aae-6bf0-4340-9e57-51b388049f40\") " pod="openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t" Oct 31 04:53:35.704184 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:35.704107 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51ef5aae-6bf0-4340-9e57-51b388049f40-istiod-ca-cert\") pod \"e2e-inf-a-670795-openshift-default-764f54c759-gtm6t\" (UID: \"51ef5aae-6bf0-4340-9e57-51b388049f40\") " pod="openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t" Oct 31 04:53:35.735359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:35.710089 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51ef5aae-6bf0-4340-9e57-51b388049f40-istio-podinfo\") pod \"e2e-inf-a-670795-openshift-default-764f54c759-gtm6t\" (UID: \"51ef5aae-6bf0-4340-9e57-51b388049f40\") " pod="openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t" Oct 31 04:53:35.735359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:35.723048 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51ef5aae-6bf0-4340-9e57-51b388049f40-istio-envoy\") pod \"e2e-inf-a-670795-openshift-default-764f54c759-gtm6t\" (UID: \"51ef5aae-6bf0-4340-9e57-51b388049f40\") " pod="openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t" Oct 31 04:53:35.743726 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:35.741370 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5qbx\" (UniqueName: \"kubernetes.io/projected/51ef5aae-6bf0-4340-9e57-51b388049f40-kube-api-access-p5qbx\") pod \"e2e-inf-a-670795-openshift-default-764f54c759-gtm6t\" (UID: \"51ef5aae-6bf0-4340-9e57-51b388049f40\") " pod="openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t" Oct 31 04:53:35.743726 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:35.743055 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51ef5aae-6bf0-4340-9e57-51b388049f40-istio-token\") pod \"e2e-inf-a-670795-openshift-default-764f54c759-gtm6t\" (UID: \"51ef5aae-6bf0-4340-9e57-51b388049f40\") " pod="openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t" Oct 31 04:53:35.886879 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:35.886841 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t" Oct 31 04:53:36.037407 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:36.037359 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e38e6ad-n8vmc"] Oct 31 04:53:36.045843 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:36.045798 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e38e6ad-n8vmc"] Oct 31 04:53:36.214850 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:53:36.214812 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51ef5aae_6bf0_4340_9e57_51b388049f40.slice/crio-b367595ed17f8672dc48ec3501380c7e6e9a612c6f2030b6d0a2dbc8209a40d7 WatchSource:0}: Error finding container b367595ed17f8672dc48ec3501380c7e6e9a612c6f2030b6d0a2dbc8209a40d7: Status 404 returned error can't find the container with id b367595ed17f8672dc48ec3501380c7e6e9a612c6f2030b6d0a2dbc8209a40d7 Oct 31 04:53:36.215888 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:36.215862 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t"] Oct 31 04:53:36.218541 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:36.218494 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:53:36.218659 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:36.218599 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:53:36.218659 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:36.218641 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:53:36.768171 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:36.768122 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t" event={"ID":"51ef5aae-6bf0-4340-9e57-51b388049f40","Type":"ContainerStarted","Data":"9e0f5aa541516eb722af32733f2ec9f5f8818409039487c81c42b5a180b7f265"} Oct 31 04:53:36.768647 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:36.768199 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t" event={"ID":"51ef5aae-6bf0-4340-9e57-51b388049f40","Type":"ContainerStarted","Data":"b367595ed17f8672dc48ec3501380c7e6e9a612c6f2030b6d0a2dbc8209a40d7"} Oct 31 04:53:36.818923 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:36.818860 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t" podStartSLOduration=1.818841991 podStartE2EDuration="1.818841991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:53:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:53:36.815062162 +0000 UTC m=+2971.571400203" watchObservedRunningTime="2026-10-31 04:53:36.818841991 +0000 UTC m=+2971.575180030" Oct 31 04:53:36.887725 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:36.887681 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t" Oct 31 04:53:37.031099 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:37.031001 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-236b5948-rpkxk"] Oct 31 04:53:37.036543 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:37.036504 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-236b5948-rpkxk"] Oct 31 04:53:37.414806 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:37.414717 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f2e1623-8b13-486d-87b7-e1c206687932" path="/var/lib/kubelet/pods/1f2e1623-8b13-486d-87b7-e1c206687932/volumes" Oct 31 04:53:37.415345 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:37.415314 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be76e744-5738-416c-8125-feb6a1cac37f" path="/var/lib/kubelet/pods/be76e744-5738-416c-8125-feb6a1cac37f/volumes" Oct 31 04:53:37.895478 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:37.895443 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t" Oct 31 04:53:38.790949 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:38.790901 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t" Oct 31 04:53:38.792231 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:53:38.792202 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t" Oct 31 04:53:52.640638 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:53:52.640589 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:54:04.961169 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:04.961102 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl"] Oct 31 04:54:04.977238 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:04.977194 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl" Oct 31 04:54:05.066356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:05.057692 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qslv\" (UniqueName: \"kubernetes.io/projected/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-kube-api-access-7qslv\") pod \"e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl\" (UID: \"1d46e1a8-ef71-4bbf-afec-a074d0fcfe05\") " pod="openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl" Oct 31 04:54:05.066356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:05.057760 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-istio-token\") pod \"e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl\" (UID: \"1d46e1a8-ef71-4bbf-afec-a074d0fcfe05\") " pod="openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl" Oct 31 04:54:05.066356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:05.057874 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-istiod-ca-cert\") pod \"e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl\" (UID: \"1d46e1a8-ef71-4bbf-afec-a074d0fcfe05\") " pod="openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl" Oct 31 04:54:05.066356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:05.057943 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-workload-socket\") pod \"e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl\" (UID: \"1d46e1a8-ef71-4bbf-afec-a074d0fcfe05\") " pod="openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl" Oct 31 04:54:05.066356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:05.057968 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-istio-podinfo\") pod \"e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl\" (UID: \"1d46e1a8-ef71-4bbf-afec-a074d0fcfe05\") " pod="openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl" Oct 31 04:54:05.066356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:05.057998 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-credential-socket\") pod \"e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl\" (UID: \"1d46e1a8-ef71-4bbf-afec-a074d0fcfe05\") " pod="openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl" Oct 31 04:54:05.066356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:05.058082 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-istio-envoy\") pod \"e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl\" (UID: \"1d46e1a8-ef71-4bbf-afec-a074d0fcfe05\") " pod="openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl" Oct 31 04:54:05.066356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:05.058152 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-workload-certs\") pod \"e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl\" (UID: \"1d46e1a8-ef71-4bbf-afec-a074d0fcfe05\") " pod="openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl" Oct 31 04:54:05.066356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:05.058206 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-istio-data\") pod \"e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl\" (UID: \"1d46e1a8-ef71-4bbf-afec-a074d0fcfe05\") " pod="openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl" Oct 31 04:54:05.098922 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:05.098891 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-171edd-openshift-default-dockercfg-84kd6\"" Oct 31 04:54:05.135375 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:05.127962 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl"] Oct 31 04:54:05.231167 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:05.229164 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-workload-socket\") pod \"e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl\" (UID: \"1d46e1a8-ef71-4bbf-afec-a074d0fcfe05\") " pod="openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl" Oct 31 04:54:05.231167 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:05.229211 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-istio-podinfo\") pod \"e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl\" (UID: \"1d46e1a8-ef71-4bbf-afec-a074d0fcfe05\") " pod="openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl" Oct 31 04:54:05.231167 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:05.229245 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-credential-socket\") pod \"e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl\" (UID: \"1d46e1a8-ef71-4bbf-afec-a074d0fcfe05\") " pod="openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl" Oct 31 04:54:05.231167 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:05.229417 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-istio-envoy\") pod \"e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl\" (UID: \"1d46e1a8-ef71-4bbf-afec-a074d0fcfe05\") " pod="openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl" Oct 31 04:54:05.231167 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:05.229495 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-workload-certs\") pod \"e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl\" (UID: \"1d46e1a8-ef71-4bbf-afec-a074d0fcfe05\") " pod="openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl" Oct 31 04:54:05.231167 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:05.229552 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-istio-data\") pod \"e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl\" (UID: \"1d46e1a8-ef71-4bbf-afec-a074d0fcfe05\") " pod="openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl" Oct 31 04:54:05.231167 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:05.229810 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qslv\" (UniqueName: \"kubernetes.io/projected/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-kube-api-access-7qslv\") pod \"e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl\" (UID: \"1d46e1a8-ef71-4bbf-afec-a074d0fcfe05\") " pod="openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl" Oct 31 04:54:05.231167 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:05.229844 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-istio-token\") pod \"e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl\" (UID: \"1d46e1a8-ef71-4bbf-afec-a074d0fcfe05\") " pod="openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl" Oct 31 04:54:05.231167 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:05.229972 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-istiod-ca-cert\") pod \"e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl\" (UID: \"1d46e1a8-ef71-4bbf-afec-a074d0fcfe05\") " pod="openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl" Oct 31 04:54:05.231167 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:05.230862 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-istiod-ca-cert\") pod \"e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl\" (UID: \"1d46e1a8-ef71-4bbf-afec-a074d0fcfe05\") " pod="openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl" Oct 31 04:54:05.231167 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:05.231158 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-workload-socket\") pod \"e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl\" (UID: \"1d46e1a8-ef71-4bbf-afec-a074d0fcfe05\") " pod="openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl" Oct 31 04:54:05.243393 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:05.243354 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-workload-certs\") pod \"e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl\" (UID: \"1d46e1a8-ef71-4bbf-afec-a074d0fcfe05\") " pod="openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl" Oct 31 04:54:05.245524 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:05.245486 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-credential-socket\") pod \"e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl\" (UID: \"1d46e1a8-ef71-4bbf-afec-a074d0fcfe05\") " pod="openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl" Oct 31 04:54:05.248857 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:05.248236 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-istio-data\") pod \"e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl\" (UID: \"1d46e1a8-ef71-4bbf-afec-a074d0fcfe05\") " pod="openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl" Oct 31 04:54:05.301130 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:05.299427 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-istio-envoy\") pod \"e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl\" (UID: \"1d46e1a8-ef71-4bbf-afec-a074d0fcfe05\") " pod="openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl" Oct 31 04:54:05.354388 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:05.353133 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qslv\" (UniqueName: \"kubernetes.io/projected/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-kube-api-access-7qslv\") pod \"e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl\" (UID: \"1d46e1a8-ef71-4bbf-afec-a074d0fcfe05\") " pod="openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl" Oct 31 04:54:05.354388 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:05.353151 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-istio-podinfo\") pod \"e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl\" (UID: \"1d46e1a8-ef71-4bbf-afec-a074d0fcfe05\") " pod="openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl" Oct 31 04:54:05.354388 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:05.353566 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-istio-token\") pod \"e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl\" (UID: \"1d46e1a8-ef71-4bbf-afec-a074d0fcfe05\") " pod="openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl" Oct 31 04:54:05.414425 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:05.414382 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl" Oct 31 04:54:05.751731 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:54:05.751683 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d46e1a8_ef71_4bbf_afec_a074d0fcfe05.slice/crio-3edb970d19488826c187d891451b3ae8802c13c366b054fd67f3a308345822f4 WatchSource:0}: Error finding container 3edb970d19488826c187d891451b3ae8802c13c366b054fd67f3a308345822f4: Status 404 returned error can't find the container with id 3edb970d19488826c187d891451b3ae8802c13c366b054fd67f3a308345822f4 Oct 31 04:54:05.753000 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:05.752953 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl"] Oct 31 04:54:05.756090 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:05.756041 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:54:05.756363 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:05.756313 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:54:05.756479 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:05.756391 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:54:06.749770 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:06.749726 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl" event={"ID":"1d46e1a8-ef71-4bbf-afec-a074d0fcfe05","Type":"ContainerStarted","Data":"91b8e9f8d818fa4ee6637299edea1421c8b1ccf53c85215faee87a40c7e0d95e"} Oct 31 04:54:06.750219 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:06.749788 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl" event={"ID":"1d46e1a8-ef71-4bbf-afec-a074d0fcfe05","Type":"ContainerStarted","Data":"3edb970d19488826c187d891451b3ae8802c13c366b054fd67f3a308345822f4"} Oct 31 04:54:06.813466 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:06.813383 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl" podStartSLOduration=2.813359322 podStartE2EDuration="2.813359322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:54:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:54:06.810024817 +0000 UTC m=+3001.566362857" watchObservedRunningTime="2026-10-31 04:54:06.813359322 +0000 UTC m=+3001.569697363" Oct 31 04:54:07.414855 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:07.414809 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl" Oct 31 04:54:07.422405 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:07.422369 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl" Oct 31 04:54:07.815635 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:07.815502 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl" Oct 31 04:54:07.821928 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:07.820861 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl" Oct 31 04:54:09.234022 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:09.233985 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s88kp_3945ff65-e877-4372-9334-3968c68b7295/console-operator/2.log" Oct 31 04:54:09.237544 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:09.237491 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s88kp_3945ff65-e877-4372-9334-3968c68b7295/console-operator/2.log" Oct 31 04:54:11.071956 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:11.071913 5019 scope.go:122] "RemoveContainer" containerID="bec1244edceaed40eda78054330cac8bf6a2bb0f14d2b8b05b36759f2a73b94f" Oct 31 04:54:11.101914 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:11.101876 5019 scope.go:122] "RemoveContainer" containerID="71dc248c40833079a458d00ae8b39197d140db07289b8d0888a74521e012d10c" Oct 31 04:54:11.118264 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:11.118233 5019 scope.go:122] "RemoveContainer" containerID="642ac8ae05c2d248bf950a6d69458a7ec71848118be2a82f47c4b66faa134613" Oct 31 04:54:11.150592 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:11.137534 5019 scope.go:122] "RemoveContainer" containerID="20d8a7a15abeca2fac23bb5cf5e7c7a2f4ff001e9eae76e0a529a0f0d7491f36" Oct 31 04:54:11.546776 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:11.546699 5019 scope.go:122] "RemoveContainer" containerID="f0e60eca91549acd179bc444543229deb6b81dbae6b68247a4e5bc307d150fd4" Oct 31 04:54:11.565522 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:11.565499 5019 scope.go:122] "RemoveContainer" containerID="a2f934ca7ab02202e9c1290f1ce40075d7e960f2274cdadafd4a9d82c8a7a2ee" Oct 31 04:54:11.582319 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:11.582287 5019 scope.go:122] "RemoveContainer" containerID="d20d0c6edf652f3baf63b7cfa9a21fbb02d8f79ba4350dc6b38da4cf0a186a32" Oct 31 04:54:11.594900 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:11.594871 5019 scope.go:122] "RemoveContainer" containerID="14243c76407f35cfa75c8b2509a61e8106d323de7946353d6ae89461636a8b06" Oct 31 04:54:11.610672 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:11.610643 5019 scope.go:122] "RemoveContainer" containerID="a12f033cee8fac7725a3064944db0adf96febccc80a81b0e5fd2658f7b29bceb" Oct 31 04:54:15.321223 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:15.320461 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" podUID="b4ced3ab-41e4-4c54-93eb-95eddaae992d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 04:54:15.363540 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:15.363288 5019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" podUID="b4ced3ab-41e4-4c54-93eb-95eddaae992d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:54:15.601457 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:15.599460 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 04:54:16.179631 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:16.178429 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" podUID="e0f4b8c2-14f4-4052-bf54-dd65bf59e413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:54:16.513358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:16.512446 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" podUID="643433d8-418e-4102-8bc3-cd544932614a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:54:16.529596 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:16.521424 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:54:16.600545 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:16.600486 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:54:16.602094 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:16.602059 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" Oct 31 04:54:16.646479 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:16.646428 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" podUID="8e2c7041-ad52-49f4-8b55-d16691f4938f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:54:17.407354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:17.403205 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" Oct 31 04:54:21.283517 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:21.283463 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ba6669d-xfmmk"] Oct 31 04:54:21.292693 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:21.292652 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ba6669d-xfmmk" Oct 31 04:54:21.308195 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:21.308160 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ba6669d-xfmmk"] Oct 31 04:54:21.312653 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:21.312622 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:54:21.312859 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:21.312774 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:54:21.312859 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:21.312829 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:54:21.313002 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:21.312906 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:54:21.386688 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:21.386622 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5crpm\" (UniqueName: \"kubernetes.io/projected/457a668d-3818-4eaa-b1a2-63a9972d2de8-kube-api-access-5crpm\") pod \"maas-api-revoke-sub-6ba6669d-xfmmk\" (UID: \"457a668d-3818-4eaa-b1a2-63a9972d2de8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ba6669d-xfmmk" Oct 31 04:54:21.386981 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:21.386931 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/457a668d-3818-4eaa-b1a2-63a9972d2de8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ba6669d-xfmmk\" (UID: \"457a668d-3818-4eaa-b1a2-63a9972d2de8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ba6669d-xfmmk" Oct 31 04:54:21.387220 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:21.387193 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/457a668d-3818-4eaa-b1a2-63a9972d2de8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ba6669d-xfmmk\" (UID: \"457a668d-3818-4eaa-b1a2-63a9972d2de8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ba6669d-xfmmk" Oct 31 04:54:21.489308 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:21.489268 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5crpm\" (UniqueName: \"kubernetes.io/projected/457a668d-3818-4eaa-b1a2-63a9972d2de8-kube-api-access-5crpm\") pod \"maas-api-revoke-sub-6ba6669d-xfmmk\" (UID: \"457a668d-3818-4eaa-b1a2-63a9972d2de8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ba6669d-xfmmk" Oct 31 04:54:21.489535 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:21.489475 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/457a668d-3818-4eaa-b1a2-63a9972d2de8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ba6669d-xfmmk\" (UID: \"457a668d-3818-4eaa-b1a2-63a9972d2de8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ba6669d-xfmmk" Oct 31 04:54:21.489732 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:21.489710 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/457a668d-3818-4eaa-b1a2-63a9972d2de8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ba6669d-xfmmk\" (UID: \"457a668d-3818-4eaa-b1a2-63a9972d2de8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ba6669d-xfmmk" Oct 31 04:54:21.490198 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:21.490174 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/457a668d-3818-4eaa-b1a2-63a9972d2de8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ba6669d-xfmmk\" (UID: \"457a668d-3818-4eaa-b1a2-63a9972d2de8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ba6669d-xfmmk" Oct 31 04:54:21.492379 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:21.492350 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/457a668d-3818-4eaa-b1a2-63a9972d2de8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ba6669d-xfmmk\" (UID: \"457a668d-3818-4eaa-b1a2-63a9972d2de8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ba6669d-xfmmk" Oct 31 04:54:21.515985 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:21.515950 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5crpm\" (UniqueName: \"kubernetes.io/projected/457a668d-3818-4eaa-b1a2-63a9972d2de8-kube-api-access-5crpm\") pod \"maas-api-revoke-sub-6ba6669d-xfmmk\" (UID: \"457a668d-3818-4eaa-b1a2-63a9972d2de8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ba6669d-xfmmk" Oct 31 04:54:21.619832 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:21.619725 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ba6669d-xfmmk" Oct 31 04:54:21.956274 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:21.955842 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ba6669d-xfmmk"] Oct 31 04:54:22.226721 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:22.226676 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ba6669d-xfmmk" event={"ID":"457a668d-3818-4eaa-b1a2-63a9972d2de8","Type":"ContainerStarted","Data":"3ed993681433655087ae93ce64ecd67ac100cbaf1ab5442fbd4b5977225bcf47"} Oct 31 04:54:22.408494 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:54:22.408444 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:54:23.317447 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:23.315750 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ba6669d-xfmmk" event={"ID":"457a668d-3818-4eaa-b1a2-63a9972d2de8","Type":"ContainerStarted","Data":"3b35f2ff5308348359210086876b0da580be0759c2472f3b013a1ec806ff6a38"} Oct 31 04:54:23.396317 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:23.385448 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ba6669d-xfmmk" podStartSLOduration=2.385425089 podStartE2EDuration="2.385425089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:54:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:54:23.378765533 +0000 UTC m=+3018.135103612" watchObservedRunningTime="2026-10-31 04:54:23.385425089 +0000 UTC m=+3018.141763130" Oct 31 04:54:23.499297 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:54:23.499238 5019 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod457a668d_3818_4eaa_b1a2_63a9972d2de8.slice/crio-3b35f2ff5308348359210086876b0da580be0759c2472f3b013a1ec806ff6a38.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod457a668d_3818_4eaa_b1a2_63a9972d2de8.slice/crio-conmon-3b35f2ff5308348359210086876b0da580be0759c2472f3b013a1ec806ff6a38.scope\": RecentStats: unable to find data in memory cache]" Oct 31 04:54:24.328049 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:24.328007 5019 generic.go:356] "Generic (PLEG): container finished" podID="457a668d-3818-4eaa-b1a2-63a9972d2de8" containerID="3b35f2ff5308348359210086876b0da580be0759c2472f3b013a1ec806ff6a38" exitCode=0 Oct 31 04:54:24.328252 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:24.328092 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ba6669d-xfmmk" event={"ID":"457a668d-3818-4eaa-b1a2-63a9972d2de8","Type":"ContainerDied","Data":"3b35f2ff5308348359210086876b0da580be0759c2472f3b013a1ec806ff6a38"} Oct 31 04:54:25.583993 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:25.583960 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ba6669d-xfmmk" Oct 31 04:54:25.631200 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:25.631096 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/457a668d-3818-4eaa-b1a2-63a9972d2de8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/457a668d-3818-4eaa-b1a2-63a9972d2de8-maas-api-openshift-ca\") pod \"457a668d-3818-4eaa-b1a2-63a9972d2de8\" (UID: \"457a668d-3818-4eaa-b1a2-63a9972d2de8\") " Oct 31 04:54:25.631200 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:25.631159 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/457a668d-3818-4eaa-b1a2-63a9972d2de8-kube-api-access-5crpm\" (UniqueName: \"kubernetes.io/projected/457a668d-3818-4eaa-b1a2-63a9972d2de8-kube-api-access-5crpm\") pod \"457a668d-3818-4eaa-b1a2-63a9972d2de8\" (UID: \"457a668d-3818-4eaa-b1a2-63a9972d2de8\") " Oct 31 04:54:25.631512 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:25.631347 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/457a668d-3818-4eaa-b1a2-63a9972d2de8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/457a668d-3818-4eaa-b1a2-63a9972d2de8-maas-api-xks-ca\") pod \"457a668d-3818-4eaa-b1a2-63a9972d2de8\" (UID: \"457a668d-3818-4eaa-b1a2-63a9972d2de8\") " Oct 31 04:54:25.631624 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:25.631595 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/457a668d-3818-4eaa-b1a2-63a9972d2de8-maas-api-openshift-ca" pod "457a668d-3818-4eaa-b1a2-63a9972d2de8" (UID: "457a668d-3818-4eaa-b1a2-63a9972d2de8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:54:25.632077 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:25.632056 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/457a668d-3818-4eaa-b1a2-63a9972d2de8-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:54:25.633678 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:25.633653 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/457a668d-3818-4eaa-b1a2-63a9972d2de8-maas-api-xks-ca" pod "457a668d-3818-4eaa-b1a2-63a9972d2de8" (UID: "457a668d-3818-4eaa-b1a2-63a9972d2de8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:54:25.633826 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:25.633803 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/457a668d-3818-4eaa-b1a2-63a9972d2de8-kube-api-access-5crpm" pod "457a668d-3818-4eaa-b1a2-63a9972d2de8" (UID: "457a668d-3818-4eaa-b1a2-63a9972d2de8"). InnerVolumeSpecName "kube-api-access-5crpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:54:25.733444 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:25.733399 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5crpm\" (UniqueName: \"kubernetes.io/projected/457a668d-3818-4eaa-b1a2-63a9972d2de8-kube-api-access-5crpm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:54:25.733444 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:25.733435 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/457a668d-3818-4eaa-b1a2-63a9972d2de8-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:54:26.350727 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:26.350689 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ba6669d-xfmmk" Oct 31 04:54:26.350727 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:26.350710 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ba6669d-xfmmk" event={"ID":"457a668d-3818-4eaa-b1a2-63a9972d2de8","Type":"ContainerDied","Data":"3ed993681433655087ae93ce64ecd67ac100cbaf1ab5442fbd4b5977225bcf47"} Oct 31 04:54:26.351014 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:26.350767 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ed993681433655087ae93ce64ecd67ac100cbaf1ab5442fbd4b5977225bcf47" Oct 31 04:54:31.159046 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:31.159005 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55d0e8d0-cmmwx"] Oct 31 04:54:31.166376 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:31.161154 5019 cpu_manager.go:414] "containerMap: removing container" podUID="457a668d-3818-4eaa-b1a2-63a9972d2de8" containerName="revoke-keys" Oct 31 04:54:31.166733 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:31.166692 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="457a668d-3818-4eaa-b1a2-63a9972d2de8" containerName="revoke-keys" Oct 31 04:54:31.167787 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:31.167748 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="457a668d-3818-4eaa-b1a2-63a9972d2de8" containerName="revoke-keys" Oct 31 04:54:31.213117 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:31.213076 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d0e8d0-cmmwx" Oct 31 04:54:31.248369 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:31.244783 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:54:31.272644 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:31.272603 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:54:31.273205 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:31.273174 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:54:31.273494 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:31.273474 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:54:31.583774 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:31.583736 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/180073ab-a2eb-47ed-a8bf-9adf41d715bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55d0e8d0-cmmwx\" (UID: \"180073ab-a2eb-47ed-a8bf-9adf41d715bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d0e8d0-cmmwx" Oct 31 04:54:31.584113 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:31.584077 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpn68\" (UniqueName: \"kubernetes.io/projected/180073ab-a2eb-47ed-a8bf-9adf41d715bd-kube-api-access-mpn68\") pod \"maas-api-revoke-sub-55d0e8d0-cmmwx\" (UID: \"180073ab-a2eb-47ed-a8bf-9adf41d715bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d0e8d0-cmmwx" Oct 31 04:54:31.584211 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:31.584114 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/180073ab-a2eb-47ed-a8bf-9adf41d715bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55d0e8d0-cmmwx\" (UID: \"180073ab-a2eb-47ed-a8bf-9adf41d715bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d0e8d0-cmmwx" Oct 31 04:54:31.642860 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:31.642817 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55d0e8d0-cmmwx"] Oct 31 04:54:31.686805 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:31.686759 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpn68\" (UniqueName: \"kubernetes.io/projected/180073ab-a2eb-47ed-a8bf-9adf41d715bd-kube-api-access-mpn68\") pod \"maas-api-revoke-sub-55d0e8d0-cmmwx\" (UID: \"180073ab-a2eb-47ed-a8bf-9adf41d715bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d0e8d0-cmmwx" Oct 31 04:54:31.687050 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:31.686814 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/180073ab-a2eb-47ed-a8bf-9adf41d715bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55d0e8d0-cmmwx\" (UID: \"180073ab-a2eb-47ed-a8bf-9adf41d715bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d0e8d0-cmmwx" Oct 31 04:54:31.687050 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:31.686964 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/180073ab-a2eb-47ed-a8bf-9adf41d715bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55d0e8d0-cmmwx\" (UID: \"180073ab-a2eb-47ed-a8bf-9adf41d715bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d0e8d0-cmmwx" Oct 31 04:54:31.687752 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:31.687722 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/180073ab-a2eb-47ed-a8bf-9adf41d715bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55d0e8d0-cmmwx\" (UID: \"180073ab-a2eb-47ed-a8bf-9adf41d715bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d0e8d0-cmmwx" Oct 31 04:54:31.689691 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:31.689655 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/180073ab-a2eb-47ed-a8bf-9adf41d715bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55d0e8d0-cmmwx\" (UID: \"180073ab-a2eb-47ed-a8bf-9adf41d715bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d0e8d0-cmmwx" Oct 31 04:54:31.710541 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:31.710492 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpn68\" (UniqueName: \"kubernetes.io/projected/180073ab-a2eb-47ed-a8bf-9adf41d715bd-kube-api-access-mpn68\") pod \"maas-api-revoke-sub-55d0e8d0-cmmwx\" (UID: \"180073ab-a2eb-47ed-a8bf-9adf41d715bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d0e8d0-cmmwx" Oct 31 04:54:31.954095 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:31.953996 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d0e8d0-cmmwx" Oct 31 04:54:32.117152 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:32.107404 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b281a9df-9nr7h"] Oct 31 04:54:32.161367 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:32.158472 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b281a9df-9nr7h"] Oct 31 04:54:32.161367 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:32.158621 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b281a9df-9nr7h" Oct 31 04:54:32.221376 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:32.220504 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whl66\" (UniqueName: \"kubernetes.io/projected/379bd26a-c57d-4d08-a0c5-2eed8732e363-kube-api-access-whl66\") pod \"maas-api-revoke-sub-b281a9df-9nr7h\" (UID: \"379bd26a-c57d-4d08-a0c5-2eed8732e363\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b281a9df-9nr7h" Oct 31 04:54:32.221376 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:32.220593 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/379bd26a-c57d-4d08-a0c5-2eed8732e363-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b281a9df-9nr7h\" (UID: \"379bd26a-c57d-4d08-a0c5-2eed8732e363\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b281a9df-9nr7h" Oct 31 04:54:32.221376 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:32.220680 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/379bd26a-c57d-4d08-a0c5-2eed8732e363-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b281a9df-9nr7h\" (UID: \"379bd26a-c57d-4d08-a0c5-2eed8732e363\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b281a9df-9nr7h" Oct 31 04:54:32.322274 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:32.322222 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/379bd26a-c57d-4d08-a0c5-2eed8732e363-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b281a9df-9nr7h\" (UID: \"379bd26a-c57d-4d08-a0c5-2eed8732e363\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b281a9df-9nr7h" Oct 31 04:54:32.322556 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:32.322542 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whl66\" (UniqueName: \"kubernetes.io/projected/379bd26a-c57d-4d08-a0c5-2eed8732e363-kube-api-access-whl66\") pod \"maas-api-revoke-sub-b281a9df-9nr7h\" (UID: \"379bd26a-c57d-4d08-a0c5-2eed8732e363\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b281a9df-9nr7h" Oct 31 04:54:32.322622 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:32.322589 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/379bd26a-c57d-4d08-a0c5-2eed8732e363-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b281a9df-9nr7h\" (UID: \"379bd26a-c57d-4d08-a0c5-2eed8732e363\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b281a9df-9nr7h" Oct 31 04:54:32.323201 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:32.323158 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/379bd26a-c57d-4d08-a0c5-2eed8732e363-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b281a9df-9nr7h\" (UID: \"379bd26a-c57d-4d08-a0c5-2eed8732e363\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b281a9df-9nr7h" Oct 31 04:54:32.325404 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:32.325369 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/379bd26a-c57d-4d08-a0c5-2eed8732e363-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b281a9df-9nr7h\" (UID: \"379bd26a-c57d-4d08-a0c5-2eed8732e363\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b281a9df-9nr7h" Oct 31 04:54:32.346122 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:32.346085 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whl66\" (UniqueName: \"kubernetes.io/projected/379bd26a-c57d-4d08-a0c5-2eed8732e363-kube-api-access-whl66\") pod \"maas-api-revoke-sub-b281a9df-9nr7h\" (UID: \"379bd26a-c57d-4d08-a0c5-2eed8732e363\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b281a9df-9nr7h" Oct 31 04:54:32.511284 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:54:32.511237 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod180073ab_a2eb_47ed_a8bf_9adf41d715bd.slice/crio-a93b5fda027786ba9ef51f9f7e9271b0627386b1633492875d263a9fc74dc945 WatchSource:0}: Error finding container a93b5fda027786ba9ef51f9f7e9271b0627386b1633492875d263a9fc74dc945: Status 404 returned error can't find the container with id a93b5fda027786ba9ef51f9f7e9271b0627386b1633492875d263a9fc74dc945 Oct 31 04:54:32.511925 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:32.511895 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55d0e8d0-cmmwx"] Oct 31 04:54:32.561566 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:32.561527 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b281a9df-9nr7h" Oct 31 04:54:32.650607 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:32.650562 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d0e8d0-cmmwx" event={"ID":"180073ab-a2eb-47ed-a8bf-9adf41d715bd","Type":"ContainerStarted","Data":"a93b5fda027786ba9ef51f9f7e9271b0627386b1633492875d263a9fc74dc945"} Oct 31 04:54:32.976638 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:54:32.976597 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod379bd26a_c57d_4d08_a0c5_2eed8732e363.slice/crio-e22fa7e64d9e8619e84b8415564ddd13556d343e801c98c05c880d03858647d3 WatchSource:0}: Error finding container e22fa7e64d9e8619e84b8415564ddd13556d343e801c98c05c880d03858647d3: Status 404 returned error can't find the container with id e22fa7e64d9e8619e84b8415564ddd13556d343e801c98c05c880d03858647d3 Oct 31 04:54:32.980437 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:32.980403 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b281a9df-9nr7h"] Oct 31 04:54:33.661834 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:33.661793 5019 generic.go:356] "Generic (PLEG): container finished" podID="180073ab-a2eb-47ed-a8bf-9adf41d715bd" containerID="327503c4b02f478da954bee7d318692ed2e62a3f1ec9c1b24f199f3570112dbc" exitCode=0 Oct 31 04:54:33.662267 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:33.661868 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d0e8d0-cmmwx" event={"ID":"180073ab-a2eb-47ed-a8bf-9adf41d715bd","Type":"ContainerDied","Data":"327503c4b02f478da954bee7d318692ed2e62a3f1ec9c1b24f199f3570112dbc"} Oct 31 04:54:33.663871 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:33.663840 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b281a9df-9nr7h" event={"ID":"379bd26a-c57d-4d08-a0c5-2eed8732e363","Type":"ContainerStarted","Data":"58f210e6f89184b4282e88a0728674db6100cd26261767d13ddb74eb7950bd6d"} Oct 31 04:54:33.664016 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:33.663889 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b281a9df-9nr7h" event={"ID":"379bd26a-c57d-4d08-a0c5-2eed8732e363","Type":"ContainerStarted","Data":"e22fa7e64d9e8619e84b8415564ddd13556d343e801c98c05c880d03858647d3"} Oct 31 04:54:33.749480 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:33.749325 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b281a9df-9nr7h" podStartSLOduration=1.74929548 podStartE2EDuration="1.74929548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:54:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:54:33.741815018 +0000 UTC m=+3028.498153059" watchObservedRunningTime="2026-10-31 04:54:33.74929548 +0000 UTC m=+3028.505633522" Oct 31 04:54:34.692541 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:34.692498 5019 generic.go:356] "Generic (PLEG): container finished" podID="379bd26a-c57d-4d08-a0c5-2eed8732e363" containerID="58f210e6f89184b4282e88a0728674db6100cd26261767d13ddb74eb7950bd6d" exitCode=0 Oct 31 04:54:34.693078 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:34.692594 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b281a9df-9nr7h" event={"ID":"379bd26a-c57d-4d08-a0c5-2eed8732e363","Type":"ContainerDied","Data":"58f210e6f89184b4282e88a0728674db6100cd26261767d13ddb74eb7950bd6d"} Oct 31 04:54:34.896837 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:34.896799 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d0e8d0-cmmwx" Oct 31 04:54:34.978882 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:34.978842 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/180073ab-a2eb-47ed-a8bf-9adf41d715bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/180073ab-a2eb-47ed-a8bf-9adf41d715bd-maas-api-openshift-ca\") pod \"180073ab-a2eb-47ed-a8bf-9adf41d715bd\" (UID: \"180073ab-a2eb-47ed-a8bf-9adf41d715bd\") " Oct 31 04:54:34.979109 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:34.979013 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/180073ab-a2eb-47ed-a8bf-9adf41d715bd-kube-api-access-mpn68\" (UniqueName: \"kubernetes.io/projected/180073ab-a2eb-47ed-a8bf-9adf41d715bd-kube-api-access-mpn68\") pod \"180073ab-a2eb-47ed-a8bf-9adf41d715bd\" (UID: \"180073ab-a2eb-47ed-a8bf-9adf41d715bd\") " Oct 31 04:54:34.979247 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:34.979221 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/180073ab-a2eb-47ed-a8bf-9adf41d715bd-maas-api-openshift-ca" pod "180073ab-a2eb-47ed-a8bf-9adf41d715bd" (UID: "180073ab-a2eb-47ed-a8bf-9adf41d715bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:54:34.979313 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:34.979277 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/180073ab-a2eb-47ed-a8bf-9adf41d715bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/180073ab-a2eb-47ed-a8bf-9adf41d715bd-maas-api-xks-ca\") pod \"180073ab-a2eb-47ed-a8bf-9adf41d715bd\" (UID: \"180073ab-a2eb-47ed-a8bf-9adf41d715bd\") " Oct 31 04:54:34.980063 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:34.980037 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/180073ab-a2eb-47ed-a8bf-9adf41d715bd-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:54:34.981626 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:34.981588 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/180073ab-a2eb-47ed-a8bf-9adf41d715bd-maas-api-xks-ca" pod "180073ab-a2eb-47ed-a8bf-9adf41d715bd" (UID: "180073ab-a2eb-47ed-a8bf-9adf41d715bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:54:34.981921 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:34.981893 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/180073ab-a2eb-47ed-a8bf-9adf41d715bd-kube-api-access-mpn68" pod "180073ab-a2eb-47ed-a8bf-9adf41d715bd" (UID: "180073ab-a2eb-47ed-a8bf-9adf41d715bd"). InnerVolumeSpecName "kube-api-access-mpn68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:54:35.082320 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:35.082267 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpn68\" (UniqueName: \"kubernetes.io/projected/180073ab-a2eb-47ed-a8bf-9adf41d715bd-kube-api-access-mpn68\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:54:35.082320 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:35.082318 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/180073ab-a2eb-47ed-a8bf-9adf41d715bd-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:54:35.708038 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:35.707950 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d0e8d0-cmmwx" Oct 31 04:54:35.708038 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:35.707948 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d0e8d0-cmmwx" event={"ID":"180073ab-a2eb-47ed-a8bf-9adf41d715bd","Type":"ContainerDied","Data":"a93b5fda027786ba9ef51f9f7e9271b0627386b1633492875d263a9fc74dc945"} Oct 31 04:54:35.708588 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:35.708039 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a93b5fda027786ba9ef51f9f7e9271b0627386b1633492875d263a9fc74dc945" Oct 31 04:54:35.893036 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:35.892999 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b281a9df-9nr7h" Oct 31 04:54:35.998975 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:35.998933 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/379bd26a-c57d-4d08-a0c5-2eed8732e363-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/379bd26a-c57d-4d08-a0c5-2eed8732e363-maas-api-openshift-ca\") pod \"379bd26a-c57d-4d08-a0c5-2eed8732e363\" (UID: \"379bd26a-c57d-4d08-a0c5-2eed8732e363\") " Oct 31 04:54:35.999169 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:35.999151 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/379bd26a-c57d-4d08-a0c5-2eed8732e363-kube-api-access-whl66\" (UniqueName: \"kubernetes.io/projected/379bd26a-c57d-4d08-a0c5-2eed8732e363-kube-api-access-whl66\") pod \"379bd26a-c57d-4d08-a0c5-2eed8732e363\" (UID: \"379bd26a-c57d-4d08-a0c5-2eed8732e363\") " Oct 31 04:54:35.999287 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:35.999268 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/379bd26a-c57d-4d08-a0c5-2eed8732e363-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/379bd26a-c57d-4d08-a0c5-2eed8732e363-maas-api-xks-ca\") pod \"379bd26a-c57d-4d08-a0c5-2eed8732e363\" (UID: \"379bd26a-c57d-4d08-a0c5-2eed8732e363\") " Oct 31 04:54:35.999410 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:35.999390 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/379bd26a-c57d-4d08-a0c5-2eed8732e363-maas-api-openshift-ca" pod "379bd26a-c57d-4d08-a0c5-2eed8732e363" (UID: "379bd26a-c57d-4d08-a0c5-2eed8732e363"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:54:36.000117 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:36.000084 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/379bd26a-c57d-4d08-a0c5-2eed8732e363-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:54:36.001458 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:36.001426 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/379bd26a-c57d-4d08-a0c5-2eed8732e363-maas-api-xks-ca" pod "379bd26a-c57d-4d08-a0c5-2eed8732e363" (UID: "379bd26a-c57d-4d08-a0c5-2eed8732e363"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:54:36.001603 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:36.001583 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/379bd26a-c57d-4d08-a0c5-2eed8732e363-kube-api-access-whl66" pod "379bd26a-c57d-4d08-a0c5-2eed8732e363" (UID: "379bd26a-c57d-4d08-a0c5-2eed8732e363"). InnerVolumeSpecName "kube-api-access-whl66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:54:36.102246 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:36.102200 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whl66\" (UniqueName: \"kubernetes.io/projected/379bd26a-c57d-4d08-a0c5-2eed8732e363-kube-api-access-whl66\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:54:36.102246 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:36.102239 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/379bd26a-c57d-4d08-a0c5-2eed8732e363-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:54:36.719262 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:36.719228 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b281a9df-9nr7h" Oct 31 04:54:36.719262 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:36.719256 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b281a9df-9nr7h" event={"ID":"379bd26a-c57d-4d08-a0c5-2eed8732e363","Type":"ContainerDied","Data":"e22fa7e64d9e8619e84b8415564ddd13556d343e801c98c05c880d03858647d3"} Oct 31 04:54:36.719782 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:36.719310 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e22fa7e64d9e8619e84b8415564ddd13556d343e801c98c05c880d03858647d3" Oct 31 04:54:42.056156 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:42.056113 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82a48179-vgtsz"] Oct 31 04:54:42.057209 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:42.057186 5019 cpu_manager.go:414] "containerMap: removing container" podUID="379bd26a-c57d-4d08-a0c5-2eed8732e363" containerName="revoke-keys" Oct 31 04:54:42.057276 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:42.057211 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="379bd26a-c57d-4d08-a0c5-2eed8732e363" containerName="revoke-keys" Oct 31 04:54:42.057276 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:42.057252 5019 cpu_manager.go:414] "containerMap: removing container" podUID="180073ab-a2eb-47ed-a8bf-9adf41d715bd" containerName="revoke-keys" Oct 31 04:54:42.057276 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:42.057259 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="180073ab-a2eb-47ed-a8bf-9adf41d715bd" containerName="revoke-keys" Oct 31 04:54:42.057420 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:42.057410 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="180073ab-a2eb-47ed-a8bf-9adf41d715bd" containerName="revoke-keys" Oct 31 04:54:42.057457 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:42.057429 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="379bd26a-c57d-4d08-a0c5-2eed8732e363" containerName="revoke-keys" Oct 31 04:54:42.061971 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:42.061945 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a48179-vgtsz" Oct 31 04:54:42.081131 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:42.081096 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82a48179-vgtsz"] Oct 31 04:54:42.085583 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:42.085546 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:54:42.085836 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:42.085574 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:54:42.085945 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:42.085613 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:54:42.085945 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:42.085666 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:54:42.127885 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:42.127831 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnb6p\" (UniqueName: \"kubernetes.io/projected/de3b023e-f99d-45f7-ac11-fcbea969d26d-kube-api-access-qnb6p\") pod \"maas-api-revoke-sub-82a48179-vgtsz\" (UID: \"de3b023e-f99d-45f7-ac11-fcbea969d26d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a48179-vgtsz" Oct 31 04:54:42.128118 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:42.127901 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de3b023e-f99d-45f7-ac11-fcbea969d26d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82a48179-vgtsz\" (UID: \"de3b023e-f99d-45f7-ac11-fcbea969d26d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a48179-vgtsz" Oct 31 04:54:42.128198 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:42.128147 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de3b023e-f99d-45f7-ac11-fcbea969d26d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82a48179-vgtsz\" (UID: \"de3b023e-f99d-45f7-ac11-fcbea969d26d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a48179-vgtsz" Oct 31 04:54:42.230734 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:42.230689 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnb6p\" (UniqueName: \"kubernetes.io/projected/de3b023e-f99d-45f7-ac11-fcbea969d26d-kube-api-access-qnb6p\") pod \"maas-api-revoke-sub-82a48179-vgtsz\" (UID: \"de3b023e-f99d-45f7-ac11-fcbea969d26d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a48179-vgtsz" Oct 31 04:54:42.230949 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:42.230747 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de3b023e-f99d-45f7-ac11-fcbea969d26d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82a48179-vgtsz\" (UID: \"de3b023e-f99d-45f7-ac11-fcbea969d26d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a48179-vgtsz" Oct 31 04:54:42.230949 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:42.230885 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de3b023e-f99d-45f7-ac11-fcbea969d26d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82a48179-vgtsz\" (UID: \"de3b023e-f99d-45f7-ac11-fcbea969d26d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a48179-vgtsz" Oct 31 04:54:42.231732 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:42.231702 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de3b023e-f99d-45f7-ac11-fcbea969d26d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82a48179-vgtsz\" (UID: \"de3b023e-f99d-45f7-ac11-fcbea969d26d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a48179-vgtsz" Oct 31 04:54:42.233213 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:42.233179 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de3b023e-f99d-45f7-ac11-fcbea969d26d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82a48179-vgtsz\" (UID: \"de3b023e-f99d-45f7-ac11-fcbea969d26d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a48179-vgtsz" Oct 31 04:54:42.255470 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:42.255440 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnb6p\" (UniqueName: \"kubernetes.io/projected/de3b023e-f99d-45f7-ac11-fcbea969d26d-kube-api-access-qnb6p\") pod \"maas-api-revoke-sub-82a48179-vgtsz\" (UID: \"de3b023e-f99d-45f7-ac11-fcbea969d26d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a48179-vgtsz" Oct 31 04:54:42.381726 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:42.381626 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a48179-vgtsz" Oct 31 04:54:42.715201 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:42.715163 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82a48179-vgtsz"] Oct 31 04:54:42.715376 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:54:42.715253 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde3b023e_f99d_45f7_ac11_fcbea969d26d.slice/crio-8cdc28c8ec855e11f28ed8bf75a66cef0f9f150db9585c6c0938254a88fa21ca WatchSource:0}: Error finding container 8cdc28c8ec855e11f28ed8bf75a66cef0f9f150db9585c6c0938254a88fa21ca: Status 404 returned error can't find the container with id 8cdc28c8ec855e11f28ed8bf75a66cef0f9f150db9585c6c0938254a88fa21ca Oct 31 04:54:42.811454 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:42.811413 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a48179-vgtsz" event={"ID":"de3b023e-f99d-45f7-ac11-fcbea969d26d","Type":"ContainerStarted","Data":"8cdc28c8ec855e11f28ed8bf75a66cef0f9f150db9585c6c0938254a88fa21ca"} Oct 31 04:54:43.823342 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:43.823302 5019 generic.go:356] "Generic (PLEG): container finished" podID="de3b023e-f99d-45f7-ac11-fcbea969d26d" containerID="46ea82e2864e87be3668aa4dd08c74045a643b16edacfa13e6e4fd050c2bf604" exitCode=0 Oct 31 04:54:43.823839 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:43.823426 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a48179-vgtsz" event={"ID":"de3b023e-f99d-45f7-ac11-fcbea969d26d","Type":"ContainerDied","Data":"46ea82e2864e87be3668aa4dd08c74045a643b16edacfa13e6e4fd050c2bf604"} Oct 31 04:54:44.425354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:44.425097 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7a0ce4f-smczk"] Oct 31 04:54:44.477355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:44.474901 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7a0ce4f-smczk"] Oct 31 04:54:44.477355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:44.475044 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a0ce4f-smczk" Oct 31 04:54:44.682513 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:44.678220 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbba30e9-13f9-4518-a38d-de9fc33b21f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7a0ce4f-smczk\" (UID: \"bbba30e9-13f9-4518-a38d-de9fc33b21f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a0ce4f-smczk" Oct 31 04:54:44.786652 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:44.786603 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckgwk\" (UniqueName: \"kubernetes.io/projected/bbba30e9-13f9-4518-a38d-de9fc33b21f1-kube-api-access-ckgwk\") pod \"maas-api-revoke-sub-f7a0ce4f-smczk\" (UID: \"bbba30e9-13f9-4518-a38d-de9fc33b21f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a0ce4f-smczk" Oct 31 04:54:44.786993 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:44.786963 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbba30e9-13f9-4518-a38d-de9fc33b21f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7a0ce4f-smczk\" (UID: \"bbba30e9-13f9-4518-a38d-de9fc33b21f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a0ce4f-smczk" Oct 31 04:54:44.889136 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:44.889088 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckgwk\" (UniqueName: \"kubernetes.io/projected/bbba30e9-13f9-4518-a38d-de9fc33b21f1-kube-api-access-ckgwk\") pod \"maas-api-revoke-sub-f7a0ce4f-smczk\" (UID: \"bbba30e9-13f9-4518-a38d-de9fc33b21f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a0ce4f-smczk" Oct 31 04:54:44.889528 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:44.889164 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbba30e9-13f9-4518-a38d-de9fc33b21f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7a0ce4f-smczk\" (UID: \"bbba30e9-13f9-4518-a38d-de9fc33b21f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a0ce4f-smczk" Oct 31 04:54:44.889528 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:44.889304 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbba30e9-13f9-4518-a38d-de9fc33b21f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7a0ce4f-smczk\" (UID: \"bbba30e9-13f9-4518-a38d-de9fc33b21f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a0ce4f-smczk" Oct 31 04:54:44.891285 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:44.891251 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbba30e9-13f9-4518-a38d-de9fc33b21f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7a0ce4f-smczk\" (UID: \"bbba30e9-13f9-4518-a38d-de9fc33b21f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a0ce4f-smczk" Oct 31 04:54:44.893083 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:44.893056 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbba30e9-13f9-4518-a38d-de9fc33b21f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7a0ce4f-smczk\" (UID: \"bbba30e9-13f9-4518-a38d-de9fc33b21f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a0ce4f-smczk" Oct 31 04:54:44.913780 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:44.913746 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckgwk\" (UniqueName: \"kubernetes.io/projected/bbba30e9-13f9-4518-a38d-de9fc33b21f1-kube-api-access-ckgwk\") pod \"maas-api-revoke-sub-f7a0ce4f-smczk\" (UID: \"bbba30e9-13f9-4518-a38d-de9fc33b21f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a0ce4f-smczk" Oct 31 04:54:45.178626 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:45.178299 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a0ce4f-smczk" Oct 31 04:54:45.362733 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:45.362304 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a48179-vgtsz" Oct 31 04:54:45.424216 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:45.424163 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de3b023e-f99d-45f7-ac11-fcbea969d26d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de3b023e-f99d-45f7-ac11-fcbea969d26d-maas-api-openshift-ca\") pod \"de3b023e-f99d-45f7-ac11-fcbea969d26d\" (UID: \"de3b023e-f99d-45f7-ac11-fcbea969d26d\") " Oct 31 04:54:45.424431 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:45.424384 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de3b023e-f99d-45f7-ac11-fcbea969d26d-kube-api-access-qnb6p\" (UniqueName: \"kubernetes.io/projected/de3b023e-f99d-45f7-ac11-fcbea969d26d-kube-api-access-qnb6p\") pod \"de3b023e-f99d-45f7-ac11-fcbea969d26d\" (UID: \"de3b023e-f99d-45f7-ac11-fcbea969d26d\") " Oct 31 04:54:45.424525 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:45.424500 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de3b023e-f99d-45f7-ac11-fcbea969d26d-maas-api-openshift-ca" pod "de3b023e-f99d-45f7-ac11-fcbea969d26d" (UID: "de3b023e-f99d-45f7-ac11-fcbea969d26d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:54:45.424606 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:45.424529 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de3b023e-f99d-45f7-ac11-fcbea969d26d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de3b023e-f99d-45f7-ac11-fcbea969d26d-maas-api-xks-ca\") pod \"de3b023e-f99d-45f7-ac11-fcbea969d26d\" (UID: \"de3b023e-f99d-45f7-ac11-fcbea969d26d\") " Oct 31 04:54:45.425281 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:45.425253 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de3b023e-f99d-45f7-ac11-fcbea969d26d-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:54:45.428021 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:45.427991 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de3b023e-f99d-45f7-ac11-fcbea969d26d-kube-api-access-qnb6p" pod "de3b023e-f99d-45f7-ac11-fcbea969d26d" (UID: "de3b023e-f99d-45f7-ac11-fcbea969d26d"). InnerVolumeSpecName "kube-api-access-qnb6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:54:45.428151 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:45.428061 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de3b023e-f99d-45f7-ac11-fcbea969d26d-maas-api-xks-ca" pod "de3b023e-f99d-45f7-ac11-fcbea969d26d" (UID: "de3b023e-f99d-45f7-ac11-fcbea969d26d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:54:45.526820 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:45.526767 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnb6p\" (UniqueName: \"kubernetes.io/projected/de3b023e-f99d-45f7-ac11-fcbea969d26d-kube-api-access-qnb6p\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:54:45.526820 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:45.526808 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de3b023e-f99d-45f7-ac11-fcbea969d26d-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:54:45.631439 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:54:45.631396 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbba30e9_13f9_4518_a38d_de9fc33b21f1.slice/crio-d99010037294e15b3b705ec7237241cc2fbbbc9423ecbcfa1d7461d75743882c WatchSource:0}: Error finding container d99010037294e15b3b705ec7237241cc2fbbbc9423ecbcfa1d7461d75743882c: Status 404 returned error can't find the container with id d99010037294e15b3b705ec7237241cc2fbbbc9423ecbcfa1d7461d75743882c Oct 31 04:54:45.633578 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:45.633545 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7a0ce4f-smczk"] Oct 31 04:54:45.858729 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:45.858639 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a48179-vgtsz" Oct 31 04:54:45.858729 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:45.858673 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a48179-vgtsz" event={"ID":"de3b023e-f99d-45f7-ac11-fcbea969d26d","Type":"ContainerDied","Data":"8cdc28c8ec855e11f28ed8bf75a66cef0f9f150db9585c6c0938254a88fa21ca"} Oct 31 04:54:45.858935 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:45.858736 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8cdc28c8ec855e11f28ed8bf75a66cef0f9f150db9585c6c0938254a88fa21ca" Oct 31 04:54:45.860643 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:45.860612 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a0ce4f-smczk" event={"ID":"bbba30e9-13f9-4518-a38d-de9fc33b21f1","Type":"ContainerStarted","Data":"d99010037294e15b3b705ec7237241cc2fbbbc9423ecbcfa1d7461d75743882c"} Oct 31 04:54:46.873973 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:46.873931 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a0ce4f-smczk" event={"ID":"bbba30e9-13f9-4518-a38d-de9fc33b21f1","Type":"ContainerStarted","Data":"c9de05d58e1040ed2e0e04941f25c38f8267e7a2b729cf561c9d216be050dd97"} Oct 31 04:54:46.916441 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:46.916370 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a0ce4f-smczk" podStartSLOduration=2.916347602 podStartE2EDuration="2.916347602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:54:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:54:46.913531814 +0000 UTC m=+3041.669869858" watchObservedRunningTime="2026-10-31 04:54:46.916347602 +0000 UTC m=+3041.672685642" Oct 31 04:54:47.885228 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:47.885186 5019 generic.go:356] "Generic (PLEG): container finished" podID="bbba30e9-13f9-4518-a38d-de9fc33b21f1" containerID="c9de05d58e1040ed2e0e04941f25c38f8267e7a2b729cf561c9d216be050dd97" exitCode=0 Oct 31 04:54:47.885677 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:47.885276 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a0ce4f-smczk" event={"ID":"bbba30e9-13f9-4518-a38d-de9fc33b21f1","Type":"ContainerDied","Data":"c9de05d58e1040ed2e0e04941f25c38f8267e7a2b729cf561c9d216be050dd97"} Oct 31 04:54:49.094251 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:49.094211 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a0ce4f-smczk" Oct 31 04:54:49.194164 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:49.194042 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbba30e9-13f9-4518-a38d-de9fc33b21f1-kube-api-access-ckgwk\" (UniqueName: \"kubernetes.io/projected/bbba30e9-13f9-4518-a38d-de9fc33b21f1-kube-api-access-ckgwk\") pod \"bbba30e9-13f9-4518-a38d-de9fc33b21f1\" (UID: \"bbba30e9-13f9-4518-a38d-de9fc33b21f1\") " Oct 31 04:54:49.194425 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:49.194225 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbba30e9-13f9-4518-a38d-de9fc33b21f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbba30e9-13f9-4518-a38d-de9fc33b21f1-maas-api-openshift-ca\") pod \"bbba30e9-13f9-4518-a38d-de9fc33b21f1\" (UID: \"bbba30e9-13f9-4518-a38d-de9fc33b21f1\") " Oct 31 04:54:49.194528 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:49.194507 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbba30e9-13f9-4518-a38d-de9fc33b21f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbba30e9-13f9-4518-a38d-de9fc33b21f1-maas-api-xks-ca\") pod \"bbba30e9-13f9-4518-a38d-de9fc33b21f1\" (UID: \"bbba30e9-13f9-4518-a38d-de9fc33b21f1\") " Oct 31 04:54:49.194697 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:49.194668 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbba30e9-13f9-4518-a38d-de9fc33b21f1-maas-api-openshift-ca" pod "bbba30e9-13f9-4518-a38d-de9fc33b21f1" (UID: "bbba30e9-13f9-4518-a38d-de9fc33b21f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:54:49.195290 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:49.195265 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbba30e9-13f9-4518-a38d-de9fc33b21f1-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:54:49.196651 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:49.196624 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbba30e9-13f9-4518-a38d-de9fc33b21f1-kube-api-access-ckgwk" pod "bbba30e9-13f9-4518-a38d-de9fc33b21f1" (UID: "bbba30e9-13f9-4518-a38d-de9fc33b21f1"). InnerVolumeSpecName "kube-api-access-ckgwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:54:49.197241 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:49.197213 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbba30e9-13f9-4518-a38d-de9fc33b21f1-maas-api-xks-ca" pod "bbba30e9-13f9-4518-a38d-de9fc33b21f1" (UID: "bbba30e9-13f9-4518-a38d-de9fc33b21f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:54:49.297437 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:49.297398 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbba30e9-13f9-4518-a38d-de9fc33b21f1-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:54:49.297437 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:49.297433 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckgwk\" (UniqueName: \"kubernetes.io/projected/bbba30e9-13f9-4518-a38d-de9fc33b21f1-kube-api-access-ckgwk\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:54:49.721627 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:49.721584 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc"] Oct 31 04:54:49.721892 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:49.721862 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc" podUID="b3743e35-69e7-4544-979d-7596c44b59a0" containerName="istio-proxy" containerID="cri-o://f5e626561f82ca23306e02890938d51ca0ae716bbbac98b43f6aad8114a67338" gracePeriod=30 Oct 31 04:54:49.847019 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:49.846974 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s"] Oct 31 04:54:49.847267 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:49.847240 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s" podUID="22f2a6e7-c1d8-4834-80d3-805cf819e158" containerName="istio-proxy" containerID="cri-o://9b639dda4a875f7a915fd3f30eae68a6793d1056cc8a334a0e15f440fac2de45" gracePeriod=30 Oct 31 04:54:49.914897 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:49.914864 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a0ce4f-smczk" Oct 31 04:54:49.915123 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:49.914901 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a0ce4f-smczk" event={"ID":"bbba30e9-13f9-4518-a38d-de9fc33b21f1","Type":"ContainerDied","Data":"d99010037294e15b3b705ec7237241cc2fbbbc9423ecbcfa1d7461d75743882c"} Oct 31 04:54:49.915123 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:49.914965 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d99010037294e15b3b705ec7237241cc2fbbbc9423ecbcfa1d7461d75743882c" Oct 31 04:54:52.396706 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:54:52.396639 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:54:53.721092 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:53.721035 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-74fckxtnc"] Oct 31 04:54:54.643356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:54.643282 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bc4gchxx"] Oct 31 04:54:55.288134 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.288064 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1a932c0-nlzns"] Oct 31 04:54:55.289990 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.289955 5019 cpu_manager.go:414] "containerMap: removing container" podUID="de3b023e-f99d-45f7-ac11-fcbea969d26d" containerName="revoke-keys" Oct 31 04:54:55.290183 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.290169 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de3b023e-f99d-45f7-ac11-fcbea969d26d" containerName="revoke-keys" Oct 31 04:54:55.290312 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.290301 5019 cpu_manager.go:414] "containerMap: removing container" podUID="bbba30e9-13f9-4518-a38d-de9fc33b21f1" containerName="revoke-keys" Oct 31 04:54:55.290498 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.290471 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbba30e9-13f9-4518-a38d-de9fc33b21f1" containerName="revoke-keys" Oct 31 04:54:55.290922 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.290903 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="de3b023e-f99d-45f7-ac11-fcbea969d26d" containerName="revoke-keys" Oct 31 04:54:55.291043 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.291032 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbba30e9-13f9-4518-a38d-de9fc33b21f1" containerName="revoke-keys" Oct 31 04:54:55.308911 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.308525 5019 patch_prober.go:29] interesting pod/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Oct 31 04:54:55.308911 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.308607 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s" podUID="22f2a6e7-c1d8-4834-80d3-805cf819e158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Oct 31 04:54:55.325914 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.325876 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1a932c0-nlzns"] Oct 31 04:54:55.326268 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.326203 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1a932c0-nlzns" Oct 31 04:54:55.344219 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.344144 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:54:55.344413 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.344254 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:54:55.344685 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.344648 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:54:55.345179 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.345146 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:54:55.346255 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.346215 5019 generic.go:356] "Generic (PLEG): container finished" podID="22f2a6e7-c1d8-4834-80d3-805cf819e158" containerID="9b639dda4a875f7a915fd3f30eae68a6793d1056cc8a334a0e15f440fac2de45" exitCode=0 Oct 31 04:54:55.347095 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.346299 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s" event={"ID":"22f2a6e7-c1d8-4834-80d3-805cf819e158","Type":"ContainerDied","Data":"9b639dda4a875f7a915fd3f30eae68a6793d1056cc8a334a0e15f440fac2de45"} Oct 31 04:54:55.347095 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.346517 5019 scope.go:122] "RemoveContainer" containerID="55b10d00a9cbae70610d3dddfad1eaa18550d082df6f4faa78043ac9e03bbf31" Oct 31 04:54:55.349647 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.349613 5019 generic.go:356] "Generic (PLEG): container finished" podID="b3743e35-69e7-4544-979d-7596c44b59a0" containerID="f5e626561f82ca23306e02890938d51ca0ae716bbbac98b43f6aad8114a67338" exitCode=0 Oct 31 04:54:55.349762 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.349691 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc" event={"ID":"b3743e35-69e7-4544-979d-7596c44b59a0","Type":"ContainerDied","Data":"f5e626561f82ca23306e02890938d51ca0ae716bbbac98b43f6aad8114a67338"} Oct 31 04:54:55.362667 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.362608 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7543c659-84f2-4c09-aa0f-d517f487eaf4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1a932c0-nlzns\" (UID: \"7543c659-84f2-4c09-aa0f-d517f487eaf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1a932c0-nlzns" Oct 31 04:54:55.362846 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.362699 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2zps\" (UniqueName: \"kubernetes.io/projected/7543c659-84f2-4c09-aa0f-d517f487eaf4-kube-api-access-q2zps\") pod \"maas-api-revoke-sub-b1a932c0-nlzns\" (UID: \"7543c659-84f2-4c09-aa0f-d517f487eaf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1a932c0-nlzns" Oct 31 04:54:55.362846 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.362752 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7543c659-84f2-4c09-aa0f-d517f487eaf4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1a932c0-nlzns\" (UID: \"7543c659-84f2-4c09-aa0f-d517f487eaf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1a932c0-nlzns" Oct 31 04:54:55.399393 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.397709 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc" Oct 31 04:54:55.464672 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.464633 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3743e35-69e7-4544-979d-7596c44b59a0-kube-api-access-v5bx8\" (UniqueName: \"kubernetes.io/projected/b3743e35-69e7-4544-979d-7596c44b59a0-kube-api-access-v5bx8\") pod \"b3743e35-69e7-4544-979d-7596c44b59a0\" (UID: \"b3743e35-69e7-4544-979d-7596c44b59a0\") " Oct 31 04:54:55.464864 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.464772 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b3743e35-69e7-4544-979d-7596c44b59a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3743e35-69e7-4544-979d-7596c44b59a0-istio-podinfo\") pod \"b3743e35-69e7-4544-979d-7596c44b59a0\" (UID: \"b3743e35-69e7-4544-979d-7596c44b59a0\") " Oct 31 04:54:55.464864 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.464838 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3743e35-69e7-4544-979d-7596c44b59a0-istio-token\" (UniqueName: \"kubernetes.io/projected/b3743e35-69e7-4544-979d-7596c44b59a0-istio-token\") pod \"b3743e35-69e7-4544-979d-7596c44b59a0\" (UID: \"b3743e35-69e7-4544-979d-7596c44b59a0\") " Oct 31 04:54:55.465002 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.464867 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3743e35-69e7-4544-979d-7596c44b59a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3743e35-69e7-4544-979d-7596c44b59a0-istio-envoy\") pod \"b3743e35-69e7-4544-979d-7596c44b59a0\" (UID: \"b3743e35-69e7-4544-979d-7596c44b59a0\") " Oct 31 04:54:55.465002 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.464942 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3743e35-69e7-4544-979d-7596c44b59a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3743e35-69e7-4544-979d-7596c44b59a0-workload-certs\") pod \"b3743e35-69e7-4544-979d-7596c44b59a0\" (UID: \"b3743e35-69e7-4544-979d-7596c44b59a0\") " Oct 31 04:54:55.465002 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.464980 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3743e35-69e7-4544-979d-7596c44b59a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3743e35-69e7-4544-979d-7596c44b59a0-istiod-ca-cert\") pod \"b3743e35-69e7-4544-979d-7596c44b59a0\" (UID: \"b3743e35-69e7-4544-979d-7596c44b59a0\") " Oct 31 04:54:55.465154 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.465044 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3743e35-69e7-4544-979d-7596c44b59a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3743e35-69e7-4544-979d-7596c44b59a0-istio-data\") pod \"b3743e35-69e7-4544-979d-7596c44b59a0\" (UID: \"b3743e35-69e7-4544-979d-7596c44b59a0\") " Oct 31 04:54:55.465319 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.465303 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3743e35-69e7-4544-979d-7596c44b59a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3743e35-69e7-4544-979d-7596c44b59a0-workload-socket\") pod \"b3743e35-69e7-4544-979d-7596c44b59a0\" (UID: \"b3743e35-69e7-4544-979d-7596c44b59a0\") " Oct 31 04:54:55.465473 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.465353 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3743e35-69e7-4544-979d-7596c44b59a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3743e35-69e7-4544-979d-7596c44b59a0-credential-socket\") pod \"b3743e35-69e7-4544-979d-7596c44b59a0\" (UID: \"b3743e35-69e7-4544-979d-7596c44b59a0\") " Oct 31 04:54:55.465638 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.465618 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7543c659-84f2-4c09-aa0f-d517f487eaf4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1a932c0-nlzns\" (UID: \"7543c659-84f2-4c09-aa0f-d517f487eaf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1a932c0-nlzns" Oct 31 04:54:55.466255 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.466086 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7543c659-84f2-4c09-aa0f-d517f487eaf4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1a932c0-nlzns\" (UID: \"7543c659-84f2-4c09-aa0f-d517f487eaf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1a932c0-nlzns" Oct 31 04:54:55.466558 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.466461 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2zps\" (UniqueName: \"kubernetes.io/projected/7543c659-84f2-4c09-aa0f-d517f487eaf4-kube-api-access-q2zps\") pod \"maas-api-revoke-sub-b1a932c0-nlzns\" (UID: \"7543c659-84f2-4c09-aa0f-d517f487eaf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1a932c0-nlzns" Oct 31 04:54:55.467854 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.467824 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3743e35-69e7-4544-979d-7596c44b59a0-istio-data" pod "b3743e35-69e7-4544-979d-7596c44b59a0" (UID: "b3743e35-69e7-4544-979d-7596c44b59a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:54:55.468973 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.468544 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3743e35-69e7-4544-979d-7596c44b59a0-workload-certs" pod "b3743e35-69e7-4544-979d-7596c44b59a0" (UID: "b3743e35-69e7-4544-979d-7596c44b59a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:54:55.468973 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.468623 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3743e35-69e7-4544-979d-7596c44b59a0-istiod-ca-cert" pod "b3743e35-69e7-4544-979d-7596c44b59a0" (UID: "b3743e35-69e7-4544-979d-7596c44b59a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:54:55.469138 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.468991 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7543c659-84f2-4c09-aa0f-d517f487eaf4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1a932c0-nlzns\" (UID: \"7543c659-84f2-4c09-aa0f-d517f487eaf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1a932c0-nlzns" Oct 31 04:54:55.469205 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.469172 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3743e35-69e7-4544-979d-7596c44b59a0-credential-socket" pod "b3743e35-69e7-4544-979d-7596c44b59a0" (UID: "b3743e35-69e7-4544-979d-7596c44b59a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:54:55.469205 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.469185 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3743e35-69e7-4544-979d-7596c44b59a0-workload-socket" pod "b3743e35-69e7-4544-979d-7596c44b59a0" (UID: "b3743e35-69e7-4544-979d-7596c44b59a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:54:55.473876 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.473824 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3743e35-69e7-4544-979d-7596c44b59a0-istio-podinfo" pod "b3743e35-69e7-4544-979d-7596c44b59a0" (UID: "b3743e35-69e7-4544-979d-7596c44b59a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:54:55.474675 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.474646 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7543c659-84f2-4c09-aa0f-d517f487eaf4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1a932c0-nlzns\" (UID: \"7543c659-84f2-4c09-aa0f-d517f487eaf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1a932c0-nlzns" Oct 31 04:54:55.476586 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.476546 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3743e35-69e7-4544-979d-7596c44b59a0-istio-envoy" pod "b3743e35-69e7-4544-979d-7596c44b59a0" (UID: "b3743e35-69e7-4544-979d-7596c44b59a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:54:55.479906 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.479871 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3743e35-69e7-4544-979d-7596c44b59a0-kube-api-access-v5bx8" pod "b3743e35-69e7-4544-979d-7596c44b59a0" (UID: "b3743e35-69e7-4544-979d-7596c44b59a0"). InnerVolumeSpecName "kube-api-access-v5bx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:54:55.484046 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.484003 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3743e35-69e7-4544-979d-7596c44b59a0-istio-token" pod "b3743e35-69e7-4544-979d-7596c44b59a0" (UID: "b3743e35-69e7-4544-979d-7596c44b59a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:54:55.490967 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.490905 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2zps\" (UniqueName: \"kubernetes.io/projected/7543c659-84f2-4c09-aa0f-d517f487eaf4-kube-api-access-q2zps\") pod \"maas-api-revoke-sub-b1a932c0-nlzns\" (UID: \"7543c659-84f2-4c09-aa0f-d517f487eaf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1a932c0-nlzns" Oct 31 04:54:55.545840 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.545759 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s" Oct 31 04:54:55.567240 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.567191 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22f2a6e7-c1d8-4834-80d3-805cf819e158-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f2a6e7-c1d8-4834-80d3-805cf819e158-credential-socket\") pod \"22f2a6e7-c1d8-4834-80d3-805cf819e158\" (UID: \"22f2a6e7-c1d8-4834-80d3-805cf819e158\") " Oct 31 04:54:55.567476 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.567304 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22f2a6e7-c1d8-4834-80d3-805cf819e158-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f2a6e7-c1d8-4834-80d3-805cf819e158-istio-envoy\") pod \"22f2a6e7-c1d8-4834-80d3-805cf819e158\" (UID: \"22f2a6e7-c1d8-4834-80d3-805cf819e158\") " Oct 31 04:54:55.567476 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.567380 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22f2a6e7-c1d8-4834-80d3-805cf819e158-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f2a6e7-c1d8-4834-80d3-805cf819e158-workload-socket\") pod \"22f2a6e7-c1d8-4834-80d3-805cf819e158\" (UID: \"22f2a6e7-c1d8-4834-80d3-805cf819e158\") " Oct 31 04:54:55.567476 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.567414 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22f2a6e7-c1d8-4834-80d3-805cf819e158-istio-token\" (UniqueName: \"kubernetes.io/projected/22f2a6e7-c1d8-4834-80d3-805cf819e158-istio-token\") pod \"22f2a6e7-c1d8-4834-80d3-805cf819e158\" (UID: \"22f2a6e7-c1d8-4834-80d3-805cf819e158\") " Oct 31 04:54:55.567476 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.567449 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22f2a6e7-c1d8-4834-80d3-805cf819e158-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f2a6e7-c1d8-4834-80d3-805cf819e158-workload-certs\") pod \"22f2a6e7-c1d8-4834-80d3-805cf819e158\" (UID: \"22f2a6e7-c1d8-4834-80d3-805cf819e158\") " Oct 31 04:54:55.567476 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.567474 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22f2a6e7-c1d8-4834-80d3-805cf819e158-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f2a6e7-c1d8-4834-80d3-805cf819e158-istio-podinfo\") pod \"22f2a6e7-c1d8-4834-80d3-805cf819e158\" (UID: \"22f2a6e7-c1d8-4834-80d3-805cf819e158\") " Oct 31 04:54:55.567871 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.567524 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22f2a6e7-c1d8-4834-80d3-805cf819e158-kube-api-access-x8s8j\" (UniqueName: \"kubernetes.io/projected/22f2a6e7-c1d8-4834-80d3-805cf819e158-kube-api-access-x8s8j\") pod \"22f2a6e7-c1d8-4834-80d3-805cf819e158\" (UID: \"22f2a6e7-c1d8-4834-80d3-805cf819e158\") " Oct 31 04:54:55.567871 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.567576 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f2a6e7-c1d8-4834-80d3-805cf819e158-credential-socket" pod "22f2a6e7-c1d8-4834-80d3-805cf819e158" (UID: "22f2a6e7-c1d8-4834-80d3-805cf819e158"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:54:55.567871 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.567626 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22f2a6e7-c1d8-4834-80d3-805cf819e158-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f2a6e7-c1d8-4834-80d3-805cf819e158-istio-data\") pod \"22f2a6e7-c1d8-4834-80d3-805cf819e158\" (UID: \"22f2a6e7-c1d8-4834-80d3-805cf819e158\") " Oct 31 04:54:55.567871 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.567828 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22f2a6e7-c1d8-4834-80d3-805cf819e158-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f2a6e7-c1d8-4834-80d3-805cf819e158-istiod-ca-cert\") pod \"22f2a6e7-c1d8-4834-80d3-805cf819e158\" (UID: \"22f2a6e7-c1d8-4834-80d3-805cf819e158\") " Oct 31 04:54:55.567871 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.567832 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f2a6e7-c1d8-4834-80d3-805cf819e158-workload-certs" pod "22f2a6e7-c1d8-4834-80d3-805cf819e158" (UID: "22f2a6e7-c1d8-4834-80d3-805cf819e158"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:54:55.568171 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.568117 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22f2a6e7-c1d8-4834-80d3-805cf819e158-istiod-ca-cert" pod "22f2a6e7-c1d8-4834-80d3-805cf819e158" (UID: "22f2a6e7-c1d8-4834-80d3-805cf819e158"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:54:55.568638 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.568384 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f2a6e7-c1d8-4834-80d3-805cf819e158-istio-data" pod "22f2a6e7-c1d8-4834-80d3-805cf819e158" (UID: "22f2a6e7-c1d8-4834-80d3-805cf819e158"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:54:55.568638 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.568487 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f2a6e7-c1d8-4834-80d3-805cf819e158-workload-socket" pod "22f2a6e7-c1d8-4834-80d3-805cf819e158" (UID: "22f2a6e7-c1d8-4834-80d3-805cf819e158"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:54:55.568954 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.568934 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f2a6e7-c1d8-4834-80d3-805cf819e158-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:54:55.569043 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.568958 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3743e35-69e7-4544-979d-7596c44b59a0-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:54:55.569043 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.568971 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3743e35-69e7-4544-979d-7596c44b59a0-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:54:55.569043 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.568985 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f2a6e7-c1d8-4834-80d3-805cf819e158-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:54:55.569043 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.569001 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5bx8\" (UniqueName: \"kubernetes.io/projected/b3743e35-69e7-4544-979d-7596c44b59a0-kube-api-access-v5bx8\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:54:55.569043 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.569015 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f2a6e7-c1d8-4834-80d3-805cf819e158-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:54:55.569043 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.569027 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f2a6e7-c1d8-4834-80d3-805cf819e158-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:54:55.569043 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.569041 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3743e35-69e7-4544-979d-7596c44b59a0-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:54:55.569521 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.569055 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3743e35-69e7-4544-979d-7596c44b59a0-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:54:55.569521 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.569069 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3743e35-69e7-4544-979d-7596c44b59a0-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:54:55.569521 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.569082 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f2a6e7-c1d8-4834-80d3-805cf819e158-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:54:55.569521 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.569094 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3743e35-69e7-4544-979d-7596c44b59a0-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:54:55.569521 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.569106 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3743e35-69e7-4544-979d-7596c44b59a0-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:54:55.569521 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.569119 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3743e35-69e7-4544-979d-7596c44b59a0-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:54:55.572949 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.572911 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f2a6e7-c1d8-4834-80d3-805cf819e158-istio-envoy" pod "22f2a6e7-c1d8-4834-80d3-805cf819e158" (UID: "22f2a6e7-c1d8-4834-80d3-805cf819e158"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:54:55.573136 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.573110 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22f2a6e7-c1d8-4834-80d3-805cf819e158-kube-api-access-x8s8j" pod "22f2a6e7-c1d8-4834-80d3-805cf819e158" (UID: "22f2a6e7-c1d8-4834-80d3-805cf819e158"). InnerVolumeSpecName "kube-api-access-x8s8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:54:55.573276 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.573238 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22f2a6e7-c1d8-4834-80d3-805cf819e158-istio-token" pod "22f2a6e7-c1d8-4834-80d3-805cf819e158" (UID: "22f2a6e7-c1d8-4834-80d3-805cf819e158"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:54:55.573377 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.573320 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22f2a6e7-c1d8-4834-80d3-805cf819e158-istio-podinfo" pod "22f2a6e7-c1d8-4834-80d3-805cf819e158" (UID: "22f2a6e7-c1d8-4834-80d3-805cf819e158"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:54:55.682584 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.682532 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f2a6e7-c1d8-4834-80d3-805cf819e158-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:54:55.682584 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.682586 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22f2a6e7-c1d8-4834-80d3-805cf819e158-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:54:55.682866 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.682601 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f2a6e7-c1d8-4834-80d3-805cf819e158-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:54:55.682866 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.682616 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8s8j\" (UniqueName: \"kubernetes.io/projected/22f2a6e7-c1d8-4834-80d3-805cf819e158-kube-api-access-x8s8j\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:54:55.693311 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:55.693272 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1a932c0-nlzns" Oct 31 04:54:56.069514 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:54:56.069472 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7543c659_84f2_4c09_aa0f_d517f487eaf4.slice/crio-f13aaf745733eb6a8695977d9fa15421b75b22dc5d7c9829fc75759c2b69f780 WatchSource:0}: Error finding container f13aaf745733eb6a8695977d9fa15421b75b22dc5d7c9829fc75759c2b69f780: Status 404 returned error can't find the container with id f13aaf745733eb6a8695977d9fa15421b75b22dc5d7c9829fc75759c2b69f780 Oct 31 04:54:56.072676 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:56.072643 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1a932c0-nlzns"] Oct 31 04:54:56.361676 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:56.361637 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1a932c0-nlzns" event={"ID":"7543c659-84f2-4c09-aa0f-d517f487eaf4","Type":"ContainerStarted","Data":"f13aaf745733eb6a8695977d9fa15421b75b22dc5d7c9829fc75759c2b69f780"} Oct 31 04:54:56.363309 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:56.363284 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc" Oct 31 04:54:56.363468 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:56.363284 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc" event={"ID":"b3743e35-69e7-4544-979d-7596c44b59a0","Type":"ContainerDied","Data":"0ef939667ceafeea31b3ffe57c86d5d5c506ee07fb1d871de9fa7ca40195cbf9"} Oct 31 04:54:56.363468 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:56.363404 5019 scope.go:122] "RemoveContainer" containerID="f5e626561f82ca23306e02890938d51ca0ae716bbbac98b43f6aad8114a67338" Oct 31 04:54:56.365272 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:56.365250 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s" event={"ID":"22f2a6e7-c1d8-4834-80d3-805cf819e158","Type":"ContainerDied","Data":"57791878e1def00efad041a7f6da21b57070384e404ca1874a40d00729e877f5"} Oct 31 04:54:56.365384 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:56.365313 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s" Oct 31 04:54:56.379888 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:56.379864 5019 scope.go:122] "RemoveContainer" containerID="9b639dda4a875f7a915fd3f30eae68a6793d1056cc8a334a0e15f440fac2de45" Oct 31 04:54:56.433283 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:56.433255 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc"] Oct 31 04:54:56.439467 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:56.439429 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-7zpnc"] Oct 31 04:54:56.474550 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:56.474490 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s"] Oct 31 04:54:56.481237 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:56.481200 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-tv22s"] Oct 31 04:54:57.383597 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:57.383530 5019 generic.go:356] "Generic (PLEG): container finished" podID="7543c659-84f2-4c09-aa0f-d517f487eaf4" containerID="0e0cad32e7c8fd53718550904a956508f1221266f0162af4afedac1f44bbaa9b" exitCode=0 Oct 31 04:54:57.384102 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:57.383629 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1a932c0-nlzns" event={"ID":"7543c659-84f2-4c09-aa0f-d517f487eaf4","Type":"ContainerDied","Data":"0e0cad32e7c8fd53718550904a956508f1221266f0162af4afedac1f44bbaa9b"} Oct 31 04:54:57.424956 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:57.424866 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22f2a6e7-c1d8-4834-80d3-805cf819e158" path="/var/lib/kubelet/pods/22f2a6e7-c1d8-4834-80d3-805cf819e158/volumes" Oct 31 04:54:57.426240 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:57.426169 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3743e35-69e7-4544-979d-7596c44b59a0" path="/var/lib/kubelet/pods/b3743e35-69e7-4544-979d-7596c44b59a0/volumes" Oct 31 04:54:57.800179 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:57.800137 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-cb4fj785m"] Oct 31 04:54:58.623188 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:58.623155 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1a932c0-nlzns" Oct 31 04:54:58.721594 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:58.721545 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-74fckxtnc" podUID="5606bd96-b466-42d3-9eb8-fffe43b65bff" containerName="payload-processing" containerID="cri-o://4422794e18d1a9e77fc037eec3c2866e0e975344d3e6f9f14be54c894a1f0d1f" gracePeriod=25 Oct 31 04:54:58.777326 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:58.777268 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7543c659-84f2-4c09-aa0f-d517f487eaf4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7543c659-84f2-4c09-aa0f-d517f487eaf4-maas-api-xks-ca\") pod \"7543c659-84f2-4c09-aa0f-d517f487eaf4\" (UID: \"7543c659-84f2-4c09-aa0f-d517f487eaf4\") " Oct 31 04:54:58.777539 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:58.777404 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7543c659-84f2-4c09-aa0f-d517f487eaf4-kube-api-access-q2zps\" (UniqueName: \"kubernetes.io/projected/7543c659-84f2-4c09-aa0f-d517f487eaf4-kube-api-access-q2zps\") pod \"7543c659-84f2-4c09-aa0f-d517f487eaf4\" (UID: \"7543c659-84f2-4c09-aa0f-d517f487eaf4\") " Oct 31 04:54:58.777630 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:58.777613 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7543c659-84f2-4c09-aa0f-d517f487eaf4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7543c659-84f2-4c09-aa0f-d517f487eaf4-maas-api-openshift-ca\") pod \"7543c659-84f2-4c09-aa0f-d517f487eaf4\" (UID: \"7543c659-84f2-4c09-aa0f-d517f487eaf4\") " Oct 31 04:54:58.778064 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:58.778026 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7543c659-84f2-4c09-aa0f-d517f487eaf4-maas-api-openshift-ca" pod "7543c659-84f2-4c09-aa0f-d517f487eaf4" (UID: "7543c659-84f2-4c09-aa0f-d517f487eaf4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:54:58.778595 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:58.778567 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7543c659-84f2-4c09-aa0f-d517f487eaf4-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:54:58.779852 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:58.779823 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7543c659-84f2-4c09-aa0f-d517f487eaf4-maas-api-xks-ca" pod "7543c659-84f2-4c09-aa0f-d517f487eaf4" (UID: "7543c659-84f2-4c09-aa0f-d517f487eaf4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:54:58.779994 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:58.779858 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7543c659-84f2-4c09-aa0f-d517f487eaf4-kube-api-access-q2zps" pod "7543c659-84f2-4c09-aa0f-d517f487eaf4" (UID: "7543c659-84f2-4c09-aa0f-d517f487eaf4"). InnerVolumeSpecName "kube-api-access-q2zps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:54:58.880832 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:58.880775 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7543c659-84f2-4c09-aa0f-d517f487eaf4-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:54:58.880832 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:58.880827 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2zps\" (UniqueName: \"kubernetes.io/projected/7543c659-84f2-4c09-aa0f-d517f487eaf4-kube-api-access-q2zps\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:54:59.036274 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:59.036239 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-74fckxtnc" Oct 31 04:54:59.185635 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:59.185568 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5606bd96-b466-42d3-9eb8-fffe43b65bff-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5606bd96-b466-42d3-9eb8-fffe43b65bff-provider-credentials\") pod \"5606bd96-b466-42d3-9eb8-fffe43b65bff\" (UID: \"5606bd96-b466-42d3-9eb8-fffe43b65bff\") " Oct 31 04:54:59.185842 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:59.185808 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5606bd96-b466-42d3-9eb8-fffe43b65bff-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5606bd96-b466-42d3-9eb8-fffe43b65bff-routing-overlay-volume\") pod \"5606bd96-b466-42d3-9eb8-fffe43b65bff\" (UID: \"5606bd96-b466-42d3-9eb8-fffe43b65bff\") " Oct 31 04:54:59.185968 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:59.185944 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5606bd96-b466-42d3-9eb8-fffe43b65bff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5606bd96-b466-42d3-9eb8-fffe43b65bff-plugins-config-volume\") pod \"5606bd96-b466-42d3-9eb8-fffe43b65bff\" (UID: \"5606bd96-b466-42d3-9eb8-fffe43b65bff\") " Oct 31 04:54:59.186147 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:59.186129 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5606bd96-b466-42d3-9eb8-fffe43b65bff-kube-api-access-vvx4c\" (UniqueName: \"kubernetes.io/projected/5606bd96-b466-42d3-9eb8-fffe43b65bff-kube-api-access-vvx4c\") pod \"5606bd96-b466-42d3-9eb8-fffe43b65bff\" (UID: \"5606bd96-b466-42d3-9eb8-fffe43b65bff\") " Oct 31 04:54:59.186796 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:59.186733 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5606bd96-b466-42d3-9eb8-fffe43b65bff-routing-overlay-volume" pod "5606bd96-b466-42d3-9eb8-fffe43b65bff" (UID: "5606bd96-b466-42d3-9eb8-fffe43b65bff"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:54:59.187144 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:59.187122 5019 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5606bd96-b466-42d3-9eb8-fffe43b65bff-routing-overlay-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:54:59.187569 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:59.187530 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5606bd96-b466-42d3-9eb8-fffe43b65bff-plugins-config-volume" pod "5606bd96-b466-42d3-9eb8-fffe43b65bff" (UID: "5606bd96-b466-42d3-9eb8-fffe43b65bff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:54:59.189954 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:59.189918 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5606bd96-b466-42d3-9eb8-fffe43b65bff-provider-credentials" pod "5606bd96-b466-42d3-9eb8-fffe43b65bff" (UID: "5606bd96-b466-42d3-9eb8-fffe43b65bff"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:54:59.190236 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:59.190208 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5606bd96-b466-42d3-9eb8-fffe43b65bff-kube-api-access-vvx4c" pod "5606bd96-b466-42d3-9eb8-fffe43b65bff" (UID: "5606bd96-b466-42d3-9eb8-fffe43b65bff"). InnerVolumeSpecName "kube-api-access-vvx4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:54:59.292241 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:59.292120 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5606bd96-b466-42d3-9eb8-fffe43b65bff-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:54:59.292241 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:59.292172 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvx4c\" (UniqueName: \"kubernetes.io/projected/5606bd96-b466-42d3-9eb8-fffe43b65bff-kube-api-access-vvx4c\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:54:59.292241 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:59.292191 5019 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5606bd96-b466-42d3-9eb8-fffe43b65bff-provider-credentials\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:54:59.421503 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:59.421460 5019 generic.go:356] "Generic (PLEG): container finished" podID="5606bd96-b466-42d3-9eb8-fffe43b65bff" containerID="4422794e18d1a9e77fc037eec3c2866e0e975344d3e6f9f14be54c894a1f0d1f" exitCode=0 Oct 31 04:54:59.421979 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:59.421636 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-74fckxtnc" Oct 31 04:54:59.427418 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:59.427373 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-74fckxtnc" event={"ID":"5606bd96-b466-42d3-9eb8-fffe43b65bff","Type":"ContainerDied","Data":"4422794e18d1a9e77fc037eec3c2866e0e975344d3e6f9f14be54c894a1f0d1f"} Oct 31 04:54:59.427649 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:59.427454 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-74fckxtnc" event={"ID":"5606bd96-b466-42d3-9eb8-fffe43b65bff","Type":"ContainerDied","Data":"a6fdd51578a9d36d15be3a870ff35171ca77d83839ff2cea474ecd2e93d13e4a"} Oct 31 04:54:59.427649 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:59.427487 5019 scope.go:122] "RemoveContainer" containerID="4422794e18d1a9e77fc037eec3c2866e0e975344d3e6f9f14be54c894a1f0d1f" Oct 31 04:54:59.437783 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:59.437744 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1a932c0-nlzns" event={"ID":"7543c659-84f2-4c09-aa0f-d517f487eaf4","Type":"ContainerDied","Data":"f13aaf745733eb6a8695977d9fa15421b75b22dc5d7c9829fc75759c2b69f780"} Oct 31 04:54:59.438033 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:59.438003 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f13aaf745733eb6a8695977d9fa15421b75b22dc5d7c9829fc75759c2b69f780" Oct 31 04:54:59.438033 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:59.437838 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1a932c0-nlzns" Oct 31 04:54:59.451212 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:59.451187 5019 scope.go:122] "RemoveContainer" containerID="4422794e18d1a9e77fc037eec3c2866e0e975344d3e6f9f14be54c894a1f0d1f" Oct 31 04:54:59.452296 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:54:59.452263 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4422794e18d1a9e77fc037eec3c2866e0e975344d3e6f9f14be54c894a1f0d1f\": container with ID starting with 4422794e18d1a9e77fc037eec3c2866e0e975344d3e6f9f14be54c894a1f0d1f not found: ID does not exist" containerID="4422794e18d1a9e77fc037eec3c2866e0e975344d3e6f9f14be54c894a1f0d1f" Oct 31 04:54:59.452490 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:59.452304 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4422794e18d1a9e77fc037eec3c2866e0e975344d3e6f9f14be54c894a1f0d1f"} err="failed to get container status \"4422794e18d1a9e77fc037eec3c2866e0e975344d3e6f9f14be54c894a1f0d1f\": rpc error: code = NotFound desc = could not find container \"4422794e18d1a9e77fc037eec3c2866e0e975344d3e6f9f14be54c894a1f0d1f\": container with ID starting with 4422794e18d1a9e77fc037eec3c2866e0e975344d3e6f9f14be54c894a1f0d1f not found: ID does not exist" Oct 31 04:54:59.486777 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:59.486695 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-74fckxtnc"] Oct 31 04:54:59.494915 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:59.494874 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-74fckxtnc"] Oct 31 04:54:59.644260 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:59.644211 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bc4gchxx" podUID="23409f1d-b628-4669-a2f8-3363bdc6b09b" containerName="payload-processing" containerID="cri-o://d724386f7d82034ff29c6fa11b2b3b0c5e41f5dbbd843684c61565d45017690f" gracePeriod=25 Oct 31 04:54:59.952781 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:54:59.952750 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bc4gchxx" Oct 31 04:55:00.105437 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:00.105387 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23409f1d-b628-4669-a2f8-3363bdc6b09b-kube-api-access-qdfx5\" (UniqueName: \"kubernetes.io/projected/23409f1d-b628-4669-a2f8-3363bdc6b09b-kube-api-access-qdfx5\") pod \"23409f1d-b628-4669-a2f8-3363bdc6b09b\" (UID: \"23409f1d-b628-4669-a2f8-3363bdc6b09b\") " Oct 31 04:55:00.105670 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:00.105571 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23409f1d-b628-4669-a2f8-3363bdc6b09b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23409f1d-b628-4669-a2f8-3363bdc6b09b-plugins-config-volume\") pod \"23409f1d-b628-4669-a2f8-3363bdc6b09b\" (UID: \"23409f1d-b628-4669-a2f8-3363bdc6b09b\") " Oct 31 04:55:00.105670 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:00.105663 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23409f1d-b628-4669-a2f8-3363bdc6b09b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/23409f1d-b628-4669-a2f8-3363bdc6b09b-provider-credentials\") pod \"23409f1d-b628-4669-a2f8-3363bdc6b09b\" (UID: \"23409f1d-b628-4669-a2f8-3363bdc6b09b\") " Oct 31 04:55:00.105786 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:00.105762 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23409f1d-b628-4669-a2f8-3363bdc6b09b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/23409f1d-b628-4669-a2f8-3363bdc6b09b-routing-overlay-volume\") pod \"23409f1d-b628-4669-a2f8-3363bdc6b09b\" (UID: \"23409f1d-b628-4669-a2f8-3363bdc6b09b\") " Oct 31 04:55:00.106116 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:00.106091 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23409f1d-b628-4669-a2f8-3363bdc6b09b-plugins-config-volume" pod "23409f1d-b628-4669-a2f8-3363bdc6b09b" (UID: "23409f1d-b628-4669-a2f8-3363bdc6b09b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:55:00.106197 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:00.106172 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23409f1d-b628-4669-a2f8-3363bdc6b09b-routing-overlay-volume" pod "23409f1d-b628-4669-a2f8-3363bdc6b09b" (UID: "23409f1d-b628-4669-a2f8-3363bdc6b09b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:55:00.106523 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:00.106502 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23409f1d-b628-4669-a2f8-3363bdc6b09b-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:55:00.106614 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:00.106527 5019 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/23409f1d-b628-4669-a2f8-3363bdc6b09b-routing-overlay-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:55:00.108017 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:00.107992 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23409f1d-b628-4669-a2f8-3363bdc6b09b-provider-credentials" pod "23409f1d-b628-4669-a2f8-3363bdc6b09b" (UID: "23409f1d-b628-4669-a2f8-3363bdc6b09b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:55:00.108261 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:00.108232 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23409f1d-b628-4669-a2f8-3363bdc6b09b-kube-api-access-qdfx5" pod "23409f1d-b628-4669-a2f8-3363bdc6b09b" (UID: "23409f1d-b628-4669-a2f8-3363bdc6b09b"). InnerVolumeSpecName "kube-api-access-qdfx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:55:00.209227 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:00.209132 5019 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/23409f1d-b628-4669-a2f8-3363bdc6b09b-provider-credentials\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:55:00.209227 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:00.209175 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdfx5\" (UniqueName: \"kubernetes.io/projected/23409f1d-b628-4669-a2f8-3363bdc6b09b-kube-api-access-qdfx5\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:55:00.451752 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:00.451711 5019 generic.go:356] "Generic (PLEG): container finished" podID="23409f1d-b628-4669-a2f8-3363bdc6b09b" containerID="d724386f7d82034ff29c6fa11b2b3b0c5e41f5dbbd843684c61565d45017690f" exitCode=0 Oct 31 04:55:00.451942 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:00.451763 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bc4gchxx" Oct 31 04:55:00.451942 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:00.451797 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bc4gchxx" event={"ID":"23409f1d-b628-4669-a2f8-3363bdc6b09b","Type":"ContainerDied","Data":"d724386f7d82034ff29c6fa11b2b3b0c5e41f5dbbd843684c61565d45017690f"} Oct 31 04:55:00.451942 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:00.451859 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bc4gchxx" event={"ID":"23409f1d-b628-4669-a2f8-3363bdc6b09b","Type":"ContainerDied","Data":"97606501aa74c573fa6a91e234182114c61844cc4e6bc2037ce73eecf4a9ce2a"} Oct 31 04:55:00.451942 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:00.451892 5019 scope.go:122] "RemoveContainer" containerID="d724386f7d82034ff29c6fa11b2b3b0c5e41f5dbbd843684c61565d45017690f" Oct 31 04:55:00.465620 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:00.465595 5019 scope.go:122] "RemoveContainer" containerID="d724386f7d82034ff29c6fa11b2b3b0c5e41f5dbbd843684c61565d45017690f" Oct 31 04:55:00.465939 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:55:00.465913 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d724386f7d82034ff29c6fa11b2b3b0c5e41f5dbbd843684c61565d45017690f\": container with ID starting with d724386f7d82034ff29c6fa11b2b3b0c5e41f5dbbd843684c61565d45017690f not found: ID does not exist" containerID="d724386f7d82034ff29c6fa11b2b3b0c5e41f5dbbd843684c61565d45017690f" Oct 31 04:55:00.466063 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:00.465947 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d724386f7d82034ff29c6fa11b2b3b0c5e41f5dbbd843684c61565d45017690f"} err="failed to get container status \"d724386f7d82034ff29c6fa11b2b3b0c5e41f5dbbd843684c61565d45017690f\": rpc error: code = NotFound desc = could not find container \"d724386f7d82034ff29c6fa11b2b3b0c5e41f5dbbd843684c61565d45017690f\": container with ID starting with d724386f7d82034ff29c6fa11b2b3b0c5e41f5dbbd843684c61565d45017690f not found: ID does not exist" Oct 31 04:55:00.507469 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:00.507429 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bc4gchxx"] Oct 31 04:55:00.521134 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:00.521087 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bc4gchxx"] Oct 31 04:55:01.417429 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:01.417384 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23409f1d-b628-4669-a2f8-3363bdc6b09b" path="/var/lib/kubelet/pods/23409f1d-b628-4669-a2f8-3363bdc6b09b/volumes" Oct 31 04:55:01.419209 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:01.418177 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5606bd96-b466-42d3-9eb8-fffe43b65bff" path="/var/lib/kubelet/pods/5606bd96-b466-42d3-9eb8-fffe43b65bff/volumes" Oct 31 04:55:01.489647 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:01.489583 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7"] Oct 31 04:55:01.492828 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:01.492783 5019 cpu_manager.go:414] "containerMap: removing container" podUID="7543c659-84f2-4c09-aa0f-d517f487eaf4" containerName="revoke-keys" Oct 31 04:55:01.493062 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:01.493046 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7543c659-84f2-4c09-aa0f-d517f487eaf4" containerName="revoke-keys" Oct 31 04:55:01.493164 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:01.493153 5019 cpu_manager.go:414] "containerMap: removing container" podUID="b3743e35-69e7-4544-979d-7596c44b59a0" containerName="istio-proxy" Oct 31 04:55:01.493237 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:01.493228 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3743e35-69e7-4544-979d-7596c44b59a0" containerName="istio-proxy" Oct 31 04:55:01.493390 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:01.493317 5019 cpu_manager.go:414] "containerMap: removing container" podUID="23409f1d-b628-4669-a2f8-3363bdc6b09b" containerName="payload-processing" Oct 31 04:55:01.493519 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:01.493504 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23409f1d-b628-4669-a2f8-3363bdc6b09b" containerName="payload-processing" Oct 31 04:55:01.493608 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:01.493598 5019 cpu_manager.go:414] "containerMap: removing container" podUID="5606bd96-b466-42d3-9eb8-fffe43b65bff" containerName="payload-processing" Oct 31 04:55:01.493683 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:01.493673 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5606bd96-b466-42d3-9eb8-fffe43b65bff" containerName="payload-processing" Oct 31 04:55:01.493778 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:01.493767 5019 cpu_manager.go:414] "containerMap: removing container" podUID="22f2a6e7-c1d8-4834-80d3-805cf819e158" containerName="istio-proxy" Oct 31 04:55:01.493848 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:01.493838 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22f2a6e7-c1d8-4834-80d3-805cf819e158" containerName="istio-proxy" Oct 31 04:55:01.493952 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:01.493940 5019 cpu_manager.go:414] "containerMap: removing container" podUID="22f2a6e7-c1d8-4834-80d3-805cf819e158" containerName="istio-proxy" Oct 31 04:55:01.494027 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:01.494017 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22f2a6e7-c1d8-4834-80d3-805cf819e158" containerName="istio-proxy" Oct 31 04:55:01.494351 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:01.494313 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="7543c659-84f2-4c09-aa0f-d517f487eaf4" containerName="revoke-keys" Oct 31 04:55:01.494488 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:01.494475 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3743e35-69e7-4544-979d-7596c44b59a0" containerName="istio-proxy" Oct 31 04:55:01.494579 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:01.494568 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="23409f1d-b628-4669-a2f8-3363bdc6b09b" containerName="payload-processing" Oct 31 04:55:01.494663 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:01.494652 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="5606bd96-b466-42d3-9eb8-fffe43b65bff" containerName="payload-processing" Oct 31 04:55:01.494742 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:01.494732 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="22f2a6e7-c1d8-4834-80d3-805cf819e158" containerName="istio-proxy" Oct 31 04:55:01.494822 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:01.494813 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="22f2a6e7-c1d8-4834-80d3-805cf819e158" containerName="istio-proxy" Oct 31 04:55:01.501380 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:01.501351 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7" Oct 31 04:55:01.532855 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:01.532216 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-108ad8-openshift-default-dockercfg-5ljrf\"" Oct 31 04:55:01.543698 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:01.541700 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7"] Oct 31 04:55:01.692877 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:01.692758 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-workload-socket\") pod \"e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7\" (UID: \"7e6bd271-4285-4390-a0f9-e2fcd5a855d4\") " pod="openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7" Oct 31 04:55:01.696352 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:01.693138 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-istio-data\") pod \"e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7\" (UID: \"7e6bd271-4285-4390-a0f9-e2fcd5a855d4\") " pod="openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7" Oct 31 04:55:01.700355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:01.696657 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-istiod-ca-cert\") pod \"e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7\" (UID: \"7e6bd271-4285-4390-a0f9-e2fcd5a855d4\") " pod="openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7" Oct 31 04:55:01.708352 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:01.700696 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-istio-token\") pod \"e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7\" (UID: \"7e6bd271-4285-4390-a0f9-e2fcd5a855d4\") " pod="openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7" Oct 31 04:55:01.708733 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:01.708696 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg7hj\" (UniqueName: \"kubernetes.io/projected/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-kube-api-access-hg7hj\") pod \"e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7\" (UID: \"7e6bd271-4285-4390-a0f9-e2fcd5a855d4\") " pod="openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7" Oct 31 04:55:01.708883 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:01.708868 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-credential-socket\") pod \"e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7\" (UID: \"7e6bd271-4285-4390-a0f9-e2fcd5a855d4\") " pod="openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7" Oct 31 04:55:01.712466 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:01.712421 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-istio-podinfo\") pod \"e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7\" (UID: \"7e6bd271-4285-4390-a0f9-e2fcd5a855d4\") " pod="openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7" Oct 31 04:55:01.716481 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:01.716440 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-istio-envoy\") pod \"e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7\" (UID: \"7e6bd271-4285-4390-a0f9-e2fcd5a855d4\") " pod="openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7" Oct 31 04:55:01.720565 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:01.720525 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-workload-certs\") pod \"e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7\" (UID: \"7e6bd271-4285-4390-a0f9-e2fcd5a855d4\") " pod="openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7" Oct 31 04:55:01.829573 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:01.829518 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-istio-data\") pod \"e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7\" (UID: \"7e6bd271-4285-4390-a0f9-e2fcd5a855d4\") " pod="openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7" Oct 31 04:55:01.829906 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:01.829878 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-istiod-ca-cert\") pod \"e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7\" (UID: \"7e6bd271-4285-4390-a0f9-e2fcd5a855d4\") " pod="openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7" Oct 31 04:55:01.830077 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:01.830063 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-istio-token\") pod \"e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7\" (UID: \"7e6bd271-4285-4390-a0f9-e2fcd5a855d4\") " pod="openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7" Oct 31 04:55:01.830237 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:01.830223 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg7hj\" (UniqueName: \"kubernetes.io/projected/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-kube-api-access-hg7hj\") pod \"e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7\" (UID: \"7e6bd271-4285-4390-a0f9-e2fcd5a855d4\") " pod="openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7" Oct 31 04:55:01.830372 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:01.830355 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-credential-socket\") pod \"e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7\" (UID: \"7e6bd271-4285-4390-a0f9-e2fcd5a855d4\") " pod="openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7" Oct 31 04:55:01.830503 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:01.830488 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-istio-podinfo\") pod \"e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7\" (UID: \"7e6bd271-4285-4390-a0f9-e2fcd5a855d4\") " pod="openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7" Oct 31 04:55:01.830638 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:01.830624 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-istio-envoy\") pod \"e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7\" (UID: \"7e6bd271-4285-4390-a0f9-e2fcd5a855d4\") " pod="openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7" Oct 31 04:55:01.834356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:01.830850 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-workload-certs\") pod \"e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7\" (UID: \"7e6bd271-4285-4390-a0f9-e2fcd5a855d4\") " pod="openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7" Oct 31 04:55:01.843355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:01.834836 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-workload-socket\") pod \"e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7\" (UID: \"7e6bd271-4285-4390-a0f9-e2fcd5a855d4\") " pod="openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7" Oct 31 04:55:01.844648 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:01.844613 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-istio-data\") pod \"e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7\" (UID: \"7e6bd271-4285-4390-a0f9-e2fcd5a855d4\") " pod="openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7" Oct 31 04:55:01.845757 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:01.845082 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-workload-socket\") pod \"e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7\" (UID: \"7e6bd271-4285-4390-a0f9-e2fcd5a855d4\") " pod="openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7" Oct 31 04:55:01.857495 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:01.845291 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-credential-socket\") pod \"e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7\" (UID: \"7e6bd271-4285-4390-a0f9-e2fcd5a855d4\") " pod="openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7" Oct 31 04:55:01.857742 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:01.852464 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-istiod-ca-cert\") pod \"e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7\" (UID: \"7e6bd271-4285-4390-a0f9-e2fcd5a855d4\") " pod="openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7" Oct 31 04:55:01.858162 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:01.858137 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-workload-certs\") pod \"e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7\" (UID: \"7e6bd271-4285-4390-a0f9-e2fcd5a855d4\") " pod="openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7" Oct 31 04:55:01.871223 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:01.871146 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-istio-envoy\") pod \"e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7\" (UID: \"7e6bd271-4285-4390-a0f9-e2fcd5a855d4\") " pod="openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7" Oct 31 04:55:01.877430 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:01.877007 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-istio-podinfo\") pod \"e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7\" (UID: \"7e6bd271-4285-4390-a0f9-e2fcd5a855d4\") " pod="openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7" Oct 31 04:55:01.878529 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:01.878465 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg7hj\" (UniqueName: \"kubernetes.io/projected/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-kube-api-access-hg7hj\") pod \"e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7\" (UID: \"7e6bd271-4285-4390-a0f9-e2fcd5a855d4\") " pod="openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7" Oct 31 04:55:01.909088 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:01.909050 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-istio-token\") pod \"e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7\" (UID: \"7e6bd271-4285-4390-a0f9-e2fcd5a855d4\") " pod="openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7" Oct 31 04:55:02.152898 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:02.152853 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7" Oct 31 04:55:02.667487 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:02.666207 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7"] Oct 31 04:55:02.670617 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:55:02.670567 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e6bd271_4285_4390_a0f9_e2fcd5a855d4.slice/crio-b945e0c12d1b0ea940108f15303837354637d0b937cb86ff0b38ecd8d2a7133d WatchSource:0}: Error finding container b945e0c12d1b0ea940108f15303837354637d0b937cb86ff0b38ecd8d2a7133d: Status 404 returned error can't find the container with id b945e0c12d1b0ea940108f15303837354637d0b937cb86ff0b38ecd8d2a7133d Oct 31 04:55:02.674554 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:02.674498 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:55:02.674714 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:02.674613 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:55:02.674714 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:02.674691 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:55:02.801152 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:02.801076 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-cb4fj785m" podUID="e5cc6ba4-689b-49dc-982a-a4812e64befb" containerName="payload-processing" containerID="cri-o://50a419b59df09321c9ec81fe8a38261098415cb8e568909f3d2230567e4150d9" gracePeriod=25 Oct 31 04:55:03.579209 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:03.578781 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7" event={"ID":"7e6bd271-4285-4390-a0f9-e2fcd5a855d4","Type":"ContainerStarted","Data":"36ce6b5cb4ca668ed880432272f83fc12a7095d8cf42c073613a404058d82fe8"} Oct 31 04:55:03.579209 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:03.578862 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7" event={"ID":"7e6bd271-4285-4390-a0f9-e2fcd5a855d4","Type":"ContainerStarted","Data":"b945e0c12d1b0ea940108f15303837354637d0b937cb86ff0b38ecd8d2a7133d"} Oct 31 04:55:03.582150 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:03.581923 5019 generic.go:356] "Generic (PLEG): container finished" podID="e5cc6ba4-689b-49dc-982a-a4812e64befb" containerID="50a419b59df09321c9ec81fe8a38261098415cb8e568909f3d2230567e4150d9" exitCode=0 Oct 31 04:55:03.582531 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:03.582228 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-cb4fj785m" event={"ID":"e5cc6ba4-689b-49dc-982a-a4812e64befb","Type":"ContainerDied","Data":"50a419b59df09321c9ec81fe8a38261098415cb8e568909f3d2230567e4150d9"} Oct 31 04:55:03.582531 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:03.582300 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-cb4fj785m" event={"ID":"e5cc6ba4-689b-49dc-982a-a4812e64befb","Type":"ContainerDied","Data":"75ee894228c29acc9472a2d262a880e3b616a0045ceb182b134141131c947f1a"} Oct 31 04:55:03.582531 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:03.582326 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75ee894228c29acc9472a2d262a880e3b616a0045ceb182b134141131c947f1a" Oct 31 04:55:03.582732 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:03.582693 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-cb4fj785m" Oct 31 04:55:03.643151 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:03.643073 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7" podStartSLOduration=2.643049849 podStartE2EDuration="2.643049849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:55:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:55:03.63757471 +0000 UTC m=+3058.393912760" watchObservedRunningTime="2026-10-31 04:55:03.643049849 +0000 UTC m=+3058.399387888" Oct 31 04:55:03.651687 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:03.651640 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5cc6ba4-689b-49dc-982a-a4812e64befb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e5cc6ba4-689b-49dc-982a-a4812e64befb-routing-overlay-volume\") pod \"e5cc6ba4-689b-49dc-982a-a4812e64befb\" (UID: \"e5cc6ba4-689b-49dc-982a-a4812e64befb\") " Oct 31 04:55:03.651941 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:03.651922 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5cc6ba4-689b-49dc-982a-a4812e64befb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e5cc6ba4-689b-49dc-982a-a4812e64befb-provider-credentials\") pod \"e5cc6ba4-689b-49dc-982a-a4812e64befb\" (UID: \"e5cc6ba4-689b-49dc-982a-a4812e64befb\") " Oct 31 04:55:03.652039 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:03.652025 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5cc6ba4-689b-49dc-982a-a4812e64befb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5cc6ba4-689b-49dc-982a-a4812e64befb-plugins-config-volume\") pod \"e5cc6ba4-689b-49dc-982a-a4812e64befb\" (UID: \"e5cc6ba4-689b-49dc-982a-a4812e64befb\") " Oct 31 04:55:03.652108 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:03.652085 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5cc6ba4-689b-49dc-982a-a4812e64befb-kube-api-access-vfvcl\" (UniqueName: \"kubernetes.io/projected/e5cc6ba4-689b-49dc-982a-a4812e64befb-kube-api-access-vfvcl\") pod \"e5cc6ba4-689b-49dc-982a-a4812e64befb\" (UID: \"e5cc6ba4-689b-49dc-982a-a4812e64befb\") " Oct 31 04:55:03.652545 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:03.652510 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5cc6ba4-689b-49dc-982a-a4812e64befb-plugins-config-volume" pod "e5cc6ba4-689b-49dc-982a-a4812e64befb" (UID: "e5cc6ba4-689b-49dc-982a-a4812e64befb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:55:03.653256 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:03.653225 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5cc6ba4-689b-49dc-982a-a4812e64befb-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:55:03.654320 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:03.654293 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5cc6ba4-689b-49dc-982a-a4812e64befb-routing-overlay-volume" pod "e5cc6ba4-689b-49dc-982a-a4812e64befb" (UID: "e5cc6ba4-689b-49dc-982a-a4812e64befb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:55:03.655550 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:03.655503 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5cc6ba4-689b-49dc-982a-a4812e64befb-kube-api-access-vfvcl" pod "e5cc6ba4-689b-49dc-982a-a4812e64befb" (UID: "e5cc6ba4-689b-49dc-982a-a4812e64befb"). InnerVolumeSpecName "kube-api-access-vfvcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:55:03.657055 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:03.657023 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5cc6ba4-689b-49dc-982a-a4812e64befb-provider-credentials" pod "e5cc6ba4-689b-49dc-982a-a4812e64befb" (UID: "e5cc6ba4-689b-49dc-982a-a4812e64befb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:55:03.755078 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:03.755036 5019 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e5cc6ba4-689b-49dc-982a-a4812e64befb-routing-overlay-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:55:03.755597 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:03.755552 5019 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e5cc6ba4-689b-49dc-982a-a4812e64befb-provider-credentials\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:55:03.755700 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:03.755598 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfvcl\" (UniqueName: \"kubernetes.io/projected/e5cc6ba4-689b-49dc-982a-a4812e64befb-kube-api-access-vfvcl\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:55:04.066421 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:04.064390 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42907744-t2knt"] Oct 31 04:55:04.105358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:04.090297 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42907744-t2knt"] Oct 31 04:55:04.164356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:04.163906 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7" Oct 31 04:55:04.300358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:04.295095 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7" Oct 31 04:55:04.666358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:04.665589 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-cb4fj785m" Oct 31 04:55:04.683355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:04.678786 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7" Oct 31 04:55:04.718354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:04.690028 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7" Oct 31 04:55:05.122918 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:05.122860 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-cb4fj785m"] Oct 31 04:55:05.132643 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:05.132592 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-cb4fj785m"] Oct 31 04:55:05.413897 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:05.413816 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3d6fe36-f010-4322-95af-a7e419f76d8d" path="/var/lib/kubelet/pods/c3d6fe36-f010-4322-95af-a7e419f76d8d/volumes" Oct 31 04:55:05.414540 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:05.414521 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5cc6ba4-689b-49dc-982a-a4812e64befb" path="/var/lib/kubelet/pods/e5cc6ba4-689b-49dc-982a-a4812e64befb/volumes" Oct 31 04:55:10.168628 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:10.168584 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff105070-qmjw7"] Oct 31 04:55:10.169480 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:10.169463 5019 cpu_manager.go:414] "containerMap: removing container" podUID="e5cc6ba4-689b-49dc-982a-a4812e64befb" containerName="payload-processing" Oct 31 04:55:10.169480 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:10.169481 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5cc6ba4-689b-49dc-982a-a4812e64befb" containerName="payload-processing" Oct 31 04:55:10.169615 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:10.169605 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5cc6ba4-689b-49dc-982a-a4812e64befb" containerName="payload-processing" Oct 31 04:55:10.175307 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:10.175276 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff105070-qmjw7" Oct 31 04:55:10.194392 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:10.194348 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff105070-qmjw7"] Oct 31 04:55:10.197598 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:10.197564 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:55:10.198256 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:10.197563 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:55:10.198479 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:10.197720 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:55:10.198609 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:10.197927 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:55:10.227393 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:10.227291 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5091375-a090-497a-9fc5-50c0a0a323af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff105070-qmjw7\" (UID: \"d5091375-a090-497a-9fc5-50c0a0a323af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff105070-qmjw7" Oct 31 04:55:10.227672 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:10.227648 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mclls\" (UniqueName: \"kubernetes.io/projected/d5091375-a090-497a-9fc5-50c0a0a323af-kube-api-access-mclls\") pod \"maas-api-revoke-sub-ff105070-qmjw7\" (UID: \"d5091375-a090-497a-9fc5-50c0a0a323af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff105070-qmjw7" Oct 31 04:55:10.227929 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:10.227912 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5091375-a090-497a-9fc5-50c0a0a323af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff105070-qmjw7\" (UID: \"d5091375-a090-497a-9fc5-50c0a0a323af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff105070-qmjw7" Oct 31 04:55:10.295101 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:10.295060 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-108ad8-8d8c4c667-rbd9x"] Oct 31 04:55:10.300437 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:10.300398 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-108ad8-8d8c4c667-rbd9x" Oct 31 04:55:10.317770 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:10.317724 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-108ad8\"" Oct 31 04:55:10.318495 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:10.318467 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-108ad8-dockercfg-z2t7r\"" Oct 31 04:55:10.321690 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:10.321610 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-108ad8-8d8c4c667-rbd9x"] Oct 31 04:55:10.329543 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:10.329504 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5091375-a090-497a-9fc5-50c0a0a323af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff105070-qmjw7\" (UID: \"d5091375-a090-497a-9fc5-50c0a0a323af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff105070-qmjw7" Oct 31 04:55:10.329791 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:10.329774 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mclls\" (UniqueName: \"kubernetes.io/projected/d5091375-a090-497a-9fc5-50c0a0a323af-kube-api-access-mclls\") pod \"maas-api-revoke-sub-ff105070-qmjw7\" (UID: \"d5091375-a090-497a-9fc5-50c0a0a323af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff105070-qmjw7" Oct 31 04:55:10.329896 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:10.329881 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5091375-a090-497a-9fc5-50c0a0a323af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff105070-qmjw7\" (UID: \"d5091375-a090-497a-9fc5-50c0a0a323af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff105070-qmjw7" Oct 31 04:55:10.330520 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:10.330490 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5091375-a090-497a-9fc5-50c0a0a323af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff105070-qmjw7\" (UID: \"d5091375-a090-497a-9fc5-50c0a0a323af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff105070-qmjw7" Oct 31 04:55:10.333032 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:10.333005 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5091375-a090-497a-9fc5-50c0a0a323af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff105070-qmjw7\" (UID: \"d5091375-a090-497a-9fc5-50c0a0a323af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff105070-qmjw7" Oct 31 04:55:10.474427 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:10.461124 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fe19836-7804-40c2-b436-c947e33e979a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-108ad8-8d8c4c667-rbd9x\" (UID: \"4fe19836-7804-40c2-b436-c947e33e979a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-108ad8-8d8c4c667-rbd9x" Oct 31 04:55:10.474427 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:10.461215 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlw76\" (UniqueName: \"kubernetes.io/projected/4fe19836-7804-40c2-b436-c947e33e979a-kube-api-access-wlw76\") pod \"payload-pre-processing-e2e-inf-a-108ad8-8d8c4c667-rbd9x\" (UID: \"4fe19836-7804-40c2-b436-c947e33e979a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-108ad8-8d8c4c667-rbd9x" Oct 31 04:55:10.541372 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:10.538880 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-108ad8-7495778969-49t8z"] Oct 31 04:55:10.567848 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:10.567808 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlw76\" (UniqueName: \"kubernetes.io/projected/4fe19836-7804-40c2-b436-c947e33e979a-kube-api-access-wlw76\") pod \"payload-pre-processing-e2e-inf-a-108ad8-8d8c4c667-rbd9x\" (UID: \"4fe19836-7804-40c2-b436-c947e33e979a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-108ad8-8d8c4c667-rbd9x" Oct 31 04:55:10.568956 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:10.568925 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fe19836-7804-40c2-b436-c947e33e979a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-108ad8-8d8c4c667-rbd9x\" (UID: \"4fe19836-7804-40c2-b436-c947e33e979a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-108ad8-8d8c4c667-rbd9x" Oct 31 04:55:10.570264 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:10.568176 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-108ad8-7495778969-49t8z" Oct 31 04:55:10.578778 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:10.570092 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fe19836-7804-40c2-b436-c947e33e979a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-108ad8-8d8c4c667-rbd9x\" (UID: \"4fe19836-7804-40c2-b436-c947e33e979a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-108ad8-8d8c4c667-rbd9x" Oct 31 04:55:10.624201 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:10.623878 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mclls\" (UniqueName: \"kubernetes.io/projected/d5091375-a090-497a-9fc5-50c0a0a323af-kube-api-access-mclls\") pod \"maas-api-revoke-sub-ff105070-qmjw7\" (UID: \"d5091375-a090-497a-9fc5-50c0a0a323af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff105070-qmjw7" Oct 31 04:55:10.675431 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:10.675390 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9abd61e1-2bf2-43fb-b106-132dedc0e2d9-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-108ad8-7495778969-49t8z\" (UID: \"9abd61e1-2bf2-43fb-b106-132dedc0e2d9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-108ad8-7495778969-49t8z" Oct 31 04:55:10.675858 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:10.675833 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2shsr\" (UniqueName: \"kubernetes.io/projected/9abd61e1-2bf2-43fb-b106-132dedc0e2d9-kube-api-access-2shsr\") pod \"payload-processing-e2e-inf-a-108ad8-7495778969-49t8z\" (UID: \"9abd61e1-2bf2-43fb-b106-132dedc0e2d9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-108ad8-7495778969-49t8z" Oct 31 04:55:10.732994 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:10.732919 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlw76\" (UniqueName: \"kubernetes.io/projected/4fe19836-7804-40c2-b436-c947e33e979a-kube-api-access-wlw76\") pod \"payload-pre-processing-e2e-inf-a-108ad8-8d8c4c667-rbd9x\" (UID: \"4fe19836-7804-40c2-b436-c947e33e979a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-108ad8-8d8c4c667-rbd9x" Oct 31 04:55:10.734000 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:10.733969 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-108ad8-7495778969-49t8z"] Oct 31 04:55:10.778189 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:10.778134 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9abd61e1-2bf2-43fb-b106-132dedc0e2d9-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-108ad8-7495778969-49t8z\" (UID: \"9abd61e1-2bf2-43fb-b106-132dedc0e2d9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-108ad8-7495778969-49t8z" Oct 31 04:55:10.778436 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:10.778260 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2shsr\" (UniqueName: \"kubernetes.io/projected/9abd61e1-2bf2-43fb-b106-132dedc0e2d9-kube-api-access-2shsr\") pod \"payload-processing-e2e-inf-a-108ad8-7495778969-49t8z\" (UID: \"9abd61e1-2bf2-43fb-b106-132dedc0e2d9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-108ad8-7495778969-49t8z" Oct 31 04:55:10.779132 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:10.779107 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9abd61e1-2bf2-43fb-b106-132dedc0e2d9-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-108ad8-7495778969-49t8z\" (UID: \"9abd61e1-2bf2-43fb-b106-132dedc0e2d9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-108ad8-7495778969-49t8z" Oct 31 04:55:10.800416 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:10.800371 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff105070-qmjw7" Oct 31 04:55:10.802753 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:10.802603 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2shsr\" (UniqueName: \"kubernetes.io/projected/9abd61e1-2bf2-43fb-b106-132dedc0e2d9-kube-api-access-2shsr\") pod \"payload-processing-e2e-inf-a-108ad8-7495778969-49t8z\" (UID: \"9abd61e1-2bf2-43fb-b106-132dedc0e2d9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-108ad8-7495778969-49t8z" Oct 31 04:55:10.923656 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:10.923487 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-108ad8-8d8c4c667-rbd9x" Oct 31 04:55:10.986262 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:10.982295 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-108ad8-7495778969-49t8z" Oct 31 04:55:11.124479 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:55:11.124428 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5091375_a090_497a_9fc5_50c0a0a323af.slice/crio-06fa1c1a60bbb654f69f518110b8ccf07d14f24941e95dbfe7f40c3cc61287e7 WatchSource:0}: Error finding container 06fa1c1a60bbb654f69f518110b8ccf07d14f24941e95dbfe7f40c3cc61287e7: Status 404 returned error can't find the container with id 06fa1c1a60bbb654f69f518110b8ccf07d14f24941e95dbfe7f40c3cc61287e7 Oct 31 04:55:11.127523 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:11.127497 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff105070-qmjw7"] Oct 31 04:55:11.257939 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:11.257823 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-108ad8-8d8c4c667-rbd9x"] Oct 31 04:55:11.351938 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:55:11.351898 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9abd61e1_2bf2_43fb_b106_132dedc0e2d9.slice/crio-ef4d98776be47b578114a142adb7f7dac5d2cc4eeaa3a2bb40f947908ba40345 WatchSource:0}: Error finding container ef4d98776be47b578114a142adb7f7dac5d2cc4eeaa3a2bb40f947908ba40345: Status 404 returned error can't find the container with id ef4d98776be47b578114a142adb7f7dac5d2cc4eeaa3a2bb40f947908ba40345 Oct 31 04:55:11.352298 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:11.352271 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-108ad8-7495778969-49t8z"] Oct 31 04:55:11.763891 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:11.763849 5019 scope.go:122] "RemoveContainer" containerID="e47ebc9f47bdbb91987d85f254cd74d3f8219e7a49151b0b60711a9000d1d103" Oct 31 04:55:11.777000 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:11.776972 5019 scope.go:122] "RemoveContainer" containerID="3cd5188fca2efcd76523196c962784dbdebae23e6a00f68efcc9c916c1ef93d7" Oct 31 04:55:11.789404 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:11.789373 5019 scope.go:122] "RemoveContainer" containerID="ea6d20035b6395b44d2a168b71eef4d48063b1eb33345e6da61888575cc7f9f6" Oct 31 04:55:11.899821 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:11.899768 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-108ad8-7495778969-49t8z" event={"ID":"9abd61e1-2bf2-43fb-b106-132dedc0e2d9","Type":"ContainerStarted","Data":"78b332e04241672bbc31d92d029cb85006e662b9234a95dd0e550caec94fe215"} Oct 31 04:55:11.900038 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:11.899882 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-108ad8-7495778969-49t8z" event={"ID":"9abd61e1-2bf2-43fb-b106-132dedc0e2d9","Type":"ContainerStarted","Data":"ef4d98776be47b578114a142adb7f7dac5d2cc4eeaa3a2bb40f947908ba40345"} Oct 31 04:55:11.900038 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:11.899954 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-108ad8-7495778969-49t8z" Oct 31 04:55:11.908166 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:11.908119 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff105070-qmjw7" event={"ID":"d5091375-a090-497a-9fc5-50c0a0a323af","Type":"ContainerStarted","Data":"898a84116c5821ebc205f5de1206e03aefd95dc52dca68b9b3698f7619f9faf3"} Oct 31 04:55:11.908474 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:11.908194 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff105070-qmjw7" event={"ID":"d5091375-a090-497a-9fc5-50c0a0a323af","Type":"ContainerStarted","Data":"06fa1c1a60bbb654f69f518110b8ccf07d14f24941e95dbfe7f40c3cc61287e7"} Oct 31 04:55:11.911719 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:11.911683 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-108ad8-8d8c4c667-rbd9x" event={"ID":"4fe19836-7804-40c2-b436-c947e33e979a","Type":"ContainerStarted","Data":"64a2cc1fcd3b3a4b931f7646d4296059350aef5b872a7075400ad3bd73c6bfdb"} Oct 31 04:55:11.911863 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:11.911748 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-108ad8-8d8c4c667-rbd9x" event={"ID":"4fe19836-7804-40c2-b436-c947e33e979a","Type":"ContainerStarted","Data":"9be2f4a3dfb05cee0458cef3c639a589dc100e056198b05d980690456847031c"} Oct 31 04:55:11.911863 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:11.911778 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-108ad8-8d8c4c667-rbd9x" Oct 31 04:55:11.944735 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:11.944615 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-108ad8-7495778969-49t8z" podStartSLOduration=1.944600082 podStartE2EDuration="1.944600082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:55:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:55:11.939507578 +0000 UTC m=+3066.695845619" watchObservedRunningTime="2026-10-31 04:55:11.944600082 +0000 UTC m=+3066.700938122" Oct 31 04:55:11.981087 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:11.981024 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-108ad8-8d8c4c667-rbd9x" podStartSLOduration=1.981002767 podStartE2EDuration="1.981002767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:55:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:55:11.978867818 +0000 UTC m=+3066.735205874" watchObservedRunningTime="2026-10-31 04:55:11.981002767 +0000 UTC m=+3066.737340808" Oct 31 04:55:12.023159 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:12.023095 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff105070-qmjw7" podStartSLOduration=2.02307863 podStartE2EDuration="2.02307863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:55:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:55:12.019898523 +0000 UTC m=+3066.776236563" watchObservedRunningTime="2026-10-31 04:55:12.02307863 +0000 UTC m=+3066.779416782" Oct 31 04:55:12.925713 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:12.925676 5019 generic.go:356] "Generic (PLEG): container finished" podID="d5091375-a090-497a-9fc5-50c0a0a323af" containerID="898a84116c5821ebc205f5de1206e03aefd95dc52dca68b9b3698f7619f9faf3" exitCode=0 Oct 31 04:55:12.926158 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:12.925763 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff105070-qmjw7" event={"ID":"d5091375-a090-497a-9fc5-50c0a0a323af","Type":"ContainerDied","Data":"898a84116c5821ebc205f5de1206e03aefd95dc52dca68b9b3698f7619f9faf3"} Oct 31 04:55:14.148996 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:14.148964 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff105070-qmjw7" Oct 31 04:55:14.267653 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:14.267603 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5091375-a090-497a-9fc5-50c0a0a323af-kube-api-access-mclls\" (UniqueName: \"kubernetes.io/projected/d5091375-a090-497a-9fc5-50c0a0a323af-kube-api-access-mclls\") pod \"d5091375-a090-497a-9fc5-50c0a0a323af\" (UID: \"d5091375-a090-497a-9fc5-50c0a0a323af\") " Oct 31 04:55:14.268225 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:14.268197 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5091375-a090-497a-9fc5-50c0a0a323af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5091375-a090-497a-9fc5-50c0a0a323af-maas-api-openshift-ca\") pod \"d5091375-a090-497a-9fc5-50c0a0a323af\" (UID: \"d5091375-a090-497a-9fc5-50c0a0a323af\") " Oct 31 04:55:14.268534 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:14.268514 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5091375-a090-497a-9fc5-50c0a0a323af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5091375-a090-497a-9fc5-50c0a0a323af-maas-api-xks-ca\") pod \"d5091375-a090-497a-9fc5-50c0a0a323af\" (UID: \"d5091375-a090-497a-9fc5-50c0a0a323af\") " Oct 31 04:55:14.276355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:14.276291 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5091375-a090-497a-9fc5-50c0a0a323af-maas-api-openshift-ca" pod "d5091375-a090-497a-9fc5-50c0a0a323af" (UID: "d5091375-a090-497a-9fc5-50c0a0a323af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:55:14.281451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:14.280635 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5091375-a090-497a-9fc5-50c0a0a323af-maas-api-xks-ca" pod "d5091375-a090-497a-9fc5-50c0a0a323af" (UID: "d5091375-a090-497a-9fc5-50c0a0a323af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:55:14.281451 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:14.281045 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5091375-a090-497a-9fc5-50c0a0a323af-kube-api-access-mclls" pod "d5091375-a090-497a-9fc5-50c0a0a323af" (UID: "d5091375-a090-497a-9fc5-50c0a0a323af"). InnerVolumeSpecName "kube-api-access-mclls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:55:14.371454 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:14.371410 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5091375-a090-497a-9fc5-50c0a0a323af-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:55:14.371678 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:14.371478 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mclls\" (UniqueName: \"kubernetes.io/projected/d5091375-a090-497a-9fc5-50c0a0a323af-kube-api-access-mclls\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:55:14.371678 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:14.371497 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5091375-a090-497a-9fc5-50c0a0a323af-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:55:14.946196 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:14.946164 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff105070-qmjw7" Oct 31 04:55:14.946465 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:14.946193 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff105070-qmjw7" event={"ID":"d5091375-a090-497a-9fc5-50c0a0a323af","Type":"ContainerDied","Data":"06fa1c1a60bbb654f69f518110b8ccf07d14f24941e95dbfe7f40c3cc61287e7"} Oct 31 04:55:14.946465 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:14.946241 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06fa1c1a60bbb654f69f518110b8ccf07d14f24941e95dbfe7f40c3cc61287e7" Oct 31 04:55:20.207252 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:20.207201 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-04d111-f77b9866c-2mzsk"] Oct 31 04:55:20.208182 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:20.208160 5019 cpu_manager.go:414] "containerMap: removing container" podUID="d5091375-a090-497a-9fc5-50c0a0a323af" containerName="revoke-keys" Oct 31 04:55:20.208182 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:20.208181 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5091375-a090-497a-9fc5-50c0a0a323af" containerName="revoke-keys" Oct 31 04:55:20.208325 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:20.208311 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5091375-a090-497a-9fc5-50c0a0a323af" containerName="revoke-keys" Oct 31 04:55:20.213806 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:20.213771 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-04d111-f77b9866c-2mzsk" Oct 31 04:55:20.230712 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:20.229054 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-04d111-dockercfg-kk5vn\"" Oct 31 04:55:20.236284 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:20.233531 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-04d111-f77b9866c-2mzsk"] Oct 31 04:55:20.236667 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:20.236631 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-04d111\"" Oct 31 04:55:20.279217 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:20.279170 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-04d111-76d4bc867-jmfjn"] Oct 31 04:55:20.285862 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:20.285827 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04d111-76d4bc867-jmfjn" Oct 31 04:55:20.308057 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:20.306435 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-04d111-76d4bc867-jmfjn"] Oct 31 04:55:20.395357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:20.392817 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kkk9\" (UniqueName: \"kubernetes.io/projected/bb04ce0c-726f-4b27-b588-1b30fee2c0a0-kube-api-access-7kkk9\") pod \"payload-processing-e2e-inf-a-04d111-f77b9866c-2mzsk\" (UID: \"bb04ce0c-726f-4b27-b588-1b30fee2c0a0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-04d111-f77b9866c-2mzsk" Oct 31 04:55:20.395357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:20.393008 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb04ce0c-726f-4b27-b588-1b30fee2c0a0-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-04d111-f77b9866c-2mzsk\" (UID: \"bb04ce0c-726f-4b27-b588-1b30fee2c0a0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-04d111-f77b9866c-2mzsk" Oct 31 04:55:20.589356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:20.538013 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fb383ce-a4bd-4455-b351-34bc277bc3e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-04d111-76d4bc867-jmfjn\" (UID: \"4fb383ce-a4bd-4455-b351-34bc277bc3e5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04d111-76d4bc867-jmfjn" Oct 31 04:55:20.589356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:20.566652 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kkk9\" (UniqueName: \"kubernetes.io/projected/bb04ce0c-726f-4b27-b588-1b30fee2c0a0-kube-api-access-7kkk9\") pod \"payload-processing-e2e-inf-a-04d111-f77b9866c-2mzsk\" (UID: \"bb04ce0c-726f-4b27-b588-1b30fee2c0a0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-04d111-f77b9866c-2mzsk" Oct 31 04:55:20.589356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:20.566735 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clh9s\" (UniqueName: \"kubernetes.io/projected/4fb383ce-a4bd-4455-b351-34bc277bc3e5-kube-api-access-clh9s\") pod \"payload-pre-processing-e2e-inf-a-04d111-76d4bc867-jmfjn\" (UID: \"4fb383ce-a4bd-4455-b351-34bc277bc3e5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04d111-76d4bc867-jmfjn" Oct 31 04:55:20.589356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:20.566905 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb04ce0c-726f-4b27-b588-1b30fee2c0a0-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-04d111-f77b9866c-2mzsk\" (UID: \"bb04ce0c-726f-4b27-b588-1b30fee2c0a0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-04d111-f77b9866c-2mzsk" Oct 31 04:55:20.641366 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:20.592745 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb04ce0c-726f-4b27-b588-1b30fee2c0a0-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-04d111-f77b9866c-2mzsk\" (UID: \"bb04ce0c-726f-4b27-b588-1b30fee2c0a0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-04d111-f77b9866c-2mzsk" Oct 31 04:55:20.678851 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:20.678523 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fb383ce-a4bd-4455-b351-34bc277bc3e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-04d111-76d4bc867-jmfjn\" (UID: \"4fb383ce-a4bd-4455-b351-34bc277bc3e5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04d111-76d4bc867-jmfjn" Oct 31 04:55:20.679067 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:20.679036 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clh9s\" (UniqueName: \"kubernetes.io/projected/4fb383ce-a4bd-4455-b351-34bc277bc3e5-kube-api-access-clh9s\") pod \"payload-pre-processing-e2e-inf-a-04d111-76d4bc867-jmfjn\" (UID: \"4fb383ce-a4bd-4455-b351-34bc277bc3e5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04d111-76d4bc867-jmfjn" Oct 31 04:55:20.687985 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:20.679414 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fb383ce-a4bd-4455-b351-34bc277bc3e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-04d111-76d4bc867-jmfjn\" (UID: \"4fb383ce-a4bd-4455-b351-34bc277bc3e5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04d111-76d4bc867-jmfjn" Oct 31 04:55:20.695364 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:20.694949 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kkk9\" (UniqueName: \"kubernetes.io/projected/bb04ce0c-726f-4b27-b588-1b30fee2c0a0-kube-api-access-7kkk9\") pod \"payload-processing-e2e-inf-a-04d111-f77b9866c-2mzsk\" (UID: \"bb04ce0c-726f-4b27-b588-1b30fee2c0a0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-04d111-f77b9866c-2mzsk" Oct 31 04:55:20.707317 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:20.707281 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clh9s\" (UniqueName: \"kubernetes.io/projected/4fb383ce-a4bd-4455-b351-34bc277bc3e5-kube-api-access-clh9s\") pod \"payload-pre-processing-e2e-inf-a-04d111-76d4bc867-jmfjn\" (UID: \"4fb383ce-a4bd-4455-b351-34bc277bc3e5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04d111-76d4bc867-jmfjn" Oct 31 04:55:20.843547 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:20.843437 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-04d111-f77b9866c-2mzsk" Oct 31 04:55:20.950774 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:20.950728 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04d111-76d4bc867-jmfjn" Oct 31 04:55:21.206206 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:55:21.206165 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb04ce0c_726f_4b27_b588_1b30fee2c0a0.slice/crio-1c79ac1a300ef7f9df9c0217585000226009057685fdb6ff6dd95809ddcadf13 WatchSource:0}: Error finding container 1c79ac1a300ef7f9df9c0217585000226009057685fdb6ff6dd95809ddcadf13: Status 404 returned error can't find the container with id 1c79ac1a300ef7f9df9c0217585000226009057685fdb6ff6dd95809ddcadf13 Oct 31 04:55:21.209841 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:21.209800 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-04d111-f77b9866c-2mzsk"] Oct 31 04:55:21.244893 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:21.244857 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6eba4874-6r5zg"] Oct 31 04:55:21.252018 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:21.251983 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eba4874-6r5zg" Oct 31 04:55:21.269852 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:21.269819 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 04:55:21.270068 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:21.269890 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:55:21.270068 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:21.269907 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:55:21.270068 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:21.269831 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:55:21.270689 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:21.270662 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6eba4874-6r5zg"] Oct 31 04:55:21.297692 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:21.297661 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-04d111-76d4bc867-jmfjn"] Oct 31 04:55:21.300514 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:55:21.300475 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fb383ce_a4bd_4455_b351_34bc277bc3e5.slice/crio-2fa4f4e3ab947dfc5f74bf8398e6c94cf1449222faa51fd4597a8d1ab88fb03d WatchSource:0}: Error finding container 2fa4f4e3ab947dfc5f74bf8398e6c94cf1449222faa51fd4597a8d1ab88fb03d: Status 404 returned error can't find the container with id 2fa4f4e3ab947dfc5f74bf8398e6c94cf1449222faa51fd4597a8d1ab88fb03d Oct 31 04:55:21.336532 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:21.336479 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n65dg\" (UniqueName: \"kubernetes.io/projected/1c60e764-ed6c-48dc-b6d0-abffbbdcdc3c-kube-api-access-n65dg\") pod \"maas-api-revoke-sub-6eba4874-6r5zg\" (UID: \"1c60e764-ed6c-48dc-b6d0-abffbbdcdc3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eba4874-6r5zg" Oct 31 04:55:21.336726 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:21.336619 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c60e764-ed6c-48dc-b6d0-abffbbdcdc3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6eba4874-6r5zg\" (UID: \"1c60e764-ed6c-48dc-b6d0-abffbbdcdc3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eba4874-6r5zg" Oct 31 04:55:21.336977 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:21.336959 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c60e764-ed6c-48dc-b6d0-abffbbdcdc3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6eba4874-6r5zg\" (UID: \"1c60e764-ed6c-48dc-b6d0-abffbbdcdc3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eba4874-6r5zg" Oct 31 04:55:21.438208 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:21.438165 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c60e764-ed6c-48dc-b6d0-abffbbdcdc3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6eba4874-6r5zg\" (UID: \"1c60e764-ed6c-48dc-b6d0-abffbbdcdc3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eba4874-6r5zg" Oct 31 04:55:21.438208 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:21.438209 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n65dg\" (UniqueName: \"kubernetes.io/projected/1c60e764-ed6c-48dc-b6d0-abffbbdcdc3c-kube-api-access-n65dg\") pod \"maas-api-revoke-sub-6eba4874-6r5zg\" (UID: \"1c60e764-ed6c-48dc-b6d0-abffbbdcdc3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eba4874-6r5zg" Oct 31 04:55:21.438576 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:21.438264 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c60e764-ed6c-48dc-b6d0-abffbbdcdc3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6eba4874-6r5zg\" (UID: \"1c60e764-ed6c-48dc-b6d0-abffbbdcdc3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eba4874-6r5zg" Oct 31 04:55:21.438932 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:21.438902 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c60e764-ed6c-48dc-b6d0-abffbbdcdc3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6eba4874-6r5zg\" (UID: \"1c60e764-ed6c-48dc-b6d0-abffbbdcdc3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eba4874-6r5zg" Oct 31 04:55:21.440930 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:21.440907 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c60e764-ed6c-48dc-b6d0-abffbbdcdc3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6eba4874-6r5zg\" (UID: \"1c60e764-ed6c-48dc-b6d0-abffbbdcdc3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eba4874-6r5zg" Oct 31 04:55:21.461819 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:21.461736 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n65dg\" (UniqueName: \"kubernetes.io/projected/1c60e764-ed6c-48dc-b6d0-abffbbdcdc3c-kube-api-access-n65dg\") pod \"maas-api-revoke-sub-6eba4874-6r5zg\" (UID: \"1c60e764-ed6c-48dc-b6d0-abffbbdcdc3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eba4874-6r5zg" Oct 31 04:55:21.583270 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:21.583230 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eba4874-6r5zg" Oct 31 04:55:21.894000 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:55:21.893964 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c60e764_ed6c_48dc_b6d0_abffbbdcdc3c.slice/crio-26b8b220c20405cbbce4e199be8847aac7236b3191d8cd2e5f62be2f345b1de5 WatchSource:0}: Error finding container 26b8b220c20405cbbce4e199be8847aac7236b3191d8cd2e5f62be2f345b1de5: Status 404 returned error can't find the container with id 26b8b220c20405cbbce4e199be8847aac7236b3191d8cd2e5f62be2f345b1de5 Oct 31 04:55:21.894577 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:21.894550 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6eba4874-6r5zg"] Oct 31 04:55:22.029863 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:22.029814 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-04d111-f77b9866c-2mzsk" event={"ID":"bb04ce0c-726f-4b27-b588-1b30fee2c0a0","Type":"ContainerStarted","Data":"b731bdc617d726b85ed959ae0e70fc1aef8a110f5126aa663322b09e4ff8dc5d"} Oct 31 04:55:22.030052 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:22.029926 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-04d111-f77b9866c-2mzsk" event={"ID":"bb04ce0c-726f-4b27-b588-1b30fee2c0a0","Type":"ContainerStarted","Data":"1c79ac1a300ef7f9df9c0217585000226009057685fdb6ff6dd95809ddcadf13"} Oct 31 04:55:22.030102 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:22.030060 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-04d111-f77b9866c-2mzsk" Oct 31 04:55:22.031760 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:22.031726 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eba4874-6r5zg" event={"ID":"1c60e764-ed6c-48dc-b6d0-abffbbdcdc3c","Type":"ContainerStarted","Data":"26b8b220c20405cbbce4e199be8847aac7236b3191d8cd2e5f62be2f345b1de5"} Oct 31 04:55:22.033645 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:22.033613 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04d111-76d4bc867-jmfjn" event={"ID":"4fb383ce-a4bd-4455-b351-34bc277bc3e5","Type":"ContainerStarted","Data":"4ec33c2dfccb7259bd251ec4c7e91629209092e19f7c976acd8f583a18cffb17"} Oct 31 04:55:22.033809 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:22.033670 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04d111-76d4bc867-jmfjn" event={"ID":"4fb383ce-a4bd-4455-b351-34bc277bc3e5","Type":"ContainerStarted","Data":"2fa4f4e3ab947dfc5f74bf8398e6c94cf1449222faa51fd4597a8d1ab88fb03d"} Oct 31 04:55:22.033809 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:22.033721 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04d111-76d4bc867-jmfjn" Oct 31 04:55:22.070384 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:22.070294 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-04d111-f77b9866c-2mzsk" podStartSLOduration=2.070276769 podStartE2EDuration="2.070276769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:55:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:55:22.069532731 +0000 UTC m=+3076.825870770" watchObservedRunningTime="2026-10-31 04:55:22.070276769 +0000 UTC m=+3076.826614809" Oct 31 04:55:22.110630 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:22.110556 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04d111-76d4bc867-jmfjn" podStartSLOduration=2.110537526 podStartE2EDuration="2.110537526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:55:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:55:22.10982506 +0000 UTC m=+3076.866163099" watchObservedRunningTime="2026-10-31 04:55:22.110537526 +0000 UTC m=+3076.866875581" Oct 31 04:55:22.400597 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:55:22.400508 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:55:22.929599 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:22.929559 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-108ad8-8d8c4c667-rbd9x" Oct 31 04:55:22.929771 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:22.929660 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-108ad8-7495778969-49t8z" Oct 31 04:55:23.051523 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:23.051469 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eba4874-6r5zg" event={"ID":"1c60e764-ed6c-48dc-b6d0-abffbbdcdc3c","Type":"ContainerStarted","Data":"81454944cd236fc0d7c6690d90cbe9bdf84a2acb73e417d1e9ddf6d648ac9107"} Oct 31 04:55:23.093556 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:23.093484 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eba4874-6r5zg" podStartSLOduration=2.093462495 podStartE2EDuration="2.093462495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:55:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:55:23.091848038 +0000 UTC m=+3077.848186077" watchObservedRunningTime="2026-10-31 04:55:23.093462495 +0000 UTC m=+3077.849800536" Oct 31 04:55:24.035660 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:24.035616 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b6cfebd-q5h6g"] Oct 31 04:55:24.047005 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:24.046960 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b6cfebd-q5h6g"] Oct 31 04:55:24.064064 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:24.064025 5019 generic.go:356] "Generic (PLEG): container finished" podID="1c60e764-ed6c-48dc-b6d0-abffbbdcdc3c" containerID="81454944cd236fc0d7c6690d90cbe9bdf84a2acb73e417d1e9ddf6d648ac9107" exitCode=0 Oct 31 04:55:24.064253 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:24.064070 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eba4874-6r5zg" event={"ID":"1c60e764-ed6c-48dc-b6d0-abffbbdcdc3c","Type":"ContainerDied","Data":"81454944cd236fc0d7c6690d90cbe9bdf84a2acb73e417d1e9ddf6d648ac9107"} Oct 31 04:55:25.352197 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:25.352123 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eba4874-6r5zg" Oct 31 04:55:25.422573 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:25.422535 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a99d6b5-ee66-4b2a-b55c-1fcd5f0f6130" path="/var/lib/kubelet/pods/9a99d6b5-ee66-4b2a-b55c-1fcd5f0f6130/volumes" Oct 31 04:55:25.430042 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:25.430003 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c60e764-ed6c-48dc-b6d0-abffbbdcdc3c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c60e764-ed6c-48dc-b6d0-abffbbdcdc3c-maas-api-xks-ca\") pod \"1c60e764-ed6c-48dc-b6d0-abffbbdcdc3c\" (UID: \"1c60e764-ed6c-48dc-b6d0-abffbbdcdc3c\") " Oct 31 04:55:25.430192 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:25.430068 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c60e764-ed6c-48dc-b6d0-abffbbdcdc3c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c60e764-ed6c-48dc-b6d0-abffbbdcdc3c-maas-api-openshift-ca\") pod \"1c60e764-ed6c-48dc-b6d0-abffbbdcdc3c\" (UID: \"1c60e764-ed6c-48dc-b6d0-abffbbdcdc3c\") " Oct 31 04:55:25.430192 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:25.430135 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c60e764-ed6c-48dc-b6d0-abffbbdcdc3c-kube-api-access-n65dg\" (UniqueName: \"kubernetes.io/projected/1c60e764-ed6c-48dc-b6d0-abffbbdcdc3c-kube-api-access-n65dg\") pod \"1c60e764-ed6c-48dc-b6d0-abffbbdcdc3c\" (UID: \"1c60e764-ed6c-48dc-b6d0-abffbbdcdc3c\") " Oct 31 04:55:25.430865 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:25.430836 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c60e764-ed6c-48dc-b6d0-abffbbdcdc3c-maas-api-openshift-ca" pod "1c60e764-ed6c-48dc-b6d0-abffbbdcdc3c" (UID: "1c60e764-ed6c-48dc-b6d0-abffbbdcdc3c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:55:25.433388 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:25.433359 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c60e764-ed6c-48dc-b6d0-abffbbdcdc3c-maas-api-xks-ca" pod "1c60e764-ed6c-48dc-b6d0-abffbbdcdc3c" (UID: "1c60e764-ed6c-48dc-b6d0-abffbbdcdc3c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:55:25.434809 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:25.434780 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c60e764-ed6c-48dc-b6d0-abffbbdcdc3c-kube-api-access-n65dg" pod "1c60e764-ed6c-48dc-b6d0-abffbbdcdc3c" (UID: "1c60e764-ed6c-48dc-b6d0-abffbbdcdc3c"). InnerVolumeSpecName "kube-api-access-n65dg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:55:25.532418 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:25.532382 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c60e764-ed6c-48dc-b6d0-abffbbdcdc3c-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:55:25.532418 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:25.532417 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n65dg\" (UniqueName: \"kubernetes.io/projected/1c60e764-ed6c-48dc-b6d0-abffbbdcdc3c-kube-api-access-n65dg\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:55:25.532612 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:25.532435 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c60e764-ed6c-48dc-b6d0-abffbbdcdc3c-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:55:26.107661 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:26.107628 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eba4874-6r5zg" Oct 31 04:55:26.107661 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:26.107642 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eba4874-6r5zg" event={"ID":"1c60e764-ed6c-48dc-b6d0-abffbbdcdc3c","Type":"ContainerDied","Data":"26b8b220c20405cbbce4e199be8847aac7236b3191d8cd2e5f62be2f345b1de5"} Oct 31 04:55:26.107911 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:26.107703 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26b8b220c20405cbbce4e199be8847aac7236b3191d8cd2e5f62be2f345b1de5" Oct 31 04:55:27.538103 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:27.538058 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr"] Oct 31 04:55:27.539184 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:27.539155 5019 cpu_manager.go:414] "containerMap: removing container" podUID="1c60e764-ed6c-48dc-b6d0-abffbbdcdc3c" containerName="revoke-keys" Oct 31 04:55:27.539184 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:27.539183 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c60e764-ed6c-48dc-b6d0-abffbbdcdc3c" containerName="revoke-keys" Oct 31 04:55:27.539513 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:27.539493 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c60e764-ed6c-48dc-b6d0-abffbbdcdc3c" containerName="revoke-keys" Oct 31 04:55:27.545855 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:27.545819 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr" Oct 31 04:55:27.567989 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:27.567275 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbf01885-6e9e-405d-bf79-78a8bf2903a2-istio-token\") pod \"e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr\" (UID: \"cbf01885-6e9e-405d-bf79-78a8bf2903a2\") " pod="openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr" Oct 31 04:55:27.567989 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:27.567397 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf01885-6e9e-405d-bf79-78a8bf2903a2-workload-socket\") pod \"e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr\" (UID: \"cbf01885-6e9e-405d-bf79-78a8bf2903a2\") " pod="openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr" Oct 31 04:55:27.567989 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:27.567449 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbf01885-6e9e-405d-bf79-78a8bf2903a2-istiod-ca-cert\") pod \"e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr\" (UID: \"cbf01885-6e9e-405d-bf79-78a8bf2903a2\") " pod="openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr" Oct 31 04:55:27.567989 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:27.567526 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbpfv\" (UniqueName: \"kubernetes.io/projected/cbf01885-6e9e-405d-bf79-78a8bf2903a2-kube-api-access-pbpfv\") pod \"e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr\" (UID: \"cbf01885-6e9e-405d-bf79-78a8bf2903a2\") " pod="openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr" Oct 31 04:55:27.567989 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:27.567597 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbf01885-6e9e-405d-bf79-78a8bf2903a2-istio-envoy\") pod \"e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr\" (UID: \"cbf01885-6e9e-405d-bf79-78a8bf2903a2\") " pod="openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr" Oct 31 04:55:27.567989 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:27.567666 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbf01885-6e9e-405d-bf79-78a8bf2903a2-istio-data\") pod \"e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr\" (UID: \"cbf01885-6e9e-405d-bf79-78a8bf2903a2\") " pod="openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr" Oct 31 04:55:27.567989 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:27.567708 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbf01885-6e9e-405d-bf79-78a8bf2903a2-workload-certs\") pod \"e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr\" (UID: \"cbf01885-6e9e-405d-bf79-78a8bf2903a2\") " pod="openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr" Oct 31 04:55:27.602357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:27.568885 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf01885-6e9e-405d-bf79-78a8bf2903a2-credential-socket\") pod \"e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr\" (UID: \"cbf01885-6e9e-405d-bf79-78a8bf2903a2\") " pod="openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr" Oct 31 04:55:27.602357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:27.568937 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbf01885-6e9e-405d-bf79-78a8bf2903a2-istio-podinfo\") pod \"e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr\" (UID: \"cbf01885-6e9e-405d-bf79-78a8bf2903a2\") " pod="openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr" Oct 31 04:55:27.610350 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:27.610277 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-829ac7-openshift-default-dockercfg-9dq62\"" Oct 31 04:55:27.618859 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:27.618760 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr"] Oct 31 04:55:27.767359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:27.738744 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbf01885-6e9e-405d-bf79-78a8bf2903a2-istiod-ca-cert\") pod \"e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr\" (UID: \"cbf01885-6e9e-405d-bf79-78a8bf2903a2\") " pod="openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr" Oct 31 04:55:27.803412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:27.797772 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbpfv\" (UniqueName: \"kubernetes.io/projected/cbf01885-6e9e-405d-bf79-78a8bf2903a2-kube-api-access-pbpfv\") pod \"e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr\" (UID: \"cbf01885-6e9e-405d-bf79-78a8bf2903a2\") " pod="openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr" Oct 31 04:55:27.803412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:27.797843 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbf01885-6e9e-405d-bf79-78a8bf2903a2-istio-envoy\") pod \"e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr\" (UID: \"cbf01885-6e9e-405d-bf79-78a8bf2903a2\") " pod="openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr" Oct 31 04:55:27.803412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:27.797925 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbf01885-6e9e-405d-bf79-78a8bf2903a2-istio-data\") pod \"e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr\" (UID: \"cbf01885-6e9e-405d-bf79-78a8bf2903a2\") " pod="openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr" Oct 31 04:55:27.803412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:27.797966 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbf01885-6e9e-405d-bf79-78a8bf2903a2-workload-certs\") pod \"e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr\" (UID: \"cbf01885-6e9e-405d-bf79-78a8bf2903a2\") " pod="openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr" Oct 31 04:55:27.803412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:27.798272 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf01885-6e9e-405d-bf79-78a8bf2903a2-credential-socket\") pod \"e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr\" (UID: \"cbf01885-6e9e-405d-bf79-78a8bf2903a2\") " pod="openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr" Oct 31 04:55:27.803412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:27.798311 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbf01885-6e9e-405d-bf79-78a8bf2903a2-istio-podinfo\") pod \"e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr\" (UID: \"cbf01885-6e9e-405d-bf79-78a8bf2903a2\") " pod="openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr" Oct 31 04:55:27.803412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:27.798398 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbf01885-6e9e-405d-bf79-78a8bf2903a2-istio-token\") pod \"e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr\" (UID: \"cbf01885-6e9e-405d-bf79-78a8bf2903a2\") " pod="openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr" Oct 31 04:55:27.803412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:27.798480 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf01885-6e9e-405d-bf79-78a8bf2903a2-workload-socket\") pod \"e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr\" (UID: \"cbf01885-6e9e-405d-bf79-78a8bf2903a2\") " pod="openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr" Oct 31 04:55:27.803412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:27.799010 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf01885-6e9e-405d-bf79-78a8bf2903a2-workload-socket\") pod \"e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr\" (UID: \"cbf01885-6e9e-405d-bf79-78a8bf2903a2\") " pod="openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr" Oct 31 04:55:27.803412 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:27.770676 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbf01885-6e9e-405d-bf79-78a8bf2903a2-istiod-ca-cert\") pod \"e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr\" (UID: \"cbf01885-6e9e-405d-bf79-78a8bf2903a2\") " pod="openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr" Oct 31 04:55:27.836359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:27.830558 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbf01885-6e9e-405d-bf79-78a8bf2903a2-istio-data\") pod \"e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr\" (UID: \"cbf01885-6e9e-405d-bf79-78a8bf2903a2\") " pod="openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr" Oct 31 04:55:27.836359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:27.830822 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbf01885-6e9e-405d-bf79-78a8bf2903a2-workload-certs\") pod \"e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr\" (UID: \"cbf01885-6e9e-405d-bf79-78a8bf2903a2\") " pod="openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr" Oct 31 04:55:27.836359 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:27.831049 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf01885-6e9e-405d-bf79-78a8bf2903a2-credential-socket\") pod \"e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr\" (UID: \"cbf01885-6e9e-405d-bf79-78a8bf2903a2\") " pod="openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr" Oct 31 04:55:27.851807 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:27.851771 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbf01885-6e9e-405d-bf79-78a8bf2903a2-istio-envoy\") pod \"e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr\" (UID: \"cbf01885-6e9e-405d-bf79-78a8bf2903a2\") " pod="openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr" Oct 31 04:55:27.879357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:27.865984 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbf01885-6e9e-405d-bf79-78a8bf2903a2-istio-podinfo\") pod \"e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr\" (UID: \"cbf01885-6e9e-405d-bf79-78a8bf2903a2\") " pod="openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr" Oct 31 04:55:27.929355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:27.923028 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbf01885-6e9e-405d-bf79-78a8bf2903a2-istio-token\") pod \"e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr\" (UID: \"cbf01885-6e9e-405d-bf79-78a8bf2903a2\") " pod="openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr" Oct 31 04:55:27.929355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:27.923779 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbpfv\" (UniqueName: \"kubernetes.io/projected/cbf01885-6e9e-405d-bf79-78a8bf2903a2-kube-api-access-pbpfv\") pod \"e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr\" (UID: \"cbf01885-6e9e-405d-bf79-78a8bf2903a2\") " pod="openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr" Oct 31 04:55:28.193313 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:28.193212 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr" Oct 31 04:55:28.613879 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:55:28.613832 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbf01885_6e9e_405d_bf79_78a8bf2903a2.slice/crio-825a8821be0c8348dc133957650020a7831f990dee533bfa924c53f1f0d8c2a0 WatchSource:0}: Error finding container 825a8821be0c8348dc133957650020a7831f990dee533bfa924c53f1f0d8c2a0: Status 404 returned error can't find the container with id 825a8821be0c8348dc133957650020a7831f990dee533bfa924c53f1f0d8c2a0 Oct 31 04:55:28.617701 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:28.617166 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr"] Oct 31 04:55:28.621094 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:28.620385 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:55:28.621094 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:28.620490 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:55:28.621094 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:28.620534 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:55:29.141158 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:29.141117 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr" event={"ID":"cbf01885-6e9e-405d-bf79-78a8bf2903a2","Type":"ContainerStarted","Data":"8f87bddf195f526f26510317e72fd069561f58abedc758d21fa46414c3ce6c41"} Oct 31 04:55:29.141533 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:29.141503 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr" event={"ID":"cbf01885-6e9e-405d-bf79-78a8bf2903a2","Type":"ContainerStarted","Data":"825a8821be0c8348dc133957650020a7831f990dee533bfa924c53f1f0d8c2a0"} Oct 31 04:55:29.202355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:29.200753 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr" Oct 31 04:55:29.203696 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:29.203661 5019 patch_prober.go:29] interesting pod/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" start-of-body= Oct 31 04:55:29.203935 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:29.203904 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr" podUID="cbf01885-6e9e-405d-bf79-78a8bf2903a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" Oct 31 04:55:29.204291 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:29.203807 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr" podStartSLOduration=2.203788193 podStartE2EDuration="2.203788193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:55:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:55:29.201251308 +0000 UTC m=+3083.957589350" watchObservedRunningTime="2026-10-31 04:55:29.203788193 +0000 UTC m=+3083.960126235" Oct 31 04:55:30.201014 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:30.200977 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr" Oct 31 04:55:31.161180 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:31.161144 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr" Oct 31 04:55:31.162419 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:31.162393 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr" Oct 31 04:55:33.055760 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:33.055730 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04d111-76d4bc867-jmfjn" Oct 31 04:55:33.056268 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:33.055799 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-04d111-f77b9866c-2mzsk" Oct 31 04:55:33.482983 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:33.482945 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-829ac7-77fc54d55f-4npw2"] Oct 31 04:55:33.637135 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:33.637101 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-829ac7-77fc54d55f-4npw2"] Oct 31 04:55:33.704354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:33.648719 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-829ac7-77fc54d55f-4npw2" Oct 31 04:55:33.788503 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:33.736563 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-829ac7-dockercfg-49mhb\"" Oct 31 04:55:33.788503 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:33.756289 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-829ac7\"" Oct 31 04:55:33.867546 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:33.846205 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/225a0379-7184-4a57-97d8-c9eb65397e87-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-829ac7-77fc54d55f-4npw2\" (UID: \"225a0379-7184-4a57-97d8-c9eb65397e87\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-829ac7-77fc54d55f-4npw2" Oct 31 04:55:33.867546 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:33.846594 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szrch\" (UniqueName: \"kubernetes.io/projected/225a0379-7184-4a57-97d8-c9eb65397e87-kube-api-access-szrch\") pod \"payload-pre-processing-e2e-inf-b-829ac7-77fc54d55f-4npw2\" (UID: \"225a0379-7184-4a57-97d8-c9eb65397e87\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-829ac7-77fc54d55f-4npw2" Oct 31 04:55:33.949441 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:33.949392 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szrch\" (UniqueName: \"kubernetes.io/projected/225a0379-7184-4a57-97d8-c9eb65397e87-kube-api-access-szrch\") pod \"payload-pre-processing-e2e-inf-b-829ac7-77fc54d55f-4npw2\" (UID: \"225a0379-7184-4a57-97d8-c9eb65397e87\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-829ac7-77fc54d55f-4npw2" Oct 31 04:55:33.955846 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:33.955628 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/225a0379-7184-4a57-97d8-c9eb65397e87-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-829ac7-77fc54d55f-4npw2\" (UID: \"225a0379-7184-4a57-97d8-c9eb65397e87\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-829ac7-77fc54d55f-4npw2" Oct 31 04:55:33.957017 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:33.956977 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/225a0379-7184-4a57-97d8-c9eb65397e87-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-829ac7-77fc54d55f-4npw2\" (UID: \"225a0379-7184-4a57-97d8-c9eb65397e87\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-829ac7-77fc54d55f-4npw2" Oct 31 04:55:33.975421 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:33.975385 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szrch\" (UniqueName: \"kubernetes.io/projected/225a0379-7184-4a57-97d8-c9eb65397e87-kube-api-access-szrch\") pod \"payload-pre-processing-e2e-inf-b-829ac7-77fc54d55f-4npw2\" (UID: \"225a0379-7184-4a57-97d8-c9eb65397e87\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-829ac7-77fc54d55f-4npw2" Oct 31 04:55:34.129255 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:34.129161 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-829ac7-77fc54d55f-4npw2" Oct 31 04:55:35.173240 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:55:35.173171 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod225a0379_7184_4a57_97d8_c9eb65397e87.slice/crio-739b596df7d7081975a43de57f6fcfb6a9ebe02f04b4aa1d778fc4bf9a5b4e54 WatchSource:0}: Error finding container 739b596df7d7081975a43de57f6fcfb6a9ebe02f04b4aa1d778fc4bf9a5b4e54: Status 404 returned error can't find the container with id 739b596df7d7081975a43de57f6fcfb6a9ebe02f04b4aa1d778fc4bf9a5b4e54 Oct 31 04:55:35.173845 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:35.173397 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-829ac7-77fc54d55f-4npw2"] Oct 31 04:55:35.353690 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:35.353652 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-829ac7-77fc54d55f-4npw2" event={"ID":"225a0379-7184-4a57-97d8-c9eb65397e87","Type":"ContainerStarted","Data":"739b596df7d7081975a43de57f6fcfb6a9ebe02f04b4aa1d778fc4bf9a5b4e54"} Oct 31 04:55:36.365112 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:36.365066 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-829ac7-77fc54d55f-4npw2" event={"ID":"225a0379-7184-4a57-97d8-c9eb65397e87","Type":"ContainerStarted","Data":"a2181a0820acd7ad303a05fb1399da297951e9bc970f9f7119da89cb24584329"} Oct 31 04:55:36.365627 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:36.365153 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-829ac7-77fc54d55f-4npw2" Oct 31 04:55:36.412436 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:36.412316 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-829ac7-77fc54d55f-4npw2" podStartSLOduration=3.412295883 podStartE2EDuration="3.412295883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:55:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:55:36.405176394 +0000 UTC m=+3091.161514485" watchObservedRunningTime="2026-10-31 04:55:36.412295883 +0000 UTC m=+3091.168633924" Oct 31 04:55:37.764841 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:37.764800 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-b86f7hh2j"] Oct 31 04:55:39.832501 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:39.832462 5019 patch_prober.go:29] interesting pod/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:55:39.833112 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:39.832532 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t" podUID="51ef5aae-6bf0-4340-9e57-51b388049f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:55:41.546633 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:41.546595 5019 generic.go:356] "Generic (PLEG): container finished" podID="51ef5aae-6bf0-4340-9e57-51b388049f40" containerID="9e0f5aa541516eb722af32733f2ec9f5f8818409039487c81c42b5a180b7f265" exitCode=0 Oct 31 04:55:41.547189 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:41.546670 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t" event={"ID":"51ef5aae-6bf0-4340-9e57-51b388049f40","Type":"ContainerDied","Data":"9e0f5aa541516eb722af32733f2ec9f5f8818409039487c81c42b5a180b7f265"} Oct 31 04:55:41.547272 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:41.547210 5019 scope.go:122] "RemoveContainer" containerID="9e0f5aa541516eb722af32733f2ec9f5f8818409039487c81c42b5a180b7f265" Oct 31 04:55:41.549718 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:41.549686 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:55:41.549821 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:41.549786 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:55:41.549859 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:41.549825 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:55:42.694356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:42.693798 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t" event={"ID":"51ef5aae-6bf0-4340-9e57-51b388049f40","Type":"ContainerStarted","Data":"0892c25bdd29988c6f881b34255470bcff893227b26032754df5a6a24d9f389d"} Oct 31 04:55:42.796357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:42.793205 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-b86f7hh2j" podUID="24e66441-4418-4bb6-b24c-f3f3bd9f49fd" containerName="payload-processing" containerID="cri-o://f3f7bd429ba03731aeb62cb298d49f95a09eee5ba860ee71fb8e0c5a1a86a585" gracePeriod=25 Oct 31 04:55:42.900975 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:42.900941 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t" Oct 31 04:55:42.901407 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:42.901382 5019 patch_prober.go:29] interesting pod/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" start-of-body= Oct 31 04:55:42.901617 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:42.901591 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t" podUID="51ef5aae-6bf0-4340-9e57-51b388049f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" Oct 31 04:55:43.308844 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:43.308743 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-b86f7hh2j" Oct 31 04:55:43.422352 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:43.411076 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24e66441-4418-4bb6-b24c-f3f3bd9f49fd-kube-api-access-xhbwb\" (UniqueName: \"kubernetes.io/projected/24e66441-4418-4bb6-b24c-f3f3bd9f49fd-kube-api-access-xhbwb\") pod \"24e66441-4418-4bb6-b24c-f3f3bd9f49fd\" (UID: \"24e66441-4418-4bb6-b24c-f3f3bd9f49fd\") " Oct 31 04:55:43.422352 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:43.411135 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24e66441-4418-4bb6-b24c-f3f3bd9f49fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24e66441-4418-4bb6-b24c-f3f3bd9f49fd-plugins-config-volume\") pod \"24e66441-4418-4bb6-b24c-f3f3bd9f49fd\" (UID: \"24e66441-4418-4bb6-b24c-f3f3bd9f49fd\") " Oct 31 04:55:43.422352 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:43.411190 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24e66441-4418-4bb6-b24c-f3f3bd9f49fd-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/24e66441-4418-4bb6-b24c-f3f3bd9f49fd-routing-overlay-volume\") pod \"24e66441-4418-4bb6-b24c-f3f3bd9f49fd\" (UID: \"24e66441-4418-4bb6-b24c-f3f3bd9f49fd\") " Oct 31 04:55:43.422352 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:43.411452 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24e66441-4418-4bb6-b24c-f3f3bd9f49fd-provider-credentials\" (UniqueName: \"kubernetes.io/projected/24e66441-4418-4bb6-b24c-f3f3bd9f49fd-provider-credentials\") pod \"24e66441-4418-4bb6-b24c-f3f3bd9f49fd\" (UID: \"24e66441-4418-4bb6-b24c-f3f3bd9f49fd\") " Oct 31 04:55:43.422352 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:43.412216 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24e66441-4418-4bb6-b24c-f3f3bd9f49fd-routing-overlay-volume" pod "24e66441-4418-4bb6-b24c-f3f3bd9f49fd" (UID: "24e66441-4418-4bb6-b24c-f3f3bd9f49fd"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:55:43.422352 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:43.415944 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24e66441-4418-4bb6-b24c-f3f3bd9f49fd-plugins-config-volume" pod "24e66441-4418-4bb6-b24c-f3f3bd9f49fd" (UID: "24e66441-4418-4bb6-b24c-f3f3bd9f49fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:55:43.428294 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:43.428236 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24e66441-4418-4bb6-b24c-f3f3bd9f49fd-provider-credentials" pod "24e66441-4418-4bb6-b24c-f3f3bd9f49fd" (UID: "24e66441-4418-4bb6-b24c-f3f3bd9f49fd"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:55:43.430812 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:43.430768 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24e66441-4418-4bb6-b24c-f3f3bd9f49fd-kube-api-access-xhbwb" pod "24e66441-4418-4bb6-b24c-f3f3bd9f49fd" (UID: "24e66441-4418-4bb6-b24c-f3f3bd9f49fd"). InnerVolumeSpecName "kube-api-access-xhbwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:55:43.515153 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:43.515105 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhbwb\" (UniqueName: \"kubernetes.io/projected/24e66441-4418-4bb6-b24c-f3f3bd9f49fd-kube-api-access-xhbwb\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:55:43.515719 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:43.515688 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24e66441-4418-4bb6-b24c-f3f3bd9f49fd-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:55:43.515840 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:43.515720 5019 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/24e66441-4418-4bb6-b24c-f3f3bd9f49fd-routing-overlay-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:55:43.515840 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:43.515736 5019 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/24e66441-4418-4bb6-b24c-f3f3bd9f49fd-provider-credentials\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:55:43.554859 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:43.554821 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl"] Oct 31 04:55:43.555851 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:43.555826 5019 cpu_manager.go:414] "containerMap: removing container" podUID="24e66441-4418-4bb6-b24c-f3f3bd9f49fd" containerName="payload-processing" Oct 31 04:55:43.555851 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:43.555850 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24e66441-4418-4bb6-b24c-f3f3bd9f49fd" containerName="payload-processing" Oct 31 04:55:43.556106 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:43.556089 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="24e66441-4418-4bb6-b24c-f3f3bd9f49fd" containerName="payload-processing" Oct 31 04:55:43.560474 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:43.560405 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl" Oct 31 04:55:43.603357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:43.580889 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1daf30-openshift-default-dockercfg-nlvbf\"" Oct 31 04:55:43.625308 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:43.625252 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl"] Oct 31 04:55:43.789735 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:43.769608 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6d6891a-ccbc-4007-bbf3-96544ac865c2-istio-envoy\") pod \"e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl\" (UID: \"e6d6891a-ccbc-4007-bbf3-96544ac865c2\") " pod="openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl" Oct 31 04:55:43.790482 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:43.790443 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6d6891a-ccbc-4007-bbf3-96544ac865c2-credential-socket\") pod \"e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl\" (UID: \"e6d6891a-ccbc-4007-bbf3-96544ac865c2\") " pod="openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl" Oct 31 04:55:43.794354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:43.790681 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gst8\" (UniqueName: \"kubernetes.io/projected/e6d6891a-ccbc-4007-bbf3-96544ac865c2-kube-api-access-8gst8\") pod \"e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl\" (UID: \"e6d6891a-ccbc-4007-bbf3-96544ac865c2\") " pod="openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl" Oct 31 04:55:43.808875 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:43.808828 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6d6891a-ccbc-4007-bbf3-96544ac865c2-istiod-ca-cert\") pod \"e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl\" (UID: \"e6d6891a-ccbc-4007-bbf3-96544ac865c2\") " pod="openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl" Oct 31 04:55:43.817670 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:43.817560 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6d6891a-ccbc-4007-bbf3-96544ac865c2-istio-data\") pod \"e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl\" (UID: \"e6d6891a-ccbc-4007-bbf3-96544ac865c2\") " pod="openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl" Oct 31 04:55:43.825512 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:43.825468 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6d6891a-ccbc-4007-bbf3-96544ac865c2-workload-socket\") pod \"e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl\" (UID: \"e6d6891a-ccbc-4007-bbf3-96544ac865c2\") " pod="openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl" Oct 31 04:55:43.840357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:43.825843 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6d6891a-ccbc-4007-bbf3-96544ac865c2-istio-token\") pod \"e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl\" (UID: \"e6d6891a-ccbc-4007-bbf3-96544ac865c2\") " pod="openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl" Oct 31 04:55:43.840819 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:43.840775 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6d6891a-ccbc-4007-bbf3-96544ac865c2-workload-certs\") pod \"e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl\" (UID: \"e6d6891a-ccbc-4007-bbf3-96544ac865c2\") " pod="openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl" Oct 31 04:55:43.844709 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:43.844676 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6d6891a-ccbc-4007-bbf3-96544ac865c2-istio-podinfo\") pod \"e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl\" (UID: \"e6d6891a-ccbc-4007-bbf3-96544ac865c2\") " pod="openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl" Oct 31 04:55:43.899722 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:43.898713 5019 patch_prober.go:29] interesting pod/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" start-of-body= Oct 31 04:55:43.899722 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:43.898777 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t" podUID="51ef5aae-6bf0-4340-9e57-51b388049f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" Oct 31 04:55:43.921355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:43.901073 5019 generic.go:356] "Generic (PLEG): container finished" podID="24e66441-4418-4bb6-b24c-f3f3bd9f49fd" containerID="f3f7bd429ba03731aeb62cb298d49f95a09eee5ba860ee71fb8e0c5a1a86a585" exitCode=0 Oct 31 04:55:43.921355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:43.911821 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-b86f7hh2j" Oct 31 04:55:43.921727 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:43.921690 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-b86f7hh2j" event={"ID":"24e66441-4418-4bb6-b24c-f3f3bd9f49fd","Type":"ContainerDied","Data":"f3f7bd429ba03731aeb62cb298d49f95a09eee5ba860ee71fb8e0c5a1a86a585"} Oct 31 04:55:43.921902 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:43.921878 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-b86f7hh2j" event={"ID":"24e66441-4418-4bb6-b24c-f3f3bd9f49fd","Type":"ContainerDied","Data":"f31caea86f63cadd579d7aefeb75423c2f4db11b56c28c19b943134e6945e07c"} Oct 31 04:55:43.922050 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:43.922034 5019 scope.go:122] "RemoveContainer" containerID="f3f7bd429ba03731aeb62cb298d49f95a09eee5ba860ee71fb8e0c5a1a86a585" Oct 31 04:55:43.948039 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:43.948006 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6d6891a-ccbc-4007-bbf3-96544ac865c2-istio-podinfo\") pod \"e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl\" (UID: \"e6d6891a-ccbc-4007-bbf3-96544ac865c2\") " pod="openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl" Oct 31 04:55:43.953479 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:43.953446 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6d6891a-ccbc-4007-bbf3-96544ac865c2-istio-envoy\") pod \"e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl\" (UID: \"e6d6891a-ccbc-4007-bbf3-96544ac865c2\") " pod="openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl" Oct 31 04:55:43.953784 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:43.953762 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6d6891a-ccbc-4007-bbf3-96544ac865c2-credential-socket\") pod \"e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl\" (UID: \"e6d6891a-ccbc-4007-bbf3-96544ac865c2\") " pod="openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl" Oct 31 04:55:43.953944 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:43.953927 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gst8\" (UniqueName: \"kubernetes.io/projected/e6d6891a-ccbc-4007-bbf3-96544ac865c2-kube-api-access-8gst8\") pod \"e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl\" (UID: \"e6d6891a-ccbc-4007-bbf3-96544ac865c2\") " pod="openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl" Oct 31 04:55:43.954179 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:43.954161 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6d6891a-ccbc-4007-bbf3-96544ac865c2-istiod-ca-cert\") pod \"e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl\" (UID: \"e6d6891a-ccbc-4007-bbf3-96544ac865c2\") " pod="openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl" Oct 31 04:55:43.954375 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:43.954357 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6d6891a-ccbc-4007-bbf3-96544ac865c2-istio-data\") pod \"e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl\" (UID: \"e6d6891a-ccbc-4007-bbf3-96544ac865c2\") " pod="openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl" Oct 31 04:55:43.954641 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:43.954531 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6d6891a-ccbc-4007-bbf3-96544ac865c2-workload-socket\") pod \"e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl\" (UID: \"e6d6891a-ccbc-4007-bbf3-96544ac865c2\") " pod="openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl" Oct 31 04:55:43.954787 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:43.954774 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6d6891a-ccbc-4007-bbf3-96544ac865c2-istio-token\") pod \"e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl\" (UID: \"e6d6891a-ccbc-4007-bbf3-96544ac865c2\") " pod="openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl" Oct 31 04:55:43.958526 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:43.958500 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6d6891a-ccbc-4007-bbf3-96544ac865c2-workload-certs\") pod \"e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl\" (UID: \"e6d6891a-ccbc-4007-bbf3-96544ac865c2\") " pod="openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl" Oct 31 04:55:43.973619 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:43.958957 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6d6891a-ccbc-4007-bbf3-96544ac865c2-credential-socket\") pod \"e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl\" (UID: \"e6d6891a-ccbc-4007-bbf3-96544ac865c2\") " pod="openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl" Oct 31 04:55:43.989037 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:43.966663 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6d6891a-ccbc-4007-bbf3-96544ac865c2-workload-certs\") pod \"e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl\" (UID: \"e6d6891a-ccbc-4007-bbf3-96544ac865c2\") " pod="openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl" Oct 31 04:55:43.989302 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:43.967651 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6d6891a-ccbc-4007-bbf3-96544ac865c2-istiod-ca-cert\") pod \"e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl\" (UID: \"e6d6891a-ccbc-4007-bbf3-96544ac865c2\") " pod="openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl" Oct 31 04:55:43.989436 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:43.969149 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6d6891a-ccbc-4007-bbf3-96544ac865c2-istio-data\") pod \"e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl\" (UID: \"e6d6891a-ccbc-4007-bbf3-96544ac865c2\") " pod="openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl" Oct 31 04:55:43.999705 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:43.992582 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6d6891a-ccbc-4007-bbf3-96544ac865c2-workload-socket\") pod \"e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl\" (UID: \"e6d6891a-ccbc-4007-bbf3-96544ac865c2\") " pod="openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl" Oct 31 04:55:44.092356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:44.081486 5019 scope.go:122] "RemoveContainer" containerID="f3f7bd429ba03731aeb62cb298d49f95a09eee5ba860ee71fb8e0c5a1a86a585" Oct 31 04:55:44.092356 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:55:44.082357 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3f7bd429ba03731aeb62cb298d49f95a09eee5ba860ee71fb8e0c5a1a86a585\": container with ID starting with f3f7bd429ba03731aeb62cb298d49f95a09eee5ba860ee71fb8e0c5a1a86a585 not found: ID does not exist" containerID="f3f7bd429ba03731aeb62cb298d49f95a09eee5ba860ee71fb8e0c5a1a86a585" Oct 31 04:55:44.092356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:44.082396 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3f7bd429ba03731aeb62cb298d49f95a09eee5ba860ee71fb8e0c5a1a86a585"} err="failed to get container status \"f3f7bd429ba03731aeb62cb298d49f95a09eee5ba860ee71fb8e0c5a1a86a585\": rpc error: code = NotFound desc = could not find container \"f3f7bd429ba03731aeb62cb298d49f95a09eee5ba860ee71fb8e0c5a1a86a585\": container with ID starting with f3f7bd429ba03731aeb62cb298d49f95a09eee5ba860ee71fb8e0c5a1a86a585 not found: ID does not exist" Oct 31 04:55:44.121990 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:44.121914 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6d6891a-ccbc-4007-bbf3-96544ac865c2-istio-envoy\") pod \"e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl\" (UID: \"e6d6891a-ccbc-4007-bbf3-96544ac865c2\") " pod="openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl" Oct 31 04:55:44.122707 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:44.122646 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-b86f7hh2j"] Oct 31 04:55:44.122863 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:44.122756 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6d6891a-ccbc-4007-bbf3-96544ac865c2-istio-podinfo\") pod \"e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl\" (UID: \"e6d6891a-ccbc-4007-bbf3-96544ac865c2\") " pod="openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl" Oct 31 04:55:44.124131 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:44.124098 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gst8\" (UniqueName: \"kubernetes.io/projected/e6d6891a-ccbc-4007-bbf3-96544ac865c2-kube-api-access-8gst8\") pod \"e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl\" (UID: \"e6d6891a-ccbc-4007-bbf3-96544ac865c2\") " pod="openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl" Oct 31 04:55:44.125141 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:44.125108 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6d6891a-ccbc-4007-bbf3-96544ac865c2-istio-token\") pod \"e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl\" (UID: \"e6d6891a-ccbc-4007-bbf3-96544ac865c2\") " pod="openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl" Oct 31 04:55:44.128863 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:44.128842 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-b86f7hh2j"] Oct 31 04:55:44.236993 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:44.236957 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl" Oct 31 04:55:44.619210 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:44.619176 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl"] Oct 31 04:55:44.619705 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:55:44.619645 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6d6891a_ccbc_4007_bbf3_96544ac865c2.slice/crio-5fcd94493452423c1bba98fc05882bf25ce67409195a36b2898abafdcce81d60 WatchSource:0}: Error finding container 5fcd94493452423c1bba98fc05882bf25ce67409195a36b2898abafdcce81d60: Status 404 returned error can't find the container with id 5fcd94493452423c1bba98fc05882bf25ce67409195a36b2898abafdcce81d60 Oct 31 04:55:44.623015 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:44.622976 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:55:44.623128 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:44.623068 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:55:44.623128 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:44.623119 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:55:44.894972 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:44.894874 5019 patch_prober.go:29] interesting pod/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" start-of-body= Oct 31 04:55:44.895621 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:44.895564 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t" podUID="51ef5aae-6bf0-4340-9e57-51b388049f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" Oct 31 04:55:44.929666 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:44.929622 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl" event={"ID":"e6d6891a-ccbc-4007-bbf3-96544ac865c2","Type":"ContainerStarted","Data":"a72ce1d5dea6406c5a3b57a7313087e532a7b3cbf47f24ce3815913e72d984d3"} Oct 31 04:55:44.929858 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:44.929694 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl" event={"ID":"e6d6891a-ccbc-4007-bbf3-96544ac865c2","Type":"ContainerStarted","Data":"5fcd94493452423c1bba98fc05882bf25ce67409195a36b2898abafdcce81d60"} Oct 31 04:55:45.013364 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:45.005918 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl" podStartSLOduration=2.005893283 podStartE2EDuration="2.005893283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:55:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:55:45.004094724 +0000 UTC m=+3099.760432777" watchObservedRunningTime="2026-10-31 04:55:45.005893283 +0000 UTC m=+3099.762231326" Oct 31 04:55:45.238436 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:45.238377 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl" Oct 31 04:55:45.241171 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:45.241140 5019 patch_prober.go:29] interesting pod/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" start-of-body= Oct 31 04:55:45.241383 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:45.241198 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl" podUID="e6d6891a-ccbc-4007-bbf3-96544ac865c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" Oct 31 04:55:45.423651 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:45.423525 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24e66441-4418-4bb6-b24c-f3f3bd9f49fd" path="/var/lib/kubelet/pods/24e66441-4418-4bb6-b24c-f3f3bd9f49fd/volumes" Oct 31 04:55:45.887878 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:45.887830 5019 patch_prober.go:29] interesting pod/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" start-of-body= Oct 31 04:55:45.888224 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:45.888191 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t" podUID="51ef5aae-6bf0-4340-9e57-51b388049f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" Oct 31 04:55:46.243845 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:46.243800 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl" Oct 31 04:55:46.891358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:46.891017 5019 patch_prober.go:29] interesting pod/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" start-of-body= Oct 31 04:55:46.891358 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:46.891079 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t" podUID="51ef5aae-6bf0-4340-9e57-51b388049f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" Oct 31 04:55:46.952349 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:46.952305 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl" Oct 31 04:55:46.953322 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:46.953293 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl" Oct 31 04:55:47.517354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:47.509665 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-829ac7-77fc54d55f-4npw2" Oct 31 04:55:47.900581 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:47.900470 5019 patch_prober.go:29] interesting pod/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" start-of-body= Oct 31 04:55:47.900581 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:47.900551 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t" podUID="51ef5aae-6bf0-4340-9e57-51b388049f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" Oct 31 04:55:48.900021 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:48.899989 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t" Oct 31 04:55:48.901304 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:48.901274 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t" Oct 31 04:55:48.902553 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:55:48.902525 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t" Oct 31 04:55:52.402960 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:55:52.402906 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:56:05.038795 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:56:05.038753 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b803b38-8rvz9"] Oct 31 04:56:05.051480 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:56:05.051439 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b803b38-8rvz9"] Oct 31 04:56:05.415623 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:56:05.415524 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="980bcf39-71be-432a-b878-e5fbfdec5dba" path="/var/lib/kubelet/pods/980bcf39-71be-432a-b878-e5fbfdec5dba/volumes" Oct 31 04:56:06.031342 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:56:06.031298 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f18f16e8-hlq5c"] Oct 31 04:56:06.043148 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:56:06.043110 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f18f16e8-hlq5c"] Oct 31 04:56:07.411367 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:56:07.411311 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dec4623-e861-413c-8222-0c34801f3a79" path="/var/lib/kubelet/pods/6dec4623-e861-413c-8222-0c34801f3a79/volumes" Oct 31 04:56:08.856680 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:56:08.856621 5019 patch_prober.go:29] interesting pod/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:56:08.857227 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:56:08.856712 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl" podUID="1d46e1a8-ef71-4bbf-afec-a074d0fcfe05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:56:11.878052 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:56:11.878016 5019 scope.go:122] "RemoveContainer" containerID="3af4c4e24bfd30d2da9b238c03217ca6e943486cf2f41200c79178e919e1584c" Oct 31 04:56:15.033817 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:56:15.033768 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8768315-fsv74"] Oct 31 04:56:15.042249 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:56:15.042210 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8768315-fsv74"] Oct 31 04:56:15.415191 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:56:15.415102 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="506507c1-3d2b-413c-8f8f-0bcf84ce0ed8" path="/var/lib/kubelet/pods/506507c1-3d2b-413c-8f8f-0bcf84ce0ed8/volumes" Oct 31 04:56:17.693692 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:56:17.693635 5019 generic.go:356] "Generic (PLEG): container finished" podID="1d46e1a8-ef71-4bbf-afec-a074d0fcfe05" containerID="91b8e9f8d818fa4ee6637299edea1421c8b1ccf53c85215faee87a40c7e0d95e" exitCode=0 Oct 31 04:56:17.694304 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:56:17.693864 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl" event={"ID":"1d46e1a8-ef71-4bbf-afec-a074d0fcfe05","Type":"ContainerDied","Data":"91b8e9f8d818fa4ee6637299edea1421c8b1ccf53c85215faee87a40c7e0d95e"} Oct 31 04:56:17.694610 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:56:17.694584 5019 scope.go:122] "RemoveContainer" containerID="91b8e9f8d818fa4ee6637299edea1421c8b1ccf53c85215faee87a40c7e0d95e" Oct 31 04:56:17.697740 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:56:17.697691 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:56:17.698042 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:56:17.698011 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:56:17.698264 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:56:17.698205 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:56:18.708377 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:56:18.708309 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl" event={"ID":"1d46e1a8-ef71-4bbf-afec-a074d0fcfe05","Type":"ContainerStarted","Data":"36edf30386af908410f4e4c029a008a267547565f3f9e126b1fc8054065d98b1"} Oct 31 04:56:19.417067 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:56:19.417025 5019 patch_prober.go:29] interesting pod/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" start-of-body= Oct 31 04:56:19.417285 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:56:19.417090 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl" podUID="1d46e1a8-ef71-4bbf-afec-a074d0fcfe05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" Oct 31 04:56:19.417285 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:56:19.417109 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl" Oct 31 04:56:20.414868 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:56:20.414835 5019 patch_prober.go:29] interesting pod/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" start-of-body= Oct 31 04:56:20.415531 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:56:20.415498 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl" podUID="1d46e1a8-ef71-4bbf-afec-a074d0fcfe05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" Oct 31 04:56:21.415180 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:56:21.415120 5019 patch_prober.go:29] interesting pod/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" start-of-body= Oct 31 04:56:21.415815 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:56:21.415193 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl" podUID="1d46e1a8-ef71-4bbf-afec-a074d0fcfe05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" Oct 31 04:56:22.415250 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:56:22.415208 5019 patch_prober.go:29] interesting pod/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" start-of-body= Oct 31 04:56:22.415820 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:56:22.415283 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl" podUID="1d46e1a8-ef71-4bbf-afec-a074d0fcfe05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" Oct 31 04:56:22.444997 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:56:22.444958 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:56:22.815700 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:56:22.815669 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl" Oct 31 04:56:23.415474 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:56:23.415435 5019 patch_prober.go:29] interesting pod/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" start-of-body= Oct 31 04:56:23.415961 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:56:23.415515 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl" podUID="1d46e1a8-ef71-4bbf-afec-a074d0fcfe05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" Oct 31 04:56:24.416598 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:56:24.416555 5019 patch_prober.go:29] interesting pod/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" start-of-body= Oct 31 04:56:24.417130 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:56:24.416636 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl" podUID="1d46e1a8-ef71-4bbf-afec-a074d0fcfe05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" Oct 31 04:56:25.430948 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:56:25.430906 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl" Oct 31 04:56:25.432411 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:56:25.432386 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl" Oct 31 04:56:44.089381 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:56:44.088444 5019 generic.go:356] "Generic (PLEG): container finished" podID="7e6bd271-4285-4390-a0f9-e2fcd5a855d4" containerID="36ce6b5cb4ca668ed880432272f83fc12a7095d8cf42c073613a404058d82fe8" exitCode=0 Oct 31 04:56:44.089381 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:56:44.088527 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7" event={"ID":"7e6bd271-4285-4390-a0f9-e2fcd5a855d4","Type":"ContainerDied","Data":"36ce6b5cb4ca668ed880432272f83fc12a7095d8cf42c073613a404058d82fe8"} Oct 31 04:56:44.089381 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:56:44.089098 5019 scope.go:122] "RemoveContainer" containerID="36ce6b5cb4ca668ed880432272f83fc12a7095d8cf42c073613a404058d82fe8" Oct 31 04:56:44.091570 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:56:44.091524 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:56:44.091695 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:56:44.091600 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:56:44.091695 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:56:44.091636 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:56:45.102109 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:56:45.102072 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7" event={"ID":"7e6bd271-4285-4390-a0f9-e2fcd5a855d4","Type":"ContainerStarted","Data":"b09319d67033b91da26336d733c47a3526a290dbf08cefd50290c446a792bf2f"} Oct 31 04:56:45.155425 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:56:45.154718 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7" Oct 31 04:56:45.155425 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:56:45.154846 5019 patch_prober.go:29] interesting pod/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" start-of-body= Oct 31 04:56:45.155425 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:56:45.154893 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7" podUID="7e6bd271-4285-4390-a0f9-e2fcd5a855d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" Oct 31 04:56:46.154700 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:56:46.154661 5019 patch_prober.go:29] interesting pod/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" start-of-body= Oct 31 04:56:46.155113 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:56:46.154712 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7" podUID="7e6bd271-4285-4390-a0f9-e2fcd5a855d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" Oct 31 04:56:47.167087 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:56:47.164448 5019 patch_prober.go:29] interesting pod/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" start-of-body= Oct 31 04:56:47.167087 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:56:47.164524 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7" podUID="7e6bd271-4285-4390-a0f9-e2fcd5a855d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" Oct 31 04:56:48.153717 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:56:48.153678 5019 patch_prober.go:29] interesting pod/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" start-of-body= Oct 31 04:56:48.153896 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:56:48.153728 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7" podUID="7e6bd271-4285-4390-a0f9-e2fcd5a855d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" Oct 31 04:56:49.154578 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:56:49.154535 5019 patch_prober.go:29] interesting pod/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" start-of-body= Oct 31 04:56:49.155095 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:56:49.154601 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7" podUID="7e6bd271-4285-4390-a0f9-e2fcd5a855d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" Oct 31 04:56:49.701541 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:56:49.701506 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7" Oct 31 04:56:50.154141 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:56:50.154105 5019 patch_prober.go:29] interesting pod/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" start-of-body= Oct 31 04:56:50.154324 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:56:50.154183 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7" podUID="7e6bd271-4285-4390-a0f9-e2fcd5a855d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" Oct 31 04:56:51.159497 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:56:51.159465 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7" Oct 31 04:56:51.160497 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:56:51.160472 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7" Oct 31 04:56:52.392764 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:56:52.392723 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:57:02.203835 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:02.203466 5019 patch_prober.go:29] interesting pod/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:57:02.203835 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:02.203553 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr" podUID="cbf01885-6e9e-405d-bf79-78a8bf2903a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:57:04.795762 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:04.795720 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff"] Oct 31 04:57:04.933871 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:04.933829 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff"] Oct 31 04:57:04.952366 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:04.934227 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff" Oct 31 04:57:04.995015 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:04.994982 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-47e9af-openshift-default-dockercfg-k45vf\"" Oct 31 04:57:05.112984 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:05.076131 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9eb4117-200f-4720-9047-a79a3b227cb3-istio-podinfo\") pod \"e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff\" (UID: \"a9eb4117-200f-4720-9047-a79a3b227cb3\") " pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff" Oct 31 04:57:05.113542 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:05.113507 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9eb4117-200f-4720-9047-a79a3b227cb3-workload-certs\") pod \"e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff\" (UID: \"a9eb4117-200f-4720-9047-a79a3b227cb3\") " pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff" Oct 31 04:57:05.127398 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:05.125602 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9eb4117-200f-4720-9047-a79a3b227cb3-credential-socket\") pod \"e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff\" (UID: \"a9eb4117-200f-4720-9047-a79a3b227cb3\") " pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff" Oct 31 04:57:05.127398 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:05.125696 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9eb4117-200f-4720-9047-a79a3b227cb3-workload-socket\") pod \"e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff\" (UID: \"a9eb4117-200f-4720-9047-a79a3b227cb3\") " pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff" Oct 31 04:57:05.127398 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:05.125728 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9eb4117-200f-4720-9047-a79a3b227cb3-istio-token\") pod \"e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff\" (UID: \"a9eb4117-200f-4720-9047-a79a3b227cb3\") " pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff" Oct 31 04:57:05.127398 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:05.125779 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9eb4117-200f-4720-9047-a79a3b227cb3-istio-envoy\") pod \"e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff\" (UID: \"a9eb4117-200f-4720-9047-a79a3b227cb3\") " pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff" Oct 31 04:57:05.127398 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:05.125858 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9eb4117-200f-4720-9047-a79a3b227cb3-istio-data\") pod \"e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff\" (UID: \"a9eb4117-200f-4720-9047-a79a3b227cb3\") " pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff" Oct 31 04:57:05.127398 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:05.125942 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9eb4117-200f-4720-9047-a79a3b227cb3-istiod-ca-cert\") pod \"e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff\" (UID: \"a9eb4117-200f-4720-9047-a79a3b227cb3\") " pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff" Oct 31 04:57:05.127398 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:05.126004 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7f8f\" (UniqueName: \"kubernetes.io/projected/a9eb4117-200f-4720-9047-a79a3b227cb3-kube-api-access-g7f8f\") pod \"e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff\" (UID: \"a9eb4117-200f-4720-9047-a79a3b227cb3\") " pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff" Oct 31 04:57:05.242454 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:05.241260 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9eb4117-200f-4720-9047-a79a3b227cb3-istio-data\") pod \"e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff\" (UID: \"a9eb4117-200f-4720-9047-a79a3b227cb3\") " pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff" Oct 31 04:57:05.242454 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:05.241375 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9eb4117-200f-4720-9047-a79a3b227cb3-istiod-ca-cert\") pod \"e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff\" (UID: \"a9eb4117-200f-4720-9047-a79a3b227cb3\") " pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff" Oct 31 04:57:05.242454 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:05.241429 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7f8f\" (UniqueName: \"kubernetes.io/projected/a9eb4117-200f-4720-9047-a79a3b227cb3-kube-api-access-g7f8f\") pod \"e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff\" (UID: \"a9eb4117-200f-4720-9047-a79a3b227cb3\") " pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff" Oct 31 04:57:05.242454 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:05.241606 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9eb4117-200f-4720-9047-a79a3b227cb3-istio-podinfo\") pod \"e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff\" (UID: \"a9eb4117-200f-4720-9047-a79a3b227cb3\") " pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff" Oct 31 04:57:05.242454 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:05.241684 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9eb4117-200f-4720-9047-a79a3b227cb3-workload-certs\") pod \"e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff\" (UID: \"a9eb4117-200f-4720-9047-a79a3b227cb3\") " pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff" Oct 31 04:57:05.242454 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:05.241814 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9eb4117-200f-4720-9047-a79a3b227cb3-credential-socket\") pod \"e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff\" (UID: \"a9eb4117-200f-4720-9047-a79a3b227cb3\") " pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff" Oct 31 04:57:05.242454 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:05.241872 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9eb4117-200f-4720-9047-a79a3b227cb3-workload-socket\") pod \"e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff\" (UID: \"a9eb4117-200f-4720-9047-a79a3b227cb3\") " pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff" Oct 31 04:57:05.242454 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:05.241899 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9eb4117-200f-4720-9047-a79a3b227cb3-istio-token\") pod \"e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff\" (UID: \"a9eb4117-200f-4720-9047-a79a3b227cb3\") " pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff" Oct 31 04:57:05.242454 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:05.241935 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9eb4117-200f-4720-9047-a79a3b227cb3-istio-envoy\") pod \"e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff\" (UID: \"a9eb4117-200f-4720-9047-a79a3b227cb3\") " pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff" Oct 31 04:57:05.262748 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:05.253679 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9eb4117-200f-4720-9047-a79a3b227cb3-credential-socket\") pod \"e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff\" (UID: \"a9eb4117-200f-4720-9047-a79a3b227cb3\") " pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff" Oct 31 04:57:05.262748 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:05.253930 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9eb4117-200f-4720-9047-a79a3b227cb3-workload-certs\") pod \"e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff\" (UID: \"a9eb4117-200f-4720-9047-a79a3b227cb3\") " pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff" Oct 31 04:57:05.262748 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:05.260381 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9eb4117-200f-4720-9047-a79a3b227cb3-istiod-ca-cert\") pod \"e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff\" (UID: \"a9eb4117-200f-4720-9047-a79a3b227cb3\") " pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff" Oct 31 04:57:05.262748 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:05.260658 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9eb4117-200f-4720-9047-a79a3b227cb3-istio-data\") pod \"e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff\" (UID: \"a9eb4117-200f-4720-9047-a79a3b227cb3\") " pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff" Oct 31 04:57:05.277145 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:05.277015 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9eb4117-200f-4720-9047-a79a3b227cb3-workload-socket\") pod \"e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff\" (UID: \"a9eb4117-200f-4720-9047-a79a3b227cb3\") " pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff" Oct 31 04:57:05.329352 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:05.329203 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9eb4117-200f-4720-9047-a79a3b227cb3-istio-envoy\") pod \"e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff\" (UID: \"a9eb4117-200f-4720-9047-a79a3b227cb3\") " pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff" Oct 31 04:57:05.330455 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:05.330134 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9eb4117-200f-4720-9047-a79a3b227cb3-istio-podinfo\") pod \"e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff\" (UID: \"a9eb4117-200f-4720-9047-a79a3b227cb3\") " pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff" Oct 31 04:57:05.330455 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:05.330187 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7f8f\" (UniqueName: \"kubernetes.io/projected/a9eb4117-200f-4720-9047-a79a3b227cb3-kube-api-access-g7f8f\") pod \"e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff\" (UID: \"a9eb4117-200f-4720-9047-a79a3b227cb3\") " pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff" Oct 31 04:57:05.338847 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:05.338791 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9eb4117-200f-4720-9047-a79a3b227cb3-istio-token\") pod \"e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff\" (UID: \"a9eb4117-200f-4720-9047-a79a3b227cb3\") " pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff" Oct 31 04:57:05.342353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:05.341998 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff" Oct 31 04:57:06.763803 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:06.763757 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff"] Oct 31 04:57:06.781546 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:06.781505 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:57:06.781663 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:06.781599 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:57:06.781663 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:06.781643 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:57:07.589630 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:07.589589 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff" event={"ID":"a9eb4117-200f-4720-9047-a79a3b227cb3","Type":"ContainerStarted","Data":"9812edb363bf790a0f082efc6ce5626754191c3c58ce55fa197a367921e130a7"} Oct 31 04:57:07.589838 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:07.589662 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff" event={"ID":"a9eb4117-200f-4720-9047-a79a3b227cb3","Type":"ContainerStarted","Data":"36280872daaa89fed60b4fb80fad43d08473fcc16d8b5247b62a4d73857ff2d7"} Oct 31 04:57:07.650016 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:07.649968 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff" podStartSLOduration=3.649954907 podStartE2EDuration="3.649954907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:57:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:57:07.645538252 +0000 UTC m=+3182.401876293" watchObservedRunningTime="2026-10-31 04:57:07.649954907 +0000 UTC m=+3182.406292943" Oct 31 04:57:08.343734 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:08.343699 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff" Oct 31 04:57:08.350425 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:08.350399 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff" Oct 31 04:57:08.636135 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:08.636057 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff" Oct 31 04:57:08.639559 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:08.639504 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff" Oct 31 04:57:10.657932 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:10.657901 5019 generic.go:356] "Generic (PLEG): container finished" podID="cbf01885-6e9e-405d-bf79-78a8bf2903a2" containerID="8f87bddf195f526f26510317e72fd069561f58abedc758d21fa46414c3ce6c41" exitCode=0 Oct 31 04:57:10.658503 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:10.657971 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr" event={"ID":"cbf01885-6e9e-405d-bf79-78a8bf2903a2","Type":"ContainerDied","Data":"8f87bddf195f526f26510317e72fd069561f58abedc758d21fa46414c3ce6c41"} Oct 31 04:57:10.658605 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:10.658511 5019 scope.go:122] "RemoveContainer" containerID="8f87bddf195f526f26510317e72fd069561f58abedc758d21fa46414c3ce6c41" Oct 31 04:57:10.660609 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:10.660577 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:57:10.660736 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:10.660663 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:57:10.660736 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:10.660712 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:57:11.688402 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:11.688362 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr" event={"ID":"cbf01885-6e9e-405d-bf79-78a8bf2903a2","Type":"ContainerStarted","Data":"789859bc940166363d562c703ece64478381f2c08a18213f921ace41d38c93c0"} Oct 31 04:57:11.949943 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:11.949869 5019 scope.go:122] "RemoveContainer" containerID="181053a9118aee7f5dcc3aabdec008593874cfd3a139ac47df2568fa32ae2d89" Oct 31 04:57:11.963914 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:11.963728 5019 scope.go:122] "RemoveContainer" containerID="59431a92e60a9677cbe9b1faab7dddefb49303af6fd19591bb45d57284cc79ef" Oct 31 04:57:12.014858 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:12.014833 5019 scope.go:122] "RemoveContainer" containerID="2f34943474931ccc257e133d9060b397941bb412a56708e245f6e09fd4b8beb0" Oct 31 04:57:12.059822 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:12.059795 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-47e9af-7cb88c595f-s7jrb"] Oct 31 04:57:12.067220 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:12.067200 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-47e9af-7cb88c595f-s7jrb" Oct 31 04:57:12.105190 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:12.105147 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-47e9af\"" Oct 31 04:57:12.117869 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:12.117832 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-47e9af-7cb88c595f-s7jrb"] Oct 31 04:57:12.188177 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:12.169423 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d22c47b5-d2a0-423e-a2b5-c83bf4afbf96-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-47e9af-7cb88c595f-s7jrb\" (UID: \"d22c47b5-d2a0-423e-a2b5-c83bf4afbf96\") " pod="openshift-ingress/payload-processing-e2e-inf-b-47e9af-7cb88c595f-s7jrb" Oct 31 04:57:12.188177 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:12.169534 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgvk8\" (UniqueName: \"kubernetes.io/projected/d22c47b5-d2a0-423e-a2b5-c83bf4afbf96-kube-api-access-sgvk8\") pod \"payload-processing-e2e-inf-b-47e9af-7cb88c595f-s7jrb\" (UID: \"d22c47b5-d2a0-423e-a2b5-c83bf4afbf96\") " pod="openshift-ingress/payload-processing-e2e-inf-b-47e9af-7cb88c595f-s7jrb" Oct 31 04:57:12.251356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:12.250451 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr" Oct 31 04:57:12.298356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:12.297858 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d22c47b5-d2a0-423e-a2b5-c83bf4afbf96-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-47e9af-7cb88c595f-s7jrb\" (UID: \"d22c47b5-d2a0-423e-a2b5-c83bf4afbf96\") " pod="openshift-ingress/payload-processing-e2e-inf-b-47e9af-7cb88c595f-s7jrb" Oct 31 04:57:12.298356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:12.297975 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgvk8\" (UniqueName: \"kubernetes.io/projected/d22c47b5-d2a0-423e-a2b5-c83bf4afbf96-kube-api-access-sgvk8\") pod \"payload-processing-e2e-inf-b-47e9af-7cb88c595f-s7jrb\" (UID: \"d22c47b5-d2a0-423e-a2b5-c83bf4afbf96\") " pod="openshift-ingress/payload-processing-e2e-inf-b-47e9af-7cb88c595f-s7jrb" Oct 31 04:57:12.319999 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:12.299318 5019 patch_prober.go:29] interesting pod/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" start-of-body= Oct 31 04:57:12.319999 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:12.299400 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr" podUID="cbf01885-6e9e-405d-bf79-78a8bf2903a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" Oct 31 04:57:12.319999 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:12.300908 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d22c47b5-d2a0-423e-a2b5-c83bf4afbf96-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-47e9af-7cb88c595f-s7jrb\" (UID: \"d22c47b5-d2a0-423e-a2b5-c83bf4afbf96\") " pod="openshift-ingress/payload-processing-e2e-inf-b-47e9af-7cb88c595f-s7jrb" Oct 31 04:57:12.659404 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:12.657256 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgvk8\" (UniqueName: \"kubernetes.io/projected/d22c47b5-d2a0-423e-a2b5-c83bf4afbf96-kube-api-access-sgvk8\") pod \"payload-processing-e2e-inf-b-47e9af-7cb88c595f-s7jrb\" (UID: \"d22c47b5-d2a0-423e-a2b5-c83bf4afbf96\") " pod="openshift-ingress/payload-processing-e2e-inf-b-47e9af-7cb88c595f-s7jrb" Oct 31 04:57:12.732132 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:12.732102 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-47e9af-7cb88c595f-s7jrb" Oct 31 04:57:13.117227 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:13.117202 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-47e9af-7cb88c595f-s7jrb"] Oct 31 04:57:13.195479 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:13.194664 5019 patch_prober.go:29] interesting pod/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" start-of-body= Oct 31 04:57:13.195479 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:13.194717 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr" podUID="cbf01885-6e9e-405d-bf79-78a8bf2903a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" Oct 31 04:57:13.733525 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:13.733486 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-47e9af-7cb88c595f-s7jrb" event={"ID":"d22c47b5-d2a0-423e-a2b5-c83bf4afbf96","Type":"ContainerStarted","Data":"18710fcb0d8fb5f4ff0be5042b9e1283fe037a4ee822c509a59f7c51a4f006b2"} Oct 31 04:57:13.733947 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:13.733551 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-47e9af-7cb88c595f-s7jrb" event={"ID":"d22c47b5-d2a0-423e-a2b5-c83bf4afbf96","Type":"ContainerStarted","Data":"be369535179b332120ede41289e4dc6a5364af96b3fb5c0ac9ce03dbd2e4f38b"} Oct 31 04:57:13.733947 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:13.733579 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-47e9af-7cb88c595f-s7jrb" Oct 31 04:57:13.788232 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:13.788164 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-47e9af-7cb88c595f-s7jrb" podStartSLOduration=1.7881461600000002 podStartE2EDuration="1.78814616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:57:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:57:13.787578103 +0000 UTC m=+3188.543916143" watchObservedRunningTime="2026-10-31 04:57:13.78814616 +0000 UTC m=+3188.544484201" Oct 31 04:57:14.194703 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:14.194611 5019 patch_prober.go:29] interesting pod/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" start-of-body= Oct 31 04:57:14.194979 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:14.194944 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr" podUID="cbf01885-6e9e-405d-bf79-78a8bf2903a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" Oct 31 04:57:15.202353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:15.194386 5019 patch_prober.go:29] interesting pod/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" start-of-body= Oct 31 04:57:15.202353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:15.194456 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr" podUID="cbf01885-6e9e-405d-bf79-78a8bf2903a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" Oct 31 04:57:16.161742 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:16.161704 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr" Oct 31 04:57:16.199389 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:16.197917 5019 patch_prober.go:29] interesting pod/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" start-of-body= Oct 31 04:57:16.199389 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:16.197980 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr" podUID="cbf01885-6e9e-405d-bf79-78a8bf2903a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" Oct 31 04:57:17.298409 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:17.298374 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr" Oct 31 04:57:17.299924 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:17.299894 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr" Oct 31 04:57:22.040058 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:22.037517 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24"] Oct 31 04:57:22.045072 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:22.045046 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24" Oct 31 04:57:22.061706 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:22.061674 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-c1fe26-openshift-default-dockercfg-dmvwf\"" Oct 31 04:57:22.108355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:22.094173 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7lmq\" (UniqueName: \"kubernetes.io/projected/e33bca6f-714f-44a3-bf0c-e99ddb526c67-kube-api-access-k7lmq\") pod \"e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24\" (UID: \"e33bca6f-714f-44a3-bf0c-e99ddb526c67\") " pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24" Oct 31 04:57:22.108355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:22.103552 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e33bca6f-714f-44a3-bf0c-e99ddb526c67-workload-socket\") pod \"e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24\" (UID: \"e33bca6f-714f-44a3-bf0c-e99ddb526c67\") " pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24" Oct 31 04:57:22.108355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:22.103648 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e33bca6f-714f-44a3-bf0c-e99ddb526c67-istio-data\") pod \"e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24\" (UID: \"e33bca6f-714f-44a3-bf0c-e99ddb526c67\") " pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24" Oct 31 04:57:22.108355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:22.103677 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e33bca6f-714f-44a3-bf0c-e99ddb526c67-istio-podinfo\") pod \"e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24\" (UID: \"e33bca6f-714f-44a3-bf0c-e99ddb526c67\") " pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24" Oct 31 04:57:22.108355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:22.103722 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e33bca6f-714f-44a3-bf0c-e99ddb526c67-istio-envoy\") pod \"e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24\" (UID: \"e33bca6f-714f-44a3-bf0c-e99ddb526c67\") " pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24" Oct 31 04:57:22.108355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:22.103821 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e33bca6f-714f-44a3-bf0c-e99ddb526c67-credential-socket\") pod \"e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24\" (UID: \"e33bca6f-714f-44a3-bf0c-e99ddb526c67\") " pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24" Oct 31 04:57:22.108355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:22.103857 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e33bca6f-714f-44a3-bf0c-e99ddb526c67-istio-token\") pod \"e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24\" (UID: \"e33bca6f-714f-44a3-bf0c-e99ddb526c67\") " pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24" Oct 31 04:57:22.108355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:22.103907 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e33bca6f-714f-44a3-bf0c-e99ddb526c67-istiod-ca-cert\") pod \"e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24\" (UID: \"e33bca6f-714f-44a3-bf0c-e99ddb526c67\") " pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24" Oct 31 04:57:22.108355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:22.103946 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e33bca6f-714f-44a3-bf0c-e99ddb526c67-workload-certs\") pod \"e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24\" (UID: \"e33bca6f-714f-44a3-bf0c-e99ddb526c67\") " pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24" Oct 31 04:57:22.122414 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:22.122386 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24"] Oct 31 04:57:22.216584 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:22.212158 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e33bca6f-714f-44a3-bf0c-e99ddb526c67-istio-data\") pod \"e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24\" (UID: \"e33bca6f-714f-44a3-bf0c-e99ddb526c67\") " pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24" Oct 31 04:57:22.216584 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:22.212209 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e33bca6f-714f-44a3-bf0c-e99ddb526c67-istio-podinfo\") pod \"e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24\" (UID: \"e33bca6f-714f-44a3-bf0c-e99ddb526c67\") " pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24" Oct 31 04:57:22.216584 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:22.212263 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e33bca6f-714f-44a3-bf0c-e99ddb526c67-istio-envoy\") pod \"e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24\" (UID: \"e33bca6f-714f-44a3-bf0c-e99ddb526c67\") " pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24" Oct 31 04:57:22.216584 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:22.212367 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e33bca6f-714f-44a3-bf0c-e99ddb526c67-credential-socket\") pod \"e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24\" (UID: \"e33bca6f-714f-44a3-bf0c-e99ddb526c67\") " pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24" Oct 31 04:57:22.216584 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:22.212405 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e33bca6f-714f-44a3-bf0c-e99ddb526c67-istio-token\") pod \"e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24\" (UID: \"e33bca6f-714f-44a3-bf0c-e99ddb526c67\") " pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24" Oct 31 04:57:22.216584 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:22.212455 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e33bca6f-714f-44a3-bf0c-e99ddb526c67-istiod-ca-cert\") pod \"e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24\" (UID: \"e33bca6f-714f-44a3-bf0c-e99ddb526c67\") " pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24" Oct 31 04:57:22.216584 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:22.212498 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e33bca6f-714f-44a3-bf0c-e99ddb526c67-workload-certs\") pod \"e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24\" (UID: \"e33bca6f-714f-44a3-bf0c-e99ddb526c67\") " pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24" Oct 31 04:57:22.216584 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:22.212622 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7lmq\" (UniqueName: \"kubernetes.io/projected/e33bca6f-714f-44a3-bf0c-e99ddb526c67-kube-api-access-k7lmq\") pod \"e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24\" (UID: \"e33bca6f-714f-44a3-bf0c-e99ddb526c67\") " pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24" Oct 31 04:57:22.216584 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:22.212882 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e33bca6f-714f-44a3-bf0c-e99ddb526c67-workload-socket\") pod \"e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24\" (UID: \"e33bca6f-714f-44a3-bf0c-e99ddb526c67\") " pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24" Oct 31 04:57:22.216584 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:22.213384 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e33bca6f-714f-44a3-bf0c-e99ddb526c67-workload-socket\") pod \"e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24\" (UID: \"e33bca6f-714f-44a3-bf0c-e99ddb526c67\") " pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24" Oct 31 04:57:22.216584 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:22.213653 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e33bca6f-714f-44a3-bf0c-e99ddb526c67-istio-data\") pod \"e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24\" (UID: \"e33bca6f-714f-44a3-bf0c-e99ddb526c67\") " pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24" Oct 31 04:57:22.216584 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:22.214933 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e33bca6f-714f-44a3-bf0c-e99ddb526c67-workload-certs\") pod \"e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24\" (UID: \"e33bca6f-714f-44a3-bf0c-e99ddb526c67\") " pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24" Oct 31 04:57:22.216584 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:22.215673 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e33bca6f-714f-44a3-bf0c-e99ddb526c67-istiod-ca-cert\") pod \"e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24\" (UID: \"e33bca6f-714f-44a3-bf0c-e99ddb526c67\") " pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24" Oct 31 04:57:22.216584 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:22.216166 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e33bca6f-714f-44a3-bf0c-e99ddb526c67-credential-socket\") pod \"e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24\" (UID: \"e33bca6f-714f-44a3-bf0c-e99ddb526c67\") " pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24" Oct 31 04:57:22.286357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:22.275585 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e33bca6f-714f-44a3-bf0c-e99ddb526c67-istio-token\") pod \"e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24\" (UID: \"e33bca6f-714f-44a3-bf0c-e99ddb526c67\") " pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24" Oct 31 04:57:22.381861 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:22.381778 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e33bca6f-714f-44a3-bf0c-e99ddb526c67-istio-envoy\") pod \"e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24\" (UID: \"e33bca6f-714f-44a3-bf0c-e99ddb526c67\") " pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24" Oct 31 04:57:22.382871 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:22.382847 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e33bca6f-714f-44a3-bf0c-e99ddb526c67-istio-podinfo\") pod \"e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24\" (UID: \"e33bca6f-714f-44a3-bf0c-e99ddb526c67\") " pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24" Oct 31 04:57:22.407365 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:22.399226 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7lmq\" (UniqueName: \"kubernetes.io/projected/e33bca6f-714f-44a3-bf0c-e99ddb526c67-kube-api-access-k7lmq\") pod \"e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24\" (UID: \"e33bca6f-714f-44a3-bf0c-e99ddb526c67\") " pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24" Oct 31 04:57:22.694370 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:22.692936 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24" Oct 31 04:57:22.696352 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:57:22.695378 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:57:23.131815 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:57:23.131784 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode33bca6f_714f_44a3_bf0c_e99ddb526c67.slice/crio-47b1cc2308eac241df4b4b3b4f3aa888d02c2ce218cb06152d22fdc8bfc28f4b WatchSource:0}: Error finding container 47b1cc2308eac241df4b4b3b4f3aa888d02c2ce218cb06152d22fdc8bfc28f4b: Status 404 returned error can't find the container with id 47b1cc2308eac241df4b4b3b4f3aa888d02c2ce218cb06152d22fdc8bfc28f4b Oct 31 04:57:23.132908 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:23.132880 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24"] Oct 31 04:57:23.135175 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:23.135136 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:57:23.135309 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:23.135223 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:57:23.135309 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:23.135273 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:57:23.901382 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:23.899960 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24" event={"ID":"e33bca6f-714f-44a3-bf0c-e99ddb526c67","Type":"ContainerStarted","Data":"741d41265c76a6aa2bda535cd07bf196705321b2d1deb6018cd9dd01b467ea54"} Oct 31 04:57:23.901382 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:23.900062 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24" event={"ID":"e33bca6f-714f-44a3-bf0c-e99ddb526c67","Type":"ContainerStarted","Data":"47b1cc2308eac241df4b4b3b4f3aa888d02c2ce218cb06152d22fdc8bfc28f4b"} Oct 31 04:57:23.960238 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:23.960184 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24" podStartSLOduration=1.960169992 podStartE2EDuration="1.960169992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:57:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:57:23.955985737 +0000 UTC m=+3198.712323778" watchObservedRunningTime="2026-10-31 04:57:23.960169992 +0000 UTC m=+3198.716508106" Oct 31 04:57:24.697611 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:24.697048 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24" Oct 31 04:57:24.704722 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:24.704702 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24" Oct 31 04:57:24.745635 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:24.745614 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-47e9af-7cb88c595f-s7jrb" Oct 31 04:57:24.910142 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:24.910105 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24" Oct 31 04:57:24.911663 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:24.911642 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24" Oct 31 04:57:32.296812 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:32.296774 5019 patch_prober.go:29] interesting pod/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.125:15021/healthz/ready\": read tcp 10.132.0.2:37710->10.132.1.125:15021: read: connection reset by peer" start-of-body= Oct 31 04:57:32.297268 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:32.296848 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl" podUID="e6d6891a-ccbc-4007-bbf3-96544ac865c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.125:15021/healthz/ready\": read tcp 10.132.0.2:37710->10.132.1.125:15021: read: connection reset by peer" Oct 31 04:57:32.993234 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:32.993198 5019 generic.go:356] "Generic (PLEG): container finished" podID="e6d6891a-ccbc-4007-bbf3-96544ac865c2" containerID="a72ce1d5dea6406c5a3b57a7313087e532a7b3cbf47f24ce3815913e72d984d3" exitCode=0 Oct 31 04:57:32.993495 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:32.993264 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl" event={"ID":"e6d6891a-ccbc-4007-bbf3-96544ac865c2","Type":"ContainerDied","Data":"a72ce1d5dea6406c5a3b57a7313087e532a7b3cbf47f24ce3815913e72d984d3"} Oct 31 04:57:32.993756 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:32.993737 5019 scope.go:122] "RemoveContainer" containerID="a72ce1d5dea6406c5a3b57a7313087e532a7b3cbf47f24ce3815913e72d984d3" Oct 31 04:57:32.996393 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:32.996353 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:57:32.996481 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:32.996448 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:57:32.996525 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:32.996499 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:57:34.125515 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:34.125477 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl" event={"ID":"e6d6891a-ccbc-4007-bbf3-96544ac865c2","Type":"ContainerStarted","Data":"8f0392245a281d1f0a00b0f7485b92ee412a6e40dd120d83eb35513376e05de3"} Oct 31 04:57:34.241364 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:34.239524 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl" Oct 31 04:57:34.241364 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:34.239673 5019 patch_prober.go:29] interesting pod/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" start-of-body= Oct 31 04:57:34.241364 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:34.239736 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl" podUID="e6d6891a-ccbc-4007-bbf3-96544ac865c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" Oct 31 04:57:35.237895 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:35.237855 5019 patch_prober.go:29] interesting pod/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" start-of-body= Oct 31 04:57:35.238434 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:35.237912 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl" podUID="e6d6891a-ccbc-4007-bbf3-96544ac865c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" Oct 31 04:57:36.237209 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:36.237176 5019 patch_prober.go:29] interesting pod/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" start-of-body= Oct 31 04:57:36.237423 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:36.237225 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl" podUID="e6d6891a-ccbc-4007-bbf3-96544ac865c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" Oct 31 04:57:37.238218 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:37.238185 5019 patch_prober.go:29] interesting pod/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" start-of-body= Oct 31 04:57:37.238657 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:37.238238 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl" podUID="e6d6891a-ccbc-4007-bbf3-96544ac865c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" Oct 31 04:57:38.238050 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:38.238013 5019 patch_prober.go:29] interesting pod/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" start-of-body= Oct 31 04:57:38.238295 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:38.238090 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl" podUID="e6d6891a-ccbc-4007-bbf3-96544ac865c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" Oct 31 04:57:39.237907 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:39.237872 5019 patch_prober.go:29] interesting pod/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" start-of-body= Oct 31 04:57:39.238102 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:39.237935 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl" podUID="e6d6891a-ccbc-4007-bbf3-96544ac865c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" Oct 31 04:57:40.243087 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:40.243050 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl" Oct 31 04:57:40.243632 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:40.243608 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl" Oct 31 04:57:40.244376 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:40.244355 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl" Oct 31 04:57:41.305529 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:41.305492 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-670795/test-model-670795-kserve-7c6cfc8755-9gk2c"] Oct 31 04:57:41.313252 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:41.313226 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-670795/test-model-670795-kserve-7c6cfc8755-9gk2c" Oct 31 04:57:41.330455 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:41.330427 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-670795/test-model-670795-kserve-7c6cfc8755-9gk2c"] Oct 31 04:57:41.334001 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:41.333969 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-670795\"/\"openshift-service-ca.crt\"" Oct 31 04:57:41.334001 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:41.333987 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-670795\"/\"kube-root-ca.crt\"" Oct 31 04:57:41.334142 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:41.334033 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-670795\"/\"test-model-670795-kserve-self-signed-certs\"" Oct 31 04:57:41.341829 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:41.341807 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-670795\"/\"default-dockercfg-rv7gx\"" Oct 31 04:57:41.377667 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:41.377635 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3984da2-0b96-49e7-a555-6bca50f5f45a-dshm\") pod \"test-model-670795-kserve-7c6cfc8755-9gk2c\" (UID: \"f3984da2-0b96-49e7-a555-6bca50f5f45a\") " pod="ai-tenant-e2e-inf-a-670795/test-model-670795-kserve-7c6cfc8755-9gk2c" Oct 31 04:57:41.377822 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:41.377706 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3984da2-0b96-49e7-a555-6bca50f5f45a-home\") pod \"test-model-670795-kserve-7c6cfc8755-9gk2c\" (UID: \"f3984da2-0b96-49e7-a555-6bca50f5f45a\") " pod="ai-tenant-e2e-inf-a-670795/test-model-670795-kserve-7c6cfc8755-9gk2c" Oct 31 04:57:41.377822 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:41.377802 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3984da2-0b96-49e7-a555-6bca50f5f45a-model-cache\") pod \"test-model-670795-kserve-7c6cfc8755-9gk2c\" (UID: \"f3984da2-0b96-49e7-a555-6bca50f5f45a\") " pod="ai-tenant-e2e-inf-a-670795/test-model-670795-kserve-7c6cfc8755-9gk2c" Oct 31 04:57:41.377956 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:41.377859 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3984da2-0b96-49e7-a555-6bca50f5f45a-tls-certs\") pod \"test-model-670795-kserve-7c6cfc8755-9gk2c\" (UID: \"f3984da2-0b96-49e7-a555-6bca50f5f45a\") " pod="ai-tenant-e2e-inf-a-670795/test-model-670795-kserve-7c6cfc8755-9gk2c" Oct 31 04:57:41.378037 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:41.378021 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3984da2-0b96-49e7-a555-6bca50f5f45a-tmp-dir\") pod \"test-model-670795-kserve-7c6cfc8755-9gk2c\" (UID: \"f3984da2-0b96-49e7-a555-6bca50f5f45a\") " pod="ai-tenant-e2e-inf-a-670795/test-model-670795-kserve-7c6cfc8755-9gk2c" Oct 31 04:57:41.378102 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:41.378071 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfqvc\" (UniqueName: \"kubernetes.io/projected/f3984da2-0b96-49e7-a555-6bca50f5f45a-kube-api-access-rfqvc\") pod \"test-model-670795-kserve-7c6cfc8755-9gk2c\" (UID: \"f3984da2-0b96-49e7-a555-6bca50f5f45a\") " pod="ai-tenant-e2e-inf-a-670795/test-model-670795-kserve-7c6cfc8755-9gk2c" Oct 31 04:57:41.479185 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:41.479153 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfqvc\" (UniqueName: \"kubernetes.io/projected/f3984da2-0b96-49e7-a555-6bca50f5f45a-kube-api-access-rfqvc\") pod \"test-model-670795-kserve-7c6cfc8755-9gk2c\" (UID: \"f3984da2-0b96-49e7-a555-6bca50f5f45a\") " pod="ai-tenant-e2e-inf-a-670795/test-model-670795-kserve-7c6cfc8755-9gk2c" Oct 31 04:57:41.479406 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:41.479231 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3984da2-0b96-49e7-a555-6bca50f5f45a-dshm\") pod \"test-model-670795-kserve-7c6cfc8755-9gk2c\" (UID: \"f3984da2-0b96-49e7-a555-6bca50f5f45a\") " pod="ai-tenant-e2e-inf-a-670795/test-model-670795-kserve-7c6cfc8755-9gk2c" Oct 31 04:57:41.479406 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:41.479294 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3984da2-0b96-49e7-a555-6bca50f5f45a-home\") pod \"test-model-670795-kserve-7c6cfc8755-9gk2c\" (UID: \"f3984da2-0b96-49e7-a555-6bca50f5f45a\") " pod="ai-tenant-e2e-inf-a-670795/test-model-670795-kserve-7c6cfc8755-9gk2c" Oct 31 04:57:41.479406 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:41.479366 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3984da2-0b96-49e7-a555-6bca50f5f45a-model-cache\") pod \"test-model-670795-kserve-7c6cfc8755-9gk2c\" (UID: \"f3984da2-0b96-49e7-a555-6bca50f5f45a\") " pod="ai-tenant-e2e-inf-a-670795/test-model-670795-kserve-7c6cfc8755-9gk2c" Oct 31 04:57:41.479601 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:41.479427 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3984da2-0b96-49e7-a555-6bca50f5f45a-tls-certs\") pod \"test-model-670795-kserve-7c6cfc8755-9gk2c\" (UID: \"f3984da2-0b96-49e7-a555-6bca50f5f45a\") " pod="ai-tenant-e2e-inf-a-670795/test-model-670795-kserve-7c6cfc8755-9gk2c" Oct 31 04:57:41.479698 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:41.479667 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3984da2-0b96-49e7-a555-6bca50f5f45a-tmp-dir\") pod \"test-model-670795-kserve-7c6cfc8755-9gk2c\" (UID: \"f3984da2-0b96-49e7-a555-6bca50f5f45a\") " pod="ai-tenant-e2e-inf-a-670795/test-model-670795-kserve-7c6cfc8755-9gk2c" Oct 31 04:57:41.479813 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:41.479789 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3984da2-0b96-49e7-a555-6bca50f5f45a-home\") pod \"test-model-670795-kserve-7c6cfc8755-9gk2c\" (UID: \"f3984da2-0b96-49e7-a555-6bca50f5f45a\") " pod="ai-tenant-e2e-inf-a-670795/test-model-670795-kserve-7c6cfc8755-9gk2c" Oct 31 04:57:41.479813 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:41.479802 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3984da2-0b96-49e7-a555-6bca50f5f45a-model-cache\") pod \"test-model-670795-kserve-7c6cfc8755-9gk2c\" (UID: \"f3984da2-0b96-49e7-a555-6bca50f5f45a\") " pod="ai-tenant-e2e-inf-a-670795/test-model-670795-kserve-7c6cfc8755-9gk2c" Oct 31 04:57:41.480097 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:41.480077 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3984da2-0b96-49e7-a555-6bca50f5f45a-tmp-dir\") pod \"test-model-670795-kserve-7c6cfc8755-9gk2c\" (UID: \"f3984da2-0b96-49e7-a555-6bca50f5f45a\") " pod="ai-tenant-e2e-inf-a-670795/test-model-670795-kserve-7c6cfc8755-9gk2c" Oct 31 04:57:41.482014 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:41.481987 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3984da2-0b96-49e7-a555-6bca50f5f45a-dshm\") pod \"test-model-670795-kserve-7c6cfc8755-9gk2c\" (UID: \"f3984da2-0b96-49e7-a555-6bca50f5f45a\") " pod="ai-tenant-e2e-inf-a-670795/test-model-670795-kserve-7c6cfc8755-9gk2c" Oct 31 04:57:41.482184 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:41.482163 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3984da2-0b96-49e7-a555-6bca50f5f45a-tls-certs\") pod \"test-model-670795-kserve-7c6cfc8755-9gk2c\" (UID: \"f3984da2-0b96-49e7-a555-6bca50f5f45a\") " pod="ai-tenant-e2e-inf-a-670795/test-model-670795-kserve-7c6cfc8755-9gk2c" Oct 31 04:57:41.503223 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:41.503191 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfqvc\" (UniqueName: \"kubernetes.io/projected/f3984da2-0b96-49e7-a555-6bca50f5f45a-kube-api-access-rfqvc\") pod \"test-model-670795-kserve-7c6cfc8755-9gk2c\" (UID: \"f3984da2-0b96-49e7-a555-6bca50f5f45a\") " pod="ai-tenant-e2e-inf-a-670795/test-model-670795-kserve-7c6cfc8755-9gk2c" Oct 31 04:57:41.647447 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:41.645036 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-670795/test-model-670795-kserve-7c6cfc8755-9gk2c" Oct 31 04:57:42.036586 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:57:42.036554 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3984da2_0b96_49e7_a555_6bca50f5f45a.slice/crio-09b6ffbf84d96ae814eee43fa2e4adf89741ff5d996e02e2574948237ae11d34 WatchSource:0}: Error finding container 09b6ffbf84d96ae814eee43fa2e4adf89741ff5d996e02e2574948237ae11d34: Status 404 returned error can't find the container with id 09b6ffbf84d96ae814eee43fa2e4adf89741ff5d996e02e2574948237ae11d34 Oct 31 04:57:42.038202 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:42.038073 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-670795/test-model-670795-kserve-7c6cfc8755-9gk2c"] Oct 31 04:57:42.209074 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:42.209030 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-670795/test-model-670795-kserve-7c6cfc8755-9gk2c" event={"ID":"f3984da2-0b96-49e7-a555-6bca50f5f45a","Type":"ContainerStarted","Data":"8d9ba0a4cddc94c2f3e2452056d6c501cdbfe2da74215fb2c6d668b477d408a1"} Oct 31 04:57:42.209229 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:42.209083 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-670795/test-model-670795-kserve-7c6cfc8755-9gk2c" event={"ID":"f3984da2-0b96-49e7-a555-6bca50f5f45a","Type":"ContainerStarted","Data":"09b6ffbf84d96ae814eee43fa2e4adf89741ff5d996e02e2574948237ae11d34"} Oct 31 04:57:42.255178 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:42.255130 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-670795/test-model-670795-kserve-7c6cfc8755-9gk2c" podStartSLOduration=1.255117427 podStartE2EDuration="1.255117427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:57:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:57:42.24970081 +0000 UTC m=+3217.006038850" watchObservedRunningTime="2026-10-31 04:57:42.255117427 +0000 UTC m=+3217.011455467" Oct 31 04:57:42.645606 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:42.645576 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-670795/test-model-670795-kserve-7c6cfc8755-9gk2c" Oct 31 04:57:51.645940 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:51.645903 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-670795/test-model-670795-kserve-7c6cfc8755-9gk2c" Oct 31 04:57:51.658395 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:51.658370 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-670795/test-model-670795-kserve-7c6cfc8755-9gk2c" Oct 31 04:57:52.396823 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:57:52.396782 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:57:52.417154 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:57:52.417125 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-670795/test-model-670795-kserve-7c6cfc8755-9gk2c" Oct 31 04:58:02.163715 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:02.163680 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-12c0bc-6785c7bbf7-8928h"] Oct 31 04:58:02.169862 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:02.169840 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-12c0bc-6785c7bbf7-8928h" Oct 31 04:58:02.191240 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:02.191207 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-12c0bc-6785c7bbf7-8928h"] Oct 31 04:58:02.194170 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:02.194142 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-12c0bc-dockercfg-ls2c9\"" Oct 31 04:58:02.194306 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:02.194276 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-12c0bc\"" Oct 31 04:58:02.365448 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:02.318304 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5da17e2b-6c07-4222-b09c-310cf0805666-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-12c0bc-6785c7bbf7-8928h\" (UID: \"5da17e2b-6c07-4222-b09c-310cf0805666\") " pod="openshift-ingress/payload-processing-e2e-inf-b-12c0bc-6785c7bbf7-8928h" Oct 31 04:58:02.395665 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:02.395625 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76d27\" (UniqueName: \"kubernetes.io/projected/5da17e2b-6c07-4222-b09c-310cf0805666-kube-api-access-76d27\") pod \"payload-processing-e2e-inf-b-12c0bc-6785c7bbf7-8928h\" (UID: \"5da17e2b-6c07-4222-b09c-310cf0805666\") " pod="openshift-ingress/payload-processing-e2e-inf-b-12c0bc-6785c7bbf7-8928h" Oct 31 04:58:02.540355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:02.527902 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5da17e2b-6c07-4222-b09c-310cf0805666-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-12c0bc-6785c7bbf7-8928h\" (UID: \"5da17e2b-6c07-4222-b09c-310cf0805666\") " pod="openshift-ingress/payload-processing-e2e-inf-b-12c0bc-6785c7bbf7-8928h" Oct 31 04:58:02.540355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:02.528062 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76d27\" (UniqueName: \"kubernetes.io/projected/5da17e2b-6c07-4222-b09c-310cf0805666-kube-api-access-76d27\") pod \"payload-processing-e2e-inf-b-12c0bc-6785c7bbf7-8928h\" (UID: \"5da17e2b-6c07-4222-b09c-310cf0805666\") " pod="openshift-ingress/payload-processing-e2e-inf-b-12c0bc-6785c7bbf7-8928h" Oct 31 04:58:02.540355 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:02.529930 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5da17e2b-6c07-4222-b09c-310cf0805666-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-12c0bc-6785c7bbf7-8928h\" (UID: \"5da17e2b-6c07-4222-b09c-310cf0805666\") " pod="openshift-ingress/payload-processing-e2e-inf-b-12c0bc-6785c7bbf7-8928h" Oct 31 04:58:02.620038 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:02.620002 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76d27\" (UniqueName: \"kubernetes.io/projected/5da17e2b-6c07-4222-b09c-310cf0805666-kube-api-access-76d27\") pod \"payload-processing-e2e-inf-b-12c0bc-6785c7bbf7-8928h\" (UID: \"5da17e2b-6c07-4222-b09c-310cf0805666\") " pod="openshift-ingress/payload-processing-e2e-inf-b-12c0bc-6785c7bbf7-8928h" Oct 31 04:58:02.788466 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:02.788434 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-12c0bc-6785c7bbf7-8928h" Oct 31 04:58:03.101933 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:58:03.101883 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5da17e2b_6c07_4222_b09c_310cf0805666.slice/crio-0fce3db77a0b9c8b17cb0aeda0c67c31e1bbb9b0c4ed39675ee3bb662d5148e4 WatchSource:0}: Error finding container 0fce3db77a0b9c8b17cb0aeda0c67c31e1bbb9b0c4ed39675ee3bb662d5148e4: Status 404 returned error can't find the container with id 0fce3db77a0b9c8b17cb0aeda0c67c31e1bbb9b0c4ed39675ee3bb662d5148e4 Oct 31 04:58:03.102178 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:03.102147 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-12c0bc-6785c7bbf7-8928h"] Oct 31 04:58:03.635066 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:03.635028 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-12c0bc-6785c7bbf7-8928h" event={"ID":"5da17e2b-6c07-4222-b09c-310cf0805666","Type":"ContainerStarted","Data":"8c056a3b10cf20c966662b589476bb22440896a842bf1bdc6fe733b97df1f031"} Oct 31 04:58:03.635506 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:03.635088 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-12c0bc-6785c7bbf7-8928h" event={"ID":"5da17e2b-6c07-4222-b09c-310cf0805666","Type":"ContainerStarted","Data":"0fce3db77a0b9c8b17cb0aeda0c67c31e1bbb9b0c4ed39675ee3bb662d5148e4"} Oct 31 04:58:03.635506 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:03.635118 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-12c0bc-6785c7bbf7-8928h" Oct 31 04:58:03.720707 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:03.720631 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-12c0bc-6785c7bbf7-8928h" podStartSLOduration=1.720610202 podStartE2EDuration="1.720610202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:58:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:58:03.673840054 +0000 UTC m=+3238.430178094" watchObservedRunningTime="2026-10-31 04:58:03.720610202 +0000 UTC m=+3238.476948242" Oct 31 04:58:12.068906 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:12.068873 5019 scope.go:122] "RemoveContainer" containerID="6be82bc75b5ba7bd69d15554547b1bfd13bad2b5548a04ef8b620333c09f965e" Oct 31 04:58:14.647017 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:14.646980 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-12c0bc-6785c7bbf7-8928h" Oct 31 04:58:16.534730 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:16.534691 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-171edd/test-model-171edd-kserve-7b677876c5-v5zw4"] Oct 31 04:58:16.542671 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:16.542642 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-171edd/test-model-171edd-kserve-7b677876c5-v5zw4" Oct 31 04:58:16.563555 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:16.563509 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-171edd\"/\"default-dockercfg-rxdlf\"" Oct 31 04:58:16.563777 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:16.563597 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-171edd\"/\"openshift-service-ca.crt\"" Oct 31 04:58:16.564984 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:16.564952 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-171edd/test-model-171edd-kserve-7b677876c5-v5zw4"] Oct 31 04:58:16.569529 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:16.569496 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-171edd\"/\"test-model-171edd-kserve-self-signed-certs\"" Oct 31 04:58:16.569529 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:16.569496 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-171edd\"/\"kube-root-ca.crt\"" Oct 31 04:58:16.639435 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:16.639395 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96c515d1-a206-4fcd-a43e-dcc811e966bb-tmp-dir\") pod \"test-model-171edd-kserve-7b677876c5-v5zw4\" (UID: \"96c515d1-a206-4fcd-a43e-dcc811e966bb\") " pod="ai-tenant-e2e-inf-a-171edd/test-model-171edd-kserve-7b677876c5-v5zw4" Oct 31 04:58:16.639672 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:16.639554 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96c515d1-a206-4fcd-a43e-dcc811e966bb-model-cache\") pod \"test-model-171edd-kserve-7b677876c5-v5zw4\" (UID: \"96c515d1-a206-4fcd-a43e-dcc811e966bb\") " pod="ai-tenant-e2e-inf-a-171edd/test-model-171edd-kserve-7b677876c5-v5zw4" Oct 31 04:58:16.639780 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:16.639754 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96c515d1-a206-4fcd-a43e-dcc811e966bb-tls-certs\") pod \"test-model-171edd-kserve-7b677876c5-v5zw4\" (UID: \"96c515d1-a206-4fcd-a43e-dcc811e966bb\") " pod="ai-tenant-e2e-inf-a-171edd/test-model-171edd-kserve-7b677876c5-v5zw4" Oct 31 04:58:16.640052 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:16.640026 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96c515d1-a206-4fcd-a43e-dcc811e966bb-home\") pod \"test-model-171edd-kserve-7b677876c5-v5zw4\" (UID: \"96c515d1-a206-4fcd-a43e-dcc811e966bb\") " pod="ai-tenant-e2e-inf-a-171edd/test-model-171edd-kserve-7b677876c5-v5zw4" Oct 31 04:58:16.640164 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:16.640133 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkjfw\" (UniqueName: \"kubernetes.io/projected/96c515d1-a206-4fcd-a43e-dcc811e966bb-kube-api-access-jkjfw\") pod \"test-model-171edd-kserve-7b677876c5-v5zw4\" (UID: \"96c515d1-a206-4fcd-a43e-dcc811e966bb\") " pod="ai-tenant-e2e-inf-a-171edd/test-model-171edd-kserve-7b677876c5-v5zw4" Oct 31 04:58:16.640233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:16.640213 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96c515d1-a206-4fcd-a43e-dcc811e966bb-dshm\") pod \"test-model-171edd-kserve-7b677876c5-v5zw4\" (UID: \"96c515d1-a206-4fcd-a43e-dcc811e966bb\") " pod="ai-tenant-e2e-inf-a-171edd/test-model-171edd-kserve-7b677876c5-v5zw4" Oct 31 04:58:16.742233 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:16.742182 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96c515d1-a206-4fcd-a43e-dcc811e966bb-home\") pod \"test-model-171edd-kserve-7b677876c5-v5zw4\" (UID: \"96c515d1-a206-4fcd-a43e-dcc811e966bb\") " pod="ai-tenant-e2e-inf-a-171edd/test-model-171edd-kserve-7b677876c5-v5zw4" Oct 31 04:58:16.742478 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:16.742260 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkjfw\" (UniqueName: \"kubernetes.io/projected/96c515d1-a206-4fcd-a43e-dcc811e966bb-kube-api-access-jkjfw\") pod \"test-model-171edd-kserve-7b677876c5-v5zw4\" (UID: \"96c515d1-a206-4fcd-a43e-dcc811e966bb\") " pod="ai-tenant-e2e-inf-a-171edd/test-model-171edd-kserve-7b677876c5-v5zw4" Oct 31 04:58:16.742478 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:16.742327 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96c515d1-a206-4fcd-a43e-dcc811e966bb-dshm\") pod \"test-model-171edd-kserve-7b677876c5-v5zw4\" (UID: \"96c515d1-a206-4fcd-a43e-dcc811e966bb\") " pod="ai-tenant-e2e-inf-a-171edd/test-model-171edd-kserve-7b677876c5-v5zw4" Oct 31 04:58:16.742478 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:16.742433 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96c515d1-a206-4fcd-a43e-dcc811e966bb-tmp-dir\") pod \"test-model-171edd-kserve-7b677876c5-v5zw4\" (UID: \"96c515d1-a206-4fcd-a43e-dcc811e966bb\") " pod="ai-tenant-e2e-inf-a-171edd/test-model-171edd-kserve-7b677876c5-v5zw4" Oct 31 04:58:16.742667 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:16.742512 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96c515d1-a206-4fcd-a43e-dcc811e966bb-model-cache\") pod \"test-model-171edd-kserve-7b677876c5-v5zw4\" (UID: \"96c515d1-a206-4fcd-a43e-dcc811e966bb\") " pod="ai-tenant-e2e-inf-a-171edd/test-model-171edd-kserve-7b677876c5-v5zw4" Oct 31 04:58:16.742667 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:16.742621 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96c515d1-a206-4fcd-a43e-dcc811e966bb-home\") pod \"test-model-171edd-kserve-7b677876c5-v5zw4\" (UID: \"96c515d1-a206-4fcd-a43e-dcc811e966bb\") " pod="ai-tenant-e2e-inf-a-171edd/test-model-171edd-kserve-7b677876c5-v5zw4" Oct 31 04:58:16.742667 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:16.742648 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96c515d1-a206-4fcd-a43e-dcc811e966bb-tls-certs\") pod \"test-model-171edd-kserve-7b677876c5-v5zw4\" (UID: \"96c515d1-a206-4fcd-a43e-dcc811e966bb\") " pod="ai-tenant-e2e-inf-a-171edd/test-model-171edd-kserve-7b677876c5-v5zw4" Oct 31 04:58:16.742825 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:16.742787 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96c515d1-a206-4fcd-a43e-dcc811e966bb-tmp-dir\") pod \"test-model-171edd-kserve-7b677876c5-v5zw4\" (UID: \"96c515d1-a206-4fcd-a43e-dcc811e966bb\") " pod="ai-tenant-e2e-inf-a-171edd/test-model-171edd-kserve-7b677876c5-v5zw4" Oct 31 04:58:16.743108 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:16.742926 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96c515d1-a206-4fcd-a43e-dcc811e966bb-model-cache\") pod \"test-model-171edd-kserve-7b677876c5-v5zw4\" (UID: \"96c515d1-a206-4fcd-a43e-dcc811e966bb\") " pod="ai-tenant-e2e-inf-a-171edd/test-model-171edd-kserve-7b677876c5-v5zw4" Oct 31 04:58:16.744649 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:16.744625 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96c515d1-a206-4fcd-a43e-dcc811e966bb-dshm\") pod \"test-model-171edd-kserve-7b677876c5-v5zw4\" (UID: \"96c515d1-a206-4fcd-a43e-dcc811e966bb\") " pod="ai-tenant-e2e-inf-a-171edd/test-model-171edd-kserve-7b677876c5-v5zw4" Oct 31 04:58:16.763367 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:16.745460 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96c515d1-a206-4fcd-a43e-dcc811e966bb-tls-certs\") pod \"test-model-171edd-kserve-7b677876c5-v5zw4\" (UID: \"96c515d1-a206-4fcd-a43e-dcc811e966bb\") " pod="ai-tenant-e2e-inf-a-171edd/test-model-171edd-kserve-7b677876c5-v5zw4" Oct 31 04:58:16.834423 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:16.829475 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkjfw\" (UniqueName: \"kubernetes.io/projected/96c515d1-a206-4fcd-a43e-dcc811e966bb-kube-api-access-jkjfw\") pod \"test-model-171edd-kserve-7b677876c5-v5zw4\" (UID: \"96c515d1-a206-4fcd-a43e-dcc811e966bb\") " pod="ai-tenant-e2e-inf-a-171edd/test-model-171edd-kserve-7b677876c5-v5zw4" Oct 31 04:58:16.876320 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:16.876282 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-171edd/test-model-171edd-kserve-7b677876c5-v5zw4" Oct 31 04:58:17.278637 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:58:17.278599 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96c515d1_a206_4fcd_a43e_dcc811e966bb.slice/crio-566c5f4cde3ac29e77889f8aee72c0948bab999850bccaf7818079e4393e4953 WatchSource:0}: Error finding container 566c5f4cde3ac29e77889f8aee72c0948bab999850bccaf7818079e4393e4953: Status 404 returned error can't find the container with id 566c5f4cde3ac29e77889f8aee72c0948bab999850bccaf7818079e4393e4953 Oct 31 04:58:17.279858 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:17.279826 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-171edd/test-model-171edd-kserve-7b677876c5-v5zw4"] Oct 31 04:58:17.839707 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:17.839663 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-171edd/test-model-171edd-kserve-7b677876c5-v5zw4" event={"ID":"96c515d1-a206-4fcd-a43e-dcc811e966bb","Type":"ContainerStarted","Data":"5273a094861a304eae2ae699ae9094185bc4cde7afbb020684485d163a638cad"} Oct 31 04:58:17.839707 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:17.839713 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-171edd/test-model-171edd-kserve-7b677876c5-v5zw4" event={"ID":"96c515d1-a206-4fcd-a43e-dcc811e966bb","Type":"ContainerStarted","Data":"566c5f4cde3ac29e77889f8aee72c0948bab999850bccaf7818079e4393e4953"} Oct 31 04:58:17.877585 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:17.877536 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-171edd/test-model-171edd-kserve-7b677876c5-v5zw4" Oct 31 04:58:17.881973 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:17.880755 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-171edd/test-model-171edd-kserve-7b677876c5-v5zw4" podStartSLOduration=1.880737578 podStartE2EDuration="1.880737578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:58:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:58:17.880472889 +0000 UTC m=+3252.636810955" watchObservedRunningTime="2026-10-31 04:58:17.880737578 +0000 UTC m=+3252.637075618" Oct 31 04:58:21.031932 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:21.031887 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9893a9cc-jm4zm"] Oct 31 04:58:21.044131 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:21.044083 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9893a9cc-jm4zm"] Oct 31 04:58:21.416350 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:21.416236 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a334173-d81b-4b31-8a77-5ad4241215b4" path="/var/lib/kubelet/pods/3a334173-d81b-4b31-8a77-5ad4241215b4/volumes" Oct 31 04:58:22.658767 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:58:22.658725 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:58:25.034540 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:25.034497 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-826dd91f-rzcxg"] Oct 31 04:58:25.046763 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:25.046715 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-826dd91f-rzcxg"] Oct 31 04:58:25.412663 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:25.412570 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2d931e9-f094-4771-8af5-a11bf84d8978" path="/var/lib/kubelet/pods/d2d931e9-f094-4771-8af5-a11bf84d8978/volumes" Oct 31 04:58:26.877152 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:26.877094 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-171edd/test-model-171edd-kserve-7b677876c5-v5zw4" Oct 31 04:58:26.890267 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:26.890231 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-171edd/test-model-171edd-kserve-7b677876c5-v5zw4" Oct 31 04:58:26.962625 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:26.962591 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-171edd/test-model-171edd-kserve-7b677876c5-v5zw4" Oct 31 04:58:34.041489 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:34.041433 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-340168f9-8mnh2"] Oct 31 04:58:34.047636 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:34.047595 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-340168f9-8mnh2"] Oct 31 04:58:35.412492 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:35.412447 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="481beb7c-221e-430f-800c-ad11adc67b4f" path="/var/lib/kubelet/pods/481beb7c-221e-430f-800c-ad11adc67b4f/volumes" Oct 31 04:58:40.030496 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:40.030456 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-fb346b/test-model-fb346b-kserve-6d4b9b47b5-f7mnx"] Oct 31 04:58:40.036158 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:40.036128 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fb346b/test-model-fb346b-kserve-6d4b9b47b5-f7mnx" Oct 31 04:58:40.054563 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:40.054524 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-fb346b/test-model-fb346b-kserve-6d4b9b47b5-f7mnx"] Oct 31 04:58:40.059590 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:40.059552 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-fb346b\"/\"openshift-service-ca.crt\"" Oct 31 04:58:40.059780 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:40.059652 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-fb346b\"/\"kube-root-ca.crt\"" Oct 31 04:58:40.063012 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:40.062981 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-fb346b\"/\"default-dockercfg-6vv5l\"" Oct 31 04:58:40.063185 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:40.063011 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-fb346b\"/\"test-model-fb346b-kserve-self-signed-certs\"" Oct 31 04:58:40.202858 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:40.202814 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60b4b3fb-a758-45b8-933f-b62536b3661e-tmp-dir\") pod \"test-model-fb346b-kserve-6d4b9b47b5-f7mnx\" (UID: \"60b4b3fb-a758-45b8-933f-b62536b3661e\") " pod="ai-tenant-e2e-inf-b-fb346b/test-model-fb346b-kserve-6d4b9b47b5-f7mnx" Oct 31 04:58:40.203108 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:40.202988 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60b4b3fb-a758-45b8-933f-b62536b3661e-tls-certs\") pod \"test-model-fb346b-kserve-6d4b9b47b5-f7mnx\" (UID: \"60b4b3fb-a758-45b8-933f-b62536b3661e\") " pod="ai-tenant-e2e-inf-b-fb346b/test-model-fb346b-kserve-6d4b9b47b5-f7mnx" Oct 31 04:58:40.203190 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:40.203111 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60b4b3fb-a758-45b8-933f-b62536b3661e-model-cache\") pod \"test-model-fb346b-kserve-6d4b9b47b5-f7mnx\" (UID: \"60b4b3fb-a758-45b8-933f-b62536b3661e\") " pod="ai-tenant-e2e-inf-b-fb346b/test-model-fb346b-kserve-6d4b9b47b5-f7mnx" Oct 31 04:58:40.203260 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:40.203235 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60b4b3fb-a758-45b8-933f-b62536b3661e-home\") pod \"test-model-fb346b-kserve-6d4b9b47b5-f7mnx\" (UID: \"60b4b3fb-a758-45b8-933f-b62536b3661e\") " pod="ai-tenant-e2e-inf-b-fb346b/test-model-fb346b-kserve-6d4b9b47b5-f7mnx" Oct 31 04:58:40.203318 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:40.203289 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g82bq\" (UniqueName: \"kubernetes.io/projected/60b4b3fb-a758-45b8-933f-b62536b3661e-kube-api-access-g82bq\") pod \"test-model-fb346b-kserve-6d4b9b47b5-f7mnx\" (UID: \"60b4b3fb-a758-45b8-933f-b62536b3661e\") " pod="ai-tenant-e2e-inf-b-fb346b/test-model-fb346b-kserve-6d4b9b47b5-f7mnx" Oct 31 04:58:40.203397 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:40.203382 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60b4b3fb-a758-45b8-933f-b62536b3661e-dshm\") pod \"test-model-fb346b-kserve-6d4b9b47b5-f7mnx\" (UID: \"60b4b3fb-a758-45b8-933f-b62536b3661e\") " pod="ai-tenant-e2e-inf-b-fb346b/test-model-fb346b-kserve-6d4b9b47b5-f7mnx" Oct 31 04:58:40.313845 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:40.313732 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60b4b3fb-a758-45b8-933f-b62536b3661e-tmp-dir\") pod \"test-model-fb346b-kserve-6d4b9b47b5-f7mnx\" (UID: \"60b4b3fb-a758-45b8-933f-b62536b3661e\") " pod="ai-tenant-e2e-inf-b-fb346b/test-model-fb346b-kserve-6d4b9b47b5-f7mnx" Oct 31 04:58:40.314945 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:40.314855 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60b4b3fb-a758-45b8-933f-b62536b3661e-tls-certs\") pod \"test-model-fb346b-kserve-6d4b9b47b5-f7mnx\" (UID: \"60b4b3fb-a758-45b8-933f-b62536b3661e\") " pod="ai-tenant-e2e-inf-b-fb346b/test-model-fb346b-kserve-6d4b9b47b5-f7mnx" Oct 31 04:58:40.315353 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:40.315306 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60b4b3fb-a758-45b8-933f-b62536b3661e-model-cache\") pod \"test-model-fb346b-kserve-6d4b9b47b5-f7mnx\" (UID: \"60b4b3fb-a758-45b8-933f-b62536b3661e\") " pod="ai-tenant-e2e-inf-b-fb346b/test-model-fb346b-kserve-6d4b9b47b5-f7mnx" Oct 31 04:58:40.315662 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:40.315641 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60b4b3fb-a758-45b8-933f-b62536b3661e-home\") pod \"test-model-fb346b-kserve-6d4b9b47b5-f7mnx\" (UID: \"60b4b3fb-a758-45b8-933f-b62536b3661e\") " pod="ai-tenant-e2e-inf-b-fb346b/test-model-fb346b-kserve-6d4b9b47b5-f7mnx" Oct 31 04:58:40.315858 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:40.315835 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g82bq\" (UniqueName: \"kubernetes.io/projected/60b4b3fb-a758-45b8-933f-b62536b3661e-kube-api-access-g82bq\") pod \"test-model-fb346b-kserve-6d4b9b47b5-f7mnx\" (UID: \"60b4b3fb-a758-45b8-933f-b62536b3661e\") " pod="ai-tenant-e2e-inf-b-fb346b/test-model-fb346b-kserve-6d4b9b47b5-f7mnx" Oct 31 04:58:40.316069 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:40.316053 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60b4b3fb-a758-45b8-933f-b62536b3661e-dshm\") pod \"test-model-fb346b-kserve-6d4b9b47b5-f7mnx\" (UID: \"60b4b3fb-a758-45b8-933f-b62536b3661e\") " pod="ai-tenant-e2e-inf-b-fb346b/test-model-fb346b-kserve-6d4b9b47b5-f7mnx" Oct 31 04:58:40.317043 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:40.317019 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60b4b3fb-a758-45b8-933f-b62536b3661e-model-cache\") pod \"test-model-fb346b-kserve-6d4b9b47b5-f7mnx\" (UID: \"60b4b3fb-a758-45b8-933f-b62536b3661e\") " pod="ai-tenant-e2e-inf-b-fb346b/test-model-fb346b-kserve-6d4b9b47b5-f7mnx" Oct 31 04:58:40.317184 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:40.314689 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60b4b3fb-a758-45b8-933f-b62536b3661e-tmp-dir\") pod \"test-model-fb346b-kserve-6d4b9b47b5-f7mnx\" (UID: \"60b4b3fb-a758-45b8-933f-b62536b3661e\") " pod="ai-tenant-e2e-inf-b-fb346b/test-model-fb346b-kserve-6d4b9b47b5-f7mnx" Oct 31 04:58:40.327356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:40.323994 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60b4b3fb-a758-45b8-933f-b62536b3661e-dshm\") pod \"test-model-fb346b-kserve-6d4b9b47b5-f7mnx\" (UID: \"60b4b3fb-a758-45b8-933f-b62536b3661e\") " pod="ai-tenant-e2e-inf-b-fb346b/test-model-fb346b-kserve-6d4b9b47b5-f7mnx" Oct 31 04:58:40.328036 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:40.328004 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60b4b3fb-a758-45b8-933f-b62536b3661e-home\") pod \"test-model-fb346b-kserve-6d4b9b47b5-f7mnx\" (UID: \"60b4b3fb-a758-45b8-933f-b62536b3661e\") " pod="ai-tenant-e2e-inf-b-fb346b/test-model-fb346b-kserve-6d4b9b47b5-f7mnx" Oct 31 04:58:40.364289 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:40.364248 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60b4b3fb-a758-45b8-933f-b62536b3661e-tls-certs\") pod \"test-model-fb346b-kserve-6d4b9b47b5-f7mnx\" (UID: \"60b4b3fb-a758-45b8-933f-b62536b3661e\") " pod="ai-tenant-e2e-inf-b-fb346b/test-model-fb346b-kserve-6d4b9b47b5-f7mnx" Oct 31 04:58:40.371360 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:40.368057 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g82bq\" (UniqueName: \"kubernetes.io/projected/60b4b3fb-a758-45b8-933f-b62536b3661e-kube-api-access-g82bq\") pod \"test-model-fb346b-kserve-6d4b9b47b5-f7mnx\" (UID: \"60b4b3fb-a758-45b8-933f-b62536b3661e\") " pod="ai-tenant-e2e-inf-b-fb346b/test-model-fb346b-kserve-6d4b9b47b5-f7mnx" Oct 31 04:58:40.664324 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:40.664230 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fb346b/test-model-fb346b-kserve-6d4b9b47b5-f7mnx" Oct 31 04:58:41.010874 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:41.010839 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-fb346b/test-model-fb346b-kserve-6d4b9b47b5-f7mnx"] Oct 31 04:58:41.014481 ip-10-0-130-57 kubenswrapper[5019]: W1031 04:58:41.014447 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60b4b3fb_a758_45b8_933f_b62536b3661e.slice/crio-363c049e744dc8d3e47e613de4cf63670ed44df4c3d3d4d4fda38ddf6f4a30eb WatchSource:0}: Error finding container 363c049e744dc8d3e47e613de4cf63670ed44df4c3d3d4d4fda38ddf6f4a30eb: Status 404 returned error can't find the container with id 363c049e744dc8d3e47e613de4cf63670ed44df4c3d3d4d4fda38ddf6f4a30eb Oct 31 04:58:41.124646 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:41.124600 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fb346b/test-model-fb346b-kserve-6d4b9b47b5-f7mnx" event={"ID":"60b4b3fb-a758-45b8-933f-b62536b3661e","Type":"ContainerStarted","Data":"7f0bd5805834837c1605d6f20d1a5585dc99f0d232d0a8da0b671ace0998d661"} Oct 31 04:58:41.125072 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:41.124658 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fb346b/test-model-fb346b-kserve-6d4b9b47b5-f7mnx" event={"ID":"60b4b3fb-a758-45b8-933f-b62536b3661e","Type":"ContainerStarted","Data":"363c049e744dc8d3e47e613de4cf63670ed44df4c3d3d4d4fda38ddf6f4a30eb"} Oct 31 04:58:41.173551 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:41.173473 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-fb346b/test-model-fb346b-kserve-6d4b9b47b5-f7mnx" podStartSLOduration=1.173451922 podStartE2EDuration="1.173451922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:58:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:58:41.165800237 +0000 UTC m=+3275.922138278" watchObservedRunningTime="2026-10-31 04:58:41.173451922 +0000 UTC m=+3275.929789962" Oct 31 04:58:41.665034 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:41.664991 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-fb346b/test-model-fb346b-kserve-6d4b9b47b5-f7mnx" Oct 31 04:58:50.664858 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:50.664768 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-fb346b/test-model-fb346b-kserve-6d4b9b47b5-f7mnx" Oct 31 04:58:50.683520 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:50.683481 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-fb346b/test-model-fb346b-kserve-6d4b9b47b5-f7mnx" Oct 31 04:58:51.249615 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:58:51.249580 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-fb346b/test-model-fb346b-kserve-6d4b9b47b5-f7mnx" Oct 31 04:58:52.401965 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:58:52.401916 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:59:06.740023 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:06.739983 5019 generic.go:356] "Generic (PLEG): container finished" podID="a9eb4117-200f-4720-9047-a79a3b227cb3" containerID="9812edb363bf790a0f082efc6ce5626754191c3c58ce55fa197a367921e130a7" exitCode=0 Oct 31 04:59:06.740523 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:06.740049 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff" event={"ID":"a9eb4117-200f-4720-9047-a79a3b227cb3","Type":"ContainerDied","Data":"9812edb363bf790a0f082efc6ce5626754191c3c58ce55fa197a367921e130a7"} Oct 31 04:59:06.740592 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:06.740575 5019 scope.go:122] "RemoveContainer" containerID="9812edb363bf790a0f082efc6ce5626754191c3c58ce55fa197a367921e130a7" Oct 31 04:59:06.742921 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:06.742876 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:59:06.743065 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:06.742978 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:59:06.743065 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:06.743030 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:59:07.761204 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:07.761158 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff" event={"ID":"a9eb4117-200f-4720-9047-a79a3b227cb3","Type":"ContainerStarted","Data":"1f2e5804df9faef55dc8046ab52ebda9b96754213f699733811699119b1f1c24"} Oct 31 04:59:08.343700 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:08.343655 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff" Oct 31 04:59:08.344933 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:08.344906 5019 patch_prober.go:29] interesting pod/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" start-of-body= Oct 31 04:59:08.345013 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:08.344974 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff" podUID="a9eb4117-200f-4720-9047-a79a3b227cb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" Oct 31 04:59:08.636674 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:08.636557 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff" Oct 31 04:59:09.343816 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:09.343776 5019 patch_prober.go:29] interesting pod/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" start-of-body= Oct 31 04:59:09.344311 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:09.343848 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff" podUID="a9eb4117-200f-4720-9047-a79a3b227cb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" Oct 31 04:59:09.433414 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:09.433385 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s88kp_3945ff65-e877-4372-9334-3968c68b7295/console-operator/2.log" Oct 31 04:59:09.433627 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:09.433385 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s88kp_3945ff65-e877-4372-9334-3968c68b7295/console-operator/2.log" Oct 31 04:59:10.344563 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:10.344515 5019 patch_prober.go:29] interesting pod/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" start-of-body= Oct 31 04:59:10.345000 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:10.344580 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff" podUID="a9eb4117-200f-4720-9047-a79a3b227cb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" Oct 31 04:59:11.344366 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:11.344290 5019 patch_prober.go:29] interesting pod/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" start-of-body= Oct 31 04:59:11.344683 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:11.344378 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff" podUID="a9eb4117-200f-4720-9047-a79a3b227cb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" Oct 31 04:59:12.108528 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:12.108484 5019 scope.go:122] "RemoveContainer" containerID="50a419b59df09321c9ec81fe8a38261098415cb8e568909f3d2230567e4150d9" Oct 31 04:59:12.344604 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:12.344564 5019 patch_prober.go:29] interesting pod/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" start-of-body= Oct 31 04:59:12.344788 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:12.344623 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff" podUID="a9eb4117-200f-4720-9047-a79a3b227cb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" Oct 31 04:59:13.343938 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:13.343890 5019 patch_prober.go:29] interesting pod/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" start-of-body= Oct 31 04:59:13.344152 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:13.344007 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff" podUID="a9eb4117-200f-4720-9047-a79a3b227cb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" Oct 31 04:59:14.350373 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:14.350255 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff" Oct 31 04:59:14.351766 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:14.351737 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff" Oct 31 04:59:22.743052 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:59:22.743006 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:59:26.034296 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:26.034259 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ba6669d-xfmmk"] Oct 31 04:59:26.049467 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:26.049428 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ba6669d-xfmmk"] Oct 31 04:59:27.414024 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:27.413980 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="457a668d-3818-4eaa-b1a2-63a9972d2de8" path="/var/lib/kubelet/pods/457a668d-3818-4eaa-b1a2-63a9972d2de8/volumes" Oct 31 04:59:35.041154 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:35.032820 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55d0e8d0-cmmwx"] Oct 31 04:59:35.046968 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:35.046933 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55d0e8d0-cmmwx"] Oct 31 04:59:35.423675 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:35.423563 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="180073ab-a2eb-47ed-a8bf-9adf41d715bd" path="/var/lib/kubelet/pods/180073ab-a2eb-47ed-a8bf-9adf41d715bd/volumes" Oct 31 04:59:36.033162 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:36.033120 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b281a9df-9nr7h"] Oct 31 04:59:36.045592 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:36.045553 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b281a9df-9nr7h"] Oct 31 04:59:37.415351 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:37.414358 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="379bd26a-c57d-4d08-a0c5-2eed8732e363" path="/var/lib/kubelet/pods/379bd26a-c57d-4d08-a0c5-2eed8732e363/volumes" Oct 31 04:59:46.035238 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:46.035194 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82a48179-vgtsz"] Oct 31 04:59:46.046876 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:46.046829 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82a48179-vgtsz"] Oct 31 04:59:46.606116 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:46.606076 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k"] Oct 31 04:59:46.606402 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:46.606353 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k" podUID="29c8fd65-abc5-4bc2-8ea8-02d11e203e36" containerName="istio-proxy" containerID="cri-o://938a6ddb68d4afc5adbb6e31684b7c235d50c8777545539eb62215f6c6ac30a5" gracePeriod=30 Oct 31 04:59:46.621702 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:46.621658 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4"] Oct 31 04:59:46.621955 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:46.621925 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4" podUID="7de5a068-ea2b-4e78-8c43-3ae8dddabfd7" containerName="istio-proxy" containerID="cri-o://681977346a36ec8b3ad827764ccb563ce12fa653303fb275231a79c591325f46" gracePeriod=30 Oct 31 04:59:47.420520 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:47.420476 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de3b023e-f99d-45f7-ac11-fcbea969d26d" path="/var/lib/kubelet/pods/de3b023e-f99d-45f7-ac11-fcbea969d26d/volumes" Oct 31 04:59:49.030021 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:49.029981 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7a0ce4f-smczk"] Oct 31 04:59:49.035425 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:49.035396 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7a0ce4f-smczk"] Oct 31 04:59:49.785356 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:49.759008 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbba30e9-13f9-4518-a38d-de9fc33b21f1" path="/var/lib/kubelet/pods/bbba30e9-13f9-4518-a38d-de9fc33b21f1/volumes" Oct 31 04:59:52.063994 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.063953 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k" Oct 31 04:59:52.105716 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.105681 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-workload-certs\") pod \"29c8fd65-abc5-4bc2-8ea8-02d11e203e36\" (UID: \"29c8fd65-abc5-4bc2-8ea8-02d11e203e36\") " Oct 31 04:59:52.105860 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.105770 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-istio-podinfo\") pod \"29c8fd65-abc5-4bc2-8ea8-02d11e203e36\" (UID: \"29c8fd65-abc5-4bc2-8ea8-02d11e203e36\") " Oct 31 04:59:52.105860 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.105814 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-kube-api-access-mqll5\" (UniqueName: \"kubernetes.io/projected/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-kube-api-access-mqll5\") pod \"29c8fd65-abc5-4bc2-8ea8-02d11e203e36\" (UID: \"29c8fd65-abc5-4bc2-8ea8-02d11e203e36\") " Oct 31 04:59:52.106049 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.106028 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-istio-envoy\") pod \"29c8fd65-abc5-4bc2-8ea8-02d11e203e36\" (UID: \"29c8fd65-abc5-4bc2-8ea8-02d11e203e36\") " Oct 31 04:59:52.106124 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.106103 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-istio-data\") pod \"29c8fd65-abc5-4bc2-8ea8-02d11e203e36\" (UID: \"29c8fd65-abc5-4bc2-8ea8-02d11e203e36\") " Oct 31 04:59:52.106252 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.106240 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-workload-socket\") pod \"29c8fd65-abc5-4bc2-8ea8-02d11e203e36\" (UID: \"29c8fd65-abc5-4bc2-8ea8-02d11e203e36\") " Oct 31 04:59:52.106320 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.106290 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-istio-token\" (UniqueName: \"kubernetes.io/projected/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-istio-token\") pod \"29c8fd65-abc5-4bc2-8ea8-02d11e203e36\" (UID: \"29c8fd65-abc5-4bc2-8ea8-02d11e203e36\") " Oct 31 04:59:52.106421 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.106321 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-istiod-ca-cert\") pod \"29c8fd65-abc5-4bc2-8ea8-02d11e203e36\" (UID: \"29c8fd65-abc5-4bc2-8ea8-02d11e203e36\") " Oct 31 04:59:52.106966 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.106570 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-workload-socket" pod "29c8fd65-abc5-4bc2-8ea8-02d11e203e36" (UID: "29c8fd65-abc5-4bc2-8ea8-02d11e203e36"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:59:52.106966 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.106922 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-credential-socket\") pod \"29c8fd65-abc5-4bc2-8ea8-02d11e203e36\" (UID: \"29c8fd65-abc5-4bc2-8ea8-02d11e203e36\") " Oct 31 04:59:52.106966 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.106669 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-istio-data" pod "29c8fd65-abc5-4bc2-8ea8-02d11e203e36" (UID: "29c8fd65-abc5-4bc2-8ea8-02d11e203e36"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:59:52.106966 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.106876 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-istiod-ca-cert" pod "29c8fd65-abc5-4bc2-8ea8-02d11e203e36" (UID: "29c8fd65-abc5-4bc2-8ea8-02d11e203e36"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:59:52.107203 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.106937 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-workload-certs" pod "29c8fd65-abc5-4bc2-8ea8-02d11e203e36" (UID: "29c8fd65-abc5-4bc2-8ea8-02d11e203e36"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:59:52.107254 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.107216 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-credential-socket" pod "29c8fd65-abc5-4bc2-8ea8-02d11e203e36" (UID: "29c8fd65-abc5-4bc2-8ea8-02d11e203e36"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:59:52.108038 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.108013 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:59:52.108155 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.108045 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:59:52.108155 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.108063 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:59:52.108155 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.108078 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:59:52.108155 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.108093 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:59:52.109619 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.109586 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-istio-podinfo" pod "29c8fd65-abc5-4bc2-8ea8-02d11e203e36" (UID: "29c8fd65-abc5-4bc2-8ea8-02d11e203e36"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:59:52.109772 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.109750 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-kube-api-access-mqll5" pod "29c8fd65-abc5-4bc2-8ea8-02d11e203e36" (UID: "29c8fd65-abc5-4bc2-8ea8-02d11e203e36"). InnerVolumeSpecName "kube-api-access-mqll5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:59:52.110846 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.110816 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-istio-token" pod "29c8fd65-abc5-4bc2-8ea8-02d11e203e36" (UID: "29c8fd65-abc5-4bc2-8ea8-02d11e203e36"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:59:52.111147 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.111124 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-istio-envoy" pod "29c8fd65-abc5-4bc2-8ea8-02d11e203e36" (UID: "29c8fd65-abc5-4bc2-8ea8-02d11e203e36"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:59:52.139746 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.139707 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4" Oct 31 04:59:52.209174 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.209097 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-workload-certs\") pod \"7de5a068-ea2b-4e78-8c43-3ae8dddabfd7\" (UID: \"7de5a068-ea2b-4e78-8c43-3ae8dddabfd7\") " Oct 31 04:59:52.209446 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.209421 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-istio-data\") pod \"7de5a068-ea2b-4e78-8c43-3ae8dddabfd7\" (UID: \"7de5a068-ea2b-4e78-8c43-3ae8dddabfd7\") " Oct 31 04:59:52.209553 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.209475 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-kube-api-access-h8b85\" (UniqueName: \"kubernetes.io/projected/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-kube-api-access-h8b85\") pod \"7de5a068-ea2b-4e78-8c43-3ae8dddabfd7\" (UID: \"7de5a068-ea2b-4e78-8c43-3ae8dddabfd7\") " Oct 31 04:59:52.209553 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.209522 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-istio-podinfo\") pod \"7de5a068-ea2b-4e78-8c43-3ae8dddabfd7\" (UID: \"7de5a068-ea2b-4e78-8c43-3ae8dddabfd7\") " Oct 31 04:59:52.209683 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.209565 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-credential-socket\") pod \"7de5a068-ea2b-4e78-8c43-3ae8dddabfd7\" (UID: \"7de5a068-ea2b-4e78-8c43-3ae8dddabfd7\") " Oct 31 04:59:52.209746 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.209684 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-workload-socket\") pod \"7de5a068-ea2b-4e78-8c43-3ae8dddabfd7\" (UID: \"7de5a068-ea2b-4e78-8c43-3ae8dddabfd7\") " Oct 31 04:59:52.209868 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.209853 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-istiod-ca-cert\") pod \"7de5a068-ea2b-4e78-8c43-3ae8dddabfd7\" (UID: \"7de5a068-ea2b-4e78-8c43-3ae8dddabfd7\") " Oct 31 04:59:52.209931 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.209895 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-istio-token\" (UniqueName: \"kubernetes.io/projected/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-istio-token\") pod \"7de5a068-ea2b-4e78-8c43-3ae8dddabfd7\" (UID: \"7de5a068-ea2b-4e78-8c43-3ae8dddabfd7\") " Oct 31 04:59:52.210097 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.210080 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-istio-envoy\") pod \"7de5a068-ea2b-4e78-8c43-3ae8dddabfd7\" (UID: \"7de5a068-ea2b-4e78-8c43-3ae8dddabfd7\") " Oct 31 04:59:52.211097 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.211072 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:59:52.211213 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.211100 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:59:52.211213 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.211114 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqll5\" (UniqueName: \"kubernetes.io/projected/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-kube-api-access-mqll5\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:59:52.211213 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.211127 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29c8fd65-abc5-4bc2-8ea8-02d11e203e36-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:59:52.212647 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.212250 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-istiod-ca-cert" pod "7de5a068-ea2b-4e78-8c43-3ae8dddabfd7" (UID: "7de5a068-ea2b-4e78-8c43-3ae8dddabfd7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:59:52.212828 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.212803 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-workload-socket" pod "7de5a068-ea2b-4e78-8c43-3ae8dddabfd7" (UID: "7de5a068-ea2b-4e78-8c43-3ae8dddabfd7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:59:52.213139 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.212992 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-workload-certs" pod "7de5a068-ea2b-4e78-8c43-3ae8dddabfd7" (UID: "7de5a068-ea2b-4e78-8c43-3ae8dddabfd7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:59:52.213262 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.213165 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-istio-data" pod "7de5a068-ea2b-4e78-8c43-3ae8dddabfd7" (UID: "7de5a068-ea2b-4e78-8c43-3ae8dddabfd7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:59:52.213852 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.213830 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-credential-socket" pod "7de5a068-ea2b-4e78-8c43-3ae8dddabfd7" (UID: "7de5a068-ea2b-4e78-8c43-3ae8dddabfd7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:59:52.218271 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.218232 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-istio-envoy" pod "7de5a068-ea2b-4e78-8c43-3ae8dddabfd7" (UID: "7de5a068-ea2b-4e78-8c43-3ae8dddabfd7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:59:52.221613 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.221547 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-istio-podinfo" pod "7de5a068-ea2b-4e78-8c43-3ae8dddabfd7" (UID: "7de5a068-ea2b-4e78-8c43-3ae8dddabfd7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:59:52.222587 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.222547 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-kube-api-access-h8b85" pod "7de5a068-ea2b-4e78-8c43-3ae8dddabfd7" (UID: "7de5a068-ea2b-4e78-8c43-3ae8dddabfd7"). InnerVolumeSpecName "kube-api-access-h8b85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:59:52.223319 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.223298 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-istio-token" pod "7de5a068-ea2b-4e78-8c43-3ae8dddabfd7" (UID: "7de5a068-ea2b-4e78-8c43-3ae8dddabfd7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:59:52.350722 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.350464 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:59:52.350961 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.350938 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:59:52.351083 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.351068 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8b85\" (UniqueName: \"kubernetes.io/projected/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-kube-api-access-h8b85\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:59:52.351174 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.351160 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:59:52.351256 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.351246 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:59:52.351354 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.351326 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:59:52.351455 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.351445 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:59:52.351545 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.351526 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:59:52.351637 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.351626 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 04:59:52.400468 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:59:52.400387 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 04:59:52.698890 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.698847 5019 generic.go:356] "Generic (PLEG): container finished" podID="29c8fd65-abc5-4bc2-8ea8-02d11e203e36" containerID="938a6ddb68d4afc5adbb6e31684b7c235d50c8777545539eb62215f6c6ac30a5" exitCode=0 Oct 31 04:59:52.699076 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.698919 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k" Oct 31 04:59:52.699076 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.698926 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k" event={"ID":"29c8fd65-abc5-4bc2-8ea8-02d11e203e36","Type":"ContainerDied","Data":"938a6ddb68d4afc5adbb6e31684b7c235d50c8777545539eb62215f6c6ac30a5"} Oct 31 04:59:52.699076 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.699001 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k" event={"ID":"29c8fd65-abc5-4bc2-8ea8-02d11e203e36","Type":"ContainerDied","Data":"ad5f4c9f439a4b2b185cf732c558e1f686b4542815e645a8353c4b1733e3bf99"} Oct 31 04:59:52.699076 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.699035 5019 scope.go:122] "RemoveContainer" containerID="938a6ddb68d4afc5adbb6e31684b7c235d50c8777545539eb62215f6c6ac30a5" Oct 31 04:59:52.701925 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.701896 5019 generic.go:356] "Generic (PLEG): container finished" podID="7de5a068-ea2b-4e78-8c43-3ae8dddabfd7" containerID="681977346a36ec8b3ad827764ccb563ce12fa653303fb275231a79c591325f46" exitCode=0 Oct 31 04:59:52.702074 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.701952 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4" event={"ID":"7de5a068-ea2b-4e78-8c43-3ae8dddabfd7","Type":"ContainerDied","Data":"681977346a36ec8b3ad827764ccb563ce12fa653303fb275231a79c591325f46"} Oct 31 04:59:52.702074 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.701976 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4" Oct 31 04:59:52.702074 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.702014 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4" event={"ID":"7de5a068-ea2b-4e78-8c43-3ae8dddabfd7","Type":"ContainerDied","Data":"19a6c376e6ccc769905e368d91d7ba087f27ab51d20f611722f3eb205124b9c6"} Oct 31 04:59:52.741103 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.741070 5019 scope.go:122] "RemoveContainer" containerID="938a6ddb68d4afc5adbb6e31684b7c235d50c8777545539eb62215f6c6ac30a5" Oct 31 04:59:52.741546 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:59:52.741517 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"938a6ddb68d4afc5adbb6e31684b7c235d50c8777545539eb62215f6c6ac30a5\": container with ID starting with 938a6ddb68d4afc5adbb6e31684b7c235d50c8777545539eb62215f6c6ac30a5 not found: ID does not exist" containerID="938a6ddb68d4afc5adbb6e31684b7c235d50c8777545539eb62215f6c6ac30a5" Oct 31 04:59:52.741676 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.741559 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"938a6ddb68d4afc5adbb6e31684b7c235d50c8777545539eb62215f6c6ac30a5"} err="failed to get container status \"938a6ddb68d4afc5adbb6e31684b7c235d50c8777545539eb62215f6c6ac30a5\": rpc error: code = NotFound desc = could not find container \"938a6ddb68d4afc5adbb6e31684b7c235d50c8777545539eb62215f6c6ac30a5\": container with ID starting with 938a6ddb68d4afc5adbb6e31684b7c235d50c8777545539eb62215f6c6ac30a5 not found: ID does not exist" Oct 31 04:59:52.741676 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.741587 5019 scope.go:122] "RemoveContainer" containerID="681977346a36ec8b3ad827764ccb563ce12fa653303fb275231a79c591325f46" Oct 31 04:59:52.759398 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.759362 5019 scope.go:122] "RemoveContainer" containerID="681977346a36ec8b3ad827764ccb563ce12fa653303fb275231a79c591325f46" Oct 31 04:59:52.759869 ip-10-0-130-57 kubenswrapper[5019]: E1031 04:59:52.759841 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"681977346a36ec8b3ad827764ccb563ce12fa653303fb275231a79c591325f46\": container with ID starting with 681977346a36ec8b3ad827764ccb563ce12fa653303fb275231a79c591325f46 not found: ID does not exist" containerID="681977346a36ec8b3ad827764ccb563ce12fa653303fb275231a79c591325f46" Oct 31 04:59:52.759970 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.759911 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"681977346a36ec8b3ad827764ccb563ce12fa653303fb275231a79c591325f46"} err="failed to get container status \"681977346a36ec8b3ad827764ccb563ce12fa653303fb275231a79c591325f46\": rpc error: code = NotFound desc = could not find container \"681977346a36ec8b3ad827764ccb563ce12fa653303fb275231a79c591325f46\": container with ID starting with 681977346a36ec8b3ad827764ccb563ce12fa653303fb275231a79c591325f46 not found: ID does not exist" Oct 31 04:59:52.772011 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.771966 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4"] Oct 31 04:59:52.784122 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.784088 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qfbk4"] Oct 31 04:59:52.810516 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.810474 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k"] Oct 31 04:59:52.816683 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:52.816644 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-dlm7k"] Oct 31 04:59:53.433961 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:53.433921 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29c8fd65-abc5-4bc2-8ea8-02d11e203e36" path="/var/lib/kubelet/pods/29c8fd65-abc5-4bc2-8ea8-02d11e203e36/volumes" Oct 31 04:59:53.434690 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:53.434671 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7de5a068-ea2b-4e78-8c43-3ae8dddabfd7" path="/var/lib/kubelet/pods/7de5a068-ea2b-4e78-8c43-3ae8dddabfd7/volumes" Oct 31 04:59:53.716496 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:53.716464 5019 generic.go:356] "Generic (PLEG): container finished" podID="e33bca6f-714f-44a3-bf0c-e99ddb526c67" containerID="741d41265c76a6aa2bda535cd07bf196705321b2d1deb6018cd9dd01b467ea54" exitCode=0 Oct 31 04:59:53.716683 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:53.716533 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24" event={"ID":"e33bca6f-714f-44a3-bf0c-e99ddb526c67","Type":"ContainerDied","Data":"741d41265c76a6aa2bda535cd07bf196705321b2d1deb6018cd9dd01b467ea54"} Oct 31 04:59:53.717142 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:53.717123 5019 scope.go:122] "RemoveContainer" containerID="741d41265c76a6aa2bda535cd07bf196705321b2d1deb6018cd9dd01b467ea54" Oct 31 04:59:53.719707 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:53.719671 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:59:53.719793 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:53.719752 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:59:53.719793 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:53.719786 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 04:59:54.732303 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:54.732260 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24" event={"ID":"e33bca6f-714f-44a3-bf0c-e99ddb526c67","Type":"ContainerStarted","Data":"f2238467d3faf57e776bc406adaef9495cf2fdc989606c413288bf9c07850aad"} Oct 31 04:59:54.910483 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:54.910441 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24" Oct 31 04:59:55.697344 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:55.696780 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24" Oct 31 04:59:55.697344 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:55.696961 5019 patch_prober.go:29] interesting pod/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.128:15021/healthz/ready\": dial tcp 10.132.1.128:15021: connect: connection refused" start-of-body= Oct 31 04:59:55.697344 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:55.697012 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24" podUID="e33bca6f-714f-44a3-bf0c-e99ddb526c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.128:15021/healthz/ready\": dial tcp 10.132.1.128:15021: connect: connection refused" Oct 31 04:59:56.697417 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:56.693870 5019 patch_prober.go:29] interesting pod/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.128:15021/healthz/ready\": dial tcp 10.132.1.128:15021: connect: connection refused" start-of-body= Oct 31 04:59:56.697417 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:56.693940 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24" podUID="e33bca6f-714f-44a3-bf0c-e99ddb526c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.128:15021/healthz/ready\": dial tcp 10.132.1.128:15021: connect: connection refused" Oct 31 04:59:57.694357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:57.693773 5019 patch_prober.go:29] interesting pod/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.128:15021/healthz/ready\": dial tcp 10.132.1.128:15021: connect: connection refused" start-of-body= Oct 31 04:59:57.694357 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:57.693859 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24" podUID="e33bca6f-714f-44a3-bf0c-e99ddb526c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.128:15021/healthz/ready\": dial tcp 10.132.1.128:15021: connect: connection refused" Oct 31 04:59:58.696801 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:58.696756 5019 patch_prober.go:29] interesting pod/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.128:15021/healthz/ready\": dial tcp 10.132.1.128:15021: connect: connection refused" start-of-body= Oct 31 04:59:58.697462 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:58.697422 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24" podUID="e33bca6f-714f-44a3-bf0c-e99ddb526c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.128:15021/healthz/ready\": dial tcp 10.132.1.128:15021: connect: connection refused" Oct 31 04:59:59.040581 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:59.040535 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1a932c0-nlzns"] Oct 31 04:59:59.052441 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:59.052397 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1a932c0-nlzns"] Oct 31 04:59:59.418047 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:59.417811 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7543c659-84f2-4c09-aa0f-d517f487eaf4" path="/var/lib/kubelet/pods/7543c659-84f2-4c09-aa0f-d517f487eaf4/volumes" Oct 31 04:59:59.694660 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:59.694561 5019 patch_prober.go:29] interesting pod/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.128:15021/healthz/ready\": dial tcp 10.132.1.128:15021: connect: connection refused" start-of-body= Oct 31 04:59:59.694995 ip-10-0-130-57 kubenswrapper[5019]: I1031 04:59:59.694963 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24" podUID="e33bca6f-714f-44a3-bf0c-e99ddb526c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.128:15021/healthz/ready\": dial tcp 10.132.1.128:15021: connect: connection refused" Oct 31 05:00:00.164411 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.164328 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a04e4e-29890380-jgchh"] Oct 31 05:00:00.168405 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.168370 5019 cpu_manager.go:414] "containerMap: removing container" podUID="7de5a068-ea2b-4e78-8c43-3ae8dddabfd7" containerName="istio-proxy" Oct 31 05:00:00.168405 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.168402 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7de5a068-ea2b-4e78-8c43-3ae8dddabfd7" containerName="istio-proxy" Oct 31 05:00:00.168644 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.168513 5019 cpu_manager.go:414] "containerMap: removing container" podUID="29c8fd65-abc5-4bc2-8ea8-02d11e203e36" containerName="istio-proxy" Oct 31 05:00:00.168644 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.168521 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29c8fd65-abc5-4bc2-8ea8-02d11e203e36" containerName="istio-proxy" Oct 31 05:00:00.168790 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.168773 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="29c8fd65-abc5-4bc2-8ea8-02d11e203e36" containerName="istio-proxy" Oct 31 05:00:00.168859 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.168797 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="7de5a068-ea2b-4e78-8c43-3ae8dddabfd7" containerName="istio-proxy" Oct 31 05:00:00.178355 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.175348 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-47c409-29890380-cqm6g"] Oct 31 05:00:00.178355 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.175455 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a04e4e-29890380-jgchh" Oct 31 05:00:00.185355 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.181830 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890380-rmxrz"] Oct 31 05:00:00.185355 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.182015 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-47c409-29890380-cqm6g" Oct 31 05:00:00.193767 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.190784 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-567ca6-29890380-kcwdg"] Oct 31 05:00:00.194204 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.194180 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890380-rmxrz" Oct 31 05:00:00.199471 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.199430 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890380-ldsqw"] Oct 31 05:00:00.199868 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.199814 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-567ca6-29890380-kcwdg" Oct 31 05:00:00.208373 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.207323 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890380-ldsqw" Oct 31 05:00:00.222252 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.213170 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:00:00.222252 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.213291 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:00:00.225203 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.223499 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a04e4e-29890380-jgchh"] Oct 31 05:00:00.235635 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.235034 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-47c409-29890380-cqm6g"] Oct 31 05:00:00.248622 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.245046 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890380-rmxrz"] Oct 31 05:00:00.251956 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.251924 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 05:00:00.262975 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.252379 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890380-ldsqw"] Oct 31 05:00:00.272061 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.271824 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-567ca6-29890380-kcwdg"] Oct 31 05:00:00.274433 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.274398 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzstl\" (UniqueName: \"kubernetes.io/projected/982ad40e-a719-4f61-be85-ea665503c299-kube-api-access-xzstl\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890380-rmxrz\" (UID: \"982ad40e-a719-4f61-be85-ea665503c299\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890380-rmxrz" Oct 31 05:00:00.274661 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.274638 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smndt\" (UniqueName: \"kubernetes.io/projected/d13131fa-5137-4401-aef8-d53fd8bd4e5d-kube-api-access-smndt\") pod \"maas-api-key-cleanup-e2e-extmod-a-47c409-29890380-cqm6g\" (UID: \"d13131fa-5137-4401-aef8-d53fd8bd4e5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-47c409-29890380-cqm6g" Oct 31 05:00:00.274744 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.274691 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/982ad40e-a719-4f61-be85-ea665503c299-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890380-rmxrz\" (UID: \"982ad40e-a719-4f61-be85-ea665503c299\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890380-rmxrz" Oct 31 05:00:00.274817 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.274742 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/703c28bc-a603-4001-9538-4a751cf9b7c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-567ca6-29890380-kcwdg\" (UID: \"703c28bc-a603-4001-9538-4a751cf9b7c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-567ca6-29890380-kcwdg" Oct 31 05:00:00.274817 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.274776 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1e9a647-0e9e-4528-a5a6-7022b244f1ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-8973e2-29890380-ldsqw\" (UID: \"e1e9a647-0e9e-4528-a5a6-7022b244f1ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890380-ldsqw" Oct 31 05:00:00.274933 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.274837 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdfcg\" (UniqueName: \"kubernetes.io/projected/703c28bc-a603-4001-9538-4a751cf9b7c0-kube-api-access-xdfcg\") pod \"maas-api-key-cleanup-e2e-extmod-b-567ca6-29890380-kcwdg\" (UID: \"703c28bc-a603-4001-9538-4a751cf9b7c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-567ca6-29890380-kcwdg" Oct 31 05:00:00.274995 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.274952 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59sfq\" (UniqueName: \"kubernetes.io/projected/e1e9a647-0e9e-4528-a5a6-7022b244f1ba-kube-api-access-59sfq\") pod \"maas-api-key-cleanup-e2e-extmod-a-8973e2-29890380-ldsqw\" (UID: \"e1e9a647-0e9e-4528-a5a6-7022b244f1ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890380-ldsqw" Oct 31 05:00:00.275061 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.274993 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b6b5e89-5900-4db2-8552-e4f029de7573-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-a04e4e-29890380-jgchh\" (UID: \"0b6b5e89-5900-4db2-8552-e4f029de7573\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a04e4e-29890380-jgchh" Oct 31 05:00:00.275061 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.275037 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb8ht\" (UniqueName: \"kubernetes.io/projected/0b6b5e89-5900-4db2-8552-e4f029de7573-kube-api-access-lb8ht\") pod \"maas-api-key-cleanup-e2e-extmod-b-a04e4e-29890380-jgchh\" (UID: \"0b6b5e89-5900-4db2-8552-e4f029de7573\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a04e4e-29890380-jgchh" Oct 31 05:00:00.275177 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.275070 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d13131fa-5137-4401-aef8-d53fd8bd4e5d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-47c409-29890380-cqm6g\" (UID: \"d13131fa-5137-4401-aef8-d53fd8bd4e5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-47c409-29890380-cqm6g" Oct 31 05:00:00.308672 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.308386 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890380-v7gl7"] Oct 31 05:00:00.321314 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.321271 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890380-v7gl7" Oct 31 05:00:00.322069 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.321170 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-401761-29890380-zvx8f"] Oct 31 05:00:00.340929 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.340888 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-401761-29890380-zvx8f" Oct 31 05:00:00.363832 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.361737 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890380-7g9d5"] Oct 31 05:00:00.375616 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.375581 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-031745-29890380-9jpj5"] Oct 31 05:00:00.376826 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.376792 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smndt\" (UniqueName: \"kubernetes.io/projected/d13131fa-5137-4401-aef8-d53fd8bd4e5d-kube-api-access-smndt\") pod \"maas-api-key-cleanup-e2e-extmod-a-47c409-29890380-cqm6g\" (UID: \"d13131fa-5137-4401-aef8-d53fd8bd4e5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-47c409-29890380-cqm6g" Oct 31 05:00:00.376991 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.376845 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/982ad40e-a719-4f61-be85-ea665503c299-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890380-rmxrz\" (UID: \"982ad40e-a719-4f61-be85-ea665503c299\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890380-rmxrz" Oct 31 05:00:00.376991 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.376898 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/703c28bc-a603-4001-9538-4a751cf9b7c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-567ca6-29890380-kcwdg\" (UID: \"703c28bc-a603-4001-9538-4a751cf9b7c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-567ca6-29890380-kcwdg" Oct 31 05:00:00.376991 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.376924 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1e9a647-0e9e-4528-a5a6-7022b244f1ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-8973e2-29890380-ldsqw\" (UID: \"e1e9a647-0e9e-4528-a5a6-7022b244f1ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890380-ldsqw" Oct 31 05:00:00.376991 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.376977 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdfcg\" (UniqueName: \"kubernetes.io/projected/703c28bc-a603-4001-9538-4a751cf9b7c0-kube-api-access-xdfcg\") pod \"maas-api-key-cleanup-e2e-extmod-b-567ca6-29890380-kcwdg\" (UID: \"703c28bc-a603-4001-9538-4a751cf9b7c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-567ca6-29890380-kcwdg" Oct 31 05:00:00.377230 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.377010 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k6h9\" (UniqueName: \"kubernetes.io/projected/4fd6b126-7a7c-4825-a374-5dec6fb94d06-kube-api-access-5k6h9\") pod \"maas-api-key-cleanup-e2e-extmod-a-401761-29890380-zvx8f\" (UID: \"4fd6b126-7a7c-4825-a374-5dec6fb94d06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-401761-29890380-zvx8f" Oct 31 05:00:00.377230 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.377118 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59sfq\" (UniqueName: \"kubernetes.io/projected/e1e9a647-0e9e-4528-a5a6-7022b244f1ba-kube-api-access-59sfq\") pod \"maas-api-key-cleanup-e2e-extmod-a-8973e2-29890380-ldsqw\" (UID: \"e1e9a647-0e9e-4528-a5a6-7022b244f1ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890380-ldsqw" Oct 31 05:00:00.377230 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.377158 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b6b5e89-5900-4db2-8552-e4f029de7573-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-a04e4e-29890380-jgchh\" (UID: \"0b6b5e89-5900-4db2-8552-e4f029de7573\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a04e4e-29890380-jgchh" Oct 31 05:00:00.377230 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.377193 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb8ht\" (UniqueName: \"kubernetes.io/projected/0b6b5e89-5900-4db2-8552-e4f029de7573-kube-api-access-lb8ht\") pod \"maas-api-key-cleanup-e2e-extmod-b-a04e4e-29890380-jgchh\" (UID: \"0b6b5e89-5900-4db2-8552-e4f029de7573\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a04e4e-29890380-jgchh" Oct 31 05:00:00.377230 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.377217 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d13131fa-5137-4401-aef8-d53fd8bd4e5d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-47c409-29890380-cqm6g\" (UID: \"d13131fa-5137-4401-aef8-d53fd8bd4e5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-47c409-29890380-cqm6g" Oct 31 05:00:00.377467 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.377370 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzstl\" (UniqueName: \"kubernetes.io/projected/982ad40e-a719-4f61-be85-ea665503c299-kube-api-access-xzstl\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890380-rmxrz\" (UID: \"982ad40e-a719-4f61-be85-ea665503c299\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890380-rmxrz" Oct 31 05:00:00.377467 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.377453 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmczm\" (UniqueName: \"kubernetes.io/projected/f59877aa-80e8-418d-9617-2f9bb9c8f3ef-kube-api-access-hmczm\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890380-v7gl7\" (UID: \"f59877aa-80e8-418d-9617-2f9bb9c8f3ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890380-v7gl7" Oct 31 05:00:00.377543 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.377511 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f59877aa-80e8-418d-9617-2f9bb9c8f3ef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890380-v7gl7\" (UID: \"f59877aa-80e8-418d-9617-2f9bb9c8f3ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890380-v7gl7" Oct 31 05:00:00.377579 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.377554 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fd6b126-7a7c-4825-a374-5dec6fb94d06-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-401761-29890380-zvx8f\" (UID: \"4fd6b126-7a7c-4825-a374-5dec6fb94d06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-401761-29890380-zvx8f" Oct 31 05:00:00.377916 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.377892 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890380-7g9d5" Oct 31 05:00:00.378791 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.378766 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/982ad40e-a719-4f61-be85-ea665503c299-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890380-rmxrz\" (UID: \"982ad40e-a719-4f61-be85-ea665503c299\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890380-rmxrz" Oct 31 05:00:00.379610 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.379585 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/703c28bc-a603-4001-9538-4a751cf9b7c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-567ca6-29890380-kcwdg\" (UID: \"703c28bc-a603-4001-9538-4a751cf9b7c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-567ca6-29890380-kcwdg" Oct 31 05:00:00.380394 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.380372 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1e9a647-0e9e-4528-a5a6-7022b244f1ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-8973e2-29890380-ldsqw\" (UID: \"e1e9a647-0e9e-4528-a5a6-7022b244f1ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890380-ldsqw" Oct 31 05:00:00.381594 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.381569 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b6b5e89-5900-4db2-8552-e4f029de7573-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-a04e4e-29890380-jgchh\" (UID: \"0b6b5e89-5900-4db2-8552-e4f029de7573\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a04e4e-29890380-jgchh" Oct 31 05:00:00.382503 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.382478 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d13131fa-5137-4401-aef8-d53fd8bd4e5d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-47c409-29890380-cqm6g\" (UID: \"d13131fa-5137-4401-aef8-d53fd8bd4e5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-47c409-29890380-cqm6g" Oct 31 05:00:00.403274 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.403240 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-031745-29890380-9jpj5" Oct 31 05:00:00.407669 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.407639 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890380-tx4lq"] Oct 31 05:00:00.418464 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.418165 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890380-tx4lq" Oct 31 05:00:00.427854 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.427765 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890380-7g9d5"] Oct 31 05:00:00.451499 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.450922 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59sfq\" (UniqueName: \"kubernetes.io/projected/e1e9a647-0e9e-4528-a5a6-7022b244f1ba-kube-api-access-59sfq\") pod \"maas-api-key-cleanup-e2e-extmod-a-8973e2-29890380-ldsqw\" (UID: \"e1e9a647-0e9e-4528-a5a6-7022b244f1ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890380-ldsqw" Oct 31 05:00:00.461757 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.458918 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890380-ldsqw" Oct 31 05:00:00.486825 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.481486 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93bfb362-fcc2-4341-9881-1fd06b546590-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-227c93-29890380-7g9d5\" (UID: \"93bfb362-fcc2-4341-9881-1fd06b546590\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890380-7g9d5" Oct 31 05:00:00.486825 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.481632 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k6h9\" (UniqueName: \"kubernetes.io/projected/4fd6b126-7a7c-4825-a374-5dec6fb94d06-kube-api-access-5k6h9\") pod \"maas-api-key-cleanup-e2e-extmod-a-401761-29890380-zvx8f\" (UID: \"4fd6b126-7a7c-4825-a374-5dec6fb94d06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-401761-29890380-zvx8f" Oct 31 05:00:00.486825 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.481939 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmczm\" (UniqueName: \"kubernetes.io/projected/f59877aa-80e8-418d-9617-2f9bb9c8f3ef-kube-api-access-hmczm\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890380-v7gl7\" (UID: \"f59877aa-80e8-418d-9617-2f9bb9c8f3ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890380-v7gl7" Oct 31 05:00:00.486825 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.481971 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l2gk\" (UniqueName: \"kubernetes.io/projected/93bfb362-fcc2-4341-9881-1fd06b546590-kube-api-access-6l2gk\") pod \"maas-api-key-cleanup-e2e-extmod-b-227c93-29890380-7g9d5\" (UID: \"93bfb362-fcc2-4341-9881-1fd06b546590\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890380-7g9d5" Oct 31 05:00:00.486825 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.482021 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f59877aa-80e8-418d-9617-2f9bb9c8f3ef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890380-v7gl7\" (UID: \"f59877aa-80e8-418d-9617-2f9bb9c8f3ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890380-v7gl7" Oct 31 05:00:00.486825 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.482074 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fd6b126-7a7c-4825-a374-5dec6fb94d06-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-401761-29890380-zvx8f\" (UID: \"4fd6b126-7a7c-4825-a374-5dec6fb94d06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-401761-29890380-zvx8f" Oct 31 05:00:00.491934 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.491647 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f59877aa-80e8-418d-9617-2f9bb9c8f3ef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890380-v7gl7\" (UID: \"f59877aa-80e8-418d-9617-2f9bb9c8f3ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890380-v7gl7" Oct 31 05:00:00.531859 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.529018 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzstl\" (UniqueName: \"kubernetes.io/projected/982ad40e-a719-4f61-be85-ea665503c299-kube-api-access-xzstl\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890380-rmxrz\" (UID: \"982ad40e-a719-4f61-be85-ea665503c299\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890380-rmxrz" Oct 31 05:00:00.531859 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.530308 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb8ht\" (UniqueName: \"kubernetes.io/projected/0b6b5e89-5900-4db2-8552-e4f029de7573-kube-api-access-lb8ht\") pod \"maas-api-key-cleanup-e2e-extmod-b-a04e4e-29890380-jgchh\" (UID: \"0b6b5e89-5900-4db2-8552-e4f029de7573\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a04e4e-29890380-jgchh" Oct 31 05:00:00.531859 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.530391 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890380-b7jtg"] Oct 31 05:00:00.533680 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.533265 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdfcg\" (UniqueName: \"kubernetes.io/projected/703c28bc-a603-4001-9538-4a751cf9b7c0-kube-api-access-xdfcg\") pod \"maas-api-key-cleanup-e2e-extmod-b-567ca6-29890380-kcwdg\" (UID: \"703c28bc-a603-4001-9538-4a751cf9b7c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-567ca6-29890380-kcwdg" Oct 31 05:00:00.533680 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.533511 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a04e4e-29890380-jgchh" Oct 31 05:00:00.534280 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.534205 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fd6b126-7a7c-4825-a374-5dec6fb94d06-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-401761-29890380-zvx8f\" (UID: \"4fd6b126-7a7c-4825-a374-5dec6fb94d06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-401761-29890380-zvx8f" Oct 31 05:00:00.539206 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.539156 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smndt\" (UniqueName: \"kubernetes.io/projected/d13131fa-5137-4401-aef8-d53fd8bd4e5d-kube-api-access-smndt\") pod \"maas-api-key-cleanup-e2e-extmod-a-47c409-29890380-cqm6g\" (UID: \"d13131fa-5137-4401-aef8-d53fd8bd4e5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-47c409-29890380-cqm6g" Oct 31 05:00:00.544595 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.544521 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmczm\" (UniqueName: \"kubernetes.io/projected/f59877aa-80e8-418d-9617-2f9bb9c8f3ef-kube-api-access-hmczm\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890380-v7gl7\" (UID: \"f59877aa-80e8-418d-9617-2f9bb9c8f3ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890380-v7gl7" Oct 31 05:00:00.558212 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.556430 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k6h9\" (UniqueName: \"kubernetes.io/projected/4fd6b126-7a7c-4825-a374-5dec6fb94d06-kube-api-access-5k6h9\") pod \"maas-api-key-cleanup-e2e-extmod-a-401761-29890380-zvx8f\" (UID: \"4fd6b126-7a7c-4825-a374-5dec6fb94d06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-401761-29890380-zvx8f" Oct 31 05:00:00.558212 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.558026 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890380-rmxrz" Oct 31 05:00:00.564690 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.564659 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fb346b-29890380-l8fpr"] Oct 31 05:00:00.570492 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.570310 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890380-b7jtg" Oct 31 05:00:00.571748 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.571720 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890380-kr424"] Oct 31 05:00:00.574800 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.574133 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fb346b-29890380-l8fpr" Oct 31 05:00:00.588692 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.588346 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3d4f5cd-c755-4e4f-a86f-14874e597d08-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-031745-29890380-9jpj5\" (UID: \"d3d4f5cd-c755-4e4f-a86f-14874e597d08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-031745-29890380-9jpj5" Oct 31 05:00:00.588692 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.588442 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9790f69f-4de3-4837-a29f-17866661a3b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c4006d-29890380-tx4lq\" (UID: \"9790f69f-4de3-4837-a29f-17866661a3b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890380-tx4lq" Oct 31 05:00:00.588692 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.588500 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93bfb362-fcc2-4341-9881-1fd06b546590-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-227c93-29890380-7g9d5\" (UID: \"93bfb362-fcc2-4341-9881-1fd06b546590\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890380-7g9d5" Oct 31 05:00:00.588692 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.588547 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp8tj\" (UniqueName: \"kubernetes.io/projected/e27f22dd-e82f-493c-9745-6d901308b907-kube-api-access-mp8tj\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890380-b7jtg\" (UID: \"e27f22dd-e82f-493c-9745-6d901308b907\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890380-b7jtg" Oct 31 05:00:00.588692 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.588603 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwz2r\" (UniqueName: \"kubernetes.io/projected/829dc86b-4ce1-427e-86db-cbaab8f15198-kube-api-access-dwz2r\") pod \"maas-api-key-cleanup-e2e-inf-b-fb346b-29890380-l8fpr\" (UID: \"829dc86b-4ce1-427e-86db-cbaab8f15198\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fb346b-29890380-l8fpr" Oct 31 05:00:00.589674 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.589636 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93bfb362-fcc2-4341-9881-1fd06b546590-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-227c93-29890380-7g9d5\" (UID: \"93bfb362-fcc2-4341-9881-1fd06b546590\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890380-7g9d5" Oct 31 05:00:00.589838 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.589814 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpdc2\" (UniqueName: \"kubernetes.io/projected/9790f69f-4de3-4837-a29f-17866661a3b3-kube-api-access-gpdc2\") pod \"maas-api-key-cleanup-e2e-worker-w4-c4006d-29890380-tx4lq\" (UID: \"9790f69f-4de3-4837-a29f-17866661a3b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890380-tx4lq" Oct 31 05:00:00.590057 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.590033 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l2gk\" (UniqueName: \"kubernetes.io/projected/93bfb362-fcc2-4341-9881-1fd06b546590-kube-api-access-6l2gk\") pod \"maas-api-key-cleanup-e2e-extmod-b-227c93-29890380-7g9d5\" (UID: \"93bfb362-fcc2-4341-9881-1fd06b546590\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890380-7g9d5" Oct 31 05:00:00.590181 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.590095 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e27f22dd-e82f-493c-9745-6d901308b907-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890380-b7jtg\" (UID: \"e27f22dd-e82f-493c-9745-6d901308b907\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890380-b7jtg" Oct 31 05:00:00.590181 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.590132 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/829dc86b-4ce1-427e-86db-cbaab8f15198-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-fb346b-29890380-l8fpr\" (UID: \"829dc86b-4ce1-427e-86db-cbaab8f15198\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fb346b-29890380-l8fpr" Oct 31 05:00:00.590181 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.590169 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57r8h\" (UniqueName: \"kubernetes.io/projected/d3d4f5cd-c755-4e4f-a86f-14874e597d08-kube-api-access-57r8h\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-031745-29890380-9jpj5\" (UID: \"d3d4f5cd-c755-4e4f-a86f-14874e597d08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-031745-29890380-9jpj5" Oct 31 05:00:00.662383 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.657397 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l2gk\" (UniqueName: \"kubernetes.io/projected/93bfb362-fcc2-4341-9881-1fd06b546590-kube-api-access-6l2gk\") pod \"maas-api-key-cleanup-e2e-extmod-b-227c93-29890380-7g9d5\" (UID: \"93bfb362-fcc2-4341-9881-1fd06b546590\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890380-7g9d5" Oct 31 05:00:00.673409 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.671434 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890380-v7gl7"] Oct 31 05:00:00.673409 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.671476 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-401761-29890380-zvx8f"] Oct 31 05:00:00.673409 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.671497 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890380-tx4lq"] Oct 31 05:00:00.673409 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.671521 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-031745-29890380-9jpj5"] Oct 31 05:00:00.673409 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.671615 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890380-kr424" Oct 31 05:00:00.677371 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.676377 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890380-b7jtg"] Oct 31 05:00:00.684603 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.684562 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fb346b-29890380-l8fpr"] Oct 31 05:00:00.701106 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.701058 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3d4f5cd-c755-4e4f-a86f-14874e597d08-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-031745-29890380-9jpj5\" (UID: \"d3d4f5cd-c755-4e4f-a86f-14874e597d08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-031745-29890380-9jpj5" Oct 31 05:00:00.701367 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.701146 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9790f69f-4de3-4837-a29f-17866661a3b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c4006d-29890380-tx4lq\" (UID: \"9790f69f-4de3-4837-a29f-17866661a3b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890380-tx4lq" Oct 31 05:00:00.701367 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.701210 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slb2v\" (UniqueName: \"kubernetes.io/projected/17f1de0f-3edd-4d07-910a-f6e6f356203c-kube-api-access-slb2v\") pod \"maas-api-key-cleanup-e2e-extmod-a-afc519-29890380-kr424\" (UID: \"17f1de0f-3edd-4d07-910a-f6e6f356203c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890380-kr424" Oct 31 05:00:00.701367 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.701253 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp8tj\" (UniqueName: \"kubernetes.io/projected/e27f22dd-e82f-493c-9745-6d901308b907-kube-api-access-mp8tj\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890380-b7jtg\" (UID: \"e27f22dd-e82f-493c-9745-6d901308b907\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890380-b7jtg" Oct 31 05:00:00.701367 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.701302 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwz2r\" (UniqueName: \"kubernetes.io/projected/829dc86b-4ce1-427e-86db-cbaab8f15198-kube-api-access-dwz2r\") pod \"maas-api-key-cleanup-e2e-inf-b-fb346b-29890380-l8fpr\" (UID: \"829dc86b-4ce1-427e-86db-cbaab8f15198\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fb346b-29890380-l8fpr" Oct 31 05:00:00.701629 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.701380 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpdc2\" (UniqueName: \"kubernetes.io/projected/9790f69f-4de3-4837-a29f-17866661a3b3-kube-api-access-gpdc2\") pod \"maas-api-key-cleanup-e2e-worker-w4-c4006d-29890380-tx4lq\" (UID: \"9790f69f-4de3-4837-a29f-17866661a3b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890380-tx4lq" Oct 31 05:00:00.701629 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.701495 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17f1de0f-3edd-4d07-910a-f6e6f356203c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-afc519-29890380-kr424\" (UID: \"17f1de0f-3edd-4d07-910a-f6e6f356203c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890380-kr424" Oct 31 05:00:00.715406 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.706299 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3d4f5cd-c755-4e4f-a86f-14874e597d08-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-031745-29890380-9jpj5\" (UID: \"d3d4f5cd-c755-4e4f-a86f-14874e597d08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-031745-29890380-9jpj5" Oct 31 05:00:00.715406 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.706652 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9790f69f-4de3-4837-a29f-17866661a3b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c4006d-29890380-tx4lq\" (UID: \"9790f69f-4de3-4837-a29f-17866661a3b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890380-tx4lq" Oct 31 05:00:00.715406 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.706856 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-567ca6-29890380-kcwdg" Oct 31 05:00:00.715406 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.711504 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e27f22dd-e82f-493c-9745-6d901308b907-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890380-b7jtg\" (UID: \"e27f22dd-e82f-493c-9745-6d901308b907\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890380-b7jtg" Oct 31 05:00:00.715406 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.711922 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/829dc86b-4ce1-427e-86db-cbaab8f15198-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-fb346b-29890380-l8fpr\" (UID: \"829dc86b-4ce1-427e-86db-cbaab8f15198\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fb346b-29890380-l8fpr" Oct 31 05:00:00.715406 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.711972 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57r8h\" (UniqueName: \"kubernetes.io/projected/d3d4f5cd-c755-4e4f-a86f-14874e597d08-kube-api-access-57r8h\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-031745-29890380-9jpj5\" (UID: \"d3d4f5cd-c755-4e4f-a86f-14874e597d08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-031745-29890380-9jpj5" Oct 31 05:00:00.715406 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.713097 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e27f22dd-e82f-493c-9745-6d901308b907-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890380-b7jtg\" (UID: \"e27f22dd-e82f-493c-9745-6d901308b907\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890380-b7jtg" Oct 31 05:00:00.715406 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.713325 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/829dc86b-4ce1-427e-86db-cbaab8f15198-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-fb346b-29890380-l8fpr\" (UID: \"829dc86b-4ce1-427e-86db-cbaab8f15198\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fb346b-29890380-l8fpr" Oct 31 05:00:00.715406 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.713504 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24" Oct 31 05:00:00.722849 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.722431 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-401761-29890380-zvx8f" Oct 31 05:00:00.724273 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.723917 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890380-fw6dq"] Oct 31 05:00:00.742193 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.742159 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24" Oct 31 05:00:00.742475 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.742454 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890380-fh8kl"] Oct 31 05:00:00.749216 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.749102 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890380-fw6dq" Oct 31 05:00:00.753765 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.753734 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890380-v7gl7" Oct 31 05:00:00.758532 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.758229 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwz2r\" (UniqueName: \"kubernetes.io/projected/829dc86b-4ce1-427e-86db-cbaab8f15198-kube-api-access-dwz2r\") pod \"maas-api-key-cleanup-e2e-inf-b-fb346b-29890380-l8fpr\" (UID: \"829dc86b-4ce1-427e-86db-cbaab8f15198\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fb346b-29890380-l8fpr" Oct 31 05:00:00.761315 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.760037 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpdc2\" (UniqueName: \"kubernetes.io/projected/9790f69f-4de3-4837-a29f-17866661a3b3-kube-api-access-gpdc2\") pod \"maas-api-key-cleanup-e2e-worker-w4-c4006d-29890380-tx4lq\" (UID: \"9790f69f-4de3-4837-a29f-17866661a3b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890380-tx4lq" Oct 31 05:00:00.763966 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.763177 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp8tj\" (UniqueName: \"kubernetes.io/projected/e27f22dd-e82f-493c-9745-6d901308b907-kube-api-access-mp8tj\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890380-b7jtg\" (UID: \"e27f22dd-e82f-493c-9745-6d901308b907\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890380-b7jtg" Oct 31 05:00:00.765570 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.765511 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890380-tx4lq" Oct 31 05:00:00.765900 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.765825 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-108ad8-29890380-ng62l"] Oct 31 05:00:00.768609 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.768538 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890380-fh8kl" Oct 31 05:00:00.775675 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.774582 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890380-6qwg6"] Oct 31 05:00:00.777013 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.776264 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-108ad8-29890380-ng62l" Oct 31 05:00:00.789095 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.788909 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890380-6qwg6" Oct 31 05:00:00.792378 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.792179 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57r8h\" (UniqueName: \"kubernetes.io/projected/d3d4f5cd-c755-4e4f-a86f-14874e597d08-kube-api-access-57r8h\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-031745-29890380-9jpj5\" (UID: \"d3d4f5cd-c755-4e4f-a86f-14874e597d08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-031745-29890380-9jpj5" Oct 31 05:00:00.794301 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.794103 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890380-b7jtg" Oct 31 05:00:00.802442 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.802393 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-654741-29890380-85tdm"] Oct 31 05:00:00.817987 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.817952 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17f1de0f-3edd-4d07-910a-f6e6f356203c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-afc519-29890380-kr424\" (UID: \"17f1de0f-3edd-4d07-910a-f6e6f356203c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890380-kr424" Oct 31 05:00:00.818238 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.818216 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndq2s\" (UniqueName: \"kubernetes.io/projected/96e0a71b-2eee-4760-ade1-5cdd604c4c81-kube-api-access-ndq2s\") pod \"maas-api-key-cleanup-e2e-worker-w2-09a20a-29890380-fw6dq\" (UID: \"96e0a71b-2eee-4760-ade1-5cdd604c4c81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890380-fw6dq" Oct 31 05:00:00.818458 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.818439 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9b19f33-a012-48c9-b670-3b8eff773e49-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890380-6qwg6\" (UID: \"e9b19f33-a012-48c9-b670-3b8eff773e49\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890380-6qwg6" Oct 31 05:00:00.819413 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.819385 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvkkw\" (UniqueName: \"kubernetes.io/projected/d0dacaaf-e177-420b-a953-f8cdb4961a27-kube-api-access-gvkkw\") pod \"maas-api-key-cleanup-e2e-worker-w0-965f61-29890380-fh8kl\" (UID: \"d0dacaaf-e177-420b-a953-f8cdb4961a27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890380-fh8kl" Oct 31 05:00:00.820002 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.819970 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96e0a71b-2eee-4760-ade1-5cdd604c4c81-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-09a20a-29890380-fw6dq\" (UID: \"96e0a71b-2eee-4760-ade1-5cdd604c4c81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890380-fw6dq" Oct 31 05:00:00.820123 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.820018 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fb346b-29890380-l8fpr" Oct 31 05:00:00.820318 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.820065 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-47c409-29890380-cqm6g" Oct 31 05:00:00.821158 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.821113 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17f1de0f-3edd-4d07-910a-f6e6f356203c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-afc519-29890380-kr424\" (UID: \"17f1de0f-3edd-4d07-910a-f6e6f356203c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890380-kr424" Oct 31 05:00:00.823728 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.823703 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5259d89-590d-4fc7-b521-8f6feabbd2cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-108ad8-29890380-ng62l\" (UID: \"e5259d89-590d-4fc7-b521-8f6feabbd2cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-108ad8-29890380-ng62l" Oct 31 05:00:00.823970 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.823946 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0dacaaf-e177-420b-a953-f8cdb4961a27-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-965f61-29890380-fh8kl\" (UID: \"d0dacaaf-e177-420b-a953-f8cdb4961a27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890380-fh8kl" Oct 31 05:00:00.824881 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.824859 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slb2v\" (UniqueName: \"kubernetes.io/projected/17f1de0f-3edd-4d07-910a-f6e6f356203c-kube-api-access-slb2v\") pod \"maas-api-key-cleanup-e2e-extmod-a-afc519-29890380-kr424\" (UID: \"17f1de0f-3edd-4d07-910a-f6e6f356203c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890380-kr424" Oct 31 05:00:00.825072 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.825055 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8p2p\" (UniqueName: \"kubernetes.io/projected/e9b19f33-a012-48c9-b670-3b8eff773e49-kube-api-access-t8p2p\") pod \"maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890380-6qwg6\" (UID: \"e9b19f33-a012-48c9-b670-3b8eff773e49\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890380-6qwg6" Oct 31 05:00:00.825349 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.825311 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-672zl\" (UniqueName: \"kubernetes.io/projected/e5259d89-590d-4fc7-b521-8f6feabbd2cc-kube-api-access-672zl\") pod \"maas-api-key-cleanup-e2e-inf-a-108ad8-29890380-ng62l\" (UID: \"e5259d89-590d-4fc7-b521-8f6feabbd2cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-108ad8-29890380-ng62l" Oct 31 05:00:00.826209 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.826188 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890380-fh8kl"] Oct 31 05:00:00.826380 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.826365 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890380-kr424"] Oct 31 05:00:00.832875 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.827787 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-654741-29890380-85tdm" Oct 31 05:00:00.837041 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.836829 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890380-fw6dq"] Oct 31 05:00:00.846129 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.845418 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-47e9af-29890380-jk7b6"] Oct 31 05:00:00.857563 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.856945 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890380-kz5pd"] Oct 31 05:00:00.857563 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.857071 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-47e9af-29890380-jk7b6" Oct 31 05:00:00.860264 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.860192 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slb2v\" (UniqueName: \"kubernetes.io/projected/17f1de0f-3edd-4d07-910a-f6e6f356203c-kube-api-access-slb2v\") pod \"maas-api-key-cleanup-e2e-extmod-a-afc519-29890380-kr424\" (UID: \"17f1de0f-3edd-4d07-910a-f6e6f356203c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890380-kr424" Oct 31 05:00:00.870119 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.866281 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-654741-29890380-85tdm"] Oct 31 05:00:00.870119 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.866419 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890380-kz5pd" Oct 31 05:00:00.941745 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.936584 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0dacaaf-e177-420b-a953-f8cdb4961a27-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-965f61-29890380-fh8kl\" (UID: \"d0dacaaf-e177-420b-a953-f8cdb4961a27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890380-fh8kl" Oct 31 05:00:00.941745 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.936665 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmts9\" (UniqueName: \"kubernetes.io/projected/7c20d99d-7ce4-4bb8-a4d9-188cec401d75-kube-api-access-jmts9\") pod \"maas-api-key-cleanup-e2e-inf-b-47e9af-29890380-jk7b6\" (UID: \"7c20d99d-7ce4-4bb8-a4d9-188cec401d75\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-47e9af-29890380-jk7b6" Oct 31 05:00:00.941745 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.936704 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8p2p\" (UniqueName: \"kubernetes.io/projected/e9b19f33-a012-48c9-b670-3b8eff773e49-kube-api-access-t8p2p\") pod \"maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890380-6qwg6\" (UID: \"e9b19f33-a012-48c9-b670-3b8eff773e49\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890380-6qwg6" Oct 31 05:00:00.941745 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.936817 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e23416e6-954a-4ee6-8c4f-f6f8d7e1c506-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-654741-29890380-85tdm\" (UID: \"e23416e6-954a-4ee6-8c4f-f6f8d7e1c506\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-654741-29890380-85tdm" Oct 31 05:00:00.941745 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.936859 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-672zl\" (UniqueName: \"kubernetes.io/projected/e5259d89-590d-4fc7-b521-8f6feabbd2cc-kube-api-access-672zl\") pod \"maas-api-key-cleanup-e2e-inf-a-108ad8-29890380-ng62l\" (UID: \"e5259d89-590d-4fc7-b521-8f6feabbd2cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-108ad8-29890380-ng62l" Oct 31 05:00:00.941745 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.936900 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ab61246-5d2d-4a2a-8c42-fff22960018f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890380-kz5pd\" (UID: \"7ab61246-5d2d-4a2a-8c42-fff22960018f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890380-kz5pd" Oct 31 05:00:00.941745 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.936939 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rstjm\" (UniqueName: \"kubernetes.io/projected/e23416e6-954a-4ee6-8c4f-f6f8d7e1c506-kube-api-access-rstjm\") pod \"maas-api-key-cleanup-e2e-extmod-a-654741-29890380-85tdm\" (UID: \"e23416e6-954a-4ee6-8c4f-f6f8d7e1c506\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-654741-29890380-85tdm" Oct 31 05:00:00.941745 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.937056 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndq2s\" (UniqueName: \"kubernetes.io/projected/96e0a71b-2eee-4760-ade1-5cdd604c4c81-kube-api-access-ndq2s\") pod \"maas-api-key-cleanup-e2e-worker-w2-09a20a-29890380-fw6dq\" (UID: \"96e0a71b-2eee-4760-ade1-5cdd604c4c81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890380-fw6dq" Oct 31 05:00:00.941745 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.937129 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9b19f33-a012-48c9-b670-3b8eff773e49-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890380-6qwg6\" (UID: \"e9b19f33-a012-48c9-b670-3b8eff773e49\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890380-6qwg6" Oct 31 05:00:00.941745 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.937223 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c20d99d-7ce4-4bb8-a4d9-188cec401d75-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-47e9af-29890380-jk7b6\" (UID: \"7c20d99d-7ce4-4bb8-a4d9-188cec401d75\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-47e9af-29890380-jk7b6" Oct 31 05:00:00.941745 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.937250 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvkkw\" (UniqueName: \"kubernetes.io/projected/d0dacaaf-e177-420b-a953-f8cdb4961a27-kube-api-access-gvkkw\") pod \"maas-api-key-cleanup-e2e-worker-w0-965f61-29890380-fh8kl\" (UID: \"d0dacaaf-e177-420b-a953-f8cdb4961a27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890380-fh8kl" Oct 31 05:00:00.941745 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.937291 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96e0a71b-2eee-4760-ade1-5cdd604c4c81-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-09a20a-29890380-fw6dq\" (UID: \"96e0a71b-2eee-4760-ade1-5cdd604c4c81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890380-fw6dq" Oct 31 05:00:00.941745 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.937361 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5259d89-590d-4fc7-b521-8f6feabbd2cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-108ad8-29890380-ng62l\" (UID: \"e5259d89-590d-4fc7-b521-8f6feabbd2cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-108ad8-29890380-ng62l" Oct 31 05:00:00.941745 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.937386 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm2s5\" (UniqueName: \"kubernetes.io/projected/7ab61246-5d2d-4a2a-8c42-fff22960018f-kube-api-access-sm2s5\") pod \"maas-api-key-cleanup-29890380-kz5pd\" (UID: \"7ab61246-5d2d-4a2a-8c42-fff22960018f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890380-kz5pd" Oct 31 05:00:00.941745 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.938143 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0dacaaf-e177-420b-a953-f8cdb4961a27-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-965f61-29890380-fh8kl\" (UID: \"d0dacaaf-e177-420b-a953-f8cdb4961a27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890380-fh8kl" Oct 31 05:00:00.941745 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.941581 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890380-7g9d5" Oct 31 05:00:00.943125 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.941984 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96e0a71b-2eee-4760-ade1-5cdd604c4c81-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-09a20a-29890380-fw6dq\" (UID: \"96e0a71b-2eee-4760-ade1-5cdd604c4c81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890380-fw6dq" Oct 31 05:00:00.943125 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.943038 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5259d89-590d-4fc7-b521-8f6feabbd2cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-108ad8-29890380-ng62l\" (UID: \"e5259d89-590d-4fc7-b521-8f6feabbd2cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-108ad8-29890380-ng62l" Oct 31 05:00:00.946871 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.944729 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9b19f33-a012-48c9-b670-3b8eff773e49-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890380-6qwg6\" (UID: \"e9b19f33-a012-48c9-b670-3b8eff773e49\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890380-6qwg6" Oct 31 05:00:00.963859 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.963825 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8p2p\" (UniqueName: \"kubernetes.io/projected/e9b19f33-a012-48c9-b670-3b8eff773e49-kube-api-access-t8p2p\") pod \"maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890380-6qwg6\" (UID: \"e9b19f33-a012-48c9-b670-3b8eff773e49\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890380-6qwg6" Oct 31 05:00:00.969068 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.969036 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-672zl\" (UniqueName: \"kubernetes.io/projected/e5259d89-590d-4fc7-b521-8f6feabbd2cc-kube-api-access-672zl\") pod \"maas-api-key-cleanup-e2e-inf-a-108ad8-29890380-ng62l\" (UID: \"e5259d89-590d-4fc7-b521-8f6feabbd2cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-108ad8-29890380-ng62l" Oct 31 05:00:00.977236 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.977198 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890380-k8lq8"] Oct 31 05:00:00.983126 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:00.983095 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndq2s\" (UniqueName: \"kubernetes.io/projected/96e0a71b-2eee-4760-ade1-5cdd604c4c81-kube-api-access-ndq2s\") pod \"maas-api-key-cleanup-e2e-worker-w2-09a20a-29890380-fw6dq\" (UID: \"96e0a71b-2eee-4760-ade1-5cdd604c4c81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890380-fw6dq" Oct 31 05:00:01.003475 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.003447 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890380-k8lq8" Oct 31 05:00:01.005390 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.005367 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-108ad8-29890380-ng62l" Oct 31 05:00:01.018927 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.009182 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890380-6qwg6"] Oct 31 05:00:01.020768 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.020744 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-031745-29890380-9jpj5" Oct 31 05:00:01.030638 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.022944 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-670795-29890380-hzmsc"] Oct 31 05:00:01.030638 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.025452 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890380-6qwg6" Oct 31 05:00:01.035453 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.035418 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvkkw\" (UniqueName: \"kubernetes.io/projected/d0dacaaf-e177-420b-a953-f8cdb4961a27-kube-api-access-gvkkw\") pod \"maas-api-key-cleanup-e2e-worker-w0-965f61-29890380-fh8kl\" (UID: \"d0dacaaf-e177-420b-a953-f8cdb4961a27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890380-fh8kl" Oct 31 05:00:01.045927 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.045901 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmzqt\" (UniqueName: \"kubernetes.io/projected/deaadd33-dcef-43df-a66e-9fd41434de6f-kube-api-access-hmzqt\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890380-k8lq8\" (UID: \"deaadd33-dcef-43df-a66e-9fd41434de6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890380-k8lq8" Oct 31 05:00:01.046226 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.046201 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c20d99d-7ce4-4bb8-a4d9-188cec401d75-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-47e9af-29890380-jk7b6\" (UID: \"7c20d99d-7ce4-4bb8-a4d9-188cec401d75\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-47e9af-29890380-jk7b6" Oct 31 05:00:01.046514 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.046472 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/deaadd33-dcef-43df-a66e-9fd41434de6f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890380-k8lq8\" (UID: \"deaadd33-dcef-43df-a66e-9fd41434de6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890380-k8lq8" Oct 31 05:00:01.046745 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.046728 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm2s5\" (UniqueName: \"kubernetes.io/projected/7ab61246-5d2d-4a2a-8c42-fff22960018f-kube-api-access-sm2s5\") pod \"maas-api-key-cleanup-29890380-kz5pd\" (UID: \"7ab61246-5d2d-4a2a-8c42-fff22960018f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890380-kz5pd" Oct 31 05:00:01.046977 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.046961 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmts9\" (UniqueName: \"kubernetes.io/projected/7c20d99d-7ce4-4bb8-a4d9-188cec401d75-kube-api-access-jmts9\") pod \"maas-api-key-cleanup-e2e-inf-b-47e9af-29890380-jk7b6\" (UID: \"7c20d99d-7ce4-4bb8-a4d9-188cec401d75\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-47e9af-29890380-jk7b6" Oct 31 05:00:01.047199 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.047184 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e23416e6-954a-4ee6-8c4f-f6f8d7e1c506-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-654741-29890380-85tdm\" (UID: \"e23416e6-954a-4ee6-8c4f-f6f8d7e1c506\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-654741-29890380-85tdm" Oct 31 05:00:01.047324 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.047311 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ab61246-5d2d-4a2a-8c42-fff22960018f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890380-kz5pd\" (UID: \"7ab61246-5d2d-4a2a-8c42-fff22960018f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890380-kz5pd" Oct 31 05:00:01.047462 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.047446 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rstjm\" (UniqueName: \"kubernetes.io/projected/e23416e6-954a-4ee6-8c4f-f6f8d7e1c506-kube-api-access-rstjm\") pod \"maas-api-key-cleanup-e2e-extmod-a-654741-29890380-85tdm\" (UID: \"e23416e6-954a-4ee6-8c4f-f6f8d7e1c506\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-654741-29890380-85tdm" Oct 31 05:00:01.065743 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.049727 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c20d99d-7ce4-4bb8-a4d9-188cec401d75-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-47e9af-29890380-jk7b6\" (UID: \"7c20d99d-7ce4-4bb8-a4d9-188cec401d75\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-47e9af-29890380-jk7b6" Oct 31 05:00:01.065743 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.050787 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e23416e6-954a-4ee6-8c4f-f6f8d7e1c506-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-654741-29890380-85tdm\" (UID: \"e23416e6-954a-4ee6-8c4f-f6f8d7e1c506\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-654741-29890380-85tdm" Oct 31 05:00:01.065743 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.060777 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-47e9af-29890380-jk7b6"] Oct 31 05:00:01.065743 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.060896 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-670795-29890380-hzmsc" Oct 31 05:00:01.072939 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.066466 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ab61246-5d2d-4a2a-8c42-fff22960018f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890380-kz5pd\" (UID: \"7ab61246-5d2d-4a2a-8c42-fff22960018f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890380-kz5pd" Oct 31 05:00:01.072939 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.067167 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890380-kz5pd"] Oct 31 05:00:01.077890 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.074217 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1daf30-29890380-pbtq2"] Oct 31 05:00:01.083571 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.083413 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1daf30-29890380-pbtq2" Oct 31 05:00:01.085220 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.085161 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rstjm\" (UniqueName: \"kubernetes.io/projected/e23416e6-954a-4ee6-8c4f-f6f8d7e1c506-kube-api-access-rstjm\") pod \"maas-api-key-cleanup-e2e-extmod-a-654741-29890380-85tdm\" (UID: \"e23416e6-954a-4ee6-8c4f-f6f8d7e1c506\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-654741-29890380-85tdm" Oct 31 05:00:01.085953 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.085927 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-108ad8-29890380-ng62l"] Oct 31 05:00:01.105352 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.104010 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890380-tkvr7"] Oct 31 05:00:01.118673 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.111774 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890380-tkvr7" Oct 31 05:00:01.124162 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.124129 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890380-k8lq8"] Oct 31 05:00:01.126168 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.126131 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm2s5\" (UniqueName: \"kubernetes.io/projected/7ab61246-5d2d-4a2a-8c42-fff22960018f-kube-api-access-sm2s5\") pod \"maas-api-key-cleanup-29890380-kz5pd\" (UID: \"7ab61246-5d2d-4a2a-8c42-fff22960018f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890380-kz5pd" Oct 31 05:00:01.127562 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.127522 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmts9\" (UniqueName: \"kubernetes.io/projected/7c20d99d-7ce4-4bb8-a4d9-188cec401d75-kube-api-access-jmts9\") pod \"maas-api-key-cleanup-e2e-inf-b-47e9af-29890380-jk7b6\" (UID: \"7c20d99d-7ce4-4bb8-a4d9-188cec401d75\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-47e9af-29890380-jk7b6" Oct 31 05:00:01.135592 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.135500 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-670795-29890380-hzmsc"] Oct 31 05:00:01.136535 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.136125 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890380-kz5pd" Oct 31 05:00:01.146179 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.146142 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1daf30-29890380-pbtq2"] Oct 31 05:00:01.149323 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.149293 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwql4\" (UniqueName: \"kubernetes.io/projected/e478c8f1-9e68-4f6f-b919-28ba397e66c4-kube-api-access-wwql4\") pod \"maas-api-key-cleanup-e2e-worker-w3-4c2387-29890380-tkvr7\" (UID: \"e478c8f1-9e68-4f6f-b919-28ba397e66c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890380-tkvr7" Oct 31 05:00:01.151835 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.149548 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9515dc7-4ea3-4a1b-aad8-1e78c7d0e13c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-670795-29890380-hzmsc\" (UID: \"c9515dc7-4ea3-4a1b-aad8-1e78c7d0e13c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-670795-29890380-hzmsc" Oct 31 05:00:01.151835 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.149724 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnsgn\" (UniqueName: \"kubernetes.io/projected/cd0723e3-dcf6-4907-9ce1-99ca9c77b144-kube-api-access-wnsgn\") pod \"maas-api-key-cleanup-e2e-inf-a-1daf30-29890380-pbtq2\" (UID: \"cd0723e3-dcf6-4907-9ce1-99ca9c77b144\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1daf30-29890380-pbtq2" Oct 31 05:00:01.151835 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.149785 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e478c8f1-9e68-4f6f-b919-28ba397e66c4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-4c2387-29890380-tkvr7\" (UID: \"e478c8f1-9e68-4f6f-b919-28ba397e66c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890380-tkvr7" Oct 31 05:00:01.151835 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.150281 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6wbf\" (UniqueName: \"kubernetes.io/projected/c9515dc7-4ea3-4a1b-aad8-1e78c7d0e13c-kube-api-access-f6wbf\") pod \"maas-api-key-cleanup-e2e-inf-a-670795-29890380-hzmsc\" (UID: \"c9515dc7-4ea3-4a1b-aad8-1e78c7d0e13c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-670795-29890380-hzmsc" Oct 31 05:00:01.151835 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.150417 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmzqt\" (UniqueName: \"kubernetes.io/projected/deaadd33-dcef-43df-a66e-9fd41434de6f-kube-api-access-hmzqt\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890380-k8lq8\" (UID: \"deaadd33-dcef-43df-a66e-9fd41434de6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890380-k8lq8" Oct 31 05:00:01.151835 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.150550 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd0723e3-dcf6-4907-9ce1-99ca9c77b144-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-1daf30-29890380-pbtq2\" (UID: \"cd0723e3-dcf6-4907-9ce1-99ca9c77b144\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1daf30-29890380-pbtq2" Oct 31 05:00:01.151835 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.150593 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/deaadd33-dcef-43df-a66e-9fd41434de6f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890380-k8lq8\" (UID: \"deaadd33-dcef-43df-a66e-9fd41434de6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890380-k8lq8" Oct 31 05:00:01.152309 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.151919 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/deaadd33-dcef-43df-a66e-9fd41434de6f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890380-k8lq8\" (UID: \"deaadd33-dcef-43df-a66e-9fd41434de6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890380-k8lq8" Oct 31 05:00:01.157187 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.157166 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890380-kr424" Oct 31 05:00:01.159614 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.159584 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890380-tkvr7"] Oct 31 05:00:01.248406 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.248293 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmzqt\" (UniqueName: \"kubernetes.io/projected/deaadd33-dcef-43df-a66e-9fd41434de6f-kube-api-access-hmzqt\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890380-k8lq8\" (UID: \"deaadd33-dcef-43df-a66e-9fd41434de6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890380-k8lq8" Oct 31 05:00:01.250403 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.250379 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890380-fw6dq" Oct 31 05:00:01.261417 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.253639 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd0723e3-dcf6-4907-9ce1-99ca9c77b144-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-1daf30-29890380-pbtq2\" (UID: \"cd0723e3-dcf6-4907-9ce1-99ca9c77b144\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1daf30-29890380-pbtq2" Oct 31 05:00:01.261417 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.253872 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwql4\" (UniqueName: \"kubernetes.io/projected/e478c8f1-9e68-4f6f-b919-28ba397e66c4-kube-api-access-wwql4\") pod \"maas-api-key-cleanup-e2e-worker-w3-4c2387-29890380-tkvr7\" (UID: \"e478c8f1-9e68-4f6f-b919-28ba397e66c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890380-tkvr7" Oct 31 05:00:01.261417 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.253999 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9515dc7-4ea3-4a1b-aad8-1e78c7d0e13c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-670795-29890380-hzmsc\" (UID: \"c9515dc7-4ea3-4a1b-aad8-1e78c7d0e13c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-670795-29890380-hzmsc" Oct 31 05:00:01.261417 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.254118 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnsgn\" (UniqueName: \"kubernetes.io/projected/cd0723e3-dcf6-4907-9ce1-99ca9c77b144-kube-api-access-wnsgn\") pod \"maas-api-key-cleanup-e2e-inf-a-1daf30-29890380-pbtq2\" (UID: \"cd0723e3-dcf6-4907-9ce1-99ca9c77b144\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1daf30-29890380-pbtq2" Oct 31 05:00:01.261417 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.254164 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e478c8f1-9e68-4f6f-b919-28ba397e66c4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-4c2387-29890380-tkvr7\" (UID: \"e478c8f1-9e68-4f6f-b919-28ba397e66c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890380-tkvr7" Oct 31 05:00:01.261417 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.254190 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6wbf\" (UniqueName: \"kubernetes.io/projected/c9515dc7-4ea3-4a1b-aad8-1e78c7d0e13c-kube-api-access-f6wbf\") pod \"maas-api-key-cleanup-e2e-inf-a-670795-29890380-hzmsc\" (UID: \"c9515dc7-4ea3-4a1b-aad8-1e78c7d0e13c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-670795-29890380-hzmsc" Oct 31 05:00:01.261417 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.255533 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd0723e3-dcf6-4907-9ce1-99ca9c77b144-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-1daf30-29890380-pbtq2\" (UID: \"cd0723e3-dcf6-4907-9ce1-99ca9c77b144\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1daf30-29890380-pbtq2" Oct 31 05:00:01.261417 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.256738 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e478c8f1-9e68-4f6f-b919-28ba397e66c4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-4c2387-29890380-tkvr7\" (UID: \"e478c8f1-9e68-4f6f-b919-28ba397e66c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890380-tkvr7" Oct 31 05:00:01.261417 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.257645 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9515dc7-4ea3-4a1b-aad8-1e78c7d0e13c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-670795-29890380-hzmsc\" (UID: \"c9515dc7-4ea3-4a1b-aad8-1e78c7d0e13c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-670795-29890380-hzmsc" Oct 31 05:00:01.285676 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.285616 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6wbf\" (UniqueName: \"kubernetes.io/projected/c9515dc7-4ea3-4a1b-aad8-1e78c7d0e13c-kube-api-access-f6wbf\") pod \"maas-api-key-cleanup-e2e-inf-a-670795-29890380-hzmsc\" (UID: \"c9515dc7-4ea3-4a1b-aad8-1e78c7d0e13c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-670795-29890380-hzmsc" Oct 31 05:00:01.287157 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.287106 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwql4\" (UniqueName: \"kubernetes.io/projected/e478c8f1-9e68-4f6f-b919-28ba397e66c4-kube-api-access-wwql4\") pod \"maas-api-key-cleanup-e2e-worker-w3-4c2387-29890380-tkvr7\" (UID: \"e478c8f1-9e68-4f6f-b919-28ba397e66c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890380-tkvr7" Oct 31 05:00:01.289633 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.289603 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnsgn\" (UniqueName: \"kubernetes.io/projected/cd0723e3-dcf6-4907-9ce1-99ca9c77b144-kube-api-access-wnsgn\") pod \"maas-api-key-cleanup-e2e-inf-a-1daf30-29890380-pbtq2\" (UID: \"cd0723e3-dcf6-4907-9ce1-99ca9c77b144\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1daf30-29890380-pbtq2" Oct 31 05:00:01.291307 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.290944 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890380-fh8kl" Oct 31 05:00:01.306244 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:00:01.306189 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b6b5e89_5900_4db2_8552_e4f029de7573.slice/crio-d5b6a486c3a1ef24a63ad1db8e4b5fccf56225cae51e403206c48ed14fba167f WatchSource:0}: Error finding container d5b6a486c3a1ef24a63ad1db8e4b5fccf56225cae51e403206c48ed14fba167f: Status 404 returned error can't find the container with id d5b6a486c3a1ef24a63ad1db8e4b5fccf56225cae51e403206c48ed14fba167f Oct 31 05:00:01.369687 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.369588 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-654741-29890380-85tdm" Oct 31 05:00:01.385199 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.385117 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-47e9af-29890380-jk7b6" Oct 31 05:00:01.460741 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.460695 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890380-k8lq8" Oct 31 05:00:01.515971 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.515716 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1daf30-29890380-pbtq2" Oct 31 05:00:01.518709 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.516392 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-670795-29890380-hzmsc" Oct 31 05:00:01.552790 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.552727 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a04e4e-29890380-jgchh"] Oct 31 05:00:01.557771 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.557737 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890380-tkvr7" Oct 31 05:00:01.708565 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.707213 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890380-ldsqw"] Oct 31 05:00:01.727861 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:00:01.727821 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf59877aa_80e8_418d_9617_2f9bb9c8f3ef.slice/crio-ecba0638ca26c057aa747aa2c1d54d6a8e5bd63e738e761739bf05313b6de8f2 WatchSource:0}: Error finding container ecba0638ca26c057aa747aa2c1d54d6a8e5bd63e738e761739bf05313b6de8f2: Status 404 returned error can't find the container with id ecba0638ca26c057aa747aa2c1d54d6a8e5bd63e738e761739bf05313b6de8f2 Oct 31 05:00:01.739859 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:00:01.739832 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod703c28bc_a603_4001_9538_4a751cf9b7c0.slice/crio-92244f9601f37d573523d7edf1d8d2b206678697a7b97e2e8f2b221670d94682 WatchSource:0}: Error finding container 92244f9601f37d573523d7edf1d8d2b206678697a7b97e2e8f2b221670d94682: Status 404 returned error can't find the container with id 92244f9601f37d573523d7edf1d8d2b206678697a7b97e2e8f2b221670d94682 Oct 31 05:00:01.817267 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.817212 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-567ca6-29890380-kcwdg"] Oct 31 05:00:01.829116 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.829079 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890380-v7gl7"] Oct 31 05:00:01.839049 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.839007 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890380-rmxrz"] Oct 31 05:00:01.850132 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.850038 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-401761-29890380-zvx8f"] Oct 31 05:00:01.868434 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.864212 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890380-tx4lq"] Oct 31 05:00:01.924036 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:00:01.923998 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9790f69f_4de3_4837_a29f_17866661a3b3.slice/crio-60096ab33c9ce7c34ce1bce12d5ef1de0f50fb592f4b41cf235b56b656d1b767 WatchSource:0}: Error finding container 60096ab33c9ce7c34ce1bce12d5ef1de0f50fb592f4b41cf235b56b656d1b767: Status 404 returned error can't find the container with id 60096ab33c9ce7c34ce1bce12d5ef1de0f50fb592f4b41cf235b56b656d1b767 Oct 31 05:00:01.930909 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.930432 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-47c409-29890380-cqm6g"] Oct 31 05:00:01.973637 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:01.973307 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fb346b-29890380-l8fpr"] Oct 31 05:00:02.030969 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:02.030937 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-031745-29890380-9jpj5"] Oct 31 05:00:02.116923 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:00:02.116829 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3d4f5cd_c755_4e4f_a86f_14874e597d08.slice/crio-1c09ce316a7b3487db94c307631aef0c20800570ced220a14b4ed22ea44b69f6 WatchSource:0}: Error finding container 1c09ce316a7b3487db94c307631aef0c20800570ced220a14b4ed22ea44b69f6: Status 404 returned error can't find the container with id 1c09ce316a7b3487db94c307631aef0c20800570ced220a14b4ed22ea44b69f6 Oct 31 05:00:02.280208 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:02.280104 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890380-ldsqw" event={"ID":"e1e9a647-0e9e-4528-a5a6-7022b244f1ba","Type":"ContainerStarted","Data":"7f3897ec7afe082ea962d40153a5b916006f10fe62ed43c7d25943636d590e10"} Oct 31 05:00:02.321294 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:02.321268 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890380-6qwg6"] Oct 31 05:00:02.333406 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:02.328878 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890380-b7jtg"] Oct 31 05:00:02.335317 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:02.333985 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fb346b-29890380-l8fpr" event={"ID":"829dc86b-4ce1-427e-86db-cbaab8f15198","Type":"ContainerStarted","Data":"476f6c49f70b0d41fcdb86295fe365600cf8ec1c26702f8d0db601a340ebccfb"} Oct 31 05:00:02.338612 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:02.338580 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-401761-29890380-zvx8f" event={"ID":"4fd6b126-7a7c-4825-a374-5dec6fb94d06","Type":"ContainerStarted","Data":"719e130ba10827ce5bc82b3e0007523bdd820203947486455e82b0422b6355e6"} Oct 31 05:00:02.345695 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:00:02.345657 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9b19f33_a012_48c9_b670_3b8eff773e49.slice/crio-cc615af6760a6d4e9279c8e41ca11559540a1580e467045e52f5d0f5bb0d8ad7 WatchSource:0}: Error finding container cc615af6760a6d4e9279c8e41ca11559540a1580e467045e52f5d0f5bb0d8ad7: Status 404 returned error can't find the container with id cc615af6760a6d4e9279c8e41ca11559540a1580e467045e52f5d0f5bb0d8ad7 Oct 31 05:00:02.347112 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:02.345992 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890380-tx4lq" event={"ID":"9790f69f-4de3-4837-a29f-17866661a3b3","Type":"ContainerStarted","Data":"60096ab33c9ce7c34ce1bce12d5ef1de0f50fb592f4b41cf235b56b656d1b767"} Oct 31 05:00:02.353842 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:02.353648 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890380-v7gl7" event={"ID":"f59877aa-80e8-418d-9617-2f9bb9c8f3ef","Type":"ContainerStarted","Data":"ecba0638ca26c057aa747aa2c1d54d6a8e5bd63e738e761739bf05313b6de8f2"} Oct 31 05:00:02.359688 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:02.359484 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890380-rmxrz" event={"ID":"982ad40e-a719-4f61-be85-ea665503c299","Type":"ContainerStarted","Data":"0c7efa996e8c8c81ac76087696f0b213b0635ad19dc11a6640b92c932ed33d45"} Oct 31 05:00:02.399350 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:02.399216 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-567ca6-29890380-kcwdg" event={"ID":"703c28bc-a603-4001-9538-4a751cf9b7c0","Type":"ContainerStarted","Data":"92244f9601f37d573523d7edf1d8d2b206678697a7b97e2e8f2b221670d94682"} Oct 31 05:00:02.400266 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:02.400090 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-108ad8-29890380-ng62l"] Oct 31 05:00:02.402395 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:02.402275 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-47c409-29890380-cqm6g" event={"ID":"d13131fa-5137-4401-aef8-d53fd8bd4e5d","Type":"ContainerStarted","Data":"b026302e47440a8d9c43a310b6cea86e3de97c1e7227274e78f7b08d34faca31"} Oct 31 05:00:02.411104 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:02.411037 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a04e4e-29890380-jgchh" event={"ID":"0b6b5e89-5900-4db2-8552-e4f029de7573","Type":"ContainerStarted","Data":"d5b6a486c3a1ef24a63ad1db8e4b5fccf56225cae51e403206c48ed14fba167f"} Oct 31 05:00:02.457675 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:02.457644 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890380-7g9d5"] Oct 31 05:00:02.579571 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:00:02.579511 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5259d89_590d_4fc7_b521_8f6feabbd2cc.slice/crio-42e716007154a4aca25d539c13737d651d4d7685dabba7f685eb8c575cefadae WatchSource:0}: Error finding container 42e716007154a4aca25d539c13737d651d4d7685dabba7f685eb8c575cefadae: Status 404 returned error can't find the container with id 42e716007154a4aca25d539c13737d651d4d7685dabba7f685eb8c575cefadae Oct 31 05:00:02.593124 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:02.592957 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890380-kr424"] Oct 31 05:00:02.608308 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:02.608264 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890380-fw6dq"] Oct 31 05:00:02.611362 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:00:02.611275 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ab61246_5d2d_4a2a_8c42_fff22960018f.slice/crio-73d6c1cc82c1a129fd9c34184d5357a1698ad3a659995ae9972d4409f1f3e716 WatchSource:0}: Error finding container 73d6c1cc82c1a129fd9c34184d5357a1698ad3a659995ae9972d4409f1f3e716: Status 404 returned error can't find the container with id 73d6c1cc82c1a129fd9c34184d5357a1698ad3a659995ae9972d4409f1f3e716 Oct 31 05:00:02.625990 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:02.617864 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890380-kz5pd"] Oct 31 05:00:02.764800 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:02.764741 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890380-tkvr7"] Oct 31 05:00:02.773634 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:02.773548 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890380-fh8kl"] Oct 31 05:00:02.780891 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:02.780765 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-47e9af-29890380-jk7b6"] Oct 31 05:00:02.816683 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:02.815570 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890380-k8lq8"] Oct 31 05:00:02.843864 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:02.843833 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-670795-29890380-hzmsc"] Oct 31 05:00:02.850716 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:02.850681 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1daf30-29890380-pbtq2"] Oct 31 05:00:02.857278 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:02.857233 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-654741-29890380-85tdm"] Oct 31 05:00:03.007486 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:00:03.007441 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd0723e3_dcf6_4907_9ce1_99ca9c77b144.slice/crio-f03e20c3227a816c02d73feb9f768b67f8a51acbe0e03fcbdb7a0af9b5e3a30e WatchSource:0}: Error finding container f03e20c3227a816c02d73feb9f768b67f8a51acbe0e03fcbdb7a0af9b5e3a30e: Status 404 returned error can't find the container with id f03e20c3227a816c02d73feb9f768b67f8a51acbe0e03fcbdb7a0af9b5e3a30e Oct 31 05:00:03.011477 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:00:03.011435 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9515dc7_4ea3_4a1b_aad8_1e78c7d0e13c.slice/crio-bcd8e69a5a92881ae3b7f82c3c7a38b043bbbd0c755ffb7288fb4cbc0709c5e4 WatchSource:0}: Error finding container bcd8e69a5a92881ae3b7f82c3c7a38b043bbbd0c755ffb7288fb4cbc0709c5e4: Status 404 returned error can't find the container with id bcd8e69a5a92881ae3b7f82c3c7a38b043bbbd0c755ffb7288fb4cbc0709c5e4 Oct 31 05:00:03.442029 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:03.441931 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890380-6qwg6" event={"ID":"e9b19f33-a012-48c9-b670-3b8eff773e49","Type":"ContainerStarted","Data":"cc615af6760a6d4e9279c8e41ca11559540a1580e467045e52f5d0f5bb0d8ad7"} Oct 31 05:00:03.444644 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:03.444605 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-670795-29890380-hzmsc" event={"ID":"c9515dc7-4ea3-4a1b-aad8-1e78c7d0e13c","Type":"ContainerStarted","Data":"bcd8e69a5a92881ae3b7f82c3c7a38b043bbbd0c755ffb7288fb4cbc0709c5e4"} Oct 31 05:00:03.450693 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:03.450568 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890380-ldsqw" event={"ID":"e1e9a647-0e9e-4528-a5a6-7022b244f1ba","Type":"ContainerStarted","Data":"15457cd712676393b895b973612547c284d609f298b560788f8deefbe5631d6f"} Oct 31 05:00:03.452140 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:03.452110 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-654741-29890380-85tdm" event={"ID":"e23416e6-954a-4ee6-8c4f-f6f8d7e1c506","Type":"ContainerStarted","Data":"87da266a80fd1b2317935e7974c99f808cc9d0dfe6ad4664bb0a24a1dd5be3ba"} Oct 31 05:00:03.453881 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:03.453852 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890380-fh8kl" event={"ID":"d0dacaaf-e177-420b-a953-f8cdb4961a27","Type":"ContainerStarted","Data":"fc4008d62599e7e298351217caca81c5ff4c300975d4e52f6dad33a797d15083"} Oct 31 05:00:03.456000 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:03.455968 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fb346b-29890380-l8fpr" event={"ID":"829dc86b-4ce1-427e-86db-cbaab8f15198","Type":"ContainerStarted","Data":"68ca4fd6cb2d5c2738c21e0645161b22c96a5f78fe4b4a596a135aec9c97af7b"} Oct 31 05:00:03.457561 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:03.457535 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890380-kz5pd" event={"ID":"7ab61246-5d2d-4a2a-8c42-fff22960018f","Type":"ContainerStarted","Data":"73d6c1cc82c1a129fd9c34184d5357a1698ad3a659995ae9972d4409f1f3e716"} Oct 31 05:00:03.459312 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:03.459288 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-031745-29890380-9jpj5" event={"ID":"d3d4f5cd-c755-4e4f-a86f-14874e597d08","Type":"ContainerStarted","Data":"555da1a1e8b971b314b2d3fd7fc38f5096d719630ff9bcee75b3c03b2fff24ae"} Oct 31 05:00:03.459436 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:03.459390 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-031745-29890380-9jpj5" event={"ID":"d3d4f5cd-c755-4e4f-a86f-14874e597d08","Type":"ContainerStarted","Data":"1c09ce316a7b3487db94c307631aef0c20800570ced220a14b4ed22ea44b69f6"} Oct 31 05:00:03.460805 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:03.460780 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890380-fw6dq" event={"ID":"96e0a71b-2eee-4760-ade1-5cdd604c4c81","Type":"ContainerStarted","Data":"75bb012bf2acb18f8e1b88177d0f3ddb647176cb7ea75d3f8f01a6a42ba6109f"} Oct 31 05:00:03.462640 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:03.462609 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-401761-29890380-zvx8f" event={"ID":"4fd6b126-7a7c-4825-a374-5dec6fb94d06","Type":"ContainerStarted","Data":"7964a83895049c03f2b385afb6320b913f15ccb2dc893145d9de6f1d28499636"} Oct 31 05:00:03.464141 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:03.464115 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890380-kr424" event={"ID":"17f1de0f-3edd-4d07-910a-f6e6f356203c","Type":"ContainerStarted","Data":"05a7a22244007878c09d7cc14463eec41b70e7257943325c6181b7ce1ed984a0"} Oct 31 05:00:03.465959 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:03.465932 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-108ad8-29890380-ng62l" event={"ID":"e5259d89-590d-4fc7-b521-8f6feabbd2cc","Type":"ContainerStarted","Data":"42e716007154a4aca25d539c13737d651d4d7685dabba7f685eb8c575cefadae"} Oct 31 05:00:03.467591 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:03.467564 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890380-b7jtg" event={"ID":"e27f22dd-e82f-493c-9745-6d901308b907","Type":"ContainerStarted","Data":"6827956b8d908213a0999139105c6775bb9effbd6ab4cbfc232ed53dd6e58bc8"} Oct 31 05:00:03.478417 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:03.475146 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890380-tx4lq" event={"ID":"9790f69f-4de3-4837-a29f-17866661a3b3","Type":"ContainerStarted","Data":"51a7208722da3395ac4d14c37e69b6d845c509de6bc00558303c3fc8f679e1cf"} Oct 31 05:00:03.488552 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:03.488513 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890380-v7gl7" event={"ID":"f59877aa-80e8-418d-9617-2f9bb9c8f3ef","Type":"ContainerStarted","Data":"6d97a1a4faaa99cfb2890bcc32b0562f5126a0f94ebc1c7fc11da3692df248ed"} Oct 31 05:00:03.500509 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:03.500403 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890380-rmxrz" event={"ID":"982ad40e-a719-4f61-be85-ea665503c299","Type":"ContainerStarted","Data":"f7d432a1307fb2b83c4d8a6fd4001137df73dfd63e893388becb147010bc647c"} Oct 31 05:00:03.503757 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:03.503722 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890380-tkvr7" event={"ID":"e478c8f1-9e68-4f6f-b919-28ba397e66c4","Type":"ContainerStarted","Data":"c6a09d6300c609e78e460cc801be813ea805b5850be2b40de7d1efad203d48d4"} Oct 31 05:00:03.507076 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:03.507018 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890380-ldsqw" podStartSLOduration=3.506998048 podStartE2EDuration="3.506998048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:00:03.489706813 +0000 UTC m=+3358.246044854" watchObservedRunningTime="2026-10-31 05:00:03.506998048 +0000 UTC m=+3358.263336090" Oct 31 05:00:03.509907 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:03.509864 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-567ca6-29890380-kcwdg" event={"ID":"703c28bc-a603-4001-9538-4a751cf9b7c0","Type":"ContainerStarted","Data":"d807a180acbc5a50860443a4eb8afc9f9d045c59ba10cc96437fa81439113566"} Oct 31 05:00:03.512839 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:03.512809 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890380-k8lq8" event={"ID":"deaadd33-dcef-43df-a66e-9fd41434de6f","Type":"ContainerStarted","Data":"3ada75c9613f21a6928f73ea2d1d809bdb3681a50ffacdb9fb3db38ff3241efd"} Oct 31 05:00:03.514510 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:03.514483 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-47c409-29890380-cqm6g" event={"ID":"d13131fa-5137-4401-aef8-d53fd8bd4e5d","Type":"ContainerStarted","Data":"3654e96b7edaf039e141cbdea019474e2cdaeada5203c50c9914498da2f0c4d8"} Oct 31 05:00:03.516705 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:03.516679 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1daf30-29890380-pbtq2" event={"ID":"cd0723e3-dcf6-4907-9ce1-99ca9c77b144","Type":"ContainerStarted","Data":"f03e20c3227a816c02d73feb9f768b67f8a51acbe0e03fcbdb7a0af9b5e3a30e"} Oct 31 05:00:03.518856 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:03.518835 5019 generic.go:356] "Generic (PLEG): container finished" podID="0b6b5e89-5900-4db2-8552-e4f029de7573" containerID="a69459f298d08b0f90dc302b73eb69c44f016b9a39e48801db4cb4524301d774" exitCode=60 Oct 31 05:00:03.519021 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:03.519008 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a04e4e-29890380-jgchh" event={"ID":"0b6b5e89-5900-4db2-8552-e4f029de7573","Type":"ContainerDied","Data":"a69459f298d08b0f90dc302b73eb69c44f016b9a39e48801db4cb4524301d774"} Oct 31 05:00:03.519557 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:03.519534 5019 scope.go:122] "RemoveContainer" containerID="a69459f298d08b0f90dc302b73eb69c44f016b9a39e48801db4cb4524301d774" Oct 31 05:00:03.523165 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:03.523135 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-47e9af-29890380-jk7b6" event={"ID":"7c20d99d-7ce4-4bb8-a4d9-188cec401d75","Type":"ContainerStarted","Data":"d0abf60006040dd3a67b940994c050cccddc563c0bc480d49b5f6f274453e493"} Oct 31 05:00:03.529143 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:03.529116 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890380-7g9d5" event={"ID":"93bfb362-fcc2-4341-9881-1fd06b546590","Type":"ContainerStarted","Data":"31d155ef347108aee11b43ea61a5eacd06883df1056f14b570a03111b694d2ca"} Oct 31 05:00:03.535217 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:03.535163 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-401761-29890380-zvx8f" podStartSLOduration=3.5351486789999997 podStartE2EDuration="3.535148679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:00:03.530989801 +0000 UTC m=+3358.287327841" watchObservedRunningTime="2026-10-31 05:00:03.535148679 +0000 UTC m=+3358.291486712" Oct 31 05:00:03.583754 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:03.583661 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890380-tx4lq" podStartSLOduration=3.583637558 podStartE2EDuration="3.583637558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:00:03.577915306 +0000 UTC m=+3358.334253377" watchObservedRunningTime="2026-10-31 05:00:03.583637558 +0000 UTC m=+3358.339975600" Oct 31 05:00:03.625747 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:03.625676 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890380-v7gl7" podStartSLOduration=3.625654316 podStartE2EDuration="3.625654316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:00:03.617768654 +0000 UTC m=+3358.374106695" watchObservedRunningTime="2026-10-31 05:00:03.625654316 +0000 UTC m=+3358.381992357" Oct 31 05:00:03.658222 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:03.658165 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fb346b-29890380-l8fpr" podStartSLOduration=3.658147497 podStartE2EDuration="3.658147497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:00:03.656233787 +0000 UTC m=+3358.412571827" watchObservedRunningTime="2026-10-31 05:00:03.658147497 +0000 UTC m=+3358.414485535" Oct 31 05:00:03.718057 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:03.717986 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-031745-29890380-9jpj5" podStartSLOduration=3.717970607 podStartE2EDuration="3.717970607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:00:03.714796528 +0000 UTC m=+3358.471134567" watchObservedRunningTime="2026-10-31 05:00:03.717970607 +0000 UTC m=+3358.474308646" Oct 31 05:00:03.758088 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:03.758031 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-567ca6-29890380-kcwdg" podStartSLOduration=3.758016958 podStartE2EDuration="3.758016958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:00:03.754007304 +0000 UTC m=+3358.510345343" watchObservedRunningTime="2026-10-31 05:00:03.758016958 +0000 UTC m=+3358.514355024" Oct 31 05:00:03.800477 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:03.800388 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890380-rmxrz" podStartSLOduration=3.800370525 podStartE2EDuration="3.800370525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:00:03.798589415 +0000 UTC m=+3358.554927458" watchObservedRunningTime="2026-10-31 05:00:03.800370525 +0000 UTC m=+3358.556708566" Oct 31 05:00:03.878642 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:03.878580 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-47c409-29890380-cqm6g" podStartSLOduration=3.87856136 podStartE2EDuration="3.87856136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:00:03.877181747 +0000 UTC m=+3358.633519786" watchObservedRunningTime="2026-10-31 05:00:03.87856136 +0000 UTC m=+3358.634899404" Oct 31 05:00:04.544071 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:04.544031 5019 generic.go:356] "Generic (PLEG): container finished" podID="e1e9a647-0e9e-4528-a5a6-7022b244f1ba" containerID="15457cd712676393b895b973612547c284d609f298b560788f8deefbe5631d6f" exitCode=60 Oct 31 05:00:04.544544 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:04.544134 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890380-ldsqw" event={"ID":"e1e9a647-0e9e-4528-a5a6-7022b244f1ba","Type":"ContainerDied","Data":"15457cd712676393b895b973612547c284d609f298b560788f8deefbe5631d6f"} Oct 31 05:00:04.544608 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:04.544590 5019 scope.go:122] "RemoveContainer" containerID="15457cd712676393b895b973612547c284d609f298b560788f8deefbe5631d6f" Oct 31 05:00:04.546216 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:04.546185 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890380-fh8kl" event={"ID":"d0dacaaf-e177-420b-a953-f8cdb4961a27","Type":"ContainerStarted","Data":"34c97cc998485973569636bdc63787eb972b4e9523cf7c16298a5f60c9f6282d"} Oct 31 05:00:04.548934 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:04.548901 5019 generic.go:356] "Generic (PLEG): container finished" podID="829dc86b-4ce1-427e-86db-cbaab8f15198" containerID="68ca4fd6cb2d5c2738c21e0645161b22c96a5f78fe4b4a596a135aec9c97af7b" exitCode=60 Oct 31 05:00:04.549050 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:04.548981 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fb346b-29890380-l8fpr" event={"ID":"829dc86b-4ce1-427e-86db-cbaab8f15198","Type":"ContainerDied","Data":"68ca4fd6cb2d5c2738c21e0645161b22c96a5f78fe4b4a596a135aec9c97af7b"} Oct 31 05:00:04.549963 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:04.549943 5019 scope.go:122] "RemoveContainer" containerID="68ca4fd6cb2d5c2738c21e0645161b22c96a5f78fe4b4a596a135aec9c97af7b" Oct 31 05:00:04.551045 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:04.551021 5019 generic.go:356] "Generic (PLEG): container finished" podID="d3d4f5cd-c755-4e4f-a86f-14874e597d08" containerID="555da1a1e8b971b314b2d3fd7fc38f5096d719630ff9bcee75b3c03b2fff24ae" exitCode=60 Oct 31 05:00:04.551131 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:04.551099 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-031745-29890380-9jpj5" event={"ID":"d3d4f5cd-c755-4e4f-a86f-14874e597d08","Type":"ContainerDied","Data":"555da1a1e8b971b314b2d3fd7fc38f5096d719630ff9bcee75b3c03b2fff24ae"} Oct 31 05:00:04.551399 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:04.551370 5019 scope.go:122] "RemoveContainer" containerID="555da1a1e8b971b314b2d3fd7fc38f5096d719630ff9bcee75b3c03b2fff24ae" Oct 31 05:00:04.553157 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:04.553134 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890380-fw6dq" event={"ID":"96e0a71b-2eee-4760-ade1-5cdd604c4c81","Type":"ContainerStarted","Data":"74ad998b84a4ff8fbf4f04304ced47eb6b91784d4c7a65a6a4e03d075982d011"} Oct 31 05:00:04.554924 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:04.554900 5019 generic.go:356] "Generic (PLEG): container finished" podID="4fd6b126-7a7c-4825-a374-5dec6fb94d06" containerID="7964a83895049c03f2b385afb6320b913f15ccb2dc893145d9de6f1d28499636" exitCode=60 Oct 31 05:00:04.555012 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:04.554974 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-401761-29890380-zvx8f" event={"ID":"4fd6b126-7a7c-4825-a374-5dec6fb94d06","Type":"ContainerDied","Data":"7964a83895049c03f2b385afb6320b913f15ccb2dc893145d9de6f1d28499636"} Oct 31 05:00:04.555225 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:04.555204 5019 scope.go:122] "RemoveContainer" containerID="7964a83895049c03f2b385afb6320b913f15ccb2dc893145d9de6f1d28499636" Oct 31 05:00:04.557116 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:04.557090 5019 generic.go:356] "Generic (PLEG): container finished" podID="e5259d89-590d-4fc7-b521-8f6feabbd2cc" containerID="1d73f7724438a8711f7371e99ea6fb64e1a089a7f8d55d699a6ac063205c330a" exitCode=60 Oct 31 05:00:04.557219 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:04.557163 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-108ad8-29890380-ng62l" event={"ID":"e5259d89-590d-4fc7-b521-8f6feabbd2cc","Type":"ContainerDied","Data":"1d73f7724438a8711f7371e99ea6fb64e1a089a7f8d55d699a6ac063205c330a"} Oct 31 05:00:04.557443 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:04.557424 5019 scope.go:122] "RemoveContainer" containerID="1d73f7724438a8711f7371e99ea6fb64e1a089a7f8d55d699a6ac063205c330a" Oct 31 05:00:04.559188 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:04.559164 5019 generic.go:356] "Generic (PLEG): container finished" podID="e27f22dd-e82f-493c-9745-6d901308b907" containerID="ee1d28ad3481b569977e8791b92e8961c3c0d6254d11acea704028dad80ba52f" exitCode=60 Oct 31 05:00:04.559291 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:04.559240 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890380-b7jtg" event={"ID":"e27f22dd-e82f-493c-9745-6d901308b907","Type":"ContainerDied","Data":"ee1d28ad3481b569977e8791b92e8961c3c0d6254d11acea704028dad80ba52f"} Oct 31 05:00:04.559490 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:04.559472 5019 scope.go:122] "RemoveContainer" containerID="ee1d28ad3481b569977e8791b92e8961c3c0d6254d11acea704028dad80ba52f" Oct 31 05:00:04.561922 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:04.561795 5019 generic.go:356] "Generic (PLEG): container finished" podID="9790f69f-4de3-4837-a29f-17866661a3b3" containerID="51a7208722da3395ac4d14c37e69b6d845c509de6bc00558303c3fc8f679e1cf" exitCode=60 Oct 31 05:00:04.561922 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:04.561867 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890380-tx4lq" event={"ID":"9790f69f-4de3-4837-a29f-17866661a3b3","Type":"ContainerDied","Data":"51a7208722da3395ac4d14c37e69b6d845c509de6bc00558303c3fc8f679e1cf"} Oct 31 05:00:04.562166 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:04.562147 5019 scope.go:122] "RemoveContainer" containerID="51a7208722da3395ac4d14c37e69b6d845c509de6bc00558303c3fc8f679e1cf" Oct 31 05:00:04.566782 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:04.566756 5019 generic.go:356] "Generic (PLEG): container finished" podID="f59877aa-80e8-418d-9617-2f9bb9c8f3ef" containerID="6d97a1a4faaa99cfb2890bcc32b0562f5126a0f94ebc1c7fc11da3692df248ed" exitCode=60 Oct 31 05:00:04.566900 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:04.566883 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890380-v7gl7" event={"ID":"f59877aa-80e8-418d-9617-2f9bb9c8f3ef","Type":"ContainerDied","Data":"6d97a1a4faaa99cfb2890bcc32b0562f5126a0f94ebc1c7fc11da3692df248ed"} Oct 31 05:00:04.567195 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:04.567177 5019 scope.go:122] "RemoveContainer" containerID="6d97a1a4faaa99cfb2890bcc32b0562f5126a0f94ebc1c7fc11da3692df248ed" Oct 31 05:00:04.573408 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:04.569625 5019 generic.go:356] "Generic (PLEG): container finished" podID="982ad40e-a719-4f61-be85-ea665503c299" containerID="f7d432a1307fb2b83c4d8a6fd4001137df73dfd63e893388becb147010bc647c" exitCode=60 Oct 31 05:00:04.588695 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:04.587755 5019 scope.go:122] "RemoveContainer" containerID="f7d432a1307fb2b83c4d8a6fd4001137df73dfd63e893388becb147010bc647c" Oct 31 05:00:04.588853 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:04.588699 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890380-rmxrz" event={"ID":"982ad40e-a719-4f61-be85-ea665503c299","Type":"ContainerDied","Data":"f7d432a1307fb2b83c4d8a6fd4001137df73dfd63e893388becb147010bc647c"} Oct 31 05:00:04.614690 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:04.614652 5019 generic.go:356] "Generic (PLEG): container finished" podID="703c28bc-a603-4001-9538-4a751cf9b7c0" containerID="d807a180acbc5a50860443a4eb8afc9f9d045c59ba10cc96437fa81439113566" exitCode=60 Oct 31 05:00:04.614955 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:04.614888 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-567ca6-29890380-kcwdg" event={"ID":"703c28bc-a603-4001-9538-4a751cf9b7c0","Type":"ContainerDied","Data":"d807a180acbc5a50860443a4eb8afc9f9d045c59ba10cc96437fa81439113566"} Oct 31 05:00:04.616576 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:04.616019 5019 scope.go:122] "RemoveContainer" containerID="d807a180acbc5a50860443a4eb8afc9f9d045c59ba10cc96437fa81439113566" Oct 31 05:00:04.619705 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:04.619668 5019 generic.go:356] "Generic (PLEG): container finished" podID="d13131fa-5137-4401-aef8-d53fd8bd4e5d" containerID="3654e96b7edaf039e141cbdea019474e2cdaeada5203c50c9914498da2f0c4d8" exitCode=60 Oct 31 05:00:04.619869 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:04.619849 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-47c409-29890380-cqm6g" event={"ID":"d13131fa-5137-4401-aef8-d53fd8bd4e5d","Type":"ContainerDied","Data":"3654e96b7edaf039e141cbdea019474e2cdaeada5203c50c9914498da2f0c4d8"} Oct 31 05:00:04.620463 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:04.620437 5019 scope.go:122] "RemoveContainer" containerID="3654e96b7edaf039e141cbdea019474e2cdaeada5203c50c9914498da2f0c4d8" Oct 31 05:00:04.627504 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:04.627470 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890380-7g9d5" event={"ID":"93bfb362-fcc2-4341-9881-1fd06b546590","Type":"ContainerStarted","Data":"c77a304d3332e55e91f7b0105d5efbc776deeaeeef86901a4e5e810c1d5aea9c"} Oct 31 05:00:04.630809 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:04.630777 5019 generic.go:356] "Generic (PLEG): container finished" podID="e9b19f33-a012-48c9-b670-3b8eff773e49" containerID="25e8929523907b64d88120a2d9e64a67f48ecf50bb65dc1e3cbf9cc5f94987ab" exitCode=60 Oct 31 05:00:04.630925 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:04.630846 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890380-6qwg6" event={"ID":"e9b19f33-a012-48c9-b670-3b8eff773e49","Type":"ContainerDied","Data":"25e8929523907b64d88120a2d9e64a67f48ecf50bb65dc1e3cbf9cc5f94987ab"} Oct 31 05:00:04.632643 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:04.631413 5019 scope.go:122] "RemoveContainer" containerID="25e8929523907b64d88120a2d9e64a67f48ecf50bb65dc1e3cbf9cc5f94987ab" Oct 31 05:00:04.798295 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:04.798229 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890380-fh8kl" podStartSLOduration=4.798210353 podStartE2EDuration="4.798210353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:00:04.794317196 +0000 UTC m=+3359.550655238" watchObservedRunningTime="2026-10-31 05:00:04.798210353 +0000 UTC m=+3359.554548397" Oct 31 05:00:04.930791 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:04.930731 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890380-fw6dq" podStartSLOduration=4.9307104200000005 podStartE2EDuration="4.93071042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:00:04.921670173 +0000 UTC m=+3359.678008213" watchObservedRunningTime="2026-10-31 05:00:04.93071042 +0000 UTC m=+3359.687048456" Oct 31 05:00:05.165452 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:05.165327 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890380-7g9d5" podStartSLOduration=5.165310836 podStartE2EDuration="5.165310836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:00:05.163975473 +0000 UTC m=+3359.920313514" watchObservedRunningTime="2026-10-31 05:00:05.165310836 +0000 UTC m=+3359.921648876" Oct 31 05:00:05.645288 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:05.645253 5019 generic.go:356] "Generic (PLEG): container finished" podID="e478c8f1-9e68-4f6f-b919-28ba397e66c4" containerID="de1e3cb77f9b581a3d051a71aa2cd4b4a163b8e71ef12a5092db722abb6add93" exitCode=60 Oct 31 05:00:05.646262 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:05.645327 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890380-tkvr7" event={"ID":"e478c8f1-9e68-4f6f-b919-28ba397e66c4","Type":"ContainerDied","Data":"de1e3cb77f9b581a3d051a71aa2cd4b4a163b8e71ef12a5092db722abb6add93"} Oct 31 05:00:05.646262 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:05.645813 5019 scope.go:122] "RemoveContainer" containerID="de1e3cb77f9b581a3d051a71aa2cd4b4a163b8e71ef12a5092db722abb6add93" Oct 31 05:00:05.648778 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:05.648151 5019 generic.go:356] "Generic (PLEG): container finished" podID="deaadd33-dcef-43df-a66e-9fd41434de6f" containerID="cd9ca19be0efd1f15845ee46b04967e99ef188ee00fb65fabd59f523ebad1ce3" exitCode=60 Oct 31 05:00:05.648778 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:05.648195 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890380-k8lq8" event={"ID":"deaadd33-dcef-43df-a66e-9fd41434de6f","Type":"ContainerDied","Data":"cd9ca19be0efd1f15845ee46b04967e99ef188ee00fb65fabd59f523ebad1ce3"} Oct 31 05:00:05.648778 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:05.648607 5019 scope.go:122] "RemoveContainer" containerID="cd9ca19be0efd1f15845ee46b04967e99ef188ee00fb65fabd59f523ebad1ce3" Oct 31 05:00:05.650535 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:05.650503 5019 generic.go:356] "Generic (PLEG): container finished" podID="cd0723e3-dcf6-4907-9ce1-99ca9c77b144" containerID="3438843aeecdeb2ce68af9c9b78ce7bdc1d33a2d6ea87a3da15beeeb103ff737" exitCode=60 Oct 31 05:00:05.650641 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:05.650594 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1daf30-29890380-pbtq2" event={"ID":"cd0723e3-dcf6-4907-9ce1-99ca9c77b144","Type":"ContainerDied","Data":"3438843aeecdeb2ce68af9c9b78ce7bdc1d33a2d6ea87a3da15beeeb103ff737"} Oct 31 05:00:05.650980 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:05.650892 5019 scope.go:122] "RemoveContainer" containerID="3438843aeecdeb2ce68af9c9b78ce7bdc1d33a2d6ea87a3da15beeeb103ff737" Oct 31 05:00:05.653446 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:05.653422 5019 generic.go:356] "Generic (PLEG): container finished" podID="0b6b5e89-5900-4db2-8552-e4f029de7573" containerID="4b4377c5db90806371b2b551279f4bbc84522ccc6de2d0d0967bd6da2e216700" exitCode=60 Oct 31 05:00:05.653547 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:05.653477 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a04e4e-29890380-jgchh" event={"ID":"0b6b5e89-5900-4db2-8552-e4f029de7573","Type":"ContainerDied","Data":"4b4377c5db90806371b2b551279f4bbc84522ccc6de2d0d0967bd6da2e216700"} Oct 31 05:00:05.653643 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:05.653583 5019 scope.go:122] "RemoveContainer" containerID="a69459f298d08b0f90dc302b73eb69c44f016b9a39e48801db4cb4524301d774" Oct 31 05:00:05.653929 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:05.653911 5019 scope.go:122] "RemoveContainer" containerID="4b4377c5db90806371b2b551279f4bbc84522ccc6de2d0d0967bd6da2e216700" Oct 31 05:00:05.654235 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:05.654206 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-a04e4e-29890380-jgchh_odh-ai-gateway-infra(0b6b5e89-5900-4db2-8552-e4f029de7573)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a04e4e-29890380-jgchh" podUID="0b6b5e89-5900-4db2-8552-e4f029de7573" Oct 31 05:00:05.655813 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:05.655787 5019 generic.go:356] "Generic (PLEG): container finished" podID="7c20d99d-7ce4-4bb8-a4d9-188cec401d75" containerID="3f6b4961e4d3cab25aa21421f9e4585b116ff0fea23bc674e76b94b8b1732404" exitCode=60 Oct 31 05:00:05.655928 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:05.655898 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-47e9af-29890380-jk7b6" event={"ID":"7c20d99d-7ce4-4bb8-a4d9-188cec401d75","Type":"ContainerDied","Data":"3f6b4961e4d3cab25aa21421f9e4585b116ff0fea23bc674e76b94b8b1732404"} Oct 31 05:00:05.656541 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:05.656522 5019 scope.go:122] "RemoveContainer" containerID="3f6b4961e4d3cab25aa21421f9e4585b116ff0fea23bc674e76b94b8b1732404" Oct 31 05:00:05.658166 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:05.658142 5019 generic.go:356] "Generic (PLEG): container finished" podID="93bfb362-fcc2-4341-9881-1fd06b546590" containerID="c77a304d3332e55e91f7b0105d5efbc776deeaeeef86901a4e5e810c1d5aea9c" exitCode=60 Oct 31 05:00:05.658253 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:05.658221 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890380-7g9d5" event={"ID":"93bfb362-fcc2-4341-9881-1fd06b546590","Type":"ContainerDied","Data":"c77a304d3332e55e91f7b0105d5efbc776deeaeeef86901a4e5e810c1d5aea9c"} Oct 31 05:00:05.658521 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:05.658502 5019 scope.go:122] "RemoveContainer" containerID="c77a304d3332e55e91f7b0105d5efbc776deeaeeef86901a4e5e810c1d5aea9c" Oct 31 05:00:05.661752 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:05.661731 5019 generic.go:356] "Generic (PLEG): container finished" podID="c9515dc7-4ea3-4a1b-aad8-1e78c7d0e13c" containerID="8662a6aa5422c851875c67cf8a73f258959627296961118f2f4f32d6729f9728" exitCode=60 Oct 31 05:00:05.661854 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:05.661808 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-670795-29890380-hzmsc" event={"ID":"c9515dc7-4ea3-4a1b-aad8-1e78c7d0e13c","Type":"ContainerDied","Data":"8662a6aa5422c851875c67cf8a73f258959627296961118f2f4f32d6729f9728"} Oct 31 05:00:05.662029 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:05.662011 5019 scope.go:122] "RemoveContainer" containerID="8662a6aa5422c851875c67cf8a73f258959627296961118f2f4f32d6729f9728" Oct 31 05:00:05.665892 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:05.665861 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890380-ldsqw" event={"ID":"e1e9a647-0e9e-4528-a5a6-7022b244f1ba","Type":"ContainerStarted","Data":"b249f4ff0351d3704e4d3f0a0ab4e5428603ad8f889f3e0546a3773ac369dc9d"} Oct 31 05:00:05.668169 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:05.668146 5019 generic.go:356] "Generic (PLEG): container finished" podID="e23416e6-954a-4ee6-8c4f-f6f8d7e1c506" containerID="71083e8b3f82b9a55409a59770bbf6a1a66ac24369a218a5b9441598b8e97bc0" exitCode=60 Oct 31 05:00:05.668259 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:05.668219 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-654741-29890380-85tdm" event={"ID":"e23416e6-954a-4ee6-8c4f-f6f8d7e1c506","Type":"ContainerDied","Data":"71083e8b3f82b9a55409a59770bbf6a1a66ac24369a218a5b9441598b8e97bc0"} Oct 31 05:00:05.668610 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:05.668588 5019 scope.go:122] "RemoveContainer" containerID="71083e8b3f82b9a55409a59770bbf6a1a66ac24369a218a5b9441598b8e97bc0" Oct 31 05:00:05.675669 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:05.674713 5019 generic.go:356] "Generic (PLEG): container finished" podID="d0dacaaf-e177-420b-a953-f8cdb4961a27" containerID="34c97cc998485973569636bdc63787eb972b4e9523cf7c16298a5f60c9f6282d" exitCode=60 Oct 31 05:00:05.675669 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:05.674862 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890380-fh8kl" event={"ID":"d0dacaaf-e177-420b-a953-f8cdb4961a27","Type":"ContainerDied","Data":"34c97cc998485973569636bdc63787eb972b4e9523cf7c16298a5f60c9f6282d"} Oct 31 05:00:05.675669 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:05.675288 5019 scope.go:122] "RemoveContainer" containerID="34c97cc998485973569636bdc63787eb972b4e9523cf7c16298a5f60c9f6282d" Oct 31 05:00:05.682953 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:05.682921 5019 generic.go:356] "Generic (PLEG): container finished" podID="7ab61246-5d2d-4a2a-8c42-fff22960018f" containerID="09f295a65e46b70f6868801fb8302bfcf6209d92ea5aca45c4d6fbc665a053ed" exitCode=60 Oct 31 05:00:05.683949 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:05.683874 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890380-kz5pd" event={"ID":"7ab61246-5d2d-4a2a-8c42-fff22960018f","Type":"ContainerDied","Data":"09f295a65e46b70f6868801fb8302bfcf6209d92ea5aca45c4d6fbc665a053ed"} Oct 31 05:00:05.684180 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:05.684130 5019 scope.go:122] "RemoveContainer" containerID="09f295a65e46b70f6868801fb8302bfcf6209d92ea5aca45c4d6fbc665a053ed" Oct 31 05:00:05.692214 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:05.692169 5019 generic.go:356] "Generic (PLEG): container finished" podID="96e0a71b-2eee-4760-ade1-5cdd604c4c81" containerID="74ad998b84a4ff8fbf4f04304ced47eb6b91784d4c7a65a6a4e03d075982d011" exitCode=60 Oct 31 05:00:05.692581 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:05.692550 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890380-fw6dq" event={"ID":"96e0a71b-2eee-4760-ade1-5cdd604c4c81","Type":"ContainerDied","Data":"74ad998b84a4ff8fbf4f04304ced47eb6b91784d4c7a65a6a4e03d075982d011"} Oct 31 05:00:05.693429 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:05.693406 5019 scope.go:122] "RemoveContainer" containerID="74ad998b84a4ff8fbf4f04304ced47eb6b91784d4c7a65a6a4e03d075982d011" Oct 31 05:00:05.720396 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:05.720303 5019 generic.go:356] "Generic (PLEG): container finished" podID="17f1de0f-3edd-4d07-910a-f6e6f356203c" containerID="10128b3d93fc46591eb41e380a3f73d494d2b30cd4494a54511f4f6b313af267" exitCode=60 Oct 31 05:00:05.720593 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:05.720403 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890380-kr424" event={"ID":"17f1de0f-3edd-4d07-910a-f6e6f356203c","Type":"ContainerDied","Data":"10128b3d93fc46591eb41e380a3f73d494d2b30cd4494a54511f4f6b313af267"} Oct 31 05:00:05.720829 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:05.720813 5019 scope.go:122] "RemoveContainer" containerID="10128b3d93fc46591eb41e380a3f73d494d2b30cd4494a54511f4f6b313af267" Oct 31 05:00:06.737750 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:06.737704 5019 generic.go:356] "Generic (PLEG): container finished" podID="e9b19f33-a012-48c9-b670-3b8eff773e49" containerID="fa1c952e213fc2db874f7214d12dd9757aed0c6fc1e318427ea08af5651005c4" exitCode=60 Oct 31 05:00:06.738215 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:06.737784 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890380-6qwg6" event={"ID":"e9b19f33-a012-48c9-b670-3b8eff773e49","Type":"ContainerDied","Data":"fa1c952e213fc2db874f7214d12dd9757aed0c6fc1e318427ea08af5651005c4"} Oct 31 05:00:06.738302 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:06.738280 5019 scope.go:122] "RemoveContainer" containerID="fa1c952e213fc2db874f7214d12dd9757aed0c6fc1e318427ea08af5651005c4" Oct 31 05:00:06.738815 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:06.738787 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890380-6qwg6_odh-ai-gateway-infra(e9b19f33-a012-48c9-b670-3b8eff773e49)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890380-6qwg6" podUID="e9b19f33-a012-48c9-b670-3b8eff773e49" Oct 31 05:00:06.740552 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:06.740525 5019 generic.go:356] "Generic (PLEG): container finished" podID="e1e9a647-0e9e-4528-a5a6-7022b244f1ba" containerID="b249f4ff0351d3704e4d3f0a0ab4e5428603ad8f889f3e0546a3773ac369dc9d" exitCode=60 Oct 31 05:00:06.740649 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:06.740594 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890380-ldsqw" event={"ID":"e1e9a647-0e9e-4528-a5a6-7022b244f1ba","Type":"ContainerDied","Data":"b249f4ff0351d3704e4d3f0a0ab4e5428603ad8f889f3e0546a3773ac369dc9d"} Oct 31 05:00:06.741041 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:06.741018 5019 scope.go:122] "RemoveContainer" containerID="b249f4ff0351d3704e4d3f0a0ab4e5428603ad8f889f3e0546a3773ac369dc9d" Oct 31 05:00:06.741533 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:06.741506 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-8973e2-29890380-ldsqw_odh-ai-gateway-infra(e1e9a647-0e9e-4528-a5a6-7022b244f1ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890380-ldsqw" podUID="e1e9a647-0e9e-4528-a5a6-7022b244f1ba" Oct 31 05:00:06.742915 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:06.742895 5019 generic.go:356] "Generic (PLEG): container finished" podID="829dc86b-4ce1-427e-86db-cbaab8f15198" containerID="307dc9b0647618d9e86a3118ba70766b75e9974d27930eae830018e534d93edd" exitCode=60 Oct 31 05:00:06.743010 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:06.742975 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fb346b-29890380-l8fpr" event={"ID":"829dc86b-4ce1-427e-86db-cbaab8f15198","Type":"ContainerDied","Data":"307dc9b0647618d9e86a3118ba70766b75e9974d27930eae830018e534d93edd"} Oct 31 05:00:06.743322 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:06.743298 5019 scope.go:122] "RemoveContainer" containerID="307dc9b0647618d9e86a3118ba70766b75e9974d27930eae830018e534d93edd" Oct 31 05:00:06.743771 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:06.743751 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-fb346b-29890380-l8fpr_odh-ai-gateway-infra(829dc86b-4ce1-427e-86db-cbaab8f15198)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fb346b-29890380-l8fpr" podUID="829dc86b-4ce1-427e-86db-cbaab8f15198" Oct 31 05:00:06.745153 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:06.745135 5019 generic.go:356] "Generic (PLEG): container finished" podID="d3d4f5cd-c755-4e4f-a86f-14874e597d08" containerID="4b2241ffbbf74bceaaa581e0c48f1143ec969f2a7343bd8f08f814ed3ea2b57f" exitCode=60 Oct 31 05:00:06.745237 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:06.745217 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-031745-29890380-9jpj5" event={"ID":"d3d4f5cd-c755-4e4f-a86f-14874e597d08","Type":"ContainerDied","Data":"4b2241ffbbf74bceaaa581e0c48f1143ec969f2a7343bd8f08f814ed3ea2b57f"} Oct 31 05:00:06.745579 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:06.745559 5019 scope.go:122] "RemoveContainer" containerID="4b2241ffbbf74bceaaa581e0c48f1143ec969f2a7343bd8f08f814ed3ea2b57f" Oct 31 05:00:06.745971 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:06.745949 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-031745-29890380-9jpj5_odh-ai-gateway-infra(d3d4f5cd-c755-4e4f-a86f-14874e597d08)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-031745-29890380-9jpj5" podUID="d3d4f5cd-c755-4e4f-a86f-14874e597d08" Oct 31 05:00:06.747582 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:06.747561 5019 generic.go:356] "Generic (PLEG): container finished" podID="4fd6b126-7a7c-4825-a374-5dec6fb94d06" containerID="0ae3171d94a4e9fc4512711515cb6300d725282ddfb9097757734314266dfcb2" exitCode=60 Oct 31 05:00:06.747678 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:06.747635 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-401761-29890380-zvx8f" event={"ID":"4fd6b126-7a7c-4825-a374-5dec6fb94d06","Type":"ContainerDied","Data":"0ae3171d94a4e9fc4512711515cb6300d725282ddfb9097757734314266dfcb2"} Oct 31 05:00:06.748049 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:06.747932 5019 scope.go:122] "RemoveContainer" containerID="0ae3171d94a4e9fc4512711515cb6300d725282ddfb9097757734314266dfcb2" Oct 31 05:00:06.748324 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:06.748301 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-401761-29890380-zvx8f_odh-ai-gateway-infra(4fd6b126-7a7c-4825-a374-5dec6fb94d06)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-401761-29890380-zvx8f" podUID="4fd6b126-7a7c-4825-a374-5dec6fb94d06" Oct 31 05:00:06.749991 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:06.749977 5019 generic.go:356] "Generic (PLEG): container finished" podID="e5259d89-590d-4fc7-b521-8f6feabbd2cc" containerID="ed02bf60ecb10a21a7f0e07c1abb134e22852b8304d083726ba9c1258db41d2e" exitCode=60 Oct 31 05:00:06.750065 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:06.750042 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-108ad8-29890380-ng62l" event={"ID":"e5259d89-590d-4fc7-b521-8f6feabbd2cc","Type":"ContainerDied","Data":"ed02bf60ecb10a21a7f0e07c1abb134e22852b8304d083726ba9c1258db41d2e"} Oct 31 05:00:06.750454 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:06.750435 5019 scope.go:122] "RemoveContainer" containerID="ed02bf60ecb10a21a7f0e07c1abb134e22852b8304d083726ba9c1258db41d2e" Oct 31 05:00:06.750863 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:06.750812 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-108ad8-29890380-ng62l_odh-ai-gateway-infra(e5259d89-590d-4fc7-b521-8f6feabbd2cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-108ad8-29890380-ng62l" podUID="e5259d89-590d-4fc7-b521-8f6feabbd2cc" Oct 31 05:00:06.752698 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:06.752674 5019 generic.go:356] "Generic (PLEG): container finished" podID="e27f22dd-e82f-493c-9745-6d901308b907" containerID="3ce1abbdae6f4b5a0ac4e38ce85ec955c29d603a732f9dc363e5456ad51607b8" exitCode=60 Oct 31 05:00:06.752778 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:06.752757 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890380-b7jtg" event={"ID":"e27f22dd-e82f-493c-9745-6d901308b907","Type":"ContainerDied","Data":"3ce1abbdae6f4b5a0ac4e38ce85ec955c29d603a732f9dc363e5456ad51607b8"} Oct 31 05:00:06.753137 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:06.753119 5019 scope.go:122] "RemoveContainer" containerID="3ce1abbdae6f4b5a0ac4e38ce85ec955c29d603a732f9dc363e5456ad51607b8" Oct 31 05:00:06.753607 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:06.753585 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890380-b7jtg_odh-ai-gateway-infra(e27f22dd-e82f-493c-9745-6d901308b907)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890380-b7jtg" podUID="e27f22dd-e82f-493c-9745-6d901308b907" Oct 31 05:00:06.755000 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:06.754977 5019 generic.go:356] "Generic (PLEG): container finished" podID="9790f69f-4de3-4837-a29f-17866661a3b3" containerID="ccda27e6a77112b7770bb93f1c01871a0a0327f5eae723af09f6d86bad4cffee" exitCode=60 Oct 31 05:00:06.755118 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:06.755042 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890380-tx4lq" event={"ID":"9790f69f-4de3-4837-a29f-17866661a3b3","Type":"ContainerDied","Data":"ccda27e6a77112b7770bb93f1c01871a0a0327f5eae723af09f6d86bad4cffee"} Oct 31 05:00:06.755431 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:06.755410 5019 scope.go:122] "RemoveContainer" containerID="ccda27e6a77112b7770bb93f1c01871a0a0327f5eae723af09f6d86bad4cffee" Oct 31 05:00:06.755781 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:06.755761 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c4006d-29890380-tx4lq_odh-ai-gateway-infra(9790f69f-4de3-4837-a29f-17866661a3b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890380-tx4lq" podUID="9790f69f-4de3-4837-a29f-17866661a3b3" Oct 31 05:00:06.757233 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:06.757205 5019 generic.go:356] "Generic (PLEG): container finished" podID="f59877aa-80e8-418d-9617-2f9bb9c8f3ef" containerID="67cd3cea94dc92f993f190f26c65f6025368fc4b53f21e939d8fd988687f839a" exitCode=60 Oct 31 05:00:06.757323 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:06.757232 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890380-v7gl7" event={"ID":"f59877aa-80e8-418d-9617-2f9bb9c8f3ef","Type":"ContainerDied","Data":"67cd3cea94dc92f993f190f26c65f6025368fc4b53f21e939d8fd988687f839a"} Oct 31 05:00:06.758730 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:06.757840 5019 scope.go:122] "RemoveContainer" containerID="67cd3cea94dc92f993f190f26c65f6025368fc4b53f21e939d8fd988687f839a" Oct 31 05:00:06.758730 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:06.758236 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890380-v7gl7_odh-ai-gateway-infra(f59877aa-80e8-418d-9617-2f9bb9c8f3ef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890380-v7gl7" podUID="f59877aa-80e8-418d-9617-2f9bb9c8f3ef" Oct 31 05:00:06.759741 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:06.759720 5019 generic.go:356] "Generic (PLEG): container finished" podID="982ad40e-a719-4f61-be85-ea665503c299" containerID="73c4fe02e89785fe6202c4d2bdfe58135b720afe245e6f95dbc706d9ca2e4aa7" exitCode=60 Oct 31 05:00:06.759851 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:06.759811 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890380-rmxrz" event={"ID":"982ad40e-a719-4f61-be85-ea665503c299","Type":"ContainerDied","Data":"73c4fe02e89785fe6202c4d2bdfe58135b720afe245e6f95dbc706d9ca2e4aa7"} Oct 31 05:00:06.760201 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:06.760183 5019 scope.go:122] "RemoveContainer" containerID="73c4fe02e89785fe6202c4d2bdfe58135b720afe245e6f95dbc706d9ca2e4aa7" Oct 31 05:00:06.760628 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:06.760603 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890380-rmxrz_odh-ai-gateway-infra(982ad40e-a719-4f61-be85-ea665503c299)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890380-rmxrz" podUID="982ad40e-a719-4f61-be85-ea665503c299" Oct 31 05:00:06.762544 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:06.762520 5019 generic.go:356] "Generic (PLEG): container finished" podID="d13131fa-5137-4401-aef8-d53fd8bd4e5d" containerID="4815a20852906bfd87655f3881d5c0eda440c1eb65c7a7435a2e2d0cc2af711a" exitCode=60 Oct 31 05:00:06.762641 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:06.762595 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-47c409-29890380-cqm6g" event={"ID":"d13131fa-5137-4401-aef8-d53fd8bd4e5d","Type":"ContainerDied","Data":"4815a20852906bfd87655f3881d5c0eda440c1eb65c7a7435a2e2d0cc2af711a"} Oct 31 05:00:06.762951 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:06.762935 5019 scope.go:122] "RemoveContainer" containerID="4815a20852906bfd87655f3881d5c0eda440c1eb65c7a7435a2e2d0cc2af711a" Oct 31 05:00:06.763106 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:06.763090 5019 scope.go:122] "RemoveContainer" containerID="4b4377c5db90806371b2b551279f4bbc84522ccc6de2d0d0967bd6da2e216700" Oct 31 05:00:06.763429 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:06.763402 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-47c409-29890380-cqm6g_odh-ai-gateway-infra(d13131fa-5137-4401-aef8-d53fd8bd4e5d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-47c409-29890380-cqm6g" podUID="d13131fa-5137-4401-aef8-d53fd8bd4e5d" Oct 31 05:00:06.763492 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:06.763434 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-a04e4e-29890380-jgchh_odh-ai-gateway-infra(0b6b5e89-5900-4db2-8552-e4f029de7573)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a04e4e-29890380-jgchh" podUID="0b6b5e89-5900-4db2-8552-e4f029de7573" Oct 31 05:00:07.050778 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:07.050637 5019 scope.go:122] "RemoveContainer" containerID="25e8929523907b64d88120a2d9e64a67f48ecf50bb65dc1e3cbf9cc5f94987ab" Oct 31 05:00:07.102570 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:07.102537 5019 scope.go:122] "RemoveContainer" containerID="15457cd712676393b895b973612547c284d609f298b560788f8deefbe5631d6f" Oct 31 05:00:07.128955 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:07.128929 5019 scope.go:122] "RemoveContainer" containerID="68ca4fd6cb2d5c2738c21e0645161b22c96a5f78fe4b4a596a135aec9c97af7b" Oct 31 05:00:07.147689 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:07.147663 5019 scope.go:122] "RemoveContainer" containerID="555da1a1e8b971b314b2d3fd7fc38f5096d719630ff9bcee75b3c03b2fff24ae" Oct 31 05:00:07.167834 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:07.167802 5019 scope.go:122] "RemoveContainer" containerID="7964a83895049c03f2b385afb6320b913f15ccb2dc893145d9de6f1d28499636" Oct 31 05:00:07.201535 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:07.201371 5019 scope.go:122] "RemoveContainer" containerID="1d73f7724438a8711f7371e99ea6fb64e1a089a7f8d55d699a6ac063205c330a" Oct 31 05:00:07.220107 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:07.220015 5019 scope.go:122] "RemoveContainer" containerID="ee1d28ad3481b569977e8791b92e8961c3c0d6254d11acea704028dad80ba52f" Oct 31 05:00:07.239630 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:07.239498 5019 scope.go:122] "RemoveContainer" containerID="51a7208722da3395ac4d14c37e69b6d845c509de6bc00558303c3fc8f679e1cf" Oct 31 05:00:07.264074 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:07.263715 5019 scope.go:122] "RemoveContainer" containerID="6d97a1a4faaa99cfb2890bcc32b0562f5126a0f94ebc1c7fc11da3692df248ed" Oct 31 05:00:07.288190 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:07.288142 5019 scope.go:122] "RemoveContainer" containerID="f7d432a1307fb2b83c4d8a6fd4001137df73dfd63e893388becb147010bc647c" Oct 31 05:00:07.306488 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:07.306455 5019 scope.go:122] "RemoveContainer" containerID="3654e96b7edaf039e141cbdea019474e2cdaeada5203c50c9914498da2f0c4d8" Oct 31 05:00:07.785025 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:07.783497 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890380-7g9d5" event={"ID":"93bfb362-fcc2-4341-9881-1fd06b546590","Type":"ContainerStarted","Data":"b4376c22d8410400f9b4b22621b14be20195284a8a55e77f02351f9ca8e3af04"} Oct 31 05:00:07.790284 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:07.790080 5019 scope.go:122] "RemoveContainer" containerID="fa1c952e213fc2db874f7214d12dd9757aed0c6fc1e318427ea08af5651005c4" Oct 31 05:00:07.790637 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:07.790608 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890380-6qwg6_odh-ai-gateway-infra(e9b19f33-a012-48c9-b670-3b8eff773e49)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890380-6qwg6" podUID="e9b19f33-a012-48c9-b670-3b8eff773e49" Oct 31 05:00:07.800816 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:07.800788 5019 scope.go:122] "RemoveContainer" containerID="307dc9b0647618d9e86a3118ba70766b75e9974d27930eae830018e534d93edd" Oct 31 05:00:07.801347 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:07.801307 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-fb346b-29890380-l8fpr_odh-ai-gateway-infra(829dc86b-4ce1-427e-86db-cbaab8f15198)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fb346b-29890380-l8fpr" podUID="829dc86b-4ce1-427e-86db-cbaab8f15198" Oct 31 05:00:07.801672 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:07.801602 5019 scope.go:122] "RemoveContainer" containerID="b249f4ff0351d3704e4d3f0a0ab4e5428603ad8f889f3e0546a3773ac369dc9d" Oct 31 05:00:07.802086 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:07.802058 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-8973e2-29890380-ldsqw_odh-ai-gateway-infra(e1e9a647-0e9e-4528-a5a6-7022b244f1ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890380-ldsqw" podUID="e1e9a647-0e9e-4528-a5a6-7022b244f1ba" Oct 31 05:00:07.803814 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:07.803792 5019 scope.go:122] "RemoveContainer" containerID="4b2241ffbbf74bceaaa581e0c48f1143ec969f2a7343bd8f08f814ed3ea2b57f" Oct 31 05:00:07.804196 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:07.804171 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-031745-29890380-9jpj5_odh-ai-gateway-infra(d3d4f5cd-c755-4e4f-a86f-14874e597d08)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-031745-29890380-9jpj5" podUID="d3d4f5cd-c755-4e4f-a86f-14874e597d08" Oct 31 05:00:07.806003 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:07.805975 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890380-fw6dq" event={"ID":"96e0a71b-2eee-4760-ade1-5cdd604c4c81","Type":"ContainerStarted","Data":"7a47eb7c3a9c9aef47dbc58bb5b2801ca687ced5155cb619265909c644783e64"} Oct 31 05:00:07.808082 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:07.808062 5019 scope.go:122] "RemoveContainer" containerID="0ae3171d94a4e9fc4512711515cb6300d725282ddfb9097757734314266dfcb2" Oct 31 05:00:07.808479 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:07.808453 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-401761-29890380-zvx8f_odh-ai-gateway-infra(4fd6b126-7a7c-4825-a374-5dec6fb94d06)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-401761-29890380-zvx8f" podUID="4fd6b126-7a7c-4825-a374-5dec6fb94d06" Oct 31 05:00:07.810197 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:07.810177 5019 scope.go:122] "RemoveContainer" containerID="ed02bf60ecb10a21a7f0e07c1abb134e22852b8304d083726ba9c1258db41d2e" Oct 31 05:00:07.810538 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:07.810515 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-108ad8-29890380-ng62l_odh-ai-gateway-infra(e5259d89-590d-4fc7-b521-8f6feabbd2cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-108ad8-29890380-ng62l" podUID="e5259d89-590d-4fc7-b521-8f6feabbd2cc" Oct 31 05:00:07.813018 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:07.812135 5019 scope.go:122] "RemoveContainer" containerID="3ce1abbdae6f4b5a0ac4e38ce85ec955c29d603a732f9dc363e5456ad51607b8" Oct 31 05:00:07.813018 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:07.812484 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890380-b7jtg_odh-ai-gateway-infra(e27f22dd-e82f-493c-9745-6d901308b907)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890380-b7jtg" podUID="e27f22dd-e82f-493c-9745-6d901308b907" Oct 31 05:00:07.814314 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:07.814234 5019 scope.go:122] "RemoveContainer" containerID="ccda27e6a77112b7770bb93f1c01871a0a0327f5eae723af09f6d86bad4cffee" Oct 31 05:00:07.814619 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:07.814593 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c4006d-29890380-tx4lq_odh-ai-gateway-infra(9790f69f-4de3-4837-a29f-17866661a3b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890380-tx4lq" podUID="9790f69f-4de3-4837-a29f-17866661a3b3" Oct 31 05:00:07.816179 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:07.816157 5019 scope.go:122] "RemoveContainer" containerID="67cd3cea94dc92f993f190f26c65f6025368fc4b53f21e939d8fd988687f839a" Oct 31 05:00:07.816487 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:07.816462 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890380-v7gl7_odh-ai-gateway-infra(f59877aa-80e8-418d-9617-2f9bb9c8f3ef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890380-v7gl7" podUID="f59877aa-80e8-418d-9617-2f9bb9c8f3ef" Oct 31 05:00:07.818386 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:07.818366 5019 scope.go:122] "RemoveContainer" containerID="73c4fe02e89785fe6202c4d2bdfe58135b720afe245e6f95dbc706d9ca2e4aa7" Oct 31 05:00:07.818688 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:07.818664 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890380-rmxrz_odh-ai-gateway-infra(982ad40e-a719-4f61-be85-ea665503c299)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890380-rmxrz" podUID="982ad40e-a719-4f61-be85-ea665503c299" Oct 31 05:00:07.820586 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:07.820561 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-567ca6-29890380-kcwdg" event={"ID":"703c28bc-a603-4001-9538-4a751cf9b7c0","Type":"ContainerStarted","Data":"b1dd400398607ff40e67dca107dd07ed4a12b5dce7fb7eb481453dcce873925d"} Oct 31 05:00:07.831620 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:07.831590 5019 scope.go:122] "RemoveContainer" containerID="4815a20852906bfd87655f3881d5c0eda440c1eb65c7a7435a2e2d0cc2af711a" Oct 31 05:00:07.832039 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:07.831997 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-47c409-29890380-cqm6g_odh-ai-gateway-infra(d13131fa-5137-4401-aef8-d53fd8bd4e5d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-47c409-29890380-cqm6g" podUID="d13131fa-5137-4401-aef8-d53fd8bd4e5d" Oct 31 05:00:07.836719 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:07.836688 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-47e9af-29890380-jk7b6" event={"ID":"7c20d99d-7ce4-4bb8-a4d9-188cec401d75","Type":"ContainerStarted","Data":"e6aa6c0788c5a4e2329732970dd459152f7207dcaa204cc9226ef9c93dfec3d2"} Oct 31 05:00:08.360827 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:08.360757 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-47e9af-29890380-jk7b6" podStartSLOduration=8.360742296 podStartE2EDuration="8.360742296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:00:08.360113906 +0000 UTC m=+3363.116451945" watchObservedRunningTime="2026-10-31 05:00:08.360742296 +0000 UTC m=+3363.117080340" Oct 31 05:00:08.863159 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:08.863118 5019 generic.go:356] "Generic (PLEG): container finished" podID="703c28bc-a603-4001-9538-4a751cf9b7c0" containerID="b1dd400398607ff40e67dca107dd07ed4a12b5dce7fb7eb481453dcce873925d" exitCode=60 Oct 31 05:00:08.863709 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:08.863197 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-567ca6-29890380-kcwdg" event={"ID":"703c28bc-a603-4001-9538-4a751cf9b7c0","Type":"ContainerDied","Data":"b1dd400398607ff40e67dca107dd07ed4a12b5dce7fb7eb481453dcce873925d"} Oct 31 05:00:08.863709 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:08.863302 5019 scope.go:122] "RemoveContainer" containerID="d807a180acbc5a50860443a4eb8afc9f9d045c59ba10cc96437fa81439113566" Oct 31 05:00:08.863709 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:08.863648 5019 scope.go:122] "RemoveContainer" containerID="b1dd400398607ff40e67dca107dd07ed4a12b5dce7fb7eb481453dcce873925d" Oct 31 05:00:08.864138 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:08.864107 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-567ca6-29890380-kcwdg_odh-ai-gateway-infra(703c28bc-a603-4001-9538-4a751cf9b7c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-567ca6-29890380-kcwdg" podUID="703c28bc-a603-4001-9538-4a751cf9b7c0" Oct 31 05:00:08.865546 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:08.865523 5019 generic.go:356] "Generic (PLEG): container finished" podID="deaadd33-dcef-43df-a66e-9fd41434de6f" containerID="bf7f393c48f1a8d8d00b56318c44ebe02be183b0f93246deb2ffe76e2e1ec8e1" exitCode=60 Oct 31 05:00:08.865652 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:08.865605 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890380-k8lq8" event={"ID":"deaadd33-dcef-43df-a66e-9fd41434de6f","Type":"ContainerDied","Data":"bf7f393c48f1a8d8d00b56318c44ebe02be183b0f93246deb2ffe76e2e1ec8e1"} Oct 31 05:00:08.865962 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:08.865945 5019 scope.go:122] "RemoveContainer" containerID="bf7f393c48f1a8d8d00b56318c44ebe02be183b0f93246deb2ffe76e2e1ec8e1" Oct 31 05:00:08.866408 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:08.866387 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890380-k8lq8_odh-ai-gateway-infra(deaadd33-dcef-43df-a66e-9fd41434de6f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890380-k8lq8" podUID="deaadd33-dcef-43df-a66e-9fd41434de6f" Oct 31 05:00:08.867706 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:08.867686 5019 generic.go:356] "Generic (PLEG): container finished" podID="cd0723e3-dcf6-4907-9ce1-99ca9c77b144" containerID="42c9c9d6a5eb52a1726c02f5a06ad5de76f43254a6628953e9ff3f19367afe38" exitCode=60 Oct 31 05:00:08.867797 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:08.867757 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1daf30-29890380-pbtq2" event={"ID":"cd0723e3-dcf6-4907-9ce1-99ca9c77b144","Type":"ContainerDied","Data":"42c9c9d6a5eb52a1726c02f5a06ad5de76f43254a6628953e9ff3f19367afe38"} Oct 31 05:00:08.868145 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:08.868125 5019 scope.go:122] "RemoveContainer" containerID="42c9c9d6a5eb52a1726c02f5a06ad5de76f43254a6628953e9ff3f19367afe38" Oct 31 05:00:08.868584 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:08.868562 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-1daf30-29890380-pbtq2_odh-ai-gateway-infra(cd0723e3-dcf6-4907-9ce1-99ca9c77b144)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1daf30-29890380-pbtq2" podUID="cd0723e3-dcf6-4907-9ce1-99ca9c77b144" Oct 31 05:00:08.875176 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:08.875149 5019 generic.go:356] "Generic (PLEG): container finished" podID="7c20d99d-7ce4-4bb8-a4d9-188cec401d75" containerID="e6aa6c0788c5a4e2329732970dd459152f7207dcaa204cc9226ef9c93dfec3d2" exitCode=60 Oct 31 05:00:08.875311 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:08.875229 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-47e9af-29890380-jk7b6" event={"ID":"7c20d99d-7ce4-4bb8-a4d9-188cec401d75","Type":"ContainerDied","Data":"e6aa6c0788c5a4e2329732970dd459152f7207dcaa204cc9226ef9c93dfec3d2"} Oct 31 05:00:08.875684 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:08.875662 5019 scope.go:122] "RemoveContainer" containerID="e6aa6c0788c5a4e2329732970dd459152f7207dcaa204cc9226ef9c93dfec3d2" Oct 31 05:00:08.876130 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:08.876103 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-47e9af-29890380-jk7b6_odh-ai-gateway-infra(7c20d99d-7ce4-4bb8-a4d9-188cec401d75)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-47e9af-29890380-jk7b6" podUID="7c20d99d-7ce4-4bb8-a4d9-188cec401d75" Oct 31 05:00:08.877616 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:08.877592 5019 generic.go:356] "Generic (PLEG): container finished" podID="93bfb362-fcc2-4341-9881-1fd06b546590" containerID="b4376c22d8410400f9b4b22621b14be20195284a8a55e77f02351f9ca8e3af04" exitCode=60 Oct 31 05:00:08.877726 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:08.877670 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890380-7g9d5" event={"ID":"93bfb362-fcc2-4341-9881-1fd06b546590","Type":"ContainerDied","Data":"b4376c22d8410400f9b4b22621b14be20195284a8a55e77f02351f9ca8e3af04"} Oct 31 05:00:08.877971 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:08.877952 5019 scope.go:122] "RemoveContainer" containerID="b4376c22d8410400f9b4b22621b14be20195284a8a55e77f02351f9ca8e3af04" Oct 31 05:00:08.878385 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:08.878356 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-227c93-29890380-7g9d5_odh-ai-gateway-infra(93bfb362-fcc2-4341-9881-1fd06b546590)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890380-7g9d5" podUID="93bfb362-fcc2-4341-9881-1fd06b546590" Oct 31 05:00:08.880079 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:08.880059 5019 generic.go:356] "Generic (PLEG): container finished" podID="c9515dc7-4ea3-4a1b-aad8-1e78c7d0e13c" containerID="4341f8d50a9838ae7ff595641b379c4b037a565f6b3e5bb5e483ee638ccd3d3d" exitCode=60 Oct 31 05:00:08.880181 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:08.880084 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-670795-29890380-hzmsc" event={"ID":"c9515dc7-4ea3-4a1b-aad8-1e78c7d0e13c","Type":"ContainerDied","Data":"4341f8d50a9838ae7ff595641b379c4b037a565f6b3e5bb5e483ee638ccd3d3d"} Oct 31 05:00:08.880481 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:08.880463 5019 scope.go:122] "RemoveContainer" containerID="4341f8d50a9838ae7ff595641b379c4b037a565f6b3e5bb5e483ee638ccd3d3d" Oct 31 05:00:08.880883 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:08.880858 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-670795-29890380-hzmsc_odh-ai-gateway-infra(c9515dc7-4ea3-4a1b-aad8-1e78c7d0e13c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-670795-29890380-hzmsc" podUID="c9515dc7-4ea3-4a1b-aad8-1e78c7d0e13c" Oct 31 05:00:08.882527 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:08.882497 5019 generic.go:356] "Generic (PLEG): container finished" podID="e23416e6-954a-4ee6-8c4f-f6f8d7e1c506" containerID="1b602417b78f0fb3ff0a24019758993c48d43aa12e36e7521950ee2edaad5b01" exitCode=60 Oct 31 05:00:08.882625 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:08.882579 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-654741-29890380-85tdm" event={"ID":"e23416e6-954a-4ee6-8c4f-f6f8d7e1c506","Type":"ContainerDied","Data":"1b602417b78f0fb3ff0a24019758993c48d43aa12e36e7521950ee2edaad5b01"} Oct 31 05:00:08.882869 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:08.882852 5019 scope.go:122] "RemoveContainer" containerID="1b602417b78f0fb3ff0a24019758993c48d43aa12e36e7521950ee2edaad5b01" Oct 31 05:00:08.883196 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:08.883177 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-654741-29890380-85tdm_odh-ai-gateway-infra(e23416e6-954a-4ee6-8c4f-f6f8d7e1c506)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-654741-29890380-85tdm" podUID="e23416e6-954a-4ee6-8c4f-f6f8d7e1c506" Oct 31 05:00:08.885130 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:08.885113 5019 generic.go:356] "Generic (PLEG): container finished" podID="d0dacaaf-e177-420b-a953-f8cdb4961a27" containerID="75453d9bd304c57b9a965b1089fdbb266c733384e0286e68e493f5ac791b89aa" exitCode=60 Oct 31 05:00:08.885219 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:08.885199 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890380-fh8kl" event={"ID":"d0dacaaf-e177-420b-a953-f8cdb4961a27","Type":"ContainerDied","Data":"75453d9bd304c57b9a965b1089fdbb266c733384e0286e68e493f5ac791b89aa"} Oct 31 05:00:08.885489 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:08.885473 5019 scope.go:122] "RemoveContainer" containerID="75453d9bd304c57b9a965b1089fdbb266c733384e0286e68e493f5ac791b89aa" Oct 31 05:00:08.885758 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:08.885731 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-965f61-29890380-fh8kl_odh-ai-gateway-infra(d0dacaaf-e177-420b-a953-f8cdb4961a27)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890380-fh8kl" podUID="d0dacaaf-e177-420b-a953-f8cdb4961a27" Oct 31 05:00:08.887493 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:08.887452 5019 generic.go:356] "Generic (PLEG): container finished" podID="7ab61246-5d2d-4a2a-8c42-fff22960018f" containerID="939033db9d1fbe818fc40ca51fefca9989c43a8489b97b983c36af647203a762" exitCode=60 Oct 31 05:00:08.887783 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:08.887755 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890380-kz5pd" event={"ID":"7ab61246-5d2d-4a2a-8c42-fff22960018f","Type":"ContainerDied","Data":"939033db9d1fbe818fc40ca51fefca9989c43a8489b97b983c36af647203a762"} Oct 31 05:00:08.887852 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:08.887833 5019 scope.go:122] "RemoveContainer" containerID="939033db9d1fbe818fc40ca51fefca9989c43a8489b97b983c36af647203a762" Oct 31 05:00:08.888124 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:08.888096 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890380-kz5pd_odh-ai-gateway-infra(7ab61246-5d2d-4a2a-8c42-fff22960018f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890380-kz5pd" podUID="7ab61246-5d2d-4a2a-8c42-fff22960018f" Oct 31 05:00:08.890178 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:08.890145 5019 generic.go:356] "Generic (PLEG): container finished" podID="96e0a71b-2eee-4760-ade1-5cdd604c4c81" containerID="7a47eb7c3a9c9aef47dbc58bb5b2801ca687ced5155cb619265909c644783e64" exitCode=60 Oct 31 05:00:08.890248 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:08.890219 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890380-fw6dq" event={"ID":"96e0a71b-2eee-4760-ade1-5cdd604c4c81","Type":"ContainerDied","Data":"7a47eb7c3a9c9aef47dbc58bb5b2801ca687ced5155cb619265909c644783e64"} Oct 31 05:00:08.890930 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:08.890910 5019 scope.go:122] "RemoveContainer" containerID="7a47eb7c3a9c9aef47dbc58bb5b2801ca687ced5155cb619265909c644783e64" Oct 31 05:00:08.891478 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:08.891226 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-09a20a-29890380-fw6dq_odh-ai-gateway-infra(96e0a71b-2eee-4760-ade1-5cdd604c4c81)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890380-fw6dq" podUID="96e0a71b-2eee-4760-ade1-5cdd604c4c81" Oct 31 05:00:08.892761 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:08.892737 5019 generic.go:356] "Generic (PLEG): container finished" podID="17f1de0f-3edd-4d07-910a-f6e6f356203c" containerID="df8d8adef2a9b7d4d7bf0ae9182e41601e535a6890077ca74cfe319a8821a03b" exitCode=60 Oct 31 05:00:08.892846 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:08.892817 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890380-kr424" event={"ID":"17f1de0f-3edd-4d07-910a-f6e6f356203c","Type":"ContainerDied","Data":"df8d8adef2a9b7d4d7bf0ae9182e41601e535a6890077ca74cfe319a8821a03b"} Oct 31 05:00:08.893055 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:08.893035 5019 scope.go:122] "RemoveContainer" containerID="df8d8adef2a9b7d4d7bf0ae9182e41601e535a6890077ca74cfe319a8821a03b" Oct 31 05:00:08.893303 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:08.893285 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-afc519-29890380-kr424_odh-ai-gateway-infra(17f1de0f-3edd-4d07-910a-f6e6f356203c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890380-kr424" podUID="17f1de0f-3edd-4d07-910a-f6e6f356203c" Oct 31 05:00:08.894967 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:08.894945 5019 generic.go:356] "Generic (PLEG): container finished" podID="e478c8f1-9e68-4f6f-b919-28ba397e66c4" containerID="93cfacf43741883f8f7341fea7111a8a28f11b3dad97d1dbe446e59c5b77c842" exitCode=60 Oct 31 05:00:08.895066 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:08.894990 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890380-tkvr7" event={"ID":"e478c8f1-9e68-4f6f-b919-28ba397e66c4","Type":"ContainerDied","Data":"93cfacf43741883f8f7341fea7111a8a28f11b3dad97d1dbe446e59c5b77c842"} Oct 31 05:00:08.895261 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:08.895246 5019 scope.go:122] "RemoveContainer" containerID="93cfacf43741883f8f7341fea7111a8a28f11b3dad97d1dbe446e59c5b77c842" Oct 31 05:00:08.895514 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:08.895491 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-4c2387-29890380-tkvr7_odh-ai-gateway-infra(e478c8f1-9e68-4f6f-b919-28ba397e66c4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890380-tkvr7" podUID="e478c8f1-9e68-4f6f-b919-28ba397e66c4" Oct 31 05:00:09.047866 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:09.047836 5019 scope.go:122] "RemoveContainer" containerID="cd9ca19be0efd1f15845ee46b04967e99ef188ee00fb65fabd59f523ebad1ce3" Oct 31 05:00:09.064740 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:09.064715 5019 scope.go:122] "RemoveContainer" containerID="3438843aeecdeb2ce68af9c9b78ce7bdc1d33a2d6ea87a3da15beeeb103ff737" Oct 31 05:00:09.083920 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:09.083888 5019 scope.go:122] "RemoveContainer" containerID="3f6b4961e4d3cab25aa21421f9e4585b116ff0fea23bc674e76b94b8b1732404" Oct 31 05:00:09.104902 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:09.104868 5019 scope.go:122] "RemoveContainer" containerID="c77a304d3332e55e91f7b0105d5efbc776deeaeeef86901a4e5e810c1d5aea9c" Oct 31 05:00:09.123657 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:09.123628 5019 scope.go:122] "RemoveContainer" containerID="8662a6aa5422c851875c67cf8a73f258959627296961118f2f4f32d6729f9728" Oct 31 05:00:09.144121 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:09.144083 5019 scope.go:122] "RemoveContainer" containerID="71083e8b3f82b9a55409a59770bbf6a1a66ac24369a218a5b9441598b8e97bc0" Oct 31 05:00:09.161603 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:09.161569 5019 scope.go:122] "RemoveContainer" containerID="34c97cc998485973569636bdc63787eb972b4e9523cf7c16298a5f60c9f6282d" Oct 31 05:00:09.181210 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:09.181172 5019 scope.go:122] "RemoveContainer" containerID="09f295a65e46b70f6868801fb8302bfcf6209d92ea5aca45c4d6fbc665a053ed" Oct 31 05:00:09.200118 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:09.200086 5019 scope.go:122] "RemoveContainer" containerID="74ad998b84a4ff8fbf4f04304ced47eb6b91784d4c7a65a6a4e03d075982d011" Oct 31 05:00:09.219456 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:09.219409 5019 scope.go:122] "RemoveContainer" containerID="10128b3d93fc46591eb41e380a3f73d494d2b30cd4494a54511f4f6b313af267" Oct 31 05:00:09.236162 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:09.236139 5019 scope.go:122] "RemoveContainer" containerID="de1e3cb77f9b581a3d051a71aa2cd4b4a163b8e71ef12a5092db722abb6add93" Oct 31 05:00:09.908704 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:09.908665 5019 scope.go:122] "RemoveContainer" containerID="4341f8d50a9838ae7ff595641b379c4b037a565f6b3e5bb5e483ee638ccd3d3d" Oct 31 05:00:09.909242 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:09.909027 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-670795-29890380-hzmsc_odh-ai-gateway-infra(c9515dc7-4ea3-4a1b-aad8-1e78c7d0e13c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-670795-29890380-hzmsc" podUID="c9515dc7-4ea3-4a1b-aad8-1e78c7d0e13c" Oct 31 05:00:09.910904 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:09.910884 5019 scope.go:122] "RemoveContainer" containerID="1b602417b78f0fb3ff0a24019758993c48d43aa12e36e7521950ee2edaad5b01" Oct 31 05:00:09.911180 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:09.911160 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-654741-29890380-85tdm_odh-ai-gateway-infra(e23416e6-954a-4ee6-8c4f-f6f8d7e1c506)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-654741-29890380-85tdm" podUID="e23416e6-954a-4ee6-8c4f-f6f8d7e1c506" Oct 31 05:00:09.913113 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:09.913096 5019 scope.go:122] "RemoveContainer" containerID="75453d9bd304c57b9a965b1089fdbb266c733384e0286e68e493f5ac791b89aa" Oct 31 05:00:09.913402 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:09.913377 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-965f61-29890380-fh8kl_odh-ai-gateway-infra(d0dacaaf-e177-420b-a953-f8cdb4961a27)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890380-fh8kl" podUID="d0dacaaf-e177-420b-a953-f8cdb4961a27" Oct 31 05:00:09.915092 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:09.915062 5019 scope.go:122] "RemoveContainer" containerID="939033db9d1fbe818fc40ca51fefca9989c43a8489b97b983c36af647203a762" Oct 31 05:00:09.915369 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:09.915327 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890380-kz5pd_odh-ai-gateway-infra(7ab61246-5d2d-4a2a-8c42-fff22960018f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890380-kz5pd" podUID="7ab61246-5d2d-4a2a-8c42-fff22960018f" Oct 31 05:00:09.917055 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:09.917035 5019 scope.go:122] "RemoveContainer" containerID="7a47eb7c3a9c9aef47dbc58bb5b2801ca687ced5155cb619265909c644783e64" Oct 31 05:00:09.917317 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:09.917292 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-09a20a-29890380-fw6dq_odh-ai-gateway-infra(96e0a71b-2eee-4760-ade1-5cdd604c4c81)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890380-fw6dq" podUID="96e0a71b-2eee-4760-ade1-5cdd604c4c81" Oct 31 05:00:09.918983 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:09.918954 5019 scope.go:122] "RemoveContainer" containerID="df8d8adef2a9b7d4d7bf0ae9182e41601e535a6890077ca74cfe319a8821a03b" Oct 31 05:00:09.919259 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:09.919229 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-afc519-29890380-kr424_odh-ai-gateway-infra(17f1de0f-3edd-4d07-910a-f6e6f356203c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890380-kr424" podUID="17f1de0f-3edd-4d07-910a-f6e6f356203c" Oct 31 05:00:09.920842 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:09.920823 5019 scope.go:122] "RemoveContainer" containerID="93cfacf43741883f8f7341fea7111a8a28f11b3dad97d1dbe446e59c5b77c842" Oct 31 05:00:09.921128 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:09.921106 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-4c2387-29890380-tkvr7_odh-ai-gateway-infra(e478c8f1-9e68-4f6f-b919-28ba397e66c4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890380-tkvr7" podUID="e478c8f1-9e68-4f6f-b919-28ba397e66c4" Oct 31 05:00:09.922559 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:09.922536 5019 scope.go:122] "RemoveContainer" containerID="b1dd400398607ff40e67dca107dd07ed4a12b5dce7fb7eb481453dcce873925d" Oct 31 05:00:09.922784 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:09.922758 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-567ca6-29890380-kcwdg_odh-ai-gateway-infra(703c28bc-a603-4001-9538-4a751cf9b7c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-567ca6-29890380-kcwdg" podUID="703c28bc-a603-4001-9538-4a751cf9b7c0" Oct 31 05:00:09.924232 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:09.924216 5019 scope.go:122] "RemoveContainer" containerID="bf7f393c48f1a8d8d00b56318c44ebe02be183b0f93246deb2ffe76e2e1ec8e1" Oct 31 05:00:09.924524 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:09.924491 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890380-k8lq8_odh-ai-gateway-infra(deaadd33-dcef-43df-a66e-9fd41434de6f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890380-k8lq8" podUID="deaadd33-dcef-43df-a66e-9fd41434de6f" Oct 31 05:00:09.926046 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:09.926013 5019 scope.go:122] "RemoveContainer" containerID="42c9c9d6a5eb52a1726c02f5a06ad5de76f43254a6628953e9ff3f19367afe38" Oct 31 05:00:09.926308 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:09.926281 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-1daf30-29890380-pbtq2_odh-ai-gateway-infra(cd0723e3-dcf6-4907-9ce1-99ca9c77b144)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1daf30-29890380-pbtq2" podUID="cd0723e3-dcf6-4907-9ce1-99ca9c77b144" Oct 31 05:00:09.927902 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:09.927860 5019 scope.go:122] "RemoveContainer" containerID="e6aa6c0788c5a4e2329732970dd459152f7207dcaa204cc9226ef9c93dfec3d2" Oct 31 05:00:09.928143 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:09.928119 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-47e9af-29890380-jk7b6_odh-ai-gateway-infra(7c20d99d-7ce4-4bb8-a4d9-188cec401d75)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-47e9af-29890380-jk7b6" podUID="7c20d99d-7ce4-4bb8-a4d9-188cec401d75" Oct 31 05:00:09.929492 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:09.929474 5019 scope.go:122] "RemoveContainer" containerID="b4376c22d8410400f9b4b22621b14be20195284a8a55e77f02351f9ca8e3af04" Oct 31 05:00:09.929731 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:09.929709 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-227c93-29890380-7g9d5_odh-ai-gateway-infra(93bfb362-fcc2-4341-9881-1fd06b546590)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890380-7g9d5" podUID="93bfb362-fcc2-4341-9881-1fd06b546590" Oct 31 05:00:12.151590 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:12.151551 5019 scope.go:122] "RemoveContainer" containerID="f57907e303a7dc599e3e49f8e946b8a8bcbc302aed9da2e3aac4a2e9979df2bb" Oct 31 05:00:12.165270 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:12.165237 5019 scope.go:122] "RemoveContainer" containerID="7d5e9fb0e58e3de2848acadea0f1aa8f96465b5dc337246addbb580d75a6e384" Oct 31 05:00:12.179536 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:12.179498 5019 scope.go:122] "RemoveContainer" containerID="b175dcd2444cc6289ab7a2a9d7f0fc0168ac7d174d4c0d1bbb40fa487aeab5c1" Oct 31 05:00:13.974676 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:13.974354 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-c1fe26/test-model-c1fe26-kserve-7fd8b8b8f-qkbvb"] Oct 31 05:00:14.000685 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:13.999326 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c1fe26/test-model-c1fe26-kserve-7fd8b8b8f-qkbvb" Oct 31 05:00:14.027312 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:14.027269 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35fb8fb1-3a58-465b-b53a-08fe806af299-tls-certs\") pod \"test-model-c1fe26-kserve-7fd8b8b8f-qkbvb\" (UID: \"35fb8fb1-3a58-465b-b53a-08fe806af299\") " pod="ai-tenant-e2e-inf-b-c1fe26/test-model-c1fe26-kserve-7fd8b8b8f-qkbvb" Oct 31 05:00:14.027689 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:14.027660 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35fb8fb1-3a58-465b-b53a-08fe806af299-dshm\") pod \"test-model-c1fe26-kserve-7fd8b8b8f-qkbvb\" (UID: \"35fb8fb1-3a58-465b-b53a-08fe806af299\") " pod="ai-tenant-e2e-inf-b-c1fe26/test-model-c1fe26-kserve-7fd8b8b8f-qkbvb" Oct 31 05:00:14.027997 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:14.027970 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8g6t\" (UniqueName: \"kubernetes.io/projected/35fb8fb1-3a58-465b-b53a-08fe806af299-kube-api-access-l8g6t\") pod \"test-model-c1fe26-kserve-7fd8b8b8f-qkbvb\" (UID: \"35fb8fb1-3a58-465b-b53a-08fe806af299\") " pod="ai-tenant-e2e-inf-b-c1fe26/test-model-c1fe26-kserve-7fd8b8b8f-qkbvb" Oct 31 05:00:14.028198 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:14.028179 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35fb8fb1-3a58-465b-b53a-08fe806af299-model-cache\") pod \"test-model-c1fe26-kserve-7fd8b8b8f-qkbvb\" (UID: \"35fb8fb1-3a58-465b-b53a-08fe806af299\") " pod="ai-tenant-e2e-inf-b-c1fe26/test-model-c1fe26-kserve-7fd8b8b8f-qkbvb" Oct 31 05:00:14.028426 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:14.028406 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35fb8fb1-3a58-465b-b53a-08fe806af299-tmp-dir\") pod \"test-model-c1fe26-kserve-7fd8b8b8f-qkbvb\" (UID: \"35fb8fb1-3a58-465b-b53a-08fe806af299\") " pod="ai-tenant-e2e-inf-b-c1fe26/test-model-c1fe26-kserve-7fd8b8b8f-qkbvb" Oct 31 05:00:14.028580 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:14.028563 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35fb8fb1-3a58-465b-b53a-08fe806af299-home\") pod \"test-model-c1fe26-kserve-7fd8b8b8f-qkbvb\" (UID: \"35fb8fb1-3a58-465b-b53a-08fe806af299\") " pod="ai-tenant-e2e-inf-b-c1fe26/test-model-c1fe26-kserve-7fd8b8b8f-qkbvb" Oct 31 05:00:14.034145 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:14.034118 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c1fe26\"/\"kube-root-ca.crt\"" Oct 31 05:00:14.045402 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:14.038536 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c1fe26\"/\"default-dockercfg-hpbln\"" Oct 31 05:00:14.046968 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:14.046933 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c1fe26\"/\"openshift-service-ca.crt\"" Oct 31 05:00:14.046968 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:14.046943 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c1fe26\"/\"test-model-c1fe26-kserve-self-signed-certs\"" Oct 31 05:00:14.049000 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:14.048965 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c1fe26/test-model-c1fe26-kserve-7fd8b8b8f-qkbvb"] Oct 31 05:00:14.130535 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:14.130500 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35fb8fb1-3a58-465b-b53a-08fe806af299-home\") pod \"test-model-c1fe26-kserve-7fd8b8b8f-qkbvb\" (UID: \"35fb8fb1-3a58-465b-b53a-08fe806af299\") " pod="ai-tenant-e2e-inf-b-c1fe26/test-model-c1fe26-kserve-7fd8b8b8f-qkbvb" Oct 31 05:00:14.130734 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:14.130620 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35fb8fb1-3a58-465b-b53a-08fe806af299-tls-certs\") pod \"test-model-c1fe26-kserve-7fd8b8b8f-qkbvb\" (UID: \"35fb8fb1-3a58-465b-b53a-08fe806af299\") " pod="ai-tenant-e2e-inf-b-c1fe26/test-model-c1fe26-kserve-7fd8b8b8f-qkbvb" Oct 31 05:00:14.130734 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:14.130694 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35fb8fb1-3a58-465b-b53a-08fe806af299-dshm\") pod \"test-model-c1fe26-kserve-7fd8b8b8f-qkbvb\" (UID: \"35fb8fb1-3a58-465b-b53a-08fe806af299\") " pod="ai-tenant-e2e-inf-b-c1fe26/test-model-c1fe26-kserve-7fd8b8b8f-qkbvb" Oct 31 05:00:14.130851 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:14.130810 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8g6t\" (UniqueName: \"kubernetes.io/projected/35fb8fb1-3a58-465b-b53a-08fe806af299-kube-api-access-l8g6t\") pod \"test-model-c1fe26-kserve-7fd8b8b8f-qkbvb\" (UID: \"35fb8fb1-3a58-465b-b53a-08fe806af299\") " pod="ai-tenant-e2e-inf-b-c1fe26/test-model-c1fe26-kserve-7fd8b8b8f-qkbvb" Oct 31 05:00:14.130851 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:14.130840 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35fb8fb1-3a58-465b-b53a-08fe806af299-model-cache\") pod \"test-model-c1fe26-kserve-7fd8b8b8f-qkbvb\" (UID: \"35fb8fb1-3a58-465b-b53a-08fe806af299\") " pod="ai-tenant-e2e-inf-b-c1fe26/test-model-c1fe26-kserve-7fd8b8b8f-qkbvb" Oct 31 05:00:14.130954 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:14.130902 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35fb8fb1-3a58-465b-b53a-08fe806af299-tmp-dir\") pod \"test-model-c1fe26-kserve-7fd8b8b8f-qkbvb\" (UID: \"35fb8fb1-3a58-465b-b53a-08fe806af299\") " pod="ai-tenant-e2e-inf-b-c1fe26/test-model-c1fe26-kserve-7fd8b8b8f-qkbvb" Oct 31 05:00:14.131018 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:14.130978 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35fb8fb1-3a58-465b-b53a-08fe806af299-home\") pod \"test-model-c1fe26-kserve-7fd8b8b8f-qkbvb\" (UID: \"35fb8fb1-3a58-465b-b53a-08fe806af299\") " pod="ai-tenant-e2e-inf-b-c1fe26/test-model-c1fe26-kserve-7fd8b8b8f-qkbvb" Oct 31 05:00:14.131243 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:14.131224 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35fb8fb1-3a58-465b-b53a-08fe806af299-model-cache\") pod \"test-model-c1fe26-kserve-7fd8b8b8f-qkbvb\" (UID: \"35fb8fb1-3a58-465b-b53a-08fe806af299\") " pod="ai-tenant-e2e-inf-b-c1fe26/test-model-c1fe26-kserve-7fd8b8b8f-qkbvb" Oct 31 05:00:14.131349 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:14.131287 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35fb8fb1-3a58-465b-b53a-08fe806af299-tmp-dir\") pod \"test-model-c1fe26-kserve-7fd8b8b8f-qkbvb\" (UID: \"35fb8fb1-3a58-465b-b53a-08fe806af299\") " pod="ai-tenant-e2e-inf-b-c1fe26/test-model-c1fe26-kserve-7fd8b8b8f-qkbvb" Oct 31 05:00:14.133103 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:14.133082 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35fb8fb1-3a58-465b-b53a-08fe806af299-dshm\") pod \"test-model-c1fe26-kserve-7fd8b8b8f-qkbvb\" (UID: \"35fb8fb1-3a58-465b-b53a-08fe806af299\") " pod="ai-tenant-e2e-inf-b-c1fe26/test-model-c1fe26-kserve-7fd8b8b8f-qkbvb" Oct 31 05:00:14.133520 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:14.133501 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35fb8fb1-3a58-465b-b53a-08fe806af299-tls-certs\") pod \"test-model-c1fe26-kserve-7fd8b8b8f-qkbvb\" (UID: \"35fb8fb1-3a58-465b-b53a-08fe806af299\") " pod="ai-tenant-e2e-inf-b-c1fe26/test-model-c1fe26-kserve-7fd8b8b8f-qkbvb" Oct 31 05:00:14.154144 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:14.154114 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8g6t\" (UniqueName: \"kubernetes.io/projected/35fb8fb1-3a58-465b-b53a-08fe806af299-kube-api-access-l8g6t\") pod \"test-model-c1fe26-kserve-7fd8b8b8f-qkbvb\" (UID: \"35fb8fb1-3a58-465b-b53a-08fe806af299\") " pod="ai-tenant-e2e-inf-b-c1fe26/test-model-c1fe26-kserve-7fd8b8b8f-qkbvb" Oct 31 05:00:14.381405 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:14.370398 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c1fe26/test-model-c1fe26-kserve-7fd8b8b8f-qkbvb" Oct 31 05:00:14.732478 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:00:14.732442 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35fb8fb1_3a58_465b_b53a_08fe806af299.slice/crio-de95869a7cacc2a47d8c1132c34eccca87814839550e486835b9ab597761dcab WatchSource:0}: Error finding container de95869a7cacc2a47d8c1132c34eccca87814839550e486835b9ab597761dcab: Status 404 returned error can't find the container with id de95869a7cacc2a47d8c1132c34eccca87814839550e486835b9ab597761dcab Oct 31 05:00:14.732478 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:14.732457 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c1fe26/test-model-c1fe26-kserve-7fd8b8b8f-qkbvb"] Oct 31 05:00:15.031941 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:15.031834 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff105070-qmjw7"] Oct 31 05:00:15.040394 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:15.040360 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff105070-qmjw7"] Oct 31 05:00:15.075168 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:15.075128 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c1fe26/test-model-c1fe26-kserve-7fd8b8b8f-qkbvb" event={"ID":"35fb8fb1-3a58-465b-b53a-08fe806af299","Type":"ContainerStarted","Data":"0284ca77e086b05c3227ff3e59342c13dd43454270a68826bbf8c610ad09ac50"} Oct 31 05:00:15.075411 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:15.075179 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c1fe26/test-model-c1fe26-kserve-7fd8b8b8f-qkbvb" event={"ID":"35fb8fb1-3a58-465b-b53a-08fe806af299","Type":"ContainerStarted","Data":"de95869a7cacc2a47d8c1132c34eccca87814839550e486835b9ab597761dcab"} Oct 31 05:00:15.122289 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:15.122229 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-c1fe26/test-model-c1fe26-kserve-7fd8b8b8f-qkbvb" podStartSLOduration=2.122212854 podStartE2EDuration="2.122212854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:00:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:00:15.115737137 +0000 UTC m=+3369.872075177" watchObservedRunningTime="2026-10-31 05:00:15.122212854 +0000 UTC m=+3369.878550893" Oct 31 05:00:15.371643 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:15.371521 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-c1fe26/test-model-c1fe26-kserve-7fd8b8b8f-qkbvb" Oct 31 05:00:15.417705 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:15.417668 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5091375-a090-497a-9fc5-50c0a0a323af" path="/var/lib/kubelet/pods/d5091375-a090-497a-9fc5-50c0a0a323af/volumes" Oct 31 05:00:17.400178 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:17.400133 5019 scope.go:122] "RemoveContainer" containerID="fa1c952e213fc2db874f7214d12dd9757aed0c6fc1e318427ea08af5651005c4" Oct 31 05:00:17.400826 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:17.400794 5019 scope.go:122] "RemoveContainer" containerID="67cd3cea94dc92f993f190f26c65f6025368fc4b53f21e939d8fd988687f839a" Oct 31 05:00:17.400999 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:17.400978 5019 scope.go:122] "RemoveContainer" containerID="0ae3171d94a4e9fc4512711515cb6300d725282ddfb9097757734314266dfcb2" Oct 31 05:00:18.147604 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:18.147566 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890380-6qwg6" event={"ID":"e9b19f33-a012-48c9-b670-3b8eff773e49","Type":"ContainerStarted","Data":"e5b5d33d8a2fa4f0a91e145b42e88f748d49144ddad1c97f9094c5b6a4dbd0eb"} Oct 31 05:00:18.150155 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:18.150122 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890380-v7gl7" event={"ID":"f59877aa-80e8-418d-9617-2f9bb9c8f3ef","Type":"ContainerStarted","Data":"7f22f83d9d7551c9cdc5ffd73f24b5af2c3e1baf30c8301f001e5d9409896479"} Oct 31 05:00:18.192713 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:18.192650 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890380-6qwg6" podStartSLOduration=18.192636269 podStartE2EDuration="18.192636269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:00:18.18708439 +0000 UTC m=+3372.943422447" watchObservedRunningTime="2026-10-31 05:00:18.192636269 +0000 UTC m=+3372.948974308" Oct 31 05:00:18.400500 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:18.400400 5019 scope.go:122] "RemoveContainer" containerID="307dc9b0647618d9e86a3118ba70766b75e9974d27930eae830018e534d93edd" Oct 31 05:00:18.400964 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:18.400530 5019 scope.go:122] "RemoveContainer" containerID="b249f4ff0351d3704e4d3f0a0ab4e5428603ad8f889f3e0546a3773ac369dc9d" Oct 31 05:00:19.162688 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:19.162651 5019 generic.go:356] "Generic (PLEG): container finished" podID="f59877aa-80e8-418d-9617-2f9bb9c8f3ef" containerID="7f22f83d9d7551c9cdc5ffd73f24b5af2c3e1baf30c8301f001e5d9409896479" exitCode=60 Oct 31 05:00:19.162911 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:19.162720 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890380-v7gl7" event={"ID":"f59877aa-80e8-418d-9617-2f9bb9c8f3ef","Type":"ContainerDied","Data":"7f22f83d9d7551c9cdc5ffd73f24b5af2c3e1baf30c8301f001e5d9409896479"} Oct 31 05:00:19.162911 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:19.162793 5019 scope.go:122] "RemoveContainer" containerID="67cd3cea94dc92f993f190f26c65f6025368fc4b53f21e939d8fd988687f839a" Oct 31 05:00:19.163322 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:19.163296 5019 scope.go:122] "RemoveContainer" containerID="7f22f83d9d7551c9cdc5ffd73f24b5af2c3e1baf30c8301f001e5d9409896479" Oct 31 05:00:19.163322 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:19.163848 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890380-v7gl7_odh-ai-gateway-infra(f59877aa-80e8-418d-9617-2f9bb9c8f3ef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890380-v7gl7" podUID="f59877aa-80e8-418d-9617-2f9bb9c8f3ef" Oct 31 05:00:19.166596 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:19.166570 5019 generic.go:356] "Generic (PLEG): container finished" podID="e9b19f33-a012-48c9-b670-3b8eff773e49" containerID="e5b5d33d8a2fa4f0a91e145b42e88f748d49144ddad1c97f9094c5b6a4dbd0eb" exitCode=60 Oct 31 05:00:19.166709 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:19.166645 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890380-6qwg6" event={"ID":"e9b19f33-a012-48c9-b670-3b8eff773e49","Type":"ContainerDied","Data":"e5b5d33d8a2fa4f0a91e145b42e88f748d49144ddad1c97f9094c5b6a4dbd0eb"} Oct 31 05:00:19.167121 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:19.167102 5019 scope.go:122] "RemoveContainer" containerID="e5b5d33d8a2fa4f0a91e145b42e88f748d49144ddad1c97f9094c5b6a4dbd0eb" Oct 31 05:00:19.167518 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:19.167495 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890380-6qwg6_odh-ai-gateway-infra(e9b19f33-a012-48c9-b670-3b8eff773e49)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890380-6qwg6" podUID="e9b19f33-a012-48c9-b670-3b8eff773e49" Oct 31 05:00:19.169228 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:19.169202 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890380-ldsqw" event={"ID":"e1e9a647-0e9e-4528-a5a6-7022b244f1ba","Type":"ContainerStarted","Data":"1ce3724951a5ef4bf3802f42217f1a2eb300859ac66b9cfa53f76a3419ce8ae7"} Oct 31 05:00:19.173277 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:19.173223 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fb346b-29890380-l8fpr" event={"ID":"829dc86b-4ce1-427e-86db-cbaab8f15198","Type":"ContainerStarted","Data":"7dbf80bcafdda9bd900d444771f828bc700e86e1890652d938506098c1ca90d9"} Oct 31 05:00:19.176260 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:19.176237 5019 generic.go:356] "Generic (PLEG): container finished" podID="4fd6b126-7a7c-4825-a374-5dec6fb94d06" containerID="bfc07edfa586ce19e4ec2530428ba729d622cc164bc1fd7d037e5b249bfecf2a" exitCode=60 Oct 31 05:00:19.176415 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:19.176395 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-401761-29890380-zvx8f" event={"ID":"4fd6b126-7a7c-4825-a374-5dec6fb94d06","Type":"ContainerDied","Data":"bfc07edfa586ce19e4ec2530428ba729d622cc164bc1fd7d037e5b249bfecf2a"} Oct 31 05:00:19.176741 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:19.176723 5019 scope.go:122] "RemoveContainer" containerID="bfc07edfa586ce19e4ec2530428ba729d622cc164bc1fd7d037e5b249bfecf2a" Oct 31 05:00:19.177027 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:19.177002 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-401761-29890380-zvx8f_odh-ai-gateway-infra(4fd6b126-7a7c-4825-a374-5dec6fb94d06)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-401761-29890380-zvx8f" podUID="4fd6b126-7a7c-4825-a374-5dec6fb94d06" Oct 31 05:00:19.189094 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:19.189071 5019 scope.go:122] "RemoveContainer" containerID="fa1c952e213fc2db874f7214d12dd9757aed0c6fc1e318427ea08af5651005c4" Oct 31 05:00:19.209341 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:19.206911 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890380-6qwg6"] Oct 31 05:00:19.214408 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:19.214381 5019 scope.go:122] "RemoveContainer" containerID="0ae3171d94a4e9fc4512711515cb6300d725282ddfb9097757734314266dfcb2" Oct 31 05:00:19.249459 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:19.249422 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890380-v7gl7"] Oct 31 05:00:19.416512 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:19.416482 5019 scope.go:122] "RemoveContainer" containerID="4341f8d50a9838ae7ff595641b379c4b037a565f6b3e5bb5e483ee638ccd3d3d" Oct 31 05:00:19.417001 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:19.416829 5019 scope.go:122] "RemoveContainer" containerID="4815a20852906bfd87655f3881d5c0eda440c1eb65c7a7435a2e2d0cc2af711a" Oct 31 05:00:19.417001 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:19.416879 5019 scope.go:122] "RemoveContainer" containerID="ccda27e6a77112b7770bb93f1c01871a0a0327f5eae723af09f6d86bad4cffee" Oct 31 05:00:19.417140 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:19.417067 5019 scope.go:122] "RemoveContainer" containerID="b4376c22d8410400f9b4b22621b14be20195284a8a55e77f02351f9ca8e3af04" Oct 31 05:00:19.417867 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:19.417251 5019 scope.go:122] "RemoveContainer" containerID="75453d9bd304c57b9a965b1089fdbb266c733384e0286e68e493f5ac791b89aa" Oct 31 05:00:20.195924 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:20.195893 5019 generic.go:356] "Generic (PLEG): container finished" podID="e1e9a647-0e9e-4528-a5a6-7022b244f1ba" containerID="1ce3724951a5ef4bf3802f42217f1a2eb300859ac66b9cfa53f76a3419ce8ae7" exitCode=60 Oct 31 05:00:20.196042 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:20.196003 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890380-ldsqw" event={"ID":"e1e9a647-0e9e-4528-a5a6-7022b244f1ba","Type":"ContainerDied","Data":"1ce3724951a5ef4bf3802f42217f1a2eb300859ac66b9cfa53f76a3419ce8ae7"} Oct 31 05:00:20.196120 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:20.196048 5019 scope.go:122] "RemoveContainer" containerID="b249f4ff0351d3704e4d3f0a0ab4e5428603ad8f889f3e0546a3773ac369dc9d" Oct 31 05:00:20.196543 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:20.196524 5019 scope.go:122] "RemoveContainer" containerID="1ce3724951a5ef4bf3802f42217f1a2eb300859ac66b9cfa53f76a3419ce8ae7" Oct 31 05:00:20.196995 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:20.196971 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-8973e2-29890380-ldsqw_odh-ai-gateway-infra(e1e9a647-0e9e-4528-a5a6-7022b244f1ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890380-ldsqw" podUID="e1e9a647-0e9e-4528-a5a6-7022b244f1ba" Oct 31 05:00:20.200713 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:20.200681 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890380-fh8kl" event={"ID":"d0dacaaf-e177-420b-a953-f8cdb4961a27","Type":"ContainerStarted","Data":"5b50df8436b5070148821f554633c5bda1bbf53638aa6bc8607b5689691dc49b"} Oct 31 05:00:20.203456 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:20.203431 5019 generic.go:356] "Generic (PLEG): container finished" podID="829dc86b-4ce1-427e-86db-cbaab8f15198" containerID="7dbf80bcafdda9bd900d444771f828bc700e86e1890652d938506098c1ca90d9" exitCode=60 Oct 31 05:00:20.203597 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:20.203493 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fb346b-29890380-l8fpr" event={"ID":"829dc86b-4ce1-427e-86db-cbaab8f15198","Type":"ContainerDied","Data":"7dbf80bcafdda9bd900d444771f828bc700e86e1890652d938506098c1ca90d9"} Oct 31 05:00:20.203995 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:20.203971 5019 scope.go:122] "RemoveContainer" containerID="7dbf80bcafdda9bd900d444771f828bc700e86e1890652d938506098c1ca90d9" Oct 31 05:00:20.204508 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:20.204480 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-fb346b-29890380-l8fpr_odh-ai-gateway-infra(829dc86b-4ce1-427e-86db-cbaab8f15198)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fb346b-29890380-l8fpr" podUID="829dc86b-4ce1-427e-86db-cbaab8f15198" Oct 31 05:00:20.219503 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:20.219478 5019 scope.go:122] "RemoveContainer" containerID="307dc9b0647618d9e86a3118ba70766b75e9974d27930eae830018e534d93edd" Oct 31 05:00:20.298039 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:20.298008 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-401761-29890380-zvx8f"] Oct 31 05:00:20.366365 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:20.363090 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890380-ldsqw"] Oct 31 05:00:20.402629 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:20.402593 5019 scope.go:122] "RemoveContainer" containerID="4b2241ffbbf74bceaaa581e0c48f1143ec969f2a7343bd8f08f814ed3ea2b57f" Oct 31 05:00:20.403900 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:20.403297 5019 scope.go:122] "RemoveContainer" containerID="7a47eb7c3a9c9aef47dbc58bb5b2801ca687ced5155cb619265909c644783e64" Oct 31 05:00:20.403900 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:20.403554 5019 scope.go:122] "RemoveContainer" containerID="ed02bf60ecb10a21a7f0e07c1abb134e22852b8304d083726ba9c1258db41d2e" Oct 31 05:00:20.420146 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:20.404418 5019 scope.go:122] "RemoveContainer" containerID="4b4377c5db90806371b2b551279f4bbc84522ccc6de2d0d0967bd6da2e216700" Oct 31 05:00:20.420146 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:20.404530 5019 scope.go:122] "RemoveContainer" containerID="42c9c9d6a5eb52a1726c02f5a06ad5de76f43254a6628953e9ff3f19367afe38" Oct 31 05:00:20.420146 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:20.405221 5019 scope.go:122] "RemoveContainer" containerID="93cfacf43741883f8f7341fea7111a8a28f11b3dad97d1dbe446e59c5b77c842" Oct 31 05:00:20.420146 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:20.405673 5019 scope.go:122] "RemoveContainer" containerID="73c4fe02e89785fe6202c4d2bdfe58135b720afe245e6f95dbc706d9ca2e4aa7" Oct 31 05:00:20.420146 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:20.406313 5019 scope.go:122] "RemoveContainer" containerID="3ce1abbdae6f4b5a0ac4e38ce85ec955c29d603a732f9dc363e5456ad51607b8" Oct 31 05:00:20.435454 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:20.435372 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fb346b-29890380-l8fpr"] Oct 31 05:00:20.587493 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:20.586594 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890380-6qwg6" Oct 31 05:00:20.701883 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:20.701825 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9b19f33-a012-48c9-b670-3b8eff773e49-kube-api-access-t8p2p\" (UniqueName: \"kubernetes.io/projected/e9b19f33-a012-48c9-b670-3b8eff773e49-kube-api-access-t8p2p\") pod \"e9b19f33-a012-48c9-b670-3b8eff773e49\" (UID: \"e9b19f33-a012-48c9-b670-3b8eff773e49\") " Oct 31 05:00:20.702268 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:20.702126 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9b19f33-a012-48c9-b670-3b8eff773e49-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9b19f33-a012-48c9-b670-3b8eff773e49-maas-api-service-ca\") pod \"e9b19f33-a012-48c9-b670-3b8eff773e49\" (UID: \"e9b19f33-a012-48c9-b670-3b8eff773e49\") " Oct 31 05:00:20.703725 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:20.703655 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9b19f33-a012-48c9-b670-3b8eff773e49-maas-api-service-ca" pod "e9b19f33-a012-48c9-b670-3b8eff773e49" (UID: "e9b19f33-a012-48c9-b670-3b8eff773e49"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:00:20.704150 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:20.704126 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9b19f33-a012-48c9-b670-3b8eff773e49-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:00:20.707567 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:20.707071 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9b19f33-a012-48c9-b670-3b8eff773e49-kube-api-access-t8p2p" pod "e9b19f33-a012-48c9-b670-3b8eff773e49" (UID: "e9b19f33-a012-48c9-b670-3b8eff773e49"). InnerVolumeSpecName "kube-api-access-t8p2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:00:20.806111 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:20.806082 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8p2p\" (UniqueName: \"kubernetes.io/projected/e9b19f33-a012-48c9-b670-3b8eff773e49-kube-api-access-t8p2p\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:00:20.821933 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:20.821909 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890380-v7gl7" Oct 31 05:00:20.831685 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:20.831660 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-401761-29890380-zvx8f" Oct 31 05:00:20.908726 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:20.908625 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fd6b126-7a7c-4825-a374-5dec6fb94d06-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fd6b126-7a7c-4825-a374-5dec6fb94d06-maas-api-service-ca\") pod \"4fd6b126-7a7c-4825-a374-5dec6fb94d06\" (UID: \"4fd6b126-7a7c-4825-a374-5dec6fb94d06\") " Oct 31 05:00:20.908877 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:20.908819 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f59877aa-80e8-418d-9617-2f9bb9c8f3ef-kube-api-access-hmczm\" (UniqueName: \"kubernetes.io/projected/f59877aa-80e8-418d-9617-2f9bb9c8f3ef-kube-api-access-hmczm\") pod \"f59877aa-80e8-418d-9617-2f9bb9c8f3ef\" (UID: \"f59877aa-80e8-418d-9617-2f9bb9c8f3ef\") " Oct 31 05:00:20.909531 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:20.908955 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f59877aa-80e8-418d-9617-2f9bb9c8f3ef-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f59877aa-80e8-418d-9617-2f9bb9c8f3ef-maas-api-service-ca\") pod \"f59877aa-80e8-418d-9617-2f9bb9c8f3ef\" (UID: \"f59877aa-80e8-418d-9617-2f9bb9c8f3ef\") " Oct 31 05:00:20.909531 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:20.909220 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fd6b126-7a7c-4825-a374-5dec6fb94d06-kube-api-access-5k6h9\" (UniqueName: \"kubernetes.io/projected/4fd6b126-7a7c-4825-a374-5dec6fb94d06-kube-api-access-5k6h9\") pod \"4fd6b126-7a7c-4825-a374-5dec6fb94d06\" (UID: \"4fd6b126-7a7c-4825-a374-5dec6fb94d06\") " Oct 31 05:00:20.912074 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:20.911409 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f59877aa-80e8-418d-9617-2f9bb9c8f3ef-maas-api-service-ca" pod "f59877aa-80e8-418d-9617-2f9bb9c8f3ef" (UID: "f59877aa-80e8-418d-9617-2f9bb9c8f3ef"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:00:20.912074 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:20.911873 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fd6b126-7a7c-4825-a374-5dec6fb94d06-maas-api-service-ca" pod "4fd6b126-7a7c-4825-a374-5dec6fb94d06" (UID: "4fd6b126-7a7c-4825-a374-5dec6fb94d06"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:00:20.984702 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:20.984660 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fd6b126-7a7c-4825-a374-5dec6fb94d06-kube-api-access-5k6h9" pod "4fd6b126-7a7c-4825-a374-5dec6fb94d06" (UID: "4fd6b126-7a7c-4825-a374-5dec6fb94d06"). InnerVolumeSpecName "kube-api-access-5k6h9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:00:20.984828 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:20.984775 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f59877aa-80e8-418d-9617-2f9bb9c8f3ef-kube-api-access-hmczm" pod "f59877aa-80e8-418d-9617-2f9bb9c8f3ef" (UID: "f59877aa-80e8-418d-9617-2f9bb9c8f3ef"). InnerVolumeSpecName "kube-api-access-hmczm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:00:21.013095 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:21.013052 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5k6h9\" (UniqueName: \"kubernetes.io/projected/4fd6b126-7a7c-4825-a374-5dec6fb94d06-kube-api-access-5k6h9\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:00:21.013095 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:21.013092 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fd6b126-7a7c-4825-a374-5dec6fb94d06-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:00:21.013291 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:21.013102 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmczm\" (UniqueName: \"kubernetes.io/projected/f59877aa-80e8-418d-9617-2f9bb9c8f3ef-kube-api-access-hmczm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:00:21.013291 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:21.013112 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f59877aa-80e8-418d-9617-2f9bb9c8f3ef-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:00:21.236734 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:21.236697 5019 generic.go:356] "Generic (PLEG): container finished" podID="93bfb362-fcc2-4341-9881-1fd06b546590" containerID="45e3d3507691ebebd539f807a422993919a5d86db0cd53ec1e0eab2fc9cfe06f" exitCode=60 Oct 31 05:00:21.236938 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:21.236736 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890380-7g9d5" event={"ID":"93bfb362-fcc2-4341-9881-1fd06b546590","Type":"ContainerDied","Data":"45e3d3507691ebebd539f807a422993919a5d86db0cd53ec1e0eab2fc9cfe06f"} Oct 31 05:00:21.236938 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:21.236813 5019 scope.go:122] "RemoveContainer" containerID="b4376c22d8410400f9b4b22621b14be20195284a8a55e77f02351f9ca8e3af04" Oct 31 05:00:21.237380 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:21.237362 5019 scope.go:122] "RemoveContainer" containerID="45e3d3507691ebebd539f807a422993919a5d86db0cd53ec1e0eab2fc9cfe06f" Oct 31 05:00:21.237903 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:21.237864 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-227c93-29890380-7g9d5_odh-ai-gateway-infra(93bfb362-fcc2-4341-9881-1fd06b546590)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890380-7g9d5" podUID="93bfb362-fcc2-4341-9881-1fd06b546590" Oct 31 05:00:21.239507 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:21.239471 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890380-6qwg6" event={"ID":"e9b19f33-a012-48c9-b670-3b8eff773e49","Type":"ContainerDied","Data":"cc615af6760a6d4e9279c8e41ca11559540a1580e467045e52f5d0f5bb0d8ad7"} Oct 31 05:00:21.239634 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:21.239585 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890380-6qwg6" Oct 31 05:00:21.242730 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:21.242706 5019 generic.go:356] "Generic (PLEG): container finished" podID="c9515dc7-4ea3-4a1b-aad8-1e78c7d0e13c" containerID="5942c0d16ab14ed6dc80ac228b23fae45023dda719b99b028106b375bd575c9f" exitCode=60 Oct 31 05:00:21.242825 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:21.242781 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-670795-29890380-hzmsc" event={"ID":"c9515dc7-4ea3-4a1b-aad8-1e78c7d0e13c","Type":"ContainerDied","Data":"5942c0d16ab14ed6dc80ac228b23fae45023dda719b99b028106b375bd575c9f"} Oct 31 05:00:21.243355 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:21.243320 5019 scope.go:122] "RemoveContainer" containerID="5942c0d16ab14ed6dc80ac228b23fae45023dda719b99b028106b375bd575c9f" Oct 31 05:00:21.243749 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:21.243727 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-670795-29890380-hzmsc_odh-ai-gateway-infra(c9515dc7-4ea3-4a1b-aad8-1e78c7d0e13c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-670795-29890380-hzmsc" podUID="c9515dc7-4ea3-4a1b-aad8-1e78c7d0e13c" Oct 31 05:00:21.250350 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:21.250299 5019 generic.go:356] "Generic (PLEG): container finished" podID="d0dacaaf-e177-420b-a953-f8cdb4961a27" containerID="5b50df8436b5070148821f554633c5bda1bbf53638aa6bc8607b5689691dc49b" exitCode=60 Oct 31 05:00:21.250468 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:21.250434 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890380-fh8kl" event={"ID":"d0dacaaf-e177-420b-a953-f8cdb4961a27","Type":"ContainerDied","Data":"5b50df8436b5070148821f554633c5bda1bbf53638aa6bc8607b5689691dc49b"} Oct 31 05:00:21.251362 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:21.251323 5019 scope.go:122] "RemoveContainer" containerID="5b50df8436b5070148821f554633c5bda1bbf53638aa6bc8607b5689691dc49b" Oct 31 05:00:21.251865 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:21.251842 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-965f61-29890380-fh8kl_odh-ai-gateway-infra(d0dacaaf-e177-420b-a953-f8cdb4961a27)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890380-fh8kl" podUID="d0dacaaf-e177-420b-a953-f8cdb4961a27" Oct 31 05:00:21.255118 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:21.255098 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-401761-29890380-zvx8f" Oct 31 05:00:21.255229 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:21.255138 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-401761-29890380-zvx8f" event={"ID":"4fd6b126-7a7c-4825-a374-5dec6fb94d06","Type":"ContainerDied","Data":"719e130ba10827ce5bc82b3e0007523bdd820203947486455e82b0422b6355e6"} Oct 31 05:00:21.259025 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:21.258936 5019 scope.go:122] "RemoveContainer" containerID="e5b5d33d8a2fa4f0a91e145b42e88f748d49144ddad1c97f9094c5b6a4dbd0eb" Oct 31 05:00:21.263213 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:21.263156 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890380-b7jtg" event={"ID":"e27f22dd-e82f-493c-9745-6d901308b907","Type":"ContainerStarted","Data":"941ad460acc314b48df4126bf8744d1717bde3636b6ace08af19190679ef51ec"} Oct 31 05:00:21.267640 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:21.267614 5019 generic.go:356] "Generic (PLEG): container finished" podID="9790f69f-4de3-4837-a29f-17866661a3b3" containerID="9a695d1ea0d6377263e86811802ddf8be2e1b3b456e438751257b0167e071c93" exitCode=60 Oct 31 05:00:21.267766 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:21.267699 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890380-tx4lq" event={"ID":"9790f69f-4de3-4837-a29f-17866661a3b3","Type":"ContainerDied","Data":"9a695d1ea0d6377263e86811802ddf8be2e1b3b456e438751257b0167e071c93"} Oct 31 05:00:21.268844 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:21.268285 5019 scope.go:122] "RemoveContainer" containerID="9a695d1ea0d6377263e86811802ddf8be2e1b3b456e438751257b0167e071c93" Oct 31 05:00:21.268844 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:21.268795 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c4006d-29890380-tx4lq_odh-ai-gateway-infra(9790f69f-4de3-4837-a29f-17866661a3b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890380-tx4lq" podUID="9790f69f-4de3-4837-a29f-17866661a3b3" Oct 31 05:00:21.270087 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:21.270066 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890380-v7gl7" event={"ID":"f59877aa-80e8-418d-9617-2f9bb9c8f3ef","Type":"ContainerDied","Data":"ecba0638ca26c057aa747aa2c1d54d6a8e5bd63e738e761739bf05313b6de8f2"} Oct 31 05:00:21.270263 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:21.270238 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890380-v7gl7" Oct 31 05:00:21.272973 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:21.272950 5019 generic.go:356] "Generic (PLEG): container finished" podID="d13131fa-5137-4401-aef8-d53fd8bd4e5d" containerID="3c9e867331d221180d4ced9ab5f6ce4c37b45d1553799585d0d38b0838e49118" exitCode=60 Oct 31 05:00:21.273074 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:21.272991 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-47c409-29890380-cqm6g" event={"ID":"d13131fa-5137-4401-aef8-d53fd8bd4e5d","Type":"ContainerDied","Data":"3c9e867331d221180d4ced9ab5f6ce4c37b45d1553799585d0d38b0838e49118"} Oct 31 05:00:21.273457 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:21.273439 5019 scope.go:122] "RemoveContainer" containerID="3c9e867331d221180d4ced9ab5f6ce4c37b45d1553799585d0d38b0838e49118" Oct 31 05:00:21.273822 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:21.273797 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-47c409-29890380-cqm6g_odh-ai-gateway-infra(d13131fa-5137-4401-aef8-d53fd8bd4e5d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-47c409-29890380-cqm6g" podUID="d13131fa-5137-4401-aef8-d53fd8bd4e5d" Oct 31 05:00:21.274909 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:21.274889 5019 scope.go:122] "RemoveContainer" containerID="4341f8d50a9838ae7ff595641b379c4b037a565f6b3e5bb5e483ee638ccd3d3d" Oct 31 05:00:21.301555 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:21.301524 5019 scope.go:122] "RemoveContainer" containerID="75453d9bd304c57b9a965b1089fdbb266c733384e0286e68e493f5ac791b89aa" Oct 31 05:00:21.331812 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:21.331742 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890380-b7jtg" podStartSLOduration=21.331718607 podStartE2EDuration="21.331718607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:00:21.322612687 +0000 UTC m=+3376.078950730" watchObservedRunningTime="2026-10-31 05:00:21.331718607 +0000 UTC m=+3376.088056648" Oct 31 05:00:21.338774 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:21.338745 5019 scope.go:122] "RemoveContainer" containerID="bfc07edfa586ce19e4ec2530428ba729d622cc164bc1fd7d037e5b249bfecf2a" Oct 31 05:00:21.366573 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:21.366544 5019 scope.go:122] "RemoveContainer" containerID="ccda27e6a77112b7770bb93f1c01871a0a0327f5eae723af09f6d86bad4cffee" Oct 31 05:00:21.408165 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:21.407452 5019 scope.go:122] "RemoveContainer" containerID="b1dd400398607ff40e67dca107dd07ed4a12b5dce7fb7eb481453dcce873925d" Oct 31 05:00:21.410583 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:21.409048 5019 scope.go:122] "RemoveContainer" containerID="1b602417b78f0fb3ff0a24019758993c48d43aa12e36e7521950ee2edaad5b01" Oct 31 05:00:21.424021 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:21.411178 5019 scope.go:122] "RemoveContainer" containerID="df8d8adef2a9b7d4d7bf0ae9182e41601e535a6890077ca74cfe319a8821a03b" Oct 31 05:00:21.459886 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:21.459859 5019 scope.go:122] "RemoveContainer" containerID="7f22f83d9d7551c9cdc5ffd73f24b5af2c3e1baf30c8301f001e5d9409896479" Oct 31 05:00:21.467993 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:21.467901 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890380-fh8kl"] Oct 31 05:00:21.499274 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:21.499242 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-401761-29890380-zvx8f"] Oct 31 05:00:21.523298 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:21.523043 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-401761-29890380-zvx8f"] Oct 31 05:00:21.602268 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:21.602188 5019 scope.go:122] "RemoveContainer" containerID="4815a20852906bfd87655f3881d5c0eda440c1eb65c7a7435a2e2d0cc2af711a" Oct 31 05:00:21.712360 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:21.712194 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890380-6qwg6"] Oct 31 05:00:21.718467 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:21.716656 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fb346b-29890380-l8fpr" Oct 31 05:00:21.729160 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:21.729122 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a99f2a-29890380-6qwg6"] Oct 31 05:00:21.832380 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:21.831454 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/829dc86b-4ce1-427e-86db-cbaab8f15198-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/829dc86b-4ce1-427e-86db-cbaab8f15198-maas-api-service-ca\") pod \"829dc86b-4ce1-427e-86db-cbaab8f15198\" (UID: \"829dc86b-4ce1-427e-86db-cbaab8f15198\") " Oct 31 05:00:21.832380 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:21.831728 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/829dc86b-4ce1-427e-86db-cbaab8f15198-kube-api-access-dwz2r\" (UniqueName: \"kubernetes.io/projected/829dc86b-4ce1-427e-86db-cbaab8f15198-kube-api-access-dwz2r\") pod \"829dc86b-4ce1-427e-86db-cbaab8f15198\" (UID: \"829dc86b-4ce1-427e-86db-cbaab8f15198\") " Oct 31 05:00:21.834985 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:21.834954 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890380-v7gl7"] Oct 31 05:00:21.835229 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:21.835208 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/829dc86b-4ce1-427e-86db-cbaab8f15198-maas-api-service-ca" pod "829dc86b-4ce1-427e-86db-cbaab8f15198" (UID: "829dc86b-4ce1-427e-86db-cbaab8f15198"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:00:21.839921 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:21.839887 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890380-v7gl7"] Oct 31 05:00:21.845411 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:21.845358 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890380-ldsqw" Oct 31 05:00:21.881358 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:21.878980 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/829dc86b-4ce1-427e-86db-cbaab8f15198-kube-api-access-dwz2r" pod "829dc86b-4ce1-427e-86db-cbaab8f15198" (UID: "829dc86b-4ce1-427e-86db-cbaab8f15198"). InnerVolumeSpecName "kube-api-access-dwz2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:00:21.933306 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:21.933272 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1e9a647-0e9e-4528-a5a6-7022b244f1ba-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1e9a647-0e9e-4528-a5a6-7022b244f1ba-maas-api-service-ca\") pod \"e1e9a647-0e9e-4528-a5a6-7022b244f1ba\" (UID: \"e1e9a647-0e9e-4528-a5a6-7022b244f1ba\") " Oct 31 05:00:21.933762 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:21.933739 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1e9a647-0e9e-4528-a5a6-7022b244f1ba-kube-api-access-59sfq\" (UniqueName: \"kubernetes.io/projected/e1e9a647-0e9e-4528-a5a6-7022b244f1ba-kube-api-access-59sfq\") pod \"e1e9a647-0e9e-4528-a5a6-7022b244f1ba\" (UID: \"e1e9a647-0e9e-4528-a5a6-7022b244f1ba\") " Oct 31 05:00:21.934371 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:21.934299 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1e9a647-0e9e-4528-a5a6-7022b244f1ba-maas-api-service-ca" pod "e1e9a647-0e9e-4528-a5a6-7022b244f1ba" (UID: "e1e9a647-0e9e-4528-a5a6-7022b244f1ba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:00:21.934467 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:21.934407 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/829dc86b-4ce1-427e-86db-cbaab8f15198-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:00:21.934467 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:21.934430 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwz2r\" (UniqueName: \"kubernetes.io/projected/829dc86b-4ce1-427e-86db-cbaab8f15198-kube-api-access-dwz2r\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:00:21.976209 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:21.975343 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1e9a647-0e9e-4528-a5a6-7022b244f1ba-kube-api-access-59sfq" pod "e1e9a647-0e9e-4528-a5a6-7022b244f1ba" (UID: "e1e9a647-0e9e-4528-a5a6-7022b244f1ba"). InnerVolumeSpecName "kube-api-access-59sfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:00:22.036677 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.036588 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59sfq\" (UniqueName: \"kubernetes.io/projected/e1e9a647-0e9e-4528-a5a6-7022b244f1ba-kube-api-access-59sfq\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:00:22.036677 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.036623 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1e9a647-0e9e-4528-a5a6-7022b244f1ba-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:00:22.298189 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.298159 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890380-ldsqw" Oct 31 05:00:22.298414 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.298152 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890380-ldsqw" event={"ID":"e1e9a647-0e9e-4528-a5a6-7022b244f1ba","Type":"ContainerDied","Data":"7f3897ec7afe082ea962d40153a5b916006f10fe62ed43c7d25943636d590e10"} Oct 31 05:00:22.298414 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.298392 5019 scope.go:122] "RemoveContainer" containerID="1ce3724951a5ef4bf3802f42217f1a2eb300859ac66b9cfa53f76a3419ce8ae7" Oct 31 05:00:22.299247 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.299219 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890380-7g9d5"] Oct 31 05:00:22.304785 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.304751 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fb346b-29890380-l8fpr" Oct 31 05:00:22.304785 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.304759 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fb346b-29890380-l8fpr" event={"ID":"829dc86b-4ce1-427e-86db-cbaab8f15198","Type":"ContainerDied","Data":"476f6c49f70b0d41fcdb86295fe365600cf8ec1c26702f8d0db601a340ebccfb"} Oct 31 05:00:22.309046 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.309016 5019 generic.go:356] "Generic (PLEG): container finished" podID="d3d4f5cd-c755-4e4f-a86f-14874e597d08" containerID="8adfa676587097a0c6c458a31f7ed1e98d35d5dc05718c41b615bf29e0cd29bb" exitCode=60 Oct 31 05:00:22.309183 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.309097 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-031745-29890380-9jpj5" event={"ID":"d3d4f5cd-c755-4e4f-a86f-14874e597d08","Type":"ContainerDied","Data":"8adfa676587097a0c6c458a31f7ed1e98d35d5dc05718c41b615bf29e0cd29bb"} Oct 31 05:00:22.309684 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.309661 5019 scope.go:122] "RemoveContainer" containerID="8adfa676587097a0c6c458a31f7ed1e98d35d5dc05718c41b615bf29e0cd29bb" Oct 31 05:00:22.310151 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:22.310121 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-031745-29890380-9jpj5_odh-ai-gateway-infra(d3d4f5cd-c755-4e4f-a86f-14874e597d08)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-031745-29890380-9jpj5" podUID="d3d4f5cd-c755-4e4f-a86f-14874e597d08" Oct 31 05:00:22.311634 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.311609 5019 generic.go:356] "Generic (PLEG): container finished" podID="96e0a71b-2eee-4760-ade1-5cdd604c4c81" containerID="6eaa5cb9139a1bbcdd323bee1fe7831ad6dd3e149737606df029236f27c1a274" exitCode=60 Oct 31 05:00:22.311731 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.311678 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890380-fw6dq" event={"ID":"96e0a71b-2eee-4760-ade1-5cdd604c4c81","Type":"ContainerDied","Data":"6eaa5cb9139a1bbcdd323bee1fe7831ad6dd3e149737606df029236f27c1a274"} Oct 31 05:00:22.312179 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.312159 5019 scope.go:122] "RemoveContainer" containerID="6eaa5cb9139a1bbcdd323bee1fe7831ad6dd3e149737606df029236f27c1a274" Oct 31 05:00:22.312602 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:22.312577 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-09a20a-29890380-fw6dq_odh-ai-gateway-infra(96e0a71b-2eee-4760-ade1-5cdd604c4c81)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890380-fw6dq" podUID="96e0a71b-2eee-4760-ade1-5cdd604c4c81" Oct 31 05:00:22.315406 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.315316 5019 generic.go:356] "Generic (PLEG): container finished" podID="e5259d89-590d-4fc7-b521-8f6feabbd2cc" containerID="3a80fa17fb3a6e024cdfdaacd056af64be3a2f075308f38a95f1d7c5b4c95370" exitCode=60 Oct 31 05:00:22.315517 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.315401 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-108ad8-29890380-ng62l" event={"ID":"e5259d89-590d-4fc7-b521-8f6feabbd2cc","Type":"ContainerDied","Data":"3a80fa17fb3a6e024cdfdaacd056af64be3a2f075308f38a95f1d7c5b4c95370"} Oct 31 05:00:22.315881 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.315862 5019 scope.go:122] "RemoveContainer" containerID="3a80fa17fb3a6e024cdfdaacd056af64be3a2f075308f38a95f1d7c5b4c95370" Oct 31 05:00:22.316210 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:22.316190 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-108ad8-29890380-ng62l_odh-ai-gateway-infra(e5259d89-590d-4fc7-b521-8f6feabbd2cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-108ad8-29890380-ng62l" podUID="e5259d89-590d-4fc7-b521-8f6feabbd2cc" Oct 31 05:00:22.318378 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.318312 5019 generic.go:356] "Generic (PLEG): container finished" podID="e27f22dd-e82f-493c-9745-6d901308b907" containerID="941ad460acc314b48df4126bf8744d1717bde3636b6ace08af19190679ef51ec" exitCode=60 Oct 31 05:00:22.318483 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.318422 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890380-b7jtg" event={"ID":"e27f22dd-e82f-493c-9745-6d901308b907","Type":"ContainerDied","Data":"941ad460acc314b48df4126bf8744d1717bde3636b6ace08af19190679ef51ec"} Oct 31 05:00:22.318836 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.318820 5019 scope.go:122] "RemoveContainer" containerID="941ad460acc314b48df4126bf8744d1717bde3636b6ace08af19190679ef51ec" Oct 31 05:00:22.319186 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:22.319165 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890380-b7jtg_odh-ai-gateway-infra(e27f22dd-e82f-493c-9745-6d901308b907)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890380-b7jtg" podUID="e27f22dd-e82f-493c-9745-6d901308b907" Oct 31 05:00:22.324339 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.324310 5019 generic.go:356] "Generic (PLEG): container finished" podID="982ad40e-a719-4f61-be85-ea665503c299" containerID="a209cfa1faa3f7f1c3b2181992def0d9b21e2c74035f70b9b239b7e75efd8e2c" exitCode=60 Oct 31 05:00:22.324456 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.324380 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890380-rmxrz" event={"ID":"982ad40e-a719-4f61-be85-ea665503c299","Type":"ContainerDied","Data":"a209cfa1faa3f7f1c3b2181992def0d9b21e2c74035f70b9b239b7e75efd8e2c"} Oct 31 05:00:22.324839 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.324811 5019 scope.go:122] "RemoveContainer" containerID="a209cfa1faa3f7f1c3b2181992def0d9b21e2c74035f70b9b239b7e75efd8e2c" Oct 31 05:00:22.325241 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:22.325213 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890380-rmxrz_odh-ai-gateway-infra(982ad40e-a719-4f61-be85-ea665503c299)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890380-rmxrz" podUID="982ad40e-a719-4f61-be85-ea665503c299" Oct 31 05:00:22.327814 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.327783 5019 generic.go:356] "Generic (PLEG): container finished" podID="e478c8f1-9e68-4f6f-b919-28ba397e66c4" containerID="6a165e20357201538611a06791aaf559a24898b41e7e06352438f831a5412462" exitCode=60 Oct 31 05:00:22.327944 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.327817 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890380-tkvr7" event={"ID":"e478c8f1-9e68-4f6f-b919-28ba397e66c4","Type":"ContainerDied","Data":"6a165e20357201538611a06791aaf559a24898b41e7e06352438f831a5412462"} Oct 31 05:00:22.328288 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.328269 5019 scope.go:122] "RemoveContainer" containerID="6a165e20357201538611a06791aaf559a24898b41e7e06352438f831a5412462" Oct 31 05:00:22.328727 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:22.328703 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-4c2387-29890380-tkvr7_odh-ai-gateway-infra(e478c8f1-9e68-4f6f-b919-28ba397e66c4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890380-tkvr7" podUID="e478c8f1-9e68-4f6f-b919-28ba397e66c4" Oct 31 05:00:22.333985 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.333943 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890380-b7jtg"] Oct 31 05:00:22.334485 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.334454 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1daf30-29890380-pbtq2" event={"ID":"cd0723e3-dcf6-4907-9ce1-99ca9c77b144","Type":"ContainerStarted","Data":"83aa2aeb549688d462bb067fe5ddd79f2976af9fb9849bf81526b5a3679a984e"} Oct 31 05:00:22.335133 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.335107 5019 scope.go:122] "RemoveContainer" containerID="83aa2aeb549688d462bb067fe5ddd79f2976af9fb9849bf81526b5a3679a984e" Oct 31 05:00:22.335630 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:22.335600 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-1daf30-29890380-pbtq2_odh-ai-gateway-infra(cd0723e3-dcf6-4907-9ce1-99ca9c77b144)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1daf30-29890380-pbtq2" podUID="cd0723e3-dcf6-4907-9ce1-99ca9c77b144" Oct 31 05:00:22.337314 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.337290 5019 generic.go:356] "Generic (PLEG): container finished" podID="0b6b5e89-5900-4db2-8552-e4f029de7573" containerID="4114cf636ee176574808e2e50eee7af545e46773cb85c03d06311d4bfdf823a4" exitCode=60 Oct 31 05:00:22.337434 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.337385 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a04e4e-29890380-jgchh" event={"ID":"0b6b5e89-5900-4db2-8552-e4f029de7573","Type":"ContainerDied","Data":"4114cf636ee176574808e2e50eee7af545e46773cb85c03d06311d4bfdf823a4"} Oct 31 05:00:22.338173 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.338046 5019 scope.go:122] "RemoveContainer" containerID="4114cf636ee176574808e2e50eee7af545e46773cb85c03d06311d4bfdf823a4" Oct 31 05:00:22.338690 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:22.338671 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-a04e4e-29890380-jgchh_odh-ai-gateway-infra(0b6b5e89-5900-4db2-8552-e4f029de7573)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a04e4e-29890380-jgchh" podUID="0b6b5e89-5900-4db2-8552-e4f029de7573" Oct 31 05:00:22.424934 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.421840 5019 scope.go:122] "RemoveContainer" containerID="bf7f393c48f1a8d8d00b56318c44ebe02be183b0f93246deb2ffe76e2e1ec8e1" Oct 31 05:00:22.424934 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.422653 5019 scope.go:122] "RemoveContainer" containerID="e6aa6c0788c5a4e2329732970dd459152f7207dcaa204cc9226ef9c93dfec3d2" Oct 31 05:00:22.424934 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.424038 5019 scope.go:122] "RemoveContainer" containerID="939033db9d1fbe818fc40ca51fefca9989c43a8489b97b983c36af647203a762" Oct 31 05:00:22.427555 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.425860 5019 scope.go:122] "RemoveContainer" containerID="7dbf80bcafdda9bd900d444771f828bc700e86e1890652d938506098c1ca90d9" Oct 31 05:00:22.462075 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:22.461694 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:00:22.567517 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.558185 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-47c409-29890380-cqm6g"] Oct 31 05:00:22.609742 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.609323 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890380-tx4lq"] Oct 31 05:00:22.634217 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.633864 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-670795-29890380-hzmsc"] Oct 31 05:00:22.660547 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.660508 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fb346b-29890380-l8fpr"] Oct 31 05:00:22.673011 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.667659 5019 scope.go:122] "RemoveContainer" containerID="4b2241ffbbf74bceaaa581e0c48f1143ec969f2a7343bd8f08f814ed3ea2b57f" Oct 31 05:00:22.687859 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.687816 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fb346b-29890380-l8fpr"] Oct 31 05:00:22.701504 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.697527 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890380-fh8kl" Oct 31 05:00:22.759135 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.759066 5019 scope.go:122] "RemoveContainer" containerID="7a47eb7c3a9c9aef47dbc58bb5b2801ca687ced5155cb619265909c644783e64" Oct 31 05:00:22.763292 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.763097 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0dacaaf-e177-420b-a953-f8cdb4961a27-kube-api-access-gvkkw\" (UniqueName: \"kubernetes.io/projected/d0dacaaf-e177-420b-a953-f8cdb4961a27-kube-api-access-gvkkw\") pod \"d0dacaaf-e177-420b-a953-f8cdb4961a27\" (UID: \"d0dacaaf-e177-420b-a953-f8cdb4961a27\") " Oct 31 05:00:22.763524 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.763376 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0dacaaf-e177-420b-a953-f8cdb4961a27-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0dacaaf-e177-420b-a953-f8cdb4961a27-maas-api-service-ca\") pod \"d0dacaaf-e177-420b-a953-f8cdb4961a27\" (UID: \"d0dacaaf-e177-420b-a953-f8cdb4961a27\") " Oct 31 05:00:22.764835 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.764809 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0dacaaf-e177-420b-a953-f8cdb4961a27-maas-api-service-ca" pod "d0dacaaf-e177-420b-a953-f8cdb4961a27" (UID: "d0dacaaf-e177-420b-a953-f8cdb4961a27"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:00:22.824382 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.823688 5019 scope.go:122] "RemoveContainer" containerID="ed02bf60ecb10a21a7f0e07c1abb134e22852b8304d083726ba9c1258db41d2e" Oct 31 05:00:22.866815 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.866768 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0dacaaf-e177-420b-a953-f8cdb4961a27-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:00:22.915955 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.915646 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0dacaaf-e177-420b-a953-f8cdb4961a27-kube-api-access-gvkkw" pod "d0dacaaf-e177-420b-a953-f8cdb4961a27" (UID: "d0dacaaf-e177-420b-a953-f8cdb4961a27"). InnerVolumeSpecName "kube-api-access-gvkkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:00:22.915955 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.915665 5019 scope.go:122] "RemoveContainer" containerID="3ce1abbdae6f4b5a0ac4e38ce85ec955c29d603a732f9dc363e5456ad51607b8" Oct 31 05:00:22.927297 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.925245 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890380-7g9d5" Oct 31 05:00:22.949899 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.949873 5019 scope.go:122] "RemoveContainer" containerID="73c4fe02e89785fe6202c4d2bdfe58135b720afe245e6f95dbc706d9ca2e4aa7" Oct 31 05:00:22.952736 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.952708 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-47c409-29890380-cqm6g" Oct 31 05:00:22.967770 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.967730 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93bfb362-fcc2-4341-9881-1fd06b546590-kube-api-access-6l2gk\" (UniqueName: \"kubernetes.io/projected/93bfb362-fcc2-4341-9881-1fd06b546590-kube-api-access-6l2gk\") pod \"93bfb362-fcc2-4341-9881-1fd06b546590\" (UID: \"93bfb362-fcc2-4341-9881-1fd06b546590\") " Oct 31 05:00:22.968216 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.968184 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93bfb362-fcc2-4341-9881-1fd06b546590-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93bfb362-fcc2-4341-9881-1fd06b546590-maas-api-service-ca\") pod \"93bfb362-fcc2-4341-9881-1fd06b546590\" (UID: \"93bfb362-fcc2-4341-9881-1fd06b546590\") " Oct 31 05:00:22.969047 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.969022 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvkkw\" (UniqueName: \"kubernetes.io/projected/d0dacaaf-e177-420b-a953-f8cdb4961a27-kube-api-access-gvkkw\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:00:22.979136 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.969431 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93bfb362-fcc2-4341-9881-1fd06b546590-maas-api-service-ca" pod "93bfb362-fcc2-4341-9881-1fd06b546590" (UID: "93bfb362-fcc2-4341-9881-1fd06b546590"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:00:22.979136 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:22.973034 5019 scope.go:122] "RemoveContainer" containerID="93cfacf43741883f8f7341fea7111a8a28f11b3dad97d1dbe446e59c5b77c842" Oct 31 05:00:23.011199 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.010094 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93bfb362-fcc2-4341-9881-1fd06b546590-kube-api-access-6l2gk" pod "93bfb362-fcc2-4341-9881-1fd06b546590" (UID: "93bfb362-fcc2-4341-9881-1fd06b546590"). InnerVolumeSpecName "kube-api-access-6l2gk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:00:23.070393 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.069579 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890380-ldsqw"] Oct 31 05:00:23.070594 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.070411 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d13131fa-5137-4401-aef8-d53fd8bd4e5d-kube-api-access-smndt\" (UniqueName: \"kubernetes.io/projected/d13131fa-5137-4401-aef8-d53fd8bd4e5d-kube-api-access-smndt\") pod \"d13131fa-5137-4401-aef8-d53fd8bd4e5d\" (UID: \"d13131fa-5137-4401-aef8-d53fd8bd4e5d\") " Oct 31 05:00:23.070594 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.070536 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d13131fa-5137-4401-aef8-d53fd8bd4e5d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d13131fa-5137-4401-aef8-d53fd8bd4e5d-maas-api-service-ca\") pod \"d13131fa-5137-4401-aef8-d53fd8bd4e5d\" (UID: \"d13131fa-5137-4401-aef8-d53fd8bd4e5d\") " Oct 31 05:00:23.071882 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.071839 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d13131fa-5137-4401-aef8-d53fd8bd4e5d-maas-api-service-ca" pod "d13131fa-5137-4401-aef8-d53fd8bd4e5d" (UID: "d13131fa-5137-4401-aef8-d53fd8bd4e5d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:00:23.074508 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.071793 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6l2gk\" (UniqueName: \"kubernetes.io/projected/93bfb362-fcc2-4341-9881-1fd06b546590-kube-api-access-6l2gk\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:00:23.074508 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.074436 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93bfb362-fcc2-4341-9881-1fd06b546590-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:00:23.074950 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.074920 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d13131fa-5137-4401-aef8-d53fd8bd4e5d-kube-api-access-smndt" pod "d13131fa-5137-4401-aef8-d53fd8bd4e5d" (UID: "d13131fa-5137-4401-aef8-d53fd8bd4e5d"). InnerVolumeSpecName "kube-api-access-smndt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:00:23.079962 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.079924 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890380-ldsqw"] Oct 31 05:00:23.136392 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.132853 5019 scope.go:122] "RemoveContainer" containerID="4b4377c5db90806371b2b551279f4bbc84522ccc6de2d0d0967bd6da2e216700" Oct 31 05:00:23.174033 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.173996 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890380-tx4lq" Oct 31 05:00:23.182315 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.182282 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smndt\" (UniqueName: \"kubernetes.io/projected/d13131fa-5137-4401-aef8-d53fd8bd4e5d-kube-api-access-smndt\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:00:23.182476 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.182319 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d13131fa-5137-4401-aef8-d53fd8bd4e5d-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:00:23.208046 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.208003 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-670795-29890380-hzmsc" Oct 31 05:00:23.284124 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.283343 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9790f69f-4de3-4837-a29f-17866661a3b3-kube-api-access-gpdc2\" (UniqueName: \"kubernetes.io/projected/9790f69f-4de3-4837-a29f-17866661a3b3-kube-api-access-gpdc2\") pod \"9790f69f-4de3-4837-a29f-17866661a3b3\" (UID: \"9790f69f-4de3-4837-a29f-17866661a3b3\") " Oct 31 05:00:23.284124 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.283406 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9515dc7-4ea3-4a1b-aad8-1e78c7d0e13c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9515dc7-4ea3-4a1b-aad8-1e78c7d0e13c-maas-api-service-ca\") pod \"c9515dc7-4ea3-4a1b-aad8-1e78c7d0e13c\" (UID: \"c9515dc7-4ea3-4a1b-aad8-1e78c7d0e13c\") " Oct 31 05:00:23.284124 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.283732 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9515dc7-4ea3-4a1b-aad8-1e78c7d0e13c-kube-api-access-f6wbf\" (UniqueName: \"kubernetes.io/projected/c9515dc7-4ea3-4a1b-aad8-1e78c7d0e13c-kube-api-access-f6wbf\") pod \"c9515dc7-4ea3-4a1b-aad8-1e78c7d0e13c\" (UID: \"c9515dc7-4ea3-4a1b-aad8-1e78c7d0e13c\") " Oct 31 05:00:23.284124 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.283822 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9790f69f-4de3-4837-a29f-17866661a3b3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9790f69f-4de3-4837-a29f-17866661a3b3-maas-api-service-ca\") pod \"9790f69f-4de3-4837-a29f-17866661a3b3\" (UID: \"9790f69f-4de3-4837-a29f-17866661a3b3\") " Oct 31 05:00:23.284124 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.284072 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9515dc7-4ea3-4a1b-aad8-1e78c7d0e13c-maas-api-service-ca" pod "c9515dc7-4ea3-4a1b-aad8-1e78c7d0e13c" (UID: "c9515dc7-4ea3-4a1b-aad8-1e78c7d0e13c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:00:23.284943 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.284574 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9790f69f-4de3-4837-a29f-17866661a3b3-maas-api-service-ca" pod "9790f69f-4de3-4837-a29f-17866661a3b3" (UID: "9790f69f-4de3-4837-a29f-17866661a3b3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:00:23.284943 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.284885 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9515dc7-4ea3-4a1b-aad8-1e78c7d0e13c-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:00:23.284943 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.284904 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9790f69f-4de3-4837-a29f-17866661a3b3-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:00:23.287972 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.287935 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9790f69f-4de3-4837-a29f-17866661a3b3-kube-api-access-gpdc2" pod "9790f69f-4de3-4837-a29f-17866661a3b3" (UID: "9790f69f-4de3-4837-a29f-17866661a3b3"). InnerVolumeSpecName "kube-api-access-gpdc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:00:23.288506 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.288480 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9515dc7-4ea3-4a1b-aad8-1e78c7d0e13c-kube-api-access-f6wbf" pod "c9515dc7-4ea3-4a1b-aad8-1e78c7d0e13c" (UID: "c9515dc7-4ea3-4a1b-aad8-1e78c7d0e13c"). InnerVolumeSpecName "kube-api-access-f6wbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:00:23.367133 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.365124 5019 generic.go:356] "Generic (PLEG): container finished" podID="e23416e6-954a-4ee6-8c4f-f6f8d7e1c506" containerID="679b7f9e6e2b4671b86e674c6f455972228023212f1f103ba7a1e00dcdc554be" exitCode=60 Oct 31 05:00:23.367941 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.366443 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-654741-29890380-85tdm" event={"ID":"e23416e6-954a-4ee6-8c4f-f6f8d7e1c506","Type":"ContainerDied","Data":"679b7f9e6e2b4671b86e674c6f455972228023212f1f103ba7a1e00dcdc554be"} Oct 31 05:00:23.370918 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.370538 5019 scope.go:122] "RemoveContainer" containerID="1b602417b78f0fb3ff0a24019758993c48d43aa12e36e7521950ee2edaad5b01" Oct 31 05:00:23.371346 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.371171 5019 scope.go:122] "RemoveContainer" containerID="679b7f9e6e2b4671b86e674c6f455972228023212f1f103ba7a1e00dcdc554be" Oct 31 05:00:23.374362 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.371729 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890380-fh8kl" Oct 31 05:00:23.374362 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:23.371865 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-654741-29890380-85tdm_odh-ai-gateway-infra(e23416e6-954a-4ee6-8c4f-f6f8d7e1c506)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-654741-29890380-85tdm" podUID="e23416e6-954a-4ee6-8c4f-f6f8d7e1c506" Oct 31 05:00:23.374362 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.372008 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890380-fh8kl" event={"ID":"d0dacaaf-e177-420b-a953-f8cdb4961a27","Type":"ContainerDied","Data":"fc4008d62599e7e298351217caca81c5ff4c300975d4e52f6dad33a797d15083"} Oct 31 05:00:23.384733 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.384691 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890380-kz5pd" event={"ID":"7ab61246-5d2d-4a2a-8c42-fff22960018f","Type":"ContainerStarted","Data":"83c8db674b9c8ef30fcca9a937550888c5dfefd8c017ee58f6ae27cc6b00cd58"} Oct 31 05:00:23.386802 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.386765 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6wbf\" (UniqueName: \"kubernetes.io/projected/c9515dc7-4ea3-4a1b-aad8-1e78c7d0e13c-kube-api-access-f6wbf\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:00:23.386974 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.386841 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpdc2\" (UniqueName: \"kubernetes.io/projected/9790f69f-4de3-4837-a29f-17866661a3b3-kube-api-access-gpdc2\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:00:23.394794 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.394761 5019 generic.go:356] "Generic (PLEG): container finished" podID="17f1de0f-3edd-4d07-910a-f6e6f356203c" containerID="5cffd7d31f60cc876e89b630ab8ecd2cb44c150d01d9150f3d2299b5b4d58cf5" exitCode=60 Oct 31 05:00:23.394967 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.394860 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890380-kr424" event={"ID":"17f1de0f-3edd-4d07-910a-f6e6f356203c","Type":"ContainerDied","Data":"5cffd7d31f60cc876e89b630ab8ecd2cb44c150d01d9150f3d2299b5b4d58cf5"} Oct 31 05:00:23.395436 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.395417 5019 scope.go:122] "RemoveContainer" containerID="5cffd7d31f60cc876e89b630ab8ecd2cb44c150d01d9150f3d2299b5b4d58cf5" Oct 31 05:00:23.395933 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:23.395906 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-afc519-29890380-kr424_odh-ai-gateway-infra(17f1de0f-3edd-4d07-910a-f6e6f356203c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890380-kr424" podUID="17f1de0f-3edd-4d07-910a-f6e6f356203c" Oct 31 05:00:23.402427 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.402400 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890380-tx4lq" Oct 31 05:00:23.405002 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.404974 5019 scope.go:122] "RemoveContainer" containerID="5b50df8436b5070148821f554633c5bda1bbf53638aa6bc8607b5689691dc49b" Oct 31 05:00:23.429167 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.429129 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fd6b126-7a7c-4825-a374-5dec6fb94d06" path="/var/lib/kubelet/pods/4fd6b126-7a7c-4825-a374-5dec6fb94d06/volumes" Oct 31 05:00:23.432316 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.432279 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="829dc86b-4ce1-427e-86db-cbaab8f15198" path="/var/lib/kubelet/pods/829dc86b-4ce1-427e-86db-cbaab8f15198/volumes" Oct 31 05:00:23.433743 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.433555 5019 scope.go:122] "RemoveContainer" containerID="df8d8adef2a9b7d4d7bf0ae9182e41601e535a6890077ca74cfe319a8821a03b" Oct 31 05:00:23.434315 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.434288 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1e9a647-0e9e-4528-a5a6-7022b244f1ba" path="/var/lib/kubelet/pods/e1e9a647-0e9e-4528-a5a6-7022b244f1ba/volumes" Oct 31 05:00:23.436683 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.435520 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9b19f33-a012-48c9-b670-3b8eff773e49" path="/var/lib/kubelet/pods/e9b19f33-a012-48c9-b670-3b8eff773e49/volumes" Oct 31 05:00:23.438002 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.437978 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f59877aa-80e8-418d-9617-2f9bb9c8f3ef" path="/var/lib/kubelet/pods/f59877aa-80e8-418d-9617-2f9bb9c8f3ef/volumes" Oct 31 05:00:23.441781 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.441753 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-031745-29890380-9jpj5"] Oct 31 05:00:23.442255 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.442198 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890380-tx4lq" event={"ID":"9790f69f-4de3-4837-a29f-17866661a3b3","Type":"ContainerDied","Data":"60096ab33c9ce7c34ce1bce12d5ef1de0f50fb592f4b41cf235b56b656d1b767"} Oct 31 05:00:23.445086 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.445059 5019 generic.go:356] "Generic (PLEG): container finished" podID="703c28bc-a603-4001-9538-4a751cf9b7c0" containerID="a02aa289de92c2867d117b29a5b931494288fb4a7dfd8b4c01326334413a4302" exitCode=60 Oct 31 05:00:23.445234 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.445211 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-567ca6-29890380-kcwdg" event={"ID":"703c28bc-a603-4001-9538-4a751cf9b7c0","Type":"ContainerDied","Data":"a02aa289de92c2867d117b29a5b931494288fb4a7dfd8b4c01326334413a4302"} Oct 31 05:00:23.446005 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.445798 5019 scope.go:122] "RemoveContainer" containerID="a02aa289de92c2867d117b29a5b931494288fb4a7dfd8b4c01326334413a4302" Oct 31 05:00:23.446266 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:23.446239 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-567ca6-29890380-kcwdg_odh-ai-gateway-infra(703c28bc-a603-4001-9538-4a751cf9b7c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-567ca6-29890380-kcwdg" podUID="703c28bc-a603-4001-9538-4a751cf9b7c0" Oct 31 05:00:23.476357 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.476304 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890380-k8lq8" event={"ID":"deaadd33-dcef-43df-a66e-9fd41434de6f","Type":"ContainerStarted","Data":"cab9f0b3ba1b2b9d5c1d88af25038fa7bc10e63c4184cb680d2242ffea84ecfc"} Oct 31 05:00:23.483352 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.480032 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-47c409-29890380-cqm6g" event={"ID":"d13131fa-5137-4401-aef8-d53fd8bd4e5d","Type":"ContainerDied","Data":"b026302e47440a8d9c43a310b6cea86e3de97c1e7227274e78f7b08d34faca31"} Oct 31 05:00:23.483352 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.480157 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-47c409-29890380-cqm6g" Oct 31 05:00:23.495589 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.493876 5019 generic.go:356] "Generic (PLEG): container finished" podID="cd0723e3-dcf6-4907-9ce1-99ca9c77b144" containerID="83aa2aeb549688d462bb067fe5ddd79f2976af9fb9849bf81526b5a3679a984e" exitCode=60 Oct 31 05:00:23.495589 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.493972 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1daf30-29890380-pbtq2" event={"ID":"cd0723e3-dcf6-4907-9ce1-99ca9c77b144","Type":"ContainerDied","Data":"83aa2aeb549688d462bb067fe5ddd79f2976af9fb9849bf81526b5a3679a984e"} Oct 31 05:00:23.495589 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.494534 5019 scope.go:122] "RemoveContainer" containerID="83aa2aeb549688d462bb067fe5ddd79f2976af9fb9849bf81526b5a3679a984e" Oct 31 05:00:23.495589 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:23.494971 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-1daf30-29890380-pbtq2_odh-ai-gateway-infra(cd0723e3-dcf6-4907-9ce1-99ca9c77b144)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1daf30-29890380-pbtq2" podUID="cd0723e3-dcf6-4907-9ce1-99ca9c77b144" Oct 31 05:00:23.495589 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.495359 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890380-fw6dq"] Oct 31 05:00:23.513011 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.511829 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-47e9af-29890380-jk7b6" event={"ID":"7c20d99d-7ce4-4bb8-a4d9-188cec401d75","Type":"ContainerStarted","Data":"2a7402fe5081d2bb226a52cd353f662389c29dce54cac1d01b401b71e95f6a2d"} Oct 31 05:00:23.515630 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.515601 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890380-7g9d5" event={"ID":"93bfb362-fcc2-4341-9881-1fd06b546590","Type":"ContainerDied","Data":"31d155ef347108aee11b43ea61a5eacd06883df1056f14b570a03111b694d2ca"} Oct 31 05:00:23.515913 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.515896 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890380-7g9d5" Oct 31 05:00:23.518728 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.518699 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-670795-29890380-hzmsc" event={"ID":"c9515dc7-4ea3-4a1b-aad8-1e78c7d0e13c","Type":"ContainerDied","Data":"bcd8e69a5a92881ae3b7f82c3c7a38b043bbbd0c755ffb7288fb4cbc0709c5e4"} Oct 31 05:00:23.519312 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.519034 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-670795-29890380-hzmsc" Oct 31 05:00:23.584519 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.559528 5019 scope.go:122] "RemoveContainer" containerID="9a695d1ea0d6377263e86811802ddf8be2e1b3b456e438751257b0167e071c93" Oct 31 05:00:23.584519 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.564215 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890380-kz5pd" podStartSLOduration=23.564200493 podStartE2EDuration="23.564200493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:00:23.553419803 +0000 UTC m=+3378.309757844" watchObservedRunningTime="2026-10-31 05:00:23.564200493 +0000 UTC m=+3378.320538533" Oct 31 05:00:23.590987 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.587030 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-108ad8-29890380-ng62l"] Oct 31 05:00:23.654721 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.652157 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890380-fh8kl"] Oct 31 05:00:23.664044 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.662489 5019 scope.go:122] "RemoveContainer" containerID="b1dd400398607ff40e67dca107dd07ed4a12b5dce7fb7eb481453dcce873925d" Oct 31 05:00:23.664044 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.663592 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890380-fh8kl"] Oct 31 05:00:23.790715 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.790670 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-47c409-29890380-cqm6g"] Oct 31 05:00:23.796147 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.796099 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-47c409-29890380-cqm6g"] Oct 31 05:00:23.798785 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.798753 5019 scope.go:122] "RemoveContainer" containerID="3c9e867331d221180d4ced9ab5f6ce4c37b45d1553799585d0d38b0838e49118" Oct 31 05:00:23.812141 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.812111 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890380-tx4lq"] Oct 31 05:00:23.824743 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.824709 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890380-tx4lq"] Oct 31 05:00:23.828438 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.828321 5019 scope.go:122] "RemoveContainer" containerID="42c9c9d6a5eb52a1726c02f5a06ad5de76f43254a6628953e9ff3f19367afe38" Oct 31 05:00:23.833919 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.833892 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890380-b7jtg" Oct 31 05:00:23.835435 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.835218 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890380-tkvr7"] Oct 31 05:00:23.878397 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.874064 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-670795-29890380-hzmsc"] Oct 31 05:00:23.894002 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.893029 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-670795-29890380-hzmsc"] Oct 31 05:00:23.903970 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.903695 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890380-rmxrz"] Oct 31 05:00:23.982152 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.961460 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e27f22dd-e82f-493c-9745-6d901308b907-kube-api-access-mp8tj\" (UniqueName: \"kubernetes.io/projected/e27f22dd-e82f-493c-9745-6d901308b907-kube-api-access-mp8tj\") pod \"e27f22dd-e82f-493c-9745-6d901308b907\" (UID: \"e27f22dd-e82f-493c-9745-6d901308b907\") " Oct 31 05:00:23.982152 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.961535 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e27f22dd-e82f-493c-9745-6d901308b907-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e27f22dd-e82f-493c-9745-6d901308b907-maas-api-service-ca\") pod \"e27f22dd-e82f-493c-9745-6d901308b907\" (UID: \"e27f22dd-e82f-493c-9745-6d901308b907\") " Oct 31 05:00:23.982152 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.963265 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e27f22dd-e82f-493c-9745-6d901308b907-maas-api-service-ca" pod "e27f22dd-e82f-493c-9745-6d901308b907" (UID: "e27f22dd-e82f-493c-9745-6d901308b907"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:00:23.982152 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.965110 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e27f22dd-e82f-493c-9745-6d901308b907-kube-api-access-mp8tj" pod "e27f22dd-e82f-493c-9745-6d901308b907" (UID: "e27f22dd-e82f-493c-9745-6d901308b907"). InnerVolumeSpecName "kube-api-access-mp8tj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:00:23.996248 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:23.996146 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a04e4e-29890380-jgchh"] Oct 31 05:00:24.017505 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.015697 5019 scope.go:122] "RemoveContainer" containerID="45e3d3507691ebebd539f807a422993919a5d86db0cd53ec1e0eab2fc9cfe06f" Oct 31 05:00:24.030313 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.030000 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1daf30-29890380-pbtq2"] Oct 31 05:00:24.064941 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.064775 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp8tj\" (UniqueName: \"kubernetes.io/projected/e27f22dd-e82f-493c-9745-6d901308b907-kube-api-access-mp8tj\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:00:24.064941 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.064805 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e27f22dd-e82f-493c-9745-6d901308b907-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:00:24.086405 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:24.077886 5019 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeaadd33_dcef_43df_a66e_9fd41434de6f.slice/crio-cab9f0b3ba1b2b9d5c1d88af25038fa7bc10e63c4184cb680d2242ffea84ecfc.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ab61246_5d2d_4a2a_8c42_fff22960018f.slice/crio-conmon-83c8db674b9c8ef30fcca9a937550888c5dfefd8c017ee58f6ae27cc6b00cd58.scope\": RecentStats: unable to find data in memory cache]" Oct 31 05:00:24.102517 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.102405 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890380-7g9d5"] Oct 31 05:00:24.102517 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.102474 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890380-7g9d5"] Oct 31 05:00:24.104261 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.104201 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890380-k8lq8" podStartSLOduration=24.104185002 podStartE2EDuration="24.104185002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:00:24.046082971 +0000 UTC m=+3378.802421012" watchObservedRunningTime="2026-10-31 05:00:24.104185002 +0000 UTC m=+3378.860523042" Oct 31 05:00:24.187057 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.187031 5019 scope.go:122] "RemoveContainer" containerID="5942c0d16ab14ed6dc80ac228b23fae45023dda719b99b028106b375bd575c9f" Oct 31 05:00:24.193466 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.193438 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890380-fw6dq" Oct 31 05:00:24.200208 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.200178 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-031745-29890380-9jpj5" Oct 31 05:00:24.219251 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.219216 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-108ad8-29890380-ng62l" Oct 31 05:00:24.270901 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.270162 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3d4f5cd-c755-4e4f-a86f-14874e597d08-kube-api-access-57r8h\" (UniqueName: \"kubernetes.io/projected/d3d4f5cd-c755-4e4f-a86f-14874e597d08-kube-api-access-57r8h\") pod \"d3d4f5cd-c755-4e4f-a86f-14874e597d08\" (UID: \"d3d4f5cd-c755-4e4f-a86f-14874e597d08\") " Oct 31 05:00:24.270901 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.270311 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3d4f5cd-c755-4e4f-a86f-14874e597d08-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3d4f5cd-c755-4e4f-a86f-14874e597d08-maas-api-service-ca\") pod \"d3d4f5cd-c755-4e4f-a86f-14874e597d08\" (UID: \"d3d4f5cd-c755-4e4f-a86f-14874e597d08\") " Oct 31 05:00:24.270901 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.270362 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96e0a71b-2eee-4760-ade1-5cdd604c4c81-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96e0a71b-2eee-4760-ade1-5cdd604c4c81-maas-api-service-ca\") pod \"96e0a71b-2eee-4760-ade1-5cdd604c4c81\" (UID: \"96e0a71b-2eee-4760-ade1-5cdd604c4c81\") " Oct 31 05:00:24.270901 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.270461 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96e0a71b-2eee-4760-ade1-5cdd604c4c81-kube-api-access-ndq2s\" (UniqueName: \"kubernetes.io/projected/96e0a71b-2eee-4760-ade1-5cdd604c4c81-kube-api-access-ndq2s\") pod \"96e0a71b-2eee-4760-ade1-5cdd604c4c81\" (UID: \"96e0a71b-2eee-4760-ade1-5cdd604c4c81\") " Oct 31 05:00:24.271784 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.271726 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3d4f5cd-c755-4e4f-a86f-14874e597d08-maas-api-service-ca" pod "d3d4f5cd-c755-4e4f-a86f-14874e597d08" (UID: "d3d4f5cd-c755-4e4f-a86f-14874e597d08"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:00:24.271943 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.271824 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96e0a71b-2eee-4760-ade1-5cdd604c4c81-maas-api-service-ca" pod "96e0a71b-2eee-4760-ade1-5cdd604c4c81" (UID: "96e0a71b-2eee-4760-ade1-5cdd604c4c81"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:00:24.274132 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.274089 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3d4f5cd-c755-4e4f-a86f-14874e597d08-kube-api-access-57r8h" pod "d3d4f5cd-c755-4e4f-a86f-14874e597d08" (UID: "d3d4f5cd-c755-4e4f-a86f-14874e597d08"). InnerVolumeSpecName "kube-api-access-57r8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:00:24.274486 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.274455 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96e0a71b-2eee-4760-ade1-5cdd604c4c81-kube-api-access-ndq2s" pod "96e0a71b-2eee-4760-ade1-5cdd604c4c81" (UID: "96e0a71b-2eee-4760-ade1-5cdd604c4c81"). InnerVolumeSpecName "kube-api-access-ndq2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:00:24.296741 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.291022 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890380-rmxrz" Oct 31 05:00:24.372377 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.372323 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-c1fe26/test-model-c1fe26-kserve-7fd8b8b8f-qkbvb" Oct 31 05:00:24.373538 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.373505 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5259d89-590d-4fc7-b521-8f6feabbd2cc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5259d89-590d-4fc7-b521-8f6feabbd2cc-maas-api-service-ca\") pod \"e5259d89-590d-4fc7-b521-8f6feabbd2cc\" (UID: \"e5259d89-590d-4fc7-b521-8f6feabbd2cc\") " Oct 31 05:00:24.373686 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.373603 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/982ad40e-a719-4f61-be85-ea665503c299-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/982ad40e-a719-4f61-be85-ea665503c299-maas-api-service-ca\") pod \"982ad40e-a719-4f61-be85-ea665503c299\" (UID: \"982ad40e-a719-4f61-be85-ea665503c299\") " Oct 31 05:00:24.373828 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.373803 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/982ad40e-a719-4f61-be85-ea665503c299-kube-api-access-xzstl\" (UniqueName: \"kubernetes.io/projected/982ad40e-a719-4f61-be85-ea665503c299-kube-api-access-xzstl\") pod \"982ad40e-a719-4f61-be85-ea665503c299\" (UID: \"982ad40e-a719-4f61-be85-ea665503c299\") " Oct 31 05:00:24.374142 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.374126 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5259d89-590d-4fc7-b521-8f6feabbd2cc-kube-api-access-672zl\" (UniqueName: \"kubernetes.io/projected/e5259d89-590d-4fc7-b521-8f6feabbd2cc-kube-api-access-672zl\") pod \"e5259d89-590d-4fc7-b521-8f6feabbd2cc\" (UID: \"e5259d89-590d-4fc7-b521-8f6feabbd2cc\") " Oct 31 05:00:24.374800 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.374776 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57r8h\" (UniqueName: \"kubernetes.io/projected/d3d4f5cd-c755-4e4f-a86f-14874e597d08-kube-api-access-57r8h\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:00:24.374900 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.374803 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3d4f5cd-c755-4e4f-a86f-14874e597d08-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:00:24.374900 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.374817 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96e0a71b-2eee-4760-ade1-5cdd604c4c81-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:00:24.374900 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.374830 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndq2s\" (UniqueName: \"kubernetes.io/projected/96e0a71b-2eee-4760-ade1-5cdd604c4c81-kube-api-access-ndq2s\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:00:24.375114 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.374787 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5259d89-590d-4fc7-b521-8f6feabbd2cc-maas-api-service-ca" pod "e5259d89-590d-4fc7-b521-8f6feabbd2cc" (UID: "e5259d89-590d-4fc7-b521-8f6feabbd2cc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:00:24.377923 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.377896 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5259d89-590d-4fc7-b521-8f6feabbd2cc-kube-api-access-672zl" pod "e5259d89-590d-4fc7-b521-8f6feabbd2cc" (UID: "e5259d89-590d-4fc7-b521-8f6feabbd2cc"). InnerVolumeSpecName "kube-api-access-672zl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:00:24.379720 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.379703 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/982ad40e-a719-4f61-be85-ea665503c299-maas-api-service-ca" pod "982ad40e-a719-4f61-be85-ea665503c299" (UID: "982ad40e-a719-4f61-be85-ea665503c299"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:00:24.388587 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.388551 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/982ad40e-a719-4f61-be85-ea665503c299-kube-api-access-xzstl" pod "982ad40e-a719-4f61-be85-ea665503c299" (UID: "982ad40e-a719-4f61-be85-ea665503c299"). InnerVolumeSpecName "kube-api-access-xzstl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:00:24.392835 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.392813 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-c1fe26/test-model-c1fe26-kserve-7fd8b8b8f-qkbvb" Oct 31 05:00:24.409110 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.409084 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890380-tkvr7" Oct 31 05:00:24.432386 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.432349 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-654741-29890380-85tdm"] Oct 31 05:00:24.449308 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.449222 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a04e4e-29890380-jgchh" Oct 31 05:00:24.476293 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.476252 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e478c8f1-9e68-4f6f-b919-28ba397e66c4-kube-api-access-wwql4\" (UniqueName: \"kubernetes.io/projected/e478c8f1-9e68-4f6f-b919-28ba397e66c4-kube-api-access-wwql4\") pod \"e478c8f1-9e68-4f6f-b919-28ba397e66c4\" (UID: \"e478c8f1-9e68-4f6f-b919-28ba397e66c4\") " Oct 31 05:00:24.476538 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.476405 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e478c8f1-9e68-4f6f-b919-28ba397e66c4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e478c8f1-9e68-4f6f-b919-28ba397e66c4-maas-api-service-ca\") pod \"e478c8f1-9e68-4f6f-b919-28ba397e66c4\" (UID: \"e478c8f1-9e68-4f6f-b919-28ba397e66c4\") " Oct 31 05:00:24.476821 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.476795 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e478c8f1-9e68-4f6f-b919-28ba397e66c4-maas-api-service-ca" pod "e478c8f1-9e68-4f6f-b919-28ba397e66c4" (UID: "e478c8f1-9e68-4f6f-b919-28ba397e66c4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:00:24.477950 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.477534 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e478c8f1-9e68-4f6f-b919-28ba397e66c4-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:00:24.477950 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.477564 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-672zl\" (UniqueName: \"kubernetes.io/projected/e5259d89-590d-4fc7-b521-8f6feabbd2cc-kube-api-access-672zl\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:00:24.477950 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.477578 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5259d89-590d-4fc7-b521-8f6feabbd2cc-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:00:24.477950 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.477592 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/982ad40e-a719-4f61-be85-ea665503c299-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:00:24.477950 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.477607 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzstl\" (UniqueName: \"kubernetes.io/projected/982ad40e-a719-4f61-be85-ea665503c299-kube-api-access-xzstl\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:00:24.479884 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.479858 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e478c8f1-9e68-4f6f-b919-28ba397e66c4-kube-api-access-wwql4" pod "e478c8f1-9e68-4f6f-b919-28ba397e66c4" (UID: "e478c8f1-9e68-4f6f-b919-28ba397e66c4"). InnerVolumeSpecName "kube-api-access-wwql4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:00:24.533407 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.533299 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890380-kr424"] Oct 31 05:00:24.554269 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.554225 5019 generic.go:356] "Generic (PLEG): container finished" podID="7ab61246-5d2d-4a2a-8c42-fff22960018f" containerID="83c8db674b9c8ef30fcca9a937550888c5dfefd8c017ee58f6ae27cc6b00cd58" exitCode=60 Oct 31 05:00:24.554504 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.554319 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890380-kz5pd" event={"ID":"7ab61246-5d2d-4a2a-8c42-fff22960018f","Type":"ContainerDied","Data":"83c8db674b9c8ef30fcca9a937550888c5dfefd8c017ee58f6ae27cc6b00cd58"} Oct 31 05:00:24.554504 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.554418 5019 scope.go:122] "RemoveContainer" containerID="939033db9d1fbe818fc40ca51fefca9989c43a8489b97b983c36af647203a762" Oct 31 05:00:24.554954 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.554932 5019 scope.go:122] "RemoveContainer" containerID="83c8db674b9c8ef30fcca9a937550888c5dfefd8c017ee58f6ae27cc6b00cd58" Oct 31 05:00:24.555908 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:24.555428 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890380-kz5pd_odh-ai-gateway-infra(7ab61246-5d2d-4a2a-8c42-fff22960018f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890380-kz5pd" podUID="7ab61246-5d2d-4a2a-8c42-fff22960018f" Oct 31 05:00:24.557877 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.557848 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-031745-29890380-9jpj5" event={"ID":"d3d4f5cd-c755-4e4f-a86f-14874e597d08","Type":"ContainerDied","Data":"1c09ce316a7b3487db94c307631aef0c20800570ced220a14b4ed22ea44b69f6"} Oct 31 05:00:24.557967 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.557943 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-031745-29890380-9jpj5" Oct 31 05:00:24.559609 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.559587 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890380-fw6dq" Oct 31 05:00:24.559738 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.559634 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890380-fw6dq" event={"ID":"96e0a71b-2eee-4760-ade1-5cdd604c4c81","Type":"ContainerDied","Data":"75bb012bf2acb18f8e1b88177d0f3ddb647176cb7ea75d3f8f01a6a42ba6109f"} Oct 31 05:00:24.571364 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.569935 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-108ad8-29890380-ng62l" event={"ID":"e5259d89-590d-4fc7-b521-8f6feabbd2cc","Type":"ContainerDied","Data":"42e716007154a4aca25d539c13737d651d4d7685dabba7f685eb8c575cefadae"} Oct 31 05:00:24.571364 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.570031 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-108ad8-29890380-ng62l" Oct 31 05:00:24.581850 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.574585 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890380-b7jtg" event={"ID":"e27f22dd-e82f-493c-9745-6d901308b907","Type":"ContainerDied","Data":"6827956b8d908213a0999139105c6775bb9effbd6ab4cbfc232ed53dd6e58bc8"} Oct 31 05:00:24.581850 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.574682 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890380-b7jtg" Oct 31 05:00:24.581850 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.578133 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890380-kz5pd"] Oct 31 05:00:24.581850 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.578377 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b6b5e89-5900-4db2-8552-e4f029de7573-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b6b5e89-5900-4db2-8552-e4f029de7573-maas-api-service-ca\") pod \"0b6b5e89-5900-4db2-8552-e4f029de7573\" (UID: \"0b6b5e89-5900-4db2-8552-e4f029de7573\") " Oct 31 05:00:24.581850 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.578610 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b6b5e89-5900-4db2-8552-e4f029de7573-kube-api-access-lb8ht\" (UniqueName: \"kubernetes.io/projected/0b6b5e89-5900-4db2-8552-e4f029de7573-kube-api-access-lb8ht\") pod \"0b6b5e89-5900-4db2-8552-e4f029de7573\" (UID: \"0b6b5e89-5900-4db2-8552-e4f029de7573\") " Oct 31 05:00:24.581850 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.578810 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b6b5e89-5900-4db2-8552-e4f029de7573-maas-api-service-ca" pod "0b6b5e89-5900-4db2-8552-e4f029de7573" (UID: "0b6b5e89-5900-4db2-8552-e4f029de7573"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:00:24.581850 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.579250 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b6b5e89-5900-4db2-8552-e4f029de7573-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:00:24.581850 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.579268 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwql4\" (UniqueName: \"kubernetes.io/projected/e478c8f1-9e68-4f6f-b919-28ba397e66c4-kube-api-access-wwql4\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:00:24.581850 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.580240 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890380-rmxrz" event={"ID":"982ad40e-a719-4f61-be85-ea665503c299","Type":"ContainerDied","Data":"0c7efa996e8c8c81ac76087696f0b213b0635ad19dc11a6640b92c932ed33d45"} Oct 31 05:00:24.581850 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.580326 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890380-rmxrz" Oct 31 05:00:24.594690 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.594321 5019 scope.go:122] "RemoveContainer" containerID="8adfa676587097a0c6c458a31f7ed1e98d35d5dc05718c41b615bf29e0cd29bb" Oct 31 05:00:24.601086 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.596837 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890380-tkvr7" event={"ID":"e478c8f1-9e68-4f6f-b919-28ba397e66c4","Type":"ContainerDied","Data":"c6a09d6300c609e78e460cc801be813ea805b5850be2b40de7d1efad203d48d4"} Oct 31 05:00:24.601086 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.596938 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890380-tkvr7" Oct 31 05:00:24.611119 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.611087 5019 generic.go:356] "Generic (PLEG): container finished" podID="deaadd33-dcef-43df-a66e-9fd41434de6f" containerID="cab9f0b3ba1b2b9d5c1d88af25038fa7bc10e63c4184cb680d2242ffea84ecfc" exitCode=60 Oct 31 05:00:24.611235 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.611217 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890380-k8lq8" event={"ID":"deaadd33-dcef-43df-a66e-9fd41434de6f","Type":"ContainerDied","Data":"cab9f0b3ba1b2b9d5c1d88af25038fa7bc10e63c4184cb680d2242ffea84ecfc"} Oct 31 05:00:24.613688 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.611725 5019 scope.go:122] "RemoveContainer" containerID="cab9f0b3ba1b2b9d5c1d88af25038fa7bc10e63c4184cb680d2242ffea84ecfc" Oct 31 05:00:24.613688 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:24.612159 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890380-k8lq8_odh-ai-gateway-infra(deaadd33-dcef-43df-a66e-9fd41434de6f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890380-k8lq8" podUID="deaadd33-dcef-43df-a66e-9fd41434de6f" Oct 31 05:00:24.635481 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.635458 5019 scope.go:122] "RemoveContainer" containerID="6eaa5cb9139a1bbcdd323bee1fe7831ad6dd3e149737606df029236f27c1a274" Oct 31 05:00:24.639703 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.639678 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a04e4e-29890380-jgchh" event={"ID":"0b6b5e89-5900-4db2-8552-e4f029de7573","Type":"ContainerDied","Data":"d5b6a486c3a1ef24a63ad1db8e4b5fccf56225cae51e403206c48ed14fba167f"} Oct 31 05:00:24.639959 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.639899 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a04e4e-29890380-jgchh" Oct 31 05:00:24.644190 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.642642 5019 generic.go:356] "Generic (PLEG): container finished" podID="7c20d99d-7ce4-4bb8-a4d9-188cec401d75" containerID="2a7402fe5081d2bb226a52cd353f662389c29dce54cac1d01b401b71e95f6a2d" exitCode=60 Oct 31 05:00:24.644190 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.642746 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-47e9af-29890380-jk7b6" event={"ID":"7c20d99d-7ce4-4bb8-a4d9-188cec401d75","Type":"ContainerDied","Data":"2a7402fe5081d2bb226a52cd353f662389c29dce54cac1d01b401b71e95f6a2d"} Oct 31 05:00:24.644190 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.643209 5019 scope.go:122] "RemoveContainer" containerID="2a7402fe5081d2bb226a52cd353f662389c29dce54cac1d01b401b71e95f6a2d" Oct 31 05:00:24.644190 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:24.643629 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-47e9af-29890380-jk7b6_odh-ai-gateway-infra(7c20d99d-7ce4-4bb8-a4d9-188cec401d75)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-47e9af-29890380-jk7b6" podUID="7c20d99d-7ce4-4bb8-a4d9-188cec401d75" Oct 31 05:00:24.674557 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.672727 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b6b5e89-5900-4db2-8552-e4f029de7573-kube-api-access-lb8ht" pod "0b6b5e89-5900-4db2-8552-e4f029de7573" (UID: "0b6b5e89-5900-4db2-8552-e4f029de7573"). InnerVolumeSpecName "kube-api-access-lb8ht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:00:24.680044 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.680019 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-c1fe26/test-model-c1fe26-kserve-7fd8b8b8f-qkbvb" Oct 31 05:00:24.689660 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.681735 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb8ht\" (UniqueName: \"kubernetes.io/projected/0b6b5e89-5900-4db2-8552-e4f029de7573-kube-api-access-lb8ht\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:00:24.689660 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.687117 5019 scope.go:122] "RemoveContainer" containerID="3a80fa17fb3a6e024cdfdaacd056af64be3a2f075308f38a95f1d7c5b4c95370" Oct 31 05:00:24.704035 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.703980 5019 scope.go:122] "RemoveContainer" containerID="941ad460acc314b48df4126bf8744d1717bde3636b6ace08af19190679ef51ec" Oct 31 05:00:24.727282 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.726074 5019 scope.go:122] "RemoveContainer" containerID="a209cfa1faa3f7f1c3b2181992def0d9b21e2c74035f70b9b239b7e75efd8e2c" Oct 31 05:00:24.763084 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.762774 5019 scope.go:122] "RemoveContainer" containerID="6a165e20357201538611a06791aaf559a24898b41e7e06352438f831a5412462" Oct 31 05:00:24.786920 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.786887 5019 scope.go:122] "RemoveContainer" containerID="bf7f393c48f1a8d8d00b56318c44ebe02be183b0f93246deb2ffe76e2e1ec8e1" Oct 31 05:00:24.816529 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.816498 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890380-b7jtg"] Oct 31 05:00:24.819374 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.819344 5019 scope.go:122] "RemoveContainer" containerID="4114cf636ee176574808e2e50eee7af545e46773cb85c03d06311d4bfdf823a4" Oct 31 05:00:24.823997 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.823969 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890380-b7jtg"] Oct 31 05:00:24.863297 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.862903 5019 scope.go:122] "RemoveContainer" containerID="e6aa6c0788c5a4e2329732970dd459152f7207dcaa204cc9226ef9c93dfec3d2" Oct 31 05:00:24.919816 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.911282 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-47e9af-29890380-jk7b6"] Oct 31 05:00:24.947032 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.947001 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890380-kr424" Oct 31 05:00:24.951091 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.951064 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-654741-29890380-85tdm" Oct 31 05:00:24.990971 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.990918 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-567ca6-29890380-kcwdg"] Oct 31 05:00:24.994183 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.994162 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1daf30-29890380-pbtq2" Oct 31 05:00:24.999240 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.999214 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17f1de0f-3edd-4d07-910a-f6e6f356203c-kube-api-access-slb2v\" (UniqueName: \"kubernetes.io/projected/17f1de0f-3edd-4d07-910a-f6e6f356203c-kube-api-access-slb2v\") pod \"17f1de0f-3edd-4d07-910a-f6e6f356203c\" (UID: \"17f1de0f-3edd-4d07-910a-f6e6f356203c\") " Oct 31 05:00:24.999591 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:24.999569 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17f1de0f-3edd-4d07-910a-f6e6f356203c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17f1de0f-3edd-4d07-910a-f6e6f356203c-maas-api-service-ca\") pod \"17f1de0f-3edd-4d07-910a-f6e6f356203c\" (UID: \"17f1de0f-3edd-4d07-910a-f6e6f356203c\") " Oct 31 05:00:25.000670 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.000647 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17f1de0f-3edd-4d07-910a-f6e6f356203c-maas-api-service-ca" pod "17f1de0f-3edd-4d07-910a-f6e6f356203c" (UID: "17f1de0f-3edd-4d07-910a-f6e6f356203c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:00:25.002876 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.002843 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17f1de0f-3edd-4d07-910a-f6e6f356203c-kube-api-access-slb2v" pod "17f1de0f-3edd-4d07-910a-f6e6f356203c" (UID: "17f1de0f-3edd-4d07-910a-f6e6f356203c"). InnerVolumeSpecName "kube-api-access-slb2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:00:25.058970 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.058934 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890380-k8lq8"] Oct 31 05:00:25.065873 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.065594 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-031745-29890380-9jpj5"] Oct 31 05:00:25.080663 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.080614 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-031745-29890380-9jpj5"] Oct 31 05:00:25.101934 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.101904 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e23416e6-954a-4ee6-8c4f-f6f8d7e1c506-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e23416e6-954a-4ee6-8c4f-f6f8d7e1c506-maas-api-service-ca\") pod \"e23416e6-954a-4ee6-8c4f-f6f8d7e1c506\" (UID: \"e23416e6-954a-4ee6-8c4f-f6f8d7e1c506\") " Oct 31 05:00:25.102243 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.102211 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd0723e3-dcf6-4907-9ce1-99ca9c77b144-kube-api-access-wnsgn\" (UniqueName: \"kubernetes.io/projected/cd0723e3-dcf6-4907-9ce1-99ca9c77b144-kube-api-access-wnsgn\") pod \"cd0723e3-dcf6-4907-9ce1-99ca9c77b144\" (UID: \"cd0723e3-dcf6-4907-9ce1-99ca9c77b144\") " Oct 31 05:00:25.102416 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.102305 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e23416e6-954a-4ee6-8c4f-f6f8d7e1c506-maas-api-service-ca" pod "e23416e6-954a-4ee6-8c4f-f6f8d7e1c506" (UID: "e23416e6-954a-4ee6-8c4f-f6f8d7e1c506"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:00:25.102509 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.102472 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd0723e3-dcf6-4907-9ce1-99ca9c77b144-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd0723e3-dcf6-4907-9ce1-99ca9c77b144-maas-api-service-ca\") pod \"cd0723e3-dcf6-4907-9ce1-99ca9c77b144\" (UID: \"cd0723e3-dcf6-4907-9ce1-99ca9c77b144\") " Oct 31 05:00:25.102605 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.102586 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e23416e6-954a-4ee6-8c4f-f6f8d7e1c506-kube-api-access-rstjm\" (UniqueName: \"kubernetes.io/projected/e23416e6-954a-4ee6-8c4f-f6f8d7e1c506-kube-api-access-rstjm\") pod \"e23416e6-954a-4ee6-8c4f-f6f8d7e1c506\" (UID: \"e23416e6-954a-4ee6-8c4f-f6f8d7e1c506\") " Oct 31 05:00:25.103443 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.103422 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17f1de0f-3edd-4d07-910a-f6e6f356203c-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:00:25.103598 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.103581 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slb2v\" (UniqueName: \"kubernetes.io/projected/17f1de0f-3edd-4d07-910a-f6e6f356203c-kube-api-access-slb2v\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:00:25.103735 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.103684 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e23416e6-954a-4ee6-8c4f-f6f8d7e1c506-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:00:25.104301 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.104280 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd0723e3-dcf6-4907-9ce1-99ca9c77b144-maas-api-service-ca" pod "cd0723e3-dcf6-4907-9ce1-99ca9c77b144" (UID: "cd0723e3-dcf6-4907-9ce1-99ca9c77b144"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:00:25.107818 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.107793 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e23416e6-954a-4ee6-8c4f-f6f8d7e1c506-kube-api-access-rstjm" pod "e23416e6-954a-4ee6-8c4f-f6f8d7e1c506" (UID: "e23416e6-954a-4ee6-8c4f-f6f8d7e1c506"). InnerVolumeSpecName "kube-api-access-rstjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:00:25.108099 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.108068 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd0723e3-dcf6-4907-9ce1-99ca9c77b144-kube-api-access-wnsgn" pod "cd0723e3-dcf6-4907-9ce1-99ca9c77b144" (UID: "cd0723e3-dcf6-4907-9ce1-99ca9c77b144"). InnerVolumeSpecName "kube-api-access-wnsgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:00:25.171709 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.171675 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-567ca6-29890380-kcwdg" Oct 31 05:00:25.175716 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.175682 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890380-tkvr7"] Oct 31 05:00:25.185730 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.185695 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890380-tkvr7"] Oct 31 05:00:25.206041 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.206002 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnsgn\" (UniqueName: \"kubernetes.io/projected/cd0723e3-dcf6-4907-9ce1-99ca9c77b144-kube-api-access-wnsgn\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:00:25.206041 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.206037 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd0723e3-dcf6-4907-9ce1-99ca9c77b144-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:00:25.206041 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.206049 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rstjm\" (UniqueName: \"kubernetes.io/projected/e23416e6-954a-4ee6-8c4f-f6f8d7e1c506-kube-api-access-rstjm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:00:25.220726 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.220635 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890380-rmxrz"] Oct 31 05:00:25.226724 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.226687 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-57dc6d-29890380-rmxrz"] Oct 31 05:00:25.263476 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.263438 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890380-fw6dq"] Oct 31 05:00:25.266600 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.266570 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890380-fw6dq"] Oct 31 05:00:25.307713 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.307669 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/703c28bc-a603-4001-9538-4a751cf9b7c0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/703c28bc-a603-4001-9538-4a751cf9b7c0-maas-api-service-ca\") pod \"703c28bc-a603-4001-9538-4a751cf9b7c0\" (UID: \"703c28bc-a603-4001-9538-4a751cf9b7c0\") " Oct 31 05:00:25.307713 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.307715 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/703c28bc-a603-4001-9538-4a751cf9b7c0-kube-api-access-xdfcg\" (UniqueName: \"kubernetes.io/projected/703c28bc-a603-4001-9538-4a751cf9b7c0-kube-api-access-xdfcg\") pod \"703c28bc-a603-4001-9538-4a751cf9b7c0\" (UID: \"703c28bc-a603-4001-9538-4a751cf9b7c0\") " Oct 31 05:00:25.308636 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.308604 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/703c28bc-a603-4001-9538-4a751cf9b7c0-maas-api-service-ca" pod "703c28bc-a603-4001-9538-4a751cf9b7c0" (UID: "703c28bc-a603-4001-9538-4a751cf9b7c0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:00:25.308781 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.308746 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/703c28bc-a603-4001-9538-4a751cf9b7c0-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:00:25.310571 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.310542 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/703c28bc-a603-4001-9538-4a751cf9b7c0-kube-api-access-xdfcg" pod "703c28bc-a603-4001-9538-4a751cf9b7c0" (UID: "703c28bc-a603-4001-9538-4a751cf9b7c0"). InnerVolumeSpecName "kube-api-access-xdfcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:00:25.312931 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.312906 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-108ad8-29890380-ng62l"] Oct 31 05:00:25.318613 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.318583 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-108ad8-29890380-ng62l"] Oct 31 05:00:25.412296 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.412226 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdfcg\" (UniqueName: \"kubernetes.io/projected/703c28bc-a603-4001-9538-4a751cf9b7c0-kube-api-access-xdfcg\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:00:25.429407 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.429370 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93bfb362-fcc2-4341-9881-1fd06b546590" path="/var/lib/kubelet/pods/93bfb362-fcc2-4341-9881-1fd06b546590/volumes" Oct 31 05:00:25.430080 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.430053 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96e0a71b-2eee-4760-ade1-5cdd604c4c81" path="/var/lib/kubelet/pods/96e0a71b-2eee-4760-ade1-5cdd604c4c81/volumes" Oct 31 05:00:25.430762 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.430738 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9790f69f-4de3-4837-a29f-17866661a3b3" path="/var/lib/kubelet/pods/9790f69f-4de3-4837-a29f-17866661a3b3/volumes" Oct 31 05:00:25.437500 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.437468 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="982ad40e-a719-4f61-be85-ea665503c299" path="/var/lib/kubelet/pods/982ad40e-a719-4f61-be85-ea665503c299/volumes" Oct 31 05:00:25.438865 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.438836 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9515dc7-4ea3-4a1b-aad8-1e78c7d0e13c" path="/var/lib/kubelet/pods/c9515dc7-4ea3-4a1b-aad8-1e78c7d0e13c/volumes" Oct 31 05:00:25.440127 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.440099 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0dacaaf-e177-420b-a953-f8cdb4961a27" path="/var/lib/kubelet/pods/d0dacaaf-e177-420b-a953-f8cdb4961a27/volumes" Oct 31 05:00:25.440793 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.440771 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d13131fa-5137-4401-aef8-d53fd8bd4e5d" path="/var/lib/kubelet/pods/d13131fa-5137-4401-aef8-d53fd8bd4e5d/volumes" Oct 31 05:00:25.441438 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.441416 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3d4f5cd-c755-4e4f-a86f-14874e597d08" path="/var/lib/kubelet/pods/d3d4f5cd-c755-4e4f-a86f-14874e597d08/volumes" Oct 31 05:00:25.442488 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.442468 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e27f22dd-e82f-493c-9745-6d901308b907" path="/var/lib/kubelet/pods/e27f22dd-e82f-493c-9745-6d901308b907/volumes" Oct 31 05:00:25.443118 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.443100 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e478c8f1-9e68-4f6f-b919-28ba397e66c4" path="/var/lib/kubelet/pods/e478c8f1-9e68-4f6f-b919-28ba397e66c4/volumes" Oct 31 05:00:25.443772 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.443752 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5259d89-590d-4fc7-b521-8f6feabbd2cc" path="/var/lib/kubelet/pods/e5259d89-590d-4fc7-b521-8f6feabbd2cc/volumes" Oct 31 05:00:25.531327 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.530500 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a04e4e-29890380-jgchh"] Oct 31 05:00:25.536083 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.536047 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a04e4e-29890380-jgchh"] Oct 31 05:00:25.670357 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.669942 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890380-kr424" Oct 31 05:00:25.670561 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.670322 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890380-kr424" event={"ID":"17f1de0f-3edd-4d07-910a-f6e6f356203c","Type":"ContainerDied","Data":"05a7a22244007878c09d7cc14463eec41b70e7257943325c6181b7ce1ed984a0"} Oct 31 05:00:25.670561 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.670441 5019 scope.go:122] "RemoveContainer" containerID="5cffd7d31f60cc876e89b630ab8ecd2cb44c150d01d9150f3d2299b5b4d58cf5" Oct 31 05:00:25.678155 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.678134 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-567ca6-29890380-kcwdg" Oct 31 05:00:25.678155 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.678145 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-567ca6-29890380-kcwdg" event={"ID":"703c28bc-a603-4001-9538-4a751cf9b7c0","Type":"ContainerDied","Data":"92244f9601f37d573523d7edf1d8d2b206678697a7b97e2e8f2b221670d94682"} Oct 31 05:00:25.681798 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.681770 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1daf30-29890380-pbtq2" Oct 31 05:00:25.681798 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.681784 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1daf30-29890380-pbtq2" event={"ID":"cd0723e3-dcf6-4907-9ce1-99ca9c77b144","Type":"ContainerDied","Data":"f03e20c3227a816c02d73feb9f768b67f8a51acbe0e03fcbdb7a0af9b5e3a30e"} Oct 31 05:00:25.686629 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.686606 5019 scope.go:122] "RemoveContainer" containerID="a02aa289de92c2867d117b29a5b931494288fb4a7dfd8b4c01326334413a4302" Oct 31 05:00:25.687193 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.687176 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-654741-29890380-85tdm" Oct 31 05:00:25.687296 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.687209 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-654741-29890380-85tdm" event={"ID":"e23416e6-954a-4ee6-8c4f-f6f8d7e1c506","Type":"ContainerDied","Data":"87da266a80fd1b2317935e7974c99f808cc9d0dfe6ad4664bb0a24a1dd5be3ba"} Oct 31 05:00:25.706863 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.706708 5019 scope.go:122] "RemoveContainer" containerID="83aa2aeb549688d462bb067fe5ddd79f2976af9fb9849bf81526b5a3679a984e" Oct 31 05:00:25.720129 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.720097 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-1daf30/test-model-1daf30-kserve-8686666dc7-xc4n6"] Oct 31 05:00:25.721568 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.721542 5019 cpu_manager.go:414] "containerMap: removing container" podUID="703c28bc-a603-4001-9538-4a751cf9b7c0" containerName="cleanup" Oct 31 05:00:25.721701 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.721685 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="703c28bc-a603-4001-9538-4a751cf9b7c0" containerName="cleanup" Oct 31 05:00:25.721794 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.721711 5019 cpu_manager.go:414] "containerMap: removing container" podUID="f59877aa-80e8-418d-9617-2f9bb9c8f3ef" containerName="cleanup" Oct 31 05:00:25.721794 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.721719 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f59877aa-80e8-418d-9617-2f9bb9c8f3ef" containerName="cleanup" Oct 31 05:00:25.721794 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.721731 5019 cpu_manager.go:414] "containerMap: removing container" podUID="703c28bc-a603-4001-9538-4a751cf9b7c0" containerName="cleanup" Oct 31 05:00:25.721794 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.721739 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="703c28bc-a603-4001-9538-4a751cf9b7c0" containerName="cleanup" Oct 31 05:00:25.721794 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.721762 5019 cpu_manager.go:414] "containerMap: removing container" podUID="4fd6b126-7a7c-4825-a374-5dec6fb94d06" containerName="cleanup" Oct 31 05:00:25.721794 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.721770 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fd6b126-7a7c-4825-a374-5dec6fb94d06" containerName="cleanup" Oct 31 05:00:25.721794 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.721780 5019 cpu_manager.go:414] "containerMap: removing container" podUID="17f1de0f-3edd-4d07-910a-f6e6f356203c" containerName="cleanup" Oct 31 05:00:25.721794 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.721787 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17f1de0f-3edd-4d07-910a-f6e6f356203c" containerName="cleanup" Oct 31 05:00:25.721794 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.721802 5019 cpu_manager.go:414] "containerMap: removing container" podUID="d3d4f5cd-c755-4e4f-a86f-14874e597d08" containerName="cleanup" Oct 31 05:00:25.722320 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.721810 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3d4f5cd-c755-4e4f-a86f-14874e597d08" containerName="cleanup" Oct 31 05:00:25.722320 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.721820 5019 cpu_manager.go:414] "containerMap: removing container" podUID="e478c8f1-9e68-4f6f-b919-28ba397e66c4" containerName="cleanup" Oct 31 05:00:25.722320 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.721828 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e478c8f1-9e68-4f6f-b919-28ba397e66c4" containerName="cleanup" Oct 31 05:00:25.722320 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.721844 5019 cpu_manager.go:414] "containerMap: removing container" podUID="e9b19f33-a012-48c9-b670-3b8eff773e49" containerName="cleanup" Oct 31 05:00:25.722320 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.721852 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9b19f33-a012-48c9-b670-3b8eff773e49" containerName="cleanup" Oct 31 05:00:25.722320 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.721868 5019 cpu_manager.go:414] "containerMap: removing container" podUID="93bfb362-fcc2-4341-9881-1fd06b546590" containerName="cleanup" Oct 31 05:00:25.722320 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.721875 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93bfb362-fcc2-4341-9881-1fd06b546590" containerName="cleanup" Oct 31 05:00:25.722320 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.721889 5019 cpu_manager.go:414] "containerMap: removing container" podUID="c9515dc7-4ea3-4a1b-aad8-1e78c7d0e13c" containerName="cleanup" Oct 31 05:00:25.722320 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.721896 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9515dc7-4ea3-4a1b-aad8-1e78c7d0e13c" containerName="cleanup" Oct 31 05:00:25.722320 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.721910 5019 cpu_manager.go:414] "containerMap: removing container" podUID="d13131fa-5137-4401-aef8-d53fd8bd4e5d" containerName="cleanup" Oct 31 05:00:25.722320 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.721918 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d13131fa-5137-4401-aef8-d53fd8bd4e5d" containerName="cleanup" Oct 31 05:00:25.722320 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.721933 5019 cpu_manager.go:414] "containerMap: removing container" podUID="e23416e6-954a-4ee6-8c4f-f6f8d7e1c506" containerName="cleanup" Oct 31 05:00:25.722320 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.721940 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e23416e6-954a-4ee6-8c4f-f6f8d7e1c506" containerName="cleanup" Oct 31 05:00:25.722320 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.721954 5019 cpu_manager.go:414] "containerMap: removing container" podUID="cd0723e3-dcf6-4907-9ce1-99ca9c77b144" containerName="cleanup" Oct 31 05:00:25.722320 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.721961 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd0723e3-dcf6-4907-9ce1-99ca9c77b144" containerName="cleanup" Oct 31 05:00:25.722320 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.721974 5019 cpu_manager.go:414] "containerMap: removing container" podUID="0b6b5e89-5900-4db2-8552-e4f029de7573" containerName="cleanup" Oct 31 05:00:25.722320 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.721981 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b6b5e89-5900-4db2-8552-e4f029de7573" containerName="cleanup" Oct 31 05:00:25.722320 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.721992 5019 cpu_manager.go:414] "containerMap: removing container" podUID="829dc86b-4ce1-427e-86db-cbaab8f15198" containerName="cleanup" Oct 31 05:00:25.722320 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.721999 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="829dc86b-4ce1-427e-86db-cbaab8f15198" containerName="cleanup" Oct 31 05:00:25.722320 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.722015 5019 cpu_manager.go:414] "containerMap: removing container" podUID="e9b19f33-a012-48c9-b670-3b8eff773e49" containerName="cleanup" Oct 31 05:00:25.722320 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.722023 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9b19f33-a012-48c9-b670-3b8eff773e49" containerName="cleanup" Oct 31 05:00:25.722320 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.722038 5019 cpu_manager.go:414] "containerMap: removing container" podUID="93bfb362-fcc2-4341-9881-1fd06b546590" containerName="cleanup" Oct 31 05:00:25.722320 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.722045 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93bfb362-fcc2-4341-9881-1fd06b546590" containerName="cleanup" Oct 31 05:00:25.722320 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.722065 5019 cpu_manager.go:414] "containerMap: removing container" podUID="e5259d89-590d-4fc7-b521-8f6feabbd2cc" containerName="cleanup" Oct 31 05:00:25.722320 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.722073 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5259d89-590d-4fc7-b521-8f6feabbd2cc" containerName="cleanup" Oct 31 05:00:25.722320 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.722083 5019 cpu_manager.go:414] "containerMap: removing container" podUID="e23416e6-954a-4ee6-8c4f-f6f8d7e1c506" containerName="cleanup" Oct 31 05:00:25.722320 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.722090 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e23416e6-954a-4ee6-8c4f-f6f8d7e1c506" containerName="cleanup" Oct 31 05:00:25.722320 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.722105 5019 cpu_manager.go:414] "containerMap: removing container" podUID="96e0a71b-2eee-4760-ade1-5cdd604c4c81" containerName="cleanup" Oct 31 05:00:25.722320 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.722112 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96e0a71b-2eee-4760-ade1-5cdd604c4c81" containerName="cleanup" Oct 31 05:00:25.722320 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.722130 5019 cpu_manager.go:414] "containerMap: removing container" podUID="c9515dc7-4ea3-4a1b-aad8-1e78c7d0e13c" containerName="cleanup" Oct 31 05:00:25.722320 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.722138 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9515dc7-4ea3-4a1b-aad8-1e78c7d0e13c" containerName="cleanup" Oct 31 05:00:25.722320 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.722153 5019 cpu_manager.go:414] "containerMap: removing container" podUID="0b6b5e89-5900-4db2-8552-e4f029de7573" containerName="cleanup" Oct 31 05:00:25.722320 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.722160 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b6b5e89-5900-4db2-8552-e4f029de7573" containerName="cleanup" Oct 31 05:00:25.722320 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.722175 5019 cpu_manager.go:414] "containerMap: removing container" podUID="e23416e6-954a-4ee6-8c4f-f6f8d7e1c506" containerName="cleanup" Oct 31 05:00:25.722320 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.722182 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e23416e6-954a-4ee6-8c4f-f6f8d7e1c506" containerName="cleanup" Oct 31 05:00:25.722320 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.722201 5019 cpu_manager.go:414] "containerMap: removing container" podUID="f59877aa-80e8-418d-9617-2f9bb9c8f3ef" containerName="cleanup" Oct 31 05:00:25.722320 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.722207 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f59877aa-80e8-418d-9617-2f9bb9c8f3ef" containerName="cleanup" Oct 31 05:00:25.722320 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.722218 5019 cpu_manager.go:414] "containerMap: removing container" podUID="d3d4f5cd-c755-4e4f-a86f-14874e597d08" containerName="cleanup" Oct 31 05:00:25.722320 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.722225 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3d4f5cd-c755-4e4f-a86f-14874e597d08" containerName="cleanup" Oct 31 05:00:25.722320 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.722237 5019 cpu_manager.go:414] "containerMap: removing container" podUID="e5259d89-590d-4fc7-b521-8f6feabbd2cc" containerName="cleanup" Oct 31 05:00:25.722320 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.722244 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5259d89-590d-4fc7-b521-8f6feabbd2cc" containerName="cleanup" Oct 31 05:00:25.722320 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.722255 5019 cpu_manager.go:414] "containerMap: removing container" podUID="829dc86b-4ce1-427e-86db-cbaab8f15198" containerName="cleanup" Oct 31 05:00:25.722320 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.722262 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="829dc86b-4ce1-427e-86db-cbaab8f15198" containerName="cleanup" Oct 31 05:00:25.722320 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.722276 5019 cpu_manager.go:414] "containerMap: removing container" podUID="93bfb362-fcc2-4341-9881-1fd06b546590" containerName="cleanup" Oct 31 05:00:25.722320 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.722284 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93bfb362-fcc2-4341-9881-1fd06b546590" containerName="cleanup" Oct 31 05:00:25.722320 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.722295 5019 cpu_manager.go:414] "containerMap: removing container" podUID="d13131fa-5137-4401-aef8-d53fd8bd4e5d" containerName="cleanup" Oct 31 05:00:25.722320 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.722302 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d13131fa-5137-4401-aef8-d53fd8bd4e5d" containerName="cleanup" Oct 31 05:00:25.722320 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.722314 5019 cpu_manager.go:414] "containerMap: removing container" podUID="e478c8f1-9e68-4f6f-b919-28ba397e66c4" containerName="cleanup" Oct 31 05:00:25.722320 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.722321 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e478c8f1-9e68-4f6f-b919-28ba397e66c4" containerName="cleanup" Oct 31 05:00:25.729088 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.722564 5019 cpu_manager.go:414] "containerMap: removing container" podUID="e1e9a647-0e9e-4528-a5a6-7022b244f1ba" containerName="cleanup" Oct 31 05:00:25.729088 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.722578 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1e9a647-0e9e-4528-a5a6-7022b244f1ba" containerName="cleanup" Oct 31 05:00:25.729088 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.722592 5019 cpu_manager.go:414] "containerMap: removing container" podUID="9790f69f-4de3-4837-a29f-17866661a3b3" containerName="cleanup" Oct 31 05:00:25.729088 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.722598 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9790f69f-4de3-4837-a29f-17866661a3b3" containerName="cleanup" Oct 31 05:00:25.729088 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.722608 5019 cpu_manager.go:414] "containerMap: removing container" podUID="e478c8f1-9e68-4f6f-b919-28ba397e66c4" containerName="cleanup" Oct 31 05:00:25.729088 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.722614 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e478c8f1-9e68-4f6f-b919-28ba397e66c4" containerName="cleanup" Oct 31 05:00:25.729088 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.722632 5019 cpu_manager.go:414] "containerMap: removing container" podUID="e27f22dd-e82f-493c-9745-6d901308b907" containerName="cleanup" Oct 31 05:00:25.729088 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.722638 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e27f22dd-e82f-493c-9745-6d901308b907" containerName="cleanup" Oct 31 05:00:25.729088 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.722650 5019 cpu_manager.go:414] "containerMap: removing container" podUID="96e0a71b-2eee-4760-ade1-5cdd604c4c81" containerName="cleanup" Oct 31 05:00:25.729088 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.722655 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96e0a71b-2eee-4760-ade1-5cdd604c4c81" containerName="cleanup" Oct 31 05:00:25.729088 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.722665 5019 cpu_manager.go:414] "containerMap: removing container" podUID="d0dacaaf-e177-420b-a953-f8cdb4961a27" containerName="cleanup" Oct 31 05:00:25.729088 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.722671 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0dacaaf-e177-420b-a953-f8cdb4961a27" containerName="cleanup" Oct 31 05:00:25.734671 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.734607 5019 cpu_manager.go:414] "containerMap: removing container" podUID="9790f69f-4de3-4837-a29f-17866661a3b3" containerName="cleanup" Oct 31 05:00:25.734671 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.734629 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9790f69f-4de3-4837-a29f-17866661a3b3" containerName="cleanup" Oct 31 05:00:25.734988 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.734954 5019 cpu_manager.go:414] "containerMap: removing container" podUID="d0dacaaf-e177-420b-a953-f8cdb4961a27" containerName="cleanup" Oct 31 05:00:25.735143 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.735112 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0dacaaf-e177-420b-a953-f8cdb4961a27" containerName="cleanup" Oct 31 05:00:25.735310 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.735267 5019 cpu_manager.go:414] "containerMap: removing container" podUID="d0dacaaf-e177-420b-a953-f8cdb4961a27" containerName="cleanup" Oct 31 05:00:25.735558 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.735283 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0dacaaf-e177-420b-a953-f8cdb4961a27" containerName="cleanup" Oct 31 05:00:25.735558 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.735523 5019 cpu_manager.go:414] "containerMap: removing container" podUID="4fd6b126-7a7c-4825-a374-5dec6fb94d06" containerName="cleanup" Oct 31 05:00:25.735789 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.735736 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fd6b126-7a7c-4825-a374-5dec6fb94d06" containerName="cleanup" Oct 31 05:00:25.735789 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.735764 5019 cpu_manager.go:414] "containerMap: removing container" podUID="9790f69f-4de3-4837-a29f-17866661a3b3" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.735772 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9790f69f-4de3-4837-a29f-17866661a3b3" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.737318 5019 cpu_manager.go:414] "containerMap: removing container" podUID="982ad40e-a719-4f61-be85-ea665503c299" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.737340 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="982ad40e-a719-4f61-be85-ea665503c299" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.737361 5019 cpu_manager.go:414] "containerMap: removing container" podUID="e1e9a647-0e9e-4528-a5a6-7022b244f1ba" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.737369 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1e9a647-0e9e-4528-a5a6-7022b244f1ba" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.737383 5019 cpu_manager.go:414] "containerMap: removing container" podUID="96e0a71b-2eee-4760-ade1-5cdd604c4c81" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.737390 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96e0a71b-2eee-4760-ade1-5cdd604c4c81" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.737401 5019 cpu_manager.go:414] "containerMap: removing container" podUID="e5259d89-590d-4fc7-b521-8f6feabbd2cc" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.737409 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5259d89-590d-4fc7-b521-8f6feabbd2cc" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.737430 5019 cpu_manager.go:414] "containerMap: removing container" podUID="e27f22dd-e82f-493c-9745-6d901308b907" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.737438 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e27f22dd-e82f-493c-9745-6d901308b907" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.737462 5019 cpu_manager.go:414] "containerMap: removing container" podUID="982ad40e-a719-4f61-be85-ea665503c299" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.737470 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="982ad40e-a719-4f61-be85-ea665503c299" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.737501 5019 cpu_manager.go:414] "containerMap: removing container" podUID="cd0723e3-dcf6-4907-9ce1-99ca9c77b144" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.737507 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd0723e3-dcf6-4907-9ce1-99ca9c77b144" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.737516 5019 cpu_manager.go:414] "containerMap: removing container" podUID="17f1de0f-3edd-4d07-910a-f6e6f356203c" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.737521 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17f1de0f-3edd-4d07-910a-f6e6f356203c" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.737770 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b6b5e89-5900-4db2-8552-e4f029de7573" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.737784 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="829dc86b-4ce1-427e-86db-cbaab8f15198" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.737795 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9b19f33-a012-48c9-b670-3b8eff773e49" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.737811 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="93bfb362-fcc2-4341-9881-1fd06b546590" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.737825 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5259d89-590d-4fc7-b521-8f6feabbd2cc" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.737836 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e23416e6-954a-4ee6-8c4f-f6f8d7e1c506" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.737848 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="96e0a71b-2eee-4760-ade1-5cdd604c4c81" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.737858 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9515dc7-4ea3-4a1b-aad8-1e78c7d0e13c" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.737868 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e23416e6-954a-4ee6-8c4f-f6f8d7e1c506" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.737880 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="f59877aa-80e8-418d-9617-2f9bb9c8f3ef" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.737886 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3d4f5cd-c755-4e4f-a86f-14874e597d08" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.737892 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5259d89-590d-4fc7-b521-8f6feabbd2cc" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.737897 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="829dc86b-4ce1-427e-86db-cbaab8f15198" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.737902 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="93bfb362-fcc2-4341-9881-1fd06b546590" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.737911 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="d13131fa-5137-4401-aef8-d53fd8bd4e5d" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.737919 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e478c8f1-9e68-4f6f-b919-28ba397e66c4" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.737929 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1e9a647-0e9e-4528-a5a6-7022b244f1ba" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.737937 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="9790f69f-4de3-4837-a29f-17866661a3b3" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.737944 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e478c8f1-9e68-4f6f-b919-28ba397e66c4" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.737956 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b6b5e89-5900-4db2-8552-e4f029de7573" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.737964 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e27f22dd-e82f-493c-9745-6d901308b907" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.737970 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="96e0a71b-2eee-4760-ade1-5cdd604c4c81" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.737978 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0dacaaf-e177-420b-a953-f8cdb4961a27" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.737984 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="703c28bc-a603-4001-9538-4a751cf9b7c0" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.737990 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="9790f69f-4de3-4837-a29f-17866661a3b3" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.737997 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0dacaaf-e177-420b-a953-f8cdb4961a27" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.738004 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0dacaaf-e177-420b-a953-f8cdb4961a27" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.738012 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fd6b126-7a7c-4825-a374-5dec6fb94d06" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.738020 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="f59877aa-80e8-418d-9617-2f9bb9c8f3ef" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.738029 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="9790f69f-4de3-4837-a29f-17866661a3b3" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.738038 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="982ad40e-a719-4f61-be85-ea665503c299" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.738047 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1e9a647-0e9e-4528-a5a6-7022b244f1ba" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.738061 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3d4f5cd-c755-4e4f-a86f-14874e597d08" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.738068 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="982ad40e-a719-4f61-be85-ea665503c299" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.738075 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e27f22dd-e82f-493c-9745-6d901308b907" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.738082 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="d13131fa-5137-4401-aef8-d53fd8bd4e5d" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.738094 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd0723e3-dcf6-4907-9ce1-99ca9c77b144" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.738102 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="982ad40e-a719-4f61-be85-ea665503c299" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.738110 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fd6b126-7a7c-4825-a374-5dec6fb94d06" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.738117 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd0723e3-dcf6-4907-9ce1-99ca9c77b144" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.738126 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="17f1de0f-3edd-4d07-910a-f6e6f356203c" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.738137 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="17f1de0f-3edd-4d07-910a-f6e6f356203c" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.738146 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="f59877aa-80e8-418d-9617-2f9bb9c8f3ef" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.738154 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="703c28bc-a603-4001-9538-4a751cf9b7c0" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.738161 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fd6b126-7a7c-4825-a374-5dec6fb94d06" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.738166 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="17f1de0f-3edd-4d07-910a-f6e6f356203c" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.738173 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3d4f5cd-c755-4e4f-a86f-14874e597d08" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.738179 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9b19f33-a012-48c9-b670-3b8eff773e49" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.738186 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9515dc7-4ea3-4a1b-aad8-1e78c7d0e13c" containerName="cleanup" Oct 31 05:00:25.739645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.738193 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="d13131fa-5137-4401-aef8-d53fd8bd4e5d" containerName="cleanup" Oct 31 05:00:25.746019 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.745766 5019 scope.go:122] "RemoveContainer" containerID="679b7f9e6e2b4671b86e674c6f455972228023212f1f103ba7a1e00dcdc554be" Oct 31 05:00:25.766695 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.761496 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1daf30/test-model-1daf30-kserve-8686666dc7-xc4n6" Oct 31 05:00:25.788491 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.788117 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1daf30/test-model-1daf30-kserve-8686666dc7-xc4n6"] Oct 31 05:00:25.795229 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.793687 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1daf30\"/\"kube-root-ca.crt\"" Oct 31 05:00:25.795229 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.793794 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1daf30\"/\"openshift-service-ca.crt\"" Oct 31 05:00:25.795229 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.793887 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1daf30\"/\"test-model-1daf30-kserve-self-signed-certs\"" Oct 31 05:00:25.795229 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.793980 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1daf30\"/\"default-dockercfg-6zqhf\"" Oct 31 05:00:25.811711 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.811653 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890380-kr424"] Oct 31 05:00:25.819718 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.819042 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890380-kr424"] Oct 31 05:00:25.830260 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.830236 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-567ca6-29890380-kcwdg"] Oct 31 05:00:25.842583 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.842564 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-567ca6-29890380-kcwdg"] Oct 31 05:00:25.964370 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:25.956967 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7-model-cache\") pod \"test-model-1daf30-kserve-8686666dc7-xc4n6\" (UID: \"f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7\") " pod="ai-tenant-e2e-inf-a-1daf30/test-model-1daf30-kserve-8686666dc7-xc4n6" Oct 31 05:00:26.192556 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.192473 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7-home\") pod \"test-model-1daf30-kserve-8686666dc7-xc4n6\" (UID: \"f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7\") " pod="ai-tenant-e2e-inf-a-1daf30/test-model-1daf30-kserve-8686666dc7-xc4n6" Oct 31 05:00:26.192920 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.192895 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7-tmp-dir\") pod \"test-model-1daf30-kserve-8686666dc7-xc4n6\" (UID: \"f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7\") " pod="ai-tenant-e2e-inf-a-1daf30/test-model-1daf30-kserve-8686666dc7-xc4n6" Oct 31 05:00:26.193209 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.193190 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgb62\" (UniqueName: \"kubernetes.io/projected/f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7-kube-api-access-kgb62\") pod \"test-model-1daf30-kserve-8686666dc7-xc4n6\" (UID: \"f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7\") " pod="ai-tenant-e2e-inf-a-1daf30/test-model-1daf30-kserve-8686666dc7-xc4n6" Oct 31 05:00:26.193391 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.193377 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7-tls-certs\") pod \"test-model-1daf30-kserve-8686666dc7-xc4n6\" (UID: \"f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7\") " pod="ai-tenant-e2e-inf-a-1daf30/test-model-1daf30-kserve-8686666dc7-xc4n6" Oct 31 05:00:26.193655 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.193639 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7-dshm\") pod \"test-model-1daf30-kserve-8686666dc7-xc4n6\" (UID: \"f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7\") " pod="ai-tenant-e2e-inf-a-1daf30/test-model-1daf30-kserve-8686666dc7-xc4n6" Oct 31 05:00:26.198120 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.198097 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1daf30-29890380-pbtq2"] Oct 31 05:00:26.206427 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.206376 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1daf30-29890380-pbtq2"] Oct 31 05:00:26.233929 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.233903 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6eba4874-6r5zg"] Oct 31 05:00:26.246477 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.246374 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6eba4874-6r5zg"] Oct 31 05:00:26.263430 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.263400 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-654741-29890380-85tdm"] Oct 31 05:00:26.266513 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.266489 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-654741-29890380-85tdm"] Oct 31 05:00:26.283239 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.283196 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890380-k8lq8" Oct 31 05:00:26.286784 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.286759 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-47e9af-29890380-jk7b6" Oct 31 05:00:26.291098 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.291075 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890380-kz5pd" Oct 31 05:00:26.294913 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.294889 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7-home\") pod \"test-model-1daf30-kserve-8686666dc7-xc4n6\" (UID: \"f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7\") " pod="ai-tenant-e2e-inf-a-1daf30/test-model-1daf30-kserve-8686666dc7-xc4n6" Oct 31 05:00:26.295013 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.294987 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7-tmp-dir\") pod \"test-model-1daf30-kserve-8686666dc7-xc4n6\" (UID: \"f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7\") " pod="ai-tenant-e2e-inf-a-1daf30/test-model-1daf30-kserve-8686666dc7-xc4n6" Oct 31 05:00:26.295074 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.295033 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgb62\" (UniqueName: \"kubernetes.io/projected/f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7-kube-api-access-kgb62\") pod \"test-model-1daf30-kserve-8686666dc7-xc4n6\" (UID: \"f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7\") " pod="ai-tenant-e2e-inf-a-1daf30/test-model-1daf30-kserve-8686666dc7-xc4n6" Oct 31 05:00:26.295148 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.295078 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7-tls-certs\") pod \"test-model-1daf30-kserve-8686666dc7-xc4n6\" (UID: \"f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7\") " pod="ai-tenant-e2e-inf-a-1daf30/test-model-1daf30-kserve-8686666dc7-xc4n6" Oct 31 05:00:26.295216 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.295191 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7-dshm\") pod \"test-model-1daf30-kserve-8686666dc7-xc4n6\" (UID: \"f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7\") " pod="ai-tenant-e2e-inf-a-1daf30/test-model-1daf30-kserve-8686666dc7-xc4n6" Oct 31 05:00:26.295366 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.295346 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7-home\") pod \"test-model-1daf30-kserve-8686666dc7-xc4n6\" (UID: \"f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7\") " pod="ai-tenant-e2e-inf-a-1daf30/test-model-1daf30-kserve-8686666dc7-xc4n6" Oct 31 05:00:26.295460 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.295359 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7-tmp-dir\") pod \"test-model-1daf30-kserve-8686666dc7-xc4n6\" (UID: \"f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7\") " pod="ai-tenant-e2e-inf-a-1daf30/test-model-1daf30-kserve-8686666dc7-xc4n6" Oct 31 05:00:26.295460 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.295391 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7-model-cache\") pod \"test-model-1daf30-kserve-8686666dc7-xc4n6\" (UID: \"f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7\") " pod="ai-tenant-e2e-inf-a-1daf30/test-model-1daf30-kserve-8686666dc7-xc4n6" Oct 31 05:00:26.295664 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.295642 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7-model-cache\") pod \"test-model-1daf30-kserve-8686666dc7-xc4n6\" (UID: \"f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7\") " pod="ai-tenant-e2e-inf-a-1daf30/test-model-1daf30-kserve-8686666dc7-xc4n6" Oct 31 05:00:26.297524 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.297494 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7-dshm\") pod \"test-model-1daf30-kserve-8686666dc7-xc4n6\" (UID: \"f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7\") " pod="ai-tenant-e2e-inf-a-1daf30/test-model-1daf30-kserve-8686666dc7-xc4n6" Oct 31 05:00:26.297955 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.297935 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7-tls-certs\") pod \"test-model-1daf30-kserve-8686666dc7-xc4n6\" (UID: \"f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7\") " pod="ai-tenant-e2e-inf-a-1daf30/test-model-1daf30-kserve-8686666dc7-xc4n6" Oct 31 05:00:26.317860 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.317835 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgb62\" (UniqueName: \"kubernetes.io/projected/f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7-kube-api-access-kgb62\") pod \"test-model-1daf30-kserve-8686666dc7-xc4n6\" (UID: \"f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7\") " pod="ai-tenant-e2e-inf-a-1daf30/test-model-1daf30-kserve-8686666dc7-xc4n6" Oct 31 05:00:26.396261 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.396224 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/deaadd33-dcef-43df-a66e-9fd41434de6f-kube-api-access-hmzqt\" (UniqueName: \"kubernetes.io/projected/deaadd33-dcef-43df-a66e-9fd41434de6f-kube-api-access-hmzqt\") pod \"deaadd33-dcef-43df-a66e-9fd41434de6f\" (UID: \"deaadd33-dcef-43df-a66e-9fd41434de6f\") " Oct 31 05:00:26.396521 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.396500 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c20d99d-7ce4-4bb8-a4d9-188cec401d75-kube-api-access-jmts9\" (UniqueName: \"kubernetes.io/projected/7c20d99d-7ce4-4bb8-a4d9-188cec401d75-kube-api-access-jmts9\") pod \"7c20d99d-7ce4-4bb8-a4d9-188cec401d75\" (UID: \"7c20d99d-7ce4-4bb8-a4d9-188cec401d75\") " Oct 31 05:00:26.396604 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.396552 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/deaadd33-dcef-43df-a66e-9fd41434de6f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/deaadd33-dcef-43df-a66e-9fd41434de6f-maas-api-service-ca\") pod \"deaadd33-dcef-43df-a66e-9fd41434de6f\" (UID: \"deaadd33-dcef-43df-a66e-9fd41434de6f\") " Oct 31 05:00:26.396662 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.396646 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c20d99d-7ce4-4bb8-a4d9-188cec401d75-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c20d99d-7ce4-4bb8-a4d9-188cec401d75-maas-api-service-ca\") pod \"7c20d99d-7ce4-4bb8-a4d9-188cec401d75\" (UID: \"7c20d99d-7ce4-4bb8-a4d9-188cec401d75\") " Oct 31 05:00:26.396720 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.396687 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ab61246-5d2d-4a2a-8c42-fff22960018f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ab61246-5d2d-4a2a-8c42-fff22960018f-maas-api-service-ca\") pod \"7ab61246-5d2d-4a2a-8c42-fff22960018f\" (UID: \"7ab61246-5d2d-4a2a-8c42-fff22960018f\") " Oct 31 05:00:26.396931 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.396911 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ab61246-5d2d-4a2a-8c42-fff22960018f-kube-api-access-sm2s5\" (UniqueName: \"kubernetes.io/projected/7ab61246-5d2d-4a2a-8c42-fff22960018f-kube-api-access-sm2s5\") pod \"7ab61246-5d2d-4a2a-8c42-fff22960018f\" (UID: \"7ab61246-5d2d-4a2a-8c42-fff22960018f\") " Oct 31 05:00:26.397047 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.396961 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deaadd33-dcef-43df-a66e-9fd41434de6f-maas-api-service-ca" pod "deaadd33-dcef-43df-a66e-9fd41434de6f" (UID: "deaadd33-dcef-43df-a66e-9fd41434de6f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:00:26.397325 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.397295 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c20d99d-7ce4-4bb8-a4d9-188cec401d75-maas-api-service-ca" pod "7c20d99d-7ce4-4bb8-a4d9-188cec401d75" (UID: "7c20d99d-7ce4-4bb8-a4d9-188cec401d75"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:00:26.398228 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.397716 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/deaadd33-dcef-43df-a66e-9fd41434de6f-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:00:26.398228 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.397742 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c20d99d-7ce4-4bb8-a4d9-188cec401d75-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:00:26.398228 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.397920 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ab61246-5d2d-4a2a-8c42-fff22960018f-maas-api-service-ca" pod "7ab61246-5d2d-4a2a-8c42-fff22960018f" (UID: "7ab61246-5d2d-4a2a-8c42-fff22960018f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:00:26.398745 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.398724 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c20d99d-7ce4-4bb8-a4d9-188cec401d75-kube-api-access-jmts9" pod "7c20d99d-7ce4-4bb8-a4d9-188cec401d75" (UID: "7c20d99d-7ce4-4bb8-a4d9-188cec401d75"). InnerVolumeSpecName "kube-api-access-jmts9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:00:26.399892 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.399870 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1daf30/test-model-1daf30-kserve-8686666dc7-xc4n6" Oct 31 05:00:26.400157 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.400135 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ab61246-5d2d-4a2a-8c42-fff22960018f-kube-api-access-sm2s5" pod "7ab61246-5d2d-4a2a-8c42-fff22960018f" (UID: "7ab61246-5d2d-4a2a-8c42-fff22960018f"). InnerVolumeSpecName "kube-api-access-sm2s5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:00:26.402023 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.401996 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deaadd33-dcef-43df-a66e-9fd41434de6f-kube-api-access-hmzqt" pod "deaadd33-dcef-43df-a66e-9fd41434de6f" (UID: "deaadd33-dcef-43df-a66e-9fd41434de6f"). InnerVolumeSpecName "kube-api-access-hmzqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:00:26.499999 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.499956 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmts9\" (UniqueName: \"kubernetes.io/projected/7c20d99d-7ce4-4bb8-a4d9-188cec401d75-kube-api-access-jmts9\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:00:26.499999 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.499993 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ab61246-5d2d-4a2a-8c42-fff22960018f-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:00:26.500573 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.500009 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sm2s5\" (UniqueName: \"kubernetes.io/projected/7ab61246-5d2d-4a2a-8c42-fff22960018f-kube-api-access-sm2s5\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:00:26.500573 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.500024 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmzqt\" (UniqueName: \"kubernetes.io/projected/deaadd33-dcef-43df-a66e-9fd41434de6f-kube-api-access-hmzqt\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:00:26.712098 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.712067 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890380-kz5pd" Oct 31 05:00:26.712267 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.712096 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890380-kz5pd" event={"ID":"7ab61246-5d2d-4a2a-8c42-fff22960018f","Type":"ContainerDied","Data":"73d6c1cc82c1a129fd9c34184d5357a1698ad3a659995ae9972d4409f1f3e716"} Oct 31 05:00:26.712267 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.712174 5019 scope.go:122] "RemoveContainer" containerID="83c8db674b9c8ef30fcca9a937550888c5dfefd8c017ee58f6ae27cc6b00cd58" Oct 31 05:00:26.717132 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.716841 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890380-k8lq8" Oct 31 05:00:26.717132 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.716868 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890380-k8lq8" event={"ID":"deaadd33-dcef-43df-a66e-9fd41434de6f","Type":"ContainerDied","Data":"3ada75c9613f21a6928f73ea2d1d809bdb3681a50ffacdb9fb3db38ff3241efd"} Oct 31 05:00:26.720063 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.720033 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-47e9af-29890380-jk7b6" event={"ID":"7c20d99d-7ce4-4bb8-a4d9-188cec401d75","Type":"ContainerDied","Data":"d0abf60006040dd3a67b940994c050cccddc563c0bc480d49b5f6f274453e493"} Oct 31 05:00:26.720201 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.720137 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-47e9af-29890380-jk7b6" Oct 31 05:00:26.726133 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.726114 5019 scope.go:122] "RemoveContainer" containerID="cab9f0b3ba1b2b9d5c1d88af25038fa7bc10e63c4184cb680d2242ffea84ecfc" Oct 31 05:00:26.737924 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.737904 5019 scope.go:122] "RemoveContainer" containerID="2a7402fe5081d2bb226a52cd353f662389c29dce54cac1d01b401b71e95f6a2d" Oct 31 05:00:26.749690 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.749660 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1daf30/test-model-1daf30-kserve-8686666dc7-xc4n6"] Oct 31 05:00:26.752038 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:00:26.751982 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1d29fd3_0e84_4e08_8522_ba6ae2d79ef7.slice/crio-8d674b03a2ced65525cb16a198e65c14d8dbd8ac1649cd982c2691eedf0f5fe0 WatchSource:0}: Error finding container 8d674b03a2ced65525cb16a198e65c14d8dbd8ac1649cd982c2691eedf0f5fe0: Status 404 returned error can't find the container with id 8d674b03a2ced65525cb16a198e65c14d8dbd8ac1649cd982c2691eedf0f5fe0 Oct 31 05:00:26.789376 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.789317 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890380-kz5pd"] Oct 31 05:00:26.795721 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.795682 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890380-kz5pd"] Oct 31 05:00:26.845202 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.845170 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-47e9af-29890380-jk7b6"] Oct 31 05:00:26.850431 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.850402 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-47e9af-29890380-jk7b6"] Oct 31 05:00:26.887860 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.887814 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890380-k8lq8"] Oct 31 05:00:26.896367 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:26.896317 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890380-k8lq8"] Oct 31 05:00:27.038664 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.038571 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-12c0bc/test-model-12c0bc-kserve-66b5ccb945-4sg7l"] Oct 31 05:00:27.039967 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.039940 5019 cpu_manager.go:414] "containerMap: removing container" podUID="deaadd33-dcef-43df-a66e-9fd41434de6f" containerName="cleanup" Oct 31 05:00:27.039967 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.039968 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deaadd33-dcef-43df-a66e-9fd41434de6f" containerName="cleanup" Oct 31 05:00:27.040113 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.039985 5019 cpu_manager.go:414] "containerMap: removing container" podUID="cd0723e3-dcf6-4907-9ce1-99ca9c77b144" containerName="cleanup" Oct 31 05:00:27.040113 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.039993 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd0723e3-dcf6-4907-9ce1-99ca9c77b144" containerName="cleanup" Oct 31 05:00:27.040113 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.040017 5019 cpu_manager.go:414] "containerMap: removing container" podUID="e9b19f33-a012-48c9-b670-3b8eff773e49" containerName="cleanup" Oct 31 05:00:27.040113 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.040024 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9b19f33-a012-48c9-b670-3b8eff773e49" containerName="cleanup" Oct 31 05:00:27.040113 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.040048 5019 cpu_manager.go:414] "containerMap: removing container" podUID="7ab61246-5d2d-4a2a-8c42-fff22960018f" containerName="cleanup" Oct 31 05:00:27.040113 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.040055 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ab61246-5d2d-4a2a-8c42-fff22960018f" containerName="cleanup" Oct 31 05:00:27.040113 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.040068 5019 cpu_manager.go:414] "containerMap: removing container" podUID="c9515dc7-4ea3-4a1b-aad8-1e78c7d0e13c" containerName="cleanup" Oct 31 05:00:27.040113 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.040076 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9515dc7-4ea3-4a1b-aad8-1e78c7d0e13c" containerName="cleanup" Oct 31 05:00:27.040113 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.040094 5019 cpu_manager.go:414] "containerMap: removing container" podUID="0b6b5e89-5900-4db2-8552-e4f029de7573" containerName="cleanup" Oct 31 05:00:27.040113 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.040101 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b6b5e89-5900-4db2-8552-e4f029de7573" containerName="cleanup" Oct 31 05:00:27.040537 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.040120 5019 cpu_manager.go:414] "containerMap: removing container" podUID="f59877aa-80e8-418d-9617-2f9bb9c8f3ef" containerName="cleanup" Oct 31 05:00:27.040537 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.040128 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f59877aa-80e8-418d-9617-2f9bb9c8f3ef" containerName="cleanup" Oct 31 05:00:27.040537 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.040138 5019 cpu_manager.go:414] "containerMap: removing container" podUID="d3d4f5cd-c755-4e4f-a86f-14874e597d08" containerName="cleanup" Oct 31 05:00:27.040537 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.040145 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3d4f5cd-c755-4e4f-a86f-14874e597d08" containerName="cleanup" Oct 31 05:00:27.040537 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.040155 5019 cpu_manager.go:414] "containerMap: removing container" podUID="829dc86b-4ce1-427e-86db-cbaab8f15198" containerName="cleanup" Oct 31 05:00:27.040537 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.040163 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="829dc86b-4ce1-427e-86db-cbaab8f15198" containerName="cleanup" Oct 31 05:00:27.040537 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.040175 5019 cpu_manager.go:414] "containerMap: removing container" podUID="d13131fa-5137-4401-aef8-d53fd8bd4e5d" containerName="cleanup" Oct 31 05:00:27.040537 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.040183 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d13131fa-5137-4401-aef8-d53fd8bd4e5d" containerName="cleanup" Oct 31 05:00:27.040537 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.040209 5019 cpu_manager.go:414] "containerMap: removing container" podUID="7ab61246-5d2d-4a2a-8c42-fff22960018f" containerName="cleanup" Oct 31 05:00:27.040537 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.040216 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ab61246-5d2d-4a2a-8c42-fff22960018f" containerName="cleanup" Oct 31 05:00:27.040537 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.040240 5019 cpu_manager.go:414] "containerMap: removing container" podUID="7c20d99d-7ce4-4bb8-a4d9-188cec401d75" containerName="cleanup" Oct 31 05:00:27.040537 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.040247 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c20d99d-7ce4-4bb8-a4d9-188cec401d75" containerName="cleanup" Oct 31 05:00:27.040537 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.040263 5019 cpu_manager.go:414] "containerMap: removing container" podUID="e1e9a647-0e9e-4528-a5a6-7022b244f1ba" containerName="cleanup" Oct 31 05:00:27.040537 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.040270 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1e9a647-0e9e-4528-a5a6-7022b244f1ba" containerName="cleanup" Oct 31 05:00:27.040537 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.040290 5019 cpu_manager.go:414] "containerMap: removing container" podUID="e27f22dd-e82f-493c-9745-6d901308b907" containerName="cleanup" Oct 31 05:00:27.040537 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.040296 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e27f22dd-e82f-493c-9745-6d901308b907" containerName="cleanup" Oct 31 05:00:27.040537 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.040311 5019 cpu_manager.go:414] "containerMap: removing container" podUID="deaadd33-dcef-43df-a66e-9fd41434de6f" containerName="cleanup" Oct 31 05:00:27.040537 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.040318 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deaadd33-dcef-43df-a66e-9fd41434de6f" containerName="cleanup" Oct 31 05:00:27.040537 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.040345 5019 cpu_manager.go:414] "containerMap: removing container" podUID="982ad40e-a719-4f61-be85-ea665503c299" containerName="cleanup" Oct 31 05:00:27.040537 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.040353 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="982ad40e-a719-4f61-be85-ea665503c299" containerName="cleanup" Oct 31 05:00:27.040537 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.040392 5019 cpu_manager.go:414] "containerMap: removing container" podUID="703c28bc-a603-4001-9538-4a751cf9b7c0" containerName="cleanup" Oct 31 05:00:27.040537 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.040399 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="703c28bc-a603-4001-9538-4a751cf9b7c0" containerName="cleanup" Oct 31 05:00:27.040537 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.040414 5019 cpu_manager.go:414] "containerMap: removing container" podUID="4fd6b126-7a7c-4825-a374-5dec6fb94d06" containerName="cleanup" Oct 31 05:00:27.040537 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.040421 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fd6b126-7a7c-4825-a374-5dec6fb94d06" containerName="cleanup" Oct 31 05:00:27.040537 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.040433 5019 cpu_manager.go:414] "containerMap: removing container" podUID="17f1de0f-3edd-4d07-910a-f6e6f356203c" containerName="cleanup" Oct 31 05:00:27.040537 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.040440 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17f1de0f-3edd-4d07-910a-f6e6f356203c" containerName="cleanup" Oct 31 05:00:27.040537 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.040456 5019 cpu_manager.go:414] "containerMap: removing container" podUID="7c20d99d-7ce4-4bb8-a4d9-188cec401d75" containerName="cleanup" Oct 31 05:00:27.040537 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.040463 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c20d99d-7ce4-4bb8-a4d9-188cec401d75" containerName="cleanup" Oct 31 05:00:27.041522 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.040740 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1e9a647-0e9e-4528-a5a6-7022b244f1ba" containerName="cleanup" Oct 31 05:00:27.041522 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.040767 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="deaadd33-dcef-43df-a66e-9fd41434de6f" containerName="cleanup" Oct 31 05:00:27.041522 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.040780 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e27f22dd-e82f-493c-9745-6d901308b907" containerName="cleanup" Oct 31 05:00:27.041522 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.040791 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd0723e3-dcf6-4907-9ce1-99ca9c77b144" containerName="cleanup" Oct 31 05:00:27.041522 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.040804 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="703c28bc-a603-4001-9538-4a751cf9b7c0" containerName="cleanup" Oct 31 05:00:27.041522 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.040813 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ab61246-5d2d-4a2a-8c42-fff22960018f" containerName="cleanup" Oct 31 05:00:27.041522 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.040822 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c20d99d-7ce4-4bb8-a4d9-188cec401d75" containerName="cleanup" Oct 31 05:00:27.041522 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.040829 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c20d99d-7ce4-4bb8-a4d9-188cec401d75" containerName="cleanup" Oct 31 05:00:27.041522 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.040841 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="deaadd33-dcef-43df-a66e-9fd41434de6f" containerName="cleanup" Oct 31 05:00:27.041522 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.040850 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9b19f33-a012-48c9-b670-3b8eff773e49" containerName="cleanup" Oct 31 05:00:27.041522 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.040862 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="93bfb362-fcc2-4341-9881-1fd06b546590" containerName="cleanup" Oct 31 05:00:27.041522 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.040874 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ab61246-5d2d-4a2a-8c42-fff22960018f" containerName="cleanup" Oct 31 05:00:27.041522 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.040884 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9515dc7-4ea3-4a1b-aad8-1e78c7d0e13c" containerName="cleanup" Oct 31 05:00:27.041522 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.040894 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e23416e6-954a-4ee6-8c4f-f6f8d7e1c506" containerName="cleanup" Oct 31 05:00:27.041522 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.040903 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5259d89-590d-4fc7-b521-8f6feabbd2cc" containerName="cleanup" Oct 31 05:00:27.041522 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.040914 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="829dc86b-4ce1-427e-86db-cbaab8f15198" containerName="cleanup" Oct 31 05:00:27.041522 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.040926 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e478c8f1-9e68-4f6f-b919-28ba397e66c4" containerName="cleanup" Oct 31 05:00:27.041522 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.040934 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b6b5e89-5900-4db2-8552-e4f029de7573" containerName="cleanup" Oct 31 05:00:27.041522 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.040942 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="96e0a71b-2eee-4760-ade1-5cdd604c4c81" containerName="cleanup" Oct 31 05:00:27.041522 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.040953 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ab61246-5d2d-4a2a-8c42-fff22960018f" containerName="cleanup" Oct 31 05:00:27.045877 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.045852 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-12c0bc/test-model-12c0bc-kserve-66b5ccb945-4sg7l" Oct 31 05:00:27.062049 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.061970 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-12c0bc\"/\"kube-root-ca.crt\"" Oct 31 05:00:27.062248 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.062214 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-12c0bc\"/\"test-model-12c0bc-kserve-self-signed-certs\"" Oct 31 05:00:27.062429 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.062400 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-12c0bc\"/\"openshift-service-ca.crt\"" Oct 31 05:00:27.062555 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.062542 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-12c0bc\"/\"default-dockercfg-fl6rd\"" Oct 31 05:00:27.063991 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.063962 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-12c0bc/test-model-12c0bc-kserve-66b5ccb945-4sg7l"] Oct 31 05:00:27.213528 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.213481 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c357c8c-ca12-43ba-9dfa-b9bfb0867e43-tls-certs\") pod \"test-model-12c0bc-kserve-66b5ccb945-4sg7l\" (UID: \"0c357c8c-ca12-43ba-9dfa-b9bfb0867e43\") " pod="ai-tenant-e2e-inf-b-12c0bc/test-model-12c0bc-kserve-66b5ccb945-4sg7l" Oct 31 05:00:27.213727 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.213682 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c357c8c-ca12-43ba-9dfa-b9bfb0867e43-model-cache\") pod \"test-model-12c0bc-kserve-66b5ccb945-4sg7l\" (UID: \"0c357c8c-ca12-43ba-9dfa-b9bfb0867e43\") " pod="ai-tenant-e2e-inf-b-12c0bc/test-model-12c0bc-kserve-66b5ccb945-4sg7l" Oct 31 05:00:27.213815 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.213794 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c357c8c-ca12-43ba-9dfa-b9bfb0867e43-home\") pod \"test-model-12c0bc-kserve-66b5ccb945-4sg7l\" (UID: \"0c357c8c-ca12-43ba-9dfa-b9bfb0867e43\") " pod="ai-tenant-e2e-inf-b-12c0bc/test-model-12c0bc-kserve-66b5ccb945-4sg7l" Oct 31 05:00:27.213876 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.213861 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c357c8c-ca12-43ba-9dfa-b9bfb0867e43-dshm\") pod \"test-model-12c0bc-kserve-66b5ccb945-4sg7l\" (UID: \"0c357c8c-ca12-43ba-9dfa-b9bfb0867e43\") " pod="ai-tenant-e2e-inf-b-12c0bc/test-model-12c0bc-kserve-66b5ccb945-4sg7l" Oct 31 05:00:27.214123 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.214098 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw926\" (UniqueName: \"kubernetes.io/projected/0c357c8c-ca12-43ba-9dfa-b9bfb0867e43-kube-api-access-rw926\") pod \"test-model-12c0bc-kserve-66b5ccb945-4sg7l\" (UID: \"0c357c8c-ca12-43ba-9dfa-b9bfb0867e43\") " pod="ai-tenant-e2e-inf-b-12c0bc/test-model-12c0bc-kserve-66b5ccb945-4sg7l" Oct 31 05:00:27.214174 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.214142 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c357c8c-ca12-43ba-9dfa-b9bfb0867e43-tmp-dir\") pod \"test-model-12c0bc-kserve-66b5ccb945-4sg7l\" (UID: \"0c357c8c-ca12-43ba-9dfa-b9bfb0867e43\") " pod="ai-tenant-e2e-inf-b-12c0bc/test-model-12c0bc-kserve-66b5ccb945-4sg7l" Oct 31 05:00:27.340413 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.329975 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c357c8c-ca12-43ba-9dfa-b9bfb0867e43-tls-certs\") pod \"test-model-12c0bc-kserve-66b5ccb945-4sg7l\" (UID: \"0c357c8c-ca12-43ba-9dfa-b9bfb0867e43\") " pod="ai-tenant-e2e-inf-b-12c0bc/test-model-12c0bc-kserve-66b5ccb945-4sg7l" Oct 31 05:00:27.340413 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.330174 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c357c8c-ca12-43ba-9dfa-b9bfb0867e43-model-cache\") pod \"test-model-12c0bc-kserve-66b5ccb945-4sg7l\" (UID: \"0c357c8c-ca12-43ba-9dfa-b9bfb0867e43\") " pod="ai-tenant-e2e-inf-b-12c0bc/test-model-12c0bc-kserve-66b5ccb945-4sg7l" Oct 31 05:00:27.340413 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.330244 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c357c8c-ca12-43ba-9dfa-b9bfb0867e43-home\") pod \"test-model-12c0bc-kserve-66b5ccb945-4sg7l\" (UID: \"0c357c8c-ca12-43ba-9dfa-b9bfb0867e43\") " pod="ai-tenant-e2e-inf-b-12c0bc/test-model-12c0bc-kserve-66b5ccb945-4sg7l" Oct 31 05:00:27.340413 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.330307 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c357c8c-ca12-43ba-9dfa-b9bfb0867e43-dshm\") pod \"test-model-12c0bc-kserve-66b5ccb945-4sg7l\" (UID: \"0c357c8c-ca12-43ba-9dfa-b9bfb0867e43\") " pod="ai-tenant-e2e-inf-b-12c0bc/test-model-12c0bc-kserve-66b5ccb945-4sg7l" Oct 31 05:00:27.340413 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.330583 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw926\" (UniqueName: \"kubernetes.io/projected/0c357c8c-ca12-43ba-9dfa-b9bfb0867e43-kube-api-access-rw926\") pod \"test-model-12c0bc-kserve-66b5ccb945-4sg7l\" (UID: \"0c357c8c-ca12-43ba-9dfa-b9bfb0867e43\") " pod="ai-tenant-e2e-inf-b-12c0bc/test-model-12c0bc-kserve-66b5ccb945-4sg7l" Oct 31 05:00:27.340413 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.330615 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c357c8c-ca12-43ba-9dfa-b9bfb0867e43-tmp-dir\") pod \"test-model-12c0bc-kserve-66b5ccb945-4sg7l\" (UID: \"0c357c8c-ca12-43ba-9dfa-b9bfb0867e43\") " pod="ai-tenant-e2e-inf-b-12c0bc/test-model-12c0bc-kserve-66b5ccb945-4sg7l" Oct 31 05:00:27.345363 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.343832 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c357c8c-ca12-43ba-9dfa-b9bfb0867e43-home\") pod \"test-model-12c0bc-kserve-66b5ccb945-4sg7l\" (UID: \"0c357c8c-ca12-43ba-9dfa-b9bfb0867e43\") " pod="ai-tenant-e2e-inf-b-12c0bc/test-model-12c0bc-kserve-66b5ccb945-4sg7l" Oct 31 05:00:27.349366 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.346850 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c357c8c-ca12-43ba-9dfa-b9bfb0867e43-tmp-dir\") pod \"test-model-12c0bc-kserve-66b5ccb945-4sg7l\" (UID: \"0c357c8c-ca12-43ba-9dfa-b9bfb0867e43\") " pod="ai-tenant-e2e-inf-b-12c0bc/test-model-12c0bc-kserve-66b5ccb945-4sg7l" Oct 31 05:00:27.349366 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.347132 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c357c8c-ca12-43ba-9dfa-b9bfb0867e43-model-cache\") pod \"test-model-12c0bc-kserve-66b5ccb945-4sg7l\" (UID: \"0c357c8c-ca12-43ba-9dfa-b9bfb0867e43\") " pod="ai-tenant-e2e-inf-b-12c0bc/test-model-12c0bc-kserve-66b5ccb945-4sg7l" Oct 31 05:00:27.380352 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.379719 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c357c8c-ca12-43ba-9dfa-b9bfb0867e43-dshm\") pod \"test-model-12c0bc-kserve-66b5ccb945-4sg7l\" (UID: \"0c357c8c-ca12-43ba-9dfa-b9bfb0867e43\") " pod="ai-tenant-e2e-inf-b-12c0bc/test-model-12c0bc-kserve-66b5ccb945-4sg7l" Oct 31 05:00:27.418407 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.386051 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c357c8c-ca12-43ba-9dfa-b9bfb0867e43-tls-certs\") pod \"test-model-12c0bc-kserve-66b5ccb945-4sg7l\" (UID: \"0c357c8c-ca12-43ba-9dfa-b9bfb0867e43\") " pod="ai-tenant-e2e-inf-b-12c0bc/test-model-12c0bc-kserve-66b5ccb945-4sg7l" Oct 31 05:00:27.418407 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.402151 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw926\" (UniqueName: \"kubernetes.io/projected/0c357c8c-ca12-43ba-9dfa-b9bfb0867e43-kube-api-access-rw926\") pod \"test-model-12c0bc-kserve-66b5ccb945-4sg7l\" (UID: \"0c357c8c-ca12-43ba-9dfa-b9bfb0867e43\") " pod="ai-tenant-e2e-inf-b-12c0bc/test-model-12c0bc-kserve-66b5ccb945-4sg7l" Oct 31 05:00:27.473559 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.473523 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b6b5e89-5900-4db2-8552-e4f029de7573" path="/var/lib/kubelet/pods/0b6b5e89-5900-4db2-8552-e4f029de7573/volumes" Oct 31 05:00:27.474629 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.474600 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17f1de0f-3edd-4d07-910a-f6e6f356203c" path="/var/lib/kubelet/pods/17f1de0f-3edd-4d07-910a-f6e6f356203c/volumes" Oct 31 05:00:27.475537 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.475515 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c60e764-ed6c-48dc-b6d0-abffbbdcdc3c" path="/var/lib/kubelet/pods/1c60e764-ed6c-48dc-b6d0-abffbbdcdc3c/volumes" Oct 31 05:00:27.476754 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.476734 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="703c28bc-a603-4001-9538-4a751cf9b7c0" path="/var/lib/kubelet/pods/703c28bc-a603-4001-9538-4a751cf9b7c0/volumes" Oct 31 05:00:27.477614 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.477593 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ab61246-5d2d-4a2a-8c42-fff22960018f" path="/var/lib/kubelet/pods/7ab61246-5d2d-4a2a-8c42-fff22960018f/volumes" Oct 31 05:00:27.478585 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.478564 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c20d99d-7ce4-4bb8-a4d9-188cec401d75" path="/var/lib/kubelet/pods/7c20d99d-7ce4-4bb8-a4d9-188cec401d75/volumes" Oct 31 05:00:27.479826 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.479805 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd0723e3-dcf6-4907-9ce1-99ca9c77b144" path="/var/lib/kubelet/pods/cd0723e3-dcf6-4907-9ce1-99ca9c77b144/volumes" Oct 31 05:00:27.481393 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.481359 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="deaadd33-dcef-43df-a66e-9fd41434de6f" path="/var/lib/kubelet/pods/deaadd33-dcef-43df-a66e-9fd41434de6f/volumes" Oct 31 05:00:27.482124 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.482104 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e23416e6-954a-4ee6-8c4f-f6f8d7e1c506" path="/var/lib/kubelet/pods/e23416e6-954a-4ee6-8c4f-f6f8d7e1c506/volumes" Oct 31 05:00:27.665878 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.665791 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-12c0bc/test-model-12c0bc-kserve-66b5ccb945-4sg7l" Oct 31 05:00:27.741568 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.741531 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1daf30/test-model-1daf30-kserve-8686666dc7-xc4n6" event={"ID":"f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7","Type":"ContainerStarted","Data":"a391f949b7c22762f84393b8897b0c5dd3f4aa344f17d27d1c538ac54d6fded0"} Oct 31 05:00:27.741796 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.741768 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1daf30/test-model-1daf30-kserve-8686666dc7-xc4n6" event={"ID":"f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7","Type":"ContainerStarted","Data":"8d674b03a2ced65525cb16a198e65c14d8dbd8ac1649cd982c2691eedf0f5fe0"} Oct 31 05:00:27.789263 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:27.789099 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-1daf30/test-model-1daf30-kserve-8686666dc7-xc4n6" podStartSLOduration=2.789083221 podStartE2EDuration="2.789083221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:00:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:00:27.782622554 +0000 UTC m=+3382.538960594" watchObservedRunningTime="2026-10-31 05:00:27.789083221 +0000 UTC m=+3382.545421307" Oct 31 05:00:28.001085 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:28.001048 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-12c0bc/test-model-12c0bc-kserve-66b5ccb945-4sg7l"] Oct 31 05:00:28.004510 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:00:28.004464 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c357c8c_ca12_43ba_9dfa_b9bfb0867e43.slice/crio-a9447f08318d0d5d998891ded2a7c3580169d7c1802668d616f40e8184570c89 WatchSource:0}: Error finding container a9447f08318d0d5d998891ded2a7c3580169d7c1802668d616f40e8184570c89: Status 404 returned error can't find the container with id a9447f08318d0d5d998891ded2a7c3580169d7c1802668d616f40e8184570c89 Oct 31 05:00:28.400144 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:28.400051 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-1daf30/test-model-1daf30-kserve-8686666dc7-xc4n6" Oct 31 05:00:28.753618 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:28.753574 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-12c0bc/test-model-12c0bc-kserve-66b5ccb945-4sg7l" event={"ID":"0c357c8c-ca12-43ba-9dfa-b9bfb0867e43","Type":"ContainerStarted","Data":"2f4c4eecb0d628f8fd35fdb655d5620424307a007f06736f567f6d67646e5c72"} Oct 31 05:00:28.754149 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:28.753628 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-12c0bc/test-model-12c0bc-kserve-66b5ccb945-4sg7l" event={"ID":"0c357c8c-ca12-43ba-9dfa-b9bfb0867e43","Type":"ContainerStarted","Data":"a9447f08318d0d5d998891ded2a7c3580169d7c1802668d616f40e8184570c89"} Oct 31 05:00:28.795712 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:28.795657 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-12c0bc/test-model-12c0bc-kserve-66b5ccb945-4sg7l" podStartSLOduration=1.795641436 podStartE2EDuration="1.795641436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:00:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:00:28.7935765 +0000 UTC m=+3383.549914541" watchObservedRunningTime="2026-10-31 05:00:28.795641436 +0000 UTC m=+3383.551979481" Oct 31 05:00:29.666237 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:29.666204 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-12c0bc/test-model-12c0bc-kserve-66b5ccb945-4sg7l" Oct 31 05:00:36.428128 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:36.419075 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-1daf30/test-model-1daf30-kserve-8686666dc7-xc4n6" Oct 31 05:00:36.451349 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:36.451301 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-1daf30/test-model-1daf30-kserve-8686666dc7-xc4n6" Oct 31 05:00:36.856303 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:36.856270 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-1daf30/test-model-1daf30-kserve-8686666dc7-xc4n6" Oct 31 05:00:37.666124 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:37.666086 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-12c0bc/test-model-12c0bc-kserve-66b5ccb945-4sg7l" Oct 31 05:00:37.678742 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:37.678710 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-12c0bc/test-model-12c0bc-kserve-66b5ccb945-4sg7l" Oct 31 05:00:37.866611 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:37.866576 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-12c0bc/test-model-12c0bc-kserve-66b5ccb945-4sg7l" Oct 31 05:00:38.054928 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:38.054890 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx"] Oct 31 05:00:38.055109 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:38.055087 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx" podUID="cc1adc72-6548-4daf-9dc1-2b96c853cdbf" containerName="istio-proxy" containerID="cri-o://2289634591bd47dc3795be818dac69b25930c63f38c5395757dae4755423d8ca" gracePeriod=30 Oct 31 05:00:43.381744 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:43.381717 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx" Oct 31 05:00:43.480614 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:43.480578 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-istio-podinfo\") pod \"cc1adc72-6548-4daf-9dc1-2b96c853cdbf\" (UID: \"cc1adc72-6548-4daf-9dc1-2b96c853cdbf\") " Oct 31 05:00:43.480887 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:43.480862 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-istio-data\") pod \"cc1adc72-6548-4daf-9dc1-2b96c853cdbf\" (UID: \"cc1adc72-6548-4daf-9dc1-2b96c853cdbf\") " Oct 31 05:00:43.481111 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:43.481091 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-istiod-ca-cert\") pod \"cc1adc72-6548-4daf-9dc1-2b96c853cdbf\" (UID: \"cc1adc72-6548-4daf-9dc1-2b96c853cdbf\") " Oct 31 05:00:43.481301 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:43.481278 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-workload-socket\") pod \"cc1adc72-6548-4daf-9dc1-2b96c853cdbf\" (UID: \"cc1adc72-6548-4daf-9dc1-2b96c853cdbf\") " Oct 31 05:00:43.481452 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:43.481323 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-credential-socket\") pod \"cc1adc72-6548-4daf-9dc1-2b96c853cdbf\" (UID: \"cc1adc72-6548-4daf-9dc1-2b96c853cdbf\") " Oct 31 05:00:43.481452 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:43.481380 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-istio-data" pod "cc1adc72-6548-4daf-9dc1-2b96c853cdbf" (UID: "cc1adc72-6548-4daf-9dc1-2b96c853cdbf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:00:43.481452 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:43.481404 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-istio-envoy\") pod \"cc1adc72-6548-4daf-9dc1-2b96c853cdbf\" (UID: \"cc1adc72-6548-4daf-9dc1-2b96c853cdbf\") " Oct 31 05:00:43.481452 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:43.481448 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-workload-certs\") pod \"cc1adc72-6548-4daf-9dc1-2b96c853cdbf\" (UID: \"cc1adc72-6548-4daf-9dc1-2b96c853cdbf\") " Oct 31 05:00:43.481740 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:43.481595 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-workload-socket" pod "cc1adc72-6548-4daf-9dc1-2b96c853cdbf" (UID: "cc1adc72-6548-4daf-9dc1-2b96c853cdbf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:00:43.481740 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:43.481615 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-istio-token\" (UniqueName: \"kubernetes.io/projected/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-istio-token\") pod \"cc1adc72-6548-4daf-9dc1-2b96c853cdbf\" (UID: \"cc1adc72-6548-4daf-9dc1-2b96c853cdbf\") " Oct 31 05:00:43.481853 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:43.481789 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-kube-api-access-rlnj8\" (UniqueName: \"kubernetes.io/projected/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-kube-api-access-rlnj8\") pod \"cc1adc72-6548-4daf-9dc1-2b96c853cdbf\" (UID: \"cc1adc72-6548-4daf-9dc1-2b96c853cdbf\") " Oct 31 05:00:43.481920 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:43.481881 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-istiod-ca-cert" pod "cc1adc72-6548-4daf-9dc1-2b96c853cdbf" (UID: "cc1adc72-6548-4daf-9dc1-2b96c853cdbf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:00:43.482497 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:43.482461 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-credential-socket" pod "cc1adc72-6548-4daf-9dc1-2b96c853cdbf" (UID: "cc1adc72-6548-4daf-9dc1-2b96c853cdbf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:00:43.482650 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:43.482526 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-workload-certs" pod "cc1adc72-6548-4daf-9dc1-2b96c853cdbf" (UID: "cc1adc72-6548-4daf-9dc1-2b96c853cdbf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:00:43.483302 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:43.483277 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:00:43.483408 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:43.483309 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:00:43.483408 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:43.483324 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:00:43.483408 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:43.483362 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:00:43.483408 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:43.483375 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:00:43.485167 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:43.485130 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-istio-envoy" pod "cc1adc72-6548-4daf-9dc1-2b96c853cdbf" (UID: "cc1adc72-6548-4daf-9dc1-2b96c853cdbf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:00:43.486095 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:43.486062 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-kube-api-access-rlnj8" pod "cc1adc72-6548-4daf-9dc1-2b96c853cdbf" (UID: "cc1adc72-6548-4daf-9dc1-2b96c853cdbf"). InnerVolumeSpecName "kube-api-access-rlnj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:00:43.487607 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:43.487581 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-istio-token" pod "cc1adc72-6548-4daf-9dc1-2b96c853cdbf" (UID: "cc1adc72-6548-4daf-9dc1-2b96c853cdbf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:00:43.491281 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:43.491236 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-istio-podinfo" pod "cc1adc72-6548-4daf-9dc1-2b96c853cdbf" (UID: "cc1adc72-6548-4daf-9dc1-2b96c853cdbf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:00:43.591998 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:43.590323 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlnj8\" (UniqueName: \"kubernetes.io/projected/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-kube-api-access-rlnj8\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:00:43.591998 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:43.590376 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:00:43.591998 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:43.590393 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:00:43.591998 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:43.590405 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc1adc72-6548-4daf-9dc1-2b96c853cdbf-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:00:43.925539 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:43.925448 5019 generic.go:356] "Generic (PLEG): container finished" podID="cc1adc72-6548-4daf-9dc1-2b96c853cdbf" containerID="2289634591bd47dc3795be818dac69b25930c63f38c5395757dae4755423d8ca" exitCode=0 Oct 31 05:00:43.925539 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:43.925504 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx" Oct 31 05:00:43.925539 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:43.925525 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx" event={"ID":"cc1adc72-6548-4daf-9dc1-2b96c853cdbf","Type":"ContainerDied","Data":"2289634591bd47dc3795be818dac69b25930c63f38c5395757dae4755423d8ca"} Oct 31 05:00:43.925805 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:43.925587 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx" event={"ID":"cc1adc72-6548-4daf-9dc1-2b96c853cdbf","Type":"ContainerDied","Data":"abeba8990513c0e70f11cd581c43ed10c9259b29230b4729fdb9aa347937146d"} Oct 31 05:00:43.925805 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:43.925610 5019 scope.go:122] "RemoveContainer" containerID="2289634591bd47dc3795be818dac69b25930c63f38c5395757dae4755423d8ca" Oct 31 05:00:43.938974 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:43.938953 5019 scope.go:122] "RemoveContainer" containerID="8b49042a4d5200c98a78e4465ba5fbf55837632fc4c8f40d09980add49bdd0fc" Oct 31 05:00:43.954792 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:43.954751 5019 scope.go:122] "RemoveContainer" containerID="2289634591bd47dc3795be818dac69b25930c63f38c5395757dae4755423d8ca" Oct 31 05:00:43.955131 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:43.955102 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2289634591bd47dc3795be818dac69b25930c63f38c5395757dae4755423d8ca\": container with ID starting with 2289634591bd47dc3795be818dac69b25930c63f38c5395757dae4755423d8ca not found: ID does not exist" containerID="2289634591bd47dc3795be818dac69b25930c63f38c5395757dae4755423d8ca" Oct 31 05:00:43.955257 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:43.955138 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2289634591bd47dc3795be818dac69b25930c63f38c5395757dae4755423d8ca"} err="failed to get container status \"2289634591bd47dc3795be818dac69b25930c63f38c5395757dae4755423d8ca\": rpc error: code = NotFound desc = could not find container \"2289634591bd47dc3795be818dac69b25930c63f38c5395757dae4755423d8ca\": container with ID starting with 2289634591bd47dc3795be818dac69b25930c63f38c5395757dae4755423d8ca not found: ID does not exist" Oct 31 05:00:43.955257 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:43.955157 5019 scope.go:122] "RemoveContainer" containerID="8b49042a4d5200c98a78e4465ba5fbf55837632fc4c8f40d09980add49bdd0fc" Oct 31 05:00:43.955466 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:43.955445 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b49042a4d5200c98a78e4465ba5fbf55837632fc4c8f40d09980add49bdd0fc\": container with ID starting with 8b49042a4d5200c98a78e4465ba5fbf55837632fc4c8f40d09980add49bdd0fc not found: ID does not exist" containerID="8b49042a4d5200c98a78e4465ba5fbf55837632fc4c8f40d09980add49bdd0fc" Oct 31 05:00:43.955544 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:43.955474 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b49042a4d5200c98a78e4465ba5fbf55837632fc4c8f40d09980add49bdd0fc"} err="failed to get container status \"8b49042a4d5200c98a78e4465ba5fbf55837632fc4c8f40d09980add49bdd0fc\": rpc error: code = NotFound desc = could not find container \"8b49042a4d5200c98a78e4465ba5fbf55837632fc4c8f40d09980add49bdd0fc\": container with ID starting with 8b49042a4d5200c98a78e4465ba5fbf55837632fc4c8f40d09980add49bdd0fc not found: ID does not exist" Oct 31 05:00:43.994273 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:43.994209 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx"] Oct 31 05:00:43.999720 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:43.999672 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-9q9wx"] Oct 31 05:00:45.589421 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:00:45.586047 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc1adc72-6548-4daf-9dc1-2b96c853cdbf" path="/var/lib/kubelet/pods/cc1adc72-6548-4daf-9dc1-2b96c853cdbf/volumes" Oct 31 05:00:52.398733 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:00:52.398685 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:01:12.330722 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:12.330687 5019 scope.go:122] "RemoveContainer" containerID="327503c4b02f478da954bee7d318692ed2e62a3f1ec9c1b24f199f3570112dbc" Oct 31 05:01:12.344348 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:12.344308 5019 scope.go:122] "RemoveContainer" containerID="46ea82e2864e87be3668aa4dd08c74045a643b16edacfa13e6e4fd050c2bf604" Oct 31 05:01:12.377125 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:12.377093 5019 scope.go:122] "RemoveContainer" containerID="3b35f2ff5308348359210086876b0da580be0759c2472f3b013a1ec806ff6a38" Oct 31 05:01:12.435817 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:12.435789 5019 scope.go:122] "RemoveContainer" containerID="898a84116c5821ebc205f5de1206e03aefd95dc52dca68b9b3698f7619f9faf3" Oct 31 05:01:12.449287 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:12.449256 5019 scope.go:122] "RemoveContainer" containerID="c9de05d58e1040ed2e0e04941f25c38f8267e7a2b729cf561c9d216be050dd97" Oct 31 05:01:12.456724 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:12.456607 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf"] Oct 31 05:01:12.456891 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:12.456861 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf" podUID="7ce5abc5-55a0-4b15-80a6-7e21d360e389" containerName="istio-proxy" containerID="cri-o://7ac9c8e148e3b8eb2c8dd3ded1ee78d693cca6c7855b72d66c4a5b69328290d6" gracePeriod=30 Oct 31 05:01:12.464682 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:12.464655 5019 scope.go:122] "RemoveContainer" containerID="58f210e6f89184b4282e88a0728674db6100cd26261767d13ddb74eb7950bd6d" Oct 31 05:01:12.487447 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:12.483193 5019 scope.go:122] "RemoveContainer" containerID="0e0cad32e7c8fd53718550904a956508f1221266f0162af4afedac1f44bbaa9b" Oct 31 05:01:14.977280 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:14.977243 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg"] Oct 31 05:01:14.977731 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:14.977476 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg" podUID="d12de313-f7d1-4d6e-b2d7-0727073b032c" containerName="istio-proxy" containerID="cri-o://4d2ad2b39c00ac5b93bac98d010d32ed1103e0db1d05e46753c724c707f4fc0f" gracePeriod=30 Oct 31 05:01:15.740667 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:15.740594 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j"] Oct 31 05:01:15.740927 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:15.740854 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j" podUID="e107852e-0f1f-4705-bf39-a676808806ef" containerName="istio-proxy" containerID="cri-o://b988f154cd8443fafa6814fc2771b9b20a9b97df6edae4aaa1d7b3b4e25f5950" gracePeriod=30 Oct 31 05:01:15.776770 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:15.776731 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v"] Oct 31 05:01:15.777179 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:15.777145 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v" podUID="c4e33d7c-168f-4001-8e39-22bea132574a" containerName="istio-proxy" containerID="cri-o://1264bd0ec634bcf3505b1094dbd8ceec6d8fda0fdd152b41795dbc63e4c43b18" gracePeriod=30 Oct 31 05:01:17.773890 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:17.773852 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf" Oct 31 05:01:17.828739 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:17.828710 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ce5abc5-55a0-4b15-80a6-7e21d360e389-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ce5abc5-55a0-4b15-80a6-7e21d360e389-istio-podinfo\") pod \"7ce5abc5-55a0-4b15-80a6-7e21d360e389\" (UID: \"7ce5abc5-55a0-4b15-80a6-7e21d360e389\") " Oct 31 05:01:17.828904 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:17.828816 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ce5abc5-55a0-4b15-80a6-7e21d360e389-istio-token\" (UniqueName: \"kubernetes.io/projected/7ce5abc5-55a0-4b15-80a6-7e21d360e389-istio-token\") pod \"7ce5abc5-55a0-4b15-80a6-7e21d360e389\" (UID: \"7ce5abc5-55a0-4b15-80a6-7e21d360e389\") " Oct 31 05:01:17.828904 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:17.828878 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ce5abc5-55a0-4b15-80a6-7e21d360e389-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ce5abc5-55a0-4b15-80a6-7e21d360e389-istio-envoy\") pod \"7ce5abc5-55a0-4b15-80a6-7e21d360e389\" (UID: \"7ce5abc5-55a0-4b15-80a6-7e21d360e389\") " Oct 31 05:01:17.829048 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:17.829002 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ce5abc5-55a0-4b15-80a6-7e21d360e389-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ce5abc5-55a0-4b15-80a6-7e21d360e389-workload-certs\") pod \"7ce5abc5-55a0-4b15-80a6-7e21d360e389\" (UID: \"7ce5abc5-55a0-4b15-80a6-7e21d360e389\") " Oct 31 05:01:17.829112 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:17.829093 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ce5abc5-55a0-4b15-80a6-7e21d360e389-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce5abc5-55a0-4b15-80a6-7e21d360e389-workload-socket\") pod \"7ce5abc5-55a0-4b15-80a6-7e21d360e389\" (UID: \"7ce5abc5-55a0-4b15-80a6-7e21d360e389\") " Oct 31 05:01:17.829171 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:17.829151 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ce5abc5-55a0-4b15-80a6-7e21d360e389-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce5abc5-55a0-4b15-80a6-7e21d360e389-credential-socket\") pod \"7ce5abc5-55a0-4b15-80a6-7e21d360e389\" (UID: \"7ce5abc5-55a0-4b15-80a6-7e21d360e389\") " Oct 31 05:01:17.829266 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:17.829248 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ce5abc5-55a0-4b15-80a6-7e21d360e389-kube-api-access-xr9q9\" (UniqueName: \"kubernetes.io/projected/7ce5abc5-55a0-4b15-80a6-7e21d360e389-kube-api-access-xr9q9\") pod \"7ce5abc5-55a0-4b15-80a6-7e21d360e389\" (UID: \"7ce5abc5-55a0-4b15-80a6-7e21d360e389\") " Oct 31 05:01:17.829367 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:17.829265 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ce5abc5-55a0-4b15-80a6-7e21d360e389-workload-certs" pod "7ce5abc5-55a0-4b15-80a6-7e21d360e389" (UID: "7ce5abc5-55a0-4b15-80a6-7e21d360e389"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:01:17.829367 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:17.829314 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ce5abc5-55a0-4b15-80a6-7e21d360e389-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ce5abc5-55a0-4b15-80a6-7e21d360e389-istio-data\") pod \"7ce5abc5-55a0-4b15-80a6-7e21d360e389\" (UID: \"7ce5abc5-55a0-4b15-80a6-7e21d360e389\") " Oct 31 05:01:17.829494 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:17.829428 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ce5abc5-55a0-4b15-80a6-7e21d360e389-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ce5abc5-55a0-4b15-80a6-7e21d360e389-istiod-ca-cert\") pod \"7ce5abc5-55a0-4b15-80a6-7e21d360e389\" (UID: \"7ce5abc5-55a0-4b15-80a6-7e21d360e389\") " Oct 31 05:01:17.829851 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:17.829403 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ce5abc5-55a0-4b15-80a6-7e21d360e389-workload-socket" pod "7ce5abc5-55a0-4b15-80a6-7e21d360e389" (UID: "7ce5abc5-55a0-4b15-80a6-7e21d360e389"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:01:17.829851 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:17.829437 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ce5abc5-55a0-4b15-80a6-7e21d360e389-credential-socket" pod "7ce5abc5-55a0-4b15-80a6-7e21d360e389" (UID: "7ce5abc5-55a0-4b15-80a6-7e21d360e389"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:01:17.830048 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:17.829561 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ce5abc5-55a0-4b15-80a6-7e21d360e389-istio-data" pod "7ce5abc5-55a0-4b15-80a6-7e21d360e389" (UID: "7ce5abc5-55a0-4b15-80a6-7e21d360e389"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:01:17.830048 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:17.829744 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ce5abc5-55a0-4b15-80a6-7e21d360e389-istiod-ca-cert" pod "7ce5abc5-55a0-4b15-80a6-7e21d360e389" (UID: "7ce5abc5-55a0-4b15-80a6-7e21d360e389"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:01:17.830273 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:17.830245 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce5abc5-55a0-4b15-80a6-7e21d360e389-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:01:17.830273 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:17.830270 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce5abc5-55a0-4b15-80a6-7e21d360e389-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:01:17.830451 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:17.830286 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ce5abc5-55a0-4b15-80a6-7e21d360e389-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:01:17.830451 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:17.830299 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ce5abc5-55a0-4b15-80a6-7e21d360e389-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:01:17.830451 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:17.830325 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ce5abc5-55a0-4b15-80a6-7e21d360e389-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:01:17.831392 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:17.831366 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ce5abc5-55a0-4b15-80a6-7e21d360e389-istio-podinfo" pod "7ce5abc5-55a0-4b15-80a6-7e21d360e389" (UID: "7ce5abc5-55a0-4b15-80a6-7e21d360e389"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:01:17.831500 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:17.831459 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ce5abc5-55a0-4b15-80a6-7e21d360e389-istio-token" pod "7ce5abc5-55a0-4b15-80a6-7e21d360e389" (UID: "7ce5abc5-55a0-4b15-80a6-7e21d360e389"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:01:17.831894 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:17.831871 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ce5abc5-55a0-4b15-80a6-7e21d360e389-kube-api-access-xr9q9" pod "7ce5abc5-55a0-4b15-80a6-7e21d360e389" (UID: "7ce5abc5-55a0-4b15-80a6-7e21d360e389"). InnerVolumeSpecName "kube-api-access-xr9q9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:01:17.832652 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:17.832631 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ce5abc5-55a0-4b15-80a6-7e21d360e389-istio-envoy" pod "7ce5abc5-55a0-4b15-80a6-7e21d360e389" (UID: "7ce5abc5-55a0-4b15-80a6-7e21d360e389"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:01:17.932446 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:17.932417 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ce5abc5-55a0-4b15-80a6-7e21d360e389-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:01:17.932446 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:17.932444 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ce5abc5-55a0-4b15-80a6-7e21d360e389-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:01:17.932631 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:17.932454 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr9q9\" (UniqueName: \"kubernetes.io/projected/7ce5abc5-55a0-4b15-80a6-7e21d360e389-kube-api-access-xr9q9\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:01:17.932631 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:17.932465 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ce5abc5-55a0-4b15-80a6-7e21d360e389-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:01:18.216940 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:18.216906 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz"] Oct 31 05:01:18.217155 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:18.217131 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz" podUID="2d771867-36b5-4b55-afb4-f889d1d91784" containerName="istio-proxy" containerID="cri-o://a8c165f1cf4f8c49f2f4ed1ad38f945ec38654b0460f84dd3abe49d97607631f" gracePeriod=30 Oct 31 05:01:18.349520 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:18.349485 5019 generic.go:356] "Generic (PLEG): container finished" podID="7ce5abc5-55a0-4b15-80a6-7e21d360e389" containerID="7ac9c8e148e3b8eb2c8dd3ded1ee78d693cca6c7855b72d66c4a5b69328290d6" exitCode=0 Oct 31 05:01:18.349694 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:18.349540 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf" Oct 31 05:01:18.349694 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:18.349568 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf" event={"ID":"7ce5abc5-55a0-4b15-80a6-7e21d360e389","Type":"ContainerDied","Data":"7ac9c8e148e3b8eb2c8dd3ded1ee78d693cca6c7855b72d66c4a5b69328290d6"} Oct 31 05:01:18.349694 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:18.349622 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf" event={"ID":"7ce5abc5-55a0-4b15-80a6-7e21d360e389","Type":"ContainerDied","Data":"c100eed766336bded3bba90fcd6304a45f691e1e1d94f41defcd0fe7e27084f1"} Oct 31 05:01:18.349694 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:18.349644 5019 scope.go:122] "RemoveContainer" containerID="7ac9c8e148e3b8eb2c8dd3ded1ee78d693cca6c7855b72d66c4a5b69328290d6" Oct 31 05:01:18.363428 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:18.363403 5019 scope.go:122] "RemoveContainer" containerID="0e29406cdd0f4aeca185f400307c928d1f7c7040da088704a7b1e5c45dca56ac" Oct 31 05:01:18.376951 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:18.376933 5019 scope.go:122] "RemoveContainer" containerID="7ac9c8e148e3b8eb2c8dd3ded1ee78d693cca6c7855b72d66c4a5b69328290d6" Oct 31 05:01:18.377234 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:01:18.377207 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ac9c8e148e3b8eb2c8dd3ded1ee78d693cca6c7855b72d66c4a5b69328290d6\": container with ID starting with 7ac9c8e148e3b8eb2c8dd3ded1ee78d693cca6c7855b72d66c4a5b69328290d6 not found: ID does not exist" containerID="7ac9c8e148e3b8eb2c8dd3ded1ee78d693cca6c7855b72d66c4a5b69328290d6" Oct 31 05:01:18.377317 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:18.377241 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ac9c8e148e3b8eb2c8dd3ded1ee78d693cca6c7855b72d66c4a5b69328290d6"} err="failed to get container status \"7ac9c8e148e3b8eb2c8dd3ded1ee78d693cca6c7855b72d66c4a5b69328290d6\": rpc error: code = NotFound desc = could not find container \"7ac9c8e148e3b8eb2c8dd3ded1ee78d693cca6c7855b72d66c4a5b69328290d6\": container with ID starting with 7ac9c8e148e3b8eb2c8dd3ded1ee78d693cca6c7855b72d66c4a5b69328290d6 not found: ID does not exist" Oct 31 05:01:18.377317 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:18.377258 5019 scope.go:122] "RemoveContainer" containerID="0e29406cdd0f4aeca185f400307c928d1f7c7040da088704a7b1e5c45dca56ac" Oct 31 05:01:18.377588 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:01:18.377569 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e29406cdd0f4aeca185f400307c928d1f7c7040da088704a7b1e5c45dca56ac\": container with ID starting with 0e29406cdd0f4aeca185f400307c928d1f7c7040da088704a7b1e5c45dca56ac not found: ID does not exist" containerID="0e29406cdd0f4aeca185f400307c928d1f7c7040da088704a7b1e5c45dca56ac" Oct 31 05:01:18.377655 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:18.377593 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e29406cdd0f4aeca185f400307c928d1f7c7040da088704a7b1e5c45dca56ac"} err="failed to get container status \"0e29406cdd0f4aeca185f400307c928d1f7c7040da088704a7b1e5c45dca56ac\": rpc error: code = NotFound desc = could not find container \"0e29406cdd0f4aeca185f400307c928d1f7c7040da088704a7b1e5c45dca56ac\": container with ID starting with 0e29406cdd0f4aeca185f400307c928d1f7c7040da088704a7b1e5c45dca56ac not found: ID does not exist" Oct 31 05:01:18.407816 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:18.406044 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf"] Oct 31 05:01:18.411388 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:18.411359 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-47c409-openshift-default-796cbb7889-bm9cf"] Oct 31 05:01:18.869888 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:18.869857 5019 patch_prober.go:29] interesting pod/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 05:01:18.870375 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:18.869913 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg" podUID="d12de313-f7d1-4d6e-b2d7-0727073b032c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 05:01:19.415124 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:19.415089 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ce5abc5-55a0-4b15-80a6-7e21d360e389" path="/var/lib/kubelet/pods/7ce5abc5-55a0-4b15-80a6-7e21d360e389/volumes" Oct 31 05:01:20.294739 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:20.294706 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg" Oct 31 05:01:20.370868 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:20.370832 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d12de313-f7d1-4d6e-b2d7-0727073b032c-kube-api-access-jcfnn\" (UniqueName: \"kubernetes.io/projected/d12de313-f7d1-4d6e-b2d7-0727073b032c-kube-api-access-jcfnn\") pod \"d12de313-f7d1-4d6e-b2d7-0727073b032c\" (UID: \"d12de313-f7d1-4d6e-b2d7-0727073b032c\") " Oct 31 05:01:20.371087 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:20.370896 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d12de313-f7d1-4d6e-b2d7-0727073b032c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d12de313-f7d1-4d6e-b2d7-0727073b032c-istio-data\") pod \"d12de313-f7d1-4d6e-b2d7-0727073b032c\" (UID: \"d12de313-f7d1-4d6e-b2d7-0727073b032c\") " Oct 31 05:01:20.371087 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:20.371043 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d12de313-f7d1-4d6e-b2d7-0727073b032c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d12de313-f7d1-4d6e-b2d7-0727073b032c-credential-socket\") pod \"d12de313-f7d1-4d6e-b2d7-0727073b032c\" (UID: \"d12de313-f7d1-4d6e-b2d7-0727073b032c\") " Oct 31 05:01:20.371218 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:20.371092 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d12de313-f7d1-4d6e-b2d7-0727073b032c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d12de313-f7d1-4d6e-b2d7-0727073b032c-workload-socket\") pod \"d12de313-f7d1-4d6e-b2d7-0727073b032c\" (UID: \"d12de313-f7d1-4d6e-b2d7-0727073b032c\") " Oct 31 05:01:20.371218 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:20.371129 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d12de313-f7d1-4d6e-b2d7-0727073b032c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d12de313-f7d1-4d6e-b2d7-0727073b032c-istiod-ca-cert\") pod \"d12de313-f7d1-4d6e-b2d7-0727073b032c\" (UID: \"d12de313-f7d1-4d6e-b2d7-0727073b032c\") " Oct 31 05:01:20.371369 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:20.371247 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d12de313-f7d1-4d6e-b2d7-0727073b032c-istio-data" pod "d12de313-f7d1-4d6e-b2d7-0727073b032c" (UID: "d12de313-f7d1-4d6e-b2d7-0727073b032c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:01:20.371546 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:20.371519 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d12de313-f7d1-4d6e-b2d7-0727073b032c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d12de313-f7d1-4d6e-b2d7-0727073b032c-istio-podinfo\") pod \"d12de313-f7d1-4d6e-b2d7-0727073b032c\" (UID: \"d12de313-f7d1-4d6e-b2d7-0727073b032c\") " Oct 31 05:01:20.371635 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:20.371548 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d12de313-f7d1-4d6e-b2d7-0727073b032c-workload-socket" pod "d12de313-f7d1-4d6e-b2d7-0727073b032c" (UID: "d12de313-f7d1-4d6e-b2d7-0727073b032c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:01:20.371696 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:20.371662 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d12de313-f7d1-4d6e-b2d7-0727073b032c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d12de313-f7d1-4d6e-b2d7-0727073b032c-workload-certs\") pod \"d12de313-f7d1-4d6e-b2d7-0727073b032c\" (UID: \"d12de313-f7d1-4d6e-b2d7-0727073b032c\") " Oct 31 05:01:20.371800 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:20.371749 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d12de313-f7d1-4d6e-b2d7-0727073b032c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d12de313-f7d1-4d6e-b2d7-0727073b032c-istio-envoy\") pod \"d12de313-f7d1-4d6e-b2d7-0727073b032c\" (UID: \"d12de313-f7d1-4d6e-b2d7-0727073b032c\") " Oct 31 05:01:20.371864 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:20.371829 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d12de313-f7d1-4d6e-b2d7-0727073b032c-istio-token\" (UniqueName: \"kubernetes.io/projected/d12de313-f7d1-4d6e-b2d7-0727073b032c-istio-token\") pod \"d12de313-f7d1-4d6e-b2d7-0727073b032c\" (UID: \"d12de313-f7d1-4d6e-b2d7-0727073b032c\") " Oct 31 05:01:20.371923 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:20.371906 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d12de313-f7d1-4d6e-b2d7-0727073b032c-workload-certs" pod "d12de313-f7d1-4d6e-b2d7-0727073b032c" (UID: "d12de313-f7d1-4d6e-b2d7-0727073b032c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:01:20.372357 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:20.372123 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d12de313-f7d1-4d6e-b2d7-0727073b032c-credential-socket" pod "d12de313-f7d1-4d6e-b2d7-0727073b032c" (UID: "d12de313-f7d1-4d6e-b2d7-0727073b032c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:01:20.372976 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:20.372601 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d12de313-f7d1-4d6e-b2d7-0727073b032c-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:01:20.372976 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:20.372628 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d12de313-f7d1-4d6e-b2d7-0727073b032c-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:01:20.372976 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:20.372642 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d12de313-f7d1-4d6e-b2d7-0727073b032c-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:01:20.372976 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:20.372656 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d12de313-f7d1-4d6e-b2d7-0727073b032c-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:01:20.372976 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:20.372674 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d12de313-f7d1-4d6e-b2d7-0727073b032c-istiod-ca-cert" pod "d12de313-f7d1-4d6e-b2d7-0727073b032c" (UID: "d12de313-f7d1-4d6e-b2d7-0727073b032c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:01:20.374640 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:20.374073 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d12de313-f7d1-4d6e-b2d7-0727073b032c-kube-api-access-jcfnn" pod "d12de313-f7d1-4d6e-b2d7-0727073b032c" (UID: "d12de313-f7d1-4d6e-b2d7-0727073b032c"). InnerVolumeSpecName "kube-api-access-jcfnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:01:20.374640 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:20.374092 5019 generic.go:356] "Generic (PLEG): container finished" podID="d12de313-f7d1-4d6e-b2d7-0727073b032c" containerID="4d2ad2b39c00ac5b93bac98d010d32ed1103e0db1d05e46753c724c707f4fc0f" exitCode=0 Oct 31 05:01:20.374640 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:20.374162 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg" Oct 31 05:01:20.374640 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:20.374466 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg" event={"ID":"d12de313-f7d1-4d6e-b2d7-0727073b032c","Type":"ContainerDied","Data":"4d2ad2b39c00ac5b93bac98d010d32ed1103e0db1d05e46753c724c707f4fc0f"} Oct 31 05:01:20.374640 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:20.374516 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg" event={"ID":"d12de313-f7d1-4d6e-b2d7-0727073b032c","Type":"ContainerDied","Data":"db4374bb5e4e0ddacb0295a966a4e6ebff6fd406886d24596e19f09b3c424509"} Oct 31 05:01:20.374640 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:20.374546 5019 scope.go:122] "RemoveContainer" containerID="4d2ad2b39c00ac5b93bac98d010d32ed1103e0db1d05e46753c724c707f4fc0f" Oct 31 05:01:20.379600 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:20.379563 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d12de313-f7d1-4d6e-b2d7-0727073b032c-istio-podinfo" pod "d12de313-f7d1-4d6e-b2d7-0727073b032c" (UID: "d12de313-f7d1-4d6e-b2d7-0727073b032c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:01:20.379724 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:20.379704 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d12de313-f7d1-4d6e-b2d7-0727073b032c-istio-envoy" pod "d12de313-f7d1-4d6e-b2d7-0727073b032c" (UID: "d12de313-f7d1-4d6e-b2d7-0727073b032c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:01:20.379937 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:20.379918 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d12de313-f7d1-4d6e-b2d7-0727073b032c-istio-token" pod "d12de313-f7d1-4d6e-b2d7-0727073b032c" (UID: "d12de313-f7d1-4d6e-b2d7-0727073b032c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:01:20.438084 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:20.438062 5019 scope.go:122] "RemoveContainer" containerID="4d2ad2b39c00ac5b93bac98d010d32ed1103e0db1d05e46753c724c707f4fc0f" Oct 31 05:01:20.438526 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:01:20.438484 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d2ad2b39c00ac5b93bac98d010d32ed1103e0db1d05e46753c724c707f4fc0f\": container with ID starting with 4d2ad2b39c00ac5b93bac98d010d32ed1103e0db1d05e46753c724c707f4fc0f not found: ID does not exist" containerID="4d2ad2b39c00ac5b93bac98d010d32ed1103e0db1d05e46753c724c707f4fc0f" Oct 31 05:01:20.438659 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:20.438527 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d2ad2b39c00ac5b93bac98d010d32ed1103e0db1d05e46753c724c707f4fc0f"} err="failed to get container status \"4d2ad2b39c00ac5b93bac98d010d32ed1103e0db1d05e46753c724c707f4fc0f\": rpc error: code = NotFound desc = could not find container \"4d2ad2b39c00ac5b93bac98d010d32ed1103e0db1d05e46753c724c707f4fc0f\": container with ID starting with 4d2ad2b39c00ac5b93bac98d010d32ed1103e0db1d05e46753c724c707f4fc0f not found: ID does not exist" Oct 31 05:01:20.489433 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:20.479115 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d12de313-f7d1-4d6e-b2d7-0727073b032c-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:01:20.489433 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:20.479159 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d12de313-f7d1-4d6e-b2d7-0727073b032c-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:01:20.489433 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:20.479182 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d12de313-f7d1-4d6e-b2d7-0727073b032c-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:01:20.489433 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:20.479195 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcfnn\" (UniqueName: \"kubernetes.io/projected/d12de313-f7d1-4d6e-b2d7-0727073b032c-kube-api-access-jcfnn\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:01:20.489433 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:20.479211 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d12de313-f7d1-4d6e-b2d7-0727073b032c-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:01:20.726607 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:20.726571 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg"] Oct 31 05:01:20.731364 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:20.731318 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-6b9mg"] Oct 31 05:01:20.790252 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:20.790216 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-x829c"] Oct 31 05:01:20.790633 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:20.790596 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-x829c" podUID="22bd2484-d363-4287-a424-29a99c3153ac" containerName="istio-proxy" containerID="cri-o://cc47c4219876b046a1e1f07e9ba418fccb8269f89357a67c4ae0288b24ae6b85" gracePeriod=30 Oct 31 05:01:21.180625 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.180590 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j" Oct 31 05:01:21.185174 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.185144 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v" Oct 31 05:01:21.307406 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.303485 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e107852e-0f1f-4705-bf39-a676808806ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e107852e-0f1f-4705-bf39-a676808806ef-istio-podinfo\") pod \"e107852e-0f1f-4705-bf39-a676808806ef\" (UID: \"e107852e-0f1f-4705-bf39-a676808806ef\") " Oct 31 05:01:21.307406 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.303541 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e107852e-0f1f-4705-bf39-a676808806ef-istio-token\" (UniqueName: \"kubernetes.io/projected/e107852e-0f1f-4705-bf39-a676808806ef-istio-token\") pod \"e107852e-0f1f-4705-bf39-a676808806ef\" (UID: \"e107852e-0f1f-4705-bf39-a676808806ef\") " Oct 31 05:01:21.307406 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.303599 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4e33d7c-168f-4001-8e39-22bea132574a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4e33d7c-168f-4001-8e39-22bea132574a-istio-envoy\") pod \"c4e33d7c-168f-4001-8e39-22bea132574a\" (UID: \"c4e33d7c-168f-4001-8e39-22bea132574a\") " Oct 31 05:01:21.307406 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.303652 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4e33d7c-168f-4001-8e39-22bea132574a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e33d7c-168f-4001-8e39-22bea132574a-workload-socket\") pod \"c4e33d7c-168f-4001-8e39-22bea132574a\" (UID: \"c4e33d7c-168f-4001-8e39-22bea132574a\") " Oct 31 05:01:21.307406 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.303687 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e107852e-0f1f-4705-bf39-a676808806ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e107852e-0f1f-4705-bf39-a676808806ef-istiod-ca-cert\") pod \"e107852e-0f1f-4705-bf39-a676808806ef\" (UID: \"e107852e-0f1f-4705-bf39-a676808806ef\") " Oct 31 05:01:21.307406 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.303788 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4e33d7c-168f-4001-8e39-22bea132574a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4e33d7c-168f-4001-8e39-22bea132574a-istiod-ca-cert\") pod \"c4e33d7c-168f-4001-8e39-22bea132574a\" (UID: \"c4e33d7c-168f-4001-8e39-22bea132574a\") " Oct 31 05:01:21.307406 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.303866 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e107852e-0f1f-4705-bf39-a676808806ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e107852e-0f1f-4705-bf39-a676808806ef-workload-socket\") pod \"e107852e-0f1f-4705-bf39-a676808806ef\" (UID: \"e107852e-0f1f-4705-bf39-a676808806ef\") " Oct 31 05:01:21.307406 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.303936 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4e33d7c-168f-4001-8e39-22bea132574a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4e33d7c-168f-4001-8e39-22bea132574a-workload-certs\") pod \"c4e33d7c-168f-4001-8e39-22bea132574a\" (UID: \"c4e33d7c-168f-4001-8e39-22bea132574a\") " Oct 31 05:01:21.307406 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.303968 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4e33d7c-168f-4001-8e39-22bea132574a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4e33d7c-168f-4001-8e39-22bea132574a-istio-podinfo\") pod \"c4e33d7c-168f-4001-8e39-22bea132574a\" (UID: \"c4e33d7c-168f-4001-8e39-22bea132574a\") " Oct 31 05:01:21.307406 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.304026 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e107852e-0f1f-4705-bf39-a676808806ef-kube-api-access-6gkt8\" (UniqueName: \"kubernetes.io/projected/e107852e-0f1f-4705-bf39-a676808806ef-kube-api-access-6gkt8\") pod \"e107852e-0f1f-4705-bf39-a676808806ef\" (UID: \"e107852e-0f1f-4705-bf39-a676808806ef\") " Oct 31 05:01:21.307406 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.304081 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e107852e-0f1f-4705-bf39-a676808806ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e107852e-0f1f-4705-bf39-a676808806ef-workload-certs\") pod \"e107852e-0f1f-4705-bf39-a676808806ef\" (UID: \"e107852e-0f1f-4705-bf39-a676808806ef\") " Oct 31 05:01:21.307406 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.304130 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e107852e-0f1f-4705-bf39-a676808806ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e107852e-0f1f-4705-bf39-a676808806ef-credential-socket\") pod \"e107852e-0f1f-4705-bf39-a676808806ef\" (UID: \"e107852e-0f1f-4705-bf39-a676808806ef\") " Oct 31 05:01:21.307406 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.304179 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e107852e-0f1f-4705-bf39-a676808806ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e107852e-0f1f-4705-bf39-a676808806ef-istio-envoy\") pod \"e107852e-0f1f-4705-bf39-a676808806ef\" (UID: \"e107852e-0f1f-4705-bf39-a676808806ef\") " Oct 31 05:01:21.307406 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.304291 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e107852e-0f1f-4705-bf39-a676808806ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e107852e-0f1f-4705-bf39-a676808806ef-istio-data\") pod \"e107852e-0f1f-4705-bf39-a676808806ef\" (UID: \"e107852e-0f1f-4705-bf39-a676808806ef\") " Oct 31 05:01:21.307406 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.304350 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4e33d7c-168f-4001-8e39-22bea132574a-istio-token\" (UniqueName: \"kubernetes.io/projected/c4e33d7c-168f-4001-8e39-22bea132574a-istio-token\") pod \"c4e33d7c-168f-4001-8e39-22bea132574a\" (UID: \"c4e33d7c-168f-4001-8e39-22bea132574a\") " Oct 31 05:01:21.307406 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.304384 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4e33d7c-168f-4001-8e39-22bea132574a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4e33d7c-168f-4001-8e39-22bea132574a-istio-data\") pod \"c4e33d7c-168f-4001-8e39-22bea132574a\" (UID: \"c4e33d7c-168f-4001-8e39-22bea132574a\") " Oct 31 05:01:21.307406 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.304421 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4e33d7c-168f-4001-8e39-22bea132574a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e33d7c-168f-4001-8e39-22bea132574a-credential-socket\") pod \"c4e33d7c-168f-4001-8e39-22bea132574a\" (UID: \"c4e33d7c-168f-4001-8e39-22bea132574a\") " Oct 31 05:01:21.307406 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.304489 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4e33d7c-168f-4001-8e39-22bea132574a-kube-api-access-dg428\" (UniqueName: \"kubernetes.io/projected/c4e33d7c-168f-4001-8e39-22bea132574a-kube-api-access-dg428\") pod \"c4e33d7c-168f-4001-8e39-22bea132574a\" (UID: \"c4e33d7c-168f-4001-8e39-22bea132574a\") " Oct 31 05:01:21.308845 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.308777 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4e33d7c-168f-4001-8e39-22bea132574a-istiod-ca-cert" pod "c4e33d7c-168f-4001-8e39-22bea132574a" (UID: "c4e33d7c-168f-4001-8e39-22bea132574a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:01:21.309144 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.309086 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4e33d7c-168f-4001-8e39-22bea132574a-credential-socket" pod "c4e33d7c-168f-4001-8e39-22bea132574a" (UID: "c4e33d7c-168f-4001-8e39-22bea132574a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:01:21.309305 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.309284 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e107852e-0f1f-4705-bf39-a676808806ef-workload-socket" pod "e107852e-0f1f-4705-bf39-a676808806ef" (UID: "e107852e-0f1f-4705-bf39-a676808806ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:01:21.309608 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.309585 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e107852e-0f1f-4705-bf39-a676808806ef-istiod-ca-cert" pod "e107852e-0f1f-4705-bf39-a676808806ef" (UID: "e107852e-0f1f-4705-bf39-a676808806ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:01:21.309762 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.309742 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e107852e-0f1f-4705-bf39-a676808806ef-workload-certs" pod "e107852e-0f1f-4705-bf39-a676808806ef" (UID: "e107852e-0f1f-4705-bf39-a676808806ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:01:21.310563 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.310532 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4e33d7c-168f-4001-8e39-22bea132574a-istio-data" pod "c4e33d7c-168f-4001-8e39-22bea132574a" (UID: "c4e33d7c-168f-4001-8e39-22bea132574a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:01:21.310798 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.310773 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4e33d7c-168f-4001-8e39-22bea132574a-workload-certs" pod "c4e33d7c-168f-4001-8e39-22bea132574a" (UID: "c4e33d7c-168f-4001-8e39-22bea132574a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:01:21.311101 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.311078 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4e33d7c-168f-4001-8e39-22bea132574a-workload-socket" pod "c4e33d7c-168f-4001-8e39-22bea132574a" (UID: "c4e33d7c-168f-4001-8e39-22bea132574a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:01:21.312848 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.312804 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e107852e-0f1f-4705-bf39-a676808806ef-istio-data" pod "e107852e-0f1f-4705-bf39-a676808806ef" (UID: "e107852e-0f1f-4705-bf39-a676808806ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:01:21.313045 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.313021 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e107852e-0f1f-4705-bf39-a676808806ef-credential-socket" pod "e107852e-0f1f-4705-bf39-a676808806ef" (UID: "e107852e-0f1f-4705-bf39-a676808806ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:01:21.313874 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.313844 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e107852e-0f1f-4705-bf39-a676808806ef-kube-api-access-6gkt8" pod "e107852e-0f1f-4705-bf39-a676808806ef" (UID: "e107852e-0f1f-4705-bf39-a676808806ef"). InnerVolumeSpecName "kube-api-access-6gkt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:01:21.321106 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.321076 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e107852e-0f1f-4705-bf39-a676808806ef-istio-envoy" pod "e107852e-0f1f-4705-bf39-a676808806ef" (UID: "e107852e-0f1f-4705-bf39-a676808806ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:01:21.321543 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.321180 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e107852e-0f1f-4705-bf39-a676808806ef-istio-token" pod "e107852e-0f1f-4705-bf39-a676808806ef" (UID: "e107852e-0f1f-4705-bf39-a676808806ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:01:21.321543 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.321201 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4e33d7c-168f-4001-8e39-22bea132574a-kube-api-access-dg428" pod "c4e33d7c-168f-4001-8e39-22bea132574a" (UID: "c4e33d7c-168f-4001-8e39-22bea132574a"). InnerVolumeSpecName "kube-api-access-dg428". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:01:21.321543 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.321324 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4e33d7c-168f-4001-8e39-22bea132574a-istio-podinfo" pod "c4e33d7c-168f-4001-8e39-22bea132574a" (UID: "c4e33d7c-168f-4001-8e39-22bea132574a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:01:21.321543 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.321528 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4e33d7c-168f-4001-8e39-22bea132574a-istio-token" pod "c4e33d7c-168f-4001-8e39-22bea132574a" (UID: "c4e33d7c-168f-4001-8e39-22bea132574a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:01:21.321543 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.321529 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e107852e-0f1f-4705-bf39-a676808806ef-istio-podinfo" pod "e107852e-0f1f-4705-bf39-a676808806ef" (UID: "e107852e-0f1f-4705-bf39-a676808806ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:01:21.322511 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.322492 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4e33d7c-168f-4001-8e39-22bea132574a-istio-envoy" pod "c4e33d7c-168f-4001-8e39-22bea132574a" (UID: "c4e33d7c-168f-4001-8e39-22bea132574a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:01:21.400536 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.400503 5019 generic.go:356] "Generic (PLEG): container finished" podID="e107852e-0f1f-4705-bf39-a676808806ef" containerID="b988f154cd8443fafa6814fc2771b9b20a9b97df6edae4aaa1d7b3b4e25f5950" exitCode=0 Oct 31 05:01:21.400735 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.400609 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j" Oct 31 05:01:21.404480 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.404449 5019 generic.go:356] "Generic (PLEG): container finished" podID="c4e33d7c-168f-4001-8e39-22bea132574a" containerID="1264bd0ec634bcf3505b1094dbd8ceec6d8fda0fdd152b41795dbc63e4c43b18" exitCode=0 Oct 31 05:01:21.404671 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.404556 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v" Oct 31 05:01:21.408905 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.408877 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4e33d7c-168f-4001-8e39-22bea132574a-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:01:21.409074 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.409048 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e107852e-0f1f-4705-bf39-a676808806ef-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:01:21.409074 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.409070 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4e33d7c-168f-4001-8e39-22bea132574a-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:01:21.409291 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.409084 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4e33d7c-168f-4001-8e39-22bea132574a-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:01:21.409291 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.409098 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gkt8\" (UniqueName: \"kubernetes.io/projected/e107852e-0f1f-4705-bf39-a676808806ef-kube-api-access-6gkt8\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:01:21.409291 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.409112 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e107852e-0f1f-4705-bf39-a676808806ef-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:01:21.409291 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.409125 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e107852e-0f1f-4705-bf39-a676808806ef-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:01:21.409291 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.409138 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e107852e-0f1f-4705-bf39-a676808806ef-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:01:21.409291 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.409164 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e107852e-0f1f-4705-bf39-a676808806ef-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:01:21.409291 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.409176 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4e33d7c-168f-4001-8e39-22bea132574a-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:01:21.409291 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.409188 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4e33d7c-168f-4001-8e39-22bea132574a-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:01:21.409291 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.409201 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e33d7c-168f-4001-8e39-22bea132574a-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:01:21.409291 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.409214 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg428\" (UniqueName: \"kubernetes.io/projected/c4e33d7c-168f-4001-8e39-22bea132574a-kube-api-access-dg428\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:01:21.409291 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.409228 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e107852e-0f1f-4705-bf39-a676808806ef-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:01:21.409291 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.409241 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e107852e-0f1f-4705-bf39-a676808806ef-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:01:21.409291 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.409254 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4e33d7c-168f-4001-8e39-22bea132574a-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:01:21.409291 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.409267 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e33d7c-168f-4001-8e39-22bea132574a-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:01:21.409291 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.409279 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e107852e-0f1f-4705-bf39-a676808806ef-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:01:21.423344 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.423301 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d12de313-f7d1-4d6e-b2d7-0727073b032c" path="/var/lib/kubelet/pods/d12de313-f7d1-4d6e-b2d7-0727073b032c/volumes" Oct 31 05:01:21.424270 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.424244 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j" event={"ID":"e107852e-0f1f-4705-bf39-a676808806ef","Type":"ContainerDied","Data":"b988f154cd8443fafa6814fc2771b9b20a9b97df6edae4aaa1d7b3b4e25f5950"} Oct 31 05:01:21.424415 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.424276 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j" event={"ID":"e107852e-0f1f-4705-bf39-a676808806ef","Type":"ContainerDied","Data":"e809418e6d2904b1514bfb9be8e45e2c5dddae293bb7d30fa1ea0a8d11967ced"} Oct 31 05:01:21.424415 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.424292 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v" event={"ID":"c4e33d7c-168f-4001-8e39-22bea132574a","Type":"ContainerDied","Data":"1264bd0ec634bcf3505b1094dbd8ceec6d8fda0fdd152b41795dbc63e4c43b18"} Oct 31 05:01:21.424415 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.424305 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v" event={"ID":"c4e33d7c-168f-4001-8e39-22bea132574a","Type":"ContainerDied","Data":"330833c6127d895bb14be86276cc63041d5ec617006184cc090b5564e51b408e"} Oct 31 05:01:21.424415 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.424324 5019 scope.go:122] "RemoveContainer" containerID="b988f154cd8443fafa6814fc2771b9b20a9b97df6edae4aaa1d7b3b4e25f5950" Oct 31 05:01:21.442164 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.442140 5019 scope.go:122] "RemoveContainer" containerID="b988f154cd8443fafa6814fc2771b9b20a9b97df6edae4aaa1d7b3b4e25f5950" Oct 31 05:01:21.442514 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:01:21.442493 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b988f154cd8443fafa6814fc2771b9b20a9b97df6edae4aaa1d7b3b4e25f5950\": container with ID starting with b988f154cd8443fafa6814fc2771b9b20a9b97df6edae4aaa1d7b3b4e25f5950 not found: ID does not exist" containerID="b988f154cd8443fafa6814fc2771b9b20a9b97df6edae4aaa1d7b3b4e25f5950" Oct 31 05:01:21.442586 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.442524 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b988f154cd8443fafa6814fc2771b9b20a9b97df6edae4aaa1d7b3b4e25f5950"} err="failed to get container status \"b988f154cd8443fafa6814fc2771b9b20a9b97df6edae4aaa1d7b3b4e25f5950\": rpc error: code = NotFound desc = could not find container \"b988f154cd8443fafa6814fc2771b9b20a9b97df6edae4aaa1d7b3b4e25f5950\": container with ID starting with b988f154cd8443fafa6814fc2771b9b20a9b97df6edae4aaa1d7b3b4e25f5950 not found: ID does not exist" Oct 31 05:01:21.442586 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.442544 5019 scope.go:122] "RemoveContainer" containerID="1264bd0ec634bcf3505b1094dbd8ceec6d8fda0fdd152b41795dbc63e4c43b18" Oct 31 05:01:21.454924 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.454902 5019 scope.go:122] "RemoveContainer" containerID="13130f4cbef0ca565dad709c9689dde75da0cc0e67e923aa4f059cdd2af27a83" Oct 31 05:01:21.466409 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.466381 5019 scope.go:122] "RemoveContainer" containerID="1264bd0ec634bcf3505b1094dbd8ceec6d8fda0fdd152b41795dbc63e4c43b18" Oct 31 05:01:21.466773 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:01:21.466740 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1264bd0ec634bcf3505b1094dbd8ceec6d8fda0fdd152b41795dbc63e4c43b18\": container with ID starting with 1264bd0ec634bcf3505b1094dbd8ceec6d8fda0fdd152b41795dbc63e4c43b18 not found: ID does not exist" containerID="1264bd0ec634bcf3505b1094dbd8ceec6d8fda0fdd152b41795dbc63e4c43b18" Oct 31 05:01:21.466895 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.466782 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1264bd0ec634bcf3505b1094dbd8ceec6d8fda0fdd152b41795dbc63e4c43b18"} err="failed to get container status \"1264bd0ec634bcf3505b1094dbd8ceec6d8fda0fdd152b41795dbc63e4c43b18\": rpc error: code = NotFound desc = could not find container \"1264bd0ec634bcf3505b1094dbd8ceec6d8fda0fdd152b41795dbc63e4c43b18\": container with ID starting with 1264bd0ec634bcf3505b1094dbd8ceec6d8fda0fdd152b41795dbc63e4c43b18 not found: ID does not exist" Oct 31 05:01:21.466895 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.466807 5019 scope.go:122] "RemoveContainer" containerID="13130f4cbef0ca565dad709c9689dde75da0cc0e67e923aa4f059cdd2af27a83" Oct 31 05:01:21.467430 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:01:21.467401 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13130f4cbef0ca565dad709c9689dde75da0cc0e67e923aa4f059cdd2af27a83\": container with ID starting with 13130f4cbef0ca565dad709c9689dde75da0cc0e67e923aa4f059cdd2af27a83 not found: ID does not exist" containerID="13130f4cbef0ca565dad709c9689dde75da0cc0e67e923aa4f059cdd2af27a83" Oct 31 05:01:21.467541 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.467437 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13130f4cbef0ca565dad709c9689dde75da0cc0e67e923aa4f059cdd2af27a83"} err="failed to get container status \"13130f4cbef0ca565dad709c9689dde75da0cc0e67e923aa4f059cdd2af27a83\": rpc error: code = NotFound desc = could not find container \"13130f4cbef0ca565dad709c9689dde75da0cc0e67e923aa4f059cdd2af27a83\": container with ID starting with 13130f4cbef0ca565dad709c9689dde75da0cc0e67e923aa4f059cdd2af27a83 not found: ID does not exist" Oct 31 05:01:21.477386 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.475385 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j"] Oct 31 05:01:21.483469 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.483440 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-md28j"] Oct 31 05:01:21.511271 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.511167 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v"] Oct 31 05:01:21.519454 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:21.519419 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5547v"] Oct 31 05:01:22.394216 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:01:22.394117 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:01:23.468528 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:23.468481 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4e33d7c-168f-4001-8e39-22bea132574a" path="/var/lib/kubelet/pods/c4e33d7c-168f-4001-8e39-22bea132574a/volumes" Oct 31 05:01:23.469377 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:23.469352 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e107852e-0f1f-4705-bf39-a676808806ef" path="/var/lib/kubelet/pods/e107852e-0f1f-4705-bf39-a676808806ef/volumes" Oct 31 05:01:23.476120 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:23.476093 5019 generic.go:356] "Generic (PLEG): container finished" podID="2d771867-36b5-4b55-afb4-f889d1d91784" containerID="a8c165f1cf4f8c49f2f4ed1ad38f945ec38654b0460f84dd3abe49d97607631f" exitCode=0 Oct 31 05:01:23.476481 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:23.476456 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz" event={"ID":"2d771867-36b5-4b55-afb4-f889d1d91784","Type":"ContainerDied","Data":"a8c165f1cf4f8c49f2f4ed1ad38f945ec38654b0460f84dd3abe49d97607631f"} Oct 31 05:01:23.476630 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:23.476613 5019 scope.go:122] "RemoveContainer" containerID="85a2b9dd4c1a3477ce87dd382e1a0c28fde2e96c96b8d364cc1769bfa3a2ae46" Oct 31 05:01:23.690405 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:23.690377 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz" Oct 31 05:01:23.753227 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:23.753150 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d771867-36b5-4b55-afb4-f889d1d91784-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d771867-36b5-4b55-afb4-f889d1d91784-istio-data\") pod \"2d771867-36b5-4b55-afb4-f889d1d91784\" (UID: \"2d771867-36b5-4b55-afb4-f889d1d91784\") " Oct 31 05:01:23.753227 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:23.753214 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d771867-36b5-4b55-afb4-f889d1d91784-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d771867-36b5-4b55-afb4-f889d1d91784-workload-socket\") pod \"2d771867-36b5-4b55-afb4-f889d1d91784\" (UID: \"2d771867-36b5-4b55-afb4-f889d1d91784\") " Oct 31 05:01:23.753485 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:23.753357 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d771867-36b5-4b55-afb4-f889d1d91784-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d771867-36b5-4b55-afb4-f889d1d91784-istio-envoy\") pod \"2d771867-36b5-4b55-afb4-f889d1d91784\" (UID: \"2d771867-36b5-4b55-afb4-f889d1d91784\") " Oct 31 05:01:23.753485 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:23.753415 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d771867-36b5-4b55-afb4-f889d1d91784-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d771867-36b5-4b55-afb4-f889d1d91784-istio-podinfo\") pod \"2d771867-36b5-4b55-afb4-f889d1d91784\" (UID: \"2d771867-36b5-4b55-afb4-f889d1d91784\") " Oct 31 05:01:23.753598 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:23.753496 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d771867-36b5-4b55-afb4-f889d1d91784-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d771867-36b5-4b55-afb4-f889d1d91784-workload-certs\") pod \"2d771867-36b5-4b55-afb4-f889d1d91784\" (UID: \"2d771867-36b5-4b55-afb4-f889d1d91784\") " Oct 31 05:01:23.753598 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:23.753564 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d771867-36b5-4b55-afb4-f889d1d91784-kube-api-access-ts69w\" (UniqueName: \"kubernetes.io/projected/2d771867-36b5-4b55-afb4-f889d1d91784-kube-api-access-ts69w\") pod \"2d771867-36b5-4b55-afb4-f889d1d91784\" (UID: \"2d771867-36b5-4b55-afb4-f889d1d91784\") " Oct 31 05:01:23.753715 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:23.753664 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d771867-36b5-4b55-afb4-f889d1d91784-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d771867-36b5-4b55-afb4-f889d1d91784-credential-socket\") pod \"2d771867-36b5-4b55-afb4-f889d1d91784\" (UID: \"2d771867-36b5-4b55-afb4-f889d1d91784\") " Oct 31 05:01:23.753778 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:23.753761 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d771867-36b5-4b55-afb4-f889d1d91784-istio-token\" (UniqueName: \"kubernetes.io/projected/2d771867-36b5-4b55-afb4-f889d1d91784-istio-token\") pod \"2d771867-36b5-4b55-afb4-f889d1d91784\" (UID: \"2d771867-36b5-4b55-afb4-f889d1d91784\") " Oct 31 05:01:23.753869 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:23.753855 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d771867-36b5-4b55-afb4-f889d1d91784-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d771867-36b5-4b55-afb4-f889d1d91784-istiod-ca-cert\") pod \"2d771867-36b5-4b55-afb4-f889d1d91784\" (UID: \"2d771867-36b5-4b55-afb4-f889d1d91784\") " Oct 31 05:01:23.754966 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:23.754930 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d771867-36b5-4b55-afb4-f889d1d91784-workload-socket" pod "2d771867-36b5-4b55-afb4-f889d1d91784" (UID: "2d771867-36b5-4b55-afb4-f889d1d91784"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:01:23.755118 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:23.755019 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d771867-36b5-4b55-afb4-f889d1d91784-istio-data" pod "2d771867-36b5-4b55-afb4-f889d1d91784" (UID: "2d771867-36b5-4b55-afb4-f889d1d91784"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:01:23.755238 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:23.755216 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d771867-36b5-4b55-afb4-f889d1d91784-workload-certs" pod "2d771867-36b5-4b55-afb4-f889d1d91784" (UID: "2d771867-36b5-4b55-afb4-f889d1d91784"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:01:23.755465 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:23.755442 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d771867-36b5-4b55-afb4-f889d1d91784-credential-socket" pod "2d771867-36b5-4b55-afb4-f889d1d91784" (UID: "2d771867-36b5-4b55-afb4-f889d1d91784"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:01:23.755465 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:23.755457 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d771867-36b5-4b55-afb4-f889d1d91784-istiod-ca-cert" pod "2d771867-36b5-4b55-afb4-f889d1d91784" (UID: "2d771867-36b5-4b55-afb4-f889d1d91784"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:01:23.757964 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:23.757936 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d771867-36b5-4b55-afb4-f889d1d91784-kube-api-access-ts69w" pod "2d771867-36b5-4b55-afb4-f889d1d91784" (UID: "2d771867-36b5-4b55-afb4-f889d1d91784"). InnerVolumeSpecName "kube-api-access-ts69w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:01:23.759980 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:23.759954 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d771867-36b5-4b55-afb4-f889d1d91784-istio-envoy" pod "2d771867-36b5-4b55-afb4-f889d1d91784" (UID: "2d771867-36b5-4b55-afb4-f889d1d91784"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:01:23.760732 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:23.760710 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d771867-36b5-4b55-afb4-f889d1d91784-istio-podinfo" pod "2d771867-36b5-4b55-afb4-f889d1d91784" (UID: "2d771867-36b5-4b55-afb4-f889d1d91784"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:01:23.761186 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:23.761163 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d771867-36b5-4b55-afb4-f889d1d91784-istio-token" pod "2d771867-36b5-4b55-afb4-f889d1d91784" (UID: "2d771867-36b5-4b55-afb4-f889d1d91784"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:01:23.874230 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:23.862125 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d771867-36b5-4b55-afb4-f889d1d91784-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:01:23.874230 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:23.862167 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d771867-36b5-4b55-afb4-f889d1d91784-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:01:23.874230 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:23.862191 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d771867-36b5-4b55-afb4-f889d1d91784-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:01:23.874230 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:23.862369 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d771867-36b5-4b55-afb4-f889d1d91784-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:01:23.874230 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:23.862387 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d771867-36b5-4b55-afb4-f889d1d91784-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:01:23.874230 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:23.862422 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d771867-36b5-4b55-afb4-f889d1d91784-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:01:23.874230 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:23.862442 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d771867-36b5-4b55-afb4-f889d1d91784-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:01:23.874230 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:23.862456 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts69w\" (UniqueName: \"kubernetes.io/projected/2d771867-36b5-4b55-afb4-f889d1d91784-kube-api-access-ts69w\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:01:23.874230 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:23.862469 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d771867-36b5-4b55-afb4-f889d1d91784-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:01:24.495427 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:24.495393 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz" Oct 31 05:01:24.496422 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:24.495430 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz" event={"ID":"2d771867-36b5-4b55-afb4-f889d1d91784","Type":"ContainerDied","Data":"bfa6b5be2d47068f00e3bb2b7f3bf24be0d46d3eaf7502c387e9ac7d92a68a13"} Oct 31 05:01:24.496609 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:24.496463 5019 scope.go:122] "RemoveContainer" containerID="a8c165f1cf4f8c49f2f4ed1ad38f945ec38654b0460f84dd3abe49d97607631f" Oct 31 05:01:24.566157 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:24.566110 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz"] Oct 31 05:01:24.579815 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:24.579776 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-77a153-openshift-default-7778f45d97-t2jqz"] Oct 31 05:01:25.414954 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:25.414918 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d771867-36b5-4b55-afb4-f889d1d91784" path="/var/lib/kubelet/pods/2d771867-36b5-4b55-afb4-f889d1d91784/volumes" Oct 31 05:01:26.132316 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:26.132289 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-x829c" Oct 31 05:01:26.309971 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:26.309931 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22bd2484-d363-4287-a424-29a99c3153ac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22bd2484-d363-4287-a424-29a99c3153ac-workload-socket\") pod \"22bd2484-d363-4287-a424-29a99c3153ac\" (UID: \"22bd2484-d363-4287-a424-29a99c3153ac\") " Oct 31 05:01:26.309971 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:26.309995 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22bd2484-d363-4287-a424-29a99c3153ac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22bd2484-d363-4287-a424-29a99c3153ac-istiod-ca-cert\") pod \"22bd2484-d363-4287-a424-29a99c3153ac\" (UID: \"22bd2484-d363-4287-a424-29a99c3153ac\") " Oct 31 05:01:26.310852 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:26.310818 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22bd2484-d363-4287-a424-29a99c3153ac-istiod-ca-cert" pod "22bd2484-d363-4287-a424-29a99c3153ac" (UID: "22bd2484-d363-4287-a424-29a99c3153ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:01:26.310981 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:26.310034 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22bd2484-d363-4287-a424-29a99c3153ac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22bd2484-d363-4287-a424-29a99c3153ac-istio-data\") pod \"22bd2484-d363-4287-a424-29a99c3153ac\" (UID: \"22bd2484-d363-4287-a424-29a99c3153ac\") " Oct 31 05:01:26.311046 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:26.310983 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22bd2484-d363-4287-a424-29a99c3153ac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22bd2484-d363-4287-a424-29a99c3153ac-istio-envoy\") pod \"22bd2484-d363-4287-a424-29a99c3153ac\" (UID: \"22bd2484-d363-4287-a424-29a99c3153ac\") " Oct 31 05:01:26.311109 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:26.311071 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22bd2484-d363-4287-a424-29a99c3153ac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22bd2484-d363-4287-a424-29a99c3153ac-istio-podinfo\") pod \"22bd2484-d363-4287-a424-29a99c3153ac\" (UID: \"22bd2484-d363-4287-a424-29a99c3153ac\") " Oct 31 05:01:26.311165 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:26.311145 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22bd2484-d363-4287-a424-29a99c3153ac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22bd2484-d363-4287-a424-29a99c3153ac-credential-socket\") pod \"22bd2484-d363-4287-a424-29a99c3153ac\" (UID: \"22bd2484-d363-4287-a424-29a99c3153ac\") " Oct 31 05:01:26.311246 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:26.311228 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22bd2484-d363-4287-a424-29a99c3153ac-istio-token\" (UniqueName: \"kubernetes.io/projected/22bd2484-d363-4287-a424-29a99c3153ac-istio-token\") pod \"22bd2484-d363-4287-a424-29a99c3153ac\" (UID: \"22bd2484-d363-4287-a424-29a99c3153ac\") " Oct 31 05:01:26.311477 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:26.311452 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22bd2484-d363-4287-a424-29a99c3153ac-istio-data" pod "22bd2484-d363-4287-a424-29a99c3153ac" (UID: "22bd2484-d363-4287-a424-29a99c3153ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:01:26.311545 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:26.311508 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22bd2484-d363-4287-a424-29a99c3153ac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22bd2484-d363-4287-a424-29a99c3153ac-workload-certs\") pod \"22bd2484-d363-4287-a424-29a99c3153ac\" (UID: \"22bd2484-d363-4287-a424-29a99c3153ac\") " Oct 31 05:01:26.311623 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:26.311552 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22bd2484-d363-4287-a424-29a99c3153ac-kube-api-access-xrpjb\" (UniqueName: \"kubernetes.io/projected/22bd2484-d363-4287-a424-29a99c3153ac-kube-api-access-xrpjb\") pod \"22bd2484-d363-4287-a424-29a99c3153ac\" (UID: \"22bd2484-d363-4287-a424-29a99c3153ac\") " Oct 31 05:01:26.312141 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:26.312113 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22bd2484-d363-4287-a424-29a99c3153ac-workload-socket" pod "22bd2484-d363-4287-a424-29a99c3153ac" (UID: "22bd2484-d363-4287-a424-29a99c3153ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:01:26.312141 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:26.312131 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22bd2484-d363-4287-a424-29a99c3153ac-credential-socket" pod "22bd2484-d363-4287-a424-29a99c3153ac" (UID: "22bd2484-d363-4287-a424-29a99c3153ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:01:26.312390 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:26.312364 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22bd2484-d363-4287-a424-29a99c3153ac-workload-certs" pod "22bd2484-d363-4287-a424-29a99c3153ac" (UID: "22bd2484-d363-4287-a424-29a99c3153ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:01:26.312626 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:26.312605 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22bd2484-d363-4287-a424-29a99c3153ac-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:01:26.312719 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:26.312633 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22bd2484-d363-4287-a424-29a99c3153ac-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:01:26.312719 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:26.312648 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22bd2484-d363-4287-a424-29a99c3153ac-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:01:26.312719 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:26.312661 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22bd2484-d363-4287-a424-29a99c3153ac-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:01:26.312719 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:26.312673 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22bd2484-d363-4287-a424-29a99c3153ac-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:01:26.314890 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:26.314865 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22bd2484-d363-4287-a424-29a99c3153ac-istio-podinfo" pod "22bd2484-d363-4287-a424-29a99c3153ac" (UID: "22bd2484-d363-4287-a424-29a99c3153ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:01:26.315048 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:26.314929 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22bd2484-d363-4287-a424-29a99c3153ac-istio-envoy" pod "22bd2484-d363-4287-a424-29a99c3153ac" (UID: "22bd2484-d363-4287-a424-29a99c3153ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:01:26.315048 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:26.314937 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22bd2484-d363-4287-a424-29a99c3153ac-istio-token" pod "22bd2484-d363-4287-a424-29a99c3153ac" (UID: "22bd2484-d363-4287-a424-29a99c3153ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:01:26.315048 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:26.315043 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22bd2484-d363-4287-a424-29a99c3153ac-kube-api-access-xrpjb" pod "22bd2484-d363-4287-a424-29a99c3153ac" (UID: "22bd2484-d363-4287-a424-29a99c3153ac"). InnerVolumeSpecName "kube-api-access-xrpjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:01:26.414772 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:26.414680 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrpjb\" (UniqueName: \"kubernetes.io/projected/22bd2484-d363-4287-a424-29a99c3153ac-kube-api-access-xrpjb\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:01:26.414772 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:26.414715 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22bd2484-d363-4287-a424-29a99c3153ac-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:01:26.414772 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:26.414728 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22bd2484-d363-4287-a424-29a99c3153ac-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:01:26.414772 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:26.414739 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22bd2484-d363-4287-a424-29a99c3153ac-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:01:26.520735 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:26.520697 5019 generic.go:356] "Generic (PLEG): container finished" podID="22bd2484-d363-4287-a424-29a99c3153ac" containerID="cc47c4219876b046a1e1f07e9ba418fccb8269f89357a67c4ae0288b24ae6b85" exitCode=0 Oct 31 05:01:26.520907 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:26.520760 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-x829c" Oct 31 05:01:26.520907 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:26.520784 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-x829c" event={"ID":"22bd2484-d363-4287-a424-29a99c3153ac","Type":"ContainerDied","Data":"cc47c4219876b046a1e1f07e9ba418fccb8269f89357a67c4ae0288b24ae6b85"} Oct 31 05:01:26.520907 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:26.520866 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-x829c" event={"ID":"22bd2484-d363-4287-a424-29a99c3153ac","Type":"ContainerDied","Data":"a28e4ebe40f3403bc5f54aa5afef6172080e9b78150c6571a8d351ddc8836624"} Oct 31 05:01:26.520907 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:26.520895 5019 scope.go:122] "RemoveContainer" containerID="cc47c4219876b046a1e1f07e9ba418fccb8269f89357a67c4ae0288b24ae6b85" Oct 31 05:01:26.535725 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:26.535703 5019 scope.go:122] "RemoveContainer" containerID="cc47c4219876b046a1e1f07e9ba418fccb8269f89357a67c4ae0288b24ae6b85" Oct 31 05:01:26.536006 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:01:26.535986 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc47c4219876b046a1e1f07e9ba418fccb8269f89357a67c4ae0288b24ae6b85\": container with ID starting with cc47c4219876b046a1e1f07e9ba418fccb8269f89357a67c4ae0288b24ae6b85 not found: ID does not exist" containerID="cc47c4219876b046a1e1f07e9ba418fccb8269f89357a67c4ae0288b24ae6b85" Oct 31 05:01:26.536057 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:26.536015 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc47c4219876b046a1e1f07e9ba418fccb8269f89357a67c4ae0288b24ae6b85"} err="failed to get container status \"cc47c4219876b046a1e1f07e9ba418fccb8269f89357a67c4ae0288b24ae6b85\": rpc error: code = NotFound desc = could not find container \"cc47c4219876b046a1e1f07e9ba418fccb8269f89357a67c4ae0288b24ae6b85\": container with ID starting with cc47c4219876b046a1e1f07e9ba418fccb8269f89357a67c4ae0288b24ae6b85 not found: ID does not exist" Oct 31 05:01:26.580603 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:26.580579 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-x829c"] Oct 31 05:01:26.587948 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:26.587923 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-x829c"] Oct 31 05:01:27.414774 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:01:27.414737 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22bd2484-d363-4287-a424-29a99c3153ac" path="/var/lib/kubelet/pods/22bd2484-d363-4287-a424-29a99c3153ac/volumes" Oct 31 05:01:52.395211 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:01:52.395117 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:02:12.792126 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:12.792076 5019 scope.go:122] "RemoveContainer" containerID="81454944cd236fc0d7c6690d90cbe9bdf84a2acb73e417d1e9ddf6d648ac9107" Oct 31 05:02:22.397622 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:02:22.397577 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:02:31.174386 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:31.174327 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b9214b0-zmjk5"] Oct 31 05:02:31.175297 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:31.175277 5019 cpu_manager.go:414] "containerMap: removing container" podUID="22bd2484-d363-4287-a424-29a99c3153ac" containerName="istio-proxy" Oct 31 05:02:31.175366 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:31.175298 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22bd2484-d363-4287-a424-29a99c3153ac" containerName="istio-proxy" Oct 31 05:02:31.175366 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:31.175317 5019 cpu_manager.go:414] "containerMap: removing container" podUID="deaadd33-dcef-43df-a66e-9fd41434de6f" containerName="cleanup" Oct 31 05:02:31.175366 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:31.175322 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deaadd33-dcef-43df-a66e-9fd41434de6f" containerName="cleanup" Oct 31 05:02:31.175366 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:31.175348 5019 cpu_manager.go:414] "containerMap: removing container" podUID="c4e33d7c-168f-4001-8e39-22bea132574a" containerName="istio-proxy" Oct 31 05:02:31.175366 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:31.175354 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4e33d7c-168f-4001-8e39-22bea132574a" containerName="istio-proxy" Oct 31 05:02:31.175366 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:31.175362 5019 cpu_manager.go:414] "containerMap: removing container" podUID="c4e33d7c-168f-4001-8e39-22bea132574a" containerName="istio-proxy" Oct 31 05:02:31.175582 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:31.175368 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4e33d7c-168f-4001-8e39-22bea132574a" containerName="istio-proxy" Oct 31 05:02:31.175582 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:31.175378 5019 cpu_manager.go:414] "containerMap: removing container" podUID="7ce5abc5-55a0-4b15-80a6-7e21d360e389" containerName="istio-proxy" Oct 31 05:02:31.175582 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:31.175385 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ce5abc5-55a0-4b15-80a6-7e21d360e389" containerName="istio-proxy" Oct 31 05:02:31.175582 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:31.175391 5019 cpu_manager.go:414] "containerMap: removing container" podUID="e107852e-0f1f-4705-bf39-a676808806ef" containerName="istio-proxy" Oct 31 05:02:31.175582 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:31.175395 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e107852e-0f1f-4705-bf39-a676808806ef" containerName="istio-proxy" Oct 31 05:02:31.175582 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:31.175402 5019 cpu_manager.go:414] "containerMap: removing container" podUID="7ce5abc5-55a0-4b15-80a6-7e21d360e389" containerName="istio-proxy" Oct 31 05:02:31.175582 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:31.175406 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ce5abc5-55a0-4b15-80a6-7e21d360e389" containerName="istio-proxy" Oct 31 05:02:31.175582 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:31.175424 5019 cpu_manager.go:414] "containerMap: removing container" podUID="7c20d99d-7ce4-4bb8-a4d9-188cec401d75" containerName="cleanup" Oct 31 05:02:31.175582 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:31.175429 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c20d99d-7ce4-4bb8-a4d9-188cec401d75" containerName="cleanup" Oct 31 05:02:31.175582 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:31.175440 5019 cpu_manager.go:414] "containerMap: removing container" podUID="d12de313-f7d1-4d6e-b2d7-0727073b032c" containerName="istio-proxy" Oct 31 05:02:31.175582 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:31.175444 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d12de313-f7d1-4d6e-b2d7-0727073b032c" containerName="istio-proxy" Oct 31 05:02:31.175582 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:31.175461 5019 cpu_manager.go:414] "containerMap: removing container" podUID="cc1adc72-6548-4daf-9dc1-2b96c853cdbf" containerName="istio-proxy" Oct 31 05:02:31.175582 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:31.175465 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc1adc72-6548-4daf-9dc1-2b96c853cdbf" containerName="istio-proxy" Oct 31 05:02:31.175582 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:31.175485 5019 cpu_manager.go:414] "containerMap: removing container" podUID="2d771867-36b5-4b55-afb4-f889d1d91784" containerName="istio-proxy" Oct 31 05:02:31.175582 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:31.175491 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d771867-36b5-4b55-afb4-f889d1d91784" containerName="istio-proxy" Oct 31 05:02:31.175582 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:31.175498 5019 cpu_manager.go:414] "containerMap: removing container" podUID="2d771867-36b5-4b55-afb4-f889d1d91784" containerName="istio-proxy" Oct 31 05:02:31.175582 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:31.175502 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d771867-36b5-4b55-afb4-f889d1d91784" containerName="istio-proxy" Oct 31 05:02:31.175582 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:31.175511 5019 cpu_manager.go:414] "containerMap: removing container" podUID="7ab61246-5d2d-4a2a-8c42-fff22960018f" containerName="cleanup" Oct 31 05:02:31.175582 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:31.175515 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ab61246-5d2d-4a2a-8c42-fff22960018f" containerName="cleanup" Oct 31 05:02:31.176371 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:31.175663 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d771867-36b5-4b55-afb4-f889d1d91784" containerName="istio-proxy" Oct 31 05:02:31.176371 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:31.175672 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d771867-36b5-4b55-afb4-f889d1d91784" containerName="istio-proxy" Oct 31 05:02:31.176371 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:31.175682 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc1adc72-6548-4daf-9dc1-2b96c853cdbf" containerName="istio-proxy" Oct 31 05:02:31.176371 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:31.175690 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="22bd2484-d363-4287-a424-29a99c3153ac" containerName="istio-proxy" Oct 31 05:02:31.176371 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:31.175698 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="deaadd33-dcef-43df-a66e-9fd41434de6f" containerName="cleanup" Oct 31 05:02:31.176371 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:31.175704 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4e33d7c-168f-4001-8e39-22bea132574a" containerName="istio-proxy" Oct 31 05:02:31.176371 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:31.175713 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4e33d7c-168f-4001-8e39-22bea132574a" containerName="istio-proxy" Oct 31 05:02:31.176371 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:31.175720 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ce5abc5-55a0-4b15-80a6-7e21d360e389" containerName="istio-proxy" Oct 31 05:02:31.176371 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:31.175725 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e107852e-0f1f-4705-bf39-a676808806ef" containerName="istio-proxy" Oct 31 05:02:31.176371 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:31.175730 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ce5abc5-55a0-4b15-80a6-7e21d360e389" containerName="istio-proxy" Oct 31 05:02:31.176371 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:31.175738 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c20d99d-7ce4-4bb8-a4d9-188cec401d75" containerName="cleanup" Oct 31 05:02:31.176371 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:31.175745 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="d12de313-f7d1-4d6e-b2d7-0727073b032c" containerName="istio-proxy" Oct 31 05:02:31.176371 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:31.175753 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc1adc72-6548-4daf-9dc1-2b96c853cdbf" containerName="istio-proxy" Oct 31 05:02:31.181784 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:31.181757 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b9214b0-zmjk5" Oct 31 05:02:31.202259 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:31.202225 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b9214b0-zmjk5"] Oct 31 05:02:31.203632 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:31.203546 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:02:31.203632 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:31.203574 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:02:31.203632 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:31.203584 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:02:31.220403 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:31.209788 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 05:02:31.300813 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:31.300766 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/856f1832-9e23-4a5d-ace0-cbd6ac20d469-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b9214b0-zmjk5\" (UID: \"856f1832-9e23-4a5d-ace0-cbd6ac20d469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b9214b0-zmjk5" Oct 31 05:02:31.301142 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:31.301114 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/856f1832-9e23-4a5d-ace0-cbd6ac20d469-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b9214b0-zmjk5\" (UID: \"856f1832-9e23-4a5d-ace0-cbd6ac20d469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b9214b0-zmjk5" Oct 31 05:02:31.301204 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:31.301170 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p4tj\" (UniqueName: \"kubernetes.io/projected/856f1832-9e23-4a5d-ace0-cbd6ac20d469-kube-api-access-6p4tj\") pod \"maas-api-revoke-sub-2b9214b0-zmjk5\" (UID: \"856f1832-9e23-4a5d-ace0-cbd6ac20d469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b9214b0-zmjk5" Oct 31 05:02:31.402965 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:31.402919 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/856f1832-9e23-4a5d-ace0-cbd6ac20d469-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b9214b0-zmjk5\" (UID: \"856f1832-9e23-4a5d-ace0-cbd6ac20d469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b9214b0-zmjk5" Oct 31 05:02:31.403212 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:31.402997 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p4tj\" (UniqueName: \"kubernetes.io/projected/856f1832-9e23-4a5d-ace0-cbd6ac20d469-kube-api-access-6p4tj\") pod \"maas-api-revoke-sub-2b9214b0-zmjk5\" (UID: \"856f1832-9e23-4a5d-ace0-cbd6ac20d469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b9214b0-zmjk5" Oct 31 05:02:31.403212 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:31.403123 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/856f1832-9e23-4a5d-ace0-cbd6ac20d469-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b9214b0-zmjk5\" (UID: \"856f1832-9e23-4a5d-ace0-cbd6ac20d469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b9214b0-zmjk5" Oct 31 05:02:31.403946 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:31.403897 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/856f1832-9e23-4a5d-ace0-cbd6ac20d469-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b9214b0-zmjk5\" (UID: \"856f1832-9e23-4a5d-ace0-cbd6ac20d469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b9214b0-zmjk5" Oct 31 05:02:31.406509 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:31.406470 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/856f1832-9e23-4a5d-ace0-cbd6ac20d469-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b9214b0-zmjk5\" (UID: \"856f1832-9e23-4a5d-ace0-cbd6ac20d469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b9214b0-zmjk5" Oct 31 05:02:31.429492 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:31.429404 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p4tj\" (UniqueName: \"kubernetes.io/projected/856f1832-9e23-4a5d-ace0-cbd6ac20d469-kube-api-access-6p4tj\") pod \"maas-api-revoke-sub-2b9214b0-zmjk5\" (UID: \"856f1832-9e23-4a5d-ace0-cbd6ac20d469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b9214b0-zmjk5" Oct 31 05:02:31.511612 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:31.511564 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b9214b0-zmjk5" Oct 31 05:02:31.831702 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:02:31.831662 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod856f1832_9e23_4a5d_ace0_cbd6ac20d469.slice/crio-4ef33addd0bc8d51e002159da8ffc3c00195226eea21c2f572bd96140312dad7 WatchSource:0}: Error finding container 4ef33addd0bc8d51e002159da8ffc3c00195226eea21c2f572bd96140312dad7: Status 404 returned error can't find the container with id 4ef33addd0bc8d51e002159da8ffc3c00195226eea21c2f572bd96140312dad7 Oct 31 05:02:31.832854 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:31.832821 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b9214b0-zmjk5"] Oct 31 05:02:32.285776 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:32.285722 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b9214b0-zmjk5" event={"ID":"856f1832-9e23-4a5d-ace0-cbd6ac20d469","Type":"ContainerStarted","Data":"4ef33addd0bc8d51e002159da8ffc3c00195226eea21c2f572bd96140312dad7"} Oct 31 05:02:33.298848 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:33.298800 5019 generic.go:356] "Generic (PLEG): container finished" podID="856f1832-9e23-4a5d-ace0-cbd6ac20d469" containerID="805791bcf69cec7e7df394c2c9b9a6388e6f02aa509257676769e95ff8535d67" exitCode=0 Oct 31 05:02:33.299393 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:33.299049 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b9214b0-zmjk5" event={"ID":"856f1832-9e23-4a5d-ace0-cbd6ac20d469","Type":"ContainerDied","Data":"805791bcf69cec7e7df394c2c9b9a6388e6f02aa509257676769e95ff8535d67"} Oct 31 05:02:34.527519 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:34.527473 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b9214b0-zmjk5" Oct 31 05:02:34.659859 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:34.659757 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/856f1832-9e23-4a5d-ace0-cbd6ac20d469-kube-api-access-6p4tj\" (UniqueName: \"kubernetes.io/projected/856f1832-9e23-4a5d-ace0-cbd6ac20d469-kube-api-access-6p4tj\") pod \"856f1832-9e23-4a5d-ace0-cbd6ac20d469\" (UID: \"856f1832-9e23-4a5d-ace0-cbd6ac20d469\") " Oct 31 05:02:34.660030 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:34.660014 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/856f1832-9e23-4a5d-ace0-cbd6ac20d469-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/856f1832-9e23-4a5d-ace0-cbd6ac20d469-maas-api-openshift-ca\") pod \"856f1832-9e23-4a5d-ace0-cbd6ac20d469\" (UID: \"856f1832-9e23-4a5d-ace0-cbd6ac20d469\") " Oct 31 05:02:34.660280 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:34.660260 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/856f1832-9e23-4a5d-ace0-cbd6ac20d469-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/856f1832-9e23-4a5d-ace0-cbd6ac20d469-maas-api-xks-ca\") pod \"856f1832-9e23-4a5d-ace0-cbd6ac20d469\" (UID: \"856f1832-9e23-4a5d-ace0-cbd6ac20d469\") " Oct 31 05:02:34.660460 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:34.660436 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/856f1832-9e23-4a5d-ace0-cbd6ac20d469-maas-api-openshift-ca" pod "856f1832-9e23-4a5d-ace0-cbd6ac20d469" (UID: "856f1832-9e23-4a5d-ace0-cbd6ac20d469"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:02:34.660936 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:34.660913 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/856f1832-9e23-4a5d-ace0-cbd6ac20d469-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:02:34.662286 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:34.662260 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/856f1832-9e23-4a5d-ace0-cbd6ac20d469-kube-api-access-6p4tj" pod "856f1832-9e23-4a5d-ace0-cbd6ac20d469" (UID: "856f1832-9e23-4a5d-ace0-cbd6ac20d469"). InnerVolumeSpecName "kube-api-access-6p4tj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:02:34.662621 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:34.662601 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/856f1832-9e23-4a5d-ace0-cbd6ac20d469-maas-api-xks-ca" pod "856f1832-9e23-4a5d-ace0-cbd6ac20d469" (UID: "856f1832-9e23-4a5d-ace0-cbd6ac20d469"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:02:34.762446 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:34.762391 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/856f1832-9e23-4a5d-ace0-cbd6ac20d469-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:02:34.762446 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:34.762436 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p4tj\" (UniqueName: \"kubernetes.io/projected/856f1832-9e23-4a5d-ace0-cbd6ac20d469-kube-api-access-6p4tj\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:02:35.319001 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:35.318968 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b9214b0-zmjk5" Oct 31 05:02:35.319249 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:35.318997 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b9214b0-zmjk5" event={"ID":"856f1832-9e23-4a5d-ace0-cbd6ac20d469","Type":"ContainerDied","Data":"4ef33addd0bc8d51e002159da8ffc3c00195226eea21c2f572bd96140312dad7"} Oct 31 05:02:35.319249 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:35.319053 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ef33addd0bc8d51e002159da8ffc3c00195226eea21c2f572bd96140312dad7" Oct 31 05:02:42.055196 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:42.055147 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-670795/test-model-670795-kserve-7c6cfc8755-9gk2c"] Oct 31 05:02:42.604715 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:42.604675 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d55c772-7w6f5"] Oct 31 05:02:42.605636 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:42.605614 5019 cpu_manager.go:414] "containerMap: removing container" podUID="cc1adc72-6548-4daf-9dc1-2b96c853cdbf" containerName="istio-proxy" Oct 31 05:02:42.605636 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:42.605636 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc1adc72-6548-4daf-9dc1-2b96c853cdbf" containerName="istio-proxy" Oct 31 05:02:42.605787 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:42.605660 5019 cpu_manager.go:414] "containerMap: removing container" podUID="856f1832-9e23-4a5d-ace0-cbd6ac20d469" containerName="revoke-keys" Oct 31 05:02:42.605787 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:42.605666 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="856f1832-9e23-4a5d-ace0-cbd6ac20d469" containerName="revoke-keys" Oct 31 05:02:42.605865 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:42.605854 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="856f1832-9e23-4a5d-ace0-cbd6ac20d469" containerName="revoke-keys" Oct 31 05:02:42.610059 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:42.610035 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d55c772-7w6f5" Oct 31 05:02:42.629459 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:42.629419 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d55c772-7w6f5"] Oct 31 05:02:42.634789 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:42.634755 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:02:42.634998 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:42.634809 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:02:42.634998 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:42.634833 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:02:42.634998 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:42.634847 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 05:02:42.746792 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:42.746747 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8p2x\" (UniqueName: \"kubernetes.io/projected/0cf9f031-dd87-4964-8038-a8dc3ecb8917-kube-api-access-m8p2x\") pod \"maas-api-revoke-sub-8d55c772-7w6f5\" (UID: \"0cf9f031-dd87-4964-8038-a8dc3ecb8917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d55c772-7w6f5" Oct 31 05:02:42.747006 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:42.746979 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cf9f031-dd87-4964-8038-a8dc3ecb8917-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d55c772-7w6f5\" (UID: \"0cf9f031-dd87-4964-8038-a8dc3ecb8917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d55c772-7w6f5" Oct 31 05:02:42.747068 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:42.747042 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cf9f031-dd87-4964-8038-a8dc3ecb8917-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d55c772-7w6f5\" (UID: \"0cf9f031-dd87-4964-8038-a8dc3ecb8917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d55c772-7w6f5" Oct 31 05:02:42.848519 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:42.848473 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cf9f031-dd87-4964-8038-a8dc3ecb8917-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d55c772-7w6f5\" (UID: \"0cf9f031-dd87-4964-8038-a8dc3ecb8917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d55c772-7w6f5" Oct 31 05:02:42.848786 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:42.848555 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cf9f031-dd87-4964-8038-a8dc3ecb8917-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d55c772-7w6f5\" (UID: \"0cf9f031-dd87-4964-8038-a8dc3ecb8917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d55c772-7w6f5" Oct 31 05:02:42.848786 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:42.848761 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8p2x\" (UniqueName: \"kubernetes.io/projected/0cf9f031-dd87-4964-8038-a8dc3ecb8917-kube-api-access-m8p2x\") pod \"maas-api-revoke-sub-8d55c772-7w6f5\" (UID: \"0cf9f031-dd87-4964-8038-a8dc3ecb8917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d55c772-7w6f5" Oct 31 05:02:42.849170 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:42.849146 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cf9f031-dd87-4964-8038-a8dc3ecb8917-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d55c772-7w6f5\" (UID: \"0cf9f031-dd87-4964-8038-a8dc3ecb8917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d55c772-7w6f5" Oct 31 05:02:42.851085 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:42.851063 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cf9f031-dd87-4964-8038-a8dc3ecb8917-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d55c772-7w6f5\" (UID: \"0cf9f031-dd87-4964-8038-a8dc3ecb8917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d55c772-7w6f5" Oct 31 05:02:42.872285 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:42.872202 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8p2x\" (UniqueName: \"kubernetes.io/projected/0cf9f031-dd87-4964-8038-a8dc3ecb8917-kube-api-access-m8p2x\") pod \"maas-api-revoke-sub-8d55c772-7w6f5\" (UID: \"0cf9f031-dd87-4964-8038-a8dc3ecb8917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d55c772-7w6f5" Oct 31 05:02:42.928939 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:42.928904 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d55c772-7w6f5" Oct 31 05:02:43.249555 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:43.249519 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d55c772-7w6f5"] Oct 31 05:02:43.250320 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:02:43.250294 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cf9f031_dd87_4964_8038_a8dc3ecb8917.slice/crio-312dbdeba5f86a7e9c88be6bab5a3103bd0863edbd6e85bcf673379b3d4e6f57 WatchSource:0}: Error finding container 312dbdeba5f86a7e9c88be6bab5a3103bd0863edbd6e85bcf673379b3d4e6f57: Status 404 returned error can't find the container with id 312dbdeba5f86a7e9c88be6bab5a3103bd0863edbd6e85bcf673379b3d4e6f57 Oct 31 05:02:43.441417 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:43.441367 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d55c772-7w6f5" event={"ID":"0cf9f031-dd87-4964-8038-a8dc3ecb8917","Type":"ContainerStarted","Data":"312dbdeba5f86a7e9c88be6bab5a3103bd0863edbd6e85bcf673379b3d4e6f57"} Oct 31 05:02:44.452461 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:44.452420 5019 generic.go:356] "Generic (PLEG): container finished" podID="0cf9f031-dd87-4964-8038-a8dc3ecb8917" containerID="7eae9c2f09b905ca5e0d624defeb44e9921e66479cf0eac465807453f60efcf9" exitCode=0 Oct 31 05:02:44.452899 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:44.452507 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d55c772-7w6f5" event={"ID":"0cf9f031-dd87-4964-8038-a8dc3ecb8917","Type":"ContainerDied","Data":"7eae9c2f09b905ca5e0d624defeb44e9921e66479cf0eac465807453f60efcf9"} Oct 31 05:02:45.695364 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:45.695289 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d55c772-7w6f5" Oct 31 05:02:45.823211 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:45.823173 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cf9f031-dd87-4964-8038-a8dc3ecb8917-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cf9f031-dd87-4964-8038-a8dc3ecb8917-maas-api-xks-ca\") pod \"0cf9f031-dd87-4964-8038-a8dc3ecb8917\" (UID: \"0cf9f031-dd87-4964-8038-a8dc3ecb8917\") " Oct 31 05:02:45.823433 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:45.823290 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cf9f031-dd87-4964-8038-a8dc3ecb8917-kube-api-access-m8p2x\" (UniqueName: \"kubernetes.io/projected/0cf9f031-dd87-4964-8038-a8dc3ecb8917-kube-api-access-m8p2x\") pod \"0cf9f031-dd87-4964-8038-a8dc3ecb8917\" (UID: \"0cf9f031-dd87-4964-8038-a8dc3ecb8917\") " Oct 31 05:02:45.823503 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:45.823489 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cf9f031-dd87-4964-8038-a8dc3ecb8917-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cf9f031-dd87-4964-8038-a8dc3ecb8917-maas-api-openshift-ca\") pod \"0cf9f031-dd87-4964-8038-a8dc3ecb8917\" (UID: \"0cf9f031-dd87-4964-8038-a8dc3ecb8917\") " Oct 31 05:02:45.823901 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:45.823870 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cf9f031-dd87-4964-8038-a8dc3ecb8917-maas-api-openshift-ca" pod "0cf9f031-dd87-4964-8038-a8dc3ecb8917" (UID: "0cf9f031-dd87-4964-8038-a8dc3ecb8917"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:02:45.824289 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:45.824268 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cf9f031-dd87-4964-8038-a8dc3ecb8917-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:02:45.825637 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:45.825610 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cf9f031-dd87-4964-8038-a8dc3ecb8917-kube-api-access-m8p2x" pod "0cf9f031-dd87-4964-8038-a8dc3ecb8917" (UID: "0cf9f031-dd87-4964-8038-a8dc3ecb8917"). InnerVolumeSpecName "kube-api-access-m8p2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:02:45.825748 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:45.825613 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cf9f031-dd87-4964-8038-a8dc3ecb8917-maas-api-xks-ca" pod "0cf9f031-dd87-4964-8038-a8dc3ecb8917" (UID: "0cf9f031-dd87-4964-8038-a8dc3ecb8917"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:02:45.926242 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:45.926197 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cf9f031-dd87-4964-8038-a8dc3ecb8917-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:02:45.926242 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:45.926236 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8p2x\" (UniqueName: \"kubernetes.io/projected/0cf9f031-dd87-4964-8038-a8dc3ecb8917-kube-api-access-m8p2x\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:02:46.473289 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:46.473254 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d55c772-7w6f5" Oct 31 05:02:46.473561 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:46.473291 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d55c772-7w6f5" event={"ID":"0cf9f031-dd87-4964-8038-a8dc3ecb8917","Type":"ContainerDied","Data":"312dbdeba5f86a7e9c88be6bab5a3103bd0863edbd6e85bcf673379b3d4e6f57"} Oct 31 05:02:46.473561 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:46.473359 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="312dbdeba5f86a7e9c88be6bab5a3103bd0863edbd6e85bcf673379b3d4e6f57" Oct 31 05:02:47.106894 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:47.106850 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7"] Oct 31 05:02:47.107437 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:47.107110 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7" podUID="1daeedb6-a292-4bad-8d14-22a40dc22100" containerName="istio-proxy" containerID="cri-o://9c1233731ce94edf4f073b3ea698c07838accc458169eab254675391fc1e64a4" gracePeriod=30 Oct 31 05:02:52.407428 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:02:52.407225 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:02:52.436448 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:52.436412 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7" Oct 31 05:02:52.547804 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:52.547762 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1daeedb6-a292-4bad-8d14-22a40dc22100-istio-token\" (UniqueName: \"kubernetes.io/projected/1daeedb6-a292-4bad-8d14-22a40dc22100-istio-token\") pod \"1daeedb6-a292-4bad-8d14-22a40dc22100\" (UID: \"1daeedb6-a292-4bad-8d14-22a40dc22100\") " Oct 31 05:02:52.548011 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:52.547824 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1daeedb6-a292-4bad-8d14-22a40dc22100-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1daeedb6-a292-4bad-8d14-22a40dc22100-istio-envoy\") pod \"1daeedb6-a292-4bad-8d14-22a40dc22100\" (UID: \"1daeedb6-a292-4bad-8d14-22a40dc22100\") " Oct 31 05:02:52.548011 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:52.547862 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1daeedb6-a292-4bad-8d14-22a40dc22100-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1daeedb6-a292-4bad-8d14-22a40dc22100-workload-certs\") pod \"1daeedb6-a292-4bad-8d14-22a40dc22100\" (UID: \"1daeedb6-a292-4bad-8d14-22a40dc22100\") " Oct 31 05:02:52.548011 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:52.547937 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1daeedb6-a292-4bad-8d14-22a40dc22100-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1daeedb6-a292-4bad-8d14-22a40dc22100-istio-podinfo\") pod \"1daeedb6-a292-4bad-8d14-22a40dc22100\" (UID: \"1daeedb6-a292-4bad-8d14-22a40dc22100\") " Oct 31 05:02:52.548194 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:52.548030 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1daeedb6-a292-4bad-8d14-22a40dc22100-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1daeedb6-a292-4bad-8d14-22a40dc22100-istio-data\") pod \"1daeedb6-a292-4bad-8d14-22a40dc22100\" (UID: \"1daeedb6-a292-4bad-8d14-22a40dc22100\") " Oct 31 05:02:52.548194 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:52.548127 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1daeedb6-a292-4bad-8d14-22a40dc22100-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1daeedb6-a292-4bad-8d14-22a40dc22100-credential-socket\") pod \"1daeedb6-a292-4bad-8d14-22a40dc22100\" (UID: \"1daeedb6-a292-4bad-8d14-22a40dc22100\") " Oct 31 05:02:52.548321 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:52.548191 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1daeedb6-a292-4bad-8d14-22a40dc22100-kube-api-access-vgtbq\" (UniqueName: \"kubernetes.io/projected/1daeedb6-a292-4bad-8d14-22a40dc22100-kube-api-access-vgtbq\") pod \"1daeedb6-a292-4bad-8d14-22a40dc22100\" (UID: \"1daeedb6-a292-4bad-8d14-22a40dc22100\") " Oct 31 05:02:52.548321 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:52.548284 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1daeedb6-a292-4bad-8d14-22a40dc22100-workload-certs" pod "1daeedb6-a292-4bad-8d14-22a40dc22100" (UID: "1daeedb6-a292-4bad-8d14-22a40dc22100"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:02:52.548321 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:52.548307 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1daeedb6-a292-4bad-8d14-22a40dc22100-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1daeedb6-a292-4bad-8d14-22a40dc22100-istiod-ca-cert\") pod \"1daeedb6-a292-4bad-8d14-22a40dc22100\" (UID: \"1daeedb6-a292-4bad-8d14-22a40dc22100\") " Oct 31 05:02:52.548541 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:52.548381 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1daeedb6-a292-4bad-8d14-22a40dc22100-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1daeedb6-a292-4bad-8d14-22a40dc22100-workload-socket\") pod \"1daeedb6-a292-4bad-8d14-22a40dc22100\" (UID: \"1daeedb6-a292-4bad-8d14-22a40dc22100\") " Oct 31 05:02:52.548541 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:52.548530 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1daeedb6-a292-4bad-8d14-22a40dc22100-credential-socket" pod "1daeedb6-a292-4bad-8d14-22a40dc22100" (UID: "1daeedb6-a292-4bad-8d14-22a40dc22100"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:02:52.548749 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:52.548724 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1daeedb6-a292-4bad-8d14-22a40dc22100-istio-data" pod "1daeedb6-a292-4bad-8d14-22a40dc22100" (UID: "1daeedb6-a292-4bad-8d14-22a40dc22100"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:02:52.549522 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:52.548962 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1daeedb6-a292-4bad-8d14-22a40dc22100-workload-socket" pod "1daeedb6-a292-4bad-8d14-22a40dc22100" (UID: "1daeedb6-a292-4bad-8d14-22a40dc22100"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:02:52.549522 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:52.549362 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1daeedb6-a292-4bad-8d14-22a40dc22100-istiod-ca-cert" pod "1daeedb6-a292-4bad-8d14-22a40dc22100" (UID: "1daeedb6-a292-4bad-8d14-22a40dc22100"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:02:52.549522 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:52.549371 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1daeedb6-a292-4bad-8d14-22a40dc22100-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:02:52.549522 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:52.549402 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1daeedb6-a292-4bad-8d14-22a40dc22100-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:02:52.549522 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:52.549419 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1daeedb6-a292-4bad-8d14-22a40dc22100-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:02:52.549522 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:52.549433 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1daeedb6-a292-4bad-8d14-22a40dc22100-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:02:52.550928 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:52.550893 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1daeedb6-a292-4bad-8d14-22a40dc22100-istio-podinfo" pod "1daeedb6-a292-4bad-8d14-22a40dc22100" (UID: "1daeedb6-a292-4bad-8d14-22a40dc22100"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:02:52.551379 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:52.551309 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1daeedb6-a292-4bad-8d14-22a40dc22100-istio-token" pod "1daeedb6-a292-4bad-8d14-22a40dc22100" (UID: "1daeedb6-a292-4bad-8d14-22a40dc22100"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:02:52.551827 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:52.551803 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1daeedb6-a292-4bad-8d14-22a40dc22100-istio-envoy" pod "1daeedb6-a292-4bad-8d14-22a40dc22100" (UID: "1daeedb6-a292-4bad-8d14-22a40dc22100"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:02:52.551827 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:52.551811 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1daeedb6-a292-4bad-8d14-22a40dc22100-kube-api-access-vgtbq" pod "1daeedb6-a292-4bad-8d14-22a40dc22100" (UID: "1daeedb6-a292-4bad-8d14-22a40dc22100"). InnerVolumeSpecName "kube-api-access-vgtbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:02:52.556887 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:52.556851 5019 generic.go:356] "Generic (PLEG): container finished" podID="1daeedb6-a292-4bad-8d14-22a40dc22100" containerID="9c1233731ce94edf4f073b3ea698c07838accc458169eab254675391fc1e64a4" exitCode=0 Oct 31 05:02:52.557060 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:52.556908 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7" Oct 31 05:02:52.557060 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:52.556919 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7" event={"ID":"1daeedb6-a292-4bad-8d14-22a40dc22100","Type":"ContainerDied","Data":"9c1233731ce94edf4f073b3ea698c07838accc458169eab254675391fc1e64a4"} Oct 31 05:02:52.557060 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:52.556976 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7" event={"ID":"1daeedb6-a292-4bad-8d14-22a40dc22100","Type":"ContainerDied","Data":"61f8b9b7aedf43c5278f568b6abade6e6a17a4c3c460c0bd764cf73d0b037dee"} Oct 31 05:02:52.557060 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:52.557003 5019 scope.go:122] "RemoveContainer" containerID="9c1233731ce94edf4f073b3ea698c07838accc458169eab254675391fc1e64a4" Oct 31 05:02:52.633144 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:52.633120 5019 scope.go:122] "RemoveContainer" containerID="9c1233731ce94edf4f073b3ea698c07838accc458169eab254675391fc1e64a4" Oct 31 05:02:52.634004 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:02:52.633977 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c1233731ce94edf4f073b3ea698c07838accc458169eab254675391fc1e64a4\": container with ID starting with 9c1233731ce94edf4f073b3ea698c07838accc458169eab254675391fc1e64a4 not found: ID does not exist" containerID="9c1233731ce94edf4f073b3ea698c07838accc458169eab254675391fc1e64a4" Oct 31 05:02:52.634105 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:52.634012 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c1233731ce94edf4f073b3ea698c07838accc458169eab254675391fc1e64a4"} err="failed to get container status \"9c1233731ce94edf4f073b3ea698c07838accc458169eab254675391fc1e64a4\": rpc error: code = NotFound desc = could not find container \"9c1233731ce94edf4f073b3ea698c07838accc458169eab254675391fc1e64a4\": container with ID starting with 9c1233731ce94edf4f073b3ea698c07838accc458169eab254675391fc1e64a4 not found: ID does not exist" Oct 31 05:02:52.651201 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:52.651098 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1daeedb6-a292-4bad-8d14-22a40dc22100-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:02:52.651201 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:52.651144 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1daeedb6-a292-4bad-8d14-22a40dc22100-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:02:52.651201 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:52.651158 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1daeedb6-a292-4bad-8d14-22a40dc22100-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:02:52.651201 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:52.651171 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1daeedb6-a292-4bad-8d14-22a40dc22100-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:02:52.651201 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:52.651185 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgtbq\" (UniqueName: \"kubernetes.io/projected/1daeedb6-a292-4bad-8d14-22a40dc22100-kube-api-access-vgtbq\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:02:52.651573 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:52.651548 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7"] Oct 31 05:02:52.670137 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:52.658300 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rzxt7"] Oct 31 05:02:53.415864 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:53.415810 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1daeedb6-a292-4bad-8d14-22a40dc22100" path="/var/lib/kubelet/pods/1daeedb6-a292-4bad-8d14-22a40dc22100/volumes" Oct 31 05:02:57.091950 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:57.091901 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-670795/test-model-670795-kserve-7c6cfc8755-9gk2c" podUID="f3984da2-0b96-49e7-a555-6bca50f5f45a" containerName="main" containerID="cri-o://8d9ba0a4cddc94c2f3e2452056d6c501cdbfe2da74215fb2c6d668b477d408a1" gracePeriod=45 Oct 31 05:02:57.427967 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:57.427923 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-670795/test-model-670795-kserve-7c6cfc8755-9gk2c" Oct 31 05:02:57.551927 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:57.551879 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3984da2-0b96-49e7-a555-6bca50f5f45a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3984da2-0b96-49e7-a555-6bca50f5f45a-model-cache\") pod \"f3984da2-0b96-49e7-a555-6bca50f5f45a\" (UID: \"f3984da2-0b96-49e7-a555-6bca50f5f45a\") " Oct 31 05:02:57.552148 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:57.551950 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3984da2-0b96-49e7-a555-6bca50f5f45a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3984da2-0b96-49e7-a555-6bca50f5f45a-tmp-dir\") pod \"f3984da2-0b96-49e7-a555-6bca50f5f45a\" (UID: \"f3984da2-0b96-49e7-a555-6bca50f5f45a\") " Oct 31 05:02:57.552148 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:57.552083 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3984da2-0b96-49e7-a555-6bca50f5f45a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3984da2-0b96-49e7-a555-6bca50f5f45a-dshm\") pod \"f3984da2-0b96-49e7-a555-6bca50f5f45a\" (UID: \"f3984da2-0b96-49e7-a555-6bca50f5f45a\") " Oct 31 05:02:57.552279 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:57.552216 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3984da2-0b96-49e7-a555-6bca50f5f45a-tls-certs\" (UniqueName: \"kubernetes.io/secret/f3984da2-0b96-49e7-a555-6bca50f5f45a-tls-certs\") pod \"f3984da2-0b96-49e7-a555-6bca50f5f45a\" (UID: \"f3984da2-0b96-49e7-a555-6bca50f5f45a\") " Oct 31 05:02:57.552279 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:57.552197 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3984da2-0b96-49e7-a555-6bca50f5f45a-model-cache" pod "f3984da2-0b96-49e7-a555-6bca50f5f45a" (UID: "f3984da2-0b96-49e7-a555-6bca50f5f45a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:02:57.552421 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:57.552274 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3984da2-0b96-49e7-a555-6bca50f5f45a-tmp-dir" pod "f3984da2-0b96-49e7-a555-6bca50f5f45a" (UID: "f3984da2-0b96-49e7-a555-6bca50f5f45a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:02:57.552421 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:57.552350 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3984da2-0b96-49e7-a555-6bca50f5f45a-home\" (UniqueName: \"kubernetes.io/empty-dir/f3984da2-0b96-49e7-a555-6bca50f5f45a-home\") pod \"f3984da2-0b96-49e7-a555-6bca50f5f45a\" (UID: \"f3984da2-0b96-49e7-a555-6bca50f5f45a\") " Oct 31 05:02:57.552421 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:57.552402 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3984da2-0b96-49e7-a555-6bca50f5f45a-kube-api-access-rfqvc\" (UniqueName: \"kubernetes.io/projected/f3984da2-0b96-49e7-a555-6bca50f5f45a-kube-api-access-rfqvc\") pod \"f3984da2-0b96-49e7-a555-6bca50f5f45a\" (UID: \"f3984da2-0b96-49e7-a555-6bca50f5f45a\") " Oct 31 05:02:57.552638 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:57.552609 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3984da2-0b96-49e7-a555-6bca50f5f45a-home" pod "f3984da2-0b96-49e7-a555-6bca50f5f45a" (UID: "f3984da2-0b96-49e7-a555-6bca50f5f45a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:02:57.553626 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:57.553596 5019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3984da2-0b96-49e7-a555-6bca50f5f45a-tmp-dir\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:02:57.553626 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:57.553627 5019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3984da2-0b96-49e7-a555-6bca50f5f45a-home\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:02:57.553845 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:57.553641 5019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3984da2-0b96-49e7-a555-6bca50f5f45a-model-cache\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:02:57.554901 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:57.554870 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3984da2-0b96-49e7-a555-6bca50f5f45a-tls-certs" pod "f3984da2-0b96-49e7-a555-6bca50f5f45a" (UID: "f3984da2-0b96-49e7-a555-6bca50f5f45a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:02:57.555056 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:57.555032 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3984da2-0b96-49e7-a555-6bca50f5f45a-dshm" pod "f3984da2-0b96-49e7-a555-6bca50f5f45a" (UID: "f3984da2-0b96-49e7-a555-6bca50f5f45a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:02:57.555389 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:57.555365 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3984da2-0b96-49e7-a555-6bca50f5f45a-kube-api-access-rfqvc" pod "f3984da2-0b96-49e7-a555-6bca50f5f45a" (UID: "f3984da2-0b96-49e7-a555-6bca50f5f45a"). InnerVolumeSpecName "kube-api-access-rfqvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:02:57.615475 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:57.615309 5019 generic.go:356] "Generic (PLEG): container finished" podID="f3984da2-0b96-49e7-a555-6bca50f5f45a" containerID="8d9ba0a4cddc94c2f3e2452056d6c501cdbfe2da74215fb2c6d668b477d408a1" exitCode=0 Oct 31 05:02:57.615475 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:57.615390 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-670795/test-model-670795-kserve-7c6cfc8755-9gk2c" event={"ID":"f3984da2-0b96-49e7-a555-6bca50f5f45a","Type":"ContainerDied","Data":"8d9ba0a4cddc94c2f3e2452056d6c501cdbfe2da74215fb2c6d668b477d408a1"} Oct 31 05:02:57.615475 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:57.615407 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-670795/test-model-670795-kserve-7c6cfc8755-9gk2c" Oct 31 05:02:57.615475 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:57.615440 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-670795/test-model-670795-kserve-7c6cfc8755-9gk2c" event={"ID":"f3984da2-0b96-49e7-a555-6bca50f5f45a","Type":"ContainerDied","Data":"09b6ffbf84d96ae814eee43fa2e4adf89741ff5d996e02e2574948237ae11d34"} Oct 31 05:02:57.615475 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:57.615465 5019 scope.go:122] "RemoveContainer" containerID="8d9ba0a4cddc94c2f3e2452056d6c501cdbfe2da74215fb2c6d668b477d408a1" Oct 31 05:02:57.629440 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:57.629409 5019 scope.go:122] "RemoveContainer" containerID="8d9ba0a4cddc94c2f3e2452056d6c501cdbfe2da74215fb2c6d668b477d408a1" Oct 31 05:02:57.629776 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:02:57.629752 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d9ba0a4cddc94c2f3e2452056d6c501cdbfe2da74215fb2c6d668b477d408a1\": container with ID starting with 8d9ba0a4cddc94c2f3e2452056d6c501cdbfe2da74215fb2c6d668b477d408a1 not found: ID does not exist" containerID="8d9ba0a4cddc94c2f3e2452056d6c501cdbfe2da74215fb2c6d668b477d408a1" Oct 31 05:02:57.629832 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:57.629793 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d9ba0a4cddc94c2f3e2452056d6c501cdbfe2da74215fb2c6d668b477d408a1"} err="failed to get container status \"8d9ba0a4cddc94c2f3e2452056d6c501cdbfe2da74215fb2c6d668b477d408a1\": rpc error: code = NotFound desc = could not find container \"8d9ba0a4cddc94c2f3e2452056d6c501cdbfe2da74215fb2c6d668b477d408a1\": container with ID starting with 8d9ba0a4cddc94c2f3e2452056d6c501cdbfe2da74215fb2c6d668b477d408a1 not found: ID does not exist" Oct 31 05:02:57.655827 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:57.655790 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfqvc\" (UniqueName: \"kubernetes.io/projected/f3984da2-0b96-49e7-a555-6bca50f5f45a-kube-api-access-rfqvc\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:02:57.655827 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:57.655825 5019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3984da2-0b96-49e7-a555-6bca50f5f45a-dshm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:02:57.655827 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:57.655838 5019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3984da2-0b96-49e7-a555-6bca50f5f45a-tls-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:02:57.670889 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:57.670836 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-670795/test-model-670795-kserve-7c6cfc8755-9gk2c"] Oct 31 05:02:57.678859 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:57.678816 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-670795/test-model-670795-kserve-7c6cfc8755-9gk2c"] Oct 31 05:02:58.415469 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:58.415415 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-670795/test-model-670795-kserve-7c6cfc8755-9gk2c" podUID="f3984da2-0b96-49e7-a555-6bca50f5f45a" containerName="main" probeResult="failure" output="Get \"https://10.132.1.129:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 05:02:59.413805 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:02:59.413761 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3984da2-0b96-49e7-a555-6bca50f5f45a" path="/var/lib/kubelet/pods/f3984da2-0b96-49e7-a555-6bca50f5f45a/volumes" Oct 31 05:03:22.395430 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:03:22.395380 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:03:52.397693 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:03:52.397641 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:04:07.781572 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:07.781533 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t"] Oct 31 05:04:08.415499 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:08.415454 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t" podUID="51ef5aae-6bf0-4340-9e57-51b388049f40" containerName="istio-proxy" containerID="cri-o://0892c25bdd29988c6f881b34255470bcff893227b26032754df5a6a24d9f389d" gracePeriod=30 Oct 31 05:04:09.502672 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:09.502638 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s88kp_3945ff65-e877-4372-9334-3968c68b7295/console-operator/2.log" Oct 31 05:04:09.505069 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:09.505043 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s88kp_3945ff65-e877-4372-9334-3968c68b7295/console-operator/2.log" Oct 31 05:04:12.409910 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:12.409862 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed65b197-8fb99"] Oct 31 05:04:12.411191 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:12.411167 5019 cpu_manager.go:414] "containerMap: removing container" podUID="1daeedb6-a292-4bad-8d14-22a40dc22100" containerName="istio-proxy" Oct 31 05:04:12.411251 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:12.411194 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1daeedb6-a292-4bad-8d14-22a40dc22100" containerName="istio-proxy" Oct 31 05:04:12.411251 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:12.411218 5019 cpu_manager.go:414] "containerMap: removing container" podUID="0cf9f031-dd87-4964-8038-a8dc3ecb8917" containerName="revoke-keys" Oct 31 05:04:12.411251 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:12.411226 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cf9f031-dd87-4964-8038-a8dc3ecb8917" containerName="revoke-keys" Oct 31 05:04:12.411408 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:12.411293 5019 cpu_manager.go:414] "containerMap: removing container" podUID="f3984da2-0b96-49e7-a555-6bca50f5f45a" containerName="main" Oct 31 05:04:12.411408 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:12.411301 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3984da2-0b96-49e7-a555-6bca50f5f45a" containerName="main" Oct 31 05:04:12.411538 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:12.411523 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="1daeedb6-a292-4bad-8d14-22a40dc22100" containerName="istio-proxy" Oct 31 05:04:12.411588 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:12.411547 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cf9f031-dd87-4964-8038-a8dc3ecb8917" containerName="revoke-keys" Oct 31 05:04:12.411588 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:12.411571 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3984da2-0b96-49e7-a555-6bca50f5f45a" containerName="main" Oct 31 05:04:12.416456 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:12.416431 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed65b197-8fb99" Oct 31 05:04:12.432895 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:12.432853 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed65b197-8fb99"] Oct 31 05:04:12.437567 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:12.437515 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:04:12.437730 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:12.437571 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 05:04:12.443234 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:12.443202 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:04:12.443431 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:12.443366 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:04:12.544893 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:12.544852 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbee8d43-9d80-4412-99c7-6218bb6d0f5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed65b197-8fb99\" (UID: \"dbee8d43-9d80-4412-99c7-6218bb6d0f5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed65b197-8fb99" Oct 31 05:04:12.545094 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:12.545059 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x59mt\" (UniqueName: \"kubernetes.io/projected/dbee8d43-9d80-4412-99c7-6218bb6d0f5e-kube-api-access-x59mt\") pod \"maas-api-revoke-sub-ed65b197-8fb99\" (UID: \"dbee8d43-9d80-4412-99c7-6218bb6d0f5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed65b197-8fb99" Oct 31 05:04:12.545253 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:12.545234 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbee8d43-9d80-4412-99c7-6218bb6d0f5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed65b197-8fb99\" (UID: \"dbee8d43-9d80-4412-99c7-6218bb6d0f5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed65b197-8fb99" Oct 31 05:04:12.646658 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:12.646620 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbee8d43-9d80-4412-99c7-6218bb6d0f5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed65b197-8fb99\" (UID: \"dbee8d43-9d80-4412-99c7-6218bb6d0f5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed65b197-8fb99" Oct 31 05:04:12.646871 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:12.646779 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbee8d43-9d80-4412-99c7-6218bb6d0f5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed65b197-8fb99\" (UID: \"dbee8d43-9d80-4412-99c7-6218bb6d0f5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed65b197-8fb99" Oct 31 05:04:12.646986 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:12.646967 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x59mt\" (UniqueName: \"kubernetes.io/projected/dbee8d43-9d80-4412-99c7-6218bb6d0f5e-kube-api-access-x59mt\") pod \"maas-api-revoke-sub-ed65b197-8fb99\" (UID: \"dbee8d43-9d80-4412-99c7-6218bb6d0f5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed65b197-8fb99" Oct 31 05:04:12.647502 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:12.647472 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbee8d43-9d80-4412-99c7-6218bb6d0f5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed65b197-8fb99\" (UID: \"dbee8d43-9d80-4412-99c7-6218bb6d0f5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed65b197-8fb99" Oct 31 05:04:12.649531 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:12.649503 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbee8d43-9d80-4412-99c7-6218bb6d0f5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed65b197-8fb99\" (UID: \"dbee8d43-9d80-4412-99c7-6218bb6d0f5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed65b197-8fb99" Oct 31 05:04:12.670838 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:12.670759 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x59mt\" (UniqueName: \"kubernetes.io/projected/dbee8d43-9d80-4412-99c7-6218bb6d0f5e-kube-api-access-x59mt\") pod \"maas-api-revoke-sub-ed65b197-8fb99\" (UID: \"dbee8d43-9d80-4412-99c7-6218bb6d0f5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed65b197-8fb99" Oct 31 05:04:12.734280 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:12.734243 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed65b197-8fb99" Oct 31 05:04:13.043063 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:04:13.043030 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbee8d43_9d80_4412_99c7_6218bb6d0f5e.slice/crio-e889c7b93a58a1216a569d98f982e1a3174c2e64e8567262a12e913749b104b7 WatchSource:0}: Error finding container e889c7b93a58a1216a569d98f982e1a3174c2e64e8567262a12e913749b104b7: Status 404 returned error can't find the container with id e889c7b93a58a1216a569d98f982e1a3174c2e64e8567262a12e913749b104b7 Oct 31 05:04:13.043894 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:13.043865 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed65b197-8fb99"] Oct 31 05:04:13.474614 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:13.474575 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed65b197-8fb99" event={"ID":"dbee8d43-9d80-4412-99c7-6218bb6d0f5e","Type":"ContainerStarted","Data":"e889c7b93a58a1216a569d98f982e1a3174c2e64e8567262a12e913749b104b7"} Oct 31 05:04:13.735169 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:13.735134 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t" Oct 31 05:04:13.766798 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:13.766759 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51ef5aae-6bf0-4340-9e57-51b388049f40-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51ef5aae-6bf0-4340-9e57-51b388049f40-istio-envoy\") pod \"51ef5aae-6bf0-4340-9e57-51b388049f40\" (UID: \"51ef5aae-6bf0-4340-9e57-51b388049f40\") " Oct 31 05:04:13.767021 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:13.766820 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51ef5aae-6bf0-4340-9e57-51b388049f40-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51ef5aae-6bf0-4340-9e57-51b388049f40-istio-data\") pod \"51ef5aae-6bf0-4340-9e57-51b388049f40\" (UID: \"51ef5aae-6bf0-4340-9e57-51b388049f40\") " Oct 31 05:04:13.767021 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:13.766860 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51ef5aae-6bf0-4340-9e57-51b388049f40-kube-api-access-p5qbx\" (UniqueName: \"kubernetes.io/projected/51ef5aae-6bf0-4340-9e57-51b388049f40-kube-api-access-p5qbx\") pod \"51ef5aae-6bf0-4340-9e57-51b388049f40\" (UID: \"51ef5aae-6bf0-4340-9e57-51b388049f40\") " Oct 31 05:04:13.767021 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:13.766932 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51ef5aae-6bf0-4340-9e57-51b388049f40-istio-token\" (UniqueName: \"kubernetes.io/projected/51ef5aae-6bf0-4340-9e57-51b388049f40-istio-token\") pod \"51ef5aae-6bf0-4340-9e57-51b388049f40\" (UID: \"51ef5aae-6bf0-4340-9e57-51b388049f40\") " Oct 31 05:04:13.767021 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:13.766990 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51ef5aae-6bf0-4340-9e57-51b388049f40-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51ef5aae-6bf0-4340-9e57-51b388049f40-workload-socket\") pod \"51ef5aae-6bf0-4340-9e57-51b388049f40\" (UID: \"51ef5aae-6bf0-4340-9e57-51b388049f40\") " Oct 31 05:04:13.767285 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:13.767037 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51ef5aae-6bf0-4340-9e57-51b388049f40-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51ef5aae-6bf0-4340-9e57-51b388049f40-workload-certs\") pod \"51ef5aae-6bf0-4340-9e57-51b388049f40\" (UID: \"51ef5aae-6bf0-4340-9e57-51b388049f40\") " Oct 31 05:04:13.767285 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:13.767067 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51ef5aae-6bf0-4340-9e57-51b388049f40-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51ef5aae-6bf0-4340-9e57-51b388049f40-istio-podinfo\") pod \"51ef5aae-6bf0-4340-9e57-51b388049f40\" (UID: \"51ef5aae-6bf0-4340-9e57-51b388049f40\") " Oct 31 05:04:13.767285 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:13.767150 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51ef5aae-6bf0-4340-9e57-51b388049f40-istio-data" pod "51ef5aae-6bf0-4340-9e57-51b388049f40" (UID: "51ef5aae-6bf0-4340-9e57-51b388049f40"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:04:13.767285 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:13.767173 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51ef5aae-6bf0-4340-9e57-51b388049f40-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51ef5aae-6bf0-4340-9e57-51b388049f40-istiod-ca-cert\") pod \"51ef5aae-6bf0-4340-9e57-51b388049f40\" (UID: \"51ef5aae-6bf0-4340-9e57-51b388049f40\") " Oct 31 05:04:13.767587 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:13.767289 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51ef5aae-6bf0-4340-9e57-51b388049f40-workload-socket" pod "51ef5aae-6bf0-4340-9e57-51b388049f40" (UID: "51ef5aae-6bf0-4340-9e57-51b388049f40"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:04:13.767587 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:13.767417 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51ef5aae-6bf0-4340-9e57-51b388049f40-workload-certs" pod "51ef5aae-6bf0-4340-9e57-51b388049f40" (UID: "51ef5aae-6bf0-4340-9e57-51b388049f40"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:04:13.767817 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:13.767787 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51ef5aae-6bf0-4340-9e57-51b388049f40-istiod-ca-cert" pod "51ef5aae-6bf0-4340-9e57-51b388049f40" (UID: "51ef5aae-6bf0-4340-9e57-51b388049f40"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:04:13.768149 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:13.768122 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51ef5aae-6bf0-4340-9e57-51b388049f40-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:04:13.768253 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:13.768152 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51ef5aae-6bf0-4340-9e57-51b388049f40-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:04:13.768253 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:13.768161 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51ef5aae-6bf0-4340-9e57-51b388049f40-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:04:13.768253 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:13.768169 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51ef5aae-6bf0-4340-9e57-51b388049f40-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:04:13.769407 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:13.769375 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51ef5aae-6bf0-4340-9e57-51b388049f40-istio-envoy" pod "51ef5aae-6bf0-4340-9e57-51b388049f40" (UID: "51ef5aae-6bf0-4340-9e57-51b388049f40"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:04:13.769667 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:13.769637 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51ef5aae-6bf0-4340-9e57-51b388049f40-kube-api-access-p5qbx" pod "51ef5aae-6bf0-4340-9e57-51b388049f40" (UID: "51ef5aae-6bf0-4340-9e57-51b388049f40"). InnerVolumeSpecName "kube-api-access-p5qbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:04:13.769855 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:13.769826 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51ef5aae-6bf0-4340-9e57-51b388049f40-istio-podinfo" pod "51ef5aae-6bf0-4340-9e57-51b388049f40" (UID: "51ef5aae-6bf0-4340-9e57-51b388049f40"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:04:13.769935 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:13.769856 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51ef5aae-6bf0-4340-9e57-51b388049f40-istio-token" pod "51ef5aae-6bf0-4340-9e57-51b388049f40" (UID: "51ef5aae-6bf0-4340-9e57-51b388049f40"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:04:13.869038 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:13.869001 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51ef5aae-6bf0-4340-9e57-51b388049f40-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51ef5aae-6bf0-4340-9e57-51b388049f40-credential-socket\") pod \"51ef5aae-6bf0-4340-9e57-51b388049f40\" (UID: \"51ef5aae-6bf0-4340-9e57-51b388049f40\") " Oct 31 05:04:13.869384 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:13.869325 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51ef5aae-6bf0-4340-9e57-51b388049f40-credential-socket" pod "51ef5aae-6bf0-4340-9e57-51b388049f40" (UID: "51ef5aae-6bf0-4340-9e57-51b388049f40"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:04:13.869766 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:13.869748 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51ef5aae-6bf0-4340-9e57-51b388049f40-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:04:13.869766 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:13.869767 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51ef5aae-6bf0-4340-9e57-51b388049f40-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:04:13.869929 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:13.869777 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51ef5aae-6bf0-4340-9e57-51b388049f40-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:04:13.869929 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:13.869787 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5qbx\" (UniqueName: \"kubernetes.io/projected/51ef5aae-6bf0-4340-9e57-51b388049f40-kube-api-access-p5qbx\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:04:13.869929 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:13.869795 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51ef5aae-6bf0-4340-9e57-51b388049f40-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:04:14.485195 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:14.485156 5019 generic.go:356] "Generic (PLEG): container finished" podID="51ef5aae-6bf0-4340-9e57-51b388049f40" containerID="0892c25bdd29988c6f881b34255470bcff893227b26032754df5a6a24d9f389d" exitCode=0 Oct 31 05:04:14.485760 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:14.485221 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t" Oct 31 05:04:14.485760 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:14.485239 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t" event={"ID":"51ef5aae-6bf0-4340-9e57-51b388049f40","Type":"ContainerDied","Data":"0892c25bdd29988c6f881b34255470bcff893227b26032754df5a6a24d9f389d"} Oct 31 05:04:14.485760 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:14.485300 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t" event={"ID":"51ef5aae-6bf0-4340-9e57-51b388049f40","Type":"ContainerDied","Data":"b367595ed17f8672dc48ec3501380c7e6e9a612c6f2030b6d0a2dbc8209a40d7"} Oct 31 05:04:14.485760 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:14.485323 5019 scope.go:122] "RemoveContainer" containerID="0892c25bdd29988c6f881b34255470bcff893227b26032754df5a6a24d9f389d" Oct 31 05:04:14.487495 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:14.487471 5019 generic.go:356] "Generic (PLEG): container finished" podID="dbee8d43-9d80-4412-99c7-6218bb6d0f5e" containerID="931d8c1d41faf08939d890398ba63872d983cd52002b866ec24852ecc10e4c72" exitCode=0 Oct 31 05:04:14.487644 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:14.487563 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed65b197-8fb99" event={"ID":"dbee8d43-9d80-4412-99c7-6218bb6d0f5e","Type":"ContainerDied","Data":"931d8c1d41faf08939d890398ba63872d983cd52002b866ec24852ecc10e4c72"} Oct 31 05:04:14.499188 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:14.499156 5019 scope.go:122] "RemoveContainer" containerID="9e0f5aa541516eb722af32733f2ec9f5f8818409039487c81c42b5a180b7f265" Oct 31 05:04:14.518498 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:14.518471 5019 scope.go:122] "RemoveContainer" containerID="0892c25bdd29988c6f881b34255470bcff893227b26032754df5a6a24d9f389d" Oct 31 05:04:14.518904 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:04:14.518870 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0892c25bdd29988c6f881b34255470bcff893227b26032754df5a6a24d9f389d\": container with ID starting with 0892c25bdd29988c6f881b34255470bcff893227b26032754df5a6a24d9f389d not found: ID does not exist" containerID="0892c25bdd29988c6f881b34255470bcff893227b26032754df5a6a24d9f389d" Oct 31 05:04:14.518977 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:14.518918 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0892c25bdd29988c6f881b34255470bcff893227b26032754df5a6a24d9f389d"} err="failed to get container status \"0892c25bdd29988c6f881b34255470bcff893227b26032754df5a6a24d9f389d\": rpc error: code = NotFound desc = could not find container \"0892c25bdd29988c6f881b34255470bcff893227b26032754df5a6a24d9f389d\": container with ID starting with 0892c25bdd29988c6f881b34255470bcff893227b26032754df5a6a24d9f389d not found: ID does not exist" Oct 31 05:04:14.518977 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:14.518943 5019 scope.go:122] "RemoveContainer" containerID="9e0f5aa541516eb722af32733f2ec9f5f8818409039487c81c42b5a180b7f265" Oct 31 05:04:14.519298 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:04:14.519272 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e0f5aa541516eb722af32733f2ec9f5f8818409039487c81c42b5a180b7f265\": container with ID starting with 9e0f5aa541516eb722af32733f2ec9f5f8818409039487c81c42b5a180b7f265 not found: ID does not exist" containerID="9e0f5aa541516eb722af32733f2ec9f5f8818409039487c81c42b5a180b7f265" Oct 31 05:04:14.519422 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:14.519310 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e0f5aa541516eb722af32733f2ec9f5f8818409039487c81c42b5a180b7f265"} err="failed to get container status \"9e0f5aa541516eb722af32733f2ec9f5f8818409039487c81c42b5a180b7f265\": rpc error: code = NotFound desc = could not find container \"9e0f5aa541516eb722af32733f2ec9f5f8818409039487c81c42b5a180b7f265\": container with ID starting with 9e0f5aa541516eb722af32733f2ec9f5f8818409039487c81c42b5a180b7f265 not found: ID does not exist" Oct 31 05:04:14.593106 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:14.593009 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t"] Oct 31 05:04:14.598125 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:14.598089 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-670795-openshift-default-764f54c759-gtm6t"] Oct 31 05:04:14.638210 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:14.638164 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-829ac7-77fc54d55f-4npw2"] Oct 31 05:04:15.438262 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:15.428134 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51ef5aae-6bf0-4340-9e57-51b388049f40" path="/var/lib/kubelet/pods/51ef5aae-6bf0-4340-9e57-51b388049f40/volumes" Oct 31 05:04:15.698647 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:15.698581 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed65b197-8fb99" Oct 31 05:04:15.752842 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:15.752801 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbee8d43-9d80-4412-99c7-6218bb6d0f5e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbee8d43-9d80-4412-99c7-6218bb6d0f5e-maas-api-openshift-ca\") pod \"dbee8d43-9d80-4412-99c7-6218bb6d0f5e\" (UID: \"dbee8d43-9d80-4412-99c7-6218bb6d0f5e\") " Oct 31 05:04:15.753217 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:15.753194 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbee8d43-9d80-4412-99c7-6218bb6d0f5e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbee8d43-9d80-4412-99c7-6218bb6d0f5e-maas-api-xks-ca\") pod \"dbee8d43-9d80-4412-99c7-6218bb6d0f5e\" (UID: \"dbee8d43-9d80-4412-99c7-6218bb6d0f5e\") " Oct 31 05:04:15.753271 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:15.753204 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbee8d43-9d80-4412-99c7-6218bb6d0f5e-maas-api-openshift-ca" pod "dbee8d43-9d80-4412-99c7-6218bb6d0f5e" (UID: "dbee8d43-9d80-4412-99c7-6218bb6d0f5e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:04:15.753323 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:15.753286 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbee8d43-9d80-4412-99c7-6218bb6d0f5e-kube-api-access-x59mt\" (UniqueName: \"kubernetes.io/projected/dbee8d43-9d80-4412-99c7-6218bb6d0f5e-kube-api-access-x59mt\") pod \"dbee8d43-9d80-4412-99c7-6218bb6d0f5e\" (UID: \"dbee8d43-9d80-4412-99c7-6218bb6d0f5e\") " Oct 31 05:04:15.753812 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:15.753786 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbee8d43-9d80-4412-99c7-6218bb6d0f5e-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:04:15.755510 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:15.755476 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbee8d43-9d80-4412-99c7-6218bb6d0f5e-kube-api-access-x59mt" pod "dbee8d43-9d80-4412-99c7-6218bb6d0f5e" (UID: "dbee8d43-9d80-4412-99c7-6218bb6d0f5e"). InnerVolumeSpecName "kube-api-access-x59mt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:04:15.755618 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:15.755538 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbee8d43-9d80-4412-99c7-6218bb6d0f5e-maas-api-xks-ca" pod "dbee8d43-9d80-4412-99c7-6218bb6d0f5e" (UID: "dbee8d43-9d80-4412-99c7-6218bb6d0f5e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:04:15.857615 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:15.855147 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbee8d43-9d80-4412-99c7-6218bb6d0f5e-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:04:15.857615 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:15.855186 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x59mt\" (UniqueName: \"kubernetes.io/projected/dbee8d43-9d80-4412-99c7-6218bb6d0f5e-kube-api-access-x59mt\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:04:16.512085 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:16.512052 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed65b197-8fb99" Oct 31 05:04:16.512307 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:16.512110 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed65b197-8fb99" event={"ID":"dbee8d43-9d80-4412-99c7-6218bb6d0f5e","Type":"ContainerDied","Data":"e889c7b93a58a1216a569d98f982e1a3174c2e64e8567262a12e913749b104b7"} Oct 31 05:04:16.512307 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:16.512180 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e889c7b93a58a1216a569d98f982e1a3174c2e64e8567262a12e913749b104b7" Oct 31 05:04:19.639424 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:19.639360 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-829ac7-77fc54d55f-4npw2" podUID="225a0379-7184-4a57-97d8-c9eb65397e87" containerName="payload-pre-processing" containerID="cri-o://a2181a0820acd7ad303a05fb1399da297951e9bc970f9f7119da89cb24584329" gracePeriod=25 Oct 31 05:04:22.393691 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:04:22.393649 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:04:23.279069 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:23.279021 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-171edd/test-model-171edd-kserve-7b677876c5-v5zw4"] Oct 31 05:04:23.821316 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:23.821277 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40ffde1b-t6xrq"] Oct 31 05:04:23.822376 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:23.822345 5019 cpu_manager.go:414] "containerMap: removing container" podUID="dbee8d43-9d80-4412-99c7-6218bb6d0f5e" containerName="revoke-keys" Oct 31 05:04:23.822376 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:23.822373 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbee8d43-9d80-4412-99c7-6218bb6d0f5e" containerName="revoke-keys" Oct 31 05:04:23.822580 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:23.822420 5019 cpu_manager.go:414] "containerMap: removing container" podUID="51ef5aae-6bf0-4340-9e57-51b388049f40" containerName="istio-proxy" Oct 31 05:04:23.822580 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:23.822428 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51ef5aae-6bf0-4340-9e57-51b388049f40" containerName="istio-proxy" Oct 31 05:04:23.822580 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:23.822454 5019 cpu_manager.go:414] "containerMap: removing container" podUID="51ef5aae-6bf0-4340-9e57-51b388049f40" containerName="istio-proxy" Oct 31 05:04:23.822580 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:23.822463 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51ef5aae-6bf0-4340-9e57-51b388049f40" containerName="istio-proxy" Oct 31 05:04:23.822803 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:23.822713 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="51ef5aae-6bf0-4340-9e57-51b388049f40" containerName="istio-proxy" Oct 31 05:04:23.822803 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:23.822736 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="51ef5aae-6bf0-4340-9e57-51b388049f40" containerName="istio-proxy" Oct 31 05:04:23.822803 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:23.822760 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbee8d43-9d80-4412-99c7-6218bb6d0f5e" containerName="revoke-keys" Oct 31 05:04:23.826541 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:23.826521 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ffde1b-t6xrq" Oct 31 05:04:23.846849 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:23.846820 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40ffde1b-t6xrq"] Oct 31 05:04:23.851798 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:23.851770 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:04:23.851928 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:23.851777 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:04:23.851928 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:23.851845 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:04:23.853138 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:23.853118 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 05:04:24.011256 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:24.011222 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd920eaf-9ce8-4c6b-aa4f-c3ce59ef13c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40ffde1b-t6xrq\" (UID: \"dd920eaf-9ce8-4c6b-aa4f-c3ce59ef13c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ffde1b-t6xrq" Oct 31 05:04:24.011509 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:24.011483 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdzct\" (UniqueName: \"kubernetes.io/projected/dd920eaf-9ce8-4c6b-aa4f-c3ce59ef13c0-kube-api-access-pdzct\") pod \"maas-api-revoke-sub-40ffde1b-t6xrq\" (UID: \"dd920eaf-9ce8-4c6b-aa4f-c3ce59ef13c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ffde1b-t6xrq" Oct 31 05:04:24.011649 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:24.011632 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd920eaf-9ce8-4c6b-aa4f-c3ce59ef13c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40ffde1b-t6xrq\" (UID: \"dd920eaf-9ce8-4c6b-aa4f-c3ce59ef13c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ffde1b-t6xrq" Oct 31 05:04:24.112987 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:24.112888 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd920eaf-9ce8-4c6b-aa4f-c3ce59ef13c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40ffde1b-t6xrq\" (UID: \"dd920eaf-9ce8-4c6b-aa4f-c3ce59ef13c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ffde1b-t6xrq" Oct 31 05:04:24.113178 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:24.113081 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdzct\" (UniqueName: \"kubernetes.io/projected/dd920eaf-9ce8-4c6b-aa4f-c3ce59ef13c0-kube-api-access-pdzct\") pod \"maas-api-revoke-sub-40ffde1b-t6xrq\" (UID: \"dd920eaf-9ce8-4c6b-aa4f-c3ce59ef13c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ffde1b-t6xrq" Oct 31 05:04:24.113258 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:24.113201 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd920eaf-9ce8-4c6b-aa4f-c3ce59ef13c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40ffde1b-t6xrq\" (UID: \"dd920eaf-9ce8-4c6b-aa4f-c3ce59ef13c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ffde1b-t6xrq" Oct 31 05:04:24.113868 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:24.113847 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd920eaf-9ce8-4c6b-aa4f-c3ce59ef13c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40ffde1b-t6xrq\" (UID: \"dd920eaf-9ce8-4c6b-aa4f-c3ce59ef13c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ffde1b-t6xrq" Oct 31 05:04:24.115328 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:24.115297 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd920eaf-9ce8-4c6b-aa4f-c3ce59ef13c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40ffde1b-t6xrq\" (UID: \"dd920eaf-9ce8-4c6b-aa4f-c3ce59ef13c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ffde1b-t6xrq" Oct 31 05:04:24.136000 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:24.135969 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdzct\" (UniqueName: \"kubernetes.io/projected/dd920eaf-9ce8-4c6b-aa4f-c3ce59ef13c0-kube-api-access-pdzct\") pod \"maas-api-revoke-sub-40ffde1b-t6xrq\" (UID: \"dd920eaf-9ce8-4c6b-aa4f-c3ce59ef13c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ffde1b-t6xrq" Oct 31 05:04:24.145890 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:24.145863 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ffde1b-t6xrq" Oct 31 05:04:24.455351 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:04:24.455303 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd920eaf_9ce8_4c6b_aa4f_c3ce59ef13c0.slice/crio-a9be42b07ea54a9ebb3dafcaacb2318abd65b55ab6b11044b5ba0af70714aaf5 WatchSource:0}: Error finding container a9be42b07ea54a9ebb3dafcaacb2318abd65b55ab6b11044b5ba0af70714aaf5: Status 404 returned error can't find the container with id a9be42b07ea54a9ebb3dafcaacb2318abd65b55ab6b11044b5ba0af70714aaf5 Oct 31 05:04:24.456452 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:24.456423 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40ffde1b-t6xrq"] Oct 31 05:04:24.587343 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:24.587302 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ffde1b-t6xrq" event={"ID":"dd920eaf-9ce8-4c6b-aa4f-c3ce59ef13c0","Type":"ContainerStarted","Data":"a9be42b07ea54a9ebb3dafcaacb2318abd65b55ab6b11044b5ba0af70714aaf5"} Oct 31 05:04:25.599054 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:25.599011 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ffde1b-t6xrq" event={"ID":"dd920eaf-9ce8-4c6b-aa4f-c3ce59ef13c0","Type":"ContainerStarted","Data":"bcc550dbdf369fa5e598cd2b29c211a17047798755e30910abb2a1ab6b20fd35"} Oct 31 05:04:25.639573 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:25.639511 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ffde1b-t6xrq" podStartSLOduration=2.639494242 podStartE2EDuration="2.639494242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:04:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:04:25.638728097 +0000 UTC m=+3620.395066137" watchObservedRunningTime="2026-10-31 05:04:25.639494242 +0000 UTC m=+3620.395832282" Oct 31 05:04:26.611413 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:26.611367 5019 generic.go:356] "Generic (PLEG): container finished" podID="dd920eaf-9ce8-4c6b-aa4f-c3ce59ef13c0" containerID="bcc550dbdf369fa5e598cd2b29c211a17047798755e30910abb2a1ab6b20fd35" exitCode=0 Oct 31 05:04:26.611949 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:26.611432 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ffde1b-t6xrq" event={"ID":"dd920eaf-9ce8-4c6b-aa4f-c3ce59ef13c0","Type":"ContainerDied","Data":"bcc550dbdf369fa5e598cd2b29c211a17047798755e30910abb2a1ab6b20fd35"} Oct 31 05:04:27.502416 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:27.502377 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-829ac7-77fc54d55f-4npw2" podUID="225a0379-7184-4a57-97d8-c9eb65397e87" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:04:27.800481 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:27.800444 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ffde1b-t6xrq" Oct 31 05:04:27.976762 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:27.976720 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd920eaf-9ce8-4c6b-aa4f-c3ce59ef13c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd920eaf-9ce8-4c6b-aa4f-c3ce59ef13c0-maas-api-openshift-ca\") pod \"dd920eaf-9ce8-4c6b-aa4f-c3ce59ef13c0\" (UID: \"dd920eaf-9ce8-4c6b-aa4f-c3ce59ef13c0\") " Oct 31 05:04:27.976971 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:27.976840 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd920eaf-9ce8-4c6b-aa4f-c3ce59ef13c0-kube-api-access-pdzct\" (UniqueName: \"kubernetes.io/projected/dd920eaf-9ce8-4c6b-aa4f-c3ce59ef13c0-kube-api-access-pdzct\") pod \"dd920eaf-9ce8-4c6b-aa4f-c3ce59ef13c0\" (UID: \"dd920eaf-9ce8-4c6b-aa4f-c3ce59ef13c0\") " Oct 31 05:04:27.976971 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:27.976964 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd920eaf-9ce8-4c6b-aa4f-c3ce59ef13c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd920eaf-9ce8-4c6b-aa4f-c3ce59ef13c0-maas-api-xks-ca\") pod \"dd920eaf-9ce8-4c6b-aa4f-c3ce59ef13c0\" (UID: \"dd920eaf-9ce8-4c6b-aa4f-c3ce59ef13c0\") " Oct 31 05:04:27.977170 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:27.977141 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd920eaf-9ce8-4c6b-aa4f-c3ce59ef13c0-maas-api-openshift-ca" pod "dd920eaf-9ce8-4c6b-aa4f-c3ce59ef13c0" (UID: "dd920eaf-9ce8-4c6b-aa4f-c3ce59ef13c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:04:27.977740 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:27.977711 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd920eaf-9ce8-4c6b-aa4f-c3ce59ef13c0-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:04:27.979101 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:27.979073 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd920eaf-9ce8-4c6b-aa4f-c3ce59ef13c0-maas-api-xks-ca" pod "dd920eaf-9ce8-4c6b-aa4f-c3ce59ef13c0" (UID: "dd920eaf-9ce8-4c6b-aa4f-c3ce59ef13c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:04:27.979198 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:27.979145 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd920eaf-9ce8-4c6b-aa4f-c3ce59ef13c0-kube-api-access-pdzct" pod "dd920eaf-9ce8-4c6b-aa4f-c3ce59ef13c0" (UID: "dd920eaf-9ce8-4c6b-aa4f-c3ce59ef13c0"). InnerVolumeSpecName "kube-api-access-pdzct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:04:28.078942 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:28.078858 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdzct\" (UniqueName: \"kubernetes.io/projected/dd920eaf-9ce8-4c6b-aa4f-c3ce59ef13c0-kube-api-access-pdzct\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:04:28.078942 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:28.078892 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd920eaf-9ce8-4c6b-aa4f-c3ce59ef13c0-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:04:28.630261 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:28.630233 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ffde1b-t6xrq" Oct 31 05:04:28.630437 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:28.630258 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ffde1b-t6xrq" event={"ID":"dd920eaf-9ce8-4c6b-aa4f-c3ce59ef13c0","Type":"ContainerDied","Data":"a9be42b07ea54a9ebb3dafcaacb2318abd65b55ab6b11044b5ba0af70714aaf5"} Oct 31 05:04:28.630437 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:28.630303 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9be42b07ea54a9ebb3dafcaacb2318abd65b55ab6b11044b5ba0af70714aaf5" Oct 31 05:04:37.500867 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:37.500823 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-829ac7-77fc54d55f-4npw2" podUID="225a0379-7184-4a57-97d8-c9eb65397e87" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:04:38.307698 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:38.307637 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-171edd/test-model-171edd-kserve-7b677876c5-v5zw4" podUID="96c515d1-a206-4fcd-a43e-dcc811e966bb" containerName="main" containerID="cri-o://5273a094861a304eae2ae699ae9094185bc4cde7afbb020684485d163a638cad" gracePeriod=45 Oct 31 05:04:38.632756 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:38.632722 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-171edd/test-model-171edd-kserve-7b677876c5-v5zw4" Oct 31 05:04:38.745532 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:38.745495 5019 generic.go:356] "Generic (PLEG): container finished" podID="96c515d1-a206-4fcd-a43e-dcc811e966bb" containerID="5273a094861a304eae2ae699ae9094185bc4cde7afbb020684485d163a638cad" exitCode=0 Oct 31 05:04:38.745727 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:38.745543 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-171edd/test-model-171edd-kserve-7b677876c5-v5zw4" Oct 31 05:04:38.745727 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:38.745576 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-171edd/test-model-171edd-kserve-7b677876c5-v5zw4" event={"ID":"96c515d1-a206-4fcd-a43e-dcc811e966bb","Type":"ContainerDied","Data":"5273a094861a304eae2ae699ae9094185bc4cde7afbb020684485d163a638cad"} Oct 31 05:04:38.745727 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:38.745625 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-171edd/test-model-171edd-kserve-7b677876c5-v5zw4" event={"ID":"96c515d1-a206-4fcd-a43e-dcc811e966bb","Type":"ContainerDied","Data":"566c5f4cde3ac29e77889f8aee72c0948bab999850bccaf7818079e4393e4953"} Oct 31 05:04:38.745727 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:38.745643 5019 scope.go:122] "RemoveContainer" containerID="5273a094861a304eae2ae699ae9094185bc4cde7afbb020684485d163a638cad" Oct 31 05:04:38.754185 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:38.754156 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96c515d1-a206-4fcd-a43e-dcc811e966bb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96c515d1-a206-4fcd-a43e-dcc811e966bb-model-cache\") pod \"96c515d1-a206-4fcd-a43e-dcc811e966bb\" (UID: \"96c515d1-a206-4fcd-a43e-dcc811e966bb\") " Oct 31 05:04:38.754397 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:38.754376 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96c515d1-a206-4fcd-a43e-dcc811e966bb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96c515d1-a206-4fcd-a43e-dcc811e966bb-tmp-dir\") pod \"96c515d1-a206-4fcd-a43e-dcc811e966bb\" (UID: \"96c515d1-a206-4fcd-a43e-dcc811e966bb\") " Oct 31 05:04:38.754497 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:38.754473 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96c515d1-a206-4fcd-a43e-dcc811e966bb-tls-certs\" (UniqueName: \"kubernetes.io/secret/96c515d1-a206-4fcd-a43e-dcc811e966bb-tls-certs\") pod \"96c515d1-a206-4fcd-a43e-dcc811e966bb\" (UID: \"96c515d1-a206-4fcd-a43e-dcc811e966bb\") " Oct 31 05:04:38.754497 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:38.754473 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96c515d1-a206-4fcd-a43e-dcc811e966bb-model-cache" pod "96c515d1-a206-4fcd-a43e-dcc811e966bb" (UID: "96c515d1-a206-4fcd-a43e-dcc811e966bb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:04:38.754618 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:38.754565 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96c515d1-a206-4fcd-a43e-dcc811e966bb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/96c515d1-a206-4fcd-a43e-dcc811e966bb-dshm\") pod \"96c515d1-a206-4fcd-a43e-dcc811e966bb\" (UID: \"96c515d1-a206-4fcd-a43e-dcc811e966bb\") " Oct 31 05:04:38.754675 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:38.754650 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96c515d1-a206-4fcd-a43e-dcc811e966bb-tmp-dir" pod "96c515d1-a206-4fcd-a43e-dcc811e966bb" (UID: "96c515d1-a206-4fcd-a43e-dcc811e966bb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:04:38.754675 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:38.754660 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96c515d1-a206-4fcd-a43e-dcc811e966bb-kube-api-access-jkjfw\" (UniqueName: \"kubernetes.io/projected/96c515d1-a206-4fcd-a43e-dcc811e966bb-kube-api-access-jkjfw\") pod \"96c515d1-a206-4fcd-a43e-dcc811e966bb\" (UID: \"96c515d1-a206-4fcd-a43e-dcc811e966bb\") " Oct 31 05:04:38.754794 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:38.754778 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96c515d1-a206-4fcd-a43e-dcc811e966bb-home\" (UniqueName: \"kubernetes.io/empty-dir/96c515d1-a206-4fcd-a43e-dcc811e966bb-home\") pod \"96c515d1-a206-4fcd-a43e-dcc811e966bb\" (UID: \"96c515d1-a206-4fcd-a43e-dcc811e966bb\") " Oct 31 05:04:38.755084 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:38.755045 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96c515d1-a206-4fcd-a43e-dcc811e966bb-home" pod "96c515d1-a206-4fcd-a43e-dcc811e966bb" (UID: "96c515d1-a206-4fcd-a43e-dcc811e966bb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:04:38.755709 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:38.755684 5019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96c515d1-a206-4fcd-a43e-dcc811e966bb-home\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:04:38.755832 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:38.755715 5019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96c515d1-a206-4fcd-a43e-dcc811e966bb-model-cache\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:04:38.755832 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:38.755734 5019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96c515d1-a206-4fcd-a43e-dcc811e966bb-tmp-dir\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:04:38.756841 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:38.756810 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96c515d1-a206-4fcd-a43e-dcc811e966bb-dshm" pod "96c515d1-a206-4fcd-a43e-dcc811e966bb" (UID: "96c515d1-a206-4fcd-a43e-dcc811e966bb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:04:38.756961 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:38.756939 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96c515d1-a206-4fcd-a43e-dcc811e966bb-tls-certs" pod "96c515d1-a206-4fcd-a43e-dcc811e966bb" (UID: "96c515d1-a206-4fcd-a43e-dcc811e966bb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:04:38.757020 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:38.756954 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96c515d1-a206-4fcd-a43e-dcc811e966bb-kube-api-access-jkjfw" pod "96c515d1-a206-4fcd-a43e-dcc811e966bb" (UID: "96c515d1-a206-4fcd-a43e-dcc811e966bb"). InnerVolumeSpecName "kube-api-access-jkjfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:04:38.757444 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:38.757427 5019 scope.go:122] "RemoveContainer" containerID="5273a094861a304eae2ae699ae9094185bc4cde7afbb020684485d163a638cad" Oct 31 05:04:38.757697 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:04:38.757676 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5273a094861a304eae2ae699ae9094185bc4cde7afbb020684485d163a638cad\": container with ID starting with 5273a094861a304eae2ae699ae9094185bc4cde7afbb020684485d163a638cad not found: ID does not exist" containerID="5273a094861a304eae2ae699ae9094185bc4cde7afbb020684485d163a638cad" Oct 31 05:04:38.757760 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:38.757707 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5273a094861a304eae2ae699ae9094185bc4cde7afbb020684485d163a638cad"} err="failed to get container status \"5273a094861a304eae2ae699ae9094185bc4cde7afbb020684485d163a638cad\": rpc error: code = NotFound desc = could not find container \"5273a094861a304eae2ae699ae9094185bc4cde7afbb020684485d163a638cad\": container with ID starting with 5273a094861a304eae2ae699ae9094185bc4cde7afbb020684485d163a638cad not found: ID does not exist" Oct 31 05:04:38.857122 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:38.857032 5019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96c515d1-a206-4fcd-a43e-dcc811e966bb-tls-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:04:38.857122 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:38.857068 5019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96c515d1-a206-4fcd-a43e-dcc811e966bb-dshm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:04:38.857122 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:38.857083 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkjfw\" (UniqueName: \"kubernetes.io/projected/96c515d1-a206-4fcd-a43e-dcc811e966bb-kube-api-access-jkjfw\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:04:39.095834 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:39.095796 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-171edd/test-model-171edd-kserve-7b677876c5-v5zw4"] Oct 31 05:04:39.100424 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:39.100396 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-171edd/test-model-171edd-kserve-7b677876c5-v5zw4"] Oct 31 05:04:39.413879 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:39.413845 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96c515d1-a206-4fcd-a43e-dcc811e966bb" path="/var/lib/kubelet/pods/96c515d1-a206-4fcd-a43e-dcc811e966bb/volumes" Oct 31 05:04:39.756035 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:39.756009 5019 generic.go:356] "Generic (PLEG): container finished" podID="225a0379-7184-4a57-97d8-c9eb65397e87" containerID="a2181a0820acd7ad303a05fb1399da297951e9bc970f9f7119da89cb24584329" exitCode=0 Oct 31 05:04:39.756468 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:39.756076 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-829ac7-77fc54d55f-4npw2" event={"ID":"225a0379-7184-4a57-97d8-c9eb65397e87","Type":"ContainerDied","Data":"a2181a0820acd7ad303a05fb1399da297951e9bc970f9f7119da89cb24584329"} Oct 31 05:04:40.010535 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:40.010509 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-829ac7-77fc54d55f-4npw2" Oct 31 05:04:40.089819 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:40.089785 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/225a0379-7184-4a57-97d8-c9eb65397e87-kube-api-access-szrch\" (UniqueName: \"kubernetes.io/projected/225a0379-7184-4a57-97d8-c9eb65397e87-kube-api-access-szrch\") pod \"225a0379-7184-4a57-97d8-c9eb65397e87\" (UID: \"225a0379-7184-4a57-97d8-c9eb65397e87\") " Oct 31 05:04:40.090001 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:40.089879 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/225a0379-7184-4a57-97d8-c9eb65397e87-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/225a0379-7184-4a57-97d8-c9eb65397e87-plugins-config-volume\") pod \"225a0379-7184-4a57-97d8-c9eb65397e87\" (UID: \"225a0379-7184-4a57-97d8-c9eb65397e87\") " Oct 31 05:04:40.090389 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:40.090317 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/225a0379-7184-4a57-97d8-c9eb65397e87-plugins-config-volume" pod "225a0379-7184-4a57-97d8-c9eb65397e87" (UID: "225a0379-7184-4a57-97d8-c9eb65397e87"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:04:40.090760 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:40.090734 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/225a0379-7184-4a57-97d8-c9eb65397e87-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:04:40.092211 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:40.092183 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/225a0379-7184-4a57-97d8-c9eb65397e87-kube-api-access-szrch" pod "225a0379-7184-4a57-97d8-c9eb65397e87" (UID: "225a0379-7184-4a57-97d8-c9eb65397e87"). InnerVolumeSpecName "kube-api-access-szrch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:04:40.192615 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:40.192563 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szrch\" (UniqueName: \"kubernetes.io/projected/225a0379-7184-4a57-97d8-c9eb65397e87-kube-api-access-szrch\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:04:40.767572 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:40.767537 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-829ac7-77fc54d55f-4npw2" Oct 31 05:04:40.768061 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:40.767580 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-829ac7-77fc54d55f-4npw2" event={"ID":"225a0379-7184-4a57-97d8-c9eb65397e87","Type":"ContainerDied","Data":"739b596df7d7081975a43de57f6fcfb6a9ebe02f04b4aa1d778fc4bf9a5b4e54"} Oct 31 05:04:40.768061 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:40.767653 5019 scope.go:122] "RemoveContainer" containerID="a2181a0820acd7ad303a05fb1399da297951e9bc970f9f7119da89cb24584329" Oct 31 05:04:40.827892 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:40.827856 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-829ac7-77fc54d55f-4npw2"] Oct 31 05:04:40.839941 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:40.839905 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-829ac7-77fc54d55f-4npw2"] Oct 31 05:04:41.412690 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:41.412654 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="225a0379-7184-4a57-97d8-c9eb65397e87" path="/var/lib/kubelet/pods/225a0379-7184-4a57-97d8-c9eb65397e87/volumes" Oct 31 05:04:45.150224 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:04:45.150183 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-fb346b/test-model-fb346b-kserve-6d4b9b47b5-f7mnx"] Oct 31 05:04:52.399180 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:04:52.399135 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:05:00.177370 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:00.177274 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-fb346b/test-model-fb346b-kserve-6d4b9b47b5-f7mnx" podUID="60b4b3fb-a758-45b8-933f-b62536b3661e" containerName="main" containerID="cri-o://7f0bd5805834837c1605d6f20d1a5585dc99f0d232d0a8da0b671ace0998d661" gracePeriod=45 Oct 31 05:05:00.238241 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:00.238199 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-fb346b/test-model-fb346b-kserve-6d4b9b47b5-f7mnx" podUID="60b4b3fb-a758-45b8-933f-b62536b3661e" containerName="main" probeResult="failure" output="Get \"https://10.132.1.132:8000/ready\": dial tcp 10.132.1.132:8000: connect: connection refused" Oct 31 05:05:00.527720 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:00.527693 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fb346b/test-model-fb346b-kserve-6d4b9b47b5-f7mnx" Oct 31 05:05:00.657851 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:00.657816 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60b4b3fb-a758-45b8-933f-b62536b3661e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/60b4b3fb-a758-45b8-933f-b62536b3661e-dshm\") pod \"60b4b3fb-a758-45b8-933f-b62536b3661e\" (UID: \"60b4b3fb-a758-45b8-933f-b62536b3661e\") " Oct 31 05:05:00.658073 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:00.658008 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60b4b3fb-a758-45b8-933f-b62536b3661e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60b4b3fb-a758-45b8-933f-b62536b3661e-model-cache\") pod \"60b4b3fb-a758-45b8-933f-b62536b3661e\" (UID: \"60b4b3fb-a758-45b8-933f-b62536b3661e\") " Oct 31 05:05:00.658181 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:00.658162 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60b4b3fb-a758-45b8-933f-b62536b3661e-kube-api-access-g82bq\" (UniqueName: \"kubernetes.io/projected/60b4b3fb-a758-45b8-933f-b62536b3661e-kube-api-access-g82bq\") pod \"60b4b3fb-a758-45b8-933f-b62536b3661e\" (UID: \"60b4b3fb-a758-45b8-933f-b62536b3661e\") " Oct 31 05:05:00.658246 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:00.658207 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60b4b3fb-a758-45b8-933f-b62536b3661e-tls-certs\" (UniqueName: \"kubernetes.io/secret/60b4b3fb-a758-45b8-933f-b62536b3661e-tls-certs\") pod \"60b4b3fb-a758-45b8-933f-b62536b3661e\" (UID: \"60b4b3fb-a758-45b8-933f-b62536b3661e\") " Oct 31 05:05:00.658316 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:00.658243 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60b4b3fb-a758-45b8-933f-b62536b3661e-home\" (UniqueName: \"kubernetes.io/empty-dir/60b4b3fb-a758-45b8-933f-b62536b3661e-home\") pod \"60b4b3fb-a758-45b8-933f-b62536b3661e\" (UID: \"60b4b3fb-a758-45b8-933f-b62536b3661e\") " Oct 31 05:05:00.658316 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:00.658281 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60b4b3fb-a758-45b8-933f-b62536b3661e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60b4b3fb-a758-45b8-933f-b62536b3661e-tmp-dir\") pod \"60b4b3fb-a758-45b8-933f-b62536b3661e\" (UID: \"60b4b3fb-a758-45b8-933f-b62536b3661e\") " Oct 31 05:05:00.658316 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:00.658293 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60b4b3fb-a758-45b8-933f-b62536b3661e-model-cache" pod "60b4b3fb-a758-45b8-933f-b62536b3661e" (UID: "60b4b3fb-a758-45b8-933f-b62536b3661e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:05:00.658661 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:00.658588 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60b4b3fb-a758-45b8-933f-b62536b3661e-home" pod "60b4b3fb-a758-45b8-933f-b62536b3661e" (UID: "60b4b3fb-a758-45b8-933f-b62536b3661e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:05:00.658745 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:00.658682 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60b4b3fb-a758-45b8-933f-b62536b3661e-tmp-dir" pod "60b4b3fb-a758-45b8-933f-b62536b3661e" (UID: "60b4b3fb-a758-45b8-933f-b62536b3661e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:05:00.659155 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:00.659135 5019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60b4b3fb-a758-45b8-933f-b62536b3661e-model-cache\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:00.659211 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:00.659157 5019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60b4b3fb-a758-45b8-933f-b62536b3661e-home\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:00.659211 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:00.659165 5019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60b4b3fb-a758-45b8-933f-b62536b3661e-tmp-dir\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:00.664900 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:00.664869 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60b4b3fb-a758-45b8-933f-b62536b3661e-dshm" pod "60b4b3fb-a758-45b8-933f-b62536b3661e" (UID: "60b4b3fb-a758-45b8-933f-b62536b3661e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:05:00.665044 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:00.664927 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60b4b3fb-a758-45b8-933f-b62536b3661e-tls-certs" pod "60b4b3fb-a758-45b8-933f-b62536b3661e" (UID: "60b4b3fb-a758-45b8-933f-b62536b3661e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:05:00.666647 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:00.666602 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60b4b3fb-a758-45b8-933f-b62536b3661e-kube-api-access-g82bq" pod "60b4b3fb-a758-45b8-933f-b62536b3661e" (UID: "60b4b3fb-a758-45b8-933f-b62536b3661e"). InnerVolumeSpecName "kube-api-access-g82bq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:05:00.760764 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:00.760671 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g82bq\" (UniqueName: \"kubernetes.io/projected/60b4b3fb-a758-45b8-933f-b62536b3661e-kube-api-access-g82bq\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:00.760764 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:00.760706 5019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60b4b3fb-a758-45b8-933f-b62536b3661e-tls-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:00.760764 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:00.760719 5019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60b4b3fb-a758-45b8-933f-b62536b3661e-dshm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:01.082288 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:01.082187 5019 generic.go:356] "Generic (PLEG): container finished" podID="60b4b3fb-a758-45b8-933f-b62536b3661e" containerID="7f0bd5805834837c1605d6f20d1a5585dc99f0d232d0a8da0b671ace0998d661" exitCode=0 Oct 31 05:05:01.082288 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:01.082238 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fb346b/test-model-fb346b-kserve-6d4b9b47b5-f7mnx" Oct 31 05:05:01.082288 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:01.082261 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fb346b/test-model-fb346b-kserve-6d4b9b47b5-f7mnx" event={"ID":"60b4b3fb-a758-45b8-933f-b62536b3661e","Type":"ContainerDied","Data":"7f0bd5805834837c1605d6f20d1a5585dc99f0d232d0a8da0b671ace0998d661"} Oct 31 05:05:01.082633 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:01.082314 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fb346b/test-model-fb346b-kserve-6d4b9b47b5-f7mnx" event={"ID":"60b4b3fb-a758-45b8-933f-b62536b3661e","Type":"ContainerDied","Data":"363c049e744dc8d3e47e613de4cf63670ed44df4c3d3d4d4fda38ddf6f4a30eb"} Oct 31 05:05:01.082633 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:01.082346 5019 scope.go:122] "RemoveContainer" containerID="7f0bd5805834837c1605d6f20d1a5585dc99f0d232d0a8da0b671ace0998d661" Oct 31 05:05:01.094886 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:01.094865 5019 scope.go:122] "RemoveContainer" containerID="7f0bd5805834837c1605d6f20d1a5585dc99f0d232d0a8da0b671ace0998d661" Oct 31 05:05:01.095193 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:05:01.095164 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f0bd5805834837c1605d6f20d1a5585dc99f0d232d0a8da0b671ace0998d661\": container with ID starting with 7f0bd5805834837c1605d6f20d1a5585dc99f0d232d0a8da0b671ace0998d661 not found: ID does not exist" containerID="7f0bd5805834837c1605d6f20d1a5585dc99f0d232d0a8da0b671ace0998d661" Oct 31 05:05:01.095299 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:01.095202 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f0bd5805834837c1605d6f20d1a5585dc99f0d232d0a8da0b671ace0998d661"} err="failed to get container status \"7f0bd5805834837c1605d6f20d1a5585dc99f0d232d0a8da0b671ace0998d661\": rpc error: code = NotFound desc = could not find container \"7f0bd5805834837c1605d6f20d1a5585dc99f0d232d0a8da0b671ace0998d661\": container with ID starting with 7f0bd5805834837c1605d6f20d1a5585dc99f0d232d0a8da0b671ace0998d661 not found: ID does not exist" Oct 31 05:05:01.136710 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:01.136678 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-fb346b/test-model-fb346b-kserve-6d4b9b47b5-f7mnx"] Oct 31 05:05:01.148370 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:01.148315 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-fb346b/test-model-fb346b-kserve-6d4b9b47b5-f7mnx"] Oct 31 05:05:01.412673 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:01.412592 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60b4b3fb-a758-45b8-933f-b62536b3661e" path="/var/lib/kubelet/pods/60b4b3fb-a758-45b8-933f-b62536b3661e/volumes" Oct 31 05:05:11.424740 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:11.424704 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d52321c9-mc599"] Oct 31 05:05:11.426013 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:11.425990 5019 cpu_manager.go:414] "containerMap: removing container" podUID="96c515d1-a206-4fcd-a43e-dcc811e966bb" containerName="main" Oct 31 05:05:11.426198 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:11.426175 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96c515d1-a206-4fcd-a43e-dcc811e966bb" containerName="main" Oct 31 05:05:11.426384 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:11.426217 5019 cpu_manager.go:414] "containerMap: removing container" podUID="225a0379-7184-4a57-97d8-c9eb65397e87" containerName="payload-pre-processing" Oct 31 05:05:11.426384 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:11.426226 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="225a0379-7184-4a57-97d8-c9eb65397e87" containerName="payload-pre-processing" Oct 31 05:05:11.426384 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:11.426246 5019 cpu_manager.go:414] "containerMap: removing container" podUID="60b4b3fb-a758-45b8-933f-b62536b3661e" containerName="main" Oct 31 05:05:11.426384 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:11.426254 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60b4b3fb-a758-45b8-933f-b62536b3661e" containerName="main" Oct 31 05:05:11.426384 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:11.426266 5019 cpu_manager.go:414] "containerMap: removing container" podUID="dd920eaf-9ce8-4c6b-aa4f-c3ce59ef13c0" containerName="revoke-keys" Oct 31 05:05:11.426384 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:11.426273 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd920eaf-9ce8-4c6b-aa4f-c3ce59ef13c0" containerName="revoke-keys" Oct 31 05:05:11.426725 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:11.426480 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="96c515d1-a206-4fcd-a43e-dcc811e966bb" containerName="main" Oct 31 05:05:11.426725 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:11.426501 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="225a0379-7184-4a57-97d8-c9eb65397e87" containerName="payload-pre-processing" Oct 31 05:05:11.426725 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:11.426513 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="60b4b3fb-a758-45b8-933f-b62536b3661e" containerName="main" Oct 31 05:05:11.426725 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:11.426524 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd920eaf-9ce8-4c6b-aa4f-c3ce59ef13c0" containerName="revoke-keys" Oct 31 05:05:11.431143 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:11.431117 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52321c9-mc599" Oct 31 05:05:11.434149 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:11.434123 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d52321c9-mc599"] Oct 31 05:05:11.450184 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:11.450152 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:05:11.461839 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:11.461796 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:05:11.461839 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:11.461814 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:05:11.461839 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:11.461814 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 05:05:11.603318 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:11.603278 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7cgb\" (UniqueName: \"kubernetes.io/projected/d4900084-1bee-46c6-8d1d-f6504e65a828-kube-api-access-h7cgb\") pod \"maas-api-revoke-sub-d52321c9-mc599\" (UID: \"d4900084-1bee-46c6-8d1d-f6504e65a828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52321c9-mc599" Oct 31 05:05:11.603544 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:11.603364 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4900084-1bee-46c6-8d1d-f6504e65a828-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d52321c9-mc599\" (UID: \"d4900084-1bee-46c6-8d1d-f6504e65a828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52321c9-mc599" Oct 31 05:05:11.603544 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:11.603464 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4900084-1bee-46c6-8d1d-f6504e65a828-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d52321c9-mc599\" (UID: \"d4900084-1bee-46c6-8d1d-f6504e65a828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52321c9-mc599" Oct 31 05:05:11.705075 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:11.704979 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7cgb\" (UniqueName: \"kubernetes.io/projected/d4900084-1bee-46c6-8d1d-f6504e65a828-kube-api-access-h7cgb\") pod \"maas-api-revoke-sub-d52321c9-mc599\" (UID: \"d4900084-1bee-46c6-8d1d-f6504e65a828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52321c9-mc599" Oct 31 05:05:11.705075 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:11.705044 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4900084-1bee-46c6-8d1d-f6504e65a828-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d52321c9-mc599\" (UID: \"d4900084-1bee-46c6-8d1d-f6504e65a828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52321c9-mc599" Oct 31 05:05:11.705311 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:11.705085 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4900084-1bee-46c6-8d1d-f6504e65a828-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d52321c9-mc599\" (UID: \"d4900084-1bee-46c6-8d1d-f6504e65a828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52321c9-mc599" Oct 31 05:05:11.705890 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:11.705863 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4900084-1bee-46c6-8d1d-f6504e65a828-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d52321c9-mc599\" (UID: \"d4900084-1bee-46c6-8d1d-f6504e65a828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52321c9-mc599" Oct 31 05:05:11.707894 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:11.707866 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4900084-1bee-46c6-8d1d-f6504e65a828-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d52321c9-mc599\" (UID: \"d4900084-1bee-46c6-8d1d-f6504e65a828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52321c9-mc599" Oct 31 05:05:11.728518 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:11.728489 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7cgb\" (UniqueName: \"kubernetes.io/projected/d4900084-1bee-46c6-8d1d-f6504e65a828-kube-api-access-h7cgb\") pod \"maas-api-revoke-sub-d52321c9-mc599\" (UID: \"d4900084-1bee-46c6-8d1d-f6504e65a828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52321c9-mc599" Oct 31 05:05:11.755004 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:11.754979 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52321c9-mc599" Oct 31 05:05:11.900893 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:11.900858 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0614feb6-glvs5"] Oct 31 05:05:11.906461 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:11.906435 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0614feb6-glvs5" Oct 31 05:05:11.930305 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:11.930278 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0614feb6-glvs5"] Oct 31 05:05:12.009932 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:12.009896 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chh4r\" (UniqueName: \"kubernetes.io/projected/e9e616de-209d-46e9-b3b6-def8d0d0d844-kube-api-access-chh4r\") pod \"maas-api-revoke-sub-0614feb6-glvs5\" (UID: \"e9e616de-209d-46e9-b3b6-def8d0d0d844\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0614feb6-glvs5" Oct 31 05:05:12.010111 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:12.009947 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9e616de-209d-46e9-b3b6-def8d0d0d844-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0614feb6-glvs5\" (UID: \"e9e616de-209d-46e9-b3b6-def8d0d0d844\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0614feb6-glvs5" Oct 31 05:05:12.010166 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:12.010100 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9e616de-209d-46e9-b3b6-def8d0d0d844-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0614feb6-glvs5\" (UID: \"e9e616de-209d-46e9-b3b6-def8d0d0d844\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0614feb6-glvs5" Oct 31 05:05:12.065316 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:12.065273 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d52321c9-mc599"] Oct 31 05:05:12.065800 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:05:12.065777 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4900084_1bee_46c6_8d1d_f6504e65a828.slice/crio-35386a331b08f49c4d925fe9fe1229088b9b4e99b6bdc0c8aa972d5e394b6e90 WatchSource:0}: Error finding container 35386a331b08f49c4d925fe9fe1229088b9b4e99b6bdc0c8aa972d5e394b6e90: Status 404 returned error can't find the container with id 35386a331b08f49c4d925fe9fe1229088b9b4e99b6bdc0c8aa972d5e394b6e90 Oct 31 05:05:12.111718 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:12.111694 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9e616de-209d-46e9-b3b6-def8d0d0d844-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0614feb6-glvs5\" (UID: \"e9e616de-209d-46e9-b3b6-def8d0d0d844\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0614feb6-glvs5" Oct 31 05:05:12.112024 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:12.112001 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chh4r\" (UniqueName: \"kubernetes.io/projected/e9e616de-209d-46e9-b3b6-def8d0d0d844-kube-api-access-chh4r\") pod \"maas-api-revoke-sub-0614feb6-glvs5\" (UID: \"e9e616de-209d-46e9-b3b6-def8d0d0d844\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0614feb6-glvs5" Oct 31 05:05:12.112081 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:12.112044 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9e616de-209d-46e9-b3b6-def8d0d0d844-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0614feb6-glvs5\" (UID: \"e9e616de-209d-46e9-b3b6-def8d0d0d844\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0614feb6-glvs5" Oct 31 05:05:12.112748 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:12.112721 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9e616de-209d-46e9-b3b6-def8d0d0d844-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0614feb6-glvs5\" (UID: \"e9e616de-209d-46e9-b3b6-def8d0d0d844\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0614feb6-glvs5" Oct 31 05:05:12.114142 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:12.114121 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9e616de-209d-46e9-b3b6-def8d0d0d844-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0614feb6-glvs5\" (UID: \"e9e616de-209d-46e9-b3b6-def8d0d0d844\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0614feb6-glvs5" Oct 31 05:05:12.136308 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:12.136282 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chh4r\" (UniqueName: \"kubernetes.io/projected/e9e616de-209d-46e9-b3b6-def8d0d0d844-kube-api-access-chh4r\") pod \"maas-api-revoke-sub-0614feb6-glvs5\" (UID: \"e9e616de-209d-46e9-b3b6-def8d0d0d844\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0614feb6-glvs5" Oct 31 05:05:12.228077 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:12.228034 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52321c9-mc599" event={"ID":"d4900084-1bee-46c6-8d1d-f6504e65a828","Type":"ContainerStarted","Data":"35386a331b08f49c4d925fe9fe1229088b9b4e99b6bdc0c8aa972d5e394b6e90"} Oct 31 05:05:12.232488 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:12.232462 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0614feb6-glvs5" Oct 31 05:05:12.285132 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:12.285059 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b22a1dd5-7g82g"] Oct 31 05:05:12.291599 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:12.291571 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b22a1dd5-7g82g" Oct 31 05:05:12.307047 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:12.307018 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b22a1dd5-7g82g"] Oct 31 05:05:12.417256 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:12.417222 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rrkx\" (UniqueName: \"kubernetes.io/projected/fb6bda0f-355e-413f-b122-4d52c1b1fae0-kube-api-access-5rrkx\") pod \"maas-api-revoke-sub-b22a1dd5-7g82g\" (UID: \"fb6bda0f-355e-413f-b122-4d52c1b1fae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b22a1dd5-7g82g" Oct 31 05:05:12.417489 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:12.417462 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb6bda0f-355e-413f-b122-4d52c1b1fae0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b22a1dd5-7g82g\" (UID: \"fb6bda0f-355e-413f-b122-4d52c1b1fae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b22a1dd5-7g82g" Oct 31 05:05:12.417679 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:12.417654 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb6bda0f-355e-413f-b122-4d52c1b1fae0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b22a1dd5-7g82g\" (UID: \"fb6bda0f-355e-413f-b122-4d52c1b1fae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b22a1dd5-7g82g" Oct 31 05:05:12.518874 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:12.518840 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rrkx\" (UniqueName: \"kubernetes.io/projected/fb6bda0f-355e-413f-b122-4d52c1b1fae0-kube-api-access-5rrkx\") pod \"maas-api-revoke-sub-b22a1dd5-7g82g\" (UID: \"fb6bda0f-355e-413f-b122-4d52c1b1fae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b22a1dd5-7g82g" Oct 31 05:05:12.519367 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:12.518958 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb6bda0f-355e-413f-b122-4d52c1b1fae0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b22a1dd5-7g82g\" (UID: \"fb6bda0f-355e-413f-b122-4d52c1b1fae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b22a1dd5-7g82g" Oct 31 05:05:12.519367 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:12.519088 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb6bda0f-355e-413f-b122-4d52c1b1fae0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b22a1dd5-7g82g\" (UID: \"fb6bda0f-355e-413f-b122-4d52c1b1fae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b22a1dd5-7g82g" Oct 31 05:05:12.519943 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:12.519912 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb6bda0f-355e-413f-b122-4d52c1b1fae0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b22a1dd5-7g82g\" (UID: \"fb6bda0f-355e-413f-b122-4d52c1b1fae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b22a1dd5-7g82g" Oct 31 05:05:12.522018 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:12.521995 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb6bda0f-355e-413f-b122-4d52c1b1fae0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b22a1dd5-7g82g\" (UID: \"fb6bda0f-355e-413f-b122-4d52c1b1fae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b22a1dd5-7g82g" Oct 31 05:05:12.542540 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:12.542516 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rrkx\" (UniqueName: \"kubernetes.io/projected/fb6bda0f-355e-413f-b122-4d52c1b1fae0-kube-api-access-5rrkx\") pod \"maas-api-revoke-sub-b22a1dd5-7g82g\" (UID: \"fb6bda0f-355e-413f-b122-4d52c1b1fae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b22a1dd5-7g82g" Oct 31 05:05:12.545576 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:12.545548 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0614feb6-glvs5"] Oct 31 05:05:12.546694 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:05:12.546671 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9e616de_209d_46e9_b3b6_def8d0d0d844.slice/crio-d0df7f8a4bf74753072b426e5989582fec1b77794c7d1d5bf90cfea1f4c4db76 WatchSource:0}: Error finding container d0df7f8a4bf74753072b426e5989582fec1b77794c7d1d5bf90cfea1f4c4db76: Status 404 returned error can't find the container with id d0df7f8a4bf74753072b426e5989582fec1b77794c7d1d5bf90cfea1f4c4db76 Oct 31 05:05:12.622464 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:12.622435 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b22a1dd5-7g82g" Oct 31 05:05:12.927724 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:05:12.927670 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb6bda0f_355e_413f_b122_4d52c1b1fae0.slice/crio-7fa28e32a4e53ade69eef2a520f407b4fc63470bb8d41d400034f2c7bd49cb7f WatchSource:0}: Error finding container 7fa28e32a4e53ade69eef2a520f407b4fc63470bb8d41d400034f2c7bd49cb7f: Status 404 returned error can't find the container with id 7fa28e32a4e53ade69eef2a520f407b4fc63470bb8d41d400034f2c7bd49cb7f Oct 31 05:05:12.929081 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:12.929052 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b22a1dd5-7g82g"] Oct 31 05:05:13.238397 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:13.238360 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b22a1dd5-7g82g" event={"ID":"fb6bda0f-355e-413f-b122-4d52c1b1fae0","Type":"ContainerStarted","Data":"7fa28e32a4e53ade69eef2a520f407b4fc63470bb8d41d400034f2c7bd49cb7f"} Oct 31 05:05:13.240221 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:13.240194 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0614feb6-glvs5" event={"ID":"e9e616de-209d-46e9-b3b6-def8d0d0d844","Type":"ContainerStarted","Data":"e2b7a5f02242acc6c687cdd3d06c3b10e4574b7ce20a4dd0c532fd3a57e031f5"} Oct 31 05:05:13.240381 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:13.240242 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0614feb6-glvs5" event={"ID":"e9e616de-209d-46e9-b3b6-def8d0d0d844","Type":"ContainerStarted","Data":"d0df7f8a4bf74753072b426e5989582fec1b77794c7d1d5bf90cfea1f4c4db76"} Oct 31 05:05:13.241907 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:13.241883 5019 generic.go:356] "Generic (PLEG): container finished" podID="d4900084-1bee-46c6-8d1d-f6504e65a828" containerID="a71c894bd09a4bc8579bca97228cd0f4b7c3b46b41403131077ad8ccc6fde0e2" exitCode=0 Oct 31 05:05:13.241996 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:13.241931 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52321c9-mc599" event={"ID":"d4900084-1bee-46c6-8d1d-f6504e65a828","Type":"ContainerDied","Data":"a71c894bd09a4bc8579bca97228cd0f4b7c3b46b41403131077ad8ccc6fde0e2"} Oct 31 05:05:13.282721 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:13.282674 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0614feb6-glvs5" podStartSLOduration=2.282661747 podStartE2EDuration="2.282661747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:05:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:05:13.279164896 +0000 UTC m=+3668.035502946" watchObservedRunningTime="2026-10-31 05:05:13.282661747 +0000 UTC m=+3668.038999787" Oct 31 05:05:14.253658 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:14.253620 5019 generic.go:356] "Generic (PLEG): container finished" podID="fb6bda0f-355e-413f-b122-4d52c1b1fae0" containerID="8232a8be47521aabe89348e85ccddc0bd2665b3d5b7137b1dc66bc490b26c520" exitCode=0 Oct 31 05:05:14.254158 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:14.253704 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b22a1dd5-7g82g" event={"ID":"fb6bda0f-355e-413f-b122-4d52c1b1fae0","Type":"ContainerDied","Data":"8232a8be47521aabe89348e85ccddc0bd2665b3d5b7137b1dc66bc490b26c520"} Oct 31 05:05:14.255729 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:14.255703 5019 generic.go:356] "Generic (PLEG): container finished" podID="e9e616de-209d-46e9-b3b6-def8d0d0d844" containerID="e2b7a5f02242acc6c687cdd3d06c3b10e4574b7ce20a4dd0c532fd3a57e031f5" exitCode=0 Oct 31 05:05:14.255894 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:14.255768 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0614feb6-glvs5" event={"ID":"e9e616de-209d-46e9-b3b6-def8d0d0d844","Type":"ContainerDied","Data":"e2b7a5f02242acc6c687cdd3d06c3b10e4574b7ce20a4dd0c532fd3a57e031f5"} Oct 31 05:05:14.464196 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:14.464163 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52321c9-mc599" Oct 31 05:05:14.651934 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:14.651894 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4900084-1bee-46c6-8d1d-f6504e65a828-kube-api-access-h7cgb\" (UniqueName: \"kubernetes.io/projected/d4900084-1bee-46c6-8d1d-f6504e65a828-kube-api-access-h7cgb\") pod \"d4900084-1bee-46c6-8d1d-f6504e65a828\" (UID: \"d4900084-1bee-46c6-8d1d-f6504e65a828\") " Oct 31 05:05:14.652124 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:14.652039 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4900084-1bee-46c6-8d1d-f6504e65a828-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4900084-1bee-46c6-8d1d-f6504e65a828-maas-api-xks-ca\") pod \"d4900084-1bee-46c6-8d1d-f6504e65a828\" (UID: \"d4900084-1bee-46c6-8d1d-f6504e65a828\") " Oct 31 05:05:14.652181 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:14.652142 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4900084-1bee-46c6-8d1d-f6504e65a828-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4900084-1bee-46c6-8d1d-f6504e65a828-maas-api-openshift-ca\") pod \"d4900084-1bee-46c6-8d1d-f6504e65a828\" (UID: \"d4900084-1bee-46c6-8d1d-f6504e65a828\") " Oct 31 05:05:14.652610 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:14.652571 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4900084-1bee-46c6-8d1d-f6504e65a828-maas-api-openshift-ca" pod "d4900084-1bee-46c6-8d1d-f6504e65a828" (UID: "d4900084-1bee-46c6-8d1d-f6504e65a828"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:05:14.653194 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:14.653166 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4900084-1bee-46c6-8d1d-f6504e65a828-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:14.654324 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:14.654300 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4900084-1bee-46c6-8d1d-f6504e65a828-kube-api-access-h7cgb" pod "d4900084-1bee-46c6-8d1d-f6504e65a828" (UID: "d4900084-1bee-46c6-8d1d-f6504e65a828"). InnerVolumeSpecName "kube-api-access-h7cgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:05:14.654454 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:14.654370 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4900084-1bee-46c6-8d1d-f6504e65a828-maas-api-xks-ca" pod "d4900084-1bee-46c6-8d1d-f6504e65a828" (UID: "d4900084-1bee-46c6-8d1d-f6504e65a828"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:05:14.755154 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:14.755107 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7cgb\" (UniqueName: \"kubernetes.io/projected/d4900084-1bee-46c6-8d1d-f6504e65a828-kube-api-access-h7cgb\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:14.755154 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:14.755149 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4900084-1bee-46c6-8d1d-f6504e65a828-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:14.896411 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:14.896358 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr"] Oct 31 05:05:14.896664 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:14.896631 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr" podUID="cbf01885-6e9e-405d-bf79-78a8bf2903a2" containerName="istio-proxy" containerID="cri-o://789859bc940166363d562c703ece64478381f2c08a18213f921ace41d38c93c0" gracePeriod=30 Oct 31 05:05:15.266592 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:15.266560 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52321c9-mc599" Oct 31 05:05:15.266592 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:15.266582 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52321c9-mc599" event={"ID":"d4900084-1bee-46c6-8d1d-f6504e65a828","Type":"ContainerDied","Data":"35386a331b08f49c4d925fe9fe1229088b9b4e99b6bdc0c8aa972d5e394b6e90"} Oct 31 05:05:15.267091 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:15.266639 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35386a331b08f49c4d925fe9fe1229088b9b4e99b6bdc0c8aa972d5e394b6e90" Oct 31 05:05:15.517063 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:15.516990 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0614feb6-glvs5" Oct 31 05:05:15.520158 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:15.520131 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b22a1dd5-7g82g" Oct 31 05:05:15.573603 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:15.573563 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9e616de-209d-46e9-b3b6-def8d0d0d844-kube-api-access-chh4r\" (UniqueName: \"kubernetes.io/projected/e9e616de-209d-46e9-b3b6-def8d0d0d844-kube-api-access-chh4r\") pod \"e9e616de-209d-46e9-b3b6-def8d0d0d844\" (UID: \"e9e616de-209d-46e9-b3b6-def8d0d0d844\") " Oct 31 05:05:15.573816 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:15.573618 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9e616de-209d-46e9-b3b6-def8d0d0d844-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9e616de-209d-46e9-b3b6-def8d0d0d844-maas-api-openshift-ca\") pod \"e9e616de-209d-46e9-b3b6-def8d0d0d844\" (UID: \"e9e616de-209d-46e9-b3b6-def8d0d0d844\") " Oct 31 05:05:15.573816 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:15.573653 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb6bda0f-355e-413f-b122-4d52c1b1fae0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb6bda0f-355e-413f-b122-4d52c1b1fae0-maas-api-openshift-ca\") pod \"fb6bda0f-355e-413f-b122-4d52c1b1fae0\" (UID: \"fb6bda0f-355e-413f-b122-4d52c1b1fae0\") " Oct 31 05:05:15.573816 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:15.573707 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb6bda0f-355e-413f-b122-4d52c1b1fae0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb6bda0f-355e-413f-b122-4d52c1b1fae0-maas-api-xks-ca\") pod \"fb6bda0f-355e-413f-b122-4d52c1b1fae0\" (UID: \"fb6bda0f-355e-413f-b122-4d52c1b1fae0\") " Oct 31 05:05:15.574024 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:15.573912 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb6bda0f-355e-413f-b122-4d52c1b1fae0-kube-api-access-5rrkx\" (UniqueName: \"kubernetes.io/projected/fb6bda0f-355e-413f-b122-4d52c1b1fae0-kube-api-access-5rrkx\") pod \"fb6bda0f-355e-413f-b122-4d52c1b1fae0\" (UID: \"fb6bda0f-355e-413f-b122-4d52c1b1fae0\") " Oct 31 05:05:15.574024 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:15.573998 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9e616de-209d-46e9-b3b6-def8d0d0d844-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9e616de-209d-46e9-b3b6-def8d0d0d844-maas-api-xks-ca\") pod \"e9e616de-209d-46e9-b3b6-def8d0d0d844\" (UID: \"e9e616de-209d-46e9-b3b6-def8d0d0d844\") " Oct 31 05:05:15.574515 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:15.574483 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9e616de-209d-46e9-b3b6-def8d0d0d844-maas-api-openshift-ca" pod "e9e616de-209d-46e9-b3b6-def8d0d0d844" (UID: "e9e616de-209d-46e9-b3b6-def8d0d0d844"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:05:15.574774 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:15.574748 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb6bda0f-355e-413f-b122-4d52c1b1fae0-maas-api-openshift-ca" pod "fb6bda0f-355e-413f-b122-4d52c1b1fae0" (UID: "fb6bda0f-355e-413f-b122-4d52c1b1fae0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:05:15.574985 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:15.574962 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9e616de-209d-46e9-b3b6-def8d0d0d844-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:15.574985 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:15.574986 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb6bda0f-355e-413f-b122-4d52c1b1fae0-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:15.576947 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:15.576915 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9e616de-209d-46e9-b3b6-def8d0d0d844-kube-api-access-chh4r" pod "e9e616de-209d-46e9-b3b6-def8d0d0d844" (UID: "e9e616de-209d-46e9-b3b6-def8d0d0d844"). InnerVolumeSpecName "kube-api-access-chh4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:05:15.578085 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:15.578053 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb6bda0f-355e-413f-b122-4d52c1b1fae0-maas-api-xks-ca" pod "fb6bda0f-355e-413f-b122-4d52c1b1fae0" (UID: "fb6bda0f-355e-413f-b122-4d52c1b1fae0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:05:15.578239 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:15.578220 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb6bda0f-355e-413f-b122-4d52c1b1fae0-kube-api-access-5rrkx" pod "fb6bda0f-355e-413f-b122-4d52c1b1fae0" (UID: "fb6bda0f-355e-413f-b122-4d52c1b1fae0"). InnerVolumeSpecName "kube-api-access-5rrkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:05:15.578696 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:15.578673 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9e616de-209d-46e9-b3b6-def8d0d0d844-maas-api-xks-ca" pod "e9e616de-209d-46e9-b3b6-def8d0d0d844" (UID: "e9e616de-209d-46e9-b3b6-def8d0d0d844"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:05:15.676600 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:15.676563 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chh4r\" (UniqueName: \"kubernetes.io/projected/e9e616de-209d-46e9-b3b6-def8d0d0d844-kube-api-access-chh4r\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:15.676600 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:15.676597 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb6bda0f-355e-413f-b122-4d52c1b1fae0-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:15.676600 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:15.676607 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rrkx\" (UniqueName: \"kubernetes.io/projected/fb6bda0f-355e-413f-b122-4d52c1b1fae0-kube-api-access-5rrkx\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:15.676853 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:15.676616 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9e616de-209d-46e9-b3b6-def8d0d0d844-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:16.280136 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:16.280101 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0614feb6-glvs5" Oct 31 05:05:16.280639 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:16.280114 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0614feb6-glvs5" event={"ID":"e9e616de-209d-46e9-b3b6-def8d0d0d844","Type":"ContainerDied","Data":"d0df7f8a4bf74753072b426e5989582fec1b77794c7d1d5bf90cfea1f4c4db76"} Oct 31 05:05:16.280639 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:16.280221 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0df7f8a4bf74753072b426e5989582fec1b77794c7d1d5bf90cfea1f4c4db76" Oct 31 05:05:16.281842 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:16.281816 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b22a1dd5-7g82g" Oct 31 05:05:16.281977 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:16.281815 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b22a1dd5-7g82g" event={"ID":"fb6bda0f-355e-413f-b122-4d52c1b1fae0","Type":"ContainerDied","Data":"7fa28e32a4e53ade69eef2a520f407b4fc63470bb8d41d400034f2c7bd49cb7f"} Oct 31 05:05:16.281977 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:16.281948 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7fa28e32a4e53ade69eef2a520f407b4fc63470bb8d41d400034f2c7bd49cb7f" Oct 31 05:05:17.301805 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:17.301765 5019 patch_prober.go:29] interesting pod/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 05:05:17.302224 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:17.301828 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr" podUID="cbf01885-6e9e-405d-bf79-78a8bf2903a2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 05:05:20.209535 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:20.209501 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr" Oct 31 05:05:20.245966 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:20.245935 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbf01885-6e9e-405d-bf79-78a8bf2903a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbf01885-6e9e-405d-bf79-78a8bf2903a2-istio-envoy\") pod \"cbf01885-6e9e-405d-bf79-78a8bf2903a2\" (UID: \"cbf01885-6e9e-405d-bf79-78a8bf2903a2\") " Oct 31 05:05:20.245966 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:20.245970 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbf01885-6e9e-405d-bf79-78a8bf2903a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbf01885-6e9e-405d-bf79-78a8bf2903a2-workload-certs\") pod \"cbf01885-6e9e-405d-bf79-78a8bf2903a2\" (UID: \"cbf01885-6e9e-405d-bf79-78a8bf2903a2\") " Oct 31 05:05:20.246163 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:20.246006 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbf01885-6e9e-405d-bf79-78a8bf2903a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf01885-6e9e-405d-bf79-78a8bf2903a2-credential-socket\") pod \"cbf01885-6e9e-405d-bf79-78a8bf2903a2\" (UID: \"cbf01885-6e9e-405d-bf79-78a8bf2903a2\") " Oct 31 05:05:20.246163 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:20.246088 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cbf01885-6e9e-405d-bf79-78a8bf2903a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbf01885-6e9e-405d-bf79-78a8bf2903a2-istio-podinfo\") pod \"cbf01885-6e9e-405d-bf79-78a8bf2903a2\" (UID: \"cbf01885-6e9e-405d-bf79-78a8bf2903a2\") " Oct 31 05:05:20.246163 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:20.246122 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbf01885-6e9e-405d-bf79-78a8bf2903a2-istio-token\" (UniqueName: \"kubernetes.io/projected/cbf01885-6e9e-405d-bf79-78a8bf2903a2-istio-token\") pod \"cbf01885-6e9e-405d-bf79-78a8bf2903a2\" (UID: \"cbf01885-6e9e-405d-bf79-78a8bf2903a2\") " Oct 31 05:05:20.246352 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:20.246204 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbf01885-6e9e-405d-bf79-78a8bf2903a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbf01885-6e9e-405d-bf79-78a8bf2903a2-istio-data\") pod \"cbf01885-6e9e-405d-bf79-78a8bf2903a2\" (UID: \"cbf01885-6e9e-405d-bf79-78a8bf2903a2\") " Oct 31 05:05:20.246352 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:20.246251 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbf01885-6e9e-405d-bf79-78a8bf2903a2-kube-api-access-pbpfv\" (UniqueName: \"kubernetes.io/projected/cbf01885-6e9e-405d-bf79-78a8bf2903a2-kube-api-access-pbpfv\") pod \"cbf01885-6e9e-405d-bf79-78a8bf2903a2\" (UID: \"cbf01885-6e9e-405d-bf79-78a8bf2903a2\") " Oct 31 05:05:20.246352 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:20.246258 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbf01885-6e9e-405d-bf79-78a8bf2903a2-workload-certs" pod "cbf01885-6e9e-405d-bf79-78a8bf2903a2" (UID: "cbf01885-6e9e-405d-bf79-78a8bf2903a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:05:20.246352 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:20.246312 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbf01885-6e9e-405d-bf79-78a8bf2903a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf01885-6e9e-405d-bf79-78a8bf2903a2-workload-socket\") pod \"cbf01885-6e9e-405d-bf79-78a8bf2903a2\" (UID: \"cbf01885-6e9e-405d-bf79-78a8bf2903a2\") " Oct 31 05:05:20.246599 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:20.246425 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbf01885-6e9e-405d-bf79-78a8bf2903a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbf01885-6e9e-405d-bf79-78a8bf2903a2-istiod-ca-cert\") pod \"cbf01885-6e9e-405d-bf79-78a8bf2903a2\" (UID: \"cbf01885-6e9e-405d-bf79-78a8bf2903a2\") " Oct 31 05:05:20.246599 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:20.246426 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbf01885-6e9e-405d-bf79-78a8bf2903a2-credential-socket" pod "cbf01885-6e9e-405d-bf79-78a8bf2903a2" (UID: "cbf01885-6e9e-405d-bf79-78a8bf2903a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:05:20.246720 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:20.246630 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbf01885-6e9e-405d-bf79-78a8bf2903a2-workload-socket" pod "cbf01885-6e9e-405d-bf79-78a8bf2903a2" (UID: "cbf01885-6e9e-405d-bf79-78a8bf2903a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:05:20.246804 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:20.246786 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbf01885-6e9e-405d-bf79-78a8bf2903a2-istiod-ca-cert" pod "cbf01885-6e9e-405d-bf79-78a8bf2903a2" (UID: "cbf01885-6e9e-405d-bf79-78a8bf2903a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:05:20.246865 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:20.246781 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbf01885-6e9e-405d-bf79-78a8bf2903a2-istio-data" pod "cbf01885-6e9e-405d-bf79-78a8bf2903a2" (UID: "cbf01885-6e9e-405d-bf79-78a8bf2903a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:05:20.247213 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:20.247192 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbf01885-6e9e-405d-bf79-78a8bf2903a2-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:20.247378 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:20.247214 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf01885-6e9e-405d-bf79-78a8bf2903a2-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:20.247378 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:20.247229 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbf01885-6e9e-405d-bf79-78a8bf2903a2-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:20.247378 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:20.247244 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf01885-6e9e-405d-bf79-78a8bf2903a2-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:20.247378 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:20.247255 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbf01885-6e9e-405d-bf79-78a8bf2903a2-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:20.248460 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:20.248433 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cbf01885-6e9e-405d-bf79-78a8bf2903a2-istio-podinfo" pod "cbf01885-6e9e-405d-bf79-78a8bf2903a2" (UID: "cbf01885-6e9e-405d-bf79-78a8bf2903a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:05:20.248567 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:20.248499 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbf01885-6e9e-405d-bf79-78a8bf2903a2-istio-envoy" pod "cbf01885-6e9e-405d-bf79-78a8bf2903a2" (UID: "cbf01885-6e9e-405d-bf79-78a8bf2903a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:05:20.248892 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:20.248874 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbf01885-6e9e-405d-bf79-78a8bf2903a2-kube-api-access-pbpfv" pod "cbf01885-6e9e-405d-bf79-78a8bf2903a2" (UID: "cbf01885-6e9e-405d-bf79-78a8bf2903a2"). InnerVolumeSpecName "kube-api-access-pbpfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:05:20.248951 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:20.248903 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbf01885-6e9e-405d-bf79-78a8bf2903a2-istio-token" pod "cbf01885-6e9e-405d-bf79-78a8bf2903a2" (UID: "cbf01885-6e9e-405d-bf79-78a8bf2903a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:05:20.321153 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:20.321110 5019 generic.go:356] "Generic (PLEG): container finished" podID="cbf01885-6e9e-405d-bf79-78a8bf2903a2" containerID="789859bc940166363d562c703ece64478381f2c08a18213f921ace41d38c93c0" exitCode=0 Oct 31 05:05:20.321379 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:20.321161 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr" event={"ID":"cbf01885-6e9e-405d-bf79-78a8bf2903a2","Type":"ContainerDied","Data":"789859bc940166363d562c703ece64478381f2c08a18213f921ace41d38c93c0"} Oct 31 05:05:20.321379 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:20.321233 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr" event={"ID":"cbf01885-6e9e-405d-bf79-78a8bf2903a2","Type":"ContainerDied","Data":"825a8821be0c8348dc133957650020a7831f990dee533bfa924c53f1f0d8c2a0"} Oct 31 05:05:20.321379 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:20.321172 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr" Oct 31 05:05:20.321379 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:20.321264 5019 scope.go:122] "RemoveContainer" containerID="789859bc940166363d562c703ece64478381f2c08a18213f921ace41d38c93c0" Oct 31 05:05:20.342155 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:20.342134 5019 scope.go:122] "RemoveContainer" containerID="8f87bddf195f526f26510317e72fd069561f58abedc758d21fa46414c3ce6c41" Oct 31 05:05:20.349287 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:20.349261 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbpfv\" (UniqueName: \"kubernetes.io/projected/cbf01885-6e9e-405d-bf79-78a8bf2903a2-kube-api-access-pbpfv\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:20.349411 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:20.349291 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbf01885-6e9e-405d-bf79-78a8bf2903a2-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:20.349411 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:20.349306 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbf01885-6e9e-405d-bf79-78a8bf2903a2-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:20.349411 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:20.349383 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbf01885-6e9e-405d-bf79-78a8bf2903a2-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:20.354310 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:20.354287 5019 scope.go:122] "RemoveContainer" containerID="789859bc940166363d562c703ece64478381f2c08a18213f921ace41d38c93c0" Oct 31 05:05:20.354715 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:05:20.354684 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"789859bc940166363d562c703ece64478381f2c08a18213f921ace41d38c93c0\": container with ID starting with 789859bc940166363d562c703ece64478381f2c08a18213f921ace41d38c93c0 not found: ID does not exist" containerID="789859bc940166363d562c703ece64478381f2c08a18213f921ace41d38c93c0" Oct 31 05:05:20.354799 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:20.354727 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"789859bc940166363d562c703ece64478381f2c08a18213f921ace41d38c93c0"} err="failed to get container status \"789859bc940166363d562c703ece64478381f2c08a18213f921ace41d38c93c0\": rpc error: code = NotFound desc = could not find container \"789859bc940166363d562c703ece64478381f2c08a18213f921ace41d38c93c0\": container with ID starting with 789859bc940166363d562c703ece64478381f2c08a18213f921ace41d38c93c0 not found: ID does not exist" Oct 31 05:05:20.354799 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:20.354753 5019 scope.go:122] "RemoveContainer" containerID="8f87bddf195f526f26510317e72fd069561f58abedc758d21fa46414c3ce6c41" Oct 31 05:05:20.355048 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:05:20.355028 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f87bddf195f526f26510317e72fd069561f58abedc758d21fa46414c3ce6c41\": container with ID starting with 8f87bddf195f526f26510317e72fd069561f58abedc758d21fa46414c3ce6c41 not found: ID does not exist" containerID="8f87bddf195f526f26510317e72fd069561f58abedc758d21fa46414c3ce6c41" Oct 31 05:05:20.355118 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:20.355054 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f87bddf195f526f26510317e72fd069561f58abedc758d21fa46414c3ce6c41"} err="failed to get container status \"8f87bddf195f526f26510317e72fd069561f58abedc758d21fa46414c3ce6c41\": rpc error: code = NotFound desc = could not find container \"8f87bddf195f526f26510317e72fd069561f58abedc758d21fa46414c3ce6c41\": container with ID starting with 8f87bddf195f526f26510317e72fd069561f58abedc758d21fa46414c3ce6c41 not found: ID does not exist" Oct 31 05:05:20.381998 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:20.381973 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr"] Oct 31 05:05:20.387643 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:20.387622 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-829ac7-openshift-default-7f8ccddb6f-q48vr"] Oct 31 05:05:21.410777 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:21.410739 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbf01885-6e9e-405d-bf79-78a8bf2903a2" path="/var/lib/kubelet/pods/cbf01885-6e9e-405d-bf79-78a8bf2903a2/volumes" Oct 31 05:05:22.393370 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:05:22.393310 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:05:22.960986 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:22.960950 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-191198b1-lhwmk"] Oct 31 05:05:22.961792 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:22.961771 5019 cpu_manager.go:414] "containerMap: removing container" podUID="d4900084-1bee-46c6-8d1d-f6504e65a828" containerName="revoke-keys" Oct 31 05:05:22.961792 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:22.961790 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4900084-1bee-46c6-8d1d-f6504e65a828" containerName="revoke-keys" Oct 31 05:05:22.961963 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:22.961813 5019 cpu_manager.go:414] "containerMap: removing container" podUID="fb6bda0f-355e-413f-b122-4d52c1b1fae0" containerName="revoke-keys" Oct 31 05:05:22.961963 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:22.961819 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb6bda0f-355e-413f-b122-4d52c1b1fae0" containerName="revoke-keys" Oct 31 05:05:22.961963 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:22.961830 5019 cpu_manager.go:414] "containerMap: removing container" podUID="e9e616de-209d-46e9-b3b6-def8d0d0d844" containerName="revoke-keys" Oct 31 05:05:22.961963 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:22.961835 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9e616de-209d-46e9-b3b6-def8d0d0d844" containerName="revoke-keys" Oct 31 05:05:22.961963 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:22.961853 5019 cpu_manager.go:414] "containerMap: removing container" podUID="cbf01885-6e9e-405d-bf79-78a8bf2903a2" containerName="istio-proxy" Oct 31 05:05:22.961963 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:22.961858 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbf01885-6e9e-405d-bf79-78a8bf2903a2" containerName="istio-proxy" Oct 31 05:05:22.961963 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:22.961881 5019 cpu_manager.go:414] "containerMap: removing container" podUID="cbf01885-6e9e-405d-bf79-78a8bf2903a2" containerName="istio-proxy" Oct 31 05:05:22.961963 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:22.961886 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbf01885-6e9e-405d-bf79-78a8bf2903a2" containerName="istio-proxy" Oct 31 05:05:22.962412 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:22.961993 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbf01885-6e9e-405d-bf79-78a8bf2903a2" containerName="istio-proxy" Oct 31 05:05:22.962412 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:22.962005 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4900084-1bee-46c6-8d1d-f6504e65a828" containerName="revoke-keys" Oct 31 05:05:22.962412 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:22.962018 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb6bda0f-355e-413f-b122-4d52c1b1fae0" containerName="revoke-keys" Oct 31 05:05:22.962412 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:22.962025 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9e616de-209d-46e9-b3b6-def8d0d0d844" containerName="revoke-keys" Oct 31 05:05:22.962412 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:22.962035 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbf01885-6e9e-405d-bf79-78a8bf2903a2" containerName="istio-proxy" Oct 31 05:05:22.967249 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:22.967227 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-191198b1-lhwmk" Oct 31 05:05:22.984347 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:22.984307 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-191198b1-lhwmk"] Oct 31 05:05:22.989914 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:22.989892 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:05:22.990148 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:22.990126 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 05:05:22.990272 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:22.990248 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:05:22.999431 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:22.999406 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:05:23.117850 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:23.117813 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08a4cd92-dc05-43bd-b0a6-07a9b3c31cf0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-191198b1-lhwmk\" (UID: \"08a4cd92-dc05-43bd-b0a6-07a9b3c31cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-191198b1-lhwmk" Oct 31 05:05:23.118062 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:23.118034 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rqsz\" (UniqueName: \"kubernetes.io/projected/08a4cd92-dc05-43bd-b0a6-07a9b3c31cf0-kube-api-access-6rqsz\") pod \"maas-api-revoke-sub-191198b1-lhwmk\" (UID: \"08a4cd92-dc05-43bd-b0a6-07a9b3c31cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-191198b1-lhwmk" Oct 31 05:05:23.118175 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:23.118156 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08a4cd92-dc05-43bd-b0a6-07a9b3c31cf0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-191198b1-lhwmk\" (UID: \"08a4cd92-dc05-43bd-b0a6-07a9b3c31cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-191198b1-lhwmk" Oct 31 05:05:23.220061 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:23.220028 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rqsz\" (UniqueName: \"kubernetes.io/projected/08a4cd92-dc05-43bd-b0a6-07a9b3c31cf0-kube-api-access-6rqsz\") pod \"maas-api-revoke-sub-191198b1-lhwmk\" (UID: \"08a4cd92-dc05-43bd-b0a6-07a9b3c31cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-191198b1-lhwmk" Oct 31 05:05:23.220242 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:23.220092 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08a4cd92-dc05-43bd-b0a6-07a9b3c31cf0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-191198b1-lhwmk\" (UID: \"08a4cd92-dc05-43bd-b0a6-07a9b3c31cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-191198b1-lhwmk" Oct 31 05:05:23.220368 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:23.220255 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08a4cd92-dc05-43bd-b0a6-07a9b3c31cf0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-191198b1-lhwmk\" (UID: \"08a4cd92-dc05-43bd-b0a6-07a9b3c31cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-191198b1-lhwmk" Oct 31 05:05:23.220827 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:23.220801 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08a4cd92-dc05-43bd-b0a6-07a9b3c31cf0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-191198b1-lhwmk\" (UID: \"08a4cd92-dc05-43bd-b0a6-07a9b3c31cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-191198b1-lhwmk" Oct 31 05:05:23.222745 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:23.222718 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08a4cd92-dc05-43bd-b0a6-07a9b3c31cf0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-191198b1-lhwmk\" (UID: \"08a4cd92-dc05-43bd-b0a6-07a9b3c31cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-191198b1-lhwmk" Oct 31 05:05:23.243728 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:23.243703 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rqsz\" (UniqueName: \"kubernetes.io/projected/08a4cd92-dc05-43bd-b0a6-07a9b3c31cf0-kube-api-access-6rqsz\") pod \"maas-api-revoke-sub-191198b1-lhwmk\" (UID: \"08a4cd92-dc05-43bd-b0a6-07a9b3c31cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-191198b1-lhwmk" Oct 31 05:05:23.285973 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:23.285940 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-191198b1-lhwmk" Oct 31 05:05:23.659255 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:23.659229 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-191198b1-lhwmk"] Oct 31 05:05:23.670066 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:05:23.662708 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08a4cd92_dc05_43bd_b0a6_07a9b3c31cf0.slice/crio-824b13186b33876163c4609221cc051e8f392d02ecb0717d916b8623d906bf0c WatchSource:0}: Error finding container 824b13186b33876163c4609221cc051e8f392d02ecb0717d916b8623d906bf0c: Status 404 returned error can't find the container with id 824b13186b33876163c4609221cc051e8f392d02ecb0717d916b8623d906bf0c Oct 31 05:05:24.099979 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:24.099928 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50e186c4-d68px"] Oct 31 05:05:24.107756 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:24.107730 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50e186c4-d68px" Oct 31 05:05:24.122748 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:24.122719 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50e186c4-d68px"] Oct 31 05:05:24.262949 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:24.262852 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80f6dda9-d1b5-4990-be72-199a370a953a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50e186c4-d68px\" (UID: \"80f6dda9-d1b5-4990-be72-199a370a953a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50e186c4-d68px" Oct 31 05:05:24.263131 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:24.262987 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80f6dda9-d1b5-4990-be72-199a370a953a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50e186c4-d68px\" (UID: \"80f6dda9-d1b5-4990-be72-199a370a953a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50e186c4-d68px" Oct 31 05:05:24.263131 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:24.263081 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdtcx\" (UniqueName: \"kubernetes.io/projected/80f6dda9-d1b5-4990-be72-199a370a953a-kube-api-access-tdtcx\") pod \"maas-api-revoke-sub-50e186c4-d68px\" (UID: \"80f6dda9-d1b5-4990-be72-199a370a953a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50e186c4-d68px" Oct 31 05:05:24.364747 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:24.364711 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80f6dda9-d1b5-4990-be72-199a370a953a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50e186c4-d68px\" (UID: \"80f6dda9-d1b5-4990-be72-199a370a953a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50e186c4-d68px" Oct 31 05:05:24.364914 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:24.364803 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80f6dda9-d1b5-4990-be72-199a370a953a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50e186c4-d68px\" (UID: \"80f6dda9-d1b5-4990-be72-199a370a953a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50e186c4-d68px" Oct 31 05:05:24.364914 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:24.364861 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdtcx\" (UniqueName: \"kubernetes.io/projected/80f6dda9-d1b5-4990-be72-199a370a953a-kube-api-access-tdtcx\") pod \"maas-api-revoke-sub-50e186c4-d68px\" (UID: \"80f6dda9-d1b5-4990-be72-199a370a953a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50e186c4-d68px" Oct 31 05:05:24.365428 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:24.365407 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80f6dda9-d1b5-4990-be72-199a370a953a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50e186c4-d68px\" (UID: \"80f6dda9-d1b5-4990-be72-199a370a953a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50e186c4-d68px" Oct 31 05:05:24.367450 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:24.367422 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80f6dda9-d1b5-4990-be72-199a370a953a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50e186c4-d68px\" (UID: \"80f6dda9-d1b5-4990-be72-199a370a953a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50e186c4-d68px" Oct 31 05:05:24.368698 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:24.368673 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-191198b1-lhwmk" event={"ID":"08a4cd92-dc05-43bd-b0a6-07a9b3c31cf0","Type":"ContainerStarted","Data":"98149429bf613b96906c313e9935e122a789a4bc56dcc716aeb7de949211f9b4"} Oct 31 05:05:24.368814 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:24.368725 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-191198b1-lhwmk" event={"ID":"08a4cd92-dc05-43bd-b0a6-07a9b3c31cf0","Type":"ContainerStarted","Data":"824b13186b33876163c4609221cc051e8f392d02ecb0717d916b8623d906bf0c"} Oct 31 05:05:24.388556 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:24.388529 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdtcx\" (UniqueName: \"kubernetes.io/projected/80f6dda9-d1b5-4990-be72-199a370a953a-kube-api-access-tdtcx\") pod \"maas-api-revoke-sub-50e186c4-d68px\" (UID: \"80f6dda9-d1b5-4990-be72-199a370a953a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50e186c4-d68px" Oct 31 05:05:24.415559 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:24.415499 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-191198b1-lhwmk" podStartSLOduration=2.415485107 podStartE2EDuration="2.415485107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:05:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:05:24.411201927 +0000 UTC m=+3679.167539968" watchObservedRunningTime="2026-10-31 05:05:24.415485107 +0000 UTC m=+3679.171823144" Oct 31 05:05:24.429121 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:24.429091 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50e186c4-d68px" Oct 31 05:05:24.746709 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:05:24.746672 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80f6dda9_d1b5_4990_be72_199a370a953a.slice/crio-2958057e979cbdbbe022e02adc9a812ab75bbb146efa711d68c35c9348adf34b WatchSource:0}: Error finding container 2958057e979cbdbbe022e02adc9a812ab75bbb146efa711d68c35c9348adf34b: Status 404 returned error can't find the container with id 2958057e979cbdbbe022e02adc9a812ab75bbb146efa711d68c35c9348adf34b Oct 31 05:05:24.747905 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:24.747882 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50e186c4-d68px"] Oct 31 05:05:25.381591 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:25.381552 5019 generic.go:356] "Generic (PLEG): container finished" podID="08a4cd92-dc05-43bd-b0a6-07a9b3c31cf0" containerID="98149429bf613b96906c313e9935e122a789a4bc56dcc716aeb7de949211f9b4" exitCode=0 Oct 31 05:05:25.382078 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:25.381643 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-191198b1-lhwmk" event={"ID":"08a4cd92-dc05-43bd-b0a6-07a9b3c31cf0","Type":"ContainerDied","Data":"98149429bf613b96906c313e9935e122a789a4bc56dcc716aeb7de949211f9b4"} Oct 31 05:05:25.383936 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:25.383903 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50e186c4-d68px" event={"ID":"80f6dda9-d1b5-4990-be72-199a370a953a","Type":"ContainerStarted","Data":"6dec045c18a305092161f0cdf8811661b3a23aba3256d0b0601831a04c6ea3e5"} Oct 31 05:05:25.384082 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:25.383962 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50e186c4-d68px" event={"ID":"80f6dda9-d1b5-4990-be72-199a370a953a","Type":"ContainerStarted","Data":"2958057e979cbdbbe022e02adc9a812ab75bbb146efa711d68c35c9348adf34b"} Oct 31 05:05:25.488632 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:25.488534 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50e186c4-d68px" podStartSLOduration=1.488514945 podStartE2EDuration="1.488514945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:05:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:05:25.461134722 +0000 UTC m=+3680.217472787" watchObservedRunningTime="2026-10-31 05:05:25.488514945 +0000 UTC m=+3680.244852987" Oct 31 05:05:26.395409 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:26.395373 5019 generic.go:356] "Generic (PLEG): container finished" podID="80f6dda9-d1b5-4990-be72-199a370a953a" containerID="6dec045c18a305092161f0cdf8811661b3a23aba3256d0b0601831a04c6ea3e5" exitCode=0 Oct 31 05:05:26.395913 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:26.395464 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50e186c4-d68px" event={"ID":"80f6dda9-d1b5-4990-be72-199a370a953a","Type":"ContainerDied","Data":"6dec045c18a305092161f0cdf8811661b3a23aba3256d0b0601831a04c6ea3e5"} Oct 31 05:05:26.600614 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:26.600587 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-191198b1-lhwmk" Oct 31 05:05:26.708979 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:26.708896 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08a4cd92-dc05-43bd-b0a6-07a9b3c31cf0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08a4cd92-dc05-43bd-b0a6-07a9b3c31cf0-maas-api-xks-ca\") pod \"08a4cd92-dc05-43bd-b0a6-07a9b3c31cf0\" (UID: \"08a4cd92-dc05-43bd-b0a6-07a9b3c31cf0\") " Oct 31 05:05:26.709157 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:26.709020 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08a4cd92-dc05-43bd-b0a6-07a9b3c31cf0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08a4cd92-dc05-43bd-b0a6-07a9b3c31cf0-maas-api-openshift-ca\") pod \"08a4cd92-dc05-43bd-b0a6-07a9b3c31cf0\" (UID: \"08a4cd92-dc05-43bd-b0a6-07a9b3c31cf0\") " Oct 31 05:05:26.709157 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:26.709146 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08a4cd92-dc05-43bd-b0a6-07a9b3c31cf0-kube-api-access-6rqsz\" (UniqueName: \"kubernetes.io/projected/08a4cd92-dc05-43bd-b0a6-07a9b3c31cf0-kube-api-access-6rqsz\") pod \"08a4cd92-dc05-43bd-b0a6-07a9b3c31cf0\" (UID: \"08a4cd92-dc05-43bd-b0a6-07a9b3c31cf0\") " Oct 31 05:05:26.709731 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:26.709702 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08a4cd92-dc05-43bd-b0a6-07a9b3c31cf0-maas-api-openshift-ca" pod "08a4cd92-dc05-43bd-b0a6-07a9b3c31cf0" (UID: "08a4cd92-dc05-43bd-b0a6-07a9b3c31cf0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:05:26.709955 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:26.709933 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08a4cd92-dc05-43bd-b0a6-07a9b3c31cf0-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:26.712073 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:26.712047 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08a4cd92-dc05-43bd-b0a6-07a9b3c31cf0-kube-api-access-6rqsz" pod "08a4cd92-dc05-43bd-b0a6-07a9b3c31cf0" (UID: "08a4cd92-dc05-43bd-b0a6-07a9b3c31cf0"). InnerVolumeSpecName "kube-api-access-6rqsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:05:26.712208 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:26.712084 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08a4cd92-dc05-43bd-b0a6-07a9b3c31cf0-maas-api-xks-ca" pod "08a4cd92-dc05-43bd-b0a6-07a9b3c31cf0" (UID: "08a4cd92-dc05-43bd-b0a6-07a9b3c31cf0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:05:26.811137 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:26.811101 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08a4cd92-dc05-43bd-b0a6-07a9b3c31cf0-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:26.811137 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:26.811132 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rqsz\" (UniqueName: \"kubernetes.io/projected/08a4cd92-dc05-43bd-b0a6-07a9b3c31cf0-kube-api-access-6rqsz\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:27.364783 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:27.364738 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1daf30/test-model-1daf30-kserve-8686666dc7-xc4n6"] Oct 31 05:05:27.425359 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:27.424733 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-191198b1-lhwmk" Oct 31 05:05:27.427791 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:27.427762 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-191198b1-lhwmk" event={"ID":"08a4cd92-dc05-43bd-b0a6-07a9b3c31cf0","Type":"ContainerDied","Data":"824b13186b33876163c4609221cc051e8f392d02ecb0717d916b8623d906bf0c"} Oct 31 05:05:27.427944 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:27.427803 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="824b13186b33876163c4609221cc051e8f392d02ecb0717d916b8623d906bf0c" Oct 31 05:05:27.769129 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:27.769100 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50e186c4-d68px" Oct 31 05:05:27.932828 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:27.932791 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80f6dda9-d1b5-4990-be72-199a370a953a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80f6dda9-d1b5-4990-be72-199a370a953a-maas-api-xks-ca\") pod \"80f6dda9-d1b5-4990-be72-199a370a953a\" (UID: \"80f6dda9-d1b5-4990-be72-199a370a953a\") " Oct 31 05:05:27.933062 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:27.933026 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80f6dda9-d1b5-4990-be72-199a370a953a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80f6dda9-d1b5-4990-be72-199a370a953a-maas-api-openshift-ca\") pod \"80f6dda9-d1b5-4990-be72-199a370a953a\" (UID: \"80f6dda9-d1b5-4990-be72-199a370a953a\") " Oct 31 05:05:27.933210 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:27.933190 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80f6dda9-d1b5-4990-be72-199a370a953a-kube-api-access-tdtcx\" (UniqueName: \"kubernetes.io/projected/80f6dda9-d1b5-4990-be72-199a370a953a-kube-api-access-tdtcx\") pod \"80f6dda9-d1b5-4990-be72-199a370a953a\" (UID: \"80f6dda9-d1b5-4990-be72-199a370a953a\") " Oct 31 05:05:27.933468 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:27.933440 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80f6dda9-d1b5-4990-be72-199a370a953a-maas-api-openshift-ca" pod "80f6dda9-d1b5-4990-be72-199a370a953a" (UID: "80f6dda9-d1b5-4990-be72-199a370a953a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:05:27.933982 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:27.933957 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80f6dda9-d1b5-4990-be72-199a370a953a-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:27.935438 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:27.935408 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80f6dda9-d1b5-4990-be72-199a370a953a-maas-api-xks-ca" pod "80f6dda9-d1b5-4990-be72-199a370a953a" (UID: "80f6dda9-d1b5-4990-be72-199a370a953a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:05:27.935566 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:27.935457 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80f6dda9-d1b5-4990-be72-199a370a953a-kube-api-access-tdtcx" pod "80f6dda9-d1b5-4990-be72-199a370a953a" (UID: "80f6dda9-d1b5-4990-be72-199a370a953a"). InnerVolumeSpecName "kube-api-access-tdtcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:05:28.035050 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:28.034959 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdtcx\" (UniqueName: \"kubernetes.io/projected/80f6dda9-d1b5-4990-be72-199a370a953a-kube-api-access-tdtcx\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:28.035050 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:28.034992 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80f6dda9-d1b5-4990-be72-199a370a953a-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:28.048983 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:28.048945 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a76d3e66-hb2pf"] Oct 31 05:05:28.049816 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:28.049795 5019 cpu_manager.go:414] "containerMap: removing container" podUID="80f6dda9-d1b5-4990-be72-199a370a953a" containerName="revoke-keys" Oct 31 05:05:28.049816 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:28.049814 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80f6dda9-d1b5-4990-be72-199a370a953a" containerName="revoke-keys" Oct 31 05:05:28.050002 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:28.049836 5019 cpu_manager.go:414] "containerMap: removing container" podUID="08a4cd92-dc05-43bd-b0a6-07a9b3c31cf0" containerName="revoke-keys" Oct 31 05:05:28.050002 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:28.049842 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08a4cd92-dc05-43bd-b0a6-07a9b3c31cf0" containerName="revoke-keys" Oct 31 05:05:28.050002 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:28.049982 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="80f6dda9-d1b5-4990-be72-199a370a953a" containerName="revoke-keys" Oct 31 05:05:28.050002 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:28.049995 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="08a4cd92-dc05-43bd-b0a6-07a9b3c31cf0" containerName="revoke-keys" Oct 31 05:05:28.054028 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:28.054005 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76d3e66-hb2pf" Oct 31 05:05:28.090114 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:28.090076 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a76d3e66-hb2pf"] Oct 31 05:05:28.136175 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:28.136139 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd7eece6-db42-4033-9f69-74bfd818bece-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a76d3e66-hb2pf\" (UID: \"bd7eece6-db42-4033-9f69-74bfd818bece\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76d3e66-hb2pf" Oct 31 05:05:28.136393 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:28.136222 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd7eece6-db42-4033-9f69-74bfd818bece-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a76d3e66-hb2pf\" (UID: \"bd7eece6-db42-4033-9f69-74bfd818bece\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76d3e66-hb2pf" Oct 31 05:05:28.136393 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:28.136262 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m628\" (UniqueName: \"kubernetes.io/projected/bd7eece6-db42-4033-9f69-74bfd818bece-kube-api-access-8m628\") pod \"maas-api-revoke-sub-a76d3e66-hb2pf\" (UID: \"bd7eece6-db42-4033-9f69-74bfd818bece\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76d3e66-hb2pf" Oct 31 05:05:28.238270 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:28.238235 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m628\" (UniqueName: \"kubernetes.io/projected/bd7eece6-db42-4033-9f69-74bfd818bece-kube-api-access-8m628\") pod \"maas-api-revoke-sub-a76d3e66-hb2pf\" (UID: \"bd7eece6-db42-4033-9f69-74bfd818bece\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76d3e66-hb2pf" Oct 31 05:05:28.238590 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:28.238567 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd7eece6-db42-4033-9f69-74bfd818bece-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a76d3e66-hb2pf\" (UID: \"bd7eece6-db42-4033-9f69-74bfd818bece\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76d3e66-hb2pf" Oct 31 05:05:28.238761 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:28.238652 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd7eece6-db42-4033-9f69-74bfd818bece-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a76d3e66-hb2pf\" (UID: \"bd7eece6-db42-4033-9f69-74bfd818bece\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76d3e66-hb2pf" Oct 31 05:05:28.239322 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:28.239297 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd7eece6-db42-4033-9f69-74bfd818bece-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a76d3e66-hb2pf\" (UID: \"bd7eece6-db42-4033-9f69-74bfd818bece\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76d3e66-hb2pf" Oct 31 05:05:28.241086 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:28.241064 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd7eece6-db42-4033-9f69-74bfd818bece-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a76d3e66-hb2pf\" (UID: \"bd7eece6-db42-4033-9f69-74bfd818bece\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76d3e66-hb2pf" Oct 31 05:05:28.261098 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:28.261074 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m628\" (UniqueName: \"kubernetes.io/projected/bd7eece6-db42-4033-9f69-74bfd818bece-kube-api-access-8m628\") pod \"maas-api-revoke-sub-a76d3e66-hb2pf\" (UID: \"bd7eece6-db42-4033-9f69-74bfd818bece\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76d3e66-hb2pf" Oct 31 05:05:28.396420 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:28.396280 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76d3e66-hb2pf" Oct 31 05:05:28.436772 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:28.436743 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50e186c4-d68px" Oct 31 05:05:28.437203 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:28.436824 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50e186c4-d68px" event={"ID":"80f6dda9-d1b5-4990-be72-199a370a953a","Type":"ContainerDied","Data":"2958057e979cbdbbe022e02adc9a812ab75bbb146efa711d68c35c9348adf34b"} Oct 31 05:05:28.437203 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:28.436907 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2958057e979cbdbbe022e02adc9a812ab75bbb146efa711d68c35c9348adf34b" Oct 31 05:05:28.719393 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:28.719356 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a76d3e66-hb2pf"] Oct 31 05:05:28.719599 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:05:28.719572 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd7eece6_db42_4033_9f69_74bfd818bece.slice/crio-178a03450cf4a7bac0dc6b6c671b5c8446ee81864868fcef702fe05e867e7203 WatchSource:0}: Error finding container 178a03450cf4a7bac0dc6b6c671b5c8446ee81864868fcef702fe05e867e7203: Status 404 returned error can't find the container with id 178a03450cf4a7bac0dc6b6c671b5c8446ee81864868fcef702fe05e867e7203 Oct 31 05:05:29.449245 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:29.449145 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76d3e66-hb2pf" event={"ID":"bd7eece6-db42-4033-9f69-74bfd818bece","Type":"ContainerStarted","Data":"5ddf8695c1aa044aa22478aedd977245e8e0c35d1a1640a32d14476ba7bb4402"} Oct 31 05:05:29.449245 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:29.449217 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76d3e66-hb2pf" event={"ID":"bd7eece6-db42-4033-9f69-74bfd818bece","Type":"ContainerStarted","Data":"178a03450cf4a7bac0dc6b6c671b5c8446ee81864868fcef702fe05e867e7203"} Oct 31 05:05:29.500275 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:29.500211 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76d3e66-hb2pf" podStartSLOduration=1.500193793 podStartE2EDuration="1.500193793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:05:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:05:29.499302089 +0000 UTC m=+3684.255640128" watchObservedRunningTime="2026-10-31 05:05:29.500193793 +0000 UTC m=+3684.256532236" Oct 31 05:05:30.459868 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:30.459778 5019 generic.go:356] "Generic (PLEG): container finished" podID="bd7eece6-db42-4033-9f69-74bfd818bece" containerID="5ddf8695c1aa044aa22478aedd977245e8e0c35d1a1640a32d14476ba7bb4402" exitCode=0 Oct 31 05:05:30.460250 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:30.459860 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76d3e66-hb2pf" event={"ID":"bd7eece6-db42-4033-9f69-74bfd818bece","Type":"ContainerDied","Data":"5ddf8695c1aa044aa22478aedd977245e8e0c35d1a1640a32d14476ba7bb4402"} Oct 31 05:05:31.669100 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:31.669067 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76d3e66-hb2pf" Oct 31 05:05:31.695920 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:31.695888 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd7eece6-db42-4033-9f69-74bfd818bece-kube-api-access-8m628\" (UniqueName: \"kubernetes.io/projected/bd7eece6-db42-4033-9f69-74bfd818bece-kube-api-access-8m628\") pod \"bd7eece6-db42-4033-9f69-74bfd818bece\" (UID: \"bd7eece6-db42-4033-9f69-74bfd818bece\") " Oct 31 05:05:31.696136 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:31.696029 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd7eece6-db42-4033-9f69-74bfd818bece-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd7eece6-db42-4033-9f69-74bfd818bece-maas-api-xks-ca\") pod \"bd7eece6-db42-4033-9f69-74bfd818bece\" (UID: \"bd7eece6-db42-4033-9f69-74bfd818bece\") " Oct 31 05:05:31.696136 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:31.696087 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd7eece6-db42-4033-9f69-74bfd818bece-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd7eece6-db42-4033-9f69-74bfd818bece-maas-api-openshift-ca\") pod \"bd7eece6-db42-4033-9f69-74bfd818bece\" (UID: \"bd7eece6-db42-4033-9f69-74bfd818bece\") " Oct 31 05:05:31.696621 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:31.696589 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd7eece6-db42-4033-9f69-74bfd818bece-maas-api-openshift-ca" pod "bd7eece6-db42-4033-9f69-74bfd818bece" (UID: "bd7eece6-db42-4033-9f69-74bfd818bece"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:05:31.696843 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:31.696820 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd7eece6-db42-4033-9f69-74bfd818bece-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:31.698255 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:31.698229 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd7eece6-db42-4033-9f69-74bfd818bece-kube-api-access-8m628" pod "bd7eece6-db42-4033-9f69-74bfd818bece" (UID: "bd7eece6-db42-4033-9f69-74bfd818bece"). InnerVolumeSpecName "kube-api-access-8m628". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:05:31.698408 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:31.698277 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd7eece6-db42-4033-9f69-74bfd818bece-maas-api-xks-ca" pod "bd7eece6-db42-4033-9f69-74bfd818bece" (UID: "bd7eece6-db42-4033-9f69-74bfd818bece"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:05:31.798159 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:31.798060 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd7eece6-db42-4033-9f69-74bfd818bece-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:31.798159 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:31.798103 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m628\" (UniqueName: \"kubernetes.io/projected/bd7eece6-db42-4033-9f69-74bfd818bece-kube-api-access-8m628\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:32.484218 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:32.484185 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76d3e66-hb2pf" Oct 31 05:05:32.484446 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:32.484210 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76d3e66-hb2pf" event={"ID":"bd7eece6-db42-4033-9f69-74bfd818bece","Type":"ContainerDied","Data":"178a03450cf4a7bac0dc6b6c671b5c8446ee81864868fcef702fe05e867e7203"} Oct 31 05:05:32.484446 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:32.484273 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="178a03450cf4a7bac0dc6b6c671b5c8446ee81864868fcef702fe05e867e7203" Oct 31 05:05:36.913929 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:36.913890 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c1fe26/test-model-c1fe26-kserve-7fd8b8b8f-qkbvb"] Oct 31 05:05:42.409601 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:42.409553 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-1daf30/test-model-1daf30-kserve-8686666dc7-xc4n6" podUID="f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7" containerName="main" containerID="cri-o://a391f949b7c22762f84393b8897b0c5dd3f4aa344f17d27d1c538ac54d6fded0" gracePeriod=45 Oct 31 05:05:42.703430 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:42.703290 5019 generic.go:356] "Generic (PLEG): container finished" podID="f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7" containerID="a391f949b7c22762f84393b8897b0c5dd3f4aa344f17d27d1c538ac54d6fded0" exitCode=0 Oct 31 05:05:42.703430 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:42.703408 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1daf30/test-model-1daf30-kserve-8686666dc7-xc4n6" event={"ID":"f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7","Type":"ContainerDied","Data":"a391f949b7c22762f84393b8897b0c5dd3f4aa344f17d27d1c538ac54d6fded0"} Oct 31 05:05:42.746124 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:42.746093 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1daf30/test-model-1daf30-kserve-8686666dc7-xc4n6" Oct 31 05:05:42.859432 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:42.859391 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7-kube-api-access-kgb62\" (UniqueName: \"kubernetes.io/projected/f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7-kube-api-access-kgb62\") pod \"f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7\" (UID: \"f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7\") " Oct 31 05:05:42.859639 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:42.859519 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7-dshm\") pod \"f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7\" (UID: \"f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7\") " Oct 31 05:05:42.859639 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:42.859605 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7-tmp-dir\") pod \"f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7\" (UID: \"f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7\") " Oct 31 05:05:42.859787 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:42.859676 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7-home\" (UniqueName: \"kubernetes.io/empty-dir/f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7-home\") pod \"f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7\" (UID: \"f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7\") " Oct 31 05:05:42.859787 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:42.859766 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7-model-cache\") pod \"f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7\" (UID: \"f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7\") " Oct 31 05:05:42.859907 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:42.859832 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7-tls-certs\" (UniqueName: \"kubernetes.io/secret/f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7-tls-certs\") pod \"f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7\" (UID: \"f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7\") " Oct 31 05:05:42.859965 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:42.859895 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7-tmp-dir" pod "f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7" (UID: "f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:05:42.860022 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:42.859985 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7-home" pod "f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7" (UID: "f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:05:42.860281 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:42.860087 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7-model-cache" pod "f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7" (UID: "f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:05:42.860568 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:42.860541 5019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7-tmp-dir\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:42.860678 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:42.860572 5019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7-home\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:42.860678 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:42.860586 5019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7-model-cache\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:42.862076 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:42.861968 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7-dshm" pod "f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7" (UID: "f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:05:42.862076 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:42.862046 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7-kube-api-access-kgb62" pod "f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7" (UID: "f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7"). InnerVolumeSpecName "kube-api-access-kgb62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:05:42.862284 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:42.862262 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7-tls-certs" pod "f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7" (UID: "f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:05:42.961699 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:42.961621 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgb62\" (UniqueName: \"kubernetes.io/projected/f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7-kube-api-access-kgb62\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:42.961699 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:42.961653 5019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7-dshm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:42.961699 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:42.961663 5019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7-tls-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:43.716165 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:43.716133 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1daf30/test-model-1daf30-kserve-8686666dc7-xc4n6" event={"ID":"f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7","Type":"ContainerDied","Data":"8d674b03a2ced65525cb16a198e65c14d8dbd8ac1649cd982c2691eedf0f5fe0"} Oct 31 05:05:43.716657 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:43.716175 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1daf30/test-model-1daf30-kserve-8686666dc7-xc4n6" Oct 31 05:05:43.716657 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:43.716186 5019 scope.go:122] "RemoveContainer" containerID="a391f949b7c22762f84393b8897b0c5dd3f4aa344f17d27d1c538ac54d6fded0" Oct 31 05:05:43.770611 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:43.770574 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1daf30/test-model-1daf30-kserve-8686666dc7-xc4n6"] Oct 31 05:05:43.781691 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:43.781660 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-1daf30/test-model-1daf30-kserve-8686666dc7-xc4n6"] Oct 31 05:05:43.860655 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:43.860620 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-04d111-f77b9866c-2mzsk"] Oct 31 05:05:43.889645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:43.889616 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-04d111-76d4bc867-jmfjn"] Oct 31 05:05:44.296984 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:44.296943 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-12c0bc/test-model-12c0bc-kserve-66b5ccb945-4sg7l"] Oct 31 05:05:44.583446 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:44.583358 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl"] Oct 31 05:05:44.583631 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:44.583605 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl" podUID="1d46e1a8-ef71-4bbf-afec-a074d0fcfe05" containerName="istio-proxy" containerID="cri-o://36edf30386af908410f4e4c029a008a267547565f3f9e126b1fc8054065d98b1" gracePeriod=30 Oct 31 05:05:45.410813 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:45.410772 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7" path="/var/lib/kubelet/pods/f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7/volumes" Oct 31 05:05:48.861773 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:48.861730 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-04d111-f77b9866c-2mzsk" podUID="bb04ce0c-726f-4b27-b588-1b30fee2c0a0" containerName="payload-processing" containerID="cri-o://b731bdc617d726b85ed959ae0e70fc1aef8a110f5126aa663322b09e4ff8dc5d" gracePeriod=25 Oct 31 05:05:48.890575 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:48.890537 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04d111-76d4bc867-jmfjn" podUID="4fb383ce-a4bd-4455-b351-34bc277bc3e5" containerName="payload-pre-processing" containerID="cri-o://4ec33c2dfccb7259bd251ec4c7e91629209092e19f7c976acd8f583a18cffb17" gracePeriod=25 Oct 31 05:05:49.178783 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:49.178752 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-04d111-f77b9866c-2mzsk" Oct 31 05:05:49.280778 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:49.280737 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb04ce0c-726f-4b27-b588-1b30fee2c0a0-kube-api-access-7kkk9\" (UniqueName: \"kubernetes.io/projected/bb04ce0c-726f-4b27-b588-1b30fee2c0a0-kube-api-access-7kkk9\") pod \"bb04ce0c-726f-4b27-b588-1b30fee2c0a0\" (UID: \"bb04ce0c-726f-4b27-b588-1b30fee2c0a0\") " Oct 31 05:05:49.281114 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:49.281092 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb04ce0c-726f-4b27-b588-1b30fee2c0a0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb04ce0c-726f-4b27-b588-1b30fee2c0a0-plugins-config-volume\") pod \"bb04ce0c-726f-4b27-b588-1b30fee2c0a0\" (UID: \"bb04ce0c-726f-4b27-b588-1b30fee2c0a0\") " Oct 31 05:05:49.281663 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:49.281615 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb04ce0c-726f-4b27-b588-1b30fee2c0a0-plugins-config-volume" pod "bb04ce0c-726f-4b27-b588-1b30fee2c0a0" (UID: "bb04ce0c-726f-4b27-b588-1b30fee2c0a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:05:49.283659 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:49.283628 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb04ce0c-726f-4b27-b588-1b30fee2c0a0-kube-api-access-7kkk9" pod "bb04ce0c-726f-4b27-b588-1b30fee2c0a0" (UID: "bb04ce0c-726f-4b27-b588-1b30fee2c0a0"). InnerVolumeSpecName "kube-api-access-7kkk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:05:49.382858 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:49.382769 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb04ce0c-726f-4b27-b588-1b30fee2c0a0-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:49.382858 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:49.382806 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kkk9\" (UniqueName: \"kubernetes.io/projected/bb04ce0c-726f-4b27-b588-1b30fee2c0a0-kube-api-access-7kkk9\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:49.846263 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:49.846228 5019 generic.go:356] "Generic (PLEG): container finished" podID="bb04ce0c-726f-4b27-b588-1b30fee2c0a0" containerID="b731bdc617d726b85ed959ae0e70fc1aef8a110f5126aa663322b09e4ff8dc5d" exitCode=0 Oct 31 05:05:49.846478 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:49.846278 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-04d111-f77b9866c-2mzsk" Oct 31 05:05:49.846478 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:49.846313 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-04d111-f77b9866c-2mzsk" event={"ID":"bb04ce0c-726f-4b27-b588-1b30fee2c0a0","Type":"ContainerDied","Data":"b731bdc617d726b85ed959ae0e70fc1aef8a110f5126aa663322b09e4ff8dc5d"} Oct 31 05:05:49.846478 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:49.846403 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-04d111-f77b9866c-2mzsk" event={"ID":"bb04ce0c-726f-4b27-b588-1b30fee2c0a0","Type":"ContainerDied","Data":"1c79ac1a300ef7f9df9c0217585000226009057685fdb6ff6dd95809ddcadf13"} Oct 31 05:05:49.846478 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:49.846436 5019 scope.go:122] "RemoveContainer" containerID="b731bdc617d726b85ed959ae0e70fc1aef8a110f5126aa663322b09e4ff8dc5d" Oct 31 05:05:49.849883 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:49.849856 5019 generic.go:356] "Generic (PLEG): container finished" podID="1d46e1a8-ef71-4bbf-afec-a074d0fcfe05" containerID="36edf30386af908410f4e4c029a008a267547565f3f9e126b1fc8054065d98b1" exitCode=0 Oct 31 05:05:49.850117 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:49.850098 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl" event={"ID":"1d46e1a8-ef71-4bbf-afec-a074d0fcfe05","Type":"ContainerDied","Data":"36edf30386af908410f4e4c029a008a267547565f3f9e126b1fc8054065d98b1"} Oct 31 05:05:49.862040 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:49.862017 5019 scope.go:122] "RemoveContainer" containerID="b731bdc617d726b85ed959ae0e70fc1aef8a110f5126aa663322b09e4ff8dc5d" Oct 31 05:05:49.862445 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:05:49.862299 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b731bdc617d726b85ed959ae0e70fc1aef8a110f5126aa663322b09e4ff8dc5d\": container with ID starting with b731bdc617d726b85ed959ae0e70fc1aef8a110f5126aa663322b09e4ff8dc5d not found: ID does not exist" containerID="b731bdc617d726b85ed959ae0e70fc1aef8a110f5126aa663322b09e4ff8dc5d" Oct 31 05:05:49.862445 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:49.862394 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b731bdc617d726b85ed959ae0e70fc1aef8a110f5126aa663322b09e4ff8dc5d"} err="failed to get container status \"b731bdc617d726b85ed959ae0e70fc1aef8a110f5126aa663322b09e4ff8dc5d\": rpc error: code = NotFound desc = could not find container \"b731bdc617d726b85ed959ae0e70fc1aef8a110f5126aa663322b09e4ff8dc5d\": container with ID starting with b731bdc617d726b85ed959ae0e70fc1aef8a110f5126aa663322b09e4ff8dc5d not found: ID does not exist" Oct 31 05:05:49.862445 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:49.862422 5019 scope.go:122] "RemoveContainer" containerID="91b8e9f8d818fa4ee6637299edea1421c8b1ccf53c85215faee87a40c7e0d95e" Oct 31 05:05:49.866197 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:49.866174 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl" Oct 31 05:05:49.911154 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:49.911118 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-04d111-f77b9866c-2mzsk"] Oct 31 05:05:49.922854 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:49.922821 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-04d111-f77b9866c-2mzsk"] Oct 31 05:05:49.995006 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:49.994960 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-workload-certs\") pod \"1d46e1a8-ef71-4bbf-afec-a074d0fcfe05\" (UID: \"1d46e1a8-ef71-4bbf-afec-a074d0fcfe05\") " Oct 31 05:05:49.995292 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:49.995267 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-istio-envoy\") pod \"1d46e1a8-ef71-4bbf-afec-a074d0fcfe05\" (UID: \"1d46e1a8-ef71-4bbf-afec-a074d0fcfe05\") " Oct 31 05:05:49.995602 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:49.995581 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-istiod-ca-cert\") pod \"1d46e1a8-ef71-4bbf-afec-a074d0fcfe05\" (UID: \"1d46e1a8-ef71-4bbf-afec-a074d0fcfe05\") " Oct 31 05:05:49.995801 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:49.995782 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-istio-data\") pod \"1d46e1a8-ef71-4bbf-afec-a074d0fcfe05\" (UID: \"1d46e1a8-ef71-4bbf-afec-a074d0fcfe05\") " Oct 31 05:05:49.995942 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:49.995925 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-istio-token\" (UniqueName: \"kubernetes.io/projected/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-istio-token\") pod \"1d46e1a8-ef71-4bbf-afec-a074d0fcfe05\" (UID: \"1d46e1a8-ef71-4bbf-afec-a074d0fcfe05\") " Oct 31 05:05:49.996097 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:49.996080 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-istio-podinfo\") pod \"1d46e1a8-ef71-4bbf-afec-a074d0fcfe05\" (UID: \"1d46e1a8-ef71-4bbf-afec-a074d0fcfe05\") " Oct 31 05:05:49.996313 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:49.996258 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-kube-api-access-7qslv\" (UniqueName: \"kubernetes.io/projected/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-kube-api-access-7qslv\") pod \"1d46e1a8-ef71-4bbf-afec-a074d0fcfe05\" (UID: \"1d46e1a8-ef71-4bbf-afec-a074d0fcfe05\") " Oct 31 05:05:49.996568 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:49.996545 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-workload-socket\") pod \"1d46e1a8-ef71-4bbf-afec-a074d0fcfe05\" (UID: \"1d46e1a8-ef71-4bbf-afec-a074d0fcfe05\") " Oct 31 05:05:49.996710 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:49.996693 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-credential-socket\") pod \"1d46e1a8-ef71-4bbf-afec-a074d0fcfe05\" (UID: \"1d46e1a8-ef71-4bbf-afec-a074d0fcfe05\") " Oct 31 05:05:49.998112 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:49.998089 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-credential-socket" pod "1d46e1a8-ef71-4bbf-afec-a074d0fcfe05" (UID: "1d46e1a8-ef71-4bbf-afec-a074d0fcfe05"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:05:49.998501 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:49.998480 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-workload-certs" pod "1d46e1a8-ef71-4bbf-afec-a074d0fcfe05" (UID: "1d46e1a8-ef71-4bbf-afec-a074d0fcfe05"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:05:50.001286 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:50.001246 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-istiod-ca-cert" pod "1d46e1a8-ef71-4bbf-afec-a074d0fcfe05" (UID: "1d46e1a8-ef71-4bbf-afec-a074d0fcfe05"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:05:50.001574 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:50.001532 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-workload-socket" pod "1d46e1a8-ef71-4bbf-afec-a074d0fcfe05" (UID: "1d46e1a8-ef71-4bbf-afec-a074d0fcfe05"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:05:50.001737 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:50.001590 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-istio-data" pod "1d46e1a8-ef71-4bbf-afec-a074d0fcfe05" (UID: "1d46e1a8-ef71-4bbf-afec-a074d0fcfe05"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:05:50.022074 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:50.020979 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-istio-podinfo" pod "1d46e1a8-ef71-4bbf-afec-a074d0fcfe05" (UID: "1d46e1a8-ef71-4bbf-afec-a074d0fcfe05"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:05:50.022074 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:50.021269 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-istio-token" pod "1d46e1a8-ef71-4bbf-afec-a074d0fcfe05" (UID: "1d46e1a8-ef71-4bbf-afec-a074d0fcfe05"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:05:50.022578 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:50.022548 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-kube-api-access-7qslv" pod "1d46e1a8-ef71-4bbf-afec-a074d0fcfe05" (UID: "1d46e1a8-ef71-4bbf-afec-a074d0fcfe05"). InnerVolumeSpecName "kube-api-access-7qslv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:05:50.022649 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:50.022602 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-istio-envoy" pod "1d46e1a8-ef71-4bbf-afec-a074d0fcfe05" (UID: "1d46e1a8-ef71-4bbf-afec-a074d0fcfe05"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:05:50.099830 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:50.099786 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:50.099830 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:50.099823 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:50.099830 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:50.099833 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:50.099830 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:50.099844 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:50.100213 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:50.099852 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:50.100213 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:50.099860 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:50.100213 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:50.099868 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qslv\" (UniqueName: \"kubernetes.io/projected/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-kube-api-access-7qslv\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:50.100213 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:50.099880 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:50.100213 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:50.099889 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:50.861648 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:50.861608 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl" Oct 31 05:05:50.861648 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:50.861622 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl" event={"ID":"1d46e1a8-ef71-4bbf-afec-a074d0fcfe05","Type":"ContainerDied","Data":"3edb970d19488826c187d891451b3ae8802c13c366b054fd67f3a308345822f4"} Oct 31 05:05:50.861913 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:50.861687 5019 scope.go:122] "RemoveContainer" containerID="36edf30386af908410f4e4c029a008a267547565f3f9e126b1fc8054065d98b1" Oct 31 05:05:50.960593 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:50.960555 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl"] Oct 31 05:05:50.966023 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:50.965994 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-171edd-openshift-default-78cdf84cf9-8rtzl"] Oct 31 05:05:51.414034 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:51.413990 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d46e1a8-ef71-4bbf-afec-a074d0fcfe05" path="/var/lib/kubelet/pods/1d46e1a8-ef71-4bbf-afec-a074d0fcfe05/volumes" Oct 31 05:05:51.414690 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:51.414666 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb04ce0c-726f-4b27-b588-1b30fee2c0a0" path="/var/lib/kubelet/pods/bb04ce0c-726f-4b27-b588-1b30fee2c0a0/volumes" Oct 31 05:05:52.024358 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:52.022879 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-c1fe26/test-model-c1fe26-kserve-7fd8b8b8f-qkbvb" podUID="35fb8fb1-3a58-465b-b53a-08fe806af299" containerName="main" containerID="cri-o://0284ca77e086b05c3227ff3e59342c13dd43454270a68826bbf8c610ad09ac50" gracePeriod=45 Oct 31 05:05:52.509387 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:05:52.507983 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:05:52.579114 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:52.579075 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c1fe26/test-model-c1fe26-kserve-7fd8b8b8f-qkbvb" Oct 31 05:05:52.732779 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:52.732750 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35fb8fb1-3a58-465b-b53a-08fe806af299-tls-certs\" (UniqueName: \"kubernetes.io/secret/35fb8fb1-3a58-465b-b53a-08fe806af299-tls-certs\") pod \"35fb8fb1-3a58-465b-b53a-08fe806af299\" (UID: \"35fb8fb1-3a58-465b-b53a-08fe806af299\") " Oct 31 05:05:52.732985 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:52.732872 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35fb8fb1-3a58-465b-b53a-08fe806af299-home\" (UniqueName: \"kubernetes.io/empty-dir/35fb8fb1-3a58-465b-b53a-08fe806af299-home\") pod \"35fb8fb1-3a58-465b-b53a-08fe806af299\" (UID: \"35fb8fb1-3a58-465b-b53a-08fe806af299\") " Oct 31 05:05:52.733063 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:52.733009 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35fb8fb1-3a58-465b-b53a-08fe806af299-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35fb8fb1-3a58-465b-b53a-08fe806af299-tmp-dir\") pod \"35fb8fb1-3a58-465b-b53a-08fe806af299\" (UID: \"35fb8fb1-3a58-465b-b53a-08fe806af299\") " Oct 31 05:05:52.733140 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:52.733068 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35fb8fb1-3a58-465b-b53a-08fe806af299-kube-api-access-l8g6t\" (UniqueName: \"kubernetes.io/projected/35fb8fb1-3a58-465b-b53a-08fe806af299-kube-api-access-l8g6t\") pod \"35fb8fb1-3a58-465b-b53a-08fe806af299\" (UID: \"35fb8fb1-3a58-465b-b53a-08fe806af299\") " Oct 31 05:05:52.733140 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:52.733130 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35fb8fb1-3a58-465b-b53a-08fe806af299-dshm\" (UniqueName: \"kubernetes.io/empty-dir/35fb8fb1-3a58-465b-b53a-08fe806af299-dshm\") pod \"35fb8fb1-3a58-465b-b53a-08fe806af299\" (UID: \"35fb8fb1-3a58-465b-b53a-08fe806af299\") " Oct 31 05:05:52.733255 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:52.733179 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35fb8fb1-3a58-465b-b53a-08fe806af299-home" pod "35fb8fb1-3a58-465b-b53a-08fe806af299" (UID: "35fb8fb1-3a58-465b-b53a-08fe806af299"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:05:52.733255 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:52.733199 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35fb8fb1-3a58-465b-b53a-08fe806af299-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35fb8fb1-3a58-465b-b53a-08fe806af299-model-cache\") pod \"35fb8fb1-3a58-465b-b53a-08fe806af299\" (UID: \"35fb8fb1-3a58-465b-b53a-08fe806af299\") " Oct 31 05:05:52.734002 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:52.733900 5019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35fb8fb1-3a58-465b-b53a-08fe806af299-home\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:52.734002 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:52.733931 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35fb8fb1-3a58-465b-b53a-08fe806af299-model-cache" pod "35fb8fb1-3a58-465b-b53a-08fe806af299" (UID: "35fb8fb1-3a58-465b-b53a-08fe806af299"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:05:52.734002 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:52.733980 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35fb8fb1-3a58-465b-b53a-08fe806af299-tmp-dir" pod "35fb8fb1-3a58-465b-b53a-08fe806af299" (UID: "35fb8fb1-3a58-465b-b53a-08fe806af299"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:05:52.735692 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:52.735665 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35fb8fb1-3a58-465b-b53a-08fe806af299-kube-api-access-l8g6t" pod "35fb8fb1-3a58-465b-b53a-08fe806af299" (UID: "35fb8fb1-3a58-465b-b53a-08fe806af299"). InnerVolumeSpecName "kube-api-access-l8g6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:05:52.735798 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:52.735713 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35fb8fb1-3a58-465b-b53a-08fe806af299-dshm" pod "35fb8fb1-3a58-465b-b53a-08fe806af299" (UID: "35fb8fb1-3a58-465b-b53a-08fe806af299"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:05:52.735798 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:52.735730 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35fb8fb1-3a58-465b-b53a-08fe806af299-tls-certs" pod "35fb8fb1-3a58-465b-b53a-08fe806af299" (UID: "35fb8fb1-3a58-465b-b53a-08fe806af299"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:05:52.835396 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:52.835293 5019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35fb8fb1-3a58-465b-b53a-08fe806af299-tls-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:52.835396 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:52.835323 5019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35fb8fb1-3a58-465b-b53a-08fe806af299-tmp-dir\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:52.835396 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:52.835351 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8g6t\" (UniqueName: \"kubernetes.io/projected/35fb8fb1-3a58-465b-b53a-08fe806af299-kube-api-access-l8g6t\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:52.835396 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:52.835364 5019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35fb8fb1-3a58-465b-b53a-08fe806af299-dshm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:52.835396 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:52.835379 5019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35fb8fb1-3a58-465b-b53a-08fe806af299-model-cache\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:52.929653 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:52.929622 5019 generic.go:356] "Generic (PLEG): container finished" podID="35fb8fb1-3a58-465b-b53a-08fe806af299" containerID="0284ca77e086b05c3227ff3e59342c13dd43454270a68826bbf8c610ad09ac50" exitCode=0 Oct 31 05:05:52.929838 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:52.929665 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c1fe26/test-model-c1fe26-kserve-7fd8b8b8f-qkbvb" Oct 31 05:05:52.929838 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:52.929701 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c1fe26/test-model-c1fe26-kserve-7fd8b8b8f-qkbvb" event={"ID":"35fb8fb1-3a58-465b-b53a-08fe806af299","Type":"ContainerDied","Data":"0284ca77e086b05c3227ff3e59342c13dd43454270a68826bbf8c610ad09ac50"} Oct 31 05:05:52.929838 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:52.929746 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c1fe26/test-model-c1fe26-kserve-7fd8b8b8f-qkbvb" event={"ID":"35fb8fb1-3a58-465b-b53a-08fe806af299","Type":"ContainerDied","Data":"de95869a7cacc2a47d8c1132c34eccca87814839550e486835b9ab597761dcab"} Oct 31 05:05:52.929838 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:52.929770 5019 scope.go:122] "RemoveContainer" containerID="0284ca77e086b05c3227ff3e59342c13dd43454270a68826bbf8c610ad09ac50" Oct 31 05:05:52.946068 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:52.946046 5019 scope.go:122] "RemoveContainer" containerID="0284ca77e086b05c3227ff3e59342c13dd43454270a68826bbf8c610ad09ac50" Oct 31 05:05:52.946369 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:05:52.946341 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0284ca77e086b05c3227ff3e59342c13dd43454270a68826bbf8c610ad09ac50\": container with ID starting with 0284ca77e086b05c3227ff3e59342c13dd43454270a68826bbf8c610ad09ac50 not found: ID does not exist" containerID="0284ca77e086b05c3227ff3e59342c13dd43454270a68826bbf8c610ad09ac50" Oct 31 05:05:52.946478 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:52.946376 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0284ca77e086b05c3227ff3e59342c13dd43454270a68826bbf8c610ad09ac50"} err="failed to get container status \"0284ca77e086b05c3227ff3e59342c13dd43454270a68826bbf8c610ad09ac50\": rpc error: code = NotFound desc = could not find container \"0284ca77e086b05c3227ff3e59342c13dd43454270a68826bbf8c610ad09ac50\": container with ID starting with 0284ca77e086b05c3227ff3e59342c13dd43454270a68826bbf8c610ad09ac50 not found: ID does not exist" Oct 31 05:05:52.986632 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:52.986596 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c1fe26/test-model-c1fe26-kserve-7fd8b8b8f-qkbvb"] Oct 31 05:05:52.994387 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:52.994362 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-c1fe26/test-model-c1fe26-kserve-7fd8b8b8f-qkbvb"] Oct 31 05:05:53.079354 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:53.074424 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04d111-76d4bc867-jmfjn" podUID="4fb383ce-a4bd-4455-b351-34bc277bc3e5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:05:53.192447 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:53.188023 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-108ad8-7495778969-49t8z"] Oct 31 05:05:53.236447 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:53.235706 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-108ad8-8d8c4c667-rbd9x"] Oct 31 05:05:53.413435 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:53.413395 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35fb8fb1-3a58-465b-b53a-08fe806af299" path="/var/lib/kubelet/pods/35fb8fb1-3a58-465b-b53a-08fe806af299/volumes" Oct 31 05:05:58.188885 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:58.188833 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-108ad8-7495778969-49t8z" podUID="9abd61e1-2bf2-43fb-b106-132dedc0e2d9" containerName="payload-processing" containerID="cri-o://78b332e04241672bbc31d92d029cb85006e662b9234a95dd0e550caec94fe215" gracePeriod=25 Oct 31 05:05:58.236722 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:58.236660 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-108ad8-8d8c4c667-rbd9x" podUID="4fe19836-7804-40c2-b436-c947e33e979a" containerName="payload-pre-processing" containerID="cri-o://64a2cc1fcd3b3a4b931f7646d4296059350aef5b872a7075400ad3bd73c6bfdb" gracePeriod=25 Oct 31 05:05:58.533761 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:58.533732 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-108ad8-7495778969-49t8z" Oct 31 05:05:58.632299 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:58.632261 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9abd61e1-2bf2-43fb-b106-132dedc0e2d9-kube-api-access-2shsr\" (UniqueName: \"kubernetes.io/projected/9abd61e1-2bf2-43fb-b106-132dedc0e2d9-kube-api-access-2shsr\") pod \"9abd61e1-2bf2-43fb-b106-132dedc0e2d9\" (UID: \"9abd61e1-2bf2-43fb-b106-132dedc0e2d9\") " Oct 31 05:05:58.632567 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:58.632362 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9abd61e1-2bf2-43fb-b106-132dedc0e2d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9abd61e1-2bf2-43fb-b106-132dedc0e2d9-plugins-config-volume\") pod \"9abd61e1-2bf2-43fb-b106-132dedc0e2d9\" (UID: \"9abd61e1-2bf2-43fb-b106-132dedc0e2d9\") " Oct 31 05:05:58.632890 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:58.632856 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9abd61e1-2bf2-43fb-b106-132dedc0e2d9-plugins-config-volume" pod "9abd61e1-2bf2-43fb-b106-132dedc0e2d9" (UID: "9abd61e1-2bf2-43fb-b106-132dedc0e2d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:05:58.633105 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:58.633080 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9abd61e1-2bf2-43fb-b106-132dedc0e2d9-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:58.641284 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:58.639867 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9abd61e1-2bf2-43fb-b106-132dedc0e2d9-kube-api-access-2shsr" pod "9abd61e1-2bf2-43fb-b106-132dedc0e2d9" (UID: "9abd61e1-2bf2-43fb-b106-132dedc0e2d9"). InnerVolumeSpecName "kube-api-access-2shsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:05:58.734942 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:58.734894 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2shsr\" (UniqueName: \"kubernetes.io/projected/9abd61e1-2bf2-43fb-b106-132dedc0e2d9-kube-api-access-2shsr\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:59.124285 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:59.124243 5019 generic.go:356] "Generic (PLEG): container finished" podID="9abd61e1-2bf2-43fb-b106-132dedc0e2d9" containerID="78b332e04241672bbc31d92d029cb85006e662b9234a95dd0e550caec94fe215" exitCode=0 Oct 31 05:05:59.124495 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:59.124306 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-108ad8-7495778969-49t8z" Oct 31 05:05:59.124495 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:59.124322 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-108ad8-7495778969-49t8z" event={"ID":"9abd61e1-2bf2-43fb-b106-132dedc0e2d9","Type":"ContainerDied","Data":"78b332e04241672bbc31d92d029cb85006e662b9234a95dd0e550caec94fe215"} Oct 31 05:05:59.124495 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:59.124404 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-108ad8-7495778969-49t8z" event={"ID":"9abd61e1-2bf2-43fb-b106-132dedc0e2d9","Type":"ContainerDied","Data":"ef4d98776be47b578114a142adb7f7dac5d2cc4eeaa3a2bb40f947908ba40345"} Oct 31 05:05:59.124495 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:59.124444 5019 scope.go:122] "RemoveContainer" containerID="78b332e04241672bbc31d92d029cb85006e662b9234a95dd0e550caec94fe215" Oct 31 05:05:59.137106 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:59.137085 5019 scope.go:122] "RemoveContainer" containerID="78b332e04241672bbc31d92d029cb85006e662b9234a95dd0e550caec94fe215" Oct 31 05:05:59.137358 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:05:59.137322 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78b332e04241672bbc31d92d029cb85006e662b9234a95dd0e550caec94fe215\": container with ID starting with 78b332e04241672bbc31d92d029cb85006e662b9234a95dd0e550caec94fe215 not found: ID does not exist" containerID="78b332e04241672bbc31d92d029cb85006e662b9234a95dd0e550caec94fe215" Oct 31 05:05:59.137447 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:59.137369 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78b332e04241672bbc31d92d029cb85006e662b9234a95dd0e550caec94fe215"} err="failed to get container status \"78b332e04241672bbc31d92d029cb85006e662b9234a95dd0e550caec94fe215\": rpc error: code = NotFound desc = could not find container \"78b332e04241672bbc31d92d029cb85006e662b9234a95dd0e550caec94fe215\": container with ID starting with 78b332e04241672bbc31d92d029cb85006e662b9234a95dd0e550caec94fe215 not found: ID does not exist" Oct 31 05:05:59.180628 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:59.180578 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-108ad8-7495778969-49t8z"] Oct 31 05:05:59.186370 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:59.186321 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-108ad8-7495778969-49t8z"] Oct 31 05:05:59.330666 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:59.330621 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-12c0bc/test-model-12c0bc-kserve-66b5ccb945-4sg7l" podUID="0c357c8c-ca12-43ba-9dfa-b9bfb0867e43" containerName="main" containerID="cri-o://2f4c4eecb0d628f8fd35fdb655d5620424307a007f06736f567f6d67646e5c72" gracePeriod=45 Oct 31 05:05:59.416208 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:59.416127 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9abd61e1-2bf2-43fb-b106-132dedc0e2d9" path="/var/lib/kubelet/pods/9abd61e1-2bf2-43fb-b106-132dedc0e2d9/volumes" Oct 31 05:05:59.640797 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:59.640766 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-12c0bc/test-model-12c0bc-kserve-66b5ccb945-4sg7l" Oct 31 05:05:59.751145 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:59.751108 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c357c8c-ca12-43ba-9dfa-b9bfb0867e43-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c357c8c-ca12-43ba-9dfa-b9bfb0867e43-dshm\") pod \"0c357c8c-ca12-43ba-9dfa-b9bfb0867e43\" (UID: \"0c357c8c-ca12-43ba-9dfa-b9bfb0867e43\") " Oct 31 05:05:59.751405 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:59.751163 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c357c8c-ca12-43ba-9dfa-b9bfb0867e43-kube-api-access-rw926\" (UniqueName: \"kubernetes.io/projected/0c357c8c-ca12-43ba-9dfa-b9bfb0867e43-kube-api-access-rw926\") pod \"0c357c8c-ca12-43ba-9dfa-b9bfb0867e43\" (UID: \"0c357c8c-ca12-43ba-9dfa-b9bfb0867e43\") " Oct 31 05:05:59.751405 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:59.751216 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c357c8c-ca12-43ba-9dfa-b9bfb0867e43-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c357c8c-ca12-43ba-9dfa-b9bfb0867e43-model-cache\") pod \"0c357c8c-ca12-43ba-9dfa-b9bfb0867e43\" (UID: \"0c357c8c-ca12-43ba-9dfa-b9bfb0867e43\") " Oct 31 05:05:59.751405 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:59.751297 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c357c8c-ca12-43ba-9dfa-b9bfb0867e43-home\" (UniqueName: \"kubernetes.io/empty-dir/0c357c8c-ca12-43ba-9dfa-b9bfb0867e43-home\") pod \"0c357c8c-ca12-43ba-9dfa-b9bfb0867e43\" (UID: \"0c357c8c-ca12-43ba-9dfa-b9bfb0867e43\") " Oct 31 05:05:59.751405 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:59.751377 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c357c8c-ca12-43ba-9dfa-b9bfb0867e43-tls-certs\" (UniqueName: \"kubernetes.io/secret/0c357c8c-ca12-43ba-9dfa-b9bfb0867e43-tls-certs\") pod \"0c357c8c-ca12-43ba-9dfa-b9bfb0867e43\" (UID: \"0c357c8c-ca12-43ba-9dfa-b9bfb0867e43\") " Oct 31 05:05:59.751684 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:59.751475 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c357c8c-ca12-43ba-9dfa-b9bfb0867e43-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c357c8c-ca12-43ba-9dfa-b9bfb0867e43-tmp-dir\") pod \"0c357c8c-ca12-43ba-9dfa-b9bfb0867e43\" (UID: \"0c357c8c-ca12-43ba-9dfa-b9bfb0867e43\") " Oct 31 05:05:59.751684 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:59.751582 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c357c8c-ca12-43ba-9dfa-b9bfb0867e43-model-cache" pod "0c357c8c-ca12-43ba-9dfa-b9bfb0867e43" (UID: "0c357c8c-ca12-43ba-9dfa-b9bfb0867e43"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:05:59.751684 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:59.751639 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c357c8c-ca12-43ba-9dfa-b9bfb0867e43-home" pod "0c357c8c-ca12-43ba-9dfa-b9bfb0867e43" (UID: "0c357c8c-ca12-43ba-9dfa-b9bfb0867e43"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:05:59.751861 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:59.751825 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c357c8c-ca12-43ba-9dfa-b9bfb0867e43-tmp-dir" pod "0c357c8c-ca12-43ba-9dfa-b9bfb0867e43" (UID: "0c357c8c-ca12-43ba-9dfa-b9bfb0867e43"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:05:59.752380 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:59.752361 5019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c357c8c-ca12-43ba-9dfa-b9bfb0867e43-model-cache\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:59.752380 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:59.752380 5019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c357c8c-ca12-43ba-9dfa-b9bfb0867e43-home\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:59.752565 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:59.752389 5019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c357c8c-ca12-43ba-9dfa-b9bfb0867e43-tmp-dir\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:59.753728 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:59.753703 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c357c8c-ca12-43ba-9dfa-b9bfb0867e43-tls-certs" pod "0c357c8c-ca12-43ba-9dfa-b9bfb0867e43" (UID: "0c357c8c-ca12-43ba-9dfa-b9bfb0867e43"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:05:59.753901 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:59.753877 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c357c8c-ca12-43ba-9dfa-b9bfb0867e43-dshm" pod "0c357c8c-ca12-43ba-9dfa-b9bfb0867e43" (UID: "0c357c8c-ca12-43ba-9dfa-b9bfb0867e43"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:05:59.753989 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:59.753905 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c357c8c-ca12-43ba-9dfa-b9bfb0867e43-kube-api-access-rw926" pod "0c357c8c-ca12-43ba-9dfa-b9bfb0867e43" (UID: "0c357c8c-ca12-43ba-9dfa-b9bfb0867e43"). InnerVolumeSpecName "kube-api-access-rw926". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:05:59.853394 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:59.853326 5019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c357c8c-ca12-43ba-9dfa-b9bfb0867e43-dshm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:59.853394 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:59.853395 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw926\" (UniqueName: \"kubernetes.io/projected/0c357c8c-ca12-43ba-9dfa-b9bfb0867e43-kube-api-access-rw926\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:05:59.853620 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:05:59.853406 5019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c357c8c-ca12-43ba-9dfa-b9bfb0867e43-tls-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:06:00.135224 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:00.135149 5019 generic.go:356] "Generic (PLEG): container finished" podID="0c357c8c-ca12-43ba-9dfa-b9bfb0867e43" containerID="2f4c4eecb0d628f8fd35fdb655d5620424307a007f06736f567f6d67646e5c72" exitCode=0 Oct 31 05:06:00.135224 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:00.135213 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-12c0bc/test-model-12c0bc-kserve-66b5ccb945-4sg7l" Oct 31 05:06:00.135449 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:00.135233 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-12c0bc/test-model-12c0bc-kserve-66b5ccb945-4sg7l" event={"ID":"0c357c8c-ca12-43ba-9dfa-b9bfb0867e43","Type":"ContainerDied","Data":"2f4c4eecb0d628f8fd35fdb655d5620424307a007f06736f567f6d67646e5c72"} Oct 31 05:06:00.135449 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:00.135280 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-12c0bc/test-model-12c0bc-kserve-66b5ccb945-4sg7l" event={"ID":"0c357c8c-ca12-43ba-9dfa-b9bfb0867e43","Type":"ContainerDied","Data":"a9447f08318d0d5d998891ded2a7c3580169d7c1802668d616f40e8184570c89"} Oct 31 05:06:00.135449 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:00.135299 5019 scope.go:122] "RemoveContainer" containerID="2f4c4eecb0d628f8fd35fdb655d5620424307a007f06736f567f6d67646e5c72" Oct 31 05:06:00.147551 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:00.147525 5019 scope.go:122] "RemoveContainer" containerID="2f4c4eecb0d628f8fd35fdb655d5620424307a007f06736f567f6d67646e5c72" Oct 31 05:06:00.147827 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:06:00.147808 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f4c4eecb0d628f8fd35fdb655d5620424307a007f06736f567f6d67646e5c72\": container with ID starting with 2f4c4eecb0d628f8fd35fdb655d5620424307a007f06736f567f6d67646e5c72 not found: ID does not exist" containerID="2f4c4eecb0d628f8fd35fdb655d5620424307a007f06736f567f6d67646e5c72" Oct 31 05:06:00.147897 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:00.147838 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f4c4eecb0d628f8fd35fdb655d5620424307a007f06736f567f6d67646e5c72"} err="failed to get container status \"2f4c4eecb0d628f8fd35fdb655d5620424307a007f06736f567f6d67646e5c72\": rpc error: code = NotFound desc = could not find container \"2f4c4eecb0d628f8fd35fdb655d5620424307a007f06736f567f6d67646e5c72\": container with ID starting with 2f4c4eecb0d628f8fd35fdb655d5620424307a007f06736f567f6d67646e5c72 not found: ID does not exist" Oct 31 05:06:00.191977 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:00.191947 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-12c0bc/test-model-12c0bc-kserve-66b5ccb945-4sg7l"] Oct 31 05:06:00.204396 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:00.204369 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-12c0bc/test-model-12c0bc-kserve-66b5ccb945-4sg7l"] Oct 31 05:06:01.411503 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:01.411464 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c357c8c-ca12-43ba-9dfa-b9bfb0867e43" path="/var/lib/kubelet/pods/0c357c8c-ca12-43ba-9dfa-b9bfb0867e43/volumes" Oct 31 05:06:02.689007 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:02.688971 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq"] Oct 31 05:06:02.690194 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:02.690165 5019 cpu_manager.go:414] "containerMap: removing container" podUID="9abd61e1-2bf2-43fb-b106-132dedc0e2d9" containerName="payload-processing" Oct 31 05:06:02.690194 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:02.690194 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9abd61e1-2bf2-43fb-b106-132dedc0e2d9" containerName="payload-processing" Oct 31 05:06:02.690422 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:02.690227 5019 cpu_manager.go:414] "containerMap: removing container" podUID="35fb8fb1-3a58-465b-b53a-08fe806af299" containerName="main" Oct 31 05:06:02.690422 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:02.690235 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35fb8fb1-3a58-465b-b53a-08fe806af299" containerName="main" Oct 31 05:06:02.690422 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:02.690255 5019 cpu_manager.go:414] "containerMap: removing container" podUID="bd7eece6-db42-4033-9f69-74bfd818bece" containerName="revoke-keys" Oct 31 05:06:02.690422 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:02.690264 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd7eece6-db42-4033-9f69-74bfd818bece" containerName="revoke-keys" Oct 31 05:06:02.690422 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:02.690283 5019 cpu_manager.go:414] "containerMap: removing container" podUID="0c357c8c-ca12-43ba-9dfa-b9bfb0867e43" containerName="main" Oct 31 05:06:02.690422 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:02.690291 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c357c8c-ca12-43ba-9dfa-b9bfb0867e43" containerName="main" Oct 31 05:06:02.690422 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:02.690368 5019 cpu_manager.go:414] "containerMap: removing container" podUID="1d46e1a8-ef71-4bbf-afec-a074d0fcfe05" containerName="istio-proxy" Oct 31 05:06:02.690422 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:02.690378 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d46e1a8-ef71-4bbf-afec-a074d0fcfe05" containerName="istio-proxy" Oct 31 05:06:02.690422 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:02.690393 5019 cpu_manager.go:414] "containerMap: removing container" podUID="1d46e1a8-ef71-4bbf-afec-a074d0fcfe05" containerName="istio-proxy" Oct 31 05:06:02.690422 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:02.690402 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d46e1a8-ef71-4bbf-afec-a074d0fcfe05" containerName="istio-proxy" Oct 31 05:06:02.690422 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:02.690423 5019 cpu_manager.go:414] "containerMap: removing container" podUID="f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7" containerName="main" Oct 31 05:06:02.691074 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:02.690431 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7" containerName="main" Oct 31 05:06:02.691074 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:02.690445 5019 cpu_manager.go:414] "containerMap: removing container" podUID="bb04ce0c-726f-4b27-b588-1b30fee2c0a0" containerName="payload-processing" Oct 31 05:06:02.691074 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:02.690454 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb04ce0c-726f-4b27-b588-1b30fee2c0a0" containerName="payload-processing" Oct 31 05:06:02.691074 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:02.690664 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="9abd61e1-2bf2-43fb-b106-132dedc0e2d9" containerName="payload-processing" Oct 31 05:06:02.691074 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:02.690681 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="35fb8fb1-3a58-465b-b53a-08fe806af299" containerName="main" Oct 31 05:06:02.691074 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:02.690692 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd7eece6-db42-4033-9f69-74bfd818bece" containerName="revoke-keys" Oct 31 05:06:02.691074 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:02.690703 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c357c8c-ca12-43ba-9dfa-b9bfb0867e43" containerName="main" Oct 31 05:06:02.691074 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:02.690719 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d46e1a8-ef71-4bbf-afec-a074d0fcfe05" containerName="istio-proxy" Oct 31 05:06:02.691074 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:02.690725 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d46e1a8-ef71-4bbf-afec-a074d0fcfe05" containerName="istio-proxy" Oct 31 05:06:02.691074 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:02.690732 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1d29fd3-0e84-4e08-8522-ba6ae2d79ef7" containerName="main" Oct 31 05:06:02.691074 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:02.690739 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb04ce0c-726f-4b27-b588-1b30fee2c0a0" containerName="payload-processing" Oct 31 05:06:02.696055 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:02.696034 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq" Oct 31 05:06:02.722740 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:02.722709 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq"] Oct 31 05:06:02.812887 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:02.812848 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf9a806-d96f-478b-9ce1-2e67ec594739-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pr6rq\" (UID: \"1cf9a806-d96f-478b-9ce1-2e67ec594739\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq" Oct 31 05:06:02.813071 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:02.812909 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cf9a806-d96f-478b-9ce1-2e67ec594739-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pr6rq\" (UID: \"1cf9a806-d96f-478b-9ce1-2e67ec594739\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq" Oct 31 05:06:02.813071 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:02.813044 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cf9a806-d96f-478b-9ce1-2e67ec594739-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pr6rq\" (UID: \"1cf9a806-d96f-478b-9ce1-2e67ec594739\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq" Oct 31 05:06:02.813235 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:02.813131 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cf9a806-d96f-478b-9ce1-2e67ec594739-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pr6rq\" (UID: \"1cf9a806-d96f-478b-9ce1-2e67ec594739\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq" Oct 31 05:06:02.813315 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:02.813290 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzvbb\" (UniqueName: \"kubernetes.io/projected/1cf9a806-d96f-478b-9ce1-2e67ec594739-kube-api-access-rzvbb\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pr6rq\" (UID: \"1cf9a806-d96f-478b-9ce1-2e67ec594739\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq" Oct 31 05:06:02.813493 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:02.813478 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cf9a806-d96f-478b-9ce1-2e67ec594739-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pr6rq\" (UID: \"1cf9a806-d96f-478b-9ce1-2e67ec594739\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq" Oct 31 05:06:02.813584 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:02.813569 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf9a806-d96f-478b-9ce1-2e67ec594739-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pr6rq\" (UID: \"1cf9a806-d96f-478b-9ce1-2e67ec594739\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq" Oct 31 05:06:02.813637 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:02.813622 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cf9a806-d96f-478b-9ce1-2e67ec594739-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pr6rq\" (UID: \"1cf9a806-d96f-478b-9ce1-2e67ec594739\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq" Oct 31 05:06:02.813675 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:02.813664 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cf9a806-d96f-478b-9ce1-2e67ec594739-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pr6rq\" (UID: \"1cf9a806-d96f-478b-9ce1-2e67ec594739\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq" Oct 31 05:06:02.914795 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:02.914760 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzvbb\" (UniqueName: \"kubernetes.io/projected/1cf9a806-d96f-478b-9ce1-2e67ec594739-kube-api-access-rzvbb\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pr6rq\" (UID: \"1cf9a806-d96f-478b-9ce1-2e67ec594739\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq" Oct 31 05:06:02.914973 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:02.914856 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cf9a806-d96f-478b-9ce1-2e67ec594739-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pr6rq\" (UID: \"1cf9a806-d96f-478b-9ce1-2e67ec594739\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq" Oct 31 05:06:02.914973 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:02.914938 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf9a806-d96f-478b-9ce1-2e67ec594739-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pr6rq\" (UID: \"1cf9a806-d96f-478b-9ce1-2e67ec594739\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq" Oct 31 05:06:02.915102 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:02.914976 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cf9a806-d96f-478b-9ce1-2e67ec594739-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pr6rq\" (UID: \"1cf9a806-d96f-478b-9ce1-2e67ec594739\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq" Oct 31 05:06:02.915102 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:02.915000 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cf9a806-d96f-478b-9ce1-2e67ec594739-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pr6rq\" (UID: \"1cf9a806-d96f-478b-9ce1-2e67ec594739\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq" Oct 31 05:06:02.915216 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:02.915101 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf9a806-d96f-478b-9ce1-2e67ec594739-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pr6rq\" (UID: \"1cf9a806-d96f-478b-9ce1-2e67ec594739\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq" Oct 31 05:06:02.915216 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:02.915135 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cf9a806-d96f-478b-9ce1-2e67ec594739-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pr6rq\" (UID: \"1cf9a806-d96f-478b-9ce1-2e67ec594739\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq" Oct 31 05:06:02.915216 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:02.915189 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cf9a806-d96f-478b-9ce1-2e67ec594739-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pr6rq\" (UID: \"1cf9a806-d96f-478b-9ce1-2e67ec594739\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq" Oct 31 05:06:02.915411 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:02.915230 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cf9a806-d96f-478b-9ce1-2e67ec594739-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pr6rq\" (UID: \"1cf9a806-d96f-478b-9ce1-2e67ec594739\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq" Oct 31 05:06:02.915411 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:02.915344 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cf9a806-d96f-478b-9ce1-2e67ec594739-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pr6rq\" (UID: \"1cf9a806-d96f-478b-9ce1-2e67ec594739\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq" Oct 31 05:06:02.915518 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:02.915424 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf9a806-d96f-478b-9ce1-2e67ec594739-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pr6rq\" (UID: \"1cf9a806-d96f-478b-9ce1-2e67ec594739\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq" Oct 31 05:06:02.915652 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:02.915629 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cf9a806-d96f-478b-9ce1-2e67ec594739-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pr6rq\" (UID: \"1cf9a806-d96f-478b-9ce1-2e67ec594739\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq" Oct 31 05:06:02.915726 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:02.915652 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf9a806-d96f-478b-9ce1-2e67ec594739-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pr6rq\" (UID: \"1cf9a806-d96f-478b-9ce1-2e67ec594739\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq" Oct 31 05:06:02.915918 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:02.915899 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cf9a806-d96f-478b-9ce1-2e67ec594739-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pr6rq\" (UID: \"1cf9a806-d96f-478b-9ce1-2e67ec594739\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq" Oct 31 05:06:02.917566 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:02.917538 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cf9a806-d96f-478b-9ce1-2e67ec594739-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pr6rq\" (UID: \"1cf9a806-d96f-478b-9ce1-2e67ec594739\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq" Oct 31 05:06:02.917745 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:02.917730 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cf9a806-d96f-478b-9ce1-2e67ec594739-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pr6rq\" (UID: \"1cf9a806-d96f-478b-9ce1-2e67ec594739\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq" Oct 31 05:06:02.927651 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:02.927614 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-108ad8-8d8c4c667-rbd9x" podUID="4fe19836-7804-40c2-b436-c947e33e979a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:06:02.937927 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:02.937904 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzvbb\" (UniqueName: \"kubernetes.io/projected/1cf9a806-d96f-478b-9ce1-2e67ec594739-kube-api-access-rzvbb\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pr6rq\" (UID: \"1cf9a806-d96f-478b-9ce1-2e67ec594739\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq" Oct 31 05:06:02.938035 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:02.937909 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cf9a806-d96f-478b-9ce1-2e67ec594739-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pr6rq\" (UID: \"1cf9a806-d96f-478b-9ce1-2e67ec594739\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq" Oct 31 05:06:03.019246 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:03.014776 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq" Oct 31 05:06:03.097563 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:03.096453 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04d111-76d4bc867-jmfjn" podUID="4fb383ce-a4bd-4455-b351-34bc277bc3e5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:06:03.517435 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:03.517400 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq"] Oct 31 05:06:03.518070 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:06:03.518044 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cf9a806_d96f_478b_9ce1_2e67ec594739.slice/crio-4ddb6b032c18ca1b4b3b2f3638fff120bfa8e2dbb708c54eaccfda6d9e660498 WatchSource:0}: Error finding container 4ddb6b032c18ca1b4b3b2f3638fff120bfa8e2dbb708c54eaccfda6d9e660498: Status 404 returned error can't find the container with id 4ddb6b032c18ca1b4b3b2f3638fff120bfa8e2dbb708c54eaccfda6d9e660498 Oct 31 05:06:03.522125 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:03.522088 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:06:03.522248 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:03.522169 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:06:03.522248 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:03.522216 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:06:04.211718 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:04.211672 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq" event={"ID":"1cf9a806-d96f-478b-9ce1-2e67ec594739","Type":"ContainerStarted","Data":"1622cd76a3f47c2dc335a11e6857175d67f351df07c11d04fab8ce562097102b"} Oct 31 05:06:04.212247 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:04.211778 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq" event={"ID":"1cf9a806-d96f-478b-9ce1-2e67ec594739","Type":"ContainerStarted","Data":"4ddb6b032c18ca1b4b3b2f3638fff120bfa8e2dbb708c54eaccfda6d9e660498"} Oct 31 05:06:04.282423 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:04.282344 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq" podStartSLOduration=2.282306265 podStartE2EDuration="2.282306265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:06:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:06:04.267742896 +0000 UTC m=+3719.024080937" watchObservedRunningTime="2026-10-31 05:06:04.282306265 +0000 UTC m=+3719.038644306" Oct 31 05:06:05.016283 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:05.016237 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq" Oct 31 05:06:05.018302 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:05.018275 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.170:15021/healthz/ready\": dial tcp 10.132.1.170:15021: connect: connection refused" start-of-body= Oct 31 05:06:05.018438 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:05.018343 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq" podUID="1cf9a806-d96f-478b-9ce1-2e67ec594739" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.170:15021/healthz/ready\": dial tcp 10.132.1.170:15021: connect: connection refused" Oct 31 05:06:06.015760 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:06.015718 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.170:15021/healthz/ready\": dial tcp 10.132.1.170:15021: connect: connection refused" start-of-body= Oct 31 05:06:06.016259 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:06.015780 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq" podUID="1cf9a806-d96f-478b-9ce1-2e67ec594739" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.170:15021/healthz/ready\": dial tcp 10.132.1.170:15021: connect: connection refused" Oct 31 05:06:07.016831 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:07.016789 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.170:15021/healthz/ready\": dial tcp 10.132.1.170:15021: connect: connection refused" start-of-body= Oct 31 05:06:07.017522 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:07.017483 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq" podUID="1cf9a806-d96f-478b-9ce1-2e67ec594739" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.170:15021/healthz/ready\": dial tcp 10.132.1.170:15021: connect: connection refused" Oct 31 05:06:08.015846 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:08.015802 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.170:15021/healthz/ready\": dial tcp 10.132.1.170:15021: connect: connection refused" start-of-body= Oct 31 05:06:08.016052 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:08.015897 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq" podUID="1cf9a806-d96f-478b-9ce1-2e67ec594739" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.170:15021/healthz/ready\": dial tcp 10.132.1.170:15021: connect: connection refused" Oct 31 05:06:09.015742 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:09.015696 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.170:15021/healthz/ready\": dial tcp 10.132.1.170:15021: connect: connection refused" start-of-body= Oct 31 05:06:09.016202 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:09.015771 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq" podUID="1cf9a806-d96f-478b-9ce1-2e67ec594739" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.170:15021/healthz/ready\": dial tcp 10.132.1.170:15021: connect: connection refused" Oct 31 05:06:09.314135 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:09.314099 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04d111-76d4bc867-jmfjn" Oct 31 05:06:09.330446 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:09.330412 5019 generic.go:356] "Generic (PLEG): container finished" podID="4fb383ce-a4bd-4455-b351-34bc277bc3e5" containerID="4ec33c2dfccb7259bd251ec4c7e91629209092e19f7c976acd8f583a18cffb17" exitCode=0 Oct 31 05:06:09.330647 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:09.330460 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04d111-76d4bc867-jmfjn" Oct 31 05:06:09.330647 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:09.330497 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04d111-76d4bc867-jmfjn" event={"ID":"4fb383ce-a4bd-4455-b351-34bc277bc3e5","Type":"ContainerDied","Data":"4ec33c2dfccb7259bd251ec4c7e91629209092e19f7c976acd8f583a18cffb17"} Oct 31 05:06:09.330723 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:09.330655 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04d111-76d4bc867-jmfjn" event={"ID":"4fb383ce-a4bd-4455-b351-34bc277bc3e5","Type":"ContainerDied","Data":"2fa4f4e3ab947dfc5f74bf8398e6c94cf1449222faa51fd4597a8d1ab88fb03d"} Oct 31 05:06:09.330723 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:09.330688 5019 scope.go:122] "RemoveContainer" containerID="4ec33c2dfccb7259bd251ec4c7e91629209092e19f7c976acd8f583a18cffb17" Oct 31 05:06:09.337695 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:09.337662 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fb383ce-a4bd-4455-b351-34bc277bc3e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fb383ce-a4bd-4455-b351-34bc277bc3e5-plugins-config-volume\") pod \"4fb383ce-a4bd-4455-b351-34bc277bc3e5\" (UID: \"4fb383ce-a4bd-4455-b351-34bc277bc3e5\") " Oct 31 05:06:09.337854 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:09.337787 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fb383ce-a4bd-4455-b351-34bc277bc3e5-kube-api-access-clh9s\" (UniqueName: \"kubernetes.io/projected/4fb383ce-a4bd-4455-b351-34bc277bc3e5-kube-api-access-clh9s\") pod \"4fb383ce-a4bd-4455-b351-34bc277bc3e5\" (UID: \"4fb383ce-a4bd-4455-b351-34bc277bc3e5\") " Oct 31 05:06:09.338146 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:09.338119 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fb383ce-a4bd-4455-b351-34bc277bc3e5-plugins-config-volume" pod "4fb383ce-a4bd-4455-b351-34bc277bc3e5" (UID: "4fb383ce-a4bd-4455-b351-34bc277bc3e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:06:09.338639 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:09.338614 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fb383ce-a4bd-4455-b351-34bc277bc3e5-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:06:09.340224 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:09.340194 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fb383ce-a4bd-4455-b351-34bc277bc3e5-kube-api-access-clh9s" pod "4fb383ce-a4bd-4455-b351-34bc277bc3e5" (UID: "4fb383ce-a4bd-4455-b351-34bc277bc3e5"). InnerVolumeSpecName "kube-api-access-clh9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:06:09.343821 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:09.343790 5019 scope.go:122] "RemoveContainer" containerID="4ec33c2dfccb7259bd251ec4c7e91629209092e19f7c976acd8f583a18cffb17" Oct 31 05:06:09.344188 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:06:09.344163 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ec33c2dfccb7259bd251ec4c7e91629209092e19f7c976acd8f583a18cffb17\": container with ID starting with 4ec33c2dfccb7259bd251ec4c7e91629209092e19f7c976acd8f583a18cffb17 not found: ID does not exist" containerID="4ec33c2dfccb7259bd251ec4c7e91629209092e19f7c976acd8f583a18cffb17" Oct 31 05:06:09.344247 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:09.344202 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ec33c2dfccb7259bd251ec4c7e91629209092e19f7c976acd8f583a18cffb17"} err="failed to get container status \"4ec33c2dfccb7259bd251ec4c7e91629209092e19f7c976acd8f583a18cffb17\": rpc error: code = NotFound desc = could not find container \"4ec33c2dfccb7259bd251ec4c7e91629209092e19f7c976acd8f583a18cffb17\": container with ID starting with 4ec33c2dfccb7259bd251ec4c7e91629209092e19f7c976acd8f583a18cffb17 not found: ID does not exist" Oct 31 05:06:09.440216 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:09.440177 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clh9s\" (UniqueName: \"kubernetes.io/projected/4fb383ce-a4bd-4455-b351-34bc277bc3e5-kube-api-access-clh9s\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:06:09.693419 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:09.692581 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-04d111-76d4bc867-jmfjn"] Oct 31 05:06:09.698094 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:09.698046 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-04d111-76d4bc867-jmfjn"] Oct 31 05:06:10.015969 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:10.015923 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.170:15021/healthz/ready\": dial tcp 10.132.1.170:15021: connect: connection refused" start-of-body= Oct 31 05:06:10.016461 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:10.015984 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq" podUID="1cf9a806-d96f-478b-9ce1-2e67ec594739" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.170:15021/healthz/ready\": dial tcp 10.132.1.170:15021: connect: connection refused" Oct 31 05:06:11.015768 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:11.015717 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.170:15021/healthz/ready\": dial tcp 10.132.1.170:15021: connect: connection refused" start-of-body= Oct 31 05:06:11.016006 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:11.015789 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq" podUID="1cf9a806-d96f-478b-9ce1-2e67ec594739" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.170:15021/healthz/ready\": dial tcp 10.132.1.170:15021: connect: connection refused" Oct 31 05:06:11.413745 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:11.413622 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fb383ce-a4bd-4455-b351-34bc277bc3e5" path="/var/lib/kubelet/pods/4fb383ce-a4bd-4455-b351-34bc277bc3e5/volumes" Oct 31 05:06:12.368817 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:12.368779 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq" Oct 31 05:06:12.370111 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:12.369966 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq" Oct 31 05:06:12.371201 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:12.371173 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq" Oct 31 05:06:12.927597 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:12.927552 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-108ad8-8d8c4c667-rbd9x" podUID="4fe19836-7804-40c2-b436-c947e33e979a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:06:17.732109 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:17.732016 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rdnft"] Oct 31 05:06:17.732908 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:17.732889 5019 cpu_manager.go:414] "containerMap: removing container" podUID="4fb383ce-a4bd-4455-b351-34bc277bc3e5" containerName="payload-pre-processing" Oct 31 05:06:17.732977 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:17.732908 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fb383ce-a4bd-4455-b351-34bc277bc3e5" containerName="payload-pre-processing" Oct 31 05:06:17.733116 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:17.733102 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fb383ce-a4bd-4455-b351-34bc277bc3e5" containerName="payload-pre-processing" Oct 31 05:06:17.739142 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:17.739114 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rdnft" Oct 31 05:06:17.778252 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:17.777836 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rdnft"] Oct 31 05:06:17.821801 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:17.821750 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rdnft\" (UID: \"5fc2a87b-394a-4c26-adfd-9a57a1fc463b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rdnft" Oct 31 05:06:17.822034 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:17.821842 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5kx5\" (UniqueName: \"kubernetes.io/projected/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-kube-api-access-d5kx5\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rdnft\" (UID: \"5fc2a87b-394a-4c26-adfd-9a57a1fc463b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rdnft" Oct 31 05:06:17.822034 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:17.821900 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rdnft\" (UID: \"5fc2a87b-394a-4c26-adfd-9a57a1fc463b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rdnft" Oct 31 05:06:17.822147 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:17.822016 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rdnft\" (UID: \"5fc2a87b-394a-4c26-adfd-9a57a1fc463b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rdnft" Oct 31 05:06:17.822300 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:17.822277 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rdnft\" (UID: \"5fc2a87b-394a-4c26-adfd-9a57a1fc463b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rdnft" Oct 31 05:06:17.822436 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:17.822310 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rdnft\" (UID: \"5fc2a87b-394a-4c26-adfd-9a57a1fc463b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rdnft" Oct 31 05:06:17.822436 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:17.822406 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rdnft\" (UID: \"5fc2a87b-394a-4c26-adfd-9a57a1fc463b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rdnft" Oct 31 05:06:17.822556 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:17.822446 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rdnft\" (UID: \"5fc2a87b-394a-4c26-adfd-9a57a1fc463b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rdnft" Oct 31 05:06:17.822634 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:17.822610 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rdnft\" (UID: \"5fc2a87b-394a-4c26-adfd-9a57a1fc463b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rdnft" Oct 31 05:06:17.924040 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:17.923995 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rdnft\" (UID: \"5fc2a87b-394a-4c26-adfd-9a57a1fc463b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rdnft" Oct 31 05:06:17.924040 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:17.924041 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rdnft\" (UID: \"5fc2a87b-394a-4c26-adfd-9a57a1fc463b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rdnft" Oct 31 05:06:17.924310 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:17.924096 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rdnft\" (UID: \"5fc2a87b-394a-4c26-adfd-9a57a1fc463b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rdnft" Oct 31 05:06:17.924310 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:17.924117 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rdnft\" (UID: \"5fc2a87b-394a-4c26-adfd-9a57a1fc463b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rdnft" Oct 31 05:06:17.924310 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:17.924176 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rdnft\" (UID: \"5fc2a87b-394a-4c26-adfd-9a57a1fc463b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rdnft" Oct 31 05:06:17.924310 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:17.924233 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rdnft\" (UID: \"5fc2a87b-394a-4c26-adfd-9a57a1fc463b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rdnft" Oct 31 05:06:17.924310 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:17.924283 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5kx5\" (UniqueName: \"kubernetes.io/projected/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-kube-api-access-d5kx5\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rdnft\" (UID: \"5fc2a87b-394a-4c26-adfd-9a57a1fc463b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rdnft" Oct 31 05:06:17.924666 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:17.924327 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rdnft\" (UID: \"5fc2a87b-394a-4c26-adfd-9a57a1fc463b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rdnft" Oct 31 05:06:17.924666 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:17.924425 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rdnft\" (UID: \"5fc2a87b-394a-4c26-adfd-9a57a1fc463b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rdnft" Oct 31 05:06:17.924666 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:17.924606 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rdnft\" (UID: \"5fc2a87b-394a-4c26-adfd-9a57a1fc463b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rdnft" Oct 31 05:06:17.924832 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:17.924709 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rdnft\" (UID: \"5fc2a87b-394a-4c26-adfd-9a57a1fc463b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rdnft" Oct 31 05:06:17.924832 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:17.924774 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rdnft\" (UID: \"5fc2a87b-394a-4c26-adfd-9a57a1fc463b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rdnft" Oct 31 05:06:17.924970 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:17.924849 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rdnft\" (UID: \"5fc2a87b-394a-4c26-adfd-9a57a1fc463b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rdnft" Oct 31 05:06:17.925095 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:17.925030 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rdnft\" (UID: \"5fc2a87b-394a-4c26-adfd-9a57a1fc463b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rdnft" Oct 31 05:06:17.926946 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:17.926910 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rdnft\" (UID: \"5fc2a87b-394a-4c26-adfd-9a57a1fc463b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rdnft" Oct 31 05:06:17.927065 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:17.926959 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rdnft\" (UID: \"5fc2a87b-394a-4c26-adfd-9a57a1fc463b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rdnft" Oct 31 05:06:17.954158 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:17.954077 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rdnft\" (UID: \"5fc2a87b-394a-4c26-adfd-9a57a1fc463b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rdnft" Oct 31 05:06:17.995497 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:17.992605 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5kx5\" (UniqueName: \"kubernetes.io/projected/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-kube-api-access-d5kx5\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rdnft\" (UID: \"5fc2a87b-394a-4c26-adfd-9a57a1fc463b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rdnft" Oct 31 05:06:18.058450 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:18.058398 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rdnft" Oct 31 05:06:18.392160 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:18.389601 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rdnft"] Oct 31 05:06:18.406712 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:18.406655 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:06:18.406893 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:18.406748 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:06:18.406893 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:18.406785 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:06:18.533597 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:18.533547 5019 generic.go:356] "Generic (PLEG): container finished" podID="4fe19836-7804-40c2-b436-c947e33e979a" containerID="64a2cc1fcd3b3a4b931f7646d4296059350aef5b872a7075400ad3bd73c6bfdb" exitCode=0 Oct 31 05:06:18.533863 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:18.533838 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-108ad8-8d8c4c667-rbd9x" event={"ID":"4fe19836-7804-40c2-b436-c947e33e979a","Type":"ContainerDied","Data":"64a2cc1fcd3b3a4b931f7646d4296059350aef5b872a7075400ad3bd73c6bfdb"} Oct 31 05:06:18.536883 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:18.536842 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rdnft" event={"ID":"5fc2a87b-394a-4c26-adfd-9a57a1fc463b","Type":"ContainerStarted","Data":"9cb688020331b5b25269cdcabc6ef390ed966339aa5c38048b85635200eebd08"} Oct 31 05:06:18.554612 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:18.554573 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-108ad8-8d8c4c667-rbd9x" Oct 31 05:06:18.589621 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:18.589534 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rdnft" podStartSLOduration=1.589510839 podStartE2EDuration="1.589510839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:06:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:06:18.584723874 +0000 UTC m=+3733.341061913" watchObservedRunningTime="2026-10-31 05:06:18.589510839 +0000 UTC m=+3733.345848880" Oct 31 05:06:18.640804 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:18.640768 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fe19836-7804-40c2-b436-c947e33e979a-kube-api-access-wlw76\" (UniqueName: \"kubernetes.io/projected/4fe19836-7804-40c2-b436-c947e33e979a-kube-api-access-wlw76\") pod \"4fe19836-7804-40c2-b436-c947e33e979a\" (UID: \"4fe19836-7804-40c2-b436-c947e33e979a\") " Oct 31 05:06:18.640989 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:18.640973 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fe19836-7804-40c2-b436-c947e33e979a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fe19836-7804-40c2-b436-c947e33e979a-plugins-config-volume\") pod \"4fe19836-7804-40c2-b436-c947e33e979a\" (UID: \"4fe19836-7804-40c2-b436-c947e33e979a\") " Oct 31 05:06:18.641870 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:18.641641 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fe19836-7804-40c2-b436-c947e33e979a-plugins-config-volume" pod "4fe19836-7804-40c2-b436-c947e33e979a" (UID: "4fe19836-7804-40c2-b436-c947e33e979a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:06:18.642045 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:18.641932 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fe19836-7804-40c2-b436-c947e33e979a-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:06:18.644754 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:18.644711 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fe19836-7804-40c2-b436-c947e33e979a-kube-api-access-wlw76" pod "4fe19836-7804-40c2-b436-c947e33e979a" (UID: "4fe19836-7804-40c2-b436-c947e33e979a"). InnerVolumeSpecName "kube-api-access-wlw76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:06:18.744068 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:18.744016 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlw76\" (UniqueName: \"kubernetes.io/projected/4fe19836-7804-40c2-b436-c947e33e979a-kube-api-access-wlw76\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:06:19.058918 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:19.058872 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rdnft" Oct 31 05:06:19.060599 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:19.060566 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-rdnft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.171:15021/healthz/ready\": dial tcp 10.132.1.171:15021: connect: connection refused" start-of-body= Oct 31 05:06:19.060753 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:19.060634 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rdnft" podUID="5fc2a87b-394a-4c26-adfd-9a57a1fc463b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.171:15021/healthz/ready\": dial tcp 10.132.1.171:15021: connect: connection refused" Oct 31 05:06:19.548035 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:19.548004 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-108ad8-8d8c4c667-rbd9x" Oct 31 05:06:19.548276 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:19.548234 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-108ad8-8d8c4c667-rbd9x" event={"ID":"4fe19836-7804-40c2-b436-c947e33e979a","Type":"ContainerDied","Data":"9be2f4a3dfb05cee0458cef3c639a589dc100e056198b05d980690456847031c"} Oct 31 05:06:19.548421 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:19.548320 5019 scope.go:122] "RemoveContainer" containerID="64a2cc1fcd3b3a4b931f7646d4296059350aef5b872a7075400ad3bd73c6bfdb" Oct 31 05:06:19.550212 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:19.550183 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rdnft" event={"ID":"5fc2a87b-394a-4c26-adfd-9a57a1fc463b","Type":"ContainerStarted","Data":"6d58868d6eb7edd1c2938c35528b8a887945575e704d2f8158d3d9a1012a1f14"} Oct 31 05:06:19.620435 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:19.620393 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-108ad8-8d8c4c667-rbd9x"] Oct 31 05:06:19.631954 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:19.631918 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-108ad8-8d8c4c667-rbd9x"] Oct 31 05:06:20.058829 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:20.058792 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-rdnft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.171:15021/healthz/ready\": dial tcp 10.132.1.171:15021: connect: connection refused" start-of-body= Oct 31 05:06:20.059283 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:20.058847 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rdnft" podUID="5fc2a87b-394a-4c26-adfd-9a57a1fc463b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.171:15021/healthz/ready\": dial tcp 10.132.1.171:15021: connect: connection refused" Oct 31 05:06:21.059581 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:21.059533 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-rdnft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.171:15021/healthz/ready\": dial tcp 10.132.1.171:15021: connect: connection refused" start-of-body= Oct 31 05:06:21.060001 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:21.059607 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rdnft" podUID="5fc2a87b-394a-4c26-adfd-9a57a1fc463b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.171:15021/healthz/ready\": dial tcp 10.132.1.171:15021: connect: connection refused" Oct 31 05:06:21.415147 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:21.415053 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fe19836-7804-40c2-b436-c947e33e979a" path="/var/lib/kubelet/pods/4fe19836-7804-40c2-b436-c947e33e979a/volumes" Oct 31 05:06:22.059463 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:22.059417 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-rdnft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.171:15021/healthz/ready\": dial tcp 10.132.1.171:15021: connect: connection refused" start-of-body= Oct 31 05:06:22.059683 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:22.059501 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rdnft" podUID="5fc2a87b-394a-4c26-adfd-9a57a1fc463b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.171:15021/healthz/ready\": dial tcp 10.132.1.171:15021: connect: connection refused" Oct 31 05:06:22.431716 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:06:22.431618 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:06:23.059742 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:23.059702 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-rdnft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.171:15021/healthz/ready\": dial tcp 10.132.1.171:15021: connect: connection refused" start-of-body= Oct 31 05:06:23.060222 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:23.059773 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rdnft" podUID="5fc2a87b-394a-4c26-adfd-9a57a1fc463b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.171:15021/healthz/ready\": dial tcp 10.132.1.171:15021: connect: connection refused" Oct 31 05:06:24.059027 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:24.058981 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-rdnft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.171:15021/healthz/ready\": dial tcp 10.132.1.171:15021: connect: connection refused" start-of-body= Oct 31 05:06:24.059512 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:24.059061 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rdnft" podUID="5fc2a87b-394a-4c26-adfd-9a57a1fc463b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.171:15021/healthz/ready\": dial tcp 10.132.1.171:15021: connect: connection refused" Oct 31 05:06:25.059734 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:25.059678 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-rdnft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.171:15021/healthz/ready\": dial tcp 10.132.1.171:15021: connect: connection refused" start-of-body= Oct 31 05:06:25.060321 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:25.059760 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rdnft" podUID="5fc2a87b-394a-4c26-adfd-9a57a1fc463b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.171:15021/healthz/ready\": dial tcp 10.132.1.171:15021: connect: connection refused" Oct 31 05:06:26.058961 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:26.058916 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-rdnft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.171:15021/healthz/ready\": dial tcp 10.132.1.171:15021: connect: connection refused" start-of-body= Oct 31 05:06:26.059200 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:26.058981 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rdnft" podUID="5fc2a87b-394a-4c26-adfd-9a57a1fc463b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.171:15021/healthz/ready\": dial tcp 10.132.1.171:15021: connect: connection refused" Oct 31 05:06:27.595000 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:27.594961 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rdnft" Oct 31 05:06:27.595556 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:27.595520 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rdnft" Oct 31 05:06:27.596460 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:27.596440 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rdnft" Oct 31 05:06:33.127442 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:33.127221 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl"] Oct 31 05:06:33.128569 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:33.128517 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl" podUID="e6d6891a-ccbc-4007-bbf3-96544ac865c2" containerName="istio-proxy" containerID="cri-o://8f0392245a281d1f0a00b0f7485b92ee412a6e40dd120d83eb35513376e05de3" gracePeriod=30 Oct 31 05:06:38.474295 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:38.474265 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl" Oct 31 05:06:38.620472 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:38.620428 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6d6891a-ccbc-4007-bbf3-96544ac865c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6d6891a-ccbc-4007-bbf3-96544ac865c2-workload-certs\") pod \"e6d6891a-ccbc-4007-bbf3-96544ac865c2\" (UID: \"e6d6891a-ccbc-4007-bbf3-96544ac865c2\") " Oct 31 05:06:38.620688 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:38.620492 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6d6891a-ccbc-4007-bbf3-96544ac865c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6d6891a-ccbc-4007-bbf3-96544ac865c2-istio-data\") pod \"e6d6891a-ccbc-4007-bbf3-96544ac865c2\" (UID: \"e6d6891a-ccbc-4007-bbf3-96544ac865c2\") " Oct 31 05:06:38.620688 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:38.620518 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6d6891a-ccbc-4007-bbf3-96544ac865c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6d6891a-ccbc-4007-bbf3-96544ac865c2-workload-socket\") pod \"e6d6891a-ccbc-4007-bbf3-96544ac865c2\" (UID: \"e6d6891a-ccbc-4007-bbf3-96544ac865c2\") " Oct 31 05:06:38.620688 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:38.620585 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6d6891a-ccbc-4007-bbf3-96544ac865c2-istio-token\" (UniqueName: \"kubernetes.io/projected/e6d6891a-ccbc-4007-bbf3-96544ac865c2-istio-token\") pod \"e6d6891a-ccbc-4007-bbf3-96544ac865c2\" (UID: \"e6d6891a-ccbc-4007-bbf3-96544ac865c2\") " Oct 31 05:06:38.620688 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:38.620655 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6d6891a-ccbc-4007-bbf3-96544ac865c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6d6891a-ccbc-4007-bbf3-96544ac865c2-istio-envoy\") pod \"e6d6891a-ccbc-4007-bbf3-96544ac865c2\" (UID: \"e6d6891a-ccbc-4007-bbf3-96544ac865c2\") " Oct 31 05:06:38.620893 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:38.620688 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6d6891a-ccbc-4007-bbf3-96544ac865c2-kube-api-access-8gst8\" (UniqueName: \"kubernetes.io/projected/e6d6891a-ccbc-4007-bbf3-96544ac865c2-kube-api-access-8gst8\") pod \"e6d6891a-ccbc-4007-bbf3-96544ac865c2\" (UID: \"e6d6891a-ccbc-4007-bbf3-96544ac865c2\") " Oct 31 05:06:38.620893 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:38.620730 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6d6891a-ccbc-4007-bbf3-96544ac865c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6d6891a-ccbc-4007-bbf3-96544ac865c2-istiod-ca-cert\") pod \"e6d6891a-ccbc-4007-bbf3-96544ac865c2\" (UID: \"e6d6891a-ccbc-4007-bbf3-96544ac865c2\") " Oct 31 05:06:38.620893 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:38.620765 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6d6891a-ccbc-4007-bbf3-96544ac865c2-workload-certs" pod "e6d6891a-ccbc-4007-bbf3-96544ac865c2" (UID: "e6d6891a-ccbc-4007-bbf3-96544ac865c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:06:38.620893 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:38.620778 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6d6891a-ccbc-4007-bbf3-96544ac865c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6d6891a-ccbc-4007-bbf3-96544ac865c2-credential-socket\") pod \"e6d6891a-ccbc-4007-bbf3-96544ac865c2\" (UID: \"e6d6891a-ccbc-4007-bbf3-96544ac865c2\") " Oct 31 05:06:38.620893 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:38.620803 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6d6891a-ccbc-4007-bbf3-96544ac865c2-istio-data" pod "e6d6891a-ccbc-4007-bbf3-96544ac865c2" (UID: "e6d6891a-ccbc-4007-bbf3-96544ac865c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:06:38.621176 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:38.620944 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6d6891a-ccbc-4007-bbf3-96544ac865c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6d6891a-ccbc-4007-bbf3-96544ac865c2-istio-podinfo\") pod \"e6d6891a-ccbc-4007-bbf3-96544ac865c2\" (UID: \"e6d6891a-ccbc-4007-bbf3-96544ac865c2\") " Oct 31 05:06:38.621176 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:38.620992 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6d6891a-ccbc-4007-bbf3-96544ac865c2-workload-socket" pod "e6d6891a-ccbc-4007-bbf3-96544ac865c2" (UID: "e6d6891a-ccbc-4007-bbf3-96544ac865c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:06:38.621303 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:38.621196 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6d6891a-ccbc-4007-bbf3-96544ac865c2-credential-socket" pod "e6d6891a-ccbc-4007-bbf3-96544ac865c2" (UID: "e6d6891a-ccbc-4007-bbf3-96544ac865c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:06:38.621303 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:38.621283 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6d6891a-ccbc-4007-bbf3-96544ac865c2-istiod-ca-cert" pod "e6d6891a-ccbc-4007-bbf3-96544ac865c2" (UID: "e6d6891a-ccbc-4007-bbf3-96544ac865c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:06:38.621755 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:38.621733 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6d6891a-ccbc-4007-bbf3-96544ac865c2-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:06:38.621755 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:38.621754 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6d6891a-ccbc-4007-bbf3-96544ac865c2-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:06:38.621923 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:38.621764 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6d6891a-ccbc-4007-bbf3-96544ac865c2-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:06:38.621923 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:38.621774 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6d6891a-ccbc-4007-bbf3-96544ac865c2-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:06:38.621923 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:38.621784 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6d6891a-ccbc-4007-bbf3-96544ac865c2-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:06:38.623225 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:38.623193 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6d6891a-ccbc-4007-bbf3-96544ac865c2-istio-token" pod "e6d6891a-ccbc-4007-bbf3-96544ac865c2" (UID: "e6d6891a-ccbc-4007-bbf3-96544ac865c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:06:38.623379 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:38.623263 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6d6891a-ccbc-4007-bbf3-96544ac865c2-kube-api-access-8gst8" pod "e6d6891a-ccbc-4007-bbf3-96544ac865c2" (UID: "e6d6891a-ccbc-4007-bbf3-96544ac865c2"). InnerVolumeSpecName "kube-api-access-8gst8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:06:38.623583 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:38.623561 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6d6891a-ccbc-4007-bbf3-96544ac865c2-istio-podinfo" pod "e6d6891a-ccbc-4007-bbf3-96544ac865c2" (UID: "e6d6891a-ccbc-4007-bbf3-96544ac865c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:06:38.623583 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:38.623569 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6d6891a-ccbc-4007-bbf3-96544ac865c2-istio-envoy" pod "e6d6891a-ccbc-4007-bbf3-96544ac865c2" (UID: "e6d6891a-ccbc-4007-bbf3-96544ac865c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:06:38.723835 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:38.723790 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6d6891a-ccbc-4007-bbf3-96544ac865c2-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:06:38.723835 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:38.723830 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6d6891a-ccbc-4007-bbf3-96544ac865c2-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:06:38.723835 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:38.723842 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gst8\" (UniqueName: \"kubernetes.io/projected/e6d6891a-ccbc-4007-bbf3-96544ac865c2-kube-api-access-8gst8\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:06:38.724113 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:38.723853 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6d6891a-ccbc-4007-bbf3-96544ac865c2-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:06:38.916025 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:38.915988 5019 generic.go:356] "Generic (PLEG): container finished" podID="e6d6891a-ccbc-4007-bbf3-96544ac865c2" containerID="8f0392245a281d1f0a00b0f7485b92ee412a6e40dd120d83eb35513376e05de3" exitCode=0 Oct 31 05:06:38.916218 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:38.916051 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl" Oct 31 05:06:38.916218 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:38.916049 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl" event={"ID":"e6d6891a-ccbc-4007-bbf3-96544ac865c2","Type":"ContainerDied","Data":"8f0392245a281d1f0a00b0f7485b92ee412a6e40dd120d83eb35513376e05de3"} Oct 31 05:06:38.916218 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:38.916150 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl" event={"ID":"e6d6891a-ccbc-4007-bbf3-96544ac865c2","Type":"ContainerDied","Data":"5fcd94493452423c1bba98fc05882bf25ce67409195a36b2898abafdcce81d60"} Oct 31 05:06:38.916218 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:38.916175 5019 scope.go:122] "RemoveContainer" containerID="8f0392245a281d1f0a00b0f7485b92ee412a6e40dd120d83eb35513376e05de3" Oct 31 05:06:38.933821 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:38.933794 5019 scope.go:122] "RemoveContainer" containerID="a72ce1d5dea6406c5a3b57a7313087e532a7b3cbf47f24ce3815913e72d984d3" Oct 31 05:06:38.945879 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:38.945853 5019 scope.go:122] "RemoveContainer" containerID="8f0392245a281d1f0a00b0f7485b92ee412a6e40dd120d83eb35513376e05de3" Oct 31 05:06:38.946259 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:06:38.946232 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f0392245a281d1f0a00b0f7485b92ee412a6e40dd120d83eb35513376e05de3\": container with ID starting with 8f0392245a281d1f0a00b0f7485b92ee412a6e40dd120d83eb35513376e05de3 not found: ID does not exist" containerID="8f0392245a281d1f0a00b0f7485b92ee412a6e40dd120d83eb35513376e05de3" Oct 31 05:06:38.946392 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:38.946270 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f0392245a281d1f0a00b0f7485b92ee412a6e40dd120d83eb35513376e05de3"} err="failed to get container status \"8f0392245a281d1f0a00b0f7485b92ee412a6e40dd120d83eb35513376e05de3\": rpc error: code = NotFound desc = could not find container \"8f0392245a281d1f0a00b0f7485b92ee412a6e40dd120d83eb35513376e05de3\": container with ID starting with 8f0392245a281d1f0a00b0f7485b92ee412a6e40dd120d83eb35513376e05de3 not found: ID does not exist" Oct 31 05:06:38.946392 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:38.946291 5019 scope.go:122] "RemoveContainer" containerID="a72ce1d5dea6406c5a3b57a7313087e532a7b3cbf47f24ce3815913e72d984d3" Oct 31 05:06:38.946600 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:06:38.946583 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a72ce1d5dea6406c5a3b57a7313087e532a7b3cbf47f24ce3815913e72d984d3\": container with ID starting with a72ce1d5dea6406c5a3b57a7313087e532a7b3cbf47f24ce3815913e72d984d3 not found: ID does not exist" containerID="a72ce1d5dea6406c5a3b57a7313087e532a7b3cbf47f24ce3815913e72d984d3" Oct 31 05:06:38.946644 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:38.946608 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a72ce1d5dea6406c5a3b57a7313087e532a7b3cbf47f24ce3815913e72d984d3"} err="failed to get container status \"a72ce1d5dea6406c5a3b57a7313087e532a7b3cbf47f24ce3815913e72d984d3\": rpc error: code = NotFound desc = could not find container \"a72ce1d5dea6406c5a3b57a7313087e532a7b3cbf47f24ce3815913e72d984d3\": container with ID starting with a72ce1d5dea6406c5a3b57a7313087e532a7b3cbf47f24ce3815913e72d984d3 not found: ID does not exist" Oct 31 05:06:38.972387 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:38.972349 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl"] Oct 31 05:06:38.986012 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:38.985926 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1daf30-openshift-default-58c8c5d5df-hhfzl"] Oct 31 05:06:39.412795 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:39.412712 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6d6891a-ccbc-4007-bbf3-96544ac865c2" path="/var/lib/kubelet/pods/e6d6891a-ccbc-4007-bbf3-96544ac865c2/volumes" Oct 31 05:06:40.211358 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:40.211261 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7cb3ca6-9v7n2"] Oct 31 05:06:40.213367 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:40.212589 5019 cpu_manager.go:414] "containerMap: removing container" podUID="4fe19836-7804-40c2-b436-c947e33e979a" containerName="payload-pre-processing" Oct 31 05:06:40.213367 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:40.212620 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fe19836-7804-40c2-b436-c947e33e979a" containerName="payload-pre-processing" Oct 31 05:06:40.213367 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:40.212640 5019 cpu_manager.go:414] "containerMap: removing container" podUID="e6d6891a-ccbc-4007-bbf3-96544ac865c2" containerName="istio-proxy" Oct 31 05:06:40.213367 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:40.212648 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6d6891a-ccbc-4007-bbf3-96544ac865c2" containerName="istio-proxy" Oct 31 05:06:40.213367 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:40.212965 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6d6891a-ccbc-4007-bbf3-96544ac865c2" containerName="istio-proxy" Oct 31 05:06:40.213367 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:40.212988 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fe19836-7804-40c2-b436-c947e33e979a" containerName="payload-pre-processing" Oct 31 05:06:40.213367 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:40.213004 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6d6891a-ccbc-4007-bbf3-96544ac865c2" containerName="istio-proxy" Oct 31 05:06:40.217671 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:40.217625 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7cb3ca6-9v7n2" Oct 31 05:06:40.224218 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:40.224183 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7cb3ca6-9v7n2"] Oct 31 05:06:40.238434 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:40.238317 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:06:40.248069 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:40.248034 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:06:40.270606 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:40.270571 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 05:06:40.270871 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:40.270612 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:06:40.348951 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:40.348908 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4068532b-36ea-4ef1-bb0f-366e10073c49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7cb3ca6-9v7n2\" (UID: \"4068532b-36ea-4ef1-bb0f-366e10073c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7cb3ca6-9v7n2" Oct 31 05:06:40.349173 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:40.348969 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgxg4\" (UniqueName: \"kubernetes.io/projected/4068532b-36ea-4ef1-bb0f-366e10073c49-kube-api-access-sgxg4\") pod \"maas-api-revoke-sub-d7cb3ca6-9v7n2\" (UID: \"4068532b-36ea-4ef1-bb0f-366e10073c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7cb3ca6-9v7n2" Oct 31 05:06:40.349173 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:40.349134 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4068532b-36ea-4ef1-bb0f-366e10073c49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7cb3ca6-9v7n2\" (UID: \"4068532b-36ea-4ef1-bb0f-366e10073c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7cb3ca6-9v7n2" Oct 31 05:06:40.450819 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:40.450772 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4068532b-36ea-4ef1-bb0f-366e10073c49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7cb3ca6-9v7n2\" (UID: \"4068532b-36ea-4ef1-bb0f-366e10073c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7cb3ca6-9v7n2" Oct 31 05:06:40.451025 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:40.451000 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4068532b-36ea-4ef1-bb0f-366e10073c49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7cb3ca6-9v7n2\" (UID: \"4068532b-36ea-4ef1-bb0f-366e10073c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7cb3ca6-9v7n2" Oct 31 05:06:40.451071 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:40.451038 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgxg4\" (UniqueName: \"kubernetes.io/projected/4068532b-36ea-4ef1-bb0f-366e10073c49-kube-api-access-sgxg4\") pod \"maas-api-revoke-sub-d7cb3ca6-9v7n2\" (UID: \"4068532b-36ea-4ef1-bb0f-366e10073c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7cb3ca6-9v7n2" Oct 31 05:06:40.451569 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:40.451535 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4068532b-36ea-4ef1-bb0f-366e10073c49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7cb3ca6-9v7n2\" (UID: \"4068532b-36ea-4ef1-bb0f-366e10073c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7cb3ca6-9v7n2" Oct 31 05:06:40.453834 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:40.453801 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4068532b-36ea-4ef1-bb0f-366e10073c49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7cb3ca6-9v7n2\" (UID: \"4068532b-36ea-4ef1-bb0f-366e10073c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7cb3ca6-9v7n2" Oct 31 05:06:40.475686 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:40.475640 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgxg4\" (UniqueName: \"kubernetes.io/projected/4068532b-36ea-4ef1-bb0f-366e10073c49-kube-api-access-sgxg4\") pod \"maas-api-revoke-sub-d7cb3ca6-9v7n2\" (UID: \"4068532b-36ea-4ef1-bb0f-366e10073c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7cb3ca6-9v7n2" Oct 31 05:06:40.548082 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:40.548041 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7cb3ca6-9v7n2" Oct 31 05:06:40.749704 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:40.749604 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-12c0bc-6785c7bbf7-8928h"] Oct 31 05:06:40.876459 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:40.876411 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7cb3ca6-9v7n2"] Oct 31 05:06:40.940881 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:40.940835 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7cb3ca6-9v7n2" event={"ID":"4068532b-36ea-4ef1-bb0f-366e10073c49","Type":"ContainerStarted","Data":"78f72c08455c9a1fbd1826cdafb61c70963f1824ab47396cef4b51a3f2015aa1"} Oct 31 05:06:41.953218 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:41.953168 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7cb3ca6-9v7n2" event={"ID":"4068532b-36ea-4ef1-bb0f-366e10073c49","Type":"ContainerStarted","Data":"2e86d0c70d9817b9795d6283820a5ab603f9b56b8b29ea956690922afb061a0c"} Oct 31 05:06:41.994983 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:41.994926 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7cb3ca6-9v7n2" podStartSLOduration=1.9949106269999999 podStartE2EDuration="1.994910627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:06:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:06:41.992836956 +0000 UTC m=+3756.749175022" watchObservedRunningTime="2026-10-31 05:06:41.994910627 +0000 UTC m=+3756.751248667" Oct 31 05:06:42.964095 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:42.964053 5019 generic.go:356] "Generic (PLEG): container finished" podID="4068532b-36ea-4ef1-bb0f-366e10073c49" containerID="2e86d0c70d9817b9795d6283820a5ab603f9b56b8b29ea956690922afb061a0c" exitCode=0 Oct 31 05:06:42.964609 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:42.964145 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7cb3ca6-9v7n2" event={"ID":"4068532b-36ea-4ef1-bb0f-366e10073c49","Type":"ContainerDied","Data":"2e86d0c70d9817b9795d6283820a5ab603f9b56b8b29ea956690922afb061a0c"} Oct 31 05:06:44.262452 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:44.261869 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7cb3ca6-9v7n2" Oct 31 05:06:44.344290 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:44.344246 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4068532b-36ea-4ef1-bb0f-366e10073c49-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4068532b-36ea-4ef1-bb0f-366e10073c49-maas-api-xks-ca\") pod \"4068532b-36ea-4ef1-bb0f-366e10073c49\" (UID: \"4068532b-36ea-4ef1-bb0f-366e10073c49\") " Oct 31 05:06:44.344531 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:44.344444 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4068532b-36ea-4ef1-bb0f-366e10073c49-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4068532b-36ea-4ef1-bb0f-366e10073c49-maas-api-openshift-ca\") pod \"4068532b-36ea-4ef1-bb0f-366e10073c49\" (UID: \"4068532b-36ea-4ef1-bb0f-366e10073c49\") " Oct 31 05:06:44.344613 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:44.344565 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4068532b-36ea-4ef1-bb0f-366e10073c49-kube-api-access-sgxg4\" (UniqueName: \"kubernetes.io/projected/4068532b-36ea-4ef1-bb0f-366e10073c49-kube-api-access-sgxg4\") pod \"4068532b-36ea-4ef1-bb0f-366e10073c49\" (UID: \"4068532b-36ea-4ef1-bb0f-366e10073c49\") " Oct 31 05:06:44.344908 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:44.344879 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4068532b-36ea-4ef1-bb0f-366e10073c49-maas-api-openshift-ca" pod "4068532b-36ea-4ef1-bb0f-366e10073c49" (UID: "4068532b-36ea-4ef1-bb0f-366e10073c49"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:06:44.345419 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:44.345391 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4068532b-36ea-4ef1-bb0f-366e10073c49-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:06:44.346973 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:44.346939 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4068532b-36ea-4ef1-bb0f-366e10073c49-kube-api-access-sgxg4" pod "4068532b-36ea-4ef1-bb0f-366e10073c49" (UID: "4068532b-36ea-4ef1-bb0f-366e10073c49"). InnerVolumeSpecName "kube-api-access-sgxg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:06:44.347120 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:44.347092 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4068532b-36ea-4ef1-bb0f-366e10073c49-maas-api-xks-ca" pod "4068532b-36ea-4ef1-bb0f-366e10073c49" (UID: "4068532b-36ea-4ef1-bb0f-366e10073c49"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:06:44.469359 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:44.461649 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4068532b-36ea-4ef1-bb0f-366e10073c49-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:06:44.483399 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:44.469632 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgxg4\" (UniqueName: \"kubernetes.io/projected/4068532b-36ea-4ef1-bb0f-366e10073c49-kube-api-access-sgxg4\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:06:44.993218 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:44.993180 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7cb3ca6-9v7n2" Oct 31 05:06:44.993521 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:44.993422 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7cb3ca6-9v7n2" event={"ID":"4068532b-36ea-4ef1-bb0f-366e10073c49","Type":"ContainerDied","Data":"78f72c08455c9a1fbd1826cdafb61c70963f1824ab47396cef4b51a3f2015aa1"} Oct 31 05:06:44.993521 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:44.993480 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78f72c08455c9a1fbd1826cdafb61c70963f1824ab47396cef4b51a3f2015aa1" Oct 31 05:06:45.750780 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:45.750713 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-12c0bc-6785c7bbf7-8928h" podUID="5da17e2b-6c07-4222-b09c-310cf0805666" containerName="payload-processing" containerID="cri-o://8c056a3b10cf20c966662b589476bb22440896a842bf1bdc6fe733b97df1f031" gracePeriod=25 Oct 31 05:06:46.013679 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:46.013420 5019 generic.go:356] "Generic (PLEG): container finished" podID="5da17e2b-6c07-4222-b09c-310cf0805666" containerID="8c056a3b10cf20c966662b589476bb22440896a842bf1bdc6fe733b97df1f031" exitCode=0 Oct 31 05:06:46.013679 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:46.013528 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-12c0bc-6785c7bbf7-8928h" event={"ID":"5da17e2b-6c07-4222-b09c-310cf0805666","Type":"ContainerDied","Data":"8c056a3b10cf20c966662b589476bb22440896a842bf1bdc6fe733b97df1f031"} Oct 31 05:06:46.069351 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:46.069311 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-12c0bc-6785c7bbf7-8928h" Oct 31 05:06:46.213424 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:46.213385 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5da17e2b-6c07-4222-b09c-310cf0805666-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5da17e2b-6c07-4222-b09c-310cf0805666-plugins-config-volume\") pod \"5da17e2b-6c07-4222-b09c-310cf0805666\" (UID: \"5da17e2b-6c07-4222-b09c-310cf0805666\") " Oct 31 05:06:46.213654 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:46.213640 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5da17e2b-6c07-4222-b09c-310cf0805666-kube-api-access-76d27\" (UniqueName: \"kubernetes.io/projected/5da17e2b-6c07-4222-b09c-310cf0805666-kube-api-access-76d27\") pod \"5da17e2b-6c07-4222-b09c-310cf0805666\" (UID: \"5da17e2b-6c07-4222-b09c-310cf0805666\") " Oct 31 05:06:46.213830 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:46.213804 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5da17e2b-6c07-4222-b09c-310cf0805666-plugins-config-volume" pod "5da17e2b-6c07-4222-b09c-310cf0805666" (UID: "5da17e2b-6c07-4222-b09c-310cf0805666"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:06:46.214150 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:46.214133 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5da17e2b-6c07-4222-b09c-310cf0805666-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:06:46.216017 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:46.215988 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5da17e2b-6c07-4222-b09c-310cf0805666-kube-api-access-76d27" pod "5da17e2b-6c07-4222-b09c-310cf0805666" (UID: "5da17e2b-6c07-4222-b09c-310cf0805666"). InnerVolumeSpecName "kube-api-access-76d27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:06:46.315596 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:46.315500 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76d27\" (UniqueName: \"kubernetes.io/projected/5da17e2b-6c07-4222-b09c-310cf0805666-kube-api-access-76d27\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:06:47.025312 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:47.025272 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-12c0bc-6785c7bbf7-8928h" Oct 31 05:06:47.025830 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:47.025283 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-12c0bc-6785c7bbf7-8928h" event={"ID":"5da17e2b-6c07-4222-b09c-310cf0805666","Type":"ContainerDied","Data":"0fce3db77a0b9c8b17cb0aeda0c67c31e1bbb9b0c4ed39675ee3bb662d5148e4"} Oct 31 05:06:47.025830 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:47.025455 5019 scope.go:122] "RemoveContainer" containerID="8c056a3b10cf20c966662b589476bb22440896a842bf1bdc6fe733b97df1f031" Oct 31 05:06:47.085620 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:47.085583 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-12c0bc-6785c7bbf7-8928h"] Oct 31 05:06:47.090583 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:47.090543 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-12c0bc-6785c7bbf7-8928h"] Oct 31 05:06:47.411932 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:47.411834 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5da17e2b-6c07-4222-b09c-310cf0805666" path="/var/lib/kubelet/pods/5da17e2b-6c07-4222-b09c-310cf0805666/volumes" Oct 31 05:06:52.394074 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:06:52.394027 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:06:54.946739 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:54.946700 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7"] Oct 31 05:06:54.947170 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:06:54.946999 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7" podUID="7e6bd271-4285-4390-a0f9-e2fcd5a855d4" containerName="istio-proxy" containerID="cri-o://b09319d67033b91da26336d733c47a3526a290dbf08cefd50290c446a792bf2f" gracePeriod=30 Oct 31 05:07:00.196088 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:00.196032 5019 generic.go:356] "Generic (PLEG): container finished" podID="7e6bd271-4285-4390-a0f9-e2fcd5a855d4" containerID="b09319d67033b91da26336d733c47a3526a290dbf08cefd50290c446a792bf2f" exitCode=0 Oct 31 05:07:00.196669 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:00.196348 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7" event={"ID":"7e6bd271-4285-4390-a0f9-e2fcd5a855d4","Type":"ContainerDied","Data":"b09319d67033b91da26336d733c47a3526a290dbf08cefd50290c446a792bf2f"} Oct 31 05:07:00.196669 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:00.196414 5019 scope.go:122] "RemoveContainer" containerID="36ce6b5cb4ca668ed880432272f83fc12a7095d8cf42c073613a404058d82fe8" Oct 31 05:07:00.277898 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:00.277855 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7" Oct 31 05:07:00.418761 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:00.418714 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-credential-socket\") pod \"7e6bd271-4285-4390-a0f9-e2fcd5a855d4\" (UID: \"7e6bd271-4285-4390-a0f9-e2fcd5a855d4\") " Oct 31 05:07:00.418994 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:00.418798 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-workload-certs\") pod \"7e6bd271-4285-4390-a0f9-e2fcd5a855d4\" (UID: \"7e6bd271-4285-4390-a0f9-e2fcd5a855d4\") " Oct 31 05:07:00.418994 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:00.418878 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-istio-token\" (UniqueName: \"kubernetes.io/projected/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-istio-token\") pod \"7e6bd271-4285-4390-a0f9-e2fcd5a855d4\" (UID: \"7e6bd271-4285-4390-a0f9-e2fcd5a855d4\") " Oct 31 05:07:00.418994 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:00.418920 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-istiod-ca-cert\") pod \"7e6bd271-4285-4390-a0f9-e2fcd5a855d4\" (UID: \"7e6bd271-4285-4390-a0f9-e2fcd5a855d4\") " Oct 31 05:07:00.418994 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:00.418986 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-istio-data\") pod \"7e6bd271-4285-4390-a0f9-e2fcd5a855d4\" (UID: \"7e6bd271-4285-4390-a0f9-e2fcd5a855d4\") " Oct 31 05:07:00.419238 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:00.419034 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-kube-api-access-hg7hj\" (UniqueName: \"kubernetes.io/projected/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-kube-api-access-hg7hj\") pod \"7e6bd271-4285-4390-a0f9-e2fcd5a855d4\" (UID: \"7e6bd271-4285-4390-a0f9-e2fcd5a855d4\") " Oct 31 05:07:00.419238 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:00.419041 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-credential-socket" pod "7e6bd271-4285-4390-a0f9-e2fcd5a855d4" (UID: "7e6bd271-4285-4390-a0f9-e2fcd5a855d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:07:00.419238 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:00.419121 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-workload-certs" pod "7e6bd271-4285-4390-a0f9-e2fcd5a855d4" (UID: "7e6bd271-4285-4390-a0f9-e2fcd5a855d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:07:00.419238 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:00.419209 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-istio-data" pod "7e6bd271-4285-4390-a0f9-e2fcd5a855d4" (UID: "7e6bd271-4285-4390-a0f9-e2fcd5a855d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:07:00.419488 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:00.419290 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-istiod-ca-cert" pod "7e6bd271-4285-4390-a0f9-e2fcd5a855d4" (UID: "7e6bd271-4285-4390-a0f9-e2fcd5a855d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:07:00.419488 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:00.419465 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-workload-socket\") pod \"7e6bd271-4285-4390-a0f9-e2fcd5a855d4\" (UID: \"7e6bd271-4285-4390-a0f9-e2fcd5a855d4\") " Oct 31 05:07:00.419682 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:00.419520 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-istio-podinfo\") pod \"7e6bd271-4285-4390-a0f9-e2fcd5a855d4\" (UID: \"7e6bd271-4285-4390-a0f9-e2fcd5a855d4\") " Oct 31 05:07:00.419682 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:00.419604 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-istio-envoy\") pod \"7e6bd271-4285-4390-a0f9-e2fcd5a855d4\" (UID: \"7e6bd271-4285-4390-a0f9-e2fcd5a855d4\") " Oct 31 05:07:00.419839 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:00.419789 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-workload-socket" pod "7e6bd271-4285-4390-a0f9-e2fcd5a855d4" (UID: "7e6bd271-4285-4390-a0f9-e2fcd5a855d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:07:00.420520 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:00.420495 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:07:00.420663 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:00.420527 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:07:00.420663 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:00.420542 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:07:00.420663 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:00.420556 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:07:00.420663 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:00.420570 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:07:00.421938 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:00.421816 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-istio-podinfo" pod "7e6bd271-4285-4390-a0f9-e2fcd5a855d4" (UID: "7e6bd271-4285-4390-a0f9-e2fcd5a855d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:07:00.421938 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:00.421876 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-istio-envoy" pod "7e6bd271-4285-4390-a0f9-e2fcd5a855d4" (UID: "7e6bd271-4285-4390-a0f9-e2fcd5a855d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:07:00.421938 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:00.421901 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-kube-api-access-hg7hj" pod "7e6bd271-4285-4390-a0f9-e2fcd5a855d4" (UID: "7e6bd271-4285-4390-a0f9-e2fcd5a855d4"). InnerVolumeSpecName "kube-api-access-hg7hj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:07:00.421938 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:00.421923 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-istio-token" pod "7e6bd271-4285-4390-a0f9-e2fcd5a855d4" (UID: "7e6bd271-4285-4390-a0f9-e2fcd5a855d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:07:00.522053 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:00.521965 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:07:00.522053 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:00.522001 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:07:00.522053 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:00.522013 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg7hj\" (UniqueName: \"kubernetes.io/projected/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-kube-api-access-hg7hj\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:07:00.522053 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:00.522024 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e6bd271-4285-4390-a0f9-e2fcd5a855d4-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:07:01.210635 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:01.210602 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7" Oct 31 05:07:01.211126 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:01.210598 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7" event={"ID":"7e6bd271-4285-4390-a0f9-e2fcd5a855d4","Type":"ContainerDied","Data":"b945e0c12d1b0ea940108f15303837354637d0b937cb86ff0b38ecd8d2a7133d"} Oct 31 05:07:01.211126 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:01.210758 5019 scope.go:122] "RemoveContainer" containerID="b09319d67033b91da26336d733c47a3526a290dbf08cefd50290c446a792bf2f" Oct 31 05:07:01.265943 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:01.265902 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7"] Oct 31 05:07:01.278088 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:01.278044 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-108ad8-openshift-default-5f6b668f95-4m7k7"] Oct 31 05:07:01.427477 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:01.427440 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e6bd271-4285-4390-a0f9-e2fcd5a855d4" path="/var/lib/kubelet/pods/7e6bd271-4285-4390-a0f9-e2fcd5a855d4/volumes" Oct 31 05:07:01.982625 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:01.982585 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-47e9af-7cb88c595f-s7jrb"] Oct 31 05:07:06.983301 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:06.983241 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-47e9af-7cb88c595f-s7jrb" podUID="d22c47b5-d2a0-423e-a2b5-c83bf4afbf96" containerName="payload-processing" containerID="cri-o://18710fcb0d8fb5f4ff0be5042b9e1283fe037a4ee822c509a59f7c51a4f006b2" gracePeriod=25 Oct 31 05:07:07.308386 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:07.308244 5019 generic.go:356] "Generic (PLEG): container finished" podID="d22c47b5-d2a0-423e-a2b5-c83bf4afbf96" containerID="18710fcb0d8fb5f4ff0be5042b9e1283fe037a4ee822c509a59f7c51a4f006b2" exitCode=0 Oct 31 05:07:07.308569 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:07.308394 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-47e9af-7cb88c595f-s7jrb" event={"ID":"d22c47b5-d2a0-423e-a2b5-c83bf4afbf96","Type":"ContainerDied","Data":"18710fcb0d8fb5f4ff0be5042b9e1283fe037a4ee822c509a59f7c51a4f006b2"} Oct 31 05:07:07.349565 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:07.349533 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-47e9af-7cb88c595f-s7jrb" Oct 31 05:07:07.377199 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:07.377144 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d22c47b5-d2a0-423e-a2b5-c83bf4afbf96-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d22c47b5-d2a0-423e-a2b5-c83bf4afbf96-plugins-config-volume\") pod \"d22c47b5-d2a0-423e-a2b5-c83bf4afbf96\" (UID: \"d22c47b5-d2a0-423e-a2b5-c83bf4afbf96\") " Oct 31 05:07:07.377417 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:07.377275 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d22c47b5-d2a0-423e-a2b5-c83bf4afbf96-kube-api-access-sgvk8\" (UniqueName: \"kubernetes.io/projected/d22c47b5-d2a0-423e-a2b5-c83bf4afbf96-kube-api-access-sgvk8\") pod \"d22c47b5-d2a0-423e-a2b5-c83bf4afbf96\" (UID: \"d22c47b5-d2a0-423e-a2b5-c83bf4afbf96\") " Oct 31 05:07:07.377591 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:07.377567 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d22c47b5-d2a0-423e-a2b5-c83bf4afbf96-plugins-config-volume" pod "d22c47b5-d2a0-423e-a2b5-c83bf4afbf96" (UID: "d22c47b5-d2a0-423e-a2b5-c83bf4afbf96"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:07:07.378001 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:07.377967 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d22c47b5-d2a0-423e-a2b5-c83bf4afbf96-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:07:07.379555 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:07.379520 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d22c47b5-d2a0-423e-a2b5-c83bf4afbf96-kube-api-access-sgvk8" pod "d22c47b5-d2a0-423e-a2b5-c83bf4afbf96" (UID: "d22c47b5-d2a0-423e-a2b5-c83bf4afbf96"). InnerVolumeSpecName "kube-api-access-sgvk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:07:07.479857 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:07.479811 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgvk8\" (UniqueName: \"kubernetes.io/projected/d22c47b5-d2a0-423e-a2b5-c83bf4afbf96-kube-api-access-sgvk8\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:07:08.320321 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:08.320287 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-47e9af-7cb88c595f-s7jrb" Oct 31 05:07:08.320916 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:08.320326 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-47e9af-7cb88c595f-s7jrb" event={"ID":"d22c47b5-d2a0-423e-a2b5-c83bf4afbf96","Type":"ContainerDied","Data":"be369535179b332120ede41289e4dc6a5364af96b3fb5c0ac9ce03dbd2e4f38b"} Oct 31 05:07:08.320916 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:08.320441 5019 scope.go:122] "RemoveContainer" containerID="18710fcb0d8fb5f4ff0be5042b9e1283fe037a4ee822c509a59f7c51a4f006b2" Oct 31 05:07:08.374728 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:08.374681 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-47e9af-7cb88c595f-s7jrb"] Oct 31 05:07:08.379629 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:08.379586 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-47e9af-7cb88c595f-s7jrb"] Oct 31 05:07:09.413998 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:09.413954 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d22c47b5-d2a0-423e-a2b5-c83bf4afbf96" path="/var/lib/kubelet/pods/d22c47b5-d2a0-423e-a2b5-c83bf4afbf96/volumes" Oct 31 05:07:14.643820 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:14.643765 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24"] Oct 31 05:07:14.644386 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:14.644041 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24" podUID="e33bca6f-714f-44a3-bf0c-e99ddb526c67" containerName="istio-proxy" containerID="cri-o://f2238467d3faf57e776bc406adaef9495cf2fdc989606c413288bf9c07850aad" gracePeriod=30 Oct 31 05:07:15.724668 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:15.724619 5019 patch_prober.go:29] interesting pod/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 05:07:15.725101 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:15.724717 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24" podUID="e33bca6f-714f-44a3-bf0c-e99ddb526c67" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 05:07:19.972368 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:19.972323 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24" Oct 31 05:07:19.998410 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:19.998360 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e33bca6f-714f-44a3-bf0c-e99ddb526c67-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e33bca6f-714f-44a3-bf0c-e99ddb526c67-workload-certs\") pod \"e33bca6f-714f-44a3-bf0c-e99ddb526c67\" (UID: \"e33bca6f-714f-44a3-bf0c-e99ddb526c67\") " Oct 31 05:07:19.998632 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:19.998427 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e33bca6f-714f-44a3-bf0c-e99ddb526c67-kube-api-access-k7lmq\" (UniqueName: \"kubernetes.io/projected/e33bca6f-714f-44a3-bf0c-e99ddb526c67-kube-api-access-k7lmq\") pod \"e33bca6f-714f-44a3-bf0c-e99ddb526c67\" (UID: \"e33bca6f-714f-44a3-bf0c-e99ddb526c67\") " Oct 31 05:07:19.998632 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:19.998480 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e33bca6f-714f-44a3-bf0c-e99ddb526c67-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e33bca6f-714f-44a3-bf0c-e99ddb526c67-istio-podinfo\") pod \"e33bca6f-714f-44a3-bf0c-e99ddb526c67\" (UID: \"e33bca6f-714f-44a3-bf0c-e99ddb526c67\") " Oct 31 05:07:19.998632 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:19.998555 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e33bca6f-714f-44a3-bf0c-e99ddb526c67-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e33bca6f-714f-44a3-bf0c-e99ddb526c67-workload-socket\") pod \"e33bca6f-714f-44a3-bf0c-e99ddb526c67\" (UID: \"e33bca6f-714f-44a3-bf0c-e99ddb526c67\") " Oct 31 05:07:19.998870 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:19.998699 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e33bca6f-714f-44a3-bf0c-e99ddb526c67-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e33bca6f-714f-44a3-bf0c-e99ddb526c67-istiod-ca-cert\") pod \"e33bca6f-714f-44a3-bf0c-e99ddb526c67\" (UID: \"e33bca6f-714f-44a3-bf0c-e99ddb526c67\") " Oct 31 05:07:19.998870 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:19.998701 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e33bca6f-714f-44a3-bf0c-e99ddb526c67-workload-certs" pod "e33bca6f-714f-44a3-bf0c-e99ddb526c67" (UID: "e33bca6f-714f-44a3-bf0c-e99ddb526c67"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:07:19.998870 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:19.998733 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e33bca6f-714f-44a3-bf0c-e99ddb526c67-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e33bca6f-714f-44a3-bf0c-e99ddb526c67-credential-socket\") pod \"e33bca6f-714f-44a3-bf0c-e99ddb526c67\" (UID: \"e33bca6f-714f-44a3-bf0c-e99ddb526c67\") " Oct 31 05:07:19.998870 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:19.998770 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e33bca6f-714f-44a3-bf0c-e99ddb526c67-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e33bca6f-714f-44a3-bf0c-e99ddb526c67-istio-envoy\") pod \"e33bca6f-714f-44a3-bf0c-e99ddb526c67\" (UID: \"e33bca6f-714f-44a3-bf0c-e99ddb526c67\") " Oct 31 05:07:19.998870 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:19.998839 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e33bca6f-714f-44a3-bf0c-e99ddb526c67-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e33bca6f-714f-44a3-bf0c-e99ddb526c67-istio-data\") pod \"e33bca6f-714f-44a3-bf0c-e99ddb526c67\" (UID: \"e33bca6f-714f-44a3-bf0c-e99ddb526c67\") " Oct 31 05:07:19.999165 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:19.998944 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e33bca6f-714f-44a3-bf0c-e99ddb526c67-istio-token\" (UniqueName: \"kubernetes.io/projected/e33bca6f-714f-44a3-bf0c-e99ddb526c67-istio-token\") pod \"e33bca6f-714f-44a3-bf0c-e99ddb526c67\" (UID: \"e33bca6f-714f-44a3-bf0c-e99ddb526c67\") " Oct 31 05:07:19.999165 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:19.999022 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e33bca6f-714f-44a3-bf0c-e99ddb526c67-credential-socket" pod "e33bca6f-714f-44a3-bf0c-e99ddb526c67" (UID: "e33bca6f-714f-44a3-bf0c-e99ddb526c67"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:07:19.999165 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:19.999119 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e33bca6f-714f-44a3-bf0c-e99ddb526c67-workload-socket" pod "e33bca6f-714f-44a3-bf0c-e99ddb526c67" (UID: "e33bca6f-714f-44a3-bf0c-e99ddb526c67"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:07:19.999429 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:19.999312 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e33bca6f-714f-44a3-bf0c-e99ddb526c67-istio-data" pod "e33bca6f-714f-44a3-bf0c-e99ddb526c67" (UID: "e33bca6f-714f-44a3-bf0c-e99ddb526c67"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:07:20.000044 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:19.999666 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e33bca6f-714f-44a3-bf0c-e99ddb526c67-istiod-ca-cert" pod "e33bca6f-714f-44a3-bf0c-e99ddb526c67" (UID: "e33bca6f-714f-44a3-bf0c-e99ddb526c67"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:07:20.000044 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:19.999689 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e33bca6f-714f-44a3-bf0c-e99ddb526c67-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:07:20.000044 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:19.999732 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e33bca6f-714f-44a3-bf0c-e99ddb526c67-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:07:20.000044 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:19.999746 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e33bca6f-714f-44a3-bf0c-e99ddb526c67-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:07:20.000044 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:19.999761 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e33bca6f-714f-44a3-bf0c-e99ddb526c67-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:07:20.001521 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:20.001491 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e33bca6f-714f-44a3-bf0c-e99ddb526c67-istio-podinfo" pod "e33bca6f-714f-44a3-bf0c-e99ddb526c67" (UID: "e33bca6f-714f-44a3-bf0c-e99ddb526c67"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:07:20.001643 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:20.001551 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e33bca6f-714f-44a3-bf0c-e99ddb526c67-kube-api-access-k7lmq" pod "e33bca6f-714f-44a3-bf0c-e99ddb526c67" (UID: "e33bca6f-714f-44a3-bf0c-e99ddb526c67"). InnerVolumeSpecName "kube-api-access-k7lmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:07:20.002793 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:20.002763 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e33bca6f-714f-44a3-bf0c-e99ddb526c67-istio-envoy" pod "e33bca6f-714f-44a3-bf0c-e99ddb526c67" (UID: "e33bca6f-714f-44a3-bf0c-e99ddb526c67"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:07:20.002899 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:20.002768 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e33bca6f-714f-44a3-bf0c-e99ddb526c67-istio-token" pod "e33bca6f-714f-44a3-bf0c-e99ddb526c67" (UID: "e33bca6f-714f-44a3-bf0c-e99ddb526c67"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:07:20.101098 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:20.101049 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e33bca6f-714f-44a3-bf0c-e99ddb526c67-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:07:20.101098 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:20.101083 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e33bca6f-714f-44a3-bf0c-e99ddb526c67-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:07:20.101098 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:20.101095 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7lmq\" (UniqueName: \"kubernetes.io/projected/e33bca6f-714f-44a3-bf0c-e99ddb526c67-kube-api-access-k7lmq\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:07:20.101098 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:20.101104 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e33bca6f-714f-44a3-bf0c-e99ddb526c67-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:07:20.101512 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:20.101113 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e33bca6f-714f-44a3-bf0c-e99ddb526c67-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:07:20.444253 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:20.444205 5019 generic.go:356] "Generic (PLEG): container finished" podID="e33bca6f-714f-44a3-bf0c-e99ddb526c67" containerID="f2238467d3faf57e776bc406adaef9495cf2fdc989606c413288bf9c07850aad" exitCode=0 Oct 31 05:07:20.444528 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:20.444278 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24" Oct 31 05:07:20.444528 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:20.444296 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24" event={"ID":"e33bca6f-714f-44a3-bf0c-e99ddb526c67","Type":"ContainerDied","Data":"f2238467d3faf57e776bc406adaef9495cf2fdc989606c413288bf9c07850aad"} Oct 31 05:07:20.444528 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:20.444408 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24" event={"ID":"e33bca6f-714f-44a3-bf0c-e99ddb526c67","Type":"ContainerDied","Data":"47b1cc2308eac241df4b4b3b4f3aa888d02c2ce218cb06152d22fdc8bfc28f4b"} Oct 31 05:07:20.444528 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:20.444443 5019 scope.go:122] "RemoveContainer" containerID="f2238467d3faf57e776bc406adaef9495cf2fdc989606c413288bf9c07850aad" Oct 31 05:07:20.458696 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:20.458671 5019 scope.go:122] "RemoveContainer" containerID="741d41265c76a6aa2bda535cd07bf196705321b2d1deb6018cd9dd01b467ea54" Oct 31 05:07:20.473578 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:20.473548 5019 scope.go:122] "RemoveContainer" containerID="f2238467d3faf57e776bc406adaef9495cf2fdc989606c413288bf9c07850aad" Oct 31 05:07:20.474021 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:07:20.473992 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2238467d3faf57e776bc406adaef9495cf2fdc989606c413288bf9c07850aad\": container with ID starting with f2238467d3faf57e776bc406adaef9495cf2fdc989606c413288bf9c07850aad not found: ID does not exist" containerID="f2238467d3faf57e776bc406adaef9495cf2fdc989606c413288bf9c07850aad" Oct 31 05:07:20.474131 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:20.474031 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2238467d3faf57e776bc406adaef9495cf2fdc989606c413288bf9c07850aad"} err="failed to get container status \"f2238467d3faf57e776bc406adaef9495cf2fdc989606c413288bf9c07850aad\": rpc error: code = NotFound desc = could not find container \"f2238467d3faf57e776bc406adaef9495cf2fdc989606c413288bf9c07850aad\": container with ID starting with f2238467d3faf57e776bc406adaef9495cf2fdc989606c413288bf9c07850aad not found: ID does not exist" Oct 31 05:07:20.474131 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:20.474053 5019 scope.go:122] "RemoveContainer" containerID="741d41265c76a6aa2bda535cd07bf196705321b2d1deb6018cd9dd01b467ea54" Oct 31 05:07:20.474426 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:07:20.474400 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"741d41265c76a6aa2bda535cd07bf196705321b2d1deb6018cd9dd01b467ea54\": container with ID starting with 741d41265c76a6aa2bda535cd07bf196705321b2d1deb6018cd9dd01b467ea54 not found: ID does not exist" containerID="741d41265c76a6aa2bda535cd07bf196705321b2d1deb6018cd9dd01b467ea54" Oct 31 05:07:20.474531 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:20.474439 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"741d41265c76a6aa2bda535cd07bf196705321b2d1deb6018cd9dd01b467ea54"} err="failed to get container status \"741d41265c76a6aa2bda535cd07bf196705321b2d1deb6018cd9dd01b467ea54\": rpc error: code = NotFound desc = could not find container \"741d41265c76a6aa2bda535cd07bf196705321b2d1deb6018cd9dd01b467ea54\": container with ID starting with 741d41265c76a6aa2bda535cd07bf196705321b2d1deb6018cd9dd01b467ea54 not found: ID does not exist" Oct 31 05:07:20.500094 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:20.500045 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24"] Oct 31 05:07:20.505210 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:20.505167 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c1fe26-openshift-default-88665c78c-87h24"] Oct 31 05:07:21.410246 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:21.410207 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e33bca6f-714f-44a3-bf0c-e99ddb526c67" path="/var/lib/kubelet/pods/e33bca6f-714f-44a3-bf0c-e99ddb526c67/volumes" Oct 31 05:07:22.393158 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:07:22.393115 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:07:24.029122 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:24.029078 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6190e705-ksd9m"] Oct 31 05:07:24.029934 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:24.029914 5019 cpu_manager.go:414] "containerMap: removing container" podUID="7e6bd271-4285-4390-a0f9-e2fcd5a855d4" containerName="istio-proxy" Oct 31 05:07:24.030003 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:24.029936 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e6bd271-4285-4390-a0f9-e2fcd5a855d4" containerName="istio-proxy" Oct 31 05:07:24.030003 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:24.029966 5019 cpu_manager.go:414] "containerMap: removing container" podUID="4068532b-36ea-4ef1-bb0f-366e10073c49" containerName="revoke-keys" Oct 31 05:07:24.030003 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:24.029972 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4068532b-36ea-4ef1-bb0f-366e10073c49" containerName="revoke-keys" Oct 31 05:07:24.030003 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:24.029989 5019 cpu_manager.go:414] "containerMap: removing container" podUID="e33bca6f-714f-44a3-bf0c-e99ddb526c67" containerName="istio-proxy" Oct 31 05:07:24.030003 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:24.029995 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e33bca6f-714f-44a3-bf0c-e99ddb526c67" containerName="istio-proxy" Oct 31 05:07:24.030169 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:24.030007 5019 cpu_manager.go:414] "containerMap: removing container" podUID="d22c47b5-d2a0-423e-a2b5-c83bf4afbf96" containerName="payload-processing" Oct 31 05:07:24.030169 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:24.030012 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d22c47b5-d2a0-423e-a2b5-c83bf4afbf96" containerName="payload-processing" Oct 31 05:07:24.030169 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:24.030027 5019 cpu_manager.go:414] "containerMap: removing container" podUID="e6d6891a-ccbc-4007-bbf3-96544ac865c2" containerName="istio-proxy" Oct 31 05:07:24.030169 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:24.030032 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6d6891a-ccbc-4007-bbf3-96544ac865c2" containerName="istio-proxy" Oct 31 05:07:24.030169 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:24.030048 5019 cpu_manager.go:414] "containerMap: removing container" podUID="e33bca6f-714f-44a3-bf0c-e99ddb526c67" containerName="istio-proxy" Oct 31 05:07:24.030169 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:24.030053 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e33bca6f-714f-44a3-bf0c-e99ddb526c67" containerName="istio-proxy" Oct 31 05:07:24.030169 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:24.030065 5019 cpu_manager.go:414] "containerMap: removing container" podUID="5da17e2b-6c07-4222-b09c-310cf0805666" containerName="payload-processing" Oct 31 05:07:24.030169 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:24.030071 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5da17e2b-6c07-4222-b09c-310cf0805666" containerName="payload-processing" Oct 31 05:07:24.030458 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:24.030173 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="5da17e2b-6c07-4222-b09c-310cf0805666" containerName="payload-processing" Oct 31 05:07:24.030458 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:24.030186 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e6bd271-4285-4390-a0f9-e2fcd5a855d4" containerName="istio-proxy" Oct 31 05:07:24.030458 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:24.030199 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="4068532b-36ea-4ef1-bb0f-366e10073c49" containerName="revoke-keys" Oct 31 05:07:24.030458 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:24.030207 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e33bca6f-714f-44a3-bf0c-e99ddb526c67" containerName="istio-proxy" Oct 31 05:07:24.030458 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:24.030213 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="d22c47b5-d2a0-423e-a2b5-c83bf4afbf96" containerName="payload-processing" Oct 31 05:07:24.030458 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:24.030219 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e6bd271-4285-4390-a0f9-e2fcd5a855d4" containerName="istio-proxy" Oct 31 05:07:24.030458 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:24.030227 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e33bca6f-714f-44a3-bf0c-e99ddb526c67" containerName="istio-proxy" Oct 31 05:07:24.035797 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:24.035768 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6190e705-ksd9m" Oct 31 05:07:24.053718 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:24.053682 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6190e705-ksd9m"] Oct 31 05:07:24.059220 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:24.059188 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:07:24.059480 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:24.059207 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:07:24.059480 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:24.059272 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:07:24.059480 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:24.059278 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 05:07:24.160353 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:24.160291 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bf2369f-ee6d-4a56-9abb-1a9204ea03ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6190e705-ksd9m\" (UID: \"4bf2369f-ee6d-4a56-9abb-1a9204ea03ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6190e705-ksd9m" Oct 31 05:07:24.160586 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:24.160385 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bf2369f-ee6d-4a56-9abb-1a9204ea03ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6190e705-ksd9m\" (UID: \"4bf2369f-ee6d-4a56-9abb-1a9204ea03ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6190e705-ksd9m" Oct 31 05:07:24.160685 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:24.160658 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htt2s\" (UniqueName: \"kubernetes.io/projected/4bf2369f-ee6d-4a56-9abb-1a9204ea03ef-kube-api-access-htt2s\") pod \"maas-api-revoke-sub-6190e705-ksd9m\" (UID: \"4bf2369f-ee6d-4a56-9abb-1a9204ea03ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6190e705-ksd9m" Oct 31 05:07:24.262624 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:24.262580 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htt2s\" (UniqueName: \"kubernetes.io/projected/4bf2369f-ee6d-4a56-9abb-1a9204ea03ef-kube-api-access-htt2s\") pod \"maas-api-revoke-sub-6190e705-ksd9m\" (UID: \"4bf2369f-ee6d-4a56-9abb-1a9204ea03ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6190e705-ksd9m" Oct 31 05:07:24.262839 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:24.262735 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bf2369f-ee6d-4a56-9abb-1a9204ea03ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6190e705-ksd9m\" (UID: \"4bf2369f-ee6d-4a56-9abb-1a9204ea03ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6190e705-ksd9m" Oct 31 05:07:24.262839 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:24.262769 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bf2369f-ee6d-4a56-9abb-1a9204ea03ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6190e705-ksd9m\" (UID: \"4bf2369f-ee6d-4a56-9abb-1a9204ea03ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6190e705-ksd9m" Oct 31 05:07:24.263703 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:24.263670 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bf2369f-ee6d-4a56-9abb-1a9204ea03ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6190e705-ksd9m\" (UID: \"4bf2369f-ee6d-4a56-9abb-1a9204ea03ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6190e705-ksd9m" Oct 31 05:07:24.265288 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:24.265264 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bf2369f-ee6d-4a56-9abb-1a9204ea03ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6190e705-ksd9m\" (UID: \"4bf2369f-ee6d-4a56-9abb-1a9204ea03ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6190e705-ksd9m" Oct 31 05:07:24.286279 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:24.286197 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htt2s\" (UniqueName: \"kubernetes.io/projected/4bf2369f-ee6d-4a56-9abb-1a9204ea03ef-kube-api-access-htt2s\") pod \"maas-api-revoke-sub-6190e705-ksd9m\" (UID: \"4bf2369f-ee6d-4a56-9abb-1a9204ea03ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6190e705-ksd9m" Oct 31 05:07:24.352142 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:24.352105 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6190e705-ksd9m" Oct 31 05:07:24.669010 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:07:24.668974 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bf2369f_ee6d_4a56_9abb_1a9204ea03ef.slice/crio-bdeb265ae60d1f3391c387778d123ad96af37c324c522961d9e4e6767b095927 WatchSource:0}: Error finding container bdeb265ae60d1f3391c387778d123ad96af37c324c522961d9e4e6767b095927: Status 404 returned error can't find the container with id bdeb265ae60d1f3391c387778d123ad96af37c324c522961d9e4e6767b095927 Oct 31 05:07:24.669816 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:24.669773 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6190e705-ksd9m"] Oct 31 05:07:25.500665 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:25.500616 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6190e705-ksd9m" event={"ID":"4bf2369f-ee6d-4a56-9abb-1a9204ea03ef","Type":"ContainerStarted","Data":"03a34729a0c00280813365d60408ec371bf4522d4294ed11f581f221a4eeff67"} Oct 31 05:07:25.501208 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:25.500692 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6190e705-ksd9m" event={"ID":"4bf2369f-ee6d-4a56-9abb-1a9204ea03ef","Type":"ContainerStarted","Data":"bdeb265ae60d1f3391c387778d123ad96af37c324c522961d9e4e6767b095927"} Oct 31 05:07:25.544303 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:25.544236 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6190e705-ksd9m" podStartSLOduration=1.5442200069999998 podStartE2EDuration="1.544220007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:07:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:07:25.540185518 +0000 UTC m=+3800.296523559" watchObservedRunningTime="2026-10-31 05:07:25.544220007 +0000 UTC m=+3800.300558046" Oct 31 05:07:26.513787 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:26.513736 5019 generic.go:356] "Generic (PLEG): container finished" podID="4bf2369f-ee6d-4a56-9abb-1a9204ea03ef" containerID="03a34729a0c00280813365d60408ec371bf4522d4294ed11f581f221a4eeff67" exitCode=0 Oct 31 05:07:26.514358 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:26.513869 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6190e705-ksd9m" event={"ID":"4bf2369f-ee6d-4a56-9abb-1a9204ea03ef","Type":"ContainerDied","Data":"03a34729a0c00280813365d60408ec371bf4522d4294ed11f581f221a4eeff67"} Oct 31 05:07:27.714719 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:27.714683 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6190e705-ksd9m" Oct 31 05:07:27.831698 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:27.831575 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bf2369f-ee6d-4a56-9abb-1a9204ea03ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bf2369f-ee6d-4a56-9abb-1a9204ea03ef-maas-api-openshift-ca\") pod \"4bf2369f-ee6d-4a56-9abb-1a9204ea03ef\" (UID: \"4bf2369f-ee6d-4a56-9abb-1a9204ea03ef\") " Oct 31 05:07:27.831900 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:27.831710 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bf2369f-ee6d-4a56-9abb-1a9204ea03ef-kube-api-access-htt2s\" (UniqueName: \"kubernetes.io/projected/4bf2369f-ee6d-4a56-9abb-1a9204ea03ef-kube-api-access-htt2s\") pod \"4bf2369f-ee6d-4a56-9abb-1a9204ea03ef\" (UID: \"4bf2369f-ee6d-4a56-9abb-1a9204ea03ef\") " Oct 31 05:07:27.831900 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:27.831839 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bf2369f-ee6d-4a56-9abb-1a9204ea03ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bf2369f-ee6d-4a56-9abb-1a9204ea03ef-maas-api-xks-ca\") pod \"4bf2369f-ee6d-4a56-9abb-1a9204ea03ef\" (UID: \"4bf2369f-ee6d-4a56-9abb-1a9204ea03ef\") " Oct 31 05:07:27.832038 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:27.832021 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bf2369f-ee6d-4a56-9abb-1a9204ea03ef-maas-api-openshift-ca" pod "4bf2369f-ee6d-4a56-9abb-1a9204ea03ef" (UID: "4bf2369f-ee6d-4a56-9abb-1a9204ea03ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:07:27.832472 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:27.832447 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bf2369f-ee6d-4a56-9abb-1a9204ea03ef-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:07:27.834209 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:27.834171 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bf2369f-ee6d-4a56-9abb-1a9204ea03ef-kube-api-access-htt2s" pod "4bf2369f-ee6d-4a56-9abb-1a9204ea03ef" (UID: "4bf2369f-ee6d-4a56-9abb-1a9204ea03ef"). InnerVolumeSpecName "kube-api-access-htt2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:07:27.834209 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:27.834191 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bf2369f-ee6d-4a56-9abb-1a9204ea03ef-maas-api-xks-ca" pod "4bf2369f-ee6d-4a56-9abb-1a9204ea03ef" (UID: "4bf2369f-ee6d-4a56-9abb-1a9204ea03ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:07:27.934480 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:27.934428 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htt2s\" (UniqueName: \"kubernetes.io/projected/4bf2369f-ee6d-4a56-9abb-1a9204ea03ef-kube-api-access-htt2s\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:07:27.934480 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:27.934473 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bf2369f-ee6d-4a56-9abb-1a9204ea03ef-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:07:28.541272 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:28.541232 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6190e705-ksd9m" Oct 31 05:07:28.541703 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:28.541443 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6190e705-ksd9m" event={"ID":"4bf2369f-ee6d-4a56-9abb-1a9204ea03ef","Type":"ContainerDied","Data":"bdeb265ae60d1f3391c387778d123ad96af37c324c522961d9e4e6767b095927"} Oct 31 05:07:28.541827 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:28.541739 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bdeb265ae60d1f3391c387778d123ad96af37c324c522961d9e4e6767b095927" Oct 31 05:07:35.033158 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:35.033114 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b9214b0-zmjk5"] Oct 31 05:07:35.045307 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:35.045261 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b9214b0-zmjk5"] Oct 31 05:07:35.411914 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:35.411829 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="856f1832-9e23-4a5d-ace0-cbd6ac20d469" path="/var/lib/kubelet/pods/856f1832-9e23-4a5d-ace0-cbd6ac20d469/volumes" Oct 31 05:07:43.121371 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:43.120450 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-pghv7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 05:07:43.121371 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:43.120547 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7" podUID="e0ccc03e-0e5e-40b7-8a40-718f199dd747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 05:07:44.276190 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:44.276152 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff"] Oct 31 05:07:44.276820 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:44.276428 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff" podUID="a9eb4117-200f-4720-9047-a79a3b227cb3" containerName="istio-proxy" containerID="cri-o://1f2e5804df9faef55dc8046ab52ebda9b96754213f699733811699119b1f1c24" gracePeriod=30 Oct 31 05:07:44.353192 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:44.353151 5019 patch_prober.go:29] interesting pod/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 05:07:44.353386 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:44.353216 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff" podUID="a9eb4117-200f-4720-9047-a79a3b227cb3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 05:07:46.033041 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:46.032926 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d55c772-7w6f5"] Oct 31 05:07:46.043112 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:46.043074 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d55c772-7w6f5"] Oct 31 05:07:47.412765 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:47.412722 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cf9f031-dd87-4964-8038-a8dc3ecb8917" path="/var/lib/kubelet/pods/0cf9f031-dd87-4964-8038-a8dc3ecb8917/volumes" Oct 31 05:07:49.584964 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:49.584934 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff" Oct 31 05:07:49.637896 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:49.637862 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9eb4117-200f-4720-9047-a79a3b227cb3-istio-token\" (UniqueName: \"kubernetes.io/projected/a9eb4117-200f-4720-9047-a79a3b227cb3-istio-token\") pod \"a9eb4117-200f-4720-9047-a79a3b227cb3\" (UID: \"a9eb4117-200f-4720-9047-a79a3b227cb3\") " Oct 31 05:07:49.637896 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:49.637913 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9eb4117-200f-4720-9047-a79a3b227cb3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9eb4117-200f-4720-9047-a79a3b227cb3-istio-data\") pod \"a9eb4117-200f-4720-9047-a79a3b227cb3\" (UID: \"a9eb4117-200f-4720-9047-a79a3b227cb3\") " Oct 31 05:07:49.638184 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:49.637952 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9eb4117-200f-4720-9047-a79a3b227cb3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9eb4117-200f-4720-9047-a79a3b227cb3-istiod-ca-cert\") pod \"a9eb4117-200f-4720-9047-a79a3b227cb3\" (UID: \"a9eb4117-200f-4720-9047-a79a3b227cb3\") " Oct 31 05:07:49.638184 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:49.637978 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9eb4117-200f-4720-9047-a79a3b227cb3-kube-api-access-g7f8f\" (UniqueName: \"kubernetes.io/projected/a9eb4117-200f-4720-9047-a79a3b227cb3-kube-api-access-g7f8f\") pod \"a9eb4117-200f-4720-9047-a79a3b227cb3\" (UID: \"a9eb4117-200f-4720-9047-a79a3b227cb3\") " Oct 31 05:07:49.638184 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:49.638049 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9eb4117-200f-4720-9047-a79a3b227cb3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9eb4117-200f-4720-9047-a79a3b227cb3-istio-envoy\") pod \"a9eb4117-200f-4720-9047-a79a3b227cb3\" (UID: \"a9eb4117-200f-4720-9047-a79a3b227cb3\") " Oct 31 05:07:49.638184 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:49.638075 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9eb4117-200f-4720-9047-a79a3b227cb3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9eb4117-200f-4720-9047-a79a3b227cb3-workload-socket\") pod \"a9eb4117-200f-4720-9047-a79a3b227cb3\" (UID: \"a9eb4117-200f-4720-9047-a79a3b227cb3\") " Oct 31 05:07:49.638184 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:49.638115 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9eb4117-200f-4720-9047-a79a3b227cb3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9eb4117-200f-4720-9047-a79a3b227cb3-credential-socket\") pod \"a9eb4117-200f-4720-9047-a79a3b227cb3\" (UID: \"a9eb4117-200f-4720-9047-a79a3b227cb3\") " Oct 31 05:07:49.638184 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:49.638158 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9eb4117-200f-4720-9047-a79a3b227cb3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9eb4117-200f-4720-9047-a79a3b227cb3-workload-certs\") pod \"a9eb4117-200f-4720-9047-a79a3b227cb3\" (UID: \"a9eb4117-200f-4720-9047-a79a3b227cb3\") " Oct 31 05:07:49.638577 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:49.638195 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9eb4117-200f-4720-9047-a79a3b227cb3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9eb4117-200f-4720-9047-a79a3b227cb3-istio-podinfo\") pod \"a9eb4117-200f-4720-9047-a79a3b227cb3\" (UID: \"a9eb4117-200f-4720-9047-a79a3b227cb3\") " Oct 31 05:07:49.638577 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:49.638304 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9eb4117-200f-4720-9047-a79a3b227cb3-istio-data" pod "a9eb4117-200f-4720-9047-a79a3b227cb3" (UID: "a9eb4117-200f-4720-9047-a79a3b227cb3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:07:49.639219 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:49.638711 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9eb4117-200f-4720-9047-a79a3b227cb3-istiod-ca-cert" pod "a9eb4117-200f-4720-9047-a79a3b227cb3" (UID: "a9eb4117-200f-4720-9047-a79a3b227cb3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:07:49.639219 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:49.639045 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9eb4117-200f-4720-9047-a79a3b227cb3-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:07:49.639219 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:49.639068 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9eb4117-200f-4720-9047-a79a3b227cb3-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:07:49.639219 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:49.639128 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9eb4117-200f-4720-9047-a79a3b227cb3-workload-socket" pod "a9eb4117-200f-4720-9047-a79a3b227cb3" (UID: "a9eb4117-200f-4720-9047-a79a3b227cb3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:07:49.639219 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:49.639151 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9eb4117-200f-4720-9047-a79a3b227cb3-workload-certs" pod "a9eb4117-200f-4720-9047-a79a3b227cb3" (UID: "a9eb4117-200f-4720-9047-a79a3b227cb3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:07:49.639219 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:49.639195 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9eb4117-200f-4720-9047-a79a3b227cb3-credential-socket" pod "a9eb4117-200f-4720-9047-a79a3b227cb3" (UID: "a9eb4117-200f-4720-9047-a79a3b227cb3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:07:49.650932 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:49.650879 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9eb4117-200f-4720-9047-a79a3b227cb3-kube-api-access-g7f8f" pod "a9eb4117-200f-4720-9047-a79a3b227cb3" (UID: "a9eb4117-200f-4720-9047-a79a3b227cb3"). InnerVolumeSpecName "kube-api-access-g7f8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:07:49.659569 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:49.659498 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9eb4117-200f-4720-9047-a79a3b227cb3-istio-token" pod "a9eb4117-200f-4720-9047-a79a3b227cb3" (UID: "a9eb4117-200f-4720-9047-a79a3b227cb3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:07:49.659569 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:49.659505 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9eb4117-200f-4720-9047-a79a3b227cb3-istio-envoy" pod "a9eb4117-200f-4720-9047-a79a3b227cb3" (UID: "a9eb4117-200f-4720-9047-a79a3b227cb3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:07:49.660062 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:49.660036 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9eb4117-200f-4720-9047-a79a3b227cb3-istio-podinfo" pod "a9eb4117-200f-4720-9047-a79a3b227cb3" (UID: "a9eb4117-200f-4720-9047-a79a3b227cb3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:07:49.740898 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:49.740860 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9eb4117-200f-4720-9047-a79a3b227cb3-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:07:49.740898 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:49.740893 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7f8f\" (UniqueName: \"kubernetes.io/projected/a9eb4117-200f-4720-9047-a79a3b227cb3-kube-api-access-g7f8f\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:07:49.740898 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:49.740905 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9eb4117-200f-4720-9047-a79a3b227cb3-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:07:49.741155 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:49.740915 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9eb4117-200f-4720-9047-a79a3b227cb3-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:07:49.741155 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:49.740923 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9eb4117-200f-4720-9047-a79a3b227cb3-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:07:49.741155 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:49.740932 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9eb4117-200f-4720-9047-a79a3b227cb3-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:07:49.741155 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:49.740941 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9eb4117-200f-4720-9047-a79a3b227cb3-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:07:49.822643 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:49.822558 5019 generic.go:356] "Generic (PLEG): container finished" podID="a9eb4117-200f-4720-9047-a79a3b227cb3" containerID="1f2e5804df9faef55dc8046ab52ebda9b96754213f699733811699119b1f1c24" exitCode=0 Oct 31 05:07:49.822643 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:49.822610 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff" Oct 31 05:07:49.822835 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:49.822638 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff" event={"ID":"a9eb4117-200f-4720-9047-a79a3b227cb3","Type":"ContainerDied","Data":"1f2e5804df9faef55dc8046ab52ebda9b96754213f699733811699119b1f1c24"} Oct 31 05:07:49.822835 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:49.822700 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff" event={"ID":"a9eb4117-200f-4720-9047-a79a3b227cb3","Type":"ContainerDied","Data":"36280872daaa89fed60b4fb80fad43d08473fcc16d8b5247b62a4d73857ff2d7"} Oct 31 05:07:49.822835 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:49.822730 5019 scope.go:122] "RemoveContainer" containerID="1f2e5804df9faef55dc8046ab52ebda9b96754213f699733811699119b1f1c24" Oct 31 05:07:49.836203 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:49.836179 5019 scope.go:122] "RemoveContainer" containerID="9812edb363bf790a0f082efc6ce5626754191c3c58ce55fa197a367921e130a7" Oct 31 05:07:49.848566 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:49.848535 5019 scope.go:122] "RemoveContainer" containerID="1f2e5804df9faef55dc8046ab52ebda9b96754213f699733811699119b1f1c24" Oct 31 05:07:49.848907 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:07:49.848883 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f2e5804df9faef55dc8046ab52ebda9b96754213f699733811699119b1f1c24\": container with ID starting with 1f2e5804df9faef55dc8046ab52ebda9b96754213f699733811699119b1f1c24 not found: ID does not exist" containerID="1f2e5804df9faef55dc8046ab52ebda9b96754213f699733811699119b1f1c24" Oct 31 05:07:49.849009 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:49.848917 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f2e5804df9faef55dc8046ab52ebda9b96754213f699733811699119b1f1c24"} err="failed to get container status \"1f2e5804df9faef55dc8046ab52ebda9b96754213f699733811699119b1f1c24\": rpc error: code = NotFound desc = could not find container \"1f2e5804df9faef55dc8046ab52ebda9b96754213f699733811699119b1f1c24\": container with ID starting with 1f2e5804df9faef55dc8046ab52ebda9b96754213f699733811699119b1f1c24 not found: ID does not exist" Oct 31 05:07:49.849009 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:49.848966 5019 scope.go:122] "RemoveContainer" containerID="9812edb363bf790a0f082efc6ce5626754191c3c58ce55fa197a367921e130a7" Oct 31 05:07:49.849299 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:07:49.849277 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9812edb363bf790a0f082efc6ce5626754191c3c58ce55fa197a367921e130a7\": container with ID starting with 9812edb363bf790a0f082efc6ce5626754191c3c58ce55fa197a367921e130a7 not found: ID does not exist" containerID="9812edb363bf790a0f082efc6ce5626754191c3c58ce55fa197a367921e130a7" Oct 31 05:07:49.849451 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:49.849306 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9812edb363bf790a0f082efc6ce5626754191c3c58ce55fa197a367921e130a7"} err="failed to get container status \"9812edb363bf790a0f082efc6ce5626754191c3c58ce55fa197a367921e130a7\": rpc error: code = NotFound desc = could not find container \"9812edb363bf790a0f082efc6ce5626754191c3c58ce55fa197a367921e130a7\": container with ID starting with 9812edb363bf790a0f082efc6ce5626754191c3c58ce55fa197a367921e130a7 not found: ID does not exist" Oct 31 05:07:49.878447 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:49.878406 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff"] Oct 31 05:07:49.883238 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:49.883200 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-47e9af-openshift-default-5b7fdd8749-wtjff"] Oct 31 05:07:51.411765 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:07:51.411724 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9eb4117-200f-4720-9047-a79a3b227cb3" path="/var/lib/kubelet/pods/a9eb4117-200f-4720-9047-a79a3b227cb3/volumes" Oct 31 05:07:52.399270 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:07:52.399232 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:08:03.085358 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:03.084785 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-67f7j"] Oct 31 05:08:03.105790 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:03.105124 5019 cpu_manager.go:414] "containerMap: removing container" podUID="a9eb4117-200f-4720-9047-a79a3b227cb3" containerName="istio-proxy" Oct 31 05:08:03.106285 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:03.106258 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9eb4117-200f-4720-9047-a79a3b227cb3" containerName="istio-proxy" Oct 31 05:08:03.106518 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:03.106502 5019 cpu_manager.go:414] "containerMap: removing container" podUID="a9eb4117-200f-4720-9047-a79a3b227cb3" containerName="istio-proxy" Oct 31 05:08:03.114458 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:03.114418 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9eb4117-200f-4720-9047-a79a3b227cb3" containerName="istio-proxy" Oct 31 05:08:03.114788 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:03.114766 5019 cpu_manager.go:414] "containerMap: removing container" podUID="4bf2369f-ee6d-4a56-9abb-1a9204ea03ef" containerName="revoke-keys" Oct 31 05:08:03.114923 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:03.114908 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bf2369f-ee6d-4a56-9abb-1a9204ea03ef" containerName="revoke-keys" Oct 31 05:08:03.115096 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:03.115080 5019 cpu_manager.go:414] "containerMap: removing container" podUID="7e6bd271-4285-4390-a0f9-e2fcd5a855d4" containerName="istio-proxy" Oct 31 05:08:03.115204 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:03.115191 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e6bd271-4285-4390-a0f9-e2fcd5a855d4" containerName="istio-proxy" Oct 31 05:08:03.115718 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:03.115605 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9eb4117-200f-4720-9047-a79a3b227cb3" containerName="istio-proxy" Oct 31 05:08:03.115876 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:03.115860 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9eb4117-200f-4720-9047-a79a3b227cb3" containerName="istio-proxy" Oct 31 05:08:03.115987 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:03.115976 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bf2369f-ee6d-4a56-9abb-1a9204ea03ef" containerName="revoke-keys" Oct 31 05:08:03.138686 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:03.138627 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-67f7j" Oct 31 05:08:03.144616 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:03.144576 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-67f7j"] Oct 31 05:08:03.380437 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:03.366756 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9573de57-8748-44eb-8949-b55b7dca0043-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-67f7j\" (UID: \"9573de57-8748-44eb-8949-b55b7dca0043\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-67f7j" Oct 31 05:08:03.380437 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:03.366817 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqhlr\" (UniqueName: \"kubernetes.io/projected/9573de57-8748-44eb-8949-b55b7dca0043-kube-api-access-wqhlr\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-67f7j\" (UID: \"9573de57-8748-44eb-8949-b55b7dca0043\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-67f7j" Oct 31 05:08:03.380437 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:03.366890 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9573de57-8748-44eb-8949-b55b7dca0043-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-67f7j\" (UID: \"9573de57-8748-44eb-8949-b55b7dca0043\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-67f7j" Oct 31 05:08:03.380437 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:03.366926 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9573de57-8748-44eb-8949-b55b7dca0043-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-67f7j\" (UID: \"9573de57-8748-44eb-8949-b55b7dca0043\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-67f7j" Oct 31 05:08:03.380437 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:03.366967 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9573de57-8748-44eb-8949-b55b7dca0043-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-67f7j\" (UID: \"9573de57-8748-44eb-8949-b55b7dca0043\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-67f7j" Oct 31 05:08:03.380437 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:03.367036 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9573de57-8748-44eb-8949-b55b7dca0043-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-67f7j\" (UID: \"9573de57-8748-44eb-8949-b55b7dca0043\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-67f7j" Oct 31 05:08:03.380437 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:03.367098 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9573de57-8748-44eb-8949-b55b7dca0043-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-67f7j\" (UID: \"9573de57-8748-44eb-8949-b55b7dca0043\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-67f7j" Oct 31 05:08:03.380437 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:03.367125 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9573de57-8748-44eb-8949-b55b7dca0043-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-67f7j\" (UID: \"9573de57-8748-44eb-8949-b55b7dca0043\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-67f7j" Oct 31 05:08:03.380437 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:03.367158 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9573de57-8748-44eb-8949-b55b7dca0043-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-67f7j\" (UID: \"9573de57-8748-44eb-8949-b55b7dca0043\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-67f7j" Oct 31 05:08:03.468204 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:03.468143 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9573de57-8748-44eb-8949-b55b7dca0043-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-67f7j\" (UID: \"9573de57-8748-44eb-8949-b55b7dca0043\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-67f7j" Oct 31 05:08:03.468499 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:03.468476 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqhlr\" (UniqueName: \"kubernetes.io/projected/9573de57-8748-44eb-8949-b55b7dca0043-kube-api-access-wqhlr\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-67f7j\" (UID: \"9573de57-8748-44eb-8949-b55b7dca0043\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-67f7j" Oct 31 05:08:03.468686 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:03.468669 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9573de57-8748-44eb-8949-b55b7dca0043-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-67f7j\" (UID: \"9573de57-8748-44eb-8949-b55b7dca0043\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-67f7j" Oct 31 05:08:03.468806 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:03.468793 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9573de57-8748-44eb-8949-b55b7dca0043-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-67f7j\" (UID: \"9573de57-8748-44eb-8949-b55b7dca0043\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-67f7j" Oct 31 05:08:03.468915 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:03.468903 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9573de57-8748-44eb-8949-b55b7dca0043-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-67f7j\" (UID: \"9573de57-8748-44eb-8949-b55b7dca0043\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-67f7j" Oct 31 05:08:03.469085 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:03.469070 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9573de57-8748-44eb-8949-b55b7dca0043-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-67f7j\" (UID: \"9573de57-8748-44eb-8949-b55b7dca0043\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-67f7j" Oct 31 05:08:03.469244 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:03.469230 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9573de57-8748-44eb-8949-b55b7dca0043-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-67f7j\" (UID: \"9573de57-8748-44eb-8949-b55b7dca0043\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-67f7j" Oct 31 05:08:03.469386 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:03.469367 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9573de57-8748-44eb-8949-b55b7dca0043-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-67f7j\" (UID: \"9573de57-8748-44eb-8949-b55b7dca0043\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-67f7j" Oct 31 05:08:03.493477 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:03.493437 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9573de57-8748-44eb-8949-b55b7dca0043-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-67f7j\" (UID: \"9573de57-8748-44eb-8949-b55b7dca0043\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-67f7j" Oct 31 05:08:03.531357 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:03.471372 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9573de57-8748-44eb-8949-b55b7dca0043-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-67f7j\" (UID: \"9573de57-8748-44eb-8949-b55b7dca0043\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-67f7j" Oct 31 05:08:03.554597 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:03.479887 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9573de57-8748-44eb-8949-b55b7dca0043-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-67f7j\" (UID: \"9573de57-8748-44eb-8949-b55b7dca0043\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-67f7j" Oct 31 05:08:03.554835 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:03.480822 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9573de57-8748-44eb-8949-b55b7dca0043-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-67f7j\" (UID: \"9573de57-8748-44eb-8949-b55b7dca0043\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-67f7j" Oct 31 05:08:03.554966 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:03.481067 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9573de57-8748-44eb-8949-b55b7dca0043-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-67f7j\" (UID: \"9573de57-8748-44eb-8949-b55b7dca0043\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-67f7j" Oct 31 05:08:03.555063 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:03.487351 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9573de57-8748-44eb-8949-b55b7dca0043-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-67f7j\" (UID: \"9573de57-8748-44eb-8949-b55b7dca0043\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-67f7j" Oct 31 05:08:03.555157 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:03.496271 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9573de57-8748-44eb-8949-b55b7dca0043-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-67f7j\" (UID: \"9573de57-8748-44eb-8949-b55b7dca0043\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-67f7j" Oct 31 05:08:03.555263 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:03.478997 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9573de57-8748-44eb-8949-b55b7dca0043-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-67f7j\" (UID: \"9573de57-8748-44eb-8949-b55b7dca0043\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-67f7j" Oct 31 05:08:03.555390 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:03.524191 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqhlr\" (UniqueName: \"kubernetes.io/projected/9573de57-8748-44eb-8949-b55b7dca0043-kube-api-access-wqhlr\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-67f7j\" (UID: \"9573de57-8748-44eb-8949-b55b7dca0043\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-67f7j" Oct 31 05:08:03.555484 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:03.553955 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9573de57-8748-44eb-8949-b55b7dca0043-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-67f7j\" (UID: \"9573de57-8748-44eb-8949-b55b7dca0043\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-67f7j" Oct 31 05:08:03.625964 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:03.625623 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-67f7j" Oct 31 05:08:04.074765 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:04.074723 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-67f7j"] Oct 31 05:08:04.075404 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:08:04.075375 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9573de57_8748_44eb_8949_b55b7dca0043.slice/crio-02f23f2ed0232e5ebbbbd023e8d938dc5592d0d86a31cc5164163399c5d2fbef WatchSource:0}: Error finding container 02f23f2ed0232e5ebbbbd023e8d938dc5592d0d86a31cc5164163399c5d2fbef: Status 404 returned error can't find the container with id 02f23f2ed0232e5ebbbbd023e8d938dc5592d0d86a31cc5164163399c5d2fbef Oct 31 05:08:04.079101 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:04.079053 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:08:04.079209 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:04.079156 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:08:04.079209 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:04.079200 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:08:04.120221 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:04.120165 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-67f7j" event={"ID":"9573de57-8748-44eb-8949-b55b7dca0043","Type":"ContainerStarted","Data":"02f23f2ed0232e5ebbbbd023e8d938dc5592d0d86a31cc5164163399c5d2fbef"} Oct 31 05:08:05.137237 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:05.137189 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-67f7j" event={"ID":"9573de57-8748-44eb-8949-b55b7dca0043","Type":"ContainerStarted","Data":"e446e225b4891af314659175ba11f9a40c9491146968b525a8fc11608e13f303"} Oct 31 05:08:05.216078 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:05.216015 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-67f7j" podStartSLOduration=2.215998131 podStartE2EDuration="2.215998131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:08:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:08:05.200832346 +0000 UTC m=+3839.957170386" watchObservedRunningTime="2026-10-31 05:08:05.215998131 +0000 UTC m=+3839.972336171" Oct 31 05:08:05.626646 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:05.626598 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-67f7j" Oct 31 05:08:05.628607 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:05.628579 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-67f7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.174:15021/healthz/ready\": dial tcp 10.132.1.174:15021: connect: connection refused" start-of-body= Oct 31 05:08:05.628770 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:05.628631 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-67f7j" podUID="9573de57-8748-44eb-8949-b55b7dca0043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.174:15021/healthz/ready\": dial tcp 10.132.1.174:15021: connect: connection refused" Oct 31 05:08:06.627488 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:06.627448 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-67f7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.174:15021/healthz/ready\": dial tcp 10.132.1.174:15021: connect: connection refused" start-of-body= Oct 31 05:08:06.627945 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:06.627506 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-67f7j" podUID="9573de57-8748-44eb-8949-b55b7dca0043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.174:15021/healthz/ready\": dial tcp 10.132.1.174:15021: connect: connection refused" Oct 31 05:08:07.627219 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:07.627176 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-67f7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.174:15021/healthz/ready\": dial tcp 10.132.1.174:15021: connect: connection refused" start-of-body= Oct 31 05:08:07.627462 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:07.627261 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-67f7j" podUID="9573de57-8748-44eb-8949-b55b7dca0043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.174:15021/healthz/ready\": dial tcp 10.132.1.174:15021: connect: connection refused" Oct 31 05:08:08.627452 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:08.627404 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-67f7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.174:15021/healthz/ready\": dial tcp 10.132.1.174:15021: connect: connection refused" start-of-body= Oct 31 05:08:08.627933 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:08.627473 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-67f7j" podUID="9573de57-8748-44eb-8949-b55b7dca0043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.174:15021/healthz/ready\": dial tcp 10.132.1.174:15021: connect: connection refused" Oct 31 05:08:09.627009 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:09.626972 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-67f7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.174:15021/healthz/ready\": dial tcp 10.132.1.174:15021: connect: connection refused" start-of-body= Oct 31 05:08:09.627180 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:09.627031 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-67f7j" podUID="9573de57-8748-44eb-8949-b55b7dca0043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.174:15021/healthz/ready\": dial tcp 10.132.1.174:15021: connect: connection refused" Oct 31 05:08:10.626610 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:10.626571 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-67f7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.174:15021/healthz/ready\": dial tcp 10.132.1.174:15021: connect: connection refused" start-of-body= Oct 31 05:08:10.627052 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:10.626627 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-67f7j" podUID="9573de57-8748-44eb-8949-b55b7dca0043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.174:15021/healthz/ready\": dial tcp 10.132.1.174:15021: connect: connection refused" Oct 31 05:08:11.626826 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:11.626786 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-67f7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.174:15021/healthz/ready\": dial tcp 10.132.1.174:15021: connect: connection refused" start-of-body= Oct 31 05:08:11.627241 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:11.626850 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-67f7j" podUID="9573de57-8748-44eb-8949-b55b7dca0043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.174:15021/healthz/ready\": dial tcp 10.132.1.174:15021: connect: connection refused" Oct 31 05:08:13.130888 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:13.130852 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-67f7j" Oct 31 05:08:13.131467 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:13.131362 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-67f7j" Oct 31 05:08:13.132691 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:13.132663 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-67f7j" Oct 31 05:08:21.694767 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:21.694728 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml"] Oct 31 05:08:21.702261 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:21.702227 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml" Oct 31 05:08:21.726527 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:21.726481 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml"] Oct 31 05:08:21.730477 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:21.730448 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-51cc4ce6-openshift-default-dockercfg-s8jd5\"" Oct 31 05:08:21.841406 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:21.833282 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3361c865-a311-4ae0-a040-e3ffade9e514-workload-socket\") pod \"e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml\" (UID: \"3361c865-a311-4ae0-a040-e3ffade9e514\") " pod="openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml" Oct 31 05:08:21.841406 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:21.833416 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3361c865-a311-4ae0-a040-e3ffade9e514-istio-envoy\") pod \"e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml\" (UID: \"3361c865-a311-4ae0-a040-e3ffade9e514\") " pod="openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml" Oct 31 05:08:21.841406 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:21.833470 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3361c865-a311-4ae0-a040-e3ffade9e514-istio-data\") pod \"e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml\" (UID: \"3361c865-a311-4ae0-a040-e3ffade9e514\") " pod="openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml" Oct 31 05:08:21.841406 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:21.833503 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3361c865-a311-4ae0-a040-e3ffade9e514-workload-certs\") pod \"e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml\" (UID: \"3361c865-a311-4ae0-a040-e3ffade9e514\") " pod="openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml" Oct 31 05:08:21.841406 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:21.833529 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3361c865-a311-4ae0-a040-e3ffade9e514-istio-podinfo\") pod \"e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml\" (UID: \"3361c865-a311-4ae0-a040-e3ffade9e514\") " pod="openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml" Oct 31 05:08:21.841406 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:21.833610 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcfj9\" (UniqueName: \"kubernetes.io/projected/3361c865-a311-4ae0-a040-e3ffade9e514-kube-api-access-vcfj9\") pod \"e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml\" (UID: \"3361c865-a311-4ae0-a040-e3ffade9e514\") " pod="openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml" Oct 31 05:08:21.841406 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:21.833652 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3361c865-a311-4ae0-a040-e3ffade9e514-istio-token\") pod \"e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml\" (UID: \"3361c865-a311-4ae0-a040-e3ffade9e514\") " pod="openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml" Oct 31 05:08:21.841406 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:21.833732 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3361c865-a311-4ae0-a040-e3ffade9e514-credential-socket\") pod \"e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml\" (UID: \"3361c865-a311-4ae0-a040-e3ffade9e514\") " pod="openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml" Oct 31 05:08:21.841406 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:21.833833 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3361c865-a311-4ae0-a040-e3ffade9e514-istiod-ca-cert\") pod \"e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml\" (UID: \"3361c865-a311-4ae0-a040-e3ffade9e514\") " pod="openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml" Oct 31 05:08:21.938355 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:21.935268 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3361c865-a311-4ae0-a040-e3ffade9e514-credential-socket\") pod \"e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml\" (UID: \"3361c865-a311-4ae0-a040-e3ffade9e514\") " pod="openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml" Oct 31 05:08:21.938355 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:21.935415 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3361c865-a311-4ae0-a040-e3ffade9e514-istiod-ca-cert\") pod \"e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml\" (UID: \"3361c865-a311-4ae0-a040-e3ffade9e514\") " pod="openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml" Oct 31 05:08:21.938355 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:21.935533 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3361c865-a311-4ae0-a040-e3ffade9e514-workload-socket\") pod \"e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml\" (UID: \"3361c865-a311-4ae0-a040-e3ffade9e514\") " pod="openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml" Oct 31 05:08:21.938355 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:21.935616 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3361c865-a311-4ae0-a040-e3ffade9e514-istio-envoy\") pod \"e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml\" (UID: \"3361c865-a311-4ae0-a040-e3ffade9e514\") " pod="openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml" Oct 31 05:08:21.938355 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:21.935667 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3361c865-a311-4ae0-a040-e3ffade9e514-istio-data\") pod \"e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml\" (UID: \"3361c865-a311-4ae0-a040-e3ffade9e514\") " pod="openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml" Oct 31 05:08:21.938355 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:21.935699 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3361c865-a311-4ae0-a040-e3ffade9e514-workload-certs\") pod \"e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml\" (UID: \"3361c865-a311-4ae0-a040-e3ffade9e514\") " pod="openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml" Oct 31 05:08:21.938355 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:21.935724 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3361c865-a311-4ae0-a040-e3ffade9e514-istio-podinfo\") pod \"e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml\" (UID: \"3361c865-a311-4ae0-a040-e3ffade9e514\") " pod="openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml" Oct 31 05:08:21.938355 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:21.935804 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcfj9\" (UniqueName: \"kubernetes.io/projected/3361c865-a311-4ae0-a040-e3ffade9e514-kube-api-access-vcfj9\") pod \"e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml\" (UID: \"3361c865-a311-4ae0-a040-e3ffade9e514\") " pod="openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml" Oct 31 05:08:21.938355 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:21.935848 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3361c865-a311-4ae0-a040-e3ffade9e514-istio-token\") pod \"e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml\" (UID: \"3361c865-a311-4ae0-a040-e3ffade9e514\") " pod="openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml" Oct 31 05:08:21.938355 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:21.936669 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3361c865-a311-4ae0-a040-e3ffade9e514-credential-socket\") pod \"e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml\" (UID: \"3361c865-a311-4ae0-a040-e3ffade9e514\") " pod="openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml" Oct 31 05:08:21.938355 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:21.937299 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3361c865-a311-4ae0-a040-e3ffade9e514-istiod-ca-cert\") pod \"e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml\" (UID: \"3361c865-a311-4ae0-a040-e3ffade9e514\") " pod="openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml" Oct 31 05:08:21.938355 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:21.937546 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3361c865-a311-4ae0-a040-e3ffade9e514-workload-socket\") pod \"e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml\" (UID: \"3361c865-a311-4ae0-a040-e3ffade9e514\") " pod="openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml" Oct 31 05:08:21.944943 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:21.944876 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3361c865-a311-4ae0-a040-e3ffade9e514-istio-envoy\") pod \"e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml\" (UID: \"3361c865-a311-4ae0-a040-e3ffade9e514\") " pod="openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml" Oct 31 05:08:21.950695 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:21.950669 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3361c865-a311-4ae0-a040-e3ffade9e514-istio-data\") pod \"e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml\" (UID: \"3361c865-a311-4ae0-a040-e3ffade9e514\") " pod="openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml" Oct 31 05:08:21.958750 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:21.951168 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3361c865-a311-4ae0-a040-e3ffade9e514-workload-certs\") pod \"e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml\" (UID: \"3361c865-a311-4ae0-a040-e3ffade9e514\") " pod="openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml" Oct 31 05:08:21.963763 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:21.963714 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3361c865-a311-4ae0-a040-e3ffade9e514-istio-token\") pod \"e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml\" (UID: \"3361c865-a311-4ae0-a040-e3ffade9e514\") " pod="openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml" Oct 31 05:08:21.978351 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:21.974313 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3361c865-a311-4ae0-a040-e3ffade9e514-istio-podinfo\") pod \"e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml\" (UID: \"3361c865-a311-4ae0-a040-e3ffade9e514\") " pod="openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml" Oct 31 05:08:22.014705 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:22.014636 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcfj9\" (UniqueName: \"kubernetes.io/projected/3361c865-a311-4ae0-a040-e3ffade9e514-kube-api-access-vcfj9\") pod \"e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml\" (UID: \"3361c865-a311-4ae0-a040-e3ffade9e514\") " pod="openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml" Oct 31 05:08:22.030150 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:22.029615 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml" Oct 31 05:08:22.364016 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:08:22.363979 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3361c865_a311_4ae0_a040_e3ffade9e514.slice/crio-2df6f79e3254a0b819a72d3f5907c3f9c8a7af06ebb023205065c5ca5fea1384 WatchSource:0}: Error finding container 2df6f79e3254a0b819a72d3f5907c3f9c8a7af06ebb023205065c5ca5fea1384: Status 404 returned error can't find the container with id 2df6f79e3254a0b819a72d3f5907c3f9c8a7af06ebb023205065c5ca5fea1384 Oct 31 05:08:22.364016 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:22.364005 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml"] Oct 31 05:08:22.367198 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:22.367159 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:08:22.367304 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:22.367227 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:08:22.367304 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:22.367258 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:08:22.397875 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:08:22.397843 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:08:23.331982 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:23.331936 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml" event={"ID":"3361c865-a311-4ae0-a040-e3ffade9e514","Type":"ContainerStarted","Data":"fdfde4c344f59c6557677bb6ce5ea995e444e764c95047966893ac06fb261af7"} Oct 31 05:08:23.332501 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:23.332044 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml" event={"ID":"3361c865-a311-4ae0-a040-e3ffade9e514","Type":"ContainerStarted","Data":"2df6f79e3254a0b819a72d3f5907c3f9c8a7af06ebb023205065c5ca5fea1384"} Oct 31 05:08:23.405731 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:23.405670 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml" podStartSLOduration=2.405651518 podStartE2EDuration="2.405651518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:08:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:08:23.390304922 +0000 UTC m=+3858.146642962" watchObservedRunningTime="2026-10-31 05:08:23.405651518 +0000 UTC m=+3858.161989559" Oct 31 05:08:24.030576 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:24.030535 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml" Oct 31 05:08:24.037013 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:24.036985 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml" Oct 31 05:08:24.340701 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:24.340607 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml" Oct 31 05:08:24.341705 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:08:24.341682 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml" Oct 31 05:08:52.393469 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:08:52.393429 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:09:05.243407 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:05.243366 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f742f79-7psld"] Oct 31 05:09:05.249365 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:05.249324 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f742f79-7psld" Oct 31 05:09:05.266212 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:05.266182 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f742f79-7psld"] Oct 31 05:09:05.271513 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:05.271480 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:09:05.271655 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:05.271539 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:09:05.295694 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:05.295661 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:09:05.295885 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:05.295870 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 05:09:05.353245 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:05.353208 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2171746-75af-4bba-bf0a-1f065df42461-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f742f79-7psld\" (UID: \"c2171746-75af-4bba-bf0a-1f065df42461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f742f79-7psld" Oct 31 05:09:05.353476 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:05.353455 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rplxx\" (UniqueName: \"kubernetes.io/projected/c2171746-75af-4bba-bf0a-1f065df42461-kube-api-access-rplxx\") pod \"maas-api-revoke-sub-3f742f79-7psld\" (UID: \"c2171746-75af-4bba-bf0a-1f065df42461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f742f79-7psld" Oct 31 05:09:05.353571 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:05.353551 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2171746-75af-4bba-bf0a-1f065df42461-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f742f79-7psld\" (UID: \"c2171746-75af-4bba-bf0a-1f065df42461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f742f79-7psld" Oct 31 05:09:05.455225 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:05.455189 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rplxx\" (UniqueName: \"kubernetes.io/projected/c2171746-75af-4bba-bf0a-1f065df42461-kube-api-access-rplxx\") pod \"maas-api-revoke-sub-3f742f79-7psld\" (UID: \"c2171746-75af-4bba-bf0a-1f065df42461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f742f79-7psld" Oct 31 05:09:05.455465 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:05.455247 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2171746-75af-4bba-bf0a-1f065df42461-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f742f79-7psld\" (UID: \"c2171746-75af-4bba-bf0a-1f065df42461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f742f79-7psld" Oct 31 05:09:05.455573 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:05.455462 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2171746-75af-4bba-bf0a-1f065df42461-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f742f79-7psld\" (UID: \"c2171746-75af-4bba-bf0a-1f065df42461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f742f79-7psld" Oct 31 05:09:05.456119 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:05.456094 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2171746-75af-4bba-bf0a-1f065df42461-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f742f79-7psld\" (UID: \"c2171746-75af-4bba-bf0a-1f065df42461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f742f79-7psld" Oct 31 05:09:05.457778 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:05.457746 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2171746-75af-4bba-bf0a-1f065df42461-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f742f79-7psld\" (UID: \"c2171746-75af-4bba-bf0a-1f065df42461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f742f79-7psld" Oct 31 05:09:05.477885 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:05.477860 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rplxx\" (UniqueName: \"kubernetes.io/projected/c2171746-75af-4bba-bf0a-1f065df42461-kube-api-access-rplxx\") pod \"maas-api-revoke-sub-3f742f79-7psld\" (UID: \"c2171746-75af-4bba-bf0a-1f065df42461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f742f79-7psld" Oct 31 05:09:05.568924 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:05.568838 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f742f79-7psld" Oct 31 05:09:05.876140 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:05.876113 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f742f79-7psld"] Oct 31 05:09:05.876656 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:09:05.876631 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2171746_75af_4bba_bf0a_1f065df42461.slice/crio-a399e8a3bb09a7ecfc58c8bee58a6e389521b1019293fe650f2e2b3d19576ef6 WatchSource:0}: Error finding container a399e8a3bb09a7ecfc58c8bee58a6e389521b1019293fe650f2e2b3d19576ef6: Status 404 returned error can't find the container with id a399e8a3bb09a7ecfc58c8bee58a6e389521b1019293fe650f2e2b3d19576ef6 Oct 31 05:09:06.737876 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:06.737837 5019 generic.go:356] "Generic (PLEG): container finished" podID="c2171746-75af-4bba-bf0a-1f065df42461" containerID="db6923af30f9e8ade71bc749eaf01be76cbc88d7be0aa26ec8206385de9a3009" exitCode=0 Oct 31 05:09:06.738701 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:06.738663 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f742f79-7psld" event={"ID":"c2171746-75af-4bba-bf0a-1f065df42461","Type":"ContainerDied","Data":"db6923af30f9e8ade71bc749eaf01be76cbc88d7be0aa26ec8206385de9a3009"} Oct 31 05:09:06.738917 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:06.738896 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f742f79-7psld" event={"ID":"c2171746-75af-4bba-bf0a-1f065df42461","Type":"ContainerStarted","Data":"a399e8a3bb09a7ecfc58c8bee58a6e389521b1019293fe650f2e2b3d19576ef6"} Oct 31 05:09:07.922798 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:07.922769 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f742f79-7psld" Oct 31 05:09:08.097665 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:08.097579 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2171746-75af-4bba-bf0a-1f065df42461-kube-api-access-rplxx\" (UniqueName: \"kubernetes.io/projected/c2171746-75af-4bba-bf0a-1f065df42461-kube-api-access-rplxx\") pod \"c2171746-75af-4bba-bf0a-1f065df42461\" (UID: \"c2171746-75af-4bba-bf0a-1f065df42461\") " Oct 31 05:09:08.097665 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:08.097630 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2171746-75af-4bba-bf0a-1f065df42461-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2171746-75af-4bba-bf0a-1f065df42461-maas-api-openshift-ca\") pod \"c2171746-75af-4bba-bf0a-1f065df42461\" (UID: \"c2171746-75af-4bba-bf0a-1f065df42461\") " Oct 31 05:09:08.097872 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:08.097737 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2171746-75af-4bba-bf0a-1f065df42461-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2171746-75af-4bba-bf0a-1f065df42461-maas-api-xks-ca\") pod \"c2171746-75af-4bba-bf0a-1f065df42461\" (UID: \"c2171746-75af-4bba-bf0a-1f065df42461\") " Oct 31 05:09:08.098053 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:08.098029 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2171746-75af-4bba-bf0a-1f065df42461-maas-api-openshift-ca" pod "c2171746-75af-4bba-bf0a-1f065df42461" (UID: "c2171746-75af-4bba-bf0a-1f065df42461"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:09:08.098248 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:08.098220 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2171746-75af-4bba-bf0a-1f065df42461-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:09:08.100007 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:08.099982 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2171746-75af-4bba-bf0a-1f065df42461-maas-api-xks-ca" pod "c2171746-75af-4bba-bf0a-1f065df42461" (UID: "c2171746-75af-4bba-bf0a-1f065df42461"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:09:08.100007 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:08.099994 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2171746-75af-4bba-bf0a-1f065df42461-kube-api-access-rplxx" pod "c2171746-75af-4bba-bf0a-1f065df42461" (UID: "c2171746-75af-4bba-bf0a-1f065df42461"). InnerVolumeSpecName "kube-api-access-rplxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:09:08.199390 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:08.199356 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2171746-75af-4bba-bf0a-1f065df42461-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:09:08.199390 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:08.199387 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rplxx\" (UniqueName: \"kubernetes.io/projected/c2171746-75af-4bba-bf0a-1f065df42461-kube-api-access-rplxx\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:09:08.757475 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:08.757444 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f742f79-7psld" Oct 31 05:09:08.757654 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:08.757484 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f742f79-7psld" event={"ID":"c2171746-75af-4bba-bf0a-1f065df42461","Type":"ContainerDied","Data":"a399e8a3bb09a7ecfc58c8bee58a6e389521b1019293fe650f2e2b3d19576ef6"} Oct 31 05:09:08.757654 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:08.757535 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a399e8a3bb09a7ecfc58c8bee58a6e389521b1019293fe650f2e2b3d19576ef6" Oct 31 05:09:09.564972 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:09.564936 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s88kp_3945ff65-e877-4372-9334-3968c68b7295/console-operator/2.log" Oct 31 05:09:09.566681 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:09.566658 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s88kp_3945ff65-e877-4372-9334-3968c68b7295/console-operator/2.log" Oct 31 05:09:13.166219 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:13.166186 5019 scope.go:122] "RemoveContainer" containerID="7eae9c2f09b905ca5e0d624defeb44e9921e66479cf0eac465807453f60efcf9" Oct 31 05:09:13.180128 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:13.180102 5019 scope.go:122] "RemoveContainer" containerID="805791bcf69cec7e7df394c2c9b9a6388e6f02aa509257676769e95ff8535d67" Oct 31 05:09:16.038665 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:16.038585 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed65b197-8fb99"] Oct 31 05:09:16.049798 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:16.049770 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed65b197-8fb99"] Oct 31 05:09:17.411183 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:17.411139 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbee8d43-9d80-4412-99c7-6218bb6d0f5e" path="/var/lib/kubelet/pods/dbee8d43-9d80-4412-99c7-6218bb6d0f5e/volumes" Oct 31 05:09:22.394946 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:09:22.394899 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:09:28.028848 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:28.028812 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40ffde1b-t6xrq"] Oct 31 05:09:28.040637 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:28.040607 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40ffde1b-t6xrq"] Oct 31 05:09:29.412951 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:29.412914 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd920eaf-9ce8-4c6b-aa4f-c3ce59ef13c0" path="/var/lib/kubelet/pods/dd920eaf-9ce8-4c6b-aa4f-c3ce59ef13c0/volumes" Oct 31 05:09:46.276767 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:46.276715 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" podUID="b4ced3ab-41e4-4c54-93eb-95eddaae992d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 05:09:46.490625 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:46.490579 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" podUID="643433d8-418e-4102-8bc3-cd544932614a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 05:09:46.527414 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:46.520598 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 05:09:47.285709 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:47.277549 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" podUID="b4ced3ab-41e4-4c54-93eb-95eddaae992d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Oct 31 05:09:47.285709 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:47.278025 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" Oct 31 05:09:47.583512 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:47.570776 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" Oct 31 05:09:52.398442 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:09:52.397845 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:09:54.700157 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:54.699561 5019 patch_prober.go:29] interesting pod/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.175:15021/healthz/ready\": read tcp 10.132.0.2:42806->10.132.1.175:15021: read: connection reset by peer" start-of-body= Oct 31 05:09:54.700157 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:54.699621 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml" podUID="3361c865-a311-4ae0-a040-e3ffade9e514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.175:15021/healthz/ready\": read tcp 10.132.0.2:42806->10.132.1.175:15021: read: connection reset by peer" Oct 31 05:09:55.711525 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:55.711488 5019 generic.go:356] "Generic (PLEG): container finished" podID="3361c865-a311-4ae0-a040-e3ffade9e514" containerID="fdfde4c344f59c6557677bb6ce5ea995e444e764c95047966893ac06fb261af7" exitCode=0 Oct 31 05:09:55.712106 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:55.711572 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml" event={"ID":"3361c865-a311-4ae0-a040-e3ffade9e514","Type":"ContainerDied","Data":"fdfde4c344f59c6557677bb6ce5ea995e444e764c95047966893ac06fb261af7"} Oct 31 05:09:55.712818 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:55.712798 5019 scope.go:122] "RemoveContainer" containerID="fdfde4c344f59c6557677bb6ce5ea995e444e764c95047966893ac06fb261af7" Oct 31 05:09:55.715351 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:55.715296 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:09:55.715497 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:55.715406 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:09:55.715497 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:55.715465 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:09:56.724041 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:56.724005 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml" event={"ID":"3361c865-a311-4ae0-a040-e3ffade9e514","Type":"ContainerStarted","Data":"ae6ace2126cc015b6e841467cfd54213649f609c523d09fc76c89eeee4d2812c"} Oct 31 05:09:57.030187 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:57.030112 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml" Oct 31 05:09:57.030352 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:57.030296 5019 patch_prober.go:29] interesting pod/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.175:15021/healthz/ready\": dial tcp 10.132.1.175:15021: connect: connection refused" start-of-body= Oct 31 05:09:57.030414 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:57.030393 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml" podUID="3361c865-a311-4ae0-a040-e3ffade9e514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.175:15021/healthz/ready\": dial tcp 10.132.1.175:15021: connect: connection refused" Oct 31 05:09:58.030438 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:58.030402 5019 patch_prober.go:29] interesting pod/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.175:15021/healthz/ready\": dial tcp 10.132.1.175:15021: connect: connection refused" start-of-body= Oct 31 05:09:58.030851 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:58.030453 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml" podUID="3361c865-a311-4ae0-a040-e3ffade9e514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.175:15021/healthz/ready\": dial tcp 10.132.1.175:15021: connect: connection refused" Oct 31 05:09:59.030572 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:59.030531 5019 patch_prober.go:29] interesting pod/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.175:15021/healthz/ready\": dial tcp 10.132.1.175:15021: connect: connection refused" start-of-body= Oct 31 05:09:59.031074 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:09:59.030589 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml" podUID="3361c865-a311-4ae0-a040-e3ffade9e514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.175:15021/healthz/ready\": dial tcp 10.132.1.175:15021: connect: connection refused" Oct 31 05:10:00.030632 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:10:00.030594 5019 patch_prober.go:29] interesting pod/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.175:15021/healthz/ready\": dial tcp 10.132.1.175:15021: connect: connection refused" start-of-body= Oct 31 05:10:00.031053 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:10:00.030647 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml" podUID="3361c865-a311-4ae0-a040-e3ffade9e514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.175:15021/healthz/ready\": dial tcp 10.132.1.175:15021: connect: connection refused" Oct 31 05:10:01.030797 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:10:01.030757 5019 patch_prober.go:29] interesting pod/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.175:15021/healthz/ready\": dial tcp 10.132.1.175:15021: connect: connection refused" start-of-body= Oct 31 05:10:01.031201 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:10:01.030864 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml" podUID="3361c865-a311-4ae0-a040-e3ffade9e514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.175:15021/healthz/ready\": dial tcp 10.132.1.175:15021: connect: connection refused" Oct 31 05:10:02.031053 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:10:02.031006 5019 patch_prober.go:29] interesting pod/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.175:15021/healthz/ready\": dial tcp 10.132.1.175:15021: connect: connection refused" start-of-body= Oct 31 05:10:02.031474 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:10:02.031078 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml" podUID="3361c865-a311-4ae0-a040-e3ffade9e514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.175:15021/healthz/ready\": dial tcp 10.132.1.175:15021: connect: connection refused" Oct 31 05:10:03.035956 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:10:03.035926 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml" Oct 31 05:10:03.036407 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:10:03.036200 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml" Oct 31 05:10:03.036948 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:10:03.036930 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml" Oct 31 05:10:13.237270 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:10:13.237235 5019 scope.go:122] "RemoveContainer" containerID="931d8c1d41faf08939d890398ba63872d983cd52002b866ec24852ecc10e4c72" Oct 31 05:10:15.030215 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:10:15.030180 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d52321c9-mc599"] Oct 31 05:10:15.041573 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:10:15.041548 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d52321c9-mc599"] Oct 31 05:10:15.409580 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:10:15.409496 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4900084-1bee-46c6-8d1d-f6504e65a828" path="/var/lib/kubelet/pods/d4900084-1bee-46c6-8d1d-f6504e65a828/volumes" Oct 31 05:10:16.030078 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:10:16.030050 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b22a1dd5-7g82g"] Oct 31 05:10:16.041599 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:10:16.041571 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b22a1dd5-7g82g"] Oct 31 05:10:16.045632 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:10:16.045604 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0614feb6-glvs5"] Oct 31 05:10:16.049206 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:10:16.049186 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0614feb6-glvs5"] Oct 31 05:10:17.411499 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:10:17.411455 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9e616de-209d-46e9-b3b6-def8d0d0d844" path="/var/lib/kubelet/pods/e9e616de-209d-46e9-b3b6-def8d0d0d844/volumes" Oct 31 05:10:17.412120 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:10:17.412095 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb6bda0f-355e-413f-b122-4d52c1b1fae0" path="/var/lib/kubelet/pods/fb6bda0f-355e-413f-b122-4d52c1b1fae0/volumes" Oct 31 05:10:22.399918 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:10:22.399880 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:10:27.030038 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:10:27.029999 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-191198b1-lhwmk"] Oct 31 05:10:27.041896 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:10:27.041871 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-191198b1-lhwmk"] Oct 31 05:10:27.411363 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:10:27.411268 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08a4cd92-dc05-43bd-b0a6-07a9b3c31cf0" path="/var/lib/kubelet/pods/08a4cd92-dc05-43bd-b0a6-07a9b3c31cf0/volumes" Oct 31 05:10:28.028549 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:10:28.028514 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50e186c4-d68px"] Oct 31 05:10:28.040139 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:10:28.040106 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50e186c4-d68px"] Oct 31 05:10:29.410020 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:10:29.409980 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80f6dda9-d1b5-4990-be72-199a370a953a" path="/var/lib/kubelet/pods/80f6dda9-d1b5-4990-be72-199a370a953a/volumes" Oct 31 05:10:32.028732 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:10:32.028699 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a76d3e66-hb2pf"] Oct 31 05:10:32.040138 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:10:32.040108 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a76d3e66-hb2pf"] Oct 31 05:10:33.411520 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:10:33.411479 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd7eece6-db42-4033-9f69-74bfd818bece" path="/var/lib/kubelet/pods/bd7eece6-db42-4033-9f69-74bfd818bece/volumes" Oct 31 05:10:40.628967 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:10:40.628932 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-47c409-d86754547-hbqjk"] Oct 31 05:10:46.190507 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:10:46.190415 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-47c409-d86754547-hbqjk" podUID="7d2f4246-7ce7-475e-87ba-a8afdfec5b00" containerName="payload-pre-processing" containerID="cri-o://daca217026a488d54600774e0a4804f22e83927d017cd6ed263aa6bbe7178644" gracePeriod=25 Oct 31 05:10:48.318065 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:10:48.318029 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-47c409-d86754547-hbqjk" podUID="7d2f4246-7ce7-475e-87ba-a8afdfec5b00" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:10:52.394809 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:10:52.394756 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:10:58.317299 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:10:58.317261 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-47c409-d86754547-hbqjk" podUID="7d2f4246-7ce7-475e-87ba-a8afdfec5b00" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:11:06.465237 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:06.465175 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-47c409-d86754547-hbqjk" event={"ID":"7d2f4246-7ce7-475e-87ba-a8afdfec5b00","Type":"ContainerDied","Data":"daca217026a488d54600774e0a4804f22e83927d017cd6ed263aa6bbe7178644"} Oct 31 05:11:06.468184 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:06.468126 5019 generic.go:356] "Generic (PLEG): container finished" podID="7d2f4246-7ce7-475e-87ba-a8afdfec5b00" containerID="daca217026a488d54600774e0a4804f22e83927d017cd6ed263aa6bbe7178644" exitCode=0 Oct 31 05:11:06.532146 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:06.532120 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-47c409-d86754547-hbqjk" Oct 31 05:11:06.631187 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:06.631150 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d2f4246-7ce7-475e-87ba-a8afdfec5b00-kube-api-access-vm8hq\" (UniqueName: \"kubernetes.io/projected/7d2f4246-7ce7-475e-87ba-a8afdfec5b00-kube-api-access-vm8hq\") pod \"7d2f4246-7ce7-475e-87ba-a8afdfec5b00\" (UID: \"7d2f4246-7ce7-475e-87ba-a8afdfec5b00\") " Oct 31 05:11:06.631444 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:06.631427 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d2f4246-7ce7-475e-87ba-a8afdfec5b00-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d2f4246-7ce7-475e-87ba-a8afdfec5b00-plugins-config-volume\") pod \"7d2f4246-7ce7-475e-87ba-a8afdfec5b00\" (UID: \"7d2f4246-7ce7-475e-87ba-a8afdfec5b00\") " Oct 31 05:11:06.631807 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:06.631782 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d2f4246-7ce7-475e-87ba-a8afdfec5b00-plugins-config-volume" pod "7d2f4246-7ce7-475e-87ba-a8afdfec5b00" (UID: "7d2f4246-7ce7-475e-87ba-a8afdfec5b00"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:11:06.632016 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:06.631997 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d2f4246-7ce7-475e-87ba-a8afdfec5b00-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:11:06.633251 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:06.633231 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d2f4246-7ce7-475e-87ba-a8afdfec5b00-kube-api-access-vm8hq" pod "7d2f4246-7ce7-475e-87ba-a8afdfec5b00" (UID: "7d2f4246-7ce7-475e-87ba-a8afdfec5b00"). InnerVolumeSpecName "kube-api-access-vm8hq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:11:06.733768 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:06.733746 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm8hq\" (UniqueName: \"kubernetes.io/projected/7d2f4246-7ce7-475e-87ba-a8afdfec5b00-kube-api-access-vm8hq\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:11:07.478178 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:07.478150 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-47c409-d86754547-hbqjk" Oct 31 05:11:07.478178 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:07.478165 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-47c409-d86754547-hbqjk" event={"ID":"7d2f4246-7ce7-475e-87ba-a8afdfec5b00","Type":"ContainerDied","Data":"7eb351c52050821d66c86f8ba8bc2ac7b5b2fbbc753bb1d461c5c25ea9b85381"} Oct 31 05:11:07.478683 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:07.478224 5019 scope.go:122] "RemoveContainer" containerID="daca217026a488d54600774e0a4804f22e83927d017cd6ed263aa6bbe7178644" Oct 31 05:11:07.532153 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:07.532108 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-47c409-d86754547-hbqjk"] Oct 31 05:11:07.545645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:07.545601 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-47c409-d86754547-hbqjk"] Oct 31 05:11:09.412148 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:09.412106 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d2f4246-7ce7-475e-87ba-a8afdfec5b00" path="/var/lib/kubelet/pods/7d2f4246-7ce7-475e-87ba-a8afdfec5b00/volumes" Oct 31 05:11:13.267907 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:13.267864 5019 scope.go:122] "RemoveContainer" containerID="a71c894bd09a4bc8579bca97228cd0f4b7c3b46b41403131077ad8ccc6fde0e2" Oct 31 05:11:13.279737 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:13.279703 5019 scope.go:122] "RemoveContainer" containerID="e2b7a5f02242acc6c687cdd3d06c3b10e4574b7ce20a4dd0c532fd3a57e031f5" Oct 31 05:11:13.291599 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:13.291569 5019 scope.go:122] "RemoveContainer" containerID="8232a8be47521aabe89348e85ccddc0bd2665b3d5b7137b1dc66bc490b26c520" Oct 31 05:11:13.310420 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:13.308920 5019 scope.go:122] "RemoveContainer" containerID="bcc550dbdf369fa5e598cd2b29c211a17047798755e30910abb2a1ab6b20fd35" Oct 31 05:11:22.129021 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:22.128969 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc"] Oct 31 05:11:22.129539 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:22.129265 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc" podUID="c4a5e443-a1d7-4882-b86c-e6022b980073" containerName="istio-proxy" containerID="cri-o://8f2cda9389a568536a9adeb8d2cca0b34bd87243df394127ec7c24e6d353d837" gracePeriod=30 Oct 31 05:11:22.463645 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:11:22.463525 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:11:26.899668 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:26.899616 5019 patch_prober.go:29] interesting pod/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 05:11:26.900474 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:26.899703 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc" podUID="c4a5e443-a1d7-4882-b86c-e6022b980073" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 05:11:27.459706 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:27.459673 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc" Oct 31 05:11:27.545734 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:27.545693 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4a5e443-a1d7-4882-b86c-e6022b980073-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4a5e443-a1d7-4882-b86c-e6022b980073-istio-envoy\") pod \"c4a5e443-a1d7-4882-b86c-e6022b980073\" (UID: \"c4a5e443-a1d7-4882-b86c-e6022b980073\") " Oct 31 05:11:27.545969 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:27.545796 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4a5e443-a1d7-4882-b86c-e6022b980073-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4a5e443-a1d7-4882-b86c-e6022b980073-workload-socket\") pod \"c4a5e443-a1d7-4882-b86c-e6022b980073\" (UID: \"c4a5e443-a1d7-4882-b86c-e6022b980073\") " Oct 31 05:11:27.545969 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:27.545859 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4a5e443-a1d7-4882-b86c-e6022b980073-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4a5e443-a1d7-4882-b86c-e6022b980073-istiod-ca-cert\") pod \"c4a5e443-a1d7-4882-b86c-e6022b980073\" (UID: \"c4a5e443-a1d7-4882-b86c-e6022b980073\") " Oct 31 05:11:27.545969 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:27.545890 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4a5e443-a1d7-4882-b86c-e6022b980073-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4a5e443-a1d7-4882-b86c-e6022b980073-workload-certs\") pod \"c4a5e443-a1d7-4882-b86c-e6022b980073\" (UID: \"c4a5e443-a1d7-4882-b86c-e6022b980073\") " Oct 31 05:11:27.545969 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:27.545954 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4a5e443-a1d7-4882-b86c-e6022b980073-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4a5e443-a1d7-4882-b86c-e6022b980073-istio-podinfo\") pod \"c4a5e443-a1d7-4882-b86c-e6022b980073\" (UID: \"c4a5e443-a1d7-4882-b86c-e6022b980073\") " Oct 31 05:11:27.546203 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:27.546049 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4a5e443-a1d7-4882-b86c-e6022b980073-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4a5e443-a1d7-4882-b86c-e6022b980073-istio-data\") pod \"c4a5e443-a1d7-4882-b86c-e6022b980073\" (UID: \"c4a5e443-a1d7-4882-b86c-e6022b980073\") " Oct 31 05:11:27.546203 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:27.546087 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4a5e443-a1d7-4882-b86c-e6022b980073-kube-api-access-djpnm\" (UniqueName: \"kubernetes.io/projected/c4a5e443-a1d7-4882-b86c-e6022b980073-kube-api-access-djpnm\") pod \"c4a5e443-a1d7-4882-b86c-e6022b980073\" (UID: \"c4a5e443-a1d7-4882-b86c-e6022b980073\") " Oct 31 05:11:27.546203 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:27.546115 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4a5e443-a1d7-4882-b86c-e6022b980073-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4a5e443-a1d7-4882-b86c-e6022b980073-credential-socket\") pod \"c4a5e443-a1d7-4882-b86c-e6022b980073\" (UID: \"c4a5e443-a1d7-4882-b86c-e6022b980073\") " Oct 31 05:11:27.546203 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:27.546174 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4a5e443-a1d7-4882-b86c-e6022b980073-istio-token\" (UniqueName: \"kubernetes.io/projected/c4a5e443-a1d7-4882-b86c-e6022b980073-istio-token\") pod \"c4a5e443-a1d7-4882-b86c-e6022b980073\" (UID: \"c4a5e443-a1d7-4882-b86c-e6022b980073\") " Oct 31 05:11:27.546479 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:27.546109 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4a5e443-a1d7-4882-b86c-e6022b980073-workload-socket" pod "c4a5e443-a1d7-4882-b86c-e6022b980073" (UID: "c4a5e443-a1d7-4882-b86c-e6022b980073"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:11:27.546479 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:27.546302 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4a5e443-a1d7-4882-b86c-e6022b980073-istiod-ca-cert" pod "c4a5e443-a1d7-4882-b86c-e6022b980073" (UID: "c4a5e443-a1d7-4882-b86c-e6022b980073"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:11:27.546589 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:27.546515 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4a5e443-a1d7-4882-b86c-e6022b980073-credential-socket" pod "c4a5e443-a1d7-4882-b86c-e6022b980073" (UID: "c4a5e443-a1d7-4882-b86c-e6022b980073"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:11:27.547458 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:27.546677 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4a5e443-a1d7-4882-b86c-e6022b980073-workload-certs" pod "c4a5e443-a1d7-4882-b86c-e6022b980073" (UID: "c4a5e443-a1d7-4882-b86c-e6022b980073"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:11:27.547458 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:27.546695 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4a5e443-a1d7-4882-b86c-e6022b980073-istio-data" pod "c4a5e443-a1d7-4882-b86c-e6022b980073" (UID: "c4a5e443-a1d7-4882-b86c-e6022b980073"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:11:27.547458 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:27.547255 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4a5e443-a1d7-4882-b86c-e6022b980073-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:11:27.547458 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:27.547278 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4a5e443-a1d7-4882-b86c-e6022b980073-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:11:27.547458 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:27.547297 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4a5e443-a1d7-4882-b86c-e6022b980073-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:11:27.547458 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:27.547310 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4a5e443-a1d7-4882-b86c-e6022b980073-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:11:27.547458 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:27.547323 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4a5e443-a1d7-4882-b86c-e6022b980073-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:11:27.549302 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:27.549266 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4a5e443-a1d7-4882-b86c-e6022b980073-istio-token" pod "c4a5e443-a1d7-4882-b86c-e6022b980073" (UID: "c4a5e443-a1d7-4882-b86c-e6022b980073"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:11:27.550484 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:27.550448 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4a5e443-a1d7-4882-b86c-e6022b980073-istio-envoy" pod "c4a5e443-a1d7-4882-b86c-e6022b980073" (UID: "c4a5e443-a1d7-4882-b86c-e6022b980073"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:11:27.551191 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:27.551162 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4a5e443-a1d7-4882-b86c-e6022b980073-istio-podinfo" pod "c4a5e443-a1d7-4882-b86c-e6022b980073" (UID: "c4a5e443-a1d7-4882-b86c-e6022b980073"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:11:27.551324 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:27.551300 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4a5e443-a1d7-4882-b86c-e6022b980073-kube-api-access-djpnm" pod "c4a5e443-a1d7-4882-b86c-e6022b980073" (UID: "c4a5e443-a1d7-4882-b86c-e6022b980073"). InnerVolumeSpecName "kube-api-access-djpnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:11:27.649282 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:27.649227 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4a5e443-a1d7-4882-b86c-e6022b980073-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:11:27.649282 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:27.649280 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4a5e443-a1d7-4882-b86c-e6022b980073-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:11:27.649565 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:27.649296 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djpnm\" (UniqueName: \"kubernetes.io/projected/c4a5e443-a1d7-4882-b86c-e6022b980073-kube-api-access-djpnm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:11:27.649565 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:27.649311 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4a5e443-a1d7-4882-b86c-e6022b980073-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:11:27.666730 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:27.666691 5019 generic.go:356] "Generic (PLEG): container finished" podID="c4a5e443-a1d7-4882-b86c-e6022b980073" containerID="8f2cda9389a568536a9adeb8d2cca0b34bd87243df394127ec7c24e6d353d837" exitCode=0 Oct 31 05:11:27.666932 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:27.666760 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc" Oct 31 05:11:27.666932 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:27.666774 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc" event={"ID":"c4a5e443-a1d7-4882-b86c-e6022b980073","Type":"ContainerDied","Data":"8f2cda9389a568536a9adeb8d2cca0b34bd87243df394127ec7c24e6d353d837"} Oct 31 05:11:27.666932 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:27.666838 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc" event={"ID":"c4a5e443-a1d7-4882-b86c-e6022b980073","Type":"ContainerDied","Data":"c592e5d5dcd3fbde67f0437356cdadee112347f0961d69568542d6661e05000a"} Oct 31 05:11:27.666932 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:27.666866 5019 scope.go:122] "RemoveContainer" containerID="8f2cda9389a568536a9adeb8d2cca0b34bd87243df394127ec7c24e6d353d837" Oct 31 05:11:27.680310 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:27.680285 5019 scope.go:122] "RemoveContainer" containerID="19b5d18539ac54474e37bed439fc85ae4f3767304b0d9d34948706c493b7a752" Oct 31 05:11:27.693375 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:27.693321 5019 scope.go:122] "RemoveContainer" containerID="8f2cda9389a568536a9adeb8d2cca0b34bd87243df394127ec7c24e6d353d837" Oct 31 05:11:27.693728 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:11:27.693706 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f2cda9389a568536a9adeb8d2cca0b34bd87243df394127ec7c24e6d353d837\": container with ID starting with 8f2cda9389a568536a9adeb8d2cca0b34bd87243df394127ec7c24e6d353d837 not found: ID does not exist" containerID="8f2cda9389a568536a9adeb8d2cca0b34bd87243df394127ec7c24e6d353d837" Oct 31 05:11:27.693790 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:27.693739 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f2cda9389a568536a9adeb8d2cca0b34bd87243df394127ec7c24e6d353d837"} err="failed to get container status \"8f2cda9389a568536a9adeb8d2cca0b34bd87243df394127ec7c24e6d353d837\": rpc error: code = NotFound desc = could not find container \"8f2cda9389a568536a9adeb8d2cca0b34bd87243df394127ec7c24e6d353d837\": container with ID starting with 8f2cda9389a568536a9adeb8d2cca0b34bd87243df394127ec7c24e6d353d837 not found: ID does not exist" Oct 31 05:11:27.693790 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:27.693758 5019 scope.go:122] "RemoveContainer" containerID="19b5d18539ac54474e37bed439fc85ae4f3767304b0d9d34948706c493b7a752" Oct 31 05:11:27.694027 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:11:27.694004 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19b5d18539ac54474e37bed439fc85ae4f3767304b0d9d34948706c493b7a752\": container with ID starting with 19b5d18539ac54474e37bed439fc85ae4f3767304b0d9d34948706c493b7a752 not found: ID does not exist" containerID="19b5d18539ac54474e37bed439fc85ae4f3767304b0d9d34948706c493b7a752" Oct 31 05:11:27.694079 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:27.694033 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19b5d18539ac54474e37bed439fc85ae4f3767304b0d9d34948706c493b7a752"} err="failed to get container status \"19b5d18539ac54474e37bed439fc85ae4f3767304b0d9d34948706c493b7a752\": rpc error: code = NotFound desc = could not find container \"19b5d18539ac54474e37bed439fc85ae4f3767304b0d9d34948706c493b7a752\": container with ID starting with 19b5d18539ac54474e37bed439fc85ae4f3767304b0d9d34948706c493b7a752 not found: ID does not exist" Oct 31 05:11:27.724799 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:27.724749 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc"] Oct 31 05:11:27.730865 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:27.730829 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-401761-openshift-default-6cdf858b5-7z2jc"] Oct 31 05:11:29.417801 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:29.417756 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4a5e443-a1d7-4882-b86c-e6022b980073" path="/var/lib/kubelet/pods/c4a5e443-a1d7-4882-b86c-e6022b980073/volumes" Oct 31 05:11:30.073714 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:30.073663 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-567ca6-7fdf68b675-mfgzr"] Oct 31 05:11:35.074153 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:35.074059 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-567ca6-7fdf68b675-mfgzr" podUID="99145b1d-f102-4e18-942d-aee106ab1bdd" containerName="payload-pre-processing" containerID="cri-o://5673d8ab5c1a17f1fa9b6a0e8a7372d6fe2e9c486de26632c98167ab54f3cc69" gracePeriod=25 Oct 31 05:11:35.094170 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:35.094123 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-567ca6-7fdf68b675-mfgzr" podUID="99145b1d-f102-4e18-942d-aee106ab1bdd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:11:36.847353 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:36.847296 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv"] Oct 31 05:11:36.847807 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:36.847619 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv" podUID="2f84895b-57a1-4c93-b20c-dc3f2ca754b4" containerName="istio-proxy" containerID="cri-o://7d8b013ace00c2653492da4e9a3e0d8942408ffbba4f3cdbfcffea5fff43b0f6" gracePeriod=30 Oct 31 05:11:38.871245 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:38.871197 5019 patch_prober.go:29] interesting pod/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 05:11:38.871813 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:38.871273 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv" podUID="2f84895b-57a1-4c93-b20c-dc3f2ca754b4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 05:11:42.199515 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:42.199476 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv" Oct 31 05:11:42.332691 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:42.332627 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-istio-podinfo\") pod \"2f84895b-57a1-4c93-b20c-dc3f2ca754b4\" (UID: \"2f84895b-57a1-4c93-b20c-dc3f2ca754b4\") " Oct 31 05:11:42.332902 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:42.332805 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-kube-api-access-bcr9b\" (UniqueName: \"kubernetes.io/projected/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-kube-api-access-bcr9b\") pod \"2f84895b-57a1-4c93-b20c-dc3f2ca754b4\" (UID: \"2f84895b-57a1-4c93-b20c-dc3f2ca754b4\") " Oct 31 05:11:42.332902 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:42.332836 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-istiod-ca-cert\") pod \"2f84895b-57a1-4c93-b20c-dc3f2ca754b4\" (UID: \"2f84895b-57a1-4c93-b20c-dc3f2ca754b4\") " Oct 31 05:11:42.332902 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:42.332891 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-istio-token\" (UniqueName: \"kubernetes.io/projected/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-istio-token\") pod \"2f84895b-57a1-4c93-b20c-dc3f2ca754b4\" (UID: \"2f84895b-57a1-4c93-b20c-dc3f2ca754b4\") " Oct 31 05:11:42.333041 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:42.332945 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-istio-envoy\") pod \"2f84895b-57a1-4c93-b20c-dc3f2ca754b4\" (UID: \"2f84895b-57a1-4c93-b20c-dc3f2ca754b4\") " Oct 31 05:11:42.333041 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:42.332970 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-workload-certs\") pod \"2f84895b-57a1-4c93-b20c-dc3f2ca754b4\" (UID: \"2f84895b-57a1-4c93-b20c-dc3f2ca754b4\") " Oct 31 05:11:42.333041 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:42.333024 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-credential-socket\") pod \"2f84895b-57a1-4c93-b20c-dc3f2ca754b4\" (UID: \"2f84895b-57a1-4c93-b20c-dc3f2ca754b4\") " Oct 31 05:11:42.333217 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:42.333116 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-workload-socket\") pod \"2f84895b-57a1-4c93-b20c-dc3f2ca754b4\" (UID: \"2f84895b-57a1-4c93-b20c-dc3f2ca754b4\") " Oct 31 05:11:42.333217 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:42.333173 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-istio-data\") pod \"2f84895b-57a1-4c93-b20c-dc3f2ca754b4\" (UID: \"2f84895b-57a1-4c93-b20c-dc3f2ca754b4\") " Oct 31 05:11:42.333402 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:42.333369 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-workload-certs" pod "2f84895b-57a1-4c93-b20c-dc3f2ca754b4" (UID: "2f84895b-57a1-4c93-b20c-dc3f2ca754b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:11:42.333504 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:42.333406 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-istiod-ca-cert" pod "2f84895b-57a1-4c93-b20c-dc3f2ca754b4" (UID: "2f84895b-57a1-4c93-b20c-dc3f2ca754b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:11:42.333650 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:42.333627 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-credential-socket" pod "2f84895b-57a1-4c93-b20c-dc3f2ca754b4" (UID: "2f84895b-57a1-4c93-b20c-dc3f2ca754b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:11:42.333721 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:42.333640 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-istio-data" pod "2f84895b-57a1-4c93-b20c-dc3f2ca754b4" (UID: "2f84895b-57a1-4c93-b20c-dc3f2ca754b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:11:42.333998 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:42.333968 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-workload-socket" pod "2f84895b-57a1-4c93-b20c-dc3f2ca754b4" (UID: "2f84895b-57a1-4c93-b20c-dc3f2ca754b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:11:42.334127 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:42.334078 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:11:42.334127 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:42.334095 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:11:42.334127 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:42.334104 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:11:42.334127 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:42.334112 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:11:42.334127 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:42.334120 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:11:42.335095 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:42.335069 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-istio-podinfo" pod "2f84895b-57a1-4c93-b20c-dc3f2ca754b4" (UID: "2f84895b-57a1-4c93-b20c-dc3f2ca754b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:11:42.335432 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:42.335412 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-istio-token" pod "2f84895b-57a1-4c93-b20c-dc3f2ca754b4" (UID: "2f84895b-57a1-4c93-b20c-dc3f2ca754b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:11:42.335687 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:42.335663 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-istio-envoy" pod "2f84895b-57a1-4c93-b20c-dc3f2ca754b4" (UID: "2f84895b-57a1-4c93-b20c-dc3f2ca754b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:11:42.335687 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:42.335671 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-kube-api-access-bcr9b" pod "2f84895b-57a1-4c93-b20c-dc3f2ca754b4" (UID: "2f84895b-57a1-4c93-b20c-dc3f2ca754b4"). InnerVolumeSpecName "kube-api-access-bcr9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:11:42.435285 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:42.435185 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:11:42.435285 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:42.435228 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcr9b\" (UniqueName: \"kubernetes.io/projected/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-kube-api-access-bcr9b\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:11:42.435285 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:42.435241 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:11:42.435285 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:42.435255 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f84895b-57a1-4c93-b20c-dc3f2ca754b4-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:11:42.952160 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:42.952117 5019 generic.go:356] "Generic (PLEG): container finished" podID="2f84895b-57a1-4c93-b20c-dc3f2ca754b4" containerID="7d8b013ace00c2653492da4e9a3e0d8942408ffbba4f3cdbfcffea5fff43b0f6" exitCode=0 Oct 31 05:11:42.952379 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:42.952179 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv" Oct 31 05:11:42.952379 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:42.952204 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv" event={"ID":"2f84895b-57a1-4c93-b20c-dc3f2ca754b4","Type":"ContainerDied","Data":"7d8b013ace00c2653492da4e9a3e0d8942408ffbba4f3cdbfcffea5fff43b0f6"} Oct 31 05:11:42.952379 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:42.952256 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv" event={"ID":"2f84895b-57a1-4c93-b20c-dc3f2ca754b4","Type":"ContainerDied","Data":"69837732349f44555008f45574b782cddff9e7179fa631e03d57ab3d1ae77e25"} Oct 31 05:11:42.952379 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:42.952281 5019 scope.go:122] "RemoveContainer" containerID="7d8b013ace00c2653492da4e9a3e0d8942408ffbba4f3cdbfcffea5fff43b0f6" Oct 31 05:11:42.966891 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:42.966862 5019 scope.go:122] "RemoveContainer" containerID="a341aa061b790b2673aeedde095428a96501b3818ee27f7127958763081c82a2" Oct 31 05:11:42.979896 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:42.979687 5019 scope.go:122] "RemoveContainer" containerID="7d8b013ace00c2653492da4e9a3e0d8942408ffbba4f3cdbfcffea5fff43b0f6" Oct 31 05:11:42.980136 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:11:42.980100 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d8b013ace00c2653492da4e9a3e0d8942408ffbba4f3cdbfcffea5fff43b0f6\": container with ID starting with 7d8b013ace00c2653492da4e9a3e0d8942408ffbba4f3cdbfcffea5fff43b0f6 not found: ID does not exist" containerID="7d8b013ace00c2653492da4e9a3e0d8942408ffbba4f3cdbfcffea5fff43b0f6" Oct 31 05:11:42.980220 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:42.980149 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d8b013ace00c2653492da4e9a3e0d8942408ffbba4f3cdbfcffea5fff43b0f6"} err="failed to get container status \"7d8b013ace00c2653492da4e9a3e0d8942408ffbba4f3cdbfcffea5fff43b0f6\": rpc error: code = NotFound desc = could not find container \"7d8b013ace00c2653492da4e9a3e0d8942408ffbba4f3cdbfcffea5fff43b0f6\": container with ID starting with 7d8b013ace00c2653492da4e9a3e0d8942408ffbba4f3cdbfcffea5fff43b0f6 not found: ID does not exist" Oct 31 05:11:42.980220 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:42.980175 5019 scope.go:122] "RemoveContainer" containerID="a341aa061b790b2673aeedde095428a96501b3818ee27f7127958763081c82a2" Oct 31 05:11:42.980507 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:11:42.980479 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a341aa061b790b2673aeedde095428a96501b3818ee27f7127958763081c82a2\": container with ID starting with a341aa061b790b2673aeedde095428a96501b3818ee27f7127958763081c82a2 not found: ID does not exist" containerID="a341aa061b790b2673aeedde095428a96501b3818ee27f7127958763081c82a2" Oct 31 05:11:42.980598 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:42.980511 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a341aa061b790b2673aeedde095428a96501b3818ee27f7127958763081c82a2"} err="failed to get container status \"a341aa061b790b2673aeedde095428a96501b3818ee27f7127958763081c82a2\": rpc error: code = NotFound desc = could not find container \"a341aa061b790b2673aeedde095428a96501b3818ee27f7127958763081c82a2\": container with ID starting with a341aa061b790b2673aeedde095428a96501b3818ee27f7127958763081c82a2 not found: ID does not exist" Oct 31 05:11:43.020344 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:43.020288 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv"] Oct 31 05:11:43.032238 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:43.032193 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-567ca6-openshift-default-74bf9cd4df-dj9fv"] Oct 31 05:11:43.410861 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:43.410816 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f84895b-57a1-4c93-b20c-dc3f2ca754b4" path="/var/lib/kubelet/pods/2f84895b-57a1-4c93-b20c-dc3f2ca754b4/volumes" Oct 31 05:11:45.028104 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:45.028055 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7cb3ca6-9v7n2"] Oct 31 05:11:45.039571 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:45.039525 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7cb3ca6-9v7n2"] Oct 31 05:11:45.094710 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:45.094653 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-567ca6-7fdf68b675-mfgzr" podUID="99145b1d-f102-4e18-942d-aee106ab1bdd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:11:45.525769 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:45.525732 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4068532b-36ea-4ef1-bb0f-366e10073c49" path="/var/lib/kubelet/pods/4068532b-36ea-4ef1-bb0f-366e10073c49/volumes" Oct 31 05:11:52.408075 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:11:52.408020 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:11:55.093800 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:11:55.093764 5019 logging.go:55] [core] [Channel #18334 SubChannel #18335]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.1.25:50052", ServerName: "10.132.1.25:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.1.25:50052: connect: connection refused" Oct 31 05:11:55.430875 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:55.430840 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-567ca6-7fdf68b675-mfgzr" Oct 31 05:11:55.447818 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:55.447780 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99145b1d-f102-4e18-942d-aee106ab1bdd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99145b1d-f102-4e18-942d-aee106ab1bdd-plugins-config-volume\") pod \"99145b1d-f102-4e18-942d-aee106ab1bdd\" (UID: \"99145b1d-f102-4e18-942d-aee106ab1bdd\") " Oct 31 05:11:55.448039 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:55.447837 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99145b1d-f102-4e18-942d-aee106ab1bdd-kube-api-access-sjgns\" (UniqueName: \"kubernetes.io/projected/99145b1d-f102-4e18-942d-aee106ab1bdd-kube-api-access-sjgns\") pod \"99145b1d-f102-4e18-942d-aee106ab1bdd\" (UID: \"99145b1d-f102-4e18-942d-aee106ab1bdd\") " Oct 31 05:11:55.448243 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:55.448212 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99145b1d-f102-4e18-942d-aee106ab1bdd-plugins-config-volume" pod "99145b1d-f102-4e18-942d-aee106ab1bdd" (UID: "99145b1d-f102-4e18-942d-aee106ab1bdd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:11:55.448820 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:55.448729 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99145b1d-f102-4e18-942d-aee106ab1bdd-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:11:55.450578 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:55.450543 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99145b1d-f102-4e18-942d-aee106ab1bdd-kube-api-access-sjgns" pod "99145b1d-f102-4e18-942d-aee106ab1bdd" (UID: "99145b1d-f102-4e18-942d-aee106ab1bdd"). InnerVolumeSpecName "kube-api-access-sjgns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:11:55.549749 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:55.549704 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjgns\" (UniqueName: \"kubernetes.io/projected/99145b1d-f102-4e18-942d-aee106ab1bdd-kube-api-access-sjgns\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:11:56.071066 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:56.071029 5019 generic.go:356] "Generic (PLEG): container finished" podID="99145b1d-f102-4e18-942d-aee106ab1bdd" containerID="5673d8ab5c1a17f1fa9b6a0e8a7372d6fe2e9c486de26632c98167ab54f3cc69" exitCode=0 Oct 31 05:11:56.071291 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:56.071105 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-567ca6-7fdf68b675-mfgzr" Oct 31 05:11:56.071291 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:56.071108 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-567ca6-7fdf68b675-mfgzr" event={"ID":"99145b1d-f102-4e18-942d-aee106ab1bdd","Type":"ContainerDied","Data":"5673d8ab5c1a17f1fa9b6a0e8a7372d6fe2e9c486de26632c98167ab54f3cc69"} Oct 31 05:11:56.071291 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:56.071206 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-567ca6-7fdf68b675-mfgzr" event={"ID":"99145b1d-f102-4e18-942d-aee106ab1bdd","Type":"ContainerDied","Data":"4ef3b1ceb0b13e04477a248a02327b483a95c811b28d8c2bf517bdf829c1c351"} Oct 31 05:11:56.071291 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:56.071241 5019 scope.go:122] "RemoveContainer" containerID="5673d8ab5c1a17f1fa9b6a0e8a7372d6fe2e9c486de26632c98167ab54f3cc69" Oct 31 05:11:56.084305 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:56.084275 5019 scope.go:122] "RemoveContainer" containerID="5673d8ab5c1a17f1fa9b6a0e8a7372d6fe2e9c486de26632c98167ab54f3cc69" Oct 31 05:11:56.084718 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:11:56.084692 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5673d8ab5c1a17f1fa9b6a0e8a7372d6fe2e9c486de26632c98167ab54f3cc69\": container with ID starting with 5673d8ab5c1a17f1fa9b6a0e8a7372d6fe2e9c486de26632c98167ab54f3cc69 not found: ID does not exist" containerID="5673d8ab5c1a17f1fa9b6a0e8a7372d6fe2e9c486de26632c98167ab54f3cc69" Oct 31 05:11:56.084825 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:56.084734 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5673d8ab5c1a17f1fa9b6a0e8a7372d6fe2e9c486de26632c98167ab54f3cc69"} err="failed to get container status \"5673d8ab5c1a17f1fa9b6a0e8a7372d6fe2e9c486de26632c98167ab54f3cc69\": rpc error: code = NotFound desc = could not find container \"5673d8ab5c1a17f1fa9b6a0e8a7372d6fe2e9c486de26632c98167ab54f3cc69\": container with ID starting with 5673d8ab5c1a17f1fa9b6a0e8a7372d6fe2e9c486de26632c98167ab54f3cc69 not found: ID does not exist" Oct 31 05:11:56.093861 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:56.093814 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-567ca6-7fdf68b675-mfgzr" podUID="99145b1d-f102-4e18-942d-aee106ab1bdd" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.1.25:50052\" within 1s: context deadline exceeded" Oct 31 05:11:56.094239 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:56.093933 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-567ca6-7fdf68b675-mfgzr" Oct 31 05:11:56.123904 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:56.123869 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-567ca6-7fdf68b675-mfgzr"] Oct 31 05:11:56.127994 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:56.127962 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-567ca6-7fdf68b675-mfgzr"] Oct 31 05:11:57.410379 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:11:57.410322 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99145b1d-f102-4e18-942d-aee106ab1bdd" path="/var/lib/kubelet/pods/99145b1d-f102-4e18-942d-aee106ab1bdd/volumes" Oct 31 05:12:13.382417 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:13.382374 5019 scope.go:122] "RemoveContainer" containerID="5ddf8695c1aa044aa22478aedd977245e8e0c35d1a1640a32d14476ba7bb4402" Oct 31 05:12:13.393841 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:13.393817 5019 scope.go:122] "RemoveContainer" containerID="98149429bf613b96906c313e9935e122a789a4bc56dcc716aeb7de949211f9b4" Oct 31 05:12:13.405653 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:13.405626 5019 scope.go:122] "RemoveContainer" containerID="6dec045c18a305092161f0cdf8811661b3a23aba3256d0b0601831a04c6ea3e5" Oct 31 05:12:15.061635 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:15.061592 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz"] Oct 31 05:12:15.062159 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:15.061872 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz" podUID="b132bea6-493c-43f7-bd51-29f02a01ae9a" containerName="istio-proxy" containerID="cri-o://0db71e0ec6717ba59c028b1f566d455a81ff4c2e5ac9854dedfc32a723851199" gracePeriod=30 Oct 31 05:12:20.331920 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:20.331879 5019 generic.go:356] "Generic (PLEG): container finished" podID="b132bea6-493c-43f7-bd51-29f02a01ae9a" containerID="0db71e0ec6717ba59c028b1f566d455a81ff4c2e5ac9854dedfc32a723851199" exitCode=0 Oct 31 05:12:20.332610 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:20.331937 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz" event={"ID":"b132bea6-493c-43f7-bd51-29f02a01ae9a","Type":"ContainerDied","Data":"0db71e0ec6717ba59c028b1f566d455a81ff4c2e5ac9854dedfc32a723851199"} Oct 31 05:12:20.332836 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:20.332792 5019 scope.go:122] "RemoveContainer" containerID="d61e05c21f05c1503bdface0acfc18118fd9973b66cd14a67890c6020f1f59a6" Oct 31 05:12:20.618010 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:20.617975 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz" Oct 31 05:12:20.726507 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:20.726456 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b132bea6-493c-43f7-bd51-29f02a01ae9a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b132bea6-493c-43f7-bd51-29f02a01ae9a-credential-socket\") pod \"b132bea6-493c-43f7-bd51-29f02a01ae9a\" (UID: \"b132bea6-493c-43f7-bd51-29f02a01ae9a\") " Oct 31 05:12:20.726742 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:20.726533 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b132bea6-493c-43f7-bd51-29f02a01ae9a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b132bea6-493c-43f7-bd51-29f02a01ae9a-workload-certs\") pod \"b132bea6-493c-43f7-bd51-29f02a01ae9a\" (UID: \"b132bea6-493c-43f7-bd51-29f02a01ae9a\") " Oct 31 05:12:20.726742 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:20.726557 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b132bea6-493c-43f7-bd51-29f02a01ae9a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b132bea6-493c-43f7-bd51-29f02a01ae9a-istio-podinfo\") pod \"b132bea6-493c-43f7-bd51-29f02a01ae9a\" (UID: \"b132bea6-493c-43f7-bd51-29f02a01ae9a\") " Oct 31 05:12:20.726742 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:20.726584 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b132bea6-493c-43f7-bd51-29f02a01ae9a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b132bea6-493c-43f7-bd51-29f02a01ae9a-istio-data\") pod \"b132bea6-493c-43f7-bd51-29f02a01ae9a\" (UID: \"b132bea6-493c-43f7-bd51-29f02a01ae9a\") " Oct 31 05:12:20.726742 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:20.726658 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b132bea6-493c-43f7-bd51-29f02a01ae9a-istio-token\" (UniqueName: \"kubernetes.io/projected/b132bea6-493c-43f7-bd51-29f02a01ae9a-istio-token\") pod \"b132bea6-493c-43f7-bd51-29f02a01ae9a\" (UID: \"b132bea6-493c-43f7-bd51-29f02a01ae9a\") " Oct 31 05:12:20.726742 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:20.726697 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b132bea6-493c-43f7-bd51-29f02a01ae9a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b132bea6-493c-43f7-bd51-29f02a01ae9a-istiod-ca-cert\") pod \"b132bea6-493c-43f7-bd51-29f02a01ae9a\" (UID: \"b132bea6-493c-43f7-bd51-29f02a01ae9a\") " Oct 31 05:12:20.727184 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:20.726752 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b132bea6-493c-43f7-bd51-29f02a01ae9a-kube-api-access-pzvwl\" (UniqueName: \"kubernetes.io/projected/b132bea6-493c-43f7-bd51-29f02a01ae9a-kube-api-access-pzvwl\") pod \"b132bea6-493c-43f7-bd51-29f02a01ae9a\" (UID: \"b132bea6-493c-43f7-bd51-29f02a01ae9a\") " Oct 31 05:12:20.727184 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:20.726793 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b132bea6-493c-43f7-bd51-29f02a01ae9a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b132bea6-493c-43f7-bd51-29f02a01ae9a-workload-socket\") pod \"b132bea6-493c-43f7-bd51-29f02a01ae9a\" (UID: \"b132bea6-493c-43f7-bd51-29f02a01ae9a\") " Oct 31 05:12:20.727184 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:20.726856 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b132bea6-493c-43f7-bd51-29f02a01ae9a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b132bea6-493c-43f7-bd51-29f02a01ae9a-istio-envoy\") pod \"b132bea6-493c-43f7-bd51-29f02a01ae9a\" (UID: \"b132bea6-493c-43f7-bd51-29f02a01ae9a\") " Oct 31 05:12:20.727184 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:20.726863 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b132bea6-493c-43f7-bd51-29f02a01ae9a-credential-socket" pod "b132bea6-493c-43f7-bd51-29f02a01ae9a" (UID: "b132bea6-493c-43f7-bd51-29f02a01ae9a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:12:20.727184 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:20.726936 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b132bea6-493c-43f7-bd51-29f02a01ae9a-workload-certs" pod "b132bea6-493c-43f7-bd51-29f02a01ae9a" (UID: "b132bea6-493c-43f7-bd51-29f02a01ae9a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:12:20.727184 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:20.727118 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b132bea6-493c-43f7-bd51-29f02a01ae9a-workload-socket" pod "b132bea6-493c-43f7-bd51-29f02a01ae9a" (UID: "b132bea6-493c-43f7-bd51-29f02a01ae9a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:12:20.727184 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:20.727153 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b132bea6-493c-43f7-bd51-29f02a01ae9a-istiod-ca-cert" pod "b132bea6-493c-43f7-bd51-29f02a01ae9a" (UID: "b132bea6-493c-43f7-bd51-29f02a01ae9a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:12:20.727757 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:20.727729 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b132bea6-493c-43f7-bd51-29f02a01ae9a-istio-data" pod "b132bea6-493c-43f7-bd51-29f02a01ae9a" (UID: "b132bea6-493c-43f7-bd51-29f02a01ae9a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:12:20.727841 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:20.727765 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b132bea6-493c-43f7-bd51-29f02a01ae9a-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:12:20.727841 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:20.727791 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b132bea6-493c-43f7-bd51-29f02a01ae9a-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:12:20.727841 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:20.727805 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b132bea6-493c-43f7-bd51-29f02a01ae9a-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:12:20.727841 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:20.727820 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b132bea6-493c-43f7-bd51-29f02a01ae9a-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:12:20.729725 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:20.729681 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b132bea6-493c-43f7-bd51-29f02a01ae9a-kube-api-access-pzvwl" pod "b132bea6-493c-43f7-bd51-29f02a01ae9a" (UID: "b132bea6-493c-43f7-bd51-29f02a01ae9a"). InnerVolumeSpecName "kube-api-access-pzvwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:12:20.730147 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:20.730124 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b132bea6-493c-43f7-bd51-29f02a01ae9a-istio-envoy" pod "b132bea6-493c-43f7-bd51-29f02a01ae9a" (UID: "b132bea6-493c-43f7-bd51-29f02a01ae9a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:12:20.730423 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:20.730399 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b132bea6-493c-43f7-bd51-29f02a01ae9a-istio-podinfo" pod "b132bea6-493c-43f7-bd51-29f02a01ae9a" (UID: "b132bea6-493c-43f7-bd51-29f02a01ae9a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:12:20.730745 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:20.730724 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b132bea6-493c-43f7-bd51-29f02a01ae9a-istio-token" pod "b132bea6-493c-43f7-bd51-29f02a01ae9a" (UID: "b132bea6-493c-43f7-bd51-29f02a01ae9a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:12:20.829084 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:20.829021 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b132bea6-493c-43f7-bd51-29f02a01ae9a-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:12:20.829084 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:20.829077 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b132bea6-493c-43f7-bd51-29f02a01ae9a-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:12:20.829084 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:20.829086 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b132bea6-493c-43f7-bd51-29f02a01ae9a-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:12:20.829084 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:20.829095 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b132bea6-493c-43f7-bd51-29f02a01ae9a-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:12:20.829084 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:20.829104 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzvwl\" (UniqueName: \"kubernetes.io/projected/b132bea6-493c-43f7-bd51-29f02a01ae9a-kube-api-access-pzvwl\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:12:21.345182 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:21.345142 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz" Oct 31 05:12:21.345182 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:21.345155 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz" event={"ID":"b132bea6-493c-43f7-bd51-29f02a01ae9a","Type":"ContainerDied","Data":"a8b9d8f6768538703a73e1008fc959d55a26d28a55ec6915912da38b8f51fd8e"} Oct 31 05:12:21.345841 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:21.345223 5019 scope.go:122] "RemoveContainer" containerID="0db71e0ec6717ba59c028b1f566d455a81ff4c2e5ac9854dedfc32a723851199" Oct 31 05:12:21.416440 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:21.416387 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz"] Oct 31 05:12:21.422909 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:21.422870 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-a04e4e-openshift-default-6fbf97c57-srtdz"] Oct 31 05:12:22.395990 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:12:22.395926 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:12:22.562787 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:22.562745 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-77a153-6c975c99f5-mp8vx"] Oct 31 05:12:22.615624 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:22.615577 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-77a153-6d7d74f579-mfw72"] Oct 31 05:12:23.318153 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:23.318105 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-031745-6d7b5f78dc-6k5z9"] Oct 31 05:12:23.361868 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:23.361819 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-031745-74cd69fbc5-s7bs2"] Oct 31 05:12:23.418949 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:23.418902 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b132bea6-493c-43f7-bd51-29f02a01ae9a" path="/var/lib/kubelet/pods/b132bea6-493c-43f7-bd51-29f02a01ae9a/volumes" Oct 31 05:12:27.563822 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:27.563743 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-77a153-6c975c99f5-mp8vx" podUID="7139f8fd-4b2e-4c1e-9111-92e9b1bd1f62" containerName="payload-processing" containerID="cri-o://6e803cd33fa32f8e539c094464e09cbca4efeb86359db641670212855553a5f6" gracePeriod=25 Oct 31 05:12:27.617052 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:27.616982 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-77a153-6d7d74f579-mfw72" podUID="b27570f7-6a81-4a8c-aad3-404126c2b7de" containerName="payload-pre-processing" containerID="cri-o://055909b4eaf8584cd37ac55ea33ad929c6f418ec6d7fedc9fe43d41caccfa244" gracePeriod=25 Oct 31 05:12:27.736496 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:27.736450 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-147fcf85-zdc8l"] Oct 31 05:12:27.737252 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:27.737232 5019 cpu_manager.go:414] "containerMap: removing container" podUID="b132bea6-493c-43f7-bd51-29f02a01ae9a" containerName="istio-proxy" Oct 31 05:12:27.737367 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:27.737254 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b132bea6-493c-43f7-bd51-29f02a01ae9a" containerName="istio-proxy" Oct 31 05:12:27.737367 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:27.737266 5019 cpu_manager.go:414] "containerMap: removing container" podUID="2f84895b-57a1-4c93-b20c-dc3f2ca754b4" containerName="istio-proxy" Oct 31 05:12:27.737367 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:27.737272 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f84895b-57a1-4c93-b20c-dc3f2ca754b4" containerName="istio-proxy" Oct 31 05:12:27.737367 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:27.737287 5019 cpu_manager.go:414] "containerMap: removing container" podUID="b132bea6-493c-43f7-bd51-29f02a01ae9a" containerName="istio-proxy" Oct 31 05:12:27.737367 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:27.737294 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b132bea6-493c-43f7-bd51-29f02a01ae9a" containerName="istio-proxy" Oct 31 05:12:27.737367 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:27.737318 5019 cpu_manager.go:414] "containerMap: removing container" podUID="7d2f4246-7ce7-475e-87ba-a8afdfec5b00" containerName="payload-pre-processing" Oct 31 05:12:27.737367 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:27.737327 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d2f4246-7ce7-475e-87ba-a8afdfec5b00" containerName="payload-pre-processing" Oct 31 05:12:27.737367 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:27.737351 5019 cpu_manager.go:414] "containerMap: removing container" podUID="c4a5e443-a1d7-4882-b86c-e6022b980073" containerName="istio-proxy" Oct 31 05:12:27.737367 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:27.737356 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4a5e443-a1d7-4882-b86c-e6022b980073" containerName="istio-proxy" Oct 31 05:12:27.737873 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:27.737375 5019 cpu_manager.go:414] "containerMap: removing container" podUID="c4a5e443-a1d7-4882-b86c-e6022b980073" containerName="istio-proxy" Oct 31 05:12:27.737873 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:27.737380 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4a5e443-a1d7-4882-b86c-e6022b980073" containerName="istio-proxy" Oct 31 05:12:27.737873 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:27.737407 5019 cpu_manager.go:414] "containerMap: removing container" podUID="99145b1d-f102-4e18-942d-aee106ab1bdd" containerName="payload-pre-processing" Oct 31 05:12:27.737873 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:27.737412 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99145b1d-f102-4e18-942d-aee106ab1bdd" containerName="payload-pre-processing" Oct 31 05:12:27.737873 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:27.737420 5019 cpu_manager.go:414] "containerMap: removing container" podUID="c2171746-75af-4bba-bf0a-1f065df42461" containerName="revoke-keys" Oct 31 05:12:27.737873 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:27.737425 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2171746-75af-4bba-bf0a-1f065df42461" containerName="revoke-keys" Oct 31 05:12:27.737873 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:27.737541 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d2f4246-7ce7-475e-87ba-a8afdfec5b00" containerName="payload-pre-processing" Oct 31 05:12:27.737873 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:27.737558 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4a5e443-a1d7-4882-b86c-e6022b980073" containerName="istio-proxy" Oct 31 05:12:27.737873 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:27.737567 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4a5e443-a1d7-4882-b86c-e6022b980073" containerName="istio-proxy" Oct 31 05:12:27.737873 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:27.737577 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="99145b1d-f102-4e18-942d-aee106ab1bdd" containerName="payload-pre-processing" Oct 31 05:12:27.737873 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:27.737583 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2171746-75af-4bba-bf0a-1f065df42461" containerName="revoke-keys" Oct 31 05:12:27.737873 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:27.737591 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="b132bea6-493c-43f7-bd51-29f02a01ae9a" containerName="istio-proxy" Oct 31 05:12:27.737873 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:27.737597 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f84895b-57a1-4c93-b20c-dc3f2ca754b4" containerName="istio-proxy" Oct 31 05:12:27.737873 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:27.737604 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="b132bea6-493c-43f7-bd51-29f02a01ae9a" containerName="istio-proxy" Oct 31 05:12:27.737873 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:27.737610 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f84895b-57a1-4c93-b20c-dc3f2ca754b4" containerName="istio-proxy" Oct 31 05:12:27.742787 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:27.742748 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-147fcf85-zdc8l" Oct 31 05:12:27.758861 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:27.758826 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-147fcf85-zdc8l"] Oct 31 05:12:27.809944 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:27.809905 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:12:27.810150 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:27.809952 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:12:27.816187 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:27.816097 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:12:27.816187 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:27.816156 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 05:12:27.823036 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:27.822998 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-77a153-6c975c99f5-mp8vx" podUID="7139f8fd-4b2e-4c1e-9111-92e9b1bd1f62" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:12:27.846822 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:27.846782 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faf366d6-a221-44f6-b362-0630b46fa956-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-147fcf85-zdc8l\" (UID: \"faf366d6-a221-44f6-b362-0630b46fa956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-147fcf85-zdc8l" Oct 31 05:12:27.846996 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:27.846894 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faf366d6-a221-44f6-b362-0630b46fa956-maas-api-xks-ca\") pod \"maas-api-revoke-sub-147fcf85-zdc8l\" (UID: \"faf366d6-a221-44f6-b362-0630b46fa956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-147fcf85-zdc8l" Oct 31 05:12:27.846996 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:27.846931 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sm5s\" (UniqueName: \"kubernetes.io/projected/faf366d6-a221-44f6-b362-0630b46fa956-kube-api-access-5sm5s\") pod \"maas-api-revoke-sub-147fcf85-zdc8l\" (UID: \"faf366d6-a221-44f6-b362-0630b46fa956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-147fcf85-zdc8l" Oct 31 05:12:27.948345 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:27.948294 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faf366d6-a221-44f6-b362-0630b46fa956-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-147fcf85-zdc8l\" (UID: \"faf366d6-a221-44f6-b362-0630b46fa956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-147fcf85-zdc8l" Oct 31 05:12:27.948538 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:27.948391 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faf366d6-a221-44f6-b362-0630b46fa956-maas-api-xks-ca\") pod \"maas-api-revoke-sub-147fcf85-zdc8l\" (UID: \"faf366d6-a221-44f6-b362-0630b46fa956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-147fcf85-zdc8l" Oct 31 05:12:27.948538 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:27.948429 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sm5s\" (UniqueName: \"kubernetes.io/projected/faf366d6-a221-44f6-b362-0630b46fa956-kube-api-access-5sm5s\") pod \"maas-api-revoke-sub-147fcf85-zdc8l\" (UID: \"faf366d6-a221-44f6-b362-0630b46fa956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-147fcf85-zdc8l" Oct 31 05:12:27.949018 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:27.948995 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faf366d6-a221-44f6-b362-0630b46fa956-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-147fcf85-zdc8l\" (UID: \"faf366d6-a221-44f6-b362-0630b46fa956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-147fcf85-zdc8l" Oct 31 05:12:27.950907 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:27.950886 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faf366d6-a221-44f6-b362-0630b46fa956-maas-api-xks-ca\") pod \"maas-api-revoke-sub-147fcf85-zdc8l\" (UID: \"faf366d6-a221-44f6-b362-0630b46fa956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-147fcf85-zdc8l" Oct 31 05:12:27.972775 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:27.972737 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sm5s\" (UniqueName: \"kubernetes.io/projected/faf366d6-a221-44f6-b362-0630b46fa956-kube-api-access-5sm5s\") pod \"maas-api-revoke-sub-147fcf85-zdc8l\" (UID: \"faf366d6-a221-44f6-b362-0630b46fa956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-147fcf85-zdc8l" Oct 31 05:12:28.028784 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:28.028733 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6190e705-ksd9m"] Oct 31 05:12:28.032645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:28.032614 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6190e705-ksd9m"] Oct 31 05:12:28.059579 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:28.059529 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-147fcf85-zdc8l" Oct 31 05:12:28.319088 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:28.319027 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-031745-6d7b5f78dc-6k5z9" podUID="3456208b-2c04-4e78-a654-fafe097e110f" containerName="payload-processing" containerID="cri-o://692485e1b18974950fc2c514d1d936c2595a8d004b1a7c4d108c2c7005c86ae7" gracePeriod=25 Oct 31 05:12:28.362564 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:28.362455 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-031745-74cd69fbc5-s7bs2" podUID="bc856b70-497c-4c46-9db5-f41282c6b91d" containerName="payload-pre-processing" containerID="cri-o://a83e70acf36eb980d48e39d998c744f45ee371cc4dadebd59c9c4e9b800dcb2b" gracePeriod=25 Oct 31 05:12:28.385529 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:12:28.385488 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaf366d6_a221_44f6_b362_0630b46fa956.slice/crio-51e113420767e10a0146d4094fcdb2d04d0243d80d8b6acb9cd5ae6a5c647011 WatchSource:0}: Error finding container 51e113420767e10a0146d4094fcdb2d04d0243d80d8b6acb9cd5ae6a5c647011: Status 404 returned error can't find the container with id 51e113420767e10a0146d4094fcdb2d04d0243d80d8b6acb9cd5ae6a5c647011 Oct 31 05:12:28.385765 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:28.385737 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-147fcf85-zdc8l"] Oct 31 05:12:28.424915 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:28.424875 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-147fcf85-zdc8l" event={"ID":"faf366d6-a221-44f6-b362-0630b46fa956","Type":"ContainerStarted","Data":"51e113420767e10a0146d4094fcdb2d04d0243d80d8b6acb9cd5ae6a5c647011"} Oct 31 05:12:29.308105 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:29.308062 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml"] Oct 31 05:12:29.308714 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:29.308305 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml" podUID="3361c865-a311-4ae0-a040-e3ffade9e514" containerName="istio-proxy" containerID="cri-o://ae6ace2126cc015b6e841467cfd54213649f609c523d09fc76c89eeee4d2812c" gracePeriod=30 Oct 31 05:12:29.542361 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:29.532922 5019 generic.go:356] "Generic (PLEG): container finished" podID="faf366d6-a221-44f6-b362-0630b46fa956" containerID="9ac92aa4b28fc42e0b2ad3b6421c5549a5520b83798cd261586e7fe65c0ee40e" exitCode=0 Oct 31 05:12:29.559777 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:29.559679 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bf2369f-ee6d-4a56-9abb-1a9204ea03ef" path="/var/lib/kubelet/pods/4bf2369f-ee6d-4a56-9abb-1a9204ea03ef/volumes" Oct 31 05:12:29.560794 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:29.560762 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-147fcf85-zdc8l" event={"ID":"faf366d6-a221-44f6-b362-0630b46fa956","Type":"ContainerDied","Data":"9ac92aa4b28fc42e0b2ad3b6421c5549a5520b83798cd261586e7fe65c0ee40e"} Oct 31 05:12:30.901848 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:30.901821 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-147fcf85-zdc8l" Oct 31 05:12:31.012527 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:31.012472 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faf366d6-a221-44f6-b362-0630b46fa956-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faf366d6-a221-44f6-b362-0630b46fa956-maas-api-openshift-ca\") pod \"faf366d6-a221-44f6-b362-0630b46fa956\" (UID: \"faf366d6-a221-44f6-b362-0630b46fa956\") " Oct 31 05:12:31.012739 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:31.012543 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faf366d6-a221-44f6-b362-0630b46fa956-kube-api-access-5sm5s\" (UniqueName: \"kubernetes.io/projected/faf366d6-a221-44f6-b362-0630b46fa956-kube-api-access-5sm5s\") pod \"faf366d6-a221-44f6-b362-0630b46fa956\" (UID: \"faf366d6-a221-44f6-b362-0630b46fa956\") " Oct 31 05:12:31.012739 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:31.012596 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/faf366d6-a221-44f6-b362-0630b46fa956-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faf366d6-a221-44f6-b362-0630b46fa956-maas-api-xks-ca\") pod \"faf366d6-a221-44f6-b362-0630b46fa956\" (UID: \"faf366d6-a221-44f6-b362-0630b46fa956\") " Oct 31 05:12:31.012970 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:31.012939 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faf366d6-a221-44f6-b362-0630b46fa956-maas-api-openshift-ca" pod "faf366d6-a221-44f6-b362-0630b46fa956" (UID: "faf366d6-a221-44f6-b362-0630b46fa956"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:12:31.013296 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:31.013274 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faf366d6-a221-44f6-b362-0630b46fa956-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:12:31.015029 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:31.014998 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/faf366d6-a221-44f6-b362-0630b46fa956-maas-api-xks-ca" pod "faf366d6-a221-44f6-b362-0630b46fa956" (UID: "faf366d6-a221-44f6-b362-0630b46fa956"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:12:31.015229 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:31.015211 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faf366d6-a221-44f6-b362-0630b46fa956-kube-api-access-5sm5s" pod "faf366d6-a221-44f6-b362-0630b46fa956" (UID: "faf366d6-a221-44f6-b362-0630b46fa956"). InnerVolumeSpecName "kube-api-access-5sm5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:12:31.115442 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:31.115395 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sm5s\" (UniqueName: \"kubernetes.io/projected/faf366d6-a221-44f6-b362-0630b46fa956-kube-api-access-5sm5s\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:12:31.115442 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:31.115441 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faf366d6-a221-44f6-b362-0630b46fa956-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:12:31.613034 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:31.613001 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-147fcf85-zdc8l" Oct 31 05:12:31.613253 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:31.613025 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-147fcf85-zdc8l" event={"ID":"faf366d6-a221-44f6-b362-0630b46fa956","Type":"ContainerDied","Data":"51e113420767e10a0146d4094fcdb2d04d0243d80d8b6acb9cd5ae6a5c647011"} Oct 31 05:12:31.613253 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:31.613081 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51e113420767e10a0146d4094fcdb2d04d0243d80d8b6acb9cd5ae6a5c647011" Oct 31 05:12:33.038584 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:33.038541 5019 patch_prober.go:29] interesting pod/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 05:12:33.039092 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:33.038624 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml" podUID="3361c865-a311-4ae0-a040-e3ffade9e514" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 05:12:33.759778 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:33.759732 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-031745-74cd69fbc5-s7bs2" podUID="bc856b70-497c-4c46-9db5-f41282c6b91d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:12:33.759981 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:33.759850 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-031745-6d7b5f78dc-6k5z9" podUID="3456208b-2c04-4e78-a654-fafe097e110f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:12:34.624998 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:34.624963 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml" Oct 31 05:12:34.644537 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:34.644494 5019 generic.go:356] "Generic (PLEG): container finished" podID="3361c865-a311-4ae0-a040-e3ffade9e514" containerID="ae6ace2126cc015b6e841467cfd54213649f609c523d09fc76c89eeee4d2812c" exitCode=0 Oct 31 05:12:34.644758 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:34.644557 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml" Oct 31 05:12:34.644758 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:34.644587 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml" event={"ID":"3361c865-a311-4ae0-a040-e3ffade9e514","Type":"ContainerDied","Data":"ae6ace2126cc015b6e841467cfd54213649f609c523d09fc76c89eeee4d2812c"} Oct 31 05:12:34.644758 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:34.644659 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml" event={"ID":"3361c865-a311-4ae0-a040-e3ffade9e514","Type":"ContainerDied","Data":"2df6f79e3254a0b819a72d3f5907c3f9c8a7af06ebb023205065c5ca5fea1384"} Oct 31 05:12:34.644758 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:34.644693 5019 scope.go:122] "RemoveContainer" containerID="ae6ace2126cc015b6e841467cfd54213649f609c523d09fc76c89eeee4d2812c" Oct 31 05:12:34.657216 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:34.657191 5019 scope.go:122] "RemoveContainer" containerID="fdfde4c344f59c6557677bb6ce5ea995e444e764c95047966893ac06fb261af7" Oct 31 05:12:34.668925 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:34.668899 5019 scope.go:122] "RemoveContainer" containerID="ae6ace2126cc015b6e841467cfd54213649f609c523d09fc76c89eeee4d2812c" Oct 31 05:12:34.669296 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:12:34.669274 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae6ace2126cc015b6e841467cfd54213649f609c523d09fc76c89eeee4d2812c\": container with ID starting with ae6ace2126cc015b6e841467cfd54213649f609c523d09fc76c89eeee4d2812c not found: ID does not exist" containerID="ae6ace2126cc015b6e841467cfd54213649f609c523d09fc76c89eeee4d2812c" Oct 31 05:12:34.669403 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:34.669308 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae6ace2126cc015b6e841467cfd54213649f609c523d09fc76c89eeee4d2812c"} err="failed to get container status \"ae6ace2126cc015b6e841467cfd54213649f609c523d09fc76c89eeee4d2812c\": rpc error: code = NotFound desc = could not find container \"ae6ace2126cc015b6e841467cfd54213649f609c523d09fc76c89eeee4d2812c\": container with ID starting with ae6ace2126cc015b6e841467cfd54213649f609c523d09fc76c89eeee4d2812c not found: ID does not exist" Oct 31 05:12:34.669403 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:34.669339 5019 scope.go:122] "RemoveContainer" containerID="fdfde4c344f59c6557677bb6ce5ea995e444e764c95047966893ac06fb261af7" Oct 31 05:12:34.669625 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:12:34.669606 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdfde4c344f59c6557677bb6ce5ea995e444e764c95047966893ac06fb261af7\": container with ID starting with fdfde4c344f59c6557677bb6ce5ea995e444e764c95047966893ac06fb261af7 not found: ID does not exist" containerID="fdfde4c344f59c6557677bb6ce5ea995e444e764c95047966893ac06fb261af7" Oct 31 05:12:34.669679 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:34.669630 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdfde4c344f59c6557677bb6ce5ea995e444e764c95047966893ac06fb261af7"} err="failed to get container status \"fdfde4c344f59c6557677bb6ce5ea995e444e764c95047966893ac06fb261af7\": rpc error: code = NotFound desc = could not find container \"fdfde4c344f59c6557677bb6ce5ea995e444e764c95047966893ac06fb261af7\": container with ID starting with fdfde4c344f59c6557677bb6ce5ea995e444e764c95047966893ac06fb261af7 not found: ID does not exist" Oct 31 05:12:34.677285 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:34.677240 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3361c865-a311-4ae0-a040-e3ffade9e514-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3361c865-a311-4ae0-a040-e3ffade9e514-istiod-ca-cert\") pod \"3361c865-a311-4ae0-a040-e3ffade9e514\" (UID: \"3361c865-a311-4ae0-a040-e3ffade9e514\") " Oct 31 05:12:34.677477 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:34.677397 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3361c865-a311-4ae0-a040-e3ffade9e514-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3361c865-a311-4ae0-a040-e3ffade9e514-workload-socket\") pod \"3361c865-a311-4ae0-a040-e3ffade9e514\" (UID: \"3361c865-a311-4ae0-a040-e3ffade9e514\") " Oct 31 05:12:34.677552 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:34.677475 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3361c865-a311-4ae0-a040-e3ffade9e514-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3361c865-a311-4ae0-a040-e3ffade9e514-credential-socket\") pod \"3361c865-a311-4ae0-a040-e3ffade9e514\" (UID: \"3361c865-a311-4ae0-a040-e3ffade9e514\") " Oct 31 05:12:34.677552 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:34.677518 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3361c865-a311-4ae0-a040-e3ffade9e514-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3361c865-a311-4ae0-a040-e3ffade9e514-istio-envoy\") pod \"3361c865-a311-4ae0-a040-e3ffade9e514\" (UID: \"3361c865-a311-4ae0-a040-e3ffade9e514\") " Oct 31 05:12:34.677675 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:34.677588 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3361c865-a311-4ae0-a040-e3ffade9e514-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3361c865-a311-4ae0-a040-e3ffade9e514-istio-podinfo\") pod \"3361c865-a311-4ae0-a040-e3ffade9e514\" (UID: \"3361c865-a311-4ae0-a040-e3ffade9e514\") " Oct 31 05:12:34.677675 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:34.677649 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3361c865-a311-4ae0-a040-e3ffade9e514-kube-api-access-vcfj9\" (UniqueName: \"kubernetes.io/projected/3361c865-a311-4ae0-a040-e3ffade9e514-kube-api-access-vcfj9\") pod \"3361c865-a311-4ae0-a040-e3ffade9e514\" (UID: \"3361c865-a311-4ae0-a040-e3ffade9e514\") " Oct 31 05:12:34.677788 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:34.677707 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3361c865-a311-4ae0-a040-e3ffade9e514-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3361c865-a311-4ae0-a040-e3ffade9e514-istio-data\") pod \"3361c865-a311-4ae0-a040-e3ffade9e514\" (UID: \"3361c865-a311-4ae0-a040-e3ffade9e514\") " Oct 31 05:12:34.677788 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:34.677745 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3361c865-a311-4ae0-a040-e3ffade9e514-workload-socket" pod "3361c865-a311-4ae0-a040-e3ffade9e514" (UID: "3361c865-a311-4ae0-a040-e3ffade9e514"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:12:34.677788 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:34.677766 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3361c865-a311-4ae0-a040-e3ffade9e514-credential-socket" pod "3361c865-a311-4ae0-a040-e3ffade9e514" (UID: "3361c865-a311-4ae0-a040-e3ffade9e514"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:12:34.677959 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:34.677664 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3361c865-a311-4ae0-a040-e3ffade9e514-istiod-ca-cert" pod "3361c865-a311-4ae0-a040-e3ffade9e514" (UID: "3361c865-a311-4ae0-a040-e3ffade9e514"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:12:34.677959 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:34.677755 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3361c865-a311-4ae0-a040-e3ffade9e514-istio-token\" (UniqueName: \"kubernetes.io/projected/3361c865-a311-4ae0-a040-e3ffade9e514-istio-token\") pod \"3361c865-a311-4ae0-a040-e3ffade9e514\" (UID: \"3361c865-a311-4ae0-a040-e3ffade9e514\") " Oct 31 05:12:34.677959 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:34.677856 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3361c865-a311-4ae0-a040-e3ffade9e514-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3361c865-a311-4ae0-a040-e3ffade9e514-workload-certs\") pod \"3361c865-a311-4ae0-a040-e3ffade9e514\" (UID: \"3361c865-a311-4ae0-a040-e3ffade9e514\") " Oct 31 05:12:34.678135 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:34.677984 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3361c865-a311-4ae0-a040-e3ffade9e514-istio-data" pod "3361c865-a311-4ae0-a040-e3ffade9e514" (UID: "3361c865-a311-4ae0-a040-e3ffade9e514"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:12:34.678459 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:34.678224 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3361c865-a311-4ae0-a040-e3ffade9e514-workload-certs" pod "3361c865-a311-4ae0-a040-e3ffade9e514" (UID: "3361c865-a311-4ae0-a040-e3ffade9e514"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:12:34.679211 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:34.679172 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3361c865-a311-4ae0-a040-e3ffade9e514-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:12:34.679211 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:34.679202 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3361c865-a311-4ae0-a040-e3ffade9e514-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:12:34.679436 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:34.679217 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3361c865-a311-4ae0-a040-e3ffade9e514-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:12:34.679436 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:34.679231 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3361c865-a311-4ae0-a040-e3ffade9e514-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:12:34.679436 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:34.679244 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3361c865-a311-4ae0-a040-e3ffade9e514-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:12:34.680205 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:34.680174 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3361c865-a311-4ae0-a040-e3ffade9e514-istio-podinfo" pod "3361c865-a311-4ae0-a040-e3ffade9e514" (UID: "3361c865-a311-4ae0-a040-e3ffade9e514"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:12:34.680422 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:34.680395 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3361c865-a311-4ae0-a040-e3ffade9e514-istio-envoy" pod "3361c865-a311-4ae0-a040-e3ffade9e514" (UID: "3361c865-a311-4ae0-a040-e3ffade9e514"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:12:34.680520 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:34.680432 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3361c865-a311-4ae0-a040-e3ffade9e514-kube-api-access-vcfj9" pod "3361c865-a311-4ae0-a040-e3ffade9e514" (UID: "3361c865-a311-4ae0-a040-e3ffade9e514"). InnerVolumeSpecName "kube-api-access-vcfj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:12:34.680520 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:34.680472 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3361c865-a311-4ae0-a040-e3ffade9e514-istio-token" pod "3361c865-a311-4ae0-a040-e3ffade9e514" (UID: "3361c865-a311-4ae0-a040-e3ffade9e514"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:12:34.781558 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:34.781515 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3361c865-a311-4ae0-a040-e3ffade9e514-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:12:34.781558 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:34.781560 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3361c865-a311-4ae0-a040-e3ffade9e514-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:12:34.781847 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:34.781574 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3361c865-a311-4ae0-a040-e3ffade9e514-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:12:34.781847 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:34.781590 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcfj9\" (UniqueName: \"kubernetes.io/projected/3361c865-a311-4ae0-a040-e3ffade9e514-kube-api-access-vcfj9\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:12:34.992992 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:34.992948 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml"] Oct 31 05:12:35.004391 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:35.004349 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-51cc4ce6-openshift-default-9f7d7c9c-7v7ml"] Oct 31 05:12:35.412816 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:35.412776 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3361c865-a311-4ae0-a040-e3ffade9e514" path="/var/lib/kubelet/pods/3361c865-a311-4ae0-a040-e3ffade9e514/volumes" Oct 31 05:12:36.805145 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:36.805101 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-77a153-6d7d74f579-mfw72" podUID="b27570f7-6a81-4a8c-aad3-404126c2b7de" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:12:37.822792 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:37.822750 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-77a153-6c975c99f5-mp8vx" podUID="7139f8fd-4b2e-4c1e-9111-92e9b1bd1f62" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:12:38.143879 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:38.143793 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b9aaa6b-l495d"] Oct 31 05:12:38.144512 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:38.144497 5019 cpu_manager.go:414] "containerMap: removing container" podUID="2f84895b-57a1-4c93-b20c-dc3f2ca754b4" containerName="istio-proxy" Oct 31 05:12:38.144578 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:38.144513 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f84895b-57a1-4c93-b20c-dc3f2ca754b4" containerName="istio-proxy" Oct 31 05:12:38.144578 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:38.144550 5019 cpu_manager.go:414] "containerMap: removing container" podUID="3361c865-a311-4ae0-a040-e3ffade9e514" containerName="istio-proxy" Oct 31 05:12:38.144578 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:38.144555 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3361c865-a311-4ae0-a040-e3ffade9e514" containerName="istio-proxy" Oct 31 05:12:38.144578 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:38.144566 5019 cpu_manager.go:414] "containerMap: removing container" podUID="faf366d6-a221-44f6-b362-0630b46fa956" containerName="revoke-keys" Oct 31 05:12:38.144578 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:38.144572 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faf366d6-a221-44f6-b362-0630b46fa956" containerName="revoke-keys" Oct 31 05:12:38.144578 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:38.144579 5019 cpu_manager.go:414] "containerMap: removing container" podUID="3361c865-a311-4ae0-a040-e3ffade9e514" containerName="istio-proxy" Oct 31 05:12:38.144779 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:38.144584 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3361c865-a311-4ae0-a040-e3ffade9e514" containerName="istio-proxy" Oct 31 05:12:38.144779 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:38.144697 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="faf366d6-a221-44f6-b362-0630b46fa956" containerName="revoke-keys" Oct 31 05:12:38.144779 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:38.144704 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="3361c865-a311-4ae0-a040-e3ffade9e514" containerName="istio-proxy" Oct 31 05:12:38.150239 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:38.150211 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9aaa6b-l495d" Oct 31 05:12:38.167818 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:38.167783 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b9aaa6b-l495d"] Oct 31 05:12:38.177097 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:38.174559 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:12:38.177097 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:38.174659 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:12:38.230562 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:38.230521 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:12:38.230562 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:38.230560 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 05:12:38.234311 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:38.234280 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk7p7\" (UniqueName: \"kubernetes.io/projected/4eb1a2ea-76e1-4b63-bf5c-3a3fa67afe37-kube-api-access-lk7p7\") pod \"maas-api-revoke-sub-8b9aaa6b-l495d\" (UID: \"4eb1a2ea-76e1-4b63-bf5c-3a3fa67afe37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9aaa6b-l495d" Oct 31 05:12:38.234491 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:38.234315 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eb1a2ea-76e1-4b63-bf5c-3a3fa67afe37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b9aaa6b-l495d\" (UID: \"4eb1a2ea-76e1-4b63-bf5c-3a3fa67afe37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9aaa6b-l495d" Oct 31 05:12:38.234549 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:38.234534 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eb1a2ea-76e1-4b63-bf5c-3a3fa67afe37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b9aaa6b-l495d\" (UID: \"4eb1a2ea-76e1-4b63-bf5c-3a3fa67afe37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9aaa6b-l495d" Oct 31 05:12:38.336291 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:38.336248 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk7p7\" (UniqueName: \"kubernetes.io/projected/4eb1a2ea-76e1-4b63-bf5c-3a3fa67afe37-kube-api-access-lk7p7\") pod \"maas-api-revoke-sub-8b9aaa6b-l495d\" (UID: \"4eb1a2ea-76e1-4b63-bf5c-3a3fa67afe37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9aaa6b-l495d" Oct 31 05:12:38.336517 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:38.336294 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eb1a2ea-76e1-4b63-bf5c-3a3fa67afe37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b9aaa6b-l495d\" (UID: \"4eb1a2ea-76e1-4b63-bf5c-3a3fa67afe37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9aaa6b-l495d" Oct 31 05:12:38.336568 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:38.336519 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eb1a2ea-76e1-4b63-bf5c-3a3fa67afe37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b9aaa6b-l495d\" (UID: \"4eb1a2ea-76e1-4b63-bf5c-3a3fa67afe37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9aaa6b-l495d" Oct 31 05:12:38.337028 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:38.337007 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eb1a2ea-76e1-4b63-bf5c-3a3fa67afe37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b9aaa6b-l495d\" (UID: \"4eb1a2ea-76e1-4b63-bf5c-3a3fa67afe37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9aaa6b-l495d" Oct 31 05:12:38.339119 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:38.339100 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eb1a2ea-76e1-4b63-bf5c-3a3fa67afe37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b9aaa6b-l495d\" (UID: \"4eb1a2ea-76e1-4b63-bf5c-3a3fa67afe37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9aaa6b-l495d" Oct 31 05:12:38.359948 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:38.359919 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk7p7\" (UniqueName: \"kubernetes.io/projected/4eb1a2ea-76e1-4b63-bf5c-3a3fa67afe37-kube-api-access-lk7p7\") pod \"maas-api-revoke-sub-8b9aaa6b-l495d\" (UID: \"4eb1a2ea-76e1-4b63-bf5c-3a3fa67afe37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9aaa6b-l495d" Oct 31 05:12:38.472210 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:38.472174 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9aaa6b-l495d" Oct 31 05:12:38.790647 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:38.790603 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b9aaa6b-l495d"] Oct 31 05:12:38.792714 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:12:38.792683 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eb1a2ea_76e1_4b63_bf5c_3a3fa67afe37.slice/crio-0cdce6654c514bb84d8c9197a7bdde1c29675e904b8d10d9f7f01eea6358e1aa WatchSource:0}: Error finding container 0cdce6654c514bb84d8c9197a7bdde1c29675e904b8d10d9f7f01eea6358e1aa: Status 404 returned error can't find the container with id 0cdce6654c514bb84d8c9197a7bdde1c29675e904b8d10d9f7f01eea6358e1aa Oct 31 05:12:39.695800 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:39.695753 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9aaa6b-l495d" event={"ID":"4eb1a2ea-76e1-4b63-bf5c-3a3fa67afe37","Type":"ContainerStarted","Data":"8c839258577a8a0cfa16d74cd944604839e85ff47434d229d3c23d029d9c0d3e"} Oct 31 05:12:39.696697 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:39.695820 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9aaa6b-l495d" event={"ID":"4eb1a2ea-76e1-4b63-bf5c-3a3fa67afe37","Type":"ContainerStarted","Data":"0cdce6654c514bb84d8c9197a7bdde1c29675e904b8d10d9f7f01eea6358e1aa"} Oct 31 05:12:39.736830 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:39.736769 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9aaa6b-l495d" podStartSLOduration=1.736753251 podStartE2EDuration="1.736753251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:12:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:12:39.734859786 +0000 UTC m=+4114.491197826" watchObservedRunningTime="2026-10-31 05:12:39.736753251 +0000 UTC m=+4114.493091291" Oct 31 05:12:40.705526 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:40.705476 5019 generic.go:356] "Generic (PLEG): container finished" podID="4eb1a2ea-76e1-4b63-bf5c-3a3fa67afe37" containerID="8c839258577a8a0cfa16d74cd944604839e85ff47434d229d3c23d029d9c0d3e" exitCode=0 Oct 31 05:12:40.706022 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:40.705533 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9aaa6b-l495d" event={"ID":"4eb1a2ea-76e1-4b63-bf5c-3a3fa67afe37","Type":"ContainerDied","Data":"8c839258577a8a0cfa16d74cd944604839e85ff47434d229d3c23d029d9c0d3e"} Oct 31 05:12:41.895295 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:41.895245 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9aaa6b-l495d" Oct 31 05:12:42.004546 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:42.004510 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4eb1a2ea-76e1-4b63-bf5c-3a3fa67afe37-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eb1a2ea-76e1-4b63-bf5c-3a3fa67afe37-maas-api-xks-ca\") pod \"4eb1a2ea-76e1-4b63-bf5c-3a3fa67afe37\" (UID: \"4eb1a2ea-76e1-4b63-bf5c-3a3fa67afe37\") " Oct 31 05:12:42.004747 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:42.004592 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4eb1a2ea-76e1-4b63-bf5c-3a3fa67afe37-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eb1a2ea-76e1-4b63-bf5c-3a3fa67afe37-maas-api-openshift-ca\") pod \"4eb1a2ea-76e1-4b63-bf5c-3a3fa67afe37\" (UID: \"4eb1a2ea-76e1-4b63-bf5c-3a3fa67afe37\") " Oct 31 05:12:42.004747 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:42.004625 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eb1a2ea-76e1-4b63-bf5c-3a3fa67afe37-kube-api-access-lk7p7\" (UniqueName: \"kubernetes.io/projected/4eb1a2ea-76e1-4b63-bf5c-3a3fa67afe37-kube-api-access-lk7p7\") pod \"4eb1a2ea-76e1-4b63-bf5c-3a3fa67afe37\" (UID: \"4eb1a2ea-76e1-4b63-bf5c-3a3fa67afe37\") " Oct 31 05:12:42.005067 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:42.005007 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eb1a2ea-76e1-4b63-bf5c-3a3fa67afe37-maas-api-openshift-ca" pod "4eb1a2ea-76e1-4b63-bf5c-3a3fa67afe37" (UID: "4eb1a2ea-76e1-4b63-bf5c-3a3fa67afe37"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:12:42.005152 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:42.005133 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eb1a2ea-76e1-4b63-bf5c-3a3fa67afe37-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:12:42.007035 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:42.007006 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eb1a2ea-76e1-4b63-bf5c-3a3fa67afe37-kube-api-access-lk7p7" pod "4eb1a2ea-76e1-4b63-bf5c-3a3fa67afe37" (UID: "4eb1a2ea-76e1-4b63-bf5c-3a3fa67afe37"). InnerVolumeSpecName "kube-api-access-lk7p7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:12:42.007035 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:42.007019 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eb1a2ea-76e1-4b63-bf5c-3a3fa67afe37-maas-api-xks-ca" pod "4eb1a2ea-76e1-4b63-bf5c-3a3fa67afe37" (UID: "4eb1a2ea-76e1-4b63-bf5c-3a3fa67afe37"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:12:42.106767 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:42.106724 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eb1a2ea-76e1-4b63-bf5c-3a3fa67afe37-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:12:42.106767 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:42.106762 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk7p7\" (UniqueName: \"kubernetes.io/projected/4eb1a2ea-76e1-4b63-bf5c-3a3fa67afe37-kube-api-access-lk7p7\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:12:42.726448 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:42.726412 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9aaa6b-l495d" Oct 31 05:12:42.726646 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:42.726443 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9aaa6b-l495d" event={"ID":"4eb1a2ea-76e1-4b63-bf5c-3a3fa67afe37","Type":"ContainerDied","Data":"0cdce6654c514bb84d8c9197a7bdde1c29675e904b8d10d9f7f01eea6358e1aa"} Oct 31 05:12:42.726646 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:42.726493 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0cdce6654c514bb84d8c9197a7bdde1c29675e904b8d10d9f7f01eea6358e1aa" Oct 31 05:12:43.829356 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:43.825467 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-031745-6d7b5f78dc-6k5z9" podUID="3456208b-2c04-4e78-a654-fafe097e110f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:12:43.829356 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:43.826011 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-031745-74cd69fbc5-s7bs2" podUID="bc856b70-497c-4c46-9db5-f41282c6b91d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:12:46.805629 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:46.805587 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-77a153-6d7d74f579-mfw72" podUID="b27570f7-6a81-4a8c-aad3-404126c2b7de" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:12:47.840307 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:47.840213 5019 generic.go:356] "Generic (PLEG): container finished" podID="7139f8fd-4b2e-4c1e-9111-92e9b1bd1f62" containerID="6e803cd33fa32f8e539c094464e09cbca4efeb86359db641670212855553a5f6" exitCode=0 Oct 31 05:12:47.841146 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:47.840885 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-77a153-6c975c99f5-mp8vx" event={"ID":"7139f8fd-4b2e-4c1e-9111-92e9b1bd1f62","Type":"ContainerDied","Data":"6e803cd33fa32f8e539c094464e09cbca4efeb86359db641670212855553a5f6"} Oct 31 05:12:47.844254 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:47.843839 5019 generic.go:356] "Generic (PLEG): container finished" podID="b27570f7-6a81-4a8c-aad3-404126c2b7de" containerID="055909b4eaf8584cd37ac55ea33ad929c6f418ec6d7fedc9fe43d41caccfa244" exitCode=0 Oct 31 05:12:47.844254 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:47.844126 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-77a153-6d7d74f579-mfw72" event={"ID":"b27570f7-6a81-4a8c-aad3-404126c2b7de","Type":"ContainerDied","Data":"055909b4eaf8584cd37ac55ea33ad929c6f418ec6d7fedc9fe43d41caccfa244"} Oct 31 05:12:47.933952 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:47.933920 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-77a153-6c975c99f5-mp8vx" Oct 31 05:12:48.006788 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.006748 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-77a153-6d7d74f579-mfw72" Oct 31 05:12:48.040252 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.040210 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7139f8fd-4b2e-4c1e-9111-92e9b1bd1f62-kube-api-access-scvd9\" (UniqueName: \"kubernetes.io/projected/7139f8fd-4b2e-4c1e-9111-92e9b1bd1f62-kube-api-access-scvd9\") pod \"7139f8fd-4b2e-4c1e-9111-92e9b1bd1f62\" (UID: \"7139f8fd-4b2e-4c1e-9111-92e9b1bd1f62\") " Oct 31 05:12:48.040457 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.040350 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7139f8fd-4b2e-4c1e-9111-92e9b1bd1f62-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7139f8fd-4b2e-4c1e-9111-92e9b1bd1f62-plugins-config-volume\") pod \"7139f8fd-4b2e-4c1e-9111-92e9b1bd1f62\" (UID: \"7139f8fd-4b2e-4c1e-9111-92e9b1bd1f62\") " Oct 31 05:12:48.040457 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.040418 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b27570f7-6a81-4a8c-aad3-404126c2b7de-kube-api-access-wnlpx\" (UniqueName: \"kubernetes.io/projected/b27570f7-6a81-4a8c-aad3-404126c2b7de-kube-api-access-wnlpx\") pod \"b27570f7-6a81-4a8c-aad3-404126c2b7de\" (UID: \"b27570f7-6a81-4a8c-aad3-404126c2b7de\") " Oct 31 05:12:48.040585 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.040461 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b27570f7-6a81-4a8c-aad3-404126c2b7de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b27570f7-6a81-4a8c-aad3-404126c2b7de-plugins-config-volume\") pod \"b27570f7-6a81-4a8c-aad3-404126c2b7de\" (UID: \"b27570f7-6a81-4a8c-aad3-404126c2b7de\") " Oct 31 05:12:48.040830 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.040801 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7139f8fd-4b2e-4c1e-9111-92e9b1bd1f62-plugins-config-volume" pod "7139f8fd-4b2e-4c1e-9111-92e9b1bd1f62" (UID: "7139f8fd-4b2e-4c1e-9111-92e9b1bd1f62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:12:48.040964 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.040876 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b27570f7-6a81-4a8c-aad3-404126c2b7de-plugins-config-volume" pod "b27570f7-6a81-4a8c-aad3-404126c2b7de" (UID: "b27570f7-6a81-4a8c-aad3-404126c2b7de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:12:48.041069 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.041032 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7139f8fd-4b2e-4c1e-9111-92e9b1bd1f62-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:12:48.041069 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.041050 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b27570f7-6a81-4a8c-aad3-404126c2b7de-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:12:48.043275 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.043242 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7139f8fd-4b2e-4c1e-9111-92e9b1bd1f62-kube-api-access-scvd9" pod "7139f8fd-4b2e-4c1e-9111-92e9b1bd1f62" (UID: "7139f8fd-4b2e-4c1e-9111-92e9b1bd1f62"). InnerVolumeSpecName "kube-api-access-scvd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:12:48.043451 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.043283 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b27570f7-6a81-4a8c-aad3-404126c2b7de-kube-api-access-wnlpx" pod "b27570f7-6a81-4a8c-aad3-404126c2b7de" (UID: "b27570f7-6a81-4a8c-aad3-404126c2b7de"). InnerVolumeSpecName "kube-api-access-wnlpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:12:48.142827 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.142724 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnlpx\" (UniqueName: \"kubernetes.io/projected/b27570f7-6a81-4a8c-aad3-404126c2b7de-kube-api-access-wnlpx\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:12:48.142827 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.142761 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scvd9\" (UniqueName: \"kubernetes.io/projected/7139f8fd-4b2e-4c1e-9111-92e9b1bd1f62-kube-api-access-scvd9\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:12:48.729969 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.729929 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-031745-6d7b5f78dc-6k5z9" Oct 31 05:12:48.811565 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.811523 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-031745-74cd69fbc5-s7bs2" Oct 31 05:12:48.822562 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.822514 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-77a153-6c975c99f5-mp8vx" podUID="7139f8fd-4b2e-4c1e-9111-92e9b1bd1f62" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.241:50052\" within 1s: context deadline exceeded" Oct 31 05:12:48.822765 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.822614 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-77a153-6c975c99f5-mp8vx" Oct 31 05:12:48.822765 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:12:48.822646 5019 logging.go:55] [core] [Channel #18639 SubChannel #18640]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.241:50052", ServerName: "10.132.0.241:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.241:50052: operation was canceled" Oct 31 05:12:48.851680 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.851640 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3456208b-2c04-4e78-a654-fafe097e110f-kube-api-access-x9m9m\" (UniqueName: \"kubernetes.io/projected/3456208b-2c04-4e78-a654-fafe097e110f-kube-api-access-x9m9m\") pod \"3456208b-2c04-4e78-a654-fafe097e110f\" (UID: \"3456208b-2c04-4e78-a654-fafe097e110f\") " Oct 31 05:12:48.852199 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.851747 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc856b70-497c-4c46-9db5-f41282c6b91d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc856b70-497c-4c46-9db5-f41282c6b91d-plugins-config-volume\") pod \"bc856b70-497c-4c46-9db5-f41282c6b91d\" (UID: \"bc856b70-497c-4c46-9db5-f41282c6b91d\") " Oct 31 05:12:48.852199 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.851776 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3456208b-2c04-4e78-a654-fafe097e110f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3456208b-2c04-4e78-a654-fafe097e110f-plugins-config-volume\") pod \"3456208b-2c04-4e78-a654-fafe097e110f\" (UID: \"3456208b-2c04-4e78-a654-fafe097e110f\") " Oct 31 05:12:48.852199 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.851819 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc856b70-497c-4c46-9db5-f41282c6b91d-kube-api-access-kzd76\" (UniqueName: \"kubernetes.io/projected/bc856b70-497c-4c46-9db5-f41282c6b91d-kube-api-access-kzd76\") pod \"bc856b70-497c-4c46-9db5-f41282c6b91d\" (UID: \"bc856b70-497c-4c46-9db5-f41282c6b91d\") " Oct 31 05:12:48.853100 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.853063 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc856b70-497c-4c46-9db5-f41282c6b91d-plugins-config-volume" pod "bc856b70-497c-4c46-9db5-f41282c6b91d" (UID: "bc856b70-497c-4c46-9db5-f41282c6b91d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:12:48.853595 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.853559 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3456208b-2c04-4e78-a654-fafe097e110f-plugins-config-volume" pod "3456208b-2c04-4e78-a654-fafe097e110f" (UID: "3456208b-2c04-4e78-a654-fafe097e110f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:12:48.855665 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.855632 5019 generic.go:356] "Generic (PLEG): container finished" podID="bc856b70-497c-4c46-9db5-f41282c6b91d" containerID="a83e70acf36eb980d48e39d998c744f45ee371cc4dadebd59c9c4e9b800dcb2b" exitCode=0 Oct 31 05:12:48.855796 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.855691 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-031745-74cd69fbc5-s7bs2" Oct 31 05:12:48.855796 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.855726 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-031745-74cd69fbc5-s7bs2" event={"ID":"bc856b70-497c-4c46-9db5-f41282c6b91d","Type":"ContainerDied","Data":"a83e70acf36eb980d48e39d998c744f45ee371cc4dadebd59c9c4e9b800dcb2b"} Oct 31 05:12:48.855796 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.855783 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-031745-74cd69fbc5-s7bs2" event={"ID":"bc856b70-497c-4c46-9db5-f41282c6b91d","Type":"ContainerDied","Data":"0d7ca1742a48f5cf43f9f1f21d17ddf970a1537a736458761dfbcd17aed4489e"} Oct 31 05:12:48.856008 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.855809 5019 scope.go:122] "RemoveContainer" containerID="a83e70acf36eb980d48e39d998c744f45ee371cc4dadebd59c9c4e9b800dcb2b" Oct 31 05:12:48.858708 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.858680 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3456208b-2c04-4e78-a654-fafe097e110f-kube-api-access-x9m9m" pod "3456208b-2c04-4e78-a654-fafe097e110f" (UID: "3456208b-2c04-4e78-a654-fafe097e110f"). InnerVolumeSpecName "kube-api-access-x9m9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:12:48.858826 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.858719 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc856b70-497c-4c46-9db5-f41282c6b91d-kube-api-access-kzd76" pod "bc856b70-497c-4c46-9db5-f41282c6b91d" (UID: "bc856b70-497c-4c46-9db5-f41282c6b91d"). InnerVolumeSpecName "kube-api-access-kzd76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:12:48.859136 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.859115 5019 generic.go:356] "Generic (PLEG): container finished" podID="3456208b-2c04-4e78-a654-fafe097e110f" containerID="692485e1b18974950fc2c514d1d936c2595a8d004b1a7c4d108c2c7005c86ae7" exitCode=0 Oct 31 05:12:48.859203 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.859163 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-031745-6d7b5f78dc-6k5z9" Oct 31 05:12:48.859257 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.859209 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-031745-6d7b5f78dc-6k5z9" event={"ID":"3456208b-2c04-4e78-a654-fafe097e110f","Type":"ContainerDied","Data":"692485e1b18974950fc2c514d1d936c2595a8d004b1a7c4d108c2c7005c86ae7"} Oct 31 05:12:48.859328 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.859270 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-031745-6d7b5f78dc-6k5z9" event={"ID":"3456208b-2c04-4e78-a654-fafe097e110f","Type":"ContainerDied","Data":"09c718c218f97218aebe4274411ee26e1b15ef9d06f98e133c5896a495c43c2b"} Oct 31 05:12:48.863739 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.861092 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-77a153-6c975c99f5-mp8vx" Oct 31 05:12:48.863739 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.861122 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-77a153-6c975c99f5-mp8vx" event={"ID":"7139f8fd-4b2e-4c1e-9111-92e9b1bd1f62","Type":"ContainerDied","Data":"6232203bc2fdb0cabab3b96b42aa67c41f6426f4bc0dcd45af988c4f3dc6801c"} Oct 31 05:12:48.863739 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.863599 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-77a153-6d7d74f579-mfw72" Oct 31 05:12:48.863739 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.863596 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-77a153-6d7d74f579-mfw72" event={"ID":"b27570f7-6a81-4a8c-aad3-404126c2b7de","Type":"ContainerDied","Data":"0649e8706ee0dbc33f7a43bd92a91f5923c41ae4d69a86ad67d7c5815566562c"} Oct 31 05:12:48.876062 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.876034 5019 scope.go:122] "RemoveContainer" containerID="a83e70acf36eb980d48e39d998c744f45ee371cc4dadebd59c9c4e9b800dcb2b" Oct 31 05:12:48.877658 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:12:48.876751 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a83e70acf36eb980d48e39d998c744f45ee371cc4dadebd59c9c4e9b800dcb2b\": container with ID starting with a83e70acf36eb980d48e39d998c744f45ee371cc4dadebd59c9c4e9b800dcb2b not found: ID does not exist" containerID="a83e70acf36eb980d48e39d998c744f45ee371cc4dadebd59c9c4e9b800dcb2b" Oct 31 05:12:48.877658 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.876820 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a83e70acf36eb980d48e39d998c744f45ee371cc4dadebd59c9c4e9b800dcb2b"} err="failed to get container status \"a83e70acf36eb980d48e39d998c744f45ee371cc4dadebd59c9c4e9b800dcb2b\": rpc error: code = NotFound desc = could not find container \"a83e70acf36eb980d48e39d998c744f45ee371cc4dadebd59c9c4e9b800dcb2b\": container with ID starting with a83e70acf36eb980d48e39d998c744f45ee371cc4dadebd59c9c4e9b800dcb2b not found: ID does not exist" Oct 31 05:12:48.877658 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.876851 5019 scope.go:122] "RemoveContainer" containerID="692485e1b18974950fc2c514d1d936c2595a8d004b1a7c4d108c2c7005c86ae7" Oct 31 05:12:48.916089 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.916040 5019 scope.go:122] "RemoveContainer" containerID="692485e1b18974950fc2c514d1d936c2595a8d004b1a7c4d108c2c7005c86ae7" Oct 31 05:12:48.916514 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:12:48.916480 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"692485e1b18974950fc2c514d1d936c2595a8d004b1a7c4d108c2c7005c86ae7\": container with ID starting with 692485e1b18974950fc2c514d1d936c2595a8d004b1a7c4d108c2c7005c86ae7 not found: ID does not exist" containerID="692485e1b18974950fc2c514d1d936c2595a8d004b1a7c4d108c2c7005c86ae7" Oct 31 05:12:48.916613 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.916529 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"692485e1b18974950fc2c514d1d936c2595a8d004b1a7c4d108c2c7005c86ae7"} err="failed to get container status \"692485e1b18974950fc2c514d1d936c2595a8d004b1a7c4d108c2c7005c86ae7\": rpc error: code = NotFound desc = could not find container \"692485e1b18974950fc2c514d1d936c2595a8d004b1a7c4d108c2c7005c86ae7\": container with ID starting with 692485e1b18974950fc2c514d1d936c2595a8d004b1a7c4d108c2c7005c86ae7 not found: ID does not exist" Oct 31 05:12:48.916613 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.916556 5019 scope.go:122] "RemoveContainer" containerID="6e803cd33fa32f8e539c094464e09cbca4efeb86359db641670212855553a5f6" Oct 31 05:12:48.931311 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.931258 5019 scope.go:122] "RemoveContainer" containerID="055909b4eaf8584cd37ac55ea33ad929c6f418ec6d7fedc9fe43d41caccfa244" Oct 31 05:12:48.953925 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.953881 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9m9m\" (UniqueName: \"kubernetes.io/projected/3456208b-2c04-4e78-a654-fafe097e110f-kube-api-access-x9m9m\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:12:48.953925 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.953920 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc856b70-497c-4c46-9db5-f41282c6b91d-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:12:48.953925 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.953932 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3456208b-2c04-4e78-a654-fafe097e110f-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:12:48.954193 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.953943 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzd76\" (UniqueName: \"kubernetes.io/projected/bc856b70-497c-4c46-9db5-f41282c6b91d-kube-api-access-kzd76\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:12:48.959844 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.959795 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-77a153-6d7d74f579-mfw72"] Oct 31 05:12:48.964239 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.964203 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-77a153-6d7d74f579-mfw72"] Oct 31 05:12:48.974908 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.974870 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-339900d9-vdj8t"] Oct 31 05:12:48.975998 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.975972 5019 cpu_manager.go:414] "containerMap: removing container" podUID="b27570f7-6a81-4a8c-aad3-404126c2b7de" containerName="payload-pre-processing" Oct 31 05:12:48.976080 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.975999 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b27570f7-6a81-4a8c-aad3-404126c2b7de" containerName="payload-pre-processing" Oct 31 05:12:48.976080 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.976028 5019 cpu_manager.go:414] "containerMap: removing container" podUID="4eb1a2ea-76e1-4b63-bf5c-3a3fa67afe37" containerName="revoke-keys" Oct 31 05:12:48.976080 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.976036 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eb1a2ea-76e1-4b63-bf5c-3a3fa67afe37" containerName="revoke-keys" Oct 31 05:12:48.976080 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.976053 5019 cpu_manager.go:414] "containerMap: removing container" podUID="3456208b-2c04-4e78-a654-fafe097e110f" containerName="payload-processing" Oct 31 05:12:48.976080 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.976062 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3456208b-2c04-4e78-a654-fafe097e110f" containerName="payload-processing" Oct 31 05:12:48.976245 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.976094 5019 cpu_manager.go:414] "containerMap: removing container" podUID="bc856b70-497c-4c46-9db5-f41282c6b91d" containerName="payload-pre-processing" Oct 31 05:12:48.976245 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.976101 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc856b70-497c-4c46-9db5-f41282c6b91d" containerName="payload-pre-processing" Oct 31 05:12:48.976245 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.976126 5019 cpu_manager.go:414] "containerMap: removing container" podUID="7139f8fd-4b2e-4c1e-9111-92e9b1bd1f62" containerName="payload-processing" Oct 31 05:12:48.976245 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.976134 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7139f8fd-4b2e-4c1e-9111-92e9b1bd1f62" containerName="payload-processing" Oct 31 05:12:48.976420 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.976280 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="b27570f7-6a81-4a8c-aad3-404126c2b7de" containerName="payload-pre-processing" Oct 31 05:12:48.976420 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.976294 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="3361c865-a311-4ae0-a040-e3ffade9e514" containerName="istio-proxy" Oct 31 05:12:48.976420 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.976304 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eb1a2ea-76e1-4b63-bf5c-3a3fa67afe37" containerName="revoke-keys" Oct 31 05:12:48.976420 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.976317 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="3456208b-2c04-4e78-a654-fafe097e110f" containerName="payload-processing" Oct 31 05:12:48.976420 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.976362 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc856b70-497c-4c46-9db5-f41282c6b91d" containerName="payload-pre-processing" Oct 31 05:12:48.976420 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.976377 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="7139f8fd-4b2e-4c1e-9111-92e9b1bd1f62" containerName="payload-processing" Oct 31 05:12:48.989404 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.989369 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-339900d9-vdj8t" Oct 31 05:12:48.999429 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:48.999366 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-339900d9-vdj8t"] Oct 31 05:12:49.004519 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:49.004484 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 05:12:49.004689 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:49.004562 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:12:49.006181 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:49.006152 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-77a153-6c975c99f5-mp8vx"] Oct 31 05:12:49.011677 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:49.011632 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-77a153-6c975c99f5-mp8vx"] Oct 31 05:12:49.033589 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:49.033548 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:12:49.034177 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:49.034133 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:12:49.040414 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:49.040370 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-031745-6d7b5f78dc-6k5z9"] Oct 31 05:12:49.047433 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:49.047385 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-031745-6d7b5f78dc-6k5z9"] Oct 31 05:12:49.055171 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:49.055130 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97wxr\" (UniqueName: \"kubernetes.io/projected/e7c18d7a-62a1-4ef6-a835-7e79bd7ed3a0-kube-api-access-97wxr\") pod \"maas-api-revoke-sub-339900d9-vdj8t\" (UID: \"e7c18d7a-62a1-4ef6-a835-7e79bd7ed3a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-339900d9-vdj8t" Oct 31 05:12:49.055407 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:49.055192 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7c18d7a-62a1-4ef6-a835-7e79bd7ed3a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-339900d9-vdj8t\" (UID: \"e7c18d7a-62a1-4ef6-a835-7e79bd7ed3a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-339900d9-vdj8t" Oct 31 05:12:49.055407 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:49.055308 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7c18d7a-62a1-4ef6-a835-7e79bd7ed3a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-339900d9-vdj8t\" (UID: \"e7c18d7a-62a1-4ef6-a835-7e79bd7ed3a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-339900d9-vdj8t" Oct 31 05:12:49.157116 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:49.157076 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97wxr\" (UniqueName: \"kubernetes.io/projected/e7c18d7a-62a1-4ef6-a835-7e79bd7ed3a0-kube-api-access-97wxr\") pod \"maas-api-revoke-sub-339900d9-vdj8t\" (UID: \"e7c18d7a-62a1-4ef6-a835-7e79bd7ed3a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-339900d9-vdj8t" Oct 31 05:12:49.157309 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:49.157144 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7c18d7a-62a1-4ef6-a835-7e79bd7ed3a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-339900d9-vdj8t\" (UID: \"e7c18d7a-62a1-4ef6-a835-7e79bd7ed3a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-339900d9-vdj8t" Oct 31 05:12:49.157309 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:49.157224 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7c18d7a-62a1-4ef6-a835-7e79bd7ed3a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-339900d9-vdj8t\" (UID: \"e7c18d7a-62a1-4ef6-a835-7e79bd7ed3a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-339900d9-vdj8t" Oct 31 05:12:49.157921 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:49.157894 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7c18d7a-62a1-4ef6-a835-7e79bd7ed3a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-339900d9-vdj8t\" (UID: \"e7c18d7a-62a1-4ef6-a835-7e79bd7ed3a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-339900d9-vdj8t" Oct 31 05:12:49.159824 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:49.159795 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7c18d7a-62a1-4ef6-a835-7e79bd7ed3a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-339900d9-vdj8t\" (UID: \"e7c18d7a-62a1-4ef6-a835-7e79bd7ed3a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-339900d9-vdj8t" Oct 31 05:12:49.180824 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:49.180776 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97wxr\" (UniqueName: \"kubernetes.io/projected/e7c18d7a-62a1-4ef6-a835-7e79bd7ed3a0-kube-api-access-97wxr\") pod \"maas-api-revoke-sub-339900d9-vdj8t\" (UID: \"e7c18d7a-62a1-4ef6-a835-7e79bd7ed3a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-339900d9-vdj8t" Oct 31 05:12:49.206397 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:49.206242 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-031745-74cd69fbc5-s7bs2"] Oct 31 05:12:49.212695 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:49.212656 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-031745-74cd69fbc5-s7bs2"] Oct 31 05:12:49.318134 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:49.318092 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-339900d9-vdj8t" Oct 31 05:12:49.418373 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:49.417570 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3456208b-2c04-4e78-a654-fafe097e110f" path="/var/lib/kubelet/pods/3456208b-2c04-4e78-a654-fafe097e110f/volumes" Oct 31 05:12:49.418373 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:49.418258 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7139f8fd-4b2e-4c1e-9111-92e9b1bd1f62" path="/var/lib/kubelet/pods/7139f8fd-4b2e-4c1e-9111-92e9b1bd1f62/volumes" Oct 31 05:12:49.420422 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:49.419583 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b27570f7-6a81-4a8c-aad3-404126c2b7de" path="/var/lib/kubelet/pods/b27570f7-6a81-4a8c-aad3-404126c2b7de/volumes" Oct 31 05:12:49.420422 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:49.420135 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc856b70-497c-4c46-9db5-f41282c6b91d" path="/var/lib/kubelet/pods/bc856b70-497c-4c46-9db5-f41282c6b91d/volumes" Oct 31 05:12:49.625170 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:49.625139 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-339900d9-vdj8t"] Oct 31 05:12:49.626025 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:12:49.625997 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7c18d7a_62a1_4ef6_a835_7e79bd7ed3a0.slice/crio-795f1af7d8342a1652b00dce34b2d315341795ff0496c9101c85409a42a7d2ee WatchSource:0}: Error finding container 795f1af7d8342a1652b00dce34b2d315341795ff0496c9101c85409a42a7d2ee: Status 404 returned error can't find the container with id 795f1af7d8342a1652b00dce34b2d315341795ff0496c9101c85409a42a7d2ee Oct 31 05:12:49.877391 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:49.877293 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-339900d9-vdj8t" event={"ID":"e7c18d7a-62a1-4ef6-a835-7e79bd7ed3a0","Type":"ContainerStarted","Data":"795f1af7d8342a1652b00dce34b2d315341795ff0496c9101c85409a42a7d2ee"} Oct 31 05:12:50.890281 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:50.890243 5019 generic.go:356] "Generic (PLEG): container finished" podID="e7c18d7a-62a1-4ef6-a835-7e79bd7ed3a0" containerID="2a4d5fc70509d71cb7ebd8499ac041bc1601285dfe766e3537f2a34fa6a3b561" exitCode=0 Oct 31 05:12:50.890766 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:50.890327 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-339900d9-vdj8t" event={"ID":"e7c18d7a-62a1-4ef6-a835-7e79bd7ed3a0","Type":"ContainerDied","Data":"2a4d5fc70509d71cb7ebd8499ac041bc1601285dfe766e3537f2a34fa6a3b561"} Oct 31 05:12:52.079313 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:52.079277 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq"] Oct 31 05:12:52.081227 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:52.081191 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq" podUID="3f1a76d9-21b3-4831-a4b8-fd07b96cec87" containerName="istio-proxy" containerID="cri-o://3751a15cfb38f44c5aa11802fcac35be000fc885279b6cb58dcfe76c81b49a5f" gracePeriod=30 Oct 31 05:12:52.117403 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:52.113436 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-339900d9-vdj8t" Oct 31 05:12:52.209304 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:52.207797 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7c18d7a-62a1-4ef6-a835-7e79bd7ed3a0-kube-api-access-97wxr\" (UniqueName: \"kubernetes.io/projected/e7c18d7a-62a1-4ef6-a835-7e79bd7ed3a0-kube-api-access-97wxr\") pod \"e7c18d7a-62a1-4ef6-a835-7e79bd7ed3a0\" (UID: \"e7c18d7a-62a1-4ef6-a835-7e79bd7ed3a0\") " Oct 31 05:12:52.209833 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:52.209807 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7c18d7a-62a1-4ef6-a835-7e79bd7ed3a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7c18d7a-62a1-4ef6-a835-7e79bd7ed3a0-maas-api-xks-ca\") pod \"e7c18d7a-62a1-4ef6-a835-7e79bd7ed3a0\" (UID: \"e7c18d7a-62a1-4ef6-a835-7e79bd7ed3a0\") " Oct 31 05:12:52.210178 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:52.210156 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7c18d7a-62a1-4ef6-a835-7e79bd7ed3a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7c18d7a-62a1-4ef6-a835-7e79bd7ed3a0-maas-api-openshift-ca\") pod \"e7c18d7a-62a1-4ef6-a835-7e79bd7ed3a0\" (UID: \"e7c18d7a-62a1-4ef6-a835-7e79bd7ed3a0\") " Oct 31 05:12:52.210624 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:52.210578 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7c18d7a-62a1-4ef6-a835-7e79bd7ed3a0-maas-api-openshift-ca" pod "e7c18d7a-62a1-4ef6-a835-7e79bd7ed3a0" (UID: "e7c18d7a-62a1-4ef6-a835-7e79bd7ed3a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:12:52.212439 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:52.212319 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7c18d7a-62a1-4ef6-a835-7e79bd7ed3a0-kube-api-access-97wxr" pod "e7c18d7a-62a1-4ef6-a835-7e79bd7ed3a0" (UID: "e7c18d7a-62a1-4ef6-a835-7e79bd7ed3a0"). InnerVolumeSpecName "kube-api-access-97wxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:12:52.212786 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:52.212761 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97wxr\" (UniqueName: \"kubernetes.io/projected/e7c18d7a-62a1-4ef6-a835-7e79bd7ed3a0-kube-api-access-97wxr\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:12:52.212892 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:52.212792 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7c18d7a-62a1-4ef6-a835-7e79bd7ed3a0-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:12:52.215013 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:52.214984 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7c18d7a-62a1-4ef6-a835-7e79bd7ed3a0-maas-api-xks-ca" pod "e7c18d7a-62a1-4ef6-a835-7e79bd7ed3a0" (UID: "e7c18d7a-62a1-4ef6-a835-7e79bd7ed3a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:12:52.316353 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:52.315383 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7c18d7a-62a1-4ef6-a835-7e79bd7ed3a0-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:12:52.415724 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:12:52.415679 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:12:52.926247 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:52.926209 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-339900d9-vdj8t" Oct 31 05:12:52.926511 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:52.926240 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-339900d9-vdj8t" event={"ID":"e7c18d7a-62a1-4ef6-a835-7e79bd7ed3a0","Type":"ContainerDied","Data":"795f1af7d8342a1652b00dce34b2d315341795ff0496c9101c85409a42a7d2ee"} Oct 31 05:12:52.926511 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:52.926322 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="795f1af7d8342a1652b00dce34b2d315341795ff0496c9101c85409a42a7d2ee" Oct 31 05:12:53.005136 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:53.005089 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e8d67d9-cmhxg"] Oct 31 05:12:53.005885 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:53.005861 5019 cpu_manager.go:414] "containerMap: removing container" podUID="e7c18d7a-62a1-4ef6-a835-7e79bd7ed3a0" containerName="revoke-keys" Oct 31 05:12:53.005992 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:53.005887 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7c18d7a-62a1-4ef6-a835-7e79bd7ed3a0" containerName="revoke-keys" Oct 31 05:12:53.006106 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:53.006090 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7c18d7a-62a1-4ef6-a835-7e79bd7ed3a0" containerName="revoke-keys" Oct 31 05:12:53.011005 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:53.010977 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e8d67d9-cmhxg" Oct 31 05:12:53.025781 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:53.025748 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 05:12:53.025938 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:53.025796 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:12:53.025938 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:53.025909 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:12:53.028847 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:53.028815 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e8d67d9-cmhxg"] Oct 31 05:12:53.038040 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:53.036019 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:12:53.124792 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:53.124748 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d10aad80-55fd-4e26-883d-3dc3a5c7dc40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e8d67d9-cmhxg\" (UID: \"d10aad80-55fd-4e26-883d-3dc3a5c7dc40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e8d67d9-cmhxg" Oct 31 05:12:53.125228 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:53.124880 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d10aad80-55fd-4e26-883d-3dc3a5c7dc40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e8d67d9-cmhxg\" (UID: \"d10aad80-55fd-4e26-883d-3dc3a5c7dc40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e8d67d9-cmhxg" Oct 31 05:12:53.125228 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:53.124930 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnjk4\" (UniqueName: \"kubernetes.io/projected/d10aad80-55fd-4e26-883d-3dc3a5c7dc40-kube-api-access-bnjk4\") pod \"maas-api-revoke-sub-7e8d67d9-cmhxg\" (UID: \"d10aad80-55fd-4e26-883d-3dc3a5c7dc40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e8d67d9-cmhxg" Oct 31 05:12:53.225961 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:53.225918 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d10aad80-55fd-4e26-883d-3dc3a5c7dc40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e8d67d9-cmhxg\" (UID: \"d10aad80-55fd-4e26-883d-3dc3a5c7dc40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e8d67d9-cmhxg" Oct 31 05:12:53.226175 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:53.226004 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d10aad80-55fd-4e26-883d-3dc3a5c7dc40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e8d67d9-cmhxg\" (UID: \"d10aad80-55fd-4e26-883d-3dc3a5c7dc40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e8d67d9-cmhxg" Oct 31 05:12:53.226175 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:53.226030 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnjk4\" (UniqueName: \"kubernetes.io/projected/d10aad80-55fd-4e26-883d-3dc3a5c7dc40-kube-api-access-bnjk4\") pod \"maas-api-revoke-sub-7e8d67d9-cmhxg\" (UID: \"d10aad80-55fd-4e26-883d-3dc3a5c7dc40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e8d67d9-cmhxg" Oct 31 05:12:53.226671 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:53.226636 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d10aad80-55fd-4e26-883d-3dc3a5c7dc40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e8d67d9-cmhxg\" (UID: \"d10aad80-55fd-4e26-883d-3dc3a5c7dc40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e8d67d9-cmhxg" Oct 31 05:12:53.228522 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:53.228496 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d10aad80-55fd-4e26-883d-3dc3a5c7dc40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e8d67d9-cmhxg\" (UID: \"d10aad80-55fd-4e26-883d-3dc3a5c7dc40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e8d67d9-cmhxg" Oct 31 05:12:53.249470 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:53.249429 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnjk4\" (UniqueName: \"kubernetes.io/projected/d10aad80-55fd-4e26-883d-3dc3a5c7dc40-kube-api-access-bnjk4\") pod \"maas-api-revoke-sub-7e8d67d9-cmhxg\" (UID: \"d10aad80-55fd-4e26-883d-3dc3a5c7dc40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e8d67d9-cmhxg" Oct 31 05:12:53.327294 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:53.327242 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e8d67d9-cmhxg" Oct 31 05:12:53.635208 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:53.635182 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e8d67d9-cmhxg"] Oct 31 05:12:53.635840 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:12:53.635813 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd10aad80_55fd_4e26_883d_3dc3a5c7dc40.slice/crio-52f20b4947f525af7195aad0e867045792def99206fc93a505b6a48932100ff9 WatchSource:0}: Error finding container 52f20b4947f525af7195aad0e867045792def99206fc93a505b6a48932100ff9: Status 404 returned error can't find the container with id 52f20b4947f525af7195aad0e867045792def99206fc93a505b6a48932100ff9 Oct 31 05:12:53.936014 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:53.935913 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e8d67d9-cmhxg" event={"ID":"d10aad80-55fd-4e26-883d-3dc3a5c7dc40","Type":"ContainerStarted","Data":"52f20b4947f525af7195aad0e867045792def99206fc93a505b6a48932100ff9"} Oct 31 05:12:54.945212 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:54.945112 5019 generic.go:356] "Generic (PLEG): container finished" podID="d10aad80-55fd-4e26-883d-3dc3a5c7dc40" containerID="f255ea32ecea31466a26ad9494d2cda25ea7db6528486320178a51adab2c78c1" exitCode=0 Oct 31 05:12:54.945212 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:54.945199 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e8d67d9-cmhxg" event={"ID":"d10aad80-55fd-4e26-883d-3dc3a5c7dc40","Type":"ContainerDied","Data":"f255ea32ecea31466a26ad9494d2cda25ea7db6528486320178a51adab2c78c1"} Oct 31 05:12:56.149349 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:56.149307 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e8d67d9-cmhxg" Oct 31 05:12:56.277302 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:56.277266 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d10aad80-55fd-4e26-883d-3dc3a5c7dc40-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d10aad80-55fd-4e26-883d-3dc3a5c7dc40-maas-api-openshift-ca\") pod \"d10aad80-55fd-4e26-883d-3dc3a5c7dc40\" (UID: \"d10aad80-55fd-4e26-883d-3dc3a5c7dc40\") " Oct 31 05:12:56.277561 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:56.277420 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d10aad80-55fd-4e26-883d-3dc3a5c7dc40-kube-api-access-bnjk4\" (UniqueName: \"kubernetes.io/projected/d10aad80-55fd-4e26-883d-3dc3a5c7dc40-kube-api-access-bnjk4\") pod \"d10aad80-55fd-4e26-883d-3dc3a5c7dc40\" (UID: \"d10aad80-55fd-4e26-883d-3dc3a5c7dc40\") " Oct 31 05:12:56.277561 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:56.277484 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d10aad80-55fd-4e26-883d-3dc3a5c7dc40-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d10aad80-55fd-4e26-883d-3dc3a5c7dc40-maas-api-xks-ca\") pod \"d10aad80-55fd-4e26-883d-3dc3a5c7dc40\" (UID: \"d10aad80-55fd-4e26-883d-3dc3a5c7dc40\") " Oct 31 05:12:56.277759 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:56.277727 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d10aad80-55fd-4e26-883d-3dc3a5c7dc40-maas-api-openshift-ca" pod "d10aad80-55fd-4e26-883d-3dc3a5c7dc40" (UID: "d10aad80-55fd-4e26-883d-3dc3a5c7dc40"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:12:56.278113 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:56.278087 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d10aad80-55fd-4e26-883d-3dc3a5c7dc40-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:12:56.279838 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:56.279809 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d10aad80-55fd-4e26-883d-3dc3a5c7dc40-maas-api-xks-ca" pod "d10aad80-55fd-4e26-883d-3dc3a5c7dc40" (UID: "d10aad80-55fd-4e26-883d-3dc3a5c7dc40"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:12:56.279975 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:56.279818 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d10aad80-55fd-4e26-883d-3dc3a5c7dc40-kube-api-access-bnjk4" pod "d10aad80-55fd-4e26-883d-3dc3a5c7dc40" (UID: "d10aad80-55fd-4e26-883d-3dc3a5c7dc40"). InnerVolumeSpecName "kube-api-access-bnjk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:12:56.380226 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:56.380169 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d10aad80-55fd-4e26-883d-3dc3a5c7dc40-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:12:56.380226 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:56.380224 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnjk4\" (UniqueName: \"kubernetes.io/projected/d10aad80-55fd-4e26-883d-3dc3a5c7dc40-kube-api-access-bnjk4\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:12:56.963235 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:56.963192 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e8d67d9-cmhxg" event={"ID":"d10aad80-55fd-4e26-883d-3dc3a5c7dc40","Type":"ContainerDied","Data":"52f20b4947f525af7195aad0e867045792def99206fc93a505b6a48932100ff9"} Oct 31 05:12:56.963235 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:56.963228 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e8d67d9-cmhxg" Oct 31 05:12:56.963497 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:56.963278 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52f20b4947f525af7195aad0e867045792def99206fc93a505b6a48932100ff9" Oct 31 05:12:57.392989 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:57.392960 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq" Oct 31 05:12:57.499770 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:57.498193 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-istio-token\" (UniqueName: \"kubernetes.io/projected/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-istio-token\") pod \"3f1a76d9-21b3-4831-a4b8-fd07b96cec87\" (UID: \"3f1a76d9-21b3-4831-a4b8-fd07b96cec87\") " Oct 31 05:12:57.499770 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:57.498252 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-workload-socket\") pod \"3f1a76d9-21b3-4831-a4b8-fd07b96cec87\" (UID: \"3f1a76d9-21b3-4831-a4b8-fd07b96cec87\") " Oct 31 05:12:57.499770 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:57.498299 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-istiod-ca-cert\") pod \"3f1a76d9-21b3-4831-a4b8-fd07b96cec87\" (UID: \"3f1a76d9-21b3-4831-a4b8-fd07b96cec87\") " Oct 31 05:12:57.499770 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:57.498343 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-istio-envoy\") pod \"3f1a76d9-21b3-4831-a4b8-fd07b96cec87\" (UID: \"3f1a76d9-21b3-4831-a4b8-fd07b96cec87\") " Oct 31 05:12:57.499770 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:57.498405 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-kube-api-access-xxzfh\" (UniqueName: \"kubernetes.io/projected/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-kube-api-access-xxzfh\") pod \"3f1a76d9-21b3-4831-a4b8-fd07b96cec87\" (UID: \"3f1a76d9-21b3-4831-a4b8-fd07b96cec87\") " Oct 31 05:12:57.499770 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:57.498487 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-credential-socket\") pod \"3f1a76d9-21b3-4831-a4b8-fd07b96cec87\" (UID: \"3f1a76d9-21b3-4831-a4b8-fd07b96cec87\") " Oct 31 05:12:57.499770 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:57.498638 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-istio-podinfo\") pod \"3f1a76d9-21b3-4831-a4b8-fd07b96cec87\" (UID: \"3f1a76d9-21b3-4831-a4b8-fd07b96cec87\") " Oct 31 05:12:57.499770 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:57.498671 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-workload-certs\") pod \"3f1a76d9-21b3-4831-a4b8-fd07b96cec87\" (UID: \"3f1a76d9-21b3-4831-a4b8-fd07b96cec87\") " Oct 31 05:12:57.499770 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:57.498808 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-istio-data\") pod \"3f1a76d9-21b3-4831-a4b8-fd07b96cec87\" (UID: \"3f1a76d9-21b3-4831-a4b8-fd07b96cec87\") " Oct 31 05:12:57.499770 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:57.499401 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-istio-data" pod "3f1a76d9-21b3-4831-a4b8-fd07b96cec87" (UID: "3f1a76d9-21b3-4831-a4b8-fd07b96cec87"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:12:57.501617 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:57.501087 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:12:57.502951 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:57.502921 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-istio-token" pod "3f1a76d9-21b3-4831-a4b8-fd07b96cec87" (UID: "3f1a76d9-21b3-4831-a4b8-fd07b96cec87"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:12:57.503167 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:57.503147 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-credential-socket" pod "3f1a76d9-21b3-4831-a4b8-fd07b96cec87" (UID: "3f1a76d9-21b3-4831-a4b8-fd07b96cec87"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:12:57.513401 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:57.513360 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-workload-certs" pod "3f1a76d9-21b3-4831-a4b8-fd07b96cec87" (UID: "3f1a76d9-21b3-4831-a4b8-fd07b96cec87"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:12:57.513628 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:57.513605 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-workload-socket" pod "3f1a76d9-21b3-4831-a4b8-fd07b96cec87" (UID: "3f1a76d9-21b3-4831-a4b8-fd07b96cec87"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:12:57.519550 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:57.519509 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-istiod-ca-cert" pod "3f1a76d9-21b3-4831-a4b8-fd07b96cec87" (UID: "3f1a76d9-21b3-4831-a4b8-fd07b96cec87"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:12:57.524089 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:57.524058 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-istio-envoy" pod "3f1a76d9-21b3-4831-a4b8-fd07b96cec87" (UID: "3f1a76d9-21b3-4831-a4b8-fd07b96cec87"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:12:57.524545 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:57.524517 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-kube-api-access-xxzfh" pod "3f1a76d9-21b3-4831-a4b8-fd07b96cec87" (UID: "3f1a76d9-21b3-4831-a4b8-fd07b96cec87"). InnerVolumeSpecName "kube-api-access-xxzfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:12:57.526550 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:57.526525 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-istio-podinfo" pod "3f1a76d9-21b3-4831-a4b8-fd07b96cec87" (UID: "3f1a76d9-21b3-4831-a4b8-fd07b96cec87"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:12:57.602460 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:57.602423 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:12:57.602460 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:57.602453 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:12:57.602460 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:57.602463 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:12:57.602739 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:57.602472 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:12:57.602739 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:57.602480 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxzfh\" (UniqueName: \"kubernetes.io/projected/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-kube-api-access-xxzfh\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:12:57.602739 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:57.602489 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:12:57.602739 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:57.602499 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:12:57.602739 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:57.602509 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f1a76d9-21b3-4831-a4b8-fd07b96cec87-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:12:57.652187 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:57.652145 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz"] Oct 31 05:12:58.035357 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:58.032848 5019 generic.go:356] "Generic (PLEG): container finished" podID="3f1a76d9-21b3-4831-a4b8-fd07b96cec87" containerID="3751a15cfb38f44c5aa11802fcac35be000fc885279b6cb58dcfe76c81b49a5f" exitCode=0 Oct 31 05:12:58.035357 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:58.033110 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq" event={"ID":"3f1a76d9-21b3-4831-a4b8-fd07b96cec87","Type":"ContainerDied","Data":"3751a15cfb38f44c5aa11802fcac35be000fc885279b6cb58dcfe76c81b49a5f"} Oct 31 05:12:58.035357 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:58.033172 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq" event={"ID":"3f1a76d9-21b3-4831-a4b8-fd07b96cec87","Type":"ContainerDied","Data":"ea45e52808a22c0864c1e2bdbea37593682e35308026dc329947e0f78d573f6b"} Oct 31 05:12:58.035357 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:58.033202 5019 scope.go:122] "RemoveContainer" containerID="3751a15cfb38f44c5aa11802fcac35be000fc885279b6cb58dcfe76c81b49a5f" Oct 31 05:12:58.035357 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:58.033404 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq" Oct 31 05:12:58.051429 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:58.051400 5019 scope.go:122] "RemoveContainer" containerID="90b1550534a1e97e636d4c271604b6e7b7b77a2751d5228485f696b2255c5b17" Oct 31 05:12:58.062494 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:58.062471 5019 scope.go:122] "RemoveContainer" containerID="3751a15cfb38f44c5aa11802fcac35be000fc885279b6cb58dcfe76c81b49a5f" Oct 31 05:12:58.062805 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:12:58.062781 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3751a15cfb38f44c5aa11802fcac35be000fc885279b6cb58dcfe76c81b49a5f\": container with ID starting with 3751a15cfb38f44c5aa11802fcac35be000fc885279b6cb58dcfe76c81b49a5f not found: ID does not exist" containerID="3751a15cfb38f44c5aa11802fcac35be000fc885279b6cb58dcfe76c81b49a5f" Oct 31 05:12:58.062888 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:58.062815 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3751a15cfb38f44c5aa11802fcac35be000fc885279b6cb58dcfe76c81b49a5f"} err="failed to get container status \"3751a15cfb38f44c5aa11802fcac35be000fc885279b6cb58dcfe76c81b49a5f\": rpc error: code = NotFound desc = could not find container \"3751a15cfb38f44c5aa11802fcac35be000fc885279b6cb58dcfe76c81b49a5f\": container with ID starting with 3751a15cfb38f44c5aa11802fcac35be000fc885279b6cb58dcfe76c81b49a5f not found: ID does not exist" Oct 31 05:12:58.062888 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:58.062835 5019 scope.go:122] "RemoveContainer" containerID="90b1550534a1e97e636d4c271604b6e7b7b77a2751d5228485f696b2255c5b17" Oct 31 05:12:58.063082 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:12:58.063065 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90b1550534a1e97e636d4c271604b6e7b7b77a2751d5228485f696b2255c5b17\": container with ID starting with 90b1550534a1e97e636d4c271604b6e7b7b77a2751d5228485f696b2255c5b17 not found: ID does not exist" containerID="90b1550534a1e97e636d4c271604b6e7b7b77a2751d5228485f696b2255c5b17" Oct 31 05:12:58.063151 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:58.063086 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90b1550534a1e97e636d4c271604b6e7b7b77a2751d5228485f696b2255c5b17"} err="failed to get container status \"90b1550534a1e97e636d4c271604b6e7b7b77a2751d5228485f696b2255c5b17\": rpc error: code = NotFound desc = could not find container \"90b1550534a1e97e636d4c271604b6e7b7b77a2751d5228485f696b2255c5b17\": container with ID starting with 90b1550534a1e97e636d4c271604b6e7b7b77a2751d5228485f696b2255c5b17 not found: ID does not exist" Oct 31 05:12:58.092593 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:58.092557 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq"] Oct 31 05:12:58.104523 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:58.104489 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d6ac82-openshift-default-56596855f-5chnq"] Oct 31 05:12:58.412663 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:58.412560 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-b2a372/rate-test-model-b2a372-kserve-8454cdd6df-bkqc8"] Oct 31 05:12:58.413611 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:58.413583 5019 cpu_manager.go:414] "containerMap: removing container" podUID="d10aad80-55fd-4e26-883d-3dc3a5c7dc40" containerName="revoke-keys" Oct 31 05:12:58.413611 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:58.413611 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d10aad80-55fd-4e26-883d-3dc3a5c7dc40" containerName="revoke-keys" Oct 31 05:12:58.413811 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:58.413688 5019 cpu_manager.go:414] "containerMap: removing container" podUID="3f1a76d9-21b3-4831-a4b8-fd07b96cec87" containerName="istio-proxy" Oct 31 05:12:58.413811 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:58.413698 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f1a76d9-21b3-4831-a4b8-fd07b96cec87" containerName="istio-proxy" Oct 31 05:12:58.413811 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:58.413713 5019 cpu_manager.go:414] "containerMap: removing container" podUID="3f1a76d9-21b3-4831-a4b8-fd07b96cec87" containerName="istio-proxy" Oct 31 05:12:58.413811 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:58.413721 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f1a76d9-21b3-4831-a4b8-fd07b96cec87" containerName="istio-proxy" Oct 31 05:12:58.414049 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:58.413900 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f1a76d9-21b3-4831-a4b8-fd07b96cec87" containerName="istio-proxy" Oct 31 05:12:58.414049 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:58.413917 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f1a76d9-21b3-4831-a4b8-fd07b96cec87" containerName="istio-proxy" Oct 31 05:12:58.414049 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:58.413941 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="d10aad80-55fd-4e26-883d-3dc3a5c7dc40" containerName="revoke-keys" Oct 31 05:12:58.419578 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:58.419547 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b2a372/rate-test-model-b2a372-kserve-8454cdd6df-bkqc8" Oct 31 05:12:58.436760 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:58.436723 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-b2a372\"/\"openshift-service-ca.crt\"" Oct 31 05:12:58.436980 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:58.436959 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-b2a372\"/\"default-dockercfg-dqlrt\"" Oct 31 05:12:58.437096 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:58.437079 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-b2a372\"/\"rate-test-model-b2a372-kserve-self-signed-certs\"" Oct 31 05:12:58.438208 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:58.437849 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-b2a372\"/\"kube-root-ca.crt\"" Oct 31 05:12:58.442577 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:58.442549 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b2a372/rate-test-model-b2a372-kserve-8454cdd6df-bkqc8"] Oct 31 05:12:58.516414 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:58.516318 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66fj2\" (UniqueName: \"kubernetes.io/projected/0b8d5e17-5809-46a1-b0da-d11c15832f1d-kube-api-access-66fj2\") pod \"rate-test-model-b2a372-kserve-8454cdd6df-bkqc8\" (UID: \"0b8d5e17-5809-46a1-b0da-d11c15832f1d\") " pod="ai-tenant-e2e-shared-b-w3-b2a372/rate-test-model-b2a372-kserve-8454cdd6df-bkqc8" Oct 31 05:12:58.516611 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:58.516429 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b8d5e17-5809-46a1-b0da-d11c15832f1d-model-cache\") pod \"rate-test-model-b2a372-kserve-8454cdd6df-bkqc8\" (UID: \"0b8d5e17-5809-46a1-b0da-d11c15832f1d\") " pod="ai-tenant-e2e-shared-b-w3-b2a372/rate-test-model-b2a372-kserve-8454cdd6df-bkqc8" Oct 31 05:12:58.516611 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:58.516464 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b8d5e17-5809-46a1-b0da-d11c15832f1d-tls-certs\") pod \"rate-test-model-b2a372-kserve-8454cdd6df-bkqc8\" (UID: \"0b8d5e17-5809-46a1-b0da-d11c15832f1d\") " pod="ai-tenant-e2e-shared-b-w3-b2a372/rate-test-model-b2a372-kserve-8454cdd6df-bkqc8" Oct 31 05:12:58.516763 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:58.516624 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b8d5e17-5809-46a1-b0da-d11c15832f1d-tmp-dir\") pod \"rate-test-model-b2a372-kserve-8454cdd6df-bkqc8\" (UID: \"0b8d5e17-5809-46a1-b0da-d11c15832f1d\") " pod="ai-tenant-e2e-shared-b-w3-b2a372/rate-test-model-b2a372-kserve-8454cdd6df-bkqc8" Oct 31 05:12:58.516821 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:58.516775 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b8d5e17-5809-46a1-b0da-d11c15832f1d-home\") pod \"rate-test-model-b2a372-kserve-8454cdd6df-bkqc8\" (UID: \"0b8d5e17-5809-46a1-b0da-d11c15832f1d\") " pod="ai-tenant-e2e-shared-b-w3-b2a372/rate-test-model-b2a372-kserve-8454cdd6df-bkqc8" Oct 31 05:12:58.516948 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:58.516932 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b8d5e17-5809-46a1-b0da-d11c15832f1d-dshm\") pod \"rate-test-model-b2a372-kserve-8454cdd6df-bkqc8\" (UID: \"0b8d5e17-5809-46a1-b0da-d11c15832f1d\") " pod="ai-tenant-e2e-shared-b-w3-b2a372/rate-test-model-b2a372-kserve-8454cdd6df-bkqc8" Oct 31 05:12:58.618328 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:58.618289 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b8d5e17-5809-46a1-b0da-d11c15832f1d-dshm\") pod \"rate-test-model-b2a372-kserve-8454cdd6df-bkqc8\" (UID: \"0b8d5e17-5809-46a1-b0da-d11c15832f1d\") " pod="ai-tenant-e2e-shared-b-w3-b2a372/rate-test-model-b2a372-kserve-8454cdd6df-bkqc8" Oct 31 05:12:58.618723 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:58.618441 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66fj2\" (UniqueName: \"kubernetes.io/projected/0b8d5e17-5809-46a1-b0da-d11c15832f1d-kube-api-access-66fj2\") pod \"rate-test-model-b2a372-kserve-8454cdd6df-bkqc8\" (UID: \"0b8d5e17-5809-46a1-b0da-d11c15832f1d\") " pod="ai-tenant-e2e-shared-b-w3-b2a372/rate-test-model-b2a372-kserve-8454cdd6df-bkqc8" Oct 31 05:12:58.618723 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:58.618476 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b8d5e17-5809-46a1-b0da-d11c15832f1d-model-cache\") pod \"rate-test-model-b2a372-kserve-8454cdd6df-bkqc8\" (UID: \"0b8d5e17-5809-46a1-b0da-d11c15832f1d\") " pod="ai-tenant-e2e-shared-b-w3-b2a372/rate-test-model-b2a372-kserve-8454cdd6df-bkqc8" Oct 31 05:12:58.618723 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:58.618505 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b8d5e17-5809-46a1-b0da-d11c15832f1d-tls-certs\") pod \"rate-test-model-b2a372-kserve-8454cdd6df-bkqc8\" (UID: \"0b8d5e17-5809-46a1-b0da-d11c15832f1d\") " pod="ai-tenant-e2e-shared-b-w3-b2a372/rate-test-model-b2a372-kserve-8454cdd6df-bkqc8" Oct 31 05:12:58.618723 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:58.618533 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b8d5e17-5809-46a1-b0da-d11c15832f1d-tmp-dir\") pod \"rate-test-model-b2a372-kserve-8454cdd6df-bkqc8\" (UID: \"0b8d5e17-5809-46a1-b0da-d11c15832f1d\") " pod="ai-tenant-e2e-shared-b-w3-b2a372/rate-test-model-b2a372-kserve-8454cdd6df-bkqc8" Oct 31 05:12:58.618723 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:58.618623 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b8d5e17-5809-46a1-b0da-d11c15832f1d-home\") pod \"rate-test-model-b2a372-kserve-8454cdd6df-bkqc8\" (UID: \"0b8d5e17-5809-46a1-b0da-d11c15832f1d\") " pod="ai-tenant-e2e-shared-b-w3-b2a372/rate-test-model-b2a372-kserve-8454cdd6df-bkqc8" Oct 31 05:12:58.619036 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:58.618965 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b8d5e17-5809-46a1-b0da-d11c15832f1d-tmp-dir\") pod \"rate-test-model-b2a372-kserve-8454cdd6df-bkqc8\" (UID: \"0b8d5e17-5809-46a1-b0da-d11c15832f1d\") " pod="ai-tenant-e2e-shared-b-w3-b2a372/rate-test-model-b2a372-kserve-8454cdd6df-bkqc8" Oct 31 05:12:58.619036 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:58.619012 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b8d5e17-5809-46a1-b0da-d11c15832f1d-home\") pod \"rate-test-model-b2a372-kserve-8454cdd6df-bkqc8\" (UID: \"0b8d5e17-5809-46a1-b0da-d11c15832f1d\") " pod="ai-tenant-e2e-shared-b-w3-b2a372/rate-test-model-b2a372-kserve-8454cdd6df-bkqc8" Oct 31 05:12:58.619174 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:58.619157 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b8d5e17-5809-46a1-b0da-d11c15832f1d-model-cache\") pod \"rate-test-model-b2a372-kserve-8454cdd6df-bkqc8\" (UID: \"0b8d5e17-5809-46a1-b0da-d11c15832f1d\") " pod="ai-tenant-e2e-shared-b-w3-b2a372/rate-test-model-b2a372-kserve-8454cdd6df-bkqc8" Oct 31 05:12:58.621381 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:58.621316 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b8d5e17-5809-46a1-b0da-d11c15832f1d-dshm\") pod \"rate-test-model-b2a372-kserve-8454cdd6df-bkqc8\" (UID: \"0b8d5e17-5809-46a1-b0da-d11c15832f1d\") " pod="ai-tenant-e2e-shared-b-w3-b2a372/rate-test-model-b2a372-kserve-8454cdd6df-bkqc8" Oct 31 05:12:58.621617 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:58.621596 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b8d5e17-5809-46a1-b0da-d11c15832f1d-tls-certs\") pod \"rate-test-model-b2a372-kserve-8454cdd6df-bkqc8\" (UID: \"0b8d5e17-5809-46a1-b0da-d11c15832f1d\") " pod="ai-tenant-e2e-shared-b-w3-b2a372/rate-test-model-b2a372-kserve-8454cdd6df-bkqc8" Oct 31 05:12:58.677680 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:58.677523 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66fj2\" (UniqueName: \"kubernetes.io/projected/0b8d5e17-5809-46a1-b0da-d11c15832f1d-kube-api-access-66fj2\") pod \"rate-test-model-b2a372-kserve-8454cdd6df-bkqc8\" (UID: \"0b8d5e17-5809-46a1-b0da-d11c15832f1d\") " pod="ai-tenant-e2e-shared-b-w3-b2a372/rate-test-model-b2a372-kserve-8454cdd6df-bkqc8" Oct 31 05:12:58.742794 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:58.742756 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b2a372/rate-test-model-b2a372-kserve-8454cdd6df-bkqc8" Oct 31 05:12:59.057297 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:59.057261 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b2a372/rate-test-model-b2a372-kserve-8454cdd6df-bkqc8"] Oct 31 05:12:59.057793 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:12:59.057764 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b8d5e17_5809_46a1_b0da_d11c15832f1d.slice/crio-1e1e0c0f8b912f66af75a756f1eed170c84f69d93ce3a2b1b8a5219e5f641016 WatchSource:0}: Error finding container 1e1e0c0f8b912f66af75a756f1eed170c84f69d93ce3a2b1b8a5219e5f641016: Status 404 returned error can't find the container with id 1e1e0c0f8b912f66af75a756f1eed170c84f69d93ce3a2b1b8a5219e5f641016 Oct 31 05:12:59.415187 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:59.415088 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f1a76d9-21b3-4831-a4b8-fd07b96cec87" path="/var/lib/kubelet/pods/3f1a76d9-21b3-4831-a4b8-fd07b96cec87/volumes" Oct 31 05:12:59.597888 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:59.597851 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ef3d94d-w6hdj"] Oct 31 05:12:59.602928 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:59.602897 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef3d94d-w6hdj" Oct 31 05:12:59.618665 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:59.618631 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:12:59.632114 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:59.631134 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 05:12:59.632114 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:59.631273 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:12:59.632114 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:59.631371 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:12:59.646308 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:59.646268 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ef3d94d-w6hdj"] Oct 31 05:12:59.741083 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:59.741036 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/227b5b23-079f-4fef-b140-2835d2df2745-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ef3d94d-w6hdj\" (UID: \"227b5b23-079f-4fef-b140-2835d2df2745\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef3d94d-w6hdj" Oct 31 05:12:59.741390 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:59.741347 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpscw\" (UniqueName: \"kubernetes.io/projected/227b5b23-079f-4fef-b140-2835d2df2745-kube-api-access-mpscw\") pod \"maas-api-revoke-sub-4ef3d94d-w6hdj\" (UID: \"227b5b23-079f-4fef-b140-2835d2df2745\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef3d94d-w6hdj" Oct 31 05:12:59.741535 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:59.741428 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/227b5b23-079f-4fef-b140-2835d2df2745-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ef3d94d-w6hdj\" (UID: \"227b5b23-079f-4fef-b140-2835d2df2745\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef3d94d-w6hdj" Oct 31 05:12:59.843397 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:59.843314 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/227b5b23-079f-4fef-b140-2835d2df2745-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ef3d94d-w6hdj\" (UID: \"227b5b23-079f-4fef-b140-2835d2df2745\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef3d94d-w6hdj" Oct 31 05:12:59.843584 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:59.843515 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpscw\" (UniqueName: \"kubernetes.io/projected/227b5b23-079f-4fef-b140-2835d2df2745-kube-api-access-mpscw\") pod \"maas-api-revoke-sub-4ef3d94d-w6hdj\" (UID: \"227b5b23-079f-4fef-b140-2835d2df2745\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef3d94d-w6hdj" Oct 31 05:12:59.843584 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:59.843568 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/227b5b23-079f-4fef-b140-2835d2df2745-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ef3d94d-w6hdj\" (UID: \"227b5b23-079f-4fef-b140-2835d2df2745\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef3d94d-w6hdj" Oct 31 05:12:59.844243 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:59.844216 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/227b5b23-079f-4fef-b140-2835d2df2745-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ef3d94d-w6hdj\" (UID: \"227b5b23-079f-4fef-b140-2835d2df2745\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef3d94d-w6hdj" Oct 31 05:12:59.845937 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:59.845916 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/227b5b23-079f-4fef-b140-2835d2df2745-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ef3d94d-w6hdj\" (UID: \"227b5b23-079f-4fef-b140-2835d2df2745\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef3d94d-w6hdj" Oct 31 05:12:59.867688 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:59.867647 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpscw\" (UniqueName: \"kubernetes.io/projected/227b5b23-079f-4fef-b140-2835d2df2745-kube-api-access-mpscw\") pod \"maas-api-revoke-sub-4ef3d94d-w6hdj\" (UID: \"227b5b23-079f-4fef-b140-2835d2df2745\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef3d94d-w6hdj" Oct 31 05:12:59.946760 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:12:59.946721 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef3d94d-w6hdj" Oct 31 05:13:00.190215 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:00.190164 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b2a372/rate-test-model-b2a372-kserve-8454cdd6df-bkqc8" event={"ID":"0b8d5e17-5809-46a1-b0da-d11c15832f1d","Type":"ContainerStarted","Data":"b52e90aa933e021314a542d34462e93149df95cabd2111dc34e8005a08548fd7"} Oct 31 05:13:00.190508 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:00.190478 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b2a372/rate-test-model-b2a372-kserve-8454cdd6df-bkqc8" event={"ID":"0b8d5e17-5809-46a1-b0da-d11c15832f1d","Type":"ContainerStarted","Data":"1e1e0c0f8b912f66af75a756f1eed170c84f69d93ce3a2b1b8a5219e5f641016"} Oct 31 05:13:00.255018 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:00.254568 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-b2a372/rate-test-model-b2a372-kserve-8454cdd6df-bkqc8" podStartSLOduration=2.254546995 podStartE2EDuration="2.254546995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:12:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:13:00.245789351 +0000 UTC m=+4135.002127383" watchObservedRunningTime="2026-10-31 05:13:00.254546995 +0000 UTC m=+4135.010885037" Oct 31 05:13:00.650320 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:00.650277 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ef3d94d-w6hdj"] Oct 31 05:13:00.654234 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:13:00.654172 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod227b5b23_079f_4fef_b140_2835d2df2745.slice/crio-74538613f96506c224568781c12419f178ecad8dc3ba0b868500cbda85742e39 WatchSource:0}: Error finding container 74538613f96506c224568781c12419f178ecad8dc3ba0b868500cbda85742e39: Status 404 returned error can't find the container with id 74538613f96506c224568781c12419f178ecad8dc3ba0b868500cbda85742e39 Oct 31 05:13:00.743663 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:00.743611 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-b2a372/rate-test-model-b2a372-kserve-8454cdd6df-bkqc8" Oct 31 05:13:01.265501 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:01.265360 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef3d94d-w6hdj" event={"ID":"227b5b23-079f-4fef-b140-2835d2df2745","Type":"ContainerStarted","Data":"74538613f96506c224568781c12419f178ecad8dc3ba0b868500cbda85742e39"} Oct 31 05:13:02.274642 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:02.274606 5019 generic.go:356] "Generic (PLEG): container finished" podID="227b5b23-079f-4fef-b140-2835d2df2745" containerID="57eccfc623dcb6911a57be6e992f35e00a9dd27a3e6560408297bc6b418c250f" exitCode=0 Oct 31 05:13:02.275104 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:02.274693 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef3d94d-w6hdj" event={"ID":"227b5b23-079f-4fef-b140-2835d2df2745","Type":"ContainerDied","Data":"57eccfc623dcb6911a57be6e992f35e00a9dd27a3e6560408297bc6b418c250f"} Oct 31 05:13:03.454392 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:03.454365 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef3d94d-w6hdj" Oct 31 05:13:03.563703 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:03.563597 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/227b5b23-079f-4fef-b140-2835d2df2745-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/227b5b23-079f-4fef-b140-2835d2df2745-maas-api-xks-ca\") pod \"227b5b23-079f-4fef-b140-2835d2df2745\" (UID: \"227b5b23-079f-4fef-b140-2835d2df2745\") " Oct 31 05:13:03.563883 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:03.563731 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/227b5b23-079f-4fef-b140-2835d2df2745-kube-api-access-mpscw\" (UniqueName: \"kubernetes.io/projected/227b5b23-079f-4fef-b140-2835d2df2745-kube-api-access-mpscw\") pod \"227b5b23-079f-4fef-b140-2835d2df2745\" (UID: \"227b5b23-079f-4fef-b140-2835d2df2745\") " Oct 31 05:13:03.563883 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:03.563808 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/227b5b23-079f-4fef-b140-2835d2df2745-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/227b5b23-079f-4fef-b140-2835d2df2745-maas-api-openshift-ca\") pod \"227b5b23-079f-4fef-b140-2835d2df2745\" (UID: \"227b5b23-079f-4fef-b140-2835d2df2745\") " Oct 31 05:13:03.564175 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:03.564136 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/227b5b23-079f-4fef-b140-2835d2df2745-maas-api-openshift-ca" pod "227b5b23-079f-4fef-b140-2835d2df2745" (UID: "227b5b23-079f-4fef-b140-2835d2df2745"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:13:03.564511 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:03.564489 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/227b5b23-079f-4fef-b140-2835d2df2745-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:13:03.565981 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:03.565953 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/227b5b23-079f-4fef-b140-2835d2df2745-maas-api-xks-ca" pod "227b5b23-079f-4fef-b140-2835d2df2745" (UID: "227b5b23-079f-4fef-b140-2835d2df2745"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:13:03.566319 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:03.566297 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/227b5b23-079f-4fef-b140-2835d2df2745-kube-api-access-mpscw" pod "227b5b23-079f-4fef-b140-2835d2df2745" (UID: "227b5b23-079f-4fef-b140-2835d2df2745"). InnerVolumeSpecName "kube-api-access-mpscw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:13:03.665764 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:03.665717 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/227b5b23-079f-4fef-b140-2835d2df2745-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:13:03.665764 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:03.665755 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpscw\" (UniqueName: \"kubernetes.io/projected/227b5b23-079f-4fef-b140-2835d2df2745-kube-api-access-mpscw\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:13:04.293034 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:04.293000 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef3d94d-w6hdj" Oct 31 05:13:04.293034 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:04.293015 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef3d94d-w6hdj" event={"ID":"227b5b23-079f-4fef-b140-2835d2df2745","Type":"ContainerDied","Data":"74538613f96506c224568781c12419f178ecad8dc3ba0b868500cbda85742e39"} Oct 31 05:13:04.293273 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:04.293121 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74538613f96506c224568781c12419f178ecad8dc3ba0b868500cbda85742e39" Oct 31 05:13:05.970448 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:05.970378 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1e47f72-kn24j"] Oct 31 05:13:05.971378 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:05.971320 5019 cpu_manager.go:414] "containerMap: removing container" podUID="227b5b23-079f-4fef-b140-2835d2df2745" containerName="revoke-keys" Oct 31 05:13:05.971378 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:05.971363 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="227b5b23-079f-4fef-b140-2835d2df2745" containerName="revoke-keys" Oct 31 05:13:05.971570 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:05.971547 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="227b5b23-079f-4fef-b140-2835d2df2745" containerName="revoke-keys" Oct 31 05:13:05.975547 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:05.975510 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1e47f72-kn24j" Oct 31 05:13:05.994868 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:05.994836 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1e47f72-kn24j"] Oct 31 05:13:06.004883 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:06.001471 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 05:13:06.004883 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:06.001557 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:13:06.004883 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:06.001769 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5ed9a44-4e39-403c-a936-7f44d5f0fdad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1e47f72-kn24j\" (UID: \"a5ed9a44-4e39-403c-a936-7f44d5f0fdad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1e47f72-kn24j" Oct 31 05:13:06.004883 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:06.001833 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htrq6\" (UniqueName: \"kubernetes.io/projected/a5ed9a44-4e39-403c-a936-7f44d5f0fdad-kube-api-access-htrq6\") pod \"maas-api-revoke-sub-e1e47f72-kn24j\" (UID: \"a5ed9a44-4e39-403c-a936-7f44d5f0fdad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1e47f72-kn24j" Oct 31 05:13:06.004883 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:06.001869 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5ed9a44-4e39-403c-a936-7f44d5f0fdad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1e47f72-kn24j\" (UID: \"a5ed9a44-4e39-403c-a936-7f44d5f0fdad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1e47f72-kn24j" Oct 31 05:13:06.004883 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:06.002129 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:13:06.004883 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:06.002169 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:13:06.103274 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:06.103225 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5ed9a44-4e39-403c-a936-7f44d5f0fdad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1e47f72-kn24j\" (UID: \"a5ed9a44-4e39-403c-a936-7f44d5f0fdad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1e47f72-kn24j" Oct 31 05:13:06.103542 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:06.103294 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htrq6\" (UniqueName: \"kubernetes.io/projected/a5ed9a44-4e39-403c-a936-7f44d5f0fdad-kube-api-access-htrq6\") pod \"maas-api-revoke-sub-e1e47f72-kn24j\" (UID: \"a5ed9a44-4e39-403c-a936-7f44d5f0fdad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1e47f72-kn24j" Oct 31 05:13:06.103542 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:06.103320 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5ed9a44-4e39-403c-a936-7f44d5f0fdad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1e47f72-kn24j\" (UID: \"a5ed9a44-4e39-403c-a936-7f44d5f0fdad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1e47f72-kn24j" Oct 31 05:13:06.104288 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:06.104258 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5ed9a44-4e39-403c-a936-7f44d5f0fdad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1e47f72-kn24j\" (UID: \"a5ed9a44-4e39-403c-a936-7f44d5f0fdad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1e47f72-kn24j" Oct 31 05:13:06.105872 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:06.105844 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5ed9a44-4e39-403c-a936-7f44d5f0fdad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1e47f72-kn24j\" (UID: \"a5ed9a44-4e39-403c-a936-7f44d5f0fdad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1e47f72-kn24j" Oct 31 05:13:06.126550 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:06.126521 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htrq6\" (UniqueName: \"kubernetes.io/projected/a5ed9a44-4e39-403c-a936-7f44d5f0fdad-kube-api-access-htrq6\") pod \"maas-api-revoke-sub-e1e47f72-kn24j\" (UID: \"a5ed9a44-4e39-403c-a936-7f44d5f0fdad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1e47f72-kn24j" Oct 31 05:13:06.292210 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:06.292106 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1e47f72-kn24j" Oct 31 05:13:06.607177 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:06.607142 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1e47f72-kn24j"] Oct 31 05:13:06.607559 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:13:06.607526 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5ed9a44_4e39_403c_a936_7f44d5f0fdad.slice/crio-a1b207b4882fc0ab532ffab004ec7dcc47722b438af343e5f21d7881d27f95b6 WatchSource:0}: Error finding container a1b207b4882fc0ab532ffab004ec7dcc47722b438af343e5f21d7881d27f95b6: Status 404 returned error can't find the container with id a1b207b4882fc0ab532ffab004ec7dcc47722b438af343e5f21d7881d27f95b6 Oct 31 05:13:07.321948 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:07.321907 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1e47f72-kn24j" event={"ID":"a5ed9a44-4e39-403c-a936-7f44d5f0fdad","Type":"ContainerStarted","Data":"e4e64566e2b37b7df4e32fdee6292abc265dad6b21e12f9c33f55821bd14de95"} Oct 31 05:13:07.322412 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:07.321965 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1e47f72-kn24j" event={"ID":"a5ed9a44-4e39-403c-a936-7f44d5f0fdad","Type":"ContainerStarted","Data":"a1b207b4882fc0ab532ffab004ec7dcc47722b438af343e5f21d7881d27f95b6"} Oct 31 05:13:07.366960 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:07.366895 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1e47f72-kn24j" podStartSLOduration=2.366878663 podStartE2EDuration="2.366878663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:13:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:13:07.363197539 +0000 UTC m=+4142.119535605" watchObservedRunningTime="2026-10-31 05:13:07.366878663 +0000 UTC m=+4142.123216702" Oct 31 05:13:07.877389 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:07.877286 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d48ecf64-6n45m"] Oct 31 05:13:07.882885 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:07.882854 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48ecf64-6n45m" Oct 31 05:13:07.900661 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:07.900621 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d48ecf64-6n45m"] Oct 31 05:13:07.933986 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:07.933943 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df1885b0-4e23-47ce-a3aa-ae4e5045750c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d48ecf64-6n45m\" (UID: \"df1885b0-4e23-47ce-a3aa-ae4e5045750c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48ecf64-6n45m" Oct 31 05:13:07.934215 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:07.934132 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df1885b0-4e23-47ce-a3aa-ae4e5045750c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d48ecf64-6n45m\" (UID: \"df1885b0-4e23-47ce-a3aa-ae4e5045750c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48ecf64-6n45m" Oct 31 05:13:07.934215 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:07.934196 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp474\" (UniqueName: \"kubernetes.io/projected/df1885b0-4e23-47ce-a3aa-ae4e5045750c-kube-api-access-vp474\") pod \"maas-api-revoke-sub-d48ecf64-6n45m\" (UID: \"df1885b0-4e23-47ce-a3aa-ae4e5045750c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48ecf64-6n45m" Oct 31 05:13:08.036109 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:08.036063 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df1885b0-4e23-47ce-a3aa-ae4e5045750c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d48ecf64-6n45m\" (UID: \"df1885b0-4e23-47ce-a3aa-ae4e5045750c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48ecf64-6n45m" Oct 31 05:13:08.036313 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:08.036149 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp474\" (UniqueName: \"kubernetes.io/projected/df1885b0-4e23-47ce-a3aa-ae4e5045750c-kube-api-access-vp474\") pod \"maas-api-revoke-sub-d48ecf64-6n45m\" (UID: \"df1885b0-4e23-47ce-a3aa-ae4e5045750c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48ecf64-6n45m" Oct 31 05:13:08.036439 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:08.036312 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df1885b0-4e23-47ce-a3aa-ae4e5045750c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d48ecf64-6n45m\" (UID: \"df1885b0-4e23-47ce-a3aa-ae4e5045750c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48ecf64-6n45m" Oct 31 05:13:08.036929 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:08.036896 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df1885b0-4e23-47ce-a3aa-ae4e5045750c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d48ecf64-6n45m\" (UID: \"df1885b0-4e23-47ce-a3aa-ae4e5045750c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48ecf64-6n45m" Oct 31 05:13:08.038754 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:08.038727 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df1885b0-4e23-47ce-a3aa-ae4e5045750c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d48ecf64-6n45m\" (UID: \"df1885b0-4e23-47ce-a3aa-ae4e5045750c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48ecf64-6n45m" Oct 31 05:13:08.060581 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:08.060543 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp474\" (UniqueName: \"kubernetes.io/projected/df1885b0-4e23-47ce-a3aa-ae4e5045750c-kube-api-access-vp474\") pod \"maas-api-revoke-sub-d48ecf64-6n45m\" (UID: \"df1885b0-4e23-47ce-a3aa-ae4e5045750c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48ecf64-6n45m" Oct 31 05:13:08.201531 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:08.201433 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48ecf64-6n45m" Oct 31 05:13:08.334918 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:08.334883 5019 generic.go:356] "Generic (PLEG): container finished" podID="a5ed9a44-4e39-403c-a936-7f44d5f0fdad" containerID="e4e64566e2b37b7df4e32fdee6292abc265dad6b21e12f9c33f55821bd14de95" exitCode=0 Oct 31 05:13:08.335325 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:08.334973 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1e47f72-kn24j" event={"ID":"a5ed9a44-4e39-403c-a936-7f44d5f0fdad","Type":"ContainerDied","Data":"e4e64566e2b37b7df4e32fdee6292abc265dad6b21e12f9c33f55821bd14de95"} Oct 31 05:13:08.508744 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:08.508712 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d48ecf64-6n45m"] Oct 31 05:13:08.510139 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:13:08.510111 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf1885b0_4e23_47ce_a3aa_ae4e5045750c.slice/crio-2ee739ccf4f3818d00018b9be5151a66f6d0930b45b4bbcb20fa530e268b4a11 WatchSource:0}: Error finding container 2ee739ccf4f3818d00018b9be5151a66f6d0930b45b4bbcb20fa530e268b4a11: Status 404 returned error can't find the container with id 2ee739ccf4f3818d00018b9be5151a66f6d0930b45b4bbcb20fa530e268b4a11 Oct 31 05:13:08.743837 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:08.743800 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-b2a372/rate-test-model-b2a372-kserve-8454cdd6df-bkqc8" Oct 31 05:13:08.756857 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:08.756784 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-b2a372/rate-test-model-b2a372-kserve-8454cdd6df-bkqc8" Oct 31 05:13:08.974251 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:08.974200 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1df64e1e-hdx4d"] Oct 31 05:13:08.979902 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:08.979870 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1df64e1e-hdx4d" Oct 31 05:13:08.997063 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:08.996741 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1df64e1e-hdx4d"] Oct 31 05:13:09.050589 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:09.050489 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t9qj\" (UniqueName: \"kubernetes.io/projected/4972f293-aad0-4dbf-b5cf-925d46e74a3f-kube-api-access-6t9qj\") pod \"maas-api-revoke-sub-1df64e1e-hdx4d\" (UID: \"4972f293-aad0-4dbf-b5cf-925d46e74a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1df64e1e-hdx4d" Oct 31 05:13:09.050743 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:09.050702 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4972f293-aad0-4dbf-b5cf-925d46e74a3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1df64e1e-hdx4d\" (UID: \"4972f293-aad0-4dbf-b5cf-925d46e74a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1df64e1e-hdx4d" Oct 31 05:13:09.050896 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:09.050876 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4972f293-aad0-4dbf-b5cf-925d46e74a3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1df64e1e-hdx4d\" (UID: \"4972f293-aad0-4dbf-b5cf-925d46e74a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1df64e1e-hdx4d" Oct 31 05:13:09.152092 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:09.152056 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4972f293-aad0-4dbf-b5cf-925d46e74a3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1df64e1e-hdx4d\" (UID: \"4972f293-aad0-4dbf-b5cf-925d46e74a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1df64e1e-hdx4d" Oct 31 05:13:09.152295 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:09.152118 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t9qj\" (UniqueName: \"kubernetes.io/projected/4972f293-aad0-4dbf-b5cf-925d46e74a3f-kube-api-access-6t9qj\") pod \"maas-api-revoke-sub-1df64e1e-hdx4d\" (UID: \"4972f293-aad0-4dbf-b5cf-925d46e74a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1df64e1e-hdx4d" Oct 31 05:13:09.152295 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:09.152277 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4972f293-aad0-4dbf-b5cf-925d46e74a3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1df64e1e-hdx4d\" (UID: \"4972f293-aad0-4dbf-b5cf-925d46e74a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1df64e1e-hdx4d" Oct 31 05:13:09.152917 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:09.152896 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4972f293-aad0-4dbf-b5cf-925d46e74a3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1df64e1e-hdx4d\" (UID: \"4972f293-aad0-4dbf-b5cf-925d46e74a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1df64e1e-hdx4d" Oct 31 05:13:09.154438 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:09.154413 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4972f293-aad0-4dbf-b5cf-925d46e74a3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1df64e1e-hdx4d\" (UID: \"4972f293-aad0-4dbf-b5cf-925d46e74a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1df64e1e-hdx4d" Oct 31 05:13:09.177051 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:09.177024 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t9qj\" (UniqueName: \"kubernetes.io/projected/4972f293-aad0-4dbf-b5cf-925d46e74a3f-kube-api-access-6t9qj\") pod \"maas-api-revoke-sub-1df64e1e-hdx4d\" (UID: \"4972f293-aad0-4dbf-b5cf-925d46e74a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1df64e1e-hdx4d" Oct 31 05:13:09.297534 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:09.297491 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1df64e1e-hdx4d" Oct 31 05:13:09.347242 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:09.347199 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48ecf64-6n45m" event={"ID":"df1885b0-4e23-47ce-a3aa-ae4e5045750c","Type":"ContainerStarted","Data":"5728f9525210bcd3d9384d86aa8b03744a9ffce2b0ecb52ec62aa86a0dc4aa50"} Oct 31 05:13:09.347756 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:09.347268 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48ecf64-6n45m" event={"ID":"df1885b0-4e23-47ce-a3aa-ae4e5045750c","Type":"ContainerStarted","Data":"2ee739ccf4f3818d00018b9be5151a66f6d0930b45b4bbcb20fa530e268b4a11"} Oct 31 05:13:09.363158 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:09.362040 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-b2a372/rate-test-model-b2a372-kserve-8454cdd6df-bkqc8" Oct 31 05:13:09.395790 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:09.395733 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48ecf64-6n45m" podStartSLOduration=2.39571663 podStartE2EDuration="2.39571663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:13:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:13:09.388817249 +0000 UTC m=+4144.145155290" watchObservedRunningTime="2026-10-31 05:13:09.39571663 +0000 UTC m=+4144.152054669" Oct 31 05:13:09.568618 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:09.568580 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1e47f72-kn24j" Oct 31 05:13:09.613593 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:13:09.611525 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4972f293_aad0_4dbf_b5cf_925d46e74a3f.slice/crio-8dfe5aa7524ec560c57c045aeb8158b8e3e845cfef1b3a49b8573a1741b0179a WatchSource:0}: Error finding container 8dfe5aa7524ec560c57c045aeb8158b8e3e845cfef1b3a49b8573a1741b0179a: Status 404 returned error can't find the container with id 8dfe5aa7524ec560c57c045aeb8158b8e3e845cfef1b3a49b8573a1741b0179a Oct 31 05:13:09.613593 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:09.612464 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1df64e1e-hdx4d"] Oct 31 05:13:09.659080 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:09.659049 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5ed9a44-4e39-403c-a936-7f44d5f0fdad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5ed9a44-4e39-403c-a936-7f44d5f0fdad-maas-api-openshift-ca\") pod \"a5ed9a44-4e39-403c-a936-7f44d5f0fdad\" (UID: \"a5ed9a44-4e39-403c-a936-7f44d5f0fdad\") " Oct 31 05:13:09.659242 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:09.659103 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5ed9a44-4e39-403c-a936-7f44d5f0fdad-kube-api-access-htrq6\" (UniqueName: \"kubernetes.io/projected/a5ed9a44-4e39-403c-a936-7f44d5f0fdad-kube-api-access-htrq6\") pod \"a5ed9a44-4e39-403c-a936-7f44d5f0fdad\" (UID: \"a5ed9a44-4e39-403c-a936-7f44d5f0fdad\") " Oct 31 05:13:09.659398 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:09.659380 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5ed9a44-4e39-403c-a936-7f44d5f0fdad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5ed9a44-4e39-403c-a936-7f44d5f0fdad-maas-api-xks-ca\") pod \"a5ed9a44-4e39-403c-a936-7f44d5f0fdad\" (UID: \"a5ed9a44-4e39-403c-a936-7f44d5f0fdad\") " Oct 31 05:13:09.659525 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:09.659492 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5ed9a44-4e39-403c-a936-7f44d5f0fdad-maas-api-openshift-ca" pod "a5ed9a44-4e39-403c-a936-7f44d5f0fdad" (UID: "a5ed9a44-4e39-403c-a936-7f44d5f0fdad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:13:09.659966 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:09.659939 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5ed9a44-4e39-403c-a936-7f44d5f0fdad-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:13:09.661679 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:09.661650 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5ed9a44-4e39-403c-a936-7f44d5f0fdad-maas-api-xks-ca" pod "a5ed9a44-4e39-403c-a936-7f44d5f0fdad" (UID: "a5ed9a44-4e39-403c-a936-7f44d5f0fdad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:13:09.661920 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:09.661900 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5ed9a44-4e39-403c-a936-7f44d5f0fdad-kube-api-access-htrq6" pod "a5ed9a44-4e39-403c-a936-7f44d5f0fdad" (UID: "a5ed9a44-4e39-403c-a936-7f44d5f0fdad"). InnerVolumeSpecName "kube-api-access-htrq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:13:09.761704 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:09.761662 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5ed9a44-4e39-403c-a936-7f44d5f0fdad-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:13:09.761704 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:09.761697 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htrq6\" (UniqueName: \"kubernetes.io/projected/a5ed9a44-4e39-403c-a936-7f44d5f0fdad-kube-api-access-htrq6\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:13:10.064619 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:10.064577 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-439989-6b4f868455-v228h"] Oct 31 05:13:10.361395 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:10.361350 5019 generic.go:356] "Generic (PLEG): container finished" podID="df1885b0-4e23-47ce-a3aa-ae4e5045750c" containerID="5728f9525210bcd3d9384d86aa8b03744a9ffce2b0ecb52ec62aa86a0dc4aa50" exitCode=0 Oct 31 05:13:10.361911 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:10.361491 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48ecf64-6n45m" event={"ID":"df1885b0-4e23-47ce-a3aa-ae4e5045750c","Type":"ContainerDied","Data":"5728f9525210bcd3d9384d86aa8b03744a9ffce2b0ecb52ec62aa86a0dc4aa50"} Oct 31 05:13:10.363590 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:10.363558 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1e47f72-kn24j" event={"ID":"a5ed9a44-4e39-403c-a936-7f44d5f0fdad","Type":"ContainerDied","Data":"a1b207b4882fc0ab532ffab004ec7dcc47722b438af343e5f21d7881d27f95b6"} Oct 31 05:13:10.363744 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:10.363606 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1b207b4882fc0ab532ffab004ec7dcc47722b438af343e5f21d7881d27f95b6" Oct 31 05:13:10.363744 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:10.363681 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1e47f72-kn24j" Oct 31 05:13:10.371391 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:10.370262 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1df64e1e-hdx4d" event={"ID":"4972f293-aad0-4dbf-b5cf-925d46e74a3f","Type":"ContainerStarted","Data":"0f9a960171226f10b26990e357211084a65d717f6e5834d500f6a2f7372ab1f3"} Oct 31 05:13:10.371391 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:10.370396 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1df64e1e-hdx4d" event={"ID":"4972f293-aad0-4dbf-b5cf-925d46e74a3f","Type":"ContainerStarted","Data":"8dfe5aa7524ec560c57c045aeb8158b8e3e845cfef1b3a49b8573a1741b0179a"} Oct 31 05:13:10.434640 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:10.434595 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z"] Oct 31 05:13:10.454190 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:10.454050 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1df64e1e-hdx4d" podStartSLOduration=2.45403078 podStartE2EDuration="2.45403078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:13:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:13:10.449591543 +0000 UTC m=+4145.205929585" watchObservedRunningTime="2026-10-31 05:13:10.45403078 +0000 UTC m=+4145.210368820" Oct 31 05:13:11.383607 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:11.383567 5019 generic.go:356] "Generic (PLEG): container finished" podID="4972f293-aad0-4dbf-b5cf-925d46e74a3f" containerID="0f9a960171226f10b26990e357211084a65d717f6e5834d500f6a2f7372ab1f3" exitCode=0 Oct 31 05:13:11.384111 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:11.383657 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1df64e1e-hdx4d" event={"ID":"4972f293-aad0-4dbf-b5cf-925d46e74a3f","Type":"ContainerDied","Data":"0f9a960171226f10b26990e357211084a65d717f6e5834d500f6a2f7372ab1f3"} Oct 31 05:13:11.584254 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:11.584225 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48ecf64-6n45m" Oct 31 05:13:11.710194 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:11.710094 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df1885b0-4e23-47ce-a3aa-ae4e5045750c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df1885b0-4e23-47ce-a3aa-ae4e5045750c-maas-api-xks-ca\") pod \"df1885b0-4e23-47ce-a3aa-ae4e5045750c\" (UID: \"df1885b0-4e23-47ce-a3aa-ae4e5045750c\") " Oct 31 05:13:11.710399 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:11.710244 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df1885b0-4e23-47ce-a3aa-ae4e5045750c-kube-api-access-vp474\" (UniqueName: \"kubernetes.io/projected/df1885b0-4e23-47ce-a3aa-ae4e5045750c-kube-api-access-vp474\") pod \"df1885b0-4e23-47ce-a3aa-ae4e5045750c\" (UID: \"df1885b0-4e23-47ce-a3aa-ae4e5045750c\") " Oct 31 05:13:11.710399 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:11.710283 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df1885b0-4e23-47ce-a3aa-ae4e5045750c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df1885b0-4e23-47ce-a3aa-ae4e5045750c-maas-api-openshift-ca\") pod \"df1885b0-4e23-47ce-a3aa-ae4e5045750c\" (UID: \"df1885b0-4e23-47ce-a3aa-ae4e5045750c\") " Oct 31 05:13:11.710808 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:11.710774 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df1885b0-4e23-47ce-a3aa-ae4e5045750c-maas-api-openshift-ca" pod "df1885b0-4e23-47ce-a3aa-ae4e5045750c" (UID: "df1885b0-4e23-47ce-a3aa-ae4e5045750c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:13:11.712547 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:11.712516 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df1885b0-4e23-47ce-a3aa-ae4e5045750c-maas-api-xks-ca" pod "df1885b0-4e23-47ce-a3aa-ae4e5045750c" (UID: "df1885b0-4e23-47ce-a3aa-ae4e5045750c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:13:11.712659 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:11.712561 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df1885b0-4e23-47ce-a3aa-ae4e5045750c-kube-api-access-vp474" pod "df1885b0-4e23-47ce-a3aa-ae4e5045750c" (UID: "df1885b0-4e23-47ce-a3aa-ae4e5045750c"). InnerVolumeSpecName "kube-api-access-vp474". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:13:11.812780 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:11.812727 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df1885b0-4e23-47ce-a3aa-ae4e5045750c-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:13:11.812780 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:11.812780 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vp474\" (UniqueName: \"kubernetes.io/projected/df1885b0-4e23-47ce-a3aa-ae4e5045750c-kube-api-access-vp474\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:13:11.813014 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:11.812797 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df1885b0-4e23-47ce-a3aa-ae4e5045750c-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:13:12.394963 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:12.394922 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48ecf64-6n45m" Oct 31 05:13:12.395410 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:12.394959 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48ecf64-6n45m" event={"ID":"df1885b0-4e23-47ce-a3aa-ae4e5045750c","Type":"ContainerDied","Data":"2ee739ccf4f3818d00018b9be5151a66f6d0930b45b4bbcb20fa530e268b4a11"} Oct 31 05:13:12.395410 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:12.395010 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ee739ccf4f3818d00018b9be5151a66f6d0930b45b4bbcb20fa530e268b4a11" Oct 31 05:13:12.579895 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:12.579852 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1df64e1e-hdx4d" Oct 31 05:13:12.674200 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:12.674068 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerName="main" containerID="cri-o://89ce61196420a55bb6c098348a2bab85a5be7f15a962f9020650e8734aac397f" gracePeriod=45 Oct 31 05:13:12.730379 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:12.730325 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4972f293-aad0-4dbf-b5cf-925d46e74a3f-kube-api-access-6t9qj\" (UniqueName: \"kubernetes.io/projected/4972f293-aad0-4dbf-b5cf-925d46e74a3f-kube-api-access-6t9qj\") pod \"4972f293-aad0-4dbf-b5cf-925d46e74a3f\" (UID: \"4972f293-aad0-4dbf-b5cf-925d46e74a3f\") " Oct 31 05:13:12.730561 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:12.730460 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4972f293-aad0-4dbf-b5cf-925d46e74a3f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4972f293-aad0-4dbf-b5cf-925d46e74a3f-maas-api-xks-ca\") pod \"4972f293-aad0-4dbf-b5cf-925d46e74a3f\" (UID: \"4972f293-aad0-4dbf-b5cf-925d46e74a3f\") " Oct 31 05:13:12.730561 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:12.730483 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4972f293-aad0-4dbf-b5cf-925d46e74a3f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4972f293-aad0-4dbf-b5cf-925d46e74a3f-maas-api-openshift-ca\") pod \"4972f293-aad0-4dbf-b5cf-925d46e74a3f\" (UID: \"4972f293-aad0-4dbf-b5cf-925d46e74a3f\") " Oct 31 05:13:12.730983 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:12.730955 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4972f293-aad0-4dbf-b5cf-925d46e74a3f-maas-api-openshift-ca" pod "4972f293-aad0-4dbf-b5cf-925d46e74a3f" (UID: "4972f293-aad0-4dbf-b5cf-925d46e74a3f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:13:12.732541 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:12.732516 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4972f293-aad0-4dbf-b5cf-925d46e74a3f-maas-api-xks-ca" pod "4972f293-aad0-4dbf-b5cf-925d46e74a3f" (UID: "4972f293-aad0-4dbf-b5cf-925d46e74a3f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:13:12.732622 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:12.732538 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4972f293-aad0-4dbf-b5cf-925d46e74a3f-kube-api-access-6t9qj" pod "4972f293-aad0-4dbf-b5cf-925d46e74a3f" (UID: "4972f293-aad0-4dbf-b5cf-925d46e74a3f"). InnerVolumeSpecName "kube-api-access-6t9qj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:13:12.832514 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:12.832466 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4972f293-aad0-4dbf-b5cf-925d46e74a3f-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:13:12.832514 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:12.832516 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4972f293-aad0-4dbf-b5cf-925d46e74a3f-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:13:12.832514 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:12.832532 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6t9qj\" (UniqueName: \"kubernetes.io/projected/4972f293-aad0-4dbf-b5cf-925d46e74a3f-kube-api-access-6t9qj\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:13:12.962929 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:12.962892 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" Oct 31 05:13:13.034568 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:13.034528 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/379847f6-6418-43cd-ae5e-dad6fb408eb0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/379847f6-6418-43cd-ae5e-dad6fb408eb0-tmp-dir\") pod \"379847f6-6418-43cd-ae5e-dad6fb408eb0\" (UID: \"379847f6-6418-43cd-ae5e-dad6fb408eb0\") " Oct 31 05:13:13.034785 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:13.034603 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/379847f6-6418-43cd-ae5e-dad6fb408eb0-home\" (UniqueName: \"kubernetes.io/empty-dir/379847f6-6418-43cd-ae5e-dad6fb408eb0-home\") pod \"379847f6-6418-43cd-ae5e-dad6fb408eb0\" (UID: \"379847f6-6418-43cd-ae5e-dad6fb408eb0\") " Oct 31 05:13:13.034785 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:13.034700 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/379847f6-6418-43cd-ae5e-dad6fb408eb0-tls-certs\" (UniqueName: \"kubernetes.io/secret/379847f6-6418-43cd-ae5e-dad6fb408eb0-tls-certs\") pod \"379847f6-6418-43cd-ae5e-dad6fb408eb0\" (UID: \"379847f6-6418-43cd-ae5e-dad6fb408eb0\") " Oct 31 05:13:13.034785 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:13.034750 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/379847f6-6418-43cd-ae5e-dad6fb408eb0-kube-api-access-fklmw\" (UniqueName: \"kubernetes.io/projected/379847f6-6418-43cd-ae5e-dad6fb408eb0-kube-api-access-fklmw\") pod \"379847f6-6418-43cd-ae5e-dad6fb408eb0\" (UID: \"379847f6-6418-43cd-ae5e-dad6fb408eb0\") " Oct 31 05:13:13.034993 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:13.034852 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/379847f6-6418-43cd-ae5e-dad6fb408eb0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/379847f6-6418-43cd-ae5e-dad6fb408eb0-model-cache\") pod \"379847f6-6418-43cd-ae5e-dad6fb408eb0\" (UID: \"379847f6-6418-43cd-ae5e-dad6fb408eb0\") " Oct 31 05:13:13.034993 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:13.034885 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/379847f6-6418-43cd-ae5e-dad6fb408eb0-home" pod "379847f6-6418-43cd-ae5e-dad6fb408eb0" (UID: "379847f6-6418-43cd-ae5e-dad6fb408eb0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:13:13.034993 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:13.034899 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/379847f6-6418-43cd-ae5e-dad6fb408eb0-tmp-dir" pod "379847f6-6418-43cd-ae5e-dad6fb408eb0" (UID: "379847f6-6418-43cd-ae5e-dad6fb408eb0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:13:13.034993 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:13.034913 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/379847f6-6418-43cd-ae5e-dad6fb408eb0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/379847f6-6418-43cd-ae5e-dad6fb408eb0-dshm\") pod \"379847f6-6418-43cd-ae5e-dad6fb408eb0\" (UID: \"379847f6-6418-43cd-ae5e-dad6fb408eb0\") " Oct 31 05:13:13.035211 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:13.035138 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/379847f6-6418-43cd-ae5e-dad6fb408eb0-model-cache" pod "379847f6-6418-43cd-ae5e-dad6fb408eb0" (UID: "379847f6-6418-43cd-ae5e-dad6fb408eb0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:13:13.035734 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:13.035707 5019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/379847f6-6418-43cd-ae5e-dad6fb408eb0-tmp-dir\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:13:13.035734 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:13.035735 5019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/379847f6-6418-43cd-ae5e-dad6fb408eb0-home\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:13:13.035910 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:13.035750 5019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/379847f6-6418-43cd-ae5e-dad6fb408eb0-model-cache\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:13:13.038833 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:13.038800 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/379847f6-6418-43cd-ae5e-dad6fb408eb0-dshm" pod "379847f6-6418-43cd-ae5e-dad6fb408eb0" (UID: "379847f6-6418-43cd-ae5e-dad6fb408eb0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:13:13.038833 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:13.038804 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/379847f6-6418-43cd-ae5e-dad6fb408eb0-kube-api-access-fklmw" pod "379847f6-6418-43cd-ae5e-dad6fb408eb0" (UID: "379847f6-6418-43cd-ae5e-dad6fb408eb0"). InnerVolumeSpecName "kube-api-access-fklmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:13:13.039003 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:13.038804 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/379847f6-6418-43cd-ae5e-dad6fb408eb0-tls-certs" pod "379847f6-6418-43cd-ae5e-dad6fb408eb0" (UID: "379847f6-6418-43cd-ae5e-dad6fb408eb0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:13:13.136943 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:13.136901 5019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/379847f6-6418-43cd-ae5e-dad6fb408eb0-tls-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:13:13.136943 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:13.136940 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fklmw\" (UniqueName: \"kubernetes.io/projected/379847f6-6418-43cd-ae5e-dad6fb408eb0-kube-api-access-fklmw\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:13:13.136943 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:13.136951 5019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/379847f6-6418-43cd-ae5e-dad6fb408eb0-dshm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:13:13.407905 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:13.407865 5019 generic.go:356] "Generic (PLEG): container finished" podID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerID="89ce61196420a55bb6c098348a2bab85a5be7f15a962f9020650e8734aac397f" exitCode=0 Oct 31 05:13:13.408446 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:13.408002 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" Oct 31 05:13:13.409969 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:13.409942 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1df64e1e-hdx4d" Oct 31 05:13:13.412894 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:13.412854 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" event={"ID":"379847f6-6418-43cd-ae5e-dad6fb408eb0","Type":"ContainerDied","Data":"89ce61196420a55bb6c098348a2bab85a5be7f15a962f9020650e8734aac397f"} Oct 31 05:13:13.413025 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:13.412906 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz" event={"ID":"379847f6-6418-43cd-ae5e-dad6fb408eb0","Type":"ContainerDied","Data":"7c1a9f3f22c988127940a3bbb7bacf0cf0519a4f1e951ceec968f462bfdb9d1c"} Oct 31 05:13:13.413025 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:13.412924 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1df64e1e-hdx4d" event={"ID":"4972f293-aad0-4dbf-b5cf-925d46e74a3f","Type":"ContainerDied","Data":"8dfe5aa7524ec560c57c045aeb8158b8e3e845cfef1b3a49b8573a1741b0179a"} Oct 31 05:13:13.413025 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:13.412954 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8dfe5aa7524ec560c57c045aeb8158b8e3e845cfef1b3a49b8573a1741b0179a" Oct 31 05:13:13.413025 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:13.412974 5019 scope.go:122] "RemoveContainer" containerID="89ce61196420a55bb6c098348a2bab85a5be7f15a962f9020650e8734aac397f" Oct 31 05:13:13.431034 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:13.431012 5019 scope.go:122] "RemoveContainer" containerID="89ce61196420a55bb6c098348a2bab85a5be7f15a962f9020650e8734aac397f" Oct 31 05:13:13.431405 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:13:13.431386 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89ce61196420a55bb6c098348a2bab85a5be7f15a962f9020650e8734aac397f\": container with ID starting with 89ce61196420a55bb6c098348a2bab85a5be7f15a962f9020650e8734aac397f not found: ID does not exist" containerID="89ce61196420a55bb6c098348a2bab85a5be7f15a962f9020650e8734aac397f" Oct 31 05:13:13.431476 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:13.431413 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89ce61196420a55bb6c098348a2bab85a5be7f15a962f9020650e8734aac397f"} err="failed to get container status \"89ce61196420a55bb6c098348a2bab85a5be7f15a962f9020650e8734aac397f\": rpc error: code = NotFound desc = could not find container \"89ce61196420a55bb6c098348a2bab85a5be7f15a962f9020650e8734aac397f\": container with ID starting with 89ce61196420a55bb6c098348a2bab85a5be7f15a962f9020650e8734aac397f not found: ID does not exist" Oct 31 05:13:13.464111 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:13.464066 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz"] Oct 31 05:13:13.468793 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:13.468757 5019 scope.go:122] "RemoveContainer" containerID="2e86d0c70d9817b9795d6283820a5ab603f9b56b8b29ea956690922afb061a0c" Oct 31 05:13:13.469153 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:13.469123 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-439989/simulator-w1-439989-kserve-75445c78dd-77sjz"] Oct 31 05:13:15.065787 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:15.065733 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-439989-6b4f868455-v228h" podUID="406761c9-51b9-4105-88ce-bd0fac558eb1" containerName="payload-pre-processing" containerID="cri-o://e3de073ce9e0303b08fd111494884cd51743cb376c8b21e0cb83879589726517" gracePeriod=25 Oct 31 05:13:15.412165 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:15.412074 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" path="/var/lib/kubelet/pods/379847f6-6418-43cd-ae5e-dad6fb408eb0/volumes" Oct 31 05:13:18.278681 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:18.278642 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01c821fb-shvcn"] Oct 31 05:13:18.279355 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:18.279319 5019 cpu_manager.go:414] "containerMap: removing container" podUID="4972f293-aad0-4dbf-b5cf-925d46e74a3f" containerName="revoke-keys" Oct 31 05:13:18.279355 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:18.279351 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4972f293-aad0-4dbf-b5cf-925d46e74a3f" containerName="revoke-keys" Oct 31 05:13:18.279558 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:18.279395 5019 cpu_manager.go:414] "containerMap: removing container" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerName="main" Oct 31 05:13:18.279558 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:18.279401 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerName="main" Oct 31 05:13:18.279558 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:18.279424 5019 cpu_manager.go:414] "containerMap: removing container" podUID="df1885b0-4e23-47ce-a3aa-ae4e5045750c" containerName="revoke-keys" Oct 31 05:13:18.279558 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:18.279430 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df1885b0-4e23-47ce-a3aa-ae4e5045750c" containerName="revoke-keys" Oct 31 05:13:18.279558 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:18.279442 5019 cpu_manager.go:414] "containerMap: removing container" podUID="a5ed9a44-4e39-403c-a936-7f44d5f0fdad" containerName="revoke-keys" Oct 31 05:13:18.279558 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:18.279447 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5ed9a44-4e39-403c-a936-7f44d5f0fdad" containerName="revoke-keys" Oct 31 05:13:18.279782 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:18.279586 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="379847f6-6418-43cd-ae5e-dad6fb408eb0" containerName="main" Oct 31 05:13:18.279782 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:18.279612 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="df1885b0-4e23-47ce-a3aa-ae4e5045750c" containerName="revoke-keys" Oct 31 05:13:18.279782 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:18.279623 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5ed9a44-4e39-403c-a936-7f44d5f0fdad" containerName="revoke-keys" Oct 31 05:13:18.279782 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:18.279637 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="4972f293-aad0-4dbf-b5cf-925d46e74a3f" containerName="revoke-keys" Oct 31 05:13:18.284611 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:18.284589 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c821fb-shvcn" Oct 31 05:13:18.302919 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:18.302879 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01c821fb-shvcn"] Oct 31 05:13:18.312372 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:18.309735 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:13:18.312372 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:18.309800 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 05:13:18.312372 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:18.309808 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:13:18.312372 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:18.310042 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:13:18.410731 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:18.410690 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djr5f\" (UniqueName: \"kubernetes.io/projected/aa224080-7d7b-4c02-b09f-b8dd69076408-kube-api-access-djr5f\") pod \"maas-api-revoke-sub-01c821fb-shvcn\" (UID: \"aa224080-7d7b-4c02-b09f-b8dd69076408\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c821fb-shvcn" Oct 31 05:13:18.410956 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:18.410798 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa224080-7d7b-4c02-b09f-b8dd69076408-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01c821fb-shvcn\" (UID: \"aa224080-7d7b-4c02-b09f-b8dd69076408\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c821fb-shvcn" Oct 31 05:13:18.410956 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:18.410839 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa224080-7d7b-4c02-b09f-b8dd69076408-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01c821fb-shvcn\" (UID: \"aa224080-7d7b-4c02-b09f-b8dd69076408\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c821fb-shvcn" Oct 31 05:13:18.512827 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:18.512788 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djr5f\" (UniqueName: \"kubernetes.io/projected/aa224080-7d7b-4c02-b09f-b8dd69076408-kube-api-access-djr5f\") pod \"maas-api-revoke-sub-01c821fb-shvcn\" (UID: \"aa224080-7d7b-4c02-b09f-b8dd69076408\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c821fb-shvcn" Oct 31 05:13:18.513039 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:18.512984 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa224080-7d7b-4c02-b09f-b8dd69076408-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01c821fb-shvcn\" (UID: \"aa224080-7d7b-4c02-b09f-b8dd69076408\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c821fb-shvcn" Oct 31 05:13:18.513169 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:18.513129 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa224080-7d7b-4c02-b09f-b8dd69076408-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01c821fb-shvcn\" (UID: \"aa224080-7d7b-4c02-b09f-b8dd69076408\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c821fb-shvcn" Oct 31 05:13:18.513861 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:18.513837 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa224080-7d7b-4c02-b09f-b8dd69076408-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01c821fb-shvcn\" (UID: \"aa224080-7d7b-4c02-b09f-b8dd69076408\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c821fb-shvcn" Oct 31 05:13:18.515563 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:18.515542 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa224080-7d7b-4c02-b09f-b8dd69076408-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01c821fb-shvcn\" (UID: \"aa224080-7d7b-4c02-b09f-b8dd69076408\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c821fb-shvcn" Oct 31 05:13:18.536448 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:18.536320 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djr5f\" (UniqueName: \"kubernetes.io/projected/aa224080-7d7b-4c02-b09f-b8dd69076408-kube-api-access-djr5f\") pod \"maas-api-revoke-sub-01c821fb-shvcn\" (UID: \"aa224080-7d7b-4c02-b09f-b8dd69076408\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c821fb-shvcn" Oct 31 05:13:18.600289 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:18.600254 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c821fb-shvcn" Oct 31 05:13:18.717181 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:18.717138 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-439989-6b4f868455-v228h" podUID="406761c9-51b9-4105-88ce-bd0fac558eb1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:13:18.937432 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:18.937293 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01c821fb-shvcn"] Oct 31 05:13:19.467346 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:19.467293 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c821fb-shvcn" event={"ID":"aa224080-7d7b-4c02-b09f-b8dd69076408","Type":"ContainerStarted","Data":"c1a5a492e11ae9f239042814de50379ae2dd961c1be57948d30d9b4ba983244d"} Oct 31 05:13:19.467878 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:19.467376 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c821fb-shvcn" event={"ID":"aa224080-7d7b-4c02-b09f-b8dd69076408","Type":"ContainerStarted","Data":"a529cce5309788fc4321d86c3812cfd36b60200ceac7a089eaf3e08851d6a234"} Oct 31 05:13:19.508908 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:19.508850 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c821fb-shvcn" podStartSLOduration=1.5088342049999999 podStartE2EDuration="1.508834205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:13:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:13:19.506294393 +0000 UTC m=+4154.262632435" watchObservedRunningTime="2026-10-31 05:13:19.508834205 +0000 UTC m=+4154.265172285" Oct 31 05:13:20.477208 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:20.477175 5019 generic.go:356] "Generic (PLEG): container finished" podID="aa224080-7d7b-4c02-b09f-b8dd69076408" containerID="c1a5a492e11ae9f239042814de50379ae2dd961c1be57948d30d9b4ba983244d" exitCode=0 Oct 31 05:13:20.477638 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:20.477361 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c821fb-shvcn" event={"ID":"aa224080-7d7b-4c02-b09f-b8dd69076408","Type":"ContainerDied","Data":"c1a5a492e11ae9f239042814de50379ae2dd961c1be57948d30d9b4ba983244d"} Oct 31 05:13:21.684841 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:21.684809 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c821fb-shvcn" Oct 31 05:13:21.863607 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:21.863491 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa224080-7d7b-4c02-b09f-b8dd69076408-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa224080-7d7b-4c02-b09f-b8dd69076408-maas-api-openshift-ca\") pod \"aa224080-7d7b-4c02-b09f-b8dd69076408\" (UID: \"aa224080-7d7b-4c02-b09f-b8dd69076408\") " Oct 31 05:13:21.863607 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:21.863570 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa224080-7d7b-4c02-b09f-b8dd69076408-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa224080-7d7b-4c02-b09f-b8dd69076408-maas-api-xks-ca\") pod \"aa224080-7d7b-4c02-b09f-b8dd69076408\" (UID: \"aa224080-7d7b-4c02-b09f-b8dd69076408\") " Oct 31 05:13:21.863880 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:21.863698 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa224080-7d7b-4c02-b09f-b8dd69076408-kube-api-access-djr5f\" (UniqueName: \"kubernetes.io/projected/aa224080-7d7b-4c02-b09f-b8dd69076408-kube-api-access-djr5f\") pod \"aa224080-7d7b-4c02-b09f-b8dd69076408\" (UID: \"aa224080-7d7b-4c02-b09f-b8dd69076408\") " Oct 31 05:13:21.863995 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:21.863966 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa224080-7d7b-4c02-b09f-b8dd69076408-maas-api-openshift-ca" pod "aa224080-7d7b-4c02-b09f-b8dd69076408" (UID: "aa224080-7d7b-4c02-b09f-b8dd69076408"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:13:21.864277 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:21.864255 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa224080-7d7b-4c02-b09f-b8dd69076408-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:13:21.866033 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:21.866000 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa224080-7d7b-4c02-b09f-b8dd69076408-kube-api-access-djr5f" pod "aa224080-7d7b-4c02-b09f-b8dd69076408" (UID: "aa224080-7d7b-4c02-b09f-b8dd69076408"). InnerVolumeSpecName "kube-api-access-djr5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:13:21.866142 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:21.866008 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa224080-7d7b-4c02-b09f-b8dd69076408-maas-api-xks-ca" pod "aa224080-7d7b-4c02-b09f-b8dd69076408" (UID: "aa224080-7d7b-4c02-b09f-b8dd69076408"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:13:21.965657 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:21.965612 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa224080-7d7b-4c02-b09f-b8dd69076408-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:13:21.965657 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:21.965654 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djr5f\" (UniqueName: \"kubernetes.io/projected/aa224080-7d7b-4c02-b09f-b8dd69076408-kube-api-access-djr5f\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:13:22.453907 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:13:22.453860 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:13:22.501310 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:22.501272 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c821fb-shvcn" Oct 31 05:13:22.501564 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:22.501265 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c821fb-shvcn" event={"ID":"aa224080-7d7b-4c02-b09f-b8dd69076408","Type":"ContainerDied","Data":"a529cce5309788fc4321d86c3812cfd36b60200ceac7a089eaf3e08851d6a234"} Oct 31 05:13:22.501564 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:22.501405 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a529cce5309788fc4321d86c3812cfd36b60200ceac7a089eaf3e08851d6a234" Oct 31 05:13:23.417010 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:23.416959 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a182845-st9jv"] Oct 31 05:13:23.417701 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:23.417682 5019 cpu_manager.go:414] "containerMap: removing container" podUID="aa224080-7d7b-4c02-b09f-b8dd69076408" containerName="revoke-keys" Oct 31 05:13:23.417761 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:23.417702 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa224080-7d7b-4c02-b09f-b8dd69076408" containerName="revoke-keys" Oct 31 05:13:23.417847 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:23.417838 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa224080-7d7b-4c02-b09f-b8dd69076408" containerName="revoke-keys" Oct 31 05:13:23.421752 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:23.421728 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a182845-st9jv" Oct 31 05:13:23.438742 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:23.438710 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a182845-st9jv"] Oct 31 05:13:23.445567 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:23.445529 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:13:23.445798 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:23.445629 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 05:13:23.494186 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:23.494146 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jcvj\" (UniqueName: \"kubernetes.io/projected/2bea695c-448e-42b8-bc47-d381c431e1c8-kube-api-access-2jcvj\") pod \"maas-api-revoke-sub-5a182845-st9jv\" (UID: \"2bea695c-448e-42b8-bc47-d381c431e1c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a182845-st9jv" Oct 31 05:13:23.494457 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:23.494208 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bea695c-448e-42b8-bc47-d381c431e1c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a182845-st9jv\" (UID: \"2bea695c-448e-42b8-bc47-d381c431e1c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a182845-st9jv" Oct 31 05:13:23.494457 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:23.494317 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bea695c-448e-42b8-bc47-d381c431e1c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a182845-st9jv\" (UID: \"2bea695c-448e-42b8-bc47-d381c431e1c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a182845-st9jv" Oct 31 05:13:23.527709 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:23.527669 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:13:23.527876 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:23.527723 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:13:23.596163 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:23.596121 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jcvj\" (UniqueName: \"kubernetes.io/projected/2bea695c-448e-42b8-bc47-d381c431e1c8-kube-api-access-2jcvj\") pod \"maas-api-revoke-sub-5a182845-st9jv\" (UID: \"2bea695c-448e-42b8-bc47-d381c431e1c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a182845-st9jv" Oct 31 05:13:23.596411 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:23.596182 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bea695c-448e-42b8-bc47-d381c431e1c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a182845-st9jv\" (UID: \"2bea695c-448e-42b8-bc47-d381c431e1c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a182845-st9jv" Oct 31 05:13:23.596411 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:23.596245 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bea695c-448e-42b8-bc47-d381c431e1c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a182845-st9jv\" (UID: \"2bea695c-448e-42b8-bc47-d381c431e1c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a182845-st9jv" Oct 31 05:13:23.596974 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:23.596948 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bea695c-448e-42b8-bc47-d381c431e1c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a182845-st9jv\" (UID: \"2bea695c-448e-42b8-bc47-d381c431e1c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a182845-st9jv" Oct 31 05:13:23.598839 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:23.598814 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bea695c-448e-42b8-bc47-d381c431e1c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a182845-st9jv\" (UID: \"2bea695c-448e-42b8-bc47-d381c431e1c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a182845-st9jv" Oct 31 05:13:23.619557 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:23.619513 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jcvj\" (UniqueName: \"kubernetes.io/projected/2bea695c-448e-42b8-bc47-d381c431e1c8-kube-api-access-2jcvj\") pod \"maas-api-revoke-sub-5a182845-st9jv\" (UID: \"2bea695c-448e-42b8-bc47-d381c431e1c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a182845-st9jv" Oct 31 05:13:23.738058 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:23.738009 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a182845-st9jv" Oct 31 05:13:24.041219 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:24.041180 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a182845-st9jv"] Oct 31 05:13:24.041592 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:13:24.041568 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bea695c_448e_42b8_bc47_d381c431e1c8.slice/crio-20828684257db953675ac5e7e913d54f12bb7a8b08bae1d6f16004e747e307a3 WatchSource:0}: Error finding container 20828684257db953675ac5e7e913d54f12bb7a8b08bae1d6f16004e747e307a3: Status 404 returned error can't find the container with id 20828684257db953675ac5e7e913d54f12bb7a8b08bae1d6f16004e747e307a3 Oct 31 05:13:24.520022 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:24.519981 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a182845-st9jv" event={"ID":"2bea695c-448e-42b8-bc47-d381c431e1c8","Type":"ContainerStarted","Data":"1c741c6c8dee8fad5e0e7c6a46c50f8756f92222365ccb310742a2099e35a20b"} Oct 31 05:13:24.520481 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:24.520048 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a182845-st9jv" event={"ID":"2bea695c-448e-42b8-bc47-d381c431e1c8","Type":"ContainerStarted","Data":"20828684257db953675ac5e7e913d54f12bb7a8b08bae1d6f16004e747e307a3"} Oct 31 05:13:24.601734 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:24.601674 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a182845-st9jv" podStartSLOduration=1.6016590000000002 podStartE2EDuration="1.601659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:13:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:13:24.597535535 +0000 UTC m=+4159.353873572" watchObservedRunningTime="2026-10-31 05:13:24.601659 +0000 UTC m=+4159.357997083" Oct 31 05:13:25.458647 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:25.458542 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" podUID="dccd15a9-47f5-4bd1-b21b-9d5ca8b05845" containerName="main" containerID="cri-o://3df64f167fe46cef28bd6b07f1ee9c922d82d33b5a7bed53ae1eae3350321d01" gracePeriod=45 Oct 31 05:13:25.529859 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:25.529816 5019 generic.go:356] "Generic (PLEG): container finished" podID="2bea695c-448e-42b8-bc47-d381c431e1c8" containerID="1c741c6c8dee8fad5e0e7c6a46c50f8756f92222365ccb310742a2099e35a20b" exitCode=0 Oct 31 05:13:25.530290 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:25.529896 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a182845-st9jv" event={"ID":"2bea695c-448e-42b8-bc47-d381c431e1c8","Type":"ContainerDied","Data":"1c741c6c8dee8fad5e0e7c6a46c50f8756f92222365ccb310742a2099e35a20b"} Oct 31 05:13:25.785024 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:25.784986 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" Oct 31 05:13:25.827317 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:25.827267 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dccd15a9-47f5-4bd1-b21b-9d5ca8b05845-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dccd15a9-47f5-4bd1-b21b-9d5ca8b05845-tmp-dir\") pod \"dccd15a9-47f5-4bd1-b21b-9d5ca8b05845\" (UID: \"dccd15a9-47f5-4bd1-b21b-9d5ca8b05845\") " Oct 31 05:13:25.827588 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:25.827383 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dccd15a9-47f5-4bd1-b21b-9d5ca8b05845-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dccd15a9-47f5-4bd1-b21b-9d5ca8b05845-dshm\") pod \"dccd15a9-47f5-4bd1-b21b-9d5ca8b05845\" (UID: \"dccd15a9-47f5-4bd1-b21b-9d5ca8b05845\") " Oct 31 05:13:25.827588 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:25.827457 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dccd15a9-47f5-4bd1-b21b-9d5ca8b05845-kube-api-access-22z6n\" (UniqueName: \"kubernetes.io/projected/dccd15a9-47f5-4bd1-b21b-9d5ca8b05845-kube-api-access-22z6n\") pod \"dccd15a9-47f5-4bd1-b21b-9d5ca8b05845\" (UID: \"dccd15a9-47f5-4bd1-b21b-9d5ca8b05845\") " Oct 31 05:13:25.827588 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:25.827579 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dccd15a9-47f5-4bd1-b21b-9d5ca8b05845-tls-certs\" (UniqueName: \"kubernetes.io/secret/dccd15a9-47f5-4bd1-b21b-9d5ca8b05845-tls-certs\") pod \"dccd15a9-47f5-4bd1-b21b-9d5ca8b05845\" (UID: \"dccd15a9-47f5-4bd1-b21b-9d5ca8b05845\") " Oct 31 05:13:25.827824 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:25.827672 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dccd15a9-47f5-4bd1-b21b-9d5ca8b05845-home\" (UniqueName: \"kubernetes.io/empty-dir/dccd15a9-47f5-4bd1-b21b-9d5ca8b05845-home\") pod \"dccd15a9-47f5-4bd1-b21b-9d5ca8b05845\" (UID: \"dccd15a9-47f5-4bd1-b21b-9d5ca8b05845\") " Oct 31 05:13:25.827824 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:25.827702 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dccd15a9-47f5-4bd1-b21b-9d5ca8b05845-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dccd15a9-47f5-4bd1-b21b-9d5ca8b05845-model-cache\") pod \"dccd15a9-47f5-4bd1-b21b-9d5ca8b05845\" (UID: \"dccd15a9-47f5-4bd1-b21b-9d5ca8b05845\") " Oct 31 05:13:25.828911 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:25.828879 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dccd15a9-47f5-4bd1-b21b-9d5ca8b05845-home" pod "dccd15a9-47f5-4bd1-b21b-9d5ca8b05845" (UID: "dccd15a9-47f5-4bd1-b21b-9d5ca8b05845"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:13:25.829066 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:25.829029 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dccd15a9-47f5-4bd1-b21b-9d5ca8b05845-model-cache" pod "dccd15a9-47f5-4bd1-b21b-9d5ca8b05845" (UID: "dccd15a9-47f5-4bd1-b21b-9d5ca8b05845"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:13:25.829208 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:25.829163 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dccd15a9-47f5-4bd1-b21b-9d5ca8b05845-tmp-dir" pod "dccd15a9-47f5-4bd1-b21b-9d5ca8b05845" (UID: "dccd15a9-47f5-4bd1-b21b-9d5ca8b05845"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:13:25.832923 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:25.832887 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dccd15a9-47f5-4bd1-b21b-9d5ca8b05845-tls-certs" pod "dccd15a9-47f5-4bd1-b21b-9d5ca8b05845" (UID: "dccd15a9-47f5-4bd1-b21b-9d5ca8b05845"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:13:25.858885 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:25.842009 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dccd15a9-47f5-4bd1-b21b-9d5ca8b05845-dshm" pod "dccd15a9-47f5-4bd1-b21b-9d5ca8b05845" (UID: "dccd15a9-47f5-4bd1-b21b-9d5ca8b05845"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:13:25.858885 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:25.842012 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dccd15a9-47f5-4bd1-b21b-9d5ca8b05845-kube-api-access-22z6n" pod "dccd15a9-47f5-4bd1-b21b-9d5ca8b05845" (UID: "dccd15a9-47f5-4bd1-b21b-9d5ca8b05845"). InnerVolumeSpecName "kube-api-access-22z6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:13:25.929994 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:25.929949 5019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dccd15a9-47f5-4bd1-b21b-9d5ca8b05845-tmp-dir\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:13:25.929994 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:25.929993 5019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dccd15a9-47f5-4bd1-b21b-9d5ca8b05845-dshm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:13:25.930266 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:25.930008 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22z6n\" (UniqueName: \"kubernetes.io/projected/dccd15a9-47f5-4bd1-b21b-9d5ca8b05845-kube-api-access-22z6n\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:13:25.930266 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:25.930024 5019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dccd15a9-47f5-4bd1-b21b-9d5ca8b05845-tls-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:13:25.930266 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:25.930037 5019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dccd15a9-47f5-4bd1-b21b-9d5ca8b05845-home\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:13:25.930266 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:25.930051 5019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dccd15a9-47f5-4bd1-b21b-9d5ca8b05845-model-cache\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:13:26.540216 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:26.540178 5019 generic.go:356] "Generic (PLEG): container finished" podID="dccd15a9-47f5-4bd1-b21b-9d5ca8b05845" containerID="3df64f167fe46cef28bd6b07f1ee9c922d82d33b5a7bed53ae1eae3350321d01" exitCode=0 Oct 31 05:13:26.540707 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:26.540226 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" Oct 31 05:13:26.540707 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:26.540267 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" event={"ID":"dccd15a9-47f5-4bd1-b21b-9d5ca8b05845","Type":"ContainerDied","Data":"3df64f167fe46cef28bd6b07f1ee9c922d82d33b5a7bed53ae1eae3350321d01"} Oct 31 05:13:26.540707 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:26.540318 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z" event={"ID":"dccd15a9-47f5-4bd1-b21b-9d5ca8b05845","Type":"ContainerDied","Data":"b995751e0675e5910d98110c0eaa2d9c11aff682da5a2617db7d80616503b78e"} Oct 31 05:13:26.540707 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:26.540373 5019 scope.go:122] "RemoveContainer" containerID="3df64f167fe46cef28bd6b07f1ee9c922d82d33b5a7bed53ae1eae3350321d01" Oct 31 05:13:26.557697 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:26.557667 5019 scope.go:122] "RemoveContainer" containerID="3df64f167fe46cef28bd6b07f1ee9c922d82d33b5a7bed53ae1eae3350321d01" Oct 31 05:13:26.558087 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:13:26.558052 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3df64f167fe46cef28bd6b07f1ee9c922d82d33b5a7bed53ae1eae3350321d01\": container with ID starting with 3df64f167fe46cef28bd6b07f1ee9c922d82d33b5a7bed53ae1eae3350321d01 not found: ID does not exist" containerID="3df64f167fe46cef28bd6b07f1ee9c922d82d33b5a7bed53ae1eae3350321d01" Oct 31 05:13:26.558195 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:26.558098 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3df64f167fe46cef28bd6b07f1ee9c922d82d33b5a7bed53ae1eae3350321d01"} err="failed to get container status \"3df64f167fe46cef28bd6b07f1ee9c922d82d33b5a7bed53ae1eae3350321d01\": rpc error: code = NotFound desc = could not find container \"3df64f167fe46cef28bd6b07f1ee9c922d82d33b5a7bed53ae1eae3350321d01\": container with ID starting with 3df64f167fe46cef28bd6b07f1ee9c922d82d33b5a7bed53ae1eae3350321d01 not found: ID does not exist" Oct 31 05:13:26.597002 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:26.596953 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z"] Oct 31 05:13:26.608642 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:26.608602 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-09a20a/simulator-w2-09a20a-kserve-5dc88fc844-8jf6z"] Oct 31 05:13:26.731621 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:26.731582 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a182845-st9jv" Oct 31 05:13:26.844397 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:26.844248 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bea695c-448e-42b8-bc47-d381c431e1c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bea695c-448e-42b8-bc47-d381c431e1c8-maas-api-xks-ca\") pod \"2bea695c-448e-42b8-bc47-d381c431e1c8\" (UID: \"2bea695c-448e-42b8-bc47-d381c431e1c8\") " Oct 31 05:13:26.844570 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:26.844407 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bea695c-448e-42b8-bc47-d381c431e1c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bea695c-448e-42b8-bc47-d381c431e1c8-maas-api-openshift-ca\") pod \"2bea695c-448e-42b8-bc47-d381c431e1c8\" (UID: \"2bea695c-448e-42b8-bc47-d381c431e1c8\") " Oct 31 05:13:26.844619 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:26.844578 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bea695c-448e-42b8-bc47-d381c431e1c8-kube-api-access-2jcvj\" (UniqueName: \"kubernetes.io/projected/2bea695c-448e-42b8-bc47-d381c431e1c8-kube-api-access-2jcvj\") pod \"2bea695c-448e-42b8-bc47-d381c431e1c8\" (UID: \"2bea695c-448e-42b8-bc47-d381c431e1c8\") " Oct 31 05:13:26.845445 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:26.845407 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bea695c-448e-42b8-bc47-d381c431e1c8-maas-api-openshift-ca" pod "2bea695c-448e-42b8-bc47-d381c431e1c8" (UID: "2bea695c-448e-42b8-bc47-d381c431e1c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:13:26.861445 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:26.861394 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bea695c-448e-42b8-bc47-d381c431e1c8-kube-api-access-2jcvj" pod "2bea695c-448e-42b8-bc47-d381c431e1c8" (UID: "2bea695c-448e-42b8-bc47-d381c431e1c8"). InnerVolumeSpecName "kube-api-access-2jcvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:13:26.861445 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:26.861407 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bea695c-448e-42b8-bc47-d381c431e1c8-maas-api-xks-ca" pod "2bea695c-448e-42b8-bc47-d381c431e1c8" (UID: "2bea695c-448e-42b8-bc47-d381c431e1c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:13:26.947536 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:26.947474 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jcvj\" (UniqueName: \"kubernetes.io/projected/2bea695c-448e-42b8-bc47-d381c431e1c8-kube-api-access-2jcvj\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:13:26.947536 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:26.947520 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bea695c-448e-42b8-bc47-d381c431e1c8-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:13:26.947536 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:26.947540 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bea695c-448e-42b8-bc47-d381c431e1c8-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:13:27.411911 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:27.411868 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dccd15a9-47f5-4bd1-b21b-9d5ca8b05845" path="/var/lib/kubelet/pods/dccd15a9-47f5-4bd1-b21b-9d5ca8b05845/volumes" Oct 31 05:13:27.550255 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:27.550217 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a182845-st9jv" Oct 31 05:13:27.550255 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:27.550247 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a182845-st9jv" event={"ID":"2bea695c-448e-42b8-bc47-d381c431e1c8","Type":"ContainerDied","Data":"20828684257db953675ac5e7e913d54f12bb7a8b08bae1d6f16004e747e307a3"} Oct 31 05:13:27.550784 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:27.550300 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20828684257db953675ac5e7e913d54f12bb7a8b08bae1d6f16004e747e307a3" Oct 31 05:13:28.717815 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:28.717776 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-439989-6b4f868455-v228h" podUID="406761c9-51b9-4105-88ce-bd0fac558eb1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:13:35.451344 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:35.451179 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-439989-6b4f868455-v228h" Oct 31 05:13:35.595906 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:35.595806 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/406761c9-51b9-4105-88ce-bd0fac558eb1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/406761c9-51b9-4105-88ce-bd0fac558eb1-plugins-config-volume\") pod \"406761c9-51b9-4105-88ce-bd0fac558eb1\" (UID: \"406761c9-51b9-4105-88ce-bd0fac558eb1\") " Oct 31 05:13:35.596081 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:35.595975 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/406761c9-51b9-4105-88ce-bd0fac558eb1-kube-api-access-qnf59\" (UniqueName: \"kubernetes.io/projected/406761c9-51b9-4105-88ce-bd0fac558eb1-kube-api-access-qnf59\") pod \"406761c9-51b9-4105-88ce-bd0fac558eb1\" (UID: \"406761c9-51b9-4105-88ce-bd0fac558eb1\") " Oct 31 05:13:35.596292 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:35.596269 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/406761c9-51b9-4105-88ce-bd0fac558eb1-plugins-config-volume" pod "406761c9-51b9-4105-88ce-bd0fac558eb1" (UID: "406761c9-51b9-4105-88ce-bd0fac558eb1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:13:35.596594 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:35.596573 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/406761c9-51b9-4105-88ce-bd0fac558eb1-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:13:35.598296 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:35.598224 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/406761c9-51b9-4105-88ce-bd0fac558eb1-kube-api-access-qnf59" pod "406761c9-51b9-4105-88ce-bd0fac558eb1" (UID: "406761c9-51b9-4105-88ce-bd0fac558eb1"). InnerVolumeSpecName "kube-api-access-qnf59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:13:35.621599 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:35.621559 5019 generic.go:356] "Generic (PLEG): container finished" podID="406761c9-51b9-4105-88ce-bd0fac558eb1" containerID="e3de073ce9e0303b08fd111494884cd51743cb376c8b21e0cb83879589726517" exitCode=0 Oct 31 05:13:35.621793 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:35.621608 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-439989-6b4f868455-v228h" Oct 31 05:13:35.621793 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:35.621637 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-439989-6b4f868455-v228h" event={"ID":"406761c9-51b9-4105-88ce-bd0fac558eb1","Type":"ContainerDied","Data":"e3de073ce9e0303b08fd111494884cd51743cb376c8b21e0cb83879589726517"} Oct 31 05:13:35.621793 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:35.621698 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-439989-6b4f868455-v228h" event={"ID":"406761c9-51b9-4105-88ce-bd0fac558eb1","Type":"ContainerDied","Data":"d0e6f4a5fbc8d8385469b4f1af642d66352a7889ff62dca62812e9df4667bd99"} Oct 31 05:13:35.621793 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:35.621725 5019 scope.go:122] "RemoveContainer" containerID="e3de073ce9e0303b08fd111494884cd51743cb376c8b21e0cb83879589726517" Oct 31 05:13:35.633899 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:35.633879 5019 scope.go:122] "RemoveContainer" containerID="e3de073ce9e0303b08fd111494884cd51743cb376c8b21e0cb83879589726517" Oct 31 05:13:35.634159 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:13:35.634138 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3de073ce9e0303b08fd111494884cd51743cb376c8b21e0cb83879589726517\": container with ID starting with e3de073ce9e0303b08fd111494884cd51743cb376c8b21e0cb83879589726517 not found: ID does not exist" containerID="e3de073ce9e0303b08fd111494884cd51743cb376c8b21e0cb83879589726517" Oct 31 05:13:35.634199 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:35.634171 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3de073ce9e0303b08fd111494884cd51743cb376c8b21e0cb83879589726517"} err="failed to get container status \"e3de073ce9e0303b08fd111494884cd51743cb376c8b21e0cb83879589726517\": rpc error: code = NotFound desc = could not find container \"e3de073ce9e0303b08fd111494884cd51743cb376c8b21e0cb83879589726517\": container with ID starting with e3de073ce9e0303b08fd111494884cd51743cb376c8b21e0cb83879589726517 not found: ID does not exist" Oct 31 05:13:35.676150 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:35.676117 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-439989-6b4f868455-v228h"] Oct 31 05:13:35.682246 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:35.682220 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-439989-6b4f868455-v228h"] Oct 31 05:13:35.698437 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:35.698408 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnf59\" (UniqueName: \"kubernetes.io/projected/406761c9-51b9-4105-88ce-bd0fac558eb1-kube-api-access-qnf59\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:13:37.413622 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:37.413578 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="406761c9-51b9-4105-88ce-bd0fac558eb1" path="/var/lib/kubelet/pods/406761c9-51b9-4105-88ce-bd0fac558eb1/volumes" Oct 31 05:13:41.729269 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:41.729213 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r"] Oct 31 05:13:41.729976 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:41.729910 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r" podUID="fffb7c60-8cb2-40f0-903c-9b98c62a1741" containerName="istio-proxy" containerID="cri-o://d13c99ec9cfcc52db329243fc9f3e286ec2b59763c5037648baa5571038be0f6" gracePeriod=30 Oct 31 05:13:44.913595 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:44.913549 5019 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 05:13:44.914603 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:44.914223 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r" podUID="fffb7c60-8cb2-40f0-903c-9b98c62a1741" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 05:13:47.097115 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:47.097074 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r" Oct 31 05:13:47.211258 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:47.211216 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fffb7c60-8cb2-40f0-903c-9b98c62a1741-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fffb7c60-8cb2-40f0-903c-9b98c62a1741-istio-data\") pod \"fffb7c60-8cb2-40f0-903c-9b98c62a1741\" (UID: \"fffb7c60-8cb2-40f0-903c-9b98c62a1741\") " Oct 31 05:13:47.211558 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:47.211274 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fffb7c60-8cb2-40f0-903c-9b98c62a1741-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fffb7c60-8cb2-40f0-903c-9b98c62a1741-workload-certs\") pod \"fffb7c60-8cb2-40f0-903c-9b98c62a1741\" (UID: \"fffb7c60-8cb2-40f0-903c-9b98c62a1741\") " Oct 31 05:13:47.211558 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:47.211344 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fffb7c60-8cb2-40f0-903c-9b98c62a1741-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fffb7c60-8cb2-40f0-903c-9b98c62a1741-istio-podinfo\") pod \"fffb7c60-8cb2-40f0-903c-9b98c62a1741\" (UID: \"fffb7c60-8cb2-40f0-903c-9b98c62a1741\") " Oct 31 05:13:47.211558 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:47.211388 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fffb7c60-8cb2-40f0-903c-9b98c62a1741-kube-api-access-lbq88\" (UniqueName: \"kubernetes.io/projected/fffb7c60-8cb2-40f0-903c-9b98c62a1741-kube-api-access-lbq88\") pod \"fffb7c60-8cb2-40f0-903c-9b98c62a1741\" (UID: \"fffb7c60-8cb2-40f0-903c-9b98c62a1741\") " Oct 31 05:13:47.211558 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:47.211442 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fffb7c60-8cb2-40f0-903c-9b98c62a1741-istio-token\" (UniqueName: \"kubernetes.io/projected/fffb7c60-8cb2-40f0-903c-9b98c62a1741-istio-token\") pod \"fffb7c60-8cb2-40f0-903c-9b98c62a1741\" (UID: \"fffb7c60-8cb2-40f0-903c-9b98c62a1741\") " Oct 31 05:13:47.211558 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:47.211511 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fffb7c60-8cb2-40f0-903c-9b98c62a1741-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fffb7c60-8cb2-40f0-903c-9b98c62a1741-workload-socket\") pod \"fffb7c60-8cb2-40f0-903c-9b98c62a1741\" (UID: \"fffb7c60-8cb2-40f0-903c-9b98c62a1741\") " Oct 31 05:13:47.211927 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:47.211617 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fffb7c60-8cb2-40f0-903c-9b98c62a1741-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fffb7c60-8cb2-40f0-903c-9b98c62a1741-credential-socket\") pod \"fffb7c60-8cb2-40f0-903c-9b98c62a1741\" (UID: \"fffb7c60-8cb2-40f0-903c-9b98c62a1741\") " Oct 31 05:13:47.211927 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:47.211783 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fffb7c60-8cb2-40f0-903c-9b98c62a1741-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fffb7c60-8cb2-40f0-903c-9b98c62a1741-istiod-ca-cert\") pod \"fffb7c60-8cb2-40f0-903c-9b98c62a1741\" (UID: \"fffb7c60-8cb2-40f0-903c-9b98c62a1741\") " Oct 31 05:13:47.211927 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:47.211860 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fffb7c60-8cb2-40f0-903c-9b98c62a1741-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fffb7c60-8cb2-40f0-903c-9b98c62a1741-istio-envoy\") pod \"fffb7c60-8cb2-40f0-903c-9b98c62a1741\" (UID: \"fffb7c60-8cb2-40f0-903c-9b98c62a1741\") " Oct 31 05:13:47.213453 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:47.212889 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fffb7c60-8cb2-40f0-903c-9b98c62a1741-credential-socket" pod "fffb7c60-8cb2-40f0-903c-9b98c62a1741" (UID: "fffb7c60-8cb2-40f0-903c-9b98c62a1741"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:13:47.213453 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:47.213044 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fffb7c60-8cb2-40f0-903c-9b98c62a1741-istio-data" pod "fffb7c60-8cb2-40f0-903c-9b98c62a1741" (UID: "fffb7c60-8cb2-40f0-903c-9b98c62a1741"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:13:47.213453 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:47.213109 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fffb7c60-8cb2-40f0-903c-9b98c62a1741-workload-socket" pod "fffb7c60-8cb2-40f0-903c-9b98c62a1741" (UID: "fffb7c60-8cb2-40f0-903c-9b98c62a1741"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:13:47.213453 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:47.213167 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fffb7c60-8cb2-40f0-903c-9b98c62a1741-istiod-ca-cert" pod "fffb7c60-8cb2-40f0-903c-9b98c62a1741" (UID: "fffb7c60-8cb2-40f0-903c-9b98c62a1741"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:13:47.213453 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:47.213421 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fffb7c60-8cb2-40f0-903c-9b98c62a1741-workload-certs" pod "fffb7c60-8cb2-40f0-903c-9b98c62a1741" (UID: "fffb7c60-8cb2-40f0-903c-9b98c62a1741"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:13:47.217399 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:47.216035 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fffb7c60-8cb2-40f0-903c-9b98c62a1741-kube-api-access-lbq88" pod "fffb7c60-8cb2-40f0-903c-9b98c62a1741" (UID: "fffb7c60-8cb2-40f0-903c-9b98c62a1741"). InnerVolumeSpecName "kube-api-access-lbq88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:13:47.217571 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:47.217494 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fffb7c60-8cb2-40f0-903c-9b98c62a1741-istio-token" pod "fffb7c60-8cb2-40f0-903c-9b98c62a1741" (UID: "fffb7c60-8cb2-40f0-903c-9b98c62a1741"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:13:47.223499 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:47.222215 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fffb7c60-8cb2-40f0-903c-9b98c62a1741-istio-envoy" pod "fffb7c60-8cb2-40f0-903c-9b98c62a1741" (UID: "fffb7c60-8cb2-40f0-903c-9b98c62a1741"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:13:47.223918 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:47.223894 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fffb7c60-8cb2-40f0-903c-9b98c62a1741-istio-podinfo" pod "fffb7c60-8cb2-40f0-903c-9b98c62a1741" (UID: "fffb7c60-8cb2-40f0-903c-9b98c62a1741"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:13:47.324208 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:47.324099 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fffb7c60-8cb2-40f0-903c-9b98c62a1741-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:13:47.324208 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:47.324143 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fffb7c60-8cb2-40f0-903c-9b98c62a1741-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:13:47.324208 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:47.324159 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fffb7c60-8cb2-40f0-903c-9b98c62a1741-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:13:47.324208 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:47.324173 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbq88\" (UniqueName: \"kubernetes.io/projected/fffb7c60-8cb2-40f0-903c-9b98c62a1741-kube-api-access-lbq88\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:13:47.324208 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:47.324189 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fffb7c60-8cb2-40f0-903c-9b98c62a1741-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:13:47.324208 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:47.324203 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fffb7c60-8cb2-40f0-903c-9b98c62a1741-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:13:47.324815 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:47.324217 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fffb7c60-8cb2-40f0-903c-9b98c62a1741-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:13:47.324815 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:47.324230 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fffb7c60-8cb2-40f0-903c-9b98c62a1741-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:13:47.324815 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:47.324243 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fffb7c60-8cb2-40f0-903c-9b98c62a1741-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:13:47.813662 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:47.813620 5019 generic.go:356] "Generic (PLEG): container finished" podID="fffb7c60-8cb2-40f0-903c-9b98c62a1741" containerID="d13c99ec9cfcc52db329243fc9f3e286ec2b59763c5037648baa5571038be0f6" exitCode=0 Oct 31 05:13:47.813890 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:47.813694 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r" event={"ID":"fffb7c60-8cb2-40f0-903c-9b98c62a1741","Type":"ContainerDied","Data":"d13c99ec9cfcc52db329243fc9f3e286ec2b59763c5037648baa5571038be0f6"} Oct 31 05:13:47.813890 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:47.813733 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r" Oct 31 05:13:47.813890 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:47.813764 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r" event={"ID":"fffb7c60-8cb2-40f0-903c-9b98c62a1741","Type":"ContainerDied","Data":"86e8307da6c3ff5799de89d143615d2f97529769b6b70660bacf8dd77f009d8d"} Oct 31 05:13:47.813890 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:47.813789 5019 scope.go:122] "RemoveContainer" containerID="d13c99ec9cfcc52db329243fc9f3e286ec2b59763c5037648baa5571038be0f6" Oct 31 05:13:47.827958 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:47.827929 5019 scope.go:122] "RemoveContainer" containerID="d13c99ec9cfcc52db329243fc9f3e286ec2b59763c5037648baa5571038be0f6" Oct 31 05:13:47.828414 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:13:47.828353 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d13c99ec9cfcc52db329243fc9f3e286ec2b59763c5037648baa5571038be0f6\": container with ID starting with d13c99ec9cfcc52db329243fc9f3e286ec2b59763c5037648baa5571038be0f6 not found: ID does not exist" containerID="d13c99ec9cfcc52db329243fc9f3e286ec2b59763c5037648baa5571038be0f6" Oct 31 05:13:47.828414 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:47.828386 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d13c99ec9cfcc52db329243fc9f3e286ec2b59763c5037648baa5571038be0f6"} err="failed to get container status \"d13c99ec9cfcc52db329243fc9f3e286ec2b59763c5037648baa5571038be0f6\": rpc error: code = NotFound desc = could not find container \"d13c99ec9cfcc52db329243fc9f3e286ec2b59763c5037648baa5571038be0f6\": container with ID starting with d13c99ec9cfcc52db329243fc9f3e286ec2b59763c5037648baa5571038be0f6 not found: ID does not exist" Oct 31 05:13:47.868579 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:47.868535 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r"] Oct 31 05:13:47.876297 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:47.876239 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-q7m9r"] Oct 31 05:13:49.414528 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:49.414490 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fffb7c60-8cb2-40f0-903c-9b98c62a1741" path="/var/lib/kubelet/pods/fffb7c60-8cb2-40f0-903c-9b98c62a1741/volumes" Oct 31 05:13:52.394460 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:13:52.394417 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:13:55.330359 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.328226 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh"] Oct 31 05:13:55.377759 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.377643 5019 cpu_manager.go:414] "containerMap: removing container" podUID="fffb7c60-8cb2-40f0-903c-9b98c62a1741" containerName="istio-proxy" Oct 31 05:13:55.378033 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.378011 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fffb7c60-8cb2-40f0-903c-9b98c62a1741" containerName="istio-proxy" Oct 31 05:13:55.378195 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.378171 5019 cpu_manager.go:414] "containerMap: removing container" podUID="dccd15a9-47f5-4bd1-b21b-9d5ca8b05845" containerName="main" Oct 31 05:13:55.378312 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.378297 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dccd15a9-47f5-4bd1-b21b-9d5ca8b05845" containerName="main" Oct 31 05:13:55.378459 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.378445 5019 cpu_manager.go:414] "containerMap: removing container" podUID="2bea695c-448e-42b8-bc47-d381c431e1c8" containerName="revoke-keys" Oct 31 05:13:55.378569 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.378547 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bea695c-448e-42b8-bc47-d381c431e1c8" containerName="revoke-keys" Oct 31 05:13:55.378761 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.378731 5019 cpu_manager.go:414] "containerMap: removing container" podUID="406761c9-51b9-4105-88ce-bd0fac558eb1" containerName="payload-pre-processing" Oct 31 05:13:55.378875 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.378861 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="406761c9-51b9-4105-88ce-bd0fac558eb1" containerName="payload-pre-processing" Oct 31 05:13:55.379419 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.379400 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="406761c9-51b9-4105-88ce-bd0fac558eb1" containerName="payload-pre-processing" Oct 31 05:13:55.379578 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.379565 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="fffb7c60-8cb2-40f0-903c-9b98c62a1741" containerName="istio-proxy" Oct 31 05:13:55.379705 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.379692 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="dccd15a9-47f5-4bd1-b21b-9d5ca8b05845" containerName="main" Oct 31 05:13:55.379813 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.379795 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bea695c-448e-42b8-bc47-d381c431e1c8" containerName="revoke-keys" Oct 31 05:13:55.386088 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.386053 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c"] Oct 31 05:13:55.393464 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.393428 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh" Oct 31 05:13:55.395620 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.395585 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh"] Oct 31 05:13:55.395786 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.395640 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c"] Oct 31 05:13:55.395786 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.395733 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c" Oct 31 05:13:55.448635 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.448597 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b66ab368-8869-4a6d-a83e-e40d92df0042-istio-token\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh\" (UID: \"b66ab368-8869-4a6d-a83e-e40d92df0042\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh" Oct 31 05:13:55.448830 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.448655 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b66ab368-8869-4a6d-a83e-e40d92df0042-istio-data\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh\" (UID: \"b66ab368-8869-4a6d-a83e-e40d92df0042\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh" Oct 31 05:13:55.448830 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.448696 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-credential-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c\" (UID: \"93d2b6ca-e93d-4dd1-805a-b8985e4da4b9\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c" Oct 31 05:13:55.448830 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.448724 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-istio-envoy\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c\" (UID: \"93d2b6ca-e93d-4dd1-805a-b8985e4da4b9\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c" Oct 31 05:13:55.449034 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.448855 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwhvq\" (UniqueName: \"kubernetes.io/projected/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-kube-api-access-mwhvq\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c\" (UID: \"93d2b6ca-e93d-4dd1-805a-b8985e4da4b9\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c" Oct 31 05:13:55.449034 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.448950 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b66ab368-8869-4a6d-a83e-e40d92df0042-istiod-ca-cert\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh\" (UID: \"b66ab368-8869-4a6d-a83e-e40d92df0042\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh" Oct 31 05:13:55.449127 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.449035 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-workload-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c\" (UID: \"93d2b6ca-e93d-4dd1-805a-b8985e4da4b9\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c" Oct 31 05:13:55.449127 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.449081 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-istio-token\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c\" (UID: \"93d2b6ca-e93d-4dd1-805a-b8985e4da4b9\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c" Oct 31 05:13:55.449226 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.449202 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-istio-podinfo\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c\" (UID: \"93d2b6ca-e93d-4dd1-805a-b8985e4da4b9\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c" Oct 31 05:13:55.449285 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.449244 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b66ab368-8869-4a6d-a83e-e40d92df0042-workload-certs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh\" (UID: \"b66ab368-8869-4a6d-a83e-e40d92df0042\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh" Oct 31 05:13:55.449389 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.449369 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b66ab368-8869-4a6d-a83e-e40d92df0042-istio-podinfo\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh\" (UID: \"b66ab368-8869-4a6d-a83e-e40d92df0042\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh" Oct 31 05:13:55.449467 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.449443 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znt8d\" (UniqueName: \"kubernetes.io/projected/b66ab368-8869-4a6d-a83e-e40d92df0042-kube-api-access-znt8d\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh\" (UID: \"b66ab368-8869-4a6d-a83e-e40d92df0042\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh" Oct 31 05:13:55.449631 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.449607 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-workload-certs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c\" (UID: \"93d2b6ca-e93d-4dd1-805a-b8985e4da4b9\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c" Oct 31 05:13:55.449720 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.449660 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b66ab368-8869-4a6d-a83e-e40d92df0042-workload-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh\" (UID: \"b66ab368-8869-4a6d-a83e-e40d92df0042\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh" Oct 31 05:13:55.449791 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.449742 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b66ab368-8869-4a6d-a83e-e40d92df0042-credential-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh\" (UID: \"b66ab368-8869-4a6d-a83e-e40d92df0042\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh" Oct 31 05:13:55.449837 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.449799 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-istiod-ca-cert\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c\" (UID: \"93d2b6ca-e93d-4dd1-805a-b8985e4da4b9\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c" Oct 31 05:13:55.449908 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.449888 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-istio-data\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c\" (UID: \"93d2b6ca-e93d-4dd1-805a-b8985e4da4b9\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c" Oct 31 05:13:55.449975 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.449959 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b66ab368-8869-4a6d-a83e-e40d92df0042-istio-envoy\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh\" (UID: \"b66ab368-8869-4a6d-a83e-e40d92df0042\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh" Oct 31 05:13:55.471898 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.469016 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-09a20a-openshift-default-dockercfg-hf9qb\"" Oct 31 05:13:55.567105 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.567049 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-workload-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c\" (UID: \"93d2b6ca-e93d-4dd1-805a-b8985e4da4b9\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c" Oct 31 05:13:55.567400 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.567366 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-istio-token\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c\" (UID: \"93d2b6ca-e93d-4dd1-805a-b8985e4da4b9\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c" Oct 31 05:13:55.567633 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.567606 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-istio-podinfo\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c\" (UID: \"93d2b6ca-e93d-4dd1-805a-b8985e4da4b9\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c" Oct 31 05:13:55.567774 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.567758 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b66ab368-8869-4a6d-a83e-e40d92df0042-workload-certs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh\" (UID: \"b66ab368-8869-4a6d-a83e-e40d92df0042\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh" Oct 31 05:13:55.567923 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.567909 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b66ab368-8869-4a6d-a83e-e40d92df0042-istio-podinfo\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh\" (UID: \"b66ab368-8869-4a6d-a83e-e40d92df0042\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh" Oct 31 05:13:55.568054 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.568040 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znt8d\" (UniqueName: \"kubernetes.io/projected/b66ab368-8869-4a6d-a83e-e40d92df0042-kube-api-access-znt8d\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh\" (UID: \"b66ab368-8869-4a6d-a83e-e40d92df0042\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh" Oct 31 05:13:55.568209 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.568195 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-workload-certs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c\" (UID: \"93d2b6ca-e93d-4dd1-805a-b8985e4da4b9\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c" Oct 31 05:13:55.568323 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.568309 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b66ab368-8869-4a6d-a83e-e40d92df0042-workload-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh\" (UID: \"b66ab368-8869-4a6d-a83e-e40d92df0042\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh" Oct 31 05:13:55.568492 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.568476 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b66ab368-8869-4a6d-a83e-e40d92df0042-credential-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh\" (UID: \"b66ab368-8869-4a6d-a83e-e40d92df0042\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh" Oct 31 05:13:55.568616 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.568600 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-istiod-ca-cert\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c\" (UID: \"93d2b6ca-e93d-4dd1-805a-b8985e4da4b9\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c" Oct 31 05:13:55.568747 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.568731 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-istio-data\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c\" (UID: \"93d2b6ca-e93d-4dd1-805a-b8985e4da4b9\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c" Oct 31 05:13:55.568874 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.568858 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b66ab368-8869-4a6d-a83e-e40d92df0042-istio-envoy\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh\" (UID: \"b66ab368-8869-4a6d-a83e-e40d92df0042\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh" Oct 31 05:13:55.568976 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.568963 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b66ab368-8869-4a6d-a83e-e40d92df0042-istio-token\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh\" (UID: \"b66ab368-8869-4a6d-a83e-e40d92df0042\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh" Oct 31 05:13:55.569104 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.569089 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b66ab368-8869-4a6d-a83e-e40d92df0042-istio-data\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh\" (UID: \"b66ab368-8869-4a6d-a83e-e40d92df0042\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh" Oct 31 05:13:55.569237 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.569219 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-credential-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c\" (UID: \"93d2b6ca-e93d-4dd1-805a-b8985e4da4b9\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c" Oct 31 05:13:55.569375 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.569357 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-istio-envoy\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c\" (UID: \"93d2b6ca-e93d-4dd1-805a-b8985e4da4b9\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c" Oct 31 05:13:55.581483 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.573435 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwhvq\" (UniqueName: \"kubernetes.io/projected/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-kube-api-access-mwhvq\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c\" (UID: \"93d2b6ca-e93d-4dd1-805a-b8985e4da4b9\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c" Oct 31 05:13:55.600362 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.581769 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b66ab368-8869-4a6d-a83e-e40d92df0042-istiod-ca-cert\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh\" (UID: \"b66ab368-8869-4a6d-a83e-e40d92df0042\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh" Oct 31 05:13:55.613410 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.591489 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b66ab368-8869-4a6d-a83e-e40d92df0042-istiod-ca-cert\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh\" (UID: \"b66ab368-8869-4a6d-a83e-e40d92df0042\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh" Oct 31 05:13:55.613410 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.591914 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-workload-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c\" (UID: \"93d2b6ca-e93d-4dd1-805a-b8985e4da4b9\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c" Oct 31 05:13:55.613410 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.593564 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b66ab368-8869-4a6d-a83e-e40d92df0042-workload-certs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh\" (UID: \"b66ab368-8869-4a6d-a83e-e40d92df0042\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh" Oct 31 05:13:55.613410 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.595468 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-istio-podinfo\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c\" (UID: \"93d2b6ca-e93d-4dd1-805a-b8985e4da4b9\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c" Oct 31 05:13:55.613410 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.596361 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-istio-data\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c\" (UID: \"93d2b6ca-e93d-4dd1-805a-b8985e4da4b9\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c" Oct 31 05:13:55.613410 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.596598 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b66ab368-8869-4a6d-a83e-e40d92df0042-credential-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh\" (UID: \"b66ab368-8869-4a6d-a83e-e40d92df0042\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh" Oct 31 05:13:55.613410 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.597392 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-istiod-ca-cert\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c\" (UID: \"93d2b6ca-e93d-4dd1-805a-b8985e4da4b9\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c" Oct 31 05:13:55.613410 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.597665 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b66ab368-8869-4a6d-a83e-e40d92df0042-workload-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh\" (UID: \"b66ab368-8869-4a6d-a83e-e40d92df0042\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh" Oct 31 05:13:55.613410 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.597687 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b66ab368-8869-4a6d-a83e-e40d92df0042-istio-data\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh\" (UID: \"b66ab368-8869-4a6d-a83e-e40d92df0042\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh" Oct 31 05:13:55.613410 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.598163 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-workload-certs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c\" (UID: \"93d2b6ca-e93d-4dd1-805a-b8985e4da4b9\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c" Oct 31 05:13:55.613410 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.602384 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-credential-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c\" (UID: \"93d2b6ca-e93d-4dd1-805a-b8985e4da4b9\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c" Oct 31 05:13:55.613410 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.602823 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b66ab368-8869-4a6d-a83e-e40d92df0042-istio-podinfo\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh\" (UID: \"b66ab368-8869-4a6d-a83e-e40d92df0042\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh" Oct 31 05:13:55.613410 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.607516 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-istio-envoy\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c\" (UID: \"93d2b6ca-e93d-4dd1-805a-b8985e4da4b9\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c" Oct 31 05:13:55.613410 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.607516 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b66ab368-8869-4a6d-a83e-e40d92df0042-istio-envoy\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh\" (UID: \"b66ab368-8869-4a6d-a83e-e40d92df0042\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh" Oct 31 05:13:55.628993 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.623024 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-istio-token\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c\" (UID: \"93d2b6ca-e93d-4dd1-805a-b8985e4da4b9\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c" Oct 31 05:13:55.628993 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.624990 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwhvq\" (UniqueName: \"kubernetes.io/projected/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-kube-api-access-mwhvq\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c\" (UID: \"93d2b6ca-e93d-4dd1-805a-b8985e4da4b9\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c" Oct 31 05:13:55.628993 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.626826 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b66ab368-8869-4a6d-a83e-e40d92df0042-istio-token\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh\" (UID: \"b66ab368-8869-4a6d-a83e-e40d92df0042\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh" Oct 31 05:13:55.629179 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.629050 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znt8d\" (UniqueName: \"kubernetes.io/projected/b66ab368-8869-4a6d-a83e-e40d92df0042-kube-api-access-znt8d\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh\" (UID: \"b66ab368-8869-4a6d-a83e-e40d92df0042\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh" Oct 31 05:13:55.724526 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.724475 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh" Oct 31 05:13:55.739814 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:55.735974 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c" Oct 31 05:13:56.080999 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:13:56.080965 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93d2b6ca_e93d_4dd1_805a_b8985e4da4b9.slice/crio-b8f3bbd92f2db21f64fc7355d7bdd389715eff1074d8137a28715e640f58a35e WatchSource:0}: Error finding container b8f3bbd92f2db21f64fc7355d7bdd389715eff1074d8137a28715e640f58a35e: Status 404 returned error can't find the container with id b8f3bbd92f2db21f64fc7355d7bdd389715eff1074d8137a28715e640f58a35e Oct 31 05:13:56.081887 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:56.081860 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c"] Oct 31 05:13:56.084191 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:56.084096 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:13:56.084378 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:56.084185 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:13:56.084378 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:56.084233 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:13:56.088365 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:56.088318 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh"] Oct 31 05:13:56.097516 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:56.097468 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:13:56.097637 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:56.097570 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:13:56.097637 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:56.097610 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:13:56.906357 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:56.905618 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh" event={"ID":"b66ab368-8869-4a6d-a83e-e40d92df0042","Type":"ContainerStarted","Data":"1300428e307e65aa890ebd3bb9be80f2ad8d6d86d532b6a0e0e6f4de39a90c9f"} Oct 31 05:13:56.906357 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:56.905688 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh" event={"ID":"b66ab368-8869-4a6d-a83e-e40d92df0042","Type":"ContainerStarted","Data":"1f03f988015f3dc5a01b3ae86bc6ce7970318ed6b72e54004a782016a6eda0a6"} Oct 31 05:13:56.909370 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:56.909254 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c" event={"ID":"93d2b6ca-e93d-4dd1-805a-b8985e4da4b9","Type":"ContainerStarted","Data":"6407221f951a57026bec4e3553ed3affef2bb18c58a079151e2a836554120df4"} Oct 31 05:13:56.926368 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:56.909323 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c" event={"ID":"93d2b6ca-e93d-4dd1-805a-b8985e4da4b9","Type":"ContainerStarted","Data":"b8f3bbd92f2db21f64fc7355d7bdd389715eff1074d8137a28715e640f58a35e"} Oct 31 05:13:56.992440 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:56.992373 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh" podStartSLOduration=1.992352542 podStartE2EDuration="1.992352542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:13:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:13:56.966786359 +0000 UTC m=+4191.723124399" watchObservedRunningTime="2026-10-31 05:13:56.992352542 +0000 UTC m=+4191.748690578" Oct 31 05:13:57.031497 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:57.031423 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c" podStartSLOduration=2.031399122 podStartE2EDuration="2.031399122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:13:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:13:57.027843335 +0000 UTC m=+4191.784181376" watchObservedRunningTime="2026-10-31 05:13:57.031399122 +0000 UTC m=+4191.787737163" Oct 31 05:13:57.727431 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:57.727387 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh" Oct 31 05:13:57.731490 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:57.731454 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.188:15021/healthz/ready\": dial tcp 10.132.1.188:15021: connect: connection refused" start-of-body= Oct 31 05:13:57.731811 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:57.731778 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh" podUID="b66ab368-8869-4a6d-a83e-e40d92df0042" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.188:15021/healthz/ready\": dial tcp 10.132.1.188:15021: connect: connection refused" Oct 31 05:13:57.748792 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:57.748754 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c" Oct 31 05:13:57.749506 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:57.749478 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.189:15021/healthz/ready\": dial tcp 10.132.1.189:15021: connect: connection refused" start-of-body= Oct 31 05:13:57.752432 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:57.752393 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c" podUID="93d2b6ca-e93d-4dd1-805a-b8985e4da4b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.189:15021/healthz/ready\": dial tcp 10.132.1.189:15021: connect: connection refused" Oct 31 05:13:58.732135 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:58.732093 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.188:15021/healthz/ready\": dial tcp 10.132.1.188:15021: connect: connection refused" start-of-body= Oct 31 05:13:58.734238 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:58.734200 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh" podUID="b66ab368-8869-4a6d-a83e-e40d92df0042" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.188:15021/healthz/ready\": dial tcp 10.132.1.188:15021: connect: connection refused" Oct 31 05:13:58.740525 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:58.740244 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.189:15021/healthz/ready\": dial tcp 10.132.1.189:15021: connect: connection refused" start-of-body= Oct 31 05:13:58.740525 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:58.740310 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c" podUID="93d2b6ca-e93d-4dd1-805a-b8985e4da4b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.189:15021/healthz/ready\": dial tcp 10.132.1.189:15021: connect: connection refused" Oct 31 05:13:59.725979 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:59.725923 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.188:15021/healthz/ready\": dial tcp 10.132.1.188:15021: connect: connection refused" start-of-body= Oct 31 05:13:59.726183 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:59.725987 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh" podUID="b66ab368-8869-4a6d-a83e-e40d92df0042" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.188:15021/healthz/ready\": dial tcp 10.132.1.188:15021: connect: connection refused" Oct 31 05:13:59.740646 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:59.740604 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.189:15021/healthz/ready\": dial tcp 10.132.1.189:15021: connect: connection refused" start-of-body= Oct 31 05:13:59.741028 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:13:59.740663 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c" podUID="93d2b6ca-e93d-4dd1-805a-b8985e4da4b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.189:15021/healthz/ready\": dial tcp 10.132.1.189:15021: connect: connection refused" Oct 31 05:14:00.726497 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:00.725725 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.188:15021/healthz/ready\": dial tcp 10.132.1.188:15021: connect: connection refused" start-of-body= Oct 31 05:14:00.726497 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:00.725787 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh" podUID="b66ab368-8869-4a6d-a83e-e40d92df0042" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.188:15021/healthz/ready\": dial tcp 10.132.1.188:15021: connect: connection refused" Oct 31 05:14:00.741372 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:00.740707 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.189:15021/healthz/ready\": dial tcp 10.132.1.189:15021: connect: connection refused" start-of-body= Oct 31 05:14:00.741372 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:00.740763 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c" podUID="93d2b6ca-e93d-4dd1-805a-b8985e4da4b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.189:15021/healthz/ready\": dial tcp 10.132.1.189:15021: connect: connection refused" Oct 31 05:14:01.729355 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:01.728787 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.188:15021/healthz/ready\": dial tcp 10.132.1.188:15021: connect: connection refused" start-of-body= Oct 31 05:14:01.729355 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:01.728852 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh" podUID="b66ab368-8869-4a6d-a83e-e40d92df0042" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.188:15021/healthz/ready\": dial tcp 10.132.1.188:15021: connect: connection refused" Oct 31 05:14:01.744497 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:01.741825 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.189:15021/healthz/ready\": dial tcp 10.132.1.189:15021: connect: connection refused" start-of-body= Oct 31 05:14:01.744497 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:01.741893 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c" podUID="93d2b6ca-e93d-4dd1-805a-b8985e4da4b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.189:15021/healthz/ready\": dial tcp 10.132.1.189:15021: connect: connection refused" Oct 31 05:14:02.726102 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:02.726045 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.188:15021/healthz/ready\": dial tcp 10.132.1.188:15021: connect: connection refused" start-of-body= Oct 31 05:14:02.726385 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:02.726108 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh" podUID="b66ab368-8869-4a6d-a83e-e40d92df0042" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.188:15021/healthz/ready\": dial tcp 10.132.1.188:15021: connect: connection refused" Oct 31 05:14:02.740940 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:02.740888 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.189:15021/healthz/ready\": dial tcp 10.132.1.189:15021: connect: connection refused" start-of-body= Oct 31 05:14:02.741160 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:02.740959 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c" podUID="93d2b6ca-e93d-4dd1-805a-b8985e4da4b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.189:15021/healthz/ready\": dial tcp 10.132.1.189:15021: connect: connection refused" Oct 31 05:14:03.886900 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:03.886854 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c" Oct 31 05:14:03.888244 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:03.887931 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c" Oct 31 05:14:03.889192 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:03.889153 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c" Oct 31 05:14:04.019356 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:04.018629 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh" Oct 31 05:14:04.021663 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:04.019776 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh" Oct 31 05:14:04.021663 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:04.021140 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh" Oct 31 05:14:08.043908 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:08.043867 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f742f79-7psld"] Oct 31 05:14:08.053019 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:08.052985 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f742f79-7psld"] Oct 31 05:14:08.716592 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:08.716552 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-ded435-5cbbf45d4c-j9btt"] Oct 31 05:14:08.744474 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:08.744433 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-ded435-5787fc7f9-c9dqc"] Oct 31 05:14:09.416022 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:09.415982 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2171746-75af-4bba-bf0a-1f065df42461" path="/var/lib/kubelet/pods/c2171746-75af-4bba-bf0a-1f065df42461/volumes" Oct 31 05:14:09.625518 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:09.625488 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s88kp_3945ff65-e877-4372-9334-3968c68b7295/console-operator/2.log" Oct 31 05:14:09.629106 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:09.629076 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s88kp_3945ff65-e877-4372-9334-3968c68b7295/console-operator/2.log" Oct 31 05:14:13.610884 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:13.610845 5019 scope.go:122] "RemoveContainer" containerID="03a34729a0c00280813365d60408ec371bf4522d4294ed11f581f221a4eeff67" Oct 31 05:14:13.717158 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:13.717109 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-ded435-5cbbf45d4c-j9btt" podUID="91ac1809-b6e5-4c0e-a324-cfdc8c088e74" containerName="payload-processing" containerID="cri-o://91cb972e9136e8af3bb377ede3771c827c383ec6692bae6165ed723119b92755" gracePeriod=25 Oct 31 05:14:13.745068 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:13.745020 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ded435-5787fc7f9-c9dqc" podUID="74e04a61-5277-4180-bc46-68af7640b25e" containerName="payload-pre-processing" containerID="cri-o://d664a380cf9fdd7c09c421f5e2960a6b2db4cc531109ee9946434ab68acc60f0" gracePeriod=25 Oct 31 05:14:14.594400 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:14.594306 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c"] Oct 31 05:14:14.594994 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:14.594947 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c" podUID="93d2b6ca-e93d-4dd1-805a-b8985e4da4b9" containerName="istio-proxy" containerID="cri-o://6407221f951a57026bec4e3553ed3affef2bb18c58a079151e2a836554120df4" gracePeriod=30 Oct 31 05:14:14.603389 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:14.603295 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh"] Oct 31 05:14:14.603679 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:14.603634 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh" podUID="b66ab368-8869-4a6d-a83e-e40d92df0042" containerName="istio-proxy" containerID="cri-o://1300428e307e65aa890ebd3bb9be80f2ad8d6d86d532b6a0e0e6f4de39a90c9f" gracePeriod=30 Oct 31 05:14:18.710599 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:18.710550 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ded435-5787fc7f9-c9dqc" podUID="74e04a61-5277-4180-bc46-68af7640b25e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:14:18.710599 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:18.710571 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-ded435-5cbbf45d4c-j9btt" podUID="91ac1809-b6e5-4c0e-a324-cfdc8c088e74" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:14:18.890730 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:18.890689 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 05:14:18.890911 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:18.890764 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c" podUID="93d2b6ca-e93d-4dd1-805a-b8985e4da4b9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 05:14:19.022986 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:19.022944 5019 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 05:14:19.023182 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:19.023007 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh" podUID="b66ab368-8869-4a6d-a83e-e40d92df0042" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 05:14:19.917233 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:19.917200 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh" Oct 31 05:14:19.966474 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:19.966444 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c" Oct 31 05:14:20.024946 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.024904 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b66ab368-8869-4a6d-a83e-e40d92df0042-istio-token\" (UniqueName: \"kubernetes.io/projected/b66ab368-8869-4a6d-a83e-e40d92df0042-istio-token\") pod \"b66ab368-8869-4a6d-a83e-e40d92df0042\" (UID: \"b66ab368-8869-4a6d-a83e-e40d92df0042\") " Oct 31 05:14:20.025173 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.024965 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-credential-socket\") pod \"93d2b6ca-e93d-4dd1-805a-b8985e4da4b9\" (UID: \"93d2b6ca-e93d-4dd1-805a-b8985e4da4b9\") " Oct 31 05:14:20.025173 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.025042 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-istio-podinfo\") pod \"93d2b6ca-e93d-4dd1-805a-b8985e4da4b9\" (UID: \"93d2b6ca-e93d-4dd1-805a-b8985e4da4b9\") " Oct 31 05:14:20.025173 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.025079 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-workload-socket\") pod \"93d2b6ca-e93d-4dd1-805a-b8985e4da4b9\" (UID: \"93d2b6ca-e93d-4dd1-805a-b8985e4da4b9\") " Oct 31 05:14:20.025173 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.025119 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b66ab368-8869-4a6d-a83e-e40d92df0042-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b66ab368-8869-4a6d-a83e-e40d92df0042-istiod-ca-cert\") pod \"b66ab368-8869-4a6d-a83e-e40d92df0042\" (UID: \"b66ab368-8869-4a6d-a83e-e40d92df0042\") " Oct 31 05:14:20.025576 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.025180 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-istiod-ca-cert\") pod \"93d2b6ca-e93d-4dd1-805a-b8985e4da4b9\" (UID: \"93d2b6ca-e93d-4dd1-805a-b8985e4da4b9\") " Oct 31 05:14:20.025576 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.025209 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b66ab368-8869-4a6d-a83e-e40d92df0042-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b66ab368-8869-4a6d-a83e-e40d92df0042-credential-socket\") pod \"b66ab368-8869-4a6d-a83e-e40d92df0042\" (UID: \"b66ab368-8869-4a6d-a83e-e40d92df0042\") " Oct 31 05:14:20.025576 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.025282 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b66ab368-8869-4a6d-a83e-e40d92df0042-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b66ab368-8869-4a6d-a83e-e40d92df0042-workload-socket\") pod \"b66ab368-8869-4a6d-a83e-e40d92df0042\" (UID: \"b66ab368-8869-4a6d-a83e-e40d92df0042\") " Oct 31 05:14:20.025576 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.025314 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-kube-api-access-mwhvq\" (UniqueName: \"kubernetes.io/projected/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-kube-api-access-mwhvq\") pod \"93d2b6ca-e93d-4dd1-805a-b8985e4da4b9\" (UID: \"93d2b6ca-e93d-4dd1-805a-b8985e4da4b9\") " Oct 31 05:14:20.025576 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.025357 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b66ab368-8869-4a6d-a83e-e40d92df0042-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b66ab368-8869-4a6d-a83e-e40d92df0042-workload-certs\") pod \"b66ab368-8869-4a6d-a83e-e40d92df0042\" (UID: \"b66ab368-8869-4a6d-a83e-e40d92df0042\") " Oct 31 05:14:20.025576 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.025470 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-istio-envoy\") pod \"93d2b6ca-e93d-4dd1-805a-b8985e4da4b9\" (UID: \"93d2b6ca-e93d-4dd1-805a-b8985e4da4b9\") " Oct 31 05:14:20.025929 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.025577 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b66ab368-8869-4a6d-a83e-e40d92df0042-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b66ab368-8869-4a6d-a83e-e40d92df0042-istio-podinfo\") pod \"b66ab368-8869-4a6d-a83e-e40d92df0042\" (UID: \"b66ab368-8869-4a6d-a83e-e40d92df0042\") " Oct 31 05:14:20.025929 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.025609 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b66ab368-8869-4a6d-a83e-e40d92df0042-kube-api-access-znt8d\" (UniqueName: \"kubernetes.io/projected/b66ab368-8869-4a6d-a83e-e40d92df0042-kube-api-access-znt8d\") pod \"b66ab368-8869-4a6d-a83e-e40d92df0042\" (UID: \"b66ab368-8869-4a6d-a83e-e40d92df0042\") " Oct 31 05:14:20.025929 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.025638 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-workload-certs\") pod \"93d2b6ca-e93d-4dd1-805a-b8985e4da4b9\" (UID: \"93d2b6ca-e93d-4dd1-805a-b8985e4da4b9\") " Oct 31 05:14:20.025929 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.025663 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b66ab368-8869-4a6d-a83e-e40d92df0042-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b66ab368-8869-4a6d-a83e-e40d92df0042-istio-envoy\") pod \"b66ab368-8869-4a6d-a83e-e40d92df0042\" (UID: \"b66ab368-8869-4a6d-a83e-e40d92df0042\") " Oct 31 05:14:20.025929 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.025711 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-istio-token\" (UniqueName: \"kubernetes.io/projected/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-istio-token\") pod \"93d2b6ca-e93d-4dd1-805a-b8985e4da4b9\" (UID: \"93d2b6ca-e93d-4dd1-805a-b8985e4da4b9\") " Oct 31 05:14:20.025929 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.025819 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b66ab368-8869-4a6d-a83e-e40d92df0042-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b66ab368-8869-4a6d-a83e-e40d92df0042-istio-data\") pod \"b66ab368-8869-4a6d-a83e-e40d92df0042\" (UID: \"b66ab368-8869-4a6d-a83e-e40d92df0042\") " Oct 31 05:14:20.026273 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.025965 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-istio-data\") pod \"93d2b6ca-e93d-4dd1-805a-b8985e4da4b9\" (UID: \"93d2b6ca-e93d-4dd1-805a-b8985e4da4b9\") " Oct 31 05:14:20.026273 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.026257 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-workload-socket" pod "93d2b6ca-e93d-4dd1-805a-b8985e4da4b9" (UID: "93d2b6ca-e93d-4dd1-805a-b8985e4da4b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:14:20.026495 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.026467 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b66ab368-8869-4a6d-a83e-e40d92df0042-credential-socket" pod "b66ab368-8869-4a6d-a83e-e40d92df0042" (UID: "b66ab368-8869-4a6d-a83e-e40d92df0042"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:14:20.026568 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.026503 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-istio-data" pod "93d2b6ca-e93d-4dd1-805a-b8985e4da4b9" (UID: "93d2b6ca-e93d-4dd1-805a-b8985e4da4b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:14:20.031392 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.026836 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b66ab368-8869-4a6d-a83e-e40d92df0042-istiod-ca-cert" pod "b66ab368-8869-4a6d-a83e-e40d92df0042" (UID: "b66ab368-8869-4a6d-a83e-e40d92df0042"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:14:20.031392 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.026846 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b66ab368-8869-4a6d-a83e-e40d92df0042-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:14:20.031392 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.026873 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:14:20.031392 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.026888 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:14:20.031392 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.027019 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b66ab368-8869-4a6d-a83e-e40d92df0042-workload-socket" pod "b66ab368-8869-4a6d-a83e-e40d92df0042" (UID: "b66ab368-8869-4a6d-a83e-e40d92df0042"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:14:20.031392 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.027374 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-istiod-ca-cert" pod "93d2b6ca-e93d-4dd1-805a-b8985e4da4b9" (UID: "93d2b6ca-e93d-4dd1-805a-b8985e4da4b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:14:20.031392 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.029360 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b66ab368-8869-4a6d-a83e-e40d92df0042-istio-token" pod "b66ab368-8869-4a6d-a83e-e40d92df0042" (UID: "b66ab368-8869-4a6d-a83e-e40d92df0042"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:14:20.031392 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.029630 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-credential-socket" pod "93d2b6ca-e93d-4dd1-805a-b8985e4da4b9" (UID: "93d2b6ca-e93d-4dd1-805a-b8985e4da4b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:14:20.031392 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.029741 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-kube-api-access-mwhvq" pod "93d2b6ca-e93d-4dd1-805a-b8985e4da4b9" (UID: "93d2b6ca-e93d-4dd1-805a-b8985e4da4b9"). InnerVolumeSpecName "kube-api-access-mwhvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:14:20.031392 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.029775 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b66ab368-8869-4a6d-a83e-e40d92df0042-kube-api-access-znt8d" pod "b66ab368-8869-4a6d-a83e-e40d92df0042" (UID: "b66ab368-8869-4a6d-a83e-e40d92df0042"). InnerVolumeSpecName "kube-api-access-znt8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:14:20.031392 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.029859 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b66ab368-8869-4a6d-a83e-e40d92df0042-workload-certs" pod "b66ab368-8869-4a6d-a83e-e40d92df0042" (UID: "b66ab368-8869-4a6d-a83e-e40d92df0042"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:14:20.031392 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.029989 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-workload-certs" pod "93d2b6ca-e93d-4dd1-805a-b8985e4da4b9" (UID: "93d2b6ca-e93d-4dd1-805a-b8985e4da4b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:14:20.032268 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.032107 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-istio-podinfo" pod "93d2b6ca-e93d-4dd1-805a-b8985e4da4b9" (UID: "93d2b6ca-e93d-4dd1-805a-b8985e4da4b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:14:20.032268 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.032252 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-istio-envoy" pod "93d2b6ca-e93d-4dd1-805a-b8985e4da4b9" (UID: "93d2b6ca-e93d-4dd1-805a-b8985e4da4b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:14:20.032415 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.032270 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b66ab368-8869-4a6d-a83e-e40d92df0042-istio-envoy" pod "b66ab368-8869-4a6d-a83e-e40d92df0042" (UID: "b66ab368-8869-4a6d-a83e-e40d92df0042"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:14:20.032611 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.032558 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b66ab368-8869-4a6d-a83e-e40d92df0042-istio-data" pod "b66ab368-8869-4a6d-a83e-e40d92df0042" (UID: "b66ab368-8869-4a6d-a83e-e40d92df0042"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:14:20.034281 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.034252 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b66ab368-8869-4a6d-a83e-e40d92df0042-istio-podinfo" pod "b66ab368-8869-4a6d-a83e-e40d92df0042" (UID: "b66ab368-8869-4a6d-a83e-e40d92df0042"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:14:20.034838 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.034810 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-istio-token" pod "93d2b6ca-e93d-4dd1-805a-b8985e4da4b9" (UID: "93d2b6ca-e93d-4dd1-805a-b8985e4da4b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:14:20.128893 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.128857 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b66ab368-8869-4a6d-a83e-e40d92df0042-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:14:20.128893 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.128893 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:14:20.128893 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.128907 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:14:20.129198 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.128920 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b66ab368-8869-4a6d-a83e-e40d92df0042-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:14:20.129198 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.128933 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:14:20.129198 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.128944 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b66ab368-8869-4a6d-a83e-e40d92df0042-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:14:20.129198 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.128956 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwhvq\" (UniqueName: \"kubernetes.io/projected/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-kube-api-access-mwhvq\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:14:20.129198 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.128967 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b66ab368-8869-4a6d-a83e-e40d92df0042-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:14:20.129198 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.128982 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:14:20.129198 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.128993 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b66ab368-8869-4a6d-a83e-e40d92df0042-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:14:20.129198 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.129004 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znt8d\" (UniqueName: \"kubernetes.io/projected/b66ab368-8869-4a6d-a83e-e40d92df0042-kube-api-access-znt8d\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:14:20.129198 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.129015 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:14:20.129198 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.129027 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b66ab368-8869-4a6d-a83e-e40d92df0042-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:14:20.129198 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.129038 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:14:20.129198 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.129049 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b66ab368-8869-4a6d-a83e-e40d92df0042-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:14:20.154869 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.154792 5019 generic.go:356] "Generic (PLEG): container finished" podID="b66ab368-8869-4a6d-a83e-e40d92df0042" containerID="1300428e307e65aa890ebd3bb9be80f2ad8d6d86d532b6a0e0e6f4de39a90c9f" exitCode=0 Oct 31 05:14:20.154869 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.154842 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh" Oct 31 05:14:20.155098 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.154871 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh" event={"ID":"b66ab368-8869-4a6d-a83e-e40d92df0042","Type":"ContainerDied","Data":"1300428e307e65aa890ebd3bb9be80f2ad8d6d86d532b6a0e0e6f4de39a90c9f"} Oct 31 05:14:20.155098 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.154925 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh" event={"ID":"b66ab368-8869-4a6d-a83e-e40d92df0042","Type":"ContainerDied","Data":"1f03f988015f3dc5a01b3ae86bc6ce7970318ed6b72e54004a782016a6eda0a6"} Oct 31 05:14:20.155098 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.154957 5019 scope.go:122] "RemoveContainer" containerID="1300428e307e65aa890ebd3bb9be80f2ad8d6d86d532b6a0e0e6f4de39a90c9f" Oct 31 05:14:20.156670 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.156640 5019 generic.go:356] "Generic (PLEG): container finished" podID="93d2b6ca-e93d-4dd1-805a-b8985e4da4b9" containerID="6407221f951a57026bec4e3553ed3affef2bb18c58a079151e2a836554120df4" exitCode=0 Oct 31 05:14:20.156802 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.156695 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c" Oct 31 05:14:20.156802 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.156731 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c" event={"ID":"93d2b6ca-e93d-4dd1-805a-b8985e4da4b9","Type":"ContainerDied","Data":"6407221f951a57026bec4e3553ed3affef2bb18c58a079151e2a836554120df4"} Oct 31 05:14:20.156802 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.156795 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c" event={"ID":"93d2b6ca-e93d-4dd1-805a-b8985e4da4b9","Type":"ContainerDied","Data":"b8f3bbd92f2db21f64fc7355d7bdd389715eff1074d8137a28715e640f58a35e"} Oct 31 05:14:20.173966 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.173938 5019 scope.go:122] "RemoveContainer" containerID="1300428e307e65aa890ebd3bb9be80f2ad8d6d86d532b6a0e0e6f4de39a90c9f" Oct 31 05:14:20.174377 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:14:20.174326 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1300428e307e65aa890ebd3bb9be80f2ad8d6d86d532b6a0e0e6f4de39a90c9f\": container with ID starting with 1300428e307e65aa890ebd3bb9be80f2ad8d6d86d532b6a0e0e6f4de39a90c9f not found: ID does not exist" containerID="1300428e307e65aa890ebd3bb9be80f2ad8d6d86d532b6a0e0e6f4de39a90c9f" Oct 31 05:14:20.174481 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.174384 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1300428e307e65aa890ebd3bb9be80f2ad8d6d86d532b6a0e0e6f4de39a90c9f"} err="failed to get container status \"1300428e307e65aa890ebd3bb9be80f2ad8d6d86d532b6a0e0e6f4de39a90c9f\": rpc error: code = NotFound desc = could not find container \"1300428e307e65aa890ebd3bb9be80f2ad8d6d86d532b6a0e0e6f4de39a90c9f\": container with ID starting with 1300428e307e65aa890ebd3bb9be80f2ad8d6d86d532b6a0e0e6f4de39a90c9f not found: ID does not exist" Oct 31 05:14:20.174481 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.174411 5019 scope.go:122] "RemoveContainer" containerID="6407221f951a57026bec4e3553ed3affef2bb18c58a079151e2a836554120df4" Oct 31 05:14:20.188252 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.188226 5019 scope.go:122] "RemoveContainer" containerID="6407221f951a57026bec4e3553ed3affef2bb18c58a079151e2a836554120df4" Oct 31 05:14:20.188607 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:14:20.188579 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6407221f951a57026bec4e3553ed3affef2bb18c58a079151e2a836554120df4\": container with ID starting with 6407221f951a57026bec4e3553ed3affef2bb18c58a079151e2a836554120df4 not found: ID does not exist" containerID="6407221f951a57026bec4e3553ed3affef2bb18c58a079151e2a836554120df4" Oct 31 05:14:20.188711 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.188619 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6407221f951a57026bec4e3553ed3affef2bb18c58a079151e2a836554120df4"} err="failed to get container status \"6407221f951a57026bec4e3553ed3affef2bb18c58a079151e2a836554120df4\": rpc error: code = NotFound desc = could not find container \"6407221f951a57026bec4e3553ed3affef2bb18c58a079151e2a836554120df4\": container with ID starting with 6407221f951a57026bec4e3553ed3affef2bb18c58a079151e2a836554120df4 not found: ID does not exist" Oct 31 05:14:20.226281 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.226250 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh"] Oct 31 05:14:20.232506 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.232480 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-qjghh"] Oct 31 05:14:20.266691 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.266653 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c"] Oct 31 05:14:20.271752 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:20.271723 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-sll2c"] Oct 31 05:14:21.408188 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:21.408154 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93d2b6ca-e93d-4dd1-805a-b8985e4da4b9" path="/var/lib/kubelet/pods/93d2b6ca-e93d-4dd1-805a-b8985e4da4b9/volumes" Oct 31 05:14:21.408750 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:21.408731 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b66ab368-8869-4a6d-a83e-e40d92df0042" path="/var/lib/kubelet/pods/b66ab368-8869-4a6d-a83e-e40d92df0042/volumes" Oct 31 05:14:22.393390 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:14:22.393345 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:14:28.710677 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:28.710627 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-ded435-5cbbf45d4c-j9btt" podUID="91ac1809-b6e5-4c0e-a324-cfdc8c088e74" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:14:28.711244 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:28.710646 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ded435-5787fc7f9-c9dqc" podUID="74e04a61-5277-4180-bc46-68af7640b25e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:14:34.041689 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:34.041658 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-ded435-5cbbf45d4c-j9btt" Oct 31 05:14:34.072713 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:34.072681 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ded435-5787fc7f9-c9dqc" Oct 31 05:14:34.114310 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:34.114272 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91ac1809-b6e5-4c0e-a324-cfdc8c088e74-kube-api-access-md48j\" (UniqueName: \"kubernetes.io/projected/91ac1809-b6e5-4c0e-a324-cfdc8c088e74-kube-api-access-md48j\") pod \"91ac1809-b6e5-4c0e-a324-cfdc8c088e74\" (UID: \"91ac1809-b6e5-4c0e-a324-cfdc8c088e74\") " Oct 31 05:14:34.114544 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:34.114361 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74e04a61-5277-4180-bc46-68af7640b25e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74e04a61-5277-4180-bc46-68af7640b25e-plugins-config-volume\") pod \"74e04a61-5277-4180-bc46-68af7640b25e\" (UID: \"74e04a61-5277-4180-bc46-68af7640b25e\") " Oct 31 05:14:34.114544 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:34.114445 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91ac1809-b6e5-4c0e-a324-cfdc8c088e74-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91ac1809-b6e5-4c0e-a324-cfdc8c088e74-plugins-config-volume\") pod \"91ac1809-b6e5-4c0e-a324-cfdc8c088e74\" (UID: \"91ac1809-b6e5-4c0e-a324-cfdc8c088e74\") " Oct 31 05:14:34.114544 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:34.114515 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74e04a61-5277-4180-bc46-68af7640b25e-kube-api-access-wxzzz\" (UniqueName: \"kubernetes.io/projected/74e04a61-5277-4180-bc46-68af7640b25e-kube-api-access-wxzzz\") pod \"74e04a61-5277-4180-bc46-68af7640b25e\" (UID: \"74e04a61-5277-4180-bc46-68af7640b25e\") " Oct 31 05:14:34.115408 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:34.114980 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91ac1809-b6e5-4c0e-a324-cfdc8c088e74-plugins-config-volume" pod "91ac1809-b6e5-4c0e-a324-cfdc8c088e74" (UID: "91ac1809-b6e5-4c0e-a324-cfdc8c088e74"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:14:34.115408 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:34.115094 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91ac1809-b6e5-4c0e-a324-cfdc8c088e74-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:14:34.115408 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:34.115366 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74e04a61-5277-4180-bc46-68af7640b25e-plugins-config-volume" pod "74e04a61-5277-4180-bc46-68af7640b25e" (UID: "74e04a61-5277-4180-bc46-68af7640b25e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:14:34.116883 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:34.116860 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74e04a61-5277-4180-bc46-68af7640b25e-kube-api-access-wxzzz" pod "74e04a61-5277-4180-bc46-68af7640b25e" (UID: "74e04a61-5277-4180-bc46-68af7640b25e"). InnerVolumeSpecName "kube-api-access-wxzzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:14:34.117125 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:34.117110 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91ac1809-b6e5-4c0e-a324-cfdc8c088e74-kube-api-access-md48j" pod "91ac1809-b6e5-4c0e-a324-cfdc8c088e74" (UID: "91ac1809-b6e5-4c0e-a324-cfdc8c088e74"). InnerVolumeSpecName "kube-api-access-md48j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:14:34.216481 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:34.216447 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-md48j\" (UniqueName: \"kubernetes.io/projected/91ac1809-b6e5-4c0e-a324-cfdc8c088e74-kube-api-access-md48j\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:14:34.216481 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:34.216477 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74e04a61-5277-4180-bc46-68af7640b25e-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:14:34.216481 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:34.216489 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxzzz\" (UniqueName: \"kubernetes.io/projected/74e04a61-5277-4180-bc46-68af7640b25e-kube-api-access-wxzzz\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:14:34.276456 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:34.276365 5019 generic.go:356] "Generic (PLEG): container finished" podID="74e04a61-5277-4180-bc46-68af7640b25e" containerID="d664a380cf9fdd7c09c421f5e2960a6b2db4cc531109ee9946434ab68acc60f0" exitCode=0 Oct 31 05:14:34.276456 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:34.276405 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ded435-5787fc7f9-c9dqc" Oct 31 05:14:34.276681 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:34.276448 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ded435-5787fc7f9-c9dqc" event={"ID":"74e04a61-5277-4180-bc46-68af7640b25e","Type":"ContainerDied","Data":"d664a380cf9fdd7c09c421f5e2960a6b2db4cc531109ee9946434ab68acc60f0"} Oct 31 05:14:34.276681 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:34.276505 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ded435-5787fc7f9-c9dqc" event={"ID":"74e04a61-5277-4180-bc46-68af7640b25e","Type":"ContainerDied","Data":"da7e56fa98d953fac990d82d820bd1f840e83df0f95901f56d5148d4332805cf"} Oct 31 05:14:34.276681 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:34.276533 5019 scope.go:122] "RemoveContainer" containerID="d664a380cf9fdd7c09c421f5e2960a6b2db4cc531109ee9946434ab68acc60f0" Oct 31 05:14:34.278071 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:34.278045 5019 generic.go:356] "Generic (PLEG): container finished" podID="91ac1809-b6e5-4c0e-a324-cfdc8c088e74" containerID="91cb972e9136e8af3bb377ede3771c827c383ec6692bae6165ed723119b92755" exitCode=0 Oct 31 05:14:34.278198 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:34.278074 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-ded435-5cbbf45d4c-j9btt" event={"ID":"91ac1809-b6e5-4c0e-a324-cfdc8c088e74","Type":"ContainerDied","Data":"91cb972e9136e8af3bb377ede3771c827c383ec6692bae6165ed723119b92755"} Oct 31 05:14:34.278198 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:34.278097 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-ded435-5cbbf45d4c-j9btt" Oct 31 05:14:34.278198 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:34.278125 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-ded435-5cbbf45d4c-j9btt" event={"ID":"91ac1809-b6e5-4c0e-a324-cfdc8c088e74","Type":"ContainerDied","Data":"c53bdadb908f29cd2f26efa882ed52b321760a89a9af6a17a1e5064093d762e0"} Oct 31 05:14:34.289843 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:34.289825 5019 scope.go:122] "RemoveContainer" containerID="d664a380cf9fdd7c09c421f5e2960a6b2db4cc531109ee9946434ab68acc60f0" Oct 31 05:14:34.290085 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:14:34.290064 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d664a380cf9fdd7c09c421f5e2960a6b2db4cc531109ee9946434ab68acc60f0\": container with ID starting with d664a380cf9fdd7c09c421f5e2960a6b2db4cc531109ee9946434ab68acc60f0 not found: ID does not exist" containerID="d664a380cf9fdd7c09c421f5e2960a6b2db4cc531109ee9946434ab68acc60f0" Oct 31 05:14:34.290141 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:34.290095 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d664a380cf9fdd7c09c421f5e2960a6b2db4cc531109ee9946434ab68acc60f0"} err="failed to get container status \"d664a380cf9fdd7c09c421f5e2960a6b2db4cc531109ee9946434ab68acc60f0\": rpc error: code = NotFound desc = could not find container \"d664a380cf9fdd7c09c421f5e2960a6b2db4cc531109ee9946434ab68acc60f0\": container with ID starting with d664a380cf9fdd7c09c421f5e2960a6b2db4cc531109ee9946434ab68acc60f0 not found: ID does not exist" Oct 31 05:14:34.290141 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:34.290113 5019 scope.go:122] "RemoveContainer" containerID="91cb972e9136e8af3bb377ede3771c827c383ec6692bae6165ed723119b92755" Oct 31 05:14:34.300273 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:34.300256 5019 scope.go:122] "RemoveContainer" containerID="91cb972e9136e8af3bb377ede3771c827c383ec6692bae6165ed723119b92755" Oct 31 05:14:34.300606 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:14:34.300579 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91cb972e9136e8af3bb377ede3771c827c383ec6692bae6165ed723119b92755\": container with ID starting with 91cb972e9136e8af3bb377ede3771c827c383ec6692bae6165ed723119b92755 not found: ID does not exist" containerID="91cb972e9136e8af3bb377ede3771c827c383ec6692bae6165ed723119b92755" Oct 31 05:14:34.300680 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:34.300612 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91cb972e9136e8af3bb377ede3771c827c383ec6692bae6165ed723119b92755"} err="failed to get container status \"91cb972e9136e8af3bb377ede3771c827c383ec6692bae6165ed723119b92755\": rpc error: code = NotFound desc = could not find container \"91cb972e9136e8af3bb377ede3771c827c383ec6692bae6165ed723119b92755\": container with ID starting with 91cb972e9136e8af3bb377ede3771c827c383ec6692bae6165ed723119b92755 not found: ID does not exist" Oct 31 05:14:34.330371 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:34.330325 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-ded435-5cbbf45d4c-j9btt"] Oct 31 05:14:34.334208 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:34.334181 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-ded435-5cbbf45d4c-j9btt"] Oct 31 05:14:34.367013 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:34.366988 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-ded435-5787fc7f9-c9dqc"] Oct 31 05:14:34.372051 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:34.372026 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-ded435-5787fc7f9-c9dqc"] Oct 31 05:14:35.408809 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:35.408776 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74e04a61-5277-4180-bc46-68af7640b25e" path="/var/lib/kubelet/pods/74e04a61-5277-4180-bc46-68af7640b25e/volumes" Oct 31 05:14:35.409302 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:35.409267 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91ac1809-b6e5-4c0e-a324-cfdc8c088e74" path="/var/lib/kubelet/pods/91ac1809-b6e5-4c0e-a324-cfdc8c088e74/volumes" Oct 31 05:14:38.944662 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:38.944623 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt"] Oct 31 05:14:38.945134 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:38.944889 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt" podUID="2764ba4c-dff3-4c51-a8ca-0121dd9f5b29" containerName="istio-proxy" containerID="cri-o://ba5bf75249bd21f67e87b4770c90f3f88d061af1f1f287a2250a8dda09f6d977" gracePeriod=30 Oct 31 05:14:44.233771 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:44.233740 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt" Oct 31 05:14:44.354382 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:44.354340 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-istiod-ca-cert\") pod \"2764ba4c-dff3-4c51-a8ca-0121dd9f5b29\" (UID: \"2764ba4c-dff3-4c51-a8ca-0121dd9f5b29\") " Oct 31 05:14:44.354557 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:44.354417 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-istio-podinfo\") pod \"2764ba4c-dff3-4c51-a8ca-0121dd9f5b29\" (UID: \"2764ba4c-dff3-4c51-a8ca-0121dd9f5b29\") " Oct 31 05:14:44.354557 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:44.354511 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-workload-socket\") pod \"2764ba4c-dff3-4c51-a8ca-0121dd9f5b29\" (UID: \"2764ba4c-dff3-4c51-a8ca-0121dd9f5b29\") " Oct 31 05:14:44.354681 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:44.354557 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-credential-socket\") pod \"2764ba4c-dff3-4c51-a8ca-0121dd9f5b29\" (UID: \"2764ba4c-dff3-4c51-a8ca-0121dd9f5b29\") " Oct 31 05:14:44.354681 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:44.354637 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-workload-certs\") pod \"2764ba4c-dff3-4c51-a8ca-0121dd9f5b29\" (UID: \"2764ba4c-dff3-4c51-a8ca-0121dd9f5b29\") " Oct 31 05:14:44.354802 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:44.354703 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-kube-api-access-kw574\" (UniqueName: \"kubernetes.io/projected/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-kube-api-access-kw574\") pod \"2764ba4c-dff3-4c51-a8ca-0121dd9f5b29\" (UID: \"2764ba4c-dff3-4c51-a8ca-0121dd9f5b29\") " Oct 31 05:14:44.354802 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:44.354744 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-istio-token\" (UniqueName: \"kubernetes.io/projected/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-istio-token\") pod \"2764ba4c-dff3-4c51-a8ca-0121dd9f5b29\" (UID: \"2764ba4c-dff3-4c51-a8ca-0121dd9f5b29\") " Oct 31 05:14:44.354911 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:44.354843 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-istio-data\") pod \"2764ba4c-dff3-4c51-a8ca-0121dd9f5b29\" (UID: \"2764ba4c-dff3-4c51-a8ca-0121dd9f5b29\") " Oct 31 05:14:44.354911 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:44.354777 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-istiod-ca-cert" pod "2764ba4c-dff3-4c51-a8ca-0121dd9f5b29" (UID: "2764ba4c-dff3-4c51-a8ca-0121dd9f5b29"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:14:44.354911 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:44.354870 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-workload-socket" pod "2764ba4c-dff3-4c51-a8ca-0121dd9f5b29" (UID: "2764ba4c-dff3-4c51-a8ca-0121dd9f5b29"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:14:44.354911 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:44.354878 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-istio-envoy\") pod \"2764ba4c-dff3-4c51-a8ca-0121dd9f5b29\" (UID: \"2764ba4c-dff3-4c51-a8ca-0121dd9f5b29\") " Oct 31 05:14:44.355504 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:44.355482 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:14:44.355692 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:44.355654 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-istio-data" pod "2764ba4c-dff3-4c51-a8ca-0121dd9f5b29" (UID: "2764ba4c-dff3-4c51-a8ca-0121dd9f5b29"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:14:44.355810 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:44.355791 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:14:44.355928 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:44.355835 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-credential-socket" pod "2764ba4c-dff3-4c51-a8ca-0121dd9f5b29" (UID: "2764ba4c-dff3-4c51-a8ca-0121dd9f5b29"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:14:44.356034 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:44.355877 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-workload-certs" pod "2764ba4c-dff3-4c51-a8ca-0121dd9f5b29" (UID: "2764ba4c-dff3-4c51-a8ca-0121dd9f5b29"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:14:44.357274 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:44.357238 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-istio-podinfo" pod "2764ba4c-dff3-4c51-a8ca-0121dd9f5b29" (UID: "2764ba4c-dff3-4c51-a8ca-0121dd9f5b29"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:14:44.357610 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:44.357570 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-istio-envoy" pod "2764ba4c-dff3-4c51-a8ca-0121dd9f5b29" (UID: "2764ba4c-dff3-4c51-a8ca-0121dd9f5b29"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:14:44.357722 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:44.357661 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-kube-api-access-kw574" pod "2764ba4c-dff3-4c51-a8ca-0121dd9f5b29" (UID: "2764ba4c-dff3-4c51-a8ca-0121dd9f5b29"). InnerVolumeSpecName "kube-api-access-kw574". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:14:44.358302 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:44.358275 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-istio-token" pod "2764ba4c-dff3-4c51-a8ca-0121dd9f5b29" (UID: "2764ba4c-dff3-4c51-a8ca-0121dd9f5b29"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:14:44.362274 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:44.362249 5019 generic.go:356] "Generic (PLEG): container finished" podID="2764ba4c-dff3-4c51-a8ca-0121dd9f5b29" containerID="ba5bf75249bd21f67e87b4770c90f3f88d061af1f1f287a2250a8dda09f6d977" exitCode=0 Oct 31 05:14:44.362391 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:44.362295 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt" event={"ID":"2764ba4c-dff3-4c51-a8ca-0121dd9f5b29","Type":"ContainerDied","Data":"ba5bf75249bd21f67e87b4770c90f3f88d061af1f1f287a2250a8dda09f6d977"} Oct 31 05:14:44.362391 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:44.362312 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt" Oct 31 05:14:44.362391 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:44.362375 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt" event={"ID":"2764ba4c-dff3-4c51-a8ca-0121dd9f5b29","Type":"ContainerDied","Data":"2ad9671df018a064944518fa26e0f21c22b6fe2770e7ce0493c63a4fbdb76433"} Oct 31 05:14:44.362535 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:44.362407 5019 scope.go:122] "RemoveContainer" containerID="ba5bf75249bd21f67e87b4770c90f3f88d061af1f1f287a2250a8dda09f6d977" Oct 31 05:14:44.391102 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:44.391080 5019 scope.go:122] "RemoveContainer" containerID="ae3722b96fc47f0c6468e33e2f94c81d370fd8ec6ad49c0f6464a7e744156dca" Oct 31 05:14:44.402320 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:44.402303 5019 scope.go:122] "RemoveContainer" containerID="ba5bf75249bd21f67e87b4770c90f3f88d061af1f1f287a2250a8dda09f6d977" Oct 31 05:14:44.402632 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:14:44.402611 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba5bf75249bd21f67e87b4770c90f3f88d061af1f1f287a2250a8dda09f6d977\": container with ID starting with ba5bf75249bd21f67e87b4770c90f3f88d061af1f1f287a2250a8dda09f6d977 not found: ID does not exist" containerID="ba5bf75249bd21f67e87b4770c90f3f88d061af1f1f287a2250a8dda09f6d977" Oct 31 05:14:44.402704 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:44.402639 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba5bf75249bd21f67e87b4770c90f3f88d061af1f1f287a2250a8dda09f6d977"} err="failed to get container status \"ba5bf75249bd21f67e87b4770c90f3f88d061af1f1f287a2250a8dda09f6d977\": rpc error: code = NotFound desc = could not find container \"ba5bf75249bd21f67e87b4770c90f3f88d061af1f1f287a2250a8dda09f6d977\": container with ID starting with ba5bf75249bd21f67e87b4770c90f3f88d061af1f1f287a2250a8dda09f6d977 not found: ID does not exist" Oct 31 05:14:44.402704 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:44.402654 5019 scope.go:122] "RemoveContainer" containerID="ae3722b96fc47f0c6468e33e2f94c81d370fd8ec6ad49c0f6464a7e744156dca" Oct 31 05:14:44.402927 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:14:44.402901 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae3722b96fc47f0c6468e33e2f94c81d370fd8ec6ad49c0f6464a7e744156dca\": container with ID starting with ae3722b96fc47f0c6468e33e2f94c81d370fd8ec6ad49c0f6464a7e744156dca not found: ID does not exist" containerID="ae3722b96fc47f0c6468e33e2f94c81d370fd8ec6ad49c0f6464a7e744156dca" Oct 31 05:14:44.402987 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:44.402939 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae3722b96fc47f0c6468e33e2f94c81d370fd8ec6ad49c0f6464a7e744156dca"} err="failed to get container status \"ae3722b96fc47f0c6468e33e2f94c81d370fd8ec6ad49c0f6464a7e744156dca\": rpc error: code = NotFound desc = could not find container \"ae3722b96fc47f0c6468e33e2f94c81d370fd8ec6ad49c0f6464a7e744156dca\": container with ID starting with ae3722b96fc47f0c6468e33e2f94c81d370fd8ec6ad49c0f6464a7e744156dca not found: ID does not exist" Oct 31 05:14:44.425752 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:44.425725 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt"] Oct 31 05:14:44.429938 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:44.429914 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-ded435-openshift-default-6c5655ff69-qb5zt"] Oct 31 05:14:44.457028 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:44.457000 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kw574\" (UniqueName: \"kubernetes.io/projected/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-kube-api-access-kw574\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:14:44.457227 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:44.457030 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:14:44.457227 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:44.457045 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:14:44.457227 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:44.457058 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:14:44.457227 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:44.457070 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:14:44.457227 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:44.457083 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:14:44.457227 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:44.457096 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:14:45.408096 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:14:45.408063 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2764ba4c-dff3-4c51-a8ca-0121dd9f5b29" path="/var/lib/kubelet/pods/2764ba4c-dff3-4c51-a8ca-0121dd9f5b29/volumes" Oct 31 05:14:52.392112 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:14:52.392066 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:15:00.153039 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.153005 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890395-nlstq"] Oct 31 05:15:00.153602 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.153586 5019 cpu_manager.go:414] "containerMap: removing container" podUID="2764ba4c-dff3-4c51-a8ca-0121dd9f5b29" containerName="istio-proxy" Oct 31 05:15:00.153654 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.153603 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2764ba4c-dff3-4c51-a8ca-0121dd9f5b29" containerName="istio-proxy" Oct 31 05:15:00.153654 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.153622 5019 cpu_manager.go:414] "containerMap: removing container" podUID="2764ba4c-dff3-4c51-a8ca-0121dd9f5b29" containerName="istio-proxy" Oct 31 05:15:00.153654 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.153627 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2764ba4c-dff3-4c51-a8ca-0121dd9f5b29" containerName="istio-proxy" Oct 31 05:15:00.153654 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.153640 5019 cpu_manager.go:414] "containerMap: removing container" podUID="93d2b6ca-e93d-4dd1-805a-b8985e4da4b9" containerName="istio-proxy" Oct 31 05:15:00.153654 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.153648 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93d2b6ca-e93d-4dd1-805a-b8985e4da4b9" containerName="istio-proxy" Oct 31 05:15:00.153829 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.153661 5019 cpu_manager.go:414] "containerMap: removing container" podUID="74e04a61-5277-4180-bc46-68af7640b25e" containerName="payload-pre-processing" Oct 31 05:15:00.153829 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.153669 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74e04a61-5277-4180-bc46-68af7640b25e" containerName="payload-pre-processing" Oct 31 05:15:00.153829 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.153711 5019 cpu_manager.go:414] "containerMap: removing container" podUID="b66ab368-8869-4a6d-a83e-e40d92df0042" containerName="istio-proxy" Oct 31 05:15:00.153829 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.153720 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b66ab368-8869-4a6d-a83e-e40d92df0042" containerName="istio-proxy" Oct 31 05:15:00.153829 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.153742 5019 cpu_manager.go:414] "containerMap: removing container" podUID="91ac1809-b6e5-4c0e-a324-cfdc8c088e74" containerName="payload-processing" Oct 31 05:15:00.153829 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.153749 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91ac1809-b6e5-4c0e-a324-cfdc8c088e74" containerName="payload-processing" Oct 31 05:15:00.154058 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.153895 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="2764ba4c-dff3-4c51-a8ca-0121dd9f5b29" containerName="istio-proxy" Oct 31 05:15:00.154058 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.153912 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="2764ba4c-dff3-4c51-a8ca-0121dd9f5b29" containerName="istio-proxy" Oct 31 05:15:00.154058 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.153923 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="93d2b6ca-e93d-4dd1-805a-b8985e4da4b9" containerName="istio-proxy" Oct 31 05:15:00.154058 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.153933 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="74e04a61-5277-4180-bc46-68af7640b25e" containerName="payload-pre-processing" Oct 31 05:15:00.154058 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.153944 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="b66ab368-8869-4a6d-a83e-e40d92df0042" containerName="istio-proxy" Oct 31 05:15:00.154058 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.153951 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="91ac1809-b6e5-4c0e-a324-cfdc8c088e74" containerName="payload-processing" Oct 31 05:15:00.157743 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.157715 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890395-nlstq" Oct 31 05:15:00.163105 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.160458 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890395-zd2lr"] Oct 31 05:15:00.165408 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.165388 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890395-zd2lr" Oct 31 05:15:00.166029 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.166002 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890395-gkgv4"] Oct 31 05:15:00.170729 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.170710 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890395-gkgv4" Oct 31 05:15:00.171159 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.171148 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890395-fzdrx"] Oct 31 05:15:00.175312 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.175294 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890395-fzdrx" Oct 31 05:15:00.175917 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.175895 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890395-zjqjv"] Oct 31 05:15:00.177678 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.177637 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:15:00.177832 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.177750 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:15:00.184384 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.181683 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890395-4sfpv"] Oct 31 05:15:00.184384 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.181761 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890395-zjqjv" Oct 31 05:15:00.185677 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.185097 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 05:15:00.188975 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.188949 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890395-4sfpv" Oct 31 05:15:00.193595 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.192888 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890395-8fb6v"] Oct 31 05:15:00.207972 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.207164 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890395-q6v97"] Oct 31 05:15:00.208108 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.208022 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890395-8fb6v" Oct 31 05:15:00.222202 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.222045 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890395-nlstq"] Oct 31 05:15:00.222202 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.222088 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890395-zd2lr"] Oct 31 05:15:00.222396 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.222214 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890395-fzdrx"] Oct 31 05:15:00.222396 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.222250 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890395-gkgv4"] Oct 31 05:15:00.222521 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.222309 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890395-q6v97" Oct 31 05:15:00.231254 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.231227 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890395-zjqjv"] Oct 31 05:15:00.241256 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.241235 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890395-vm7v4"] Oct 31 05:15:00.248940 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.248909 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb54g\" (UniqueName: \"kubernetes.io/projected/e099ec30-86fb-43aa-9ee0-f3305f0adcfb-kube-api-access-rb54g\") pod \"maas-api-key-cleanup-e2e-worker-w0-965f61-29890395-nlstq\" (UID: \"e099ec30-86fb-43aa-9ee0-f3305f0adcfb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890395-nlstq" Oct 31 05:15:00.249198 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.249177 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe68f52a-ede6-464c-9d03-5cc39f001cb8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890395-gkgv4\" (UID: \"fe68f52a-ede6-464c-9d03-5cc39f001cb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890395-gkgv4" Oct 31 05:15:00.249421 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.249402 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7621d02f-ad60-4445-be36-e7c253887716-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890395-q6v97\" (UID: \"7621d02f-ad60-4445-be36-e7c253887716\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890395-q6v97" Oct 31 05:15:00.249648 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.249626 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm2td\" (UniqueName: \"kubernetes.io/projected/2030df55-524b-4764-ad64-002ab5d1cc52-kube-api-access-jm2td\") pod \"maas-api-key-cleanup-e2e-extmod-a-8973e2-29890395-fzdrx\" (UID: \"2030df55-524b-4764-ad64-002ab5d1cc52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890395-fzdrx" Oct 31 05:15:00.249852 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.249832 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/754e0f92-f1dd-40a2-9c1f-031c383b4629-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890395-zd2lr\" (UID: \"754e0f92-f1dd-40a2-9c1f-031c383b4629\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890395-zd2lr" Oct 31 05:15:00.250037 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.250019 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2030df55-524b-4764-ad64-002ab5d1cc52-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-8973e2-29890395-fzdrx\" (UID: \"2030df55-524b-4764-ad64-002ab5d1cc52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890395-fzdrx" Oct 31 05:15:00.250547 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.250525 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvrms\" (UniqueName: \"kubernetes.io/projected/ed220fc2-8244-47bd-a868-472210c5e521-kube-api-access-bvrms\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-766728-29890395-4sfpv\" (UID: \"ed220fc2-8244-47bd-a868-472210c5e521\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890395-4sfpv" Oct 31 05:15:00.250761 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.250742 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dfnc\" (UniqueName: \"kubernetes.io/projected/1873945a-f1ed-43b2-b787-c29a1783b3e3-kube-api-access-7dfnc\") pod \"maas-api-key-cleanup-e2e-extmod-a-afc519-29890395-8fb6v\" (UID: \"1873945a-f1ed-43b2-b787-c29a1783b3e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890395-8fb6v" Oct 31 05:15:00.250972 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.250953 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4e14c11-5250-4d70-932f-055294e67dd4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890395-zjqjv\" (UID: \"a4e14c11-5250-4d70-932f-055294e67dd4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890395-zjqjv" Oct 31 05:15:00.251443 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.251423 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkzxd\" (UniqueName: \"kubernetes.io/projected/a4e14c11-5250-4d70-932f-055294e67dd4-kube-api-access-dkzxd\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890395-zjqjv\" (UID: \"a4e14c11-5250-4d70-932f-055294e67dd4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890395-zjqjv" Oct 31 05:15:00.251679 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.251662 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcdmz\" (UniqueName: \"kubernetes.io/projected/fe68f52a-ede6-464c-9d03-5cc39f001cb8-kube-api-access-hcdmz\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890395-gkgv4\" (UID: \"fe68f52a-ede6-464c-9d03-5cc39f001cb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890395-gkgv4" Oct 31 05:15:00.251829 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.251812 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjqnm\" (UniqueName: \"kubernetes.io/projected/754e0f92-f1dd-40a2-9c1f-031c383b4629-kube-api-access-jjqnm\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890395-zd2lr\" (UID: \"754e0f92-f1dd-40a2-9c1f-031c383b4629\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890395-zd2lr" Oct 31 05:15:00.251990 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.251974 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clzp2\" (UniqueName: \"kubernetes.io/projected/7621d02f-ad60-4445-be36-e7c253887716-kube-api-access-clzp2\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890395-q6v97\" (UID: \"7621d02f-ad60-4445-be36-e7c253887716\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890395-q6v97" Oct 31 05:15:00.252143 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.252126 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1873945a-f1ed-43b2-b787-c29a1783b3e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-afc519-29890395-8fb6v\" (UID: \"1873945a-f1ed-43b2-b787-c29a1783b3e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890395-8fb6v" Oct 31 05:15:00.252724 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.252314 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e099ec30-86fb-43aa-9ee0-f3305f0adcfb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-965f61-29890395-nlstq\" (UID: \"e099ec30-86fb-43aa-9ee0-f3305f0adcfb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890395-nlstq" Oct 31 05:15:00.253094 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.253071 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed220fc2-8244-47bd-a868-472210c5e521-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-766728-29890395-4sfpv\" (UID: \"ed220fc2-8244-47bd-a868-472210c5e521\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890395-4sfpv" Oct 31 05:15:00.253319 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.250312 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890395-q6v97"] Oct 31 05:15:00.253515 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.253499 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890395-8fb6v"] Oct 31 05:15:00.253654 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.250457 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890395-vm7v4" Oct 31 05:15:00.259410 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.259388 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890395-vm7v4"] Oct 31 05:15:00.263526 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.263504 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890395-4sfpv"] Oct 31 05:15:00.353978 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.353942 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvrms\" (UniqueName: \"kubernetes.io/projected/ed220fc2-8244-47bd-a868-472210c5e521-kube-api-access-bvrms\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-766728-29890395-4sfpv\" (UID: \"ed220fc2-8244-47bd-a868-472210c5e521\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890395-4sfpv" Oct 31 05:15:00.354147 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.353992 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk2br\" (UniqueName: \"kubernetes.io/projected/8e80ef6a-aee2-4040-a8dc-ca294ff2409c-kube-api-access-xk2br\") pod \"maas-api-key-cleanup-e2e-worker-w4-c4006d-29890395-vm7v4\" (UID: \"8e80ef6a-aee2-4040-a8dc-ca294ff2409c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890395-vm7v4" Oct 31 05:15:00.354147 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.354029 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dfnc\" (UniqueName: \"kubernetes.io/projected/1873945a-f1ed-43b2-b787-c29a1783b3e3-kube-api-access-7dfnc\") pod \"maas-api-key-cleanup-e2e-extmod-a-afc519-29890395-8fb6v\" (UID: \"1873945a-f1ed-43b2-b787-c29a1783b3e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890395-8fb6v" Oct 31 05:15:00.354147 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.354123 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4e14c11-5250-4d70-932f-055294e67dd4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890395-zjqjv\" (UID: \"a4e14c11-5250-4d70-932f-055294e67dd4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890395-zjqjv" Oct 31 05:15:00.354386 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.354160 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkzxd\" (UniqueName: \"kubernetes.io/projected/a4e14c11-5250-4d70-932f-055294e67dd4-kube-api-access-dkzxd\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890395-zjqjv\" (UID: \"a4e14c11-5250-4d70-932f-055294e67dd4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890395-zjqjv" Oct 31 05:15:00.354386 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.354201 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcdmz\" (UniqueName: \"kubernetes.io/projected/fe68f52a-ede6-464c-9d03-5cc39f001cb8-kube-api-access-hcdmz\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890395-gkgv4\" (UID: \"fe68f52a-ede6-464c-9d03-5cc39f001cb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890395-gkgv4" Oct 31 05:15:00.354386 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.354235 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjqnm\" (UniqueName: \"kubernetes.io/projected/754e0f92-f1dd-40a2-9c1f-031c383b4629-kube-api-access-jjqnm\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890395-zd2lr\" (UID: \"754e0f92-f1dd-40a2-9c1f-031c383b4629\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890395-zd2lr" Oct 31 05:15:00.354386 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.354273 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clzp2\" (UniqueName: \"kubernetes.io/projected/7621d02f-ad60-4445-be36-e7c253887716-kube-api-access-clzp2\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890395-q6v97\" (UID: \"7621d02f-ad60-4445-be36-e7c253887716\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890395-q6v97" Oct 31 05:15:00.354386 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.354300 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1873945a-f1ed-43b2-b787-c29a1783b3e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-afc519-29890395-8fb6v\" (UID: \"1873945a-f1ed-43b2-b787-c29a1783b3e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890395-8fb6v" Oct 31 05:15:00.354692 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.354397 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e099ec30-86fb-43aa-9ee0-f3305f0adcfb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-965f61-29890395-nlstq\" (UID: \"e099ec30-86fb-43aa-9ee0-f3305f0adcfb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890395-nlstq" Oct 31 05:15:00.354692 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.354481 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e80ef6a-aee2-4040-a8dc-ca294ff2409c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c4006d-29890395-vm7v4\" (UID: \"8e80ef6a-aee2-4040-a8dc-ca294ff2409c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890395-vm7v4" Oct 31 05:15:00.354692 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.354519 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed220fc2-8244-47bd-a868-472210c5e521-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-766728-29890395-4sfpv\" (UID: \"ed220fc2-8244-47bd-a868-472210c5e521\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890395-4sfpv" Oct 31 05:15:00.354692 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.354552 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb54g\" (UniqueName: \"kubernetes.io/projected/e099ec30-86fb-43aa-9ee0-f3305f0adcfb-kube-api-access-rb54g\") pod \"maas-api-key-cleanup-e2e-worker-w0-965f61-29890395-nlstq\" (UID: \"e099ec30-86fb-43aa-9ee0-f3305f0adcfb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890395-nlstq" Oct 31 05:15:00.354692 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.354590 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe68f52a-ede6-464c-9d03-5cc39f001cb8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890395-gkgv4\" (UID: \"fe68f52a-ede6-464c-9d03-5cc39f001cb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890395-gkgv4" Oct 31 05:15:00.354692 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.354618 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7621d02f-ad60-4445-be36-e7c253887716-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890395-q6v97\" (UID: \"7621d02f-ad60-4445-be36-e7c253887716\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890395-q6v97" Oct 31 05:15:00.354692 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.354687 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm2td\" (UniqueName: \"kubernetes.io/projected/2030df55-524b-4764-ad64-002ab5d1cc52-kube-api-access-jm2td\") pod \"maas-api-key-cleanup-e2e-extmod-a-8973e2-29890395-fzdrx\" (UID: \"2030df55-524b-4764-ad64-002ab5d1cc52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890395-fzdrx" Oct 31 05:15:00.355089 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.354924 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4e14c11-5250-4d70-932f-055294e67dd4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890395-zjqjv\" (UID: \"a4e14c11-5250-4d70-932f-055294e67dd4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890395-zjqjv" Oct 31 05:15:00.355252 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.355225 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1873945a-f1ed-43b2-b787-c29a1783b3e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-afc519-29890395-8fb6v\" (UID: \"1873945a-f1ed-43b2-b787-c29a1783b3e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890395-8fb6v" Oct 31 05:15:00.355454 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.355430 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe68f52a-ede6-464c-9d03-5cc39f001cb8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890395-gkgv4\" (UID: \"fe68f52a-ede6-464c-9d03-5cc39f001cb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890395-gkgv4" Oct 31 05:15:00.355543 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.355476 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed220fc2-8244-47bd-a868-472210c5e521-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-766728-29890395-4sfpv\" (UID: \"ed220fc2-8244-47bd-a868-472210c5e521\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890395-4sfpv" Oct 31 05:15:00.355543 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.355498 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e099ec30-86fb-43aa-9ee0-f3305f0adcfb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-965f61-29890395-nlstq\" (UID: \"e099ec30-86fb-43aa-9ee0-f3305f0adcfb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890395-nlstq" Oct 31 05:15:00.355543 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.355528 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/754e0f92-f1dd-40a2-9c1f-031c383b4629-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890395-zd2lr\" (UID: \"754e0f92-f1dd-40a2-9c1f-031c383b4629\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890395-zd2lr" Oct 31 05:15:00.355770 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.355570 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2030df55-524b-4764-ad64-002ab5d1cc52-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-8973e2-29890395-fzdrx\" (UID: \"2030df55-524b-4764-ad64-002ab5d1cc52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890395-fzdrx" Oct 31 05:15:00.355770 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.355605 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7621d02f-ad60-4445-be36-e7c253887716-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890395-q6v97\" (UID: \"7621d02f-ad60-4445-be36-e7c253887716\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890395-q6v97" Oct 31 05:15:00.356203 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.356179 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/754e0f92-f1dd-40a2-9c1f-031c383b4629-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890395-zd2lr\" (UID: \"754e0f92-f1dd-40a2-9c1f-031c383b4629\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890395-zd2lr" Oct 31 05:15:00.356350 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.356298 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2030df55-524b-4764-ad64-002ab5d1cc52-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-8973e2-29890395-fzdrx\" (UID: \"2030df55-524b-4764-ad64-002ab5d1cc52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890395-fzdrx" Oct 31 05:15:00.378692 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.378663 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvrms\" (UniqueName: \"kubernetes.io/projected/ed220fc2-8244-47bd-a868-472210c5e521-kube-api-access-bvrms\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-766728-29890395-4sfpv\" (UID: \"ed220fc2-8244-47bd-a868-472210c5e521\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890395-4sfpv" Oct 31 05:15:00.378867 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.378848 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcdmz\" (UniqueName: \"kubernetes.io/projected/fe68f52a-ede6-464c-9d03-5cc39f001cb8-kube-api-access-hcdmz\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890395-gkgv4\" (UID: \"fe68f52a-ede6-464c-9d03-5cc39f001cb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890395-gkgv4" Oct 31 05:15:00.379258 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.379236 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjqnm\" (UniqueName: \"kubernetes.io/projected/754e0f92-f1dd-40a2-9c1f-031c383b4629-kube-api-access-jjqnm\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890395-zd2lr\" (UID: \"754e0f92-f1dd-40a2-9c1f-031c383b4629\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890395-zd2lr" Oct 31 05:15:00.379954 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.379924 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dfnc\" (UniqueName: \"kubernetes.io/projected/1873945a-f1ed-43b2-b787-c29a1783b3e3-kube-api-access-7dfnc\") pod \"maas-api-key-cleanup-e2e-extmod-a-afc519-29890395-8fb6v\" (UID: \"1873945a-f1ed-43b2-b787-c29a1783b3e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890395-8fb6v" Oct 31 05:15:00.379954 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.379924 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkzxd\" (UniqueName: \"kubernetes.io/projected/a4e14c11-5250-4d70-932f-055294e67dd4-kube-api-access-dkzxd\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890395-zjqjv\" (UID: \"a4e14c11-5250-4d70-932f-055294e67dd4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890395-zjqjv" Oct 31 05:15:00.380149 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.380125 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm2td\" (UniqueName: \"kubernetes.io/projected/2030df55-524b-4764-ad64-002ab5d1cc52-kube-api-access-jm2td\") pod \"maas-api-key-cleanup-e2e-extmod-a-8973e2-29890395-fzdrx\" (UID: \"2030df55-524b-4764-ad64-002ab5d1cc52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890395-fzdrx" Oct 31 05:15:00.381169 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.381147 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clzp2\" (UniqueName: \"kubernetes.io/projected/7621d02f-ad60-4445-be36-e7c253887716-kube-api-access-clzp2\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890395-q6v97\" (UID: \"7621d02f-ad60-4445-be36-e7c253887716\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890395-q6v97" Oct 31 05:15:00.381472 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.381448 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb54g\" (UniqueName: \"kubernetes.io/projected/e099ec30-86fb-43aa-9ee0-f3305f0adcfb-kube-api-access-rb54g\") pod \"maas-api-key-cleanup-e2e-worker-w0-965f61-29890395-nlstq\" (UID: \"e099ec30-86fb-43aa-9ee0-f3305f0adcfb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890395-nlstq" Oct 31 05:15:00.457306 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.457247 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e80ef6a-aee2-4040-a8dc-ca294ff2409c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c4006d-29890395-vm7v4\" (UID: \"8e80ef6a-aee2-4040-a8dc-ca294ff2409c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890395-vm7v4" Oct 31 05:15:00.457420 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.457347 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk2br\" (UniqueName: \"kubernetes.io/projected/8e80ef6a-aee2-4040-a8dc-ca294ff2409c-kube-api-access-xk2br\") pod \"maas-api-key-cleanup-e2e-worker-w4-c4006d-29890395-vm7v4\" (UID: \"8e80ef6a-aee2-4040-a8dc-ca294ff2409c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890395-vm7v4" Oct 31 05:15:00.457997 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.457968 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e80ef6a-aee2-4040-a8dc-ca294ff2409c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c4006d-29890395-vm7v4\" (UID: \"8e80ef6a-aee2-4040-a8dc-ca294ff2409c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890395-vm7v4" Oct 31 05:15:00.480601 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.480575 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk2br\" (UniqueName: \"kubernetes.io/projected/8e80ef6a-aee2-4040-a8dc-ca294ff2409c-kube-api-access-xk2br\") pod \"maas-api-key-cleanup-e2e-worker-w4-c4006d-29890395-vm7v4\" (UID: \"8e80ef6a-aee2-4040-a8dc-ca294ff2409c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890395-vm7v4" Oct 31 05:15:00.480799 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.480725 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890395-nlstq" Oct 31 05:15:00.516592 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.516566 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890395-zd2lr" Oct 31 05:15:00.537171 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.537116 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890395-gkgv4" Oct 31 05:15:00.568359 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.557570 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890395-fzdrx" Oct 31 05:15:00.569598 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.569559 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890395-zjqjv" Oct 31 05:15:00.586682 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.586518 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890395-4sfpv" Oct 31 05:15:00.646848 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.643599 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890395-vm7v4" Oct 31 05:15:00.646848 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.644021 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890395-q6v97" Oct 31 05:15:00.646848 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.644113 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890395-8fb6v" Oct 31 05:15:00.900984 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.900916 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890395-nlstq"] Oct 31 05:15:00.915528 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:00.914062 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890395-zd2lr"] Oct 31 05:15:00.917549 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:15:00.916365 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode099ec30_86fb_43aa_9ee0_f3305f0adcfb.slice/crio-561053ccae4be041cca41f68202fa808469e67936a1100155a81ae0f17556fc7 WatchSource:0}: Error finding container 561053ccae4be041cca41f68202fa808469e67936a1100155a81ae0f17556fc7: Status 404 returned error can't find the container with id 561053ccae4be041cca41f68202fa808469e67936a1100155a81ae0f17556fc7 Oct 31 05:15:00.927590 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:15:00.927562 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod754e0f92_f1dd_40a2_9c1f_031c383b4629.slice/crio-174dbf52d30ec889e266b300bca342e63a48d6ee514ad6c241cb982675a72378 WatchSource:0}: Error finding container 174dbf52d30ec889e266b300bca342e63a48d6ee514ad6c241cb982675a72378: Status 404 returned error can't find the container with id 174dbf52d30ec889e266b300bca342e63a48d6ee514ad6c241cb982675a72378 Oct 31 05:15:01.065088 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:15:01.065053 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe68f52a_ede6_464c_9d03_5cc39f001cb8.slice/crio-468b40768c21e112819f32abee5db73f26f3f1ef58f9c1d3f8a14d82f76d2482 WatchSource:0}: Error finding container 468b40768c21e112819f32abee5db73f26f3f1ef58f9c1d3f8a14d82f76d2482: Status 404 returned error can't find the container with id 468b40768c21e112819f32abee5db73f26f3f1ef58f9c1d3f8a14d82f76d2482 Oct 31 05:15:01.068046 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:15:01.067969 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4e14c11_5250_4d70_932f_055294e67dd4.slice/crio-8c1c2712180235a3c1a61958b197c0af9cda92b10df56c6dea63584173b5d837 WatchSource:0}: Error finding container 8c1c2712180235a3c1a61958b197c0af9cda92b10df56c6dea63584173b5d837: Status 404 returned error can't find the container with id 8c1c2712180235a3c1a61958b197c0af9cda92b10df56c6dea63584173b5d837 Oct 31 05:15:01.069748 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:01.069721 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890395-zjqjv"] Oct 31 05:15:01.076477 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:01.076451 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890395-gkgv4"] Oct 31 05:15:01.080772 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:01.080746 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890395-fzdrx"] Oct 31 05:15:01.115055 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:01.115035 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890395-4sfpv"] Oct 31 05:15:01.136181 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:01.136161 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890395-8fb6v"] Oct 31 05:15:01.149154 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:01.149135 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890395-vm7v4"] Oct 31 05:15:01.162629 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:01.162608 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890395-q6v97"] Oct 31 05:15:01.175421 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:15:01.175366 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded220fc2_8244_47bd_a868_472210c5e521.slice/crio-51c49ec6d2e25d0e87e8ba181fb3fece94a8d8952cdd9d845b42b29a9b0ee56b WatchSource:0}: Error finding container 51c49ec6d2e25d0e87e8ba181fb3fece94a8d8952cdd9d845b42b29a9b0ee56b: Status 404 returned error can't find the container with id 51c49ec6d2e25d0e87e8ba181fb3fece94a8d8952cdd9d845b42b29a9b0ee56b Oct 31 05:15:01.175814 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:15:01.175726 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1873945a_f1ed_43b2_b787_c29a1783b3e3.slice/crio-c0112c7e37058570d18528b7c76ee14de3c848bb0eaf8231c22a3c3b4f0e4f00 WatchSource:0}: Error finding container c0112c7e37058570d18528b7c76ee14de3c848bb0eaf8231c22a3c3b4f0e4f00: Status 404 returned error can't find the container with id c0112c7e37058570d18528b7c76ee14de3c848bb0eaf8231c22a3c3b4f0e4f00 Oct 31 05:15:01.177933 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:15:01.177912 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e80ef6a_aee2_4040_a8dc_ca294ff2409c.slice/crio-02f9784b2ebd34e0d7aa2c683c16601c4ecc49d09b601c49c0c7db87c7a422b9 WatchSource:0}: Error finding container 02f9784b2ebd34e0d7aa2c683c16601c4ecc49d09b601c49c0c7db87c7a422b9: Status 404 returned error can't find the container with id 02f9784b2ebd34e0d7aa2c683c16601c4ecc49d09b601c49c0c7db87c7a422b9 Oct 31 05:15:01.186217 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:15:01.186193 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7621d02f_ad60_4445_be36_e7c253887716.slice/crio-250cc312a61a8e0d0157734f32d19cba6ce5565bbfdbc1337342e67bdb119d4b WatchSource:0}: Error finding container 250cc312a61a8e0d0157734f32d19cba6ce5565bbfdbc1337342e67bdb119d4b: Status 404 returned error can't find the container with id 250cc312a61a8e0d0157734f32d19cba6ce5565bbfdbc1337342e67bdb119d4b Oct 31 05:15:01.518484 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:01.518449 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890395-zd2lr" event={"ID":"754e0f92-f1dd-40a2-9c1f-031c383b4629","Type":"ContainerStarted","Data":"1c0fcc0945621a327a2a93b341da1834e9c30035249a2672060ac0ac3f332d90"} Oct 31 05:15:01.518636 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:01.518510 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890395-zd2lr" event={"ID":"754e0f92-f1dd-40a2-9c1f-031c383b4629","Type":"ContainerStarted","Data":"174dbf52d30ec889e266b300bca342e63a48d6ee514ad6c241cb982675a72378"} Oct 31 05:15:01.520364 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:01.520290 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890395-zjqjv" event={"ID":"a4e14c11-5250-4d70-932f-055294e67dd4","Type":"ContainerStarted","Data":"8c1c2712180235a3c1a61958b197c0af9cda92b10df56c6dea63584173b5d837"} Oct 31 05:15:01.523851 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:01.523815 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890395-8fb6v" event={"ID":"1873945a-f1ed-43b2-b787-c29a1783b3e3","Type":"ContainerStarted","Data":"c0112c7e37058570d18528b7c76ee14de3c848bb0eaf8231c22a3c3b4f0e4f00"} Oct 31 05:15:01.525483 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:01.525455 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890395-gkgv4" event={"ID":"fe68f52a-ede6-464c-9d03-5cc39f001cb8","Type":"ContainerStarted","Data":"468b40768c21e112819f32abee5db73f26f3f1ef58f9c1d3f8a14d82f76d2482"} Oct 31 05:15:01.527008 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:01.526980 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890395-vm7v4" event={"ID":"8e80ef6a-aee2-4040-a8dc-ca294ff2409c","Type":"ContainerStarted","Data":"02f9784b2ebd34e0d7aa2c683c16601c4ecc49d09b601c49c0c7db87c7a422b9"} Oct 31 05:15:01.528661 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:01.528639 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890395-fzdrx" event={"ID":"2030df55-524b-4764-ad64-002ab5d1cc52","Type":"ContainerStarted","Data":"985f3798fb5207ac539b85a6bab968558c3375ada3e148e60c8de1d7ec7ba6a3"} Oct 31 05:15:01.530014 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:01.529988 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890395-4sfpv" event={"ID":"ed220fc2-8244-47bd-a868-472210c5e521","Type":"ContainerStarted","Data":"51c49ec6d2e25d0e87e8ba181fb3fece94a8d8952cdd9d845b42b29a9b0ee56b"} Oct 31 05:15:01.531813 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:01.531786 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890395-nlstq" event={"ID":"e099ec30-86fb-43aa-9ee0-f3305f0adcfb","Type":"ContainerStarted","Data":"582b1b28fb806770552bcc8c1b71bd32e5efe59e7f6940f7fc7eff0f7e1e63ef"} Oct 31 05:15:01.531936 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:01.531834 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890395-nlstq" event={"ID":"e099ec30-86fb-43aa-9ee0-f3305f0adcfb","Type":"ContainerStarted","Data":"561053ccae4be041cca41f68202fa808469e67936a1100155a81ae0f17556fc7"} Oct 31 05:15:01.538610 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:01.538587 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890395-q6v97" event={"ID":"7621d02f-ad60-4445-be36-e7c253887716","Type":"ContainerStarted","Data":"250cc312a61a8e0d0157734f32d19cba6ce5565bbfdbc1337342e67bdb119d4b"} Oct 31 05:15:01.566042 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:01.566000 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890395-zd2lr" podStartSLOduration=1.5659877880000002 podStartE2EDuration="1.565987788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:15:01.561713445 +0000 UTC m=+4256.318051486" watchObservedRunningTime="2026-10-31 05:15:01.565987788 +0000 UTC m=+4256.322325866" Oct 31 05:15:01.602915 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:01.602860 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890395-nlstq" podStartSLOduration=1.602842801 podStartE2EDuration="1.602842801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:15:01.601687204 +0000 UTC m=+4256.358025245" watchObservedRunningTime="2026-10-31 05:15:01.602842801 +0000 UTC m=+4256.359180842" Oct 31 05:15:02.547886 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:02.547853 5019 generic.go:356] "Generic (PLEG): container finished" podID="a4e14c11-5250-4d70-932f-055294e67dd4" containerID="31aa4347d556f3e63b5c7de6f81a99f0c1cc78079439e1c43c9345bb5e9eb0aa" exitCode=60 Oct 31 05:15:02.548271 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:02.547938 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890395-zjqjv" event={"ID":"a4e14c11-5250-4d70-932f-055294e67dd4","Type":"ContainerDied","Data":"31aa4347d556f3e63b5c7de6f81a99f0c1cc78079439e1c43c9345bb5e9eb0aa"} Oct 31 05:15:02.548271 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:02.548153 5019 scope.go:122] "RemoveContainer" containerID="31aa4347d556f3e63b5c7de6f81a99f0c1cc78079439e1c43c9345bb5e9eb0aa" Oct 31 05:15:02.549863 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:02.549843 5019 generic.go:356] "Generic (PLEG): container finished" podID="1873945a-f1ed-43b2-b787-c29a1783b3e3" containerID="8edd76826aae73d8cb1ccb7a9463e1ea7bb4db3a1e9674504074bd479a717e47" exitCode=60 Oct 31 05:15:02.549934 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:02.549912 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890395-8fb6v" event={"ID":"1873945a-f1ed-43b2-b787-c29a1783b3e3","Type":"ContainerDied","Data":"8edd76826aae73d8cb1ccb7a9463e1ea7bb4db3a1e9674504074bd479a717e47"} Oct 31 05:15:02.550143 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:02.550124 5019 scope.go:122] "RemoveContainer" containerID="8edd76826aae73d8cb1ccb7a9463e1ea7bb4db3a1e9674504074bd479a717e47" Oct 31 05:15:02.551773 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:02.551748 5019 generic.go:356] "Generic (PLEG): container finished" podID="fe68f52a-ede6-464c-9d03-5cc39f001cb8" containerID="9c21b3bc70f80932ec9750b0525c750c4e6fc270dbb910e3f95825dbb7579cec" exitCode=60 Oct 31 05:15:02.551870 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:02.551819 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890395-gkgv4" event={"ID":"fe68f52a-ede6-464c-9d03-5cc39f001cb8","Type":"ContainerDied","Data":"9c21b3bc70f80932ec9750b0525c750c4e6fc270dbb910e3f95825dbb7579cec"} Oct 31 05:15:02.551986 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:02.551968 5019 scope.go:122] "RemoveContainer" containerID="9c21b3bc70f80932ec9750b0525c750c4e6fc270dbb910e3f95825dbb7579cec" Oct 31 05:15:02.553683 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:02.553662 5019 generic.go:356] "Generic (PLEG): container finished" podID="8e80ef6a-aee2-4040-a8dc-ca294ff2409c" containerID="695d5e79abf1f22e42023d0ce9f0a7d9f7613a95ceb27f9dc15b0ba89898f7c8" exitCode=60 Oct 31 05:15:02.553790 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:02.553743 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890395-vm7v4" event={"ID":"8e80ef6a-aee2-4040-a8dc-ca294ff2409c","Type":"ContainerDied","Data":"695d5e79abf1f22e42023d0ce9f0a7d9f7613a95ceb27f9dc15b0ba89898f7c8"} Oct 31 05:15:02.553924 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:02.553908 5019 scope.go:122] "RemoveContainer" containerID="695d5e79abf1f22e42023d0ce9f0a7d9f7613a95ceb27f9dc15b0ba89898f7c8" Oct 31 05:15:02.555806 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:02.555787 5019 generic.go:356] "Generic (PLEG): container finished" podID="2030df55-524b-4764-ad64-002ab5d1cc52" containerID="5873d5ccb910f04a0cad73ed97986cfd714d7d2ab75fbc29f6bebe86f070f21f" exitCode=60 Oct 31 05:15:02.555912 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:02.555871 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890395-fzdrx" event={"ID":"2030df55-524b-4764-ad64-002ab5d1cc52","Type":"ContainerDied","Data":"5873d5ccb910f04a0cad73ed97986cfd714d7d2ab75fbc29f6bebe86f070f21f"} Oct 31 05:15:02.556077 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:02.556057 5019 scope.go:122] "RemoveContainer" containerID="5873d5ccb910f04a0cad73ed97986cfd714d7d2ab75fbc29f6bebe86f070f21f" Oct 31 05:15:02.557678 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:02.557659 5019 generic.go:356] "Generic (PLEG): container finished" podID="ed220fc2-8244-47bd-a868-472210c5e521" containerID="de60aa55874dbef26922405df03dcc26762e0076145dfe5dc821e6466946f2bb" exitCode=60 Oct 31 05:15:02.557770 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:02.557726 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890395-4sfpv" event={"ID":"ed220fc2-8244-47bd-a868-472210c5e521","Type":"ContainerDied","Data":"de60aa55874dbef26922405df03dcc26762e0076145dfe5dc821e6466946f2bb"} Oct 31 05:15:02.557909 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:02.557895 5019 scope.go:122] "RemoveContainer" containerID="de60aa55874dbef26922405df03dcc26762e0076145dfe5dc821e6466946f2bb" Oct 31 05:15:02.574842 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:02.571164 5019 generic.go:356] "Generic (PLEG): container finished" podID="e099ec30-86fb-43aa-9ee0-f3305f0adcfb" containerID="582b1b28fb806770552bcc8c1b71bd32e5efe59e7f6940f7fc7eff0f7e1e63ef" exitCode=60 Oct 31 05:15:02.574842 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:02.571629 5019 scope.go:122] "RemoveContainer" containerID="582b1b28fb806770552bcc8c1b71bd32e5efe59e7f6940f7fc7eff0f7e1e63ef" Oct 31 05:15:02.574842 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:02.571669 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890395-nlstq" event={"ID":"e099ec30-86fb-43aa-9ee0-f3305f0adcfb","Type":"ContainerDied","Data":"582b1b28fb806770552bcc8c1b71bd32e5efe59e7f6940f7fc7eff0f7e1e63ef"} Oct 31 05:15:02.574842 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:02.574433 5019 generic.go:356] "Generic (PLEG): container finished" podID="7621d02f-ad60-4445-be36-e7c253887716" containerID="46846129ff26375f6267401e03244620665d03e91189f62ef8fc6eb59abf7efb" exitCode=60 Oct 31 05:15:02.574842 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:02.574645 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890395-q6v97" event={"ID":"7621d02f-ad60-4445-be36-e7c253887716","Type":"ContainerDied","Data":"46846129ff26375f6267401e03244620665d03e91189f62ef8fc6eb59abf7efb"} Oct 31 05:15:02.575275 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:02.574943 5019 scope.go:122] "RemoveContainer" containerID="46846129ff26375f6267401e03244620665d03e91189f62ef8fc6eb59abf7efb" Oct 31 05:15:02.578206 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:02.578175 5019 generic.go:356] "Generic (PLEG): container finished" podID="754e0f92-f1dd-40a2-9c1f-031c383b4629" containerID="1c0fcc0945621a327a2a93b341da1834e9c30035249a2672060ac0ac3f332d90" exitCode=60 Oct 31 05:15:02.578307 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:02.578216 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890395-zd2lr" event={"ID":"754e0f92-f1dd-40a2-9c1f-031c383b4629","Type":"ContainerDied","Data":"1c0fcc0945621a327a2a93b341da1834e9c30035249a2672060ac0ac3f332d90"} Oct 31 05:15:02.578629 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:02.578609 5019 scope.go:122] "RemoveContainer" containerID="1c0fcc0945621a327a2a93b341da1834e9c30035249a2672060ac0ac3f332d90" Oct 31 05:15:03.590785 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:03.590747 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890395-gkgv4" event={"ID":"fe68f52a-ede6-464c-9d03-5cc39f001cb8","Type":"ContainerStarted","Data":"8c9f0ef363613ae8eab592d35373a6c3335a6565cc952f7cad68481ec9450f3e"} Oct 31 05:15:03.592956 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:03.592928 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890395-vm7v4" event={"ID":"8e80ef6a-aee2-4040-a8dc-ca294ff2409c","Type":"ContainerStarted","Data":"1d19599f98afe55f0baddf1b09957e1bc5e1abcbc5c3cee362e514fe09fe6a99"} Oct 31 05:15:03.595064 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:03.595038 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890395-fzdrx" event={"ID":"2030df55-524b-4764-ad64-002ab5d1cc52","Type":"ContainerStarted","Data":"7db6315e3c71fdb96e611d15a02eb664f80a0008d4408c355851cd6b89a071b0"} Oct 31 05:15:03.597144 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:03.597121 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890395-4sfpv" event={"ID":"ed220fc2-8244-47bd-a868-472210c5e521","Type":"ContainerStarted","Data":"976503fc1e3de8b719fee54b69923f27c5f84e8c3a458673dcd6c87b45e981da"} Oct 31 05:15:03.599520 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:03.599495 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890395-nlstq" event={"ID":"e099ec30-86fb-43aa-9ee0-f3305f0adcfb","Type":"ContainerStarted","Data":"120474799df77357f725ea2220e435840e2fc0a163fb9a09c7dfe30dfb6a3fed"} Oct 31 05:15:03.601635 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:03.601610 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890395-q6v97" event={"ID":"7621d02f-ad60-4445-be36-e7c253887716","Type":"ContainerStarted","Data":"98529fed0b218751675ba9adfad1f32241f6083020fbd741ab137b4469a890fa"} Oct 31 05:15:03.603616 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:03.603586 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890395-zd2lr" event={"ID":"754e0f92-f1dd-40a2-9c1f-031c383b4629","Type":"ContainerStarted","Data":"178ba3d5a2556f2b5cb3b47330ef05396050abc0453caf1f5c13736fa54dafa2"} Oct 31 05:15:03.606172 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:03.606143 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890395-zjqjv" event={"ID":"a4e14c11-5250-4d70-932f-055294e67dd4","Type":"ContainerStarted","Data":"0756d61698e203ae0cdbe36c099595dcc246cb5ff2cfe793827ccbdeed720107"} Oct 31 05:15:03.608287 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:03.608261 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890395-8fb6v" event={"ID":"1873945a-f1ed-43b2-b787-c29a1783b3e3","Type":"ContainerStarted","Data":"7b3d25e64458f963a6a523ab9e1c98679cbc4648fde6d6c29be9a6ac7f2f8d4e"} Oct 31 05:15:03.631861 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:03.631821 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890395-gkgv4" podStartSLOduration=3.631809545 podStartE2EDuration="3.631809545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:15:03.630693147 +0000 UTC m=+4258.387031186" watchObservedRunningTime="2026-10-31 05:15:03.631809545 +0000 UTC m=+4258.388147585" Oct 31 05:15:03.685548 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:03.684518 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890395-zjqjv" podStartSLOduration=3.684497426 podStartE2EDuration="3.684497426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:15:03.677805324 +0000 UTC m=+4258.434143365" watchObservedRunningTime="2026-10-31 05:15:03.684497426 +0000 UTC m=+4258.440835466" Oct 31 05:15:03.722715 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:03.722657 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890395-fzdrx" podStartSLOduration=3.722643684 podStartE2EDuration="3.722643684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:15:03.721753293 +0000 UTC m=+4258.478091346" watchObservedRunningTime="2026-10-31 05:15:03.722643684 +0000 UTC m=+4258.478981756" Oct 31 05:15:03.764469 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:03.764424 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890395-8fb6v" podStartSLOduration=3.76441073 podStartE2EDuration="3.76441073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:15:03.76049218 +0000 UTC m=+4258.516830221" watchObservedRunningTime="2026-10-31 05:15:03.76441073 +0000 UTC m=+4258.520748767" Oct 31 05:15:03.802717 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:03.802663 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890395-vm7v4" podStartSLOduration=3.802645377 podStartE2EDuration="3.802645377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:15:03.800221074 +0000 UTC m=+4258.556559116" watchObservedRunningTime="2026-10-31 05:15:03.802645377 +0000 UTC m=+4258.558983419" Oct 31 05:15:03.840926 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:15:03.840830 5019 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded220fc2_8244_47bd_a868_472210c5e521.slice/crio-976503fc1e3de8b719fee54b69923f27c5f84e8c3a458673dcd6c87b45e981da.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4e14c11_5250_4d70_932f_055294e67dd4.slice/crio-0756d61698e203ae0cdbe36c099595dcc246cb5ff2cfe793827ccbdeed720107.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe68f52a_ede6_464c_9d03_5cc39f001cb8.slice/crio-conmon-8c9f0ef363613ae8eab592d35373a6c3335a6565cc952f7cad68481ec9450f3e.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded220fc2_8244_47bd_a868_472210c5e521.slice/crio-conmon-976503fc1e3de8b719fee54b69923f27c5f84e8c3a458673dcd6c87b45e981da.scope\": RecentStats: unable to find data in memory cache]" Oct 31 05:15:03.922681 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:03.922635 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890395-q6v97" podStartSLOduration=3.922623309 podStartE2EDuration="3.922623309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:15:03.919090691 +0000 UTC m=+4258.675428732" watchObservedRunningTime="2026-10-31 05:15:03.922623309 +0000 UTC m=+4258.678961349" Oct 31 05:15:03.960948 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:03.960902 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890395-4sfpv" podStartSLOduration=3.960889303 podStartE2EDuration="3.960889303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:15:03.957946039 +0000 UTC m=+4258.714284080" watchObservedRunningTime="2026-10-31 05:15:03.960889303 +0000 UTC m=+4258.717227337" Oct 31 05:15:04.618983 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:04.618946 5019 generic.go:356] "Generic (PLEG): container finished" podID="fe68f52a-ede6-464c-9d03-5cc39f001cb8" containerID="8c9f0ef363613ae8eab592d35373a6c3335a6565cc952f7cad68481ec9450f3e" exitCode=60 Oct 31 05:15:04.619463 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:04.619027 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890395-gkgv4" event={"ID":"fe68f52a-ede6-464c-9d03-5cc39f001cb8","Type":"ContainerDied","Data":"8c9f0ef363613ae8eab592d35373a6c3335a6565cc952f7cad68481ec9450f3e"} Oct 31 05:15:04.619463 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:04.619093 5019 scope.go:122] "RemoveContainer" containerID="9c21b3bc70f80932ec9750b0525c750c4e6fc270dbb910e3f95825dbb7579cec" Oct 31 05:15:04.619463 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:04.619390 5019 scope.go:122] "RemoveContainer" containerID="8c9f0ef363613ae8eab592d35373a6c3335a6565cc952f7cad68481ec9450f3e" Oct 31 05:15:04.619771 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:15:04.619746 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890395-gkgv4_odh-ai-gateway-infra(fe68f52a-ede6-464c-9d03-5cc39f001cb8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890395-gkgv4" podUID="fe68f52a-ede6-464c-9d03-5cc39f001cb8" Oct 31 05:15:04.621409 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:04.621383 5019 generic.go:356] "Generic (PLEG): container finished" podID="8e80ef6a-aee2-4040-a8dc-ca294ff2409c" containerID="1d19599f98afe55f0baddf1b09957e1bc5e1abcbc5c3cee362e514fe09fe6a99" exitCode=60 Oct 31 05:15:04.621513 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:04.621468 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890395-vm7v4" event={"ID":"8e80ef6a-aee2-4040-a8dc-ca294ff2409c","Type":"ContainerDied","Data":"1d19599f98afe55f0baddf1b09957e1bc5e1abcbc5c3cee362e514fe09fe6a99"} Oct 31 05:15:04.621692 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:04.621678 5019 scope.go:122] "RemoveContainer" containerID="1d19599f98afe55f0baddf1b09957e1bc5e1abcbc5c3cee362e514fe09fe6a99" Oct 31 05:15:04.621912 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:15:04.621889 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c4006d-29890395-vm7v4_odh-ai-gateway-infra(8e80ef6a-aee2-4040-a8dc-ca294ff2409c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890395-vm7v4" podUID="8e80ef6a-aee2-4040-a8dc-ca294ff2409c" Oct 31 05:15:04.624001 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:04.623976 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890395-fzdrx" event={"ID":"2030df55-524b-4764-ad64-002ab5d1cc52","Type":"ContainerDied","Data":"7db6315e3c71fdb96e611d15a02eb664f80a0008d4408c355851cd6b89a071b0"} Oct 31 05:15:04.624717 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:04.624698 5019 scope.go:122] "RemoveContainer" containerID="7db6315e3c71fdb96e611d15a02eb664f80a0008d4408c355851cd6b89a071b0" Oct 31 05:15:04.625285 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:15:04.625261 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-8973e2-29890395-fzdrx_odh-ai-gateway-infra(2030df55-524b-4764-ad64-002ab5d1cc52)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890395-fzdrx" podUID="2030df55-524b-4764-ad64-002ab5d1cc52" Oct 31 05:15:04.625447 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:04.625423 5019 generic.go:356] "Generic (PLEG): container finished" podID="2030df55-524b-4764-ad64-002ab5d1cc52" containerID="7db6315e3c71fdb96e611d15a02eb664f80a0008d4408c355851cd6b89a071b0" exitCode=60 Oct 31 05:15:04.639612 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:04.639575 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890395-4sfpv" event={"ID":"ed220fc2-8244-47bd-a868-472210c5e521","Type":"ContainerDied","Data":"976503fc1e3de8b719fee54b69923f27c5f84e8c3a458673dcd6c87b45e981da"} Oct 31 05:15:04.639724 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:04.639555 5019 generic.go:356] "Generic (PLEG): container finished" podID="ed220fc2-8244-47bd-a868-472210c5e521" containerID="976503fc1e3de8b719fee54b69923f27c5f84e8c3a458673dcd6c87b45e981da" exitCode=60 Oct 31 05:15:04.640185 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:04.640170 5019 scope.go:122] "RemoveContainer" containerID="976503fc1e3de8b719fee54b69923f27c5f84e8c3a458673dcd6c87b45e981da" Oct 31 05:15:04.640597 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:15:04.640573 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-766728-29890395-4sfpv_odh-ai-gateway-infra(ed220fc2-8244-47bd-a868-472210c5e521)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890395-4sfpv" podUID="ed220fc2-8244-47bd-a868-472210c5e521" Oct 31 05:15:04.642802 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:04.642778 5019 generic.go:356] "Generic (PLEG): container finished" podID="e099ec30-86fb-43aa-9ee0-f3305f0adcfb" containerID="120474799df77357f725ea2220e435840e2fc0a163fb9a09c7dfe30dfb6a3fed" exitCode=60 Oct 31 05:15:04.642909 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:04.642858 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890395-nlstq" event={"ID":"e099ec30-86fb-43aa-9ee0-f3305f0adcfb","Type":"ContainerDied","Data":"120474799df77357f725ea2220e435840e2fc0a163fb9a09c7dfe30dfb6a3fed"} Oct 31 05:15:04.643083 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:04.643060 5019 scope.go:122] "RemoveContainer" containerID="120474799df77357f725ea2220e435840e2fc0a163fb9a09c7dfe30dfb6a3fed" Oct 31 05:15:04.643205 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:04.643190 5019 scope.go:122] "RemoveContainer" containerID="695d5e79abf1f22e42023d0ce9f0a7d9f7613a95ceb27f9dc15b0ba89898f7c8" Oct 31 05:15:04.643365 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:15:04.643308 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-965f61-29890395-nlstq_odh-ai-gateway-infra(e099ec30-86fb-43aa-9ee0-f3305f0adcfb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890395-nlstq" podUID="e099ec30-86fb-43aa-9ee0-f3305f0adcfb" Oct 31 05:15:04.645409 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:04.645387 5019 generic.go:356] "Generic (PLEG): container finished" podID="7621d02f-ad60-4445-be36-e7c253887716" containerID="98529fed0b218751675ba9adfad1f32241f6083020fbd741ab137b4469a890fa" exitCode=60 Oct 31 05:15:04.645528 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:04.645508 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890395-q6v97" event={"ID":"7621d02f-ad60-4445-be36-e7c253887716","Type":"ContainerDied","Data":"98529fed0b218751675ba9adfad1f32241f6083020fbd741ab137b4469a890fa"} Oct 31 05:15:04.645716 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:04.645701 5019 scope.go:122] "RemoveContainer" containerID="98529fed0b218751675ba9adfad1f32241f6083020fbd741ab137b4469a890fa" Oct 31 05:15:04.645939 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:15:04.645919 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890395-q6v97_odh-ai-gateway-infra(7621d02f-ad60-4445-be36-e7c253887716)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890395-q6v97" podUID="7621d02f-ad60-4445-be36-e7c253887716" Oct 31 05:15:04.647430 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:04.647412 5019 generic.go:356] "Generic (PLEG): container finished" podID="754e0f92-f1dd-40a2-9c1f-031c383b4629" containerID="178ba3d5a2556f2b5cb3b47330ef05396050abc0453caf1f5c13736fa54dafa2" exitCode=60 Oct 31 05:15:04.647519 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:04.647491 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890395-zd2lr" event={"ID":"754e0f92-f1dd-40a2-9c1f-031c383b4629","Type":"ContainerDied","Data":"178ba3d5a2556f2b5cb3b47330ef05396050abc0453caf1f5c13736fa54dafa2"} Oct 31 05:15:04.647669 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:04.647656 5019 scope.go:122] "RemoveContainer" containerID="178ba3d5a2556f2b5cb3b47330ef05396050abc0453caf1f5c13736fa54dafa2" Oct 31 05:15:04.647882 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:15:04.647859 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890395-zd2lr_odh-ai-gateway-infra(754e0f92-f1dd-40a2-9c1f-031c383b4629)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890395-zd2lr" podUID="754e0f92-f1dd-40a2-9c1f-031c383b4629" Oct 31 05:15:04.649499 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:04.649456 5019 generic.go:356] "Generic (PLEG): container finished" podID="a4e14c11-5250-4d70-932f-055294e67dd4" containerID="0756d61698e203ae0cdbe36c099595dcc246cb5ff2cfe793827ccbdeed720107" exitCode=60 Oct 31 05:15:04.649614 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:04.649580 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890395-zjqjv" event={"ID":"a4e14c11-5250-4d70-932f-055294e67dd4","Type":"ContainerDied","Data":"0756d61698e203ae0cdbe36c099595dcc246cb5ff2cfe793827ccbdeed720107"} Oct 31 05:15:04.649947 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:04.649925 5019 scope.go:122] "RemoveContainer" containerID="0756d61698e203ae0cdbe36c099595dcc246cb5ff2cfe793827ccbdeed720107" Oct 31 05:15:04.650289 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:15:04.650256 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890395-zjqjv_odh-ai-gateway-infra(a4e14c11-5250-4d70-932f-055294e67dd4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890395-zjqjv" podUID="a4e14c11-5250-4d70-932f-055294e67dd4" Oct 31 05:15:04.657069 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:04.657046 5019 generic.go:356] "Generic (PLEG): container finished" podID="1873945a-f1ed-43b2-b787-c29a1783b3e3" containerID="7b3d25e64458f963a6a523ab9e1c98679cbc4648fde6d6c29be9a6ac7f2f8d4e" exitCode=60 Oct 31 05:15:04.657180 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:04.657068 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890395-8fb6v" event={"ID":"1873945a-f1ed-43b2-b787-c29a1783b3e3","Type":"ContainerDied","Data":"7b3d25e64458f963a6a523ab9e1c98679cbc4648fde6d6c29be9a6ac7f2f8d4e"} Oct 31 05:15:04.657379 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:04.657325 5019 scope.go:122] "RemoveContainer" containerID="7b3d25e64458f963a6a523ab9e1c98679cbc4648fde6d6c29be9a6ac7f2f8d4e" Oct 31 05:15:04.657608 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:15:04.657589 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-afc519-29890395-8fb6v_odh-ai-gateway-infra(1873945a-f1ed-43b2-b787-c29a1783b3e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890395-8fb6v" podUID="1873945a-f1ed-43b2-b787-c29a1783b3e3" Oct 31 05:15:04.663784 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:04.663763 5019 scope.go:122] "RemoveContainer" containerID="5873d5ccb910f04a0cad73ed97986cfd714d7d2ab75fbc29f6bebe86f070f21f" Oct 31 05:15:04.677728 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:04.677710 5019 scope.go:122] "RemoveContainer" containerID="de60aa55874dbef26922405df03dcc26762e0076145dfe5dc821e6466946f2bb" Oct 31 05:15:04.690529 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:04.690513 5019 scope.go:122] "RemoveContainer" containerID="582b1b28fb806770552bcc8c1b71bd32e5efe59e7f6940f7fc7eff0f7e1e63ef" Oct 31 05:15:04.703424 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:04.703407 5019 scope.go:122] "RemoveContainer" containerID="46846129ff26375f6267401e03244620665d03e91189f62ef8fc6eb59abf7efb" Oct 31 05:15:04.718222 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:04.718197 5019 scope.go:122] "RemoveContainer" containerID="1c0fcc0945621a327a2a93b341da1834e9c30035249a2672060ac0ac3f332d90" Oct 31 05:15:04.731189 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:04.731171 5019 scope.go:122] "RemoveContainer" containerID="31aa4347d556f3e63b5c7de6f81a99f0c1cc78079439e1c43c9345bb5e9eb0aa" Oct 31 05:15:04.743970 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:04.743953 5019 scope.go:122] "RemoveContainer" containerID="8edd76826aae73d8cb1ccb7a9463e1ea7bb4db3a1e9674504074bd479a717e47" Oct 31 05:15:05.670268 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:05.670239 5019 scope.go:122] "RemoveContainer" containerID="178ba3d5a2556f2b5cb3b47330ef05396050abc0453caf1f5c13736fa54dafa2" Oct 31 05:15:05.670802 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:15:05.670477 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890395-zd2lr_odh-ai-gateway-infra(754e0f92-f1dd-40a2-9c1f-031c383b4629)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890395-zd2lr" podUID="754e0f92-f1dd-40a2-9c1f-031c383b4629" Oct 31 05:15:05.671983 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:05.671962 5019 scope.go:122] "RemoveContainer" containerID="0756d61698e203ae0cdbe36c099595dcc246cb5ff2cfe793827ccbdeed720107" Oct 31 05:15:05.672167 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:15:05.672146 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890395-zjqjv_odh-ai-gateway-infra(a4e14c11-5250-4d70-932f-055294e67dd4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890395-zjqjv" podUID="a4e14c11-5250-4d70-932f-055294e67dd4" Oct 31 05:15:05.673605 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:05.673583 5019 scope.go:122] "RemoveContainer" containerID="7b3d25e64458f963a6a523ab9e1c98679cbc4648fde6d6c29be9a6ac7f2f8d4e" Oct 31 05:15:05.673777 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:15:05.673757 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-afc519-29890395-8fb6v_odh-ai-gateway-infra(1873945a-f1ed-43b2-b787-c29a1783b3e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890395-8fb6v" podUID="1873945a-f1ed-43b2-b787-c29a1783b3e3" Oct 31 05:15:05.675223 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:05.675206 5019 scope.go:122] "RemoveContainer" containerID="8c9f0ef363613ae8eab592d35373a6c3335a6565cc952f7cad68481ec9450f3e" Oct 31 05:15:05.675445 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:15:05.675419 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890395-gkgv4_odh-ai-gateway-infra(fe68f52a-ede6-464c-9d03-5cc39f001cb8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890395-gkgv4" podUID="fe68f52a-ede6-464c-9d03-5cc39f001cb8" Oct 31 05:15:05.676913 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:05.676893 5019 scope.go:122] "RemoveContainer" containerID="1d19599f98afe55f0baddf1b09957e1bc5e1abcbc5c3cee362e514fe09fe6a99" Oct 31 05:15:05.677131 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:15:05.677107 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c4006d-29890395-vm7v4_odh-ai-gateway-infra(8e80ef6a-aee2-4040-a8dc-ca294ff2409c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890395-vm7v4" podUID="8e80ef6a-aee2-4040-a8dc-ca294ff2409c" Oct 31 05:15:05.678533 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:05.678516 5019 scope.go:122] "RemoveContainer" containerID="7db6315e3c71fdb96e611d15a02eb664f80a0008d4408c355851cd6b89a071b0" Oct 31 05:15:05.678717 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:15:05.678699 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-8973e2-29890395-fzdrx_odh-ai-gateway-infra(2030df55-524b-4764-ad64-002ab5d1cc52)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890395-fzdrx" podUID="2030df55-524b-4764-ad64-002ab5d1cc52" Oct 31 05:15:05.680203 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:05.680184 5019 scope.go:122] "RemoveContainer" containerID="976503fc1e3de8b719fee54b69923f27c5f84e8c3a458673dcd6c87b45e981da" Oct 31 05:15:05.680412 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:15:05.680394 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-766728-29890395-4sfpv_odh-ai-gateway-infra(ed220fc2-8244-47bd-a868-472210c5e521)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890395-4sfpv" podUID="ed220fc2-8244-47bd-a868-472210c5e521" Oct 31 05:15:05.681920 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:05.681902 5019 scope.go:122] "RemoveContainer" containerID="120474799df77357f725ea2220e435840e2fc0a163fb9a09c7dfe30dfb6a3fed" Oct 31 05:15:05.682091 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:15:05.682073 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-965f61-29890395-nlstq_odh-ai-gateway-infra(e099ec30-86fb-43aa-9ee0-f3305f0adcfb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890395-nlstq" podUID="e099ec30-86fb-43aa-9ee0-f3305f0adcfb" Oct 31 05:15:05.683550 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:05.683526 5019 scope.go:122] "RemoveContainer" containerID="98529fed0b218751675ba9adfad1f32241f6083020fbd741ab137b4469a890fa" Oct 31 05:15:05.683709 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:15:05.683693 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890395-q6v97_odh-ai-gateway-infra(7621d02f-ad60-4445-be36-e7c253887716)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890395-q6v97" podUID="7621d02f-ad60-4445-be36-e7c253887716" Oct 31 05:15:08.581237 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:08.581200 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z"] Oct 31 05:15:08.581876 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:08.581841 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z" podUID="1132f111-a44a-4cc9-97bf-540c4f0dc0a5" containerName="istio-proxy" containerID="cri-o://43852a114edbd03c7597f3ad6140cb97401df73d73265964f13d5e6d8e0ac247" gracePeriod=30 Oct 31 05:15:12.065262 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:12.065229 5019 patch_prober.go:29] interesting pod/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 05:15:12.065668 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:12.065284 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z" podUID="1132f111-a44a-4cc9-97bf-540c4f0dc0a5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 05:15:13.655155 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:13.655126 5019 scope.go:122] "RemoveContainer" containerID="db6923af30f9e8ade71bc749eaf01be76cbc88d7be0aa26ec8206385de9a3009" Oct 31 05:15:13.774418 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:13.770157 5019 generic.go:356] "Generic (PLEG): container finished" podID="1132f111-a44a-4cc9-97bf-540c4f0dc0a5" containerID="43852a114edbd03c7597f3ad6140cb97401df73d73265964f13d5e6d8e0ac247" exitCode=0 Oct 31 05:15:13.774418 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:13.770207 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z" event={"ID":"1132f111-a44a-4cc9-97bf-540c4f0dc0a5","Type":"ContainerDied","Data":"43852a114edbd03c7597f3ad6140cb97401df73d73265964f13d5e6d8e0ac247"} Oct 31 05:15:13.774418 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:13.770250 5019 scope.go:122] "RemoveContainer" containerID="f3f42daf2ba8bd5b75932b0e38fac44b2c54c9ea6b230874a15f71a6d01cb65c" Oct 31 05:15:13.895472 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:13.895442 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z" Oct 31 05:15:13.947905 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:13.947877 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-workload-socket\") pod \"1132f111-a44a-4cc9-97bf-540c4f0dc0a5\" (UID: \"1132f111-a44a-4cc9-97bf-540c4f0dc0a5\") " Oct 31 05:15:13.948021 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:13.947928 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-workload-certs\") pod \"1132f111-a44a-4cc9-97bf-540c4f0dc0a5\" (UID: \"1132f111-a44a-4cc9-97bf-540c4f0dc0a5\") " Oct 31 05:15:13.948021 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:13.947951 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-credential-socket\") pod \"1132f111-a44a-4cc9-97bf-540c4f0dc0a5\" (UID: \"1132f111-a44a-4cc9-97bf-540c4f0dc0a5\") " Oct 31 05:15:13.948021 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:13.947975 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-istio-data\") pod \"1132f111-a44a-4cc9-97bf-540c4f0dc0a5\" (UID: \"1132f111-a44a-4cc9-97bf-540c4f0dc0a5\") " Oct 31 05:15:13.948183 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:13.948027 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-istio-token\" (UniqueName: \"kubernetes.io/projected/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-istio-token\") pod \"1132f111-a44a-4cc9-97bf-540c4f0dc0a5\" (UID: \"1132f111-a44a-4cc9-97bf-540c4f0dc0a5\") " Oct 31 05:15:13.948183 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:13.948084 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-istio-podinfo\") pod \"1132f111-a44a-4cc9-97bf-540c4f0dc0a5\" (UID: \"1132f111-a44a-4cc9-97bf-540c4f0dc0a5\") " Oct 31 05:15:13.948183 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:13.948155 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-workload-socket" pod "1132f111-a44a-4cc9-97bf-540c4f0dc0a5" (UID: "1132f111-a44a-4cc9-97bf-540c4f0dc0a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:15:13.948388 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:13.948214 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-istio-envoy\") pod \"1132f111-a44a-4cc9-97bf-540c4f0dc0a5\" (UID: \"1132f111-a44a-4cc9-97bf-540c4f0dc0a5\") " Oct 31 05:15:13.948388 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:13.948239 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-credential-socket" pod "1132f111-a44a-4cc9-97bf-540c4f0dc0a5" (UID: "1132f111-a44a-4cc9-97bf-540c4f0dc0a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:15:13.948388 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:13.948249 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-kube-api-access-4z2ws\" (UniqueName: \"kubernetes.io/projected/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-kube-api-access-4z2ws\") pod \"1132f111-a44a-4cc9-97bf-540c4f0dc0a5\" (UID: \"1132f111-a44a-4cc9-97bf-540c4f0dc0a5\") " Oct 31 05:15:13.948388 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:13.948264 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-workload-certs" pod "1132f111-a44a-4cc9-97bf-540c4f0dc0a5" (UID: "1132f111-a44a-4cc9-97bf-540c4f0dc0a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:15:13.948388 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:13.948313 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-istio-data" pod "1132f111-a44a-4cc9-97bf-540c4f0dc0a5" (UID: "1132f111-a44a-4cc9-97bf-540c4f0dc0a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:15:13.948388 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:13.948321 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-istiod-ca-cert\") pod \"1132f111-a44a-4cc9-97bf-540c4f0dc0a5\" (UID: \"1132f111-a44a-4cc9-97bf-540c4f0dc0a5\") " Oct 31 05:15:13.948739 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:13.948638 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-istiod-ca-cert" pod "1132f111-a44a-4cc9-97bf-540c4f0dc0a5" (UID: "1132f111-a44a-4cc9-97bf-540c4f0dc0a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:15:13.949042 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:13.949016 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:13.949133 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:13.949046 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:13.949133 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:13.949062 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:13.949133 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:13.949077 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:13.949133 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:13.949091 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:13.950405 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:13.950375 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-istio-envoy" pod "1132f111-a44a-4cc9-97bf-540c4f0dc0a5" (UID: "1132f111-a44a-4cc9-97bf-540c4f0dc0a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:15:13.950515 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:13.950474 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-istio-podinfo" pod "1132f111-a44a-4cc9-97bf-540c4f0dc0a5" (UID: "1132f111-a44a-4cc9-97bf-540c4f0dc0a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:15:13.950591 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:13.950518 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-istio-token" pod "1132f111-a44a-4cc9-97bf-540c4f0dc0a5" (UID: "1132f111-a44a-4cc9-97bf-540c4f0dc0a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:15:13.950591 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:13.950554 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-kube-api-access-4z2ws" pod "1132f111-a44a-4cc9-97bf-540c4f0dc0a5" (UID: "1132f111-a44a-4cc9-97bf-540c4f0dc0a5"). InnerVolumeSpecName "kube-api-access-4z2ws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:15:14.049867 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:14.049835 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:14.049867 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:14.049865 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z2ws\" (UniqueName: \"kubernetes.io/projected/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-kube-api-access-4z2ws\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:14.050048 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:14.049875 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:14.050048 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:14.049884 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1132f111-a44a-4cc9-97bf-540c4f0dc0a5-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:14.400001 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:14.399968 5019 scope.go:122] "RemoveContainer" containerID="8c9f0ef363613ae8eab592d35373a6c3335a6565cc952f7cad68481ec9450f3e" Oct 31 05:15:14.400186 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:14.400150 5019 scope.go:122] "RemoveContainer" containerID="7db6315e3c71fdb96e611d15a02eb664f80a0008d4408c355851cd6b89a071b0" Oct 31 05:15:14.780554 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:14.780526 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890395-fzdrx" event={"ID":"2030df55-524b-4764-ad64-002ab5d1cc52","Type":"ContainerStarted","Data":"609834ceb40bd054b788ad043230c8d3ee1f4d80a9214babeb73581d266e5342"} Oct 31 05:15:14.782235 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:14.782217 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z" Oct 31 05:15:14.782352 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:14.782272 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z" event={"ID":"1132f111-a44a-4cc9-97bf-540c4f0dc0a5","Type":"ContainerDied","Data":"08dc0ed097de9b499ddef102e066251079af4159272bbc97341d759c5c9ce033"} Oct 31 05:15:14.782397 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:14.782372 5019 scope.go:122] "RemoveContainer" containerID="43852a114edbd03c7597f3ad6140cb97401df73d73265964f13d5e6d8e0ac247" Oct 31 05:15:14.861884 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:14.861854 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z"] Oct 31 05:15:14.871209 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:14.871186 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-57dc6d-openshift-default-6c5bd8885c-dfg6z"] Oct 31 05:15:15.444827 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:15.444702 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1132f111-a44a-4cc9-97bf-540c4f0dc0a5" path="/var/lib/kubelet/pods/1132f111-a44a-4cc9-97bf-540c4f0dc0a5/volumes" Oct 31 05:15:15.445215 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:15.445192 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9633f0-54d5b879f5-qg5lc"] Oct 31 05:15:15.802021 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:15.801976 5019 generic.go:356] "Generic (PLEG): container finished" podID="fe68f52a-ede6-464c-9d03-5cc39f001cb8" containerID="abac4cd8c5b1d7d50086699c6721cf5299f70017620e49c730a2fa6cfca4fdd0" exitCode=60 Oct 31 05:15:15.802568 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:15.802139 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890395-gkgv4" event={"ID":"fe68f52a-ede6-464c-9d03-5cc39f001cb8","Type":"ContainerDied","Data":"abac4cd8c5b1d7d50086699c6721cf5299f70017620e49c730a2fa6cfca4fdd0"} Oct 31 05:15:15.802568 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:15.802254 5019 scope.go:122] "RemoveContainer" containerID="8c9f0ef363613ae8eab592d35373a6c3335a6565cc952f7cad68481ec9450f3e" Oct 31 05:15:15.802960 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:15.802938 5019 scope.go:122] "RemoveContainer" containerID="abac4cd8c5b1d7d50086699c6721cf5299f70017620e49c730a2fa6cfca4fdd0" Oct 31 05:15:15.803561 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:15:15.803512 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890395-gkgv4_odh-ai-gateway-infra(fe68f52a-ede6-464c-9d03-5cc39f001cb8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890395-gkgv4" podUID="fe68f52a-ede6-464c-9d03-5cc39f001cb8" Oct 31 05:15:15.805109 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:15.805087 5019 generic.go:356] "Generic (PLEG): container finished" podID="2030df55-524b-4764-ad64-002ab5d1cc52" containerID="609834ceb40bd054b788ad043230c8d3ee1f4d80a9214babeb73581d266e5342" exitCode=60 Oct 31 05:15:15.805226 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:15.805157 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890395-fzdrx" event={"ID":"2030df55-524b-4764-ad64-002ab5d1cc52","Type":"ContainerDied","Data":"609834ceb40bd054b788ad043230c8d3ee1f4d80a9214babeb73581d266e5342"} Oct 31 05:15:15.805503 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:15.805488 5019 scope.go:122] "RemoveContainer" containerID="609834ceb40bd054b788ad043230c8d3ee1f4d80a9214babeb73581d266e5342" Oct 31 05:15:15.805775 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:15:15.805746 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-8973e2-29890395-fzdrx_odh-ai-gateway-infra(2030df55-524b-4764-ad64-002ab5d1cc52)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890395-fzdrx" podUID="2030df55-524b-4764-ad64-002ab5d1cc52" Oct 31 05:15:15.816194 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:15.816177 5019 scope.go:122] "RemoveContainer" containerID="7db6315e3c71fdb96e611d15a02eb664f80a0008d4408c355851cd6b89a071b0" Oct 31 05:15:15.833115 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:15.833094 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890395-fzdrx"] Oct 31 05:15:16.400137 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:16.400109 5019 scope.go:122] "RemoveContainer" containerID="1d19599f98afe55f0baddf1b09957e1bc5e1abcbc5c3cee362e514fe09fe6a99" Oct 31 05:15:16.400370 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:16.400194 5019 scope.go:122] "RemoveContainer" containerID="0756d61698e203ae0cdbe36c099595dcc246cb5ff2cfe793827ccbdeed720107" Oct 31 05:15:16.400477 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:16.400457 5019 scope.go:122] "RemoveContainer" containerID="120474799df77357f725ea2220e435840e2fc0a163fb9a09c7dfe30dfb6a3fed" Oct 31 05:15:16.855299 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:16.855269 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890395-gkgv4"] Oct 31 05:15:17.081027 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:17.080227 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890395-fzdrx" Oct 31 05:15:17.085066 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:17.085041 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890395-gkgv4" Oct 31 05:15:17.230348 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:17.230307 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2030df55-524b-4764-ad64-002ab5d1cc52-kube-api-access-jm2td\" (UniqueName: \"kubernetes.io/projected/2030df55-524b-4764-ad64-002ab5d1cc52-kube-api-access-jm2td\") pod \"2030df55-524b-4764-ad64-002ab5d1cc52\" (UID: \"2030df55-524b-4764-ad64-002ab5d1cc52\") " Oct 31 05:15:17.230548 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:17.230374 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe68f52a-ede6-464c-9d03-5cc39f001cb8-kube-api-access-hcdmz\" (UniqueName: \"kubernetes.io/projected/fe68f52a-ede6-464c-9d03-5cc39f001cb8-kube-api-access-hcdmz\") pod \"fe68f52a-ede6-464c-9d03-5cc39f001cb8\" (UID: \"fe68f52a-ede6-464c-9d03-5cc39f001cb8\") " Oct 31 05:15:17.230548 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:17.230461 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe68f52a-ede6-464c-9d03-5cc39f001cb8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe68f52a-ede6-464c-9d03-5cc39f001cb8-maas-api-service-ca\") pod \"fe68f52a-ede6-464c-9d03-5cc39f001cb8\" (UID: \"fe68f52a-ede6-464c-9d03-5cc39f001cb8\") " Oct 31 05:15:17.230548 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:17.230499 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2030df55-524b-4764-ad64-002ab5d1cc52-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2030df55-524b-4764-ad64-002ab5d1cc52-maas-api-service-ca\") pod \"2030df55-524b-4764-ad64-002ab5d1cc52\" (UID: \"2030df55-524b-4764-ad64-002ab5d1cc52\") " Oct 31 05:15:17.231012 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:17.230980 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe68f52a-ede6-464c-9d03-5cc39f001cb8-maas-api-service-ca" pod "fe68f52a-ede6-464c-9d03-5cc39f001cb8" (UID: "fe68f52a-ede6-464c-9d03-5cc39f001cb8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:15:17.231161 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:17.231026 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2030df55-524b-4764-ad64-002ab5d1cc52-maas-api-service-ca" pod "2030df55-524b-4764-ad64-002ab5d1cc52" (UID: "2030df55-524b-4764-ad64-002ab5d1cc52"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:15:17.232608 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:17.232580 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe68f52a-ede6-464c-9d03-5cc39f001cb8-kube-api-access-hcdmz" pod "fe68f52a-ede6-464c-9d03-5cc39f001cb8" (UID: "fe68f52a-ede6-464c-9d03-5cc39f001cb8"). InnerVolumeSpecName "kube-api-access-hcdmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:15:17.232880 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:17.232863 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2030df55-524b-4764-ad64-002ab5d1cc52-kube-api-access-jm2td" pod "2030df55-524b-4764-ad64-002ab5d1cc52" (UID: "2030df55-524b-4764-ad64-002ab5d1cc52"). InnerVolumeSpecName "kube-api-access-jm2td". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:15:17.331730 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:17.331699 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe68f52a-ede6-464c-9d03-5cc39f001cb8-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:17.331730 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:17.331727 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2030df55-524b-4764-ad64-002ab5d1cc52-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:17.331730 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:17.331736 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jm2td\" (UniqueName: \"kubernetes.io/projected/2030df55-524b-4764-ad64-002ab5d1cc52-kube-api-access-jm2td\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:17.331959 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:17.331748 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcdmz\" (UniqueName: \"kubernetes.io/projected/fe68f52a-ede6-464c-9d03-5cc39f001cb8-kube-api-access-hcdmz\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:17.401061 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:17.400531 5019 scope.go:122] "RemoveContainer" containerID="178ba3d5a2556f2b5cb3b47330ef05396050abc0453caf1f5c13736fa54dafa2" Oct 31 05:15:17.405583 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:17.405560 5019 scope.go:122] "RemoveContainer" containerID="98529fed0b218751675ba9adfad1f32241f6083020fbd741ab137b4469a890fa" Oct 31 05:15:17.826513 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:17.826488 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890395-gkgv4" Oct 31 05:15:17.826513 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:17.826498 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890395-gkgv4" event={"ID":"fe68f52a-ede6-464c-9d03-5cc39f001cb8","Type":"ContainerDied","Data":"468b40768c21e112819f32abee5db73f26f3f1ef58f9c1d3f8a14d82f76d2482"} Oct 31 05:15:17.826698 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:17.826567 5019 scope.go:122] "RemoveContainer" containerID="abac4cd8c5b1d7d50086699c6721cf5299f70017620e49c730a2fa6cfca4fdd0" Oct 31 05:15:17.828871 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:17.828845 5019 generic.go:356] "Generic (PLEG): container finished" podID="8e80ef6a-aee2-4040-a8dc-ca294ff2409c" containerID="4b83c284d6ed26632065daae30a8a86104a69d99a17f52833674b230a02c541d" exitCode=60 Oct 31 05:15:17.829007 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:17.828936 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890395-vm7v4" event={"ID":"8e80ef6a-aee2-4040-a8dc-ca294ff2409c","Type":"ContainerDied","Data":"4b83c284d6ed26632065daae30a8a86104a69d99a17f52833674b230a02c541d"} Oct 31 05:15:17.829374 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:17.829322 5019 scope.go:122] "RemoveContainer" containerID="4b83c284d6ed26632065daae30a8a86104a69d99a17f52833674b230a02c541d" Oct 31 05:15:17.829656 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:15:17.829633 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c4006d-29890395-vm7v4_odh-ai-gateway-infra(8e80ef6a-aee2-4040-a8dc-ca294ff2409c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890395-vm7v4" podUID="8e80ef6a-aee2-4040-a8dc-ca294ff2409c" Oct 31 05:15:17.830431 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:17.830410 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890395-fzdrx" Oct 31 05:15:17.830431 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:17.830423 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890395-fzdrx" event={"ID":"2030df55-524b-4764-ad64-002ab5d1cc52","Type":"ContainerDied","Data":"985f3798fb5207ac539b85a6bab968558c3375ada3e148e60c8de1d7ec7ba6a3"} Oct 31 05:15:17.832561 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:17.832531 5019 generic.go:356] "Generic (PLEG): container finished" podID="e099ec30-86fb-43aa-9ee0-f3305f0adcfb" containerID="b9b6b22ce8be5b17faa56fe999af575415d9becb51737e7f83b1f80738f13861" exitCode=60 Oct 31 05:15:17.832686 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:17.832580 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890395-nlstq" event={"ID":"e099ec30-86fb-43aa-9ee0-f3305f0adcfb","Type":"ContainerDied","Data":"b9b6b22ce8be5b17faa56fe999af575415d9becb51737e7f83b1f80738f13861"} Oct 31 05:15:17.832992 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:17.832973 5019 scope.go:122] "RemoveContainer" containerID="b9b6b22ce8be5b17faa56fe999af575415d9becb51737e7f83b1f80738f13861" Oct 31 05:15:17.833230 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:15:17.833212 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-965f61-29890395-nlstq_odh-ai-gateway-infra(e099ec30-86fb-43aa-9ee0-f3305f0adcfb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890395-nlstq" podUID="e099ec30-86fb-43aa-9ee0-f3305f0adcfb" Oct 31 05:15:17.835055 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:17.835030 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890395-zd2lr" event={"ID":"754e0f92-f1dd-40a2-9c1f-031c383b4629","Type":"ContainerStarted","Data":"db2d14e4e07f918861d19b9be1aea2a664afa02a7c42048b5988bf2fd0524a8e"} Oct 31 05:15:17.836872 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:17.836852 5019 generic.go:356] "Generic (PLEG): container finished" podID="a4e14c11-5250-4d70-932f-055294e67dd4" containerID="8c933996ca50740bca2cac61b5c40b5af48e75e1001a0d97af04b47a58ccedaa" exitCode=60 Oct 31 05:15:17.836992 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:17.836891 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890395-zjqjv" event={"ID":"a4e14c11-5250-4d70-932f-055294e67dd4","Type":"ContainerDied","Data":"8c933996ca50740bca2cac61b5c40b5af48e75e1001a0d97af04b47a58ccedaa"} Oct 31 05:15:17.837225 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:17.837207 5019 scope.go:122] "RemoveContainer" containerID="8c933996ca50740bca2cac61b5c40b5af48e75e1001a0d97af04b47a58ccedaa" Oct 31 05:15:17.837460 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:15:17.837439 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890395-zjqjv_odh-ai-gateway-infra(a4e14c11-5250-4d70-932f-055294e67dd4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890395-zjqjv" podUID="a4e14c11-5250-4d70-932f-055294e67dd4" Oct 31 05:15:17.837836 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:17.837821 5019 scope.go:122] "RemoveContainer" containerID="1d19599f98afe55f0baddf1b09957e1bc5e1abcbc5c3cee362e514fe09fe6a99" Oct 31 05:15:17.852590 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:17.852573 5019 scope.go:122] "RemoveContainer" containerID="609834ceb40bd054b788ad043230c8d3ee1f4d80a9214babeb73581d266e5342" Oct 31 05:15:17.861819 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:17.861793 5019 scope.go:122] "RemoveContainer" containerID="120474799df77357f725ea2220e435840e2fc0a163fb9a09c7dfe30dfb6a3fed" Oct 31 05:15:17.881377 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:17.881352 5019 scope.go:122] "RemoveContainer" containerID="0756d61698e203ae0cdbe36c099595dcc246cb5ff2cfe793827ccbdeed720107" Oct 31 05:15:17.884381 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:17.884354 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890395-gkgv4"] Oct 31 05:15:17.894461 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:17.894435 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-25f56e-29890395-gkgv4"] Oct 31 05:15:18.053487 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:18.053408 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890395-fzdrx"] Oct 31 05:15:18.054671 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:18.054647 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8973e2-29890395-fzdrx"] Oct 31 05:15:18.400381 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:18.400352 5019 scope.go:122] "RemoveContainer" containerID="7b3d25e64458f963a6a523ab9e1c98679cbc4648fde6d6c29be9a6ac7f2f8d4e" Oct 31 05:15:18.400856 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:18.400421 5019 scope.go:122] "RemoveContainer" containerID="976503fc1e3de8b719fee54b69923f27c5f84e8c3a458673dcd6c87b45e981da" Oct 31 05:15:18.859474 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:18.859442 5019 generic.go:356] "Generic (PLEG): container finished" podID="754e0f92-f1dd-40a2-9c1f-031c383b4629" containerID="db2d14e4e07f918861d19b9be1aea2a664afa02a7c42048b5988bf2fd0524a8e" exitCode=60 Oct 31 05:15:18.859651 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:18.859517 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890395-zd2lr" event={"ID":"754e0f92-f1dd-40a2-9c1f-031c383b4629","Type":"ContainerDied","Data":"db2d14e4e07f918861d19b9be1aea2a664afa02a7c42048b5988bf2fd0524a8e"} Oct 31 05:15:18.859651 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:18.859621 5019 scope.go:122] "RemoveContainer" containerID="178ba3d5a2556f2b5cb3b47330ef05396050abc0453caf1f5c13736fa54dafa2" Oct 31 05:15:18.859931 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:18.859911 5019 scope.go:122] "RemoveContainer" containerID="db2d14e4e07f918861d19b9be1aea2a664afa02a7c42048b5988bf2fd0524a8e" Oct 31 05:15:18.860195 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:15:18.860174 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890395-zd2lr_odh-ai-gateway-infra(754e0f92-f1dd-40a2-9c1f-031c383b4629)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890395-zd2lr" podUID="754e0f92-f1dd-40a2-9c1f-031c383b4629" Oct 31 05:15:18.866493 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:18.866470 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890395-4sfpv" event={"ID":"ed220fc2-8244-47bd-a868-472210c5e521","Type":"ContainerStarted","Data":"b7ca793f4ecd2df1d78a52b39606c11d910fd4f623a3a4ea658cd40f3a6b5830"} Oct 31 05:15:18.869974 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:18.869948 5019 generic.go:356] "Generic (PLEG): container finished" podID="7621d02f-ad60-4445-be36-e7c253887716" containerID="526b9efab2c265ba9195a06722a6ede2efc5cbbc9f4d231bf52683c0875eb31f" exitCode=60 Oct 31 05:15:18.870097 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:18.870047 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890395-q6v97" event={"ID":"7621d02f-ad60-4445-be36-e7c253887716","Type":"ContainerDied","Data":"526b9efab2c265ba9195a06722a6ede2efc5cbbc9f4d231bf52683c0875eb31f"} Oct 31 05:15:18.870388 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:18.870372 5019 scope.go:122] "RemoveContainer" containerID="526b9efab2c265ba9195a06722a6ede2efc5cbbc9f4d231bf52683c0875eb31f" Oct 31 05:15:18.870590 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:15:18.870562 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890395-q6v97_odh-ai-gateway-infra(7621d02f-ad60-4445-be36-e7c253887716)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890395-q6v97" podUID="7621d02f-ad60-4445-be36-e7c253887716" Oct 31 05:15:18.875780 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:18.875757 5019 scope.go:122] "RemoveContainer" containerID="98529fed0b218751675ba9adfad1f32241f6083020fbd741ab137b4469a890fa" Oct 31 05:15:18.947212 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:18.947147 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890395-zd2lr"] Oct 31 05:15:18.983084 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:18.983057 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890395-nlstq"] Oct 31 05:15:19.025202 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:19.024044 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890395-zjqjv"] Oct 31 05:15:19.111484 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:19.111449 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890395-vm7v4"] Oct 31 05:15:19.219521 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:19.218749 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890395-nlstq" Oct 31 05:15:19.258064 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:19.258037 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-67f7j"] Oct 31 05:15:19.260702 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:19.260435 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-67f7j" podUID="9573de57-8748-44eb-8949-b55b7dca0043" containerName="istio-proxy" containerID="cri-o://e446e225b4891af314659175ba11f9a40c9491146968b525a8fc11608e13f303" gracePeriod=30 Oct 31 05:15:19.262113 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:19.261738 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rdnft"] Oct 31 05:15:19.262113 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:19.261929 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rdnft" podUID="5fc2a87b-394a-4c26-adfd-9a57a1fc463b" containerName="istio-proxy" containerID="cri-o://6d58868d6eb7edd1c2938c35528b8a887945575e704d2f8158d3d9a1012a1f14" gracePeriod=30 Oct 31 05:15:19.268354 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:19.268309 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7"] Oct 31 05:15:19.268594 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:19.268562 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7" podUID="e0ccc03e-0e5e-40b7-8a40-718f199dd747" containerName="istio-proxy" containerID="cri-o://2e4684a3042b06f9b8f590a39685056778deeec1cd39a10a0daf25c952f3b422" gracePeriod=30 Oct 31 05:15:19.288649 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:19.288620 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890395-zjqjv" Oct 31 05:15:19.295559 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:19.295431 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq"] Oct 31 05:15:19.295705 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:19.295650 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq" podUID="1cf9a806-d96f-478b-9ce1-2e67ec594739" containerName="istio-proxy" containerID="cri-o://1622cd76a3f47c2dc335a11e6857175d67f351df07c11d04fab8ce562097102b" gracePeriod=30 Oct 31 05:15:19.361429 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:19.361404 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890395-vm7v4" Oct 31 05:15:19.367410 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:19.367382 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e099ec30-86fb-43aa-9ee0-f3305f0adcfb-kube-api-access-rb54g\" (UniqueName: \"kubernetes.io/projected/e099ec30-86fb-43aa-9ee0-f3305f0adcfb-kube-api-access-rb54g\") pod \"e099ec30-86fb-43aa-9ee0-f3305f0adcfb\" (UID: \"e099ec30-86fb-43aa-9ee0-f3305f0adcfb\") " Oct 31 05:15:19.367717 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:19.367688 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e099ec30-86fb-43aa-9ee0-f3305f0adcfb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e099ec30-86fb-43aa-9ee0-f3305f0adcfb-maas-api-service-ca\") pod \"e099ec30-86fb-43aa-9ee0-f3305f0adcfb\" (UID: \"e099ec30-86fb-43aa-9ee0-f3305f0adcfb\") " Oct 31 05:15:19.367919 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:19.367746 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4e14c11-5250-4d70-932f-055294e67dd4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4e14c11-5250-4d70-932f-055294e67dd4-maas-api-service-ca\") pod \"a4e14c11-5250-4d70-932f-055294e67dd4\" (UID: \"a4e14c11-5250-4d70-932f-055294e67dd4\") " Oct 31 05:15:19.367919 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:19.367838 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4e14c11-5250-4d70-932f-055294e67dd4-kube-api-access-dkzxd\" (UniqueName: \"kubernetes.io/projected/a4e14c11-5250-4d70-932f-055294e67dd4-kube-api-access-dkzxd\") pod \"a4e14c11-5250-4d70-932f-055294e67dd4\" (UID: \"a4e14c11-5250-4d70-932f-055294e67dd4\") " Oct 31 05:15:19.369287 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:19.369259 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e099ec30-86fb-43aa-9ee0-f3305f0adcfb-maas-api-service-ca" pod "e099ec30-86fb-43aa-9ee0-f3305f0adcfb" (UID: "e099ec30-86fb-43aa-9ee0-f3305f0adcfb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:15:19.369451 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:19.369302 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4e14c11-5250-4d70-932f-055294e67dd4-maas-api-service-ca" pod "a4e14c11-5250-4d70-932f-055294e67dd4" (UID: "a4e14c11-5250-4d70-932f-055294e67dd4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:15:19.380447 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:19.380359 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e099ec30-86fb-43aa-9ee0-f3305f0adcfb-kube-api-access-rb54g" pod "e099ec30-86fb-43aa-9ee0-f3305f0adcfb" (UID: "e099ec30-86fb-43aa-9ee0-f3305f0adcfb"). InnerVolumeSpecName "kube-api-access-rb54g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:15:19.381457 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:19.381311 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4e14c11-5250-4d70-932f-055294e67dd4-kube-api-access-dkzxd" pod "a4e14c11-5250-4d70-932f-055294e67dd4" (UID: "a4e14c11-5250-4d70-932f-055294e67dd4"). InnerVolumeSpecName "kube-api-access-dkzxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:15:19.417092 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:19.413398 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2030df55-524b-4764-ad64-002ab5d1cc52" path="/var/lib/kubelet/pods/2030df55-524b-4764-ad64-002ab5d1cc52/volumes" Oct 31 05:15:19.417092 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:19.414075 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe68f52a-ede6-464c-9d03-5cc39f001cb8" path="/var/lib/kubelet/pods/fe68f52a-ede6-464c-9d03-5cc39f001cb8/volumes" Oct 31 05:15:19.469302 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:19.469278 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e80ef6a-aee2-4040-a8dc-ca294ff2409c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e80ef6a-aee2-4040-a8dc-ca294ff2409c-maas-api-service-ca\") pod \"8e80ef6a-aee2-4040-a8dc-ca294ff2409c\" (UID: \"8e80ef6a-aee2-4040-a8dc-ca294ff2409c\") " Oct 31 05:15:19.469528 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:19.469479 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e80ef6a-aee2-4040-a8dc-ca294ff2409c-kube-api-access-xk2br\" (UniqueName: \"kubernetes.io/projected/8e80ef6a-aee2-4040-a8dc-ca294ff2409c-kube-api-access-xk2br\") pod \"8e80ef6a-aee2-4040-a8dc-ca294ff2409c\" (UID: \"8e80ef6a-aee2-4040-a8dc-ca294ff2409c\") " Oct 31 05:15:19.469795 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:19.469769 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e80ef6a-aee2-4040-a8dc-ca294ff2409c-maas-api-service-ca" pod "8e80ef6a-aee2-4040-a8dc-ca294ff2409c" (UID: "8e80ef6a-aee2-4040-a8dc-ca294ff2409c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:15:19.469949 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:19.469930 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkzxd\" (UniqueName: \"kubernetes.io/projected/a4e14c11-5250-4d70-932f-055294e67dd4-kube-api-access-dkzxd\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:19.470025 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:19.469954 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e80ef6a-aee2-4040-a8dc-ca294ff2409c-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:19.470025 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:19.469968 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb54g\" (UniqueName: \"kubernetes.io/projected/e099ec30-86fb-43aa-9ee0-f3305f0adcfb-kube-api-access-rb54g\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:19.470025 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:19.469982 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e099ec30-86fb-43aa-9ee0-f3305f0adcfb-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:19.470025 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:19.469997 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4e14c11-5250-4d70-932f-055294e67dd4-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:19.487563 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:19.485537 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e80ef6a-aee2-4040-a8dc-ca294ff2409c-kube-api-access-xk2br" pod "8e80ef6a-aee2-4040-a8dc-ca294ff2409c" (UID: "8e80ef6a-aee2-4040-a8dc-ca294ff2409c"). InnerVolumeSpecName "kube-api-access-xk2br". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:15:19.571031 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:19.570992 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk2br\" (UniqueName: \"kubernetes.io/projected/8e80ef6a-aee2-4040-a8dc-ca294ff2409c-kube-api-access-xk2br\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:19.879566 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:19.879484 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890395-zjqjv" Oct 31 05:15:19.879981 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:19.879497 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890395-zjqjv" event={"ID":"a4e14c11-5250-4d70-932f-055294e67dd4","Type":"ContainerDied","Data":"8c1c2712180235a3c1a61958b197c0af9cda92b10df56c6dea63584173b5d837"} Oct 31 05:15:19.879981 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:19.879915 5019 scope.go:122] "RemoveContainer" containerID="8c933996ca50740bca2cac61b5c40b5af48e75e1001a0d97af04b47a58ccedaa" Oct 31 05:15:19.883580 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:19.883549 5019 generic.go:356] "Generic (PLEG): container finished" podID="1873945a-f1ed-43b2-b787-c29a1783b3e3" containerID="57c6a261f4486994f430fd946c82d2f8bf5c739a7bf3bd24bd79aae3ccb270c6" exitCode=60 Oct 31 05:15:19.883728 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:19.883624 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890395-8fb6v" event={"ID":"1873945a-f1ed-43b2-b787-c29a1783b3e3","Type":"ContainerDied","Data":"57c6a261f4486994f430fd946c82d2f8bf5c739a7bf3bd24bd79aae3ccb270c6"} Oct 31 05:15:19.884322 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:19.884297 5019 scope.go:122] "RemoveContainer" containerID="57c6a261f4486994f430fd946c82d2f8bf5c739a7bf3bd24bd79aae3ccb270c6" Oct 31 05:15:19.884632 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:15:19.884604 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-afc519-29890395-8fb6v_odh-ai-gateway-infra(1873945a-f1ed-43b2-b787-c29a1783b3e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890395-8fb6v" podUID="1873945a-f1ed-43b2-b787-c29a1783b3e3" Oct 31 05:15:19.885603 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:19.885577 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890395-vm7v4" event={"ID":"8e80ef6a-aee2-4040-a8dc-ca294ff2409c","Type":"ContainerDied","Data":"02f9784b2ebd34e0d7aa2c683c16601c4ecc49d09b601c49c0c7db87c7a422b9"} Oct 31 05:15:19.885716 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:19.885668 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890395-vm7v4" Oct 31 05:15:19.889234 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:19.888527 5019 generic.go:356] "Generic (PLEG): container finished" podID="ed220fc2-8244-47bd-a868-472210c5e521" containerID="b7ca793f4ecd2df1d78a52b39606c11d910fd4f623a3a4ea658cd40f3a6b5830" exitCode=60 Oct 31 05:15:19.889234 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:19.888608 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890395-4sfpv" event={"ID":"ed220fc2-8244-47bd-a868-472210c5e521","Type":"ContainerDied","Data":"b7ca793f4ecd2df1d78a52b39606c11d910fd4f623a3a4ea658cd40f3a6b5830"} Oct 31 05:15:19.889234 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:19.888939 5019 scope.go:122] "RemoveContainer" containerID="b7ca793f4ecd2df1d78a52b39606c11d910fd4f623a3a4ea658cd40f3a6b5830" Oct 31 05:15:19.889234 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:15:19.889168 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-766728-29890395-4sfpv_odh-ai-gateway-infra(ed220fc2-8244-47bd-a868-472210c5e521)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890395-4sfpv" podUID="ed220fc2-8244-47bd-a868-472210c5e521" Oct 31 05:15:19.891448 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:19.890843 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890395-nlstq" event={"ID":"e099ec30-86fb-43aa-9ee0-f3305f0adcfb","Type":"ContainerDied","Data":"561053ccae4be041cca41f68202fa808469e67936a1100155a81ae0f17556fc7"} Oct 31 05:15:19.891448 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:19.890932 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890395-nlstq" Oct 31 05:15:19.896852 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:19.896758 5019 scope.go:122] "RemoveContainer" containerID="7b3d25e64458f963a6a523ab9e1c98679cbc4648fde6d6c29be9a6ac7f2f8d4e" Oct 31 05:15:19.915273 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:19.915244 5019 scope.go:122] "RemoveContainer" containerID="4b83c284d6ed26632065daae30a8a86104a69d99a17f52833674b230a02c541d" Oct 31 05:15:19.927026 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:19.926996 5019 scope.go:122] "RemoveContainer" containerID="976503fc1e3de8b719fee54b69923f27c5f84e8c3a458673dcd6c87b45e981da" Oct 31 05:15:19.942885 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:19.942861 5019 scope.go:122] "RemoveContainer" containerID="b9b6b22ce8be5b17faa56fe999af575415d9becb51737e7f83b1f80738f13861" Oct 31 05:15:19.950224 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:19.950199 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890395-q6v97"] Oct 31 05:15:19.985500 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:19.985465 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890395-zjqjv"] Oct 31 05:15:19.990750 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:19.990723 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9633f0-29890395-zjqjv"] Oct 31 05:15:20.012645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:20.012617 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890395-4sfpv"] Oct 31 05:15:20.106410 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:20.105254 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890395-zd2lr" Oct 31 05:15:20.112758 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:20.112715 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890395-vm7v4"] Oct 31 05:15:20.131424 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:20.122814 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890395-vm7v4"] Oct 31 05:15:20.145712 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:20.145690 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890395-q6v97" Oct 31 05:15:20.179403 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:20.179377 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/754e0f92-f1dd-40a2-9c1f-031c383b4629-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/754e0f92-f1dd-40a2-9c1f-031c383b4629-maas-api-service-ca\") pod \"754e0f92-f1dd-40a2-9c1f-031c383b4629\" (UID: \"754e0f92-f1dd-40a2-9c1f-031c383b4629\") " Oct 31 05:15:20.179678 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:20.179654 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/754e0f92-f1dd-40a2-9c1f-031c383b4629-kube-api-access-jjqnm\" (UniqueName: \"kubernetes.io/projected/754e0f92-f1dd-40a2-9c1f-031c383b4629-kube-api-access-jjqnm\") pod \"754e0f92-f1dd-40a2-9c1f-031c383b4629\" (UID: \"754e0f92-f1dd-40a2-9c1f-031c383b4629\") " Oct 31 05:15:20.179771 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:20.179752 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/754e0f92-f1dd-40a2-9c1f-031c383b4629-maas-api-service-ca" pod "754e0f92-f1dd-40a2-9c1f-031c383b4629" (UID: "754e0f92-f1dd-40a2-9c1f-031c383b4629"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:15:20.180054 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:20.180037 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/754e0f92-f1dd-40a2-9c1f-031c383b4629-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:20.181783 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:20.181755 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/754e0f92-f1dd-40a2-9c1f-031c383b4629-kube-api-access-jjqnm" pod "754e0f92-f1dd-40a2-9c1f-031c383b4629" (UID: "754e0f92-f1dd-40a2-9c1f-031c383b4629"). InnerVolumeSpecName "kube-api-access-jjqnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:15:20.188906 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:20.188879 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890395-nlstq"] Oct 31 05:15:20.199351 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:20.198574 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890395-nlstq"] Oct 31 05:15:20.281004 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:20.280970 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7621d02f-ad60-4445-be36-e7c253887716-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7621d02f-ad60-4445-be36-e7c253887716-maas-api-service-ca\") pod \"7621d02f-ad60-4445-be36-e7c253887716\" (UID: \"7621d02f-ad60-4445-be36-e7c253887716\") " Oct 31 05:15:20.281222 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:20.281199 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7621d02f-ad60-4445-be36-e7c253887716-kube-api-access-clzp2\" (UniqueName: \"kubernetes.io/projected/7621d02f-ad60-4445-be36-e7c253887716-kube-api-access-clzp2\") pod \"7621d02f-ad60-4445-be36-e7c253887716\" (UID: \"7621d02f-ad60-4445-be36-e7c253887716\") " Oct 31 05:15:20.281340 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:20.281296 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7621d02f-ad60-4445-be36-e7c253887716-maas-api-service-ca" pod "7621d02f-ad60-4445-be36-e7c253887716" (UID: "7621d02f-ad60-4445-be36-e7c253887716"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:15:20.281693 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:20.281662 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjqnm\" (UniqueName: \"kubernetes.io/projected/754e0f92-f1dd-40a2-9c1f-031c383b4629-kube-api-access-jjqnm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:20.281829 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:20.281769 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7621d02f-ad60-4445-be36-e7c253887716-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:20.283971 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:20.283945 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7621d02f-ad60-4445-be36-e7c253887716-kube-api-access-clzp2" pod "7621d02f-ad60-4445-be36-e7c253887716" (UID: "7621d02f-ad60-4445-be36-e7c253887716"). InnerVolumeSpecName "kube-api-access-clzp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:15:20.382649 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:20.382572 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clzp2\" (UniqueName: \"kubernetes.io/projected/7621d02f-ad60-4445-be36-e7c253887716-kube-api-access-clzp2\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:20.453473 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:20.449628 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9633f0-54d5b879f5-qg5lc" podUID="e1803197-fd95-4f13-9cdb-4a9386cf9610" containerName="payload-pre-processing" containerID="cri-o://40c212a00c030b2dc932f08ec578be766e64d3566dde30b251214d9f0e7961f0" gracePeriod=25 Oct 31 05:15:20.918524 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:20.918493 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890395-q6v97" Oct 31 05:15:20.918925 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:20.918542 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890395-q6v97" event={"ID":"7621d02f-ad60-4445-be36-e7c253887716","Type":"ContainerDied","Data":"250cc312a61a8e0d0157734f32d19cba6ce5565bbfdbc1337342e67bdb119d4b"} Oct 31 05:15:20.918925 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:20.918596 5019 scope.go:122] "RemoveContainer" containerID="526b9efab2c265ba9195a06722a6ede2efc5cbbc9f4d231bf52683c0875eb31f" Oct 31 05:15:20.920171 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:20.920149 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890395-zd2lr" Oct 31 05:15:20.920278 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:20.920180 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890395-zd2lr" event={"ID":"754e0f92-f1dd-40a2-9c1f-031c383b4629","Type":"ContainerDied","Data":"174dbf52d30ec889e266b300bca342e63a48d6ee514ad6c241cb982675a72378"} Oct 31 05:15:20.931164 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:20.931143 5019 scope.go:122] "RemoveContainer" containerID="db2d14e4e07f918861d19b9be1aea2a664afa02a7c42048b5988bf2fd0524a8e" Oct 31 05:15:20.999353 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:20.998665 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890395-zd2lr"] Oct 31 05:15:21.005247 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.004588 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-338f2e-29890395-zd2lr"] Oct 31 05:15:21.037912 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.037882 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890395-q6v97"] Oct 31 05:15:21.043637 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.043612 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2a372-29890395-q6v97"] Oct 31 05:15:21.052757 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.052733 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890395-8fb6v"] Oct 31 05:15:21.087990 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.087959 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890395-4sfpv" Oct 31 05:15:21.193405 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.192927 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed220fc2-8244-47bd-a868-472210c5e521-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed220fc2-8244-47bd-a868-472210c5e521-maas-api-service-ca\") pod \"ed220fc2-8244-47bd-a868-472210c5e521\" (UID: \"ed220fc2-8244-47bd-a868-472210c5e521\") " Oct 31 05:15:21.193405 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.192978 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed220fc2-8244-47bd-a868-472210c5e521-kube-api-access-bvrms\" (UniqueName: \"kubernetes.io/projected/ed220fc2-8244-47bd-a868-472210c5e521-kube-api-access-bvrms\") pod \"ed220fc2-8244-47bd-a868-472210c5e521\" (UID: \"ed220fc2-8244-47bd-a868-472210c5e521\") " Oct 31 05:15:21.193405 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.193375 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed220fc2-8244-47bd-a868-472210c5e521-maas-api-service-ca" pod "ed220fc2-8244-47bd-a868-472210c5e521" (UID: "ed220fc2-8244-47bd-a868-472210c5e521"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:15:21.193710 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.193515 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed220fc2-8244-47bd-a868-472210c5e521-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:21.199916 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.195728 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed220fc2-8244-47bd-a868-472210c5e521-kube-api-access-bvrms" pod "ed220fc2-8244-47bd-a868-472210c5e521" (UID: "ed220fc2-8244-47bd-a868-472210c5e521"). InnerVolumeSpecName "kube-api-access-bvrms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:15:21.228445 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.228423 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890395-8fb6v" Oct 31 05:15:21.294279 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.294246 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1873945a-f1ed-43b2-b787-c29a1783b3e3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1873945a-f1ed-43b2-b787-c29a1783b3e3-maas-api-service-ca\") pod \"1873945a-f1ed-43b2-b787-c29a1783b3e3\" (UID: \"1873945a-f1ed-43b2-b787-c29a1783b3e3\") " Oct 31 05:15:21.294443 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.294288 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1873945a-f1ed-43b2-b787-c29a1783b3e3-kube-api-access-7dfnc\" (UniqueName: \"kubernetes.io/projected/1873945a-f1ed-43b2-b787-c29a1783b3e3-kube-api-access-7dfnc\") pod \"1873945a-f1ed-43b2-b787-c29a1783b3e3\" (UID: \"1873945a-f1ed-43b2-b787-c29a1783b3e3\") " Oct 31 05:15:21.294720 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.294691 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1873945a-f1ed-43b2-b787-c29a1783b3e3-maas-api-service-ca" pod "1873945a-f1ed-43b2-b787-c29a1783b3e3" (UID: "1873945a-f1ed-43b2-b787-c29a1783b3e3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:15:21.294963 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.294933 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvrms\" (UniqueName: \"kubernetes.io/projected/ed220fc2-8244-47bd-a868-472210c5e521-kube-api-access-bvrms\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:21.295066 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.294962 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1873945a-f1ed-43b2-b787-c29a1783b3e3-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:21.296558 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.296536 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1873945a-f1ed-43b2-b787-c29a1783b3e3-kube-api-access-7dfnc" pod "1873945a-f1ed-43b2-b787-c29a1783b3e3" (UID: "1873945a-f1ed-43b2-b787-c29a1783b3e3"). InnerVolumeSpecName "kube-api-access-7dfnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:15:21.356423 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.356394 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91889c70-sjqvh"] Oct 31 05:15:21.357009 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.356994 5019 cpu_manager.go:414] "containerMap: removing container" podUID="ed220fc2-8244-47bd-a868-472210c5e521" containerName="cleanup" Oct 31 05:15:21.357065 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357009 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed220fc2-8244-47bd-a868-472210c5e521" containerName="cleanup" Oct 31 05:15:21.357065 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357021 5019 cpu_manager.go:414] "containerMap: removing container" podUID="fe68f52a-ede6-464c-9d03-5cc39f001cb8" containerName="cleanup" Oct 31 05:15:21.357065 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357026 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe68f52a-ede6-464c-9d03-5cc39f001cb8" containerName="cleanup" Oct 31 05:15:21.357065 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357041 5019 cpu_manager.go:414] "containerMap: removing container" podUID="e099ec30-86fb-43aa-9ee0-f3305f0adcfb" containerName="cleanup" Oct 31 05:15:21.357065 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357046 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e099ec30-86fb-43aa-9ee0-f3305f0adcfb" containerName="cleanup" Oct 31 05:15:21.357065 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357053 5019 cpu_manager.go:414] "containerMap: removing container" podUID="e099ec30-86fb-43aa-9ee0-f3305f0adcfb" containerName="cleanup" Oct 31 05:15:21.357065 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357057 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e099ec30-86fb-43aa-9ee0-f3305f0adcfb" containerName="cleanup" Oct 31 05:15:21.357065 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357065 5019 cpu_manager.go:414] "containerMap: removing container" podUID="a4e14c11-5250-4d70-932f-055294e67dd4" containerName="cleanup" Oct 31 05:15:21.357322 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357070 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4e14c11-5250-4d70-932f-055294e67dd4" containerName="cleanup" Oct 31 05:15:21.357322 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357077 5019 cpu_manager.go:414] "containerMap: removing container" podUID="7621d02f-ad60-4445-be36-e7c253887716" containerName="cleanup" Oct 31 05:15:21.357322 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357081 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7621d02f-ad60-4445-be36-e7c253887716" containerName="cleanup" Oct 31 05:15:21.357322 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357093 5019 cpu_manager.go:414] "containerMap: removing container" podUID="1132f111-a44a-4cc9-97bf-540c4f0dc0a5" containerName="istio-proxy" Oct 31 05:15:21.357322 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357098 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1132f111-a44a-4cc9-97bf-540c4f0dc0a5" containerName="istio-proxy" Oct 31 05:15:21.357322 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357104 5019 cpu_manager.go:414] "containerMap: removing container" podUID="2030df55-524b-4764-ad64-002ab5d1cc52" containerName="cleanup" Oct 31 05:15:21.357322 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357109 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2030df55-524b-4764-ad64-002ab5d1cc52" containerName="cleanup" Oct 31 05:15:21.357322 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357116 5019 cpu_manager.go:414] "containerMap: removing container" podUID="fe68f52a-ede6-464c-9d03-5cc39f001cb8" containerName="cleanup" Oct 31 05:15:21.357322 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357120 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe68f52a-ede6-464c-9d03-5cc39f001cb8" containerName="cleanup" Oct 31 05:15:21.357322 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357129 5019 cpu_manager.go:414] "containerMap: removing container" podUID="754e0f92-f1dd-40a2-9c1f-031c383b4629" containerName="cleanup" Oct 31 05:15:21.357322 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357133 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="754e0f92-f1dd-40a2-9c1f-031c383b4629" containerName="cleanup" Oct 31 05:15:21.357322 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357140 5019 cpu_manager.go:414] "containerMap: removing container" podUID="2030df55-524b-4764-ad64-002ab5d1cc52" containerName="cleanup" Oct 31 05:15:21.357322 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357144 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2030df55-524b-4764-ad64-002ab5d1cc52" containerName="cleanup" Oct 31 05:15:21.357322 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357152 5019 cpu_manager.go:414] "containerMap: removing container" podUID="ed220fc2-8244-47bd-a868-472210c5e521" containerName="cleanup" Oct 31 05:15:21.357322 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357156 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed220fc2-8244-47bd-a868-472210c5e521" containerName="cleanup" Oct 31 05:15:21.357322 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357163 5019 cpu_manager.go:414] "containerMap: removing container" podUID="8e80ef6a-aee2-4040-a8dc-ca294ff2409c" containerName="cleanup" Oct 31 05:15:21.357322 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357168 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e80ef6a-aee2-4040-a8dc-ca294ff2409c" containerName="cleanup" Oct 31 05:15:21.357322 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357176 5019 cpu_manager.go:414] "containerMap: removing container" podUID="8e80ef6a-aee2-4040-a8dc-ca294ff2409c" containerName="cleanup" Oct 31 05:15:21.357322 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357181 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e80ef6a-aee2-4040-a8dc-ca294ff2409c" containerName="cleanup" Oct 31 05:15:21.357322 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357190 5019 cpu_manager.go:414] "containerMap: removing container" podUID="8e80ef6a-aee2-4040-a8dc-ca294ff2409c" containerName="cleanup" Oct 31 05:15:21.357322 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357195 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e80ef6a-aee2-4040-a8dc-ca294ff2409c" containerName="cleanup" Oct 31 05:15:21.357322 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357201 5019 cpu_manager.go:414] "containerMap: removing container" podUID="ed220fc2-8244-47bd-a868-472210c5e521" containerName="cleanup" Oct 31 05:15:21.357322 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357206 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed220fc2-8244-47bd-a868-472210c5e521" containerName="cleanup" Oct 31 05:15:21.357322 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357215 5019 cpu_manager.go:414] "containerMap: removing container" podUID="1873945a-f1ed-43b2-b787-c29a1783b3e3" containerName="cleanup" Oct 31 05:15:21.357322 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357220 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1873945a-f1ed-43b2-b787-c29a1783b3e3" containerName="cleanup" Oct 31 05:15:21.357322 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357228 5019 cpu_manager.go:414] "containerMap: removing container" podUID="7621d02f-ad60-4445-be36-e7c253887716" containerName="cleanup" Oct 31 05:15:21.357322 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357232 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7621d02f-ad60-4445-be36-e7c253887716" containerName="cleanup" Oct 31 05:15:21.357322 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357239 5019 cpu_manager.go:414] "containerMap: removing container" podUID="2030df55-524b-4764-ad64-002ab5d1cc52" containerName="cleanup" Oct 31 05:15:21.357322 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357243 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2030df55-524b-4764-ad64-002ab5d1cc52" containerName="cleanup" Oct 31 05:15:21.357322 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357254 5019 cpu_manager.go:414] "containerMap: removing container" podUID="1873945a-f1ed-43b2-b787-c29a1783b3e3" containerName="cleanup" Oct 31 05:15:21.357322 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357258 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1873945a-f1ed-43b2-b787-c29a1783b3e3" containerName="cleanup" Oct 31 05:15:21.357322 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357267 5019 cpu_manager.go:414] "containerMap: removing container" podUID="1132f111-a44a-4cc9-97bf-540c4f0dc0a5" containerName="istio-proxy" Oct 31 05:15:21.357322 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357271 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1132f111-a44a-4cc9-97bf-540c4f0dc0a5" containerName="istio-proxy" Oct 31 05:15:21.357322 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357281 5019 cpu_manager.go:414] "containerMap: removing container" podUID="1873945a-f1ed-43b2-b787-c29a1783b3e3" containerName="cleanup" Oct 31 05:15:21.357322 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357285 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1873945a-f1ed-43b2-b787-c29a1783b3e3" containerName="cleanup" Oct 31 05:15:21.357322 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357295 5019 cpu_manager.go:414] "containerMap: removing container" podUID="754e0f92-f1dd-40a2-9c1f-031c383b4629" containerName="cleanup" Oct 31 05:15:21.357322 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357300 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="754e0f92-f1dd-40a2-9c1f-031c383b4629" containerName="cleanup" Oct 31 05:15:21.357322 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357307 5019 cpu_manager.go:414] "containerMap: removing container" podUID="a4e14c11-5250-4d70-932f-055294e67dd4" containerName="cleanup" Oct 31 05:15:21.357322 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357312 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4e14c11-5250-4d70-932f-055294e67dd4" containerName="cleanup" Oct 31 05:15:21.358654 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357410 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="1873945a-f1ed-43b2-b787-c29a1783b3e3" containerName="cleanup" Oct 31 05:15:21.358654 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357422 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe68f52a-ede6-464c-9d03-5cc39f001cb8" containerName="cleanup" Oct 31 05:15:21.358654 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357428 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="1132f111-a44a-4cc9-97bf-540c4f0dc0a5" containerName="istio-proxy" Oct 31 05:15:21.358654 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357436 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="1873945a-f1ed-43b2-b787-c29a1783b3e3" containerName="cleanup" Oct 31 05:15:21.358654 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357442 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="754e0f92-f1dd-40a2-9c1f-031c383b4629" containerName="cleanup" Oct 31 05:15:21.358654 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357448 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4e14c11-5250-4d70-932f-055294e67dd4" containerName="cleanup" Oct 31 05:15:21.358654 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357454 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed220fc2-8244-47bd-a868-472210c5e521" containerName="cleanup" Oct 31 05:15:21.358654 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357458 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe68f52a-ede6-464c-9d03-5cc39f001cb8" containerName="cleanup" Oct 31 05:15:21.358654 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357465 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e099ec30-86fb-43aa-9ee0-f3305f0adcfb" containerName="cleanup" Oct 31 05:15:21.358654 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357470 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e099ec30-86fb-43aa-9ee0-f3305f0adcfb" containerName="cleanup" Oct 31 05:15:21.358654 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357474 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4e14c11-5250-4d70-932f-055294e67dd4" containerName="cleanup" Oct 31 05:15:21.358654 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357479 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="7621d02f-ad60-4445-be36-e7c253887716" containerName="cleanup" Oct 31 05:15:21.358654 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357485 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="1132f111-a44a-4cc9-97bf-540c4f0dc0a5" containerName="istio-proxy" Oct 31 05:15:21.358654 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357491 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="2030df55-524b-4764-ad64-002ab5d1cc52" containerName="cleanup" Oct 31 05:15:21.358654 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357496 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="754e0f92-f1dd-40a2-9c1f-031c383b4629" containerName="cleanup" Oct 31 05:15:21.358654 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357502 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="2030df55-524b-4764-ad64-002ab5d1cc52" containerName="cleanup" Oct 31 05:15:21.358654 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357506 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed220fc2-8244-47bd-a868-472210c5e521" containerName="cleanup" Oct 31 05:15:21.358654 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357511 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e80ef6a-aee2-4040-a8dc-ca294ff2409c" containerName="cleanup" Oct 31 05:15:21.358654 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357516 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e80ef6a-aee2-4040-a8dc-ca294ff2409c" containerName="cleanup" Oct 31 05:15:21.358654 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357522 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="7621d02f-ad60-4445-be36-e7c253887716" containerName="cleanup" Oct 31 05:15:21.358654 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357528 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e80ef6a-aee2-4040-a8dc-ca294ff2409c" containerName="cleanup" Oct 31 05:15:21.358654 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357533 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e099ec30-86fb-43aa-9ee0-f3305f0adcfb" containerName="cleanup" Oct 31 05:15:21.358654 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357539 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="754e0f92-f1dd-40a2-9c1f-031c383b4629" containerName="cleanup" Oct 31 05:15:21.358654 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.357545 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="7621d02f-ad60-4445-be36-e7c253887716" containerName="cleanup" Oct 31 05:15:21.361229 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.361213 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91889c70-sjqvh" Oct 31 05:15:21.380526 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.380501 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:15:21.381201 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.381183 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91889c70-sjqvh"] Oct 31 05:15:21.396548 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.396522 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dfnc\" (UniqueName: \"kubernetes.io/projected/1873945a-f1ed-43b2-b787-c29a1783b3e3-kube-api-access-7dfnc\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:21.410244 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.410202 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="754e0f92-f1dd-40a2-9c1f-031c383b4629" path="/var/lib/kubelet/pods/754e0f92-f1dd-40a2-9c1f-031c383b4629/volumes" Oct 31 05:15:21.410855 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.410831 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7621d02f-ad60-4445-be36-e7c253887716" path="/var/lib/kubelet/pods/7621d02f-ad60-4445-be36-e7c253887716/volumes" Oct 31 05:15:21.411462 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.411439 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e80ef6a-aee2-4040-a8dc-ca294ff2409c" path="/var/lib/kubelet/pods/8e80ef6a-aee2-4040-a8dc-ca294ff2409c/volumes" Oct 31 05:15:21.412451 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.412427 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4e14c11-5250-4d70-932f-055294e67dd4" path="/var/lib/kubelet/pods/a4e14c11-5250-4d70-932f-055294e67dd4/volumes" Oct 31 05:15:21.413010 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.412988 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e099ec30-86fb-43aa-9ee0-f3305f0adcfb" path="/var/lib/kubelet/pods/e099ec30-86fb-43aa-9ee0-f3305f0adcfb/volumes" Oct 31 05:15:21.497722 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.497695 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67d355b0-4e8e-4b21-8250-aba5bbeb4932-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91889c70-sjqvh\" (UID: \"67d355b0-4e8e-4b21-8250-aba5bbeb4932\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91889c70-sjqvh" Oct 31 05:15:21.497892 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.497873 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67d355b0-4e8e-4b21-8250-aba5bbeb4932-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91889c70-sjqvh\" (UID: \"67d355b0-4e8e-4b21-8250-aba5bbeb4932\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91889c70-sjqvh" Oct 31 05:15:21.497974 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.497911 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc2pz\" (UniqueName: \"kubernetes.io/projected/67d355b0-4e8e-4b21-8250-aba5bbeb4932-kube-api-access-hc2pz\") pod \"maas-api-revoke-sub-91889c70-sjqvh\" (UID: \"67d355b0-4e8e-4b21-8250-aba5bbeb4932\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91889c70-sjqvh" Oct 31 05:15:21.598830 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.598782 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67d355b0-4e8e-4b21-8250-aba5bbeb4932-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91889c70-sjqvh\" (UID: \"67d355b0-4e8e-4b21-8250-aba5bbeb4932\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91889c70-sjqvh" Oct 31 05:15:21.599028 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.599008 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67d355b0-4e8e-4b21-8250-aba5bbeb4932-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91889c70-sjqvh\" (UID: \"67d355b0-4e8e-4b21-8250-aba5bbeb4932\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91889c70-sjqvh" Oct 31 05:15:21.599211 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.599191 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc2pz\" (UniqueName: \"kubernetes.io/projected/67d355b0-4e8e-4b21-8250-aba5bbeb4932-kube-api-access-hc2pz\") pod \"maas-api-revoke-sub-91889c70-sjqvh\" (UID: \"67d355b0-4e8e-4b21-8250-aba5bbeb4932\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91889c70-sjqvh" Oct 31 05:15:21.599747 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.599623 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67d355b0-4e8e-4b21-8250-aba5bbeb4932-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91889c70-sjqvh\" (UID: \"67d355b0-4e8e-4b21-8250-aba5bbeb4932\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91889c70-sjqvh" Oct 31 05:15:21.601702 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.601679 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67d355b0-4e8e-4b21-8250-aba5bbeb4932-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91889c70-sjqvh\" (UID: \"67d355b0-4e8e-4b21-8250-aba5bbeb4932\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91889c70-sjqvh" Oct 31 05:15:21.622509 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.622487 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc2pz\" (UniqueName: \"kubernetes.io/projected/67d355b0-4e8e-4b21-8250-aba5bbeb4932-kube-api-access-hc2pz\") pod \"maas-api-revoke-sub-91889c70-sjqvh\" (UID: \"67d355b0-4e8e-4b21-8250-aba5bbeb4932\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91889c70-sjqvh" Oct 31 05:15:21.675866 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.675844 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91889c70-sjqvh" Oct 31 05:15:21.930789 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.930762 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890395-8fb6v" Oct 31 05:15:21.931245 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.930801 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890395-8fb6v" event={"ID":"1873945a-f1ed-43b2-b787-c29a1783b3e3","Type":"ContainerDied","Data":"c0112c7e37058570d18528b7c76ee14de3c848bb0eaf8231c22a3c3b4f0e4f00"} Oct 31 05:15:21.931245 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.930868 5019 scope.go:122] "RemoveContainer" containerID="57c6a261f4486994f430fd946c82d2f8bf5c739a7bf3bd24bd79aae3ccb270c6" Oct 31 05:15:21.932294 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.932272 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890395-4sfpv" Oct 31 05:15:21.932294 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.932283 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890395-4sfpv" event={"ID":"ed220fc2-8244-47bd-a868-472210c5e521","Type":"ContainerDied","Data":"51c49ec6d2e25d0e87e8ba181fb3fece94a8d8952cdd9d845b42b29a9b0ee56b"} Oct 31 05:15:21.941935 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.941905 5019 scope.go:122] "RemoveContainer" containerID="b7ca793f4ecd2df1d78a52b39606c11d910fd4f623a3a4ea658cd40f3a6b5830" Oct 31 05:15:21.979274 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:15:21.979247 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67d355b0_4e8e_4b21_8250_aba5bbeb4932.slice/crio-fba0e2a9ed031315ec02d85208f8b73d14318b784b55e867739cf7e3237a0283 WatchSource:0}: Error finding container fba0e2a9ed031315ec02d85208f8b73d14318b784b55e867739cf7e3237a0283: Status 404 returned error can't find the container with id fba0e2a9ed031315ec02d85208f8b73d14318b784b55e867739cf7e3237a0283 Oct 31 05:15:21.979387 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.979296 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91889c70-sjqvh"] Oct 31 05:15:21.984003 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.983980 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890395-4sfpv"] Oct 31 05:15:21.990983 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:21.990936 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-766728-29890395-4sfpv"] Oct 31 05:15:22.024629 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:22.024608 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890395-8fb6v"] Oct 31 05:15:22.027925 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:22.027901 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-afc519-29890395-8fb6v"] Oct 31 05:15:22.216693 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:22.216668 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2648aa75-lmdxc"] Oct 31 05:15:22.217250 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:22.217238 5019 cpu_manager.go:414] "containerMap: removing container" podUID="e099ec30-86fb-43aa-9ee0-f3305f0adcfb" containerName="cleanup" Oct 31 05:15:22.217302 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:22.217253 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e099ec30-86fb-43aa-9ee0-f3305f0adcfb" containerName="cleanup" Oct 31 05:15:22.217302 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:22.217263 5019 cpu_manager.go:414] "containerMap: removing container" podUID="a4e14c11-5250-4d70-932f-055294e67dd4" containerName="cleanup" Oct 31 05:15:22.217302 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:22.217268 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4e14c11-5250-4d70-932f-055294e67dd4" containerName="cleanup" Oct 31 05:15:22.217302 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:22.217279 5019 cpu_manager.go:414] "containerMap: removing container" podUID="7621d02f-ad60-4445-be36-e7c253887716" containerName="cleanup" Oct 31 05:15:22.217302 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:22.217285 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7621d02f-ad60-4445-be36-e7c253887716" containerName="cleanup" Oct 31 05:15:22.217302 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:22.217296 5019 cpu_manager.go:414] "containerMap: removing container" podUID="fe68f52a-ede6-464c-9d03-5cc39f001cb8" containerName="cleanup" Oct 31 05:15:22.217302 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:22.217301 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe68f52a-ede6-464c-9d03-5cc39f001cb8" containerName="cleanup" Oct 31 05:15:22.217584 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:22.217309 5019 cpu_manager.go:414] "containerMap: removing container" podUID="754e0f92-f1dd-40a2-9c1f-031c383b4629" containerName="cleanup" Oct 31 05:15:22.217584 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:22.217313 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="754e0f92-f1dd-40a2-9c1f-031c383b4629" containerName="cleanup" Oct 31 05:15:22.217584 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:22.217421 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="1873945a-f1ed-43b2-b787-c29a1783b3e3" containerName="cleanup" Oct 31 05:15:22.217584 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:22.217435 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe68f52a-ede6-464c-9d03-5cc39f001cb8" containerName="cleanup" Oct 31 05:15:22.217584 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:22.217446 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4e14c11-5250-4d70-932f-055294e67dd4" containerName="cleanup" Oct 31 05:15:22.217584 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:22.217454 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="2030df55-524b-4764-ad64-002ab5d1cc52" containerName="cleanup" Oct 31 05:15:22.217584 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:22.217461 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed220fc2-8244-47bd-a868-472210c5e521" containerName="cleanup" Oct 31 05:15:22.224287 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:22.224265 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2648aa75-lmdxc" Oct 31 05:15:22.240908 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:22.240885 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2648aa75-lmdxc"] Oct 31 05:15:22.307882 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:22.307858 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f61145c7-2f3f-4072-b329-5760f95c042f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2648aa75-lmdxc\" (UID: \"f61145c7-2f3f-4072-b329-5760f95c042f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2648aa75-lmdxc" Oct 31 05:15:22.308007 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:22.307909 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4gn5\" (UniqueName: \"kubernetes.io/projected/f61145c7-2f3f-4072-b329-5760f95c042f-kube-api-access-g4gn5\") pod \"maas-api-revoke-sub-2648aa75-lmdxc\" (UID: \"f61145c7-2f3f-4072-b329-5760f95c042f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2648aa75-lmdxc" Oct 31 05:15:22.308077 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:22.308024 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f61145c7-2f3f-4072-b329-5760f95c042f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2648aa75-lmdxc\" (UID: \"f61145c7-2f3f-4072-b329-5760f95c042f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2648aa75-lmdxc" Oct 31 05:15:22.397665 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:15:22.397624 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:15:22.409846 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:22.409810 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f61145c7-2f3f-4072-b329-5760f95c042f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2648aa75-lmdxc\" (UID: \"f61145c7-2f3f-4072-b329-5760f95c042f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2648aa75-lmdxc" Oct 31 05:15:22.409987 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:22.409890 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4gn5\" (UniqueName: \"kubernetes.io/projected/f61145c7-2f3f-4072-b329-5760f95c042f-kube-api-access-g4gn5\") pod \"maas-api-revoke-sub-2648aa75-lmdxc\" (UID: \"f61145c7-2f3f-4072-b329-5760f95c042f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2648aa75-lmdxc" Oct 31 05:15:22.410047 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:22.409993 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f61145c7-2f3f-4072-b329-5760f95c042f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2648aa75-lmdxc\" (UID: \"f61145c7-2f3f-4072-b329-5760f95c042f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2648aa75-lmdxc" Oct 31 05:15:22.410585 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:22.410551 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f61145c7-2f3f-4072-b329-5760f95c042f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2648aa75-lmdxc\" (UID: \"f61145c7-2f3f-4072-b329-5760f95c042f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2648aa75-lmdxc" Oct 31 05:15:22.413600 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:22.413574 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f61145c7-2f3f-4072-b329-5760f95c042f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2648aa75-lmdxc\" (UID: \"f61145c7-2f3f-4072-b329-5760f95c042f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2648aa75-lmdxc" Oct 31 05:15:22.434288 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:22.434234 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4gn5\" (UniqueName: \"kubernetes.io/projected/f61145c7-2f3f-4072-b329-5760f95c042f-kube-api-access-g4gn5\") pod \"maas-api-revoke-sub-2648aa75-lmdxc\" (UID: \"f61145c7-2f3f-4072-b329-5760f95c042f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2648aa75-lmdxc" Oct 31 05:15:22.545059 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:22.544984 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2648aa75-lmdxc" Oct 31 05:15:22.846245 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:22.846215 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2648aa75-lmdxc"] Oct 31 05:15:22.847020 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:15:22.846993 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf61145c7_2f3f_4072_b329_5760f95c042f.slice/crio-4242c3d5aac8564c043858182c89df5c5682b7e20debe566aa7f76858e6aae6a WatchSource:0}: Error finding container 4242c3d5aac8564c043858182c89df5c5682b7e20debe566aa7f76858e6aae6a: Status 404 returned error can't find the container with id 4242c3d5aac8564c043858182c89df5c5682b7e20debe566aa7f76858e6aae6a Oct 31 05:15:22.942236 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:22.942207 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2648aa75-lmdxc" event={"ID":"f61145c7-2f3f-4072-b329-5760f95c042f","Type":"ContainerStarted","Data":"4242c3d5aac8564c043858182c89df5c5682b7e20debe566aa7f76858e6aae6a"} Oct 31 05:15:22.944903 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:22.944877 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91889c70-sjqvh" event={"ID":"67d355b0-4e8e-4b21-8250-aba5bbeb4932","Type":"ContainerStarted","Data":"33eff0b96eff6feb46049ed49fd0c5ebba0877ac7da702a223b46ff850c354e5"} Oct 31 05:15:22.945013 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:22.944929 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91889c70-sjqvh" event={"ID":"67d355b0-4e8e-4b21-8250-aba5bbeb4932","Type":"ContainerStarted","Data":"fba0e2a9ed031315ec02d85208f8b73d14318b784b55e867739cf7e3237a0283"} Oct 31 05:15:22.987860 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:22.987812 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91889c70-sjqvh" podStartSLOduration=1.987796747 podStartE2EDuration="1.987796747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:15:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:15:22.983515943 +0000 UTC m=+4277.739854009" watchObservedRunningTime="2026-10-31 05:15:22.987796747 +0000 UTC m=+4277.744134788" Oct 31 05:15:23.409032 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:23.408993 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1873945a-f1ed-43b2-b787-c29a1783b3e3" path="/var/lib/kubelet/pods/1873945a-f1ed-43b2-b787-c29a1783b3e3/volumes" Oct 31 05:15:23.409446 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:23.409431 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed220fc2-8244-47bd-a868-472210c5e521" path="/var/lib/kubelet/pods/ed220fc2-8244-47bd-a868-472210c5e521/volumes" Oct 31 05:15:23.954683 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:23.954651 5019 generic.go:356] "Generic (PLEG): container finished" podID="f61145c7-2f3f-4072-b329-5760f95c042f" containerID="8e61ec6e8ac2401b5070a0e8377973597c5d94fb6fe65b6cf42f43672d38f9b1" exitCode=0 Oct 31 05:15:23.955134 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:23.954732 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2648aa75-lmdxc" event={"ID":"f61145c7-2f3f-4072-b329-5760f95c042f","Type":"ContainerDied","Data":"8e61ec6e8ac2401b5070a0e8377973597c5d94fb6fe65b6cf42f43672d38f9b1"} Oct 31 05:15:23.956766 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:23.956742 5019 generic.go:356] "Generic (PLEG): container finished" podID="67d355b0-4e8e-4b21-8250-aba5bbeb4932" containerID="33eff0b96eff6feb46049ed49fd0c5ebba0877ac7da702a223b46ff850c354e5" exitCode=0 Oct 31 05:15:23.956890 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:23.956812 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91889c70-sjqvh" event={"ID":"67d355b0-4e8e-4b21-8250-aba5bbeb4932","Type":"ContainerDied","Data":"33eff0b96eff6feb46049ed49fd0c5ebba0877ac7da702a223b46ff850c354e5"} Oct 31 05:15:24.557701 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.557673 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-67f7j" Oct 31 05:15:24.580800 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.580780 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rdnft" Oct 31 05:15:24.663492 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.646755 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9573de57-8748-44eb-8949-b55b7dca0043-kube-api-access-wqhlr\" (UniqueName: \"kubernetes.io/projected/9573de57-8748-44eb-8949-b55b7dca0043-kube-api-access-wqhlr\") pod \"9573de57-8748-44eb-8949-b55b7dca0043\" (UID: \"9573de57-8748-44eb-8949-b55b7dca0043\") " Oct 31 05:15:24.663492 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.646858 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9573de57-8748-44eb-8949-b55b7dca0043-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9573de57-8748-44eb-8949-b55b7dca0043-istiod-ca-cert\") pod \"9573de57-8748-44eb-8949-b55b7dca0043\" (UID: \"9573de57-8748-44eb-8949-b55b7dca0043\") " Oct 31 05:15:24.663492 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.646895 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9573de57-8748-44eb-8949-b55b7dca0043-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9573de57-8748-44eb-8949-b55b7dca0043-istio-data\") pod \"9573de57-8748-44eb-8949-b55b7dca0043\" (UID: \"9573de57-8748-44eb-8949-b55b7dca0043\") " Oct 31 05:15:24.663492 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.646950 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9573de57-8748-44eb-8949-b55b7dca0043-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9573de57-8748-44eb-8949-b55b7dca0043-istio-podinfo\") pod \"9573de57-8748-44eb-8949-b55b7dca0043\" (UID: \"9573de57-8748-44eb-8949-b55b7dca0043\") " Oct 31 05:15:24.663492 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.647050 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9573de57-8748-44eb-8949-b55b7dca0043-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9573de57-8748-44eb-8949-b55b7dca0043-workload-socket\") pod \"9573de57-8748-44eb-8949-b55b7dca0043\" (UID: \"9573de57-8748-44eb-8949-b55b7dca0043\") " Oct 31 05:15:24.663492 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.647077 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9573de57-8748-44eb-8949-b55b7dca0043-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9573de57-8748-44eb-8949-b55b7dca0043-workload-certs\") pod \"9573de57-8748-44eb-8949-b55b7dca0043\" (UID: \"9573de57-8748-44eb-8949-b55b7dca0043\") " Oct 31 05:15:24.663492 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.647130 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9573de57-8748-44eb-8949-b55b7dca0043-istio-token\" (UniqueName: \"kubernetes.io/projected/9573de57-8748-44eb-8949-b55b7dca0043-istio-token\") pod \"9573de57-8748-44eb-8949-b55b7dca0043\" (UID: \"9573de57-8748-44eb-8949-b55b7dca0043\") " Oct 31 05:15:24.663492 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.647246 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9573de57-8748-44eb-8949-b55b7dca0043-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9573de57-8748-44eb-8949-b55b7dca0043-istio-envoy\") pod \"9573de57-8748-44eb-8949-b55b7dca0043\" (UID: \"9573de57-8748-44eb-8949-b55b7dca0043\") " Oct 31 05:15:24.663492 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.647299 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9573de57-8748-44eb-8949-b55b7dca0043-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9573de57-8748-44eb-8949-b55b7dca0043-credential-socket\") pod \"9573de57-8748-44eb-8949-b55b7dca0043\" (UID: \"9573de57-8748-44eb-8949-b55b7dca0043\") " Oct 31 05:15:24.663492 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.649387 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9573de57-8748-44eb-8949-b55b7dca0043-credential-socket" pod "9573de57-8748-44eb-8949-b55b7dca0043" (UID: "9573de57-8748-44eb-8949-b55b7dca0043"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:15:24.663492 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.654802 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9573de57-8748-44eb-8949-b55b7dca0043-istiod-ca-cert" pod "9573de57-8748-44eb-8949-b55b7dca0043" (UID: "9573de57-8748-44eb-8949-b55b7dca0043"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:15:24.663492 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.655120 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9573de57-8748-44eb-8949-b55b7dca0043-istio-data" pod "9573de57-8748-44eb-8949-b55b7dca0043" (UID: "9573de57-8748-44eb-8949-b55b7dca0043"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:15:24.663492 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.655183 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9573de57-8748-44eb-8949-b55b7dca0043-workload-certs" pod "9573de57-8748-44eb-8949-b55b7dca0043" (UID: "9573de57-8748-44eb-8949-b55b7dca0043"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:15:24.663492 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.655485 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9573de57-8748-44eb-8949-b55b7dca0043-workload-socket" pod "9573de57-8748-44eb-8949-b55b7dca0043" (UID: "9573de57-8748-44eb-8949-b55b7dca0043"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:15:24.678642 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.678493 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9573de57-8748-44eb-8949-b55b7dca0043-istio-podinfo" pod "9573de57-8748-44eb-8949-b55b7dca0043" (UID: "9573de57-8748-44eb-8949-b55b7dca0043"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:15:24.678642 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.678522 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9573de57-8748-44eb-8949-b55b7dca0043-istio-token" pod "9573de57-8748-44eb-8949-b55b7dca0043" (UID: "9573de57-8748-44eb-8949-b55b7dca0043"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:15:24.678642 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.678585 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9573de57-8748-44eb-8949-b55b7dca0043-kube-api-access-wqhlr" pod "9573de57-8748-44eb-8949-b55b7dca0043" (UID: "9573de57-8748-44eb-8949-b55b7dca0043"). InnerVolumeSpecName "kube-api-access-wqhlr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:15:24.678642 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.678608 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9573de57-8748-44eb-8949-b55b7dca0043-istio-envoy" pod "9573de57-8748-44eb-8949-b55b7dca0043" (UID: "9573de57-8748-44eb-8949-b55b7dca0043"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:15:24.710069 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.709741 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq" Oct 31 05:15:24.712928 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.712906 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7" Oct 31 05:15:24.749084 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.749055 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-istio-podinfo\") pod \"5fc2a87b-394a-4c26-adfd-9a57a1fc463b\" (UID: \"5fc2a87b-394a-4c26-adfd-9a57a1fc463b\") " Oct 31 05:15:24.749199 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.749109 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-credential-socket\") pod \"5fc2a87b-394a-4c26-adfd-9a57a1fc463b\" (UID: \"5fc2a87b-394a-4c26-adfd-9a57a1fc463b\") " Oct 31 05:15:24.749199 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.749152 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-istio-token\" (UniqueName: \"kubernetes.io/projected/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-istio-token\") pod \"5fc2a87b-394a-4c26-adfd-9a57a1fc463b\" (UID: \"5fc2a87b-394a-4c26-adfd-9a57a1fc463b\") " Oct 31 05:15:24.749199 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.749181 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-workload-certs\") pod \"5fc2a87b-394a-4c26-adfd-9a57a1fc463b\" (UID: \"5fc2a87b-394a-4c26-adfd-9a57a1fc463b\") " Oct 31 05:15:24.749406 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.749207 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-istiod-ca-cert\") pod \"5fc2a87b-394a-4c26-adfd-9a57a1fc463b\" (UID: \"5fc2a87b-394a-4c26-adfd-9a57a1fc463b\") " Oct 31 05:15:24.749406 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.749248 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-istio-envoy\") pod \"5fc2a87b-394a-4c26-adfd-9a57a1fc463b\" (UID: \"5fc2a87b-394a-4c26-adfd-9a57a1fc463b\") " Oct 31 05:15:24.749406 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.749290 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-kube-api-access-d5kx5\" (UniqueName: \"kubernetes.io/projected/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-kube-api-access-d5kx5\") pod \"5fc2a87b-394a-4c26-adfd-9a57a1fc463b\" (UID: \"5fc2a87b-394a-4c26-adfd-9a57a1fc463b\") " Oct 31 05:15:24.749581 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.749495 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-credential-socket" pod "5fc2a87b-394a-4c26-adfd-9a57a1fc463b" (UID: "5fc2a87b-394a-4c26-adfd-9a57a1fc463b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:15:24.749581 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.749532 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-workload-certs" pod "5fc2a87b-394a-4c26-adfd-9a57a1fc463b" (UID: "5fc2a87b-394a-4c26-adfd-9a57a1fc463b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:15:24.749705 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.749663 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-istiod-ca-cert" pod "5fc2a87b-394a-4c26-adfd-9a57a1fc463b" (UID: "5fc2a87b-394a-4c26-adfd-9a57a1fc463b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:15:24.749795 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.749766 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-workload-socket\") pod \"5fc2a87b-394a-4c26-adfd-9a57a1fc463b\" (UID: \"5fc2a87b-394a-4c26-adfd-9a57a1fc463b\") " Oct 31 05:15:24.749856 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.749843 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-istio-data\") pod \"5fc2a87b-394a-4c26-adfd-9a57a1fc463b\" (UID: \"5fc2a87b-394a-4c26-adfd-9a57a1fc463b\") " Oct 31 05:15:24.750080 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.750034 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-workload-socket" pod "5fc2a87b-394a-4c26-adfd-9a57a1fc463b" (UID: "5fc2a87b-394a-4c26-adfd-9a57a1fc463b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:15:24.750142 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.750119 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-istio-data" pod "5fc2a87b-394a-4c26-adfd-9a57a1fc463b" (UID: "5fc2a87b-394a-4c26-adfd-9a57a1fc463b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:15:24.750493 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.750472 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:24.750591 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.750497 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:24.750591 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.750512 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9573de57-8748-44eb-8949-b55b7dca0043-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:24.750591 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.750526 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9573de57-8748-44eb-8949-b55b7dca0043-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:24.750591 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.750541 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:24.750591 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.750555 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqhlr\" (UniqueName: \"kubernetes.io/projected/9573de57-8748-44eb-8949-b55b7dca0043-kube-api-access-wqhlr\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:24.750591 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.750580 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:24.750926 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.750593 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9573de57-8748-44eb-8949-b55b7dca0043-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:24.750926 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.750606 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9573de57-8748-44eb-8949-b55b7dca0043-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:24.750926 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.750619 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9573de57-8748-44eb-8949-b55b7dca0043-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:24.750926 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.750632 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9573de57-8748-44eb-8949-b55b7dca0043-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:24.750926 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.750645 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9573de57-8748-44eb-8949-b55b7dca0043-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:24.750926 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.750658 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:24.750926 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.750672 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9573de57-8748-44eb-8949-b55b7dca0043-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:24.751585 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.751556 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-istio-podinfo" pod "5fc2a87b-394a-4c26-adfd-9a57a1fc463b" (UID: "5fc2a87b-394a-4c26-adfd-9a57a1fc463b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:15:24.751865 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.751841 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-istio-envoy" pod "5fc2a87b-394a-4c26-adfd-9a57a1fc463b" (UID: "5fc2a87b-394a-4c26-adfd-9a57a1fc463b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:15:24.751972 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.751878 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-istio-token" pod "5fc2a87b-394a-4c26-adfd-9a57a1fc463b" (UID: "5fc2a87b-394a-4c26-adfd-9a57a1fc463b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:15:24.751972 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.751903 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-kube-api-access-d5kx5" pod "5fc2a87b-394a-4c26-adfd-9a57a1fc463b" (UID: "5fc2a87b-394a-4c26-adfd-9a57a1fc463b"). InnerVolumeSpecName "kube-api-access-d5kx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:15:24.851752 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.851662 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0ccc03e-0e5e-40b7-8a40-718f199dd747-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ccc03e-0e5e-40b7-8a40-718f199dd747-workload-socket\") pod \"e0ccc03e-0e5e-40b7-8a40-718f199dd747\" (UID: \"e0ccc03e-0e5e-40b7-8a40-718f199dd747\") " Oct 31 05:15:24.851752 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.851698 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0ccc03e-0e5e-40b7-8a40-718f199dd747-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0ccc03e-0e5e-40b7-8a40-718f199dd747-workload-certs\") pod \"e0ccc03e-0e5e-40b7-8a40-718f199dd747\" (UID: \"e0ccc03e-0e5e-40b7-8a40-718f199dd747\") " Oct 31 05:15:24.851752 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.851721 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cf9a806-d96f-478b-9ce1-2e67ec594739-istio-token\" (UniqueName: \"kubernetes.io/projected/1cf9a806-d96f-478b-9ce1-2e67ec594739-istio-token\") pod \"1cf9a806-d96f-478b-9ce1-2e67ec594739\" (UID: \"1cf9a806-d96f-478b-9ce1-2e67ec594739\") " Oct 31 05:15:24.852055 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.851782 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1cf9a806-d96f-478b-9ce1-2e67ec594739-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cf9a806-d96f-478b-9ce1-2e67ec594739-istio-podinfo\") pod \"1cf9a806-d96f-478b-9ce1-2e67ec594739\" (UID: \"1cf9a806-d96f-478b-9ce1-2e67ec594739\") " Oct 31 05:15:24.852055 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.851809 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cf9a806-d96f-478b-9ce1-2e67ec594739-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cf9a806-d96f-478b-9ce1-2e67ec594739-workload-certs\") pod \"1cf9a806-d96f-478b-9ce1-2e67ec594739\" (UID: \"1cf9a806-d96f-478b-9ce1-2e67ec594739\") " Oct 31 05:15:24.852055 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.851856 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0ccc03e-0e5e-40b7-8a40-718f199dd747-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0ccc03e-0e5e-40b7-8a40-718f199dd747-istio-envoy\") pod \"e0ccc03e-0e5e-40b7-8a40-718f199dd747\" (UID: \"e0ccc03e-0e5e-40b7-8a40-718f199dd747\") " Oct 31 05:15:24.852055 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.851946 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0ccc03e-0e5e-40b7-8a40-718f199dd747-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0ccc03e-0e5e-40b7-8a40-718f199dd747-istio-data\") pod \"e0ccc03e-0e5e-40b7-8a40-718f199dd747\" (UID: \"e0ccc03e-0e5e-40b7-8a40-718f199dd747\") " Oct 31 05:15:24.852055 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.851985 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0ccc03e-0e5e-40b7-8a40-718f199dd747-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0ccc03e-0e5e-40b7-8a40-718f199dd747-istiod-ca-cert\") pod \"e0ccc03e-0e5e-40b7-8a40-718f199dd747\" (UID: \"e0ccc03e-0e5e-40b7-8a40-718f199dd747\") " Oct 31 05:15:24.852055 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.851986 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ccc03e-0e5e-40b7-8a40-718f199dd747-workload-certs" pod "e0ccc03e-0e5e-40b7-8a40-718f199dd747" (UID: "e0ccc03e-0e5e-40b7-8a40-718f199dd747"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:15:24.852055 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.852030 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ccc03e-0e5e-40b7-8a40-718f199dd747-workload-socket" pod "e0ccc03e-0e5e-40b7-8a40-718f199dd747" (UID: "e0ccc03e-0e5e-40b7-8a40-718f199dd747"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:15:24.852055 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.852035 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e0ccc03e-0e5e-40b7-8a40-718f199dd747-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0ccc03e-0e5e-40b7-8a40-718f199dd747-istio-podinfo\") pod \"e0ccc03e-0e5e-40b7-8a40-718f199dd747\" (UID: \"e0ccc03e-0e5e-40b7-8a40-718f199dd747\") " Oct 31 05:15:24.852512 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.852103 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0ccc03e-0e5e-40b7-8a40-718f199dd747-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ccc03e-0e5e-40b7-8a40-718f199dd747-credential-socket\") pod \"e0ccc03e-0e5e-40b7-8a40-718f199dd747\" (UID: \"e0ccc03e-0e5e-40b7-8a40-718f199dd747\") " Oct 31 05:15:24.852512 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.852174 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0ccc03e-0e5e-40b7-8a40-718f199dd747-istio-token\" (UniqueName: \"kubernetes.io/projected/e0ccc03e-0e5e-40b7-8a40-718f199dd747-istio-token\") pod \"e0ccc03e-0e5e-40b7-8a40-718f199dd747\" (UID: \"e0ccc03e-0e5e-40b7-8a40-718f199dd747\") " Oct 31 05:15:24.852512 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.852224 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cf9a806-d96f-478b-9ce1-2e67ec594739-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf9a806-d96f-478b-9ce1-2e67ec594739-workload-socket\") pod \"1cf9a806-d96f-478b-9ce1-2e67ec594739\" (UID: \"1cf9a806-d96f-478b-9ce1-2e67ec594739\") " Oct 31 05:15:24.852512 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.852243 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ccc03e-0e5e-40b7-8a40-718f199dd747-istio-data" pod "e0ccc03e-0e5e-40b7-8a40-718f199dd747" (UID: "e0ccc03e-0e5e-40b7-8a40-718f199dd747"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:15:24.852512 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.852271 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0ccc03e-0e5e-40b7-8a40-718f199dd747-kube-api-access-999n5\" (UniqueName: \"kubernetes.io/projected/e0ccc03e-0e5e-40b7-8a40-718f199dd747-kube-api-access-999n5\") pod \"e0ccc03e-0e5e-40b7-8a40-718f199dd747\" (UID: \"e0ccc03e-0e5e-40b7-8a40-718f199dd747\") " Oct 31 05:15:24.852512 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.852301 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cf9a806-d96f-478b-9ce1-2e67ec594739-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cf9a806-d96f-478b-9ce1-2e67ec594739-istio-envoy\") pod \"1cf9a806-d96f-478b-9ce1-2e67ec594739\" (UID: \"1cf9a806-d96f-478b-9ce1-2e67ec594739\") " Oct 31 05:15:24.852512 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.852364 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cf9a806-d96f-478b-9ce1-2e67ec594739-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cf9a806-d96f-478b-9ce1-2e67ec594739-istio-data\") pod \"1cf9a806-d96f-478b-9ce1-2e67ec594739\" (UID: \"1cf9a806-d96f-478b-9ce1-2e67ec594739\") " Oct 31 05:15:24.852512 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.852380 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0ccc03e-0e5e-40b7-8a40-718f199dd747-istiod-ca-cert" pod "e0ccc03e-0e5e-40b7-8a40-718f199dd747" (UID: "e0ccc03e-0e5e-40b7-8a40-718f199dd747"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:15:24.852512 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.852407 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cf9a806-d96f-478b-9ce1-2e67ec594739-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cf9a806-d96f-478b-9ce1-2e67ec594739-istiod-ca-cert\") pod \"1cf9a806-d96f-478b-9ce1-2e67ec594739\" (UID: \"1cf9a806-d96f-478b-9ce1-2e67ec594739\") " Oct 31 05:15:24.852512 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.852446 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cf9a806-d96f-478b-9ce1-2e67ec594739-kube-api-access-rzvbb\" (UniqueName: \"kubernetes.io/projected/1cf9a806-d96f-478b-9ce1-2e67ec594739-kube-api-access-rzvbb\") pod \"1cf9a806-d96f-478b-9ce1-2e67ec594739\" (UID: \"1cf9a806-d96f-478b-9ce1-2e67ec594739\") " Oct 31 05:15:24.852512 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.852499 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cf9a806-d96f-478b-9ce1-2e67ec594739-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf9a806-d96f-478b-9ce1-2e67ec594739-credential-socket\") pod \"1cf9a806-d96f-478b-9ce1-2e67ec594739\" (UID: \"1cf9a806-d96f-478b-9ce1-2e67ec594739\") " Oct 31 05:15:24.853112 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.852533 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ccc03e-0e5e-40b7-8a40-718f199dd747-credential-socket" pod "e0ccc03e-0e5e-40b7-8a40-718f199dd747" (UID: "e0ccc03e-0e5e-40b7-8a40-718f199dd747"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:15:24.853112 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.852916 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cf9a806-d96f-478b-9ce1-2e67ec594739-workload-certs" pod "1cf9a806-d96f-478b-9ce1-2e67ec594739" (UID: "1cf9a806-d96f-478b-9ce1-2e67ec594739"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:15:24.853225 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.853148 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cf9a806-d96f-478b-9ce1-2e67ec594739-credential-socket" pod "1cf9a806-d96f-478b-9ce1-2e67ec594739" (UID: "1cf9a806-d96f-478b-9ce1-2e67ec594739"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:15:24.853225 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.853188 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ccc03e-0e5e-40b7-8a40-718f199dd747-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:24.853225 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.853208 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:24.853225 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.853223 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:24.853482 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.853238 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5kx5\" (UniqueName: \"kubernetes.io/projected/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-kube-api-access-d5kx5\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:24.853482 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.853253 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ccc03e-0e5e-40b7-8a40-718f199dd747-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:24.853482 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.853268 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0ccc03e-0e5e-40b7-8a40-718f199dd747-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:24.853482 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.853285 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cf9a806-d96f-478b-9ce1-2e67ec594739-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:24.853482 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.853325 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0ccc03e-0e5e-40b7-8a40-718f199dd747-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:24.853482 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.853357 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0ccc03e-0e5e-40b7-8a40-718f199dd747-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:24.853482 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.853361 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cf9a806-d96f-478b-9ce1-2e67ec594739-istio-data" pod "1cf9a806-d96f-478b-9ce1-2e67ec594739" (UID: "1cf9a806-d96f-478b-9ce1-2e67ec594739"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:15:24.853482 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.853371 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fc2a87b-394a-4c26-adfd-9a57a1fc463b-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:24.853919 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.853679 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cf9a806-d96f-478b-9ce1-2e67ec594739-istiod-ca-cert" pod "1cf9a806-d96f-478b-9ce1-2e67ec594739" (UID: "1cf9a806-d96f-478b-9ce1-2e67ec594739"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:15:24.854701 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.854638 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1cf9a806-d96f-478b-9ce1-2e67ec594739-istio-podinfo" pod "1cf9a806-d96f-478b-9ce1-2e67ec594739" (UID: "1cf9a806-d96f-478b-9ce1-2e67ec594739"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:15:24.854701 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.854652 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0ccc03e-0e5e-40b7-8a40-718f199dd747-istio-podinfo" pod "e0ccc03e-0e5e-40b7-8a40-718f199dd747" (UID: "e0ccc03e-0e5e-40b7-8a40-718f199dd747"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:15:24.854701 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.854650 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cf9a806-d96f-478b-9ce1-2e67ec594739-istio-token" pod "1cf9a806-d96f-478b-9ce1-2e67ec594739" (UID: "1cf9a806-d96f-478b-9ce1-2e67ec594739"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:15:24.854955 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.854697 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cf9a806-d96f-478b-9ce1-2e67ec594739-workload-socket" pod "1cf9a806-d96f-478b-9ce1-2e67ec594739" (UID: "1cf9a806-d96f-478b-9ce1-2e67ec594739"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:15:24.854955 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.854889 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0ccc03e-0e5e-40b7-8a40-718f199dd747-kube-api-access-999n5" pod "e0ccc03e-0e5e-40b7-8a40-718f199dd747" (UID: "e0ccc03e-0e5e-40b7-8a40-718f199dd747"). InnerVolumeSpecName "kube-api-access-999n5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:15:24.856102 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.856058 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0ccc03e-0e5e-40b7-8a40-718f199dd747-istio-token" pod "e0ccc03e-0e5e-40b7-8a40-718f199dd747" (UID: "e0ccc03e-0e5e-40b7-8a40-718f199dd747"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:15:24.856219 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.856118 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cf9a806-d96f-478b-9ce1-2e67ec594739-kube-api-access-rzvbb" pod "1cf9a806-d96f-478b-9ce1-2e67ec594739" (UID: "1cf9a806-d96f-478b-9ce1-2e67ec594739"). InnerVolumeSpecName "kube-api-access-rzvbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:15:24.856427 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.856404 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ccc03e-0e5e-40b7-8a40-718f199dd747-istio-envoy" pod "e0ccc03e-0e5e-40b7-8a40-718f199dd747" (UID: "e0ccc03e-0e5e-40b7-8a40-718f199dd747"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:15:24.856538 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.856459 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cf9a806-d96f-478b-9ce1-2e67ec594739-istio-envoy" pod "1cf9a806-d96f-478b-9ce1-2e67ec594739" (UID: "1cf9a806-d96f-478b-9ce1-2e67ec594739"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:15:24.954849 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.954824 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0ccc03e-0e5e-40b7-8a40-718f199dd747-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:24.954849 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.954848 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf9a806-d96f-478b-9ce1-2e67ec594739-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:24.955211 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.954857 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-999n5\" (UniqueName: \"kubernetes.io/projected/e0ccc03e-0e5e-40b7-8a40-718f199dd747-kube-api-access-999n5\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:24.955211 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.954867 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cf9a806-d96f-478b-9ce1-2e67ec594739-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:24.955211 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.954875 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cf9a806-d96f-478b-9ce1-2e67ec594739-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:24.955211 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.954885 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cf9a806-d96f-478b-9ce1-2e67ec594739-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:24.955211 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.954896 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzvbb\" (UniqueName: \"kubernetes.io/projected/1cf9a806-d96f-478b-9ce1-2e67ec594739-kube-api-access-rzvbb\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:24.955211 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.954904 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf9a806-d96f-478b-9ce1-2e67ec594739-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:24.955211 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.954912 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cf9a806-d96f-478b-9ce1-2e67ec594739-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:24.955211 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.954920 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cf9a806-d96f-478b-9ce1-2e67ec594739-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:24.955211 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.954928 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0ccc03e-0e5e-40b7-8a40-718f199dd747-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:24.955211 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.954936 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0ccc03e-0e5e-40b7-8a40-718f199dd747-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:24.966828 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.966798 5019 generic.go:356] "Generic (PLEG): container finished" podID="e0ccc03e-0e5e-40b7-8a40-718f199dd747" containerID="2e4684a3042b06f9b8f590a39685056778deeec1cd39a10a0daf25c952f3b422" exitCode=0 Oct 31 05:15:24.966978 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.966841 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7" Oct 31 05:15:24.966978 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.966877 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7" event={"ID":"e0ccc03e-0e5e-40b7-8a40-718f199dd747","Type":"ContainerDied","Data":"2e4684a3042b06f9b8f590a39685056778deeec1cd39a10a0daf25c952f3b422"} Oct 31 05:15:24.966978 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.966929 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7" event={"ID":"e0ccc03e-0e5e-40b7-8a40-718f199dd747","Type":"ContainerDied","Data":"06048bc2a80e48fc0b36ab3e56a74ba9d07535cb47e67eab0f90a8115a701e93"} Oct 31 05:15:24.966978 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.966958 5019 scope.go:122] "RemoveContainer" containerID="2e4684a3042b06f9b8f590a39685056778deeec1cd39a10a0daf25c952f3b422" Oct 31 05:15:24.968667 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.968644 5019 generic.go:356] "Generic (PLEG): container finished" podID="9573de57-8748-44eb-8949-b55b7dca0043" containerID="e446e225b4891af314659175ba11f9a40c9491146968b525a8fc11608e13f303" exitCode=0 Oct 31 05:15:24.968766 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.968698 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-67f7j" Oct 31 05:15:24.968766 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.968729 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-67f7j" event={"ID":"9573de57-8748-44eb-8949-b55b7dca0043","Type":"ContainerDied","Data":"e446e225b4891af314659175ba11f9a40c9491146968b525a8fc11608e13f303"} Oct 31 05:15:24.968925 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.968783 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-67f7j" event={"ID":"9573de57-8748-44eb-8949-b55b7dca0043","Type":"ContainerDied","Data":"02f23f2ed0232e5ebbbbd023e8d938dc5592d0d86a31cc5164163399c5d2fbef"} Oct 31 05:15:24.970254 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.970234 5019 generic.go:356] "Generic (PLEG): container finished" podID="5fc2a87b-394a-4c26-adfd-9a57a1fc463b" containerID="6d58868d6eb7edd1c2938c35528b8a887945575e704d2f8158d3d9a1012a1f14" exitCode=0 Oct 31 05:15:24.970378 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.970263 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rdnft" Oct 31 05:15:24.970378 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.970323 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rdnft" event={"ID":"5fc2a87b-394a-4c26-adfd-9a57a1fc463b","Type":"ContainerDied","Data":"6d58868d6eb7edd1c2938c35528b8a887945575e704d2f8158d3d9a1012a1f14"} Oct 31 05:15:24.970533 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.970387 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rdnft" event={"ID":"5fc2a87b-394a-4c26-adfd-9a57a1fc463b","Type":"ContainerDied","Data":"9cb688020331b5b25269cdcabc6ef390ed966339aa5c38048b85635200eebd08"} Oct 31 05:15:24.973990 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.973961 5019 generic.go:356] "Generic (PLEG): container finished" podID="1cf9a806-d96f-478b-9ce1-2e67ec594739" containerID="1622cd76a3f47c2dc335a11e6857175d67f351df07c11d04fab8ce562097102b" exitCode=0 Oct 31 05:15:24.975420 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.974530 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq" Oct 31 05:15:24.975420 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.974720 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq" event={"ID":"1cf9a806-d96f-478b-9ce1-2e67ec594739","Type":"ContainerDied","Data":"1622cd76a3f47c2dc335a11e6857175d67f351df07c11d04fab8ce562097102b"} Oct 31 05:15:24.975420 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.974810 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq" event={"ID":"1cf9a806-d96f-478b-9ce1-2e67ec594739","Type":"ContainerDied","Data":"4ddb6b032c18ca1b4b3b2f3638fff120bfa8e2dbb708c54eaccfda6d9e660498"} Oct 31 05:15:24.990678 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:24.989089 5019 scope.go:122] "RemoveContainer" containerID="f0f1b0a9d04e4f7be16d6c8de71e04915a34ca21d3c7c2dd2a5caf4de378669e" Oct 31 05:15:25.004096 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:25.004071 5019 scope.go:122] "RemoveContainer" containerID="2e4684a3042b06f9b8f590a39685056778deeec1cd39a10a0daf25c952f3b422" Oct 31 05:15:25.004469 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:15:25.004430 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e4684a3042b06f9b8f590a39685056778deeec1cd39a10a0daf25c952f3b422\": container with ID starting with 2e4684a3042b06f9b8f590a39685056778deeec1cd39a10a0daf25c952f3b422 not found: ID does not exist" containerID="2e4684a3042b06f9b8f590a39685056778deeec1cd39a10a0daf25c952f3b422" Oct 31 05:15:25.004579 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:25.004479 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e4684a3042b06f9b8f590a39685056778deeec1cd39a10a0daf25c952f3b422"} err="failed to get container status \"2e4684a3042b06f9b8f590a39685056778deeec1cd39a10a0daf25c952f3b422\": rpc error: code = NotFound desc = could not find container \"2e4684a3042b06f9b8f590a39685056778deeec1cd39a10a0daf25c952f3b422\": container with ID starting with 2e4684a3042b06f9b8f590a39685056778deeec1cd39a10a0daf25c952f3b422 not found: ID does not exist" Oct 31 05:15:25.004579 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:25.004503 5019 scope.go:122] "RemoveContainer" containerID="f0f1b0a9d04e4f7be16d6c8de71e04915a34ca21d3c7c2dd2a5caf4de378669e" Oct 31 05:15:25.004796 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:15:25.004772 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0f1b0a9d04e4f7be16d6c8de71e04915a34ca21d3c7c2dd2a5caf4de378669e\": container with ID starting with f0f1b0a9d04e4f7be16d6c8de71e04915a34ca21d3c7c2dd2a5caf4de378669e not found: ID does not exist" containerID="f0f1b0a9d04e4f7be16d6c8de71e04915a34ca21d3c7c2dd2a5caf4de378669e" Oct 31 05:15:25.004870 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:25.004805 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0f1b0a9d04e4f7be16d6c8de71e04915a34ca21d3c7c2dd2a5caf4de378669e"} err="failed to get container status \"f0f1b0a9d04e4f7be16d6c8de71e04915a34ca21d3c7c2dd2a5caf4de378669e\": rpc error: code = NotFound desc = could not find container \"f0f1b0a9d04e4f7be16d6c8de71e04915a34ca21d3c7c2dd2a5caf4de378669e\": container with ID starting with f0f1b0a9d04e4f7be16d6c8de71e04915a34ca21d3c7c2dd2a5caf4de378669e not found: ID does not exist" Oct 31 05:15:25.004870 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:25.004819 5019 scope.go:122] "RemoveContainer" containerID="e446e225b4891af314659175ba11f9a40c9491146968b525a8fc11608e13f303" Oct 31 05:15:25.016212 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:25.016191 5019 scope.go:122] "RemoveContainer" containerID="e446e225b4891af314659175ba11f9a40c9491146968b525a8fc11608e13f303" Oct 31 05:15:25.016503 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:15:25.016483 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e446e225b4891af314659175ba11f9a40c9491146968b525a8fc11608e13f303\": container with ID starting with e446e225b4891af314659175ba11f9a40c9491146968b525a8fc11608e13f303 not found: ID does not exist" containerID="e446e225b4891af314659175ba11f9a40c9491146968b525a8fc11608e13f303" Oct 31 05:15:25.016590 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:25.016511 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e446e225b4891af314659175ba11f9a40c9491146968b525a8fc11608e13f303"} err="failed to get container status \"e446e225b4891af314659175ba11f9a40c9491146968b525a8fc11608e13f303\": rpc error: code = NotFound desc = could not find container \"e446e225b4891af314659175ba11f9a40c9491146968b525a8fc11608e13f303\": container with ID starting with e446e225b4891af314659175ba11f9a40c9491146968b525a8fc11608e13f303 not found: ID does not exist" Oct 31 05:15:25.016590 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:25.016527 5019 scope.go:122] "RemoveContainer" containerID="6d58868d6eb7edd1c2938c35528b8a887945575e704d2f8158d3d9a1012a1f14" Oct 31 05:15:25.028788 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:25.028715 5019 scope.go:122] "RemoveContainer" containerID="6d58868d6eb7edd1c2938c35528b8a887945575e704d2f8158d3d9a1012a1f14" Oct 31 05:15:25.029176 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:15:25.029150 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d58868d6eb7edd1c2938c35528b8a887945575e704d2f8158d3d9a1012a1f14\": container with ID starting with 6d58868d6eb7edd1c2938c35528b8a887945575e704d2f8158d3d9a1012a1f14 not found: ID does not exist" containerID="6d58868d6eb7edd1c2938c35528b8a887945575e704d2f8158d3d9a1012a1f14" Oct 31 05:15:25.029262 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:25.029189 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d58868d6eb7edd1c2938c35528b8a887945575e704d2f8158d3d9a1012a1f14"} err="failed to get container status \"6d58868d6eb7edd1c2938c35528b8a887945575e704d2f8158d3d9a1012a1f14\": rpc error: code = NotFound desc = could not find container \"6d58868d6eb7edd1c2938c35528b8a887945575e704d2f8158d3d9a1012a1f14\": container with ID starting with 6d58868d6eb7edd1c2938c35528b8a887945575e704d2f8158d3d9a1012a1f14 not found: ID does not exist" Oct 31 05:15:25.029262 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:25.029226 5019 scope.go:122] "RemoveContainer" containerID="1622cd76a3f47c2dc335a11e6857175d67f351df07c11d04fab8ce562097102b" Oct 31 05:15:25.039821 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:25.039804 5019 scope.go:122] "RemoveContainer" containerID="1622cd76a3f47c2dc335a11e6857175d67f351df07c11d04fab8ce562097102b" Oct 31 05:15:25.040069 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:15:25.040051 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1622cd76a3f47c2dc335a11e6857175d67f351df07c11d04fab8ce562097102b\": container with ID starting with 1622cd76a3f47c2dc335a11e6857175d67f351df07c11d04fab8ce562097102b not found: ID does not exist" containerID="1622cd76a3f47c2dc335a11e6857175d67f351df07c11d04fab8ce562097102b" Oct 31 05:15:25.040118 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:25.040077 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1622cd76a3f47c2dc335a11e6857175d67f351df07c11d04fab8ce562097102b"} err="failed to get container status \"1622cd76a3f47c2dc335a11e6857175d67f351df07c11d04fab8ce562097102b\": rpc error: code = NotFound desc = could not find container \"1622cd76a3f47c2dc335a11e6857175d67f351df07c11d04fab8ce562097102b\": container with ID starting with 1622cd76a3f47c2dc335a11e6857175d67f351df07c11d04fab8ce562097102b not found: ID does not exist" Oct 31 05:15:25.072378 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:25.069111 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7"] Oct 31 05:15:25.073668 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:25.073638 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pghv7"] Oct 31 05:15:25.112504 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:25.112430 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rdnft"] Oct 31 05:15:25.118219 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:25.118188 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rdnft"] Oct 31 05:15:25.145465 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:25.145435 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq"] Oct 31 05:15:25.151952 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:25.151923 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pr6rq"] Oct 31 05:15:25.155125 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:25.155097 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91889c70-sjqvh" Oct 31 05:15:25.165053 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:25.165031 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2648aa75-lmdxc" Oct 31 05:15:25.183293 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:25.183270 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-67f7j"] Oct 31 05:15:25.188557 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:25.188535 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-67f7j"] Oct 31 05:15:25.260914 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:25.260889 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67d355b0-4e8e-4b21-8250-aba5bbeb4932-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67d355b0-4e8e-4b21-8250-aba5bbeb4932-maas-api-openshift-ca\") pod \"67d355b0-4e8e-4b21-8250-aba5bbeb4932\" (UID: \"67d355b0-4e8e-4b21-8250-aba5bbeb4932\") " Oct 31 05:15:25.261059 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:25.260952 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f61145c7-2f3f-4072-b329-5760f95c042f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f61145c7-2f3f-4072-b329-5760f95c042f-maas-api-openshift-ca\") pod \"f61145c7-2f3f-4072-b329-5760f95c042f\" (UID: \"f61145c7-2f3f-4072-b329-5760f95c042f\") " Oct 31 05:15:25.261059 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:25.261019 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f61145c7-2f3f-4072-b329-5760f95c042f-kube-api-access-g4gn5\" (UniqueName: \"kubernetes.io/projected/f61145c7-2f3f-4072-b329-5760f95c042f-kube-api-access-g4gn5\") pod \"f61145c7-2f3f-4072-b329-5760f95c042f\" (UID: \"f61145c7-2f3f-4072-b329-5760f95c042f\") " Oct 31 05:15:25.261059 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:25.261051 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67d355b0-4e8e-4b21-8250-aba5bbeb4932-kube-api-access-hc2pz\" (UniqueName: \"kubernetes.io/projected/67d355b0-4e8e-4b21-8250-aba5bbeb4932-kube-api-access-hc2pz\") pod \"67d355b0-4e8e-4b21-8250-aba5bbeb4932\" (UID: \"67d355b0-4e8e-4b21-8250-aba5bbeb4932\") " Oct 31 05:15:25.261235 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:25.261103 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f61145c7-2f3f-4072-b329-5760f95c042f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f61145c7-2f3f-4072-b329-5760f95c042f-maas-api-xks-ca\") pod \"f61145c7-2f3f-4072-b329-5760f95c042f\" (UID: \"f61145c7-2f3f-4072-b329-5760f95c042f\") " Oct 31 05:15:25.261235 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:25.261162 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67d355b0-4e8e-4b21-8250-aba5bbeb4932-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67d355b0-4e8e-4b21-8250-aba5bbeb4932-maas-api-xks-ca\") pod \"67d355b0-4e8e-4b21-8250-aba5bbeb4932\" (UID: \"67d355b0-4e8e-4b21-8250-aba5bbeb4932\") " Oct 31 05:15:25.261357 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:25.261275 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67d355b0-4e8e-4b21-8250-aba5bbeb4932-maas-api-openshift-ca" pod "67d355b0-4e8e-4b21-8250-aba5bbeb4932" (UID: "67d355b0-4e8e-4b21-8250-aba5bbeb4932"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:15:25.261766 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:25.261735 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67d355b0-4e8e-4b21-8250-aba5bbeb4932-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:25.261988 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:25.261964 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f61145c7-2f3f-4072-b329-5760f95c042f-maas-api-openshift-ca" pod "f61145c7-2f3f-4072-b329-5760f95c042f" (UID: "f61145c7-2f3f-4072-b329-5760f95c042f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:15:25.264061 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:25.263972 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67d355b0-4e8e-4b21-8250-aba5bbeb4932-maas-api-xks-ca" pod "67d355b0-4e8e-4b21-8250-aba5bbeb4932" (UID: "67d355b0-4e8e-4b21-8250-aba5bbeb4932"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:15:25.264061 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:25.263984 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f61145c7-2f3f-4072-b329-5760f95c042f-maas-api-xks-ca" pod "f61145c7-2f3f-4072-b329-5760f95c042f" (UID: "f61145c7-2f3f-4072-b329-5760f95c042f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:15:25.264061 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:25.264038 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f61145c7-2f3f-4072-b329-5760f95c042f-kube-api-access-g4gn5" pod "f61145c7-2f3f-4072-b329-5760f95c042f" (UID: "f61145c7-2f3f-4072-b329-5760f95c042f"). InnerVolumeSpecName "kube-api-access-g4gn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:15:25.264592 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:25.264570 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67d355b0-4e8e-4b21-8250-aba5bbeb4932-kube-api-access-hc2pz" pod "67d355b0-4e8e-4b21-8250-aba5bbeb4932" (UID: "67d355b0-4e8e-4b21-8250-aba5bbeb4932"). InnerVolumeSpecName "kube-api-access-hc2pz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:15:25.362985 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:25.362898 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f61145c7-2f3f-4072-b329-5760f95c042f-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:25.362985 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:25.362926 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4gn5\" (UniqueName: \"kubernetes.io/projected/f61145c7-2f3f-4072-b329-5760f95c042f-kube-api-access-g4gn5\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:25.362985 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:25.362936 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc2pz\" (UniqueName: \"kubernetes.io/projected/67d355b0-4e8e-4b21-8250-aba5bbeb4932-kube-api-access-hc2pz\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:25.362985 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:25.362945 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f61145c7-2f3f-4072-b329-5760f95c042f-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:25.362985 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:25.362953 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67d355b0-4e8e-4b21-8250-aba5bbeb4932-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:25.408437 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:25.408401 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cf9a806-d96f-478b-9ce1-2e67ec594739" path="/var/lib/kubelet/pods/1cf9a806-d96f-478b-9ce1-2e67ec594739/volumes" Oct 31 05:15:25.408888 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:25.408870 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fc2a87b-394a-4c26-adfd-9a57a1fc463b" path="/var/lib/kubelet/pods/5fc2a87b-394a-4c26-adfd-9a57a1fc463b/volumes" Oct 31 05:15:25.409298 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:25.409284 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9573de57-8748-44eb-8949-b55b7dca0043" path="/var/lib/kubelet/pods/9573de57-8748-44eb-8949-b55b7dca0043/volumes" Oct 31 05:15:25.410052 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:25.410037 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0ccc03e-0e5e-40b7-8a40-718f199dd747" path="/var/lib/kubelet/pods/e0ccc03e-0e5e-40b7-8a40-718f199dd747/volumes" Oct 31 05:15:25.985464 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:25.985431 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91889c70-sjqvh" Oct 31 05:15:25.985464 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:25.985445 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91889c70-sjqvh" event={"ID":"67d355b0-4e8e-4b21-8250-aba5bbeb4932","Type":"ContainerDied","Data":"fba0e2a9ed031315ec02d85208f8b73d14318b784b55e867739cf7e3237a0283"} Oct 31 05:15:25.985960 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:25.985501 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fba0e2a9ed031315ec02d85208f8b73d14318b784b55e867739cf7e3237a0283" Oct 31 05:15:25.987199 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:25.987170 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2648aa75-lmdxc" event={"ID":"f61145c7-2f3f-4072-b329-5760f95c042f","Type":"ContainerDied","Data":"4242c3d5aac8564c043858182c89df5c5682b7e20debe566aa7f76858e6aae6a"} Oct 31 05:15:25.987199 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:25.987188 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2648aa75-lmdxc" Oct 31 05:15:25.987442 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:25.987220 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4242c3d5aac8564c043858182c89df5c5682b7e20debe566aa7f76858e6aae6a" Oct 31 05:15:28.067582 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:28.067543 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9633f0-54d5b879f5-qg5lc" podUID="e1803197-fd95-4f13-9cdb-4a9386cf9610" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:15:31.687731 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:31.687693 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-871d7380-rmvwj"] Oct 31 05:15:31.688468 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:31.688446 5019 cpu_manager.go:414] "containerMap: removing container" podUID="5fc2a87b-394a-4c26-adfd-9a57a1fc463b" containerName="istio-proxy" Oct 31 05:15:31.688468 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:31.688468 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fc2a87b-394a-4c26-adfd-9a57a1fc463b" containerName="istio-proxy" Oct 31 05:15:31.688630 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:31.688511 5019 cpu_manager.go:414] "containerMap: removing container" podUID="67d355b0-4e8e-4b21-8250-aba5bbeb4932" containerName="revoke-keys" Oct 31 05:15:31.688630 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:31.688520 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67d355b0-4e8e-4b21-8250-aba5bbeb4932" containerName="revoke-keys" Oct 31 05:15:31.688630 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:31.688537 5019 cpu_manager.go:414] "containerMap: removing container" podUID="9573de57-8748-44eb-8949-b55b7dca0043" containerName="istio-proxy" Oct 31 05:15:31.688630 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:31.688543 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9573de57-8748-44eb-8949-b55b7dca0043" containerName="istio-proxy" Oct 31 05:15:31.688630 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:31.688582 5019 cpu_manager.go:414] "containerMap: removing container" podUID="1cf9a806-d96f-478b-9ce1-2e67ec594739" containerName="istio-proxy" Oct 31 05:15:31.688630 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:31.688590 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cf9a806-d96f-478b-9ce1-2e67ec594739" containerName="istio-proxy" Oct 31 05:15:31.688630 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:31.688611 5019 cpu_manager.go:414] "containerMap: removing container" podUID="f61145c7-2f3f-4072-b329-5760f95c042f" containerName="revoke-keys" Oct 31 05:15:31.688630 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:31.688619 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f61145c7-2f3f-4072-b329-5760f95c042f" containerName="revoke-keys" Oct 31 05:15:31.689063 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:31.688635 5019 cpu_manager.go:414] "containerMap: removing container" podUID="e0ccc03e-0e5e-40b7-8a40-718f199dd747" containerName="istio-proxy" Oct 31 05:15:31.689063 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:31.688642 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0ccc03e-0e5e-40b7-8a40-718f199dd747" containerName="istio-proxy" Oct 31 05:15:31.689063 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:31.688656 5019 cpu_manager.go:414] "containerMap: removing container" podUID="e0ccc03e-0e5e-40b7-8a40-718f199dd747" containerName="istio-proxy" Oct 31 05:15:31.689063 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:31.688663 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0ccc03e-0e5e-40b7-8a40-718f199dd747" containerName="istio-proxy" Oct 31 05:15:31.689063 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:31.688797 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cf9a806-d96f-478b-9ce1-2e67ec594739" containerName="istio-proxy" Oct 31 05:15:31.689063 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:31.688814 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="f61145c7-2f3f-4072-b329-5760f95c042f" containerName="revoke-keys" Oct 31 05:15:31.689063 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:31.688827 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0ccc03e-0e5e-40b7-8a40-718f199dd747" containerName="istio-proxy" Oct 31 05:15:31.689063 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:31.688837 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0ccc03e-0e5e-40b7-8a40-718f199dd747" containerName="istio-proxy" Oct 31 05:15:31.689063 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:31.688847 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fc2a87b-394a-4c26-adfd-9a57a1fc463b" containerName="istio-proxy" Oct 31 05:15:31.689063 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:31.688861 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="67d355b0-4e8e-4b21-8250-aba5bbeb4932" containerName="revoke-keys" Oct 31 05:15:31.689063 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:31.688871 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="9573de57-8748-44eb-8949-b55b7dca0043" containerName="istio-proxy" Oct 31 05:15:31.694047 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:31.694024 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-871d7380-rmvwj" Oct 31 05:15:31.711737 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:31.711709 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-871d7380-rmvwj"] Oct 31 05:15:31.718998 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:31.718975 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:15:31.719127 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:31.719074 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 05:15:31.719189 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:31.719148 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:15:31.723473 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:31.723452 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:15:31.727950 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:31.727928 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/758a18f6-0240-4fc6-afcf-7c8180a71733-maas-api-xks-ca\") pod \"maas-api-revoke-sub-871d7380-rmvwj\" (UID: \"758a18f6-0240-4fc6-afcf-7c8180a71733\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-871d7380-rmvwj" Oct 31 05:15:31.728105 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:31.728078 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ztft\" (UniqueName: \"kubernetes.io/projected/758a18f6-0240-4fc6-afcf-7c8180a71733-kube-api-access-6ztft\") pod \"maas-api-revoke-sub-871d7380-rmvwj\" (UID: \"758a18f6-0240-4fc6-afcf-7c8180a71733\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-871d7380-rmvwj" Oct 31 05:15:31.728262 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:31.728242 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/758a18f6-0240-4fc6-afcf-7c8180a71733-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-871d7380-rmvwj\" (UID: \"758a18f6-0240-4fc6-afcf-7c8180a71733\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-871d7380-rmvwj" Oct 31 05:15:31.829935 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:31.829893 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ztft\" (UniqueName: \"kubernetes.io/projected/758a18f6-0240-4fc6-afcf-7c8180a71733-kube-api-access-6ztft\") pod \"maas-api-revoke-sub-871d7380-rmvwj\" (UID: \"758a18f6-0240-4fc6-afcf-7c8180a71733\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-871d7380-rmvwj" Oct 31 05:15:31.830119 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:31.830070 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/758a18f6-0240-4fc6-afcf-7c8180a71733-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-871d7380-rmvwj\" (UID: \"758a18f6-0240-4fc6-afcf-7c8180a71733\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-871d7380-rmvwj" Oct 31 05:15:31.830207 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:31.830151 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/758a18f6-0240-4fc6-afcf-7c8180a71733-maas-api-xks-ca\") pod \"maas-api-revoke-sub-871d7380-rmvwj\" (UID: \"758a18f6-0240-4fc6-afcf-7c8180a71733\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-871d7380-rmvwj" Oct 31 05:15:31.830982 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:31.830950 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/758a18f6-0240-4fc6-afcf-7c8180a71733-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-871d7380-rmvwj\" (UID: \"758a18f6-0240-4fc6-afcf-7c8180a71733\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-871d7380-rmvwj" Oct 31 05:15:31.832760 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:31.832731 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/758a18f6-0240-4fc6-afcf-7c8180a71733-maas-api-xks-ca\") pod \"maas-api-revoke-sub-871d7380-rmvwj\" (UID: \"758a18f6-0240-4fc6-afcf-7c8180a71733\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-871d7380-rmvwj" Oct 31 05:15:31.852898 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:31.852865 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ztft\" (UniqueName: \"kubernetes.io/projected/758a18f6-0240-4fc6-afcf-7c8180a71733-kube-api-access-6ztft\") pod \"maas-api-revoke-sub-871d7380-rmvwj\" (UID: \"758a18f6-0240-4fc6-afcf-7c8180a71733\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-871d7380-rmvwj" Oct 31 05:15:32.013393 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:32.013350 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-871d7380-rmvwj" Oct 31 05:15:32.318936 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:32.318899 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-871d7380-rmvwj"] Oct 31 05:15:32.319088 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:15:32.318932 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod758a18f6_0240_4fc6_afcf_7c8180a71733.slice/crio-f8802499de8102444537f1bd14988802a4b9a152da0fb7cd9eac40d8f012a697 WatchSource:0}: Error finding container f8802499de8102444537f1bd14988802a4b9a152da0fb7cd9eac40d8f012a697: Status 404 returned error can't find the container with id f8802499de8102444537f1bd14988802a4b9a152da0fb7cd9eac40d8f012a697 Oct 31 05:15:32.685052 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:32.685025 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59f0eb97-xt2rg"] Oct 31 05:15:32.690076 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:32.690056 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f0eb97-xt2rg" Oct 31 05:15:32.713359 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:32.713306 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59f0eb97-xt2rg"] Oct 31 05:15:32.742572 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:32.742541 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6558f\" (UniqueName: \"kubernetes.io/projected/1b4bf2ad-96b8-4db2-ac9c-38dc92511e29-kube-api-access-6558f\") pod \"maas-api-revoke-sub-59f0eb97-xt2rg\" (UID: \"1b4bf2ad-96b8-4db2-ac9c-38dc92511e29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f0eb97-xt2rg" Oct 31 05:15:32.742739 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:32.742713 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b4bf2ad-96b8-4db2-ac9c-38dc92511e29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59f0eb97-xt2rg\" (UID: \"1b4bf2ad-96b8-4db2-ac9c-38dc92511e29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f0eb97-xt2rg" Oct 31 05:15:32.742839 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:32.742825 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b4bf2ad-96b8-4db2-ac9c-38dc92511e29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59f0eb97-xt2rg\" (UID: \"1b4bf2ad-96b8-4db2-ac9c-38dc92511e29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f0eb97-xt2rg" Oct 31 05:15:32.843923 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:32.843885 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b4bf2ad-96b8-4db2-ac9c-38dc92511e29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59f0eb97-xt2rg\" (UID: \"1b4bf2ad-96b8-4db2-ac9c-38dc92511e29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f0eb97-xt2rg" Oct 31 05:15:32.844077 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:32.843987 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6558f\" (UniqueName: \"kubernetes.io/projected/1b4bf2ad-96b8-4db2-ac9c-38dc92511e29-kube-api-access-6558f\") pod \"maas-api-revoke-sub-59f0eb97-xt2rg\" (UID: \"1b4bf2ad-96b8-4db2-ac9c-38dc92511e29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f0eb97-xt2rg" Oct 31 05:15:32.844176 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:32.844159 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b4bf2ad-96b8-4db2-ac9c-38dc92511e29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59f0eb97-xt2rg\" (UID: \"1b4bf2ad-96b8-4db2-ac9c-38dc92511e29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f0eb97-xt2rg" Oct 31 05:15:32.844799 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:32.844772 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b4bf2ad-96b8-4db2-ac9c-38dc92511e29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59f0eb97-xt2rg\" (UID: \"1b4bf2ad-96b8-4db2-ac9c-38dc92511e29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f0eb97-xt2rg" Oct 31 05:15:32.846323 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:32.846304 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b4bf2ad-96b8-4db2-ac9c-38dc92511e29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59f0eb97-xt2rg\" (UID: \"1b4bf2ad-96b8-4db2-ac9c-38dc92511e29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f0eb97-xt2rg" Oct 31 05:15:32.870175 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:32.870143 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6558f\" (UniqueName: \"kubernetes.io/projected/1b4bf2ad-96b8-4db2-ac9c-38dc92511e29-kube-api-access-6558f\") pod \"maas-api-revoke-sub-59f0eb97-xt2rg\" (UID: \"1b4bf2ad-96b8-4db2-ac9c-38dc92511e29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f0eb97-xt2rg" Oct 31 05:15:33.004345 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:33.004307 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f0eb97-xt2rg" Oct 31 05:15:33.050259 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:33.050222 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-871d7380-rmvwj" event={"ID":"758a18f6-0240-4fc6-afcf-7c8180a71733","Type":"ContainerStarted","Data":"b3610314f13849689a9369afe3201510524dbc3fe10d2b1c392ec16b9258cd94"} Oct 31 05:15:33.050450 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:33.050277 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-871d7380-rmvwj" event={"ID":"758a18f6-0240-4fc6-afcf-7c8180a71733","Type":"ContainerStarted","Data":"f8802499de8102444537f1bd14988802a4b9a152da0fb7cd9eac40d8f012a697"} Oct 31 05:15:33.094702 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:33.092908 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-871d7380-rmvwj" podStartSLOduration=2.09231275 podStartE2EDuration="2.09231275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:15:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:15:33.090725643 +0000 UTC m=+4287.847063685" watchObservedRunningTime="2026-10-31 05:15:33.09231275 +0000 UTC m=+4287.848650790" Oct 31 05:15:33.315109 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:33.315079 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59f0eb97-xt2rg"] Oct 31 05:15:33.316296 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:15:33.316268 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b4bf2ad_96b8_4db2_ac9c_38dc92511e29.slice/crio-b8877acbafcd8c9939197205c80abe0a41cbadd34bdaf1cde3d4c70ac23768e6 WatchSource:0}: Error finding container b8877acbafcd8c9939197205c80abe0a41cbadd34bdaf1cde3d4c70ac23768e6: Status 404 returned error can't find the container with id b8877acbafcd8c9939197205c80abe0a41cbadd34bdaf1cde3d4c70ac23768e6 Oct 31 05:15:34.062218 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:34.062174 5019 generic.go:356] "Generic (PLEG): container finished" podID="758a18f6-0240-4fc6-afcf-7c8180a71733" containerID="b3610314f13849689a9369afe3201510524dbc3fe10d2b1c392ec16b9258cd94" exitCode=0 Oct 31 05:15:34.062718 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:34.062276 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-871d7380-rmvwj" event={"ID":"758a18f6-0240-4fc6-afcf-7c8180a71733","Type":"ContainerDied","Data":"b3610314f13849689a9369afe3201510524dbc3fe10d2b1c392ec16b9258cd94"} Oct 31 05:15:34.064616 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:34.064586 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f0eb97-xt2rg" event={"ID":"1b4bf2ad-96b8-4db2-ac9c-38dc92511e29","Type":"ContainerStarted","Data":"c4cbf0a155f6154253b7d3284455ba59a265dee7246a5cd24172b16cda7e65aa"} Oct 31 05:15:34.064725 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:34.064644 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f0eb97-xt2rg" event={"ID":"1b4bf2ad-96b8-4db2-ac9c-38dc92511e29","Type":"ContainerStarted","Data":"b8877acbafcd8c9939197205c80abe0a41cbadd34bdaf1cde3d4c70ac23768e6"} Oct 31 05:15:35.074900 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:35.074862 5019 generic.go:356] "Generic (PLEG): container finished" podID="1b4bf2ad-96b8-4db2-ac9c-38dc92511e29" containerID="c4cbf0a155f6154253b7d3284455ba59a265dee7246a5cd24172b16cda7e65aa" exitCode=0 Oct 31 05:15:35.075287 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:35.074952 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f0eb97-xt2rg" event={"ID":"1b4bf2ad-96b8-4db2-ac9c-38dc92511e29","Type":"ContainerDied","Data":"c4cbf0a155f6154253b7d3284455ba59a265dee7246a5cd24172b16cda7e65aa"} Oct 31 05:15:35.263948 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:35.263919 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-871d7380-rmvwj" Oct 31 05:15:35.378585 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:35.378511 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/758a18f6-0240-4fc6-afcf-7c8180a71733-kube-api-access-6ztft\" (UniqueName: \"kubernetes.io/projected/758a18f6-0240-4fc6-afcf-7c8180a71733-kube-api-access-6ztft\") pod \"758a18f6-0240-4fc6-afcf-7c8180a71733\" (UID: \"758a18f6-0240-4fc6-afcf-7c8180a71733\") " Oct 31 05:15:35.378738 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:35.378602 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/758a18f6-0240-4fc6-afcf-7c8180a71733-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/758a18f6-0240-4fc6-afcf-7c8180a71733-maas-api-xks-ca\") pod \"758a18f6-0240-4fc6-afcf-7c8180a71733\" (UID: \"758a18f6-0240-4fc6-afcf-7c8180a71733\") " Oct 31 05:15:35.378738 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:35.378706 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/758a18f6-0240-4fc6-afcf-7c8180a71733-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/758a18f6-0240-4fc6-afcf-7c8180a71733-maas-api-openshift-ca\") pod \"758a18f6-0240-4fc6-afcf-7c8180a71733\" (UID: \"758a18f6-0240-4fc6-afcf-7c8180a71733\") " Oct 31 05:15:35.379032 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:35.379006 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/758a18f6-0240-4fc6-afcf-7c8180a71733-maas-api-openshift-ca" pod "758a18f6-0240-4fc6-afcf-7c8180a71733" (UID: "758a18f6-0240-4fc6-afcf-7c8180a71733"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:15:35.379231 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:35.379203 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/758a18f6-0240-4fc6-afcf-7c8180a71733-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:35.380660 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:35.380630 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/758a18f6-0240-4fc6-afcf-7c8180a71733-kube-api-access-6ztft" pod "758a18f6-0240-4fc6-afcf-7c8180a71733" (UID: "758a18f6-0240-4fc6-afcf-7c8180a71733"). InnerVolumeSpecName "kube-api-access-6ztft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:15:35.380780 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:35.380695 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/758a18f6-0240-4fc6-afcf-7c8180a71733-maas-api-xks-ca" pod "758a18f6-0240-4fc6-afcf-7c8180a71733" (UID: "758a18f6-0240-4fc6-afcf-7c8180a71733"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:15:35.480173 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:35.480145 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/758a18f6-0240-4fc6-afcf-7c8180a71733-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:35.480173 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:35.480169 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ztft\" (UniqueName: \"kubernetes.io/projected/758a18f6-0240-4fc6-afcf-7c8180a71733-kube-api-access-6ztft\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:36.085138 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:36.085106 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-871d7380-rmvwj" Oct 31 05:15:36.085635 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:36.085140 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-871d7380-rmvwj" event={"ID":"758a18f6-0240-4fc6-afcf-7c8180a71733","Type":"ContainerDied","Data":"f8802499de8102444537f1bd14988802a4b9a152da0fb7cd9eac40d8f012a697"} Oct 31 05:15:36.085635 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:36.085181 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8802499de8102444537f1bd14988802a4b9a152da0fb7cd9eac40d8f012a697" Oct 31 05:15:36.238015 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:36.237990 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f0eb97-xt2rg" Oct 31 05:15:36.287704 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:36.287672 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b4bf2ad-96b8-4db2-ac9c-38dc92511e29-kube-api-access-6558f\" (UniqueName: \"kubernetes.io/projected/1b4bf2ad-96b8-4db2-ac9c-38dc92511e29-kube-api-access-6558f\") pod \"1b4bf2ad-96b8-4db2-ac9c-38dc92511e29\" (UID: \"1b4bf2ad-96b8-4db2-ac9c-38dc92511e29\") " Oct 31 05:15:36.287840 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:36.287719 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b4bf2ad-96b8-4db2-ac9c-38dc92511e29-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b4bf2ad-96b8-4db2-ac9c-38dc92511e29-maas-api-xks-ca\") pod \"1b4bf2ad-96b8-4db2-ac9c-38dc92511e29\" (UID: \"1b4bf2ad-96b8-4db2-ac9c-38dc92511e29\") " Oct 31 05:15:36.287939 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:36.287923 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b4bf2ad-96b8-4db2-ac9c-38dc92511e29-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b4bf2ad-96b8-4db2-ac9c-38dc92511e29-maas-api-openshift-ca\") pod \"1b4bf2ad-96b8-4db2-ac9c-38dc92511e29\" (UID: \"1b4bf2ad-96b8-4db2-ac9c-38dc92511e29\") " Oct 31 05:15:36.288588 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:36.288557 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b4bf2ad-96b8-4db2-ac9c-38dc92511e29-maas-api-openshift-ca" pod "1b4bf2ad-96b8-4db2-ac9c-38dc92511e29" (UID: "1b4bf2ad-96b8-4db2-ac9c-38dc92511e29"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:15:36.290087 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:36.290060 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b4bf2ad-96b8-4db2-ac9c-38dc92511e29-maas-api-xks-ca" pod "1b4bf2ad-96b8-4db2-ac9c-38dc92511e29" (UID: "1b4bf2ad-96b8-4db2-ac9c-38dc92511e29"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:15:36.290262 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:36.290242 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b4bf2ad-96b8-4db2-ac9c-38dc92511e29-kube-api-access-6558f" pod "1b4bf2ad-96b8-4db2-ac9c-38dc92511e29" (UID: "1b4bf2ad-96b8-4db2-ac9c-38dc92511e29"). InnerVolumeSpecName "kube-api-access-6558f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:15:36.389839 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:36.389780 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b4bf2ad-96b8-4db2-ac9c-38dc92511e29-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:36.389839 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:36.389813 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6558f\" (UniqueName: \"kubernetes.io/projected/1b4bf2ad-96b8-4db2-ac9c-38dc92511e29-kube-api-access-6558f\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:36.389839 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:36.389823 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b4bf2ad-96b8-4db2-ac9c-38dc92511e29-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:37.094852 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:37.094819 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f0eb97-xt2rg" Oct 31 05:15:37.095228 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:37.094843 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f0eb97-xt2rg" event={"ID":"1b4bf2ad-96b8-4db2-ac9c-38dc92511e29","Type":"ContainerDied","Data":"b8877acbafcd8c9939197205c80abe0a41cbadd34bdaf1cde3d4c70ac23768e6"} Oct 31 05:15:37.095228 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:37.094901 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8877acbafcd8c9939197205c80abe0a41cbadd34bdaf1cde3d4c70ac23768e6" Oct 31 05:15:38.067589 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:38.067548 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9633f0-54d5b879f5-qg5lc" podUID="e1803197-fd95-4f13-9cdb-4a9386cf9610" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:15:39.783134 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:39.783081 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6"] Oct 31 05:15:39.783700 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:39.783396 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6" podUID="68b1965c-0483-419e-9e73-aacf0f6a5048" containerName="istio-proxy" containerID="cri-o://c7aee1c0f539e53fdd244f018e7f9224350d313b492072495922872fb4aa6e97" gracePeriod=30 Oct 31 05:15:40.745977 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:40.745943 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9633f0-54d5b879f5-qg5lc" Oct 31 05:15:40.805189 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:40.805139 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f887033-pwvhc"] Oct 31 05:15:40.805999 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:40.805969 5019 cpu_manager.go:414] "containerMap: removing container" podUID="758a18f6-0240-4fc6-afcf-7c8180a71733" containerName="revoke-keys" Oct 31 05:15:40.806104 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:40.805999 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="758a18f6-0240-4fc6-afcf-7c8180a71733" containerName="revoke-keys" Oct 31 05:15:40.806104 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:40.806059 5019 cpu_manager.go:414] "containerMap: removing container" podUID="1b4bf2ad-96b8-4db2-ac9c-38dc92511e29" containerName="revoke-keys" Oct 31 05:15:40.806104 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:40.806069 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b4bf2ad-96b8-4db2-ac9c-38dc92511e29" containerName="revoke-keys" Oct 31 05:15:40.806297 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:40.806110 5019 cpu_manager.go:414] "containerMap: removing container" podUID="e1803197-fd95-4f13-9cdb-4a9386cf9610" containerName="payload-pre-processing" Oct 31 05:15:40.806297 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:40.806118 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1803197-fd95-4f13-9cdb-4a9386cf9610" containerName="payload-pre-processing" Oct 31 05:15:40.806297 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:40.806257 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b4bf2ad-96b8-4db2-ac9c-38dc92511e29" containerName="revoke-keys" Oct 31 05:15:40.806297 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:40.806280 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1803197-fd95-4f13-9cdb-4a9386cf9610" containerName="payload-pre-processing" Oct 31 05:15:40.806297 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:40.806294 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="758a18f6-0240-4fc6-afcf-7c8180a71733" containerName="revoke-keys" Oct 31 05:15:40.810006 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:40.809974 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f887033-pwvhc" Oct 31 05:15:40.825823 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:40.825712 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:15:40.825823 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:40.825787 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:15:40.825823 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:40.825722 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:15:40.826975 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:40.826918 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 05:15:40.829032 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:40.828994 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f887033-pwvhc"] Oct 31 05:15:40.839850 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:40.839807 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1803197-fd95-4f13-9cdb-4a9386cf9610-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1803197-fd95-4f13-9cdb-4a9386cf9610-plugins-config-volume\") pod \"e1803197-fd95-4f13-9cdb-4a9386cf9610\" (UID: \"e1803197-fd95-4f13-9cdb-4a9386cf9610\") " Oct 31 05:15:40.840068 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:40.839966 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1803197-fd95-4f13-9cdb-4a9386cf9610-kube-api-access-xc82s\" (UniqueName: \"kubernetes.io/projected/e1803197-fd95-4f13-9cdb-4a9386cf9610-kube-api-access-xc82s\") pod \"e1803197-fd95-4f13-9cdb-4a9386cf9610\" (UID: \"e1803197-fd95-4f13-9cdb-4a9386cf9610\") " Oct 31 05:15:40.840302 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:40.840263 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1803197-fd95-4f13-9cdb-4a9386cf9610-plugins-config-volume" pod "e1803197-fd95-4f13-9cdb-4a9386cf9610" (UID: "e1803197-fd95-4f13-9cdb-4a9386cf9610"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:15:40.840600 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:40.840581 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1803197-fd95-4f13-9cdb-4a9386cf9610-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:40.842597 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:40.842565 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1803197-fd95-4f13-9cdb-4a9386cf9610-kube-api-access-xc82s" pod "e1803197-fd95-4f13-9cdb-4a9386cf9610" (UID: "e1803197-fd95-4f13-9cdb-4a9386cf9610"). InnerVolumeSpecName "kube-api-access-xc82s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:15:40.941557 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:40.941513 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpng7\" (UniqueName: \"kubernetes.io/projected/edc66a25-6634-4ccf-bc03-a29e87a8ac4b-kube-api-access-bpng7\") pod \"maas-api-revoke-sub-1f887033-pwvhc\" (UID: \"edc66a25-6634-4ccf-bc03-a29e87a8ac4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f887033-pwvhc" Oct 31 05:15:40.941774 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:40.941710 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edc66a25-6634-4ccf-bc03-a29e87a8ac4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f887033-pwvhc\" (UID: \"edc66a25-6634-4ccf-bc03-a29e87a8ac4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f887033-pwvhc" Oct 31 05:15:40.941830 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:40.941777 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edc66a25-6634-4ccf-bc03-a29e87a8ac4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f887033-pwvhc\" (UID: \"edc66a25-6634-4ccf-bc03-a29e87a8ac4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f887033-pwvhc" Oct 31 05:15:40.941908 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:40.941884 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xc82s\" (UniqueName: \"kubernetes.io/projected/e1803197-fd95-4f13-9cdb-4a9386cf9610-kube-api-access-xc82s\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:41.028060 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:41.028015 5019 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 05:15:41.028230 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:41.028085 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6" podUID="68b1965c-0483-419e-9e73-aacf0f6a5048" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 05:15:41.042912 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:41.042856 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edc66a25-6634-4ccf-bc03-a29e87a8ac4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f887033-pwvhc\" (UID: \"edc66a25-6634-4ccf-bc03-a29e87a8ac4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f887033-pwvhc" Oct 31 05:15:41.043151 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:41.042946 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edc66a25-6634-4ccf-bc03-a29e87a8ac4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f887033-pwvhc\" (UID: \"edc66a25-6634-4ccf-bc03-a29e87a8ac4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f887033-pwvhc" Oct 31 05:15:41.043151 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:41.042990 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpng7\" (UniqueName: \"kubernetes.io/projected/edc66a25-6634-4ccf-bc03-a29e87a8ac4b-kube-api-access-bpng7\") pod \"maas-api-revoke-sub-1f887033-pwvhc\" (UID: \"edc66a25-6634-4ccf-bc03-a29e87a8ac4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f887033-pwvhc" Oct 31 05:15:41.045684 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:41.044669 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edc66a25-6634-4ccf-bc03-a29e87a8ac4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f887033-pwvhc\" (UID: \"edc66a25-6634-4ccf-bc03-a29e87a8ac4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f887033-pwvhc" Oct 31 05:15:41.046424 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:41.046387 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edc66a25-6634-4ccf-bc03-a29e87a8ac4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f887033-pwvhc\" (UID: \"edc66a25-6634-4ccf-bc03-a29e87a8ac4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f887033-pwvhc" Oct 31 05:15:41.066363 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:41.066315 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpng7\" (UniqueName: \"kubernetes.io/projected/edc66a25-6634-4ccf-bc03-a29e87a8ac4b-kube-api-access-bpng7\") pod \"maas-api-revoke-sub-1f887033-pwvhc\" (UID: \"edc66a25-6634-4ccf-bc03-a29e87a8ac4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f887033-pwvhc" Oct 31 05:15:41.130010 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:41.129963 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f887033-pwvhc" Oct 31 05:15:41.131754 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:41.131718 5019 generic.go:356] "Generic (PLEG): container finished" podID="e1803197-fd95-4f13-9cdb-4a9386cf9610" containerID="40c212a00c030b2dc932f08ec578be766e64d3566dde30b251214d9f0e7961f0" exitCode=0 Oct 31 05:15:41.131913 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:41.131764 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9633f0-54d5b879f5-qg5lc" Oct 31 05:15:41.131913 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:41.131798 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9633f0-54d5b879f5-qg5lc" event={"ID":"e1803197-fd95-4f13-9cdb-4a9386cf9610","Type":"ContainerDied","Data":"40c212a00c030b2dc932f08ec578be766e64d3566dde30b251214d9f0e7961f0"} Oct 31 05:15:41.131913 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:41.131857 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9633f0-54d5b879f5-qg5lc" event={"ID":"e1803197-fd95-4f13-9cdb-4a9386cf9610","Type":"ContainerDied","Data":"5bc2d5c2709f590083ec3f93d03f38f9920624f79bf38a021d3333e0ec96a15c"} Oct 31 05:15:41.131913 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:41.131879 5019 scope.go:122] "RemoveContainer" containerID="40c212a00c030b2dc932f08ec578be766e64d3566dde30b251214d9f0e7961f0" Oct 31 05:15:41.147452 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:41.147423 5019 scope.go:122] "RemoveContainer" containerID="40c212a00c030b2dc932f08ec578be766e64d3566dde30b251214d9f0e7961f0" Oct 31 05:15:41.147845 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:15:41.147817 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40c212a00c030b2dc932f08ec578be766e64d3566dde30b251214d9f0e7961f0\": container with ID starting with 40c212a00c030b2dc932f08ec578be766e64d3566dde30b251214d9f0e7961f0 not found: ID does not exist" containerID="40c212a00c030b2dc932f08ec578be766e64d3566dde30b251214d9f0e7961f0" Oct 31 05:15:41.147915 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:41.147855 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40c212a00c030b2dc932f08ec578be766e64d3566dde30b251214d9f0e7961f0"} err="failed to get container status \"40c212a00c030b2dc932f08ec578be766e64d3566dde30b251214d9f0e7961f0\": rpc error: code = NotFound desc = could not find container \"40c212a00c030b2dc932f08ec578be766e64d3566dde30b251214d9f0e7961f0\": container with ID starting with 40c212a00c030b2dc932f08ec578be766e64d3566dde30b251214d9f0e7961f0 not found: ID does not exist" Oct 31 05:15:41.186354 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:41.184867 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9633f0-54d5b879f5-qg5lc"] Oct 31 05:15:41.193178 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:41.193129 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9633f0-54d5b879f5-qg5lc"] Oct 31 05:15:41.412347 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:41.412230 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1803197-fd95-4f13-9cdb-4a9386cf9610" path="/var/lib/kubelet/pods/e1803197-fd95-4f13-9cdb-4a9386cf9610/volumes" Oct 31 05:15:41.456498 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:41.456458 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f887033-pwvhc"] Oct 31 05:15:41.457802 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:15:41.457766 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedc66a25_6634_4ccf_bc03_a29e87a8ac4b.slice/crio-21b2790c2dd162af67c45a733a0f4c17755e7b9ddc7fef3f7e59ad747f4a4a53 WatchSource:0}: Error finding container 21b2790c2dd162af67c45a733a0f4c17755e7b9ddc7fef3f7e59ad747f4a4a53: Status 404 returned error can't find the container with id 21b2790c2dd162af67c45a733a0f4c17755e7b9ddc7fef3f7e59ad747f4a4a53 Oct 31 05:15:42.141712 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:42.141667 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f887033-pwvhc" event={"ID":"edc66a25-6634-4ccf-bc03-a29e87a8ac4b","Type":"ContainerStarted","Data":"d583c38692505fe99d51de20ae98ab2db7ba1136e4637f8805d36e3d7e842707"} Oct 31 05:15:42.142225 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:42.141773 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f887033-pwvhc" event={"ID":"edc66a25-6634-4ccf-bc03-a29e87a8ac4b","Type":"ContainerStarted","Data":"21b2790c2dd162af67c45a733a0f4c17755e7b9ddc7fef3f7e59ad747f4a4a53"} Oct 31 05:15:42.185218 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:42.185147 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f887033-pwvhc" podStartSLOduration=2.185130868 podStartE2EDuration="2.185130868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:15:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:15:42.181206018 +0000 UTC m=+4296.937544057" watchObservedRunningTime="2026-10-31 05:15:42.185130868 +0000 UTC m=+4296.941468907" Oct 31 05:15:43.153412 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:43.153370 5019 generic.go:356] "Generic (PLEG): container finished" podID="edc66a25-6634-4ccf-bc03-a29e87a8ac4b" containerID="d583c38692505fe99d51de20ae98ab2db7ba1136e4637f8805d36e3d7e842707" exitCode=0 Oct 31 05:15:43.153882 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:43.153455 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f887033-pwvhc" event={"ID":"edc66a25-6634-4ccf-bc03-a29e87a8ac4b","Type":"ContainerDied","Data":"d583c38692505fe99d51de20ae98ab2db7ba1136e4637f8805d36e3d7e842707"} Oct 31 05:15:43.711512 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:43.711457 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a70042a9-px695"] Oct 31 05:15:43.716001 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:43.715965 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a70042a9-px695" Oct 31 05:15:43.734658 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:43.734619 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a70042a9-px695"] Oct 31 05:15:43.776925 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:43.776864 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39c4d335-5acc-4668-b4c3-57f6ff58a75b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a70042a9-px695\" (UID: \"39c4d335-5acc-4668-b4c3-57f6ff58a75b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a70042a9-px695" Oct 31 05:15:43.777163 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:43.777013 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdzpr\" (UniqueName: \"kubernetes.io/projected/39c4d335-5acc-4668-b4c3-57f6ff58a75b-kube-api-access-bdzpr\") pod \"maas-api-revoke-sub-a70042a9-px695\" (UID: \"39c4d335-5acc-4668-b4c3-57f6ff58a75b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a70042a9-px695" Oct 31 05:15:43.777163 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:43.777088 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39c4d335-5acc-4668-b4c3-57f6ff58a75b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a70042a9-px695\" (UID: \"39c4d335-5acc-4668-b4c3-57f6ff58a75b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a70042a9-px695" Oct 31 05:15:43.878743 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:43.878688 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39c4d335-5acc-4668-b4c3-57f6ff58a75b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a70042a9-px695\" (UID: \"39c4d335-5acc-4668-b4c3-57f6ff58a75b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a70042a9-px695" Oct 31 05:15:43.878950 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:43.878781 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdzpr\" (UniqueName: \"kubernetes.io/projected/39c4d335-5acc-4668-b4c3-57f6ff58a75b-kube-api-access-bdzpr\") pod \"maas-api-revoke-sub-a70042a9-px695\" (UID: \"39c4d335-5acc-4668-b4c3-57f6ff58a75b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a70042a9-px695" Oct 31 05:15:43.878950 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:43.878832 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39c4d335-5acc-4668-b4c3-57f6ff58a75b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a70042a9-px695\" (UID: \"39c4d335-5acc-4668-b4c3-57f6ff58a75b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a70042a9-px695" Oct 31 05:15:43.879416 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:43.879392 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39c4d335-5acc-4668-b4c3-57f6ff58a75b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a70042a9-px695\" (UID: \"39c4d335-5acc-4668-b4c3-57f6ff58a75b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a70042a9-px695" Oct 31 05:15:43.881480 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:43.881449 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39c4d335-5acc-4668-b4c3-57f6ff58a75b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a70042a9-px695\" (UID: \"39c4d335-5acc-4668-b4c3-57f6ff58a75b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a70042a9-px695" Oct 31 05:15:43.902017 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:43.901980 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdzpr\" (UniqueName: \"kubernetes.io/projected/39c4d335-5acc-4668-b4c3-57f6ff58a75b-kube-api-access-bdzpr\") pod \"maas-api-revoke-sub-a70042a9-px695\" (UID: \"39c4d335-5acc-4668-b4c3-57f6ff58a75b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a70042a9-px695" Oct 31 05:15:44.032593 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:44.032547 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a70042a9-px695" Oct 31 05:15:44.339457 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:44.339427 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f887033-pwvhc" Oct 31 05:15:44.388785 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:44.388743 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edc66a25-6634-4ccf-bc03-a29e87a8ac4b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edc66a25-6634-4ccf-bc03-a29e87a8ac4b-maas-api-openshift-ca\") pod \"edc66a25-6634-4ccf-bc03-a29e87a8ac4b\" (UID: \"edc66a25-6634-4ccf-bc03-a29e87a8ac4b\") " Oct 31 05:15:44.389002 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:44.388981 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edc66a25-6634-4ccf-bc03-a29e87a8ac4b-kube-api-access-bpng7\" (UniqueName: \"kubernetes.io/projected/edc66a25-6634-4ccf-bc03-a29e87a8ac4b-kube-api-access-bpng7\") pod \"edc66a25-6634-4ccf-bc03-a29e87a8ac4b\" (UID: \"edc66a25-6634-4ccf-bc03-a29e87a8ac4b\") " Oct 31 05:15:44.389074 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:44.389041 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edc66a25-6634-4ccf-bc03-a29e87a8ac4b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edc66a25-6634-4ccf-bc03-a29e87a8ac4b-maas-api-xks-ca\") pod \"edc66a25-6634-4ccf-bc03-a29e87a8ac4b\" (UID: \"edc66a25-6634-4ccf-bc03-a29e87a8ac4b\") " Oct 31 05:15:44.390557 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:44.390518 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edc66a25-6634-4ccf-bc03-a29e87a8ac4b-maas-api-openshift-ca" pod "edc66a25-6634-4ccf-bc03-a29e87a8ac4b" (UID: "edc66a25-6634-4ccf-bc03-a29e87a8ac4b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:15:44.391653 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:44.391617 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a70042a9-px695"] Oct 31 05:15:44.396624 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:44.396577 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edc66a25-6634-4ccf-bc03-a29e87a8ac4b-kube-api-access-bpng7" pod "edc66a25-6634-4ccf-bc03-a29e87a8ac4b" (UID: "edc66a25-6634-4ccf-bc03-a29e87a8ac4b"). InnerVolumeSpecName "kube-api-access-bpng7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:15:44.396624 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:44.396587 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edc66a25-6634-4ccf-bc03-a29e87a8ac4b-maas-api-xks-ca" pod "edc66a25-6634-4ccf-bc03-a29e87a8ac4b" (UID: "edc66a25-6634-4ccf-bc03-a29e87a8ac4b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:15:44.490205 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:44.490167 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpng7\" (UniqueName: \"kubernetes.io/projected/edc66a25-6634-4ccf-bc03-a29e87a8ac4b-kube-api-access-bpng7\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:44.490205 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:44.490200 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edc66a25-6634-4ccf-bc03-a29e87a8ac4b-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:44.490205 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:44.490212 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edc66a25-6634-4ccf-bc03-a29e87a8ac4b-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:45.101531 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:45.101500 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6" Oct 31 05:15:45.202611 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:45.198308 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68b1965c-0483-419e-9e73-aacf0f6a5048-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68b1965c-0483-419e-9e73-aacf0f6a5048-workload-certs\") pod \"68b1965c-0483-419e-9e73-aacf0f6a5048\" (UID: \"68b1965c-0483-419e-9e73-aacf0f6a5048\") " Oct 31 05:15:45.202611 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:45.198417 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68b1965c-0483-419e-9e73-aacf0f6a5048-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68b1965c-0483-419e-9e73-aacf0f6a5048-istio-data\") pod \"68b1965c-0483-419e-9e73-aacf0f6a5048\" (UID: \"68b1965c-0483-419e-9e73-aacf0f6a5048\") " Oct 31 05:15:45.202611 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:45.198467 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68b1965c-0483-419e-9e73-aacf0f6a5048-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68b1965c-0483-419e-9e73-aacf0f6a5048-istio-envoy\") pod \"68b1965c-0483-419e-9e73-aacf0f6a5048\" (UID: \"68b1965c-0483-419e-9e73-aacf0f6a5048\") " Oct 31 05:15:45.202611 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:45.198535 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68b1965c-0483-419e-9e73-aacf0f6a5048-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b1965c-0483-419e-9e73-aacf0f6a5048-credential-socket\") pod \"68b1965c-0483-419e-9e73-aacf0f6a5048\" (UID: \"68b1965c-0483-419e-9e73-aacf0f6a5048\") " Oct 31 05:15:45.202611 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:45.198614 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68b1965c-0483-419e-9e73-aacf0f6a5048-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68b1965c-0483-419e-9e73-aacf0f6a5048-istiod-ca-cert\") pod \"68b1965c-0483-419e-9e73-aacf0f6a5048\" (UID: \"68b1965c-0483-419e-9e73-aacf0f6a5048\") " Oct 31 05:15:45.202611 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:45.198690 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68b1965c-0483-419e-9e73-aacf0f6a5048-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b1965c-0483-419e-9e73-aacf0f6a5048-workload-socket\") pod \"68b1965c-0483-419e-9e73-aacf0f6a5048\" (UID: \"68b1965c-0483-419e-9e73-aacf0f6a5048\") " Oct 31 05:15:45.202611 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:45.198719 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68b1965c-0483-419e-9e73-aacf0f6a5048-istio-token\" (UniqueName: \"kubernetes.io/projected/68b1965c-0483-419e-9e73-aacf0f6a5048-istio-token\") pod \"68b1965c-0483-419e-9e73-aacf0f6a5048\" (UID: \"68b1965c-0483-419e-9e73-aacf0f6a5048\") " Oct 31 05:15:45.202611 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:45.198785 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/68b1965c-0483-419e-9e73-aacf0f6a5048-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68b1965c-0483-419e-9e73-aacf0f6a5048-istio-podinfo\") pod \"68b1965c-0483-419e-9e73-aacf0f6a5048\" (UID: \"68b1965c-0483-419e-9e73-aacf0f6a5048\") " Oct 31 05:15:45.202611 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:45.198863 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68b1965c-0483-419e-9e73-aacf0f6a5048-kube-api-access-dwjsw\" (UniqueName: \"kubernetes.io/projected/68b1965c-0483-419e-9e73-aacf0f6a5048-kube-api-access-dwjsw\") pod \"68b1965c-0483-419e-9e73-aacf0f6a5048\" (UID: \"68b1965c-0483-419e-9e73-aacf0f6a5048\") " Oct 31 05:15:45.206155 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:45.205876 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b1965c-0483-419e-9e73-aacf0f6a5048-workload-certs" pod "68b1965c-0483-419e-9e73-aacf0f6a5048" (UID: "68b1965c-0483-419e-9e73-aacf0f6a5048"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:15:45.206155 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:45.205887 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b1965c-0483-419e-9e73-aacf0f6a5048-workload-socket" pod "68b1965c-0483-419e-9e73-aacf0f6a5048" (UID: "68b1965c-0483-419e-9e73-aacf0f6a5048"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:15:45.206451 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:45.206202 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68b1965c-0483-419e-9e73-aacf0f6a5048-istiod-ca-cert" pod "68b1965c-0483-419e-9e73-aacf0f6a5048" (UID: "68b1965c-0483-419e-9e73-aacf0f6a5048"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:15:45.206718 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:45.206666 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b1965c-0483-419e-9e73-aacf0f6a5048-istio-data" pod "68b1965c-0483-419e-9e73-aacf0f6a5048" (UID: "68b1965c-0483-419e-9e73-aacf0f6a5048"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:15:45.208435 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:45.208404 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b1965c-0483-419e-9e73-aacf0f6a5048-istio-envoy" pod "68b1965c-0483-419e-9e73-aacf0f6a5048" (UID: "68b1965c-0483-419e-9e73-aacf0f6a5048"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:15:45.208573 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:45.208446 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68b1965c-0483-419e-9e73-aacf0f6a5048-kube-api-access-dwjsw" pod "68b1965c-0483-419e-9e73-aacf0f6a5048" (UID: "68b1965c-0483-419e-9e73-aacf0f6a5048"). InnerVolumeSpecName "kube-api-access-dwjsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:15:45.211146 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:45.211115 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68b1965c-0483-419e-9e73-aacf0f6a5048-istio-podinfo" pod "68b1965c-0483-419e-9e73-aacf0f6a5048" (UID: "68b1965c-0483-419e-9e73-aacf0f6a5048"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:15:45.211907 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:45.211880 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68b1965c-0483-419e-9e73-aacf0f6a5048-istio-token" pod "68b1965c-0483-419e-9e73-aacf0f6a5048" (UID: "68b1965c-0483-419e-9e73-aacf0f6a5048"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:15:45.212906 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:45.212859 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a70042a9-px695" event={"ID":"39c4d335-5acc-4668-b4c3-57f6ff58a75b","Type":"ContainerStarted","Data":"119c94048c9fb31654a855ad873133797c758d31f12e9e49a58aaa7541aa395e"} Oct 31 05:15:45.213082 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:45.212970 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a70042a9-px695" event={"ID":"39c4d335-5acc-4668-b4c3-57f6ff58a75b","Type":"ContainerStarted","Data":"83885613d5c8fc4d5264f7b11cb8e977c1ae43091d2d10011a8bed37fa5a2d51"} Oct 31 05:15:45.213446 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:45.205774 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b1965c-0483-419e-9e73-aacf0f6a5048-credential-socket" pod "68b1965c-0483-419e-9e73-aacf0f6a5048" (UID: "68b1965c-0483-419e-9e73-aacf0f6a5048"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:15:45.216064 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:45.216038 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f887033-pwvhc" event={"ID":"edc66a25-6634-4ccf-bc03-a29e87a8ac4b","Type":"ContainerDied","Data":"21b2790c2dd162af67c45a733a0f4c17755e7b9ddc7fef3f7e59ad747f4a4a53"} Oct 31 05:15:45.216185 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:45.216109 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21b2790c2dd162af67c45a733a0f4c17755e7b9ddc7fef3f7e59ad747f4a4a53" Oct 31 05:15:45.216185 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:45.216169 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f887033-pwvhc" Oct 31 05:15:45.222088 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:45.221782 5019 generic.go:356] "Generic (PLEG): container finished" podID="68b1965c-0483-419e-9e73-aacf0f6a5048" containerID="c7aee1c0f539e53fdd244f018e7f9224350d313b492072495922872fb4aa6e97" exitCode=0 Oct 31 05:15:45.222088 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:45.221868 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6" event={"ID":"68b1965c-0483-419e-9e73-aacf0f6a5048","Type":"ContainerDied","Data":"c7aee1c0f539e53fdd244f018e7f9224350d313b492072495922872fb4aa6e97"} Oct 31 05:15:45.222088 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:45.221873 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6" Oct 31 05:15:45.222362 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:45.221914 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6" event={"ID":"68b1965c-0483-419e-9e73-aacf0f6a5048","Type":"ContainerDied","Data":"ba1d39d29fed0d268ba2c7078ecc65bf5fe0c519fd2bf1810f58d0c6e890d1ee"} Oct 31 05:15:45.222362 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:45.221928 5019 scope.go:122] "RemoveContainer" containerID="c7aee1c0f539e53fdd244f018e7f9224350d313b492072495922872fb4aa6e97" Oct 31 05:15:45.242960 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:45.242929 5019 scope.go:122] "RemoveContainer" containerID="2928eb4559d6aa4257028602a5358f1f1b47cccf6cbb91de2759b7b36e0b94c1" Oct 31 05:15:45.255067 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:45.255041 5019 scope.go:122] "RemoveContainer" containerID="c7aee1c0f539e53fdd244f018e7f9224350d313b492072495922872fb4aa6e97" Oct 31 05:15:45.255562 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:15:45.255452 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7aee1c0f539e53fdd244f018e7f9224350d313b492072495922872fb4aa6e97\": container with ID starting with c7aee1c0f539e53fdd244f018e7f9224350d313b492072495922872fb4aa6e97 not found: ID does not exist" containerID="c7aee1c0f539e53fdd244f018e7f9224350d313b492072495922872fb4aa6e97" Oct 31 05:15:45.255714 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:45.255571 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7aee1c0f539e53fdd244f018e7f9224350d313b492072495922872fb4aa6e97"} err="failed to get container status \"c7aee1c0f539e53fdd244f018e7f9224350d313b492072495922872fb4aa6e97\": rpc error: code = NotFound desc = could not find container \"c7aee1c0f539e53fdd244f018e7f9224350d313b492072495922872fb4aa6e97\": container with ID starting with c7aee1c0f539e53fdd244f018e7f9224350d313b492072495922872fb4aa6e97 not found: ID does not exist" Oct 31 05:15:45.255714 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:45.255591 5019 scope.go:122] "RemoveContainer" containerID="2928eb4559d6aa4257028602a5358f1f1b47cccf6cbb91de2759b7b36e0b94c1" Oct 31 05:15:45.255910 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:15:45.255878 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2928eb4559d6aa4257028602a5358f1f1b47cccf6cbb91de2759b7b36e0b94c1\": container with ID starting with 2928eb4559d6aa4257028602a5358f1f1b47cccf6cbb91de2759b7b36e0b94c1 not found: ID does not exist" containerID="2928eb4559d6aa4257028602a5358f1f1b47cccf6cbb91de2759b7b36e0b94c1" Oct 31 05:15:45.256050 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:45.255917 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2928eb4559d6aa4257028602a5358f1f1b47cccf6cbb91de2759b7b36e0b94c1"} err="failed to get container status \"2928eb4559d6aa4257028602a5358f1f1b47cccf6cbb91de2759b7b36e0b94c1\": rpc error: code = NotFound desc = could not find container \"2928eb4559d6aa4257028602a5358f1f1b47cccf6cbb91de2759b7b36e0b94c1\": container with ID starting with 2928eb4559d6aa4257028602a5358f1f1b47cccf6cbb91de2759b7b36e0b94c1 not found: ID does not exist" Oct 31 05:15:45.257760 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:45.257710 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a70042a9-px695" podStartSLOduration=2.257698193 podStartE2EDuration="2.257698193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:15:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:15:45.254475571 +0000 UTC m=+4300.010813610" watchObservedRunningTime="2026-10-31 05:15:45.257698193 +0000 UTC m=+4300.014036276" Oct 31 05:15:45.272148 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:45.272106 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f"] Oct 31 05:15:45.280415 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:45.280374 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b"] Oct 31 05:15:45.295439 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:45.295317 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6"] Oct 31 05:15:45.301167 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:45.301010 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68b1965c-0483-419e-9e73-aacf0f6a5048-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:45.301167 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:45.301042 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68b1965c-0483-419e-9e73-aacf0f6a5048-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:45.301167 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:45.301051 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68b1965c-0483-419e-9e73-aacf0f6a5048-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:45.301167 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:45.301059 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b1965c-0483-419e-9e73-aacf0f6a5048-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:45.301167 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:45.301068 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68b1965c-0483-419e-9e73-aacf0f6a5048-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:45.301167 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:45.301079 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b1965c-0483-419e-9e73-aacf0f6a5048-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:45.301167 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:45.301086 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68b1965c-0483-419e-9e73-aacf0f6a5048-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:45.301167 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:45.301095 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68b1965c-0483-419e-9e73-aacf0f6a5048-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:45.301167 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:45.301103 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwjsw\" (UniqueName: \"kubernetes.io/projected/68b1965c-0483-419e-9e73-aacf0f6a5048-kube-api-access-dwjsw\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:45.306561 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:45.306528 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9633f0-openshift-default-84b674487f-dkkf6"] Oct 31 05:15:45.412901 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:45.412860 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68b1965c-0483-419e-9e73-aacf0f6a5048" path="/var/lib/kubelet/pods/68b1965c-0483-419e-9e73-aacf0f6a5048/volumes" Oct 31 05:15:45.631888 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:45.622151 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b2a372/rate-test-model-b2a372-kserve-8454cdd6df-bkqc8"] Oct 31 05:15:46.234248 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:46.234204 5019 generic.go:356] "Generic (PLEG): container finished" podID="39c4d335-5acc-4668-b4c3-57f6ff58a75b" containerID="119c94048c9fb31654a855ad873133797c758d31f12e9e49a58aaa7541aa395e" exitCode=0 Oct 31 05:15:46.234495 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:46.234292 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a70042a9-px695" event={"ID":"39c4d335-5acc-4668-b4c3-57f6ff58a75b","Type":"ContainerDied","Data":"119c94048c9fb31654a855ad873133797c758d31f12e9e49a58aaa7541aa395e"} Oct 31 05:15:47.432304 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:47.432271 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a70042a9-px695" Oct 31 05:15:47.529492 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:47.529376 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39c4d335-5acc-4668-b4c3-57f6ff58a75b-kube-api-access-bdzpr\" (UniqueName: \"kubernetes.io/projected/39c4d335-5acc-4668-b4c3-57f6ff58a75b-kube-api-access-bdzpr\") pod \"39c4d335-5acc-4668-b4c3-57f6ff58a75b\" (UID: \"39c4d335-5acc-4668-b4c3-57f6ff58a75b\") " Oct 31 05:15:47.529492 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:47.529447 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39c4d335-5acc-4668-b4c3-57f6ff58a75b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39c4d335-5acc-4668-b4c3-57f6ff58a75b-maas-api-xks-ca\") pod \"39c4d335-5acc-4668-b4c3-57f6ff58a75b\" (UID: \"39c4d335-5acc-4668-b4c3-57f6ff58a75b\") " Oct 31 05:15:47.529770 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:47.529625 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39c4d335-5acc-4668-b4c3-57f6ff58a75b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39c4d335-5acc-4668-b4c3-57f6ff58a75b-maas-api-openshift-ca\") pod \"39c4d335-5acc-4668-b4c3-57f6ff58a75b\" (UID: \"39c4d335-5acc-4668-b4c3-57f6ff58a75b\") " Oct 31 05:15:47.530120 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:47.530078 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39c4d335-5acc-4668-b4c3-57f6ff58a75b-maas-api-openshift-ca" pod "39c4d335-5acc-4668-b4c3-57f6ff58a75b" (UID: "39c4d335-5acc-4668-b4c3-57f6ff58a75b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:15:47.530349 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:47.530303 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39c4d335-5acc-4668-b4c3-57f6ff58a75b-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:47.531934 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:47.531903 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39c4d335-5acc-4668-b4c3-57f6ff58a75b-maas-api-xks-ca" pod "39c4d335-5acc-4668-b4c3-57f6ff58a75b" (UID: "39c4d335-5acc-4668-b4c3-57f6ff58a75b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:15:47.532041 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:47.531985 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39c4d335-5acc-4668-b4c3-57f6ff58a75b-kube-api-access-bdzpr" pod "39c4d335-5acc-4668-b4c3-57f6ff58a75b" (UID: "39c4d335-5acc-4668-b4c3-57f6ff58a75b"). InnerVolumeSpecName "kube-api-access-bdzpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:15:47.631863 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:47.631798 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdzpr\" (UniqueName: \"kubernetes.io/projected/39c4d335-5acc-4668-b4c3-57f6ff58a75b-kube-api-access-bdzpr\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:47.631863 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:47.631860 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39c4d335-5acc-4668-b4c3-57f6ff58a75b-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:48.253698 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:48.253669 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a70042a9-px695" Oct 31 05:15:48.253906 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:48.253667 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a70042a9-px695" event={"ID":"39c4d335-5acc-4668-b4c3-57f6ff58a75b","Type":"ContainerDied","Data":"83885613d5c8fc4d5264f7b11cb8e977c1ae43091d2d10011a8bed37fa5a2d51"} Oct 31 05:15:48.253906 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:48.253804 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83885613d5c8fc4d5264f7b11cb8e977c1ae43091d2d10011a8bed37fa5a2d51" Oct 31 05:15:52.396482 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:15:52.396440 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:15:53.091919 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:53.091867 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c4006d-7bd575dbdf-h26mj"] Oct 31 05:15:53.122775 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:53.122734 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c4006d-548d7c6f4c-nj2nr"] Oct 31 05:15:53.274395 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:53.274346 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbf88caf-87lbv"] Oct 31 05:15:53.275002 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:53.274984 5019 cpu_manager.go:414] "containerMap: removing container" podUID="39c4d335-5acc-4668-b4c3-57f6ff58a75b" containerName="revoke-keys" Oct 31 05:15:53.275002 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:53.275001 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39c4d335-5acc-4668-b4c3-57f6ff58a75b" containerName="revoke-keys" Oct 31 05:15:53.275175 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:53.275021 5019 cpu_manager.go:414] "containerMap: removing container" podUID="edc66a25-6634-4ccf-bc03-a29e87a8ac4b" containerName="revoke-keys" Oct 31 05:15:53.275175 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:53.275026 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edc66a25-6634-4ccf-bc03-a29e87a8ac4b" containerName="revoke-keys" Oct 31 05:15:53.275175 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:53.275063 5019 cpu_manager.go:414] "containerMap: removing container" podUID="68b1965c-0483-419e-9e73-aacf0f6a5048" containerName="istio-proxy" Oct 31 05:15:53.275175 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:53.275069 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68b1965c-0483-419e-9e73-aacf0f6a5048" containerName="istio-proxy" Oct 31 05:15:53.275175 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:53.275083 5019 cpu_manager.go:414] "containerMap: removing container" podUID="68b1965c-0483-419e-9e73-aacf0f6a5048" containerName="istio-proxy" Oct 31 05:15:53.275175 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:53.275088 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68b1965c-0483-419e-9e73-aacf0f6a5048" containerName="istio-proxy" Oct 31 05:15:53.275175 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:53.275170 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="39c4d335-5acc-4668-b4c3-57f6ff58a75b" containerName="revoke-keys" Oct 31 05:15:53.275585 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:53.275181 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="edc66a25-6634-4ccf-bc03-a29e87a8ac4b" containerName="revoke-keys" Oct 31 05:15:53.275585 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:53.275197 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="68b1965c-0483-419e-9e73-aacf0f6a5048" containerName="istio-proxy" Oct 31 05:15:53.275585 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:53.275205 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="68b1965c-0483-419e-9e73-aacf0f6a5048" containerName="istio-proxy" Oct 31 05:15:53.279280 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:53.279247 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf88caf-87lbv" Oct 31 05:15:53.299467 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:53.299425 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbf88caf-87lbv"] Oct 31 05:15:53.301814 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:53.301783 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:15:53.302060 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:53.301819 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:15:53.302060 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:53.301971 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 05:15:53.302218 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:53.302124 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:15:53.406776 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:53.406676 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25bvt\" (UniqueName: \"kubernetes.io/projected/a2c340cd-5e95-444e-af15-f94090647205-kube-api-access-25bvt\") pod \"maas-api-revoke-sub-bbf88caf-87lbv\" (UID: \"a2c340cd-5e95-444e-af15-f94090647205\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf88caf-87lbv" Oct 31 05:15:53.407283 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:53.406826 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2c340cd-5e95-444e-af15-f94090647205-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbf88caf-87lbv\" (UID: \"a2c340cd-5e95-444e-af15-f94090647205\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf88caf-87lbv" Oct 31 05:15:53.407283 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:53.406873 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2c340cd-5e95-444e-af15-f94090647205-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbf88caf-87lbv\" (UID: \"a2c340cd-5e95-444e-af15-f94090647205\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf88caf-87lbv" Oct 31 05:15:53.508399 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:53.508360 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25bvt\" (UniqueName: \"kubernetes.io/projected/a2c340cd-5e95-444e-af15-f94090647205-kube-api-access-25bvt\") pod \"maas-api-revoke-sub-bbf88caf-87lbv\" (UID: \"a2c340cd-5e95-444e-af15-f94090647205\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf88caf-87lbv" Oct 31 05:15:53.508634 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:53.508445 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2c340cd-5e95-444e-af15-f94090647205-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbf88caf-87lbv\" (UID: \"a2c340cd-5e95-444e-af15-f94090647205\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf88caf-87lbv" Oct 31 05:15:53.508634 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:53.508482 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2c340cd-5e95-444e-af15-f94090647205-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbf88caf-87lbv\" (UID: \"a2c340cd-5e95-444e-af15-f94090647205\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf88caf-87lbv" Oct 31 05:15:53.509270 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:53.509209 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2c340cd-5e95-444e-af15-f94090647205-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbf88caf-87lbv\" (UID: \"a2c340cd-5e95-444e-af15-f94090647205\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf88caf-87lbv" Oct 31 05:15:53.511441 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:53.511402 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2c340cd-5e95-444e-af15-f94090647205-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbf88caf-87lbv\" (UID: \"a2c340cd-5e95-444e-af15-f94090647205\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf88caf-87lbv" Oct 31 05:15:53.531612 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:53.531574 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25bvt\" (UniqueName: \"kubernetes.io/projected/a2c340cd-5e95-444e-af15-f94090647205-kube-api-access-25bvt\") pod \"maas-api-revoke-sub-bbf88caf-87lbv\" (UID: \"a2c340cd-5e95-444e-af15-f94090647205\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf88caf-87lbv" Oct 31 05:15:53.602212 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:53.602171 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf88caf-87lbv" Oct 31 05:15:53.912312 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:15:53.912277 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2c340cd_5e95_444e_af15_f94090647205.slice/crio-aa4e65eaef20fb6c511eab528dade29c1f7efe54c83ad905978c96693f630ba0 WatchSource:0}: Error finding container aa4e65eaef20fb6c511eab528dade29c1f7efe54c83ad905978c96693f630ba0: Status 404 returned error can't find the container with id aa4e65eaef20fb6c511eab528dade29c1f7efe54c83ad905978c96693f630ba0 Oct 31 05:15:53.912515 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:53.912403 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbf88caf-87lbv"] Oct 31 05:15:54.243367 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:54.243283 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38792383-2l5ht"] Oct 31 05:15:54.247756 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:54.247727 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38792383-2l5ht" Oct 31 05:15:54.269059 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:54.269021 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38792383-2l5ht"] Oct 31 05:15:54.317258 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:54.317210 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmvlt\" (UniqueName: \"kubernetes.io/projected/42db3486-5bc2-40fc-b5cd-b32f718255b3-kube-api-access-fmvlt\") pod \"maas-api-revoke-sub-38792383-2l5ht\" (UID: \"42db3486-5bc2-40fc-b5cd-b32f718255b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38792383-2l5ht" Oct 31 05:15:54.317497 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:54.317419 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42db3486-5bc2-40fc-b5cd-b32f718255b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38792383-2l5ht\" (UID: \"42db3486-5bc2-40fc-b5cd-b32f718255b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38792383-2l5ht" Oct 31 05:15:54.317595 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:54.317574 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42db3486-5bc2-40fc-b5cd-b32f718255b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38792383-2l5ht\" (UID: \"42db3486-5bc2-40fc-b5cd-b32f718255b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38792383-2l5ht" Oct 31 05:15:54.318720 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:54.318686 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf88caf-87lbv" event={"ID":"a2c340cd-5e95-444e-af15-f94090647205","Type":"ContainerStarted","Data":"1a284afd572e939558dd715390136c3794c6dcbb2dae7985d1506a1d80cc3f7f"} Oct 31 05:15:54.318872 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:54.318746 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf88caf-87lbv" event={"ID":"a2c340cd-5e95-444e-af15-f94090647205","Type":"ContainerStarted","Data":"aa4e65eaef20fb6c511eab528dade29c1f7efe54c83ad905978c96693f630ba0"} Oct 31 05:15:54.363821 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:54.363715 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf88caf-87lbv" podStartSLOduration=1.363693959 podStartE2EDuration="1.363693959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:15:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:15:54.358294482 +0000 UTC m=+4309.114632523" watchObservedRunningTime="2026-10-31 05:15:54.363693959 +0000 UTC m=+4309.120032005" Oct 31 05:15:54.418687 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:54.418642 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmvlt\" (UniqueName: \"kubernetes.io/projected/42db3486-5bc2-40fc-b5cd-b32f718255b3-kube-api-access-fmvlt\") pod \"maas-api-revoke-sub-38792383-2l5ht\" (UID: \"42db3486-5bc2-40fc-b5cd-b32f718255b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38792383-2l5ht" Oct 31 05:15:54.419194 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:54.418809 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42db3486-5bc2-40fc-b5cd-b32f718255b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38792383-2l5ht\" (UID: \"42db3486-5bc2-40fc-b5cd-b32f718255b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38792383-2l5ht" Oct 31 05:15:54.419194 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:54.418936 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42db3486-5bc2-40fc-b5cd-b32f718255b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38792383-2l5ht\" (UID: \"42db3486-5bc2-40fc-b5cd-b32f718255b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38792383-2l5ht" Oct 31 05:15:54.419679 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:54.419651 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42db3486-5bc2-40fc-b5cd-b32f718255b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38792383-2l5ht\" (UID: \"42db3486-5bc2-40fc-b5cd-b32f718255b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38792383-2l5ht" Oct 31 05:15:54.421523 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:54.421495 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42db3486-5bc2-40fc-b5cd-b32f718255b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38792383-2l5ht\" (UID: \"42db3486-5bc2-40fc-b5cd-b32f718255b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38792383-2l5ht" Oct 31 05:15:54.442121 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:54.442042 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmvlt\" (UniqueName: \"kubernetes.io/projected/42db3486-5bc2-40fc-b5cd-b32f718255b3-kube-api-access-fmvlt\") pod \"maas-api-revoke-sub-38792383-2l5ht\" (UID: \"42db3486-5bc2-40fc-b5cd-b32f718255b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38792383-2l5ht" Oct 31 05:15:54.564263 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:54.564220 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38792383-2l5ht" Oct 31 05:15:54.874279 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:54.874243 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38792383-2l5ht"] Oct 31 05:15:54.875157 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:15:54.875127 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42db3486_5bc2_40fc_b5cd_b32f718255b3.slice/crio-62fa7a91c29c4a1e054a0dc909db675c81c004cf4821e5d23df6fb3f3e6e6930 WatchSource:0}: Error finding container 62fa7a91c29c4a1e054a0dc909db675c81c004cf4821e5d23df6fb3f3e6e6930: Status 404 returned error can't find the container with id 62fa7a91c29c4a1e054a0dc909db675c81c004cf4821e5d23df6fb3f3e6e6930 Oct 31 05:15:55.328296 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:55.328260 5019 generic.go:356] "Generic (PLEG): container finished" podID="a2c340cd-5e95-444e-af15-f94090647205" containerID="1a284afd572e939558dd715390136c3794c6dcbb2dae7985d1506a1d80cc3f7f" exitCode=0 Oct 31 05:15:55.328521 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:55.328360 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf88caf-87lbv" event={"ID":"a2c340cd-5e95-444e-af15-f94090647205","Type":"ContainerDied","Data":"1a284afd572e939558dd715390136c3794c6dcbb2dae7985d1506a1d80cc3f7f"} Oct 31 05:15:55.330327 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:55.330294 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38792383-2l5ht" event={"ID":"42db3486-5bc2-40fc-b5cd-b32f718255b3","Type":"ContainerStarted","Data":"a4a607ab8661d7a2942850dd9b7b3cdd3d42ef1d1cf311f9adafec3e776e72b9"} Oct 31 05:15:55.330503 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:55.330394 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38792383-2l5ht" event={"ID":"42db3486-5bc2-40fc-b5cd-b32f718255b3","Type":"ContainerStarted","Data":"62fa7a91c29c4a1e054a0dc909db675c81c004cf4821e5d23df6fb3f3e6e6930"} Oct 31 05:15:56.343119 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:56.343073 5019 generic.go:356] "Generic (PLEG): container finished" podID="42db3486-5bc2-40fc-b5cd-b32f718255b3" containerID="a4a607ab8661d7a2942850dd9b7b3cdd3d42ef1d1cf311f9adafec3e776e72b9" exitCode=0 Oct 31 05:15:56.343748 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:56.343169 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38792383-2l5ht" event={"ID":"42db3486-5bc2-40fc-b5cd-b32f718255b3","Type":"ContainerDied","Data":"a4a607ab8661d7a2942850dd9b7b3cdd3d42ef1d1cf311f9adafec3e776e72b9"} Oct 31 05:15:56.528599 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:56.528565 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf88caf-87lbv" Oct 31 05:15:56.648755 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:56.648648 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2c340cd-5e95-444e-af15-f94090647205-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2c340cd-5e95-444e-af15-f94090647205-maas-api-openshift-ca\") pod \"a2c340cd-5e95-444e-af15-f94090647205\" (UID: \"a2c340cd-5e95-444e-af15-f94090647205\") " Oct 31 05:15:56.648755 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:56.648729 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2c340cd-5e95-444e-af15-f94090647205-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2c340cd-5e95-444e-af15-f94090647205-maas-api-xks-ca\") pod \"a2c340cd-5e95-444e-af15-f94090647205\" (UID: \"a2c340cd-5e95-444e-af15-f94090647205\") " Oct 31 05:15:56.649016 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:56.648789 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2c340cd-5e95-444e-af15-f94090647205-kube-api-access-25bvt\" (UniqueName: \"kubernetes.io/projected/a2c340cd-5e95-444e-af15-f94090647205-kube-api-access-25bvt\") pod \"a2c340cd-5e95-444e-af15-f94090647205\" (UID: \"a2c340cd-5e95-444e-af15-f94090647205\") " Oct 31 05:15:56.649167 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:56.649134 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2c340cd-5e95-444e-af15-f94090647205-maas-api-openshift-ca" pod "a2c340cd-5e95-444e-af15-f94090647205" (UID: "a2c340cd-5e95-444e-af15-f94090647205"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:15:56.649373 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:56.649350 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2c340cd-5e95-444e-af15-f94090647205-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:56.651058 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:56.651035 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2c340cd-5e95-444e-af15-f94090647205-maas-api-xks-ca" pod "a2c340cd-5e95-444e-af15-f94090647205" (UID: "a2c340cd-5e95-444e-af15-f94090647205"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:15:56.651130 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:56.651065 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2c340cd-5e95-444e-af15-f94090647205-kube-api-access-25bvt" pod "a2c340cd-5e95-444e-af15-f94090647205" (UID: "a2c340cd-5e95-444e-af15-f94090647205"). InnerVolumeSpecName "kube-api-access-25bvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:15:56.750706 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:56.750657 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2c340cd-5e95-444e-af15-f94090647205-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:56.750706 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:56.750697 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25bvt\" (UniqueName: \"kubernetes.io/projected/a2c340cd-5e95-444e-af15-f94090647205-kube-api-access-25bvt\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:57.356458 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:57.356421 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf88caf-87lbv" Oct 31 05:15:57.356458 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:57.356445 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf88caf-87lbv" event={"ID":"a2c340cd-5e95-444e-af15-f94090647205","Type":"ContainerDied","Data":"aa4e65eaef20fb6c511eab528dade29c1f7efe54c83ad905978c96693f630ba0"} Oct 31 05:15:57.357079 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:57.356504 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa4e65eaef20fb6c511eab528dade29c1f7efe54c83ad905978c96693f630ba0" Oct 31 05:15:57.541989 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:57.541959 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38792383-2l5ht" Oct 31 05:15:57.660423 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:57.660302 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42db3486-5bc2-40fc-b5cd-b32f718255b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42db3486-5bc2-40fc-b5cd-b32f718255b3-maas-api-openshift-ca\") pod \"42db3486-5bc2-40fc-b5cd-b32f718255b3\" (UID: \"42db3486-5bc2-40fc-b5cd-b32f718255b3\") " Oct 31 05:15:57.660423 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:57.660396 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42db3486-5bc2-40fc-b5cd-b32f718255b3-kube-api-access-fmvlt\" (UniqueName: \"kubernetes.io/projected/42db3486-5bc2-40fc-b5cd-b32f718255b3-kube-api-access-fmvlt\") pod \"42db3486-5bc2-40fc-b5cd-b32f718255b3\" (UID: \"42db3486-5bc2-40fc-b5cd-b32f718255b3\") " Oct 31 05:15:57.660679 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:57.660646 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42db3486-5bc2-40fc-b5cd-b32f718255b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42db3486-5bc2-40fc-b5cd-b32f718255b3-maas-api-xks-ca\") pod \"42db3486-5bc2-40fc-b5cd-b32f718255b3\" (UID: \"42db3486-5bc2-40fc-b5cd-b32f718255b3\") " Oct 31 05:15:57.660816 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:57.660778 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42db3486-5bc2-40fc-b5cd-b32f718255b3-maas-api-openshift-ca" pod "42db3486-5bc2-40fc-b5cd-b32f718255b3" (UID: "42db3486-5bc2-40fc-b5cd-b32f718255b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:15:57.661158 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:57.661129 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42db3486-5bc2-40fc-b5cd-b32f718255b3-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:57.662630 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:57.662599 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42db3486-5bc2-40fc-b5cd-b32f718255b3-kube-api-access-fmvlt" pod "42db3486-5bc2-40fc-b5cd-b32f718255b3" (UID: "42db3486-5bc2-40fc-b5cd-b32f718255b3"). InnerVolumeSpecName "kube-api-access-fmvlt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:15:57.662730 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:57.662709 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42db3486-5bc2-40fc-b5cd-b32f718255b3-maas-api-xks-ca" pod "42db3486-5bc2-40fc-b5cd-b32f718255b3" (UID: "42db3486-5bc2-40fc-b5cd-b32f718255b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:15:57.762374 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:57.762278 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmvlt\" (UniqueName: \"kubernetes.io/projected/42db3486-5bc2-40fc-b5cd-b32f718255b3-kube-api-access-fmvlt\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:57.762374 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:57.762321 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42db3486-5bc2-40fc-b5cd-b32f718255b3-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:15:58.092629 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:58.092575 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-c4006d-7bd575dbdf-h26mj" podUID="31f4b14c-49f7-45f4-936e-e1be3123640c" containerName="payload-processing" containerID="cri-o://8ec295a135d952b8d849c8118b1fd07cef6a9e32b87542d5b24b2dbc7205a7c8" gracePeriod=25 Oct 31 05:15:58.123277 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:58.123217 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c4006d-548d7c6f4c-nj2nr" podUID="ab2c1b1c-1f3f-43fb-aa9d-248870c31da1" containerName="payload-pre-processing" containerID="cri-o://edddbd5cf556da8f6b319a00ce62442d6539e6ba1184d6a84df98513f0c1c64a" gracePeriod=25 Oct 31 05:15:58.366157 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:58.366068 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38792383-2l5ht" Oct 31 05:15:58.366157 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:58.366093 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38792383-2l5ht" event={"ID":"42db3486-5bc2-40fc-b5cd-b32f718255b3","Type":"ContainerDied","Data":"62fa7a91c29c4a1e054a0dc909db675c81c004cf4821e5d23df6fb3f3e6e6930"} Oct 31 05:15:58.366157 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:15:58.366150 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62fa7a91c29c4a1e054a0dc909db675c81c004cf4821e5d23df6fb3f3e6e6930" Oct 31 05:16:00.295856 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:00.295802 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" podUID="b4ced3ab-41e4-4c54-93eb-95eddaae992d" containerName="main" containerID="cri-o://1572354f776386eebc19afe3eb66490a6ba9b9d9704c82e72a5a37fab4eb377a" gracePeriod=45 Oct 31 05:16:00.303548 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:00.303479 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" podUID="643433d8-418e-4102-8bc3-cd544932614a" containerName="main" containerID="cri-o://065bb5eff6b7e35473610efb70b6d9984ca6d35a9b639ceb8ea3ac7691259af1" gracePeriod=45 Oct 31 05:16:00.451640 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:00.451231 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" podUID="b4ced3ab-41e4-4c54-93eb-95eddaae992d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": dial tcp 10.132.0.96:8000: connect: connection refused" Oct 31 05:16:00.493548 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:00.489038 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" podUID="643433d8-418e-4102-8bc3-cd544932614a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": dial tcp 10.132.0.80:8000: connect: connection refused" Oct 31 05:16:00.627133 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:00.627098 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" Oct 31 05:16:00.661531 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:00.661459 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-b2a372/rate-test-model-b2a372-kserve-8454cdd6df-bkqc8" podUID="0b8d5e17-5809-46a1-b0da-d11c15832f1d" containerName="main" containerID="cri-o://b52e90aa933e021314a542d34462e93149df95cabd2111dc34e8005a08548fd7" gracePeriod=45 Oct 31 05:16:00.664095 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:00.664048 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" Oct 31 05:16:00.718415 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:00.716701 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/643433d8-418e-4102-8bc3-cd544932614a-home\" (UniqueName: \"kubernetes.io/empty-dir/643433d8-418e-4102-8bc3-cd544932614a-home\") pod \"643433d8-418e-4102-8bc3-cd544932614a\" (UID: \"643433d8-418e-4102-8bc3-cd544932614a\") " Oct 31 05:16:00.718415 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:00.716779 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/643433d8-418e-4102-8bc3-cd544932614a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/643433d8-418e-4102-8bc3-cd544932614a-dshm\") pod \"643433d8-418e-4102-8bc3-cd544932614a\" (UID: \"643433d8-418e-4102-8bc3-cd544932614a\") " Oct 31 05:16:00.718415 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:00.716832 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4ced3ab-41e4-4c54-93eb-95eddaae992d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4ced3ab-41e4-4c54-93eb-95eddaae992d-model-cache\") pod \"b4ced3ab-41e4-4c54-93eb-95eddaae992d\" (UID: \"b4ced3ab-41e4-4c54-93eb-95eddaae992d\") " Oct 31 05:16:00.718415 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:00.716869 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/643433d8-418e-4102-8bc3-cd544932614a-tls-certs\" (UniqueName: \"kubernetes.io/secret/643433d8-418e-4102-8bc3-cd544932614a-tls-certs\") pod \"643433d8-418e-4102-8bc3-cd544932614a\" (UID: \"643433d8-418e-4102-8bc3-cd544932614a\") " Oct 31 05:16:00.718415 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:00.716898 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4ced3ab-41e4-4c54-93eb-95eddaae992d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4ced3ab-41e4-4c54-93eb-95eddaae992d-tmp-dir\") pod \"b4ced3ab-41e4-4c54-93eb-95eddaae992d\" (UID: \"b4ced3ab-41e4-4c54-93eb-95eddaae992d\") " Oct 31 05:16:00.718415 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:00.717482 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/643433d8-418e-4102-8bc3-cd544932614a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/643433d8-418e-4102-8bc3-cd544932614a-tmp-dir\") pod \"643433d8-418e-4102-8bc3-cd544932614a\" (UID: \"643433d8-418e-4102-8bc3-cd544932614a\") " Oct 31 05:16:00.718917 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:00.718413 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4ced3ab-41e4-4c54-93eb-95eddaae992d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4ced3ab-41e4-4c54-93eb-95eddaae992d-dshm\") pod \"b4ced3ab-41e4-4c54-93eb-95eddaae992d\" (UID: \"b4ced3ab-41e4-4c54-93eb-95eddaae992d\") " Oct 31 05:16:00.718917 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:00.718495 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/643433d8-418e-4102-8bc3-cd544932614a-kube-api-access-j8z8k\" (UniqueName: \"kubernetes.io/projected/643433d8-418e-4102-8bc3-cd544932614a-kube-api-access-j8z8k\") pod \"643433d8-418e-4102-8bc3-cd544932614a\" (UID: \"643433d8-418e-4102-8bc3-cd544932614a\") " Oct 31 05:16:00.718917 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:00.718565 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4ced3ab-41e4-4c54-93eb-95eddaae992d-kube-api-access-gl24p\" (UniqueName: \"kubernetes.io/projected/b4ced3ab-41e4-4c54-93eb-95eddaae992d-kube-api-access-gl24p\") pod \"b4ced3ab-41e4-4c54-93eb-95eddaae992d\" (UID: \"b4ced3ab-41e4-4c54-93eb-95eddaae992d\") " Oct 31 05:16:00.718917 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:00.718645 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4ced3ab-41e4-4c54-93eb-95eddaae992d-home\" (UniqueName: \"kubernetes.io/empty-dir/b4ced3ab-41e4-4c54-93eb-95eddaae992d-home\") pod \"b4ced3ab-41e4-4c54-93eb-95eddaae992d\" (UID: \"b4ced3ab-41e4-4c54-93eb-95eddaae992d\") " Oct 31 05:16:00.718917 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:00.718698 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/643433d8-418e-4102-8bc3-cd544932614a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/643433d8-418e-4102-8bc3-cd544932614a-model-cache\") pod \"643433d8-418e-4102-8bc3-cd544932614a\" (UID: \"643433d8-418e-4102-8bc3-cd544932614a\") " Oct 31 05:16:00.718917 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:00.718748 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4ced3ab-41e4-4c54-93eb-95eddaae992d-tls-certs\" (UniqueName: \"kubernetes.io/secret/b4ced3ab-41e4-4c54-93eb-95eddaae992d-tls-certs\") pod \"b4ced3ab-41e4-4c54-93eb-95eddaae992d\" (UID: \"b4ced3ab-41e4-4c54-93eb-95eddaae992d\") " Oct 31 05:16:00.724863 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:00.719853 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4ced3ab-41e4-4c54-93eb-95eddaae992d-model-cache" pod "b4ced3ab-41e4-4c54-93eb-95eddaae992d" (UID: "b4ced3ab-41e4-4c54-93eb-95eddaae992d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:16:00.724863 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:00.720092 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/643433d8-418e-4102-8bc3-cd544932614a-home" pod "643433d8-418e-4102-8bc3-cd544932614a" (UID: "643433d8-418e-4102-8bc3-cd544932614a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:16:00.724863 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:00.720590 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4ced3ab-41e4-4c54-93eb-95eddaae992d-tmp-dir" pod "b4ced3ab-41e4-4c54-93eb-95eddaae992d" (UID: "b4ced3ab-41e4-4c54-93eb-95eddaae992d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:16:00.724863 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:00.720728 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4ced3ab-41e4-4c54-93eb-95eddaae992d-home" pod "b4ced3ab-41e4-4c54-93eb-95eddaae992d" (UID: "b4ced3ab-41e4-4c54-93eb-95eddaae992d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:16:00.724863 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:00.720822 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/643433d8-418e-4102-8bc3-cd544932614a-model-cache" pod "643433d8-418e-4102-8bc3-cd544932614a" (UID: "643433d8-418e-4102-8bc3-cd544932614a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:16:00.724863 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:00.720923 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/643433d8-418e-4102-8bc3-cd544932614a-tmp-dir" pod "643433d8-418e-4102-8bc3-cd544932614a" (UID: "643433d8-418e-4102-8bc3-cd544932614a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:16:00.725539 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:00.725455 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4ced3ab-41e4-4c54-93eb-95eddaae992d-tls-certs" pod "b4ced3ab-41e4-4c54-93eb-95eddaae992d" (UID: "b4ced3ab-41e4-4c54-93eb-95eddaae992d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:16:00.725539 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:00.725468 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4ced3ab-41e4-4c54-93eb-95eddaae992d-dshm" pod "b4ced3ab-41e4-4c54-93eb-95eddaae992d" (UID: "b4ced3ab-41e4-4c54-93eb-95eddaae992d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:16:00.725539 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:00.725498 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/643433d8-418e-4102-8bc3-cd544932614a-kube-api-access-j8z8k" pod "643433d8-418e-4102-8bc3-cd544932614a" (UID: "643433d8-418e-4102-8bc3-cd544932614a"). InnerVolumeSpecName "kube-api-access-j8z8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:16:00.725539 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:00.725468 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/643433d8-418e-4102-8bc3-cd544932614a-dshm" pod "643433d8-418e-4102-8bc3-cd544932614a" (UID: "643433d8-418e-4102-8bc3-cd544932614a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:16:00.726652 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:00.726614 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4ced3ab-41e4-4c54-93eb-95eddaae992d-kube-api-access-gl24p" pod "b4ced3ab-41e4-4c54-93eb-95eddaae992d" (UID: "b4ced3ab-41e4-4c54-93eb-95eddaae992d"). InnerVolumeSpecName "kube-api-access-gl24p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:16:00.726765 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:00.726714 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/643433d8-418e-4102-8bc3-cd544932614a-tls-certs" pod "643433d8-418e-4102-8bc3-cd544932614a" (UID: "643433d8-418e-4102-8bc3-cd544932614a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:16:00.823378 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:00.822170 5019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/643433d8-418e-4102-8bc3-cd544932614a-home\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:16:00.823378 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:00.822216 5019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/643433d8-418e-4102-8bc3-cd544932614a-dshm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:16:00.823378 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:00.822229 5019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4ced3ab-41e4-4c54-93eb-95eddaae992d-model-cache\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:16:00.823378 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:00.822245 5019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/643433d8-418e-4102-8bc3-cd544932614a-tls-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:16:00.823378 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:00.822256 5019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4ced3ab-41e4-4c54-93eb-95eddaae992d-tmp-dir\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:16:00.823378 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:00.822267 5019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/643433d8-418e-4102-8bc3-cd544932614a-tmp-dir\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:16:00.823378 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:00.822277 5019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4ced3ab-41e4-4c54-93eb-95eddaae992d-dshm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:16:00.823378 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:00.822290 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8z8k\" (UniqueName: \"kubernetes.io/projected/643433d8-418e-4102-8bc3-cd544932614a-kube-api-access-j8z8k\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:16:00.823378 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:00.822307 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gl24p\" (UniqueName: \"kubernetes.io/projected/b4ced3ab-41e4-4c54-93eb-95eddaae992d-kube-api-access-gl24p\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:16:00.823378 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:00.822320 5019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4ced3ab-41e4-4c54-93eb-95eddaae992d-home\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:16:00.823378 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:00.822352 5019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/643433d8-418e-4102-8bc3-cd544932614a-model-cache\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:16:00.823378 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:00.822436 5019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4ced3ab-41e4-4c54-93eb-95eddaae992d-tls-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:16:00.931777 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:00.931745 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b2a372/rate-test-model-b2a372-kserve-8454cdd6df-bkqc8" Oct 31 05:16:01.024105 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:01.024061 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b8d5e17-5809-46a1-b0da-d11c15832f1d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b8d5e17-5809-46a1-b0da-d11c15832f1d-tmp-dir\") pod \"0b8d5e17-5809-46a1-b0da-d11c15832f1d\" (UID: \"0b8d5e17-5809-46a1-b0da-d11c15832f1d\") " Oct 31 05:16:01.024301 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:01.024145 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b8d5e17-5809-46a1-b0da-d11c15832f1d-home\" (UniqueName: \"kubernetes.io/empty-dir/0b8d5e17-5809-46a1-b0da-d11c15832f1d-home\") pod \"0b8d5e17-5809-46a1-b0da-d11c15832f1d\" (UID: \"0b8d5e17-5809-46a1-b0da-d11c15832f1d\") " Oct 31 05:16:01.024301 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:01.024174 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b8d5e17-5809-46a1-b0da-d11c15832f1d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b8d5e17-5809-46a1-b0da-d11c15832f1d-dshm\") pod \"0b8d5e17-5809-46a1-b0da-d11c15832f1d\" (UID: \"0b8d5e17-5809-46a1-b0da-d11c15832f1d\") " Oct 31 05:16:01.024301 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:01.024205 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b8d5e17-5809-46a1-b0da-d11c15832f1d-tls-certs\" (UniqueName: \"kubernetes.io/secret/0b8d5e17-5809-46a1-b0da-d11c15832f1d-tls-certs\") pod \"0b8d5e17-5809-46a1-b0da-d11c15832f1d\" (UID: \"0b8d5e17-5809-46a1-b0da-d11c15832f1d\") " Oct 31 05:16:01.024301 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:01.024274 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b8d5e17-5809-46a1-b0da-d11c15832f1d-kube-api-access-66fj2\" (UniqueName: \"kubernetes.io/projected/0b8d5e17-5809-46a1-b0da-d11c15832f1d-kube-api-access-66fj2\") pod \"0b8d5e17-5809-46a1-b0da-d11c15832f1d\" (UID: \"0b8d5e17-5809-46a1-b0da-d11c15832f1d\") " Oct 31 05:16:01.024509 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:01.024371 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b8d5e17-5809-46a1-b0da-d11c15832f1d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b8d5e17-5809-46a1-b0da-d11c15832f1d-model-cache\") pod \"0b8d5e17-5809-46a1-b0da-d11c15832f1d\" (UID: \"0b8d5e17-5809-46a1-b0da-d11c15832f1d\") " Oct 31 05:16:01.024509 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:01.024469 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b8d5e17-5809-46a1-b0da-d11c15832f1d-tmp-dir" pod "0b8d5e17-5809-46a1-b0da-d11c15832f1d" (UID: "0b8d5e17-5809-46a1-b0da-d11c15832f1d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:16:01.024509 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:01.024489 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b8d5e17-5809-46a1-b0da-d11c15832f1d-home" pod "0b8d5e17-5809-46a1-b0da-d11c15832f1d" (UID: "0b8d5e17-5809-46a1-b0da-d11c15832f1d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:16:01.024722 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:01.024701 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b8d5e17-5809-46a1-b0da-d11c15832f1d-model-cache" pod "0b8d5e17-5809-46a1-b0da-d11c15832f1d" (UID: "0b8d5e17-5809-46a1-b0da-d11c15832f1d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:16:01.024836 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:01.024822 5019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b8d5e17-5809-46a1-b0da-d11c15832f1d-model-cache\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:16:01.024898 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:01.024839 5019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b8d5e17-5809-46a1-b0da-d11c15832f1d-tmp-dir\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:16:01.024898 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:01.024847 5019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b8d5e17-5809-46a1-b0da-d11c15832f1d-home\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:16:01.026609 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:01.026584 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b8d5e17-5809-46a1-b0da-d11c15832f1d-dshm" pod "0b8d5e17-5809-46a1-b0da-d11c15832f1d" (UID: "0b8d5e17-5809-46a1-b0da-d11c15832f1d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:16:01.026721 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:01.026632 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b8d5e17-5809-46a1-b0da-d11c15832f1d-tls-certs" pod "0b8d5e17-5809-46a1-b0da-d11c15832f1d" (UID: "0b8d5e17-5809-46a1-b0da-d11c15832f1d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:16:01.026721 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:01.026646 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b8d5e17-5809-46a1-b0da-d11c15832f1d-kube-api-access-66fj2" pod "0b8d5e17-5809-46a1-b0da-d11c15832f1d" (UID: "0b8d5e17-5809-46a1-b0da-d11c15832f1d"). InnerVolumeSpecName "kube-api-access-66fj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:16:01.126398 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:01.126249 5019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b8d5e17-5809-46a1-b0da-d11c15832f1d-dshm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:16:01.126398 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:01.126294 5019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b8d5e17-5809-46a1-b0da-d11c15832f1d-tls-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:16:01.126398 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:01.126309 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66fj2\" (UniqueName: \"kubernetes.io/projected/0b8d5e17-5809-46a1-b0da-d11c15832f1d-kube-api-access-66fj2\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:16:01.399565 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:01.399447 5019 generic.go:356] "Generic (PLEG): container finished" podID="643433d8-418e-4102-8bc3-cd544932614a" containerID="065bb5eff6b7e35473610efb70b6d9984ca6d35a9b639ceb8ea3ac7691259af1" exitCode=0 Oct 31 05:16:01.400087 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:01.399582 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" Oct 31 05:16:01.401550 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:01.401517 5019 generic.go:356] "Generic (PLEG): container finished" podID="0b8d5e17-5809-46a1-b0da-d11c15832f1d" containerID="b52e90aa933e021314a542d34462e93149df95cabd2111dc34e8005a08548fd7" exitCode=0 Oct 31 05:16:01.401732 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:01.401639 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b2a372/rate-test-model-b2a372-kserve-8454cdd6df-bkqc8" Oct 31 05:16:01.403857 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:01.403823 5019 generic.go:356] "Generic (PLEG): container finished" podID="b4ced3ab-41e4-4c54-93eb-95eddaae992d" containerID="1572354f776386eebc19afe3eb66490a6ba9b9d9704c82e72a5a37fab4eb377a" exitCode=0 Oct 31 05:16:01.404016 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:01.403953 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" Oct 31 05:16:01.411938 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:01.411900 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" event={"ID":"643433d8-418e-4102-8bc3-cd544932614a","Type":"ContainerDied","Data":"065bb5eff6b7e35473610efb70b6d9984ca6d35a9b639ceb8ea3ac7691259af1"} Oct 31 05:16:01.412224 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:01.412202 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b" event={"ID":"643433d8-418e-4102-8bc3-cd544932614a","Type":"ContainerDied","Data":"8c82e0c993d14b9f29c658408067ee1379f9b217a5cb62374cda43eddfaef55d"} Oct 31 05:16:01.412426 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:01.412409 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b2a372/rate-test-model-b2a372-kserve-8454cdd6df-bkqc8" event={"ID":"0b8d5e17-5809-46a1-b0da-d11c15832f1d","Type":"ContainerDied","Data":"b52e90aa933e021314a542d34462e93149df95cabd2111dc34e8005a08548fd7"} Oct 31 05:16:01.413244 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:01.413213 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b2a372/rate-test-model-b2a372-kserve-8454cdd6df-bkqc8" event={"ID":"0b8d5e17-5809-46a1-b0da-d11c15832f1d","Type":"ContainerDied","Data":"1e1e0c0f8b912f66af75a756f1eed170c84f69d93ce3a2b1b8a5219e5f641016"} Oct 31 05:16:01.413402 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:01.413386 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" event={"ID":"b4ced3ab-41e4-4c54-93eb-95eddaae992d","Type":"ContainerDied","Data":"1572354f776386eebc19afe3eb66490a6ba9b9d9704c82e72a5a37fab4eb377a"} Oct 31 05:16:01.413554 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:01.413535 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f" event={"ID":"b4ced3ab-41e4-4c54-93eb-95eddaae992d","Type":"ContainerDied","Data":"0dfe9a5548d56a8e23a6875ee933f84c47011085f7527e1060348b4bb72eeffe"} Oct 31 05:16:01.413694 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:01.412368 5019 scope.go:122] "RemoveContainer" containerID="065bb5eff6b7e35473610efb70b6d9984ca6d35a9b639ceb8ea3ac7691259af1" Oct 31 05:16:01.430742 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:01.430719 5019 scope.go:122] "RemoveContainer" containerID="065bb5eff6b7e35473610efb70b6d9984ca6d35a9b639ceb8ea3ac7691259af1" Oct 31 05:16:01.431090 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:16:01.431071 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"065bb5eff6b7e35473610efb70b6d9984ca6d35a9b639ceb8ea3ac7691259af1\": container with ID starting with 065bb5eff6b7e35473610efb70b6d9984ca6d35a9b639ceb8ea3ac7691259af1 not found: ID does not exist" containerID="065bb5eff6b7e35473610efb70b6d9984ca6d35a9b639ceb8ea3ac7691259af1" Oct 31 05:16:01.431173 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:01.431097 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"065bb5eff6b7e35473610efb70b6d9984ca6d35a9b639ceb8ea3ac7691259af1"} err="failed to get container status \"065bb5eff6b7e35473610efb70b6d9984ca6d35a9b639ceb8ea3ac7691259af1\": rpc error: code = NotFound desc = could not find container \"065bb5eff6b7e35473610efb70b6d9984ca6d35a9b639ceb8ea3ac7691259af1\": container with ID starting with 065bb5eff6b7e35473610efb70b6d9984ca6d35a9b639ceb8ea3ac7691259af1 not found: ID does not exist" Oct 31 05:16:01.431173 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:01.431113 5019 scope.go:122] "RemoveContainer" containerID="b52e90aa933e021314a542d34462e93149df95cabd2111dc34e8005a08548fd7" Oct 31 05:16:01.442024 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:01.441995 5019 scope.go:122] "RemoveContainer" containerID="b52e90aa933e021314a542d34462e93149df95cabd2111dc34e8005a08548fd7" Oct 31 05:16:01.442432 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:16:01.442403 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b52e90aa933e021314a542d34462e93149df95cabd2111dc34e8005a08548fd7\": container with ID starting with b52e90aa933e021314a542d34462e93149df95cabd2111dc34e8005a08548fd7 not found: ID does not exist" containerID="b52e90aa933e021314a542d34462e93149df95cabd2111dc34e8005a08548fd7" Oct 31 05:16:01.442529 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:01.442442 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b52e90aa933e021314a542d34462e93149df95cabd2111dc34e8005a08548fd7"} err="failed to get container status \"b52e90aa933e021314a542d34462e93149df95cabd2111dc34e8005a08548fd7\": rpc error: code = NotFound desc = could not find container \"b52e90aa933e021314a542d34462e93149df95cabd2111dc34e8005a08548fd7\": container with ID starting with b52e90aa933e021314a542d34462e93149df95cabd2111dc34e8005a08548fd7 not found: ID does not exist" Oct 31 05:16:01.442529 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:01.442464 5019 scope.go:122] "RemoveContainer" containerID="1572354f776386eebc19afe3eb66490a6ba9b9d9704c82e72a5a37fab4eb377a" Oct 31 05:16:01.454186 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:01.454159 5019 scope.go:122] "RemoveContainer" containerID="1572354f776386eebc19afe3eb66490a6ba9b9d9704c82e72a5a37fab4eb377a" Oct 31 05:16:01.454553 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:16:01.454531 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1572354f776386eebc19afe3eb66490a6ba9b9d9704c82e72a5a37fab4eb377a\": container with ID starting with 1572354f776386eebc19afe3eb66490a6ba9b9d9704c82e72a5a37fab4eb377a not found: ID does not exist" containerID="1572354f776386eebc19afe3eb66490a6ba9b9d9704c82e72a5a37fab4eb377a" Oct 31 05:16:01.454634 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:01.454566 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1572354f776386eebc19afe3eb66490a6ba9b9d9704c82e72a5a37fab4eb377a"} err="failed to get container status \"1572354f776386eebc19afe3eb66490a6ba9b9d9704c82e72a5a37fab4eb377a\": rpc error: code = NotFound desc = could not find container \"1572354f776386eebc19afe3eb66490a6ba9b9d9704c82e72a5a37fab4eb377a\": container with ID starting with 1572354f776386eebc19afe3eb66490a6ba9b9d9704c82e72a5a37fab4eb377a not found: ID does not exist" Oct 31 05:16:01.484480 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:01.484429 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b2a372/rate-test-model-b2a372-kserve-8454cdd6df-bkqc8"] Oct 31 05:16:01.490920 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:01.490871 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b2a372/rate-test-model-b2a372-kserve-8454cdd6df-bkqc8"] Oct 31 05:16:01.521845 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:01.521808 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b"] Oct 31 05:16:01.525640 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:01.525602 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c4006d/premium-w4-c4006d-kserve-59d5c64c77-5dv8b"] Oct 31 05:16:01.556527 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:01.556488 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f"] Oct 31 05:16:01.568790 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:01.568745 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c4006d/distinct-w4-c4006d-kserve-569c4d8784-r6s2f"] Oct 31 05:16:02.813116 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:02.813067 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-c4006d-7bd575dbdf-h26mj" podUID="31f4b14c-49f7-45f4-936e-e1be3123640c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:16:02.813704 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:02.813087 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c4006d-548d7c6f4c-nj2nr" podUID="ab2c1b1c-1f3f-43fb-aa9d-248870c31da1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:16:03.408845 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:03.408804 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b8d5e17-5809-46a1-b0da-d11c15832f1d" path="/var/lib/kubelet/pods/0b8d5e17-5809-46a1-b0da-d11c15832f1d/volumes" Oct 31 05:16:03.409315 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:03.409297 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="643433d8-418e-4102-8bc3-cd544932614a" path="/var/lib/kubelet/pods/643433d8-418e-4102-8bc3-cd544932614a/volumes" Oct 31 05:16:03.409719 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:03.409702 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4ced3ab-41e4-4c54-93eb-95eddaae992d" path="/var/lib/kubelet/pods/b4ced3ab-41e4-4c54-93eb-95eddaae992d/volumes" Oct 31 05:16:12.812782 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:12.812734 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-c4006d-7bd575dbdf-h26mj" podUID="31f4b14c-49f7-45f4-936e-e1be3123640c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:16:12.812782 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:12.812775 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c4006d-548d7c6f4c-nj2nr" podUID="ab2c1b1c-1f3f-43fb-aa9d-248870c31da1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:16:13.848273 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:13.848242 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8973e2-85c9f98f95-rgkrh"] Oct 31 05:16:18.411472 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:18.411433 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-c4006d-7bd575dbdf-h26mj" Oct 31 05:16:18.433059 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:18.433027 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c4006d-548d7c6f4c-nj2nr" Oct 31 05:16:18.538685 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:18.538641 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31f4b14c-49f7-45f4-936e-e1be3123640c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31f4b14c-49f7-45f4-936e-e1be3123640c-plugins-config-volume\") pod \"31f4b14c-49f7-45f4-936e-e1be3123640c\" (UID: \"31f4b14c-49f7-45f4-936e-e1be3123640c\") " Oct 31 05:16:18.538902 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:18.538698 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31f4b14c-49f7-45f4-936e-e1be3123640c-kube-api-access-brlqv\" (UniqueName: \"kubernetes.io/projected/31f4b14c-49f7-45f4-936e-e1be3123640c-kube-api-access-brlqv\") pod \"31f4b14c-49f7-45f4-936e-e1be3123640c\" (UID: \"31f4b14c-49f7-45f4-936e-e1be3123640c\") " Oct 31 05:16:18.538902 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:18.538811 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab2c1b1c-1f3f-43fb-aa9d-248870c31da1-kube-api-access-khjbw\" (UniqueName: \"kubernetes.io/projected/ab2c1b1c-1f3f-43fb-aa9d-248870c31da1-kube-api-access-khjbw\") pod \"ab2c1b1c-1f3f-43fb-aa9d-248870c31da1\" (UID: \"ab2c1b1c-1f3f-43fb-aa9d-248870c31da1\") " Oct 31 05:16:18.539044 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:18.538918 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab2c1b1c-1f3f-43fb-aa9d-248870c31da1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab2c1b1c-1f3f-43fb-aa9d-248870c31da1-plugins-config-volume\") pod \"ab2c1b1c-1f3f-43fb-aa9d-248870c31da1\" (UID: \"ab2c1b1c-1f3f-43fb-aa9d-248870c31da1\") " Oct 31 05:16:18.539282 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:18.539132 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31f4b14c-49f7-45f4-936e-e1be3123640c-plugins-config-volume" pod "31f4b14c-49f7-45f4-936e-e1be3123640c" (UID: "31f4b14c-49f7-45f4-936e-e1be3123640c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:16:18.539465 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:18.539311 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab2c1b1c-1f3f-43fb-aa9d-248870c31da1-plugins-config-volume" pod "ab2c1b1c-1f3f-43fb-aa9d-248870c31da1" (UID: "ab2c1b1c-1f3f-43fb-aa9d-248870c31da1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:16:18.539554 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:18.539519 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab2c1b1c-1f3f-43fb-aa9d-248870c31da1-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:16:18.539633 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:18.539553 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31f4b14c-49f7-45f4-936e-e1be3123640c-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:16:18.541104 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:18.541080 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31f4b14c-49f7-45f4-936e-e1be3123640c-kube-api-access-brlqv" pod "31f4b14c-49f7-45f4-936e-e1be3123640c" (UID: "31f4b14c-49f7-45f4-936e-e1be3123640c"). InnerVolumeSpecName "kube-api-access-brlqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:16:18.541300 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:18.541284 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab2c1b1c-1f3f-43fb-aa9d-248870c31da1-kube-api-access-khjbw" pod "ab2c1b1c-1f3f-43fb-aa9d-248870c31da1" (UID: "ab2c1b1c-1f3f-43fb-aa9d-248870c31da1"). InnerVolumeSpecName "kube-api-access-khjbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:16:18.550987 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:18.550947 5019 generic.go:356] "Generic (PLEG): container finished" podID="ab2c1b1c-1f3f-43fb-aa9d-248870c31da1" containerID="edddbd5cf556da8f6b319a00ce62442d6539e6ba1184d6a84df98513f0c1c64a" exitCode=0 Oct 31 05:16:18.551200 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:18.551000 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c4006d-548d7c6f4c-nj2nr" Oct 31 05:16:18.551200 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:18.551020 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c4006d-548d7c6f4c-nj2nr" event={"ID":"ab2c1b1c-1f3f-43fb-aa9d-248870c31da1","Type":"ContainerDied","Data":"edddbd5cf556da8f6b319a00ce62442d6539e6ba1184d6a84df98513f0c1c64a"} Oct 31 05:16:18.551200 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:18.551082 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c4006d-548d7c6f4c-nj2nr" event={"ID":"ab2c1b1c-1f3f-43fb-aa9d-248870c31da1","Type":"ContainerDied","Data":"0f6d38deab920feb0b4b809ada7c36daa3eac73371a5be1222653aa9c77e8279"} Oct 31 05:16:18.551200 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:18.551104 5019 scope.go:122] "RemoveContainer" containerID="edddbd5cf556da8f6b319a00ce62442d6539e6ba1184d6a84df98513f0c1c64a" Oct 31 05:16:18.553356 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:18.553313 5019 generic.go:356] "Generic (PLEG): container finished" podID="31f4b14c-49f7-45f4-936e-e1be3123640c" containerID="8ec295a135d952b8d849c8118b1fd07cef6a9e32b87542d5b24b2dbc7205a7c8" exitCode=0 Oct 31 05:16:18.553476 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:18.553417 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-c4006d-7bd575dbdf-h26mj" event={"ID":"31f4b14c-49f7-45f4-936e-e1be3123640c","Type":"ContainerDied","Data":"8ec295a135d952b8d849c8118b1fd07cef6a9e32b87542d5b24b2dbc7205a7c8"} Oct 31 05:16:18.553476 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:18.553467 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-c4006d-7bd575dbdf-h26mj" event={"ID":"31f4b14c-49f7-45f4-936e-e1be3123640c","Type":"ContainerDied","Data":"d91fd5b6ab45fb5517286117375299ed3bfe591d411c02db66818168cb8d24ed"} Oct 31 05:16:18.553576 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:18.553545 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-c4006d-7bd575dbdf-h26mj" Oct 31 05:16:18.566775 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:18.564579 5019 scope.go:122] "RemoveContainer" containerID="edddbd5cf556da8f6b319a00ce62442d6539e6ba1184d6a84df98513f0c1c64a" Oct 31 05:16:18.566775 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:16:18.565088 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edddbd5cf556da8f6b319a00ce62442d6539e6ba1184d6a84df98513f0c1c64a\": container with ID starting with edddbd5cf556da8f6b319a00ce62442d6539e6ba1184d6a84df98513f0c1c64a not found: ID does not exist" containerID="edddbd5cf556da8f6b319a00ce62442d6539e6ba1184d6a84df98513f0c1c64a" Oct 31 05:16:18.566775 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:18.565127 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edddbd5cf556da8f6b319a00ce62442d6539e6ba1184d6a84df98513f0c1c64a"} err="failed to get container status \"edddbd5cf556da8f6b319a00ce62442d6539e6ba1184d6a84df98513f0c1c64a\": rpc error: code = NotFound desc = could not find container \"edddbd5cf556da8f6b319a00ce62442d6539e6ba1184d6a84df98513f0c1c64a\": container with ID starting with edddbd5cf556da8f6b319a00ce62442d6539e6ba1184d6a84df98513f0c1c64a not found: ID does not exist" Oct 31 05:16:18.566775 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:18.565155 5019 scope.go:122] "RemoveContainer" containerID="8ec295a135d952b8d849c8118b1fd07cef6a9e32b87542d5b24b2dbc7205a7c8" Oct 31 05:16:18.590151 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:18.590075 5019 scope.go:122] "RemoveContainer" containerID="8ec295a135d952b8d849c8118b1fd07cef6a9e32b87542d5b24b2dbc7205a7c8" Oct 31 05:16:18.590790 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:16:18.590749 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ec295a135d952b8d849c8118b1fd07cef6a9e32b87542d5b24b2dbc7205a7c8\": container with ID starting with 8ec295a135d952b8d849c8118b1fd07cef6a9e32b87542d5b24b2dbc7205a7c8 not found: ID does not exist" containerID="8ec295a135d952b8d849c8118b1fd07cef6a9e32b87542d5b24b2dbc7205a7c8" Oct 31 05:16:18.590995 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:18.590802 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ec295a135d952b8d849c8118b1fd07cef6a9e32b87542d5b24b2dbc7205a7c8"} err="failed to get container status \"8ec295a135d952b8d849c8118b1fd07cef6a9e32b87542d5b24b2dbc7205a7c8\": rpc error: code = NotFound desc = could not find container \"8ec295a135d952b8d849c8118b1fd07cef6a9e32b87542d5b24b2dbc7205a7c8\": container with ID starting with 8ec295a135d952b8d849c8118b1fd07cef6a9e32b87542d5b24b2dbc7205a7c8 not found: ID does not exist" Oct 31 05:16:18.633468 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:18.633309 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c4006d-548d7c6f4c-nj2nr"] Oct 31 05:16:18.637319 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:18.637284 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c4006d-548d7c6f4c-nj2nr"] Oct 31 05:16:18.641185 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:18.641153 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brlqv\" (UniqueName: \"kubernetes.io/projected/31f4b14c-49f7-45f4-936e-e1be3123640c-kube-api-access-brlqv\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:16:18.641185 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:18.641186 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khjbw\" (UniqueName: \"kubernetes.io/projected/ab2c1b1c-1f3f-43fb-aa9d-248870c31da1-kube-api-access-khjbw\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:16:18.672392 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:18.672318 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c4006d-7bd575dbdf-h26mj"] Oct 31 05:16:18.676241 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:18.676202 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c4006d-7bd575dbdf-h26mj"] Oct 31 05:16:18.848846 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:18.848775 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8973e2-85c9f98f95-rgkrh" podUID="a4a19a46-de23-4984-894c-b14e0d223fab" containerName="payload-pre-processing" containerID="cri-o://af140097b359d0fca97e3965becf2a8cb63c13d3bbebadbf832c621e1398f23f" gracePeriod=25 Oct 31 05:16:19.407946 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:19.407861 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31f4b14c-49f7-45f4-936e-e1be3123640c" path="/var/lib/kubelet/pods/31f4b14c-49f7-45f4-936e-e1be3123640c/volumes" Oct 31 05:16:19.408311 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:19.408297 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab2c1b1c-1f3f-43fb-aa9d-248870c31da1" path="/var/lib/kubelet/pods/ab2c1b1c-1f3f-43fb-aa9d-248870c31da1/volumes" Oct 31 05:16:22.396852 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:16:22.396801 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:16:22.844016 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:22.843981 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn"] Oct 31 05:16:22.844292 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:22.844264 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn" podUID="2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09" containerName="istio-proxy" containerID="cri-o://1d0ca6465c7367d099b5546a74d634b5f427886ba40e83c236374ce2e874bb71" gracePeriod=30 Oct 31 05:16:23.218750 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:23.218705 5019 patch_prober.go:29] interesting pod/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 05:16:23.218980 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:23.218788 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn" podUID="2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 05:16:24.685842 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:24.685801 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a99f2a-5bb987db5-2m54d"] Oct 31 05:16:24.719007 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:24.718963 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a99f2a-74bfdc65dc-9j694"] Oct 31 05:16:28.133213 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:28.133174 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8973e2-85c9f98f95-rgkrh" podUID="a4a19a46-de23-4984-894c-b14e0d223fab" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:16:28.149695 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:28.149665 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn" Oct 31 05:16:28.266314 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:28.266267 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-istio-envoy\") pod \"2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09\" (UID: \"2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09\") " Oct 31 05:16:28.266550 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:28.266321 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-istio-data\") pod \"2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09\" (UID: \"2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09\") " Oct 31 05:16:28.266550 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:28.266374 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-istio-token\" (UniqueName: \"kubernetes.io/projected/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-istio-token\") pod \"2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09\" (UID: \"2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09\") " Oct 31 05:16:28.266550 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:28.266413 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-workload-socket\") pod \"2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09\" (UID: \"2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09\") " Oct 31 05:16:28.266550 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:28.266433 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-workload-certs\") pod \"2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09\" (UID: \"2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09\") " Oct 31 05:16:28.266550 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:28.266466 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-credential-socket\") pod \"2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09\" (UID: \"2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09\") " Oct 31 05:16:28.266550 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:28.266504 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-istiod-ca-cert\") pod \"2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09\" (UID: \"2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09\") " Oct 31 05:16:28.266909 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:28.266552 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-kube-api-access-nv6xp\" (UniqueName: \"kubernetes.io/projected/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-kube-api-access-nv6xp\") pod \"2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09\" (UID: \"2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09\") " Oct 31 05:16:28.266909 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:28.266599 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-istio-podinfo\") pod \"2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09\" (UID: \"2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09\") " Oct 31 05:16:28.266909 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:28.266675 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-istio-data" pod "2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09" (UID: "2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:16:28.266909 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:28.266850 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-workload-certs" pod "2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09" (UID: "2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:16:28.267223 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:28.267016 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-istiod-ca-cert" pod "2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09" (UID: "2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:16:28.267223 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:28.267140 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:16:28.267223 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:28.267162 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:16:28.267223 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:28.267174 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:16:28.267484 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:28.267233 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-credential-socket" pod "2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09" (UID: "2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:16:28.267484 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:28.267325 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-workload-socket" pod "2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09" (UID: "2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:16:28.269049 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:28.269013 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-kube-api-access-nv6xp" pod "2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09" (UID: "2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09"). InnerVolumeSpecName "kube-api-access-nv6xp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:16:28.269049 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:28.269032 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-istio-podinfo" pod "2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09" (UID: "2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:16:28.269237 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:28.269201 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-istio-envoy" pod "2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09" (UID: "2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:16:28.269280 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:28.269233 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-istio-token" pod "2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09" (UID: "2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:16:28.368661 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:28.368613 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:16:28.368661 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:28.368655 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv6xp\" (UniqueName: \"kubernetes.io/projected/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-kube-api-access-nv6xp\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:16:28.368661 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:28.368666 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:16:28.368944 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:28.368675 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:16:28.368944 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:28.368684 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:16:28.368944 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:28.368692 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:16:28.651351 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:28.651299 5019 generic.go:356] "Generic (PLEG): container finished" podID="2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09" containerID="1d0ca6465c7367d099b5546a74d634b5f427886ba40e83c236374ce2e874bb71" exitCode=0 Oct 31 05:16:28.651604 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:28.651383 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn" Oct 31 05:16:28.651604 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:28.651449 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn" event={"ID":"2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09","Type":"ContainerDied","Data":"1d0ca6465c7367d099b5546a74d634b5f427886ba40e83c236374ce2e874bb71"} Oct 31 05:16:28.651604 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:28.651517 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn" event={"ID":"2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09","Type":"ContainerDied","Data":"4673d8306c5aa5307116b09c77d7bde416c6ead5efba6828afe24913c2f5587b"} Oct 31 05:16:28.651604 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:28.651559 5019 scope.go:122] "RemoveContainer" containerID="1d0ca6465c7367d099b5546a74d634b5f427886ba40e83c236374ce2e874bb71" Oct 31 05:16:28.664903 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:28.664877 5019 scope.go:122] "RemoveContainer" containerID="cb00f57e539761d7c6459db710bfee12b40bb84e993b14d408ad19ebb35ab74b" Oct 31 05:16:28.676608 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:28.676571 5019 scope.go:122] "RemoveContainer" containerID="1d0ca6465c7367d099b5546a74d634b5f427886ba40e83c236374ce2e874bb71" Oct 31 05:16:28.677012 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:16:28.676986 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d0ca6465c7367d099b5546a74d634b5f427886ba40e83c236374ce2e874bb71\": container with ID starting with 1d0ca6465c7367d099b5546a74d634b5f427886ba40e83c236374ce2e874bb71 not found: ID does not exist" containerID="1d0ca6465c7367d099b5546a74d634b5f427886ba40e83c236374ce2e874bb71" Oct 31 05:16:28.677119 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:28.677025 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d0ca6465c7367d099b5546a74d634b5f427886ba40e83c236374ce2e874bb71"} err="failed to get container status \"1d0ca6465c7367d099b5546a74d634b5f427886ba40e83c236374ce2e874bb71\": rpc error: code = NotFound desc = could not find container \"1d0ca6465c7367d099b5546a74d634b5f427886ba40e83c236374ce2e874bb71\": container with ID starting with 1d0ca6465c7367d099b5546a74d634b5f427886ba40e83c236374ce2e874bb71 not found: ID does not exist" Oct 31 05:16:28.677119 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:28.677048 5019 scope.go:122] "RemoveContainer" containerID="cb00f57e539761d7c6459db710bfee12b40bb84e993b14d408ad19ebb35ab74b" Oct 31 05:16:28.677395 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:16:28.677367 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb00f57e539761d7c6459db710bfee12b40bb84e993b14d408ad19ebb35ab74b\": container with ID starting with cb00f57e539761d7c6459db710bfee12b40bb84e993b14d408ad19ebb35ab74b not found: ID does not exist" containerID="cb00f57e539761d7c6459db710bfee12b40bb84e993b14d408ad19ebb35ab74b" Oct 31 05:16:28.677485 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:28.677406 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb00f57e539761d7c6459db710bfee12b40bb84e993b14d408ad19ebb35ab74b"} err="failed to get container status \"cb00f57e539761d7c6459db710bfee12b40bb84e993b14d408ad19ebb35ab74b\": rpc error: code = NotFound desc = could not find container \"cb00f57e539761d7c6459db710bfee12b40bb84e993b14d408ad19ebb35ab74b\": container with ID starting with cb00f57e539761d7c6459db710bfee12b40bb84e993b14d408ad19ebb35ab74b not found: ID does not exist" Oct 31 05:16:28.717030 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:28.716978 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn"] Oct 31 05:16:28.722016 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:28.721976 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-8973e2-openshift-default-6bd79597f7-h2ldn"] Oct 31 05:16:29.407668 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:29.407630 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09" path="/var/lib/kubelet/pods/2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09/volumes" Oct 31 05:16:29.686725 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:29.686612 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-a99f2a-5bb987db5-2m54d" podUID="46aae214-de7a-4736-9aad-3b5499aabd6f" containerName="payload-processing" containerID="cri-o://06f3675d63e93c1f1fd74cf95d265d78d62afe73fd6fc65469c319162540ec87" gracePeriod=25 Oct 31 05:16:29.719497 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:29.719445 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a99f2a-74bfdc65dc-9j694" podUID="91300bb2-7008-42f1-ba58-f7ab87e1d748" containerName="payload-pre-processing" containerID="cri-o://c00c2a8ba6fe92f2f072929ff59f31195b42edc1837e0ea9aa38b84cb72b2fe1" gracePeriod=25 Oct 31 05:16:29.865903 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:29.865855 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-227c93-5c594ccb75-k8gp2"] Oct 31 05:16:34.400174 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:34.400134 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk"] Oct 31 05:16:34.401259 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:34.401226 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk" podUID="17f31b2c-2988-45e1-92ae-e43ff492444a" containerName="istio-proxy" containerID="cri-o://542b8882dcaccae106c7375802a129b8f749f6d1a38c3394460d4a81cc4a7ba9" gracePeriod=30 Oct 31 05:16:34.867071 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:34.867009 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-227c93-5c594ccb75-k8gp2" podUID="16674b5b-1b4d-40d5-aa1d-704a9c56b54b" containerName="payload-pre-processing" containerID="cri-o://af8420201364c24f0ba46fd4cc30670ae1f1c76da091787aa63374dd9638bd24" gracePeriod=25 Oct 31 05:16:36.111884 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:36.111830 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-227c93-5c594ccb75-k8gp2" podUID="16674b5b-1b4d-40d5-aa1d-704a9c56b54b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:16:38.132920 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:38.132875 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8973e2-85c9f98f95-rgkrh" podUID="a4a19a46-de23-4984-894c-b14e0d223fab" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:16:38.820519 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:38.820473 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a99f2a-74bfdc65dc-9j694" podUID="91300bb2-7008-42f1-ba58-f7ab87e1d748" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:16:38.820743 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:38.820518 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-a99f2a-5bb987db5-2m54d" podUID="46aae214-de7a-4736-9aad-3b5499aabd6f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:16:39.147515 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:39.147482 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8973e2-85c9f98f95-rgkrh" Oct 31 05:16:39.194252 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:39.194210 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4a19a46-de23-4984-894c-b14e0d223fab-kube-api-access-ns9tq\" (UniqueName: \"kubernetes.io/projected/a4a19a46-de23-4984-894c-b14e0d223fab-kube-api-access-ns9tq\") pod \"a4a19a46-de23-4984-894c-b14e0d223fab\" (UID: \"a4a19a46-de23-4984-894c-b14e0d223fab\") " Oct 31 05:16:39.194529 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:39.194488 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4a19a46-de23-4984-894c-b14e0d223fab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4a19a46-de23-4984-894c-b14e0d223fab-plugins-config-volume\") pod \"a4a19a46-de23-4984-894c-b14e0d223fab\" (UID: \"a4a19a46-de23-4984-894c-b14e0d223fab\") " Oct 31 05:16:39.194966 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:39.194927 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4a19a46-de23-4984-894c-b14e0d223fab-plugins-config-volume" pod "a4a19a46-de23-4984-894c-b14e0d223fab" (UID: "a4a19a46-de23-4984-894c-b14e0d223fab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:16:39.195903 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:39.195866 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4a19a46-de23-4984-894c-b14e0d223fab-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:16:39.197009 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:39.196977 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4a19a46-de23-4984-894c-b14e0d223fab-kube-api-access-ns9tq" pod "a4a19a46-de23-4984-894c-b14e0d223fab" (UID: "a4a19a46-de23-4984-894c-b14e0d223fab"). InnerVolumeSpecName "kube-api-access-ns9tq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:16:39.296615 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:39.296562 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns9tq\" (UniqueName: \"kubernetes.io/projected/a4a19a46-de23-4984-894c-b14e0d223fab-kube-api-access-ns9tq\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:16:39.690583 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:39.690554 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk" Oct 31 05:16:39.741859 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:39.741818 5019 generic.go:356] "Generic (PLEG): container finished" podID="a4a19a46-de23-4984-894c-b14e0d223fab" containerID="af140097b359d0fca97e3965becf2a8cb63c13d3bbebadbf832c621e1398f23f" exitCode=0 Oct 31 05:16:39.742040 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:39.741864 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8973e2-85c9f98f95-rgkrh" Oct 31 05:16:39.742040 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:39.741889 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8973e2-85c9f98f95-rgkrh" event={"ID":"a4a19a46-de23-4984-894c-b14e0d223fab","Type":"ContainerDied","Data":"af140097b359d0fca97e3965becf2a8cb63c13d3bbebadbf832c621e1398f23f"} Oct 31 05:16:39.742040 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:39.741945 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8973e2-85c9f98f95-rgkrh" event={"ID":"a4a19a46-de23-4984-894c-b14e0d223fab","Type":"ContainerDied","Data":"38806fdc81eab273824c83fc7d552d4a0395aab807bf86bf5cce80cb25526b50"} Oct 31 05:16:39.742040 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:39.741969 5019 scope.go:122] "RemoveContainer" containerID="af140097b359d0fca97e3965becf2a8cb63c13d3bbebadbf832c621e1398f23f" Oct 31 05:16:39.744405 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:39.744354 5019 generic.go:356] "Generic (PLEG): container finished" podID="17f31b2c-2988-45e1-92ae-e43ff492444a" containerID="542b8882dcaccae106c7375802a129b8f749f6d1a38c3394460d4a81cc4a7ba9" exitCode=0 Oct 31 05:16:39.744405 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:39.744403 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk" event={"ID":"17f31b2c-2988-45e1-92ae-e43ff492444a","Type":"ContainerDied","Data":"542b8882dcaccae106c7375802a129b8f749f6d1a38c3394460d4a81cc4a7ba9"} Oct 31 05:16:39.744707 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:39.744445 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk" event={"ID":"17f31b2c-2988-45e1-92ae-e43ff492444a","Type":"ContainerDied","Data":"2333c1006ca0e0be3207a87a376cf26f332f84b79178a890a949132db589a3a6"} Oct 31 05:16:39.744707 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:39.744613 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk" Oct 31 05:16:39.754520 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:39.754471 5019 scope.go:122] "RemoveContainer" containerID="af140097b359d0fca97e3965becf2a8cb63c13d3bbebadbf832c621e1398f23f" Oct 31 05:16:39.754806 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:16:39.754776 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af140097b359d0fca97e3965becf2a8cb63c13d3bbebadbf832c621e1398f23f\": container with ID starting with af140097b359d0fca97e3965becf2a8cb63c13d3bbebadbf832c621e1398f23f not found: ID does not exist" containerID="af140097b359d0fca97e3965becf2a8cb63c13d3bbebadbf832c621e1398f23f" Oct 31 05:16:39.754935 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:39.754833 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af140097b359d0fca97e3965becf2a8cb63c13d3bbebadbf832c621e1398f23f"} err="failed to get container status \"af140097b359d0fca97e3965becf2a8cb63c13d3bbebadbf832c621e1398f23f\": rpc error: code = NotFound desc = could not find container \"af140097b359d0fca97e3965becf2a8cb63c13d3bbebadbf832c621e1398f23f\": container with ID starting with af140097b359d0fca97e3965becf2a8cb63c13d3bbebadbf832c621e1398f23f not found: ID does not exist" Oct 31 05:16:39.754935 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:39.754857 5019 scope.go:122] "RemoveContainer" containerID="542b8882dcaccae106c7375802a129b8f749f6d1a38c3394460d4a81cc4a7ba9" Oct 31 05:16:39.773592 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:39.773483 5019 scope.go:122] "RemoveContainer" containerID="c6af17490a3d2c22c621122f5463b2021aa83b76ff7e6dc4b6dfd89f99ac1689" Oct 31 05:16:39.786186 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:39.786158 5019 scope.go:122] "RemoveContainer" containerID="542b8882dcaccae106c7375802a129b8f749f6d1a38c3394460d4a81cc4a7ba9" Oct 31 05:16:39.792555 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:39.792527 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8973e2-85c9f98f95-rgkrh"] Oct 31 05:16:39.792725 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:16:39.792696 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"542b8882dcaccae106c7375802a129b8f749f6d1a38c3394460d4a81cc4a7ba9\": container with ID starting with 542b8882dcaccae106c7375802a129b8f749f6d1a38c3394460d4a81cc4a7ba9 not found: ID does not exist" containerID="542b8882dcaccae106c7375802a129b8f749f6d1a38c3394460d4a81cc4a7ba9" Oct 31 05:16:39.792865 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:39.792732 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"542b8882dcaccae106c7375802a129b8f749f6d1a38c3394460d4a81cc4a7ba9"} err="failed to get container status \"542b8882dcaccae106c7375802a129b8f749f6d1a38c3394460d4a81cc4a7ba9\": rpc error: code = NotFound desc = could not find container \"542b8882dcaccae106c7375802a129b8f749f6d1a38c3394460d4a81cc4a7ba9\": container with ID starting with 542b8882dcaccae106c7375802a129b8f749f6d1a38c3394460d4a81cc4a7ba9 not found: ID does not exist" Oct 31 05:16:39.792865 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:39.792756 5019 scope.go:122] "RemoveContainer" containerID="c6af17490a3d2c22c621122f5463b2021aa83b76ff7e6dc4b6dfd89f99ac1689" Oct 31 05:16:39.793137 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:16:39.793093 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6af17490a3d2c22c621122f5463b2021aa83b76ff7e6dc4b6dfd89f99ac1689\": container with ID starting with c6af17490a3d2c22c621122f5463b2021aa83b76ff7e6dc4b6dfd89f99ac1689 not found: ID does not exist" containerID="c6af17490a3d2c22c621122f5463b2021aa83b76ff7e6dc4b6dfd89f99ac1689" Oct 31 05:16:39.793189 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:39.793147 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6af17490a3d2c22c621122f5463b2021aa83b76ff7e6dc4b6dfd89f99ac1689"} err="failed to get container status \"c6af17490a3d2c22c621122f5463b2021aa83b76ff7e6dc4b6dfd89f99ac1689\": rpc error: code = NotFound desc = could not find container \"c6af17490a3d2c22c621122f5463b2021aa83b76ff7e6dc4b6dfd89f99ac1689\": container with ID starting with c6af17490a3d2c22c621122f5463b2021aa83b76ff7e6dc4b6dfd89f99ac1689 not found: ID does not exist" Oct 31 05:16:39.798727 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:39.798701 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8973e2-85c9f98f95-rgkrh"] Oct 31 05:16:39.800434 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:39.800413 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17f31b2c-2988-45e1-92ae-e43ff492444a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17f31b2c-2988-45e1-92ae-e43ff492444a-istio-podinfo\") pod \"17f31b2c-2988-45e1-92ae-e43ff492444a\" (UID: \"17f31b2c-2988-45e1-92ae-e43ff492444a\") " Oct 31 05:16:39.800512 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:39.800447 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17f31b2c-2988-45e1-92ae-e43ff492444a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17f31b2c-2988-45e1-92ae-e43ff492444a-istio-data\") pod \"17f31b2c-2988-45e1-92ae-e43ff492444a\" (UID: \"17f31b2c-2988-45e1-92ae-e43ff492444a\") " Oct 31 05:16:39.800512 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:39.800474 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17f31b2c-2988-45e1-92ae-e43ff492444a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f31b2c-2988-45e1-92ae-e43ff492444a-credential-socket\") pod \"17f31b2c-2988-45e1-92ae-e43ff492444a\" (UID: \"17f31b2c-2988-45e1-92ae-e43ff492444a\") " Oct 31 05:16:39.800512 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:39.800505 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17f31b2c-2988-45e1-92ae-e43ff492444a-kube-api-access-v7h78\" (UniqueName: \"kubernetes.io/projected/17f31b2c-2988-45e1-92ae-e43ff492444a-kube-api-access-v7h78\") pod \"17f31b2c-2988-45e1-92ae-e43ff492444a\" (UID: \"17f31b2c-2988-45e1-92ae-e43ff492444a\") " Oct 31 05:16:39.800698 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:39.800588 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17f31b2c-2988-45e1-92ae-e43ff492444a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f31b2c-2988-45e1-92ae-e43ff492444a-workload-socket\") pod \"17f31b2c-2988-45e1-92ae-e43ff492444a\" (UID: \"17f31b2c-2988-45e1-92ae-e43ff492444a\") " Oct 31 05:16:39.800698 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:39.800642 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17f31b2c-2988-45e1-92ae-e43ff492444a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17f31b2c-2988-45e1-92ae-e43ff492444a-istio-envoy\") pod \"17f31b2c-2988-45e1-92ae-e43ff492444a\" (UID: \"17f31b2c-2988-45e1-92ae-e43ff492444a\") " Oct 31 05:16:39.800698 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:39.800680 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17f31b2c-2988-45e1-92ae-e43ff492444a-istio-token\" (UniqueName: \"kubernetes.io/projected/17f31b2c-2988-45e1-92ae-e43ff492444a-istio-token\") pod \"17f31b2c-2988-45e1-92ae-e43ff492444a\" (UID: \"17f31b2c-2988-45e1-92ae-e43ff492444a\") " Oct 31 05:16:39.800876 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:39.800711 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17f31b2c-2988-45e1-92ae-e43ff492444a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17f31b2c-2988-45e1-92ae-e43ff492444a-istiod-ca-cert\") pod \"17f31b2c-2988-45e1-92ae-e43ff492444a\" (UID: \"17f31b2c-2988-45e1-92ae-e43ff492444a\") " Oct 31 05:16:39.800876 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:39.800800 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17f31b2c-2988-45e1-92ae-e43ff492444a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17f31b2c-2988-45e1-92ae-e43ff492444a-workload-certs\") pod \"17f31b2c-2988-45e1-92ae-e43ff492444a\" (UID: \"17f31b2c-2988-45e1-92ae-e43ff492444a\") " Oct 31 05:16:39.801000 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:39.800907 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17f31b2c-2988-45e1-92ae-e43ff492444a-workload-socket" pod "17f31b2c-2988-45e1-92ae-e43ff492444a" (UID: "17f31b2c-2988-45e1-92ae-e43ff492444a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:16:39.801065 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:39.801039 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17f31b2c-2988-45e1-92ae-e43ff492444a-credential-socket" pod "17f31b2c-2988-45e1-92ae-e43ff492444a" (UID: "17f31b2c-2988-45e1-92ae-e43ff492444a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:16:39.801137 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:39.801113 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17f31b2c-2988-45e1-92ae-e43ff492444a-istio-data" pod "17f31b2c-2988-45e1-92ae-e43ff492444a" (UID: "17f31b2c-2988-45e1-92ae-e43ff492444a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:16:39.801250 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:39.801230 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17f31b2c-2988-45e1-92ae-e43ff492444a-workload-certs" pod "17f31b2c-2988-45e1-92ae-e43ff492444a" (UID: "17f31b2c-2988-45e1-92ae-e43ff492444a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:16:39.801323 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:39.801249 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17f31b2c-2988-45e1-92ae-e43ff492444a-istiod-ca-cert" pod "17f31b2c-2988-45e1-92ae-e43ff492444a" (UID: "17f31b2c-2988-45e1-92ae-e43ff492444a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:16:39.801422 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:39.801393 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17f31b2c-2988-45e1-92ae-e43ff492444a-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:16:39.801422 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:39.801414 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f31b2c-2988-45e1-92ae-e43ff492444a-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:16:39.801533 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:39.801430 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f31b2c-2988-45e1-92ae-e43ff492444a-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:16:39.801533 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:39.801444 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17f31b2c-2988-45e1-92ae-e43ff492444a-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:16:39.801533 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:39.801457 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17f31b2c-2988-45e1-92ae-e43ff492444a-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:16:39.802897 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:39.802874 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17f31b2c-2988-45e1-92ae-e43ff492444a-istio-token" pod "17f31b2c-2988-45e1-92ae-e43ff492444a" (UID: "17f31b2c-2988-45e1-92ae-e43ff492444a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:16:39.802897 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:39.802882 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17f31b2c-2988-45e1-92ae-e43ff492444a-istio-envoy" pod "17f31b2c-2988-45e1-92ae-e43ff492444a" (UID: "17f31b2c-2988-45e1-92ae-e43ff492444a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:16:39.803041 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:39.802906 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17f31b2c-2988-45e1-92ae-e43ff492444a-kube-api-access-v7h78" pod "17f31b2c-2988-45e1-92ae-e43ff492444a" (UID: "17f31b2c-2988-45e1-92ae-e43ff492444a"). InnerVolumeSpecName "kube-api-access-v7h78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:16:39.803292 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:39.803276 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17f31b2c-2988-45e1-92ae-e43ff492444a-istio-podinfo" pod "17f31b2c-2988-45e1-92ae-e43ff492444a" (UID: "17f31b2c-2988-45e1-92ae-e43ff492444a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:16:39.903135 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:39.903082 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17f31b2c-2988-45e1-92ae-e43ff492444a-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:16:39.903135 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:39.903133 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17f31b2c-2988-45e1-92ae-e43ff492444a-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:16:39.903381 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:39.903147 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17f31b2c-2988-45e1-92ae-e43ff492444a-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:16:39.903381 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:39.903162 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7h78\" (UniqueName: \"kubernetes.io/projected/17f31b2c-2988-45e1-92ae-e43ff492444a-kube-api-access-v7h78\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:16:40.093412 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:40.093376 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk"] Oct 31 05:16:40.099221 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:40.099190 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-a99f2a-openshift-default-77649d68d5-cmchk"] Oct 31 05:16:41.242569 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:41.242519 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-338f2e-57f7757bf-mchhm"] Oct 31 05:16:41.272636 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:41.272585 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-338f2e-6ccbc66d75-htxcd"] Oct 31 05:16:41.408971 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:41.408927 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17f31b2c-2988-45e1-92ae-e43ff492444a" path="/var/lib/kubelet/pods/17f31b2c-2988-45e1-92ae-e43ff492444a/volumes" Oct 31 05:16:41.409450 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:41.409433 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4a19a46-de23-4984-894c-b14e0d223fab" path="/var/lib/kubelet/pods/a4a19a46-de23-4984-894c-b14e0d223fab/volumes" Oct 31 05:16:43.715006 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:43.714961 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4"] Oct 31 05:16:43.715546 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:43.715255 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4" podUID="39eb6eb3-a2ae-4931-bab8-1e3e07e7d764" containerName="istio-proxy" containerID="cri-o://0f63c21083717c83ce5abebb963eb427f75e6bb26fe840ae77bcc21b23cfb4cb" gracePeriod=30 Oct 31 05:16:44.721433 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:44.721388 5019 patch_prober.go:29] interesting pod/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 05:16:44.721849 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:44.721452 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4" podUID="39eb6eb3-a2ae-4931-bab8-1e3e07e7d764" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 05:16:46.111667 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:46.111551 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-227c93-5c594ccb75-k8gp2" podUID="16674b5b-1b4d-40d5-aa1d-704a9c56b54b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:16:46.243397 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:46.243321 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-338f2e-57f7757bf-mchhm" podUID="248dfc75-12b3-4535-91d1-5c604ac06498" containerName="payload-processing" containerID="cri-o://f429c47c34413b5a965793d294fb185d27ba0c31a7f67323c08dca502c530375" gracePeriod=25 Oct 31 05:16:46.273463 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:46.273407 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-338f2e-6ccbc66d75-htxcd" podUID="53f77b18-7ca9-4e11-869d-0a1aa66d2057" containerName="payload-pre-processing" containerID="cri-o://7badcff5803238f686bc30f63db2530bff481b08d6668a94ae0a2623a2df8861" gracePeriod=25 Oct 31 05:16:48.820196 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:48.820144 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a99f2a-74bfdc65dc-9j694" podUID="91300bb2-7008-42f1-ba58-f7ab87e1d748" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:16:48.820623 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:48.820165 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-a99f2a-5bb987db5-2m54d" podUID="46aae214-de7a-4736-9aad-3b5499aabd6f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:16:48.993358 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:48.993276 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4" Oct 31 05:16:49.110667 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:49.110620 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-istio-token\" (UniqueName: \"kubernetes.io/projected/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-istio-token\") pod \"39eb6eb3-a2ae-4931-bab8-1e3e07e7d764\" (UID: \"39eb6eb3-a2ae-4931-bab8-1e3e07e7d764\") " Oct 31 05:16:49.110865 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:49.110727 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-istio-envoy\") pod \"39eb6eb3-a2ae-4931-bab8-1e3e07e7d764\" (UID: \"39eb6eb3-a2ae-4931-bab8-1e3e07e7d764\") " Oct 31 05:16:49.110865 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:49.110776 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-workload-certs\") pod \"39eb6eb3-a2ae-4931-bab8-1e3e07e7d764\" (UID: \"39eb6eb3-a2ae-4931-bab8-1e3e07e7d764\") " Oct 31 05:16:49.110865 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:49.110819 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-istio-data\") pod \"39eb6eb3-a2ae-4931-bab8-1e3e07e7d764\" (UID: \"39eb6eb3-a2ae-4931-bab8-1e3e07e7d764\") " Oct 31 05:16:49.111065 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:49.110891 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-workload-socket\") pod \"39eb6eb3-a2ae-4931-bab8-1e3e07e7d764\" (UID: \"39eb6eb3-a2ae-4931-bab8-1e3e07e7d764\") " Oct 31 05:16:49.111065 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:49.110920 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-istiod-ca-cert\") pod \"39eb6eb3-a2ae-4931-bab8-1e3e07e7d764\" (UID: \"39eb6eb3-a2ae-4931-bab8-1e3e07e7d764\") " Oct 31 05:16:49.111065 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:49.110976 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-istio-podinfo\") pod \"39eb6eb3-a2ae-4931-bab8-1e3e07e7d764\" (UID: \"39eb6eb3-a2ae-4931-bab8-1e3e07e7d764\") " Oct 31 05:16:49.111065 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:49.111028 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-kube-api-access-8r7vq\" (UniqueName: \"kubernetes.io/projected/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-kube-api-access-8r7vq\") pod \"39eb6eb3-a2ae-4931-bab8-1e3e07e7d764\" (UID: \"39eb6eb3-a2ae-4931-bab8-1e3e07e7d764\") " Oct 31 05:16:49.111065 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:49.111061 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-credential-socket\") pod \"39eb6eb3-a2ae-4931-bab8-1e3e07e7d764\" (UID: \"39eb6eb3-a2ae-4931-bab8-1e3e07e7d764\") " Oct 31 05:16:49.111715 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:49.111471 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-credential-socket" pod "39eb6eb3-a2ae-4931-bab8-1e3e07e7d764" (UID: "39eb6eb3-a2ae-4931-bab8-1e3e07e7d764"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:16:49.111715 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:49.111674 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:16:49.111906 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:49.111720 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-workload-certs" pod "39eb6eb3-a2ae-4931-bab8-1e3e07e7d764" (UID: "39eb6eb3-a2ae-4931-bab8-1e3e07e7d764"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:16:49.111906 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:49.111823 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-istio-data" pod "39eb6eb3-a2ae-4931-bab8-1e3e07e7d764" (UID: "39eb6eb3-a2ae-4931-bab8-1e3e07e7d764"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:16:49.112026 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:49.111926 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-workload-socket" pod "39eb6eb3-a2ae-4931-bab8-1e3e07e7d764" (UID: "39eb6eb3-a2ae-4931-bab8-1e3e07e7d764"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:16:49.112026 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:49.111981 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-istiod-ca-cert" pod "39eb6eb3-a2ae-4931-bab8-1e3e07e7d764" (UID: "39eb6eb3-a2ae-4931-bab8-1e3e07e7d764"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:16:49.114490 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:49.114460 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-istio-podinfo" pod "39eb6eb3-a2ae-4931-bab8-1e3e07e7d764" (UID: "39eb6eb3-a2ae-4931-bab8-1e3e07e7d764"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:16:49.116718 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:49.116691 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-istio-envoy" pod "39eb6eb3-a2ae-4931-bab8-1e3e07e7d764" (UID: "39eb6eb3-a2ae-4931-bab8-1e3e07e7d764"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:16:49.116875 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:49.116722 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-istio-token" pod "39eb6eb3-a2ae-4931-bab8-1e3e07e7d764" (UID: "39eb6eb3-a2ae-4931-bab8-1e3e07e7d764"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:16:49.118241 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:49.118223 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-kube-api-access-8r7vq" pod "39eb6eb3-a2ae-4931-bab8-1e3e07e7d764" (UID: "39eb6eb3-a2ae-4931-bab8-1e3e07e7d764"). InnerVolumeSpecName "kube-api-access-8r7vq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:16:49.212306 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:49.212267 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r7vq\" (UniqueName: \"kubernetes.io/projected/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-kube-api-access-8r7vq\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:16:49.212306 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:49.212302 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:16:49.212306 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:49.212312 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:16:49.212598 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:49.212320 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:16:49.212598 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:49.212328 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:16:49.212598 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:49.212350 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:16:49.212598 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:49.212358 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:16:49.212598 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:49.212367 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:16:49.824874 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:49.824831 5019 generic.go:356] "Generic (PLEG): container finished" podID="46aae214-de7a-4736-9aad-3b5499aabd6f" containerID="06f3675d63e93c1f1fd74cf95d265d78d62afe73fd6fc65469c319162540ec87" exitCode=0 Oct 31 05:16:49.825414 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:49.824900 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-a99f2a-5bb987db5-2m54d" event={"ID":"46aae214-de7a-4736-9aad-3b5499aabd6f","Type":"ContainerDied","Data":"06f3675d63e93c1f1fd74cf95d265d78d62afe73fd6fc65469c319162540ec87"} Oct 31 05:16:49.827074 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:49.827051 5019 generic.go:356] "Generic (PLEG): container finished" podID="39eb6eb3-a2ae-4931-bab8-1e3e07e7d764" containerID="0f63c21083717c83ce5abebb963eb427f75e6bb26fe840ae77bcc21b23cfb4cb" exitCode=0 Oct 31 05:16:49.827173 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:49.827101 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4" event={"ID":"39eb6eb3-a2ae-4931-bab8-1e3e07e7d764","Type":"ContainerDied","Data":"0f63c21083717c83ce5abebb963eb427f75e6bb26fe840ae77bcc21b23cfb4cb"} Oct 31 05:16:49.827173 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:49.827117 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4" Oct 31 05:16:49.827173 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:49.827144 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4" event={"ID":"39eb6eb3-a2ae-4931-bab8-1e3e07e7d764","Type":"ContainerDied","Data":"9e63a6c28088564bf014675e9e3c6b864aea7fb3d89a93dd0903060e00579115"} Oct 31 05:16:49.827173 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:49.827165 5019 scope.go:122] "RemoveContainer" containerID="0f63c21083717c83ce5abebb963eb427f75e6bb26fe840ae77bcc21b23cfb4cb" Oct 31 05:16:49.840700 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:49.840555 5019 scope.go:122] "RemoveContainer" containerID="ceaa0513f8cbeb4411b336dc868532a62efa55a1e389ef109e97c3eec5ade4ba" Oct 31 05:16:49.853609 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:49.853575 5019 scope.go:122] "RemoveContainer" containerID="0f63c21083717c83ce5abebb963eb427f75e6bb26fe840ae77bcc21b23cfb4cb" Oct 31 05:16:49.854054 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:16:49.854025 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f63c21083717c83ce5abebb963eb427f75e6bb26fe840ae77bcc21b23cfb4cb\": container with ID starting with 0f63c21083717c83ce5abebb963eb427f75e6bb26fe840ae77bcc21b23cfb4cb not found: ID does not exist" containerID="0f63c21083717c83ce5abebb963eb427f75e6bb26fe840ae77bcc21b23cfb4cb" Oct 31 05:16:49.854175 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:49.854068 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f63c21083717c83ce5abebb963eb427f75e6bb26fe840ae77bcc21b23cfb4cb"} err="failed to get container status \"0f63c21083717c83ce5abebb963eb427f75e6bb26fe840ae77bcc21b23cfb4cb\": rpc error: code = NotFound desc = could not find container \"0f63c21083717c83ce5abebb963eb427f75e6bb26fe840ae77bcc21b23cfb4cb\": container with ID starting with 0f63c21083717c83ce5abebb963eb427f75e6bb26fe840ae77bcc21b23cfb4cb not found: ID does not exist" Oct 31 05:16:49.854175 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:49.854094 5019 scope.go:122] "RemoveContainer" containerID="ceaa0513f8cbeb4411b336dc868532a62efa55a1e389ef109e97c3eec5ade4ba" Oct 31 05:16:49.854497 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:16:49.854465 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ceaa0513f8cbeb4411b336dc868532a62efa55a1e389ef109e97c3eec5ade4ba\": container with ID starting with ceaa0513f8cbeb4411b336dc868532a62efa55a1e389ef109e97c3eec5ade4ba not found: ID does not exist" containerID="ceaa0513f8cbeb4411b336dc868532a62efa55a1e389ef109e97c3eec5ade4ba" Oct 31 05:16:49.854610 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:49.854510 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ceaa0513f8cbeb4411b336dc868532a62efa55a1e389ef109e97c3eec5ade4ba"} err="failed to get container status \"ceaa0513f8cbeb4411b336dc868532a62efa55a1e389ef109e97c3eec5ade4ba\": rpc error: code = NotFound desc = could not find container \"ceaa0513f8cbeb4411b336dc868532a62efa55a1e389ef109e97c3eec5ade4ba\": container with ID starting with ceaa0513f8cbeb4411b336dc868532a62efa55a1e389ef109e97c3eec5ade4ba not found: ID does not exist" Oct 31 05:16:49.904846 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:49.904749 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4"] Oct 31 05:16:49.912105 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:49.912046 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-227c93-openshift-default-74bc588c4f-sqhs4"] Oct 31 05:16:50.044665 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:50.044583 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-a99f2a-5bb987db5-2m54d" Oct 31 05:16:50.086818 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:50.086783 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a99f2a-74bfdc65dc-9j694" Oct 31 05:16:50.125323 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:50.125276 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46aae214-de7a-4736-9aad-3b5499aabd6f-kube-api-access-tsjdn\" (UniqueName: \"kubernetes.io/projected/46aae214-de7a-4736-9aad-3b5499aabd6f-kube-api-access-tsjdn\") pod \"46aae214-de7a-4736-9aad-3b5499aabd6f\" (UID: \"46aae214-de7a-4736-9aad-3b5499aabd6f\") " Oct 31 05:16:50.125569 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:50.125435 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46aae214-de7a-4736-9aad-3b5499aabd6f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46aae214-de7a-4736-9aad-3b5499aabd6f-plugins-config-volume\") pod \"46aae214-de7a-4736-9aad-3b5499aabd6f\" (UID: \"46aae214-de7a-4736-9aad-3b5499aabd6f\") " Oct 31 05:16:50.126415 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:50.126375 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46aae214-de7a-4736-9aad-3b5499aabd6f-plugins-config-volume" pod "46aae214-de7a-4736-9aad-3b5499aabd6f" (UID: "46aae214-de7a-4736-9aad-3b5499aabd6f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:16:50.135153 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:50.135110 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46aae214-de7a-4736-9aad-3b5499aabd6f-kube-api-access-tsjdn" pod "46aae214-de7a-4736-9aad-3b5499aabd6f" (UID: "46aae214-de7a-4736-9aad-3b5499aabd6f"). InnerVolumeSpecName "kube-api-access-tsjdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:16:50.226715 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:50.226618 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91300bb2-7008-42f1-ba58-f7ab87e1d748-kube-api-access-vdjzw\" (UniqueName: \"kubernetes.io/projected/91300bb2-7008-42f1-ba58-f7ab87e1d748-kube-api-access-vdjzw\") pod \"91300bb2-7008-42f1-ba58-f7ab87e1d748\" (UID: \"91300bb2-7008-42f1-ba58-f7ab87e1d748\") " Oct 31 05:16:50.226715 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:50.226706 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91300bb2-7008-42f1-ba58-f7ab87e1d748-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91300bb2-7008-42f1-ba58-f7ab87e1d748-plugins-config-volume\") pod \"91300bb2-7008-42f1-ba58-f7ab87e1d748\" (UID: \"91300bb2-7008-42f1-ba58-f7ab87e1d748\") " Oct 31 05:16:50.227036 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:50.227018 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsjdn\" (UniqueName: \"kubernetes.io/projected/46aae214-de7a-4736-9aad-3b5499aabd6f-kube-api-access-tsjdn\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:16:50.227112 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:50.227038 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46aae214-de7a-4736-9aad-3b5499aabd6f-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:16:50.227183 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:50.227153 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91300bb2-7008-42f1-ba58-f7ab87e1d748-plugins-config-volume" pod "91300bb2-7008-42f1-ba58-f7ab87e1d748" (UID: "91300bb2-7008-42f1-ba58-f7ab87e1d748"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:16:50.229034 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:50.229004 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91300bb2-7008-42f1-ba58-f7ab87e1d748-kube-api-access-vdjzw" pod "91300bb2-7008-42f1-ba58-f7ab87e1d748" (UID: "91300bb2-7008-42f1-ba58-f7ab87e1d748"). InnerVolumeSpecName "kube-api-access-vdjzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:16:50.328398 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:50.328322 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdjzw\" (UniqueName: \"kubernetes.io/projected/91300bb2-7008-42f1-ba58-f7ab87e1d748-kube-api-access-vdjzw\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:16:50.328398 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:50.328380 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91300bb2-7008-42f1-ba58-f7ab87e1d748-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:16:50.842379 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:50.842306 5019 generic.go:356] "Generic (PLEG): container finished" podID="91300bb2-7008-42f1-ba58-f7ab87e1d748" containerID="c00c2a8ba6fe92f2f072929ff59f31195b42edc1837e0ea9aa38b84cb72b2fe1" exitCode=0 Oct 31 05:16:50.842379 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:50.842378 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a99f2a-74bfdc65dc-9j694" Oct 31 05:16:50.843007 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:50.842401 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a99f2a-74bfdc65dc-9j694" event={"ID":"91300bb2-7008-42f1-ba58-f7ab87e1d748","Type":"ContainerDied","Data":"c00c2a8ba6fe92f2f072929ff59f31195b42edc1837e0ea9aa38b84cb72b2fe1"} Oct 31 05:16:50.843007 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:50.842463 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a99f2a-74bfdc65dc-9j694" event={"ID":"91300bb2-7008-42f1-ba58-f7ab87e1d748","Type":"ContainerDied","Data":"9800ce632df4fb6b98c4e3b881491ff12f1217cee7f648b89381c65567b89eb6"} Oct 31 05:16:50.843007 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:50.842486 5019 scope.go:122] "RemoveContainer" containerID="c00c2a8ba6fe92f2f072929ff59f31195b42edc1837e0ea9aa38b84cb72b2fe1" Oct 31 05:16:50.844403 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:50.844372 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-a99f2a-5bb987db5-2m54d" event={"ID":"46aae214-de7a-4736-9aad-3b5499aabd6f","Type":"ContainerDied","Data":"e97d069493cf40db3489d0abd3c18a088b20c8b8725f2d0c406856bc80bf27e4"} Oct 31 05:16:50.844531 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:50.844459 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-a99f2a-5bb987db5-2m54d" Oct 31 05:16:50.855768 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:50.855681 5019 scope.go:122] "RemoveContainer" containerID="c00c2a8ba6fe92f2f072929ff59f31195b42edc1837e0ea9aa38b84cb72b2fe1" Oct 31 05:16:50.856237 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:16:50.856123 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c00c2a8ba6fe92f2f072929ff59f31195b42edc1837e0ea9aa38b84cb72b2fe1\": container with ID starting with c00c2a8ba6fe92f2f072929ff59f31195b42edc1837e0ea9aa38b84cb72b2fe1 not found: ID does not exist" containerID="c00c2a8ba6fe92f2f072929ff59f31195b42edc1837e0ea9aa38b84cb72b2fe1" Oct 31 05:16:50.856237 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:50.856162 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c00c2a8ba6fe92f2f072929ff59f31195b42edc1837e0ea9aa38b84cb72b2fe1"} err="failed to get container status \"c00c2a8ba6fe92f2f072929ff59f31195b42edc1837e0ea9aa38b84cb72b2fe1\": rpc error: code = NotFound desc = could not find container \"c00c2a8ba6fe92f2f072929ff59f31195b42edc1837e0ea9aa38b84cb72b2fe1\": container with ID starting with c00c2a8ba6fe92f2f072929ff59f31195b42edc1837e0ea9aa38b84cb72b2fe1 not found: ID does not exist" Oct 31 05:16:50.856237 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:50.856188 5019 scope.go:122] "RemoveContainer" containerID="06f3675d63e93c1f1fd74cf95d265d78d62afe73fd6fc65469c319162540ec87" Oct 31 05:16:50.894926 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:50.894882 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a99f2a-5bb987db5-2m54d"] Oct 31 05:16:50.900049 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:50.900011 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a99f2a-5bb987db5-2m54d"] Oct 31 05:16:50.934902 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:50.934856 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a99f2a-74bfdc65dc-9j694"] Oct 31 05:16:50.940151 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:50.940103 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a99f2a-74bfdc65dc-9j694"] Oct 31 05:16:51.409175 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:51.409136 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39eb6eb3-a2ae-4931-bab8-1e3e07e7d764" path="/var/lib/kubelet/pods/39eb6eb3-a2ae-4931-bab8-1e3e07e7d764/volumes" Oct 31 05:16:51.409646 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:51.409632 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46aae214-de7a-4736-9aad-3b5499aabd6f" path="/var/lib/kubelet/pods/46aae214-de7a-4736-9aad-3b5499aabd6f/volumes" Oct 31 05:16:51.409994 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:51.409982 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91300bb2-7008-42f1-ba58-f7ab87e1d748" path="/var/lib/kubelet/pods/91300bb2-7008-42f1-ba58-f7ab87e1d748/volumes" Oct 31 05:16:52.393277 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:16:52.393239 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:16:55.012050 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:55.012003 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9"] Oct 31 05:16:55.012665 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:55.012300 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9" podUID="b547bb0b-cae2-453b-aa56-4f37434a1ce8" containerName="istio-proxy" containerID="cri-o://2e652653ef68a8b7a63a7ed2034dca31b8832bc21c7f3e04f7b3f517dc962883" gracePeriod=30 Oct 31 05:16:55.106608 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:55.106561 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-338f2e-6ccbc66d75-htxcd" podUID="53f77b18-7ca9-4e11-869d-0a1aa66d2057" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:16:55.180680 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:55.180640 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-227c93-5c594ccb75-k8gp2" Oct 31 05:16:55.206891 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:55.205311 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16674b5b-1b4d-40d5-aa1d-704a9c56b54b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16674b5b-1b4d-40d5-aa1d-704a9c56b54b-plugins-config-volume\") pod \"16674b5b-1b4d-40d5-aa1d-704a9c56b54b\" (UID: \"16674b5b-1b4d-40d5-aa1d-704a9c56b54b\") " Oct 31 05:16:55.206891 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:55.205531 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16674b5b-1b4d-40d5-aa1d-704a9c56b54b-kube-api-access-2nc6v\" (UniqueName: \"kubernetes.io/projected/16674b5b-1b4d-40d5-aa1d-704a9c56b54b-kube-api-access-2nc6v\") pod \"16674b5b-1b4d-40d5-aa1d-704a9c56b54b\" (UID: \"16674b5b-1b4d-40d5-aa1d-704a9c56b54b\") " Oct 31 05:16:55.215603 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:55.215535 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16674b5b-1b4d-40d5-aa1d-704a9c56b54b-kube-api-access-2nc6v" pod "16674b5b-1b4d-40d5-aa1d-704a9c56b54b" (UID: "16674b5b-1b4d-40d5-aa1d-704a9c56b54b"). InnerVolumeSpecName "kube-api-access-2nc6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:16:55.216550 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:55.216515 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16674b5b-1b4d-40d5-aa1d-704a9c56b54b-plugins-config-volume" pod "16674b5b-1b4d-40d5-aa1d-704a9c56b54b" (UID: "16674b5b-1b4d-40d5-aa1d-704a9c56b54b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:16:55.306863 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:55.306814 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nc6v\" (UniqueName: \"kubernetes.io/projected/16674b5b-1b4d-40d5-aa1d-704a9c56b54b-kube-api-access-2nc6v\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:16:55.306863 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:55.306858 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16674b5b-1b4d-40d5-aa1d-704a9c56b54b-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:16:55.890443 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:55.890398 5019 generic.go:356] "Generic (PLEG): container finished" podID="16674b5b-1b4d-40d5-aa1d-704a9c56b54b" containerID="af8420201364c24f0ba46fd4cc30670ae1f1c76da091787aa63374dd9638bd24" exitCode=0 Oct 31 05:16:55.890666 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:55.890458 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-227c93-5c594ccb75-k8gp2" Oct 31 05:16:55.890666 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:55.890497 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-227c93-5c594ccb75-k8gp2" event={"ID":"16674b5b-1b4d-40d5-aa1d-704a9c56b54b","Type":"ContainerDied","Data":"af8420201364c24f0ba46fd4cc30670ae1f1c76da091787aa63374dd9638bd24"} Oct 31 05:16:55.890666 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:55.890597 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-227c93-5c594ccb75-k8gp2" event={"ID":"16674b5b-1b4d-40d5-aa1d-704a9c56b54b","Type":"ContainerDied","Data":"a07872063ed96ab7e6ee6ae5a9e8de94a2424d5f83600e3cc725d1398945639a"} Oct 31 05:16:55.890666 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:55.890630 5019 scope.go:122] "RemoveContainer" containerID="af8420201364c24f0ba46fd4cc30670ae1f1c76da091787aa63374dd9638bd24" Oct 31 05:16:55.904641 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:55.904610 5019 scope.go:122] "RemoveContainer" containerID="af8420201364c24f0ba46fd4cc30670ae1f1c76da091787aa63374dd9638bd24" Oct 31 05:16:55.905062 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:16:55.905030 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af8420201364c24f0ba46fd4cc30670ae1f1c76da091787aa63374dd9638bd24\": container with ID starting with af8420201364c24f0ba46fd4cc30670ae1f1c76da091787aa63374dd9638bd24 not found: ID does not exist" containerID="af8420201364c24f0ba46fd4cc30670ae1f1c76da091787aa63374dd9638bd24" Oct 31 05:16:55.905208 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:55.905072 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af8420201364c24f0ba46fd4cc30670ae1f1c76da091787aa63374dd9638bd24"} err="failed to get container status \"af8420201364c24f0ba46fd4cc30670ae1f1c76da091787aa63374dd9638bd24\": rpc error: code = NotFound desc = could not find container \"af8420201364c24f0ba46fd4cc30670ae1f1c76da091787aa63374dd9638bd24\": container with ID starting with af8420201364c24f0ba46fd4cc30670ae1f1c76da091787aa63374dd9638bd24 not found: ID does not exist" Oct 31 05:16:55.942836 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:55.942791 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-227c93-5c594ccb75-k8gp2"] Oct 31 05:16:55.954842 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:55.954804 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-227c93-5c594ccb75-k8gp2"] Oct 31 05:16:56.121499 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:56.121455 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-338f2e-57f7757bf-mchhm" podUID="248dfc75-12b3-4535-91d1-5c604ac06498" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:16:57.407634 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:57.407595 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16674b5b-1b4d-40d5-aa1d-704a9c56b54b" path="/var/lib/kubelet/pods/16674b5b-1b4d-40d5-aa1d-704a9c56b54b/volumes" Oct 31 05:16:57.767461 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:57.767424 5019 patch_prober.go:29] interesting pod/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 05:16:57.767698 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:57.767492 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9" podUID="b547bb0b-cae2-453b-aa56-4f37434a1ce8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 05:16:58.732987 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:58.732943 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz"] Oct 31 05:16:58.733447 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:16:58.733209 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz" podUID="de1f94d9-8507-47a5-aec8-88c546112c7f" containerName="istio-proxy" containerID="cri-o://f031d4562c563b9e35704ce4c384ddf3aed932bda02221db54e339d5cd5714b7" gracePeriod=30 Oct 31 05:17:00.310116 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:00.310024 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9" Oct 31 05:17:00.354894 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:00.354851 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b547bb0b-cae2-453b-aa56-4f37434a1ce8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b547bb0b-cae2-453b-aa56-4f37434a1ce8-istio-podinfo\") pod \"b547bb0b-cae2-453b-aa56-4f37434a1ce8\" (UID: \"b547bb0b-cae2-453b-aa56-4f37434a1ce8\") " Oct 31 05:17:00.355118 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:00.354931 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b547bb0b-cae2-453b-aa56-4f37434a1ce8-kube-api-access-6wmgk\" (UniqueName: \"kubernetes.io/projected/b547bb0b-cae2-453b-aa56-4f37434a1ce8-kube-api-access-6wmgk\") pod \"b547bb0b-cae2-453b-aa56-4f37434a1ce8\" (UID: \"b547bb0b-cae2-453b-aa56-4f37434a1ce8\") " Oct 31 05:17:00.355118 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:00.354960 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b547bb0b-cae2-453b-aa56-4f37434a1ce8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b547bb0b-cae2-453b-aa56-4f37434a1ce8-istio-envoy\") pod \"b547bb0b-cae2-453b-aa56-4f37434a1ce8\" (UID: \"b547bb0b-cae2-453b-aa56-4f37434a1ce8\") " Oct 31 05:17:00.355118 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:00.355081 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b547bb0b-cae2-453b-aa56-4f37434a1ce8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b547bb0b-cae2-453b-aa56-4f37434a1ce8-istiod-ca-cert\") pod \"b547bb0b-cae2-453b-aa56-4f37434a1ce8\" (UID: \"b547bb0b-cae2-453b-aa56-4f37434a1ce8\") " Oct 31 05:17:00.355269 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:00.355204 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b547bb0b-cae2-453b-aa56-4f37434a1ce8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b547bb0b-cae2-453b-aa56-4f37434a1ce8-workload-socket\") pod \"b547bb0b-cae2-453b-aa56-4f37434a1ce8\" (UID: \"b547bb0b-cae2-453b-aa56-4f37434a1ce8\") " Oct 31 05:17:00.355349 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:00.355267 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b547bb0b-cae2-453b-aa56-4f37434a1ce8-istio-token\" (UniqueName: \"kubernetes.io/projected/b547bb0b-cae2-453b-aa56-4f37434a1ce8-istio-token\") pod \"b547bb0b-cae2-453b-aa56-4f37434a1ce8\" (UID: \"b547bb0b-cae2-453b-aa56-4f37434a1ce8\") " Oct 31 05:17:00.355349 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:00.355310 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b547bb0b-cae2-453b-aa56-4f37434a1ce8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b547bb0b-cae2-453b-aa56-4f37434a1ce8-istio-data\") pod \"b547bb0b-cae2-453b-aa56-4f37434a1ce8\" (UID: \"b547bb0b-cae2-453b-aa56-4f37434a1ce8\") " Oct 31 05:17:00.355482 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:00.355396 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b547bb0b-cae2-453b-aa56-4f37434a1ce8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b547bb0b-cae2-453b-aa56-4f37434a1ce8-workload-certs\") pod \"b547bb0b-cae2-453b-aa56-4f37434a1ce8\" (UID: \"b547bb0b-cae2-453b-aa56-4f37434a1ce8\") " Oct 31 05:17:00.355574 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:00.355478 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b547bb0b-cae2-453b-aa56-4f37434a1ce8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b547bb0b-cae2-453b-aa56-4f37434a1ce8-credential-socket\") pod \"b547bb0b-cae2-453b-aa56-4f37434a1ce8\" (UID: \"b547bb0b-cae2-453b-aa56-4f37434a1ce8\") " Oct 31 05:17:00.355574 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:00.355535 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b547bb0b-cae2-453b-aa56-4f37434a1ce8-istiod-ca-cert" pod "b547bb0b-cae2-453b-aa56-4f37434a1ce8" (UID: "b547bb0b-cae2-453b-aa56-4f37434a1ce8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:17:00.355697 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:00.355596 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b547bb0b-cae2-453b-aa56-4f37434a1ce8-workload-socket" pod "b547bb0b-cae2-453b-aa56-4f37434a1ce8" (UID: "b547bb0b-cae2-453b-aa56-4f37434a1ce8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:17:00.355872 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:00.355848 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b547bb0b-cae2-453b-aa56-4f37434a1ce8-istio-data" pod "b547bb0b-cae2-453b-aa56-4f37434a1ce8" (UID: "b547bb0b-cae2-453b-aa56-4f37434a1ce8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:17:00.355872 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:00.355864 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b547bb0b-cae2-453b-aa56-4f37434a1ce8-workload-certs" pod "b547bb0b-cae2-453b-aa56-4f37434a1ce8" (UID: "b547bb0b-cae2-453b-aa56-4f37434a1ce8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:17:00.356035 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:00.355982 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b547bb0b-cae2-453b-aa56-4f37434a1ce8-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:00.356035 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:00.356004 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b547bb0b-cae2-453b-aa56-4f37434a1ce8-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:00.356035 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:00.356017 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b547bb0b-cae2-453b-aa56-4f37434a1ce8-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:00.356178 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:00.356061 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b547bb0b-cae2-453b-aa56-4f37434a1ce8-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:00.356178 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:00.356106 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b547bb0b-cae2-453b-aa56-4f37434a1ce8-credential-socket" pod "b547bb0b-cae2-453b-aa56-4f37434a1ce8" (UID: "b547bb0b-cae2-453b-aa56-4f37434a1ce8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:17:00.357785 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:00.357755 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b547bb0b-cae2-453b-aa56-4f37434a1ce8-kube-api-access-6wmgk" pod "b547bb0b-cae2-453b-aa56-4f37434a1ce8" (UID: "b547bb0b-cae2-453b-aa56-4f37434a1ce8"). InnerVolumeSpecName "kube-api-access-6wmgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:17:00.357940 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:00.357806 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b547bb0b-cae2-453b-aa56-4f37434a1ce8-istio-podinfo" pod "b547bb0b-cae2-453b-aa56-4f37434a1ce8" (UID: "b547bb0b-cae2-453b-aa56-4f37434a1ce8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:17:00.358077 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:00.358054 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b547bb0b-cae2-453b-aa56-4f37434a1ce8-istio-envoy" pod "b547bb0b-cae2-453b-aa56-4f37434a1ce8" (UID: "b547bb0b-cae2-453b-aa56-4f37434a1ce8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:17:00.358275 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:00.358256 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b547bb0b-cae2-453b-aa56-4f37434a1ce8-istio-token" pod "b547bb0b-cae2-453b-aa56-4f37434a1ce8" (UID: "b547bb0b-cae2-453b-aa56-4f37434a1ce8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:17:00.457269 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:00.457211 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b547bb0b-cae2-453b-aa56-4f37434a1ce8-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:00.457269 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:00.457263 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b547bb0b-cae2-453b-aa56-4f37434a1ce8-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:00.457269 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:00.457280 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b547bb0b-cae2-453b-aa56-4f37434a1ce8-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:00.457640 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:00.457294 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wmgk\" (UniqueName: \"kubernetes.io/projected/b547bb0b-cae2-453b-aa56-4f37434a1ce8-kube-api-access-6wmgk\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:00.457640 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:00.457309 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b547bb0b-cae2-453b-aa56-4f37434a1ce8-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:00.931321 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:00.931273 5019 generic.go:356] "Generic (PLEG): container finished" podID="b547bb0b-cae2-453b-aa56-4f37434a1ce8" containerID="2e652653ef68a8b7a63a7ed2034dca31b8832bc21c7f3e04f7b3f517dc962883" exitCode=0 Oct 31 05:17:00.931321 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:00.931325 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9" Oct 31 05:17:00.931659 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:00.931360 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9" event={"ID":"b547bb0b-cae2-453b-aa56-4f37434a1ce8","Type":"ContainerDied","Data":"2e652653ef68a8b7a63a7ed2034dca31b8832bc21c7f3e04f7b3f517dc962883"} Oct 31 05:17:00.931659 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:00.931489 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9" event={"ID":"b547bb0b-cae2-453b-aa56-4f37434a1ce8","Type":"ContainerDied","Data":"1a0345cd5def1ae4e0cb237e4c08d2f51af0687b5cb1f8a7f98e2bae828f16e1"} Oct 31 05:17:00.931659 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:00.931525 5019 scope.go:122] "RemoveContainer" containerID="2e652653ef68a8b7a63a7ed2034dca31b8832bc21c7f3e04f7b3f517dc962883" Oct 31 05:17:00.943814 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:00.943784 5019 scope.go:122] "RemoveContainer" containerID="07fe73b7b2ad40bd8c4007ec62b8acdccaf0a951bd6e972b1f4c02148cb75cd1" Oct 31 05:17:00.954988 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:00.954946 5019 scope.go:122] "RemoveContainer" containerID="2e652653ef68a8b7a63a7ed2034dca31b8832bc21c7f3e04f7b3f517dc962883" Oct 31 05:17:00.955301 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:17:00.955282 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e652653ef68a8b7a63a7ed2034dca31b8832bc21c7f3e04f7b3f517dc962883\": container with ID starting with 2e652653ef68a8b7a63a7ed2034dca31b8832bc21c7f3e04f7b3f517dc962883 not found: ID does not exist" containerID="2e652653ef68a8b7a63a7ed2034dca31b8832bc21c7f3e04f7b3f517dc962883" Oct 31 05:17:00.955401 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:00.955313 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e652653ef68a8b7a63a7ed2034dca31b8832bc21c7f3e04f7b3f517dc962883"} err="failed to get container status \"2e652653ef68a8b7a63a7ed2034dca31b8832bc21c7f3e04f7b3f517dc962883\": rpc error: code = NotFound desc = could not find container \"2e652653ef68a8b7a63a7ed2034dca31b8832bc21c7f3e04f7b3f517dc962883\": container with ID starting with 2e652653ef68a8b7a63a7ed2034dca31b8832bc21c7f3e04f7b3f517dc962883 not found: ID does not exist" Oct 31 05:17:00.955401 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:00.955345 5019 scope.go:122] "RemoveContainer" containerID="07fe73b7b2ad40bd8c4007ec62b8acdccaf0a951bd6e972b1f4c02148cb75cd1" Oct 31 05:17:00.955674 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:17:00.955653 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07fe73b7b2ad40bd8c4007ec62b8acdccaf0a951bd6e972b1f4c02148cb75cd1\": container with ID starting with 07fe73b7b2ad40bd8c4007ec62b8acdccaf0a951bd6e972b1f4c02148cb75cd1 not found: ID does not exist" containerID="07fe73b7b2ad40bd8c4007ec62b8acdccaf0a951bd6e972b1f4c02148cb75cd1" Oct 31 05:17:00.955747 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:00.955685 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07fe73b7b2ad40bd8c4007ec62b8acdccaf0a951bd6e972b1f4c02148cb75cd1"} err="failed to get container status \"07fe73b7b2ad40bd8c4007ec62b8acdccaf0a951bd6e972b1f4c02148cb75cd1\": rpc error: code = NotFound desc = could not find container \"07fe73b7b2ad40bd8c4007ec62b8acdccaf0a951bd6e972b1f4c02148cb75cd1\": container with ID starting with 07fe73b7b2ad40bd8c4007ec62b8acdccaf0a951bd6e972b1f4c02148cb75cd1 not found: ID does not exist" Oct 31 05:17:01.008000 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:01.007955 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9"] Oct 31 05:17:01.017922 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:01.017880 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-dvgp9"] Oct 31 05:17:01.408962 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:01.408922 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b547bb0b-cae2-453b-aa56-4f37434a1ce8" path="/var/lib/kubelet/pods/b547bb0b-cae2-453b-aa56-4f37434a1ce8/volumes" Oct 31 05:17:01.756259 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:01.756221 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-b2a372-dfc6bb5c9-2t4p7"] Oct 31 05:17:03.390571 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:03.390521 5019 patch_prober.go:29] interesting pod/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 05:17:03.391087 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:03.390595 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz" podUID="de1f94d9-8507-47a5-aec8-88c546112c7f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 05:17:03.960445 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:03.960394 5019 generic.go:356] "Generic (PLEG): container finished" podID="de1f94d9-8507-47a5-aec8-88c546112c7f" containerID="f031d4562c563b9e35704ce4c384ddf3aed932bda02221db54e339d5cd5714b7" exitCode=0 Oct 31 05:17:03.960651 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:03.960462 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz" event={"ID":"de1f94d9-8507-47a5-aec8-88c546112c7f","Type":"ContainerDied","Data":"f031d4562c563b9e35704ce4c384ddf3aed932bda02221db54e339d5cd5714b7"} Oct 31 05:17:03.960651 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:03.960540 5019 scope.go:122] "RemoveContainer" containerID="a3128cd2d5d93597ea6ac8b493b037d64093bbcf3f8da977dd3db858ff57b612" Oct 31 05:17:04.016420 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:04.016292 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz" Oct 31 05:17:04.099325 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:04.099276 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de1f94d9-8507-47a5-aec8-88c546112c7f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de1f94d9-8507-47a5-aec8-88c546112c7f-credential-socket\") pod \"de1f94d9-8507-47a5-aec8-88c546112c7f\" (UID: \"de1f94d9-8507-47a5-aec8-88c546112c7f\") " Oct 31 05:17:04.099325 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:04.099320 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de1f94d9-8507-47a5-aec8-88c546112c7f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de1f94d9-8507-47a5-aec8-88c546112c7f-istio-data\") pod \"de1f94d9-8507-47a5-aec8-88c546112c7f\" (UID: \"de1f94d9-8507-47a5-aec8-88c546112c7f\") " Oct 31 05:17:04.099654 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:04.099398 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de1f94d9-8507-47a5-aec8-88c546112c7f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de1f94d9-8507-47a5-aec8-88c546112c7f-istio-envoy\") pod \"de1f94d9-8507-47a5-aec8-88c546112c7f\" (UID: \"de1f94d9-8507-47a5-aec8-88c546112c7f\") " Oct 31 05:17:04.099654 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:04.099424 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de1f94d9-8507-47a5-aec8-88c546112c7f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de1f94d9-8507-47a5-aec8-88c546112c7f-workload-socket\") pod \"de1f94d9-8507-47a5-aec8-88c546112c7f\" (UID: \"de1f94d9-8507-47a5-aec8-88c546112c7f\") " Oct 31 05:17:04.099654 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:04.099452 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de1f94d9-8507-47a5-aec8-88c546112c7f-istio-token\" (UniqueName: \"kubernetes.io/projected/de1f94d9-8507-47a5-aec8-88c546112c7f-istio-token\") pod \"de1f94d9-8507-47a5-aec8-88c546112c7f\" (UID: \"de1f94d9-8507-47a5-aec8-88c546112c7f\") " Oct 31 05:17:04.099654 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:04.099485 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de1f94d9-8507-47a5-aec8-88c546112c7f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de1f94d9-8507-47a5-aec8-88c546112c7f-istio-podinfo\") pod \"de1f94d9-8507-47a5-aec8-88c546112c7f\" (UID: \"de1f94d9-8507-47a5-aec8-88c546112c7f\") " Oct 31 05:17:04.099654 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:04.099582 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de1f94d9-8507-47a5-aec8-88c546112c7f-kube-api-access-xs5q2\" (UniqueName: \"kubernetes.io/projected/de1f94d9-8507-47a5-aec8-88c546112c7f-kube-api-access-xs5q2\") pod \"de1f94d9-8507-47a5-aec8-88c546112c7f\" (UID: \"de1f94d9-8507-47a5-aec8-88c546112c7f\") " Oct 31 05:17:04.099654 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:04.099605 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de1f94d9-8507-47a5-aec8-88c546112c7f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de1f94d9-8507-47a5-aec8-88c546112c7f-workload-certs\") pod \"de1f94d9-8507-47a5-aec8-88c546112c7f\" (UID: \"de1f94d9-8507-47a5-aec8-88c546112c7f\") " Oct 31 05:17:04.100062 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:04.099676 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de1f94d9-8507-47a5-aec8-88c546112c7f-credential-socket" pod "de1f94d9-8507-47a5-aec8-88c546112c7f" (UID: "de1f94d9-8507-47a5-aec8-88c546112c7f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:17:04.100062 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:04.099743 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de1f94d9-8507-47a5-aec8-88c546112c7f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de1f94d9-8507-47a5-aec8-88c546112c7f-istiod-ca-cert\") pod \"de1f94d9-8507-47a5-aec8-88c546112c7f\" (UID: \"de1f94d9-8507-47a5-aec8-88c546112c7f\") " Oct 31 05:17:04.100062 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:04.099824 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de1f94d9-8507-47a5-aec8-88c546112c7f-istio-data" pod "de1f94d9-8507-47a5-aec8-88c546112c7f" (UID: "de1f94d9-8507-47a5-aec8-88c546112c7f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:17:04.100240 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:04.100201 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de1f94d9-8507-47a5-aec8-88c546112c7f-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:04.100240 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:04.100225 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de1f94d9-8507-47a5-aec8-88c546112c7f-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:04.100382 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:04.100243 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de1f94d9-8507-47a5-aec8-88c546112c7f-workload-socket" pod "de1f94d9-8507-47a5-aec8-88c546112c7f" (UID: "de1f94d9-8507-47a5-aec8-88c546112c7f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:17:04.100382 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:04.100261 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de1f94d9-8507-47a5-aec8-88c546112c7f-workload-certs" pod "de1f94d9-8507-47a5-aec8-88c546112c7f" (UID: "de1f94d9-8507-47a5-aec8-88c546112c7f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:17:04.100503 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:04.100400 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de1f94d9-8507-47a5-aec8-88c546112c7f-istiod-ca-cert" pod "de1f94d9-8507-47a5-aec8-88c546112c7f" (UID: "de1f94d9-8507-47a5-aec8-88c546112c7f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:17:04.102460 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:04.102431 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de1f94d9-8507-47a5-aec8-88c546112c7f-istio-podinfo" pod "de1f94d9-8507-47a5-aec8-88c546112c7f" (UID: "de1f94d9-8507-47a5-aec8-88c546112c7f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:17:04.102460 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:04.102445 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de1f94d9-8507-47a5-aec8-88c546112c7f-istio-envoy" pod "de1f94d9-8507-47a5-aec8-88c546112c7f" (UID: "de1f94d9-8507-47a5-aec8-88c546112c7f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:17:04.102646 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:04.102571 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de1f94d9-8507-47a5-aec8-88c546112c7f-istio-token" pod "de1f94d9-8507-47a5-aec8-88c546112c7f" (UID: "de1f94d9-8507-47a5-aec8-88c546112c7f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:17:04.102689 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:04.102642 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de1f94d9-8507-47a5-aec8-88c546112c7f-kube-api-access-xs5q2" pod "de1f94d9-8507-47a5-aec8-88c546112c7f" (UID: "de1f94d9-8507-47a5-aec8-88c546112c7f"). InnerVolumeSpecName "kube-api-access-xs5q2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:17:04.201710 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:04.201669 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de1f94d9-8507-47a5-aec8-88c546112c7f-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:04.201710 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:04.201707 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de1f94d9-8507-47a5-aec8-88c546112c7f-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:04.201710 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:04.201719 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de1f94d9-8507-47a5-aec8-88c546112c7f-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:04.201980 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:04.201731 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de1f94d9-8507-47a5-aec8-88c546112c7f-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:04.201980 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:04.201743 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs5q2\" (UniqueName: \"kubernetes.io/projected/de1f94d9-8507-47a5-aec8-88c546112c7f-kube-api-access-xs5q2\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:04.201980 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:04.201757 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de1f94d9-8507-47a5-aec8-88c546112c7f-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:04.201980 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:04.201769 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de1f94d9-8507-47a5-aec8-88c546112c7f-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:04.982892 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:04.982851 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz" Oct 31 05:17:04.983441 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:04.982853 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz" event={"ID":"de1f94d9-8507-47a5-aec8-88c546112c7f","Type":"ContainerDied","Data":"610c4119df441c58e59f0350da349afecbd924c14e598d62777928cedbd4eb1c"} Oct 31 05:17:04.983441 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:04.983041 5019 scope.go:122] "RemoveContainer" containerID="f031d4562c563b9e35704ce4c384ddf3aed932bda02221db54e339d5cd5714b7" Oct 31 05:17:05.050979 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:05.050943 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz"] Oct 31 05:17:05.062962 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:05.062922 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-766728-openshift-default-647554f999-xfztz"] Oct 31 05:17:05.104838 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:05.104797 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-338f2e-6ccbc66d75-htxcd" podUID="53f77b18-7ca9-4e11-869d-0a1aa66d2057" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:17:05.408022 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:05.407924 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de1f94d9-8507-47a5-aec8-88c546112c7f" path="/var/lib/kubelet/pods/de1f94d9-8507-47a5-aec8-88c546112c7f/volumes" Oct 31 05:17:06.121139 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:06.121090 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-338f2e-57f7757bf-mchhm" podUID="248dfc75-12b3-4535-91d1-5c604ac06498" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:17:06.554008 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:06.553977 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-338f2e-57f7757bf-mchhm" Oct 31 05:17:06.580665 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:06.580633 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-338f2e-6ccbc66d75-htxcd" Oct 31 05:17:06.625924 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:06.625888 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53f77b18-7ca9-4e11-869d-0a1aa66d2057-kube-api-access-4rcqb\" (UniqueName: \"kubernetes.io/projected/53f77b18-7ca9-4e11-869d-0a1aa66d2057-kube-api-access-4rcqb\") pod \"53f77b18-7ca9-4e11-869d-0a1aa66d2057\" (UID: \"53f77b18-7ca9-4e11-869d-0a1aa66d2057\") " Oct 31 05:17:06.626143 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:06.625961 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/248dfc75-12b3-4535-91d1-5c604ac06498-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/248dfc75-12b3-4535-91d1-5c604ac06498-plugins-config-volume\") pod \"248dfc75-12b3-4535-91d1-5c604ac06498\" (UID: \"248dfc75-12b3-4535-91d1-5c604ac06498\") " Oct 31 05:17:06.626143 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:06.625986 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/248dfc75-12b3-4535-91d1-5c604ac06498-kube-api-access-kvc4m\" (UniqueName: \"kubernetes.io/projected/248dfc75-12b3-4535-91d1-5c604ac06498-kube-api-access-kvc4m\") pod \"248dfc75-12b3-4535-91d1-5c604ac06498\" (UID: \"248dfc75-12b3-4535-91d1-5c604ac06498\") " Oct 31 05:17:06.626143 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:06.626060 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53f77b18-7ca9-4e11-869d-0a1aa66d2057-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53f77b18-7ca9-4e11-869d-0a1aa66d2057-plugins-config-volume\") pod \"53f77b18-7ca9-4e11-869d-0a1aa66d2057\" (UID: \"53f77b18-7ca9-4e11-869d-0a1aa66d2057\") " Oct 31 05:17:06.626462 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:06.626434 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/248dfc75-12b3-4535-91d1-5c604ac06498-plugins-config-volume" pod "248dfc75-12b3-4535-91d1-5c604ac06498" (UID: "248dfc75-12b3-4535-91d1-5c604ac06498"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:17:06.626616 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:06.626534 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53f77b18-7ca9-4e11-869d-0a1aa66d2057-plugins-config-volume" pod "53f77b18-7ca9-4e11-869d-0a1aa66d2057" (UID: "53f77b18-7ca9-4e11-869d-0a1aa66d2057"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:17:06.628344 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:06.628307 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/248dfc75-12b3-4535-91d1-5c604ac06498-kube-api-access-kvc4m" pod "248dfc75-12b3-4535-91d1-5c604ac06498" (UID: "248dfc75-12b3-4535-91d1-5c604ac06498"). InnerVolumeSpecName "kube-api-access-kvc4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:17:06.628428 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:06.628379 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53f77b18-7ca9-4e11-869d-0a1aa66d2057-kube-api-access-4rcqb" pod "53f77b18-7ca9-4e11-869d-0a1aa66d2057" (UID: "53f77b18-7ca9-4e11-869d-0a1aa66d2057"). InnerVolumeSpecName "kube-api-access-4rcqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:17:06.727667 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:06.727622 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53f77b18-7ca9-4e11-869d-0a1aa66d2057-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:06.727667 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:06.727659 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rcqb\" (UniqueName: \"kubernetes.io/projected/53f77b18-7ca9-4e11-869d-0a1aa66d2057-kube-api-access-4rcqb\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:06.727667 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:06.727669 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/248dfc75-12b3-4535-91d1-5c604ac06498-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:06.727667 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:06.727678 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvc4m\" (UniqueName: \"kubernetes.io/projected/248dfc75-12b3-4535-91d1-5c604ac06498-kube-api-access-kvc4m\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:06.757327 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:06.757274 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b2a372-dfc6bb5c9-2t4p7" podUID="3bc5984a-022f-462f-aaf3-a421eb9ee119" containerName="payload-processing" containerID="cri-o://0e33685bac1afa3700ed3f474486c1b55908d5ae440022eecb5e280f68a6c7e5" gracePeriod=25 Oct 31 05:17:07.001715 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:07.001620 5019 generic.go:356] "Generic (PLEG): container finished" podID="248dfc75-12b3-4535-91d1-5c604ac06498" containerID="f429c47c34413b5a965793d294fb185d27ba0c31a7f67323c08dca502c530375" exitCode=0 Oct 31 05:17:07.001715 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:07.001666 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-338f2e-57f7757bf-mchhm" Oct 31 05:17:07.001715 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:07.001703 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-338f2e-57f7757bf-mchhm" event={"ID":"248dfc75-12b3-4535-91d1-5c604ac06498","Type":"ContainerDied","Data":"f429c47c34413b5a965793d294fb185d27ba0c31a7f67323c08dca502c530375"} Oct 31 05:17:07.002077 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:07.001757 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-338f2e-57f7757bf-mchhm" event={"ID":"248dfc75-12b3-4535-91d1-5c604ac06498","Type":"ContainerDied","Data":"82e6f10a302dede9d175fb383eb33f97ceb1f3eb2c5460045c832f42585aa944"} Oct 31 05:17:07.002077 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:07.001779 5019 scope.go:122] "RemoveContainer" containerID="f429c47c34413b5a965793d294fb185d27ba0c31a7f67323c08dca502c530375" Oct 31 05:17:07.003425 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:07.003402 5019 generic.go:356] "Generic (PLEG): container finished" podID="53f77b18-7ca9-4e11-869d-0a1aa66d2057" containerID="7badcff5803238f686bc30f63db2530bff481b08d6668a94ae0a2623a2df8861" exitCode=0 Oct 31 05:17:07.003565 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:07.003450 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-338f2e-6ccbc66d75-htxcd" Oct 31 05:17:07.003565 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:07.003490 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-338f2e-6ccbc66d75-htxcd" event={"ID":"53f77b18-7ca9-4e11-869d-0a1aa66d2057","Type":"ContainerDied","Data":"7badcff5803238f686bc30f63db2530bff481b08d6668a94ae0a2623a2df8861"} Oct 31 05:17:07.003689 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:07.003556 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-338f2e-6ccbc66d75-htxcd" event={"ID":"53f77b18-7ca9-4e11-869d-0a1aa66d2057","Type":"ContainerDied","Data":"61497f6d2b1b41e28a1517839ec71e4970fa9fe59b23de3f45ec26459b20b266"} Oct 31 05:17:07.014469 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:07.014451 5019 scope.go:122] "RemoveContainer" containerID="f429c47c34413b5a965793d294fb185d27ba0c31a7f67323c08dca502c530375" Oct 31 05:17:07.014843 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:17:07.014820 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f429c47c34413b5a965793d294fb185d27ba0c31a7f67323c08dca502c530375\": container with ID starting with f429c47c34413b5a965793d294fb185d27ba0c31a7f67323c08dca502c530375 not found: ID does not exist" containerID="f429c47c34413b5a965793d294fb185d27ba0c31a7f67323c08dca502c530375" Oct 31 05:17:07.014926 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:07.014854 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f429c47c34413b5a965793d294fb185d27ba0c31a7f67323c08dca502c530375"} err="failed to get container status \"f429c47c34413b5a965793d294fb185d27ba0c31a7f67323c08dca502c530375\": rpc error: code = NotFound desc = could not find container \"f429c47c34413b5a965793d294fb185d27ba0c31a7f67323c08dca502c530375\": container with ID starting with f429c47c34413b5a965793d294fb185d27ba0c31a7f67323c08dca502c530375 not found: ID does not exist" Oct 31 05:17:07.014926 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:07.014873 5019 scope.go:122] "RemoveContainer" containerID="7badcff5803238f686bc30f63db2530bff481b08d6668a94ae0a2623a2df8861" Oct 31 05:17:07.026239 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:07.026216 5019 scope.go:122] "RemoveContainer" containerID="7badcff5803238f686bc30f63db2530bff481b08d6668a94ae0a2623a2df8861" Oct 31 05:17:07.026587 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:17:07.026566 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7badcff5803238f686bc30f63db2530bff481b08d6668a94ae0a2623a2df8861\": container with ID starting with 7badcff5803238f686bc30f63db2530bff481b08d6668a94ae0a2623a2df8861 not found: ID does not exist" containerID="7badcff5803238f686bc30f63db2530bff481b08d6668a94ae0a2623a2df8861" Oct 31 05:17:07.026662 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:07.026596 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7badcff5803238f686bc30f63db2530bff481b08d6668a94ae0a2623a2df8861"} err="failed to get container status \"7badcff5803238f686bc30f63db2530bff481b08d6668a94ae0a2623a2df8861\": rpc error: code = NotFound desc = could not find container \"7badcff5803238f686bc30f63db2530bff481b08d6668a94ae0a2623a2df8861\": container with ID starting with 7badcff5803238f686bc30f63db2530bff481b08d6668a94ae0a2623a2df8861 not found: ID does not exist" Oct 31 05:17:07.053944 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:07.053900 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-338f2e-6ccbc66d75-htxcd"] Oct 31 05:17:07.065830 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:07.065795 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-338f2e-6ccbc66d75-htxcd"] Oct 31 05:17:07.100318 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:07.100275 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-338f2e-57f7757bf-mchhm"] Oct 31 05:17:07.103502 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:07.103466 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-338f2e-57f7757bf-mchhm"] Oct 31 05:17:07.408085 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:07.408043 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="248dfc75-12b3-4535-91d1-5c604ac06498" path="/var/lib/kubelet/pods/248dfc75-12b3-4535-91d1-5c604ac06498/volumes" Oct 31 05:17:07.408636 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:07.408531 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53f77b18-7ca9-4e11-869d-0a1aa66d2057" path="/var/lib/kubelet/pods/53f77b18-7ca9-4e11-869d-0a1aa66d2057/volumes" Oct 31 05:17:12.955368 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:12.955283 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b2a372-dfc6bb5c9-2t4p7" podUID="3bc5984a-022f-462f-aaf3-a421eb9ee119" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:17:16.444763 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.444710 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-828488ca-5tbtz"] Oct 31 05:17:16.445212 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445200 5019 cpu_manager.go:414] "containerMap: removing container" podUID="b547bb0b-cae2-453b-aa56-4f37434a1ce8" containerName="istio-proxy" Oct 31 05:17:16.445264 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445216 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b547bb0b-cae2-453b-aa56-4f37434a1ce8" containerName="istio-proxy" Oct 31 05:17:16.445264 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445228 5019 cpu_manager.go:414] "containerMap: removing container" podUID="b547bb0b-cae2-453b-aa56-4f37434a1ce8" containerName="istio-proxy" Oct 31 05:17:16.445264 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445233 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b547bb0b-cae2-453b-aa56-4f37434a1ce8" containerName="istio-proxy" Oct 31 05:17:16.445264 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445242 5019 cpu_manager.go:414] "containerMap: removing container" podUID="91300bb2-7008-42f1-ba58-f7ab87e1d748" containerName="payload-pre-processing" Oct 31 05:17:16.445264 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445247 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91300bb2-7008-42f1-ba58-f7ab87e1d748" containerName="payload-pre-processing" Oct 31 05:17:16.445264 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445260 5019 cpu_manager.go:414] "containerMap: removing container" podUID="a4a19a46-de23-4984-894c-b14e0d223fab" containerName="payload-pre-processing" Oct 31 05:17:16.445264 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445265 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4a19a46-de23-4984-894c-b14e0d223fab" containerName="payload-pre-processing" Oct 31 05:17:16.445498 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445276 5019 cpu_manager.go:414] "containerMap: removing container" podUID="de1f94d9-8507-47a5-aec8-88c546112c7f" containerName="istio-proxy" Oct 31 05:17:16.445498 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445281 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de1f94d9-8507-47a5-aec8-88c546112c7f" containerName="istio-proxy" Oct 31 05:17:16.445498 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445289 5019 cpu_manager.go:414] "containerMap: removing container" podUID="16674b5b-1b4d-40d5-aa1d-704a9c56b54b" containerName="payload-pre-processing" Oct 31 05:17:16.445498 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445294 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16674b5b-1b4d-40d5-aa1d-704a9c56b54b" containerName="payload-pre-processing" Oct 31 05:17:16.445498 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445304 5019 cpu_manager.go:414] "containerMap: removing container" podUID="39eb6eb3-a2ae-4931-bab8-1e3e07e7d764" containerName="istio-proxy" Oct 31 05:17:16.445498 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445309 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39eb6eb3-a2ae-4931-bab8-1e3e07e7d764" containerName="istio-proxy" Oct 31 05:17:16.445498 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445317 5019 cpu_manager.go:414] "containerMap: removing container" podUID="a2c340cd-5e95-444e-af15-f94090647205" containerName="revoke-keys" Oct 31 05:17:16.445498 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445321 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2c340cd-5e95-444e-af15-f94090647205" containerName="revoke-keys" Oct 31 05:17:16.445498 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445343 5019 cpu_manager.go:414] "containerMap: removing container" podUID="ab2c1b1c-1f3f-43fb-aa9d-248870c31da1" containerName="payload-pre-processing" Oct 31 05:17:16.445498 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445348 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab2c1b1c-1f3f-43fb-aa9d-248870c31da1" containerName="payload-pre-processing" Oct 31 05:17:16.445498 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445356 5019 cpu_manager.go:414] "containerMap: removing container" podUID="b4ced3ab-41e4-4c54-93eb-95eddaae992d" containerName="main" Oct 31 05:17:16.445498 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445361 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4ced3ab-41e4-4c54-93eb-95eddaae992d" containerName="main" Oct 31 05:17:16.445498 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445369 5019 cpu_manager.go:414] "containerMap: removing container" podUID="42db3486-5bc2-40fc-b5cd-b32f718255b3" containerName="revoke-keys" Oct 31 05:17:16.445498 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445374 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42db3486-5bc2-40fc-b5cd-b32f718255b3" containerName="revoke-keys" Oct 31 05:17:16.445498 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445380 5019 cpu_manager.go:414] "containerMap: removing container" podUID="53f77b18-7ca9-4e11-869d-0a1aa66d2057" containerName="payload-pre-processing" Oct 31 05:17:16.445498 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445385 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53f77b18-7ca9-4e11-869d-0a1aa66d2057" containerName="payload-pre-processing" Oct 31 05:17:16.445498 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445394 5019 cpu_manager.go:414] "containerMap: removing container" podUID="2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09" containerName="istio-proxy" Oct 31 05:17:16.445498 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445398 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09" containerName="istio-proxy" Oct 31 05:17:16.445498 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445404 5019 cpu_manager.go:414] "containerMap: removing container" podUID="2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09" containerName="istio-proxy" Oct 31 05:17:16.445498 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445409 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09" containerName="istio-proxy" Oct 31 05:17:16.445498 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445419 5019 cpu_manager.go:414] "containerMap: removing container" podUID="17f31b2c-2988-45e1-92ae-e43ff492444a" containerName="istio-proxy" Oct 31 05:17:16.445498 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445423 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17f31b2c-2988-45e1-92ae-e43ff492444a" containerName="istio-proxy" Oct 31 05:17:16.445498 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445433 5019 cpu_manager.go:414] "containerMap: removing container" podUID="643433d8-418e-4102-8bc3-cd544932614a" containerName="main" Oct 31 05:17:16.445498 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445437 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="643433d8-418e-4102-8bc3-cd544932614a" containerName="main" Oct 31 05:17:16.445498 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445444 5019 cpu_manager.go:414] "containerMap: removing container" podUID="0b8d5e17-5809-46a1-b0da-d11c15832f1d" containerName="main" Oct 31 05:17:16.445498 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445448 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b8d5e17-5809-46a1-b0da-d11c15832f1d" containerName="main" Oct 31 05:17:16.445498 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445455 5019 cpu_manager.go:414] "containerMap: removing container" podUID="31f4b14c-49f7-45f4-936e-e1be3123640c" containerName="payload-processing" Oct 31 05:17:16.445498 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445459 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31f4b14c-49f7-45f4-936e-e1be3123640c" containerName="payload-processing" Oct 31 05:17:16.445498 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445466 5019 cpu_manager.go:414] "containerMap: removing container" podUID="248dfc75-12b3-4535-91d1-5c604ac06498" containerName="payload-processing" Oct 31 05:17:16.445498 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445471 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="248dfc75-12b3-4535-91d1-5c604ac06498" containerName="payload-processing" Oct 31 05:17:16.445498 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445477 5019 cpu_manager.go:414] "containerMap: removing container" podUID="17f31b2c-2988-45e1-92ae-e43ff492444a" containerName="istio-proxy" Oct 31 05:17:16.445498 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445481 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17f31b2c-2988-45e1-92ae-e43ff492444a" containerName="istio-proxy" Oct 31 05:17:16.445498 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445487 5019 cpu_manager.go:414] "containerMap: removing container" podUID="46aae214-de7a-4736-9aad-3b5499aabd6f" containerName="payload-processing" Oct 31 05:17:16.445498 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445492 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46aae214-de7a-4736-9aad-3b5499aabd6f" containerName="payload-processing" Oct 31 05:17:16.446556 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445557 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09" containerName="istio-proxy" Oct 31 05:17:16.446556 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445570 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="17f31b2c-2988-45e1-92ae-e43ff492444a" containerName="istio-proxy" Oct 31 05:17:16.446556 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445577 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="643433d8-418e-4102-8bc3-cd544932614a" containerName="main" Oct 31 05:17:16.446556 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445583 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b8d5e17-5809-46a1-b0da-d11c15832f1d" containerName="main" Oct 31 05:17:16.446556 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445590 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="de1f94d9-8507-47a5-aec8-88c546112c7f" containerName="istio-proxy" Oct 31 05:17:16.446556 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445595 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="31f4b14c-49f7-45f4-936e-e1be3123640c" containerName="payload-processing" Oct 31 05:17:16.446556 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445601 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="248dfc75-12b3-4535-91d1-5c604ac06498" containerName="payload-processing" Oct 31 05:17:16.446556 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445606 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="46aae214-de7a-4736-9aad-3b5499aabd6f" containerName="payload-processing" Oct 31 05:17:16.446556 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445612 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="b547bb0b-cae2-453b-aa56-4f37434a1ce8" containerName="istio-proxy" Oct 31 05:17:16.446556 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445617 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="91300bb2-7008-42f1-ba58-f7ab87e1d748" containerName="payload-pre-processing" Oct 31 05:17:16.446556 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445624 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4a19a46-de23-4984-894c-b14e0d223fab" containerName="payload-pre-processing" Oct 31 05:17:16.446556 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445629 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="de1f94d9-8507-47a5-aec8-88c546112c7f" containerName="istio-proxy" Oct 31 05:17:16.446556 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445634 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="16674b5b-1b4d-40d5-aa1d-704a9c56b54b" containerName="payload-pre-processing" Oct 31 05:17:16.446556 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445640 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="39eb6eb3-a2ae-4931-bab8-1e3e07e7d764" containerName="istio-proxy" Oct 31 05:17:16.446556 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445645 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2c340cd-5e95-444e-af15-f94090647205" containerName="revoke-keys" Oct 31 05:17:16.446556 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445650 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab2c1b1c-1f3f-43fb-aa9d-248870c31da1" containerName="payload-pre-processing" Oct 31 05:17:16.446556 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445655 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4ced3ab-41e4-4c54-93eb-95eddaae992d" containerName="main" Oct 31 05:17:16.446556 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445661 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="39eb6eb3-a2ae-4931-bab8-1e3e07e7d764" containerName="istio-proxy" Oct 31 05:17:16.446556 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445666 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="42db3486-5bc2-40fc-b5cd-b32f718255b3" containerName="revoke-keys" Oct 31 05:17:16.446556 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.445670 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="53f77b18-7ca9-4e11-869d-0a1aa66d2057" containerName="payload-pre-processing" Oct 31 05:17:16.450440 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.450413 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-828488ca-5tbtz" Oct 31 05:17:16.465069 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.465033 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:17:16.468207 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.468171 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-828488ca-5tbtz"] Oct 31 05:17:16.475531 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.475479 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:17:16.475721 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.475604 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 05:17:16.477080 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.477055 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:17:16.535192 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.535146 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxtwp\" (UniqueName: \"kubernetes.io/projected/4a209b1d-00cf-4969-83d4-6326fc84037f-kube-api-access-kxtwp\") pod \"maas-api-revoke-sub-828488ca-5tbtz\" (UID: \"4a209b1d-00cf-4969-83d4-6326fc84037f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-828488ca-5tbtz" Oct 31 05:17:16.535419 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.535212 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a209b1d-00cf-4969-83d4-6326fc84037f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-828488ca-5tbtz\" (UID: \"4a209b1d-00cf-4969-83d4-6326fc84037f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-828488ca-5tbtz" Oct 31 05:17:16.535419 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.535326 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a209b1d-00cf-4969-83d4-6326fc84037f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-828488ca-5tbtz\" (UID: \"4a209b1d-00cf-4969-83d4-6326fc84037f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-828488ca-5tbtz" Oct 31 05:17:16.636182 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.636129 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxtwp\" (UniqueName: \"kubernetes.io/projected/4a209b1d-00cf-4969-83d4-6326fc84037f-kube-api-access-kxtwp\") pod \"maas-api-revoke-sub-828488ca-5tbtz\" (UID: \"4a209b1d-00cf-4969-83d4-6326fc84037f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-828488ca-5tbtz" Oct 31 05:17:16.636430 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.636213 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a209b1d-00cf-4969-83d4-6326fc84037f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-828488ca-5tbtz\" (UID: \"4a209b1d-00cf-4969-83d4-6326fc84037f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-828488ca-5tbtz" Oct 31 05:17:16.636430 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.636251 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a209b1d-00cf-4969-83d4-6326fc84037f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-828488ca-5tbtz\" (UID: \"4a209b1d-00cf-4969-83d4-6326fc84037f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-828488ca-5tbtz" Oct 31 05:17:16.637014 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.636990 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a209b1d-00cf-4969-83d4-6326fc84037f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-828488ca-5tbtz\" (UID: \"4a209b1d-00cf-4969-83d4-6326fc84037f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-828488ca-5tbtz" Oct 31 05:17:16.638723 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.638699 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a209b1d-00cf-4969-83d4-6326fc84037f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-828488ca-5tbtz\" (UID: \"4a209b1d-00cf-4969-83d4-6326fc84037f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-828488ca-5tbtz" Oct 31 05:17:16.659702 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.659663 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxtwp\" (UniqueName: \"kubernetes.io/projected/4a209b1d-00cf-4969-83d4-6326fc84037f-kube-api-access-kxtwp\") pod \"maas-api-revoke-sub-828488ca-5tbtz\" (UID: \"4a209b1d-00cf-4969-83d4-6326fc84037f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-828488ca-5tbtz" Oct 31 05:17:16.765138 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:16.765093 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-828488ca-5tbtz" Oct 31 05:17:17.071453 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:17.071407 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-828488ca-5tbtz"] Oct 31 05:17:17.071832 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:17:17.071802 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a209b1d_00cf_4969_83d4_6326fc84037f.slice/crio-b58f1e1a5227ed3eaf05508982c3474ba7adbc99a98c0dc4c7132bfb2b20158d WatchSource:0}: Error finding container b58f1e1a5227ed3eaf05508982c3474ba7adbc99a98c0dc4c7132bfb2b20158d: Status 404 returned error can't find the container with id b58f1e1a5227ed3eaf05508982c3474ba7adbc99a98c0dc4c7132bfb2b20158d Oct 31 05:17:17.082584 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:17.082538 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-828488ca-5tbtz" event={"ID":"4a209b1d-00cf-4969-83d4-6326fc84037f","Type":"ContainerStarted","Data":"b58f1e1a5227ed3eaf05508982c3474ba7adbc99a98c0dc4c7132bfb2b20158d"} Oct 31 05:17:18.092369 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:18.092312 5019 generic.go:356] "Generic (PLEG): container finished" podID="4a209b1d-00cf-4969-83d4-6326fc84037f" containerID="f7bed4ebead583af337a6da70612c913771d4709dae8cc46fbc52699c060f8a1" exitCode=0 Oct 31 05:17:18.092369 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:18.092363 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-828488ca-5tbtz" event={"ID":"4a209b1d-00cf-4969-83d4-6326fc84037f","Type":"ContainerDied","Data":"f7bed4ebead583af337a6da70612c913771d4709dae8cc46fbc52699c060f8a1"} Oct 31 05:17:19.255226 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:19.255196 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-828488ca-5tbtz" Oct 31 05:17:19.365283 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:19.365162 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a209b1d-00cf-4969-83d4-6326fc84037f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a209b1d-00cf-4969-83d4-6326fc84037f-maas-api-openshift-ca\") pod \"4a209b1d-00cf-4969-83d4-6326fc84037f\" (UID: \"4a209b1d-00cf-4969-83d4-6326fc84037f\") " Oct 31 05:17:19.365283 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:19.365217 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a209b1d-00cf-4969-83d4-6326fc84037f-kube-api-access-kxtwp\" (UniqueName: \"kubernetes.io/projected/4a209b1d-00cf-4969-83d4-6326fc84037f-kube-api-access-kxtwp\") pod \"4a209b1d-00cf-4969-83d4-6326fc84037f\" (UID: \"4a209b1d-00cf-4969-83d4-6326fc84037f\") " Oct 31 05:17:19.365626 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:19.365321 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a209b1d-00cf-4969-83d4-6326fc84037f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a209b1d-00cf-4969-83d4-6326fc84037f-maas-api-xks-ca\") pod \"4a209b1d-00cf-4969-83d4-6326fc84037f\" (UID: \"4a209b1d-00cf-4969-83d4-6326fc84037f\") " Oct 31 05:17:19.365730 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:19.365703 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a209b1d-00cf-4969-83d4-6326fc84037f-maas-api-openshift-ca" pod "4a209b1d-00cf-4969-83d4-6326fc84037f" (UID: "4a209b1d-00cf-4969-83d4-6326fc84037f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:17:19.367647 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:19.367600 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a209b1d-00cf-4969-83d4-6326fc84037f-maas-api-xks-ca" pod "4a209b1d-00cf-4969-83d4-6326fc84037f" (UID: "4a209b1d-00cf-4969-83d4-6326fc84037f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:17:19.367647 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:19.367642 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a209b1d-00cf-4969-83d4-6326fc84037f-kube-api-access-kxtwp" pod "4a209b1d-00cf-4969-83d4-6326fc84037f" (UID: "4a209b1d-00cf-4969-83d4-6326fc84037f"). InnerVolumeSpecName "kube-api-access-kxtwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:17:19.466832 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:19.466788 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a209b1d-00cf-4969-83d4-6326fc84037f-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:19.466832 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:19.466824 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxtwp\" (UniqueName: \"kubernetes.io/projected/4a209b1d-00cf-4969-83d4-6326fc84037f-kube-api-access-kxtwp\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:19.466832 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:19.466839 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a209b1d-00cf-4969-83d4-6326fc84037f-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:20.108123 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:20.108084 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-828488ca-5tbtz" Oct 31 05:17:20.108123 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:20.108104 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-828488ca-5tbtz" event={"ID":"4a209b1d-00cf-4969-83d4-6326fc84037f","Type":"ContainerDied","Data":"b58f1e1a5227ed3eaf05508982c3474ba7adbc99a98c0dc4c7132bfb2b20158d"} Oct 31 05:17:20.108376 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:20.108159 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b58f1e1a5227ed3eaf05508982c3474ba7adbc99a98c0dc4c7132bfb2b20158d" Oct 31 05:17:20.574505 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:20.574458 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7560a85-qrxrs"] Oct 31 05:17:20.575003 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:20.574986 5019 cpu_manager.go:414] "containerMap: removing container" podUID="de1f94d9-8507-47a5-aec8-88c546112c7f" containerName="istio-proxy" Oct 31 05:17:20.575003 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:20.575004 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de1f94d9-8507-47a5-aec8-88c546112c7f" containerName="istio-proxy" Oct 31 05:17:20.575093 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:20.575018 5019 cpu_manager.go:414] "containerMap: removing container" podUID="39eb6eb3-a2ae-4931-bab8-1e3e07e7d764" containerName="istio-proxy" Oct 31 05:17:20.575093 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:20.575023 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39eb6eb3-a2ae-4931-bab8-1e3e07e7d764" containerName="istio-proxy" Oct 31 05:17:20.575093 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:20.575043 5019 cpu_manager.go:414] "containerMap: removing container" podUID="4a209b1d-00cf-4969-83d4-6326fc84037f" containerName="revoke-keys" Oct 31 05:17:20.575093 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:20.575048 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a209b1d-00cf-4969-83d4-6326fc84037f" containerName="revoke-keys" Oct 31 05:17:20.575248 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:20.575117 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a209b1d-00cf-4969-83d4-6326fc84037f" containerName="revoke-keys" Oct 31 05:17:20.575248 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:20.575129 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="17f31b2c-2988-45e1-92ae-e43ff492444a" containerName="istio-proxy" Oct 31 05:17:20.575248 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:20.575139 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="b547bb0b-cae2-453b-aa56-4f37434a1ce8" containerName="istio-proxy" Oct 31 05:17:20.575248 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:20.575151 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e2d0bd6-c4ff-4e2f-8b2c-379fde919d09" containerName="istio-proxy" Oct 31 05:17:20.579293 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:20.579268 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7560a85-qrxrs" Oct 31 05:17:20.594295 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:20.594259 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:17:20.594671 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:20.594646 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:17:20.594787 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:20.594735 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 05:17:20.594787 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:20.594757 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:17:20.598326 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:20.598289 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7560a85-qrxrs"] Oct 31 05:17:20.680509 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:20.680441 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e1ab956-2e52-4699-bbfa-57233477f537-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7560a85-qrxrs\" (UID: \"2e1ab956-2e52-4699-bbfa-57233477f537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7560a85-qrxrs" Oct 31 05:17:20.680858 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:20.680579 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nt4j\" (UniqueName: \"kubernetes.io/projected/2e1ab956-2e52-4699-bbfa-57233477f537-kube-api-access-4nt4j\") pod \"maas-api-revoke-sub-b7560a85-qrxrs\" (UID: \"2e1ab956-2e52-4699-bbfa-57233477f537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7560a85-qrxrs" Oct 31 05:17:20.680858 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:20.680666 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e1ab956-2e52-4699-bbfa-57233477f537-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7560a85-qrxrs\" (UID: \"2e1ab956-2e52-4699-bbfa-57233477f537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7560a85-qrxrs" Oct 31 05:17:20.781845 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:20.781787 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nt4j\" (UniqueName: \"kubernetes.io/projected/2e1ab956-2e52-4699-bbfa-57233477f537-kube-api-access-4nt4j\") pod \"maas-api-revoke-sub-b7560a85-qrxrs\" (UID: \"2e1ab956-2e52-4699-bbfa-57233477f537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7560a85-qrxrs" Oct 31 05:17:20.782073 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:20.781893 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e1ab956-2e52-4699-bbfa-57233477f537-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7560a85-qrxrs\" (UID: \"2e1ab956-2e52-4699-bbfa-57233477f537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7560a85-qrxrs" Oct 31 05:17:20.782073 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:20.781969 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e1ab956-2e52-4699-bbfa-57233477f537-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7560a85-qrxrs\" (UID: \"2e1ab956-2e52-4699-bbfa-57233477f537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7560a85-qrxrs" Oct 31 05:17:20.782709 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:20.782685 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e1ab956-2e52-4699-bbfa-57233477f537-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7560a85-qrxrs\" (UID: \"2e1ab956-2e52-4699-bbfa-57233477f537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7560a85-qrxrs" Oct 31 05:17:20.784441 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:20.784416 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e1ab956-2e52-4699-bbfa-57233477f537-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7560a85-qrxrs\" (UID: \"2e1ab956-2e52-4699-bbfa-57233477f537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7560a85-qrxrs" Oct 31 05:17:20.805553 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:20.805514 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nt4j\" (UniqueName: \"kubernetes.io/projected/2e1ab956-2e52-4699-bbfa-57233477f537-kube-api-access-4nt4j\") pod \"maas-api-revoke-sub-b7560a85-qrxrs\" (UID: \"2e1ab956-2e52-4699-bbfa-57233477f537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7560a85-qrxrs" Oct 31 05:17:20.893812 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:20.893702 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp"] Oct 31 05:17:20.894155 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:20.893835 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7560a85-qrxrs" Oct 31 05:17:20.901297 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:20.901256 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc"] Oct 31 05:17:21.215245 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:21.215209 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7560a85-qrxrs"] Oct 31 05:17:21.215848 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:17:21.215822 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e1ab956_2e52_4699_bbfa_57233477f537.slice/crio-c677a36b4c981b40ed08797e3cd45185158857a212c11f59d36ccd1067a2fe0b WatchSource:0}: Error finding container c677a36b4c981b40ed08797e3cd45185158857a212c11f59d36ccd1067a2fe0b: Status 404 returned error can't find the container with id c677a36b4c981b40ed08797e3cd45185158857a212c11f59d36ccd1067a2fe0b Oct 31 05:17:22.125486 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:22.125444 5019 generic.go:356] "Generic (PLEG): container finished" podID="2e1ab956-2e52-4699-bbfa-57233477f537" containerID="b3ebec688d49a5fa25b0ae608e8af2df2e36f41d656f97e7f457107616642b94" exitCode=0 Oct 31 05:17:22.125918 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:22.125529 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7560a85-qrxrs" event={"ID":"2e1ab956-2e52-4699-bbfa-57233477f537","Type":"ContainerDied","Data":"b3ebec688d49a5fa25b0ae608e8af2df2e36f41d656f97e7f457107616642b94"} Oct 31 05:17:22.125918 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:22.125583 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7560a85-qrxrs" event={"ID":"2e1ab956-2e52-4699-bbfa-57233477f537","Type":"ContainerStarted","Data":"c677a36b4c981b40ed08797e3cd45185158857a212c11f59d36ccd1067a2fe0b"} Oct 31 05:17:22.392996 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:17:22.392899 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:17:22.955132 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:22.955085 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b2a372-dfc6bb5c9-2t4p7" podUID="3bc5984a-022f-462f-aaf3-a421eb9ee119" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:17:23.292569 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:23.292531 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7560a85-qrxrs" Oct 31 05:17:23.414313 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:23.414274 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e1ab956-2e52-4699-bbfa-57233477f537-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e1ab956-2e52-4699-bbfa-57233477f537-maas-api-xks-ca\") pod \"2e1ab956-2e52-4699-bbfa-57233477f537\" (UID: \"2e1ab956-2e52-4699-bbfa-57233477f537\") " Oct 31 05:17:23.414533 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:23.414393 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e1ab956-2e52-4699-bbfa-57233477f537-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e1ab956-2e52-4699-bbfa-57233477f537-maas-api-openshift-ca\") pod \"2e1ab956-2e52-4699-bbfa-57233477f537\" (UID: \"2e1ab956-2e52-4699-bbfa-57233477f537\") " Oct 31 05:17:23.414533 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:23.414429 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e1ab956-2e52-4699-bbfa-57233477f537-kube-api-access-4nt4j\" (UniqueName: \"kubernetes.io/projected/2e1ab956-2e52-4699-bbfa-57233477f537-kube-api-access-4nt4j\") pod \"2e1ab956-2e52-4699-bbfa-57233477f537\" (UID: \"2e1ab956-2e52-4699-bbfa-57233477f537\") " Oct 31 05:17:23.414842 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:23.414805 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e1ab956-2e52-4699-bbfa-57233477f537-maas-api-openshift-ca" pod "2e1ab956-2e52-4699-bbfa-57233477f537" (UID: "2e1ab956-2e52-4699-bbfa-57233477f537"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:17:23.416801 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:23.416767 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e1ab956-2e52-4699-bbfa-57233477f537-maas-api-xks-ca" pod "2e1ab956-2e52-4699-bbfa-57233477f537" (UID: "2e1ab956-2e52-4699-bbfa-57233477f537"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:17:23.416912 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:23.416800 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e1ab956-2e52-4699-bbfa-57233477f537-kube-api-access-4nt4j" pod "2e1ab956-2e52-4699-bbfa-57233477f537" (UID: "2e1ab956-2e52-4699-bbfa-57233477f537"). InnerVolumeSpecName "kube-api-access-4nt4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:17:23.515713 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:23.515615 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e1ab956-2e52-4699-bbfa-57233477f537-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:23.515713 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:23.515660 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e1ab956-2e52-4699-bbfa-57233477f537-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:23.515713 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:23.515671 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nt4j\" (UniqueName: \"kubernetes.io/projected/2e1ab956-2e52-4699-bbfa-57233477f537-kube-api-access-4nt4j\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:24.140427 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:24.140392 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7560a85-qrxrs" Oct 31 05:17:24.140652 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:24.140395 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7560a85-qrxrs" event={"ID":"2e1ab956-2e52-4699-bbfa-57233477f537","Type":"ContainerDied","Data":"c677a36b4c981b40ed08797e3cd45185158857a212c11f59d36ccd1067a2fe0b"} Oct 31 05:17:24.140652 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:24.140550 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c677a36b4c981b40ed08797e3cd45185158857a212c11f59d36ccd1067a2fe0b" Oct 31 05:17:25.115626 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:25.115580 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q"] Oct 31 05:17:25.139565 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:25.139524 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l"] Oct 31 05:17:27.034114 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:27.034077 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b2a372-dfc6bb5c9-2t4p7" Oct 31 05:17:27.149448 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:27.149402 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bc5984a-022f-462f-aaf3-a421eb9ee119-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bc5984a-022f-462f-aaf3-a421eb9ee119-plugins-config-volume\") pod \"3bc5984a-022f-462f-aaf3-a421eb9ee119\" (UID: \"3bc5984a-022f-462f-aaf3-a421eb9ee119\") " Oct 31 05:17:27.149666 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:27.149544 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bc5984a-022f-462f-aaf3-a421eb9ee119-kube-api-access-x6dwt\" (UniqueName: \"kubernetes.io/projected/3bc5984a-022f-462f-aaf3-a421eb9ee119-kube-api-access-x6dwt\") pod \"3bc5984a-022f-462f-aaf3-a421eb9ee119\" (UID: \"3bc5984a-022f-462f-aaf3-a421eb9ee119\") " Oct 31 05:17:27.149848 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:27.149821 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bc5984a-022f-462f-aaf3-a421eb9ee119-plugins-config-volume" pod "3bc5984a-022f-462f-aaf3-a421eb9ee119" (UID: "3bc5984a-022f-462f-aaf3-a421eb9ee119"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:17:27.152226 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:27.152181 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bc5984a-022f-462f-aaf3-a421eb9ee119-kube-api-access-x6dwt" pod "3bc5984a-022f-462f-aaf3-a421eb9ee119" (UID: "3bc5984a-022f-462f-aaf3-a421eb9ee119"). InnerVolumeSpecName "kube-api-access-x6dwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:17:27.164756 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:27.164708 5019 generic.go:356] "Generic (PLEG): container finished" podID="3bc5984a-022f-462f-aaf3-a421eb9ee119" containerID="0e33685bac1afa3700ed3f474486c1b55908d5ae440022eecb5e280f68a6c7e5" exitCode=0 Oct 31 05:17:27.164756 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:27.164761 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b2a372-dfc6bb5c9-2t4p7" event={"ID":"3bc5984a-022f-462f-aaf3-a421eb9ee119","Type":"ContainerDied","Data":"0e33685bac1afa3700ed3f474486c1b55908d5ae440022eecb5e280f68a6c7e5"} Oct 31 05:17:27.165097 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:27.164765 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b2a372-dfc6bb5c9-2t4p7" Oct 31 05:17:27.165097 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:27.164810 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b2a372-dfc6bb5c9-2t4p7" event={"ID":"3bc5984a-022f-462f-aaf3-a421eb9ee119","Type":"ContainerDied","Data":"6b7a945be9e734a1f1448e25e51e33ee8579caad9c06e8212c0907cab0319ebf"} Oct 31 05:17:27.165097 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:27.164834 5019 scope.go:122] "RemoveContainer" containerID="0e33685bac1afa3700ed3f474486c1b55908d5ae440022eecb5e280f68a6c7e5" Oct 31 05:17:27.178129 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:27.178107 5019 scope.go:122] "RemoveContainer" containerID="0e33685bac1afa3700ed3f474486c1b55908d5ae440022eecb5e280f68a6c7e5" Oct 31 05:17:27.178529 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:17:27.178500 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e33685bac1afa3700ed3f474486c1b55908d5ae440022eecb5e280f68a6c7e5\": container with ID starting with 0e33685bac1afa3700ed3f474486c1b55908d5ae440022eecb5e280f68a6c7e5 not found: ID does not exist" containerID="0e33685bac1afa3700ed3f474486c1b55908d5ae440022eecb5e280f68a6c7e5" Oct 31 05:17:27.178622 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:27.178537 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e33685bac1afa3700ed3f474486c1b55908d5ae440022eecb5e280f68a6c7e5"} err="failed to get container status \"0e33685bac1afa3700ed3f474486c1b55908d5ae440022eecb5e280f68a6c7e5\": rpc error: code = NotFound desc = could not find container \"0e33685bac1afa3700ed3f474486c1b55908d5ae440022eecb5e280f68a6c7e5\": container with ID starting with 0e33685bac1afa3700ed3f474486c1b55908d5ae440022eecb5e280f68a6c7e5 not found: ID does not exist" Oct 31 05:17:27.220449 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:27.220403 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-b2a372-dfc6bb5c9-2t4p7"] Oct 31 05:17:27.225542 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:27.225504 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-b2a372-dfc6bb5c9-2t4p7"] Oct 31 05:17:27.250691 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:27.250645 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bc5984a-022f-462f-aaf3-a421eb9ee119-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:27.250691 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:27.250688 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6dwt\" (UniqueName: \"kubernetes.io/projected/3bc5984a-022f-462f-aaf3-a421eb9ee119-kube-api-access-x6dwt\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:27.408544 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:27.408439 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bc5984a-022f-462f-aaf3-a421eb9ee119" path="/var/lib/kubelet/pods/3bc5984a-022f-462f-aaf3-a421eb9ee119/volumes" Oct 31 05:17:27.686947 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:27.686849 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-4c2387-78c59569f5-pklf6"] Oct 31 05:17:27.718324 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:27.718270 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-4c2387-78fdcfbd75-2twf4"] Oct 31 05:17:28.168890 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:28.168853 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-784ca31a-nrq79"] Oct 31 05:17:28.169523 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:28.169507 5019 cpu_manager.go:414] "containerMap: removing container" podUID="3bc5984a-022f-462f-aaf3-a421eb9ee119" containerName="payload-processing" Oct 31 05:17:28.169577 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:28.169526 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bc5984a-022f-462f-aaf3-a421eb9ee119" containerName="payload-processing" Oct 31 05:17:28.169577 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:28.169557 5019 cpu_manager.go:414] "containerMap: removing container" podUID="2e1ab956-2e52-4699-bbfa-57233477f537" containerName="revoke-keys" Oct 31 05:17:28.169577 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:28.169565 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e1ab956-2e52-4699-bbfa-57233477f537" containerName="revoke-keys" Oct 31 05:17:28.169685 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:28.169677 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e1ab956-2e52-4699-bbfa-57233477f537" containerName="revoke-keys" Oct 31 05:17:28.169719 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:28.169704 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bc5984a-022f-462f-aaf3-a421eb9ee119" containerName="payload-processing" Oct 31 05:17:28.174889 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:28.174861 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-784ca31a-nrq79" Oct 31 05:17:28.190264 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:28.190231 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:17:28.190555 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:28.190250 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:17:28.190555 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:28.190454 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 05:17:28.190807 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:28.190511 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:17:28.192028 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:28.191995 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-784ca31a-nrq79"] Oct 31 05:17:28.260148 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:28.260103 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/263a72f1-a00d-4e12-baed-e9bff6a0788c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-784ca31a-nrq79\" (UID: \"263a72f1-a00d-4e12-baed-e9bff6a0788c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-784ca31a-nrq79" Oct 31 05:17:28.260358 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:28.260172 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/263a72f1-a00d-4e12-baed-e9bff6a0788c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-784ca31a-nrq79\" (UID: \"263a72f1-a00d-4e12-baed-e9bff6a0788c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-784ca31a-nrq79" Oct 31 05:17:28.260358 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:28.260272 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nx9h\" (UniqueName: \"kubernetes.io/projected/263a72f1-a00d-4e12-baed-e9bff6a0788c-kube-api-access-8nx9h\") pod \"maas-api-revoke-sub-784ca31a-nrq79\" (UID: \"263a72f1-a00d-4e12-baed-e9bff6a0788c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-784ca31a-nrq79" Oct 31 05:17:28.361079 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:28.361013 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nx9h\" (UniqueName: \"kubernetes.io/projected/263a72f1-a00d-4e12-baed-e9bff6a0788c-kube-api-access-8nx9h\") pod \"maas-api-revoke-sub-784ca31a-nrq79\" (UID: \"263a72f1-a00d-4e12-baed-e9bff6a0788c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-784ca31a-nrq79" Oct 31 05:17:28.361380 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:28.361246 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/263a72f1-a00d-4e12-baed-e9bff6a0788c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-784ca31a-nrq79\" (UID: \"263a72f1-a00d-4e12-baed-e9bff6a0788c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-784ca31a-nrq79" Oct 31 05:17:28.361380 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:28.361285 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/263a72f1-a00d-4e12-baed-e9bff6a0788c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-784ca31a-nrq79\" (UID: \"263a72f1-a00d-4e12-baed-e9bff6a0788c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-784ca31a-nrq79" Oct 31 05:17:28.362216 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:28.362185 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/263a72f1-a00d-4e12-baed-e9bff6a0788c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-784ca31a-nrq79\" (UID: \"263a72f1-a00d-4e12-baed-e9bff6a0788c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-784ca31a-nrq79" Oct 31 05:17:28.364242 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:28.364203 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/263a72f1-a00d-4e12-baed-e9bff6a0788c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-784ca31a-nrq79\" (UID: \"263a72f1-a00d-4e12-baed-e9bff6a0788c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-784ca31a-nrq79" Oct 31 05:17:28.388137 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:28.388101 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nx9h\" (UniqueName: \"kubernetes.io/projected/263a72f1-a00d-4e12-baed-e9bff6a0788c-kube-api-access-8nx9h\") pod \"maas-api-revoke-sub-784ca31a-nrq79\" (UID: \"263a72f1-a00d-4e12-baed-e9bff6a0788c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-784ca31a-nrq79" Oct 31 05:17:28.490053 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:28.490009 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-784ca31a-nrq79" Oct 31 05:17:28.792675 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:28.792637 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-784ca31a-nrq79"] Oct 31 05:17:28.794722 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:17:28.794674 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod263a72f1_a00d_4e12_baed_e9bff6a0788c.slice/crio-476e18f402d1e75612d50fcec2d8e41d670e49159f43efe21448178b08487b20 WatchSource:0}: Error finding container 476e18f402d1e75612d50fcec2d8e41d670e49159f43efe21448178b08487b20: Status 404 returned error can't find the container with id 476e18f402d1e75612d50fcec2d8e41d670e49159f43efe21448178b08487b20 Oct 31 05:17:29.182062 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:29.182010 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-784ca31a-nrq79" event={"ID":"263a72f1-a00d-4e12-baed-e9bff6a0788c","Type":"ContainerStarted","Data":"d3e702c64d7136bb34bd11248abe70a873750cf183bff1773f931f2889acccef"} Oct 31 05:17:29.182651 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:29.182080 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-784ca31a-nrq79" event={"ID":"263a72f1-a00d-4e12-baed-e9bff6a0788c","Type":"ContainerStarted","Data":"476e18f402d1e75612d50fcec2d8e41d670e49159f43efe21448178b08487b20"} Oct 31 05:17:29.224143 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:29.224070 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-784ca31a-nrq79" podStartSLOduration=1.22404998 podStartE2EDuration="1.22404998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:17:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:17:29.221197429 +0000 UTC m=+4403.977535468" watchObservedRunningTime="2026-10-31 05:17:29.22404998 +0000 UTC m=+4403.980388019" Oct 31 05:17:30.190652 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:30.190609 5019 generic.go:356] "Generic (PLEG): container finished" podID="263a72f1-a00d-4e12-baed-e9bff6a0788c" containerID="d3e702c64d7136bb34bd11248abe70a873750cf183bff1773f931f2889acccef" exitCode=0 Oct 31 05:17:30.191094 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:30.190693 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-784ca31a-nrq79" event={"ID":"263a72f1-a00d-4e12-baed-e9bff6a0788c","Type":"ContainerDied","Data":"d3e702c64d7136bb34bd11248abe70a873750cf183bff1773f931f2889acccef"} Oct 31 05:17:31.033191 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:31.033147 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-147fcf85-zdc8l"] Oct 31 05:17:31.044884 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:31.044839 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-147fcf85-zdc8l"] Oct 31 05:17:31.364559 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:31.364524 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-784ca31a-nrq79" Oct 31 05:17:31.408372 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:31.408314 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faf366d6-a221-44f6-b362-0630b46fa956" path="/var/lib/kubelet/pods/faf366d6-a221-44f6-b362-0630b46fa956/volumes" Oct 31 05:17:31.498550 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:31.498506 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/263a72f1-a00d-4e12-baed-e9bff6a0788c-kube-api-access-8nx9h\" (UniqueName: \"kubernetes.io/projected/263a72f1-a00d-4e12-baed-e9bff6a0788c-kube-api-access-8nx9h\") pod \"263a72f1-a00d-4e12-baed-e9bff6a0788c\" (UID: \"263a72f1-a00d-4e12-baed-e9bff6a0788c\") " Oct 31 05:17:31.498777 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:31.498590 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/263a72f1-a00d-4e12-baed-e9bff6a0788c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/263a72f1-a00d-4e12-baed-e9bff6a0788c-maas-api-openshift-ca\") pod \"263a72f1-a00d-4e12-baed-e9bff6a0788c\" (UID: \"263a72f1-a00d-4e12-baed-e9bff6a0788c\") " Oct 31 05:17:31.498777 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:31.498686 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/263a72f1-a00d-4e12-baed-e9bff6a0788c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/263a72f1-a00d-4e12-baed-e9bff6a0788c-maas-api-xks-ca\") pod \"263a72f1-a00d-4e12-baed-e9bff6a0788c\" (UID: \"263a72f1-a00d-4e12-baed-e9bff6a0788c\") " Oct 31 05:17:31.499103 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:31.499057 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/263a72f1-a00d-4e12-baed-e9bff6a0788c-maas-api-openshift-ca" pod "263a72f1-a00d-4e12-baed-e9bff6a0788c" (UID: "263a72f1-a00d-4e12-baed-e9bff6a0788c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:17:31.499267 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:31.499245 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/263a72f1-a00d-4e12-baed-e9bff6a0788c-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:31.501142 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:31.501100 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/263a72f1-a00d-4e12-baed-e9bff6a0788c-kube-api-access-8nx9h" pod "263a72f1-a00d-4e12-baed-e9bff6a0788c" (UID: "263a72f1-a00d-4e12-baed-e9bff6a0788c"). InnerVolumeSpecName "kube-api-access-8nx9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:17:31.501142 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:31.501137 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/263a72f1-a00d-4e12-baed-e9bff6a0788c-maas-api-xks-ca" pod "263a72f1-a00d-4e12-baed-e9bff6a0788c" (UID: "263a72f1-a00d-4e12-baed-e9bff6a0788c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:17:31.600488 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:31.600389 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nx9h\" (UniqueName: \"kubernetes.io/projected/263a72f1-a00d-4e12-baed-e9bff6a0788c-kube-api-access-8nx9h\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:31.600488 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:31.600436 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/263a72f1-a00d-4e12-baed-e9bff6a0788c-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:32.210243 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:32.210201 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-784ca31a-nrq79" Oct 31 05:17:32.210243 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:32.210217 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-784ca31a-nrq79" event={"ID":"263a72f1-a00d-4e12-baed-e9bff6a0788c","Type":"ContainerDied","Data":"476e18f402d1e75612d50fcec2d8e41d670e49159f43efe21448178b08487b20"} Oct 31 05:17:32.210517 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:32.210286 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="476e18f402d1e75612d50fcec2d8e41d670e49159f43efe21448178b08487b20" Oct 31 05:17:32.688344 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:32.688208 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-4c2387-78c59569f5-pklf6" podUID="c60c1b8f-b960-4a87-9382-1dfc846ca8b5" containerName="payload-processing" containerID="cri-o://92ed61316fe2122ecfbe33607c347f05b32a297ce2df922340397c66b31e8700" gracePeriod=25 Oct 31 05:17:32.719347 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:32.719266 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4c2387-78fdcfbd75-2twf4" podUID="af4b877c-fb8f-48be-917c-a3e99fc1eb7f" containerName="payload-pre-processing" containerID="cri-o://db954a6e84eaaf447a3bdcbe78f773fa99e31994c552c3b0d7e257bc561d8c0e" gracePeriod=25 Oct 31 05:17:32.858136 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:32.858087 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fea8164c-bfqxm"] Oct 31 05:17:32.858606 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:32.858591 5019 cpu_manager.go:414] "containerMap: removing container" podUID="263a72f1-a00d-4e12-baed-e9bff6a0788c" containerName="revoke-keys" Oct 31 05:17:32.858665 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:32.858607 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="263a72f1-a00d-4e12-baed-e9bff6a0788c" containerName="revoke-keys" Oct 31 05:17:32.858718 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:32.858709 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="263a72f1-a00d-4e12-baed-e9bff6a0788c" containerName="revoke-keys" Oct 31 05:17:32.862574 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:32.862545 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea8164c-bfqxm" Oct 31 05:17:32.891278 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:32.891214 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:17:32.891278 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:32.891214 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:17:32.891278 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:32.891252 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:17:32.891604 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:32.891288 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 05:17:32.895884 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:32.895824 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fea8164c-bfqxm"] Oct 31 05:17:33.017146 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:33.017098 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gqhn\" (UniqueName: \"kubernetes.io/projected/247374b7-c1a3-4eae-bba4-76a070eed47c-kube-api-access-2gqhn\") pod \"maas-api-revoke-sub-fea8164c-bfqxm\" (UID: \"247374b7-c1a3-4eae-bba4-76a070eed47c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea8164c-bfqxm" Oct 31 05:17:33.017363 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:33.017230 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/247374b7-c1a3-4eae-bba4-76a070eed47c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fea8164c-bfqxm\" (UID: \"247374b7-c1a3-4eae-bba4-76a070eed47c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea8164c-bfqxm" Oct 31 05:17:33.017474 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:33.017444 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/247374b7-c1a3-4eae-bba4-76a070eed47c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fea8164c-bfqxm\" (UID: \"247374b7-c1a3-4eae-bba4-76a070eed47c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea8164c-bfqxm" Oct 31 05:17:33.118443 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:33.118386 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/247374b7-c1a3-4eae-bba4-76a070eed47c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fea8164c-bfqxm\" (UID: \"247374b7-c1a3-4eae-bba4-76a070eed47c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea8164c-bfqxm" Oct 31 05:17:33.118683 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:33.118486 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/247374b7-c1a3-4eae-bba4-76a070eed47c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fea8164c-bfqxm\" (UID: \"247374b7-c1a3-4eae-bba4-76a070eed47c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea8164c-bfqxm" Oct 31 05:17:33.118683 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:33.118540 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gqhn\" (UniqueName: \"kubernetes.io/projected/247374b7-c1a3-4eae-bba4-76a070eed47c-kube-api-access-2gqhn\") pod \"maas-api-revoke-sub-fea8164c-bfqxm\" (UID: \"247374b7-c1a3-4eae-bba4-76a070eed47c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea8164c-bfqxm" Oct 31 05:17:33.119131 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:33.119101 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/247374b7-c1a3-4eae-bba4-76a070eed47c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fea8164c-bfqxm\" (UID: \"247374b7-c1a3-4eae-bba4-76a070eed47c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea8164c-bfqxm" Oct 31 05:17:33.121123 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:33.121088 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/247374b7-c1a3-4eae-bba4-76a070eed47c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fea8164c-bfqxm\" (UID: \"247374b7-c1a3-4eae-bba4-76a070eed47c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea8164c-bfqxm" Oct 31 05:17:33.142577 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:33.142532 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gqhn\" (UniqueName: \"kubernetes.io/projected/247374b7-c1a3-4eae-bba4-76a070eed47c-kube-api-access-2gqhn\") pod \"maas-api-revoke-sub-fea8164c-bfqxm\" (UID: \"247374b7-c1a3-4eae-bba4-76a070eed47c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea8164c-bfqxm" Oct 31 05:17:33.181400 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:33.181320 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea8164c-bfqxm" Oct 31 05:17:33.502564 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:33.502527 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fea8164c-bfqxm"] Oct 31 05:17:33.503794 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:17:33.503761 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod247374b7_c1a3_4eae_bba4_76a070eed47c.slice/crio-4a89a5413e6de801607cde978deb3e36ebee5bcfc155f176b0d2ce49335492d8 WatchSource:0}: Error finding container 4a89a5413e6de801607cde978deb3e36ebee5bcfc155f176b0d2ce49335492d8: Status 404 returned error can't find the container with id 4a89a5413e6de801607cde978deb3e36ebee5bcfc155f176b0d2ce49335492d8 Oct 31 05:17:34.231368 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:34.231308 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea8164c-bfqxm" event={"ID":"247374b7-c1a3-4eae-bba4-76a070eed47c","Type":"ContainerStarted","Data":"51ca0db5dff716a825c66991249c0ce4ee8b6b498297f34bb60032fcb0d9f45a"} Oct 31 05:17:34.231945 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:34.231387 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea8164c-bfqxm" event={"ID":"247374b7-c1a3-4eae-bba4-76a070eed47c","Type":"ContainerStarted","Data":"4a89a5413e6de801607cde978deb3e36ebee5bcfc155f176b0d2ce49335492d8"} Oct 31 05:17:34.274759 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:34.274697 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea8164c-bfqxm" podStartSLOduration=2.274679918 podStartE2EDuration="2.274679918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:17:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:17:34.272478082 +0000 UTC m=+4409.028816125" watchObservedRunningTime="2026-10-31 05:17:34.274679918 +0000 UTC m=+4409.031017957" Oct 31 05:17:35.240461 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:35.240419 5019 generic.go:356] "Generic (PLEG): container finished" podID="247374b7-c1a3-4eae-bba4-76a070eed47c" containerID="51ca0db5dff716a825c66991249c0ce4ee8b6b498297f34bb60032fcb0d9f45a" exitCode=0 Oct 31 05:17:35.240895 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:35.240506 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea8164c-bfqxm" event={"ID":"247374b7-c1a3-4eae-bba4-76a070eed47c","Type":"ContainerDied","Data":"51ca0db5dff716a825c66991249c0ce4ee8b6b498297f34bb60032fcb0d9f45a"} Oct 31 05:17:35.925472 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:35.925408 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerName="main" containerID="cri-o://f0acbc3f06de2463824292c21ced776771ddc11152124a00db1b005a372a6eaf" gracePeriod=45 Oct 31 05:17:35.928379 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:35.928298 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" containerID="cri-o://e80ded11cb307253a5e723b5b6ec6c1b75e53ecd91dc9683b682c4b229d361b3" gracePeriod=45 Oct 31 05:17:36.215610 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.215577 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" Oct 31 05:17:36.231463 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.231428 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" Oct 31 05:17:36.251304 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.251268 5019 generic.go:356] "Generic (PLEG): container finished" podID="1f81681e-182b-4ed4-89bd-918370538f86" containerID="e80ded11cb307253a5e723b5b6ec6c1b75e53ecd91dc9683b682c4b229d361b3" exitCode=0 Oct 31 05:17:36.251855 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.251353 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" Oct 31 05:17:36.251855 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.251362 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" event={"ID":"1f81681e-182b-4ed4-89bd-918370538f86","Type":"ContainerDied","Data":"e80ded11cb307253a5e723b5b6ec6c1b75e53ecd91dc9683b682c4b229d361b3"} Oct 31 05:17:36.251855 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.251412 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc" event={"ID":"1f81681e-182b-4ed4-89bd-918370538f86","Type":"ContainerDied","Data":"0e770de1bad92768fa2a98f3eceb0a67762f75490c2e3244fc808963295a323c"} Oct 31 05:17:36.251855 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.251438 5019 scope.go:122] "RemoveContainer" containerID="e80ded11cb307253a5e723b5b6ec6c1b75e53ecd91dc9683b682c4b229d361b3" Oct 31 05:17:36.253198 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.253169 5019 generic.go:356] "Generic (PLEG): container finished" podID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerID="f0acbc3f06de2463824292c21ced776771ddc11152124a00db1b005a372a6eaf" exitCode=0 Oct 31 05:17:36.253394 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.253225 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" Oct 31 05:17:36.253394 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.253255 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" event={"ID":"157a108f-36ab-44a7-bd7c-8131cb46c7e0","Type":"ContainerDied","Data":"f0acbc3f06de2463824292c21ced776771ddc11152124a00db1b005a372a6eaf"} Oct 31 05:17:36.253394 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.253296 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp" event={"ID":"157a108f-36ab-44a7-bd7c-8131cb46c7e0","Type":"ContainerDied","Data":"93063657b647d768fdfe399c1413a4c6bb1e4542525d9cebfa1b5ff47f5edb2a"} Oct 31 05:17:36.268449 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.260393 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f81681e-182b-4ed4-89bd-918370538f86-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f81681e-182b-4ed4-89bd-918370538f86-dshm\") pod \"1f81681e-182b-4ed4-89bd-918370538f86\" (UID: \"1f81681e-182b-4ed4-89bd-918370538f86\") " Oct 31 05:17:36.268449 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.260450 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f81681e-182b-4ed4-89bd-918370538f86-home\" (UniqueName: \"kubernetes.io/empty-dir/1f81681e-182b-4ed4-89bd-918370538f86-home\") pod \"1f81681e-182b-4ed4-89bd-918370538f86\" (UID: \"1f81681e-182b-4ed4-89bd-918370538f86\") " Oct 31 05:17:36.268449 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.260494 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/157a108f-36ab-44a7-bd7c-8131cb46c7e0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/157a108f-36ab-44a7-bd7c-8131cb46c7e0-tmp-dir\") pod \"157a108f-36ab-44a7-bd7c-8131cb46c7e0\" (UID: \"157a108f-36ab-44a7-bd7c-8131cb46c7e0\") " Oct 31 05:17:36.268449 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.260533 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f81681e-182b-4ed4-89bd-918370538f86-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f81681e-182b-4ed4-89bd-918370538f86-model-cache\") pod \"1f81681e-182b-4ed4-89bd-918370538f86\" (UID: \"1f81681e-182b-4ed4-89bd-918370538f86\") " Oct 31 05:17:36.268449 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.260577 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f81681e-182b-4ed4-89bd-918370538f86-tls-certs\" (UniqueName: \"kubernetes.io/secret/1f81681e-182b-4ed4-89bd-918370538f86-tls-certs\") pod \"1f81681e-182b-4ed4-89bd-918370538f86\" (UID: \"1f81681e-182b-4ed4-89bd-918370538f86\") " Oct 31 05:17:36.268449 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.260641 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f81681e-182b-4ed4-89bd-918370538f86-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f81681e-182b-4ed4-89bd-918370538f86-tmp-dir\") pod \"1f81681e-182b-4ed4-89bd-918370538f86\" (UID: \"1f81681e-182b-4ed4-89bd-918370538f86\") " Oct 31 05:17:36.268449 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.260672 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/157a108f-36ab-44a7-bd7c-8131cb46c7e0-tls-certs\" (UniqueName: \"kubernetes.io/secret/157a108f-36ab-44a7-bd7c-8131cb46c7e0-tls-certs\") pod \"157a108f-36ab-44a7-bd7c-8131cb46c7e0\" (UID: \"157a108f-36ab-44a7-bd7c-8131cb46c7e0\") " Oct 31 05:17:36.268449 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.260728 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f81681e-182b-4ed4-89bd-918370538f86-kube-api-access-p8swp\" (UniqueName: \"kubernetes.io/projected/1f81681e-182b-4ed4-89bd-918370538f86-kube-api-access-p8swp\") pod \"1f81681e-182b-4ed4-89bd-918370538f86\" (UID: \"1f81681e-182b-4ed4-89bd-918370538f86\") " Oct 31 05:17:36.268449 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.260761 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/157a108f-36ab-44a7-bd7c-8131cb46c7e0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/157a108f-36ab-44a7-bd7c-8131cb46c7e0-model-cache\") pod \"157a108f-36ab-44a7-bd7c-8131cb46c7e0\" (UID: \"157a108f-36ab-44a7-bd7c-8131cb46c7e0\") " Oct 31 05:17:36.268449 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.260793 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/157a108f-36ab-44a7-bd7c-8131cb46c7e0-kube-api-access-rb5r7\" (UniqueName: \"kubernetes.io/projected/157a108f-36ab-44a7-bd7c-8131cb46c7e0-kube-api-access-rb5r7\") pod \"157a108f-36ab-44a7-bd7c-8131cb46c7e0\" (UID: \"157a108f-36ab-44a7-bd7c-8131cb46c7e0\") " Oct 31 05:17:36.268449 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.260852 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/157a108f-36ab-44a7-bd7c-8131cb46c7e0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/157a108f-36ab-44a7-bd7c-8131cb46c7e0-dshm\") pod \"157a108f-36ab-44a7-bd7c-8131cb46c7e0\" (UID: \"157a108f-36ab-44a7-bd7c-8131cb46c7e0\") " Oct 31 05:17:36.268449 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.260946 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/157a108f-36ab-44a7-bd7c-8131cb46c7e0-home\" (UniqueName: \"kubernetes.io/empty-dir/157a108f-36ab-44a7-bd7c-8131cb46c7e0-home\") pod \"157a108f-36ab-44a7-bd7c-8131cb46c7e0\" (UID: \"157a108f-36ab-44a7-bd7c-8131cb46c7e0\") " Oct 31 05:17:36.268449 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.261650 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f81681e-182b-4ed4-89bd-918370538f86-model-cache" pod "1f81681e-182b-4ed4-89bd-918370538f86" (UID: "1f81681e-182b-4ed4-89bd-918370538f86"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:17:36.268449 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.261806 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/157a108f-36ab-44a7-bd7c-8131cb46c7e0-model-cache" pod "157a108f-36ab-44a7-bd7c-8131cb46c7e0" (UID: "157a108f-36ab-44a7-bd7c-8131cb46c7e0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:17:36.268449 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.262087 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f81681e-182b-4ed4-89bd-918370538f86-home" pod "1f81681e-182b-4ed4-89bd-918370538f86" (UID: "1f81681e-182b-4ed4-89bd-918370538f86"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:17:36.268449 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.262157 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/157a108f-36ab-44a7-bd7c-8131cb46c7e0-tmp-dir" pod "157a108f-36ab-44a7-bd7c-8131cb46c7e0" (UID: "157a108f-36ab-44a7-bd7c-8131cb46c7e0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:17:36.268449 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.262399 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f81681e-182b-4ed4-89bd-918370538f86-tmp-dir" pod "1f81681e-182b-4ed4-89bd-918370538f86" (UID: "1f81681e-182b-4ed4-89bd-918370538f86"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:17:36.268449 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.262449 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/157a108f-36ab-44a7-bd7c-8131cb46c7e0-home" pod "157a108f-36ab-44a7-bd7c-8131cb46c7e0" (UID: "157a108f-36ab-44a7-bd7c-8131cb46c7e0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:17:36.268449 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.266642 5019 scope.go:122] "RemoveContainer" containerID="e80ded11cb307253a5e723b5b6ec6c1b75e53ecd91dc9683b682c4b229d361b3" Oct 31 05:17:36.268449 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:17:36.267125 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e80ded11cb307253a5e723b5b6ec6c1b75e53ecd91dc9683b682c4b229d361b3\": container with ID starting with e80ded11cb307253a5e723b5b6ec6c1b75e53ecd91dc9683b682c4b229d361b3 not found: ID does not exist" containerID="e80ded11cb307253a5e723b5b6ec6c1b75e53ecd91dc9683b682c4b229d361b3" Oct 31 05:17:36.268449 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.267166 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e80ded11cb307253a5e723b5b6ec6c1b75e53ecd91dc9683b682c4b229d361b3"} err="failed to get container status \"e80ded11cb307253a5e723b5b6ec6c1b75e53ecd91dc9683b682c4b229d361b3\": rpc error: code = NotFound desc = could not find container \"e80ded11cb307253a5e723b5b6ec6c1b75e53ecd91dc9683b682c4b229d361b3\": container with ID starting with e80ded11cb307253a5e723b5b6ec6c1b75e53ecd91dc9683b682c4b229d361b3 not found: ID does not exist" Oct 31 05:17:36.268449 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.267194 5019 scope.go:122] "RemoveContainer" containerID="f0acbc3f06de2463824292c21ced776771ddc11152124a00db1b005a372a6eaf" Oct 31 05:17:36.268449 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.267996 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/157a108f-36ab-44a7-bd7c-8131cb46c7e0-tls-certs" pod "157a108f-36ab-44a7-bd7c-8131cb46c7e0" (UID: "157a108f-36ab-44a7-bd7c-8131cb46c7e0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:17:36.270574 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.270541 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f81681e-182b-4ed4-89bd-918370538f86-dshm" pod "1f81681e-182b-4ed4-89bd-918370538f86" (UID: "1f81681e-182b-4ed4-89bd-918370538f86"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:17:36.270713 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.270620 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/157a108f-36ab-44a7-bd7c-8131cb46c7e0-dshm" pod "157a108f-36ab-44a7-bd7c-8131cb46c7e0" (UID: "157a108f-36ab-44a7-bd7c-8131cb46c7e0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:17:36.270993 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.270971 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f81681e-182b-4ed4-89bd-918370538f86-kube-api-access-p8swp" pod "1f81681e-182b-4ed4-89bd-918370538f86" (UID: "1f81681e-182b-4ed4-89bd-918370538f86"). InnerVolumeSpecName "kube-api-access-p8swp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:17:36.273079 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.273040 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/157a108f-36ab-44a7-bd7c-8131cb46c7e0-kube-api-access-rb5r7" pod "157a108f-36ab-44a7-bd7c-8131cb46c7e0" (UID: "157a108f-36ab-44a7-bd7c-8131cb46c7e0"). InnerVolumeSpecName "kube-api-access-rb5r7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:17:36.273487 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.273458 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f81681e-182b-4ed4-89bd-918370538f86-tls-certs" pod "1f81681e-182b-4ed4-89bd-918370538f86" (UID: "1f81681e-182b-4ed4-89bd-918370538f86"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:17:36.281492 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.281459 5019 scope.go:122] "RemoveContainer" containerID="f0acbc3f06de2463824292c21ced776771ddc11152124a00db1b005a372a6eaf" Oct 31 05:17:36.282700 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:17:36.282671 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0acbc3f06de2463824292c21ced776771ddc11152124a00db1b005a372a6eaf\": container with ID starting with f0acbc3f06de2463824292c21ced776771ddc11152124a00db1b005a372a6eaf not found: ID does not exist" containerID="f0acbc3f06de2463824292c21ced776771ddc11152124a00db1b005a372a6eaf" Oct 31 05:17:36.282794 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.282712 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0acbc3f06de2463824292c21ced776771ddc11152124a00db1b005a372a6eaf"} err="failed to get container status \"f0acbc3f06de2463824292c21ced776771ddc11152124a00db1b005a372a6eaf\": rpc error: code = NotFound desc = could not find container \"f0acbc3f06de2463824292c21ced776771ddc11152124a00db1b005a372a6eaf\": container with ID starting with f0acbc3f06de2463824292c21ced776771ddc11152124a00db1b005a372a6eaf not found: ID does not exist" Oct 31 05:17:36.362035 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.361993 5019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/157a108f-36ab-44a7-bd7c-8131cb46c7e0-home\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:36.362035 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.362033 5019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f81681e-182b-4ed4-89bd-918370538f86-dshm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:36.362276 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.362044 5019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f81681e-182b-4ed4-89bd-918370538f86-home\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:36.362276 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.362056 5019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/157a108f-36ab-44a7-bd7c-8131cb46c7e0-tmp-dir\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:36.362276 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.362070 5019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f81681e-182b-4ed4-89bd-918370538f86-model-cache\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:36.362276 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.362082 5019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f81681e-182b-4ed4-89bd-918370538f86-tls-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:36.362276 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.362093 5019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f81681e-182b-4ed4-89bd-918370538f86-tmp-dir\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:36.362276 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.362103 5019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/157a108f-36ab-44a7-bd7c-8131cb46c7e0-tls-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:36.362276 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.362116 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8swp\" (UniqueName: \"kubernetes.io/projected/1f81681e-182b-4ed4-89bd-918370538f86-kube-api-access-p8swp\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:36.362276 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.362129 5019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/157a108f-36ab-44a7-bd7c-8131cb46c7e0-model-cache\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:36.362276 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.362142 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb5r7\" (UniqueName: \"kubernetes.io/projected/157a108f-36ab-44a7-bd7c-8131cb46c7e0-kube-api-access-rb5r7\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:36.362276 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.362153 5019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/157a108f-36ab-44a7-bd7c-8131cb46c7e0-dshm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:36.438077 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.437991 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea8164c-bfqxm" Oct 31 05:17:36.462823 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.462779 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/247374b7-c1a3-4eae-bba4-76a070eed47c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/247374b7-c1a3-4eae-bba4-76a070eed47c-maas-api-xks-ca\") pod \"247374b7-c1a3-4eae-bba4-76a070eed47c\" (UID: \"247374b7-c1a3-4eae-bba4-76a070eed47c\") " Oct 31 05:17:36.463009 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.462842 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/247374b7-c1a3-4eae-bba4-76a070eed47c-kube-api-access-2gqhn\" (UniqueName: \"kubernetes.io/projected/247374b7-c1a3-4eae-bba4-76a070eed47c-kube-api-access-2gqhn\") pod \"247374b7-c1a3-4eae-bba4-76a070eed47c\" (UID: \"247374b7-c1a3-4eae-bba4-76a070eed47c\") " Oct 31 05:17:36.463009 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.462975 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/247374b7-c1a3-4eae-bba4-76a070eed47c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/247374b7-c1a3-4eae-bba4-76a070eed47c-maas-api-openshift-ca\") pod \"247374b7-c1a3-4eae-bba4-76a070eed47c\" (UID: \"247374b7-c1a3-4eae-bba4-76a070eed47c\") " Oct 31 05:17:36.463415 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.463379 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/247374b7-c1a3-4eae-bba4-76a070eed47c-maas-api-openshift-ca" pod "247374b7-c1a3-4eae-bba4-76a070eed47c" (UID: "247374b7-c1a3-4eae-bba4-76a070eed47c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:17:36.463683 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.463657 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/247374b7-c1a3-4eae-bba4-76a070eed47c-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:36.465370 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.465343 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/247374b7-c1a3-4eae-bba4-76a070eed47c-kube-api-access-2gqhn" pod "247374b7-c1a3-4eae-bba4-76a070eed47c" (UID: "247374b7-c1a3-4eae-bba4-76a070eed47c"). InnerVolumeSpecName "kube-api-access-2gqhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:17:36.465492 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.465411 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/247374b7-c1a3-4eae-bba4-76a070eed47c-maas-api-xks-ca" pod "247374b7-c1a3-4eae-bba4-76a070eed47c" (UID: "247374b7-c1a3-4eae-bba4-76a070eed47c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:17:36.477976 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.477929 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-4c2387-78c59569f5-pklf6" podUID="c60c1b8f-b960-4a87-9382-1dfc846ca8b5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:17:36.478175 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.478133 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4c2387-78fdcfbd75-2twf4" podUID="af4b877c-fb8f-48be-917c-a3e99fc1eb7f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:17:36.564522 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.564479 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/247374b7-c1a3-4eae-bba4-76a070eed47c-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:36.564522 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.564516 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gqhn\" (UniqueName: \"kubernetes.io/projected/247374b7-c1a3-4eae-bba4-76a070eed47c-kube-api-access-2gqhn\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:36.603917 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.603875 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc"] Oct 31 05:17:36.608103 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.608060 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-4c2387/simulator-w3-4c2387-kserve-64f4d58559-jjksc"] Oct 31 05:17:36.641072 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.641015 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp"] Oct 31 05:17:36.645126 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:36.645090 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-4c2387/premium-w3-4c2387-kserve-5c59b67cbb-6trqp"] Oct 31 05:17:37.264542 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:37.264509 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea8164c-bfqxm" Oct 31 05:17:37.265004 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:37.264504 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea8164c-bfqxm" event={"ID":"247374b7-c1a3-4eae-bba4-76a070eed47c","Type":"ContainerDied","Data":"4a89a5413e6de801607cde978deb3e36ebee5bcfc155f176b0d2ce49335492d8"} Oct 31 05:17:37.265004 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:37.264654 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a89a5413e6de801607cde978deb3e36ebee5bcfc155f176b0d2ce49335492d8" Oct 31 05:17:37.407972 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:37.407928 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" path="/var/lib/kubelet/pods/157a108f-36ab-44a7-bd7c-8131cb46c7e0/volumes" Oct 31 05:17:37.408509 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:37.408484 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f81681e-182b-4ed4-89bd-918370538f86" path="/var/lib/kubelet/pods/1f81681e-182b-4ed4-89bd-918370538f86/volumes" Oct 31 05:17:40.138282 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:40.138230 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" podUID="e0f4b8c2-14f4-4052-bf54-dd65bf59e413" containerName="main" containerID="cri-o://3e11da66d7d3a56da9a97a480f10b9e1b4754852bb727e952312c442bd6f2588" gracePeriod=45 Oct 31 05:17:40.161149 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:40.161050 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" podUID="e0f4b8c2-14f4-4052-bf54-dd65bf59e413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": dial tcp 10.132.0.98:8000: connect: connection refused" Oct 31 05:17:40.161437 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:40.161142 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" podUID="8e2c7041-ad52-49f4-8b55-d16691f4938f" containerName="main" containerID="cri-o://e19ff4563ad03d33b456af1368f7f44162cd7c33aada086c298c758c8f152283" gracePeriod=45 Oct 31 05:17:40.302811 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:40.302771 5019 generic.go:356] "Generic (PLEG): container finished" podID="8e2c7041-ad52-49f4-8b55-d16691f4938f" containerID="e19ff4563ad03d33b456af1368f7f44162cd7c33aada086c298c758c8f152283" exitCode=0 Oct 31 05:17:40.303012 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:40.302892 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" event={"ID":"8e2c7041-ad52-49f4-8b55-d16691f4938f","Type":"ContainerDied","Data":"e19ff4563ad03d33b456af1368f7f44162cd7c33aada086c298c758c8f152283"} Oct 31 05:17:40.304915 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:40.304878 5019 generic.go:356] "Generic (PLEG): container finished" podID="e0f4b8c2-14f4-4052-bf54-dd65bf59e413" containerID="3e11da66d7d3a56da9a97a480f10b9e1b4754852bb727e952312c442bd6f2588" exitCode=0 Oct 31 05:17:40.305080 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:40.305045 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" event={"ID":"e0f4b8c2-14f4-4052-bf54-dd65bf59e413","Type":"ContainerDied","Data":"3e11da66d7d3a56da9a97a480f10b9e1b4754852bb727e952312c442bd6f2588"} Oct 31 05:17:40.476025 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:40.475981 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" Oct 31 05:17:40.507577 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:40.506020 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0f4b8c2-14f4-4052-bf54-dd65bf59e413-tls-certs\" (UniqueName: \"kubernetes.io/secret/e0f4b8c2-14f4-4052-bf54-dd65bf59e413-tls-certs\") pod \"e0f4b8c2-14f4-4052-bf54-dd65bf59e413\" (UID: \"e0f4b8c2-14f4-4052-bf54-dd65bf59e413\") " Oct 31 05:17:40.507577 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:40.506091 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0f4b8c2-14f4-4052-bf54-dd65bf59e413-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0f4b8c2-14f4-4052-bf54-dd65bf59e413-dshm\") pod \"e0f4b8c2-14f4-4052-bf54-dd65bf59e413\" (UID: \"e0f4b8c2-14f4-4052-bf54-dd65bf59e413\") " Oct 31 05:17:40.507577 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:40.506143 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0f4b8c2-14f4-4052-bf54-dd65bf59e413-home\" (UniqueName: \"kubernetes.io/empty-dir/e0f4b8c2-14f4-4052-bf54-dd65bf59e413-home\") pod \"e0f4b8c2-14f4-4052-bf54-dd65bf59e413\" (UID: \"e0f4b8c2-14f4-4052-bf54-dd65bf59e413\") " Oct 31 05:17:40.507577 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:40.506227 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0f4b8c2-14f4-4052-bf54-dd65bf59e413-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0f4b8c2-14f4-4052-bf54-dd65bf59e413-tmp-dir\") pod \"e0f4b8c2-14f4-4052-bf54-dd65bf59e413\" (UID: \"e0f4b8c2-14f4-4052-bf54-dd65bf59e413\") " Oct 31 05:17:40.507577 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:40.506517 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0f4b8c2-14f4-4052-bf54-dd65bf59e413-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0f4b8c2-14f4-4052-bf54-dd65bf59e413-model-cache\") pod \"e0f4b8c2-14f4-4052-bf54-dd65bf59e413\" (UID: \"e0f4b8c2-14f4-4052-bf54-dd65bf59e413\") " Oct 31 05:17:40.507577 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:40.506584 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0f4b8c2-14f4-4052-bf54-dd65bf59e413-kube-api-access-stspp\" (UniqueName: \"kubernetes.io/projected/e0f4b8c2-14f4-4052-bf54-dd65bf59e413-kube-api-access-stspp\") pod \"e0f4b8c2-14f4-4052-bf54-dd65bf59e413\" (UID: \"e0f4b8c2-14f4-4052-bf54-dd65bf59e413\") " Oct 31 05:17:40.508080 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:40.507890 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0f4b8c2-14f4-4052-bf54-dd65bf59e413-tmp-dir" pod "e0f4b8c2-14f4-4052-bf54-dd65bf59e413" (UID: "e0f4b8c2-14f4-4052-bf54-dd65bf59e413"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:17:40.509930 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:40.509869 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0f4b8c2-14f4-4052-bf54-dd65bf59e413-home" pod "e0f4b8c2-14f4-4052-bf54-dd65bf59e413" (UID: "e0f4b8c2-14f4-4052-bf54-dd65bf59e413"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:17:40.510858 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:40.510653 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0f4b8c2-14f4-4052-bf54-dd65bf59e413-model-cache" pod "e0f4b8c2-14f4-4052-bf54-dd65bf59e413" (UID: "e0f4b8c2-14f4-4052-bf54-dd65bf59e413"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:17:40.515465 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:40.514592 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0f4b8c2-14f4-4052-bf54-dd65bf59e413-dshm" pod "e0f4b8c2-14f4-4052-bf54-dd65bf59e413" (UID: "e0f4b8c2-14f4-4052-bf54-dd65bf59e413"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:17:40.515465 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:40.514598 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0f4b8c2-14f4-4052-bf54-dd65bf59e413-tls-certs" pod "e0f4b8c2-14f4-4052-bf54-dd65bf59e413" (UID: "e0f4b8c2-14f4-4052-bf54-dd65bf59e413"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:17:40.516706 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:40.516668 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0f4b8c2-14f4-4052-bf54-dd65bf59e413-kube-api-access-stspp" pod "e0f4b8c2-14f4-4052-bf54-dd65bf59e413" (UID: "e0f4b8c2-14f4-4052-bf54-dd65bf59e413"). InnerVolumeSpecName "kube-api-access-stspp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:17:40.520680 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:40.520639 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" Oct 31 05:17:40.608322 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:40.608282 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e2c7041-ad52-49f4-8b55-d16691f4938f-home\" (UniqueName: \"kubernetes.io/empty-dir/8e2c7041-ad52-49f4-8b55-d16691f4938f-home\") pod \"8e2c7041-ad52-49f4-8b55-d16691f4938f\" (UID: \"8e2c7041-ad52-49f4-8b55-d16691f4938f\") " Oct 31 05:17:40.608584 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:40.608384 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e2c7041-ad52-49f4-8b55-d16691f4938f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e2c7041-ad52-49f4-8b55-d16691f4938f-dshm\") pod \"8e2c7041-ad52-49f4-8b55-d16691f4938f\" (UID: \"8e2c7041-ad52-49f4-8b55-d16691f4938f\") " Oct 31 05:17:40.608584 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:40.608438 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e2c7041-ad52-49f4-8b55-d16691f4938f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e2c7041-ad52-49f4-8b55-d16691f4938f-tmp-dir\") pod \"8e2c7041-ad52-49f4-8b55-d16691f4938f\" (UID: \"8e2c7041-ad52-49f4-8b55-d16691f4938f\") " Oct 31 05:17:40.608584 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:40.608485 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e2c7041-ad52-49f4-8b55-d16691f4938f-tls-certs\" (UniqueName: \"kubernetes.io/secret/8e2c7041-ad52-49f4-8b55-d16691f4938f-tls-certs\") pod \"8e2c7041-ad52-49f4-8b55-d16691f4938f\" (UID: \"8e2c7041-ad52-49f4-8b55-d16691f4938f\") " Oct 31 05:17:40.608584 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:40.608515 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e2c7041-ad52-49f4-8b55-d16691f4938f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e2c7041-ad52-49f4-8b55-d16691f4938f-model-cache\") pod \"8e2c7041-ad52-49f4-8b55-d16691f4938f\" (UID: \"8e2c7041-ad52-49f4-8b55-d16691f4938f\") " Oct 31 05:17:40.608869 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:40.608595 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e2c7041-ad52-49f4-8b55-d16691f4938f-kube-api-access-xhqnr\" (UniqueName: \"kubernetes.io/projected/8e2c7041-ad52-49f4-8b55-d16691f4938f-kube-api-access-xhqnr\") pod \"8e2c7041-ad52-49f4-8b55-d16691f4938f\" (UID: \"8e2c7041-ad52-49f4-8b55-d16691f4938f\") " Oct 31 05:17:40.608869 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:40.608797 5019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0f4b8c2-14f4-4052-bf54-dd65bf59e413-model-cache\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:40.608869 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:40.608810 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stspp\" (UniqueName: \"kubernetes.io/projected/e0f4b8c2-14f4-4052-bf54-dd65bf59e413-kube-api-access-stspp\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:40.608869 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:40.608821 5019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0f4b8c2-14f4-4052-bf54-dd65bf59e413-tls-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:40.608869 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:40.608831 5019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0f4b8c2-14f4-4052-bf54-dd65bf59e413-dshm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:40.608869 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:40.608839 5019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0f4b8c2-14f4-4052-bf54-dd65bf59e413-home\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:40.608869 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:40.608847 5019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0f4b8c2-14f4-4052-bf54-dd65bf59e413-tmp-dir\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:40.609573 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:40.609542 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e2c7041-ad52-49f4-8b55-d16691f4938f-tmp-dir" pod "8e2c7041-ad52-49f4-8b55-d16691f4938f" (UID: "8e2c7041-ad52-49f4-8b55-d16691f4938f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:17:40.609694 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:40.609597 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e2c7041-ad52-49f4-8b55-d16691f4938f-home" pod "8e2c7041-ad52-49f4-8b55-d16691f4938f" (UID: "8e2c7041-ad52-49f4-8b55-d16691f4938f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:17:40.609694 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:40.609610 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e2c7041-ad52-49f4-8b55-d16691f4938f-model-cache" pod "8e2c7041-ad52-49f4-8b55-d16691f4938f" (UID: "8e2c7041-ad52-49f4-8b55-d16691f4938f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:17:40.611618 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:40.611574 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e2c7041-ad52-49f4-8b55-d16691f4938f-dshm" pod "8e2c7041-ad52-49f4-8b55-d16691f4938f" (UID: "8e2c7041-ad52-49f4-8b55-d16691f4938f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:17:40.611775 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:40.611640 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e2c7041-ad52-49f4-8b55-d16691f4938f-kube-api-access-xhqnr" pod "8e2c7041-ad52-49f4-8b55-d16691f4938f" (UID: "8e2c7041-ad52-49f4-8b55-d16691f4938f"). InnerVolumeSpecName "kube-api-access-xhqnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:17:40.611863 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:40.611836 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e2c7041-ad52-49f4-8b55-d16691f4938f-tls-certs" pod "8e2c7041-ad52-49f4-8b55-d16691f4938f" (UID: "8e2c7041-ad52-49f4-8b55-d16691f4938f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:17:40.710462 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:40.710351 5019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e2c7041-ad52-49f4-8b55-d16691f4938f-model-cache\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:40.710462 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:40.710394 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhqnr\" (UniqueName: \"kubernetes.io/projected/8e2c7041-ad52-49f4-8b55-d16691f4938f-kube-api-access-xhqnr\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:40.710462 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:40.710407 5019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e2c7041-ad52-49f4-8b55-d16691f4938f-home\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:40.710462 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:40.710421 5019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e2c7041-ad52-49f4-8b55-d16691f4938f-dshm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:40.710462 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:40.710434 5019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e2c7041-ad52-49f4-8b55-d16691f4938f-tmp-dir\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:40.710462 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:40.710441 5019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e2c7041-ad52-49f4-8b55-d16691f4938f-tls-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:41.315407 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:41.315369 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" Oct 31 05:17:41.315981 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:41.315376 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q" event={"ID":"e0f4b8c2-14f4-4052-bf54-dd65bf59e413","Type":"ContainerDied","Data":"6bb6907ed3dda1faa26f2766c6e3bff81200e1b23848c2d09862b0f66af4a63d"} Oct 31 05:17:41.315981 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:41.315611 5019 scope.go:122] "RemoveContainer" containerID="3e11da66d7d3a56da9a97a480f10b9e1b4754852bb727e952312c442bd6f2588" Oct 31 05:17:41.318153 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:41.318116 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" event={"ID":"8e2c7041-ad52-49f4-8b55-d16691f4938f","Type":"ContainerDied","Data":"6dffef76b3aa82f52bccf874354bb0501f35a3b70770f13fc85aab92a3a2bd8a"} Oct 31 05:17:41.318288 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:41.318206 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l" Oct 31 05:17:41.331667 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:41.331481 5019 scope.go:122] "RemoveContainer" containerID="e19ff4563ad03d33b456af1368f7f44162cd7c33aada086c298c758c8f152283" Oct 31 05:17:41.384464 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:41.384418 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q"] Oct 31 05:17:41.395873 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:41.395822 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-965f61/distinct-2-w0-965f61-kserve-7bbc4797cf-48b6q"] Oct 31 05:17:41.407765 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:41.407669 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0f4b8c2-14f4-4052-bf54-dd65bf59e413" path="/var/lib/kubelet/pods/e0f4b8c2-14f4-4052-bf54-dd65bf59e413/volumes" Oct 31 05:17:41.432512 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:41.432449 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l"] Oct 31 05:17:41.435992 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:41.435947 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-965f61/simulator-w0-965f61-kserve-79d9bd9b-g7s8l"] Oct 31 05:17:42.028919 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:42.028884 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b9aaa6b-l495d"] Oct 31 05:17:42.041499 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:42.041460 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b9aaa6b-l495d"] Oct 31 05:17:43.407866 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:43.407826 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4eb1a2ea-76e1-4b63-bf5c-3a3fa67afe37" path="/var/lib/kubelet/pods/4eb1a2ea-76e1-4b63-bf5c-3a3fa67afe37/volumes" Oct 31 05:17:43.408306 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:43.408290 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e2c7041-ad52-49f4-8b55-d16691f4938f" path="/var/lib/kubelet/pods/8e2c7041-ad52-49f4-8b55-d16691f4938f/volumes" Oct 31 05:17:46.477375 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:46.477316 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4c2387-78fdcfbd75-2twf4" podUID="af4b877c-fb8f-48be-917c-a3e99fc1eb7f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:17:46.477977 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:46.477313 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-4c2387-78c59569f5-pklf6" podUID="c60c1b8f-b960-4a87-9382-1dfc846ca8b5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:17:48.452462 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:48.452417 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h"] Oct 31 05:17:48.452876 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:48.452851 5019 cpu_manager.go:414] "containerMap: removing container" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerName="main" Oct 31 05:17:48.452876 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:48.452862 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerName="main" Oct 31 05:17:48.452876 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:48.452875 5019 cpu_manager.go:414] "containerMap: removing container" podUID="247374b7-c1a3-4eae-bba4-76a070eed47c" containerName="revoke-keys" Oct 31 05:17:48.453011 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:48.452881 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="247374b7-c1a3-4eae-bba4-76a070eed47c" containerName="revoke-keys" Oct 31 05:17:48.453011 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:48.452915 5019 cpu_manager.go:414] "containerMap: removing container" podUID="e0f4b8c2-14f4-4052-bf54-dd65bf59e413" containerName="main" Oct 31 05:17:48.453011 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:48.452921 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0f4b8c2-14f4-4052-bf54-dd65bf59e413" containerName="main" Oct 31 05:17:48.453011 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:48.452929 5019 cpu_manager.go:414] "containerMap: removing container" podUID="8e2c7041-ad52-49f4-8b55-d16691f4938f" containerName="main" Oct 31 05:17:48.453011 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:48.452933 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e2c7041-ad52-49f4-8b55-d16691f4938f" containerName="main" Oct 31 05:17:48.453011 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:48.452946 5019 cpu_manager.go:414] "containerMap: removing container" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" Oct 31 05:17:48.453011 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:48.452951 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" Oct 31 05:17:48.453267 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:48.453015 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0f4b8c2-14f4-4052-bf54-dd65bf59e413" containerName="main" Oct 31 05:17:48.453267 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:48.453026 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e2c7041-ad52-49f4-8b55-d16691f4938f" containerName="main" Oct 31 05:17:48.453267 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:48.453036 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f81681e-182b-4ed4-89bd-918370538f86" containerName="main" Oct 31 05:17:48.453267 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:48.453043 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="157a108f-36ab-44a7-bd7c-8131cb46c7e0" containerName="main" Oct 31 05:17:48.453267 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:48.453049 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="247374b7-c1a3-4eae-bba4-76a070eed47c" containerName="revoke-keys" Oct 31 05:17:48.456615 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:48.456584 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h" Oct 31 05:17:48.471259 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:48.471221 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-4c2387-openshift-default-dockercfg-x4j6w\"" Oct 31 05:17:48.488779 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:48.488733 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h"] Oct 31 05:17:48.586352 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:48.586276 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18a6b450-e6b0-478e-9b79-79a664c57920-istio-envoy\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h\" (UID: \"18a6b450-e6b0-478e-9b79-79a664c57920\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h" Oct 31 05:17:48.586589 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:48.586451 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95cv7\" (UniqueName: \"kubernetes.io/projected/18a6b450-e6b0-478e-9b79-79a664c57920-kube-api-access-95cv7\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h\" (UID: \"18a6b450-e6b0-478e-9b79-79a664c57920\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h" Oct 31 05:17:48.586589 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:48.586499 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a6b450-e6b0-478e-9b79-79a664c57920-credential-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h\" (UID: \"18a6b450-e6b0-478e-9b79-79a664c57920\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h" Oct 31 05:17:48.586589 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:48.586534 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18a6b450-e6b0-478e-9b79-79a664c57920-istio-token\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h\" (UID: \"18a6b450-e6b0-478e-9b79-79a664c57920\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h" Oct 31 05:17:48.586589 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:48.586569 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18a6b450-e6b0-478e-9b79-79a664c57920-istio-podinfo\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h\" (UID: \"18a6b450-e6b0-478e-9b79-79a664c57920\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h" Oct 31 05:17:48.586853 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:48.586603 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18a6b450-e6b0-478e-9b79-79a664c57920-istiod-ca-cert\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h\" (UID: \"18a6b450-e6b0-478e-9b79-79a664c57920\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h" Oct 31 05:17:48.586853 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:48.586654 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18a6b450-e6b0-478e-9b79-79a664c57920-istio-data\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h\" (UID: \"18a6b450-e6b0-478e-9b79-79a664c57920\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h" Oct 31 05:17:48.586853 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:48.586685 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18a6b450-e6b0-478e-9b79-79a664c57920-workload-certs\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h\" (UID: \"18a6b450-e6b0-478e-9b79-79a664c57920\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h" Oct 31 05:17:48.586853 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:48.586791 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a6b450-e6b0-478e-9b79-79a664c57920-workload-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h\" (UID: \"18a6b450-e6b0-478e-9b79-79a664c57920\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h" Oct 31 05:17:48.687710 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:48.687646 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95cv7\" (UniqueName: \"kubernetes.io/projected/18a6b450-e6b0-478e-9b79-79a664c57920-kube-api-access-95cv7\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h\" (UID: \"18a6b450-e6b0-478e-9b79-79a664c57920\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h" Oct 31 05:17:48.687915 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:48.687717 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a6b450-e6b0-478e-9b79-79a664c57920-credential-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h\" (UID: \"18a6b450-e6b0-478e-9b79-79a664c57920\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h" Oct 31 05:17:48.687915 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:48.687749 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18a6b450-e6b0-478e-9b79-79a664c57920-istio-token\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h\" (UID: \"18a6b450-e6b0-478e-9b79-79a664c57920\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h" Oct 31 05:17:48.687915 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:48.687779 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18a6b450-e6b0-478e-9b79-79a664c57920-istio-podinfo\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h\" (UID: \"18a6b450-e6b0-478e-9b79-79a664c57920\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h" Oct 31 05:17:48.687915 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:48.687843 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18a6b450-e6b0-478e-9b79-79a664c57920-istiod-ca-cert\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h\" (UID: \"18a6b450-e6b0-478e-9b79-79a664c57920\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h" Oct 31 05:17:48.687915 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:48.687910 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18a6b450-e6b0-478e-9b79-79a664c57920-istio-data\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h\" (UID: \"18a6b450-e6b0-478e-9b79-79a664c57920\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h" Oct 31 05:17:48.688291 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:48.687941 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18a6b450-e6b0-478e-9b79-79a664c57920-workload-certs\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h\" (UID: \"18a6b450-e6b0-478e-9b79-79a664c57920\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h" Oct 31 05:17:48.688291 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:48.688029 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a6b450-e6b0-478e-9b79-79a664c57920-workload-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h\" (UID: \"18a6b450-e6b0-478e-9b79-79a664c57920\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h" Oct 31 05:17:48.688291 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:48.688062 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18a6b450-e6b0-478e-9b79-79a664c57920-istio-envoy\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h\" (UID: \"18a6b450-e6b0-478e-9b79-79a664c57920\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h" Oct 31 05:17:48.688508 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:48.688410 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a6b450-e6b0-478e-9b79-79a664c57920-credential-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h\" (UID: \"18a6b450-e6b0-478e-9b79-79a664c57920\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h" Oct 31 05:17:48.688656 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:48.688628 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18a6b450-e6b0-478e-9b79-79a664c57920-istio-data\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h\" (UID: \"18a6b450-e6b0-478e-9b79-79a664c57920\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h" Oct 31 05:17:48.688821 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:48.688740 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18a6b450-e6b0-478e-9b79-79a664c57920-workload-certs\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h\" (UID: \"18a6b450-e6b0-478e-9b79-79a664c57920\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h" Oct 31 05:17:48.688821 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:48.688762 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a6b450-e6b0-478e-9b79-79a664c57920-workload-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h\" (UID: \"18a6b450-e6b0-478e-9b79-79a664c57920\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h" Oct 31 05:17:48.689148 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:48.689126 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18a6b450-e6b0-478e-9b79-79a664c57920-istiod-ca-cert\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h\" (UID: \"18a6b450-e6b0-478e-9b79-79a664c57920\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h" Oct 31 05:17:48.690454 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:48.690431 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18a6b450-e6b0-478e-9b79-79a664c57920-istio-envoy\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h\" (UID: \"18a6b450-e6b0-478e-9b79-79a664c57920\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h" Oct 31 05:17:48.690878 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:48.690862 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18a6b450-e6b0-478e-9b79-79a664c57920-istio-podinfo\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h\" (UID: \"18a6b450-e6b0-478e-9b79-79a664c57920\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h" Oct 31 05:17:48.711524 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:48.711424 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95cv7\" (UniqueName: \"kubernetes.io/projected/18a6b450-e6b0-478e-9b79-79a664c57920-kube-api-access-95cv7\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h\" (UID: \"18a6b450-e6b0-478e-9b79-79a664c57920\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h" Oct 31 05:17:48.713147 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:48.713118 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18a6b450-e6b0-478e-9b79-79a664c57920-istio-token\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h\" (UID: \"18a6b450-e6b0-478e-9b79-79a664c57920\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h" Oct 31 05:17:48.770973 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:48.770936 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h" Oct 31 05:17:49.094676 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:49.094629 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h"] Oct 31 05:17:49.095888 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:17:49.095855 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18a6b450_e6b0_478e_9b79_79a664c57920.slice/crio-8f0eaa80922603fca5e3fa106c2d1765f41ea045152cb3b19a1b9e685c419d0b WatchSource:0}: Error finding container 8f0eaa80922603fca5e3fa106c2d1765f41ea045152cb3b19a1b9e685c419d0b: Status 404 returned error can't find the container with id 8f0eaa80922603fca5e3fa106c2d1765f41ea045152cb3b19a1b9e685c419d0b Oct 31 05:17:49.098425 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:49.098381 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:17:49.098533 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:49.098482 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:17:49.098573 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:49.098527 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:17:49.378450 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:49.378322 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h" event={"ID":"18a6b450-e6b0-478e-9b79-79a664c57920","Type":"ContainerStarted","Data":"a2c1905514d12e62661b6a678a40f1b533dbc85b6896aa8e457974aadc43fcf9"} Oct 31 05:17:49.378450 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:49.378428 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h" event={"ID":"18a6b450-e6b0-478e-9b79-79a664c57920","Type":"ContainerStarted","Data":"8f0eaa80922603fca5e3fa106c2d1765f41ea045152cb3b19a1b9e685c419d0b"} Oct 31 05:17:49.427162 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:49.427085 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h" podStartSLOduration=1.427060662 podStartE2EDuration="1.427060662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:17:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:17:49.424662249 +0000 UTC m=+4424.181000290" watchObservedRunningTime="2026-10-31 05:17:49.427060662 +0000 UTC m=+4424.183398703" Oct 31 05:17:49.774378 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:49.773441 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h" Oct 31 05:17:50.787358 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:50.784292 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.211:15021/healthz/ready\": dial tcp 10.132.1.211:15021: connect: connection refused" start-of-body= Oct 31 05:17:50.787358 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:50.784397 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h" podUID="18a6b450-e6b0-478e-9b79-79a664c57920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.211:15021/healthz/ready\": dial tcp 10.132.1.211:15021: connect: connection refused" Oct 31 05:17:51.795297 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:51.795249 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.211:15021/healthz/ready\": dial tcp 10.132.1.211:15021: connect: connection refused" start-of-body= Oct 31 05:17:51.795902 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:51.795360 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h" podUID="18a6b450-e6b0-478e-9b79-79a664c57920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.211:15021/healthz/ready\": dial tcp 10.132.1.211:15021: connect: connection refused" Oct 31 05:17:52.415162 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:17:52.415120 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:17:52.774361 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:52.773874 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.211:15021/healthz/ready\": dial tcp 10.132.1.211:15021: connect: connection refused" start-of-body= Oct 31 05:17:52.774361 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:52.773965 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h" podUID="18a6b450-e6b0-478e-9b79-79a664c57920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.211:15021/healthz/ready\": dial tcp 10.132.1.211:15021: connect: connection refused" Oct 31 05:17:53.042709 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:53.042639 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-339900d9-vdj8t"] Oct 31 05:17:53.052252 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:53.052220 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-4c2387-78c59569f5-pklf6" Oct 31 05:17:53.053021 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:53.052992 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-339900d9-vdj8t"] Oct 31 05:17:53.108943 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:53.108914 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4c2387-78fdcfbd75-2twf4" Oct 31 05:17:53.139685 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:53.139641 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af4b877c-fb8f-48be-917c-a3e99fc1eb7f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af4b877c-fb8f-48be-917c-a3e99fc1eb7f-plugins-config-volume\") pod \"af4b877c-fb8f-48be-917c-a3e99fc1eb7f\" (UID: \"af4b877c-fb8f-48be-917c-a3e99fc1eb7f\") " Oct 31 05:17:53.139911 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:53.139699 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af4b877c-fb8f-48be-917c-a3e99fc1eb7f-kube-api-access-v2cw6\" (UniqueName: \"kubernetes.io/projected/af4b877c-fb8f-48be-917c-a3e99fc1eb7f-kube-api-access-v2cw6\") pod \"af4b877c-fb8f-48be-917c-a3e99fc1eb7f\" (UID: \"af4b877c-fb8f-48be-917c-a3e99fc1eb7f\") " Oct 31 05:17:53.139911 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:53.139722 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c60c1b8f-b960-4a87-9382-1dfc846ca8b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c60c1b8f-b960-4a87-9382-1dfc846ca8b5-plugins-config-volume\") pod \"c60c1b8f-b960-4a87-9382-1dfc846ca8b5\" (UID: \"c60c1b8f-b960-4a87-9382-1dfc846ca8b5\") " Oct 31 05:17:53.139911 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:53.139779 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c60c1b8f-b960-4a87-9382-1dfc846ca8b5-kube-api-access-9rq5r\" (UniqueName: \"kubernetes.io/projected/c60c1b8f-b960-4a87-9382-1dfc846ca8b5-kube-api-access-9rq5r\") pod \"c60c1b8f-b960-4a87-9382-1dfc846ca8b5\" (UID: \"c60c1b8f-b960-4a87-9382-1dfc846ca8b5\") " Oct 31 05:17:53.140131 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:53.140072 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af4b877c-fb8f-48be-917c-a3e99fc1eb7f-plugins-config-volume" pod "af4b877c-fb8f-48be-917c-a3e99fc1eb7f" (UID: "af4b877c-fb8f-48be-917c-a3e99fc1eb7f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:17:53.140201 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:53.140162 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c60c1b8f-b960-4a87-9382-1dfc846ca8b5-plugins-config-volume" pod "c60c1b8f-b960-4a87-9382-1dfc846ca8b5" (UID: "c60c1b8f-b960-4a87-9382-1dfc846ca8b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:17:53.142033 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:53.142006 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c60c1b8f-b960-4a87-9382-1dfc846ca8b5-kube-api-access-9rq5r" pod "c60c1b8f-b960-4a87-9382-1dfc846ca8b5" (UID: "c60c1b8f-b960-4a87-9382-1dfc846ca8b5"). InnerVolumeSpecName "kube-api-access-9rq5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:17:53.142165 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:53.142033 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af4b877c-fb8f-48be-917c-a3e99fc1eb7f-kube-api-access-v2cw6" pod "af4b877c-fb8f-48be-917c-a3e99fc1eb7f" (UID: "af4b877c-fb8f-48be-917c-a3e99fc1eb7f"). InnerVolumeSpecName "kube-api-access-v2cw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:17:53.241139 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:53.241039 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af4b877c-fb8f-48be-917c-a3e99fc1eb7f-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:53.241139 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:53.241084 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2cw6\" (UniqueName: \"kubernetes.io/projected/af4b877c-fb8f-48be-917c-a3e99fc1eb7f-kube-api-access-v2cw6\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:53.241139 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:53.241098 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c60c1b8f-b960-4a87-9382-1dfc846ca8b5-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:53.241139 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:53.241112 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rq5r\" (UniqueName: \"kubernetes.io/projected/c60c1b8f-b960-4a87-9382-1dfc846ca8b5-kube-api-access-9rq5r\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:17:53.406352 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:53.406296 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7c18d7a-62a1-4ef6-a835-7e79bd7ed3a0" path="/var/lib/kubelet/pods/e7c18d7a-62a1-4ef6-a835-7e79bd7ed3a0/volumes" Oct 31 05:17:53.423348 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:53.423301 5019 generic.go:356] "Generic (PLEG): container finished" podID="af4b877c-fb8f-48be-917c-a3e99fc1eb7f" containerID="db954a6e84eaaf447a3bdcbe78f773fa99e31994c552c3b0d7e257bc561d8c0e" exitCode=0 Oct 31 05:17:53.423550 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:53.423362 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4c2387-78fdcfbd75-2twf4" Oct 31 05:17:53.423550 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:53.423362 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4c2387-78fdcfbd75-2twf4" event={"ID":"af4b877c-fb8f-48be-917c-a3e99fc1eb7f","Type":"ContainerDied","Data":"db954a6e84eaaf447a3bdcbe78f773fa99e31994c552c3b0d7e257bc561d8c0e"} Oct 31 05:17:53.423550 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:53.423486 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4c2387-78fdcfbd75-2twf4" event={"ID":"af4b877c-fb8f-48be-917c-a3e99fc1eb7f","Type":"ContainerDied","Data":"ccf10d75895a16c011c24acd8be2bad35912c024f457e3b8ebbb028620f186cb"} Oct 31 05:17:53.423550 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:53.423508 5019 scope.go:122] "RemoveContainer" containerID="db954a6e84eaaf447a3bdcbe78f773fa99e31994c552c3b0d7e257bc561d8c0e" Oct 31 05:17:53.425318 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:53.425291 5019 generic.go:356] "Generic (PLEG): container finished" podID="c60c1b8f-b960-4a87-9382-1dfc846ca8b5" containerID="92ed61316fe2122ecfbe33607c347f05b32a297ce2df922340397c66b31e8700" exitCode=0 Oct 31 05:17:53.425481 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:53.425358 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-4c2387-78c59569f5-pklf6" Oct 31 05:17:53.425481 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:53.425415 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-4c2387-78c59569f5-pklf6" event={"ID":"c60c1b8f-b960-4a87-9382-1dfc846ca8b5","Type":"ContainerDied","Data":"92ed61316fe2122ecfbe33607c347f05b32a297ce2df922340397c66b31e8700"} Oct 31 05:17:53.425949 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:53.425501 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-4c2387-78c59569f5-pklf6" event={"ID":"c60c1b8f-b960-4a87-9382-1dfc846ca8b5","Type":"ContainerDied","Data":"495f19a16703802d4d4f6c77c79732f727c2e250aa9222976cbaa012071c4df2"} Oct 31 05:17:53.435614 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:53.435586 5019 scope.go:122] "RemoveContainer" containerID="db954a6e84eaaf447a3bdcbe78f773fa99e31994c552c3b0d7e257bc561d8c0e" Oct 31 05:17:53.435997 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:17:53.435970 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db954a6e84eaaf447a3bdcbe78f773fa99e31994c552c3b0d7e257bc561d8c0e\": container with ID starting with db954a6e84eaaf447a3bdcbe78f773fa99e31994c552c3b0d7e257bc561d8c0e not found: ID does not exist" containerID="db954a6e84eaaf447a3bdcbe78f773fa99e31994c552c3b0d7e257bc561d8c0e" Oct 31 05:17:53.436060 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:53.436010 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db954a6e84eaaf447a3bdcbe78f773fa99e31994c552c3b0d7e257bc561d8c0e"} err="failed to get container status \"db954a6e84eaaf447a3bdcbe78f773fa99e31994c552c3b0d7e257bc561d8c0e\": rpc error: code = NotFound desc = could not find container \"db954a6e84eaaf447a3bdcbe78f773fa99e31994c552c3b0d7e257bc561d8c0e\": container with ID starting with db954a6e84eaaf447a3bdcbe78f773fa99e31994c552c3b0d7e257bc561d8c0e not found: ID does not exist" Oct 31 05:17:53.436060 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:53.436028 5019 scope.go:122] "RemoveContainer" containerID="92ed61316fe2122ecfbe33607c347f05b32a297ce2df922340397c66b31e8700" Oct 31 05:17:53.446870 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:53.446822 5019 scope.go:122] "RemoveContainer" containerID="92ed61316fe2122ecfbe33607c347f05b32a297ce2df922340397c66b31e8700" Oct 31 05:17:53.447220 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:17:53.447200 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92ed61316fe2122ecfbe33607c347f05b32a297ce2df922340397c66b31e8700\": container with ID starting with 92ed61316fe2122ecfbe33607c347f05b32a297ce2df922340397c66b31e8700 not found: ID does not exist" containerID="92ed61316fe2122ecfbe33607c347f05b32a297ce2df922340397c66b31e8700" Oct 31 05:17:53.447290 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:53.447230 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92ed61316fe2122ecfbe33607c347f05b32a297ce2df922340397c66b31e8700"} err="failed to get container status \"92ed61316fe2122ecfbe33607c347f05b32a297ce2df922340397c66b31e8700\": rpc error: code = NotFound desc = could not find container \"92ed61316fe2122ecfbe33607c347f05b32a297ce2df922340397c66b31e8700\": container with ID starting with 92ed61316fe2122ecfbe33607c347f05b32a297ce2df922340397c66b31e8700 not found: ID does not exist" Oct 31 05:17:53.502052 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:53.501894 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-4c2387-78c59569f5-pklf6"] Oct 31 05:17:53.505245 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:53.505200 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-4c2387-78c59569f5-pklf6"] Oct 31 05:17:53.532575 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:53.532506 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-4c2387-78fdcfbd75-2twf4"] Oct 31 05:17:53.536846 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:53.536810 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-4c2387-78fdcfbd75-2twf4"] Oct 31 05:17:53.778857 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:53.778324 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.211:15021/healthz/ready\": dial tcp 10.132.1.211:15021: connect: connection refused" start-of-body= Oct 31 05:17:53.778857 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:53.778410 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h" podUID="18a6b450-e6b0-478e-9b79-79a664c57920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.211:15021/healthz/ready\": dial tcp 10.132.1.211:15021: connect: connection refused" Oct 31 05:17:54.777373 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:54.776789 5019 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.211:15021/healthz/ready\": dial tcp 10.132.1.211:15021: connect: connection refused" start-of-body= Oct 31 05:17:54.777373 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:54.776865 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h" podUID="18a6b450-e6b0-478e-9b79-79a664c57920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.211:15021/healthz/ready\": dial tcp 10.132.1.211:15021: connect: connection refused" Oct 31 05:17:55.407812 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:55.407764 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af4b877c-fb8f-48be-917c-a3e99fc1eb7f" path="/var/lib/kubelet/pods/af4b877c-fb8f-48be-917c-a3e99fc1eb7f/volumes" Oct 31 05:17:55.408302 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:55.408279 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c60c1b8f-b960-4a87-9382-1dfc846ca8b5" path="/var/lib/kubelet/pods/c60c1b8f-b960-4a87-9382-1dfc846ca8b5/volumes" Oct 31 05:17:55.776095 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:55.776055 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h" Oct 31 05:17:55.776553 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:55.776527 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h" Oct 31 05:17:55.777411 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:55.777391 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h" Oct 31 05:17:57.029015 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:57.028878 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e8d67d9-cmhxg"] Oct 31 05:17:57.040167 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:57.040124 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e8d67d9-cmhxg"] Oct 31 05:17:57.406460 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:57.406375 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d10aad80-55fd-4e26-883d-3dc3a5c7dc40" path="/var/lib/kubelet/pods/d10aad80-55fd-4e26-883d-3dc3a5c7dc40/volumes" Oct 31 05:17:59.818041 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:59.817997 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h"] Oct 31 05:17:59.818625 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:17:59.818261 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h" podUID="18a6b450-e6b0-478e-9b79-79a664c57920" containerName="istio-proxy" containerID="cri-o://a2c1905514d12e62661b6a678a40f1b533dbc85b6896aa8e457974aadc43fcf9" gracePeriod=30 Oct 31 05:18:04.029656 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:04.029614 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ef3d94d-w6hdj"] Oct 31 05:18:04.041938 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:04.041888 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ef3d94d-w6hdj"] Oct 31 05:18:04.867940 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:18:04.867892 5019 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18a6b450_e6b0_478e_9b79_79a664c57920.slice/crio-a2c1905514d12e62661b6a678a40f1b533dbc85b6896aa8e457974aadc43fcf9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18a6b450_e6b0_478e_9b79_79a664c57920.slice/crio-conmon-a2c1905514d12e62661b6a678a40f1b533dbc85b6896aa8e457974aadc43fcf9.scope\": RecentStats: unable to find data in memory cache]" Oct 31 05:18:05.099729 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:05.099640 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h" Oct 31 05:18:05.160760 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:05.160718 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18a6b450-e6b0-478e-9b79-79a664c57920-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18a6b450-e6b0-478e-9b79-79a664c57920-workload-certs\") pod \"18a6b450-e6b0-478e-9b79-79a664c57920\" (UID: \"18a6b450-e6b0-478e-9b79-79a664c57920\") " Oct 31 05:18:05.160991 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:05.160790 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18a6b450-e6b0-478e-9b79-79a664c57920-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a6b450-e6b0-478e-9b79-79a664c57920-workload-socket\") pod \"18a6b450-e6b0-478e-9b79-79a664c57920\" (UID: \"18a6b450-e6b0-478e-9b79-79a664c57920\") " Oct 31 05:18:05.160991 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:05.160858 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18a6b450-e6b0-478e-9b79-79a664c57920-kube-api-access-95cv7\" (UniqueName: \"kubernetes.io/projected/18a6b450-e6b0-478e-9b79-79a664c57920-kube-api-access-95cv7\") pod \"18a6b450-e6b0-478e-9b79-79a664c57920\" (UID: \"18a6b450-e6b0-478e-9b79-79a664c57920\") " Oct 31 05:18:05.160991 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:05.160886 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18a6b450-e6b0-478e-9b79-79a664c57920-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18a6b450-e6b0-478e-9b79-79a664c57920-istio-data\") pod \"18a6b450-e6b0-478e-9b79-79a664c57920\" (UID: \"18a6b450-e6b0-478e-9b79-79a664c57920\") " Oct 31 05:18:05.160991 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:05.160934 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18a6b450-e6b0-478e-9b79-79a664c57920-istio-token\" (UniqueName: \"kubernetes.io/projected/18a6b450-e6b0-478e-9b79-79a664c57920-istio-token\") pod \"18a6b450-e6b0-478e-9b79-79a664c57920\" (UID: \"18a6b450-e6b0-478e-9b79-79a664c57920\") " Oct 31 05:18:05.160991 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:05.160971 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18a6b450-e6b0-478e-9b79-79a664c57920-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a6b450-e6b0-478e-9b79-79a664c57920-credential-socket\") pod \"18a6b450-e6b0-478e-9b79-79a664c57920\" (UID: \"18a6b450-e6b0-478e-9b79-79a664c57920\") " Oct 31 05:18:05.161302 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:05.161044 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18a6b450-e6b0-478e-9b79-79a664c57920-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18a6b450-e6b0-478e-9b79-79a664c57920-istiod-ca-cert\") pod \"18a6b450-e6b0-478e-9b79-79a664c57920\" (UID: \"18a6b450-e6b0-478e-9b79-79a664c57920\") " Oct 31 05:18:05.161302 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:05.161064 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18a6b450-e6b0-478e-9b79-79a664c57920-workload-certs" pod "18a6b450-e6b0-478e-9b79-79a664c57920" (UID: "18a6b450-e6b0-478e-9b79-79a664c57920"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:18:05.161302 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:05.161087 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18a6b450-e6b0-478e-9b79-79a664c57920-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18a6b450-e6b0-478e-9b79-79a664c57920-istio-envoy\") pod \"18a6b450-e6b0-478e-9b79-79a664c57920\" (UID: \"18a6b450-e6b0-478e-9b79-79a664c57920\") " Oct 31 05:18:05.161302 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:05.161143 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18a6b450-e6b0-478e-9b79-79a664c57920-workload-socket" pod "18a6b450-e6b0-478e-9b79-79a664c57920" (UID: "18a6b450-e6b0-478e-9b79-79a664c57920"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:18:05.161302 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:05.161198 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/18a6b450-e6b0-478e-9b79-79a664c57920-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18a6b450-e6b0-478e-9b79-79a664c57920-istio-podinfo\") pod \"18a6b450-e6b0-478e-9b79-79a664c57920\" (UID: \"18a6b450-e6b0-478e-9b79-79a664c57920\") " Oct 31 05:18:05.161596 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:05.161553 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18a6b450-e6b0-478e-9b79-79a664c57920-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:18:05.161596 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:05.161574 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a6b450-e6b0-478e-9b79-79a664c57920-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:18:05.161596 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:05.161573 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18a6b450-e6b0-478e-9b79-79a664c57920-credential-socket" pod "18a6b450-e6b0-478e-9b79-79a664c57920" (UID: "18a6b450-e6b0-478e-9b79-79a664c57920"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:18:05.162003 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:05.161888 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18a6b450-e6b0-478e-9b79-79a664c57920-istiod-ca-cert" pod "18a6b450-e6b0-478e-9b79-79a664c57920" (UID: "18a6b450-e6b0-478e-9b79-79a664c57920"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:18:05.162130 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:05.162064 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18a6b450-e6b0-478e-9b79-79a664c57920-istio-data" pod "18a6b450-e6b0-478e-9b79-79a664c57920" (UID: "18a6b450-e6b0-478e-9b79-79a664c57920"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:18:05.163575 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:05.163540 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18a6b450-e6b0-478e-9b79-79a664c57920-istio-token" pod "18a6b450-e6b0-478e-9b79-79a664c57920" (UID: "18a6b450-e6b0-478e-9b79-79a664c57920"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:18:05.164084 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:05.164057 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18a6b450-e6b0-478e-9b79-79a664c57920-istio-podinfo" pod "18a6b450-e6b0-478e-9b79-79a664c57920" (UID: "18a6b450-e6b0-478e-9b79-79a664c57920"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:18:05.164394 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:05.164374 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18a6b450-e6b0-478e-9b79-79a664c57920-kube-api-access-95cv7" pod "18a6b450-e6b0-478e-9b79-79a664c57920" (UID: "18a6b450-e6b0-478e-9b79-79a664c57920"). InnerVolumeSpecName "kube-api-access-95cv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:18:05.164665 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:05.164639 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18a6b450-e6b0-478e-9b79-79a664c57920-istio-envoy" pod "18a6b450-e6b0-478e-9b79-79a664c57920" (UID: "18a6b450-e6b0-478e-9b79-79a664c57920"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:18:05.263048 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:05.263004 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18a6b450-e6b0-478e-9b79-79a664c57920-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:18:05.263048 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:05.263048 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a6b450-e6b0-478e-9b79-79a664c57920-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:18:05.263319 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:05.263062 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18a6b450-e6b0-478e-9b79-79a664c57920-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:18:05.263319 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:05.263077 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18a6b450-e6b0-478e-9b79-79a664c57920-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:18:05.263319 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:05.263088 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18a6b450-e6b0-478e-9b79-79a664c57920-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:18:05.263319 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:05.263102 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95cv7\" (UniqueName: \"kubernetes.io/projected/18a6b450-e6b0-478e-9b79-79a664c57920-kube-api-access-95cv7\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:18:05.263319 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:05.263113 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18a6b450-e6b0-478e-9b79-79a664c57920-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:18:05.408106 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:05.408019 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="227b5b23-079f-4fef-b140-2835d2df2745" path="/var/lib/kubelet/pods/227b5b23-079f-4fef-b140-2835d2df2745/volumes" Oct 31 05:18:05.514221 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:05.514180 5019 generic.go:356] "Generic (PLEG): container finished" podID="18a6b450-e6b0-478e-9b79-79a664c57920" containerID="a2c1905514d12e62661b6a678a40f1b533dbc85b6896aa8e457974aadc43fcf9" exitCode=0 Oct 31 05:18:05.514221 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:05.514210 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h" event={"ID":"18a6b450-e6b0-478e-9b79-79a664c57920","Type":"ContainerDied","Data":"a2c1905514d12e62661b6a678a40f1b533dbc85b6896aa8e457974aadc43fcf9"} Oct 31 05:18:05.514539 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:05.514233 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h" Oct 31 05:18:05.514539 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:05.514274 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h" event={"ID":"18a6b450-e6b0-478e-9b79-79a664c57920","Type":"ContainerDied","Data":"8f0eaa80922603fca5e3fa106c2d1765f41ea045152cb3b19a1b9e685c419d0b"} Oct 31 05:18:05.514539 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:05.514298 5019 scope.go:122] "RemoveContainer" containerID="a2c1905514d12e62661b6a678a40f1b533dbc85b6896aa8e457974aadc43fcf9" Oct 31 05:18:05.525033 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:05.525004 5019 scope.go:122] "RemoveContainer" containerID="a2c1905514d12e62661b6a678a40f1b533dbc85b6896aa8e457974aadc43fcf9" Oct 31 05:18:05.525435 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:18:05.525409 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2c1905514d12e62661b6a678a40f1b533dbc85b6896aa8e457974aadc43fcf9\": container with ID starting with a2c1905514d12e62661b6a678a40f1b533dbc85b6896aa8e457974aadc43fcf9 not found: ID does not exist" containerID="a2c1905514d12e62661b6a678a40f1b533dbc85b6896aa8e457974aadc43fcf9" Oct 31 05:18:05.525481 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:05.525446 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2c1905514d12e62661b6a678a40f1b533dbc85b6896aa8e457974aadc43fcf9"} err="failed to get container status \"a2c1905514d12e62661b6a678a40f1b533dbc85b6896aa8e457974aadc43fcf9\": rpc error: code = NotFound desc = could not find container \"a2c1905514d12e62661b6a678a40f1b533dbc85b6896aa8e457974aadc43fcf9\": container with ID starting with a2c1905514d12e62661b6a678a40f1b533dbc85b6896aa8e457974aadc43fcf9 not found: ID does not exist" Oct 31 05:18:05.566772 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:05.566735 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h"] Oct 31 05:18:05.579487 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:05.579432 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-h4p8h"] Oct 31 05:18:06.052398 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:06.052357 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-77ddb887cc-6csbj_720f62c6-de2c-4269-bc48-b785891f8fec/authorino/0.log" Oct 31 05:18:06.292908 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:06.292858 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-r22dt_48e6e3f6-9d22-42a3-bc22-a27e516aeda1/manager/0.log" Oct 31 05:18:06.501846 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:06.501808 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-ggcnf_cd96ff5f-fb42-4491-aae0-bcdd6e4002aa/manager/0.log" Oct 31 05:18:07.408831 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:07.408780 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18a6b450-e6b0-478e-9b79-79a664c57920" path="/var/lib/kubelet/pods/18a6b450-e6b0-478e-9b79-79a664c57920/volumes" Oct 31 05:18:07.418561 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:07.418515 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-vwcwp_1f9f7baf-a850-453c-8e16-849942ba754a/manager/0.log" Oct 31 05:18:07.894733 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:07.894691 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-gsxbf_176b5a4d-06b5-484e-b6ba-e5727c90afba/manager/0.log" Oct 31 05:18:08.845671 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:08.845632 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6dc65d95b6-gpnbc_d194e31f-b57e-490d-a5c8-9f7b12de4b2c/manager/0.log" Oct 31 05:18:10.028941 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:10.028898 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1e47f72-kn24j"] Oct 31 05:18:10.032081 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:10.032046 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1e47f72-kn24j"] Oct 31 05:18:11.408545 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:11.408503 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5ed9a44-4e39-403c-a936-7f44d5f0fdad" path="/var/lib/kubelet/pods/a5ed9a44-4e39-403c-a936-7f44d5f0fdad/volumes" Oct 31 05:18:12.026741 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:12.026704 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d48ecf64-6n45m"] Oct 31 05:18:12.038695 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:12.038651 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d48ecf64-6n45m"] Oct 31 05:18:13.027422 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:13.027383 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1df64e1e-hdx4d"] Oct 31 05:18:13.039711 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:13.039663 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1df64e1e-hdx4d"] Oct 31 05:18:13.407147 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:13.407056 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4972f293-aad0-4dbf-b5cf-925d46e74a3f" path="/var/lib/kubelet/pods/4972f293-aad0-4dbf-b5cf-925d46e74a3f/volumes" Oct 31 05:18:13.407542 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:13.407519 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df1885b0-4e23-47ce-a3aa-ae4e5045750c" path="/var/lib/kubelet/pods/df1885b0-4e23-47ce-a3aa-ae4e5045750c/volumes" Oct 31 05:18:15.571645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:15.571377 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k"] Oct 31 05:18:15.572834 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:15.572795 5019 cpu_manager.go:414] "containerMap: removing container" podUID="af4b877c-fb8f-48be-917c-a3e99fc1eb7f" containerName="payload-pre-processing" Oct 31 05:18:15.572834 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:15.572830 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af4b877c-fb8f-48be-917c-a3e99fc1eb7f" containerName="payload-pre-processing" Oct 31 05:18:15.573069 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:15.572860 5019 cpu_manager.go:414] "containerMap: removing container" podUID="c60c1b8f-b960-4a87-9382-1dfc846ca8b5" containerName="payload-processing" Oct 31 05:18:15.573069 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:15.572869 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c60c1b8f-b960-4a87-9382-1dfc846ca8b5" containerName="payload-processing" Oct 31 05:18:15.573069 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:15.572903 5019 cpu_manager.go:414] "containerMap: removing container" podUID="18a6b450-e6b0-478e-9b79-79a664c57920" containerName="istio-proxy" Oct 31 05:18:15.573069 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:15.572911 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18a6b450-e6b0-478e-9b79-79a664c57920" containerName="istio-proxy" Oct 31 05:18:15.573069 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:15.572997 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="18a6b450-e6b0-478e-9b79-79a664c57920" containerName="istio-proxy" Oct 31 05:18:15.573069 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:15.573018 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="af4b877c-fb8f-48be-917c-a3e99fc1eb7f" containerName="payload-pre-processing" Oct 31 05:18:15.573069 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:15.573031 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="c60c1b8f-b960-4a87-9382-1dfc846ca8b5" containerName="payload-processing" Oct 31 05:18:15.576851 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:15.576817 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k" Oct 31 05:18:15.593466 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:15.593424 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-8d386766-openshift-default-dockercfg-pgzkc\"" Oct 31 05:18:15.607495 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:15.607453 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k"] Oct 31 05:18:15.665206 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:15.665163 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07ba66ba-7067-4861-82ec-cf8a50a8550c-istio-token\") pod \"e2e-ait-8d386766-openshift-default-575667689c-n858k\" (UID: \"07ba66ba-7067-4861-82ec-cf8a50a8550c\") " pod="openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k" Oct 31 05:18:15.665442 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:15.665211 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07ba66ba-7067-4861-82ec-cf8a50a8550c-workload-certs\") pod \"e2e-ait-8d386766-openshift-default-575667689c-n858k\" (UID: \"07ba66ba-7067-4861-82ec-cf8a50a8550c\") " pod="openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k" Oct 31 05:18:15.665442 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:15.665240 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07ba66ba-7067-4861-82ec-cf8a50a8550c-istio-data\") pod \"e2e-ait-8d386766-openshift-default-575667689c-n858k\" (UID: \"07ba66ba-7067-4861-82ec-cf8a50a8550c\") " pod="openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k" Oct 31 05:18:15.665442 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:15.665393 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkw46\" (UniqueName: \"kubernetes.io/projected/07ba66ba-7067-4861-82ec-cf8a50a8550c-kube-api-access-jkw46\") pod \"e2e-ait-8d386766-openshift-default-575667689c-n858k\" (UID: \"07ba66ba-7067-4861-82ec-cf8a50a8550c\") " pod="openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k" Oct 31 05:18:15.665612 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:15.665449 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07ba66ba-7067-4861-82ec-cf8a50a8550c-istio-podinfo\") pod \"e2e-ait-8d386766-openshift-default-575667689c-n858k\" (UID: \"07ba66ba-7067-4861-82ec-cf8a50a8550c\") " pod="openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k" Oct 31 05:18:15.665612 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:15.665474 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07ba66ba-7067-4861-82ec-cf8a50a8550c-workload-socket\") pod \"e2e-ait-8d386766-openshift-default-575667689c-n858k\" (UID: \"07ba66ba-7067-4861-82ec-cf8a50a8550c\") " pod="openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k" Oct 31 05:18:15.665612 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:15.665488 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07ba66ba-7067-4861-82ec-cf8a50a8550c-credential-socket\") pod \"e2e-ait-8d386766-openshift-default-575667689c-n858k\" (UID: \"07ba66ba-7067-4861-82ec-cf8a50a8550c\") " pod="openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k" Oct 31 05:18:15.665731 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:15.665609 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07ba66ba-7067-4861-82ec-cf8a50a8550c-istio-envoy\") pod \"e2e-ait-8d386766-openshift-default-575667689c-n858k\" (UID: \"07ba66ba-7067-4861-82ec-cf8a50a8550c\") " pod="openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k" Oct 31 05:18:15.665731 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:15.665666 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07ba66ba-7067-4861-82ec-cf8a50a8550c-istiod-ca-cert\") pod \"e2e-ait-8d386766-openshift-default-575667689c-n858k\" (UID: \"07ba66ba-7067-4861-82ec-cf8a50a8550c\") " pod="openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k" Oct 31 05:18:15.766206 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:15.766159 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07ba66ba-7067-4861-82ec-cf8a50a8550c-istiod-ca-cert\") pod \"e2e-ait-8d386766-openshift-default-575667689c-n858k\" (UID: \"07ba66ba-7067-4861-82ec-cf8a50a8550c\") " pod="openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k" Oct 31 05:18:15.766469 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:15.766262 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07ba66ba-7067-4861-82ec-cf8a50a8550c-istio-token\") pod \"e2e-ait-8d386766-openshift-default-575667689c-n858k\" (UID: \"07ba66ba-7067-4861-82ec-cf8a50a8550c\") " pod="openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k" Oct 31 05:18:15.766469 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:15.766294 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07ba66ba-7067-4861-82ec-cf8a50a8550c-workload-certs\") pod \"e2e-ait-8d386766-openshift-default-575667689c-n858k\" (UID: \"07ba66ba-7067-4861-82ec-cf8a50a8550c\") " pod="openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k" Oct 31 05:18:15.766469 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:15.766322 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07ba66ba-7067-4861-82ec-cf8a50a8550c-istio-data\") pod \"e2e-ait-8d386766-openshift-default-575667689c-n858k\" (UID: \"07ba66ba-7067-4861-82ec-cf8a50a8550c\") " pod="openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k" Oct 31 05:18:15.766662 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:15.766467 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkw46\" (UniqueName: \"kubernetes.io/projected/07ba66ba-7067-4861-82ec-cf8a50a8550c-kube-api-access-jkw46\") pod \"e2e-ait-8d386766-openshift-default-575667689c-n858k\" (UID: \"07ba66ba-7067-4861-82ec-cf8a50a8550c\") " pod="openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k" Oct 31 05:18:15.766662 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:15.766557 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07ba66ba-7067-4861-82ec-cf8a50a8550c-istio-podinfo\") pod \"e2e-ait-8d386766-openshift-default-575667689c-n858k\" (UID: \"07ba66ba-7067-4861-82ec-cf8a50a8550c\") " pod="openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k" Oct 31 05:18:15.766662 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:15.766594 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07ba66ba-7067-4861-82ec-cf8a50a8550c-workload-socket\") pod \"e2e-ait-8d386766-openshift-default-575667689c-n858k\" (UID: \"07ba66ba-7067-4861-82ec-cf8a50a8550c\") " pod="openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k" Oct 31 05:18:15.766662 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:15.766622 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07ba66ba-7067-4861-82ec-cf8a50a8550c-credential-socket\") pod \"e2e-ait-8d386766-openshift-default-575667689c-n858k\" (UID: \"07ba66ba-7067-4861-82ec-cf8a50a8550c\") " pod="openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k" Oct 31 05:18:15.766888 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:15.766664 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07ba66ba-7067-4861-82ec-cf8a50a8550c-istio-envoy\") pod \"e2e-ait-8d386766-openshift-default-575667689c-n858k\" (UID: \"07ba66ba-7067-4861-82ec-cf8a50a8550c\") " pod="openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k" Oct 31 05:18:15.766888 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:15.766854 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07ba66ba-7067-4861-82ec-cf8a50a8550c-workload-certs\") pod \"e2e-ait-8d386766-openshift-default-575667689c-n858k\" (UID: \"07ba66ba-7067-4861-82ec-cf8a50a8550c\") " pod="openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k" Oct 31 05:18:15.767038 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:15.767008 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07ba66ba-7067-4861-82ec-cf8a50a8550c-credential-socket\") pod \"e2e-ait-8d386766-openshift-default-575667689c-n858k\" (UID: \"07ba66ba-7067-4861-82ec-cf8a50a8550c\") " pod="openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k" Oct 31 05:18:15.767191 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:15.767099 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07ba66ba-7067-4861-82ec-cf8a50a8550c-workload-socket\") pod \"e2e-ait-8d386766-openshift-default-575667689c-n858k\" (UID: \"07ba66ba-7067-4861-82ec-cf8a50a8550c\") " pod="openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k" Oct 31 05:18:15.767191 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:15.767119 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07ba66ba-7067-4861-82ec-cf8a50a8550c-istio-data\") pod \"e2e-ait-8d386766-openshift-default-575667689c-n858k\" (UID: \"07ba66ba-7067-4861-82ec-cf8a50a8550c\") " pod="openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k" Oct 31 05:18:15.767191 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:15.767133 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07ba66ba-7067-4861-82ec-cf8a50a8550c-istiod-ca-cert\") pod \"e2e-ait-8d386766-openshift-default-575667689c-n858k\" (UID: \"07ba66ba-7067-4861-82ec-cf8a50a8550c\") " pod="openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k" Oct 31 05:18:15.769007 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:15.768983 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07ba66ba-7067-4861-82ec-cf8a50a8550c-istio-envoy\") pod \"e2e-ait-8d386766-openshift-default-575667689c-n858k\" (UID: \"07ba66ba-7067-4861-82ec-cf8a50a8550c\") " pod="openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k" Oct 31 05:18:15.769387 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:15.769364 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07ba66ba-7067-4861-82ec-cf8a50a8550c-istio-podinfo\") pod \"e2e-ait-8d386766-openshift-default-575667689c-n858k\" (UID: \"07ba66ba-7067-4861-82ec-cf8a50a8550c\") " pod="openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k" Oct 31 05:18:15.790500 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:15.790461 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07ba66ba-7067-4861-82ec-cf8a50a8550c-istio-token\") pod \"e2e-ait-8d386766-openshift-default-575667689c-n858k\" (UID: \"07ba66ba-7067-4861-82ec-cf8a50a8550c\") " pod="openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k" Oct 31 05:18:15.790707 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:15.790602 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkw46\" (UniqueName: \"kubernetes.io/projected/07ba66ba-7067-4861-82ec-cf8a50a8550c-kube-api-access-jkw46\") pod \"e2e-ait-8d386766-openshift-default-575667689c-n858k\" (UID: \"07ba66ba-7067-4861-82ec-cf8a50a8550c\") " pod="openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k" Oct 31 05:18:15.897609 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:15.897477 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k" Oct 31 05:18:16.218131 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:16.218094 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k"] Oct 31 05:18:16.219725 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:18:16.219695 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07ba66ba_7067_4861_82ec_cf8a50a8550c.slice/crio-7a86366487c9c4b49031ccee6edf810cc6215ecd0a441a427368ac53f471e334 WatchSource:0}: Error finding container 7a86366487c9c4b49031ccee6edf810cc6215ecd0a441a427368ac53f471e334: Status 404 returned error can't find the container with id 7a86366487c9c4b49031ccee6edf810cc6215ecd0a441a427368ac53f471e334 Oct 31 05:18:16.223897 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:16.222524 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:18:16.223897 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:16.222710 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:18:16.223897 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:16.222825 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:18:16.590231 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:16.590121 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k" event={"ID":"07ba66ba-7067-4861-82ec-cf8a50a8550c","Type":"ContainerStarted","Data":"f551717a6662ebf006462960c7e8057780b3a4aaf885dcfa59030d90813602ad"} Oct 31 05:18:16.590231 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:16.590191 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k" event={"ID":"07ba66ba-7067-4861-82ec-cf8a50a8550c","Type":"ContainerStarted","Data":"7a86366487c9c4b49031ccee6edf810cc6215ecd0a441a427368ac53f471e334"} Oct 31 05:18:16.661515 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:16.661438 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k" podStartSLOduration=1.661418382 podStartE2EDuration="1.661418382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:18:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:18:16.647280403 +0000 UTC m=+4451.403618442" watchObservedRunningTime="2026-10-31 05:18:16.661418382 +0000 UTC m=+4451.417756423" Oct 31 05:18:16.898139 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:16.898013 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k" Oct 31 05:18:17.903884 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:17.903846 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k" Oct 31 05:18:18.603388 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:18.603312 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k" Oct 31 05:18:18.604803 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:18.604768 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k" Oct 31 05:18:20.071580 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:20.071530 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8d386766-564db46ccf-p7vc2"] Oct 31 05:18:20.076015 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:20.075984 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8d386766-564db46ccf-p7vc2" Oct 31 05:18:20.076598 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:20.076573 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8d386766-9d98995bc-8srn5"] Oct 31 05:18:20.080363 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:20.080306 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8d386766-9d98995bc-8srn5" Oct 31 05:18:20.094319 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:20.094277 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-8d386766\"" Oct 31 05:18:20.096608 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:20.096577 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8d386766-564db46ccf-p7vc2"] Oct 31 05:18:20.107936 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:20.107891 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8d386766-9d98995bc-8srn5"] Oct 31 05:18:20.213621 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:20.213574 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfe8f932-8152-4c9a-88d7-3de878efee17-plugins-config-volume\") pod \"payload-processing-e2e-ait-8d386766-564db46ccf-p7vc2\" (UID: \"dfe8f932-8152-4c9a-88d7-3de878efee17\") " pod="openshift-ingress/payload-processing-e2e-ait-8d386766-564db46ccf-p7vc2" Oct 31 05:18:20.213860 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:20.213693 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/521dab1e-e502-478c-84f5-b5735c15064f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8d386766-9d98995bc-8srn5\" (UID: \"521dab1e-e502-478c-84f5-b5735c15064f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8d386766-9d98995bc-8srn5" Oct 31 05:18:20.213860 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:20.213743 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvgsc\" (UniqueName: \"kubernetes.io/projected/dfe8f932-8152-4c9a-88d7-3de878efee17-kube-api-access-dvgsc\") pod \"payload-processing-e2e-ait-8d386766-564db46ccf-p7vc2\" (UID: \"dfe8f932-8152-4c9a-88d7-3de878efee17\") " pod="openshift-ingress/payload-processing-e2e-ait-8d386766-564db46ccf-p7vc2" Oct 31 05:18:20.213860 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:20.213801 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd7qz\" (UniqueName: \"kubernetes.io/projected/521dab1e-e502-478c-84f5-b5735c15064f-kube-api-access-pd7qz\") pod \"payload-pre-processing-e2e-ait-8d386766-9d98995bc-8srn5\" (UID: \"521dab1e-e502-478c-84f5-b5735c15064f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8d386766-9d98995bc-8srn5" Oct 31 05:18:20.314449 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:20.314403 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfe8f932-8152-4c9a-88d7-3de878efee17-plugins-config-volume\") pod \"payload-processing-e2e-ait-8d386766-564db46ccf-p7vc2\" (UID: \"dfe8f932-8152-4c9a-88d7-3de878efee17\") " pod="openshift-ingress/payload-processing-e2e-ait-8d386766-564db46ccf-p7vc2" Oct 31 05:18:20.314668 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:20.314486 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/521dab1e-e502-478c-84f5-b5735c15064f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8d386766-9d98995bc-8srn5\" (UID: \"521dab1e-e502-478c-84f5-b5735c15064f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8d386766-9d98995bc-8srn5" Oct 31 05:18:20.314668 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:20.314516 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvgsc\" (UniqueName: \"kubernetes.io/projected/dfe8f932-8152-4c9a-88d7-3de878efee17-kube-api-access-dvgsc\") pod \"payload-processing-e2e-ait-8d386766-564db46ccf-p7vc2\" (UID: \"dfe8f932-8152-4c9a-88d7-3de878efee17\") " pod="openshift-ingress/payload-processing-e2e-ait-8d386766-564db46ccf-p7vc2" Oct 31 05:18:20.314811 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:20.314654 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd7qz\" (UniqueName: \"kubernetes.io/projected/521dab1e-e502-478c-84f5-b5735c15064f-kube-api-access-pd7qz\") pod \"payload-pre-processing-e2e-ait-8d386766-9d98995bc-8srn5\" (UID: \"521dab1e-e502-478c-84f5-b5735c15064f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8d386766-9d98995bc-8srn5" Oct 31 05:18:20.315289 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:20.315267 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/521dab1e-e502-478c-84f5-b5735c15064f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8d386766-9d98995bc-8srn5\" (UID: \"521dab1e-e502-478c-84f5-b5735c15064f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8d386766-9d98995bc-8srn5" Oct 31 05:18:20.315396 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:20.315277 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfe8f932-8152-4c9a-88d7-3de878efee17-plugins-config-volume\") pod \"payload-processing-e2e-ait-8d386766-564db46ccf-p7vc2\" (UID: \"dfe8f932-8152-4c9a-88d7-3de878efee17\") " pod="openshift-ingress/payload-processing-e2e-ait-8d386766-564db46ccf-p7vc2" Oct 31 05:18:20.338745 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:20.338666 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd7qz\" (UniqueName: \"kubernetes.io/projected/521dab1e-e502-478c-84f5-b5735c15064f-kube-api-access-pd7qz\") pod \"payload-pre-processing-e2e-ait-8d386766-9d98995bc-8srn5\" (UID: \"521dab1e-e502-478c-84f5-b5735c15064f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8d386766-9d98995bc-8srn5" Oct 31 05:18:20.339051 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:20.339031 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvgsc\" (UniqueName: \"kubernetes.io/projected/dfe8f932-8152-4c9a-88d7-3de878efee17-kube-api-access-dvgsc\") pod \"payload-processing-e2e-ait-8d386766-564db46ccf-p7vc2\" (UID: \"dfe8f932-8152-4c9a-88d7-3de878efee17\") " pod="openshift-ingress/payload-processing-e2e-ait-8d386766-564db46ccf-p7vc2" Oct 31 05:18:20.392869 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:20.392825 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8d386766-564db46ccf-p7vc2" Oct 31 05:18:20.401320 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:20.401288 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8d386766-9d98995bc-8srn5" Oct 31 05:18:20.702262 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:20.702224 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8d386766-564db46ccf-p7vc2"] Oct 31 05:18:20.703571 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:18:20.703528 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfe8f932_8152_4c9a_88d7_3de878efee17.slice/crio-93fd1337d4eee385409407a2d5d244a69c1b44b23d61f18ec1867e53b1c16963 WatchSource:0}: Error finding container 93fd1337d4eee385409407a2d5d244a69c1b44b23d61f18ec1867e53b1c16963: Status 404 returned error can't find the container with id 93fd1337d4eee385409407a2d5d244a69c1b44b23d61f18ec1867e53b1c16963 Oct 31 05:18:20.711177 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:20.711144 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8d386766-9d98995bc-8srn5"] Oct 31 05:18:20.716284 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:18:20.716251 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod521dab1e_e502_478c_84f5_b5735c15064f.slice/crio-1bea3acf297522ddef49aab3faf8ccf4feb0b1754c0a653710fc388153a169b2 WatchSource:0}: Error finding container 1bea3acf297522ddef49aab3faf8ccf4feb0b1754c0a653710fc388153a169b2: Status 404 returned error can't find the container with id 1bea3acf297522ddef49aab3faf8ccf4feb0b1754c0a653710fc388153a169b2 Oct 31 05:18:21.624943 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:21.624889 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8d386766-564db46ccf-p7vc2" event={"ID":"dfe8f932-8152-4c9a-88d7-3de878efee17","Type":"ContainerStarted","Data":"48adfb6861754ef94491d21c2b659c722ac7db662f6019839ede6542ec18b101"} Oct 31 05:18:21.625504 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:21.624956 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8d386766-564db46ccf-p7vc2" event={"ID":"dfe8f932-8152-4c9a-88d7-3de878efee17","Type":"ContainerStarted","Data":"93fd1337d4eee385409407a2d5d244a69c1b44b23d61f18ec1867e53b1c16963"} Oct 31 05:18:21.625504 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:21.625008 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-8d386766-564db46ccf-p7vc2" Oct 31 05:18:21.626651 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:21.626619 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8d386766-9d98995bc-8srn5" event={"ID":"521dab1e-e502-478c-84f5-b5735c15064f","Type":"ContainerStarted","Data":"accd6730dc2917e1dfbb48c31c447e7e45919ca86e6075430d22e46b199a23f9"} Oct 31 05:18:21.626777 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:21.626673 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8d386766-9d98995bc-8srn5" event={"ID":"521dab1e-e502-478c-84f5-b5735c15064f","Type":"ContainerStarted","Data":"1bea3acf297522ddef49aab3faf8ccf4feb0b1754c0a653710fc388153a169b2"} Oct 31 05:18:21.626844 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:21.626773 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8d386766-9d98995bc-8srn5" Oct 31 05:18:21.665669 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:21.665608 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-8d386766-564db46ccf-p7vc2" podStartSLOduration=1.6655939320000002 podStartE2EDuration="1.665593932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:18:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:18:21.664662128 +0000 UTC m=+4456.421000170" watchObservedRunningTime="2026-10-31 05:18:21.665593932 +0000 UTC m=+4456.421931971" Oct 31 05:18:21.704630 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:21.704549 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-8d386766-9d98995bc-8srn5" podStartSLOduration=1.704531282 podStartE2EDuration="1.704531282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:18:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:18:21.703553323 +0000 UTC m=+4456.459891364" watchObservedRunningTime="2026-10-31 05:18:21.704531282 +0000 UTC m=+4456.460869322" Oct 31 05:18:22.040349 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:22.040287 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01c821fb-shvcn"] Oct 31 05:18:22.043252 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:22.043209 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01c821fb-shvcn"] Oct 31 05:18:22.392473 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:18:22.392382 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:18:23.406076 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:23.406036 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa224080-7d7b-4c02-b09f-b8dd69076408" path="/var/lib/kubelet/pods/aa224080-7d7b-4c02-b09f-b8dd69076408/volumes" Oct 31 05:18:27.029666 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:27.029625 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a182845-st9jv"] Oct 31 05:18:27.041830 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:27.041790 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a182845-st9jv"] Oct 31 05:18:27.409292 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:27.409200 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bea695c-448e-42b8-bc47-d381c431e1c8" path="/var/lib/kubelet/pods/2bea695c-448e-42b8-bc47-d381c431e1c8/volumes" Oct 31 05:18:32.637450 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:32.637406 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8d386766-9d98995bc-8srn5" Oct 31 05:18:32.637875 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:32.637521 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-8d386766-564db46ccf-p7vc2" Oct 31 05:18:33.605789 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:33.605688 5019 patch_prober.go:29] interesting pod/e2e-ait-8d386766-openshift-default-575667689c-n858k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.212:15021/healthz/ready\": dial tcp 10.132.1.212:15021: connect: connection refused" start-of-body= Oct 31 05:18:33.605789 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:33.605757 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k" podUID="07ba66ba-7067-4861-82ec-cf8a50a8550c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.212:15021/healthz/ready\": dial tcp 10.132.1.212:15021: connect: connection refused" Oct 31 05:18:33.735904 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:33.735864 5019 generic.go:356] "Generic (PLEG): container finished" podID="07ba66ba-7067-4861-82ec-cf8a50a8550c" containerID="f551717a6662ebf006462960c7e8057780b3a4aaf885dcfa59030d90813602ad" exitCode=0 Oct 31 05:18:33.736366 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:33.735940 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k" event={"ID":"07ba66ba-7067-4861-82ec-cf8a50a8550c","Type":"ContainerDied","Data":"f551717a6662ebf006462960c7e8057780b3a4aaf885dcfa59030d90813602ad"} Oct 31 05:18:33.736424 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:33.736379 5019 scope.go:122] "RemoveContainer" containerID="f551717a6662ebf006462960c7e8057780b3a4aaf885dcfa59030d90813602ad" Oct 31 05:18:33.737732 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:33.737698 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:18:33.737882 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:33.737794 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:18:33.737882 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:33.737851 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:18:34.285355 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:34.278729 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d386766-hdf8l"] Oct 31 05:18:34.285355 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:34.283712 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d386766-hdf8l" Oct 31 05:18:34.299385 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:34.299322 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d386766-hdf8l"] Oct 31 05:18:34.299706 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:34.299684 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:18:34.299815 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:34.299803 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\"" Oct 31 05:18:34.299863 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:34.299843 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:18:34.300002 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:34.299990 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:18:34.394392 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:34.392977 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/383b618d-21bb-4a9c-876e-271d7252aba6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8d386766-hdf8l\" (UID: \"383b618d-21bb-4a9c-876e-271d7252aba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d386766-hdf8l" Oct 31 05:18:34.394644 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:34.394561 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd2tz\" (UniqueName: \"kubernetes.io/projected/383b618d-21bb-4a9c-876e-271d7252aba6-kube-api-access-xd2tz\") pod \"maas-api-revoke-keys-e2e-ait-8d386766-hdf8l\" (UID: \"383b618d-21bb-4a9c-876e-271d7252aba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d386766-hdf8l" Oct 31 05:18:34.394816 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:34.394783 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/383b618d-21bb-4a9c-876e-271d7252aba6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8d386766-hdf8l\" (UID: \"383b618d-21bb-4a9c-876e-271d7252aba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d386766-hdf8l" Oct 31 05:18:34.495491 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:34.495441 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/383b618d-21bb-4a9c-876e-271d7252aba6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8d386766-hdf8l\" (UID: \"383b618d-21bb-4a9c-876e-271d7252aba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d386766-hdf8l" Oct 31 05:18:34.495491 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:34.495495 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/383b618d-21bb-4a9c-876e-271d7252aba6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8d386766-hdf8l\" (UID: \"383b618d-21bb-4a9c-876e-271d7252aba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d386766-hdf8l" Oct 31 05:18:34.495830 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:34.495776 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd2tz\" (UniqueName: \"kubernetes.io/projected/383b618d-21bb-4a9c-876e-271d7252aba6-kube-api-access-xd2tz\") pod \"maas-api-revoke-keys-e2e-ait-8d386766-hdf8l\" (UID: \"383b618d-21bb-4a9c-876e-271d7252aba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d386766-hdf8l" Oct 31 05:18:34.496513 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:34.496488 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/383b618d-21bb-4a9c-876e-271d7252aba6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8d386766-hdf8l\" (UID: \"383b618d-21bb-4a9c-876e-271d7252aba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d386766-hdf8l" Oct 31 05:18:34.498886 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:34.498849 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/383b618d-21bb-4a9c-876e-271d7252aba6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8d386766-hdf8l\" (UID: \"383b618d-21bb-4a9c-876e-271d7252aba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d386766-hdf8l" Oct 31 05:18:34.524498 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:18:34.524450 5019 projected.go:196] Error preparing data for projected volume kube-api-access-xd2tz for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d386766-hdf8l: failed to fetch token: serviceaccounts "maas-api-cleanup" not found Oct 31 05:18:34.525148 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:18:34.525128 5019 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/383b618d-21bb-4a9c-876e-271d7252aba6-kube-api-access-xd2tz podName:383b618d-21bb-4a9c-876e-271d7252aba6 nodeName:}" failed. No retries permitted until 2026-10-31 05:18:35.025099962 +0000 UTC m=+4469.781437980 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xd2tz" (UniqueName: "kubernetes.io/projected/383b618d-21bb-4a9c-876e-271d7252aba6-kube-api-access-xd2tz") pod "maas-api-revoke-keys-e2e-ait-8d386766-hdf8l" (UID: "383b618d-21bb-4a9c-876e-271d7252aba6") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found Oct 31 05:18:34.750987 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:34.750941 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k" event={"ID":"07ba66ba-7067-4861-82ec-cf8a50a8550c","Type":"ContainerStarted","Data":"8058a6344b055a51422441c497ed46d0db5ad6995924c098b7f4b6b69776923d"} Oct 31 05:18:34.898717 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:34.898668 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k" Oct 31 05:18:34.898982 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:34.898953 5019 patch_prober.go:29] interesting pod/e2e-ait-8d386766-openshift-default-575667689c-n858k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.212:15021/healthz/ready\": dial tcp 10.132.1.212:15021: connect: connection refused" start-of-body= Oct 31 05:18:34.899038 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:34.899013 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k" podUID="07ba66ba-7067-4861-82ec-cf8a50a8550c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.212:15021/healthz/ready\": dial tcp 10.132.1.212:15021: connect: connection refused" Oct 31 05:18:35.101479 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:35.101382 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd2tz\" (UniqueName: \"kubernetes.io/projected/383b618d-21bb-4a9c-876e-271d7252aba6-kube-api-access-xd2tz\") pod \"maas-api-revoke-keys-e2e-ait-8d386766-hdf8l\" (UID: \"383b618d-21bb-4a9c-876e-271d7252aba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d386766-hdf8l" Oct 31 05:18:35.125034 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:35.124998 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd2tz\" (UniqueName: \"kubernetes.io/projected/383b618d-21bb-4a9c-876e-271d7252aba6-kube-api-access-xd2tz\") pod \"maas-api-revoke-keys-e2e-ait-8d386766-hdf8l\" (UID: \"383b618d-21bb-4a9c-876e-271d7252aba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d386766-hdf8l" Oct 31 05:18:35.215811 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:35.215761 5019 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d386766-hdf8l" secret="" err="secret \"maas-api-cleanup-dockercfg-lzb8r\" not found" Oct 31 05:18:35.216048 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:35.215835 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d386766-hdf8l" Oct 31 05:18:35.539230 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:35.539194 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d386766-hdf8l"] Oct 31 05:18:35.540595 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:18:35.540560 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod383b618d_21bb_4a9c_876e_271d7252aba6.slice/crio-beb5b904f954061321f9c28814bdaac6de0f4553b80b07096483bafc5c730e0b WatchSource:0}: Error finding container beb5b904f954061321f9c28814bdaac6de0f4553b80b07096483bafc5c730e0b: Status 404 returned error can't find the container with id beb5b904f954061321f9c28814bdaac6de0f4553b80b07096483bafc5c730e0b Oct 31 05:18:35.758663 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:35.758621 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d386766-hdf8l" event={"ID":"383b618d-21bb-4a9c-876e-271d7252aba6","Type":"ContainerStarted","Data":"beb5b904f954061321f9c28814bdaac6de0f4553b80b07096483bafc5c730e0b"} Oct 31 05:18:35.898988 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:35.898945 5019 patch_prober.go:29] interesting pod/e2e-ait-8d386766-openshift-default-575667689c-n858k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.212:15021/healthz/ready\": dial tcp 10.132.1.212:15021: connect: connection refused" start-of-body= Oct 31 05:18:35.899236 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:35.899019 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k" podUID="07ba66ba-7067-4861-82ec-cf8a50a8550c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.212:15021/healthz/ready\": dial tcp 10.132.1.212:15021: connect: connection refused" Oct 31 05:18:36.767093 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:36.767048 5019 generic.go:356] "Generic (PLEG): container finished" podID="383b618d-21bb-4a9c-876e-271d7252aba6" containerID="b474c63ad7116fee40a6696d91c253030453da7b8b5adf44a740aa9b711ca602" exitCode=0 Oct 31 05:18:36.767630 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:36.767130 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d386766-hdf8l" event={"ID":"383b618d-21bb-4a9c-876e-271d7252aba6","Type":"ContainerDied","Data":"b474c63ad7116fee40a6696d91c253030453da7b8b5adf44a740aa9b711ca602"} Oct 31 05:18:36.899050 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:36.898947 5019 patch_prober.go:29] interesting pod/e2e-ait-8d386766-openshift-default-575667689c-n858k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.212:15021/healthz/ready\": dial tcp 10.132.1.212:15021: connect: connection refused" start-of-body= Oct 31 05:18:36.899050 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:36.899014 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k" podUID="07ba66ba-7067-4861-82ec-cf8a50a8550c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.212:15021/healthz/ready\": dial tcp 10.132.1.212:15021: connect: connection refused" Oct 31 05:18:37.898416 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:37.898301 5019 patch_prober.go:29] interesting pod/e2e-ait-8d386766-openshift-default-575667689c-n858k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.212:15021/healthz/ready\": dial tcp 10.132.1.212:15021: connect: connection refused" start-of-body= Oct 31 05:18:37.899797 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:37.899250 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k" podUID="07ba66ba-7067-4861-82ec-cf8a50a8550c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.212:15021/healthz/ready\": dial tcp 10.132.1.212:15021: connect: connection refused" Oct 31 05:18:37.962235 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:37.962195 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d386766-hdf8l" Oct 31 05:18:38.029489 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:38.029449 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/383b618d-21bb-4a9c-876e-271d7252aba6-kube-api-access-xd2tz\" (UniqueName: \"kubernetes.io/projected/383b618d-21bb-4a9c-876e-271d7252aba6-kube-api-access-xd2tz\") pod \"383b618d-21bb-4a9c-876e-271d7252aba6\" (UID: \"383b618d-21bb-4a9c-876e-271d7252aba6\") " Oct 31 05:18:38.029693 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:38.029535 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/383b618d-21bb-4a9c-876e-271d7252aba6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/383b618d-21bb-4a9c-876e-271d7252aba6-maas-api-xks-ca\") pod \"383b618d-21bb-4a9c-876e-271d7252aba6\" (UID: \"383b618d-21bb-4a9c-876e-271d7252aba6\") " Oct 31 05:18:38.029693 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:38.029632 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/383b618d-21bb-4a9c-876e-271d7252aba6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/383b618d-21bb-4a9c-876e-271d7252aba6-maas-api-openshift-ca\") pod \"383b618d-21bb-4a9c-876e-271d7252aba6\" (UID: \"383b618d-21bb-4a9c-876e-271d7252aba6\") " Oct 31 05:18:38.030013 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:38.029990 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/383b618d-21bb-4a9c-876e-271d7252aba6-maas-api-openshift-ca" pod "383b618d-21bb-4a9c-876e-271d7252aba6" (UID: "383b618d-21bb-4a9c-876e-271d7252aba6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:18:38.032003 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:38.031961 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/383b618d-21bb-4a9c-876e-271d7252aba6-maas-api-xks-ca" pod "383b618d-21bb-4a9c-876e-271d7252aba6" (UID: "383b618d-21bb-4a9c-876e-271d7252aba6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:18:38.032252 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:38.032225 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/383b618d-21bb-4a9c-876e-271d7252aba6-kube-api-access-xd2tz" pod "383b618d-21bb-4a9c-876e-271d7252aba6" (UID: "383b618d-21bb-4a9c-876e-271d7252aba6"). InnerVolumeSpecName "kube-api-access-xd2tz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:18:38.130645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:38.130543 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd2tz\" (UniqueName: \"kubernetes.io/projected/383b618d-21bb-4a9c-876e-271d7252aba6-kube-api-access-xd2tz\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:18:38.130645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:38.130583 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/383b618d-21bb-4a9c-876e-271d7252aba6-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:18:38.130645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:38.130593 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/383b618d-21bb-4a9c-876e-271d7252aba6-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:18:38.806393 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:38.806353 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d386766-hdf8l" Oct 31 05:18:38.806393 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:38.806371 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d386766-hdf8l" event={"ID":"383b618d-21bb-4a9c-876e-271d7252aba6","Type":"ContainerDied","Data":"beb5b904f954061321f9c28814bdaac6de0f4553b80b07096483bafc5c730e0b"} Oct 31 05:18:38.806643 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:38.806428 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="beb5b904f954061321f9c28814bdaac6de0f4553b80b07096483bafc5c730e0b" Oct 31 05:18:38.897922 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:38.897876 5019 patch_prober.go:29] interesting pod/e2e-ait-8d386766-openshift-default-575667689c-n858k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.212:15021/healthz/ready\": dial tcp 10.132.1.212:15021: connect: connection refused" start-of-body= Oct 31 05:18:38.898132 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:38.897940 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k" podUID="07ba66ba-7067-4861-82ec-cf8a50a8550c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.212:15021/healthz/ready\": dial tcp 10.132.1.212:15021: connect: connection refused" Oct 31 05:18:39.898876 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:39.898834 5019 patch_prober.go:29] interesting pod/e2e-ait-8d386766-openshift-default-575667689c-n858k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.212:15021/healthz/ready\": dial tcp 10.132.1.212:15021: connect: connection refused" start-of-body= Oct 31 05:18:39.899380 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:39.898910 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k" podUID="07ba66ba-7067-4861-82ec-cf8a50a8550c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.212:15021/healthz/ready\": dial tcp 10.132.1.212:15021: connect: connection refused" Oct 31 05:18:40.902927 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:40.902887 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k" Oct 31 05:18:40.903450 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:40.903186 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k" Oct 31 05:18:40.904052 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:40.904030 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k" Oct 31 05:18:44.706639 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:44.706592 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8d386766-564db46ccf-p7vc2"] Oct 31 05:18:44.736379 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:44.736323 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8d386766-9d98995bc-8srn5"] Oct 31 05:18:47.870923 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:47.870877 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d386766-hdf8l"] Oct 31 05:18:47.883183 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:47.883139 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d386766-hdf8l"] Oct 31 05:18:49.407078 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:49.407039 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="383b618d-21bb-4a9c-876e-271d7252aba6" path="/var/lib/kubelet/pods/383b618d-21bb-4a9c-876e-271d7252aba6/volumes" Oct 31 05:18:49.707816 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:49.707650 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-8d386766-564db46ccf-p7vc2" podUID="dfe8f932-8152-4c9a-88d7-3de878efee17" containerName="payload-processing" containerID="cri-o://48adfb6861754ef94491d21c2b659c722ac7db662f6019839ede6542ec18b101" gracePeriod=25 Oct 31 05:18:49.737081 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:49.737026 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-8d386766-9d98995bc-8srn5" podUID="521dab1e-e502-478c-84f5-b5735c15064f" containerName="payload-pre-processing" containerID="cri-o://accd6730dc2917e1dfbb48c31c447e7e45919ca86e6075430d22e46b199a23f9" gracePeriod=25 Oct 31 05:18:49.888811 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:49.888653 5019 generic.go:356] "Generic (PLEG): container finished" podID="521dab1e-e502-478c-84f5-b5735c15064f" containerID="accd6730dc2917e1dfbb48c31c447e7e45919ca86e6075430d22e46b199a23f9" exitCode=0 Oct 31 05:18:49.888811 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:49.888753 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8d386766-9d98995bc-8srn5" event={"ID":"521dab1e-e502-478c-84f5-b5735c15064f","Type":"ContainerDied","Data":"accd6730dc2917e1dfbb48c31c447e7e45919ca86e6075430d22e46b199a23f9"} Oct 31 05:18:49.892617 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:49.892440 5019 generic.go:356] "Generic (PLEG): container finished" podID="dfe8f932-8152-4c9a-88d7-3de878efee17" containerID="48adfb6861754ef94491d21c2b659c722ac7db662f6019839ede6542ec18b101" exitCode=0 Oct 31 05:18:49.892617 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:49.892514 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8d386766-564db46ccf-p7vc2" event={"ID":"dfe8f932-8152-4c9a-88d7-3de878efee17","Type":"ContainerDied","Data":"48adfb6861754ef94491d21c2b659c722ac7db662f6019839ede6542ec18b101"} Oct 31 05:18:50.000833 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:50.000797 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8d386766-564db46ccf-p7vc2" Oct 31 05:18:50.034150 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:50.034116 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8d386766-9d98995bc-8srn5" Oct 31 05:18:50.055730 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:50.055693 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfe8f932-8152-4c9a-88d7-3de878efee17-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfe8f932-8152-4c9a-88d7-3de878efee17-plugins-config-volume\") pod \"dfe8f932-8152-4c9a-88d7-3de878efee17\" (UID: \"dfe8f932-8152-4c9a-88d7-3de878efee17\") " Oct 31 05:18:50.055923 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:50.055744 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfe8f932-8152-4c9a-88d7-3de878efee17-kube-api-access-dvgsc\" (UniqueName: \"kubernetes.io/projected/dfe8f932-8152-4c9a-88d7-3de878efee17-kube-api-access-dvgsc\") pod \"dfe8f932-8152-4c9a-88d7-3de878efee17\" (UID: \"dfe8f932-8152-4c9a-88d7-3de878efee17\") " Oct 31 05:18:50.056164 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:50.056143 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfe8f932-8152-4c9a-88d7-3de878efee17-plugins-config-volume" pod "dfe8f932-8152-4c9a-88d7-3de878efee17" (UID: "dfe8f932-8152-4c9a-88d7-3de878efee17"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:18:50.058044 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:50.058014 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfe8f932-8152-4c9a-88d7-3de878efee17-kube-api-access-dvgsc" pod "dfe8f932-8152-4c9a-88d7-3de878efee17" (UID: "dfe8f932-8152-4c9a-88d7-3de878efee17"). InnerVolumeSpecName "kube-api-access-dvgsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:18:50.156791 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:50.156744 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/521dab1e-e502-478c-84f5-b5735c15064f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/521dab1e-e502-478c-84f5-b5735c15064f-plugins-config-volume\") pod \"521dab1e-e502-478c-84f5-b5735c15064f\" (UID: \"521dab1e-e502-478c-84f5-b5735c15064f\") " Oct 31 05:18:50.157020 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:50.156848 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/521dab1e-e502-478c-84f5-b5735c15064f-kube-api-access-pd7qz\" (UniqueName: \"kubernetes.io/projected/521dab1e-e502-478c-84f5-b5735c15064f-kube-api-access-pd7qz\") pod \"521dab1e-e502-478c-84f5-b5735c15064f\" (UID: \"521dab1e-e502-478c-84f5-b5735c15064f\") " Oct 31 05:18:50.157108 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:50.157054 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfe8f932-8152-4c9a-88d7-3de878efee17-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:18:50.157108 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:50.157066 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvgsc\" (UniqueName: \"kubernetes.io/projected/dfe8f932-8152-4c9a-88d7-3de878efee17-kube-api-access-dvgsc\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:18:50.157278 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:50.157248 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/521dab1e-e502-478c-84f5-b5735c15064f-plugins-config-volume" pod "521dab1e-e502-478c-84f5-b5735c15064f" (UID: "521dab1e-e502-478c-84f5-b5735c15064f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:18:50.159146 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:50.159117 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/521dab1e-e502-478c-84f5-b5735c15064f-kube-api-access-pd7qz" pod "521dab1e-e502-478c-84f5-b5735c15064f" (UID: "521dab1e-e502-478c-84f5-b5735c15064f"). InnerVolumeSpecName "kube-api-access-pd7qz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:18:50.258318 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:50.258219 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd7qz\" (UniqueName: \"kubernetes.io/projected/521dab1e-e502-478c-84f5-b5735c15064f-kube-api-access-pd7qz\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:18:50.258318 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:50.258252 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/521dab1e-e502-478c-84f5-b5735c15064f-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:18:50.551256 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:50.551144 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k"] Oct 31 05:18:50.551813 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:50.551449 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k" podUID="07ba66ba-7067-4861-82ec-cf8a50a8550c" containerName="istio-proxy" containerID="cri-o://8058a6344b055a51422441c497ed46d0db5ad6995924c098b7f4b6b69776923d" gracePeriod=30 Oct 31 05:18:50.900450 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:50.900357 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8d386766-9d98995bc-8srn5" Oct 31 05:18:50.900648 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:50.900357 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8d386766-9d98995bc-8srn5" event={"ID":"521dab1e-e502-478c-84f5-b5735c15064f","Type":"ContainerDied","Data":"1bea3acf297522ddef49aab3faf8ccf4feb0b1754c0a653710fc388153a169b2"} Oct 31 05:18:50.900648 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:50.900540 5019 scope.go:122] "RemoveContainer" containerID="accd6730dc2917e1dfbb48c31c447e7e45919ca86e6075430d22e46b199a23f9" Oct 31 05:18:50.902102 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:50.902077 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8d386766-564db46ccf-p7vc2" event={"ID":"dfe8f932-8152-4c9a-88d7-3de878efee17","Type":"ContainerDied","Data":"93fd1337d4eee385409407a2d5d244a69c1b44b23d61f18ec1867e53b1c16963"} Oct 31 05:18:50.902234 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:50.902117 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8d386766-564db46ccf-p7vc2" Oct 31 05:18:50.913594 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:50.913569 5019 scope.go:122] "RemoveContainer" containerID="48adfb6861754ef94491d21c2b659c722ac7db662f6019839ede6542ec18b101" Oct 31 05:18:50.951632 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:50.951589 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8d386766-9d98995bc-8srn5"] Oct 31 05:18:50.956066 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:50.956028 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8d386766-9d98995bc-8srn5"] Oct 31 05:18:50.989094 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:50.989042 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8d386766-564db46ccf-p7vc2"] Oct 31 05:18:50.992868 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:50.992830 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8d386766-564db46ccf-p7vc2"] Oct 31 05:18:51.416581 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:51.416541 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="521dab1e-e502-478c-84f5-b5735c15064f" path="/var/lib/kubelet/pods/521dab1e-e502-478c-84f5-b5735c15064f/volumes" Oct 31 05:18:51.416960 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:51.416947 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfe8f932-8152-4c9a-88d7-3de878efee17" path="/var/lib/kubelet/pods/dfe8f932-8152-4c9a-88d7-3de878efee17/volumes" Oct 31 05:18:52.391958 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:18:52.391914 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:18:55.857484 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:55.857446 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k" Oct 31 05:18:55.914374 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:55.914296 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07ba66ba-7067-4861-82ec-cf8a50a8550c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07ba66ba-7067-4861-82ec-cf8a50a8550c-istio-data\") pod \"07ba66ba-7067-4861-82ec-cf8a50a8550c\" (UID: \"07ba66ba-7067-4861-82ec-cf8a50a8550c\") " Oct 31 05:18:55.914609 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:55.914402 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07ba66ba-7067-4861-82ec-cf8a50a8550c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07ba66ba-7067-4861-82ec-cf8a50a8550c-istiod-ca-cert\") pod \"07ba66ba-7067-4861-82ec-cf8a50a8550c\" (UID: \"07ba66ba-7067-4861-82ec-cf8a50a8550c\") " Oct 31 05:18:55.914609 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:55.914476 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07ba66ba-7067-4861-82ec-cf8a50a8550c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07ba66ba-7067-4861-82ec-cf8a50a8550c-credential-socket\") pod \"07ba66ba-7067-4861-82ec-cf8a50a8550c\" (UID: \"07ba66ba-7067-4861-82ec-cf8a50a8550c\") " Oct 31 05:18:55.914773 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:55.914636 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07ba66ba-7067-4861-82ec-cf8a50a8550c-istio-token\" (UniqueName: \"kubernetes.io/projected/07ba66ba-7067-4861-82ec-cf8a50a8550c-istio-token\") pod \"07ba66ba-7067-4861-82ec-cf8a50a8550c\" (UID: \"07ba66ba-7067-4861-82ec-cf8a50a8550c\") " Oct 31 05:18:55.914773 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:55.914694 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07ba66ba-7067-4861-82ec-cf8a50a8550c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07ba66ba-7067-4861-82ec-cf8a50a8550c-workload-socket\") pod \"07ba66ba-7067-4861-82ec-cf8a50a8550c\" (UID: \"07ba66ba-7067-4861-82ec-cf8a50a8550c\") " Oct 31 05:18:55.914773 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:55.914687 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07ba66ba-7067-4861-82ec-cf8a50a8550c-istio-data" pod "07ba66ba-7067-4861-82ec-cf8a50a8550c" (UID: "07ba66ba-7067-4861-82ec-cf8a50a8550c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:18:55.914773 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:55.914757 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07ba66ba-7067-4861-82ec-cf8a50a8550c-credential-socket" pod "07ba66ba-7067-4861-82ec-cf8a50a8550c" (UID: "07ba66ba-7067-4861-82ec-cf8a50a8550c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:18:55.915124 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:55.914752 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07ba66ba-7067-4861-82ec-cf8a50a8550c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07ba66ba-7067-4861-82ec-cf8a50a8550c-istio-envoy\") pod \"07ba66ba-7067-4861-82ec-cf8a50a8550c\" (UID: \"07ba66ba-7067-4861-82ec-cf8a50a8550c\") " Oct 31 05:18:55.915124 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:55.914836 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07ba66ba-7067-4861-82ec-cf8a50a8550c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07ba66ba-7067-4861-82ec-cf8a50a8550c-istio-podinfo\") pod \"07ba66ba-7067-4861-82ec-cf8a50a8550c\" (UID: \"07ba66ba-7067-4861-82ec-cf8a50a8550c\") " Oct 31 05:18:55.915124 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:55.914872 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07ba66ba-7067-4861-82ec-cf8a50a8550c-kube-api-access-jkw46\" (UniqueName: \"kubernetes.io/projected/07ba66ba-7067-4861-82ec-cf8a50a8550c-kube-api-access-jkw46\") pod \"07ba66ba-7067-4861-82ec-cf8a50a8550c\" (UID: \"07ba66ba-7067-4861-82ec-cf8a50a8550c\") " Oct 31 05:18:55.915124 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:55.914920 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07ba66ba-7067-4861-82ec-cf8a50a8550c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07ba66ba-7067-4861-82ec-cf8a50a8550c-workload-certs\") pod \"07ba66ba-7067-4861-82ec-cf8a50a8550c\" (UID: \"07ba66ba-7067-4861-82ec-cf8a50a8550c\") " Oct 31 05:18:55.915124 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:55.915046 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07ba66ba-7067-4861-82ec-cf8a50a8550c-workload-socket" pod "07ba66ba-7067-4861-82ec-cf8a50a8550c" (UID: "07ba66ba-7067-4861-82ec-cf8a50a8550c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:18:55.916431 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:55.916373 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07ba66ba-7067-4861-82ec-cf8a50a8550c-workload-certs" pod "07ba66ba-7067-4861-82ec-cf8a50a8550c" (UID: "07ba66ba-7067-4861-82ec-cf8a50a8550c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:18:55.916604 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:55.916450 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07ba66ba-7067-4861-82ec-cf8a50a8550c-istiod-ca-cert" pod "07ba66ba-7067-4861-82ec-cf8a50a8550c" (UID: "07ba66ba-7067-4861-82ec-cf8a50a8550c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:18:55.916931 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:55.916887 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07ba66ba-7067-4861-82ec-cf8a50a8550c-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:18:55.916931 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:55.916916 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07ba66ba-7067-4861-82ec-cf8a50a8550c-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:18:55.916931 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:55.916931 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07ba66ba-7067-4861-82ec-cf8a50a8550c-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:18:55.917233 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:55.916945 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07ba66ba-7067-4861-82ec-cf8a50a8550c-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:18:55.917233 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:55.916960 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07ba66ba-7067-4861-82ec-cf8a50a8550c-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:18:55.917883 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:55.917846 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07ba66ba-7067-4861-82ec-cf8a50a8550c-kube-api-access-jkw46" pod "07ba66ba-7067-4861-82ec-cf8a50a8550c" (UID: "07ba66ba-7067-4861-82ec-cf8a50a8550c"). InnerVolumeSpecName "kube-api-access-jkw46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:18:55.917991 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:55.917940 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07ba66ba-7067-4861-82ec-cf8a50a8550c-istio-envoy" pod "07ba66ba-7067-4861-82ec-cf8a50a8550c" (UID: "07ba66ba-7067-4861-82ec-cf8a50a8550c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:18:55.920185 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:55.920153 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07ba66ba-7067-4861-82ec-cf8a50a8550c-istio-token" pod "07ba66ba-7067-4861-82ec-cf8a50a8550c" (UID: "07ba66ba-7067-4861-82ec-cf8a50a8550c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:18:55.921147 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:55.920497 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07ba66ba-7067-4861-82ec-cf8a50a8550c-istio-podinfo" pod "07ba66ba-7067-4861-82ec-cf8a50a8550c" (UID: "07ba66ba-7067-4861-82ec-cf8a50a8550c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:18:55.942447 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:55.942400 5019 generic.go:356] "Generic (PLEG): container finished" podID="07ba66ba-7067-4861-82ec-cf8a50a8550c" containerID="8058a6344b055a51422441c497ed46d0db5ad6995924c098b7f4b6b69776923d" exitCode=0 Oct 31 05:18:55.942686 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:55.942481 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k" Oct 31 05:18:55.942686 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:55.942496 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k" event={"ID":"07ba66ba-7067-4861-82ec-cf8a50a8550c","Type":"ContainerDied","Data":"8058a6344b055a51422441c497ed46d0db5ad6995924c098b7f4b6b69776923d"} Oct 31 05:18:55.942686 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:55.942603 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k" event={"ID":"07ba66ba-7067-4861-82ec-cf8a50a8550c","Type":"ContainerDied","Data":"7a86366487c9c4b49031ccee6edf810cc6215ecd0a441a427368ac53f471e334"} Oct 31 05:18:55.942686 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:55.942635 5019 scope.go:122] "RemoveContainer" containerID="8058a6344b055a51422441c497ed46d0db5ad6995924c098b7f4b6b69776923d" Oct 31 05:18:55.942925 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:55.942879 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-85c4f7957c-rzwcg"] Oct 31 05:18:55.945104 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:55.943392 5019 cpu_manager.go:414] "containerMap: removing container" podUID="383b618d-21bb-4a9c-876e-271d7252aba6" containerName="revoke-keys" Oct 31 05:18:55.945104 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:55.943415 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="383b618d-21bb-4a9c-876e-271d7252aba6" containerName="revoke-keys" Oct 31 05:18:55.945104 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:55.943450 5019 cpu_manager.go:414] "containerMap: removing container" podUID="dfe8f932-8152-4c9a-88d7-3de878efee17" containerName="payload-processing" Oct 31 05:18:55.945104 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:55.943459 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfe8f932-8152-4c9a-88d7-3de878efee17" containerName="payload-processing" Oct 31 05:18:55.945104 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:55.943482 5019 cpu_manager.go:414] "containerMap: removing container" podUID="521dab1e-e502-478c-84f5-b5735c15064f" containerName="payload-pre-processing" Oct 31 05:18:55.945104 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:55.943490 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="521dab1e-e502-478c-84f5-b5735c15064f" containerName="payload-pre-processing" Oct 31 05:18:55.945104 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:55.943506 5019 cpu_manager.go:414] "containerMap: removing container" podUID="07ba66ba-7067-4861-82ec-cf8a50a8550c" containerName="istio-proxy" Oct 31 05:18:55.945104 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:55.943515 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07ba66ba-7067-4861-82ec-cf8a50a8550c" containerName="istio-proxy" Oct 31 05:18:55.945104 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:55.943526 5019 cpu_manager.go:414] "containerMap: removing container" podUID="07ba66ba-7067-4861-82ec-cf8a50a8550c" containerName="istio-proxy" Oct 31 05:18:55.945104 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:55.943533 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07ba66ba-7067-4861-82ec-cf8a50a8550c" containerName="istio-proxy" Oct 31 05:18:55.945104 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:55.943656 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfe8f932-8152-4c9a-88d7-3de878efee17" containerName="payload-processing" Oct 31 05:18:55.945104 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:55.943674 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="521dab1e-e502-478c-84f5-b5735c15064f" containerName="payload-pre-processing" Oct 31 05:18:55.945104 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:55.943684 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="07ba66ba-7067-4861-82ec-cf8a50a8550c" containerName="istio-proxy" Oct 31 05:18:55.945104 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:55.943690 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="07ba66ba-7067-4861-82ec-cf8a50a8550c" containerName="istio-proxy" Oct 31 05:18:55.945104 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:55.943700 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="383b618d-21bb-4a9c-876e-271d7252aba6" containerName="revoke-keys" Oct 31 05:18:55.947709 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:55.947683 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-85c4f7957c-rzwcg" Oct 31 05:18:55.955137 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:55.955110 5019 scope.go:122] "RemoveContainer" containerID="f551717a6662ebf006462960c7e8057780b3a4aaf885dcfa59030d90813602ad" Oct 31 05:18:55.963568 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:55.963526 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-gw86m\"" Oct 31 05:18:55.972441 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:55.966284 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 31 05:18:55.972441 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:55.969576 5019 scope.go:122] "RemoveContainer" containerID="8058a6344b055a51422441c497ed46d0db5ad6995924c098b7f4b6b69776923d" Oct 31 05:18:55.972441 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:18:55.969968 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8058a6344b055a51422441c497ed46d0db5ad6995924c098b7f4b6b69776923d\": container with ID starting with 8058a6344b055a51422441c497ed46d0db5ad6995924c098b7f4b6b69776923d not found: ID does not exist" containerID="8058a6344b055a51422441c497ed46d0db5ad6995924c098b7f4b6b69776923d" Oct 31 05:18:55.972441 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:55.969999 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8058a6344b055a51422441c497ed46d0db5ad6995924c098b7f4b6b69776923d"} err="failed to get container status \"8058a6344b055a51422441c497ed46d0db5ad6995924c098b7f4b6b69776923d\": rpc error: code = NotFound desc = could not find container \"8058a6344b055a51422441c497ed46d0db5ad6995924c098b7f4b6b69776923d\": container with ID starting with 8058a6344b055a51422441c497ed46d0db5ad6995924c098b7f4b6b69776923d not found: ID does not exist" Oct 31 05:18:55.972441 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:55.970016 5019 scope.go:122] "RemoveContainer" containerID="f551717a6662ebf006462960c7e8057780b3a4aaf885dcfa59030d90813602ad" Oct 31 05:18:55.972441 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:18:55.970282 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f551717a6662ebf006462960c7e8057780b3a4aaf885dcfa59030d90813602ad\": container with ID starting with f551717a6662ebf006462960c7e8057780b3a4aaf885dcfa59030d90813602ad not found: ID does not exist" containerID="f551717a6662ebf006462960c7e8057780b3a4aaf885dcfa59030d90813602ad" Oct 31 05:18:55.972441 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:55.970297 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f551717a6662ebf006462960c7e8057780b3a4aaf885dcfa59030d90813602ad"} err="failed to get container status \"f551717a6662ebf006462960c7e8057780b3a4aaf885dcfa59030d90813602ad\": rpc error: code = NotFound desc = could not find container \"f551717a6662ebf006462960c7e8057780b3a4aaf885dcfa59030d90813602ad\": container with ID starting with f551717a6662ebf006462960c7e8057780b3a4aaf885dcfa59030d90813602ad not found: ID does not exist" Oct 31 05:18:55.972441 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:55.970425 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-85c4f7957c-rzwcg"] Oct 31 05:18:56.017453 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:56.017412 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ldbc\" (UniqueName: \"kubernetes.io/projected/f6078cee-8fa7-49bf-a9c9-ce433f7236ed-kube-api-access-8ldbc\") pod \"maas-controller-85c4f7957c-rzwcg\" (UID: \"f6078cee-8fa7-49bf-a9c9-ce433f7236ed\") " pod="opendatahub/maas-controller-85c4f7957c-rzwcg" Oct 31 05:18:56.017691 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:56.017563 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f6078cee-8fa7-49bf-a9c9-ce433f7236ed-webhook-cert\") pod \"maas-controller-85c4f7957c-rzwcg\" (UID: \"f6078cee-8fa7-49bf-a9c9-ce433f7236ed\") " pod="opendatahub/maas-controller-85c4f7957c-rzwcg" Oct 31 05:18:56.017742 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:56.017683 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6078cee-8fa7-49bf-a9c9-ce433f7236ed-metrics-certs\") pod \"maas-controller-85c4f7957c-rzwcg\" (UID: \"f6078cee-8fa7-49bf-a9c9-ce433f7236ed\") " pod="opendatahub/maas-controller-85c4f7957c-rzwcg" Oct 31 05:18:56.017800 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:56.017793 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07ba66ba-7067-4861-82ec-cf8a50a8550c-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:18:56.017838 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:56.017806 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07ba66ba-7067-4861-82ec-cf8a50a8550c-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:18:56.017838 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:56.017815 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07ba66ba-7067-4861-82ec-cf8a50a8550c-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:18:56.017838 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:56.017825 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkw46\" (UniqueName: \"kubernetes.io/projected/07ba66ba-7067-4861-82ec-cf8a50a8550c-kube-api-access-jkw46\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:18:56.038697 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:56.038653 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k"] Oct 31 05:18:56.043282 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:56.043245 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-8d386766-openshift-default-575667689c-n858k"] Oct 31 05:18:56.064515 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:56.064475 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 31 05:18:56.064734 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:56.064577 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 31 05:18:56.118502 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:56.118386 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6078cee-8fa7-49bf-a9c9-ce433f7236ed-metrics-certs\") pod \"maas-controller-85c4f7957c-rzwcg\" (UID: \"f6078cee-8fa7-49bf-a9c9-ce433f7236ed\") " pod="opendatahub/maas-controller-85c4f7957c-rzwcg" Oct 31 05:18:56.118502 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:56.118483 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ldbc\" (UniqueName: \"kubernetes.io/projected/f6078cee-8fa7-49bf-a9c9-ce433f7236ed-kube-api-access-8ldbc\") pod \"maas-controller-85c4f7957c-rzwcg\" (UID: \"f6078cee-8fa7-49bf-a9c9-ce433f7236ed\") " pod="opendatahub/maas-controller-85c4f7957c-rzwcg" Oct 31 05:18:56.118787 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:56.118537 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f6078cee-8fa7-49bf-a9c9-ce433f7236ed-webhook-cert\") pod \"maas-controller-85c4f7957c-rzwcg\" (UID: \"f6078cee-8fa7-49bf-a9c9-ce433f7236ed\") " pod="opendatahub/maas-controller-85c4f7957c-rzwcg" Oct 31 05:18:56.121401 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:56.121365 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6078cee-8fa7-49bf-a9c9-ce433f7236ed-metrics-certs\") pod \"maas-controller-85c4f7957c-rzwcg\" (UID: \"f6078cee-8fa7-49bf-a9c9-ce433f7236ed\") " pod="opendatahub/maas-controller-85c4f7957c-rzwcg" Oct 31 05:18:56.121401 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:56.121385 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f6078cee-8fa7-49bf-a9c9-ce433f7236ed-webhook-cert\") pod \"maas-controller-85c4f7957c-rzwcg\" (UID: \"f6078cee-8fa7-49bf-a9c9-ce433f7236ed\") " pod="opendatahub/maas-controller-85c4f7957c-rzwcg" Oct 31 05:18:56.142626 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:56.142589 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ldbc\" (UniqueName: \"kubernetes.io/projected/f6078cee-8fa7-49bf-a9c9-ce433f7236ed-kube-api-access-8ldbc\") pod \"maas-controller-85c4f7957c-rzwcg\" (UID: \"f6078cee-8fa7-49bf-a9c9-ce433f7236ed\") " pod="opendatahub/maas-controller-85c4f7957c-rzwcg" Oct 31 05:18:56.268273 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:56.268232 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-85c4f7957c-rzwcg" Oct 31 05:18:56.579892 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:56.579856 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-85c4f7957c-rzwcg"] Oct 31 05:18:56.582164 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:18:56.582129 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6078cee_8fa7_49bf_a9c9_ce433f7236ed.slice/crio-40fb90c3c76f91cc3c76e707b29638fc4c500e4558b69a9a61affcedd146f857 WatchSource:0}: Error finding container 40fb90c3c76f91cc3c76e707b29638fc4c500e4558b69a9a61affcedd146f857: Status 404 returned error can't find the container with id 40fb90c3c76f91cc3c76e707b29638fc4c500e4558b69a9a61affcedd146f857 Oct 31 05:18:56.584041 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:56.584020 5019 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 05:18:56.952177 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:56.952077 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-85c4f7957c-rzwcg" event={"ID":"f6078cee-8fa7-49bf-a9c9-ce433f7236ed","Type":"ContainerStarted","Data":"40fb90c3c76f91cc3c76e707b29638fc4c500e4558b69a9a61affcedd146f857"} Oct 31 05:18:57.411054 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:57.411008 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07ba66ba-7067-4861-82ec-cf8a50a8550c" path="/var/lib/kubelet/pods/07ba66ba-7067-4861-82ec-cf8a50a8550c/volumes" Oct 31 05:18:58.967581 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:58.967522 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-85c4f7957c-rzwcg" event={"ID":"f6078cee-8fa7-49bf-a9c9-ce433f7236ed","Type":"ContainerStarted","Data":"39a6788c11e59e54d7aec9600e81cb812e873d96ce8a4759c13be4f0e4ff974a"} Oct 31 05:18:58.968079 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:58.967624 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-85c4f7957c-rzwcg" Oct 31 05:18:59.011572 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:18:59.011490 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-85c4f7957c-rzwcg" podStartSLOduration=2.418224026 podStartE2EDuration="4.011473895s" totalImagesPullingTime="1.593249869s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:18:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:18:59.009761116 +0000 UTC m=+4493.766099157" watchObservedRunningTime="2026-10-31 05:18:59.011473895 +0000 UTC m=+4493.767811934" Oct 31 05:19:09.677477 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:09.677445 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s88kp_3945ff65-e877-4372-9334-3968c68b7295/console-operator/2.log" Oct 31 05:19:09.683026 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:09.683000 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s88kp_3945ff65-e877-4372-9334-3968c68b7295/console-operator/2.log" Oct 31 05:19:09.976837 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:09.976800 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-85c4f7957c-rzwcg" Oct 31 05:19:14.078545 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:14.078505 5019 scope.go:122] "RemoveContainer" containerID="0f9a960171226f10b26990e357211084a65d717f6e5834d500f6a2f7372ab1f3" Oct 31 05:19:14.088670 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:14.088641 5019 scope.go:122] "RemoveContainer" containerID="e4e64566e2b37b7df4e32fdee6292abc265dad6b21e12f9c33f55821bd14de95" Oct 31 05:19:14.098972 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:14.098941 5019 scope.go:122] "RemoveContainer" containerID="8c839258577a8a0cfa16d74cd944604839e85ff47434d229d3c23d029d9c0d3e" Oct 31 05:19:14.109113 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:14.109087 5019 scope.go:122] "RemoveContainer" containerID="57eccfc623dcb6911a57be6e992f35e00a9dd27a3e6560408297bc6b418c250f" Oct 31 05:19:14.120546 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:14.120516 5019 scope.go:122] "RemoveContainer" containerID="5728f9525210bcd3d9384d86aa8b03744a9ffce2b0ecb52ec62aa86a0dc4aa50" Oct 31 05:19:14.130760 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:14.130729 5019 scope.go:122] "RemoveContainer" containerID="2a4d5fc70509d71cb7ebd8499ac041bc1601285dfe766e3537f2a34fa6a3b561" Oct 31 05:19:14.141462 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:14.141431 5019 scope.go:122] "RemoveContainer" containerID="f255ea32ecea31466a26ad9494d2cda25ea7db6528486320178a51adab2c78c1" Oct 31 05:19:14.151282 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:14.151256 5019 scope.go:122] "RemoveContainer" containerID="9ac92aa4b28fc42e0b2ad3b6421c5549a5520b83798cd261586e7fe65c0ee40e" Oct 31 05:19:15.432375 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:15.432323 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d6zhg"] Oct 31 05:19:15.436393 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:15.436364 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d6zhg" Oct 31 05:19:15.452188 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:15.452151 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:19:15.464387 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:15.464314 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:19:15.464387 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:15.464326 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:19:15.464628 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:15.464433 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfhzn\"" Oct 31 05:19:15.467261 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:15.467228 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d6zhg"] Oct 31 05:19:15.514438 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:15.514393 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6c8a271-23e4-4b2c-92ad-a396721263d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-d6zhg\" (UID: \"e6c8a271-23e4-4b2c-92ad-a396721263d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d6zhg" Oct 31 05:19:15.514642 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:15.514511 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5x67\" (UniqueName: \"kubernetes.io/projected/e6c8a271-23e4-4b2c-92ad-a396721263d8-kube-api-access-p5x67\") pod \"maas-api-revoke-sub-85237e45-d6zhg\" (UID: \"e6c8a271-23e4-4b2c-92ad-a396721263d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d6zhg" Oct 31 05:19:15.514642 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:15.514575 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6c8a271-23e4-4b2c-92ad-a396721263d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-d6zhg\" (UID: \"e6c8a271-23e4-4b2c-92ad-a396721263d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d6zhg" Oct 31 05:19:15.615154 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:15.615104 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6c8a271-23e4-4b2c-92ad-a396721263d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-d6zhg\" (UID: \"e6c8a271-23e4-4b2c-92ad-a396721263d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d6zhg" Oct 31 05:19:15.615400 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:15.615166 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6c8a271-23e4-4b2c-92ad-a396721263d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-d6zhg\" (UID: \"e6c8a271-23e4-4b2c-92ad-a396721263d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d6zhg" Oct 31 05:19:15.615400 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:15.615214 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5x67\" (UniqueName: \"kubernetes.io/projected/e6c8a271-23e4-4b2c-92ad-a396721263d8-kube-api-access-p5x67\") pod \"maas-api-revoke-sub-85237e45-d6zhg\" (UID: \"e6c8a271-23e4-4b2c-92ad-a396721263d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d6zhg" Oct 31 05:19:15.616056 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:15.616021 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6c8a271-23e4-4b2c-92ad-a396721263d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-d6zhg\" (UID: \"e6c8a271-23e4-4b2c-92ad-a396721263d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d6zhg" Oct 31 05:19:15.617946 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:15.617915 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6c8a271-23e4-4b2c-92ad-a396721263d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-d6zhg\" (UID: \"e6c8a271-23e4-4b2c-92ad-a396721263d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d6zhg" Oct 31 05:19:15.638994 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:15.638949 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5x67\" (UniqueName: \"kubernetes.io/projected/e6c8a271-23e4-4b2c-92ad-a396721263d8-kube-api-access-p5x67\") pod \"maas-api-revoke-sub-85237e45-d6zhg\" (UID: \"e6c8a271-23e4-4b2c-92ad-a396721263d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d6zhg" Oct 31 05:19:15.751384 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:15.751314 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d6zhg" Oct 31 05:19:16.057080 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:16.057040 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d6zhg"] Oct 31 05:19:16.059145 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:19:16.059107 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6c8a271_23e4_4b2c_92ad_a396721263d8.slice/crio-7b2c5e3e76dd13d8bbd1b0367f57190084ceb5e0c696c4ad1c84c2d9a0458a7f WatchSource:0}: Error finding container 7b2c5e3e76dd13d8bbd1b0367f57190084ceb5e0c696c4ad1c84c2d9a0458a7f: Status 404 returned error can't find the container with id 7b2c5e3e76dd13d8bbd1b0367f57190084ceb5e0c696c4ad1c84c2d9a0458a7f Oct 31 05:19:16.079158 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:16.079120 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d6zhg" event={"ID":"e6c8a271-23e4-4b2c-92ad-a396721263d8","Type":"ContainerStarted","Data":"7b2c5e3e76dd13d8bbd1b0367f57190084ceb5e0c696c4ad1c84c2d9a0458a7f"} Oct 31 05:19:17.088064 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:17.088015 5019 generic.go:356] "Generic (PLEG): container finished" podID="e6c8a271-23e4-4b2c-92ad-a396721263d8" containerID="b25753fbef87a33181e2bdc7aa2192635cbab22bae59c20c28f5530092b0891a" exitCode=0 Oct 31 05:19:17.088568 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:17.088091 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d6zhg" event={"ID":"e6c8a271-23e4-4b2c-92ad-a396721263d8","Type":"ContainerDied","Data":"b25753fbef87a33181e2bdc7aa2192635cbab22bae59c20c28f5530092b0891a"} Oct 31 05:19:18.250272 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:18.250236 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d6zhg" Oct 31 05:19:18.338317 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:18.338274 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6c8a271-23e4-4b2c-92ad-a396721263d8-kube-api-access-p5x67\" (UniqueName: \"kubernetes.io/projected/e6c8a271-23e4-4b2c-92ad-a396721263d8-kube-api-access-p5x67\") pod \"e6c8a271-23e4-4b2c-92ad-a396721263d8\" (UID: \"e6c8a271-23e4-4b2c-92ad-a396721263d8\") " Oct 31 05:19:18.338546 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:18.338374 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6c8a271-23e4-4b2c-92ad-a396721263d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6c8a271-23e4-4b2c-92ad-a396721263d8-maas-api-xks-ca\") pod \"e6c8a271-23e4-4b2c-92ad-a396721263d8\" (UID: \"e6c8a271-23e4-4b2c-92ad-a396721263d8\") " Oct 31 05:19:18.338546 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:18.338467 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6c8a271-23e4-4b2c-92ad-a396721263d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6c8a271-23e4-4b2c-92ad-a396721263d8-maas-api-openshift-ca\") pod \"e6c8a271-23e4-4b2c-92ad-a396721263d8\" (UID: \"e6c8a271-23e4-4b2c-92ad-a396721263d8\") " Oct 31 05:19:18.338870 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:18.338845 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6c8a271-23e4-4b2c-92ad-a396721263d8-maas-api-openshift-ca" pod "e6c8a271-23e4-4b2c-92ad-a396721263d8" (UID: "e6c8a271-23e4-4b2c-92ad-a396721263d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:19:18.340732 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:18.340687 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6c8a271-23e4-4b2c-92ad-a396721263d8-maas-api-xks-ca" pod "e6c8a271-23e4-4b2c-92ad-a396721263d8" (UID: "e6c8a271-23e4-4b2c-92ad-a396721263d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:19:18.340867 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:18.340737 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6c8a271-23e4-4b2c-92ad-a396721263d8-kube-api-access-p5x67" pod "e6c8a271-23e4-4b2c-92ad-a396721263d8" (UID: "e6c8a271-23e4-4b2c-92ad-a396721263d8"). InnerVolumeSpecName "kube-api-access-p5x67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:19:18.439413 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:18.439372 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6c8a271-23e4-4b2c-92ad-a396721263d8-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:19:18.439633 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:18.439534 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6c8a271-23e4-4b2c-92ad-a396721263d8-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:19:18.439633 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:18.439570 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5x67\" (UniqueName: \"kubernetes.io/projected/e6c8a271-23e4-4b2c-92ad-a396721263d8-kube-api-access-p5x67\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:19:19.102737 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:19.102700 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d6zhg" Oct 31 05:19:19.102941 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:19.102697 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d6zhg" event={"ID":"e6c8a271-23e4-4b2c-92ad-a396721263d8","Type":"ContainerDied","Data":"7b2c5e3e76dd13d8bbd1b0367f57190084ceb5e0c696c4ad1c84c2d9a0458a7f"} Oct 31 05:19:19.102941 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:19.102831 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b2c5e3e76dd13d8bbd1b0367f57190084ceb5e0c696c4ad1c84c2d9a0458a7f" Oct 31 05:19:19.960845 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:19.960804 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rgj85"] Oct 31 05:19:19.961283 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:19.961237 5019 cpu_manager.go:414] "containerMap: removing container" podUID="e6c8a271-23e4-4b2c-92ad-a396721263d8" containerName="revoke-keys" Oct 31 05:19:19.961283 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:19.961250 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6c8a271-23e4-4b2c-92ad-a396721263d8" containerName="revoke-keys" Oct 31 05:19:19.961418 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:19.961364 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6c8a271-23e4-4b2c-92ad-a396721263d8" containerName="revoke-keys" Oct 31 05:19:19.967981 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:19.967846 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rgj85" Oct 31 05:19:19.978479 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:19.978444 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-shld4"] Oct 31 05:19:19.983319 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:19.983285 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r"] Oct 31 05:19:19.983506 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:19.983419 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-shld4" Oct 31 05:19:19.987059 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:19.987030 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-sqqs6\"" Oct 31 05:19:19.987620 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:19.987582 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r" Oct 31 05:19:19.996603 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:19.996371 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rgj85"] Oct 31 05:19:19.999997 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:19.999954 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5"] Oct 31 05:19:20.009221 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.004856 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5" Oct 31 05:19:20.009221 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.008235 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-shld4"] Oct 31 05:19:20.013685 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.012376 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r"] Oct 31 05:19:20.039605 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.039535 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5"] Oct 31 05:19:20.056262 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.056218 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8430b919-a9dd-4c7e-9266-ef6872cc0b9e-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rgj85\" (UID: \"8430b919-a9dd-4c7e-9266-ef6872cc0b9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rgj85" Oct 31 05:19:20.056488 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.056273 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a69fd138-9906-4127-ac26-38a21a5b7590-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-shld4\" (UID: \"a69fd138-9906-4127-ac26-38a21a5b7590\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-shld4" Oct 31 05:19:20.056488 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.056387 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d68d19b-5392-4308-9e2c-4b5e7593d17b-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lqrr5\" (UID: \"6d68d19b-5392-4308-9e2c-4b5e7593d17b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5" Oct 31 05:19:20.056488 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.056468 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68e465ba-2b5d-4e3c-8b63-7bdb9b695b75-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hgv9r\" (UID: \"68e465ba-2b5d-4e3c-8b63-7bdb9b695b75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r" Oct 31 05:19:20.056604 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.056505 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8430b919-a9dd-4c7e-9266-ef6872cc0b9e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rgj85\" (UID: \"8430b919-a9dd-4c7e-9266-ef6872cc0b9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rgj85" Oct 31 05:19:20.056604 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.056526 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68e465ba-2b5d-4e3c-8b63-7bdb9b695b75-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hgv9r\" (UID: \"68e465ba-2b5d-4e3c-8b63-7bdb9b695b75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r" Oct 31 05:19:20.056604 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.056542 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8430b919-a9dd-4c7e-9266-ef6872cc0b9e-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rgj85\" (UID: \"8430b919-a9dd-4c7e-9266-ef6872cc0b9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rgj85" Oct 31 05:19:20.056604 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.056594 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8430b919-a9dd-4c7e-9266-ef6872cc0b9e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rgj85\" (UID: \"8430b919-a9dd-4c7e-9266-ef6872cc0b9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rgj85" Oct 31 05:19:20.056764 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.056640 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d68d19b-5392-4308-9e2c-4b5e7593d17b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lqrr5\" (UID: \"6d68d19b-5392-4308-9e2c-4b5e7593d17b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5" Oct 31 05:19:20.056764 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.056699 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d68d19b-5392-4308-9e2c-4b5e7593d17b-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lqrr5\" (UID: \"6d68d19b-5392-4308-9e2c-4b5e7593d17b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5" Oct 31 05:19:20.056764 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.056736 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8430b919-a9dd-4c7e-9266-ef6872cc0b9e-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rgj85\" (UID: \"8430b919-a9dd-4c7e-9266-ef6872cc0b9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rgj85" Oct 31 05:19:20.056764 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.056755 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a69fd138-9906-4127-ac26-38a21a5b7590-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-shld4\" (UID: \"a69fd138-9906-4127-ac26-38a21a5b7590\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-shld4" Oct 31 05:19:20.056915 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.056773 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb2mh\" (UniqueName: \"kubernetes.io/projected/68e465ba-2b5d-4e3c-8b63-7bdb9b695b75-kube-api-access-nb2mh\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hgv9r\" (UID: \"68e465ba-2b5d-4e3c-8b63-7bdb9b695b75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r" Oct 31 05:19:20.056915 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.056793 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njf9d\" (UniqueName: \"kubernetes.io/projected/8430b919-a9dd-4c7e-9266-ef6872cc0b9e-kube-api-access-njf9d\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rgj85\" (UID: \"8430b919-a9dd-4c7e-9266-ef6872cc0b9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rgj85" Oct 31 05:19:20.056915 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.056873 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a69fd138-9906-4127-ac26-38a21a5b7590-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-shld4\" (UID: \"a69fd138-9906-4127-ac26-38a21a5b7590\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-shld4" Oct 31 05:19:20.057017 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.056914 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmlgn\" (UniqueName: \"kubernetes.io/projected/6d68d19b-5392-4308-9e2c-4b5e7593d17b-kube-api-access-bmlgn\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lqrr5\" (UID: \"6d68d19b-5392-4308-9e2c-4b5e7593d17b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5" Oct 31 05:19:20.057017 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.056950 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68e465ba-2b5d-4e3c-8b63-7bdb9b695b75-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hgv9r\" (UID: \"68e465ba-2b5d-4e3c-8b63-7bdb9b695b75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r" Oct 31 05:19:20.057090 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.057028 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68e465ba-2b5d-4e3c-8b63-7bdb9b695b75-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hgv9r\" (UID: \"68e465ba-2b5d-4e3c-8b63-7bdb9b695b75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r" Oct 31 05:19:20.057090 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.057047 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8430b919-a9dd-4c7e-9266-ef6872cc0b9e-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rgj85\" (UID: \"8430b919-a9dd-4c7e-9266-ef6872cc0b9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rgj85" Oct 31 05:19:20.057090 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.057071 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8430b919-a9dd-4c7e-9266-ef6872cc0b9e-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rgj85\" (UID: \"8430b919-a9dd-4c7e-9266-ef6872cc0b9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rgj85" Oct 31 05:19:20.057203 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.057127 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68e465ba-2b5d-4e3c-8b63-7bdb9b695b75-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hgv9r\" (UID: \"68e465ba-2b5d-4e3c-8b63-7bdb9b695b75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r" Oct 31 05:19:20.057203 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.057168 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8430b919-a9dd-4c7e-9266-ef6872cc0b9e-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rgj85\" (UID: \"8430b919-a9dd-4c7e-9266-ef6872cc0b9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rgj85" Oct 31 05:19:20.057278 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.057200 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a69fd138-9906-4127-ac26-38a21a5b7590-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-shld4\" (UID: \"a69fd138-9906-4127-ac26-38a21a5b7590\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-shld4" Oct 31 05:19:20.057278 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.057231 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a69fd138-9906-4127-ac26-38a21a5b7590-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-shld4\" (UID: \"a69fd138-9906-4127-ac26-38a21a5b7590\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-shld4" Oct 31 05:19:20.057278 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.057269 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d68d19b-5392-4308-9e2c-4b5e7593d17b-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lqrr5\" (UID: \"6d68d19b-5392-4308-9e2c-4b5e7593d17b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5" Oct 31 05:19:20.057436 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.057293 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d68d19b-5392-4308-9e2c-4b5e7593d17b-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lqrr5\" (UID: \"6d68d19b-5392-4308-9e2c-4b5e7593d17b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5" Oct 31 05:19:20.057436 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.057313 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68e465ba-2b5d-4e3c-8b63-7bdb9b695b75-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hgv9r\" (UID: \"68e465ba-2b5d-4e3c-8b63-7bdb9b695b75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r" Oct 31 05:19:20.057436 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.057371 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d68d19b-5392-4308-9e2c-4b5e7593d17b-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lqrr5\" (UID: \"6d68d19b-5392-4308-9e2c-4b5e7593d17b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5" Oct 31 05:19:20.057436 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.057403 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a69fd138-9906-4127-ac26-38a21a5b7590-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-shld4\" (UID: \"a69fd138-9906-4127-ac26-38a21a5b7590\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-shld4" Oct 31 05:19:20.057577 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.057475 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68e465ba-2b5d-4e3c-8b63-7bdb9b695b75-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hgv9r\" (UID: \"68e465ba-2b5d-4e3c-8b63-7bdb9b695b75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r" Oct 31 05:19:20.057577 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.057512 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68e465ba-2b5d-4e3c-8b63-7bdb9b695b75-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hgv9r\" (UID: \"68e465ba-2b5d-4e3c-8b63-7bdb9b695b75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r" Oct 31 05:19:20.057577 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.057557 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcjzh\" (UniqueName: \"kubernetes.io/projected/a69fd138-9906-4127-ac26-38a21a5b7590-kube-api-access-fcjzh\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-shld4\" (UID: \"a69fd138-9906-4127-ac26-38a21a5b7590\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-shld4" Oct 31 05:19:20.057681 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.057598 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a69fd138-9906-4127-ac26-38a21a5b7590-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-shld4\" (UID: \"a69fd138-9906-4127-ac26-38a21a5b7590\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-shld4" Oct 31 05:19:20.057681 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.057622 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d68d19b-5392-4308-9e2c-4b5e7593d17b-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lqrr5\" (UID: \"6d68d19b-5392-4308-9e2c-4b5e7593d17b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5" Oct 31 05:19:20.057681 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.057648 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d68d19b-5392-4308-9e2c-4b5e7593d17b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lqrr5\" (UID: \"6d68d19b-5392-4308-9e2c-4b5e7593d17b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5" Oct 31 05:19:20.057781 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.057705 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a69fd138-9906-4127-ac26-38a21a5b7590-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-shld4\" (UID: \"a69fd138-9906-4127-ac26-38a21a5b7590\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-shld4" Oct 31 05:19:20.158662 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.158611 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8430b919-a9dd-4c7e-9266-ef6872cc0b9e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rgj85\" (UID: \"8430b919-a9dd-4c7e-9266-ef6872cc0b9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rgj85" Oct 31 05:19:20.158662 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.158666 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68e465ba-2b5d-4e3c-8b63-7bdb9b695b75-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hgv9r\" (UID: \"68e465ba-2b5d-4e3c-8b63-7bdb9b695b75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r" Oct 31 05:19:20.158986 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.158787 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8430b919-a9dd-4c7e-9266-ef6872cc0b9e-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rgj85\" (UID: \"8430b919-a9dd-4c7e-9266-ef6872cc0b9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rgj85" Oct 31 05:19:20.158986 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.158829 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8430b919-a9dd-4c7e-9266-ef6872cc0b9e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rgj85\" (UID: \"8430b919-a9dd-4c7e-9266-ef6872cc0b9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rgj85" Oct 31 05:19:20.158986 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.158857 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d68d19b-5392-4308-9e2c-4b5e7593d17b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lqrr5\" (UID: \"6d68d19b-5392-4308-9e2c-4b5e7593d17b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5" Oct 31 05:19:20.158986 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.158887 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d68d19b-5392-4308-9e2c-4b5e7593d17b-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lqrr5\" (UID: \"6d68d19b-5392-4308-9e2c-4b5e7593d17b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5" Oct 31 05:19:20.158986 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.158913 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8430b919-a9dd-4c7e-9266-ef6872cc0b9e-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rgj85\" (UID: \"8430b919-a9dd-4c7e-9266-ef6872cc0b9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rgj85" Oct 31 05:19:20.158986 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.158949 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a69fd138-9906-4127-ac26-38a21a5b7590-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-shld4\" (UID: \"a69fd138-9906-4127-ac26-38a21a5b7590\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-shld4" Oct 31 05:19:20.158986 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.158979 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb2mh\" (UniqueName: \"kubernetes.io/projected/68e465ba-2b5d-4e3c-8b63-7bdb9b695b75-kube-api-access-nb2mh\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hgv9r\" (UID: \"68e465ba-2b5d-4e3c-8b63-7bdb9b695b75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r" Oct 31 05:19:20.159479 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.159009 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njf9d\" (UniqueName: \"kubernetes.io/projected/8430b919-a9dd-4c7e-9266-ef6872cc0b9e-kube-api-access-njf9d\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rgj85\" (UID: \"8430b919-a9dd-4c7e-9266-ef6872cc0b9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rgj85" Oct 31 05:19:20.159479 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.159039 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a69fd138-9906-4127-ac26-38a21a5b7590-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-shld4\" (UID: \"a69fd138-9906-4127-ac26-38a21a5b7590\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-shld4" Oct 31 05:19:20.159479 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.159064 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmlgn\" (UniqueName: \"kubernetes.io/projected/6d68d19b-5392-4308-9e2c-4b5e7593d17b-kube-api-access-bmlgn\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lqrr5\" (UID: \"6d68d19b-5392-4308-9e2c-4b5e7593d17b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5" Oct 31 05:19:20.159479 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.159090 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68e465ba-2b5d-4e3c-8b63-7bdb9b695b75-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hgv9r\" (UID: \"68e465ba-2b5d-4e3c-8b63-7bdb9b695b75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r" Oct 31 05:19:20.159479 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.159125 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68e465ba-2b5d-4e3c-8b63-7bdb9b695b75-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hgv9r\" (UID: \"68e465ba-2b5d-4e3c-8b63-7bdb9b695b75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r" Oct 31 05:19:20.159479 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.159129 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68e465ba-2b5d-4e3c-8b63-7bdb9b695b75-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hgv9r\" (UID: \"68e465ba-2b5d-4e3c-8b63-7bdb9b695b75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r" Oct 31 05:19:20.159479 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.159188 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8430b919-a9dd-4c7e-9266-ef6872cc0b9e-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rgj85\" (UID: \"8430b919-a9dd-4c7e-9266-ef6872cc0b9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rgj85" Oct 31 05:19:20.159479 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.159220 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8430b919-a9dd-4c7e-9266-ef6872cc0b9e-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rgj85\" (UID: \"8430b919-a9dd-4c7e-9266-ef6872cc0b9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rgj85" Oct 31 05:19:20.159479 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.159252 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68e465ba-2b5d-4e3c-8b63-7bdb9b695b75-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hgv9r\" (UID: \"68e465ba-2b5d-4e3c-8b63-7bdb9b695b75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r" Oct 31 05:19:20.159479 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.159293 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8430b919-a9dd-4c7e-9266-ef6872cc0b9e-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rgj85\" (UID: \"8430b919-a9dd-4c7e-9266-ef6872cc0b9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rgj85" Oct 31 05:19:20.159479 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.159323 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a69fd138-9906-4127-ac26-38a21a5b7590-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-shld4\" (UID: \"a69fd138-9906-4127-ac26-38a21a5b7590\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-shld4" Oct 31 05:19:20.159479 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.159374 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a69fd138-9906-4127-ac26-38a21a5b7590-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-shld4\" (UID: \"a69fd138-9906-4127-ac26-38a21a5b7590\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-shld4" Oct 31 05:19:20.159479 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.159408 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d68d19b-5392-4308-9e2c-4b5e7593d17b-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lqrr5\" (UID: \"6d68d19b-5392-4308-9e2c-4b5e7593d17b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5" Oct 31 05:19:20.159479 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.159423 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68e465ba-2b5d-4e3c-8b63-7bdb9b695b75-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hgv9r\" (UID: \"68e465ba-2b5d-4e3c-8b63-7bdb9b695b75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r" Oct 31 05:19:20.159479 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.159439 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d68d19b-5392-4308-9e2c-4b5e7593d17b-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lqrr5\" (UID: \"6d68d19b-5392-4308-9e2c-4b5e7593d17b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5" Oct 31 05:19:20.159479 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.159466 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68e465ba-2b5d-4e3c-8b63-7bdb9b695b75-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hgv9r\" (UID: \"68e465ba-2b5d-4e3c-8b63-7bdb9b695b75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r" Oct 31 05:19:20.160487 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.159494 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d68d19b-5392-4308-9e2c-4b5e7593d17b-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lqrr5\" (UID: \"6d68d19b-5392-4308-9e2c-4b5e7593d17b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5" Oct 31 05:19:20.160487 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.159517 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8430b919-a9dd-4c7e-9266-ef6872cc0b9e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rgj85\" (UID: \"8430b919-a9dd-4c7e-9266-ef6872cc0b9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rgj85" Oct 31 05:19:20.160487 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.159736 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8430b919-a9dd-4c7e-9266-ef6872cc0b9e-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rgj85\" (UID: \"8430b919-a9dd-4c7e-9266-ef6872cc0b9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rgj85" Oct 31 05:19:20.160487 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.159803 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8430b919-a9dd-4c7e-9266-ef6872cc0b9e-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rgj85\" (UID: \"8430b919-a9dd-4c7e-9266-ef6872cc0b9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rgj85" Oct 31 05:19:20.160487 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.159521 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a69fd138-9906-4127-ac26-38a21a5b7590-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-shld4\" (UID: \"a69fd138-9906-4127-ac26-38a21a5b7590\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-shld4" Oct 31 05:19:20.160487 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.159839 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d68d19b-5392-4308-9e2c-4b5e7593d17b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lqrr5\" (UID: \"6d68d19b-5392-4308-9e2c-4b5e7593d17b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5" Oct 31 05:19:20.160487 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.159849 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68e465ba-2b5d-4e3c-8b63-7bdb9b695b75-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hgv9r\" (UID: \"68e465ba-2b5d-4e3c-8b63-7bdb9b695b75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r" Oct 31 05:19:20.160487 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.159923 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68e465ba-2b5d-4e3c-8b63-7bdb9b695b75-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hgv9r\" (UID: \"68e465ba-2b5d-4e3c-8b63-7bdb9b695b75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r" Oct 31 05:19:20.160487 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.159968 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d68d19b-5392-4308-9e2c-4b5e7593d17b-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lqrr5\" (UID: \"6d68d19b-5392-4308-9e2c-4b5e7593d17b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5" Oct 31 05:19:20.160487 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.159985 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68e465ba-2b5d-4e3c-8b63-7bdb9b695b75-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hgv9r\" (UID: \"68e465ba-2b5d-4e3c-8b63-7bdb9b695b75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r" Oct 31 05:19:20.160487 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.160020 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcjzh\" (UniqueName: \"kubernetes.io/projected/a69fd138-9906-4127-ac26-38a21a5b7590-kube-api-access-fcjzh\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-shld4\" (UID: \"a69fd138-9906-4127-ac26-38a21a5b7590\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-shld4" Oct 31 05:19:20.160487 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.160076 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a69fd138-9906-4127-ac26-38a21a5b7590-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-shld4\" (UID: \"a69fd138-9906-4127-ac26-38a21a5b7590\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-shld4" Oct 31 05:19:20.160487 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.160105 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d68d19b-5392-4308-9e2c-4b5e7593d17b-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lqrr5\" (UID: \"6d68d19b-5392-4308-9e2c-4b5e7593d17b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5" Oct 31 05:19:20.160487 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.160132 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d68d19b-5392-4308-9e2c-4b5e7593d17b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lqrr5\" (UID: \"6d68d19b-5392-4308-9e2c-4b5e7593d17b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5" Oct 31 05:19:20.160487 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.160135 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d68d19b-5392-4308-9e2c-4b5e7593d17b-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lqrr5\" (UID: \"6d68d19b-5392-4308-9e2c-4b5e7593d17b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5" Oct 31 05:19:20.160487 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.160165 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a69fd138-9906-4127-ac26-38a21a5b7590-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-shld4\" (UID: \"a69fd138-9906-4127-ac26-38a21a5b7590\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-shld4" Oct 31 05:19:20.160487 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.160214 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a69fd138-9906-4127-ac26-38a21a5b7590-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-shld4\" (UID: \"a69fd138-9906-4127-ac26-38a21a5b7590\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-shld4" Oct 31 05:19:20.160487 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.160243 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8430b919-a9dd-4c7e-9266-ef6872cc0b9e-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rgj85\" (UID: \"8430b919-a9dd-4c7e-9266-ef6872cc0b9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rgj85" Oct 31 05:19:20.160487 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.160257 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8430b919-a9dd-4c7e-9266-ef6872cc0b9e-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rgj85\" (UID: \"8430b919-a9dd-4c7e-9266-ef6872cc0b9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rgj85" Oct 31 05:19:20.160487 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.160298 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a69fd138-9906-4127-ac26-38a21a5b7590-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-shld4\" (UID: \"a69fd138-9906-4127-ac26-38a21a5b7590\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-shld4" Oct 31 05:19:20.160487 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.160349 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d68d19b-5392-4308-9e2c-4b5e7593d17b-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lqrr5\" (UID: \"6d68d19b-5392-4308-9e2c-4b5e7593d17b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5" Oct 31 05:19:20.160487 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.160375 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68e465ba-2b5d-4e3c-8b63-7bdb9b695b75-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hgv9r\" (UID: \"68e465ba-2b5d-4e3c-8b63-7bdb9b695b75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r" Oct 31 05:19:20.160487 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.160395 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68e465ba-2b5d-4e3c-8b63-7bdb9b695b75-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hgv9r\" (UID: \"68e465ba-2b5d-4e3c-8b63-7bdb9b695b75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r" Oct 31 05:19:20.161944 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.161110 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68e465ba-2b5d-4e3c-8b63-7bdb9b695b75-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hgv9r\" (UID: \"68e465ba-2b5d-4e3c-8b63-7bdb9b695b75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r" Oct 31 05:19:20.161944 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.161396 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a69fd138-9906-4127-ac26-38a21a5b7590-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-shld4\" (UID: \"a69fd138-9906-4127-ac26-38a21a5b7590\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-shld4" Oct 31 05:19:20.161944 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.161648 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8430b919-a9dd-4c7e-9266-ef6872cc0b9e-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rgj85\" (UID: \"8430b919-a9dd-4c7e-9266-ef6872cc0b9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rgj85" Oct 31 05:19:20.161944 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.161917 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a69fd138-9906-4127-ac26-38a21a5b7590-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-shld4\" (UID: \"a69fd138-9906-4127-ac26-38a21a5b7590\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-shld4" Oct 31 05:19:20.162224 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.162168 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d68d19b-5392-4308-9e2c-4b5e7593d17b-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lqrr5\" (UID: \"6d68d19b-5392-4308-9e2c-4b5e7593d17b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5" Oct 31 05:19:20.163459 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.162811 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8430b919-a9dd-4c7e-9266-ef6872cc0b9e-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rgj85\" (UID: \"8430b919-a9dd-4c7e-9266-ef6872cc0b9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rgj85" Oct 31 05:19:20.163459 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.163111 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a69fd138-9906-4127-ac26-38a21a5b7590-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-shld4\" (UID: \"a69fd138-9906-4127-ac26-38a21a5b7590\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-shld4" Oct 31 05:19:20.163459 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.163126 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68e465ba-2b5d-4e3c-8b63-7bdb9b695b75-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hgv9r\" (UID: \"68e465ba-2b5d-4e3c-8b63-7bdb9b695b75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r" Oct 31 05:19:20.163459 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.163216 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d68d19b-5392-4308-9e2c-4b5e7593d17b-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lqrr5\" (UID: \"6d68d19b-5392-4308-9e2c-4b5e7593d17b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5" Oct 31 05:19:20.163459 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.163409 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a69fd138-9906-4127-ac26-38a21a5b7590-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-shld4\" (UID: \"a69fd138-9906-4127-ac26-38a21a5b7590\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-shld4" Oct 31 05:19:20.163459 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.163421 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d68d19b-5392-4308-9e2c-4b5e7593d17b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lqrr5\" (UID: \"6d68d19b-5392-4308-9e2c-4b5e7593d17b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5" Oct 31 05:19:20.163804 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.163473 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a69fd138-9906-4127-ac26-38a21a5b7590-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-shld4\" (UID: \"a69fd138-9906-4127-ac26-38a21a5b7590\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-shld4" Oct 31 05:19:20.163804 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.163523 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d68d19b-5392-4308-9e2c-4b5e7593d17b-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lqrr5\" (UID: \"6d68d19b-5392-4308-9e2c-4b5e7593d17b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5" Oct 31 05:19:20.164080 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.164053 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a69fd138-9906-4127-ac26-38a21a5b7590-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-shld4\" (UID: \"a69fd138-9906-4127-ac26-38a21a5b7590\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-shld4" Oct 31 05:19:20.164080 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.164074 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8430b919-a9dd-4c7e-9266-ef6872cc0b9e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rgj85\" (UID: \"8430b919-a9dd-4c7e-9266-ef6872cc0b9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rgj85" Oct 31 05:19:20.164844 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.164822 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68e465ba-2b5d-4e3c-8b63-7bdb9b695b75-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hgv9r\" (UID: \"68e465ba-2b5d-4e3c-8b63-7bdb9b695b75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r" Oct 31 05:19:20.183179 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.183137 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68e465ba-2b5d-4e3c-8b63-7bdb9b695b75-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hgv9r\" (UID: \"68e465ba-2b5d-4e3c-8b63-7bdb9b695b75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r" Oct 31 05:19:20.183403 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.183261 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d68d19b-5392-4308-9e2c-4b5e7593d17b-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lqrr5\" (UID: \"6d68d19b-5392-4308-9e2c-4b5e7593d17b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5" Oct 31 05:19:20.183645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.183615 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8430b919-a9dd-4c7e-9266-ef6872cc0b9e-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rgj85\" (UID: \"8430b919-a9dd-4c7e-9266-ef6872cc0b9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rgj85" Oct 31 05:19:20.183645 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.183627 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njf9d\" (UniqueName: \"kubernetes.io/projected/8430b919-a9dd-4c7e-9266-ef6872cc0b9e-kube-api-access-njf9d\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rgj85\" (UID: \"8430b919-a9dd-4c7e-9266-ef6872cc0b9e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rgj85" Oct 31 05:19:20.183775 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.183667 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb2mh\" (UniqueName: \"kubernetes.io/projected/68e465ba-2b5d-4e3c-8b63-7bdb9b695b75-kube-api-access-nb2mh\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hgv9r\" (UID: \"68e465ba-2b5d-4e3c-8b63-7bdb9b695b75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r" Oct 31 05:19:20.183958 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.183938 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmlgn\" (UniqueName: \"kubernetes.io/projected/6d68d19b-5392-4308-9e2c-4b5e7593d17b-kube-api-access-bmlgn\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lqrr5\" (UID: \"6d68d19b-5392-4308-9e2c-4b5e7593d17b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5" Oct 31 05:19:20.184543 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.184518 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a69fd138-9906-4127-ac26-38a21a5b7590-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-shld4\" (UID: \"a69fd138-9906-4127-ac26-38a21a5b7590\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-shld4" Oct 31 05:19:20.186044 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.186022 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcjzh\" (UniqueName: \"kubernetes.io/projected/a69fd138-9906-4127-ac26-38a21a5b7590-kube-api-access-fcjzh\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-shld4\" (UID: \"a69fd138-9906-4127-ac26-38a21a5b7590\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-shld4" Oct 31 05:19:20.284935 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.284887 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rgj85" Oct 31 05:19:20.305372 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.305288 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-shld4" Oct 31 05:19:20.320038 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.319997 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r" Oct 31 05:19:20.346938 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.346904 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5" Oct 31 05:19:20.605700 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.605665 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rgj85"] Oct 31 05:19:20.606914 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:19:20.606884 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8430b919_a9dd_4c7e_9266_ef6872cc0b9e.slice/crio-e06e2a1752bb60c4b305797486c420511fb099713ca3d6c6c3d0ac92b78971f7 WatchSource:0}: Error finding container e06e2a1752bb60c4b305797486c420511fb099713ca3d6c6c3d0ac92b78971f7: Status 404 returned error can't find the container with id e06e2a1752bb60c4b305797486c420511fb099713ca3d6c6c3d0ac92b78971f7 Oct 31 05:19:20.609207 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.609164 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:19:20.609301 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.609260 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:19:20.609301 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.609290 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:19:20.632702 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.632665 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-shld4"] Oct 31 05:19:20.641953 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:19:20.641908 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda69fd138_9906_4127_ac26_38a21a5b7590.slice/crio-494279b58c4c10e2d3e91f27bb10b0d6bc0e6e7527aeaf89595bb77a1c31bcfb WatchSource:0}: Error finding container 494279b58c4c10e2d3e91f27bb10b0d6bc0e6e7527aeaf89595bb77a1c31bcfb: Status 404 returned error can't find the container with id 494279b58c4c10e2d3e91f27bb10b0d6bc0e6e7527aeaf89595bb77a1c31bcfb Oct 31 05:19:20.642821 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.642789 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r"] Oct 31 05:19:20.644934 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:19:20.644906 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68e465ba_2b5d_4e3c_8b63_7bdb9b695b75.slice/crio-c2b830080a49c1da383ce5a799d803beefe687852efab9959178043b73ea8a06 WatchSource:0}: Error finding container c2b830080a49c1da383ce5a799d803beefe687852efab9959178043b73ea8a06: Status 404 returned error can't find the container with id c2b830080a49c1da383ce5a799d803beefe687852efab9959178043b73ea8a06 Oct 31 05:19:20.645585 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.645543 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:19:20.645667 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.645615 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:19:20.645667 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.645643 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:19:20.646741 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.646706 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:19:20.646840 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.646776 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:19:20.646840 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.646814 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:19:20.678646 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.677987 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5"] Oct 31 05:19:20.682820 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:19:20.682787 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d68d19b_5392_4308_9e2c_4b5e7593d17b.slice/crio-52932a4012ae40069320d857d2680581be356e0305739dbc9a54bf135216a0af WatchSource:0}: Error finding container 52932a4012ae40069320d857d2680581be356e0305739dbc9a54bf135216a0af: Status 404 returned error can't find the container with id 52932a4012ae40069320d857d2680581be356e0305739dbc9a54bf135216a0af Oct 31 05:19:20.686006 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.685864 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:19:20.686006 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.685960 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:19:20.686164 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:20.686007 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:19:21.129370 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:21.125885 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-shld4" event={"ID":"a69fd138-9906-4127-ac26-38a21a5b7590","Type":"ContainerStarted","Data":"5b27446f159dfb99674d411fbf273a05be23fc41f1b67d49178e3a6250522a84"} Oct 31 05:19:21.129370 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:21.125958 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-shld4" event={"ID":"a69fd138-9906-4127-ac26-38a21a5b7590","Type":"ContainerStarted","Data":"494279b58c4c10e2d3e91f27bb10b0d6bc0e6e7527aeaf89595bb77a1c31bcfb"} Oct 31 05:19:21.130020 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:21.129387 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r" event={"ID":"68e465ba-2b5d-4e3c-8b63-7bdb9b695b75","Type":"ContainerStarted","Data":"6b9ac75cffe6195d5e699fe66e6e041d80dc5c281980dd779d5eb2f4336f4db5"} Oct 31 05:19:21.130020 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:21.129456 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r" event={"ID":"68e465ba-2b5d-4e3c-8b63-7bdb9b695b75","Type":"ContainerStarted","Data":"c2b830080a49c1da383ce5a799d803beefe687852efab9959178043b73ea8a06"} Oct 31 05:19:21.135129 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:21.132473 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rgj85" event={"ID":"8430b919-a9dd-4c7e-9266-ef6872cc0b9e","Type":"ContainerStarted","Data":"c80b15d3062232d999acefac8af532ffecc26fad7ee723b1844ce02b2f26ff30"} Oct 31 05:19:21.135129 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:21.132545 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rgj85" event={"ID":"8430b919-a9dd-4c7e-9266-ef6872cc0b9e","Type":"ContainerStarted","Data":"e06e2a1752bb60c4b305797486c420511fb099713ca3d6c6c3d0ac92b78971f7"} Oct 31 05:19:21.140355 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:21.137635 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5" event={"ID":"6d68d19b-5392-4308-9e2c-4b5e7593d17b","Type":"ContainerStarted","Data":"9666e8f7e8a4346b87d330f94513af1370fbff1a5925fb7e8de7c7c692989a2b"} Oct 31 05:19:21.140355 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:21.137695 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5" event={"ID":"6d68d19b-5392-4308-9e2c-4b5e7593d17b","Type":"ContainerStarted","Data":"52932a4012ae40069320d857d2680581be356e0305739dbc9a54bf135216a0af"} Oct 31 05:19:21.288792 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:21.288704 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rgj85" Oct 31 05:19:21.296454 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:21.294736 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-shld4" podStartSLOduration=2.29471066 podStartE2EDuration="2.29471066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:19:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:19:21.216854826 +0000 UTC m=+4515.973192874" watchObservedRunningTime="2026-10-31 05:19:21.29471066 +0000 UTC m=+4516.051048700" Oct 31 05:19:21.296454 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:21.295000 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5" podStartSLOduration=2.294993093 podStartE2EDuration="2.294993093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:19:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:19:21.284525821 +0000 UTC m=+4516.040863866" watchObservedRunningTime="2026-10-31 05:19:21.294993093 +0000 UTC m=+4516.051331136" Oct 31 05:19:21.297271 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:21.296891 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-rgj85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.218:15021/healthz/ready\": dial tcp 10.132.1.218:15021: connect: connection refused" start-of-body= Oct 31 05:19:21.297271 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:21.296959 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rgj85" podUID="8430b919-a9dd-4c7e-9266-ef6872cc0b9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.218:15021/healthz/ready\": dial tcp 10.132.1.218:15021: connect: connection refused" Oct 31 05:19:21.320360 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:21.307742 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-shld4" Oct 31 05:19:21.320360 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:21.312512 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-shld4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.219:15021/healthz/ready\": dial tcp 10.132.1.219:15021: connect: connection refused" start-of-body= Oct 31 05:19:21.320360 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:21.312581 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-shld4" podUID="a69fd138-9906-4127-ac26-38a21a5b7590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.219:15021/healthz/ready\": dial tcp 10.132.1.219:15021: connect: connection refused" Oct 31 05:19:21.329425 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:21.323190 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r" Oct 31 05:19:21.329425 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:21.323360 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.220:15021/healthz/ready\": dial tcp 10.132.1.220:15021: connect: connection refused" start-of-body= Oct 31 05:19:21.329425 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:21.323410 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r" podUID="68e465ba-2b5d-4e3c-8b63-7bdb9b695b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.220:15021/healthz/ready\": dial tcp 10.132.1.220:15021: connect: connection refused" Oct 31 05:19:21.363358 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:21.351130 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5" Oct 31 05:19:21.363358 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:21.351283 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.221:15021/healthz/ready\": dial tcp 10.132.1.221:15021: connect: connection refused" start-of-body= Oct 31 05:19:21.363358 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:21.351326 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5" podUID="6d68d19b-5392-4308-9e2c-4b5e7593d17b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.221:15021/healthz/ready\": dial tcp 10.132.1.221:15021: connect: connection refused" Oct 31 05:19:21.412638 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:21.412502 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rgj85" podStartSLOduration=2.412482821 podStartE2EDuration="2.412482821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:19:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:19:21.411932567 +0000 UTC m=+4516.168270608" watchObservedRunningTime="2026-10-31 05:19:21.412482821 +0000 UTC m=+4516.168820862" Oct 31 05:19:21.439359 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:21.423299 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r" podStartSLOduration=2.423277559 podStartE2EDuration="2.423277559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:19:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:19:21.343094015 +0000 UTC m=+4516.099432056" watchObservedRunningTime="2026-10-31 05:19:21.423277559 +0000 UTC m=+4516.179615601" Oct 31 05:19:22.289457 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:22.288847 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-rgj85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.218:15021/healthz/ready\": dial tcp 10.132.1.218:15021: connect: connection refused" start-of-body= Oct 31 05:19:22.289457 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:22.288921 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rgj85" podUID="8430b919-a9dd-4c7e-9266-ef6872cc0b9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.218:15021/healthz/ready\": dial tcp 10.132.1.218:15021: connect: connection refused" Oct 31 05:19:22.307924 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:22.306939 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-shld4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.219:15021/healthz/ready\": dial tcp 10.132.1.219:15021: connect: connection refused" start-of-body= Oct 31 05:19:22.307924 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:22.307010 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-shld4" podUID="a69fd138-9906-4127-ac26-38a21a5b7590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.219:15021/healthz/ready\": dial tcp 10.132.1.219:15021: connect: connection refused" Oct 31 05:19:22.321891 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:22.321843 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.220:15021/healthz/ready\": dial tcp 10.132.1.220:15021: connect: connection refused" start-of-body= Oct 31 05:19:22.322252 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:22.322199 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r" podUID="68e465ba-2b5d-4e3c-8b63-7bdb9b695b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.220:15021/healthz/ready\": dial tcp 10.132.1.220:15021: connect: connection refused" Oct 31 05:19:22.348949 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:22.348906 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.221:15021/healthz/ready\": dial tcp 10.132.1.221:15021: connect: connection refused" start-of-body= Oct 31 05:19:22.349302 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:22.349257 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5" podUID="6d68d19b-5392-4308-9e2c-4b5e7593d17b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.221:15021/healthz/ready\": dial tcp 10.132.1.221:15021: connect: connection refused" Oct 31 05:19:22.519321 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:19:22.519278 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:19:23.287190 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:23.287143 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-rgj85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.218:15021/healthz/ready\": dial tcp 10.132.1.218:15021: connect: connection refused" start-of-body= Oct 31 05:19:23.287521 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:23.287485 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rgj85" podUID="8430b919-a9dd-4c7e-9266-ef6872cc0b9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.218:15021/healthz/ready\": dial tcp 10.132.1.218:15021: connect: connection refused" Oct 31 05:19:23.313125 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:23.313081 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-shld4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.219:15021/healthz/ready\": dial tcp 10.132.1.219:15021: connect: connection refused" start-of-body= Oct 31 05:19:23.313878 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:23.313800 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-shld4" podUID="a69fd138-9906-4127-ac26-38a21a5b7590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.219:15021/healthz/ready\": dial tcp 10.132.1.219:15021: connect: connection refused" Oct 31 05:19:23.321869 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:23.321820 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.220:15021/healthz/ready\": dial tcp 10.132.1.220:15021: connect: connection refused" start-of-body= Oct 31 05:19:23.322156 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:23.322122 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r" podUID="68e465ba-2b5d-4e3c-8b63-7bdb9b695b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.220:15021/healthz/ready\": dial tcp 10.132.1.220:15021: connect: connection refused" Oct 31 05:19:23.360724 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:23.360684 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.221:15021/healthz/ready\": dial tcp 10.132.1.221:15021: connect: connection refused" start-of-body= Oct 31 05:19:23.360991 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:23.360962 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5" podUID="6d68d19b-5392-4308-9e2c-4b5e7593d17b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.221:15021/healthz/ready\": dial tcp 10.132.1.221:15021: connect: connection refused" Oct 31 05:19:24.312410 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:24.311693 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-shld4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.219:15021/healthz/ready\": dial tcp 10.132.1.219:15021: connect: connection refused" start-of-body= Oct 31 05:19:24.312410 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:24.311766 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-shld4" podUID="a69fd138-9906-4127-ac26-38a21a5b7590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.219:15021/healthz/ready\": dial tcp 10.132.1.219:15021: connect: connection refused" Oct 31 05:19:24.312410 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:24.312261 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-rgj85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.218:15021/healthz/ready\": dial tcp 10.132.1.218:15021: connect: connection refused" start-of-body= Oct 31 05:19:24.312410 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:24.312306 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rgj85" podUID="8430b919-a9dd-4c7e-9266-ef6872cc0b9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.218:15021/healthz/ready\": dial tcp 10.132.1.218:15021: connect: connection refused" Oct 31 05:19:24.324081 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:24.323916 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.220:15021/healthz/ready\": dial tcp 10.132.1.220:15021: connect: connection refused" start-of-body= Oct 31 05:19:24.324081 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:24.324009 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r" podUID="68e465ba-2b5d-4e3c-8b63-7bdb9b695b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.220:15021/healthz/ready\": dial tcp 10.132.1.220:15021: connect: connection refused" Oct 31 05:19:24.350377 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:24.349892 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.221:15021/healthz/ready\": dial tcp 10.132.1.221:15021: connect: connection refused" start-of-body= Oct 31 05:19:24.350377 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:24.349986 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5" podUID="6d68d19b-5392-4308-9e2c-4b5e7593d17b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.221:15021/healthz/ready\": dial tcp 10.132.1.221:15021: connect: connection refused" Oct 31 05:19:25.287360 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:25.286474 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-rgj85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.218:15021/healthz/ready\": dial tcp 10.132.1.218:15021: connect: connection refused" start-of-body= Oct 31 05:19:25.287360 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:25.286566 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rgj85" podUID="8430b919-a9dd-4c7e-9266-ef6872cc0b9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.218:15021/healthz/ready\": dial tcp 10.132.1.218:15021: connect: connection refused" Oct 31 05:19:25.308357 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:25.306957 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-shld4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.219:15021/healthz/ready\": dial tcp 10.132.1.219:15021: connect: connection refused" start-of-body= Oct 31 05:19:25.308357 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:25.307055 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-shld4" podUID="a69fd138-9906-4127-ac26-38a21a5b7590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.219:15021/healthz/ready\": dial tcp 10.132.1.219:15021: connect: connection refused" Oct 31 05:19:25.336703 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:25.336663 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.220:15021/healthz/ready\": dial tcp 10.132.1.220:15021: connect: connection refused" start-of-body= Oct 31 05:19:25.337380 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:25.337310 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r" podUID="68e465ba-2b5d-4e3c-8b63-7bdb9b695b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.220:15021/healthz/ready\": dial tcp 10.132.1.220:15021: connect: connection refused" Oct 31 05:19:25.349964 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:25.349829 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.221:15021/healthz/ready\": dial tcp 10.132.1.221:15021: connect: connection refused" start-of-body= Oct 31 05:19:25.349964 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:25.349900 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5" podUID="6d68d19b-5392-4308-9e2c-4b5e7593d17b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.221:15021/healthz/ready\": dial tcp 10.132.1.221:15021: connect: connection refused" Oct 31 05:19:26.302468 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:26.302408 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-rgj85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.218:15021/healthz/ready\": dial tcp 10.132.1.218:15021: connect: connection refused" start-of-body= Oct 31 05:19:26.302873 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:26.302832 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rgj85" podUID="8430b919-a9dd-4c7e-9266-ef6872cc0b9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.218:15021/healthz/ready\": dial tcp 10.132.1.218:15021: connect: connection refused" Oct 31 05:19:26.307871 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:26.307833 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-shld4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.219:15021/healthz/ready\": dial tcp 10.132.1.219:15021: connect: connection refused" start-of-body= Oct 31 05:19:26.308172 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:26.308120 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-shld4" podUID="a69fd138-9906-4127-ac26-38a21a5b7590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.219:15021/healthz/ready\": dial tcp 10.132.1.219:15021: connect: connection refused" Oct 31 05:19:26.320927 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:26.320882 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.220:15021/healthz/ready\": dial tcp 10.132.1.220:15021: connect: connection refused" start-of-body= Oct 31 05:19:26.321404 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:26.321307 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r" podUID="68e465ba-2b5d-4e3c-8b63-7bdb9b695b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.220:15021/healthz/ready\": dial tcp 10.132.1.220:15021: connect: connection refused" Oct 31 05:19:26.348845 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:26.348798 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.221:15021/healthz/ready\": dial tcp 10.132.1.221:15021: connect: connection refused" start-of-body= Oct 31 05:19:26.349520 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:26.349479 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5" podUID="6d68d19b-5392-4308-9e2c-4b5e7593d17b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.221:15021/healthz/ready\": dial tcp 10.132.1.221:15021: connect: connection refused" Oct 31 05:19:27.288361 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:27.286796 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-rgj85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.218:15021/healthz/ready\": dial tcp 10.132.1.218:15021: connect: connection refused" start-of-body= Oct 31 05:19:27.288361 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:27.286867 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rgj85" podUID="8430b919-a9dd-4c7e-9266-ef6872cc0b9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.218:15021/healthz/ready\": dial tcp 10.132.1.218:15021: connect: connection refused" Oct 31 05:19:27.314667 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:27.314602 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-shld4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.219:15021/healthz/ready\": dial tcp 10.132.1.219:15021: connect: connection refused" start-of-body= Oct 31 05:19:27.315149 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:27.315043 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-shld4" podUID="a69fd138-9906-4127-ac26-38a21a5b7590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.219:15021/healthz/ready\": dial tcp 10.132.1.219:15021: connect: connection refused" Oct 31 05:19:27.335868 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:27.334715 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.220:15021/healthz/ready\": dial tcp 10.132.1.220:15021: connect: connection refused" start-of-body= Oct 31 05:19:27.335868 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:27.334785 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r" podUID="68e465ba-2b5d-4e3c-8b63-7bdb9b695b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.220:15021/healthz/ready\": dial tcp 10.132.1.220:15021: connect: connection refused" Oct 31 05:19:27.348943 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:27.348887 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.221:15021/healthz/ready\": dial tcp 10.132.1.221:15021: connect: connection refused" start-of-body= Oct 31 05:19:27.349659 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:27.349619 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5" podUID="6d68d19b-5392-4308-9e2c-4b5e7593d17b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.221:15021/healthz/ready\": dial tcp 10.132.1.221:15021: connect: connection refused" Oct 31 05:19:27.577768 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:27.577657 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-85c4f7957c-rzwcg"] Oct 31 05:19:27.577989 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:27.577930 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-85c4f7957c-rzwcg" podUID="f6078cee-8fa7-49bf-a9c9-ce433f7236ed" containerName="manager" containerID="cri-o://39a6788c11e59e54d7aec9600e81cb812e873d96ce8a4759c13be4f0e4ff974a" gracePeriod=30 Oct 31 05:19:27.916499 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:27.916452 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-85c4f7957c-rzwcg" Oct 31 05:19:27.995107 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:27.994108 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6078cee-8fa7-49bf-a9c9-ce433f7236ed-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6078cee-8fa7-49bf-a9c9-ce433f7236ed-metrics-certs\") pod \"f6078cee-8fa7-49bf-a9c9-ce433f7236ed\" (UID: \"f6078cee-8fa7-49bf-a9c9-ce433f7236ed\") " Oct 31 05:19:27.995107 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:27.994202 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6078cee-8fa7-49bf-a9c9-ce433f7236ed-kube-api-access-8ldbc\" (UniqueName: \"kubernetes.io/projected/f6078cee-8fa7-49bf-a9c9-ce433f7236ed-kube-api-access-8ldbc\") pod \"f6078cee-8fa7-49bf-a9c9-ce433f7236ed\" (UID: \"f6078cee-8fa7-49bf-a9c9-ce433f7236ed\") " Oct 31 05:19:27.995107 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:27.994308 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6078cee-8fa7-49bf-a9c9-ce433f7236ed-webhook-cert\" (UniqueName: \"kubernetes.io/secret/f6078cee-8fa7-49bf-a9c9-ce433f7236ed-webhook-cert\") pod \"f6078cee-8fa7-49bf-a9c9-ce433f7236ed\" (UID: \"f6078cee-8fa7-49bf-a9c9-ce433f7236ed\") " Oct 31 05:19:27.998221 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:27.998149 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6078cee-8fa7-49bf-a9c9-ce433f7236ed-webhook-cert" pod "f6078cee-8fa7-49bf-a9c9-ce433f7236ed" (UID: "f6078cee-8fa7-49bf-a9c9-ce433f7236ed"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:19:27.999120 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:27.999072 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6078cee-8fa7-49bf-a9c9-ce433f7236ed-metrics-certs" pod "f6078cee-8fa7-49bf-a9c9-ce433f7236ed" (UID: "f6078cee-8fa7-49bf-a9c9-ce433f7236ed"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:19:28.000090 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:28.000053 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6078cee-8fa7-49bf-a9c9-ce433f7236ed-kube-api-access-8ldbc" pod "f6078cee-8fa7-49bf-a9c9-ce433f7236ed" (UID: "f6078cee-8fa7-49bf-a9c9-ce433f7236ed"). InnerVolumeSpecName "kube-api-access-8ldbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:19:28.096871 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:28.096782 5019 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6078cee-8fa7-49bf-a9c9-ce433f7236ed-metrics-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:19:28.096871 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:28.096827 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ldbc\" (UniqueName: \"kubernetes.io/projected/f6078cee-8fa7-49bf-a9c9-ce433f7236ed-kube-api-access-8ldbc\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:19:28.096871 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:28.096843 5019 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f6078cee-8fa7-49bf-a9c9-ce433f7236ed-webhook-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:19:28.285946 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:28.285892 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-rgj85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.218:15021/healthz/ready\": dial tcp 10.132.1.218:15021: connect: connection refused" start-of-body= Oct 31 05:19:28.286192 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:28.285972 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rgj85" podUID="8430b919-a9dd-4c7e-9266-ef6872cc0b9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.218:15021/healthz/ready\": dial tcp 10.132.1.218:15021: connect: connection refused" Oct 31 05:19:28.305977 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:28.305928 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-shld4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.219:15021/healthz/ready\": dial tcp 10.132.1.219:15021: connect: connection refused" start-of-body= Oct 31 05:19:28.306221 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:28.306002 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-shld4" podUID="a69fd138-9906-4127-ac26-38a21a5b7590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.219:15021/healthz/ready\": dial tcp 10.132.1.219:15021: connect: connection refused" Oct 31 05:19:28.321063 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:28.321014 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.220:15021/healthz/ready\": dial tcp 10.132.1.220:15021: connect: connection refused" start-of-body= Oct 31 05:19:28.321265 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:28.321098 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r" podUID="68e465ba-2b5d-4e3c-8b63-7bdb9b695b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.220:15021/healthz/ready\": dial tcp 10.132.1.220:15021: connect: connection refused" Oct 31 05:19:28.347848 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:28.347799 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.221:15021/healthz/ready\": dial tcp 10.132.1.221:15021: connect: connection refused" start-of-body= Oct 31 05:19:28.348096 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:28.347872 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5" podUID="6d68d19b-5392-4308-9e2c-4b5e7593d17b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.221:15021/healthz/ready\": dial tcp 10.132.1.221:15021: connect: connection refused" Oct 31 05:19:28.364397 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:28.364312 5019 generic.go:356] "Generic (PLEG): container finished" podID="f6078cee-8fa7-49bf-a9c9-ce433f7236ed" containerID="39a6788c11e59e54d7aec9600e81cb812e873d96ce8a4759c13be4f0e4ff974a" exitCode=0 Oct 31 05:19:28.365023 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:28.364395 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-85c4f7957c-rzwcg" Oct 31 05:19:28.365185 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:28.364418 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-85c4f7957c-rzwcg" event={"ID":"f6078cee-8fa7-49bf-a9c9-ce433f7236ed","Type":"ContainerDied","Data":"39a6788c11e59e54d7aec9600e81cb812e873d96ce8a4759c13be4f0e4ff974a"} Oct 31 05:19:28.365185 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:28.365171 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-85c4f7957c-rzwcg" event={"ID":"f6078cee-8fa7-49bf-a9c9-ce433f7236ed","Type":"ContainerDied","Data":"40fb90c3c76f91cc3c76e707b29638fc4c500e4558b69a9a61affcedd146f857"} Oct 31 05:19:28.365390 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:28.365212 5019 scope.go:122] "RemoveContainer" containerID="39a6788c11e59e54d7aec9600e81cb812e873d96ce8a4759c13be4f0e4ff974a" Oct 31 05:19:28.381470 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:28.381438 5019 scope.go:122] "RemoveContainer" containerID="39a6788c11e59e54d7aec9600e81cb812e873d96ce8a4759c13be4f0e4ff974a" Oct 31 05:19:28.381958 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:19:28.381924 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39a6788c11e59e54d7aec9600e81cb812e873d96ce8a4759c13be4f0e4ff974a\": container with ID starting with 39a6788c11e59e54d7aec9600e81cb812e873d96ce8a4759c13be4f0e4ff974a not found: ID does not exist" containerID="39a6788c11e59e54d7aec9600e81cb812e873d96ce8a4759c13be4f0e4ff974a" Oct 31 05:19:28.382083 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:28.381973 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39a6788c11e59e54d7aec9600e81cb812e873d96ce8a4759c13be4f0e4ff974a"} err="failed to get container status \"39a6788c11e59e54d7aec9600e81cb812e873d96ce8a4759c13be4f0e4ff974a\": rpc error: code = NotFound desc = could not find container \"39a6788c11e59e54d7aec9600e81cb812e873d96ce8a4759c13be4f0e4ff974a\": container with ID starting with 39a6788c11e59e54d7aec9600e81cb812e873d96ce8a4759c13be4f0e4ff974a not found: ID does not exist" Oct 31 05:19:28.420611 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:28.420568 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-85c4f7957c-rzwcg"] Oct 31 05:19:28.425626 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:28.425579 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-85c4f7957c-rzwcg"] Oct 31 05:19:29.569352 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:29.568755 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6078cee-8fa7-49bf-a9c9-ce433f7236ed" path="/var/lib/kubelet/pods/f6078cee-8fa7-49bf-a9c9-ce433f7236ed/volumes" Oct 31 05:19:30.325706 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:30.324265 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-rgj85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.218:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 05:19:30.325706 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:30.324393 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rgj85" podUID="8430b919-a9dd-4c7e-9266-ef6872cc0b9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.218:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 05:19:30.341514 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:30.341458 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 05:19:30.341758 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:30.341532 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r" podUID="68e465ba-2b5d-4e3c-8b63-7bdb9b695b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 05:19:30.356044 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:30.355517 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-shld4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 05:19:30.356044 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:30.355602 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-shld4" podUID="a69fd138-9906-4127-ac26-38a21a5b7590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 05:19:30.365320 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:30.365282 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rgj85" Oct 31 05:19:30.366121 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:30.366092 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rgj85" Oct 31 05:19:30.372544 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:30.370900 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rgj85" Oct 31 05:19:30.391612 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:30.391574 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.221:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 05:19:30.391927 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:30.391901 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5" podUID="6d68d19b-5392-4308-9e2c-4b5e7593d17b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.221:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 05:19:30.396214 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:30.395535 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r" Oct 31 05:19:30.396449 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:30.396391 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r" Oct 31 05:19:30.397934 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:30.397882 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r" Oct 31 05:19:30.486272 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:30.485679 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-shld4" Oct 31 05:19:30.486986 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:30.486805 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-shld4" Oct 31 05:19:30.489234 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:30.489206 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-shld4" Oct 31 05:19:30.569421 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:30.569386 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5" Oct 31 05:19:30.570608 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:30.570583 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5" Oct 31 05:19:30.571268 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:30.571248 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5" Oct 31 05:19:52.513433 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:19:52.503936 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:19:54.688041 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:54.688000 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b9vls"] Oct 31 05:19:54.688519 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:54.688502 5019 cpu_manager.go:414] "containerMap: removing container" podUID="f6078cee-8fa7-49bf-a9c9-ce433f7236ed" containerName="manager" Oct 31 05:19:54.688519 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:54.688519 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6078cee-8fa7-49bf-a9c9-ce433f7236ed" containerName="manager" Oct 31 05:19:54.688624 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:54.688614 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6078cee-8fa7-49bf-a9c9-ce433f7236ed" containerName="manager" Oct 31 05:19:54.693189 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:54.693162 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b9vls" Oct 31 05:19:54.712356 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:54.712284 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b9vls"] Oct 31 05:19:54.720242 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:54.720202 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:19:54.720512 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:54.720372 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfhzn\"" Oct 31 05:19:54.720512 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:54.720218 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:19:54.720968 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:54.720940 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:19:54.803816 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:54.803769 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37f41b51-5216-424f-9e18-81c46b69ffb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-b9vls\" (UID: \"37f41b51-5216-424f-9e18-81c46b69ffb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b9vls" Oct 31 05:19:54.804033 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:54.803850 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6h7r\" (UniqueName: \"kubernetes.io/projected/37f41b51-5216-424f-9e18-81c46b69ffb2-kube-api-access-d6h7r\") pod \"maas-api-revoke-sub-f904bff9-b9vls\" (UID: \"37f41b51-5216-424f-9e18-81c46b69ffb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b9vls" Oct 31 05:19:54.804033 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:54.803952 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37f41b51-5216-424f-9e18-81c46b69ffb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-b9vls\" (UID: \"37f41b51-5216-424f-9e18-81c46b69ffb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b9vls" Oct 31 05:19:54.905449 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:54.905402 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6h7r\" (UniqueName: \"kubernetes.io/projected/37f41b51-5216-424f-9e18-81c46b69ffb2-kube-api-access-d6h7r\") pod \"maas-api-revoke-sub-f904bff9-b9vls\" (UID: \"37f41b51-5216-424f-9e18-81c46b69ffb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b9vls" Oct 31 05:19:54.905659 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:54.905459 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37f41b51-5216-424f-9e18-81c46b69ffb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-b9vls\" (UID: \"37f41b51-5216-424f-9e18-81c46b69ffb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b9vls" Oct 31 05:19:54.905659 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:54.905549 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37f41b51-5216-424f-9e18-81c46b69ffb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-b9vls\" (UID: \"37f41b51-5216-424f-9e18-81c46b69ffb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b9vls" Oct 31 05:19:54.906211 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:54.906184 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37f41b51-5216-424f-9e18-81c46b69ffb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-b9vls\" (UID: \"37f41b51-5216-424f-9e18-81c46b69ffb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b9vls" Oct 31 05:19:54.908104 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:54.908078 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37f41b51-5216-424f-9e18-81c46b69ffb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-b9vls\" (UID: \"37f41b51-5216-424f-9e18-81c46b69ffb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b9vls" Oct 31 05:19:54.929390 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:54.929322 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6h7r\" (UniqueName: \"kubernetes.io/projected/37f41b51-5216-424f-9e18-81c46b69ffb2-kube-api-access-d6h7r\") pod \"maas-api-revoke-sub-f904bff9-b9vls\" (UID: \"37f41b51-5216-424f-9e18-81c46b69ffb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b9vls" Oct 31 05:19:55.010491 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:55.010435 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b9vls" Oct 31 05:19:55.311773 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:55.311679 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b9vls"] Oct 31 05:19:55.316322 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:19:55.316280 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37f41b51_5216_424f_9e18_81c46b69ffb2.slice/crio-2af937155b2dbaf66719dd847b546ff72e17b3cb90e03a0fdd31bff6bd398303 WatchSource:0}: Error finding container 2af937155b2dbaf66719dd847b546ff72e17b3cb90e03a0fdd31bff6bd398303: Status 404 returned error can't find the container with id 2af937155b2dbaf66719dd847b546ff72e17b3cb90e03a0fdd31bff6bd398303 Oct 31 05:19:55.794821 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:55.794780 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b9vls" event={"ID":"37f41b51-5216-424f-9e18-81c46b69ffb2","Type":"ContainerStarted","Data":"9834ac8795f6a278d2c53cd862a02c3488739e27b55717b16fecdef0d3526517"} Oct 31 05:19:55.795378 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:55.794864 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b9vls" event={"ID":"37f41b51-5216-424f-9e18-81c46b69ffb2","Type":"ContainerStarted","Data":"2af937155b2dbaf66719dd847b546ff72e17b3cb90e03a0fdd31bff6bd398303"} Oct 31 05:19:55.837932 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:55.837869 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b9vls" podStartSLOduration=1.8378528570000001 podStartE2EDuration="1.837852857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:19:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:19:55.834672957 +0000 UTC m=+4550.591010998" watchObservedRunningTime="2026-10-31 05:19:55.837852857 +0000 UTC m=+4550.594190933" Oct 31 05:19:56.802673 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:56.802628 5019 generic.go:356] "Generic (PLEG): container finished" podID="37f41b51-5216-424f-9e18-81c46b69ffb2" containerID="9834ac8795f6a278d2c53cd862a02c3488739e27b55717b16fecdef0d3526517" exitCode=0 Oct 31 05:19:56.803085 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:56.802723 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b9vls" event={"ID":"37f41b51-5216-424f-9e18-81c46b69ffb2","Type":"ContainerDied","Data":"9834ac8795f6a278d2c53cd862a02c3488739e27b55717b16fecdef0d3526517"} Oct 31 05:19:57.968831 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:57.968803 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b9vls" Oct 31 05:19:58.035406 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:58.035366 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37f41b51-5216-424f-9e18-81c46b69ffb2-kube-api-access-d6h7r\" (UniqueName: \"kubernetes.io/projected/37f41b51-5216-424f-9e18-81c46b69ffb2-kube-api-access-d6h7r\") pod \"37f41b51-5216-424f-9e18-81c46b69ffb2\" (UID: \"37f41b51-5216-424f-9e18-81c46b69ffb2\") " Oct 31 05:19:58.035614 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:58.035488 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37f41b51-5216-424f-9e18-81c46b69ffb2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37f41b51-5216-424f-9e18-81c46b69ffb2-maas-api-xks-ca\") pod \"37f41b51-5216-424f-9e18-81c46b69ffb2\" (UID: \"37f41b51-5216-424f-9e18-81c46b69ffb2\") " Oct 31 05:19:58.035614 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:58.035532 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37f41b51-5216-424f-9e18-81c46b69ffb2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37f41b51-5216-424f-9e18-81c46b69ffb2-maas-api-openshift-ca\") pod \"37f41b51-5216-424f-9e18-81c46b69ffb2\" (UID: \"37f41b51-5216-424f-9e18-81c46b69ffb2\") " Oct 31 05:19:58.036020 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:58.035985 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37f41b51-5216-424f-9e18-81c46b69ffb2-maas-api-openshift-ca" pod "37f41b51-5216-424f-9e18-81c46b69ffb2" (UID: "37f41b51-5216-424f-9e18-81c46b69ffb2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:19:58.037802 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:58.037768 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37f41b51-5216-424f-9e18-81c46b69ffb2-kube-api-access-d6h7r" pod "37f41b51-5216-424f-9e18-81c46b69ffb2" (UID: "37f41b51-5216-424f-9e18-81c46b69ffb2"). InnerVolumeSpecName "kube-api-access-d6h7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:19:58.037950 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:58.037846 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37f41b51-5216-424f-9e18-81c46b69ffb2-maas-api-xks-ca" pod "37f41b51-5216-424f-9e18-81c46b69ffb2" (UID: "37f41b51-5216-424f-9e18-81c46b69ffb2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:19:58.137400 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:58.137281 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6h7r\" (UniqueName: \"kubernetes.io/projected/37f41b51-5216-424f-9e18-81c46b69ffb2-kube-api-access-d6h7r\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:19:58.137400 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:58.137325 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37f41b51-5216-424f-9e18-81c46b69ffb2-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:19:58.137400 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:58.137351 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37f41b51-5216-424f-9e18-81c46b69ffb2-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:19:58.817685 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:58.817647 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b9vls" Oct 31 05:19:58.817900 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:58.817684 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b9vls" event={"ID":"37f41b51-5216-424f-9e18-81c46b69ffb2","Type":"ContainerDied","Data":"2af937155b2dbaf66719dd847b546ff72e17b3cb90e03a0fdd31bff6bd398303"} Oct 31 05:19:58.817900 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:19:58.817739 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2af937155b2dbaf66719dd847b546ff72e17b3cb90e03a0fdd31bff6bd398303" Oct 31 05:20:14.235591 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:14.235541 5019 scope.go:122] "RemoveContainer" containerID="c1a5a492e11ae9f239042814de50379ae2dd961c1be57948d30d9b4ba983244d" Oct 31 05:20:14.247684 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:14.247657 5019 scope.go:122] "RemoveContainer" containerID="1c741c6c8dee8fad5e0e7c6a46c50f8756f92222365ccb310742a2099e35a20b" Oct 31 05:20:20.419506 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:20.419459 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hgx4g"] Oct 31 05:20:20.419985 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:20.419968 5019 cpu_manager.go:414] "containerMap: removing container" podUID="37f41b51-5216-424f-9e18-81c46b69ffb2" containerName="revoke-keys" Oct 31 05:20:20.420034 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:20.419987 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37f41b51-5216-424f-9e18-81c46b69ffb2" containerName="revoke-keys" Oct 31 05:20:20.420089 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:20.420080 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="37f41b51-5216-424f-9e18-81c46b69ffb2" containerName="revoke-keys" Oct 31 05:20:20.432277 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:20.424178 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hgx4g" Oct 31 05:20:20.445608 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:20.445565 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hgx4g"] Oct 31 05:20:20.451489 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:20.451317 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:20:20.451489 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:20.451476 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:20:20.451795 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:20.451628 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfhzn\"" Oct 31 05:20:20.451795 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:20.451689 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:20:20.507400 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:20.507289 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1884c87e-c15e-490f-b09d-c54dad973f44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-hgx4g\" (UID: \"1884c87e-c15e-490f-b09d-c54dad973f44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hgx4g" Oct 31 05:20:20.507637 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:20.507451 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhdz5\" (UniqueName: \"kubernetes.io/projected/1884c87e-c15e-490f-b09d-c54dad973f44-kube-api-access-qhdz5\") pod \"maas-api-revoke-sub-f03629db-hgx4g\" (UID: \"1884c87e-c15e-490f-b09d-c54dad973f44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hgx4g" Oct 31 05:20:20.507637 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:20.507505 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1884c87e-c15e-490f-b09d-c54dad973f44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-hgx4g\" (UID: \"1884c87e-c15e-490f-b09d-c54dad973f44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hgx4g" Oct 31 05:20:20.608319 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:20.608272 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1884c87e-c15e-490f-b09d-c54dad973f44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-hgx4g\" (UID: \"1884c87e-c15e-490f-b09d-c54dad973f44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hgx4g" Oct 31 05:20:20.608319 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:20.608321 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhdz5\" (UniqueName: \"kubernetes.io/projected/1884c87e-c15e-490f-b09d-c54dad973f44-kube-api-access-qhdz5\") pod \"maas-api-revoke-sub-f03629db-hgx4g\" (UID: \"1884c87e-c15e-490f-b09d-c54dad973f44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hgx4g" Oct 31 05:20:20.608672 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:20.608393 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1884c87e-c15e-490f-b09d-c54dad973f44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-hgx4g\" (UID: \"1884c87e-c15e-490f-b09d-c54dad973f44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hgx4g" Oct 31 05:20:20.609144 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:20.609119 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1884c87e-c15e-490f-b09d-c54dad973f44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-hgx4g\" (UID: \"1884c87e-c15e-490f-b09d-c54dad973f44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hgx4g" Oct 31 05:20:20.611094 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:20.611065 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1884c87e-c15e-490f-b09d-c54dad973f44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-hgx4g\" (UID: \"1884c87e-c15e-490f-b09d-c54dad973f44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hgx4g" Oct 31 05:20:20.632817 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:20.632777 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhdz5\" (UniqueName: \"kubernetes.io/projected/1884c87e-c15e-490f-b09d-c54dad973f44-kube-api-access-qhdz5\") pod \"maas-api-revoke-sub-f03629db-hgx4g\" (UID: \"1884c87e-c15e-490f-b09d-c54dad973f44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hgx4g" Oct 31 05:20:20.747505 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:20.747458 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hgx4g" Oct 31 05:20:21.057934 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:21.057897 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hgx4g"] Oct 31 05:20:21.058741 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:20:21.058709 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1884c87e_c15e_490f_b09d_c54dad973f44.slice/crio-265bcd560120b4f8225c8f22d5f81b536dfe2da99f2de6ec1dbca2b35fd62e62 WatchSource:0}: Error finding container 265bcd560120b4f8225c8f22d5f81b536dfe2da99f2de6ec1dbca2b35fd62e62: Status 404 returned error can't find the container with id 265bcd560120b4f8225c8f22d5f81b536dfe2da99f2de6ec1dbca2b35fd62e62 Oct 31 05:20:21.185147 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:21.185090 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hgx4g" event={"ID":"1884c87e-c15e-490f-b09d-c54dad973f44","Type":"ContainerStarted","Data":"265bcd560120b4f8225c8f22d5f81b536dfe2da99f2de6ec1dbca2b35fd62e62"} Oct 31 05:20:22.194412 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:22.194371 5019 generic.go:356] "Generic (PLEG): container finished" podID="1884c87e-c15e-490f-b09d-c54dad973f44" containerID="74a91bad9b4198d132ea668eac7e594e60ee2d9d51657b461022547b41b67e83" exitCode=0 Oct 31 05:20:22.194848 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:22.194546 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hgx4g" event={"ID":"1884c87e-c15e-490f-b09d-c54dad973f44","Type":"ContainerDied","Data":"74a91bad9b4198d132ea668eac7e594e60ee2d9d51657b461022547b41b67e83"} Oct 31 05:20:22.394661 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:20:22.394552 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:20:23.356959 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:23.356922 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hgx4g" Oct 31 05:20:23.435816 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:23.435766 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1884c87e-c15e-490f-b09d-c54dad973f44-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1884c87e-c15e-490f-b09d-c54dad973f44-maas-api-openshift-ca\") pod \"1884c87e-c15e-490f-b09d-c54dad973f44\" (UID: \"1884c87e-c15e-490f-b09d-c54dad973f44\") " Oct 31 05:20:23.436079 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:23.435848 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1884c87e-c15e-490f-b09d-c54dad973f44-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1884c87e-c15e-490f-b09d-c54dad973f44-maas-api-xks-ca\") pod \"1884c87e-c15e-490f-b09d-c54dad973f44\" (UID: \"1884c87e-c15e-490f-b09d-c54dad973f44\") " Oct 31 05:20:23.436079 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:23.435901 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1884c87e-c15e-490f-b09d-c54dad973f44-kube-api-access-qhdz5\" (UniqueName: \"kubernetes.io/projected/1884c87e-c15e-490f-b09d-c54dad973f44-kube-api-access-qhdz5\") pod \"1884c87e-c15e-490f-b09d-c54dad973f44\" (UID: \"1884c87e-c15e-490f-b09d-c54dad973f44\") " Oct 31 05:20:23.436503 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:23.436459 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1884c87e-c15e-490f-b09d-c54dad973f44-maas-api-openshift-ca" pod "1884c87e-c15e-490f-b09d-c54dad973f44" (UID: "1884c87e-c15e-490f-b09d-c54dad973f44"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:20:23.438426 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:23.438378 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1884c87e-c15e-490f-b09d-c54dad973f44-kube-api-access-qhdz5" pod "1884c87e-c15e-490f-b09d-c54dad973f44" (UID: "1884c87e-c15e-490f-b09d-c54dad973f44"). InnerVolumeSpecName "kube-api-access-qhdz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:20:23.438426 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:23.438412 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1884c87e-c15e-490f-b09d-c54dad973f44-maas-api-xks-ca" pod "1884c87e-c15e-490f-b09d-c54dad973f44" (UID: "1884c87e-c15e-490f-b09d-c54dad973f44"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:20:23.537229 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:23.537172 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1884c87e-c15e-490f-b09d-c54dad973f44-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:20:23.537229 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:23.537219 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1884c87e-c15e-490f-b09d-c54dad973f44-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:20:23.537229 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:23.537232 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhdz5\" (UniqueName: \"kubernetes.io/projected/1884c87e-c15e-490f-b09d-c54dad973f44-kube-api-access-qhdz5\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:20:24.212435 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:24.212375 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hgx4g" event={"ID":"1884c87e-c15e-490f-b09d-c54dad973f44","Type":"ContainerDied","Data":"265bcd560120b4f8225c8f22d5f81b536dfe2da99f2de6ec1dbca2b35fd62e62"} Oct 31 05:20:24.212693 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:24.212454 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="265bcd560120b4f8225c8f22d5f81b536dfe2da99f2de6ec1dbca2b35fd62e62" Oct 31 05:20:24.212693 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:24.212405 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hgx4g" Oct 31 05:20:26.038661 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:26.038588 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2648aa75-lmdxc"] Oct 31 05:20:26.051030 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:26.050994 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91889c70-sjqvh"] Oct 31 05:20:26.053968 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:26.053933 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2648aa75-lmdxc"] Oct 31 05:20:26.056869 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:26.056831 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91889c70-sjqvh"] Oct 31 05:20:27.407272 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:27.407224 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67d355b0-4e8e-4b21-8250-aba5bbeb4932" path="/var/lib/kubelet/pods/67d355b0-4e8e-4b21-8250-aba5bbeb4932/volumes" Oct 31 05:20:27.407868 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:27.407730 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f61145c7-2f3f-4072-b329-5760f95c042f" path="/var/lib/kubelet/pods/f61145c7-2f3f-4072-b329-5760f95c042f/volumes" Oct 31 05:20:36.046048 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:36.045992 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-871d7380-rmvwj"] Oct 31 05:20:36.055820 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:36.055767 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-871d7380-rmvwj"] Oct 31 05:20:37.016851 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:37.016814 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f64dd"] Oct 31 05:20:37.017283 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:37.017267 5019 cpu_manager.go:414] "containerMap: removing container" podUID="1884c87e-c15e-490f-b09d-c54dad973f44" containerName="revoke-keys" Oct 31 05:20:37.017360 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:37.017285 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1884c87e-c15e-490f-b09d-c54dad973f44" containerName="revoke-keys" Oct 31 05:20:37.017408 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:37.017393 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="1884c87e-c15e-490f-b09d-c54dad973f44" containerName="revoke-keys" Oct 31 05:20:37.021652 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:37.021624 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f64dd" Oct 31 05:20:37.031696 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:37.031659 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59f0eb97-xt2rg"] Oct 31 05:20:37.039979 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:37.039944 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:20:37.040172 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:37.040096 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfhzn\"" Oct 31 05:20:37.042482 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:37.042421 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:20:37.042482 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:37.042431 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:20:37.043685 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:37.043661 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59f0eb97-xt2rg"] Oct 31 05:20:37.047056 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:37.047025 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f64dd"] Oct 31 05:20:37.154655 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:37.154613 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb37794d-be69-4586-9e19-bf3b9600b200-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-f64dd\" (UID: \"eb37794d-be69-4586-9e19-bf3b9600b200\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f64dd" Oct 31 05:20:37.154655 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:37.154665 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6czps\" (UniqueName: \"kubernetes.io/projected/eb37794d-be69-4586-9e19-bf3b9600b200-kube-api-access-6czps\") pod \"maas-api-revoke-sub-0165d7d2-f64dd\" (UID: \"eb37794d-be69-4586-9e19-bf3b9600b200\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f64dd" Oct 31 05:20:37.154974 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:37.154849 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb37794d-be69-4586-9e19-bf3b9600b200-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-f64dd\" (UID: \"eb37794d-be69-4586-9e19-bf3b9600b200\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f64dd" Oct 31 05:20:37.255669 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:37.255624 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb37794d-be69-4586-9e19-bf3b9600b200-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-f64dd\" (UID: \"eb37794d-be69-4586-9e19-bf3b9600b200\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f64dd" Oct 31 05:20:37.255886 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:37.255763 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb37794d-be69-4586-9e19-bf3b9600b200-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-f64dd\" (UID: \"eb37794d-be69-4586-9e19-bf3b9600b200\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f64dd" Oct 31 05:20:37.255886 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:37.255803 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6czps\" (UniqueName: \"kubernetes.io/projected/eb37794d-be69-4586-9e19-bf3b9600b200-kube-api-access-6czps\") pod \"maas-api-revoke-sub-0165d7d2-f64dd\" (UID: \"eb37794d-be69-4586-9e19-bf3b9600b200\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f64dd" Oct 31 05:20:37.256577 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:37.256541 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb37794d-be69-4586-9e19-bf3b9600b200-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-f64dd\" (UID: \"eb37794d-be69-4586-9e19-bf3b9600b200\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f64dd" Oct 31 05:20:37.258193 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:37.258164 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb37794d-be69-4586-9e19-bf3b9600b200-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-f64dd\" (UID: \"eb37794d-be69-4586-9e19-bf3b9600b200\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f64dd" Oct 31 05:20:37.279623 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:37.279534 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6czps\" (UniqueName: \"kubernetes.io/projected/eb37794d-be69-4586-9e19-bf3b9600b200-kube-api-access-6czps\") pod \"maas-api-revoke-sub-0165d7d2-f64dd\" (UID: \"eb37794d-be69-4586-9e19-bf3b9600b200\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f64dd" Oct 31 05:20:37.337402 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:37.337360 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f64dd" Oct 31 05:20:37.411363 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:37.411306 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b4bf2ad-96b8-4db2-ac9c-38dc92511e29" path="/var/lib/kubelet/pods/1b4bf2ad-96b8-4db2-ac9c-38dc92511e29/volumes" Oct 31 05:20:37.412544 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:37.412508 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="758a18f6-0240-4fc6-afcf-7c8180a71733" path="/var/lib/kubelet/pods/758a18f6-0240-4fc6-afcf-7c8180a71733/volumes" Oct 31 05:20:37.638496 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:37.638462 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f64dd"] Oct 31 05:20:37.640091 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:20:37.640061 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb37794d_be69_4586_9e19_bf3b9600b200.slice/crio-bb71b3cb87826c3a4e0e641b1df5d177ca6f6fae982dd3b3d0f6f7197e61b6ef WatchSource:0}: Error finding container bb71b3cb87826c3a4e0e641b1df5d177ca6f6fae982dd3b3d0f6f7197e61b6ef: Status 404 returned error can't find the container with id bb71b3cb87826c3a4e0e641b1df5d177ca6f6fae982dd3b3d0f6f7197e61b6ef Oct 31 05:20:38.367030 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:38.366980 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f64dd" event={"ID":"eb37794d-be69-4586-9e19-bf3b9600b200","Type":"ContainerStarted","Data":"11e456b5c1c0ed5618e1325f57f4ea10d56977daaeb42372b4120a008f7166f5"} Oct 31 05:20:38.367528 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:38.367056 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f64dd" event={"ID":"eb37794d-be69-4586-9e19-bf3b9600b200","Type":"ContainerStarted","Data":"bb71b3cb87826c3a4e0e641b1df5d177ca6f6fae982dd3b3d0f6f7197e61b6ef"} Oct 31 05:20:38.407153 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:38.407086 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f64dd" podStartSLOduration=2.40707101 podStartE2EDuration="2.40707101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:20:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:20:38.406933834 +0000 UTC m=+4593.163271873" watchObservedRunningTime="2026-10-31 05:20:38.40707101 +0000 UTC m=+4593.163409050" Oct 31 05:20:39.374789 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:39.374750 5019 generic.go:356] "Generic (PLEG): container finished" podID="eb37794d-be69-4586-9e19-bf3b9600b200" containerID="11e456b5c1c0ed5618e1325f57f4ea10d56977daaeb42372b4120a008f7166f5" exitCode=0 Oct 31 05:20:39.375257 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:39.374845 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f64dd" event={"ID":"eb37794d-be69-4586-9e19-bf3b9600b200","Type":"ContainerDied","Data":"11e456b5c1c0ed5618e1325f57f4ea10d56977daaeb42372b4120a008f7166f5"} Oct 31 05:20:40.544211 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:40.544180 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f64dd" Oct 31 05:20:40.691711 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:40.691600 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb37794d-be69-4586-9e19-bf3b9600b200-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb37794d-be69-4586-9e19-bf3b9600b200-maas-api-openshift-ca\") pod \"eb37794d-be69-4586-9e19-bf3b9600b200\" (UID: \"eb37794d-be69-4586-9e19-bf3b9600b200\") " Oct 31 05:20:40.691915 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:40.691756 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb37794d-be69-4586-9e19-bf3b9600b200-kube-api-access-6czps\" (UniqueName: \"kubernetes.io/projected/eb37794d-be69-4586-9e19-bf3b9600b200-kube-api-access-6czps\") pod \"eb37794d-be69-4586-9e19-bf3b9600b200\" (UID: \"eb37794d-be69-4586-9e19-bf3b9600b200\") " Oct 31 05:20:40.691915 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:40.691800 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb37794d-be69-4586-9e19-bf3b9600b200-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb37794d-be69-4586-9e19-bf3b9600b200-maas-api-xks-ca\") pod \"eb37794d-be69-4586-9e19-bf3b9600b200\" (UID: \"eb37794d-be69-4586-9e19-bf3b9600b200\") " Oct 31 05:20:40.692072 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:40.692039 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb37794d-be69-4586-9e19-bf3b9600b200-maas-api-openshift-ca" pod "eb37794d-be69-4586-9e19-bf3b9600b200" (UID: "eb37794d-be69-4586-9e19-bf3b9600b200"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:20:40.692252 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:40.692234 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb37794d-be69-4586-9e19-bf3b9600b200-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:20:40.693978 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:40.693949 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb37794d-be69-4586-9e19-bf3b9600b200-kube-api-access-6czps" pod "eb37794d-be69-4586-9e19-bf3b9600b200" (UID: "eb37794d-be69-4586-9e19-bf3b9600b200"). InnerVolumeSpecName "kube-api-access-6czps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:20:40.694094 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:40.694015 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb37794d-be69-4586-9e19-bf3b9600b200-maas-api-xks-ca" pod "eb37794d-be69-4586-9e19-bf3b9600b200" (UID: "eb37794d-be69-4586-9e19-bf3b9600b200"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:20:40.793436 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:40.793389 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6czps\" (UniqueName: \"kubernetes.io/projected/eb37794d-be69-4586-9e19-bf3b9600b200-kube-api-access-6czps\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:20:40.793436 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:40.793427 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb37794d-be69-4586-9e19-bf3b9600b200-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:20:41.388177 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:41.388135 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f64dd" Oct 31 05:20:41.388413 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:41.388144 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f64dd" event={"ID":"eb37794d-be69-4586-9e19-bf3b9600b200","Type":"ContainerDied","Data":"bb71b3cb87826c3a4e0e641b1df5d177ca6f6fae982dd3b3d0f6f7197e61b6ef"} Oct 31 05:20:41.388413 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:41.388270 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb71b3cb87826c3a4e0e641b1df5d177ca6f6fae982dd3b3d0f6f7197e61b6ef" Oct 31 05:20:45.029846 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:45.029807 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f887033-pwvhc"] Oct 31 05:20:45.041811 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:45.041773 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f887033-pwvhc"] Oct 31 05:20:45.406289 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:45.406203 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edc66a25-6634-4ccf-bc03-a29e87a8ac4b" path="/var/lib/kubelet/pods/edc66a25-6634-4ccf-bc03-a29e87a8ac4b/volumes" Oct 31 05:20:47.972075 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:47.972030 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hrdvp"] Oct 31 05:20:47.972780 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:47.972743 5019 cpu_manager.go:414] "containerMap: removing container" podUID="eb37794d-be69-4586-9e19-bf3b9600b200" containerName="revoke-keys" Oct 31 05:20:47.972780 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:47.972769 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb37794d-be69-4586-9e19-bf3b9600b200" containerName="revoke-keys" Oct 31 05:20:47.972952 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:47.972933 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb37794d-be69-4586-9e19-bf3b9600b200" containerName="revoke-keys" Oct 31 05:20:47.976324 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:47.976293 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hrdvp" Oct 31 05:20:47.991886 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:47.991843 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:20:48.004011 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:48.003965 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:20:48.004011 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:48.003999 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:20:48.004290 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:48.004247 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfhzn\"" Oct 31 05:20:48.007160 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:48.007124 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hrdvp"] Oct 31 05:20:48.030112 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:48.030070 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a70042a9-px695"] Oct 31 05:20:48.035880 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:48.035732 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a70042a9-px695"] Oct 31 05:20:48.072672 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:48.072623 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptd92\" (UniqueName: \"kubernetes.io/projected/614fb8da-43f7-49cc-acb4-dbf2befda850-kube-api-access-ptd92\") pod \"maas-api-revoke-sub-6b19338f-hrdvp\" (UID: \"614fb8da-43f7-49cc-acb4-dbf2befda850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hrdvp" Oct 31 05:20:48.072914 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:48.072684 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/614fb8da-43f7-49cc-acb4-dbf2befda850-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-hrdvp\" (UID: \"614fb8da-43f7-49cc-acb4-dbf2befda850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hrdvp" Oct 31 05:20:48.072914 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:48.072826 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/614fb8da-43f7-49cc-acb4-dbf2befda850-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-hrdvp\" (UID: \"614fb8da-43f7-49cc-acb4-dbf2befda850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hrdvp" Oct 31 05:20:48.173763 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:48.173695 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptd92\" (UniqueName: \"kubernetes.io/projected/614fb8da-43f7-49cc-acb4-dbf2befda850-kube-api-access-ptd92\") pod \"maas-api-revoke-sub-6b19338f-hrdvp\" (UID: \"614fb8da-43f7-49cc-acb4-dbf2befda850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hrdvp" Oct 31 05:20:48.173763 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:48.173772 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/614fb8da-43f7-49cc-acb4-dbf2befda850-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-hrdvp\" (UID: \"614fb8da-43f7-49cc-acb4-dbf2befda850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hrdvp" Oct 31 05:20:48.174083 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:48.173835 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/614fb8da-43f7-49cc-acb4-dbf2befda850-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-hrdvp\" (UID: \"614fb8da-43f7-49cc-acb4-dbf2befda850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hrdvp" Oct 31 05:20:48.174705 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:48.174676 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/614fb8da-43f7-49cc-acb4-dbf2befda850-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-hrdvp\" (UID: \"614fb8da-43f7-49cc-acb4-dbf2befda850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hrdvp" Oct 31 05:20:48.176470 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:48.176447 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/614fb8da-43f7-49cc-acb4-dbf2befda850-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-hrdvp\" (UID: \"614fb8da-43f7-49cc-acb4-dbf2befda850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hrdvp" Oct 31 05:20:48.197822 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:48.197783 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptd92\" (UniqueName: \"kubernetes.io/projected/614fb8da-43f7-49cc-acb4-dbf2befda850-kube-api-access-ptd92\") pod \"maas-api-revoke-sub-6b19338f-hrdvp\" (UID: \"614fb8da-43f7-49cc-acb4-dbf2befda850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hrdvp" Oct 31 05:20:48.291582 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:48.291463 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hrdvp" Oct 31 05:20:48.622103 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:48.601169 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hrdvp"] Oct 31 05:20:49.408233 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:49.408146 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39c4d335-5acc-4668-b4c3-57f6ff58a75b" path="/var/lib/kubelet/pods/39c4d335-5acc-4668-b4c3-57f6ff58a75b/volumes" Oct 31 05:20:49.439464 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:49.439424 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hrdvp" event={"ID":"614fb8da-43f7-49cc-acb4-dbf2befda850","Type":"ContainerStarted","Data":"f76bd78545ae2191475bc86b431b2f09280282e077076623c60fa5fe8443df6b"} Oct 31 05:20:49.439704 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:49.439480 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hrdvp" event={"ID":"614fb8da-43f7-49cc-acb4-dbf2befda850","Type":"ContainerStarted","Data":"2253989af859f9b8def8e96123aed3aad211bad313fc8422cb6da6a18008937c"} Oct 31 05:20:49.482609 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:49.482546 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hrdvp" podStartSLOduration=2.482526977 podStartE2EDuration="2.482526977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:20:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:20:49.480182255 +0000 UTC m=+4604.236520296" watchObservedRunningTime="2026-10-31 05:20:49.482526977 +0000 UTC m=+4604.238865078" Oct 31 05:20:50.447866 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:50.447824 5019 generic.go:356] "Generic (PLEG): container finished" podID="614fb8da-43f7-49cc-acb4-dbf2befda850" containerID="f76bd78545ae2191475bc86b431b2f09280282e077076623c60fa5fe8443df6b" exitCode=0 Oct 31 05:20:50.448556 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:50.447902 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hrdvp" event={"ID":"614fb8da-43f7-49cc-acb4-dbf2befda850","Type":"ContainerDied","Data":"f76bd78545ae2191475bc86b431b2f09280282e077076623c60fa5fe8443df6b"} Oct 31 05:20:51.621780 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:51.621748 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hrdvp" Oct 31 05:20:51.712023 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:51.711930 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/614fb8da-43f7-49cc-acb4-dbf2befda850-kube-api-access-ptd92\" (UniqueName: \"kubernetes.io/projected/614fb8da-43f7-49cc-acb4-dbf2befda850-kube-api-access-ptd92\") pod \"614fb8da-43f7-49cc-acb4-dbf2befda850\" (UID: \"614fb8da-43f7-49cc-acb4-dbf2befda850\") " Oct 31 05:20:51.712023 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:51.711983 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/614fb8da-43f7-49cc-acb4-dbf2befda850-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/614fb8da-43f7-49cc-acb4-dbf2befda850-maas-api-openshift-ca\") pod \"614fb8da-43f7-49cc-acb4-dbf2befda850\" (UID: \"614fb8da-43f7-49cc-acb4-dbf2befda850\") " Oct 31 05:20:51.712023 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:51.712015 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/614fb8da-43f7-49cc-acb4-dbf2befda850-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/614fb8da-43f7-49cc-acb4-dbf2befda850-maas-api-xks-ca\") pod \"614fb8da-43f7-49cc-acb4-dbf2befda850\" (UID: \"614fb8da-43f7-49cc-acb4-dbf2befda850\") " Oct 31 05:20:51.712497 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:51.712471 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/614fb8da-43f7-49cc-acb4-dbf2befda850-maas-api-openshift-ca" pod "614fb8da-43f7-49cc-acb4-dbf2befda850" (UID: "614fb8da-43f7-49cc-acb4-dbf2befda850"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:20:51.714176 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:51.714147 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/614fb8da-43f7-49cc-acb4-dbf2befda850-maas-api-xks-ca" pod "614fb8da-43f7-49cc-acb4-dbf2befda850" (UID: "614fb8da-43f7-49cc-acb4-dbf2befda850"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:20:51.714292 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:51.714213 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/614fb8da-43f7-49cc-acb4-dbf2befda850-kube-api-access-ptd92" pod "614fb8da-43f7-49cc-acb4-dbf2befda850" (UID: "614fb8da-43f7-49cc-acb4-dbf2befda850"). InnerVolumeSpecName "kube-api-access-ptd92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:20:51.813698 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:51.813655 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/614fb8da-43f7-49cc-acb4-dbf2befda850-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:20:51.813698 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:51.813696 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptd92\" (UniqueName: \"kubernetes.io/projected/614fb8da-43f7-49cc-acb4-dbf2befda850-kube-api-access-ptd92\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:20:51.813934 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:51.813708 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/614fb8da-43f7-49cc-acb4-dbf2befda850-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:20:52.391046 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:20:52.391004 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:20:52.464523 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:52.464485 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hrdvp" event={"ID":"614fb8da-43f7-49cc-acb4-dbf2befda850","Type":"ContainerDied","Data":"2253989af859f9b8def8e96123aed3aad211bad313fc8422cb6da6a18008937c"} Oct 31 05:20:52.464523 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:52.464514 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hrdvp" Oct 31 05:20:52.464757 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:52.464538 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2253989af859f9b8def8e96123aed3aad211bad313fc8422cb6da6a18008937c" Oct 31 05:20:57.028996 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:57.028947 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbf88caf-87lbv"] Oct 31 05:20:57.040971 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:57.040934 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbf88caf-87lbv"] Oct 31 05:20:57.406724 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:57.406635 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2c340cd-5e95-444e-af15-f94090647205" path="/var/lib/kubelet/pods/a2c340cd-5e95-444e-af15-f94090647205/volumes" Oct 31 05:20:58.029438 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:58.029396 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38792383-2l5ht"] Oct 31 05:20:58.041534 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:58.041493 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38792383-2l5ht"] Oct 31 05:20:59.406958 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:20:59.406916 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42db3486-5bc2-40fc-b5cd-b32f718255b3" path="/var/lib/kubelet/pods/42db3486-5bc2-40fc-b5cd-b32f718255b3/volumes" Oct 31 05:21:01.416356 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:01.415595 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-rgj85 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.218:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 05:21:01.416356 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:01.415695 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rgj85" podUID="8430b919-a9dd-4c7e-9266-ef6872cc0b9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.218:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 05:21:01.448360 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:01.445458 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 05:21:01.448360 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:01.445584 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r" podUID="68e465ba-2b5d-4e3c-8b63-7bdb9b695b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 05:21:01.542585 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:01.542535 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-shld4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 05:21:01.542971 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:01.542933 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-shld4" podUID="a69fd138-9906-4127-ac26-38a21a5b7590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 05:21:01.644508 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:01.644462 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.221:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 05:21:01.644837 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:01.644799 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5" podUID="6d68d19b-5392-4308-9e2c-4b5e7593d17b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.221:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 05:21:01.649434 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:01.649389 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-119c987e-kserve-66959d954d-kln77"] Oct 31 05:21:01.663369 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:01.650325 5019 cpu_manager.go:414] "containerMap: removing container" podUID="614fb8da-43f7-49cc-acb4-dbf2befda850" containerName="revoke-keys" Oct 31 05:21:01.668428 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:01.663692 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="614fb8da-43f7-49cc-acb4-dbf2befda850" containerName="revoke-keys" Oct 31 05:21:01.678366 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:01.668914 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="614fb8da-43f7-49cc-acb4-dbf2befda850" containerName="revoke-keys" Oct 31 05:21:01.815363 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:01.808223 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-119c987e-kserve-66959d954d-kln77"] Oct 31 05:21:01.839173 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:01.812078 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-119c987e-kserve-66959d954d-kln77" Oct 31 05:21:01.857389 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:01.855399 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-119c987e-kserve-self-signed-certs\"" Oct 31 05:21:01.916357 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:01.915707 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d696792-e30e-4020-b4b5-fc6f7af3c21e-tmp-dir\") pod \"e2e-prio-pool-119c987e-kserve-66959d954d-kln77\" (UID: \"6d696792-e30e-4020-b4b5-fc6f7af3c21e\") " pod="llm/e2e-prio-pool-119c987e-kserve-66959d954d-kln77" Oct 31 05:21:01.916357 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:01.916003 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d696792-e30e-4020-b4b5-fc6f7af3c21e-home\") pod \"e2e-prio-pool-119c987e-kserve-66959d954d-kln77\" (UID: \"6d696792-e30e-4020-b4b5-fc6f7af3c21e\") " pod="llm/e2e-prio-pool-119c987e-kserve-66959d954d-kln77" Oct 31 05:21:01.916357 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:01.916044 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d696792-e30e-4020-b4b5-fc6f7af3c21e-dshm\") pod \"e2e-prio-pool-119c987e-kserve-66959d954d-kln77\" (UID: \"6d696792-e30e-4020-b4b5-fc6f7af3c21e\") " pod="llm/e2e-prio-pool-119c987e-kserve-66959d954d-kln77" Oct 31 05:21:01.916357 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:01.916071 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d696792-e30e-4020-b4b5-fc6f7af3c21e-model-cache\") pod \"e2e-prio-pool-119c987e-kserve-66959d954d-kln77\" (UID: \"6d696792-e30e-4020-b4b5-fc6f7af3c21e\") " pod="llm/e2e-prio-pool-119c987e-kserve-66959d954d-kln77" Oct 31 05:21:01.916357 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:01.916136 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t2nf\" (UniqueName: \"kubernetes.io/projected/6d696792-e30e-4020-b4b5-fc6f7af3c21e-kube-api-access-8t2nf\") pod \"e2e-prio-pool-119c987e-kserve-66959d954d-kln77\" (UID: \"6d696792-e30e-4020-b4b5-fc6f7af3c21e\") " pod="llm/e2e-prio-pool-119c987e-kserve-66959d954d-kln77" Oct 31 05:21:01.916357 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:01.916195 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d696792-e30e-4020-b4b5-fc6f7af3c21e-tls-certs\") pod \"e2e-prio-pool-119c987e-kserve-66959d954d-kln77\" (UID: \"6d696792-e30e-4020-b4b5-fc6f7af3c21e\") " pod="llm/e2e-prio-pool-119c987e-kserve-66959d954d-kln77" Oct 31 05:21:02.017358 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:02.017289 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d696792-e30e-4020-b4b5-fc6f7af3c21e-tmp-dir\") pod \"e2e-prio-pool-119c987e-kserve-66959d954d-kln77\" (UID: \"6d696792-e30e-4020-b4b5-fc6f7af3c21e\") " pod="llm/e2e-prio-pool-119c987e-kserve-66959d954d-kln77" Oct 31 05:21:02.017597 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:02.017424 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d696792-e30e-4020-b4b5-fc6f7af3c21e-home\") pod \"e2e-prio-pool-119c987e-kserve-66959d954d-kln77\" (UID: \"6d696792-e30e-4020-b4b5-fc6f7af3c21e\") " pod="llm/e2e-prio-pool-119c987e-kserve-66959d954d-kln77" Oct 31 05:21:02.017597 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:02.017446 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d696792-e30e-4020-b4b5-fc6f7af3c21e-dshm\") pod \"e2e-prio-pool-119c987e-kserve-66959d954d-kln77\" (UID: \"6d696792-e30e-4020-b4b5-fc6f7af3c21e\") " pod="llm/e2e-prio-pool-119c987e-kserve-66959d954d-kln77" Oct 31 05:21:02.017597 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:02.017461 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d696792-e30e-4020-b4b5-fc6f7af3c21e-model-cache\") pod \"e2e-prio-pool-119c987e-kserve-66959d954d-kln77\" (UID: \"6d696792-e30e-4020-b4b5-fc6f7af3c21e\") " pod="llm/e2e-prio-pool-119c987e-kserve-66959d954d-kln77" Oct 31 05:21:02.017597 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:02.017502 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t2nf\" (UniqueName: \"kubernetes.io/projected/6d696792-e30e-4020-b4b5-fc6f7af3c21e-kube-api-access-8t2nf\") pod \"e2e-prio-pool-119c987e-kserve-66959d954d-kln77\" (UID: \"6d696792-e30e-4020-b4b5-fc6f7af3c21e\") " pod="llm/e2e-prio-pool-119c987e-kserve-66959d954d-kln77" Oct 31 05:21:02.017597 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:02.017549 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d696792-e30e-4020-b4b5-fc6f7af3c21e-tls-certs\") pod \"e2e-prio-pool-119c987e-kserve-66959d954d-kln77\" (UID: \"6d696792-e30e-4020-b4b5-fc6f7af3c21e\") " pod="llm/e2e-prio-pool-119c987e-kserve-66959d954d-kln77" Oct 31 05:21:02.017920 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:02.017813 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d696792-e30e-4020-b4b5-fc6f7af3c21e-tmp-dir\") pod \"e2e-prio-pool-119c987e-kserve-66959d954d-kln77\" (UID: \"6d696792-e30e-4020-b4b5-fc6f7af3c21e\") " pod="llm/e2e-prio-pool-119c987e-kserve-66959d954d-kln77" Oct 31 05:21:02.018538 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:02.018479 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d696792-e30e-4020-b4b5-fc6f7af3c21e-model-cache\") pod \"e2e-prio-pool-119c987e-kserve-66959d954d-kln77\" (UID: \"6d696792-e30e-4020-b4b5-fc6f7af3c21e\") " pod="llm/e2e-prio-pool-119c987e-kserve-66959d954d-kln77" Oct 31 05:21:02.018538 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:02.018488 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d696792-e30e-4020-b4b5-fc6f7af3c21e-home\") pod \"e2e-prio-pool-119c987e-kserve-66959d954d-kln77\" (UID: \"6d696792-e30e-4020-b4b5-fc6f7af3c21e\") " pod="llm/e2e-prio-pool-119c987e-kserve-66959d954d-kln77" Oct 31 05:21:02.020611 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:02.020578 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d696792-e30e-4020-b4b5-fc6f7af3c21e-tls-certs\") pod \"e2e-prio-pool-119c987e-kserve-66959d954d-kln77\" (UID: \"6d696792-e30e-4020-b4b5-fc6f7af3c21e\") " pod="llm/e2e-prio-pool-119c987e-kserve-66959d954d-kln77" Oct 31 05:21:02.020967 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:02.020941 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d696792-e30e-4020-b4b5-fc6f7af3c21e-dshm\") pod \"e2e-prio-pool-119c987e-kserve-66959d954d-kln77\" (UID: \"6d696792-e30e-4020-b4b5-fc6f7af3c21e\") " pod="llm/e2e-prio-pool-119c987e-kserve-66959d954d-kln77" Oct 31 05:21:02.041766 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:02.041724 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t2nf\" (UniqueName: \"kubernetes.io/projected/6d696792-e30e-4020-b4b5-fc6f7af3c21e-kube-api-access-8t2nf\") pod \"e2e-prio-pool-119c987e-kserve-66959d954d-kln77\" (UID: \"6d696792-e30e-4020-b4b5-fc6f7af3c21e\") " pod="llm/e2e-prio-pool-119c987e-kserve-66959d954d-kln77" Oct 31 05:21:02.171864 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:02.171820 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-119c987e-kserve-66959d954d-kln77" Oct 31 05:21:02.507419 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:02.507387 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-119c987e-kserve-66959d954d-kln77"] Oct 31 05:21:02.508799 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:21:02.508770 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d696792_e30e_4020_b4b5_fc6f7af3c21e.slice/crio-e3fe57ee2a5ff2a75c9dcde84cea0c94a5ce78cf35f546454d67fe551e2c77f4 WatchSource:0}: Error finding container e3fe57ee2a5ff2a75c9dcde84cea0c94a5ce78cf35f546454d67fe551e2c77f4: Status 404 returned error can't find the container with id e3fe57ee2a5ff2a75c9dcde84cea0c94a5ce78cf35f546454d67fe551e2c77f4 Oct 31 05:21:02.574603 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:02.574568 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-119c987e-kserve-66959d954d-kln77" event={"ID":"6d696792-e30e-4020-b4b5-fc6f7af3c21e","Type":"ContainerStarted","Data":"e3fe57ee2a5ff2a75c9dcde84cea0c94a5ce78cf35f546454d67fe551e2c77f4"} Oct 31 05:21:03.584542 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:03.584491 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-119c987e-kserve-66959d954d-kln77" event={"ID":"6d696792-e30e-4020-b4b5-fc6f7af3c21e","Type":"ContainerStarted","Data":"25b485f6181e097b372ef8a89c819cdb8d536c898424e59e0cfe56904270b339"} Oct 31 05:21:03.631808 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:03.631731 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-119c987e-kserve-66959d954d-kln77" podStartSLOduration=2.631708691 podStartE2EDuration="2.631708691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:21:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:21:03.626792179 +0000 UTC m=+4618.383130220" watchObservedRunningTime="2026-10-31 05:21:03.631708691 +0000 UTC m=+4618.388046732" Oct 31 05:21:04.172501 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:04.172450 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-119c987e-kserve-66959d954d-kln77" Oct 31 05:21:12.172586 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:12.172531 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-prio-pool-119c987e-kserve-66959d954d-kln77" Oct 31 05:21:12.191239 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:12.191203 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-prio-pool-119c987e-kserve-66959d954d-kln77" Oct 31 05:21:12.688836 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:12.688806 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-119c987e-kserve-66959d954d-kln77" Oct 31 05:21:22.390821 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:21:22.390769 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:21:46.416885 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:46.415495 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-rgj85 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.218:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 05:21:46.418125 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:46.417652 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rgj85" podUID="8430b919-a9dd-4c7e-9266-ef6872cc0b9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.218:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 05:21:46.438521 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:46.438482 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 05:21:46.438823 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:46.438794 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r" podUID="68e465ba-2b5d-4e3c-8b63-7bdb9b695b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 05:21:46.533359 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:46.531460 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-shld4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 05:21:46.533359 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:46.531542 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-shld4" podUID="a69fd138-9906-4127-ac26-38a21a5b7590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 05:21:46.624042 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:46.623529 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.221:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 05:21:46.624042 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:46.623604 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5" podUID="6d68d19b-5392-4308-9e2c-4b5e7593d17b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.221:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 05:21:52.168802 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:52.168759 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ad68e6c-w4w9h"] Oct 31 05:21:52.173009 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:52.172980 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad68e6c-w4w9h" Oct 31 05:21:52.194554 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:52.194508 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ad68e6c-w4w9h"] Oct 31 05:21:52.202694 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:52.202655 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:21:52.202694 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:52.202667 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:21:52.202949 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:52.202720 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:21:52.202949 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:52.202800 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfhzn\"" Oct 31 05:21:52.218395 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:52.218342 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecfe5316-7a65-452e-b75e-a813ad0a3276-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ad68e6c-w4w9h\" (UID: \"ecfe5316-7a65-452e-b75e-a813ad0a3276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad68e6c-w4w9h" Oct 31 05:21:52.218648 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:52.218528 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bslp\" (UniqueName: \"kubernetes.io/projected/ecfe5316-7a65-452e-b75e-a813ad0a3276-kube-api-access-2bslp\") pod \"maas-api-revoke-sub-4ad68e6c-w4w9h\" (UID: \"ecfe5316-7a65-452e-b75e-a813ad0a3276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad68e6c-w4w9h" Oct 31 05:21:52.218648 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:52.218575 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecfe5316-7a65-452e-b75e-a813ad0a3276-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ad68e6c-w4w9h\" (UID: \"ecfe5316-7a65-452e-b75e-a813ad0a3276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad68e6c-w4w9h" Oct 31 05:21:52.319287 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:52.319219 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecfe5316-7a65-452e-b75e-a813ad0a3276-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ad68e6c-w4w9h\" (UID: \"ecfe5316-7a65-452e-b75e-a813ad0a3276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad68e6c-w4w9h" Oct 31 05:21:52.319577 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:52.319415 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bslp\" (UniqueName: \"kubernetes.io/projected/ecfe5316-7a65-452e-b75e-a813ad0a3276-kube-api-access-2bslp\") pod \"maas-api-revoke-sub-4ad68e6c-w4w9h\" (UID: \"ecfe5316-7a65-452e-b75e-a813ad0a3276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad68e6c-w4w9h" Oct 31 05:21:52.319577 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:52.319452 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecfe5316-7a65-452e-b75e-a813ad0a3276-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ad68e6c-w4w9h\" (UID: \"ecfe5316-7a65-452e-b75e-a813ad0a3276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad68e6c-w4w9h" Oct 31 05:21:52.320189 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:52.320158 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecfe5316-7a65-452e-b75e-a813ad0a3276-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ad68e6c-w4w9h\" (UID: \"ecfe5316-7a65-452e-b75e-a813ad0a3276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad68e6c-w4w9h" Oct 31 05:21:52.321802 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:52.321782 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecfe5316-7a65-452e-b75e-a813ad0a3276-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ad68e6c-w4w9h\" (UID: \"ecfe5316-7a65-452e-b75e-a813ad0a3276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad68e6c-w4w9h" Oct 31 05:21:52.342969 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:52.342933 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bslp\" (UniqueName: \"kubernetes.io/projected/ecfe5316-7a65-452e-b75e-a813ad0a3276-kube-api-access-2bslp\") pod \"maas-api-revoke-sub-4ad68e6c-w4w9h\" (UID: \"ecfe5316-7a65-452e-b75e-a813ad0a3276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad68e6c-w4w9h" Oct 31 05:21:52.392257 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:21:52.392213 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:21:52.487545 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:52.487495 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad68e6c-w4w9h" Oct 31 05:21:52.787438 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:52.787398 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ad68e6c-w4w9h"] Oct 31 05:21:52.789347 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:21:52.789307 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecfe5316_7a65_452e_b75e_a813ad0a3276.slice/crio-f46e24f2792d736abd04b2903558d8d976c0b98c40cce482e938f30cf518ca8a WatchSource:0}: Error finding container f46e24f2792d736abd04b2903558d8d976c0b98c40cce482e938f30cf518ca8a: Status 404 returned error can't find the container with id f46e24f2792d736abd04b2903558d8d976c0b98c40cce482e938f30cf518ca8a Oct 31 05:21:53.003311 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:53.003270 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad68e6c-w4w9h" event={"ID":"ecfe5316-7a65-452e-b75e-a813ad0a3276","Type":"ContainerStarted","Data":"f46e24f2792d736abd04b2903558d8d976c0b98c40cce482e938f30cf518ca8a"} Oct 31 05:21:54.011073 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:54.011037 5019 generic.go:356] "Generic (PLEG): container finished" podID="ecfe5316-7a65-452e-b75e-a813ad0a3276" containerID="b2028df90f0b44957a05308047f58b890daef75b8137b54c498963c98cda72f8" exitCode=0 Oct 31 05:21:54.011563 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:54.011134 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad68e6c-w4w9h" event={"ID":"ecfe5316-7a65-452e-b75e-a813ad0a3276","Type":"ContainerDied","Data":"b2028df90f0b44957a05308047f58b890daef75b8137b54c498963c98cda72f8"} Oct 31 05:21:55.167872 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:55.167840 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad68e6c-w4w9h" Oct 31 05:21:55.243662 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:55.243616 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecfe5316-7a65-452e-b75e-a813ad0a3276-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecfe5316-7a65-452e-b75e-a813ad0a3276-maas-api-xks-ca\") pod \"ecfe5316-7a65-452e-b75e-a813ad0a3276\" (UID: \"ecfe5316-7a65-452e-b75e-a813ad0a3276\") " Oct 31 05:21:55.243893 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:55.243711 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecfe5316-7a65-452e-b75e-a813ad0a3276-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecfe5316-7a65-452e-b75e-a813ad0a3276-maas-api-openshift-ca\") pod \"ecfe5316-7a65-452e-b75e-a813ad0a3276\" (UID: \"ecfe5316-7a65-452e-b75e-a813ad0a3276\") " Oct 31 05:21:55.244192 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:55.244162 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecfe5316-7a65-452e-b75e-a813ad0a3276-maas-api-openshift-ca" pod "ecfe5316-7a65-452e-b75e-a813ad0a3276" (UID: "ecfe5316-7a65-452e-b75e-a813ad0a3276"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:21:55.246101 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:55.246063 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecfe5316-7a65-452e-b75e-a813ad0a3276-maas-api-xks-ca" pod "ecfe5316-7a65-452e-b75e-a813ad0a3276" (UID: "ecfe5316-7a65-452e-b75e-a813ad0a3276"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:21:55.344782 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:55.344677 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecfe5316-7a65-452e-b75e-a813ad0a3276-kube-api-access-2bslp\" (UniqueName: \"kubernetes.io/projected/ecfe5316-7a65-452e-b75e-a813ad0a3276-kube-api-access-2bslp\") pod \"ecfe5316-7a65-452e-b75e-a813ad0a3276\" (UID: \"ecfe5316-7a65-452e-b75e-a813ad0a3276\") " Oct 31 05:21:55.344988 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:55.344973 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecfe5316-7a65-452e-b75e-a813ad0a3276-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:21:55.345041 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:55.344993 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecfe5316-7a65-452e-b75e-a813ad0a3276-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:21:55.346999 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:55.346965 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecfe5316-7a65-452e-b75e-a813ad0a3276-kube-api-access-2bslp" pod "ecfe5316-7a65-452e-b75e-a813ad0a3276" (UID: "ecfe5316-7a65-452e-b75e-a813ad0a3276"). InnerVolumeSpecName "kube-api-access-2bslp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:21:55.446454 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:55.446324 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bslp\" (UniqueName: \"kubernetes.io/projected/ecfe5316-7a65-452e-b75e-a813ad0a3276-kube-api-access-2bslp\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:21:56.024652 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:56.024617 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad68e6c-w4w9h" Oct 31 05:21:56.024880 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:56.024661 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad68e6c-w4w9h" event={"ID":"ecfe5316-7a65-452e-b75e-a813ad0a3276","Type":"ContainerDied","Data":"f46e24f2792d736abd04b2903558d8d976c0b98c40cce482e938f30cf518ca8a"} Oct 31 05:21:56.024880 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:21:56.024740 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f46e24f2792d736abd04b2903558d8d976c0b98c40cce482e938f30cf518ca8a" Oct 31 05:22:05.030567 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:05.030518 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-119c987e-kserve-66959d954d-kln77"] Oct 31 05:22:10.134173 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:10.134133 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be9d2b4d-s4wb4"] Oct 31 05:22:10.134642 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:10.134626 5019 cpu_manager.go:414] "containerMap: removing container" podUID="ecfe5316-7a65-452e-b75e-a813ad0a3276" containerName="revoke-keys" Oct 31 05:22:10.134697 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:10.134642 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecfe5316-7a65-452e-b75e-a813ad0a3276" containerName="revoke-keys" Oct 31 05:22:10.134747 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:10.134738 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecfe5316-7a65-452e-b75e-a813ad0a3276" containerName="revoke-keys" Oct 31 05:22:10.138408 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:10.138379 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9d2b4d-s4wb4" Oct 31 05:22:10.157657 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:10.157617 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be9d2b4d-s4wb4"] Oct 31 05:22:10.165322 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:10.165282 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:22:10.165664 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:10.165625 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:22:10.165803 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:10.165768 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfhzn\"" Oct 31 05:22:10.165867 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:10.165805 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:22:10.243768 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:10.243729 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c81d39f-ba54-4401-80ed-bdd66e448fb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be9d2b4d-s4wb4\" (UID: \"5c81d39f-ba54-4401-80ed-bdd66e448fb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9d2b4d-s4wb4" Oct 31 05:22:10.243982 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:10.243852 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tx6j\" (UniqueName: \"kubernetes.io/projected/5c81d39f-ba54-4401-80ed-bdd66e448fb2-kube-api-access-6tx6j\") pod \"maas-api-revoke-sub-be9d2b4d-s4wb4\" (UID: \"5c81d39f-ba54-4401-80ed-bdd66e448fb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9d2b4d-s4wb4" Oct 31 05:22:10.243982 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:10.243926 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c81d39f-ba54-4401-80ed-bdd66e448fb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be9d2b4d-s4wb4\" (UID: \"5c81d39f-ba54-4401-80ed-bdd66e448fb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9d2b4d-s4wb4" Oct 31 05:22:10.345132 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:10.345092 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c81d39f-ba54-4401-80ed-bdd66e448fb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be9d2b4d-s4wb4\" (UID: \"5c81d39f-ba54-4401-80ed-bdd66e448fb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9d2b4d-s4wb4" Oct 31 05:22:10.345350 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:10.345155 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c81d39f-ba54-4401-80ed-bdd66e448fb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be9d2b4d-s4wb4\" (UID: \"5c81d39f-ba54-4401-80ed-bdd66e448fb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9d2b4d-s4wb4" Oct 31 05:22:10.345350 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:10.345203 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tx6j\" (UniqueName: \"kubernetes.io/projected/5c81d39f-ba54-4401-80ed-bdd66e448fb2-kube-api-access-6tx6j\") pod \"maas-api-revoke-sub-be9d2b4d-s4wb4\" (UID: \"5c81d39f-ba54-4401-80ed-bdd66e448fb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9d2b4d-s4wb4" Oct 31 05:22:10.345900 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:10.345872 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c81d39f-ba54-4401-80ed-bdd66e448fb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be9d2b4d-s4wb4\" (UID: \"5c81d39f-ba54-4401-80ed-bdd66e448fb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9d2b4d-s4wb4" Oct 31 05:22:10.347548 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:10.347528 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c81d39f-ba54-4401-80ed-bdd66e448fb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be9d2b4d-s4wb4\" (UID: \"5c81d39f-ba54-4401-80ed-bdd66e448fb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9d2b4d-s4wb4" Oct 31 05:22:10.369198 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:10.369168 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tx6j\" (UniqueName: \"kubernetes.io/projected/5c81d39f-ba54-4401-80ed-bdd66e448fb2-kube-api-access-6tx6j\") pod \"maas-api-revoke-sub-be9d2b4d-s4wb4\" (UID: \"5c81d39f-ba54-4401-80ed-bdd66e448fb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9d2b4d-s4wb4" Oct 31 05:22:10.454796 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:10.454710 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9d2b4d-s4wb4" Oct 31 05:22:10.755320 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:10.755280 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be9d2b4d-s4wb4"] Oct 31 05:22:10.757009 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:22:10.756971 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c81d39f_ba54_4401_80ed_bdd66e448fb2.slice/crio-a20d9de17cea99a5e7e9861526a4ad9757fa852a387ca322b222036164233333 WatchSource:0}: Error finding container a20d9de17cea99a5e7e9861526a4ad9757fa852a387ca322b222036164233333: Status 404 returned error can't find the container with id a20d9de17cea99a5e7e9861526a4ad9757fa852a387ca322b222036164233333 Oct 31 05:22:11.158990 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:11.158945 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9d2b4d-s4wb4" event={"ID":"5c81d39f-ba54-4401-80ed-bdd66e448fb2","Type":"ContainerStarted","Data":"5ef23476d9a0c1a9f4835d7532412deb0a5956a16ef0c7d8be75084b28e8b18f"} Oct 31 05:22:11.159462 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:11.159005 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9d2b4d-s4wb4" event={"ID":"5c81d39f-ba54-4401-80ed-bdd66e448fb2","Type":"ContainerStarted","Data":"a20d9de17cea99a5e7e9861526a4ad9757fa852a387ca322b222036164233333"} Oct 31 05:22:11.201326 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:11.201251 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9d2b4d-s4wb4" podStartSLOduration=1.201227886 podStartE2EDuration="1.201227886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:22:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:22:11.198194655 +0000 UTC m=+4685.954532696" watchObservedRunningTime="2026-10-31 05:22:11.201227886 +0000 UTC m=+4685.957565927" Oct 31 05:22:12.172162 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:12.172119 5019 generic.go:356] "Generic (PLEG): container finished" podID="5c81d39f-ba54-4401-80ed-bdd66e448fb2" containerID="5ef23476d9a0c1a9f4835d7532412deb0a5956a16ef0c7d8be75084b28e8b18f" exitCode=0 Oct 31 05:22:12.172962 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:12.172931 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9d2b4d-s4wb4" event={"ID":"5c81d39f-ba54-4401-80ed-bdd66e448fb2","Type":"ContainerDied","Data":"5ef23476d9a0c1a9f4835d7532412deb0a5956a16ef0c7d8be75084b28e8b18f"} Oct 31 05:22:14.194904 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:14.194863 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9d2b4d-s4wb4" Oct 31 05:22:14.215694 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:14.215648 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c81d39f-ba54-4401-80ed-bdd66e448fb2-kube-api-access-6tx6j\" (UniqueName: \"kubernetes.io/projected/5c81d39f-ba54-4401-80ed-bdd66e448fb2-kube-api-access-6tx6j\") pod \"5c81d39f-ba54-4401-80ed-bdd66e448fb2\" (UID: \"5c81d39f-ba54-4401-80ed-bdd66e448fb2\") " Oct 31 05:22:14.216132 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:14.216105 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c81d39f-ba54-4401-80ed-bdd66e448fb2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c81d39f-ba54-4401-80ed-bdd66e448fb2-maas-api-openshift-ca\") pod \"5c81d39f-ba54-4401-80ed-bdd66e448fb2\" (UID: \"5c81d39f-ba54-4401-80ed-bdd66e448fb2\") " Oct 31 05:22:14.216292 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:14.216272 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c81d39f-ba54-4401-80ed-bdd66e448fb2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c81d39f-ba54-4401-80ed-bdd66e448fb2-maas-api-xks-ca\") pod \"5c81d39f-ba54-4401-80ed-bdd66e448fb2\" (UID: \"5c81d39f-ba54-4401-80ed-bdd66e448fb2\") " Oct 31 05:22:14.219825 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:14.219770 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c81d39f-ba54-4401-80ed-bdd66e448fb2-maas-api-openshift-ca" pod "5c81d39f-ba54-4401-80ed-bdd66e448fb2" (UID: "5c81d39f-ba54-4401-80ed-bdd66e448fb2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:22:14.248595 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:14.248551 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c81d39f-ba54-4401-80ed-bdd66e448fb2-maas-api-xks-ca" pod "5c81d39f-ba54-4401-80ed-bdd66e448fb2" (UID: "5c81d39f-ba54-4401-80ed-bdd66e448fb2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:22:14.249916 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:14.249881 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c81d39f-ba54-4401-80ed-bdd66e448fb2-kube-api-access-6tx6j" pod "5c81d39f-ba54-4401-80ed-bdd66e448fb2" (UID: "5c81d39f-ba54-4401-80ed-bdd66e448fb2"). InnerVolumeSpecName "kube-api-access-6tx6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:22:14.283380 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:14.279955 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9d2b4d-s4wb4" event={"ID":"5c81d39f-ba54-4401-80ed-bdd66e448fb2","Type":"ContainerDied","Data":"a20d9de17cea99a5e7e9861526a4ad9757fa852a387ca322b222036164233333"} Oct 31 05:22:14.283380 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:14.280027 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a20d9de17cea99a5e7e9861526a4ad9757fa852a387ca322b222036164233333" Oct 31 05:22:14.283380 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:14.280100 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9d2b4d-s4wb4" Oct 31 05:22:14.305981 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:14.305879 5019 scope.go:122] "RemoveContainer" containerID="1a284afd572e939558dd715390136c3794c6dcbb2dae7985d1506a1d80cc3f7f" Oct 31 05:22:14.335366 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:14.326778 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c81d39f-ba54-4401-80ed-bdd66e448fb2-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:22:14.347623 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:14.345600 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c81d39f-ba54-4401-80ed-bdd66e448fb2-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:22:14.347623 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:14.345733 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tx6j\" (UniqueName: \"kubernetes.io/projected/5c81d39f-ba54-4401-80ed-bdd66e448fb2-kube-api-access-6tx6j\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:22:14.356733 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:14.355532 5019 scope.go:122] "RemoveContainer" containerID="8e61ec6e8ac2401b5070a0e8377973597c5d94fb6fe65b6cf42f43672d38f9b1" Oct 31 05:22:14.376242 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:14.376094 5019 scope.go:122] "RemoveContainer" containerID="c4cbf0a155f6154253b7d3284455ba59a265dee7246a5cd24172b16cda7e65aa" Oct 31 05:22:14.408760 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:14.408279 5019 scope.go:122] "RemoveContainer" containerID="119c94048c9fb31654a855ad873133797c758d31f12e9e49a58aaa7541aa395e" Oct 31 05:22:14.427181 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:14.426736 5019 scope.go:122] "RemoveContainer" containerID="33eff0b96eff6feb46049ed49fd0c5ebba0877ac7da702a223b46ff850c354e5" Oct 31 05:22:14.444221 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:14.443753 5019 scope.go:122] "RemoveContainer" containerID="b3610314f13849689a9369afe3201510524dbc3fe10d2b1c392ec16b9258cd94" Oct 31 05:22:14.462006 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:14.461593 5019 scope.go:122] "RemoveContainer" containerID="a4a607ab8661d7a2942850dd9b7b3cdd3d42ef1d1cf311f9adafec3e776e72b9" Oct 31 05:22:14.478885 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:14.478845 5019 scope.go:122] "RemoveContainer" containerID="d583c38692505fe99d51de20ae98ab2db7ba1136e4637f8805d36e3d7e842707" Oct 31 05:22:20.030786 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:20.030746 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-828488ca-5tbtz"] Oct 31 05:22:20.042210 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:20.042178 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-828488ca-5tbtz"] Oct 31 05:22:20.053524 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:20.053467 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-119c987e-kserve-66959d954d-kln77" podUID="6d696792-e30e-4020-b4b5-fc6f7af3c21e" containerName="main" containerID="cri-o://25b485f6181e097b372ef8a89c819cdb8d536c898424e59e0cfe56904270b339" gracePeriod=45 Oct 31 05:22:20.319717 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:20.319680 5019 generic.go:356] "Generic (PLEG): container finished" podID="6d696792-e30e-4020-b4b5-fc6f7af3c21e" containerID="25b485f6181e097b372ef8a89c819cdb8d536c898424e59e0cfe56904270b339" exitCode=0 Oct 31 05:22:20.319897 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:20.319757 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-119c987e-kserve-66959d954d-kln77" event={"ID":"6d696792-e30e-4020-b4b5-fc6f7af3c21e","Type":"ContainerDied","Data":"25b485f6181e097b372ef8a89c819cdb8d536c898424e59e0cfe56904270b339"} Oct 31 05:22:20.319897 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:20.319811 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-119c987e-kserve-66959d954d-kln77" event={"ID":"6d696792-e30e-4020-b4b5-fc6f7af3c21e","Type":"ContainerDied","Data":"e3fe57ee2a5ff2a75c9dcde84cea0c94a5ce78cf35f546454d67fe551e2c77f4"} Oct 31 05:22:20.319897 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:20.319830 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3fe57ee2a5ff2a75c9dcde84cea0c94a5ce78cf35f546454d67fe551e2c77f4" Oct 31 05:22:20.324610 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:20.324590 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-119c987e-kserve-66959d954d-kln77" Oct 31 05:22:20.411194 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:20.411149 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d696792-e30e-4020-b4b5-fc6f7af3c21e-tls-certs\" (UniqueName: \"kubernetes.io/secret/6d696792-e30e-4020-b4b5-fc6f7af3c21e-tls-certs\") pod \"6d696792-e30e-4020-b4b5-fc6f7af3c21e\" (UID: \"6d696792-e30e-4020-b4b5-fc6f7af3c21e\") " Oct 31 05:22:20.411453 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:20.411222 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d696792-e30e-4020-b4b5-fc6f7af3c21e-kube-api-access-8t2nf\" (UniqueName: \"kubernetes.io/projected/6d696792-e30e-4020-b4b5-fc6f7af3c21e-kube-api-access-8t2nf\") pod \"6d696792-e30e-4020-b4b5-fc6f7af3c21e\" (UID: \"6d696792-e30e-4020-b4b5-fc6f7af3c21e\") " Oct 31 05:22:20.411453 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:20.411259 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d696792-e30e-4020-b4b5-fc6f7af3c21e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d696792-e30e-4020-b4b5-fc6f7af3c21e-tmp-dir\") pod \"6d696792-e30e-4020-b4b5-fc6f7af3c21e\" (UID: \"6d696792-e30e-4020-b4b5-fc6f7af3c21e\") " Oct 31 05:22:20.411453 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:20.411281 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d696792-e30e-4020-b4b5-fc6f7af3c21e-home\" (UniqueName: \"kubernetes.io/empty-dir/6d696792-e30e-4020-b4b5-fc6f7af3c21e-home\") pod \"6d696792-e30e-4020-b4b5-fc6f7af3c21e\" (UID: \"6d696792-e30e-4020-b4b5-fc6f7af3c21e\") " Oct 31 05:22:20.411453 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:20.411323 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d696792-e30e-4020-b4b5-fc6f7af3c21e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d696792-e30e-4020-b4b5-fc6f7af3c21e-model-cache\") pod \"6d696792-e30e-4020-b4b5-fc6f7af3c21e\" (UID: \"6d696792-e30e-4020-b4b5-fc6f7af3c21e\") " Oct 31 05:22:20.411453 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:20.411388 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d696792-e30e-4020-b4b5-fc6f7af3c21e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d696792-e30e-4020-b4b5-fc6f7af3c21e-dshm\") pod \"6d696792-e30e-4020-b4b5-fc6f7af3c21e\" (UID: \"6d696792-e30e-4020-b4b5-fc6f7af3c21e\") " Oct 31 05:22:20.411791 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:20.411701 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d696792-e30e-4020-b4b5-fc6f7af3c21e-tmp-dir" pod "6d696792-e30e-4020-b4b5-fc6f7af3c21e" (UID: "6d696792-e30e-4020-b4b5-fc6f7af3c21e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:22:20.411999 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:20.411970 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d696792-e30e-4020-b4b5-fc6f7af3c21e-home" pod "6d696792-e30e-4020-b4b5-fc6f7af3c21e" (UID: "6d696792-e30e-4020-b4b5-fc6f7af3c21e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:22:20.412111 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:20.412078 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d696792-e30e-4020-b4b5-fc6f7af3c21e-model-cache" pod "6d696792-e30e-4020-b4b5-fc6f7af3c21e" (UID: "6d696792-e30e-4020-b4b5-fc6f7af3c21e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:22:20.413745 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:20.413718 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d696792-e30e-4020-b4b5-fc6f7af3c21e-tls-certs" pod "6d696792-e30e-4020-b4b5-fc6f7af3c21e" (UID: "6d696792-e30e-4020-b4b5-fc6f7af3c21e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:22:20.413854 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:20.413762 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d696792-e30e-4020-b4b5-fc6f7af3c21e-kube-api-access-8t2nf" pod "6d696792-e30e-4020-b4b5-fc6f7af3c21e" (UID: "6d696792-e30e-4020-b4b5-fc6f7af3c21e"). InnerVolumeSpecName "kube-api-access-8t2nf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:22:20.413854 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:20.413761 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d696792-e30e-4020-b4b5-fc6f7af3c21e-dshm" pod "6d696792-e30e-4020-b4b5-fc6f7af3c21e" (UID: "6d696792-e30e-4020-b4b5-fc6f7af3c21e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:22:20.513194 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:20.513140 5019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d696792-e30e-4020-b4b5-fc6f7af3c21e-tls-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:22:20.513194 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:20.513181 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t2nf\" (UniqueName: \"kubernetes.io/projected/6d696792-e30e-4020-b4b5-fc6f7af3c21e-kube-api-access-8t2nf\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:22:20.513194 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:20.513190 5019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d696792-e30e-4020-b4b5-fc6f7af3c21e-tmp-dir\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:22:20.513194 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:20.513198 5019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d696792-e30e-4020-b4b5-fc6f7af3c21e-home\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:22:20.513194 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:20.513208 5019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d696792-e30e-4020-b4b5-fc6f7af3c21e-model-cache\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:22:20.513579 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:20.513216 5019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d696792-e30e-4020-b4b5-fc6f7af3c21e-dshm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:22:21.325549 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:21.325512 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-119c987e-kserve-66959d954d-kln77" Oct 31 05:22:21.377516 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:21.377477 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-119c987e-kserve-66959d954d-kln77"] Oct 31 05:22:21.389821 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:21.389787 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-119c987e-kserve-66959d954d-kln77"] Oct 31 05:22:21.407260 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:21.407224 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a209b1d-00cf-4969-83d4-6326fc84037f" path="/var/lib/kubelet/pods/4a209b1d-00cf-4969-83d4-6326fc84037f/volumes" Oct 31 05:22:21.407746 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:21.407729 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d696792-e30e-4020-b4b5-fc6f7af3c21e" path="/var/lib/kubelet/pods/6d696792-e30e-4020-b4b5-fc6f7af3c21e/volumes" Oct 31 05:22:22.392067 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:22:22.392028 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:22:22.602903 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:22.602861 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k"] Oct 31 05:22:22.603562 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:22.603542 5019 cpu_manager.go:414] "containerMap: removing container" podUID="5c81d39f-ba54-4401-80ed-bdd66e448fb2" containerName="revoke-keys" Oct 31 05:22:22.603650 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:22.603564 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c81d39f-ba54-4401-80ed-bdd66e448fb2" containerName="revoke-keys" Oct 31 05:22:22.603650 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:22.603583 5019 cpu_manager.go:414] "containerMap: removing container" podUID="6d696792-e30e-4020-b4b5-fc6f7af3c21e" containerName="main" Oct 31 05:22:22.603650 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:22.603591 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d696792-e30e-4020-b4b5-fc6f7af3c21e" containerName="main" Oct 31 05:22:22.603787 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:22.603723 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c81d39f-ba54-4401-80ed-bdd66e448fb2" containerName="revoke-keys" Oct 31 05:22:22.603787 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:22.603737 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d696792-e30e-4020-b4b5-fc6f7af3c21e" containerName="main" Oct 31 05:22:22.608799 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:22.608767 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k" Oct 31 05:22:22.625805 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:22.625755 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-5534f0-openshift-default-dockercfg-68fw5\"" Oct 31 05:22:22.640550 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:22.640514 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k"] Oct 31 05:22:22.745537 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:22.745489 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e010ac6-7566-49bc-ae3d-02f791df20c7-istio-data\") pod \"e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k\" (UID: \"5e010ac6-7566-49bc-ae3d-02f791df20c7\") " pod="openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k" Oct 31 05:22:22.745761 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:22.745586 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e010ac6-7566-49bc-ae3d-02f791df20c7-workload-certs\") pod \"e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k\" (UID: \"5e010ac6-7566-49bc-ae3d-02f791df20c7\") " pod="openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k" Oct 31 05:22:22.745761 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:22.745631 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e010ac6-7566-49bc-ae3d-02f791df20c7-istio-envoy\") pod \"e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k\" (UID: \"5e010ac6-7566-49bc-ae3d-02f791df20c7\") " pod="openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k" Oct 31 05:22:22.745761 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:22.745679 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e010ac6-7566-49bc-ae3d-02f791df20c7-istiod-ca-cert\") pod \"e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k\" (UID: \"5e010ac6-7566-49bc-ae3d-02f791df20c7\") " pod="openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k" Oct 31 05:22:22.745761 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:22.745730 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e010ac6-7566-49bc-ae3d-02f791df20c7-credential-socket\") pod \"e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k\" (UID: \"5e010ac6-7566-49bc-ae3d-02f791df20c7\") " pod="openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k" Oct 31 05:22:22.745921 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:22.745760 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6jn9\" (UniqueName: \"kubernetes.io/projected/5e010ac6-7566-49bc-ae3d-02f791df20c7-kube-api-access-g6jn9\") pod \"e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k\" (UID: \"5e010ac6-7566-49bc-ae3d-02f791df20c7\") " pod="openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k" Oct 31 05:22:22.745921 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:22.745827 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e010ac6-7566-49bc-ae3d-02f791df20c7-workload-socket\") pod \"e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k\" (UID: \"5e010ac6-7566-49bc-ae3d-02f791df20c7\") " pod="openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k" Oct 31 05:22:22.746002 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:22.745940 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e010ac6-7566-49bc-ae3d-02f791df20c7-istio-podinfo\") pod \"e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k\" (UID: \"5e010ac6-7566-49bc-ae3d-02f791df20c7\") " pod="openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k" Oct 31 05:22:22.746002 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:22.745989 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e010ac6-7566-49bc-ae3d-02f791df20c7-istio-token\") pod \"e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k\" (UID: \"5e010ac6-7566-49bc-ae3d-02f791df20c7\") " pod="openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k" Oct 31 05:22:22.846921 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:22.846880 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e010ac6-7566-49bc-ae3d-02f791df20c7-workload-socket\") pod \"e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k\" (UID: \"5e010ac6-7566-49bc-ae3d-02f791df20c7\") " pod="openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k" Oct 31 05:22:22.847170 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:22.846948 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e010ac6-7566-49bc-ae3d-02f791df20c7-istio-podinfo\") pod \"e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k\" (UID: \"5e010ac6-7566-49bc-ae3d-02f791df20c7\") " pod="openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k" Oct 31 05:22:22.847170 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:22.846975 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e010ac6-7566-49bc-ae3d-02f791df20c7-istio-token\") pod \"e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k\" (UID: \"5e010ac6-7566-49bc-ae3d-02f791df20c7\") " pod="openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k" Oct 31 05:22:22.847170 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:22.847024 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e010ac6-7566-49bc-ae3d-02f791df20c7-istio-data\") pod \"e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k\" (UID: \"5e010ac6-7566-49bc-ae3d-02f791df20c7\") " pod="openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k" Oct 31 05:22:22.847170 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:22.847075 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e010ac6-7566-49bc-ae3d-02f791df20c7-workload-certs\") pod \"e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k\" (UID: \"5e010ac6-7566-49bc-ae3d-02f791df20c7\") " pod="openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k" Oct 31 05:22:22.847170 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:22.847105 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e010ac6-7566-49bc-ae3d-02f791df20c7-istio-envoy\") pod \"e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k\" (UID: \"5e010ac6-7566-49bc-ae3d-02f791df20c7\") " pod="openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k" Oct 31 05:22:22.847560 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:22.847292 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e010ac6-7566-49bc-ae3d-02f791df20c7-istiod-ca-cert\") pod \"e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k\" (UID: \"5e010ac6-7566-49bc-ae3d-02f791df20c7\") " pod="openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k" Oct 31 05:22:22.847560 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:22.847372 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e010ac6-7566-49bc-ae3d-02f791df20c7-credential-socket\") pod \"e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k\" (UID: \"5e010ac6-7566-49bc-ae3d-02f791df20c7\") " pod="openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k" Oct 31 05:22:22.847560 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:22.847404 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6jn9\" (UniqueName: \"kubernetes.io/projected/5e010ac6-7566-49bc-ae3d-02f791df20c7-kube-api-access-g6jn9\") pod \"e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k\" (UID: \"5e010ac6-7566-49bc-ae3d-02f791df20c7\") " pod="openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k" Oct 31 05:22:22.847560 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:22.847428 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e010ac6-7566-49bc-ae3d-02f791df20c7-workload-socket\") pod \"e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k\" (UID: \"5e010ac6-7566-49bc-ae3d-02f791df20c7\") " pod="openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k" Oct 31 05:22:22.847822 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:22.847561 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e010ac6-7566-49bc-ae3d-02f791df20c7-istio-data\") pod \"e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k\" (UID: \"5e010ac6-7566-49bc-ae3d-02f791df20c7\") " pod="openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k" Oct 31 05:22:22.848017 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:22.847915 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e010ac6-7566-49bc-ae3d-02f791df20c7-credential-socket\") pod \"e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k\" (UID: \"5e010ac6-7566-49bc-ae3d-02f791df20c7\") " pod="openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k" Oct 31 05:22:22.848206 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:22.848181 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e010ac6-7566-49bc-ae3d-02f791df20c7-workload-certs\") pod \"e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k\" (UID: \"5e010ac6-7566-49bc-ae3d-02f791df20c7\") " pod="openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k" Oct 31 05:22:22.848900 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:22.848879 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e010ac6-7566-49bc-ae3d-02f791df20c7-istiod-ca-cert\") pod \"e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k\" (UID: \"5e010ac6-7566-49bc-ae3d-02f791df20c7\") " pod="openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k" Oct 31 05:22:22.850465 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:22.850326 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e010ac6-7566-49bc-ae3d-02f791df20c7-istio-podinfo\") pod \"e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k\" (UID: \"5e010ac6-7566-49bc-ae3d-02f791df20c7\") " pod="openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k" Oct 31 05:22:22.850465 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:22.850394 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e010ac6-7566-49bc-ae3d-02f791df20c7-istio-envoy\") pod \"e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k\" (UID: \"5e010ac6-7566-49bc-ae3d-02f791df20c7\") " pod="openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k" Oct 31 05:22:22.872026 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:22.871993 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e010ac6-7566-49bc-ae3d-02f791df20c7-istio-token\") pod \"e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k\" (UID: \"5e010ac6-7566-49bc-ae3d-02f791df20c7\") " pod="openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k" Oct 31 05:22:22.872225 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:22.872200 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6jn9\" (UniqueName: \"kubernetes.io/projected/5e010ac6-7566-49bc-ae3d-02f791df20c7-kube-api-access-g6jn9\") pod \"e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k\" (UID: \"5e010ac6-7566-49bc-ae3d-02f791df20c7\") " pod="openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k" Oct 31 05:22:22.926076 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:22.926028 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k" Oct 31 05:22:23.248073 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:23.248046 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k"] Oct 31 05:22:23.248695 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:22:23.248668 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e010ac6_7566_49bc_ae3d_02f791df20c7.slice/crio-09f42140fc6b8320f8801fc288cc072adf15d6735d159935ba49518ee7c2dc15 WatchSource:0}: Error finding container 09f42140fc6b8320f8801fc288cc072adf15d6735d159935ba49518ee7c2dc15: Status 404 returned error can't find the container with id 09f42140fc6b8320f8801fc288cc072adf15d6735d159935ba49518ee7c2dc15 Oct 31 05:22:23.250933 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:23.250900 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:22:23.251013 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:23.250969 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:22:23.251013 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:23.250996 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:22:23.338864 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:23.338819 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k" event={"ID":"5e010ac6-7566-49bc-ae3d-02f791df20c7","Type":"ContainerStarted","Data":"ce441c25c90a50804c1005f73da04a6d623ec74b796f71c5c8dc324d77d8377b"} Oct 31 05:22:23.339057 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:23.338891 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k" event={"ID":"5e010ac6-7566-49bc-ae3d-02f791df20c7","Type":"ContainerStarted","Data":"09f42140fc6b8320f8801fc288cc072adf15d6735d159935ba49518ee7c2dc15"} Oct 31 05:22:23.386436 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:23.386364 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k" podStartSLOduration=1.386342566 podStartE2EDuration="1.386342566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:22:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:22:23.383238683 +0000 UTC m=+4698.139576720" watchObservedRunningTime="2026-10-31 05:22:23.386342566 +0000 UTC m=+4698.142680599" Oct 31 05:22:23.926878 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:23.926834 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k" Oct 31 05:22:24.026609 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:24.026569 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7560a85-qrxrs"] Oct 31 05:22:24.029713 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:24.029676 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7560a85-qrxrs"] Oct 31 05:22:24.932210 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:24.932171 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k" Oct 31 05:22:25.351287 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:25.351253 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k" Oct 31 05:22:25.352308 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:25.352283 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k" Oct 31 05:22:25.407847 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:25.407810 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e1ab956-2e52-4699-bbfa-57233477f537" path="/var/lib/kubelet/pods/2e1ab956-2e52-4699-bbfa-57233477f537/volumes" Oct 31 05:22:27.657993 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:27.657951 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-5534f0-869ff6db452mp4"] Oct 31 05:22:27.662958 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:27.662923 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5534f0-869ff6db452mp4" Oct 31 05:22:27.683021 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:27.682981 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-5534f0\"" Oct 31 05:22:27.683021 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:27.683008 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-bf061496-dockercfg-cksmx\"" Oct 31 05:22:27.698296 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:27.698265 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-5534f0-869ff6db452mp4"] Oct 31 05:22:27.729519 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:27.729482 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7ltrbs"] Oct 31 05:22:27.733421 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:27.733397 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7ltrbs" Oct 31 05:22:27.757483 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:27.757456 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7ltrbs"] Oct 31 05:22:27.826301 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:27.826260 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7683a47b-f45f-4bd7-a52b-65375ee7fdf8-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-5534f0-869ff6db452mp4\" (UID: \"7683a47b-f45f-4bd7-a52b-65375ee7fdf8\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-5534f0-869ff6db452mp4" Oct 31 05:22:27.826511 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:27.826359 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lft9t\" (UniqueName: \"kubernetes.io/projected/8fc184d7-c471-4093-981f-05b01cc441a5-kube-api-access-lft9t\") pod \"payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7ltrbs\" (UID: \"8fc184d7-c471-4093-981f-05b01cc441a5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7ltrbs" Oct 31 05:22:27.826511 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:27.826428 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fc184d7-c471-4093-981f-05b01cc441a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7ltrbs\" (UID: \"8fc184d7-c471-4093-981f-05b01cc441a5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7ltrbs" Oct 31 05:22:27.826511 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:27.826475 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhmp6\" (UniqueName: \"kubernetes.io/projected/7683a47b-f45f-4bd7-a52b-65375ee7fdf8-kube-api-access-lhmp6\") pod \"payload-processing-e2e-worker-single-main-5534f0-869ff6db452mp4\" (UID: \"7683a47b-f45f-4bd7-a52b-65375ee7fdf8\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-5534f0-869ff6db452mp4" Oct 31 05:22:27.927815 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:27.927731 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7683a47b-f45f-4bd7-a52b-65375ee7fdf8-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-5534f0-869ff6db452mp4\" (UID: \"7683a47b-f45f-4bd7-a52b-65375ee7fdf8\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-5534f0-869ff6db452mp4" Oct 31 05:22:27.927815 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:27.927790 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lft9t\" (UniqueName: \"kubernetes.io/projected/8fc184d7-c471-4093-981f-05b01cc441a5-kube-api-access-lft9t\") pod \"payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7ltrbs\" (UID: \"8fc184d7-c471-4093-981f-05b01cc441a5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7ltrbs" Oct 31 05:22:27.928074 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:27.927824 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fc184d7-c471-4093-981f-05b01cc441a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7ltrbs\" (UID: \"8fc184d7-c471-4093-981f-05b01cc441a5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7ltrbs" Oct 31 05:22:27.928074 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:27.927914 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhmp6\" (UniqueName: \"kubernetes.io/projected/7683a47b-f45f-4bd7-a52b-65375ee7fdf8-kube-api-access-lhmp6\") pod \"payload-processing-e2e-worker-single-main-5534f0-869ff6db452mp4\" (UID: \"7683a47b-f45f-4bd7-a52b-65375ee7fdf8\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-5534f0-869ff6db452mp4" Oct 31 05:22:27.928528 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:27.928505 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fc184d7-c471-4093-981f-05b01cc441a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7ltrbs\" (UID: \"8fc184d7-c471-4093-981f-05b01cc441a5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7ltrbs" Oct 31 05:22:27.928640 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:27.928579 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7683a47b-f45f-4bd7-a52b-65375ee7fdf8-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-5534f0-869ff6db452mp4\" (UID: \"7683a47b-f45f-4bd7-a52b-65375ee7fdf8\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-5534f0-869ff6db452mp4" Oct 31 05:22:27.950939 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:27.950909 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lft9t\" (UniqueName: \"kubernetes.io/projected/8fc184d7-c471-4093-981f-05b01cc441a5-kube-api-access-lft9t\") pod \"payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7ltrbs\" (UID: \"8fc184d7-c471-4093-981f-05b01cc441a5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7ltrbs" Oct 31 05:22:27.951051 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:27.950952 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhmp6\" (UniqueName: \"kubernetes.io/projected/7683a47b-f45f-4bd7-a52b-65375ee7fdf8-kube-api-access-lhmp6\") pod \"payload-processing-e2e-worker-single-main-5534f0-869ff6db452mp4\" (UID: \"7683a47b-f45f-4bd7-a52b-65375ee7fdf8\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-5534f0-869ff6db452mp4" Oct 31 05:22:27.977676 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:27.977638 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5534f0-869ff6db452mp4" Oct 31 05:22:28.046958 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:28.046687 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7ltrbs" Oct 31 05:22:28.278919 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:28.278889 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-5534f0-869ff6db452mp4"] Oct 31 05:22:28.281811 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:22:28.281779 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7683a47b_f45f_4bd7_a52b_65375ee7fdf8.slice/crio-c69bcb4508ff1f2e69ccceff86e542834f2da518593db92902eda62785c621cb WatchSource:0}: Error finding container c69bcb4508ff1f2e69ccceff86e542834f2da518593db92902eda62785c621cb: Status 404 returned error can't find the container with id c69bcb4508ff1f2e69ccceff86e542834f2da518593db92902eda62785c621cb Oct 31 05:22:28.368145 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:28.368112 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7ltrbs"] Oct 31 05:22:28.368843 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:22:28.368809 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fc184d7_c471_4093_981f_05b01cc441a5.slice/crio-8f4eacf1d3d11e5d742c798ad95693e301abf53d188a874df8f3aa3c204213c3 WatchSource:0}: Error finding container 8f4eacf1d3d11e5d742c798ad95693e301abf53d188a874df8f3aa3c204213c3: Status 404 returned error can't find the container with id 8f4eacf1d3d11e5d742c798ad95693e301abf53d188a874df8f3aa3c204213c3 Oct 31 05:22:28.375722 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:28.375694 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5534f0-869ff6db452mp4" event={"ID":"7683a47b-f45f-4bd7-a52b-65375ee7fdf8","Type":"ContainerStarted","Data":"29f14f4559106932aa5d04412bd76be3fdcfec8ede970860f24e66d2b0dd46a1"} Oct 31 05:22:28.375829 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:28.375747 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5534f0-869ff6db452mp4" event={"ID":"7683a47b-f45f-4bd7-a52b-65375ee7fdf8","Type":"ContainerStarted","Data":"c69bcb4508ff1f2e69ccceff86e542834f2da518593db92902eda62785c621cb"} Oct 31 05:22:28.375829 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:28.375775 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5534f0-869ff6db452mp4" Oct 31 05:22:28.417751 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:28.417688 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5534f0-869ff6db452mp4" podStartSLOduration=1.417670945 podStartE2EDuration="1.417670945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:22:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:22:28.414759065 +0000 UTC m=+4703.171097107" watchObservedRunningTime="2026-10-31 05:22:28.417670945 +0000 UTC m=+4703.174008985" Oct 31 05:22:29.384708 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:29.384665 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7ltrbs" event={"ID":"8fc184d7-c471-4093-981f-05b01cc441a5","Type":"ContainerStarted","Data":"b3719c8b8ff548be0d9bbb9cd0fdea6b1b8027e4a4a5a37411592750dda5a0a2"} Oct 31 05:22:29.385152 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:29.384732 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7ltrbs" event={"ID":"8fc184d7-c471-4093-981f-05b01cc441a5","Type":"ContainerStarted","Data":"8f4eacf1d3d11e5d742c798ad95693e301abf53d188a874df8f3aa3c204213c3"} Oct 31 05:22:29.385152 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:29.384997 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7ltrbs" Oct 31 05:22:29.425789 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:29.425731 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7ltrbs" podStartSLOduration=2.425716376 podStartE2EDuration="2.425716376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:22:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:22:29.424685245 +0000 UTC m=+4704.181023286" watchObservedRunningTime="2026-10-31 05:22:29.425716376 +0000 UTC m=+4704.182054426" Oct 31 05:22:32.029347 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:32.029299 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-784ca31a-nrq79"] Oct 31 05:22:32.041491 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:32.041462 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-784ca31a-nrq79"] Oct 31 05:22:33.406723 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:33.406685 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="263a72f1-a00d-4e12-baed-e9bff6a0788c" path="/var/lib/kubelet/pods/263a72f1-a00d-4e12-baed-e9bff6a0788c/volumes" Oct 31 05:22:37.029077 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:37.029029 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fea8164c-bfqxm"] Oct 31 05:22:37.040120 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:37.040080 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fea8164c-bfqxm"] Oct 31 05:22:37.407504 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:37.407403 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="247374b7-c1a3-4eae-bba4-76a070eed47c" path="/var/lib/kubelet/pods/247374b7-c1a3-4eae-bba4-76a070eed47c/volumes" Oct 31 05:22:39.388226 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:39.388156 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5534f0-869ff6db452mp4" Oct 31 05:22:40.395500 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:40.395468 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7ltrbs" Oct 31 05:22:40.666759 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:40.665592 5019 patch_prober.go:29] interesting pod/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.229:15021/healthz/ready\": read tcp 10.132.0.2:50032->10.132.1.229:15021: read: connection reset by peer" start-of-body= Oct 31 05:22:40.666759 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:40.665660 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k" podUID="5e010ac6-7566-49bc-ae3d-02f791df20c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.229:15021/healthz/ready\": read tcp 10.132.0.2:50032->10.132.1.229:15021: read: connection reset by peer" Oct 31 05:22:41.478261 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:41.478223 5019 generic.go:356] "Generic (PLEG): container finished" podID="5e010ac6-7566-49bc-ae3d-02f791df20c7" containerID="ce441c25c90a50804c1005f73da04a6d623ec74b796f71c5c8dc324d77d8377b" exitCode=0 Oct 31 05:22:41.478756 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:41.478303 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k" event={"ID":"5e010ac6-7566-49bc-ae3d-02f791df20c7","Type":"ContainerDied","Data":"ce441c25c90a50804c1005f73da04a6d623ec74b796f71c5c8dc324d77d8377b"} Oct 31 05:22:41.478888 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:41.478866 5019 scope.go:122] "RemoveContainer" containerID="ce441c25c90a50804c1005f73da04a6d623ec74b796f71c5c8dc324d77d8377b" Oct 31 05:22:41.480794 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:41.480742 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:22:41.480951 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:41.480921 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:22:41.481052 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:41.481028 5019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 05:22:42.500653 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:42.500614 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k" event={"ID":"5e010ac6-7566-49bc-ae3d-02f791df20c7","Type":"ContainerStarted","Data":"0ee93aa4abf8f3f476323333c29e210d8c8054f32e8001180bc6b5fa30582c94"} Oct 31 05:22:42.927017 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:42.926909 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k" Oct 31 05:22:42.927209 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:42.927151 5019 patch_prober.go:29] interesting pod/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.229:15021/healthz/ready\": dial tcp 10.132.1.229:15021: connect: connection refused" start-of-body= Oct 31 05:22:42.927262 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:42.927202 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k" podUID="5e010ac6-7566-49bc-ae3d-02f791df20c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.229:15021/healthz/ready\": dial tcp 10.132.1.229:15021: connect: connection refused" Oct 31 05:22:43.118011 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:43.117950 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-5534f0/simulator-single-main-5534f0-kserve-6ff498fcc7-ffwlt"] Oct 31 05:22:43.123401 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:43.123356 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-5534f0/simulator-single-main-5534f0-kserve-6ff498fcc7-ffwlt" Oct 31 05:22:43.138955 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:43.138925 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-5534f0\"/\"openshift-service-ca.crt\"" Oct 31 05:22:43.139121 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:43.138985 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-5534f0\"/\"kube-root-ca.crt\"" Oct 31 05:22:43.139949 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:43.139894 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-5534f0/simulator-single-main-5534f0-kserve-6ff498fcc7-ffwlt"] Oct 31 05:22:43.148697 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:43.148655 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-5534f0\"/\"default-dockercfg-gkw4c\"" Oct 31 05:22:43.148883 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:43.148739 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-5534f0\"/\"simulator-single-main-5534f0-kserve-self-signed-certs\"" Oct 31 05:22:43.212457 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:43.212368 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c18017cd-ffd0-418f-b57b-66e8f0867b86-home\") pod \"simulator-single-main-5534f0-kserve-6ff498fcc7-ffwlt\" (UID: \"c18017cd-ffd0-418f-b57b-66e8f0867b86\") " pod="e2e-models-e2e-worker-single-main-5534f0/simulator-single-main-5534f0-kserve-6ff498fcc7-ffwlt" Oct 31 05:22:43.212457 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:43.212429 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5prm\" (UniqueName: \"kubernetes.io/projected/c18017cd-ffd0-418f-b57b-66e8f0867b86-kube-api-access-g5prm\") pod \"simulator-single-main-5534f0-kserve-6ff498fcc7-ffwlt\" (UID: \"c18017cd-ffd0-418f-b57b-66e8f0867b86\") " pod="e2e-models-e2e-worker-single-main-5534f0/simulator-single-main-5534f0-kserve-6ff498fcc7-ffwlt" Oct 31 05:22:43.212690 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:43.212510 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c18017cd-ffd0-418f-b57b-66e8f0867b86-tls-certs\") pod \"simulator-single-main-5534f0-kserve-6ff498fcc7-ffwlt\" (UID: \"c18017cd-ffd0-418f-b57b-66e8f0867b86\") " pod="e2e-models-e2e-worker-single-main-5534f0/simulator-single-main-5534f0-kserve-6ff498fcc7-ffwlt" Oct 31 05:22:43.212690 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:43.212546 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c18017cd-ffd0-418f-b57b-66e8f0867b86-dshm\") pod \"simulator-single-main-5534f0-kserve-6ff498fcc7-ffwlt\" (UID: \"c18017cd-ffd0-418f-b57b-66e8f0867b86\") " pod="e2e-models-e2e-worker-single-main-5534f0/simulator-single-main-5534f0-kserve-6ff498fcc7-ffwlt" Oct 31 05:22:43.212690 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:43.212613 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c18017cd-ffd0-418f-b57b-66e8f0867b86-model-cache\") pod \"simulator-single-main-5534f0-kserve-6ff498fcc7-ffwlt\" (UID: \"c18017cd-ffd0-418f-b57b-66e8f0867b86\") " pod="e2e-models-e2e-worker-single-main-5534f0/simulator-single-main-5534f0-kserve-6ff498fcc7-ffwlt" Oct 31 05:22:43.212690 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:43.212665 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c18017cd-ffd0-418f-b57b-66e8f0867b86-tmp-dir\") pod \"simulator-single-main-5534f0-kserve-6ff498fcc7-ffwlt\" (UID: \"c18017cd-ffd0-418f-b57b-66e8f0867b86\") " pod="e2e-models-e2e-worker-single-main-5534f0/simulator-single-main-5534f0-kserve-6ff498fcc7-ffwlt" Oct 31 05:22:43.313852 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:43.313813 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c18017cd-ffd0-418f-b57b-66e8f0867b86-home\") pod \"simulator-single-main-5534f0-kserve-6ff498fcc7-ffwlt\" (UID: \"c18017cd-ffd0-418f-b57b-66e8f0867b86\") " pod="e2e-models-e2e-worker-single-main-5534f0/simulator-single-main-5534f0-kserve-6ff498fcc7-ffwlt" Oct 31 05:22:43.314095 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:43.313879 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5prm\" (UniqueName: \"kubernetes.io/projected/c18017cd-ffd0-418f-b57b-66e8f0867b86-kube-api-access-g5prm\") pod \"simulator-single-main-5534f0-kserve-6ff498fcc7-ffwlt\" (UID: \"c18017cd-ffd0-418f-b57b-66e8f0867b86\") " pod="e2e-models-e2e-worker-single-main-5534f0/simulator-single-main-5534f0-kserve-6ff498fcc7-ffwlt" Oct 31 05:22:43.314095 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:43.313910 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c18017cd-ffd0-418f-b57b-66e8f0867b86-tls-certs\") pod \"simulator-single-main-5534f0-kserve-6ff498fcc7-ffwlt\" (UID: \"c18017cd-ffd0-418f-b57b-66e8f0867b86\") " pod="e2e-models-e2e-worker-single-main-5534f0/simulator-single-main-5534f0-kserve-6ff498fcc7-ffwlt" Oct 31 05:22:43.314095 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:43.313931 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c18017cd-ffd0-418f-b57b-66e8f0867b86-dshm\") pod \"simulator-single-main-5534f0-kserve-6ff498fcc7-ffwlt\" (UID: \"c18017cd-ffd0-418f-b57b-66e8f0867b86\") " pod="e2e-models-e2e-worker-single-main-5534f0/simulator-single-main-5534f0-kserve-6ff498fcc7-ffwlt" Oct 31 05:22:43.314095 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:43.313969 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c18017cd-ffd0-418f-b57b-66e8f0867b86-model-cache\") pod \"simulator-single-main-5534f0-kserve-6ff498fcc7-ffwlt\" (UID: \"c18017cd-ffd0-418f-b57b-66e8f0867b86\") " pod="e2e-models-e2e-worker-single-main-5534f0/simulator-single-main-5534f0-kserve-6ff498fcc7-ffwlt" Oct 31 05:22:43.314095 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:43.314013 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c18017cd-ffd0-418f-b57b-66e8f0867b86-tmp-dir\") pod \"simulator-single-main-5534f0-kserve-6ff498fcc7-ffwlt\" (UID: \"c18017cd-ffd0-418f-b57b-66e8f0867b86\") " pod="e2e-models-e2e-worker-single-main-5534f0/simulator-single-main-5534f0-kserve-6ff498fcc7-ffwlt" Oct 31 05:22:43.314455 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:43.314400 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c18017cd-ffd0-418f-b57b-66e8f0867b86-home\") pod \"simulator-single-main-5534f0-kserve-6ff498fcc7-ffwlt\" (UID: \"c18017cd-ffd0-418f-b57b-66e8f0867b86\") " pod="e2e-models-e2e-worker-single-main-5534f0/simulator-single-main-5534f0-kserve-6ff498fcc7-ffwlt" Oct 31 05:22:43.314508 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:43.314464 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c18017cd-ffd0-418f-b57b-66e8f0867b86-tmp-dir\") pod \"simulator-single-main-5534f0-kserve-6ff498fcc7-ffwlt\" (UID: \"c18017cd-ffd0-418f-b57b-66e8f0867b86\") " pod="e2e-models-e2e-worker-single-main-5534f0/simulator-single-main-5534f0-kserve-6ff498fcc7-ffwlt" Oct 31 05:22:43.314508 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:43.314486 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c18017cd-ffd0-418f-b57b-66e8f0867b86-model-cache\") pod \"simulator-single-main-5534f0-kserve-6ff498fcc7-ffwlt\" (UID: \"c18017cd-ffd0-418f-b57b-66e8f0867b86\") " pod="e2e-models-e2e-worker-single-main-5534f0/simulator-single-main-5534f0-kserve-6ff498fcc7-ffwlt" Oct 31 05:22:43.316300 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:43.316274 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c18017cd-ffd0-418f-b57b-66e8f0867b86-dshm\") pod \"simulator-single-main-5534f0-kserve-6ff498fcc7-ffwlt\" (UID: \"c18017cd-ffd0-418f-b57b-66e8f0867b86\") " pod="e2e-models-e2e-worker-single-main-5534f0/simulator-single-main-5534f0-kserve-6ff498fcc7-ffwlt" Oct 31 05:22:43.316654 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:43.316635 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c18017cd-ffd0-418f-b57b-66e8f0867b86-tls-certs\") pod \"simulator-single-main-5534f0-kserve-6ff498fcc7-ffwlt\" (UID: \"c18017cd-ffd0-418f-b57b-66e8f0867b86\") " pod="e2e-models-e2e-worker-single-main-5534f0/simulator-single-main-5534f0-kserve-6ff498fcc7-ffwlt" Oct 31 05:22:43.338681 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:43.338608 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5prm\" (UniqueName: \"kubernetes.io/projected/c18017cd-ffd0-418f-b57b-66e8f0867b86-kube-api-access-g5prm\") pod \"simulator-single-main-5534f0-kserve-6ff498fcc7-ffwlt\" (UID: \"c18017cd-ffd0-418f-b57b-66e8f0867b86\") " pod="e2e-models-e2e-worker-single-main-5534f0/simulator-single-main-5534f0-kserve-6ff498fcc7-ffwlt" Oct 31 05:22:43.437800 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:43.437749 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-5534f0/simulator-single-main-5534f0-kserve-6ff498fcc7-ffwlt" Oct 31 05:22:43.765183 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:43.765150 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-5534f0/simulator-single-main-5534f0-kserve-6ff498fcc7-ffwlt"] Oct 31 05:22:43.766534 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:22:43.766504 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc18017cd_ffd0_418f_b57b_66e8f0867b86.slice/crio-e1317f2566bb067625214dedd9a6913901ef47e1e735f3d2e5d448006c1b8bc5 WatchSource:0}: Error finding container e1317f2566bb067625214dedd9a6913901ef47e1e735f3d2e5d448006c1b8bc5: Status 404 returned error can't find the container with id e1317f2566bb067625214dedd9a6913901ef47e1e735f3d2e5d448006c1b8bc5 Oct 31 05:22:43.926997 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:43.926961 5019 patch_prober.go:29] interesting pod/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.229:15021/healthz/ready\": dial tcp 10.132.1.229:15021: connect: connection refused" start-of-body= Oct 31 05:22:43.927205 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:43.927019 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k" podUID="5e010ac6-7566-49bc-ae3d-02f791df20c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.229:15021/healthz/ready\": dial tcp 10.132.1.229:15021: connect: connection refused" Oct 31 05:22:44.518532 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:44.518491 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-5534f0/simulator-single-main-5534f0-kserve-6ff498fcc7-ffwlt" event={"ID":"c18017cd-ffd0-418f-b57b-66e8f0867b86","Type":"ContainerStarted","Data":"20d8731db07545a6fc782e0714f0852e94d6aba8428fd45eeb5fdbec87c8f77d"} Oct 31 05:22:44.518532 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:44.518538 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-5534f0/simulator-single-main-5534f0-kserve-6ff498fcc7-ffwlt" event={"ID":"c18017cd-ffd0-418f-b57b-66e8f0867b86","Type":"ContainerStarted","Data":"e1317f2566bb067625214dedd9a6913901ef47e1e735f3d2e5d448006c1b8bc5"} Oct 31 05:22:44.559945 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:44.559885 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-5534f0/simulator-single-main-5534f0-kserve-6ff498fcc7-ffwlt" podStartSLOduration=1.559869113 podStartE2EDuration="1.559869113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:22:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:22:44.55890311 +0000 UTC m=+4719.315241152" watchObservedRunningTime="2026-10-31 05:22:44.559869113 +0000 UTC m=+4719.316207152" Oct 31 05:22:44.926813 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:44.926712 5019 patch_prober.go:29] interesting pod/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.229:15021/healthz/ready\": dial tcp 10.132.1.229:15021: connect: connection refused" start-of-body= Oct 31 05:22:44.926813 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:44.926787 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k" podUID="5e010ac6-7566-49bc-ae3d-02f791df20c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.229:15021/healthz/ready\": dial tcp 10.132.1.229:15021: connect: connection refused" Oct 31 05:22:45.438676 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:45.438580 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-5534f0/simulator-single-main-5534f0-kserve-6ff498fcc7-ffwlt" Oct 31 05:22:45.926989 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:45.926941 5019 patch_prober.go:29] interesting pod/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.229:15021/healthz/ready\": dial tcp 10.132.1.229:15021: connect: connection refused" start-of-body= Oct 31 05:22:45.927484 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:45.927029 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k" podUID="5e010ac6-7566-49bc-ae3d-02f791df20c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.229:15021/healthz/ready\": dial tcp 10.132.1.229:15021: connect: connection refused" Oct 31 05:22:46.927356 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:46.926827 5019 patch_prober.go:29] interesting pod/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.229:15021/healthz/ready\": dial tcp 10.132.1.229:15021: connect: connection refused" start-of-body= Oct 31 05:22:46.927356 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:46.926899 5019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k" podUID="5e010ac6-7566-49bc-ae3d-02f791df20c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.229:15021/healthz/ready\": dial tcp 10.132.1.229:15021: connect: connection refused" Oct 31 05:22:48.144459 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:48.144411 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k" Oct 31 05:22:48.145852 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:48.145714 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k" Oct 31 05:22:48.147089 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:48.147050 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k" Oct 31 05:22:52.393040 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:22:52.392998 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:22:53.438248 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:53.438203 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-5534f0/simulator-single-main-5534f0-kserve-6ff498fcc7-ffwlt" Oct 31 05:22:53.451375 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:53.451322 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-5534f0/simulator-single-main-5534f0-kserve-6ff498fcc7-ffwlt" Oct 31 05:22:53.595848 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:22:53.595811 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-5534f0/simulator-single-main-5534f0-kserve-6ff498fcc7-ffwlt" Oct 31 05:23:13.708806 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:13.708755 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05fee954-7r5zh"] Oct 31 05:23:13.714444 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:13.714405 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05fee954-7r5zh" Oct 31 05:23:13.733294 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:13.733242 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05fee954-7r5zh"] Oct 31 05:23:13.740529 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:13.740481 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:23:13.740529 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:13.740516 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:23:13.740844 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:13.740558 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfhzn\"" Oct 31 05:23:13.745800 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:13.745763 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:23:13.774824 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:13.774772 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmfs7\" (UniqueName: \"kubernetes.io/projected/d6b22fd0-53dd-4de4-9cc5-272f6abc02ab-kube-api-access-xmfs7\") pod \"maas-api-revoke-sub-05fee954-7r5zh\" (UID: \"d6b22fd0-53dd-4de4-9cc5-272f6abc02ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05fee954-7r5zh" Oct 31 05:23:13.775083 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:13.774961 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b22fd0-53dd-4de4-9cc5-272f6abc02ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05fee954-7r5zh\" (UID: \"d6b22fd0-53dd-4de4-9cc5-272f6abc02ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05fee954-7r5zh" Oct 31 05:23:13.775175 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:13.775091 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b22fd0-53dd-4de4-9cc5-272f6abc02ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05fee954-7r5zh\" (UID: \"d6b22fd0-53dd-4de4-9cc5-272f6abc02ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05fee954-7r5zh" Oct 31 05:23:13.876699 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:13.876655 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b22fd0-53dd-4de4-9cc5-272f6abc02ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05fee954-7r5zh\" (UID: \"d6b22fd0-53dd-4de4-9cc5-272f6abc02ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05fee954-7r5zh" Oct 31 05:23:13.876923 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:13.876737 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b22fd0-53dd-4de4-9cc5-272f6abc02ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05fee954-7r5zh\" (UID: \"d6b22fd0-53dd-4de4-9cc5-272f6abc02ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05fee954-7r5zh" Oct 31 05:23:13.876923 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:13.876795 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmfs7\" (UniqueName: \"kubernetes.io/projected/d6b22fd0-53dd-4de4-9cc5-272f6abc02ab-kube-api-access-xmfs7\") pod \"maas-api-revoke-sub-05fee954-7r5zh\" (UID: \"d6b22fd0-53dd-4de4-9cc5-272f6abc02ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05fee954-7r5zh" Oct 31 05:23:13.877412 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:13.877379 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b22fd0-53dd-4de4-9cc5-272f6abc02ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05fee954-7r5zh\" (UID: \"d6b22fd0-53dd-4de4-9cc5-272f6abc02ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05fee954-7r5zh" Oct 31 05:23:13.879387 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:13.879358 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b22fd0-53dd-4de4-9cc5-272f6abc02ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05fee954-7r5zh\" (UID: \"d6b22fd0-53dd-4de4-9cc5-272f6abc02ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05fee954-7r5zh" Oct 31 05:23:13.900595 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:13.900558 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmfs7\" (UniqueName: \"kubernetes.io/projected/d6b22fd0-53dd-4de4-9cc5-272f6abc02ab-kube-api-access-xmfs7\") pod \"maas-api-revoke-sub-05fee954-7r5zh\" (UID: \"d6b22fd0-53dd-4de4-9cc5-272f6abc02ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05fee954-7r5zh" Oct 31 05:23:14.035720 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:14.035683 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05fee954-7r5zh" Oct 31 05:23:14.347613 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:14.347574 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05fee954-7r5zh"] Oct 31 05:23:14.348867 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:23:14.348838 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6b22fd0_53dd_4de4_9cc5_272f6abc02ab.slice/crio-57ab0a6e48ffc59b6b1d5ba3652413d22facc3391d45c43628a47693f6532c76 WatchSource:0}: Error finding container 57ab0a6e48ffc59b6b1d5ba3652413d22facc3391d45c43628a47693f6532c76: Status 404 returned error can't find the container with id 57ab0a6e48ffc59b6b1d5ba3652413d22facc3391d45c43628a47693f6532c76 Oct 31 05:23:14.732732 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:14.732688 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05fee954-7r5zh" event={"ID":"d6b22fd0-53dd-4de4-9cc5-272f6abc02ab","Type":"ContainerStarted","Data":"012748f7f1b7e3e7f9c05615ff98ecded562c1eb282f1f0fbd843fa8335c43fa"} Oct 31 05:23:14.733143 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:14.732753 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05fee954-7r5zh" event={"ID":"d6b22fd0-53dd-4de4-9cc5-272f6abc02ab","Type":"ContainerStarted","Data":"57ab0a6e48ffc59b6b1d5ba3652413d22facc3391d45c43628a47693f6532c76"} Oct 31 05:23:14.773217 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:14.773149 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05fee954-7r5zh" podStartSLOduration=1.773131508 podStartE2EDuration="1.773131508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:23:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:23:14.772658171 +0000 UTC m=+4749.528996213" watchObservedRunningTime="2026-10-31 05:23:14.773131508 +0000 UTC m=+4749.529469549" Oct 31 05:23:15.741740 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:15.741692 5019 generic.go:356] "Generic (PLEG): container finished" podID="d6b22fd0-53dd-4de4-9cc5-272f6abc02ab" containerID="012748f7f1b7e3e7f9c05615ff98ecded562c1eb282f1f0fbd843fa8335c43fa" exitCode=0 Oct 31 05:23:15.742300 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:15.741776 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05fee954-7r5zh" event={"ID":"d6b22fd0-53dd-4de4-9cc5-272f6abc02ab","Type":"ContainerDied","Data":"012748f7f1b7e3e7f9c05615ff98ecded562c1eb282f1f0fbd843fa8335c43fa"} Oct 31 05:23:16.903572 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:16.903540 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05fee954-7r5zh" Oct 31 05:23:16.923418 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:16.923378 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6b22fd0-53dd-4de4-9cc5-272f6abc02ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b22fd0-53dd-4de4-9cc5-272f6abc02ab-maas-api-xks-ca\") pod \"d6b22fd0-53dd-4de4-9cc5-272f6abc02ab\" (UID: \"d6b22fd0-53dd-4de4-9cc5-272f6abc02ab\") " Oct 31 05:23:16.923632 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:16.923532 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6b22fd0-53dd-4de4-9cc5-272f6abc02ab-kube-api-access-xmfs7\" (UniqueName: \"kubernetes.io/projected/d6b22fd0-53dd-4de4-9cc5-272f6abc02ab-kube-api-access-xmfs7\") pod \"d6b22fd0-53dd-4de4-9cc5-272f6abc02ab\" (UID: \"d6b22fd0-53dd-4de4-9cc5-272f6abc02ab\") " Oct 31 05:23:16.923632 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:16.923589 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6b22fd0-53dd-4de4-9cc5-272f6abc02ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b22fd0-53dd-4de4-9cc5-272f6abc02ab-maas-api-openshift-ca\") pod \"d6b22fd0-53dd-4de4-9cc5-272f6abc02ab\" (UID: \"d6b22fd0-53dd-4de4-9cc5-272f6abc02ab\") " Oct 31 05:23:16.924018 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:16.923986 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6b22fd0-53dd-4de4-9cc5-272f6abc02ab-maas-api-openshift-ca" pod "d6b22fd0-53dd-4de4-9cc5-272f6abc02ab" (UID: "d6b22fd0-53dd-4de4-9cc5-272f6abc02ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:23:16.925698 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:16.925677 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6b22fd0-53dd-4de4-9cc5-272f6abc02ab-maas-api-xks-ca" pod "d6b22fd0-53dd-4de4-9cc5-272f6abc02ab" (UID: "d6b22fd0-53dd-4de4-9cc5-272f6abc02ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:23:16.925883 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:16.925863 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6b22fd0-53dd-4de4-9cc5-272f6abc02ab-kube-api-access-xmfs7" pod "d6b22fd0-53dd-4de4-9cc5-272f6abc02ab" (UID: "d6b22fd0-53dd-4de4-9cc5-272f6abc02ab"). InnerVolumeSpecName "kube-api-access-xmfs7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:23:17.025604 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:17.025509 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmfs7\" (UniqueName: \"kubernetes.io/projected/d6b22fd0-53dd-4de4-9cc5-272f6abc02ab-kube-api-access-xmfs7\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:23:17.025604 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:17.025545 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b22fd0-53dd-4de4-9cc5-272f6abc02ab-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:23:17.025604 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:17.025558 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b22fd0-53dd-4de4-9cc5-272f6abc02ab-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:23:17.758056 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:17.758011 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05fee954-7r5zh" event={"ID":"d6b22fd0-53dd-4de4-9cc5-272f6abc02ab","Type":"ContainerDied","Data":"57ab0a6e48ffc59b6b1d5ba3652413d22facc3391d45c43628a47693f6532c76"} Oct 31 05:23:17.758280 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:17.758070 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05fee954-7r5zh" Oct 31 05:23:17.758280 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:17.758075 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57ab0a6e48ffc59b6b1d5ba3652413d22facc3391d45c43628a47693f6532c76" Oct 31 05:23:22.392881 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:23:22.392838 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:23:29.573443 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:29.573389 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-5534f0/simulator-single-main-5534f0-kserve-6ff498fcc7-ffwlt"] Oct 31 05:23:31.596142 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:31.596062 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-5534f0-869ff6db452mp4"] Oct 31 05:23:31.627212 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:31.627164 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7ltrbs"] Oct 31 05:23:36.597299 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:36.597229 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5534f0-869ff6db452mp4" podUID="7683a47b-f45f-4bd7-a52b-65375ee7fdf8" containerName="payload-processing" containerID="cri-o://29f14f4559106932aa5d04412bd76be3fdcfec8ede970860f24e66d2b0dd46a1" gracePeriod=25 Oct 31 05:23:36.627721 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:36.627672 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7ltrbs" podUID="8fc184d7-c471-4093-981f-05b01cc441a5" containerName="payload-pre-processing" containerID="cri-o://b3719c8b8ff548be0d9bbb9cd0fdea6b1b8027e4a4a5a37411592750dda5a0a2" gracePeriod=25 Oct 31 05:23:39.386796 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:39.386749 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5534f0-869ff6db452mp4" podUID="7683a47b-f45f-4bd7-a52b-65375ee7fdf8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:23:40.393896 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:40.393850 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7ltrbs" podUID="8fc184d7-c471-4093-981f-05b01cc441a5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:23:44.596387 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:44.596305 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-5534f0/simulator-single-main-5534f0-kserve-6ff498fcc7-ffwlt" podUID="c18017cd-ffd0-418f-b57b-66e8f0867b86" containerName="main" containerID="cri-o://20d8731db07545a6fc782e0714f0852e94d6aba8428fd45eeb5fdbec87c8f77d" gracePeriod=45 Oct 31 05:23:44.867435 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:44.867404 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-5534f0/simulator-single-main-5534f0-kserve-6ff498fcc7-ffwlt" Oct 31 05:23:44.943207 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:44.943164 5019 generic.go:356] "Generic (PLEG): container finished" podID="c18017cd-ffd0-418f-b57b-66e8f0867b86" containerID="20d8731db07545a6fc782e0714f0852e94d6aba8428fd45eeb5fdbec87c8f77d" exitCode=0 Oct 31 05:23:44.943439 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:44.943214 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-5534f0/simulator-single-main-5534f0-kserve-6ff498fcc7-ffwlt" Oct 31 05:23:44.943439 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:44.943245 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-5534f0/simulator-single-main-5534f0-kserve-6ff498fcc7-ffwlt" event={"ID":"c18017cd-ffd0-418f-b57b-66e8f0867b86","Type":"ContainerDied","Data":"20d8731db07545a6fc782e0714f0852e94d6aba8428fd45eeb5fdbec87c8f77d"} Oct 31 05:23:44.943439 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:44.943287 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-5534f0/simulator-single-main-5534f0-kserve-6ff498fcc7-ffwlt" event={"ID":"c18017cd-ffd0-418f-b57b-66e8f0867b86","Type":"ContainerDied","Data":"e1317f2566bb067625214dedd9a6913901ef47e1e735f3d2e5d448006c1b8bc5"} Oct 31 05:23:44.943439 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:44.943311 5019 scope.go:122] "RemoveContainer" containerID="20d8731db07545a6fc782e0714f0852e94d6aba8428fd45eeb5fdbec87c8f77d" Oct 31 05:23:44.948008 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:44.947954 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c18017cd-ffd0-418f-b57b-66e8f0867b86-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c18017cd-ffd0-418f-b57b-66e8f0867b86-tmp-dir\") pod \"c18017cd-ffd0-418f-b57b-66e8f0867b86\" (UID: \"c18017cd-ffd0-418f-b57b-66e8f0867b86\") " Oct 31 05:23:44.948181 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:44.948040 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c18017cd-ffd0-418f-b57b-66e8f0867b86-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c18017cd-ffd0-418f-b57b-66e8f0867b86-model-cache\") pod \"c18017cd-ffd0-418f-b57b-66e8f0867b86\" (UID: \"c18017cd-ffd0-418f-b57b-66e8f0867b86\") " Oct 31 05:23:44.948181 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:44.948108 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c18017cd-ffd0-418f-b57b-66e8f0867b86-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c18017cd-ffd0-418f-b57b-66e8f0867b86-dshm\") pod \"c18017cd-ffd0-418f-b57b-66e8f0867b86\" (UID: \"c18017cd-ffd0-418f-b57b-66e8f0867b86\") " Oct 31 05:23:44.948181 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:44.948140 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c18017cd-ffd0-418f-b57b-66e8f0867b86-home\" (UniqueName: \"kubernetes.io/empty-dir/c18017cd-ffd0-418f-b57b-66e8f0867b86-home\") pod \"c18017cd-ffd0-418f-b57b-66e8f0867b86\" (UID: \"c18017cd-ffd0-418f-b57b-66e8f0867b86\") " Oct 31 05:23:44.948400 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:44.948220 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c18017cd-ffd0-418f-b57b-66e8f0867b86-kube-api-access-g5prm\" (UniqueName: \"kubernetes.io/projected/c18017cd-ffd0-418f-b57b-66e8f0867b86-kube-api-access-g5prm\") pod \"c18017cd-ffd0-418f-b57b-66e8f0867b86\" (UID: \"c18017cd-ffd0-418f-b57b-66e8f0867b86\") " Oct 31 05:23:44.948400 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:44.948361 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c18017cd-ffd0-418f-b57b-66e8f0867b86-tls-certs\" (UniqueName: \"kubernetes.io/secret/c18017cd-ffd0-418f-b57b-66e8f0867b86-tls-certs\") pod \"c18017cd-ffd0-418f-b57b-66e8f0867b86\" (UID: \"c18017cd-ffd0-418f-b57b-66e8f0867b86\") " Oct 31 05:23:44.948975 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:44.948931 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c18017cd-ffd0-418f-b57b-66e8f0867b86-model-cache" pod "c18017cd-ffd0-418f-b57b-66e8f0867b86" (UID: "c18017cd-ffd0-418f-b57b-66e8f0867b86"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:23:44.948975 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:44.948950 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c18017cd-ffd0-418f-b57b-66e8f0867b86-home" pod "c18017cd-ffd0-418f-b57b-66e8f0867b86" (UID: "c18017cd-ffd0-418f-b57b-66e8f0867b86"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:23:44.949192 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:44.949169 5019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c18017cd-ffd0-418f-b57b-66e8f0867b86-model-cache\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:23:44.949311 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:44.949198 5019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c18017cd-ffd0-418f-b57b-66e8f0867b86-home\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:23:44.949435 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:44.949350 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c18017cd-ffd0-418f-b57b-66e8f0867b86-tmp-dir" pod "c18017cd-ffd0-418f-b57b-66e8f0867b86" (UID: "c18017cd-ffd0-418f-b57b-66e8f0867b86"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:23:44.951076 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:44.951047 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c18017cd-ffd0-418f-b57b-66e8f0867b86-dshm" pod "c18017cd-ffd0-418f-b57b-66e8f0867b86" (UID: "c18017cd-ffd0-418f-b57b-66e8f0867b86"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:23:44.951313 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:44.951293 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c18017cd-ffd0-418f-b57b-66e8f0867b86-tls-certs" pod "c18017cd-ffd0-418f-b57b-66e8f0867b86" (UID: "c18017cd-ffd0-418f-b57b-66e8f0867b86"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:23:44.951730 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:44.951692 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c18017cd-ffd0-418f-b57b-66e8f0867b86-kube-api-access-g5prm" pod "c18017cd-ffd0-418f-b57b-66e8f0867b86" (UID: "c18017cd-ffd0-418f-b57b-66e8f0867b86"). InnerVolumeSpecName "kube-api-access-g5prm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:23:44.955368 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:44.955298 5019 scope.go:122] "RemoveContainer" containerID="20d8731db07545a6fc782e0714f0852e94d6aba8428fd45eeb5fdbec87c8f77d" Oct 31 05:23:44.956014 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:23:44.955978 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20d8731db07545a6fc782e0714f0852e94d6aba8428fd45eeb5fdbec87c8f77d\": container with ID starting with 20d8731db07545a6fc782e0714f0852e94d6aba8428fd45eeb5fdbec87c8f77d not found: ID does not exist" containerID="20d8731db07545a6fc782e0714f0852e94d6aba8428fd45eeb5fdbec87c8f77d" Oct 31 05:23:44.956119 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:44.956031 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20d8731db07545a6fc782e0714f0852e94d6aba8428fd45eeb5fdbec87c8f77d"} err="failed to get container status \"20d8731db07545a6fc782e0714f0852e94d6aba8428fd45eeb5fdbec87c8f77d\": rpc error: code = NotFound desc = could not find container \"20d8731db07545a6fc782e0714f0852e94d6aba8428fd45eeb5fdbec87c8f77d\": container with ID starting with 20d8731db07545a6fc782e0714f0852e94d6aba8428fd45eeb5fdbec87c8f77d not found: ID does not exist" Oct 31 05:23:45.050675 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:45.050628 5019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c18017cd-ffd0-418f-b57b-66e8f0867b86-tls-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:23:45.050675 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:45.050662 5019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c18017cd-ffd0-418f-b57b-66e8f0867b86-tmp-dir\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:23:45.050675 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:45.050670 5019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c18017cd-ffd0-418f-b57b-66e8f0867b86-dshm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:23:45.050675 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:45.050679 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5prm\" (UniqueName: \"kubernetes.io/projected/c18017cd-ffd0-418f-b57b-66e8f0867b86-kube-api-access-g5prm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:23:45.290654 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:45.290608 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-5534f0/simulator-single-main-5534f0-kserve-6ff498fcc7-ffwlt"] Oct 31 05:23:45.294513 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:45.294473 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-5534f0/simulator-single-main-5534f0-kserve-6ff498fcc7-ffwlt"] Oct 31 05:23:45.407013 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:45.406973 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c18017cd-ffd0-418f-b57b-66e8f0867b86" path="/var/lib/kubelet/pods/c18017cd-ffd0-418f-b57b-66e8f0867b86/volumes" Oct 31 05:23:49.386255 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:49.386208 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5534f0-869ff6db452mp4" podUID="7683a47b-f45f-4bd7-a52b-65375ee7fdf8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:23:50.393599 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:50.393549 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7ltrbs" podUID="8fc184d7-c471-4093-981f-05b01cc441a5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:23:52.391293 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:23:52.391250 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:23:56.894787 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:56.894752 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5534f0-869ff6db452mp4" Oct 31 05:23:56.932433 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:56.932399 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7ltrbs" Oct 31 05:23:56.973427 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:56.973376 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7683a47b-f45f-4bd7-a52b-65375ee7fdf8-kube-api-access-lhmp6\" (UniqueName: \"kubernetes.io/projected/7683a47b-f45f-4bd7-a52b-65375ee7fdf8-kube-api-access-lhmp6\") pod \"7683a47b-f45f-4bd7-a52b-65375ee7fdf8\" (UID: \"7683a47b-f45f-4bd7-a52b-65375ee7fdf8\") " Oct 31 05:23:56.973643 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:56.973437 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7683a47b-f45f-4bd7-a52b-65375ee7fdf8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7683a47b-f45f-4bd7-a52b-65375ee7fdf8-plugins-config-volume\") pod \"7683a47b-f45f-4bd7-a52b-65375ee7fdf8\" (UID: \"7683a47b-f45f-4bd7-a52b-65375ee7fdf8\") " Oct 31 05:23:56.973869 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:56.973845 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7683a47b-f45f-4bd7-a52b-65375ee7fdf8-plugins-config-volume" pod "7683a47b-f45f-4bd7-a52b-65375ee7fdf8" (UID: "7683a47b-f45f-4bd7-a52b-65375ee7fdf8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:23:56.975793 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:56.975767 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7683a47b-f45f-4bd7-a52b-65375ee7fdf8-kube-api-access-lhmp6" pod "7683a47b-f45f-4bd7-a52b-65375ee7fdf8" (UID: "7683a47b-f45f-4bd7-a52b-65375ee7fdf8"). InnerVolumeSpecName "kube-api-access-lhmp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:23:57.023525 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:57.023480 5019 generic.go:356] "Generic (PLEG): container finished" podID="7683a47b-f45f-4bd7-a52b-65375ee7fdf8" containerID="29f14f4559106932aa5d04412bd76be3fdcfec8ede970860f24e66d2b0dd46a1" exitCode=0 Oct 31 05:23:57.023752 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:57.023542 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5534f0-869ff6db452mp4" Oct 31 05:23:57.023752 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:57.023555 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5534f0-869ff6db452mp4" event={"ID":"7683a47b-f45f-4bd7-a52b-65375ee7fdf8","Type":"ContainerDied","Data":"29f14f4559106932aa5d04412bd76be3fdcfec8ede970860f24e66d2b0dd46a1"} Oct 31 05:23:57.023752 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:57.023638 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5534f0-869ff6db452mp4" event={"ID":"7683a47b-f45f-4bd7-a52b-65375ee7fdf8","Type":"ContainerDied","Data":"c69bcb4508ff1f2e69ccceff86e542834f2da518593db92902eda62785c621cb"} Oct 31 05:23:57.023752 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:57.023662 5019 scope.go:122] "RemoveContainer" containerID="29f14f4559106932aa5d04412bd76be3fdcfec8ede970860f24e66d2b0dd46a1" Oct 31 05:23:57.025233 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:57.025205 5019 generic.go:356] "Generic (PLEG): container finished" podID="8fc184d7-c471-4093-981f-05b01cc441a5" containerID="b3719c8b8ff548be0d9bbb9cd0fdea6b1b8027e4a4a5a37411592750dda5a0a2" exitCode=0 Oct 31 05:23:57.025393 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:57.025265 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7ltrbs" Oct 31 05:23:57.025393 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:57.025288 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7ltrbs" event={"ID":"8fc184d7-c471-4093-981f-05b01cc441a5","Type":"ContainerDied","Data":"b3719c8b8ff548be0d9bbb9cd0fdea6b1b8027e4a4a5a37411592750dda5a0a2"} Oct 31 05:23:57.025547 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:57.025431 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7ltrbs" event={"ID":"8fc184d7-c471-4093-981f-05b01cc441a5","Type":"ContainerDied","Data":"8f4eacf1d3d11e5d742c798ad95693e301abf53d188a874df8f3aa3c204213c3"} Oct 31 05:23:57.035342 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:57.035301 5019 scope.go:122] "RemoveContainer" containerID="29f14f4559106932aa5d04412bd76be3fdcfec8ede970860f24e66d2b0dd46a1" Oct 31 05:23:57.035712 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:23:57.035691 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29f14f4559106932aa5d04412bd76be3fdcfec8ede970860f24e66d2b0dd46a1\": container with ID starting with 29f14f4559106932aa5d04412bd76be3fdcfec8ede970860f24e66d2b0dd46a1 not found: ID does not exist" containerID="29f14f4559106932aa5d04412bd76be3fdcfec8ede970860f24e66d2b0dd46a1" Oct 31 05:23:57.035769 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:57.035722 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29f14f4559106932aa5d04412bd76be3fdcfec8ede970860f24e66d2b0dd46a1"} err="failed to get container status \"29f14f4559106932aa5d04412bd76be3fdcfec8ede970860f24e66d2b0dd46a1\": rpc error: code = NotFound desc = could not find container \"29f14f4559106932aa5d04412bd76be3fdcfec8ede970860f24e66d2b0dd46a1\": container with ID starting with 29f14f4559106932aa5d04412bd76be3fdcfec8ede970860f24e66d2b0dd46a1 not found: ID does not exist" Oct 31 05:23:57.035769 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:57.035741 5019 scope.go:122] "RemoveContainer" containerID="b3719c8b8ff548be0d9bbb9cd0fdea6b1b8027e4a4a5a37411592750dda5a0a2" Oct 31 05:23:57.045518 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:57.045490 5019 scope.go:122] "RemoveContainer" containerID="b3719c8b8ff548be0d9bbb9cd0fdea6b1b8027e4a4a5a37411592750dda5a0a2" Oct 31 05:23:57.045891 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:23:57.045863 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3719c8b8ff548be0d9bbb9cd0fdea6b1b8027e4a4a5a37411592750dda5a0a2\": container with ID starting with b3719c8b8ff548be0d9bbb9cd0fdea6b1b8027e4a4a5a37411592750dda5a0a2 not found: ID does not exist" containerID="b3719c8b8ff548be0d9bbb9cd0fdea6b1b8027e4a4a5a37411592750dda5a0a2" Oct 31 05:23:57.045956 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:57.045903 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3719c8b8ff548be0d9bbb9cd0fdea6b1b8027e4a4a5a37411592750dda5a0a2"} err="failed to get container status \"b3719c8b8ff548be0d9bbb9cd0fdea6b1b8027e4a4a5a37411592750dda5a0a2\": rpc error: code = NotFound desc = could not find container \"b3719c8b8ff548be0d9bbb9cd0fdea6b1b8027e4a4a5a37411592750dda5a0a2\": container with ID starting with b3719c8b8ff548be0d9bbb9cd0fdea6b1b8027e4a4a5a37411592750dda5a0a2 not found: ID does not exist" Oct 31 05:23:57.074931 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:57.074883 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fc184d7-c471-4093-981f-05b01cc441a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fc184d7-c471-4093-981f-05b01cc441a5-plugins-config-volume\") pod \"8fc184d7-c471-4093-981f-05b01cc441a5\" (UID: \"8fc184d7-c471-4093-981f-05b01cc441a5\") " Oct 31 05:23:57.075124 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:57.075048 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fc184d7-c471-4093-981f-05b01cc441a5-kube-api-access-lft9t\" (UniqueName: \"kubernetes.io/projected/8fc184d7-c471-4093-981f-05b01cc441a5-kube-api-access-lft9t\") pod \"8fc184d7-c471-4093-981f-05b01cc441a5\" (UID: \"8fc184d7-c471-4093-981f-05b01cc441a5\") " Oct 31 05:23:57.075250 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:57.075237 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhmp6\" (UniqueName: \"kubernetes.io/projected/7683a47b-f45f-4bd7-a52b-65375ee7fdf8-kube-api-access-lhmp6\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:23:57.075323 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:57.075251 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7683a47b-f45f-4bd7-a52b-65375ee7fdf8-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:23:57.075323 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:57.075295 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fc184d7-c471-4093-981f-05b01cc441a5-plugins-config-volume" pod "8fc184d7-c471-4093-981f-05b01cc441a5" (UID: "8fc184d7-c471-4093-981f-05b01cc441a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:23:57.077377 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:57.077347 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fc184d7-c471-4093-981f-05b01cc441a5-kube-api-access-lft9t" pod "8fc184d7-c471-4093-981f-05b01cc441a5" (UID: "8fc184d7-c471-4093-981f-05b01cc441a5"). InnerVolumeSpecName "kube-api-access-lft9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:23:57.090260 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:57.090216 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-5534f0-869ff6db452mp4"] Oct 31 05:23:57.094270 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:57.094230 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-5534f0-869ff6db452mp4"] Oct 31 05:23:57.176491 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:57.176394 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lft9t\" (UniqueName: \"kubernetes.io/projected/8fc184d7-c471-4093-981f-05b01cc441a5-kube-api-access-lft9t\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:23:57.176491 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:57.176430 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fc184d7-c471-4093-981f-05b01cc441a5-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:23:57.371828 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:57.371781 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7ltrbs"] Oct 31 05:23:57.377432 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:57.377393 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-5534f0-cfbf7ltrbs"] Oct 31 05:23:57.407792 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:57.407747 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7683a47b-f45f-4bd7-a52b-65375ee7fdf8" path="/var/lib/kubelet/pods/7683a47b-f45f-4bd7-a52b-65375ee7fdf8/volumes" Oct 31 05:23:57.408239 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:23:57.408224 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fc184d7-c471-4093-981f-05b01cc441a5" path="/var/lib/kubelet/pods/8fc184d7-c471-4093-981f-05b01cc441a5/volumes" Oct 31 05:24:06.195959 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:06.195918 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-5534f0-869ff6db4pxdqx"] Oct 31 05:24:06.196641 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:06.196609 5019 cpu_manager.go:414] "containerMap: removing container" podUID="c18017cd-ffd0-418f-b57b-66e8f0867b86" containerName="main" Oct 31 05:24:06.196641 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:06.196633 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c18017cd-ffd0-418f-b57b-66e8f0867b86" containerName="main" Oct 31 05:24:06.196763 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:06.196667 5019 cpu_manager.go:414] "containerMap: removing container" podUID="7683a47b-f45f-4bd7-a52b-65375ee7fdf8" containerName="payload-processing" Oct 31 05:24:06.196763 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:06.196677 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7683a47b-f45f-4bd7-a52b-65375ee7fdf8" containerName="payload-processing" Oct 31 05:24:06.196763 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:06.196688 5019 cpu_manager.go:414] "containerMap: removing container" podUID="d6b22fd0-53dd-4de4-9cc5-272f6abc02ab" containerName="revoke-keys" Oct 31 05:24:06.196763 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:06.196697 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6b22fd0-53dd-4de4-9cc5-272f6abc02ab" containerName="revoke-keys" Oct 31 05:24:06.196763 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:06.196713 5019 cpu_manager.go:414] "containerMap: removing container" podUID="8fc184d7-c471-4093-981f-05b01cc441a5" containerName="payload-pre-processing" Oct 31 05:24:06.196763 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:06.196721 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fc184d7-c471-4093-981f-05b01cc441a5" containerName="payload-pre-processing" Oct 31 05:24:06.197093 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:06.196881 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="c18017cd-ffd0-418f-b57b-66e8f0867b86" containerName="main" Oct 31 05:24:06.197093 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:06.196898 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="7683a47b-f45f-4bd7-a52b-65375ee7fdf8" containerName="payload-processing" Oct 31 05:24:06.197093 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:06.196908 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6b22fd0-53dd-4de4-9cc5-272f6abc02ab" containerName="revoke-keys" Oct 31 05:24:06.197093 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:06.196918 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fc184d7-c471-4093-981f-05b01cc441a5" containerName="payload-pre-processing" Oct 31 05:24:06.202008 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:06.201978 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5534f0-869ff6db4pxdqx" Oct 31 05:24:06.221826 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:06.221798 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-5534f0-869ff6db4pxdqx"] Oct 31 05:24:06.230522 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:06.230491 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-5534f0\"" Oct 31 05:24:06.370811 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:06.370758 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmpwk\" (UniqueName: \"kubernetes.io/projected/d3f715ad-0733-4333-95a2-c9583be926ea-kube-api-access-xmpwk\") pod \"payload-processing-e2e-worker-single-main-5534f0-869ff6db4pxdqx\" (UID: \"d3f715ad-0733-4333-95a2-c9583be926ea\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-5534f0-869ff6db4pxdqx" Oct 31 05:24:06.371045 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:06.370995 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3f715ad-0733-4333-95a2-c9583be926ea-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-5534f0-869ff6db4pxdqx\" (UID: \"d3f715ad-0733-4333-95a2-c9583be926ea\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-5534f0-869ff6db4pxdqx" Oct 31 05:24:06.472410 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:06.472374 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmpwk\" (UniqueName: \"kubernetes.io/projected/d3f715ad-0733-4333-95a2-c9583be926ea-kube-api-access-xmpwk\") pod \"payload-processing-e2e-worker-single-main-5534f0-869ff6db4pxdqx\" (UID: \"d3f715ad-0733-4333-95a2-c9583be926ea\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-5534f0-869ff6db4pxdqx" Oct 31 05:24:06.472595 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:06.472554 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3f715ad-0733-4333-95a2-c9583be926ea-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-5534f0-869ff6db4pxdqx\" (UID: \"d3f715ad-0733-4333-95a2-c9583be926ea\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-5534f0-869ff6db4pxdqx" Oct 31 05:24:06.473301 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:06.473279 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3f715ad-0733-4333-95a2-c9583be926ea-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-5534f0-869ff6db4pxdqx\" (UID: \"d3f715ad-0733-4333-95a2-c9583be926ea\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-5534f0-869ff6db4pxdqx" Oct 31 05:24:06.496435 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:06.496404 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmpwk\" (UniqueName: \"kubernetes.io/projected/d3f715ad-0733-4333-95a2-c9583be926ea-kube-api-access-xmpwk\") pod \"payload-processing-e2e-worker-single-main-5534f0-869ff6db4pxdqx\" (UID: \"d3f715ad-0733-4333-95a2-c9583be926ea\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-5534f0-869ff6db4pxdqx" Oct 31 05:24:06.516370 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:06.516326 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5534f0-869ff6db4pxdqx" Oct 31 05:24:06.826373 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:06.821976 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-5534f0-869ff6db4pxdqx"] Oct 31 05:24:07.102772 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:07.102663 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5534f0-869ff6db4pxdqx" event={"ID":"d3f715ad-0733-4333-95a2-c9583be926ea","Type":"ContainerStarted","Data":"e0fee491a3a3e17700071f31139b424548f68fa22e6da918066d1dca747dd0f8"} Oct 31 05:24:07.102772 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:07.102730 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5534f0-869ff6db4pxdqx" event={"ID":"d3f715ad-0733-4333-95a2-c9583be926ea","Type":"ContainerStarted","Data":"edc2edaff7bf585ec5856c6e9599dd3394ea75d946632caee84256fec1ca494c"} Oct 31 05:24:07.102772 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:07.102761 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5534f0-869ff6db4pxdqx" Oct 31 05:24:07.144975 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:07.144908 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5534f0-869ff6db4pxdqx" podStartSLOduration=1.144889426 podStartE2EDuration="1.144889426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:24:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:24:07.142685393 +0000 UTC m=+4801.899023432" watchObservedRunningTime="2026-10-31 05:24:07.144889426 +0000 UTC m=+4801.901227496" Oct 31 05:24:09.747893 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:09.747858 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s88kp_3945ff65-e877-4372-9334-3968c68b7295/console-operator/2.log" Oct 31 05:24:09.750065 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:09.750039 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s88kp_3945ff65-e877-4372-9334-3968c68b7295/console-operator/2.log" Oct 31 05:24:14.602963 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:14.602924 5019 scope.go:122] "RemoveContainer" containerID="d3e702c64d7136bb34bd11248abe70a873750cf183bff1773f931f2889acccef" Oct 31 05:24:14.613644 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:14.613619 5019 scope.go:122] "RemoveContainer" containerID="b3ebec688d49a5fa25b0ae608e8af2df2e36f41d656f97e7f457107616642b94" Oct 31 05:24:14.623999 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:14.623971 5019 scope.go:122] "RemoveContainer" containerID="f7bed4ebead583af337a6da70612c913771d4709dae8cc46fbc52699c060f8a1" Oct 31 05:24:14.634233 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:14.634201 5019 scope.go:122] "RemoveContainer" containerID="51ca0db5dff716a825c66991249c0ce4ee8b6b498297f34bb60032fcb0d9f45a" Oct 31 05:24:18.112591 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:18.112505 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5534f0-869ff6db4pxdqx" Oct 31 05:24:19.027263 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:19.027224 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d6zhg"] Oct 31 05:24:19.039179 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:19.039138 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d6zhg"] Oct 31 05:24:19.406952 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:19.406849 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6c8a271-23e4-4b2c-92ad-a396721263d8" path="/var/lib/kubelet/pods/e6c8a271-23e4-4b2c-92ad-a396721263d8/volumes" Oct 31 05:24:22.391653 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:24:22.391609 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:24:24.579984 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:24.579942 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-5534f0-869ff6db4pxdqx"] Oct 31 05:24:26.792594 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:26.792556 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k"] Oct 31 05:24:26.793097 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:26.792864 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k" podUID="5e010ac6-7566-49bc-ae3d-02f791df20c7" containerName="istio-proxy" containerID="cri-o://0ee93aa4abf8f3f476323333c29e210d8c8054f32e8001180bc6b5fa30582c94" gracePeriod=30 Oct 31 05:24:29.580479 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:29.580422 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5534f0-869ff6db4pxdqx" podUID="d3f715ad-0733-4333-95a2-c9583be926ea" containerName="payload-processing" containerID="cri-o://e0fee491a3a3e17700071f31139b424548f68fa22e6da918066d1dca747dd0f8" gracePeriod=25 Oct 31 05:24:29.849859 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:29.849780 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5534f0-869ff6db4pxdqx" Oct 31 05:24:29.931435 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:29.931395 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3f715ad-0733-4333-95a2-c9583be926ea-kube-api-access-xmpwk\" (UniqueName: \"kubernetes.io/projected/d3f715ad-0733-4333-95a2-c9583be926ea-kube-api-access-xmpwk\") pod \"d3f715ad-0733-4333-95a2-c9583be926ea\" (UID: \"d3f715ad-0733-4333-95a2-c9583be926ea\") " Oct 31 05:24:29.931655 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:29.931521 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3f715ad-0733-4333-95a2-c9583be926ea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3f715ad-0733-4333-95a2-c9583be926ea-plugins-config-volume\") pod \"d3f715ad-0733-4333-95a2-c9583be926ea\" (UID: \"d3f715ad-0733-4333-95a2-c9583be926ea\") " Oct 31 05:24:29.931997 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:29.931963 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3f715ad-0733-4333-95a2-c9583be926ea-plugins-config-volume" pod "d3f715ad-0733-4333-95a2-c9583be926ea" (UID: "d3f715ad-0733-4333-95a2-c9583be926ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:24:29.933742 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:29.933712 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3f715ad-0733-4333-95a2-c9583be926ea-kube-api-access-xmpwk" pod "d3f715ad-0733-4333-95a2-c9583be926ea" (UID: "d3f715ad-0733-4333-95a2-c9583be926ea"). InnerVolumeSpecName "kube-api-access-xmpwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:24:30.033305 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:30.033261 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmpwk\" (UniqueName: \"kubernetes.io/projected/d3f715ad-0733-4333-95a2-c9583be926ea-kube-api-access-xmpwk\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:24:30.033305 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:30.033301 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3f715ad-0733-4333-95a2-c9583be926ea-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:24:30.276318 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:30.276274 5019 generic.go:356] "Generic (PLEG): container finished" podID="d3f715ad-0733-4333-95a2-c9583be926ea" containerID="e0fee491a3a3e17700071f31139b424548f68fa22e6da918066d1dca747dd0f8" exitCode=0 Oct 31 05:24:30.276318 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:30.276355 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5534f0-869ff6db4pxdqx" Oct 31 05:24:30.276615 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:30.276355 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5534f0-869ff6db4pxdqx" event={"ID":"d3f715ad-0733-4333-95a2-c9583be926ea","Type":"ContainerDied","Data":"e0fee491a3a3e17700071f31139b424548f68fa22e6da918066d1dca747dd0f8"} Oct 31 05:24:30.276615 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:30.276429 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5534f0-869ff6db4pxdqx" event={"ID":"d3f715ad-0733-4333-95a2-c9583be926ea","Type":"ContainerDied","Data":"edc2edaff7bf585ec5856c6e9599dd3394ea75d946632caee84256fec1ca494c"} Oct 31 05:24:30.276615 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:30.276456 5019 scope.go:122] "RemoveContainer" containerID="e0fee491a3a3e17700071f31139b424548f68fa22e6da918066d1dca747dd0f8" Oct 31 05:24:30.287257 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:30.287234 5019 scope.go:122] "RemoveContainer" containerID="e0fee491a3a3e17700071f31139b424548f68fa22e6da918066d1dca747dd0f8" Oct 31 05:24:30.287753 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:24:30.287724 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0fee491a3a3e17700071f31139b424548f68fa22e6da918066d1dca747dd0f8\": container with ID starting with e0fee491a3a3e17700071f31139b424548f68fa22e6da918066d1dca747dd0f8 not found: ID does not exist" containerID="e0fee491a3a3e17700071f31139b424548f68fa22e6da918066d1dca747dd0f8" Oct 31 05:24:30.287845 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:30.287763 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0fee491a3a3e17700071f31139b424548f68fa22e6da918066d1dca747dd0f8"} err="failed to get container status \"e0fee491a3a3e17700071f31139b424548f68fa22e6da918066d1dca747dd0f8\": rpc error: code = NotFound desc = could not find container \"e0fee491a3a3e17700071f31139b424548f68fa22e6da918066d1dca747dd0f8\": container with ID starting with e0fee491a3a3e17700071f31139b424548f68fa22e6da918066d1dca747dd0f8 not found: ID does not exist" Oct 31 05:24:30.328351 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:30.328294 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-5534f0-869ff6db4pxdqx"] Oct 31 05:24:30.340605 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:30.340563 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-5534f0-869ff6db4pxdqx"] Oct 31 05:24:31.408287 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:31.408247 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3f715ad-0733-4333-95a2-c9583be926ea" path="/var/lib/kubelet/pods/d3f715ad-0733-4333-95a2-c9583be926ea/volumes" Oct 31 05:24:31.705523 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:31.705412 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-784e0608-kserve-7cdf47df77-sn6vk"] Oct 31 05:24:31.706237 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:31.706205 5019 cpu_manager.go:414] "containerMap: removing container" podUID="d3f715ad-0733-4333-95a2-c9583be926ea" containerName="payload-processing" Oct 31 05:24:31.706424 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:31.706239 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3f715ad-0733-4333-95a2-c9583be926ea" containerName="payload-processing" Oct 31 05:24:31.706535 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:31.706477 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3f715ad-0733-4333-95a2-c9583be926ea" containerName="payload-processing" Oct 31 05:24:31.712303 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:31.712256 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-784e0608-kserve-7cdf47df77-sn6vk" Oct 31 05:24:31.730109 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:31.730073 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-784e0608-kserve-7cdf47df77-sn6vk"] Oct 31 05:24:31.738321 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:31.738270 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-784e0608-kserve-self-signed-certs\"" Oct 31 05:24:31.853781 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:31.853736 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/248a0e94-8fa1-4f4c-af6e-28c8035d1e73-tls-certs\") pod \"e2e-same-id-a-784e0608-kserve-7cdf47df77-sn6vk\" (UID: \"248a0e94-8fa1-4f4c-af6e-28c8035d1e73\") " pod="llm/e2e-same-id-a-784e0608-kserve-7cdf47df77-sn6vk" Oct 31 05:24:31.854003 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:31.853880 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/248a0e94-8fa1-4f4c-af6e-28c8035d1e73-dshm\") pod \"e2e-same-id-a-784e0608-kserve-7cdf47df77-sn6vk\" (UID: \"248a0e94-8fa1-4f4c-af6e-28c8035d1e73\") " pod="llm/e2e-same-id-a-784e0608-kserve-7cdf47df77-sn6vk" Oct 31 05:24:31.854003 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:31.853932 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxm7n\" (UniqueName: \"kubernetes.io/projected/248a0e94-8fa1-4f4c-af6e-28c8035d1e73-kube-api-access-wxm7n\") pod \"e2e-same-id-a-784e0608-kserve-7cdf47df77-sn6vk\" (UID: \"248a0e94-8fa1-4f4c-af6e-28c8035d1e73\") " pod="llm/e2e-same-id-a-784e0608-kserve-7cdf47df77-sn6vk" Oct 31 05:24:31.854003 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:31.853963 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/248a0e94-8fa1-4f4c-af6e-28c8035d1e73-home\") pod \"e2e-same-id-a-784e0608-kserve-7cdf47df77-sn6vk\" (UID: \"248a0e94-8fa1-4f4c-af6e-28c8035d1e73\") " pod="llm/e2e-same-id-a-784e0608-kserve-7cdf47df77-sn6vk" Oct 31 05:24:31.854122 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:31.854089 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/248a0e94-8fa1-4f4c-af6e-28c8035d1e73-tmp-dir\") pod \"e2e-same-id-a-784e0608-kserve-7cdf47df77-sn6vk\" (UID: \"248a0e94-8fa1-4f4c-af6e-28c8035d1e73\") " pod="llm/e2e-same-id-a-784e0608-kserve-7cdf47df77-sn6vk" Oct 31 05:24:31.854122 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:31.854109 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/248a0e94-8fa1-4f4c-af6e-28c8035d1e73-model-cache\") pod \"e2e-same-id-a-784e0608-kserve-7cdf47df77-sn6vk\" (UID: \"248a0e94-8fa1-4f4c-af6e-28c8035d1e73\") " pod="llm/e2e-same-id-a-784e0608-kserve-7cdf47df77-sn6vk" Oct 31 05:24:31.954964 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:31.954925 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/248a0e94-8fa1-4f4c-af6e-28c8035d1e73-tmp-dir\") pod \"e2e-same-id-a-784e0608-kserve-7cdf47df77-sn6vk\" (UID: \"248a0e94-8fa1-4f4c-af6e-28c8035d1e73\") " pod="llm/e2e-same-id-a-784e0608-kserve-7cdf47df77-sn6vk" Oct 31 05:24:31.954964 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:31.954968 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/248a0e94-8fa1-4f4c-af6e-28c8035d1e73-model-cache\") pod \"e2e-same-id-a-784e0608-kserve-7cdf47df77-sn6vk\" (UID: \"248a0e94-8fa1-4f4c-af6e-28c8035d1e73\") " pod="llm/e2e-same-id-a-784e0608-kserve-7cdf47df77-sn6vk" Oct 31 05:24:31.955273 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:31.954994 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/248a0e94-8fa1-4f4c-af6e-28c8035d1e73-tls-certs\") pod \"e2e-same-id-a-784e0608-kserve-7cdf47df77-sn6vk\" (UID: \"248a0e94-8fa1-4f4c-af6e-28c8035d1e73\") " pod="llm/e2e-same-id-a-784e0608-kserve-7cdf47df77-sn6vk" Oct 31 05:24:31.955273 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:31.955040 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/248a0e94-8fa1-4f4c-af6e-28c8035d1e73-dshm\") pod \"e2e-same-id-a-784e0608-kserve-7cdf47df77-sn6vk\" (UID: \"248a0e94-8fa1-4f4c-af6e-28c8035d1e73\") " pod="llm/e2e-same-id-a-784e0608-kserve-7cdf47df77-sn6vk" Oct 31 05:24:31.955273 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:31.955061 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxm7n\" (UniqueName: \"kubernetes.io/projected/248a0e94-8fa1-4f4c-af6e-28c8035d1e73-kube-api-access-wxm7n\") pod \"e2e-same-id-a-784e0608-kserve-7cdf47df77-sn6vk\" (UID: \"248a0e94-8fa1-4f4c-af6e-28c8035d1e73\") " pod="llm/e2e-same-id-a-784e0608-kserve-7cdf47df77-sn6vk" Oct 31 05:24:31.955273 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:31.955083 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/248a0e94-8fa1-4f4c-af6e-28c8035d1e73-home\") pod \"e2e-same-id-a-784e0608-kserve-7cdf47df77-sn6vk\" (UID: \"248a0e94-8fa1-4f4c-af6e-28c8035d1e73\") " pod="llm/e2e-same-id-a-784e0608-kserve-7cdf47df77-sn6vk" Oct 31 05:24:31.955566 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:31.955358 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/248a0e94-8fa1-4f4c-af6e-28c8035d1e73-model-cache\") pod \"e2e-same-id-a-784e0608-kserve-7cdf47df77-sn6vk\" (UID: \"248a0e94-8fa1-4f4c-af6e-28c8035d1e73\") " pod="llm/e2e-same-id-a-784e0608-kserve-7cdf47df77-sn6vk" Oct 31 05:24:31.955566 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:31.955450 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/248a0e94-8fa1-4f4c-af6e-28c8035d1e73-tmp-dir\") pod \"e2e-same-id-a-784e0608-kserve-7cdf47df77-sn6vk\" (UID: \"248a0e94-8fa1-4f4c-af6e-28c8035d1e73\") " pod="llm/e2e-same-id-a-784e0608-kserve-7cdf47df77-sn6vk" Oct 31 05:24:31.955657 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:31.955596 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/248a0e94-8fa1-4f4c-af6e-28c8035d1e73-home\") pod \"e2e-same-id-a-784e0608-kserve-7cdf47df77-sn6vk\" (UID: \"248a0e94-8fa1-4f4c-af6e-28c8035d1e73\") " pod="llm/e2e-same-id-a-784e0608-kserve-7cdf47df77-sn6vk" Oct 31 05:24:31.957802 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:31.957780 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/248a0e94-8fa1-4f4c-af6e-28c8035d1e73-tls-certs\") pod \"e2e-same-id-a-784e0608-kserve-7cdf47df77-sn6vk\" (UID: \"248a0e94-8fa1-4f4c-af6e-28c8035d1e73\") " pod="llm/e2e-same-id-a-784e0608-kserve-7cdf47df77-sn6vk" Oct 31 05:24:31.958551 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:31.958529 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/248a0e94-8fa1-4f4c-af6e-28c8035d1e73-dshm\") pod \"e2e-same-id-a-784e0608-kserve-7cdf47df77-sn6vk\" (UID: \"248a0e94-8fa1-4f4c-af6e-28c8035d1e73\") " pod="llm/e2e-same-id-a-784e0608-kserve-7cdf47df77-sn6vk" Oct 31 05:24:31.983236 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:31.983134 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxm7n\" (UniqueName: \"kubernetes.io/projected/248a0e94-8fa1-4f4c-af6e-28c8035d1e73-kube-api-access-wxm7n\") pod \"e2e-same-id-a-784e0608-kserve-7cdf47df77-sn6vk\" (UID: \"248a0e94-8fa1-4f4c-af6e-28c8035d1e73\") " pod="llm/e2e-same-id-a-784e0608-kserve-7cdf47df77-sn6vk" Oct 31 05:24:32.030564 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:32.030291 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-784e0608-kserve-7cdf47df77-sn6vk" Oct 31 05:24:32.059311 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:32.059281 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k" Oct 31 05:24:32.165829 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:32.165771 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e010ac6-7566-49bc-ae3d-02f791df20c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e010ac6-7566-49bc-ae3d-02f791df20c7-workload-socket\") pod \"5e010ac6-7566-49bc-ae3d-02f791df20c7\" (UID: \"5e010ac6-7566-49bc-ae3d-02f791df20c7\") " Oct 31 05:24:32.166083 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:32.165866 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e010ac6-7566-49bc-ae3d-02f791df20c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e010ac6-7566-49bc-ae3d-02f791df20c7-credential-socket\") pod \"5e010ac6-7566-49bc-ae3d-02f791df20c7\" (UID: \"5e010ac6-7566-49bc-ae3d-02f791df20c7\") " Oct 31 05:24:32.166083 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:32.165899 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e010ac6-7566-49bc-ae3d-02f791df20c7-istio-token\" (UniqueName: \"kubernetes.io/projected/5e010ac6-7566-49bc-ae3d-02f791df20c7-istio-token\") pod \"5e010ac6-7566-49bc-ae3d-02f791df20c7\" (UID: \"5e010ac6-7566-49bc-ae3d-02f791df20c7\") " Oct 31 05:24:32.166083 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:32.166007 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e010ac6-7566-49bc-ae3d-02f791df20c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e010ac6-7566-49bc-ae3d-02f791df20c7-istio-envoy\") pod \"5e010ac6-7566-49bc-ae3d-02f791df20c7\" (UID: \"5e010ac6-7566-49bc-ae3d-02f791df20c7\") " Oct 31 05:24:32.166083 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:32.166057 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e010ac6-7566-49bc-ae3d-02f791df20c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e010ac6-7566-49bc-ae3d-02f791df20c7-istio-data\") pod \"5e010ac6-7566-49bc-ae3d-02f791df20c7\" (UID: \"5e010ac6-7566-49bc-ae3d-02f791df20c7\") " Oct 31 05:24:32.166316 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:32.166149 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e010ac6-7566-49bc-ae3d-02f791df20c7-kube-api-access-g6jn9\" (UniqueName: \"kubernetes.io/projected/5e010ac6-7566-49bc-ae3d-02f791df20c7-kube-api-access-g6jn9\") pod \"5e010ac6-7566-49bc-ae3d-02f791df20c7\" (UID: \"5e010ac6-7566-49bc-ae3d-02f791df20c7\") " Oct 31 05:24:32.166316 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:32.166204 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e010ac6-7566-49bc-ae3d-02f791df20c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e010ac6-7566-49bc-ae3d-02f791df20c7-istio-podinfo\") pod \"5e010ac6-7566-49bc-ae3d-02f791df20c7\" (UID: \"5e010ac6-7566-49bc-ae3d-02f791df20c7\") " Oct 31 05:24:32.166316 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:32.166245 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e010ac6-7566-49bc-ae3d-02f791df20c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e010ac6-7566-49bc-ae3d-02f791df20c7-istiod-ca-cert\") pod \"5e010ac6-7566-49bc-ae3d-02f791df20c7\" (UID: \"5e010ac6-7566-49bc-ae3d-02f791df20c7\") " Oct 31 05:24:32.166316 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:32.166280 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e010ac6-7566-49bc-ae3d-02f791df20c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e010ac6-7566-49bc-ae3d-02f791df20c7-workload-certs\") pod \"5e010ac6-7566-49bc-ae3d-02f791df20c7\" (UID: \"5e010ac6-7566-49bc-ae3d-02f791df20c7\") " Oct 31 05:24:32.167189 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:32.167153 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e010ac6-7566-49bc-ae3d-02f791df20c7-workload-certs" pod "5e010ac6-7566-49bc-ae3d-02f791df20c7" (UID: "5e010ac6-7566-49bc-ae3d-02f791df20c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:24:32.167469 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:32.167439 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e010ac6-7566-49bc-ae3d-02f791df20c7-workload-socket" pod "5e010ac6-7566-49bc-ae3d-02f791df20c7" (UID: "5e010ac6-7566-49bc-ae3d-02f791df20c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:24:32.167757 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:32.167730 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e010ac6-7566-49bc-ae3d-02f791df20c7-istio-data" pod "5e010ac6-7566-49bc-ae3d-02f791df20c7" (UID: "5e010ac6-7566-49bc-ae3d-02f791df20c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:24:32.167844 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:32.167770 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e010ac6-7566-49bc-ae3d-02f791df20c7-istiod-ca-cert" pod "5e010ac6-7566-49bc-ae3d-02f791df20c7" (UID: "5e010ac6-7566-49bc-ae3d-02f791df20c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:24:32.167962 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:32.167934 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e010ac6-7566-49bc-ae3d-02f791df20c7-credential-socket" pod "5e010ac6-7566-49bc-ae3d-02f791df20c7" (UID: "5e010ac6-7566-49bc-ae3d-02f791df20c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:24:32.170154 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:32.170096 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e010ac6-7566-49bc-ae3d-02f791df20c7-istio-podinfo" pod "5e010ac6-7566-49bc-ae3d-02f791df20c7" (UID: "5e010ac6-7566-49bc-ae3d-02f791df20c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 05:24:32.170154 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:32.170122 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e010ac6-7566-49bc-ae3d-02f791df20c7-kube-api-access-g6jn9" pod "5e010ac6-7566-49bc-ae3d-02f791df20c7" (UID: "5e010ac6-7566-49bc-ae3d-02f791df20c7"). InnerVolumeSpecName "kube-api-access-g6jn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:24:32.170562 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:32.170532 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e010ac6-7566-49bc-ae3d-02f791df20c7-istio-envoy" pod "5e010ac6-7566-49bc-ae3d-02f791df20c7" (UID: "5e010ac6-7566-49bc-ae3d-02f791df20c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:24:32.170654 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:32.170589 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e010ac6-7566-49bc-ae3d-02f791df20c7-istio-token" pod "5e010ac6-7566-49bc-ae3d-02f791df20c7" (UID: "5e010ac6-7566-49bc-ae3d-02f791df20c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:24:32.268148 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:32.268104 5019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e010ac6-7566-49bc-ae3d-02f791df20c7-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:24:32.268148 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:32.268141 5019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e010ac6-7566-49bc-ae3d-02f791df20c7-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:24:32.268148 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:32.268152 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6jn9\" (UniqueName: \"kubernetes.io/projected/5e010ac6-7566-49bc-ae3d-02f791df20c7-kube-api-access-g6jn9\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:24:32.268148 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:32.268162 5019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e010ac6-7566-49bc-ae3d-02f791df20c7-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:24:32.268543 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:32.268171 5019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e010ac6-7566-49bc-ae3d-02f791df20c7-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:24:32.268543 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:32.268179 5019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e010ac6-7566-49bc-ae3d-02f791df20c7-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:24:32.268543 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:32.268187 5019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e010ac6-7566-49bc-ae3d-02f791df20c7-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:24:32.268543 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:32.268195 5019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e010ac6-7566-49bc-ae3d-02f791df20c7-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:24:32.268543 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:32.268203 5019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e010ac6-7566-49bc-ae3d-02f791df20c7-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:24:32.291661 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:32.291623 5019 generic.go:356] "Generic (PLEG): container finished" podID="5e010ac6-7566-49bc-ae3d-02f791df20c7" containerID="0ee93aa4abf8f3f476323333c29e210d8c8054f32e8001180bc6b5fa30582c94" exitCode=0 Oct 31 05:24:32.291869 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:32.291706 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k" event={"ID":"5e010ac6-7566-49bc-ae3d-02f791df20c7","Type":"ContainerDied","Data":"0ee93aa4abf8f3f476323333c29e210d8c8054f32e8001180bc6b5fa30582c94"} Oct 31 05:24:32.291869 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:32.291720 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k" Oct 31 05:24:32.291869 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:32.291821 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k" event={"ID":"5e010ac6-7566-49bc-ae3d-02f791df20c7","Type":"ContainerDied","Data":"09f42140fc6b8320f8801fc288cc072adf15d6735d159935ba49518ee7c2dc15"} Oct 31 05:24:32.291869 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:32.291854 5019 scope.go:122] "RemoveContainer" containerID="0ee93aa4abf8f3f476323333c29e210d8c8054f32e8001180bc6b5fa30582c94" Oct 31 05:24:32.302246 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:32.302225 5019 scope.go:122] "RemoveContainer" containerID="ce441c25c90a50804c1005f73da04a6d623ec74b796f71c5c8dc324d77d8377b" Oct 31 05:24:32.311979 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:32.311952 5019 scope.go:122] "RemoveContainer" containerID="0ee93aa4abf8f3f476323333c29e210d8c8054f32e8001180bc6b5fa30582c94" Oct 31 05:24:32.312353 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:24:32.312313 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ee93aa4abf8f3f476323333c29e210d8c8054f32e8001180bc6b5fa30582c94\": container with ID starting with 0ee93aa4abf8f3f476323333c29e210d8c8054f32e8001180bc6b5fa30582c94 not found: ID does not exist" containerID="0ee93aa4abf8f3f476323333c29e210d8c8054f32e8001180bc6b5fa30582c94" Oct 31 05:24:32.312440 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:32.312367 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ee93aa4abf8f3f476323333c29e210d8c8054f32e8001180bc6b5fa30582c94"} err="failed to get container status \"0ee93aa4abf8f3f476323333c29e210d8c8054f32e8001180bc6b5fa30582c94\": rpc error: code = NotFound desc = could not find container \"0ee93aa4abf8f3f476323333c29e210d8c8054f32e8001180bc6b5fa30582c94\": container with ID starting with 0ee93aa4abf8f3f476323333c29e210d8c8054f32e8001180bc6b5fa30582c94 not found: ID does not exist" Oct 31 05:24:32.312440 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:32.312385 5019 scope.go:122] "RemoveContainer" containerID="ce441c25c90a50804c1005f73da04a6d623ec74b796f71c5c8dc324d77d8377b" Oct 31 05:24:32.312682 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:24:32.312659 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce441c25c90a50804c1005f73da04a6d623ec74b796f71c5c8dc324d77d8377b\": container with ID starting with ce441c25c90a50804c1005f73da04a6d623ec74b796f71c5c8dc324d77d8377b not found: ID does not exist" containerID="ce441c25c90a50804c1005f73da04a6d623ec74b796f71c5c8dc324d77d8377b" Oct 31 05:24:32.312726 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:32.312689 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce441c25c90a50804c1005f73da04a6d623ec74b796f71c5c8dc324d77d8377b"} err="failed to get container status \"ce441c25c90a50804c1005f73da04a6d623ec74b796f71c5c8dc324d77d8377b\": rpc error: code = NotFound desc = could not find container \"ce441c25c90a50804c1005f73da04a6d623ec74b796f71c5c8dc324d77d8377b\": container with ID starting with ce441c25c90a50804c1005f73da04a6d623ec74b796f71c5c8dc324d77d8377b not found: ID does not exist" Oct 31 05:24:32.339534 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:32.339260 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-784e0608-kserve-7cdf47df77-sn6vk"] Oct 31 05:24:32.342188 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:24:32.342156 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod248a0e94_8fa1_4f4c_af6e_28c8035d1e73.slice/crio-ad6ec7c386fdd88d056f51692e810c281198af140f5636a794a49b8167f63bff WatchSource:0}: Error finding container ad6ec7c386fdd88d056f51692e810c281198af140f5636a794a49b8167f63bff: Status 404 returned error can't find the container with id ad6ec7c386fdd88d056f51692e810c281198af140f5636a794a49b8167f63bff Oct 31 05:24:32.349466 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:32.349430 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k"] Oct 31 05:24:32.354230 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:32.354190 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-5534f0-openshift-default-58cd9846b5h745k"] Oct 31 05:24:33.299871 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:33.299826 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-784e0608-kserve-7cdf47df77-sn6vk" event={"ID":"248a0e94-8fa1-4f4c-af6e-28c8035d1e73","Type":"ContainerStarted","Data":"4eb3247d11cc5f2e3b8af933f0b830d6ff3c628e8e92c908e3b9c43f6f644360"} Oct 31 05:24:33.300382 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:33.299882 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-784e0608-kserve-7cdf47df77-sn6vk" event={"ID":"248a0e94-8fa1-4f4c-af6e-28c8035d1e73","Type":"ContainerStarted","Data":"ad6ec7c386fdd88d056f51692e810c281198af140f5636a794a49b8167f63bff"} Oct 31 05:24:33.344011 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:33.343952 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-784e0608-kserve-7cdf47df77-sn6vk" podStartSLOduration=2.34393676 podStartE2EDuration="2.34393676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:24:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:24:33.341565108 +0000 UTC m=+4828.097903150" watchObservedRunningTime="2026-10-31 05:24:33.34393676 +0000 UTC m=+4828.100274799" Oct 31 05:24:33.406812 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:33.406777 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e010ac6-7566-49bc-ae3d-02f791df20c7" path="/var/lib/kubelet/pods/5e010ac6-7566-49bc-ae3d-02f791df20c7/volumes" Oct 31 05:24:34.031322 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:34.031280 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-784e0608-kserve-7cdf47df77-sn6vk" Oct 31 05:24:42.031053 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:42.031013 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-784e0608-kserve-7cdf47df77-sn6vk" Oct 31 05:24:42.043530 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:42.043499 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-784e0608-kserve-7cdf47df77-sn6vk" Oct 31 05:24:42.417970 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:42.417897 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-784e0608-kserve-7cdf47df77-sn6vk" Oct 31 05:24:47.688276 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:47.688241 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-784e0608-kserve-6d98d66ff7-hz49g"] Oct 31 05:24:47.688827 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:47.688709 5019 cpu_manager.go:414] "containerMap: removing container" podUID="5e010ac6-7566-49bc-ae3d-02f791df20c7" containerName="istio-proxy" Oct 31 05:24:47.688827 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:47.688726 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e010ac6-7566-49bc-ae3d-02f791df20c7" containerName="istio-proxy" Oct 31 05:24:47.688969 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:47.688922 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e010ac6-7566-49bc-ae3d-02f791df20c7" containerName="istio-proxy" Oct 31 05:24:47.688969 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:47.688937 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e010ac6-7566-49bc-ae3d-02f791df20c7" containerName="istio-proxy" Oct 31 05:24:47.692145 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:47.692122 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-784e0608-kserve-6d98d66ff7-hz49g" Oct 31 05:24:47.707841 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:47.707815 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-784e0608-kserve-self-signed-certs\"" Oct 31 05:24:47.711376 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:47.711348 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-784e0608-kserve-6d98d66ff7-hz49g"] Oct 31 05:24:47.791889 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:47.791856 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94212018-f635-4d56-a099-b9e4f45ec632-tmp-dir\") pod \"e2e-same-id-b-784e0608-kserve-6d98d66ff7-hz49g\" (UID: \"94212018-f635-4d56-a099-b9e4f45ec632\") " pod="llm/e2e-same-id-b-784e0608-kserve-6d98d66ff7-hz49g" Oct 31 05:24:47.792021 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:47.792005 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9k94\" (UniqueName: \"kubernetes.io/projected/94212018-f635-4d56-a099-b9e4f45ec632-kube-api-access-c9k94\") pod \"e2e-same-id-b-784e0608-kserve-6d98d66ff7-hz49g\" (UID: \"94212018-f635-4d56-a099-b9e4f45ec632\") " pod="llm/e2e-same-id-b-784e0608-kserve-6d98d66ff7-hz49g" Oct 31 05:24:47.792120 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:47.792099 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94212018-f635-4d56-a099-b9e4f45ec632-tls-certs\") pod \"e2e-same-id-b-784e0608-kserve-6d98d66ff7-hz49g\" (UID: \"94212018-f635-4d56-a099-b9e4f45ec632\") " pod="llm/e2e-same-id-b-784e0608-kserve-6d98d66ff7-hz49g" Oct 31 05:24:47.792170 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:47.792138 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94212018-f635-4d56-a099-b9e4f45ec632-dshm\") pod \"e2e-same-id-b-784e0608-kserve-6d98d66ff7-hz49g\" (UID: \"94212018-f635-4d56-a099-b9e4f45ec632\") " pod="llm/e2e-same-id-b-784e0608-kserve-6d98d66ff7-hz49g" Oct 31 05:24:47.792208 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:47.792186 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94212018-f635-4d56-a099-b9e4f45ec632-model-cache\") pod \"e2e-same-id-b-784e0608-kserve-6d98d66ff7-hz49g\" (UID: \"94212018-f635-4d56-a099-b9e4f45ec632\") " pod="llm/e2e-same-id-b-784e0608-kserve-6d98d66ff7-hz49g" Oct 31 05:24:47.792323 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:47.792307 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94212018-f635-4d56-a099-b9e4f45ec632-home\") pod \"e2e-same-id-b-784e0608-kserve-6d98d66ff7-hz49g\" (UID: \"94212018-f635-4d56-a099-b9e4f45ec632\") " pod="llm/e2e-same-id-b-784e0608-kserve-6d98d66ff7-hz49g" Oct 31 05:24:47.893706 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:47.893665 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94212018-f635-4d56-a099-b9e4f45ec632-tls-certs\") pod \"e2e-same-id-b-784e0608-kserve-6d98d66ff7-hz49g\" (UID: \"94212018-f635-4d56-a099-b9e4f45ec632\") " pod="llm/e2e-same-id-b-784e0608-kserve-6d98d66ff7-hz49g" Oct 31 05:24:47.893931 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:47.893721 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94212018-f635-4d56-a099-b9e4f45ec632-dshm\") pod \"e2e-same-id-b-784e0608-kserve-6d98d66ff7-hz49g\" (UID: \"94212018-f635-4d56-a099-b9e4f45ec632\") " pod="llm/e2e-same-id-b-784e0608-kserve-6d98d66ff7-hz49g" Oct 31 05:24:47.893931 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:47.893802 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94212018-f635-4d56-a099-b9e4f45ec632-model-cache\") pod \"e2e-same-id-b-784e0608-kserve-6d98d66ff7-hz49g\" (UID: \"94212018-f635-4d56-a099-b9e4f45ec632\") " pod="llm/e2e-same-id-b-784e0608-kserve-6d98d66ff7-hz49g" Oct 31 05:24:47.893931 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:47.893871 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94212018-f635-4d56-a099-b9e4f45ec632-home\") pod \"e2e-same-id-b-784e0608-kserve-6d98d66ff7-hz49g\" (UID: \"94212018-f635-4d56-a099-b9e4f45ec632\") " pod="llm/e2e-same-id-b-784e0608-kserve-6d98d66ff7-hz49g" Oct 31 05:24:47.894092 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:47.893946 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94212018-f635-4d56-a099-b9e4f45ec632-tmp-dir\") pod \"e2e-same-id-b-784e0608-kserve-6d98d66ff7-hz49g\" (UID: \"94212018-f635-4d56-a099-b9e4f45ec632\") " pod="llm/e2e-same-id-b-784e0608-kserve-6d98d66ff7-hz49g" Oct 31 05:24:47.894092 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:47.894072 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9k94\" (UniqueName: \"kubernetes.io/projected/94212018-f635-4d56-a099-b9e4f45ec632-kube-api-access-c9k94\") pod \"e2e-same-id-b-784e0608-kserve-6d98d66ff7-hz49g\" (UID: \"94212018-f635-4d56-a099-b9e4f45ec632\") " pod="llm/e2e-same-id-b-784e0608-kserve-6d98d66ff7-hz49g" Oct 31 05:24:47.894562 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:47.894536 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94212018-f635-4d56-a099-b9e4f45ec632-model-cache\") pod \"e2e-same-id-b-784e0608-kserve-6d98d66ff7-hz49g\" (UID: \"94212018-f635-4d56-a099-b9e4f45ec632\") " pod="llm/e2e-same-id-b-784e0608-kserve-6d98d66ff7-hz49g" Oct 31 05:24:47.896159 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:47.894798 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94212018-f635-4d56-a099-b9e4f45ec632-tmp-dir\") pod \"e2e-same-id-b-784e0608-kserve-6d98d66ff7-hz49g\" (UID: \"94212018-f635-4d56-a099-b9e4f45ec632\") " pod="llm/e2e-same-id-b-784e0608-kserve-6d98d66ff7-hz49g" Oct 31 05:24:47.896159 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:47.895059 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94212018-f635-4d56-a099-b9e4f45ec632-home\") pod \"e2e-same-id-b-784e0608-kserve-6d98d66ff7-hz49g\" (UID: \"94212018-f635-4d56-a099-b9e4f45ec632\") " pod="llm/e2e-same-id-b-784e0608-kserve-6d98d66ff7-hz49g" Oct 31 05:24:47.897706 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:47.897611 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94212018-f635-4d56-a099-b9e4f45ec632-dshm\") pod \"e2e-same-id-b-784e0608-kserve-6d98d66ff7-hz49g\" (UID: \"94212018-f635-4d56-a099-b9e4f45ec632\") " pod="llm/e2e-same-id-b-784e0608-kserve-6d98d66ff7-hz49g" Oct 31 05:24:47.897972 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:47.897951 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94212018-f635-4d56-a099-b9e4f45ec632-tls-certs\") pod \"e2e-same-id-b-784e0608-kserve-6d98d66ff7-hz49g\" (UID: \"94212018-f635-4d56-a099-b9e4f45ec632\") " pod="llm/e2e-same-id-b-784e0608-kserve-6d98d66ff7-hz49g" Oct 31 05:24:47.916973 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:47.916949 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9k94\" (UniqueName: \"kubernetes.io/projected/94212018-f635-4d56-a099-b9e4f45ec632-kube-api-access-c9k94\") pod \"e2e-same-id-b-784e0608-kserve-6d98d66ff7-hz49g\" (UID: \"94212018-f635-4d56-a099-b9e4f45ec632\") " pod="llm/e2e-same-id-b-784e0608-kserve-6d98d66ff7-hz49g" Oct 31 05:24:48.005313 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:48.005287 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-784e0608-kserve-6d98d66ff7-hz49g" Oct 31 05:24:48.301791 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:48.301760 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-784e0608-kserve-6d98d66ff7-hz49g"] Oct 31 05:24:48.301938 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:24:48.301832 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94212018_f635_4d56_a099_b9e4f45ec632.slice/crio-f9754a167c713406d86602cee71577bef4115a18812013b8a1e2e96f5e4946ca WatchSource:0}: Error finding container f9754a167c713406d86602cee71577bef4115a18812013b8a1e2e96f5e4946ca: Status 404 returned error can't find the container with id f9754a167c713406d86602cee71577bef4115a18812013b8a1e2e96f5e4946ca Oct 31 05:24:48.441765 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:48.441727 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-784e0608-kserve-6d98d66ff7-hz49g" event={"ID":"94212018-f635-4d56-a099-b9e4f45ec632","Type":"ContainerStarted","Data":"7fbd1163c4229519ab782792f6b67c041845566c2477f23d21bdf93d68b12fc4"} Oct 31 05:24:48.441765 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:48.441772 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-784e0608-kserve-6d98d66ff7-hz49g" event={"ID":"94212018-f635-4d56-a099-b9e4f45ec632","Type":"ContainerStarted","Data":"f9754a167c713406d86602cee71577bef4115a18812013b8a1e2e96f5e4946ca"} Oct 31 05:24:48.484512 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:48.484458 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-784e0608-kserve-6d98d66ff7-hz49g" podStartSLOduration=1.4844438549999999 podStartE2EDuration="1.484443855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:24:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:24:48.483265394 +0000 UTC m=+4843.239603433" watchObservedRunningTime="2026-10-31 05:24:48.484443855 +0000 UTC m=+4843.240781894" Oct 31 05:24:49.005740 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:49.005709 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-784e0608-kserve-6d98d66ff7-hz49g" Oct 31 05:24:52.584371 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:24:52.583999 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:24:58.006300 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:58.006265 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-784e0608-kserve-6d98d66ff7-hz49g" Oct 31 05:24:58.018936 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:58.018908 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-784e0608-kserve-6d98d66ff7-hz49g" Oct 31 05:24:58.027127 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:58.027101 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b9vls"] Oct 31 05:24:58.038891 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:58.038865 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b9vls"] Oct 31 05:24:58.555417 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:58.555391 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-784e0608-kserve-6d98d66ff7-hz49g" Oct 31 05:24:59.406393 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:24:59.406364 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37f41b51-5216-424f-9e18-81c46b69ffb2" path="/var/lib/kubelet/pods/37f41b51-5216-424f-9e18-81c46b69ffb2/volumes" Oct 31 05:25:05.495427 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:05.495394 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9lqcq"] Oct 31 05:25:05.496041 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:05.496019 5019 cpu_manager.go:414] "containerMap: removing container" podUID="5e010ac6-7566-49bc-ae3d-02f791df20c7" containerName="istio-proxy" Oct 31 05:25:05.496143 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:05.496044 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e010ac6-7566-49bc-ae3d-02f791df20c7" containerName="istio-proxy" Oct 31 05:25:05.503025 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:05.502992 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9lqcq" Oct 31 05:25:05.519536 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:05.519498 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9lqcq"] Oct 31 05:25:05.527604 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:05.527579 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:25:05.527845 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:05.527614 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:25:05.527845 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:05.527693 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfhzn\"" Oct 31 05:25:05.529765 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:05.529746 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:25:05.532435 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:05.532413 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5pqc\" (UniqueName: \"kubernetes.io/projected/930ebbb3-2834-4e7c-8ce6-3154e80142fb-kube-api-access-s5pqc\") pod \"maas-api-revoke-sub-452b6d7b-9lqcq\" (UID: \"930ebbb3-2834-4e7c-8ce6-3154e80142fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9lqcq" Oct 31 05:25:05.532542 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:05.532442 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/930ebbb3-2834-4e7c-8ce6-3154e80142fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-9lqcq\" (UID: \"930ebbb3-2834-4e7c-8ce6-3154e80142fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9lqcq" Oct 31 05:25:05.532606 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:05.532577 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/930ebbb3-2834-4e7c-8ce6-3154e80142fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-9lqcq\" (UID: \"930ebbb3-2834-4e7c-8ce6-3154e80142fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9lqcq" Oct 31 05:25:05.633838 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:05.633811 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/930ebbb3-2834-4e7c-8ce6-3154e80142fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-9lqcq\" (UID: \"930ebbb3-2834-4e7c-8ce6-3154e80142fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9lqcq" Oct 31 05:25:05.633991 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:05.633863 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5pqc\" (UniqueName: \"kubernetes.io/projected/930ebbb3-2834-4e7c-8ce6-3154e80142fb-kube-api-access-s5pqc\") pod \"maas-api-revoke-sub-452b6d7b-9lqcq\" (UID: \"930ebbb3-2834-4e7c-8ce6-3154e80142fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9lqcq" Oct 31 05:25:05.633991 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:05.633976 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/930ebbb3-2834-4e7c-8ce6-3154e80142fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-9lqcq\" (UID: \"930ebbb3-2834-4e7c-8ce6-3154e80142fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9lqcq" Oct 31 05:25:05.636324 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:05.636303 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/930ebbb3-2834-4e7c-8ce6-3154e80142fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-9lqcq\" (UID: \"930ebbb3-2834-4e7c-8ce6-3154e80142fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9lqcq" Oct 31 05:25:05.647431 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:05.647410 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:25:05.654897 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:05.654878 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/930ebbb3-2834-4e7c-8ce6-3154e80142fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-9lqcq\" (UID: \"930ebbb3-2834-4e7c-8ce6-3154e80142fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9lqcq" Oct 31 05:25:05.668069 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:05.668051 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:25:05.677602 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:05.677579 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5pqc\" (UniqueName: \"kubernetes.io/projected/930ebbb3-2834-4e7c-8ce6-3154e80142fb-kube-api-access-s5pqc\") pod \"maas-api-revoke-sub-452b6d7b-9lqcq\" (UID: \"930ebbb3-2834-4e7c-8ce6-3154e80142fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9lqcq" Oct 31 05:25:05.834255 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:05.834184 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfhzn\"" Oct 31 05:25:05.841349 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:05.841318 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9lqcq" Oct 31 05:25:06.138712 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:06.138682 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9lqcq"] Oct 31 05:25:06.144049 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:25:06.140189 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod930ebbb3_2834_4e7c_8ce6_3154e80142fb.slice/crio-90d79d12ab28d933f36f33ba25e5b87e8be2fb301843983fa5df9afe7a0793bb WatchSource:0}: Error finding container 90d79d12ab28d933f36f33ba25e5b87e8be2fb301843983fa5df9afe7a0793bb: Status 404 returned error can't find the container with id 90d79d12ab28d933f36f33ba25e5b87e8be2fb301843983fa5df9afe7a0793bb Oct 31 05:25:06.586550 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:06.586510 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9lqcq" event={"ID":"930ebbb3-2834-4e7c-8ce6-3154e80142fb","Type":"ContainerStarted","Data":"65842166e029e69dcb5e53e86e9c8a92b30f78cc877bf5ccc2278658026dce50"} Oct 31 05:25:06.586934 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:06.586568 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9lqcq" event={"ID":"930ebbb3-2834-4e7c-8ce6-3154e80142fb","Type":"ContainerStarted","Data":"90d79d12ab28d933f36f33ba25e5b87e8be2fb301843983fa5df9afe7a0793bb"} Oct 31 05:25:06.628801 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:06.628746 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9lqcq" podStartSLOduration=1.628733371 podStartE2EDuration="1.628733371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:25:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:25:06.627883554 +0000 UTC m=+4861.384221597" watchObservedRunningTime="2026-10-31 05:25:06.628733371 +0000 UTC m=+4861.385071411" Oct 31 05:25:07.598893 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:07.598853 5019 generic.go:356] "Generic (PLEG): container finished" podID="930ebbb3-2834-4e7c-8ce6-3154e80142fb" containerID="65842166e029e69dcb5e53e86e9c8a92b30f78cc877bf5ccc2278658026dce50" exitCode=0 Oct 31 05:25:07.599447 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:07.599025 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9lqcq" event={"ID":"930ebbb3-2834-4e7c-8ce6-3154e80142fb","Type":"ContainerDied","Data":"65842166e029e69dcb5e53e86e9c8a92b30f78cc877bf5ccc2278658026dce50"} Oct 31 05:25:09.575776 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:09.575486 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9lqcq" Oct 31 05:25:09.622269 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:09.622232 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/930ebbb3-2834-4e7c-8ce6-3154e80142fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/930ebbb3-2834-4e7c-8ce6-3154e80142fb-maas-api-xks-ca\") pod \"930ebbb3-2834-4e7c-8ce6-3154e80142fb\" (UID: \"930ebbb3-2834-4e7c-8ce6-3154e80142fb\") " Oct 31 05:25:09.626484 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:09.626454 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/930ebbb3-2834-4e7c-8ce6-3154e80142fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/930ebbb3-2834-4e7c-8ce6-3154e80142fb-maas-api-openshift-ca\") pod \"930ebbb3-2834-4e7c-8ce6-3154e80142fb\" (UID: \"930ebbb3-2834-4e7c-8ce6-3154e80142fb\") " Oct 31 05:25:09.627048 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:09.627029 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/930ebbb3-2834-4e7c-8ce6-3154e80142fb-kube-api-access-s5pqc\" (UniqueName: \"kubernetes.io/projected/930ebbb3-2834-4e7c-8ce6-3154e80142fb-kube-api-access-s5pqc\") pod \"930ebbb3-2834-4e7c-8ce6-3154e80142fb\" (UID: \"930ebbb3-2834-4e7c-8ce6-3154e80142fb\") " Oct 31 05:25:09.636438 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:09.626976 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/930ebbb3-2834-4e7c-8ce6-3154e80142fb-maas-api-openshift-ca" pod "930ebbb3-2834-4e7c-8ce6-3154e80142fb" (UID: "930ebbb3-2834-4e7c-8ce6-3154e80142fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:25:09.636438 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:09.627983 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/930ebbb3-2834-4e7c-8ce6-3154e80142fb-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:25:09.746921 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:09.746879 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9lqcq" event={"ID":"930ebbb3-2834-4e7c-8ce6-3154e80142fb","Type":"ContainerDied","Data":"90d79d12ab28d933f36f33ba25e5b87e8be2fb301843983fa5df9afe7a0793bb"} Oct 31 05:25:09.757443 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:09.757405 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90d79d12ab28d933f36f33ba25e5b87e8be2fb301843983fa5df9afe7a0793bb" Oct 31 05:25:09.757877 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:09.757859 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9lqcq" Oct 31 05:25:09.758792 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:09.758748 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/930ebbb3-2834-4e7c-8ce6-3154e80142fb-maas-api-xks-ca" pod "930ebbb3-2834-4e7c-8ce6-3154e80142fb" (UID: "930ebbb3-2834-4e7c-8ce6-3154e80142fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:25:09.857367 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:09.856548 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/930ebbb3-2834-4e7c-8ce6-3154e80142fb-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:25:09.857730 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:09.857700 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/930ebbb3-2834-4e7c-8ce6-3154e80142fb-kube-api-access-s5pqc" pod "930ebbb3-2834-4e7c-8ce6-3154e80142fb" (UID: "930ebbb3-2834-4e7c-8ce6-3154e80142fb"). InnerVolumeSpecName "kube-api-access-s5pqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:25:09.986355 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:09.966643 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5pqc\" (UniqueName: \"kubernetes.io/projected/930ebbb3-2834-4e7c-8ce6-3154e80142fb-kube-api-access-s5pqc\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:25:14.693031 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:14.692996 5019 scope.go:122] "RemoveContainer" containerID="b474c63ad7116fee40a6696d91c253030453da7b8b5adf44a740aa9b711ca602" Oct 31 05:25:16.759469 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:16.759434 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-784e0608-kserve-7cdf47df77-sn6vk"] Oct 31 05:25:22.599596 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:25:22.598047 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:25:22.603368 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:22.602627 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-784e0608-kserve-6d98d66ff7-hz49g"] Oct 31 05:25:24.029074 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:24.029039 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hgx4g"] Oct 31 05:25:24.040802 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:24.040773 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hgx4g"] Oct 31 05:25:25.437621 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:25.437578 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1884c87e-c15e-490f-b09d-c54dad973f44" path="/var/lib/kubelet/pods/1884c87e-c15e-490f-b09d-c54dad973f44/volumes" Oct 31 05:25:27.037669 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:27.037635 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cbsgr"] Oct 31 05:25:27.038102 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:27.038087 5019 cpu_manager.go:414] "containerMap: removing container" podUID="930ebbb3-2834-4e7c-8ce6-3154e80142fb" containerName="revoke-keys" Oct 31 05:25:27.038154 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:27.038102 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="930ebbb3-2834-4e7c-8ce6-3154e80142fb" containerName="revoke-keys" Oct 31 05:25:27.038191 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:27.038186 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="930ebbb3-2834-4e7c-8ce6-3154e80142fb" containerName="revoke-keys" Oct 31 05:25:27.041784 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:27.041764 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cbsgr" Oct 31 05:25:27.061156 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:27.061133 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cbsgr"] Oct 31 05:25:27.069975 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:27.069951 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:25:27.069975 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:27.069951 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:25:27.070173 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:27.069998 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfhzn\"" Oct 31 05:25:27.070173 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:27.069999 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:25:27.100578 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:27.100554 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eafb76a4-6a45-4a64-8bce-2787d5914a8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-cbsgr\" (UID: \"eafb76a4-6a45-4a64-8bce-2787d5914a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cbsgr" Oct 31 05:25:27.100670 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:27.100589 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eafb76a4-6a45-4a64-8bce-2787d5914a8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-cbsgr\" (UID: \"eafb76a4-6a45-4a64-8bce-2787d5914a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cbsgr" Oct 31 05:25:27.100718 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:27.100677 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcrt8\" (UniqueName: \"kubernetes.io/projected/eafb76a4-6a45-4a64-8bce-2787d5914a8c-kube-api-access-lcrt8\") pod \"maas-api-revoke-sub-3e31a506-cbsgr\" (UID: \"eafb76a4-6a45-4a64-8bce-2787d5914a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cbsgr" Oct 31 05:25:27.201289 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:27.201250 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eafb76a4-6a45-4a64-8bce-2787d5914a8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-cbsgr\" (UID: \"eafb76a4-6a45-4a64-8bce-2787d5914a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cbsgr" Oct 31 05:25:27.201289 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:27.201291 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eafb76a4-6a45-4a64-8bce-2787d5914a8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-cbsgr\" (UID: \"eafb76a4-6a45-4a64-8bce-2787d5914a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cbsgr" Oct 31 05:25:27.201502 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:27.201354 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcrt8\" (UniqueName: \"kubernetes.io/projected/eafb76a4-6a45-4a64-8bce-2787d5914a8c-kube-api-access-lcrt8\") pod \"maas-api-revoke-sub-3e31a506-cbsgr\" (UID: \"eafb76a4-6a45-4a64-8bce-2787d5914a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cbsgr" Oct 31 05:25:27.202152 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:27.202129 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eafb76a4-6a45-4a64-8bce-2787d5914a8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-cbsgr\" (UID: \"eafb76a4-6a45-4a64-8bce-2787d5914a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cbsgr" Oct 31 05:25:27.204265 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:27.204233 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eafb76a4-6a45-4a64-8bce-2787d5914a8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-cbsgr\" (UID: \"eafb76a4-6a45-4a64-8bce-2787d5914a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cbsgr" Oct 31 05:25:27.224850 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:27.224823 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcrt8\" (UniqueName: \"kubernetes.io/projected/eafb76a4-6a45-4a64-8bce-2787d5914a8c-kube-api-access-lcrt8\") pod \"maas-api-revoke-sub-3e31a506-cbsgr\" (UID: \"eafb76a4-6a45-4a64-8bce-2787d5914a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cbsgr" Oct 31 05:25:27.354370 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:27.354315 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cbsgr" Oct 31 05:25:27.659485 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:27.659453 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cbsgr"] Oct 31 05:25:27.660740 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:25:27.660714 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeafb76a4_6a45_4a64_8bce_2787d5914a8c.slice/crio-168ba455d3e045fb11b44e3905a43b7f0a5829026b89bebaf2101a90482f1fbc WatchSource:0}: Error finding container 168ba455d3e045fb11b44e3905a43b7f0a5829026b89bebaf2101a90482f1fbc: Status 404 returned error can't find the container with id 168ba455d3e045fb11b44e3905a43b7f0a5829026b89bebaf2101a90482f1fbc Oct 31 05:25:27.946174 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:27.946102 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cbsgr" event={"ID":"eafb76a4-6a45-4a64-8bce-2787d5914a8c","Type":"ContainerStarted","Data":"168ba455d3e045fb11b44e3905a43b7f0a5829026b89bebaf2101a90482f1fbc"} Oct 31 05:25:28.956830 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:28.956728 5019 generic.go:356] "Generic (PLEG): container finished" podID="eafb76a4-6a45-4a64-8bce-2787d5914a8c" containerID="e242b56caad567fcecc1a2d6ca7f3fa0174b025389e3cb01a64793ad1e6430a8" exitCode=0 Oct 31 05:25:28.956830 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:28.956807 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cbsgr" event={"ID":"eafb76a4-6a45-4a64-8bce-2787d5914a8c","Type":"ContainerDied","Data":"e242b56caad567fcecc1a2d6ca7f3fa0174b025389e3cb01a64793ad1e6430a8"} Oct 31 05:25:30.821766 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:30.821731 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cbsgr" Oct 31 05:25:30.956115 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:30.956022 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eafb76a4-6a45-4a64-8bce-2787d5914a8c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eafb76a4-6a45-4a64-8bce-2787d5914a8c-maas-api-openshift-ca\") pod \"eafb76a4-6a45-4a64-8bce-2787d5914a8c\" (UID: \"eafb76a4-6a45-4a64-8bce-2787d5914a8c\") " Oct 31 05:25:30.966446 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:30.966410 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eafb76a4-6a45-4a64-8bce-2787d5914a8c-kube-api-access-lcrt8\" (UniqueName: \"kubernetes.io/projected/eafb76a4-6a45-4a64-8bce-2787d5914a8c-kube-api-access-lcrt8\") pod \"eafb76a4-6a45-4a64-8bce-2787d5914a8c\" (UID: \"eafb76a4-6a45-4a64-8bce-2787d5914a8c\") " Oct 31 05:25:30.966706 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:30.966685 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eafb76a4-6a45-4a64-8bce-2787d5914a8c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eafb76a4-6a45-4a64-8bce-2787d5914a8c-maas-api-xks-ca\") pod \"eafb76a4-6a45-4a64-8bce-2787d5914a8c\" (UID: \"eafb76a4-6a45-4a64-8bce-2787d5914a8c\") " Oct 31 05:25:30.977858 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:30.977829 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eafb76a4-6a45-4a64-8bce-2787d5914a8c-maas-api-openshift-ca" pod "eafb76a4-6a45-4a64-8bce-2787d5914a8c" (UID: "eafb76a4-6a45-4a64-8bce-2787d5914a8c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:25:31.016911 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:31.016861 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eafb76a4-6a45-4a64-8bce-2787d5914a8c-maas-api-xks-ca" pod "eafb76a4-6a45-4a64-8bce-2787d5914a8c" (UID: "eafb76a4-6a45-4a64-8bce-2787d5914a8c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:25:31.051587 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:31.051511 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eafb76a4-6a45-4a64-8bce-2787d5914a8c-kube-api-access-lcrt8" pod "eafb76a4-6a45-4a64-8bce-2787d5914a8c" (UID: "eafb76a4-6a45-4a64-8bce-2787d5914a8c"). InnerVolumeSpecName "kube-api-access-lcrt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:25:31.109356 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:31.071786 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cbsgr" event={"ID":"eafb76a4-6a45-4a64-8bce-2787d5914a8c","Type":"ContainerDied","Data":"168ba455d3e045fb11b44e3905a43b7f0a5829026b89bebaf2101a90482f1fbc"} Oct 31 05:25:31.109356 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:31.071853 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="168ba455d3e045fb11b44e3905a43b7f0a5829026b89bebaf2101a90482f1fbc" Oct 31 05:25:31.109356 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:31.071931 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cbsgr" Oct 31 05:25:31.127354 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:31.111563 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eafb76a4-6a45-4a64-8bce-2787d5914a8c-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:25:31.127659 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:31.127620 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcrt8\" (UniqueName: \"kubernetes.io/projected/eafb76a4-6a45-4a64-8bce-2787d5914a8c-kube-api-access-lcrt8\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:25:31.127823 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:31.127807 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eafb76a4-6a45-4a64-8bce-2787d5914a8c-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:25:31.525558 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:31.525511 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-rgj85 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.218:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 05:25:31.525907 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:31.525855 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rgj85" podUID="8430b919-a9dd-4c7e-9266-ef6872cc0b9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.218:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 05:25:31.780012 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:31.779921 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-784e0608-kserve-7cdf47df77-sn6vk" podUID="248a0e94-8fa1-4f4c-af6e-28c8035d1e73" containerName="main" containerID="cri-o://4eb3247d11cc5f2e3b8af933f0b830d6ff3c628e8e92c908e3b9c43f6f644360" gracePeriod=45 Oct 31 05:25:32.052272 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:32.052242 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-784e0608-kserve-7cdf47df77-sn6vk" Oct 31 05:25:32.078443 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:32.078409 5019 generic.go:356] "Generic (PLEG): container finished" podID="248a0e94-8fa1-4f4c-af6e-28c8035d1e73" containerID="4eb3247d11cc5f2e3b8af933f0b830d6ff3c628e8e92c908e3b9c43f6f644360" exitCode=0 Oct 31 05:25:32.078634 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:32.078456 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-784e0608-kserve-7cdf47df77-sn6vk" Oct 31 05:25:32.078634 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:32.078497 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-784e0608-kserve-7cdf47df77-sn6vk" event={"ID":"248a0e94-8fa1-4f4c-af6e-28c8035d1e73","Type":"ContainerDied","Data":"4eb3247d11cc5f2e3b8af933f0b830d6ff3c628e8e92c908e3b9c43f6f644360"} Oct 31 05:25:32.078634 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:32.078543 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-784e0608-kserve-7cdf47df77-sn6vk" event={"ID":"248a0e94-8fa1-4f4c-af6e-28c8035d1e73","Type":"ContainerDied","Data":"ad6ec7c386fdd88d056f51692e810c281198af140f5636a794a49b8167f63bff"} Oct 31 05:25:32.078634 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:32.078561 5019 scope.go:122] "RemoveContainer" containerID="4eb3247d11cc5f2e3b8af933f0b830d6ff3c628e8e92c908e3b9c43f6f644360" Oct 31 05:25:32.087940 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:32.087914 5019 scope.go:122] "RemoveContainer" containerID="4eb3247d11cc5f2e3b8af933f0b830d6ff3c628e8e92c908e3b9c43f6f644360" Oct 31 05:25:32.088222 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:25:32.088195 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eb3247d11cc5f2e3b8af933f0b830d6ff3c628e8e92c908e3b9c43f6f644360\": container with ID starting with 4eb3247d11cc5f2e3b8af933f0b830d6ff3c628e8e92c908e3b9c43f6f644360 not found: ID does not exist" containerID="4eb3247d11cc5f2e3b8af933f0b830d6ff3c628e8e92c908e3b9c43f6f644360" Oct 31 05:25:32.088292 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:32.088236 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eb3247d11cc5f2e3b8af933f0b830d6ff3c628e8e92c908e3b9c43f6f644360"} err="failed to get container status \"4eb3247d11cc5f2e3b8af933f0b830d6ff3c628e8e92c908e3b9c43f6f644360\": rpc error: code = NotFound desc = could not find container \"4eb3247d11cc5f2e3b8af933f0b830d6ff3c628e8e92c908e3b9c43f6f644360\": container with ID starting with 4eb3247d11cc5f2e3b8af933f0b830d6ff3c628e8e92c908e3b9c43f6f644360 not found: ID does not exist" Oct 31 05:25:32.149674 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:32.149643 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/248a0e94-8fa1-4f4c-af6e-28c8035d1e73-dshm\" (UniqueName: \"kubernetes.io/empty-dir/248a0e94-8fa1-4f4c-af6e-28c8035d1e73-dshm\") pod \"248a0e94-8fa1-4f4c-af6e-28c8035d1e73\" (UID: \"248a0e94-8fa1-4f4c-af6e-28c8035d1e73\") " Oct 31 05:25:32.149852 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:32.149729 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/248a0e94-8fa1-4f4c-af6e-28c8035d1e73-home\" (UniqueName: \"kubernetes.io/empty-dir/248a0e94-8fa1-4f4c-af6e-28c8035d1e73-home\") pod \"248a0e94-8fa1-4f4c-af6e-28c8035d1e73\" (UID: \"248a0e94-8fa1-4f4c-af6e-28c8035d1e73\") " Oct 31 05:25:32.149852 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:32.149769 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/248a0e94-8fa1-4f4c-af6e-28c8035d1e73-kube-api-access-wxm7n\" (UniqueName: \"kubernetes.io/projected/248a0e94-8fa1-4f4c-af6e-28c8035d1e73-kube-api-access-wxm7n\") pod \"248a0e94-8fa1-4f4c-af6e-28c8035d1e73\" (UID: \"248a0e94-8fa1-4f4c-af6e-28c8035d1e73\") " Oct 31 05:25:32.149852 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:32.149817 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/248a0e94-8fa1-4f4c-af6e-28c8035d1e73-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/248a0e94-8fa1-4f4c-af6e-28c8035d1e73-model-cache\") pod \"248a0e94-8fa1-4f4c-af6e-28c8035d1e73\" (UID: \"248a0e94-8fa1-4f4c-af6e-28c8035d1e73\") " Oct 31 05:25:32.150053 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:32.149859 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/248a0e94-8fa1-4f4c-af6e-28c8035d1e73-tls-certs\" (UniqueName: \"kubernetes.io/secret/248a0e94-8fa1-4f4c-af6e-28c8035d1e73-tls-certs\") pod \"248a0e94-8fa1-4f4c-af6e-28c8035d1e73\" (UID: \"248a0e94-8fa1-4f4c-af6e-28c8035d1e73\") " Oct 31 05:25:32.150053 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:32.149988 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/248a0e94-8fa1-4f4c-af6e-28c8035d1e73-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/248a0e94-8fa1-4f4c-af6e-28c8035d1e73-tmp-dir\") pod \"248a0e94-8fa1-4f4c-af6e-28c8035d1e73\" (UID: \"248a0e94-8fa1-4f4c-af6e-28c8035d1e73\") " Oct 31 05:25:32.150053 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:32.150036 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/248a0e94-8fa1-4f4c-af6e-28c8035d1e73-home" pod "248a0e94-8fa1-4f4c-af6e-28c8035d1e73" (UID: "248a0e94-8fa1-4f4c-af6e-28c8035d1e73"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:25:32.150375 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:32.150327 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/248a0e94-8fa1-4f4c-af6e-28c8035d1e73-tmp-dir" pod "248a0e94-8fa1-4f4c-af6e-28c8035d1e73" (UID: "248a0e94-8fa1-4f4c-af6e-28c8035d1e73"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:25:32.150375 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:32.150368 5019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/248a0e94-8fa1-4f4c-af6e-28c8035d1e73-home\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:25:32.150571 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:32.150385 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/248a0e94-8fa1-4f4c-af6e-28c8035d1e73-model-cache" pod "248a0e94-8fa1-4f4c-af6e-28c8035d1e73" (UID: "248a0e94-8fa1-4f4c-af6e-28c8035d1e73"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:25:32.152011 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:32.151981 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/248a0e94-8fa1-4f4c-af6e-28c8035d1e73-tls-certs" pod "248a0e94-8fa1-4f4c-af6e-28c8035d1e73" (UID: "248a0e94-8fa1-4f4c-af6e-28c8035d1e73"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:25:32.152110 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:32.152014 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/248a0e94-8fa1-4f4c-af6e-28c8035d1e73-dshm" pod "248a0e94-8fa1-4f4c-af6e-28c8035d1e73" (UID: "248a0e94-8fa1-4f4c-af6e-28c8035d1e73"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:25:32.152110 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:32.152086 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/248a0e94-8fa1-4f4c-af6e-28c8035d1e73-kube-api-access-wxm7n" pod "248a0e94-8fa1-4f4c-af6e-28c8035d1e73" (UID: "248a0e94-8fa1-4f4c-af6e-28c8035d1e73"). InnerVolumeSpecName "kube-api-access-wxm7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:25:32.251116 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:32.251082 5019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/248a0e94-8fa1-4f4c-af6e-28c8035d1e73-tls-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:25:32.251116 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:32.251111 5019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/248a0e94-8fa1-4f4c-af6e-28c8035d1e73-tmp-dir\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:25:32.251116 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:32.251120 5019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/248a0e94-8fa1-4f4c-af6e-28c8035d1e73-dshm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:25:32.251455 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:32.251128 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxm7n\" (UniqueName: \"kubernetes.io/projected/248a0e94-8fa1-4f4c-af6e-28c8035d1e73-kube-api-access-wxm7n\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:25:32.251455 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:32.251137 5019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/248a0e94-8fa1-4f4c-af6e-28c8035d1e73-model-cache\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:25:32.424088 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:32.424055 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-784e0608-kserve-7cdf47df77-sn6vk"] Oct 31 05:25:32.435935 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:32.435912 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-784e0608-kserve-7cdf47df77-sn6vk"] Oct 31 05:25:33.405829 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:33.405784 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="248a0e94-8fa1-4f4c-af6e-28c8035d1e73" path="/var/lib/kubelet/pods/248a0e94-8fa1-4f4c-af6e-28c8035d1e73/volumes" Oct 31 05:25:37.669228 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:37.669163 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-784e0608-kserve-6d98d66ff7-hz49g" podUID="94212018-f635-4d56-a099-b9e4f45ec632" containerName="main" containerID="cri-o://7fbd1163c4229519ab782792f6b67c041845566c2477f23d21bdf93d68b12fc4" gracePeriod=45 Oct 31 05:25:37.930958 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:37.930902 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-784e0608-kserve-6d98d66ff7-hz49g" Oct 31 05:25:38.104720 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:38.104683 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94212018-f635-4d56-a099-b9e4f45ec632-tls-certs\" (UniqueName: \"kubernetes.io/secret/94212018-f635-4d56-a099-b9e4f45ec632-tls-certs\") pod \"94212018-f635-4d56-a099-b9e4f45ec632\" (UID: \"94212018-f635-4d56-a099-b9e4f45ec632\") " Oct 31 05:25:38.104918 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:38.104775 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94212018-f635-4d56-a099-b9e4f45ec632-dshm\" (UniqueName: \"kubernetes.io/empty-dir/94212018-f635-4d56-a099-b9e4f45ec632-dshm\") pod \"94212018-f635-4d56-a099-b9e4f45ec632\" (UID: \"94212018-f635-4d56-a099-b9e4f45ec632\") " Oct 31 05:25:38.104918 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:38.104842 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94212018-f635-4d56-a099-b9e4f45ec632-home\" (UniqueName: \"kubernetes.io/empty-dir/94212018-f635-4d56-a099-b9e4f45ec632-home\") pod \"94212018-f635-4d56-a099-b9e4f45ec632\" (UID: \"94212018-f635-4d56-a099-b9e4f45ec632\") " Oct 31 05:25:38.104918 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:38.104889 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94212018-f635-4d56-a099-b9e4f45ec632-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94212018-f635-4d56-a099-b9e4f45ec632-tmp-dir\") pod \"94212018-f635-4d56-a099-b9e4f45ec632\" (UID: \"94212018-f635-4d56-a099-b9e4f45ec632\") " Oct 31 05:25:38.105152 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:38.104924 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94212018-f635-4d56-a099-b9e4f45ec632-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94212018-f635-4d56-a099-b9e4f45ec632-model-cache\") pod \"94212018-f635-4d56-a099-b9e4f45ec632\" (UID: \"94212018-f635-4d56-a099-b9e4f45ec632\") " Oct 31 05:25:38.105152 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:38.105023 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94212018-f635-4d56-a099-b9e4f45ec632-kube-api-access-c9k94\" (UniqueName: \"kubernetes.io/projected/94212018-f635-4d56-a099-b9e4f45ec632-kube-api-access-c9k94\") pod \"94212018-f635-4d56-a099-b9e4f45ec632\" (UID: \"94212018-f635-4d56-a099-b9e4f45ec632\") " Oct 31 05:25:38.105529 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:38.105487 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94212018-f635-4d56-a099-b9e4f45ec632-model-cache" pod "94212018-f635-4d56-a099-b9e4f45ec632" (UID: "94212018-f635-4d56-a099-b9e4f45ec632"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:25:38.105693 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:38.105592 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94212018-f635-4d56-a099-b9e4f45ec632-tmp-dir" pod "94212018-f635-4d56-a099-b9e4f45ec632" (UID: "94212018-f635-4d56-a099-b9e4f45ec632"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:25:38.105693 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:38.105649 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94212018-f635-4d56-a099-b9e4f45ec632-home" pod "94212018-f635-4d56-a099-b9e4f45ec632" (UID: "94212018-f635-4d56-a099-b9e4f45ec632"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:25:38.107495 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:38.107468 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94212018-f635-4d56-a099-b9e4f45ec632-tls-certs" pod "94212018-f635-4d56-a099-b9e4f45ec632" (UID: "94212018-f635-4d56-a099-b9e4f45ec632"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:25:38.107889 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:38.107862 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94212018-f635-4d56-a099-b9e4f45ec632-kube-api-access-c9k94" pod "94212018-f635-4d56-a099-b9e4f45ec632" (UID: "94212018-f635-4d56-a099-b9e4f45ec632"). InnerVolumeSpecName "kube-api-access-c9k94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:25:38.108460 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:38.108433 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94212018-f635-4d56-a099-b9e4f45ec632-dshm" pod "94212018-f635-4d56-a099-b9e4f45ec632" (UID: "94212018-f635-4d56-a099-b9e4f45ec632"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:25:38.120653 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:38.120629 5019 generic.go:356] "Generic (PLEG): container finished" podID="94212018-f635-4d56-a099-b9e4f45ec632" containerID="7fbd1163c4229519ab782792f6b67c041845566c2477f23d21bdf93d68b12fc4" exitCode=0 Oct 31 05:25:38.120756 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:38.120690 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-784e0608-kserve-6d98d66ff7-hz49g" Oct 31 05:25:38.120756 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:38.120717 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-784e0608-kserve-6d98d66ff7-hz49g" event={"ID":"94212018-f635-4d56-a099-b9e4f45ec632","Type":"ContainerDied","Data":"7fbd1163c4229519ab782792f6b67c041845566c2477f23d21bdf93d68b12fc4"} Oct 31 05:25:38.120869 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:38.120761 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-784e0608-kserve-6d98d66ff7-hz49g" event={"ID":"94212018-f635-4d56-a099-b9e4f45ec632","Type":"ContainerDied","Data":"f9754a167c713406d86602cee71577bef4115a18812013b8a1e2e96f5e4946ca"} Oct 31 05:25:38.120869 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:38.120781 5019 scope.go:122] "RemoveContainer" containerID="7fbd1163c4229519ab782792f6b67c041845566c2477f23d21bdf93d68b12fc4" Oct 31 05:25:38.136406 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:38.136390 5019 scope.go:122] "RemoveContainer" containerID="7fbd1163c4229519ab782792f6b67c041845566c2477f23d21bdf93d68b12fc4" Oct 31 05:25:38.136654 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:25:38.136630 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fbd1163c4229519ab782792f6b67c041845566c2477f23d21bdf93d68b12fc4\": container with ID starting with 7fbd1163c4229519ab782792f6b67c041845566c2477f23d21bdf93d68b12fc4 not found: ID does not exist" containerID="7fbd1163c4229519ab782792f6b67c041845566c2477f23d21bdf93d68b12fc4" Oct 31 05:25:38.136710 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:38.136662 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fbd1163c4229519ab782792f6b67c041845566c2477f23d21bdf93d68b12fc4"} err="failed to get container status \"7fbd1163c4229519ab782792f6b67c041845566c2477f23d21bdf93d68b12fc4\": rpc error: code = NotFound desc = could not find container \"7fbd1163c4229519ab782792f6b67c041845566c2477f23d21bdf93d68b12fc4\": container with ID starting with 7fbd1163c4229519ab782792f6b67c041845566c2477f23d21bdf93d68b12fc4 not found: ID does not exist" Oct 31 05:25:38.174808 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:38.174784 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-784e0608-kserve-6d98d66ff7-hz49g"] Oct 31 05:25:38.178680 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:38.178659 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-784e0608-kserve-6d98d66ff7-hz49g"] Oct 31 05:25:38.206500 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:38.206441 5019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94212018-f635-4d56-a099-b9e4f45ec632-home\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:25:38.206500 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:38.206480 5019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94212018-f635-4d56-a099-b9e4f45ec632-tmp-dir\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:25:38.206500 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:38.206489 5019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94212018-f635-4d56-a099-b9e4f45ec632-model-cache\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:25:38.206500 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:38.206499 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9k94\" (UniqueName: \"kubernetes.io/projected/94212018-f635-4d56-a099-b9e4f45ec632-kube-api-access-c9k94\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:25:38.206773 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:38.206508 5019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94212018-f635-4d56-a099-b9e4f45ec632-tls-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:25:38.206773 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:38.206526 5019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94212018-f635-4d56-a099-b9e4f45ec632-dshm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:25:39.406326 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:39.406291 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94212018-f635-4d56-a099-b9e4f45ec632" path="/var/lib/kubelet/pods/94212018-f635-4d56-a099-b9e4f45ec632/volumes" Oct 31 05:25:41.028479 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:41.028443 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f64dd"] Oct 31 05:25:41.040139 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:41.040108 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-f64dd"] Oct 31 05:25:41.407705 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:41.407613 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb37794d-be69-4586-9e19-bf3b9600b200" path="/var/lib/kubelet/pods/eb37794d-be69-4586-9e19-bf3b9600b200/volumes" Oct 31 05:25:47.125889 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:47.125793 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42aa8c1d-snpcr"] Oct 31 05:25:47.126542 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:47.126274 5019 cpu_manager.go:414] "containerMap: removing container" podUID="248a0e94-8fa1-4f4c-af6e-28c8035d1e73" containerName="main" Oct 31 05:25:47.126542 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:47.126290 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="248a0e94-8fa1-4f4c-af6e-28c8035d1e73" containerName="main" Oct 31 05:25:47.126542 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:47.126311 5019 cpu_manager.go:414] "containerMap: removing container" podUID="eafb76a4-6a45-4a64-8bce-2787d5914a8c" containerName="revoke-keys" Oct 31 05:25:47.126542 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:47.126316 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eafb76a4-6a45-4a64-8bce-2787d5914a8c" containerName="revoke-keys" Oct 31 05:25:47.126542 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:47.126360 5019 cpu_manager.go:414] "containerMap: removing container" podUID="94212018-f635-4d56-a099-b9e4f45ec632" containerName="main" Oct 31 05:25:47.126542 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:47.126369 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94212018-f635-4d56-a099-b9e4f45ec632" containerName="main" Oct 31 05:25:47.126542 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:47.126453 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="248a0e94-8fa1-4f4c-af6e-28c8035d1e73" containerName="main" Oct 31 05:25:47.126542 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:47.126465 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="eafb76a4-6a45-4a64-8bce-2787d5914a8c" containerName="revoke-keys" Oct 31 05:25:47.126542 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:47.126472 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="94212018-f635-4d56-a099-b9e4f45ec632" containerName="main" Oct 31 05:25:47.131098 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:47.131078 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42aa8c1d-snpcr" Oct 31 05:25:47.143490 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:47.143468 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bks8t\" (UniqueName: \"kubernetes.io/projected/d1f204b0-02c0-46d1-aab4-093b54f7caca-kube-api-access-bks8t\") pod \"maas-api-revoke-sub-42aa8c1d-snpcr\" (UID: \"d1f204b0-02c0-46d1-aab4-093b54f7caca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42aa8c1d-snpcr" Oct 31 05:25:47.143613 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:47.143511 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1f204b0-02c0-46d1-aab4-093b54f7caca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42aa8c1d-snpcr\" (UID: \"d1f204b0-02c0-46d1-aab4-093b54f7caca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42aa8c1d-snpcr" Oct 31 05:25:47.143613 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:47.143537 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1f204b0-02c0-46d1-aab4-093b54f7caca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42aa8c1d-snpcr\" (UID: \"d1f204b0-02c0-46d1-aab4-093b54f7caca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42aa8c1d-snpcr" Oct 31 05:25:47.149251 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:47.149225 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42aa8c1d-snpcr"] Oct 31 05:25:47.157867 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:47.157845 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:25:47.158080 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:47.157868 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfhzn\"" Oct 31 05:25:47.158080 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:47.157896 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:25:47.158237 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:47.157909 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:25:47.244983 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:47.244952 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bks8t\" (UniqueName: \"kubernetes.io/projected/d1f204b0-02c0-46d1-aab4-093b54f7caca-kube-api-access-bks8t\") pod \"maas-api-revoke-sub-42aa8c1d-snpcr\" (UID: \"d1f204b0-02c0-46d1-aab4-093b54f7caca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42aa8c1d-snpcr" Oct 31 05:25:47.245139 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:47.244993 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1f204b0-02c0-46d1-aab4-093b54f7caca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42aa8c1d-snpcr\" (UID: \"d1f204b0-02c0-46d1-aab4-093b54f7caca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42aa8c1d-snpcr" Oct 31 05:25:47.245139 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:47.245016 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1f204b0-02c0-46d1-aab4-093b54f7caca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42aa8c1d-snpcr\" (UID: \"d1f204b0-02c0-46d1-aab4-093b54f7caca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42aa8c1d-snpcr" Oct 31 05:25:47.245698 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:47.245678 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1f204b0-02c0-46d1-aab4-093b54f7caca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42aa8c1d-snpcr\" (UID: \"d1f204b0-02c0-46d1-aab4-093b54f7caca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42aa8c1d-snpcr" Oct 31 05:25:47.247392 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:47.247371 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1f204b0-02c0-46d1-aab4-093b54f7caca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42aa8c1d-snpcr\" (UID: \"d1f204b0-02c0-46d1-aab4-093b54f7caca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42aa8c1d-snpcr" Oct 31 05:25:47.267877 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:47.267852 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bks8t\" (UniqueName: \"kubernetes.io/projected/d1f204b0-02c0-46d1-aab4-093b54f7caca-kube-api-access-bks8t\") pod \"maas-api-revoke-sub-42aa8c1d-snpcr\" (UID: \"d1f204b0-02c0-46d1-aab4-093b54f7caca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42aa8c1d-snpcr" Oct 31 05:25:47.444357 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:47.444263 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42aa8c1d-snpcr" Oct 31 05:25:47.746440 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:47.746079 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42aa8c1d-snpcr"] Oct 31 05:25:47.750811 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:25:47.750783 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1f204b0_02c0_46d1_aab4_093b54f7caca.slice/crio-67415b4e4c34adf41fe8d5d9baffd14098604ee6592f54fa0e56c1fb10cb603b WatchSource:0}: Error finding container 67415b4e4c34adf41fe8d5d9baffd14098604ee6592f54fa0e56c1fb10cb603b: Status 404 returned error can't find the container with id 67415b4e4c34adf41fe8d5d9baffd14098604ee6592f54fa0e56c1fb10cb603b Oct 31 05:25:48.212554 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:48.212517 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42aa8c1d-snpcr" event={"ID":"d1f204b0-02c0-46d1-aab4-093b54f7caca","Type":"ContainerStarted","Data":"f6d8dd3b056da28d8807fb058affcae31bf0b54c829c548fe082b425bae07055"} Oct 31 05:25:48.212897 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:48.212583 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42aa8c1d-snpcr" event={"ID":"d1f204b0-02c0-46d1-aab4-093b54f7caca","Type":"ContainerStarted","Data":"67415b4e4c34adf41fe8d5d9baffd14098604ee6592f54fa0e56c1fb10cb603b"} Oct 31 05:25:48.254212 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:48.254168 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42aa8c1d-snpcr" podStartSLOduration=1.254154276 podStartE2EDuration="1.254154276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:25:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:25:48.252885247 +0000 UTC m=+4903.009223288" watchObservedRunningTime="2026-10-31 05:25:48.254154276 +0000 UTC m=+4903.010492316" Oct 31 05:25:49.219644 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:49.219612 5019 generic.go:356] "Generic (PLEG): container finished" podID="d1f204b0-02c0-46d1-aab4-093b54f7caca" containerID="f6d8dd3b056da28d8807fb058affcae31bf0b54c829c548fe082b425bae07055" exitCode=0 Oct 31 05:25:49.220102 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:49.219673 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42aa8c1d-snpcr" event={"ID":"d1f204b0-02c0-46d1-aab4-093b54f7caca","Type":"ContainerDied","Data":"f6d8dd3b056da28d8807fb058affcae31bf0b54c829c548fe082b425bae07055"} Oct 31 05:25:50.369631 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:50.369608 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42aa8c1d-snpcr" Oct 31 05:25:50.473603 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:50.473568 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1f204b0-02c0-46d1-aab4-093b54f7caca-kube-api-access-bks8t\" (UniqueName: \"kubernetes.io/projected/d1f204b0-02c0-46d1-aab4-093b54f7caca-kube-api-access-bks8t\") pod \"d1f204b0-02c0-46d1-aab4-093b54f7caca\" (UID: \"d1f204b0-02c0-46d1-aab4-093b54f7caca\") " Oct 31 05:25:50.473750 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:50.473640 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1f204b0-02c0-46d1-aab4-093b54f7caca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1f204b0-02c0-46d1-aab4-093b54f7caca-maas-api-xks-ca\") pod \"d1f204b0-02c0-46d1-aab4-093b54f7caca\" (UID: \"d1f204b0-02c0-46d1-aab4-093b54f7caca\") " Oct 31 05:25:50.473750 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:50.473709 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1f204b0-02c0-46d1-aab4-093b54f7caca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1f204b0-02c0-46d1-aab4-093b54f7caca-maas-api-openshift-ca\") pod \"d1f204b0-02c0-46d1-aab4-093b54f7caca\" (UID: \"d1f204b0-02c0-46d1-aab4-093b54f7caca\") " Oct 31 05:25:50.474145 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:50.474099 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1f204b0-02c0-46d1-aab4-093b54f7caca-maas-api-openshift-ca" pod "d1f204b0-02c0-46d1-aab4-093b54f7caca" (UID: "d1f204b0-02c0-46d1-aab4-093b54f7caca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:25:50.475791 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:50.475742 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1f204b0-02c0-46d1-aab4-093b54f7caca-maas-api-xks-ca" pod "d1f204b0-02c0-46d1-aab4-093b54f7caca" (UID: "d1f204b0-02c0-46d1-aab4-093b54f7caca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:25:50.475791 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:50.475748 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1f204b0-02c0-46d1-aab4-093b54f7caca-kube-api-access-bks8t" pod "d1f204b0-02c0-46d1-aab4-093b54f7caca" (UID: "d1f204b0-02c0-46d1-aab4-093b54f7caca"). InnerVolumeSpecName "kube-api-access-bks8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:25:50.575166 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:50.575136 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1f204b0-02c0-46d1-aab4-093b54f7caca-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:25:50.575166 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:50.575165 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bks8t\" (UniqueName: \"kubernetes.io/projected/d1f204b0-02c0-46d1-aab4-093b54f7caca-kube-api-access-bks8t\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:25:50.575326 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:50.575175 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1f204b0-02c0-46d1-aab4-093b54f7caca-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:25:51.233368 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:51.233313 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42aa8c1d-snpcr" event={"ID":"d1f204b0-02c0-46d1-aab4-093b54f7caca","Type":"ContainerDied","Data":"67415b4e4c34adf41fe8d5d9baffd14098604ee6592f54fa0e56c1fb10cb603b"} Oct 31 05:25:51.233586 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:51.233391 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67415b4e4c34adf41fe8d5d9baffd14098604ee6592f54fa0e56c1fb10cb603b" Oct 31 05:25:51.233586 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:51.233341 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42aa8c1d-snpcr" Oct 31 05:25:52.037519 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:52.037483 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hrdvp"] Oct 31 05:25:52.049211 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:52.049169 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hrdvp"] Oct 31 05:25:52.389557 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:25:52.389480 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:25:53.406768 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:25:53.406724 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="614fb8da-43f7-49cc-acb4-dbf2befda850" path="/var/lib/kubelet/pods/614fb8da-43f7-49cc-acb4-dbf2befda850/volumes" Oct 31 05:26:02.745401 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:02.745368 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-6554894b6b-98v2m"] Oct 31 05:26:02.745803 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:02.745790 5019 cpu_manager.go:414] "containerMap: removing container" podUID="d1f204b0-02c0-46d1-aab4-093b54f7caca" containerName="revoke-keys" Oct 31 05:26:02.745864 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:02.745804 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1f204b0-02c0-46d1-aab4-093b54f7caca" containerName="revoke-keys" Oct 31 05:26:02.745909 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:02.745876 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1f204b0-02c0-46d1-aab4-093b54f7caca" containerName="revoke-keys" Oct 31 05:26:02.750410 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:02.750383 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-6554894b6b-98v2m" Oct 31 05:26:02.766423 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:02.766403 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 31 05:26:02.769568 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:02.769543 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-6554894b6b-98v2m"] Oct 31 05:26:02.778642 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:02.778615 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-sw4pr\"" Oct 31 05:26:02.780391 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:02.780360 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crzm6\" (UniqueName: \"kubernetes.io/projected/a2ca5aff-5121-4461-b5a1-51b2e0c00455-kube-api-access-crzm6\") pod \"payload-pre-processing-6554894b6b-98v2m\" (UID: \"a2ca5aff-5121-4461-b5a1-51b2e0c00455\") " pod="openshift-ingress/payload-pre-processing-6554894b6b-98v2m" Oct 31 05:26:02.780495 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:02.780421 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2ca5aff-5121-4461-b5a1-51b2e0c00455-plugins-config-volume\") pod \"payload-pre-processing-6554894b6b-98v2m\" (UID: \"a2ca5aff-5121-4461-b5a1-51b2e0c00455\") " pod="openshift-ingress/payload-pre-processing-6554894b6b-98v2m" Oct 31 05:26:02.881607 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:02.881576 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crzm6\" (UniqueName: \"kubernetes.io/projected/a2ca5aff-5121-4461-b5a1-51b2e0c00455-kube-api-access-crzm6\") pod \"payload-pre-processing-6554894b6b-98v2m\" (UID: \"a2ca5aff-5121-4461-b5a1-51b2e0c00455\") " pod="openshift-ingress/payload-pre-processing-6554894b6b-98v2m" Oct 31 05:26:02.881824 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:02.881634 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2ca5aff-5121-4461-b5a1-51b2e0c00455-plugins-config-volume\") pod \"payload-pre-processing-6554894b6b-98v2m\" (UID: \"a2ca5aff-5121-4461-b5a1-51b2e0c00455\") " pod="openshift-ingress/payload-pre-processing-6554894b6b-98v2m" Oct 31 05:26:02.882312 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:02.882293 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2ca5aff-5121-4461-b5a1-51b2e0c00455-plugins-config-volume\") pod \"payload-pre-processing-6554894b6b-98v2m\" (UID: \"a2ca5aff-5121-4461-b5a1-51b2e0c00455\") " pod="openshift-ingress/payload-pre-processing-6554894b6b-98v2m" Oct 31 05:26:02.905992 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:02.905966 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crzm6\" (UniqueName: \"kubernetes.io/projected/a2ca5aff-5121-4461-b5a1-51b2e0c00455-kube-api-access-crzm6\") pod \"payload-pre-processing-6554894b6b-98v2m\" (UID: \"a2ca5aff-5121-4461-b5a1-51b2e0c00455\") " pod="openshift-ingress/payload-pre-processing-6554894b6b-98v2m" Oct 31 05:26:03.063932 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:03.063874 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-6554894b6b-98v2m" Oct 31 05:26:03.360505 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:03.360477 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-6554894b6b-98v2m"] Oct 31 05:26:03.362433 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:26:03.362407 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2ca5aff_5121_4461_b5a1_51b2e0c00455.slice/crio-d8a9c39376431972e7751c40d62554e0267122a143466a1b4ece89b94e1a97f5 WatchSource:0}: Error finding container d8a9c39376431972e7751c40d62554e0267122a143466a1b4ece89b94e1a97f5: Status 404 returned error can't find the container with id d8a9c39376431972e7751c40d62554e0267122a143466a1b4ece89b94e1a97f5 Oct 31 05:26:04.323667 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:04.323630 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-6554894b6b-98v2m" event={"ID":"a2ca5aff-5121-4461-b5a1-51b2e0c00455","Type":"ContainerStarted","Data":"580b4c6b89efc9c97df0d066987e632425ea6604cd8250594513e0c16bef4a85"} Oct 31 05:26:04.324066 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:04.323688 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-6554894b6b-98v2m" event={"ID":"a2ca5aff-5121-4461-b5a1-51b2e0c00455","Type":"ContainerStarted","Data":"d8a9c39376431972e7751c40d62554e0267122a143466a1b4ece89b94e1a97f5"} Oct 31 05:26:04.324066 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:04.323711 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-6554894b6b-98v2m" Oct 31 05:26:04.364174 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:04.364097 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-6554894b6b-98v2m" podStartSLOduration=2.364079077 podStartE2EDuration="2.364079077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:26:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:26:04.363502072 +0000 UTC m=+4919.119840113" watchObservedRunningTime="2026-10-31 05:26:04.364079077 +0000 UTC m=+4919.120417118" Oct 31 05:26:14.723512 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:14.723477 5019 scope.go:122] "RemoveContainer" containerID="b25753fbef87a33181e2bdc7aa2192635cbab22bae59c20c28f5530092b0891a" Oct 31 05:26:14.732551 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:14.732531 5019 scope.go:122] "RemoveContainer" containerID="9834ac8795f6a278d2c53cd862a02c3488739e27b55717b16fecdef0d3526517" Oct 31 05:26:15.333179 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:15.333148 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-6554894b6b-98v2m" Oct 31 05:26:22.389620 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:26:22.389580 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:26:31.551515 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:31.551480 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-5bd7d6d88-dth2t"] Oct 31 05:26:31.555315 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:31.555296 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-5bd7d6d88-dth2t" Oct 31 05:26:31.574441 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:31.574415 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-5bd7d6d88-dth2t"] Oct 31 05:26:31.739139 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:31.739093 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfklf\" (UniqueName: \"kubernetes.io/projected/db04abaa-c7e6-46c2-86d1-edb4e4b71862-kube-api-access-rfklf\") pod \"payload-pre-processing-5bd7d6d88-dth2t\" (UID: \"db04abaa-c7e6-46c2-86d1-edb4e4b71862\") " pod="openshift-ingress/payload-pre-processing-5bd7d6d88-dth2t" Oct 31 05:26:31.739351 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:31.739245 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db04abaa-c7e6-46c2-86d1-edb4e4b71862-plugins-config-volume\") pod \"payload-pre-processing-5bd7d6d88-dth2t\" (UID: \"db04abaa-c7e6-46c2-86d1-edb4e4b71862\") " pod="openshift-ingress/payload-pre-processing-5bd7d6d88-dth2t" Oct 31 05:26:31.840617 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:31.840544 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db04abaa-c7e6-46c2-86d1-edb4e4b71862-plugins-config-volume\") pod \"payload-pre-processing-5bd7d6d88-dth2t\" (UID: \"db04abaa-c7e6-46c2-86d1-edb4e4b71862\") " pod="openshift-ingress/payload-pre-processing-5bd7d6d88-dth2t" Oct 31 05:26:31.840748 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:31.840692 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfklf\" (UniqueName: \"kubernetes.io/projected/db04abaa-c7e6-46c2-86d1-edb4e4b71862-kube-api-access-rfklf\") pod \"payload-pre-processing-5bd7d6d88-dth2t\" (UID: \"db04abaa-c7e6-46c2-86d1-edb4e4b71862\") " pod="openshift-ingress/payload-pre-processing-5bd7d6d88-dth2t" Oct 31 05:26:31.841527 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:31.841509 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db04abaa-c7e6-46c2-86d1-edb4e4b71862-plugins-config-volume\") pod \"payload-pre-processing-5bd7d6d88-dth2t\" (UID: \"db04abaa-c7e6-46c2-86d1-edb4e4b71862\") " pod="openshift-ingress/payload-pre-processing-5bd7d6d88-dth2t" Oct 31 05:26:31.863987 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:31.863958 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfklf\" (UniqueName: \"kubernetes.io/projected/db04abaa-c7e6-46c2-86d1-edb4e4b71862-kube-api-access-rfklf\") pod \"payload-pre-processing-5bd7d6d88-dth2t\" (UID: \"db04abaa-c7e6-46c2-86d1-edb4e4b71862\") " pod="openshift-ingress/payload-pre-processing-5bd7d6d88-dth2t" Oct 31 05:26:31.868758 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:31.868738 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-5bd7d6d88-dth2t" Oct 31 05:26:32.172915 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:32.172889 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-5bd7d6d88-dth2t"] Oct 31 05:26:32.174637 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:26:32.174606 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb04abaa_c7e6_46c2_86d1_edb4e4b71862.slice/crio-ccbf4b491c35c5283e61ceb6197ff7efe1e868e69fede1ea83c325fd8815eb18 WatchSource:0}: Error finding container ccbf4b491c35c5283e61ceb6197ff7efe1e868e69fede1ea83c325fd8815eb18: Status 404 returned error can't find the container with id ccbf4b491c35c5283e61ceb6197ff7efe1e868e69fede1ea83c325fd8815eb18 Oct 31 05:26:32.481716 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:32.481686 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-5bd7d6d88-dth2t" event={"ID":"db04abaa-c7e6-46c2-86d1-edb4e4b71862","Type":"ContainerStarted","Data":"d36e1a1bbc919d1cfaa2905c9dda5c004db41d41fa0a621956dbe0fb9fe8beed"} Oct 31 05:26:32.481918 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:32.481735 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-5bd7d6d88-dth2t" event={"ID":"db04abaa-c7e6-46c2-86d1-edb4e4b71862","Type":"ContainerStarted","Data":"ccbf4b491c35c5283e61ceb6197ff7efe1e868e69fede1ea83c325fd8815eb18"} Oct 31 05:26:32.481918 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:32.481761 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-5bd7d6d88-dth2t" Oct 31 05:26:32.524182 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:32.524129 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-5bd7d6d88-dth2t" podStartSLOduration=1.524115629 podStartE2EDuration="1.524115629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:26:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:26:32.521981326 +0000 UTC m=+4947.278319367" watchObservedRunningTime="2026-10-31 05:26:32.524115629 +0000 UTC m=+4947.280453665" Oct 31 05:26:43.491064 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:43.491031 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-5bd7d6d88-dth2t" Oct 31 05:26:43.560793 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:43.560751 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-6554894b6b-98v2m"] Oct 31 05:26:48.561515 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:48.561462 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-6554894b6b-98v2m" podUID="a2ca5aff-5121-4461-b5a1-51b2e0c00455" containerName="payload-pre-processing" containerID="cri-o://580b4c6b89efc9c97df0d066987e632425ea6604cd8250594513e0c16bef4a85" gracePeriod=25 Oct 31 05:26:48.905711 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:48.905366 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-6554894b6b-98v2m" Oct 31 05:26:49.021761 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:49.021701 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2ca5aff-5121-4461-b5a1-51b2e0c00455-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2ca5aff-5121-4461-b5a1-51b2e0c00455-plugins-config-volume\") pod \"a2ca5aff-5121-4461-b5a1-51b2e0c00455\" (UID: \"a2ca5aff-5121-4461-b5a1-51b2e0c00455\") " Oct 31 05:26:49.021999 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:49.021850 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2ca5aff-5121-4461-b5a1-51b2e0c00455-kube-api-access-crzm6\" (UniqueName: \"kubernetes.io/projected/a2ca5aff-5121-4461-b5a1-51b2e0c00455-kube-api-access-crzm6\") pod \"a2ca5aff-5121-4461-b5a1-51b2e0c00455\" (UID: \"a2ca5aff-5121-4461-b5a1-51b2e0c00455\") " Oct 31 05:26:49.022248 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:49.022219 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2ca5aff-5121-4461-b5a1-51b2e0c00455-plugins-config-volume" pod "a2ca5aff-5121-4461-b5a1-51b2e0c00455" (UID: "a2ca5aff-5121-4461-b5a1-51b2e0c00455"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:26:49.024877 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:49.024839 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2ca5aff-5121-4461-b5a1-51b2e0c00455-kube-api-access-crzm6" pod "a2ca5aff-5121-4461-b5a1-51b2e0c00455" (UID: "a2ca5aff-5121-4461-b5a1-51b2e0c00455"). InnerVolumeSpecName "kube-api-access-crzm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:26:49.124051 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:49.123938 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2ca5aff-5121-4461-b5a1-51b2e0c00455-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:26:49.124051 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:49.123987 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crzm6\" (UniqueName: \"kubernetes.io/projected/a2ca5aff-5121-4461-b5a1-51b2e0c00455-kube-api-access-crzm6\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:26:49.596433 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:49.596383 5019 generic.go:356] "Generic (PLEG): container finished" podID="a2ca5aff-5121-4461-b5a1-51b2e0c00455" containerID="580b4c6b89efc9c97df0d066987e632425ea6604cd8250594513e0c16bef4a85" exitCode=0 Oct 31 05:26:49.596936 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:49.596472 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-6554894b6b-98v2m" event={"ID":"a2ca5aff-5121-4461-b5a1-51b2e0c00455","Type":"ContainerDied","Data":"580b4c6b89efc9c97df0d066987e632425ea6604cd8250594513e0c16bef4a85"} Oct 31 05:26:49.596936 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:49.596526 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-6554894b6b-98v2m" event={"ID":"a2ca5aff-5121-4461-b5a1-51b2e0c00455","Type":"ContainerDied","Data":"d8a9c39376431972e7751c40d62554e0267122a143466a1b4ece89b94e1a97f5"} Oct 31 05:26:49.596936 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:49.596557 5019 scope.go:122] "RemoveContainer" containerID="580b4c6b89efc9c97df0d066987e632425ea6604cd8250594513e0c16bef4a85" Oct 31 05:26:49.596936 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:49.596744 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-6554894b6b-98v2m" Oct 31 05:26:49.615937 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:49.615845 5019 scope.go:122] "RemoveContainer" containerID="580b4c6b89efc9c97df0d066987e632425ea6604cd8250594513e0c16bef4a85" Oct 31 05:26:49.616369 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:26:49.616286 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"580b4c6b89efc9c97df0d066987e632425ea6604cd8250594513e0c16bef4a85\": container with ID starting with 580b4c6b89efc9c97df0d066987e632425ea6604cd8250594513e0c16bef4a85 not found: ID does not exist" containerID="580b4c6b89efc9c97df0d066987e632425ea6604cd8250594513e0c16bef4a85" Oct 31 05:26:49.616369 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:49.616341 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"580b4c6b89efc9c97df0d066987e632425ea6604cd8250594513e0c16bef4a85"} err="failed to get container status \"580b4c6b89efc9c97df0d066987e632425ea6604cd8250594513e0c16bef4a85\": rpc error: code = NotFound desc = could not find container \"580b4c6b89efc9c97df0d066987e632425ea6604cd8250594513e0c16bef4a85\": container with ID starting with 580b4c6b89efc9c97df0d066987e632425ea6604cd8250594513e0c16bef4a85 not found: ID does not exist" Oct 31 05:26:49.651265 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:49.651191 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-6554894b6b-98v2m"] Oct 31 05:26:49.663296 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:49.663251 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-6554894b6b-98v2m"] Oct 31 05:26:51.421657 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:51.421612 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2ca5aff-5121-4461-b5a1-51b2e0c00455" path="/var/lib/kubelet/pods/a2ca5aff-5121-4461-b5a1-51b2e0c00455/volumes" Oct 31 05:26:52.392347 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:26:52.392294 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:26:56.037841 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:56.037805 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ad68e6c-w4w9h"] Oct 31 05:26:56.049541 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:56.049503 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ad68e6c-w4w9h"] Oct 31 05:26:57.406769 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:26:57.406728 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecfe5316-7a65-452e-b75e-a813ad0a3276" path="/var/lib/kubelet/pods/ecfe5316-7a65-452e-b75e-a813ad0a3276/volumes" Oct 31 05:27:14.767580 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:27:14.767536 5019 scope.go:122] "RemoveContainer" containerID="f76bd78545ae2191475bc86b431b2f09280282e077076623c60fa5fe8443df6b" Oct 31 05:27:14.777311 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:27:14.777282 5019 scope.go:122] "RemoveContainer" containerID="74a91bad9b4198d132ea668eac7e594e60ee2d9d51657b461022547b41b67e83" Oct 31 05:27:14.786995 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:27:14.786969 5019 scope.go:122] "RemoveContainer" containerID="11e456b5c1c0ed5618e1325f57f4ea10d56977daaeb42372b4120a008f7166f5" Oct 31 05:27:14.796173 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:27:14.796150 5019 scope.go:122] "RemoveContainer" containerID="25b485f6181e097b372ef8a89c819cdb8d536c898424e59e0cfe56904270b339" Oct 31 05:27:15.029633 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:27:15.029542 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be9d2b4d-s4wb4"] Oct 31 05:27:15.041200 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:27:15.041163 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be9d2b4d-s4wb4"] Oct 31 05:27:15.406401 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:27:15.406274 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c81d39f-ba54-4401-80ed-bdd66e448fb2" path="/var/lib/kubelet/pods/5c81d39f-ba54-4401-80ed-bdd66e448fb2/volumes" Oct 31 05:27:22.391478 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:27:22.391437 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:27:52.392430 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:27:52.392389 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:28:04.951264 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:04.951216 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-zbjhj"] Oct 31 05:28:04.951809 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:04.951792 5019 cpu_manager.go:414] "containerMap: removing container" podUID="a2ca5aff-5121-4461-b5a1-51b2e0c00455" containerName="payload-pre-processing" Oct 31 05:28:04.951862 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:04.951809 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2ca5aff-5121-4461-b5a1-51b2e0c00455" containerName="payload-pre-processing" Oct 31 05:28:04.951906 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:04.951883 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2ca5aff-5121-4461-b5a1-51b2e0c00455" containerName="payload-pre-processing" Oct 31 05:28:04.955786 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:04.955757 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-zbjhj" Oct 31 05:28:04.984015 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:04.983965 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:28:04.984266 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:04.984045 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:28:04.984266 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:04.984165 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfhzn\"" Oct 31 05:28:04.984507 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:04.984472 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:28:04.986929 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:04.986894 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-zbjhj"] Oct 31 05:28:05.139595 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:05.139545 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjn74\" (UniqueName: \"kubernetes.io/projected/eb507ac0-aec2-4f6e-af5a-83fc6a3ab9b8-kube-api-access-bjn74\") pod \"maas-api-revoke-sub-e2547622-zbjhj\" (UID: \"eb507ac0-aec2-4f6e-af5a-83fc6a3ab9b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-zbjhj" Oct 31 05:28:05.139821 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:05.139629 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb507ac0-aec2-4f6e-af5a-83fc6a3ab9b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-zbjhj\" (UID: \"eb507ac0-aec2-4f6e-af5a-83fc6a3ab9b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-zbjhj" Oct 31 05:28:05.139875 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:05.139846 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb507ac0-aec2-4f6e-af5a-83fc6a3ab9b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-zbjhj\" (UID: \"eb507ac0-aec2-4f6e-af5a-83fc6a3ab9b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-zbjhj" Oct 31 05:28:05.240742 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:05.240687 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb507ac0-aec2-4f6e-af5a-83fc6a3ab9b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-zbjhj\" (UID: \"eb507ac0-aec2-4f6e-af5a-83fc6a3ab9b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-zbjhj" Oct 31 05:28:05.240964 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:05.240747 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjn74\" (UniqueName: \"kubernetes.io/projected/eb507ac0-aec2-4f6e-af5a-83fc6a3ab9b8-kube-api-access-bjn74\") pod \"maas-api-revoke-sub-e2547622-zbjhj\" (UID: \"eb507ac0-aec2-4f6e-af5a-83fc6a3ab9b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-zbjhj" Oct 31 05:28:05.240964 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:05.240788 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb507ac0-aec2-4f6e-af5a-83fc6a3ab9b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-zbjhj\" (UID: \"eb507ac0-aec2-4f6e-af5a-83fc6a3ab9b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-zbjhj" Oct 31 05:28:05.241608 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:05.241580 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb507ac0-aec2-4f6e-af5a-83fc6a3ab9b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-zbjhj\" (UID: \"eb507ac0-aec2-4f6e-af5a-83fc6a3ab9b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-zbjhj" Oct 31 05:28:05.243422 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:05.243393 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb507ac0-aec2-4f6e-af5a-83fc6a3ab9b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-zbjhj\" (UID: \"eb507ac0-aec2-4f6e-af5a-83fc6a3ab9b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-zbjhj" Oct 31 05:28:05.265234 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:05.265194 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjn74\" (UniqueName: \"kubernetes.io/projected/eb507ac0-aec2-4f6e-af5a-83fc6a3ab9b8-kube-api-access-bjn74\") pod \"maas-api-revoke-sub-e2547622-zbjhj\" (UID: \"eb507ac0-aec2-4f6e-af5a-83fc6a3ab9b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-zbjhj" Oct 31 05:28:05.271139 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:05.271110 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-zbjhj" Oct 31 05:28:05.577634 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:05.577598 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-zbjhj"] Oct 31 05:28:05.579861 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:28:05.579827 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb507ac0_aec2_4f6e_af5a_83fc6a3ab9b8.slice/crio-916e674387d88e914ae1f9c1315a3663f6f19fce978cd608fb63520e36f14500 WatchSource:0}: Error finding container 916e674387d88e914ae1f9c1315a3663f6f19fce978cd608fb63520e36f14500: Status 404 returned error can't find the container with id 916e674387d88e914ae1f9c1315a3663f6f19fce978cd608fb63520e36f14500 Oct 31 05:28:06.072734 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:06.072697 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-zbjhj" event={"ID":"eb507ac0-aec2-4f6e-af5a-83fc6a3ab9b8","Type":"ContainerStarted","Data":"717b25ad4b234b253c42da2aa531420624165056ba52dc1dfc03c564df713465"} Oct 31 05:28:06.073234 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:06.072752 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-zbjhj" event={"ID":"eb507ac0-aec2-4f6e-af5a-83fc6a3ab9b8","Type":"ContainerStarted","Data":"916e674387d88e914ae1f9c1315a3663f6f19fce978cd608fb63520e36f14500"} Oct 31 05:28:06.115076 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:06.115014 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-zbjhj" podStartSLOduration=2.11499782 podStartE2EDuration="2.11499782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:28:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:28:06.112913639 +0000 UTC m=+5040.869251682" watchObservedRunningTime="2026-10-31 05:28:06.11499782 +0000 UTC m=+5040.871335859" Oct 31 05:28:07.080952 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:07.080908 5019 generic.go:356] "Generic (PLEG): container finished" podID="eb507ac0-aec2-4f6e-af5a-83fc6a3ab9b8" containerID="717b25ad4b234b253c42da2aa531420624165056ba52dc1dfc03c564df713465" exitCode=0 Oct 31 05:28:07.081419 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:07.080994 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-zbjhj" event={"ID":"eb507ac0-aec2-4f6e-af5a-83fc6a3ab9b8","Type":"ContainerDied","Data":"717b25ad4b234b253c42da2aa531420624165056ba52dc1dfc03c564df713465"} Oct 31 05:28:08.238787 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:08.238751 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-zbjhj" Oct 31 05:28:08.371426 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:08.371281 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb507ac0-aec2-4f6e-af5a-83fc6a3ab9b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb507ac0-aec2-4f6e-af5a-83fc6a3ab9b8-maas-api-xks-ca\") pod \"eb507ac0-aec2-4f6e-af5a-83fc6a3ab9b8\" (UID: \"eb507ac0-aec2-4f6e-af5a-83fc6a3ab9b8\") " Oct 31 05:28:08.371639 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:08.371488 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb507ac0-aec2-4f6e-af5a-83fc6a3ab9b8-kube-api-access-bjn74\" (UniqueName: \"kubernetes.io/projected/eb507ac0-aec2-4f6e-af5a-83fc6a3ab9b8-kube-api-access-bjn74\") pod \"eb507ac0-aec2-4f6e-af5a-83fc6a3ab9b8\" (UID: \"eb507ac0-aec2-4f6e-af5a-83fc6a3ab9b8\") " Oct 31 05:28:08.371639 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:08.371562 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb507ac0-aec2-4f6e-af5a-83fc6a3ab9b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb507ac0-aec2-4f6e-af5a-83fc6a3ab9b8-maas-api-openshift-ca\") pod \"eb507ac0-aec2-4f6e-af5a-83fc6a3ab9b8\" (UID: \"eb507ac0-aec2-4f6e-af5a-83fc6a3ab9b8\") " Oct 31 05:28:08.371946 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:08.371915 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb507ac0-aec2-4f6e-af5a-83fc6a3ab9b8-maas-api-openshift-ca" pod "eb507ac0-aec2-4f6e-af5a-83fc6a3ab9b8" (UID: "eb507ac0-aec2-4f6e-af5a-83fc6a3ab9b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:28:08.373545 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:08.373519 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb507ac0-aec2-4f6e-af5a-83fc6a3ab9b8-maas-api-xks-ca" pod "eb507ac0-aec2-4f6e-af5a-83fc6a3ab9b8" (UID: "eb507ac0-aec2-4f6e-af5a-83fc6a3ab9b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:28:08.373673 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:08.373658 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb507ac0-aec2-4f6e-af5a-83fc6a3ab9b8-kube-api-access-bjn74" pod "eb507ac0-aec2-4f6e-af5a-83fc6a3ab9b8" (UID: "eb507ac0-aec2-4f6e-af5a-83fc6a3ab9b8"). InnerVolumeSpecName "kube-api-access-bjn74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:28:08.473178 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:08.473124 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb507ac0-aec2-4f6e-af5a-83fc6a3ab9b8-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:28:08.473178 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:08.473172 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb507ac0-aec2-4f6e-af5a-83fc6a3ab9b8-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:28:08.473178 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:08.473185 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjn74\" (UniqueName: \"kubernetes.io/projected/eb507ac0-aec2-4f6e-af5a-83fc6a3ab9b8-kube-api-access-bjn74\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:28:09.094482 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:09.094442 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-zbjhj" Oct 31 05:28:09.094678 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:09.094447 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-zbjhj" event={"ID":"eb507ac0-aec2-4f6e-af5a-83fc6a3ab9b8","Type":"ContainerDied","Data":"916e674387d88e914ae1f9c1315a3663f6f19fce978cd608fb63520e36f14500"} Oct 31 05:28:09.094678 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:09.094608 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="916e674387d88e914ae1f9c1315a3663f6f19fce978cd608fb63520e36f14500" Oct 31 05:28:14.841379 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:14.841311 5019 scope.go:122] "RemoveContainer" containerID="5ef23476d9a0c1a9f4835d7532412deb0a5956a16ef0c7d8be75084b28e8b18f" Oct 31 05:28:14.851106 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:14.851079 5019 scope.go:122] "RemoveContainer" containerID="b2028df90f0b44957a05308047f58b890daef75b8137b54c498963c98cda72f8" Oct 31 05:28:17.027151 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:17.027111 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05fee954-7r5zh"] Oct 31 05:28:17.038999 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:17.038959 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05fee954-7r5zh"] Oct 31 05:28:17.409224 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:17.409133 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6b22fd0-53dd-4de4-9cc5-272f6abc02ab" path="/var/lib/kubelet/pods/d6b22fd0-53dd-4de4-9cc5-272f6abc02ab/volumes" Oct 31 05:28:22.393627 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:28:22.393581 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:28:29.563712 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:29.563666 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rlthw"] Oct 31 05:28:29.564201 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:29.564185 5019 cpu_manager.go:414] "containerMap: removing container" podUID="eb507ac0-aec2-4f6e-af5a-83fc6a3ab9b8" containerName="revoke-keys" Oct 31 05:28:29.564256 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:29.564203 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb507ac0-aec2-4f6e-af5a-83fc6a3ab9b8" containerName="revoke-keys" Oct 31 05:28:29.564321 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:29.564310 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb507ac0-aec2-4f6e-af5a-83fc6a3ab9b8" containerName="revoke-keys" Oct 31 05:28:29.567859 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:29.567828 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rlthw" Oct 31 05:28:29.596027 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:29.595989 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:28:29.596027 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:29.595989 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:28:29.596293 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:29.596175 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfhzn\"" Oct 31 05:28:29.596421 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:29.596398 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:28:29.600117 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:29.599795 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rlthw"] Oct 31 05:28:29.667820 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:29.667777 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5c796ae-478c-4b61-886c-63ff4532433e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-rlthw\" (UID: \"a5c796ae-478c-4b61-886c-63ff4532433e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rlthw" Oct 31 05:28:29.668039 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:29.667915 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsjm2\" (UniqueName: \"kubernetes.io/projected/a5c796ae-478c-4b61-886c-63ff4532433e-kube-api-access-qsjm2\") pod \"maas-api-revoke-sub-c1e271a9-rlthw\" (UID: \"a5c796ae-478c-4b61-886c-63ff4532433e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rlthw" Oct 31 05:28:29.668039 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:29.667974 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5c796ae-478c-4b61-886c-63ff4532433e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-rlthw\" (UID: \"a5c796ae-478c-4b61-886c-63ff4532433e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rlthw" Oct 31 05:28:29.769370 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:29.769298 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsjm2\" (UniqueName: \"kubernetes.io/projected/a5c796ae-478c-4b61-886c-63ff4532433e-kube-api-access-qsjm2\") pod \"maas-api-revoke-sub-c1e271a9-rlthw\" (UID: \"a5c796ae-478c-4b61-886c-63ff4532433e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rlthw" Oct 31 05:28:29.769635 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:29.769399 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5c796ae-478c-4b61-886c-63ff4532433e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-rlthw\" (UID: \"a5c796ae-478c-4b61-886c-63ff4532433e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rlthw" Oct 31 05:28:29.769635 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:29.769456 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5c796ae-478c-4b61-886c-63ff4532433e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-rlthw\" (UID: \"a5c796ae-478c-4b61-886c-63ff4532433e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rlthw" Oct 31 05:28:29.770159 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:29.770126 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5c796ae-478c-4b61-886c-63ff4532433e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-rlthw\" (UID: \"a5c796ae-478c-4b61-886c-63ff4532433e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rlthw" Oct 31 05:28:29.772068 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:29.772042 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5c796ae-478c-4b61-886c-63ff4532433e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-rlthw\" (UID: \"a5c796ae-478c-4b61-886c-63ff4532433e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rlthw" Oct 31 05:28:29.792712 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:29.792676 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsjm2\" (UniqueName: \"kubernetes.io/projected/a5c796ae-478c-4b61-886c-63ff4532433e-kube-api-access-qsjm2\") pod \"maas-api-revoke-sub-c1e271a9-rlthw\" (UID: \"a5c796ae-478c-4b61-886c-63ff4532433e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rlthw" Oct 31 05:28:29.884305 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:29.884202 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rlthw" Oct 31 05:28:30.186369 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:30.186306 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rlthw"] Oct 31 05:28:30.188269 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:28:30.188228 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5c796ae_478c_4b61_886c_63ff4532433e.slice/crio-74450b4c497990d5e231fca96954bc47388e807a7bc2393e614f9c8cfb06d1a2 WatchSource:0}: Error finding container 74450b4c497990d5e231fca96954bc47388e807a7bc2393e614f9c8cfb06d1a2: Status 404 returned error can't find the container with id 74450b4c497990d5e231fca96954bc47388e807a7bc2393e614f9c8cfb06d1a2 Oct 31 05:28:30.329692 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:30.329648 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rlthw" event={"ID":"a5c796ae-478c-4b61-886c-63ff4532433e","Type":"ContainerStarted","Data":"74450b4c497990d5e231fca96954bc47388e807a7bc2393e614f9c8cfb06d1a2"} Oct 31 05:28:31.337365 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:31.337234 5019 generic.go:356] "Generic (PLEG): container finished" podID="a5c796ae-478c-4b61-886c-63ff4532433e" containerID="5a6853750dfe00a9ccc5305010e19eb65da299f4e6445a3a6acfc0429ea05a6a" exitCode=0 Oct 31 05:28:31.337365 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:31.337317 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rlthw" event={"ID":"a5c796ae-478c-4b61-886c-63ff4532433e","Type":"ContainerDied","Data":"5a6853750dfe00a9ccc5305010e19eb65da299f4e6445a3a6acfc0429ea05a6a"} Oct 31 05:28:32.494262 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:32.494226 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rlthw" Oct 31 05:28:32.599495 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:32.599407 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5c796ae-478c-4b61-886c-63ff4532433e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5c796ae-478c-4b61-886c-63ff4532433e-maas-api-xks-ca\") pod \"a5c796ae-478c-4b61-886c-63ff4532433e\" (UID: \"a5c796ae-478c-4b61-886c-63ff4532433e\") " Oct 31 05:28:32.599495 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:32.599477 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5c796ae-478c-4b61-886c-63ff4532433e-kube-api-access-qsjm2\" (UniqueName: \"kubernetes.io/projected/a5c796ae-478c-4b61-886c-63ff4532433e-kube-api-access-qsjm2\") pod \"a5c796ae-478c-4b61-886c-63ff4532433e\" (UID: \"a5c796ae-478c-4b61-886c-63ff4532433e\") " Oct 31 05:28:32.599727 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:32.599524 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5c796ae-478c-4b61-886c-63ff4532433e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5c796ae-478c-4b61-886c-63ff4532433e-maas-api-openshift-ca\") pod \"a5c796ae-478c-4b61-886c-63ff4532433e\" (UID: \"a5c796ae-478c-4b61-886c-63ff4532433e\") " Oct 31 05:28:32.599917 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:32.599889 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5c796ae-478c-4b61-886c-63ff4532433e-maas-api-openshift-ca" pod "a5c796ae-478c-4b61-886c-63ff4532433e" (UID: "a5c796ae-478c-4b61-886c-63ff4532433e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:28:32.601649 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:32.601623 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5c796ae-478c-4b61-886c-63ff4532433e-maas-api-xks-ca" pod "a5c796ae-478c-4b61-886c-63ff4532433e" (UID: "a5c796ae-478c-4b61-886c-63ff4532433e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:28:32.601864 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:32.601848 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5c796ae-478c-4b61-886c-63ff4532433e-kube-api-access-qsjm2" pod "a5c796ae-478c-4b61-886c-63ff4532433e" (UID: "a5c796ae-478c-4b61-886c-63ff4532433e"). InnerVolumeSpecName "kube-api-access-qsjm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:28:32.700813 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:32.700768 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5c796ae-478c-4b61-886c-63ff4532433e-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:28:32.700813 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:32.700807 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsjm2\" (UniqueName: \"kubernetes.io/projected/a5c796ae-478c-4b61-886c-63ff4532433e-kube-api-access-qsjm2\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:28:32.700813 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:32.700817 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5c796ae-478c-4b61-886c-63ff4532433e-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:28:33.350150 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:33.350116 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rlthw" Oct 31 05:28:33.350150 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:33.350139 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rlthw" event={"ID":"a5c796ae-478c-4b61-886c-63ff4532433e","Type":"ContainerDied","Data":"74450b4c497990d5e231fca96954bc47388e807a7bc2393e614f9c8cfb06d1a2"} Oct 31 05:28:33.350400 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:33.350192 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74450b4c497990d5e231fca96954bc47388e807a7bc2393e614f9c8cfb06d1a2" Oct 31 05:28:49.805046 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:49.804953 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hrlmq"] Oct 31 05:28:49.805583 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:49.805447 5019 cpu_manager.go:414] "containerMap: removing container" podUID="a5c796ae-478c-4b61-886c-63ff4532433e" containerName="revoke-keys" Oct 31 05:28:49.805583 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:49.805459 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5c796ae-478c-4b61-886c-63ff4532433e" containerName="revoke-keys" Oct 31 05:28:49.805583 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:49.805544 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5c796ae-478c-4b61-886c-63ff4532433e" containerName="revoke-keys" Oct 31 05:28:49.808845 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:49.808824 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hrlmq" Oct 31 05:28:49.824548 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:49.824236 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:28:49.836495 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:49.836463 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:28:49.837190 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:49.836538 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:28:49.837428 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:49.837403 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfhzn\"" Oct 31 05:28:49.840996 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:49.840966 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hrlmq"] Oct 31 05:28:49.864086 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:49.864031 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ec58d93-ce49-4c53-a165-ba08c5439b41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-hrlmq\" (UID: \"2ec58d93-ce49-4c53-a165-ba08c5439b41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hrlmq" Oct 31 05:28:49.864316 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:49.864207 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ec58d93-ce49-4c53-a165-ba08c5439b41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-hrlmq\" (UID: \"2ec58d93-ce49-4c53-a165-ba08c5439b41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hrlmq" Oct 31 05:28:49.864460 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:49.864403 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-592rm\" (UniqueName: \"kubernetes.io/projected/2ec58d93-ce49-4c53-a165-ba08c5439b41-kube-api-access-592rm\") pod \"maas-api-revoke-sub-2e1f5b5f-hrlmq\" (UID: \"2ec58d93-ce49-4c53-a165-ba08c5439b41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hrlmq" Oct 31 05:28:49.965157 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:49.965107 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-592rm\" (UniqueName: \"kubernetes.io/projected/2ec58d93-ce49-4c53-a165-ba08c5439b41-kube-api-access-592rm\") pod \"maas-api-revoke-sub-2e1f5b5f-hrlmq\" (UID: \"2ec58d93-ce49-4c53-a165-ba08c5439b41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hrlmq" Oct 31 05:28:49.965437 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:49.965200 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ec58d93-ce49-4c53-a165-ba08c5439b41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-hrlmq\" (UID: \"2ec58d93-ce49-4c53-a165-ba08c5439b41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hrlmq" Oct 31 05:28:49.965437 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:49.965249 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ec58d93-ce49-4c53-a165-ba08c5439b41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-hrlmq\" (UID: \"2ec58d93-ce49-4c53-a165-ba08c5439b41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hrlmq" Oct 31 05:28:49.966068 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:49.966041 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ec58d93-ce49-4c53-a165-ba08c5439b41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-hrlmq\" (UID: \"2ec58d93-ce49-4c53-a165-ba08c5439b41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hrlmq" Oct 31 05:28:49.967794 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:49.967767 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ec58d93-ce49-4c53-a165-ba08c5439b41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-hrlmq\" (UID: \"2ec58d93-ce49-4c53-a165-ba08c5439b41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hrlmq" Oct 31 05:28:49.988604 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:49.988568 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-592rm\" (UniqueName: \"kubernetes.io/projected/2ec58d93-ce49-4c53-a165-ba08c5439b41-kube-api-access-592rm\") pod \"maas-api-revoke-sub-2e1f5b5f-hrlmq\" (UID: \"2ec58d93-ce49-4c53-a165-ba08c5439b41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hrlmq" Oct 31 05:28:50.125364 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:50.125240 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hrlmq" Oct 31 05:28:50.428634 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:50.428602 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hrlmq"] Oct 31 05:28:50.430484 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:28:50.430442 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ec58d93_ce49_4c53_a165_ba08c5439b41.slice/crio-d682c3bc8506f9ae9180ded47523f5407105cd634ac064fe0de516ae2f781958 WatchSource:0}: Error finding container d682c3bc8506f9ae9180ded47523f5407105cd634ac064fe0de516ae2f781958: Status 404 returned error can't find the container with id d682c3bc8506f9ae9180ded47523f5407105cd634ac064fe0de516ae2f781958 Oct 31 05:28:50.496135 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:50.496097 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hrlmq" event={"ID":"2ec58d93-ce49-4c53-a165-ba08c5439b41","Type":"ContainerStarted","Data":"d682c3bc8506f9ae9180ded47523f5407105cd634ac064fe0de516ae2f781958"} Oct 31 05:28:51.503428 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:51.503392 5019 generic.go:356] "Generic (PLEG): container finished" podID="2ec58d93-ce49-4c53-a165-ba08c5439b41" containerID="3c2f98146ce8425b145e8db54e5f32823a454ec1e0838fb1ad8dc53c6b0e0c36" exitCode=0 Oct 31 05:28:51.503863 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:51.503475 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hrlmq" event={"ID":"2ec58d93-ce49-4c53-a165-ba08c5439b41","Type":"ContainerDied","Data":"3c2f98146ce8425b145e8db54e5f32823a454ec1e0838fb1ad8dc53c6b0e0c36"} Oct 31 05:28:52.392106 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:28:52.392064 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:28:52.662220 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:52.662189 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hrlmq" Oct 31 05:28:52.688199 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:52.688154 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ec58d93-ce49-4c53-a165-ba08c5439b41-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ec58d93-ce49-4c53-a165-ba08c5439b41-maas-api-openshift-ca\") pod \"2ec58d93-ce49-4c53-a165-ba08c5439b41\" (UID: \"2ec58d93-ce49-4c53-a165-ba08c5439b41\") " Oct 31 05:28:52.688409 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:52.688277 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ec58d93-ce49-4c53-a165-ba08c5439b41-kube-api-access-592rm\" (UniqueName: \"kubernetes.io/projected/2ec58d93-ce49-4c53-a165-ba08c5439b41-kube-api-access-592rm\") pod \"2ec58d93-ce49-4c53-a165-ba08c5439b41\" (UID: \"2ec58d93-ce49-4c53-a165-ba08c5439b41\") " Oct 31 05:28:52.688409 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:52.688388 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ec58d93-ce49-4c53-a165-ba08c5439b41-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ec58d93-ce49-4c53-a165-ba08c5439b41-maas-api-xks-ca\") pod \"2ec58d93-ce49-4c53-a165-ba08c5439b41\" (UID: \"2ec58d93-ce49-4c53-a165-ba08c5439b41\") " Oct 31 05:28:52.688679 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:52.688652 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ec58d93-ce49-4c53-a165-ba08c5439b41-maas-api-openshift-ca" pod "2ec58d93-ce49-4c53-a165-ba08c5439b41" (UID: "2ec58d93-ce49-4c53-a165-ba08c5439b41"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:28:52.690714 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:52.690678 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ec58d93-ce49-4c53-a165-ba08c5439b41-maas-api-xks-ca" pod "2ec58d93-ce49-4c53-a165-ba08c5439b41" (UID: "2ec58d93-ce49-4c53-a165-ba08c5439b41"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:28:52.690864 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:52.690740 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ec58d93-ce49-4c53-a165-ba08c5439b41-kube-api-access-592rm" pod "2ec58d93-ce49-4c53-a165-ba08c5439b41" (UID: "2ec58d93-ce49-4c53-a165-ba08c5439b41"). InnerVolumeSpecName "kube-api-access-592rm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:28:52.789809 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:52.789761 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-592rm\" (UniqueName: \"kubernetes.io/projected/2ec58d93-ce49-4c53-a165-ba08c5439b41-kube-api-access-592rm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:28:52.789809 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:52.789805 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ec58d93-ce49-4c53-a165-ba08c5439b41-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:28:52.789809 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:52.789817 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ec58d93-ce49-4c53-a165-ba08c5439b41-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:28:53.517201 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:53.517169 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hrlmq" Oct 31 05:28:53.517201 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:53.517186 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hrlmq" event={"ID":"2ec58d93-ce49-4c53-a165-ba08c5439b41","Type":"ContainerDied","Data":"d682c3bc8506f9ae9180ded47523f5407105cd634ac064fe0de516ae2f781958"} Oct 31 05:28:53.517462 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:28:53.517246 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d682c3bc8506f9ae9180ded47523f5407105cd634ac064fe0de516ae2f781958" Oct 31 05:29:01.099488 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:01.099447 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xqj88"] Oct 31 05:29:01.099997 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:01.099980 5019 cpu_manager.go:414] "containerMap: removing container" podUID="2ec58d93-ce49-4c53-a165-ba08c5439b41" containerName="revoke-keys" Oct 31 05:29:01.100045 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:01.100000 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ec58d93-ce49-4c53-a165-ba08c5439b41" containerName="revoke-keys" Oct 31 05:29:01.100104 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:01.100094 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ec58d93-ce49-4c53-a165-ba08c5439b41" containerName="revoke-keys" Oct 31 05:29:01.103783 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:01.103752 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xqj88" Oct 31 05:29:01.125147 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:01.124488 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xqj88"] Oct 31 05:29:01.136088 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:01.136048 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:29:01.136088 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:01.136083 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:29:01.136401 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:01.136051 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:29:01.136587 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:01.136568 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfhzn\"" Oct 31 05:29:01.175969 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:01.175931 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88b92fcd-a647-48a5-a06f-1767d73a51c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-xqj88\" (UID: \"88b92fcd-a647-48a5-a06f-1767d73a51c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xqj88" Oct 31 05:29:01.175969 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:01.175979 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88b92fcd-a647-48a5-a06f-1767d73a51c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-xqj88\" (UID: \"88b92fcd-a647-48a5-a06f-1767d73a51c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xqj88" Oct 31 05:29:01.176247 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:01.176085 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrd5t\" (UniqueName: \"kubernetes.io/projected/88b92fcd-a647-48a5-a06f-1767d73a51c3-kube-api-access-nrd5t\") pod \"maas-api-revoke-sub-ec843018-xqj88\" (UID: \"88b92fcd-a647-48a5-a06f-1767d73a51c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xqj88" Oct 31 05:29:01.277041 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:01.276995 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88b92fcd-a647-48a5-a06f-1767d73a51c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-xqj88\" (UID: \"88b92fcd-a647-48a5-a06f-1767d73a51c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xqj88" Oct 31 05:29:01.277041 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:01.277047 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88b92fcd-a647-48a5-a06f-1767d73a51c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-xqj88\" (UID: \"88b92fcd-a647-48a5-a06f-1767d73a51c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xqj88" Oct 31 05:29:01.277285 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:01.277086 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrd5t\" (UniqueName: \"kubernetes.io/projected/88b92fcd-a647-48a5-a06f-1767d73a51c3-kube-api-access-nrd5t\") pod \"maas-api-revoke-sub-ec843018-xqj88\" (UID: \"88b92fcd-a647-48a5-a06f-1767d73a51c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xqj88" Oct 31 05:29:01.277769 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:01.277744 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88b92fcd-a647-48a5-a06f-1767d73a51c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-xqj88\" (UID: \"88b92fcd-a647-48a5-a06f-1767d73a51c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xqj88" Oct 31 05:29:01.279744 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:01.279707 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88b92fcd-a647-48a5-a06f-1767d73a51c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-xqj88\" (UID: \"88b92fcd-a647-48a5-a06f-1767d73a51c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xqj88" Oct 31 05:29:01.300594 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:01.300558 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrd5t\" (UniqueName: \"kubernetes.io/projected/88b92fcd-a647-48a5-a06f-1767d73a51c3-kube-api-access-nrd5t\") pod \"maas-api-revoke-sub-ec843018-xqj88\" (UID: \"88b92fcd-a647-48a5-a06f-1767d73a51c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xqj88" Oct 31 05:29:01.421963 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:01.421863 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xqj88" Oct 31 05:29:01.731558 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:01.731523 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xqj88"] Oct 31 05:29:01.733897 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:29:01.733861 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88b92fcd_a647_48a5_a06f_1767d73a51c3.slice/crio-131568ed0ba03712c1a666a694c5760bdec6b739b6d604dc698be648c212ca90 WatchSource:0}: Error finding container 131568ed0ba03712c1a666a694c5760bdec6b739b6d604dc698be648c212ca90: Status 404 returned error can't find the container with id 131568ed0ba03712c1a666a694c5760bdec6b739b6d604dc698be648c212ca90 Oct 31 05:29:02.580472 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:02.580427 5019 generic.go:356] "Generic (PLEG): container finished" podID="88b92fcd-a647-48a5-a06f-1767d73a51c3" containerID="360702bfefd49e63d9b99ae2c8921f3f35e229e636392c9dec12c071715f13b6" exitCode=0 Oct 31 05:29:02.580903 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:02.580516 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xqj88" event={"ID":"88b92fcd-a647-48a5-a06f-1767d73a51c3","Type":"ContainerDied","Data":"360702bfefd49e63d9b99ae2c8921f3f35e229e636392c9dec12c071715f13b6"} Oct 31 05:29:02.580903 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:02.580575 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xqj88" event={"ID":"88b92fcd-a647-48a5-a06f-1767d73a51c3","Type":"ContainerStarted","Data":"131568ed0ba03712c1a666a694c5760bdec6b739b6d604dc698be648c212ca90"} Oct 31 05:29:03.748967 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:03.748933 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xqj88" Oct 31 05:29:03.807819 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:03.807769 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88b92fcd-a647-48a5-a06f-1767d73a51c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88b92fcd-a647-48a5-a06f-1767d73a51c3-maas-api-xks-ca\") pod \"88b92fcd-a647-48a5-a06f-1767d73a51c3\" (UID: \"88b92fcd-a647-48a5-a06f-1767d73a51c3\") " Oct 31 05:29:03.808055 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:03.807962 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88b92fcd-a647-48a5-a06f-1767d73a51c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88b92fcd-a647-48a5-a06f-1767d73a51c3-maas-api-openshift-ca\") pod \"88b92fcd-a647-48a5-a06f-1767d73a51c3\" (UID: \"88b92fcd-a647-48a5-a06f-1767d73a51c3\") " Oct 31 05:29:03.808055 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:03.808030 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88b92fcd-a647-48a5-a06f-1767d73a51c3-kube-api-access-nrd5t\" (UniqueName: \"kubernetes.io/projected/88b92fcd-a647-48a5-a06f-1767d73a51c3-kube-api-access-nrd5t\") pod \"88b92fcd-a647-48a5-a06f-1767d73a51c3\" (UID: \"88b92fcd-a647-48a5-a06f-1767d73a51c3\") " Oct 31 05:29:03.810438 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:03.808817 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88b92fcd-a647-48a5-a06f-1767d73a51c3-maas-api-openshift-ca" pod "88b92fcd-a647-48a5-a06f-1767d73a51c3" (UID: "88b92fcd-a647-48a5-a06f-1767d73a51c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:29:03.815767 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:03.815711 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88b92fcd-a647-48a5-a06f-1767d73a51c3-kube-api-access-nrd5t" pod "88b92fcd-a647-48a5-a06f-1767d73a51c3" (UID: "88b92fcd-a647-48a5-a06f-1767d73a51c3"). InnerVolumeSpecName "kube-api-access-nrd5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:29:03.819156 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:03.819098 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88b92fcd-a647-48a5-a06f-1767d73a51c3-maas-api-xks-ca" pod "88b92fcd-a647-48a5-a06f-1767d73a51c3" (UID: "88b92fcd-a647-48a5-a06f-1767d73a51c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:29:03.909465 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:03.909324 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrd5t\" (UniqueName: \"kubernetes.io/projected/88b92fcd-a647-48a5-a06f-1767d73a51c3-kube-api-access-nrd5t\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:29:03.909465 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:03.909408 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88b92fcd-a647-48a5-a06f-1767d73a51c3-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:29:03.909465 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:03.909418 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88b92fcd-a647-48a5-a06f-1767d73a51c3-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:29:04.594470 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:04.594438 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xqj88" Oct 31 05:29:04.594679 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:04.594478 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xqj88" event={"ID":"88b92fcd-a647-48a5-a06f-1767d73a51c3","Type":"ContainerDied","Data":"131568ed0ba03712c1a666a694c5760bdec6b739b6d604dc698be648c212ca90"} Oct 31 05:29:04.594679 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:04.594534 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="131568ed0ba03712c1a666a694c5760bdec6b739b6d604dc698be648c212ca90" Oct 31 05:29:09.789246 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:09.789214 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s88kp_3945ff65-e877-4372-9334-3968c68b7295/console-operator/2.log" Oct 31 05:29:09.793601 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:09.793573 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s88kp_3945ff65-e877-4372-9334-3968c68b7295/console-operator/2.log" Oct 31 05:29:14.879323 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:14.879283 5019 scope.go:122] "RemoveContainer" containerID="012748f7f1b7e3e7f9c05615ff98ecded562c1eb282f1f0fbd843fa8335c43fa" Oct 31 05:29:22.391707 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:29:22.391666 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:29:29.209908 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:29.209864 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xl58n"] Oct 31 05:29:29.210449 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:29.210430 5019 cpu_manager.go:414] "containerMap: removing container" podUID="88b92fcd-a647-48a5-a06f-1767d73a51c3" containerName="revoke-keys" Oct 31 05:29:29.210518 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:29.210451 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88b92fcd-a647-48a5-a06f-1767d73a51c3" containerName="revoke-keys" Oct 31 05:29:29.210561 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:29.210555 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="88b92fcd-a647-48a5-a06f-1767d73a51c3" containerName="revoke-keys" Oct 31 05:29:29.214195 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:29.214167 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xl58n" Oct 31 05:29:29.233885 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:29.233841 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xl58n"] Oct 31 05:29:29.241704 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:29.241666 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:29:29.242011 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:29.241760 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:29:29.242116 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:29.241847 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:29:29.242116 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:29.241957 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfhzn\"" Oct 31 05:29:29.262253 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:29.262207 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf427484-81ab-4eaa-8714-8d800e8c11d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-xl58n\" (UID: \"bf427484-81ab-4eaa-8714-8d800e8c11d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xl58n" Oct 31 05:29:29.262504 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:29.262280 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf427484-81ab-4eaa-8714-8d800e8c11d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-xl58n\" (UID: \"bf427484-81ab-4eaa-8714-8d800e8c11d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xl58n" Oct 31 05:29:29.262504 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:29.262413 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b589b\" (UniqueName: \"kubernetes.io/projected/bf427484-81ab-4eaa-8714-8d800e8c11d0-kube-api-access-b589b\") pod \"maas-api-revoke-sub-046e1343-xl58n\" (UID: \"bf427484-81ab-4eaa-8714-8d800e8c11d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xl58n" Oct 31 05:29:29.363706 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:29.363655 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf427484-81ab-4eaa-8714-8d800e8c11d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-xl58n\" (UID: \"bf427484-81ab-4eaa-8714-8d800e8c11d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xl58n" Oct 31 05:29:29.363945 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:29.363742 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf427484-81ab-4eaa-8714-8d800e8c11d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-xl58n\" (UID: \"bf427484-81ab-4eaa-8714-8d800e8c11d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xl58n" Oct 31 05:29:29.363945 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:29.363773 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b589b\" (UniqueName: \"kubernetes.io/projected/bf427484-81ab-4eaa-8714-8d800e8c11d0-kube-api-access-b589b\") pod \"maas-api-revoke-sub-046e1343-xl58n\" (UID: \"bf427484-81ab-4eaa-8714-8d800e8c11d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xl58n" Oct 31 05:29:29.364455 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:29.364432 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf427484-81ab-4eaa-8714-8d800e8c11d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-xl58n\" (UID: \"bf427484-81ab-4eaa-8714-8d800e8c11d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xl58n" Oct 31 05:29:29.366299 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:29.366269 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf427484-81ab-4eaa-8714-8d800e8c11d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-xl58n\" (UID: \"bf427484-81ab-4eaa-8714-8d800e8c11d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xl58n" Oct 31 05:29:29.387708 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:29.387666 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b589b\" (UniqueName: \"kubernetes.io/projected/bf427484-81ab-4eaa-8714-8d800e8c11d0-kube-api-access-b589b\") pod \"maas-api-revoke-sub-046e1343-xl58n\" (UID: \"bf427484-81ab-4eaa-8714-8d800e8c11d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xl58n" Oct 31 05:29:29.529770 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:29.529726 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xl58n" Oct 31 05:29:29.830340 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:29.830293 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xl58n"] Oct 31 05:29:29.831558 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:29:29.831528 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf427484_81ab_4eaa_8714_8d800e8c11d0.slice/crio-43d4203f90f18158c219db32a1731e22924971f7d5e3e3df634ac209bf5e714d WatchSource:0}: Error finding container 43d4203f90f18158c219db32a1731e22924971f7d5e3e3df634ac209bf5e714d: Status 404 returned error can't find the container with id 43d4203f90f18158c219db32a1731e22924971f7d5e3e3df634ac209bf5e714d Oct 31 05:29:30.801999 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:30.801955 5019 generic.go:356] "Generic (PLEG): container finished" podID="bf427484-81ab-4eaa-8714-8d800e8c11d0" containerID="b4195590cb12e8c2d72eebbb69c002d9787166a28c49d5c62b9a74a6fc96de1c" exitCode=0 Oct 31 05:29:30.802433 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:30.802043 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xl58n" event={"ID":"bf427484-81ab-4eaa-8714-8d800e8c11d0","Type":"ContainerDied","Data":"b4195590cb12e8c2d72eebbb69c002d9787166a28c49d5c62b9a74a6fc96de1c"} Oct 31 05:29:30.802433 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:30.802102 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xl58n" event={"ID":"bf427484-81ab-4eaa-8714-8d800e8c11d0","Type":"ContainerStarted","Data":"43d4203f90f18158c219db32a1731e22924971f7d5e3e3df634ac209bf5e714d"} Oct 31 05:29:31.964816 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:31.964777 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xl58n" Oct 31 05:29:31.991085 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:31.991042 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf427484-81ab-4eaa-8714-8d800e8c11d0-kube-api-access-b589b\" (UniqueName: \"kubernetes.io/projected/bf427484-81ab-4eaa-8714-8d800e8c11d0-kube-api-access-b589b\") pod \"bf427484-81ab-4eaa-8714-8d800e8c11d0\" (UID: \"bf427484-81ab-4eaa-8714-8d800e8c11d0\") " Oct 31 05:29:31.991298 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:31.991148 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf427484-81ab-4eaa-8714-8d800e8c11d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf427484-81ab-4eaa-8714-8d800e8c11d0-maas-api-openshift-ca\") pod \"bf427484-81ab-4eaa-8714-8d800e8c11d0\" (UID: \"bf427484-81ab-4eaa-8714-8d800e8c11d0\") " Oct 31 05:29:31.991298 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:31.991228 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf427484-81ab-4eaa-8714-8d800e8c11d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf427484-81ab-4eaa-8714-8d800e8c11d0-maas-api-xks-ca\") pod \"bf427484-81ab-4eaa-8714-8d800e8c11d0\" (UID: \"bf427484-81ab-4eaa-8714-8d800e8c11d0\") " Oct 31 05:29:31.991633 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:31.991608 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf427484-81ab-4eaa-8714-8d800e8c11d0-maas-api-openshift-ca" pod "bf427484-81ab-4eaa-8714-8d800e8c11d0" (UID: "bf427484-81ab-4eaa-8714-8d800e8c11d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:29:31.993538 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:31.993506 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf427484-81ab-4eaa-8714-8d800e8c11d0-maas-api-xks-ca" pod "bf427484-81ab-4eaa-8714-8d800e8c11d0" (UID: "bf427484-81ab-4eaa-8714-8d800e8c11d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:29:31.993622 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:31.993536 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf427484-81ab-4eaa-8714-8d800e8c11d0-kube-api-access-b589b" pod "bf427484-81ab-4eaa-8714-8d800e8c11d0" (UID: "bf427484-81ab-4eaa-8714-8d800e8c11d0"). InnerVolumeSpecName "kube-api-access-b589b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:29:32.093142 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:32.093031 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf427484-81ab-4eaa-8714-8d800e8c11d0-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:29:32.093142 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:32.093083 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf427484-81ab-4eaa-8714-8d800e8c11d0-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:29:32.093142 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:32.093094 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b589b\" (UniqueName: \"kubernetes.io/projected/bf427484-81ab-4eaa-8714-8d800e8c11d0-kube-api-access-b589b\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:29:32.815005 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:32.814970 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xl58n" Oct 31 05:29:32.815202 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:32.814976 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xl58n" event={"ID":"bf427484-81ab-4eaa-8714-8d800e8c11d0","Type":"ContainerDied","Data":"43d4203f90f18158c219db32a1731e22924971f7d5e3e3df634ac209bf5e714d"} Oct 31 05:29:32.815202 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:32.815101 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43d4203f90f18158c219db32a1731e22924971f7d5e3e3df634ac209bf5e714d" Oct 31 05:29:39.656550 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:39.656510 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kwklx"] Oct 31 05:29:39.657010 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:39.656995 5019 cpu_manager.go:414] "containerMap: removing container" podUID="bf427484-81ab-4eaa-8714-8d800e8c11d0" containerName="revoke-keys" Oct 31 05:29:39.657067 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:39.657009 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf427484-81ab-4eaa-8714-8d800e8c11d0" containerName="revoke-keys" Oct 31 05:29:39.657109 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:39.657098 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf427484-81ab-4eaa-8714-8d800e8c11d0" containerName="revoke-keys" Oct 31 05:29:39.660772 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:39.660737 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kwklx" Oct 31 05:29:39.677052 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:39.677010 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:29:39.689241 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:39.689205 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfhzn\"" Oct 31 05:29:39.689535 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:39.689397 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:29:39.690687 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:39.690659 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:29:39.693902 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:39.693866 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kwklx"] Oct 31 05:29:39.768444 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:39.768393 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f76dead-50bf-4d0e-94e0-1152a518abe3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-kwklx\" (UID: \"9f76dead-50bf-4d0e-94e0-1152a518abe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kwklx" Oct 31 05:29:39.768654 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:39.768460 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxpbn\" (UniqueName: \"kubernetes.io/projected/9f76dead-50bf-4d0e-94e0-1152a518abe3-kube-api-access-xxpbn\") pod \"maas-api-revoke-sub-d3fba056-kwklx\" (UID: \"9f76dead-50bf-4d0e-94e0-1152a518abe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kwklx" Oct 31 05:29:39.768654 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:39.768544 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f76dead-50bf-4d0e-94e0-1152a518abe3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-kwklx\" (UID: \"9f76dead-50bf-4d0e-94e0-1152a518abe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kwklx" Oct 31 05:29:39.869626 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:39.869570 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f76dead-50bf-4d0e-94e0-1152a518abe3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-kwklx\" (UID: \"9f76dead-50bf-4d0e-94e0-1152a518abe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kwklx" Oct 31 05:29:39.869875 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:39.869643 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxpbn\" (UniqueName: \"kubernetes.io/projected/9f76dead-50bf-4d0e-94e0-1152a518abe3-kube-api-access-xxpbn\") pod \"maas-api-revoke-sub-d3fba056-kwklx\" (UID: \"9f76dead-50bf-4d0e-94e0-1152a518abe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kwklx" Oct 31 05:29:39.869875 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:39.869691 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f76dead-50bf-4d0e-94e0-1152a518abe3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-kwklx\" (UID: \"9f76dead-50bf-4d0e-94e0-1152a518abe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kwklx" Oct 31 05:29:39.870418 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:39.870390 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f76dead-50bf-4d0e-94e0-1152a518abe3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-kwklx\" (UID: \"9f76dead-50bf-4d0e-94e0-1152a518abe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kwklx" Oct 31 05:29:39.872374 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:39.872322 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f76dead-50bf-4d0e-94e0-1152a518abe3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-kwklx\" (UID: \"9f76dead-50bf-4d0e-94e0-1152a518abe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kwklx" Oct 31 05:29:39.893224 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:39.893193 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxpbn\" (UniqueName: \"kubernetes.io/projected/9f76dead-50bf-4d0e-94e0-1152a518abe3-kube-api-access-xxpbn\") pod \"maas-api-revoke-sub-d3fba056-kwklx\" (UID: \"9f76dead-50bf-4d0e-94e0-1152a518abe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kwklx" Oct 31 05:29:39.976134 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:39.976090 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kwklx" Oct 31 05:29:40.282616 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:40.282580 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kwklx"] Oct 31 05:29:40.285516 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:29:40.285478 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f76dead_50bf_4d0e_94e0_1152a518abe3.slice/crio-4e87510b76391c107763ae3ef390d974b684b26c222ddb9e38627dd391d6e5d9 WatchSource:0}: Error finding container 4e87510b76391c107763ae3ef390d974b684b26c222ddb9e38627dd391d6e5d9: Status 404 returned error can't find the container with id 4e87510b76391c107763ae3ef390d974b684b26c222ddb9e38627dd391d6e5d9 Oct 31 05:29:40.865282 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:40.865153 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kwklx" event={"ID":"9f76dead-50bf-4d0e-94e0-1152a518abe3","Type":"ContainerStarted","Data":"37d8791d56ca4f417147d97b85c04fc2231dec515b2d8d90d5491f372835e49c"} Oct 31 05:29:40.865282 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:40.865231 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kwklx" event={"ID":"9f76dead-50bf-4d0e-94e0-1152a518abe3","Type":"ContainerStarted","Data":"4e87510b76391c107763ae3ef390d974b684b26c222ddb9e38627dd391d6e5d9"} Oct 31 05:29:40.905848 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:40.905781 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kwklx" podStartSLOduration=1.905764933 podStartE2EDuration="1.905764933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:29:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:29:40.904684897 +0000 UTC m=+5135.661022939" watchObservedRunningTime="2026-10-31 05:29:40.905764933 +0000 UTC m=+5135.662103031" Oct 31 05:29:41.881420 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:41.880807 5019 generic.go:356] "Generic (PLEG): container finished" podID="9f76dead-50bf-4d0e-94e0-1152a518abe3" containerID="37d8791d56ca4f417147d97b85c04fc2231dec515b2d8d90d5491f372835e49c" exitCode=0 Oct 31 05:29:41.881420 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:41.881000 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kwklx" event={"ID":"9f76dead-50bf-4d0e-94e0-1152a518abe3","Type":"ContainerDied","Data":"37d8791d56ca4f417147d97b85c04fc2231dec515b2d8d90d5491f372835e49c"} Oct 31 05:29:43.817892 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:43.817841 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kwklx" Oct 31 05:29:43.949434 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:43.940237 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f76dead-50bf-4d0e-94e0-1152a518abe3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f76dead-50bf-4d0e-94e0-1152a518abe3-maas-api-xks-ca\") pod \"9f76dead-50bf-4d0e-94e0-1152a518abe3\" (UID: \"9f76dead-50bf-4d0e-94e0-1152a518abe3\") " Oct 31 05:29:43.953661 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:43.953619 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f76dead-50bf-4d0e-94e0-1152a518abe3-kube-api-access-xxpbn\" (UniqueName: \"kubernetes.io/projected/9f76dead-50bf-4d0e-94e0-1152a518abe3-kube-api-access-xxpbn\") pod \"9f76dead-50bf-4d0e-94e0-1152a518abe3\" (UID: \"9f76dead-50bf-4d0e-94e0-1152a518abe3\") " Oct 31 05:29:43.954064 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:43.954036 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f76dead-50bf-4d0e-94e0-1152a518abe3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f76dead-50bf-4d0e-94e0-1152a518abe3-maas-api-openshift-ca\") pod \"9f76dead-50bf-4d0e-94e0-1152a518abe3\" (UID: \"9f76dead-50bf-4d0e-94e0-1152a518abe3\") " Oct 31 05:29:44.005210 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:44.000709 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f76dead-50bf-4d0e-94e0-1152a518abe3-maas-api-openshift-ca" pod "9f76dead-50bf-4d0e-94e0-1152a518abe3" (UID: "9f76dead-50bf-4d0e-94e0-1152a518abe3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:29:44.005210 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:44.004806 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f76dead-50bf-4d0e-94e0-1152a518abe3-maas-api-xks-ca" pod "9f76dead-50bf-4d0e-94e0-1152a518abe3" (UID: "9f76dead-50bf-4d0e-94e0-1152a518abe3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:29:44.005210 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:44.004972 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f76dead-50bf-4d0e-94e0-1152a518abe3-kube-api-access-xxpbn" pod "9f76dead-50bf-4d0e-94e0-1152a518abe3" (UID: "9f76dead-50bf-4d0e-94e0-1152a518abe3"). InnerVolumeSpecName "kube-api-access-xxpbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:29:44.016444 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:44.012650 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kwklx" event={"ID":"9f76dead-50bf-4d0e-94e0-1152a518abe3","Type":"ContainerDied","Data":"4e87510b76391c107763ae3ef390d974b684b26c222ddb9e38627dd391d6e5d9"} Oct 31 05:29:44.016444 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:44.012717 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e87510b76391c107763ae3ef390d974b684b26c222ddb9e38627dd391d6e5d9" Oct 31 05:29:44.016444 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:44.012794 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kwklx" Oct 31 05:29:44.073737 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:44.073634 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f76dead-50bf-4d0e-94e0-1152a518abe3-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:29:44.073737 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:44.073688 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f76dead-50bf-4d0e-94e0-1152a518abe3-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:29:44.073737 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:29:44.073707 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxpbn\" (UniqueName: \"kubernetes.io/projected/9f76dead-50bf-4d0e-94e0-1152a518abe3-kube-api-access-xxpbn\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:29:52.392642 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:29:52.392600 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:30:00.141782 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:00.141735 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890410-nf7gh"] Oct 31 05:30:00.142556 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:00.142534 5019 cpu_manager.go:414] "containerMap: removing container" podUID="9f76dead-50bf-4d0e-94e0-1152a518abe3" containerName="revoke-keys" Oct 31 05:30:00.142647 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:00.142559 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f76dead-50bf-4d0e-94e0-1152a518abe3" containerName="revoke-keys" Oct 31 05:30:00.142701 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:00.142690 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f76dead-50bf-4d0e-94e0-1152a518abe3" containerName="revoke-keys" Oct 31 05:30:00.146520 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:00.146491 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890410-nf7gh" Oct 31 05:30:00.163525 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:00.163491 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890410-nf7gh"] Oct 31 05:30:00.172425 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:00.172394 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:30:00.172667 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:00.172395 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:30:00.172667 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:00.172435 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfhzn\"" Oct 31 05:30:00.284003 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:00.283944 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6abe0c3e-cbad-49d1-89d5-a75f8421de4b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890410-nf7gh\" (UID: \"6abe0c3e-cbad-49d1-89d5-a75f8421de4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890410-nf7gh" Oct 31 05:30:00.284244 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:00.284107 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwxrs\" (UniqueName: \"kubernetes.io/projected/6abe0c3e-cbad-49d1-89d5-a75f8421de4b-kube-api-access-nwxrs\") pod \"maas-api-key-cleanup-29890410-nf7gh\" (UID: \"6abe0c3e-cbad-49d1-89d5-a75f8421de4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890410-nf7gh" Oct 31 05:30:00.385191 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:00.385142 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6abe0c3e-cbad-49d1-89d5-a75f8421de4b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890410-nf7gh\" (UID: \"6abe0c3e-cbad-49d1-89d5-a75f8421de4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890410-nf7gh" Oct 31 05:30:00.385437 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:00.385223 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwxrs\" (UniqueName: \"kubernetes.io/projected/6abe0c3e-cbad-49d1-89d5-a75f8421de4b-kube-api-access-nwxrs\") pod \"maas-api-key-cleanup-29890410-nf7gh\" (UID: \"6abe0c3e-cbad-49d1-89d5-a75f8421de4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890410-nf7gh" Oct 31 05:30:00.385832 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:00.385805 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6abe0c3e-cbad-49d1-89d5-a75f8421de4b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890410-nf7gh\" (UID: \"6abe0c3e-cbad-49d1-89d5-a75f8421de4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890410-nf7gh" Oct 31 05:30:00.409537 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:00.409463 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwxrs\" (UniqueName: \"kubernetes.io/projected/6abe0c3e-cbad-49d1-89d5-a75f8421de4b-kube-api-access-nwxrs\") pod \"maas-api-key-cleanup-29890410-nf7gh\" (UID: \"6abe0c3e-cbad-49d1-89d5-a75f8421de4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890410-nf7gh" Oct 31 05:30:00.460422 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:00.460384 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890410-nf7gh" Oct 31 05:30:00.774381 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:00.774318 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890410-nf7gh"] Oct 31 05:30:00.775797 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:30:00.775766 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6abe0c3e_cbad_49d1_89d5_a75f8421de4b.slice/crio-14eaafdc5c941eee25add402364fa3043a1802a93633ea3aef45126ea54d49a8 WatchSource:0}: Error finding container 14eaafdc5c941eee25add402364fa3043a1802a93633ea3aef45126ea54d49a8: Status 404 returned error can't find the container with id 14eaafdc5c941eee25add402364fa3043a1802a93633ea3aef45126ea54d49a8 Oct 31 05:30:01.150101 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:01.150063 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890410-nf7gh" event={"ID":"6abe0c3e-cbad-49d1-89d5-a75f8421de4b","Type":"ContainerStarted","Data":"ed70e0331f95d7c1b2f5f15b6a6170934fe81a7e2c1c79db3211245ce77f3500"} Oct 31 05:30:01.150608 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:01.150124 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890410-nf7gh" event={"ID":"6abe0c3e-cbad-49d1-89d5-a75f8421de4b","Type":"ContainerStarted","Data":"14eaafdc5c941eee25add402364fa3043a1802a93633ea3aef45126ea54d49a8"} Oct 31 05:30:01.191324 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:01.191255 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890410-nf7gh" podStartSLOduration=1.191237565 podStartE2EDuration="1.191237565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:30:01.189401013 +0000 UTC m=+5155.945739055" watchObservedRunningTime="2026-10-31 05:30:01.191237565 +0000 UTC m=+5155.947575605" Oct 31 05:30:02.157495 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:02.157458 5019 generic.go:356] "Generic (PLEG): container finished" podID="6abe0c3e-cbad-49d1-89d5-a75f8421de4b" containerID="ed70e0331f95d7c1b2f5f15b6a6170934fe81a7e2c1c79db3211245ce77f3500" exitCode=60 Oct 31 05:30:02.157949 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:02.157539 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890410-nf7gh" event={"ID":"6abe0c3e-cbad-49d1-89d5-a75f8421de4b","Type":"ContainerDied","Data":"ed70e0331f95d7c1b2f5f15b6a6170934fe81a7e2c1c79db3211245ce77f3500"} Oct 31 05:30:02.157949 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:02.157776 5019 scope.go:122] "RemoveContainer" containerID="ed70e0331f95d7c1b2f5f15b6a6170934fe81a7e2c1c79db3211245ce77f3500" Oct 31 05:30:03.165131 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:03.165089 5019 generic.go:356] "Generic (PLEG): container finished" podID="6abe0c3e-cbad-49d1-89d5-a75f8421de4b" containerID="1c20459b884611d7fdad6515f34307b153cb9d980877fa82e6364d1308a8aedf" exitCode=60 Oct 31 05:30:03.165627 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:03.165164 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890410-nf7gh" event={"ID":"6abe0c3e-cbad-49d1-89d5-a75f8421de4b","Type":"ContainerDied","Data":"1c20459b884611d7fdad6515f34307b153cb9d980877fa82e6364d1308a8aedf"} Oct 31 05:30:03.165627 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:03.165229 5019 scope.go:122] "RemoveContainer" containerID="ed70e0331f95d7c1b2f5f15b6a6170934fe81a7e2c1c79db3211245ce77f3500" Oct 31 05:30:03.165627 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:03.165487 5019 scope.go:122] "RemoveContainer" containerID="1c20459b884611d7fdad6515f34307b153cb9d980877fa82e6364d1308a8aedf" Oct 31 05:30:03.165790 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:30:03.165748 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890410-nf7gh_odh-ai-gateway-infra(6abe0c3e-cbad-49d1-89d5-a75f8421de4b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890410-nf7gh" podUID="6abe0c3e-cbad-49d1-89d5-a75f8421de4b" Oct 31 05:30:04.173040 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:04.172994 5019 scope.go:122] "RemoveContainer" containerID="1c20459b884611d7fdad6515f34307b153cb9d980877fa82e6364d1308a8aedf" Oct 31 05:30:04.173467 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:30:04.173227 5019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890410-nf7gh_odh-ai-gateway-infra(6abe0c3e-cbad-49d1-89d5-a75f8421de4b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890410-nf7gh" podUID="6abe0c3e-cbad-49d1-89d5-a75f8421de4b" Oct 31 05:30:09.028031 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:09.027995 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9lqcq"] Oct 31 05:30:09.040002 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:09.039972 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9lqcq"] Oct 31 05:30:09.406082 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:09.406000 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="930ebbb3-2834-4e7c-8ce6-3154e80142fb" path="/var/lib/kubelet/pods/930ebbb3-2834-4e7c-8ce6-3154e80142fb/volumes" Oct 31 05:30:15.403388 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:15.403296 5019 scope.go:122] "RemoveContainer" containerID="1c20459b884611d7fdad6515f34307b153cb9d980877fa82e6364d1308a8aedf" Oct 31 05:30:16.254759 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:16.254712 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890410-nf7gh" event={"ID":"6abe0c3e-cbad-49d1-89d5-a75f8421de4b","Type":"ContainerStarted","Data":"a55dd4aff3da80ce755b12562b9f734261049ee1f5b747051b2e408390e2cab3"} Oct 31 05:30:17.313357 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:17.311279 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890410-nf7gh"] Oct 31 05:30:17.364362 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:17.363577 5019 generic.go:356] "Generic (PLEG): container finished" podID="6abe0c3e-cbad-49d1-89d5-a75f8421de4b" containerID="a55dd4aff3da80ce755b12562b9f734261049ee1f5b747051b2e408390e2cab3" exitCode=60 Oct 31 05:30:17.364362 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:17.363669 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890410-nf7gh" event={"ID":"6abe0c3e-cbad-49d1-89d5-a75f8421de4b","Type":"ContainerDied","Data":"a55dd4aff3da80ce755b12562b9f734261049ee1f5b747051b2e408390e2cab3"} Oct 31 05:30:17.364362 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:17.363729 5019 scope.go:122] "RemoveContainer" containerID="1c20459b884611d7fdad6515f34307b153cb9d980877fa82e6364d1308a8aedf" Oct 31 05:30:18.586590 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:18.586549 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890410-nf7gh" Oct 31 05:30:18.923409 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:18.899955 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6abe0c3e-cbad-49d1-89d5-a75f8421de4b-kube-api-access-nwxrs\" (UniqueName: \"kubernetes.io/projected/6abe0c3e-cbad-49d1-89d5-a75f8421de4b-kube-api-access-nwxrs\") pod \"6abe0c3e-cbad-49d1-89d5-a75f8421de4b\" (UID: \"6abe0c3e-cbad-49d1-89d5-a75f8421de4b\") " Oct 31 05:30:18.923886 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:18.923855 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6abe0c3e-cbad-49d1-89d5-a75f8421de4b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6abe0c3e-cbad-49d1-89d5-a75f8421de4b-maas-api-service-ca\") pod \"6abe0c3e-cbad-49d1-89d5-a75f8421de4b\" (UID: \"6abe0c3e-cbad-49d1-89d5-a75f8421de4b\") " Oct 31 05:30:18.924718 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:18.924691 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6abe0c3e-cbad-49d1-89d5-a75f8421de4b-maas-api-service-ca" pod "6abe0c3e-cbad-49d1-89d5-a75f8421de4b" (UID: "6abe0c3e-cbad-49d1-89d5-a75f8421de4b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:30:18.957551 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:18.957516 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6abe0c3e-cbad-49d1-89d5-a75f8421de4b-kube-api-access-nwxrs" pod "6abe0c3e-cbad-49d1-89d5-a75f8421de4b" (UID: "6abe0c3e-cbad-49d1-89d5-a75f8421de4b"). InnerVolumeSpecName "kube-api-access-nwxrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:30:19.025403 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:19.025370 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6abe0c3e-cbad-49d1-89d5-a75f8421de4b-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:30:19.025403 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:19.025401 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwxrs\" (UniqueName: \"kubernetes.io/projected/6abe0c3e-cbad-49d1-89d5-a75f8421de4b-kube-api-access-nwxrs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:30:19.238919 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:19.238885 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lswwt"] Oct 31 05:30:19.239365 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:19.239348 5019 cpu_manager.go:414] "containerMap: removing container" podUID="6abe0c3e-cbad-49d1-89d5-a75f8421de4b" containerName="cleanup" Oct 31 05:30:19.239434 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:19.239368 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6abe0c3e-cbad-49d1-89d5-a75f8421de4b" containerName="cleanup" Oct 31 05:30:19.239434 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:19.239396 5019 cpu_manager.go:414] "containerMap: removing container" podUID="6abe0c3e-cbad-49d1-89d5-a75f8421de4b" containerName="cleanup" Oct 31 05:30:19.239434 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:19.239401 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6abe0c3e-cbad-49d1-89d5-a75f8421de4b" containerName="cleanup" Oct 31 05:30:19.239534 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:19.239482 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="6abe0c3e-cbad-49d1-89d5-a75f8421de4b" containerName="cleanup" Oct 31 05:30:19.239534 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:19.239498 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="6abe0c3e-cbad-49d1-89d5-a75f8421de4b" containerName="cleanup" Oct 31 05:30:19.242770 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:19.242752 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lswwt" Oct 31 05:30:19.263427 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:19.263391 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lswwt"] Oct 31 05:30:19.271379 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:19.271321 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:30:19.328804 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:19.328769 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4fddbfb-2cf9-411a-a461-7a8e8d3d5f7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-lswwt\" (UID: \"c4fddbfb-2cf9-411a-a461-7a8e8d3d5f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lswwt" Oct 31 05:30:19.328804 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:19.328804 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g8ws\" (UniqueName: \"kubernetes.io/projected/c4fddbfb-2cf9-411a-a461-7a8e8d3d5f7c-kube-api-access-4g8ws\") pod \"maas-api-revoke-sub-a4aef7d7-lswwt\" (UID: \"c4fddbfb-2cf9-411a-a461-7a8e8d3d5f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lswwt" Oct 31 05:30:19.329026 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:19.328824 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4fddbfb-2cf9-411a-a461-7a8e8d3d5f7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-lswwt\" (UID: \"c4fddbfb-2cf9-411a-a461-7a8e8d3d5f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lswwt" Oct 31 05:30:19.430276 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:19.430244 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4fddbfb-2cf9-411a-a461-7a8e8d3d5f7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-lswwt\" (UID: \"c4fddbfb-2cf9-411a-a461-7a8e8d3d5f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lswwt" Oct 31 05:30:19.430276 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:19.430280 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g8ws\" (UniqueName: \"kubernetes.io/projected/c4fddbfb-2cf9-411a-a461-7a8e8d3d5f7c-kube-api-access-4g8ws\") pod \"maas-api-revoke-sub-a4aef7d7-lswwt\" (UID: \"c4fddbfb-2cf9-411a-a461-7a8e8d3d5f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lswwt" Oct 31 05:30:19.430519 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:19.430301 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4fddbfb-2cf9-411a-a461-7a8e8d3d5f7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-lswwt\" (UID: \"c4fddbfb-2cf9-411a-a461-7a8e8d3d5f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lswwt" Oct 31 05:30:19.431115 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:19.431087 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4fddbfb-2cf9-411a-a461-7a8e8d3d5f7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-lswwt\" (UID: \"c4fddbfb-2cf9-411a-a461-7a8e8d3d5f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lswwt" Oct 31 05:30:19.432806 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:19.432776 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4fddbfb-2cf9-411a-a461-7a8e8d3d5f7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-lswwt\" (UID: \"c4fddbfb-2cf9-411a-a461-7a8e8d3d5f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lswwt" Oct 31 05:30:19.455469 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:19.455427 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g8ws\" (UniqueName: \"kubernetes.io/projected/c4fddbfb-2cf9-411a-a461-7a8e8d3d5f7c-kube-api-access-4g8ws\") pod \"maas-api-revoke-sub-a4aef7d7-lswwt\" (UID: \"c4fddbfb-2cf9-411a-a461-7a8e8d3d5f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lswwt" Oct 31 05:30:19.481861 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:19.481829 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890410-nf7gh" Oct 31 05:30:19.482067 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:19.481853 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890410-nf7gh" event={"ID":"6abe0c3e-cbad-49d1-89d5-a75f8421de4b","Type":"ContainerDied","Data":"14eaafdc5c941eee25add402364fa3043a1802a93633ea3aef45126ea54d49a8"} Oct 31 05:30:19.482067 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:19.481921 5019 scope.go:122] "RemoveContainer" containerID="a55dd4aff3da80ce755b12562b9f734261049ee1f5b747051b2e408390e2cab3" Oct 31 05:30:19.533664 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:19.533619 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890410-nf7gh"] Oct 31 05:30:19.539401 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:19.539366 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890410-nf7gh"] Oct 31 05:30:19.555823 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:19.555793 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lswwt" Oct 31 05:30:19.855111 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:19.855084 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lswwt"] Oct 31 05:30:19.856586 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:30:19.856558 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4fddbfb_2cf9_411a_a461_7a8e8d3d5f7c.slice/crio-ead34994ae4b5eb78926d95d0c024be7bfe41d80066039fe2c520e535c5d3ab0 WatchSource:0}: Error finding container ead34994ae4b5eb78926d95d0c024be7bfe41d80066039fe2c520e535c5d3ab0: Status 404 returned error can't find the container with id ead34994ae4b5eb78926d95d0c024be7bfe41d80066039fe2c520e535c5d3ab0 Oct 31 05:30:20.489518 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:20.489478 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lswwt" event={"ID":"c4fddbfb-2cf9-411a-a461-7a8e8d3d5f7c","Type":"ContainerStarted","Data":"cf0db329332f5f3e88338007a8a3400aed58eb0c818a31203cdb0b1667be5707"} Oct 31 05:30:20.489753 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:20.489549 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lswwt" event={"ID":"c4fddbfb-2cf9-411a-a461-7a8e8d3d5f7c","Type":"ContainerStarted","Data":"ead34994ae4b5eb78926d95d0c024be7bfe41d80066039fe2c520e535c5d3ab0"} Oct 31 05:30:20.536640 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:20.536573 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lswwt" podStartSLOduration=1.5365494229999999 podStartE2EDuration="1.536549423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:30:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:30:20.535430201 +0000 UTC m=+5175.291768243" watchObservedRunningTime="2026-10-31 05:30:20.536549423 +0000 UTC m=+5175.292887512" Oct 31 05:30:21.407091 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:21.407055 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6abe0c3e-cbad-49d1-89d5-a75f8421de4b" path="/var/lib/kubelet/pods/6abe0c3e-cbad-49d1-89d5-a75f8421de4b/volumes" Oct 31 05:30:21.497633 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:21.497594 5019 generic.go:356] "Generic (PLEG): container finished" podID="c4fddbfb-2cf9-411a-a461-7a8e8d3d5f7c" containerID="cf0db329332f5f3e88338007a8a3400aed58eb0c818a31203cdb0b1667be5707" exitCode=0 Oct 31 05:30:21.497823 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:21.497675 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lswwt" event={"ID":"c4fddbfb-2cf9-411a-a461-7a8e8d3d5f7c","Type":"ContainerDied","Data":"cf0db329332f5f3e88338007a8a3400aed58eb0c818a31203cdb0b1667be5707"} Oct 31 05:30:22.391711 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:30:22.391669 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:30:22.640706 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:22.640678 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lswwt" Oct 31 05:30:22.762144 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:22.762088 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4fddbfb-2cf9-411a-a461-7a8e8d3d5f7c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4fddbfb-2cf9-411a-a461-7a8e8d3d5f7c-maas-api-xks-ca\") pod \"c4fddbfb-2cf9-411a-a461-7a8e8d3d5f7c\" (UID: \"c4fddbfb-2cf9-411a-a461-7a8e8d3d5f7c\") " Oct 31 05:30:22.762364 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:22.762200 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4fddbfb-2cf9-411a-a461-7a8e8d3d5f7c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4fddbfb-2cf9-411a-a461-7a8e8d3d5f7c-maas-api-openshift-ca\") pod \"c4fddbfb-2cf9-411a-a461-7a8e8d3d5f7c\" (UID: \"c4fddbfb-2cf9-411a-a461-7a8e8d3d5f7c\") " Oct 31 05:30:22.762364 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:22.762242 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4fddbfb-2cf9-411a-a461-7a8e8d3d5f7c-kube-api-access-4g8ws\" (UniqueName: \"kubernetes.io/projected/c4fddbfb-2cf9-411a-a461-7a8e8d3d5f7c-kube-api-access-4g8ws\") pod \"c4fddbfb-2cf9-411a-a461-7a8e8d3d5f7c\" (UID: \"c4fddbfb-2cf9-411a-a461-7a8e8d3d5f7c\") " Oct 31 05:30:22.762696 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:22.762666 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4fddbfb-2cf9-411a-a461-7a8e8d3d5f7c-maas-api-openshift-ca" pod "c4fddbfb-2cf9-411a-a461-7a8e8d3d5f7c" (UID: "c4fddbfb-2cf9-411a-a461-7a8e8d3d5f7c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:30:22.764376 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:22.764312 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4fddbfb-2cf9-411a-a461-7a8e8d3d5f7c-maas-api-xks-ca" pod "c4fddbfb-2cf9-411a-a461-7a8e8d3d5f7c" (UID: "c4fddbfb-2cf9-411a-a461-7a8e8d3d5f7c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:30:22.764635 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:22.764608 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4fddbfb-2cf9-411a-a461-7a8e8d3d5f7c-kube-api-access-4g8ws" pod "c4fddbfb-2cf9-411a-a461-7a8e8d3d5f7c" (UID: "c4fddbfb-2cf9-411a-a461-7a8e8d3d5f7c"). InnerVolumeSpecName "kube-api-access-4g8ws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:30:22.863574 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:22.863533 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4fddbfb-2cf9-411a-a461-7a8e8d3d5f7c-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:30:22.863574 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:22.863569 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4fddbfb-2cf9-411a-a461-7a8e8d3d5f7c-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:30:22.863574 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:22.863579 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g8ws\" (UniqueName: \"kubernetes.io/projected/c4fddbfb-2cf9-411a-a461-7a8e8d3d5f7c-kube-api-access-4g8ws\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:30:23.510480 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:23.510440 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lswwt" event={"ID":"c4fddbfb-2cf9-411a-a461-7a8e8d3d5f7c","Type":"ContainerDied","Data":"ead34994ae4b5eb78926d95d0c024be7bfe41d80066039fe2c520e535c5d3ab0"} Oct 31 05:30:23.510480 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:23.510478 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lswwt" Oct 31 05:30:23.510711 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:23.510495 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ead34994ae4b5eb78926d95d0c024be7bfe41d80066039fe2c520e535c5d3ab0" Oct 31 05:30:31.028061 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:31.028022 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cbsgr"] Oct 31 05:30:31.039576 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:31.039544 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cbsgr"] Oct 31 05:30:31.406346 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:31.406254 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eafb76a4-6a45-4a64-8bce-2787d5914a8c" path="/var/lib/kubelet/pods/eafb76a4-6a45-4a64-8bce-2787d5914a8c/volumes" Oct 31 05:30:36.161363 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:36.161297 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fb8lv"] Oct 31 05:30:36.162708 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:36.162677 5019 cpu_manager.go:414] "containerMap: removing container" podUID="c4fddbfb-2cf9-411a-a461-7a8e8d3d5f7c" containerName="revoke-keys" Oct 31 05:30:36.162899 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:36.162880 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4fddbfb-2cf9-411a-a461-7a8e8d3d5f7c" containerName="revoke-keys" Oct 31 05:30:36.163059 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:36.163044 5019 cpu_manager.go:414] "containerMap: removing container" podUID="6abe0c3e-cbad-49d1-89d5-a75f8421de4b" containerName="cleanup" Oct 31 05:30:36.163156 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:36.163145 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6abe0c3e-cbad-49d1-89d5-a75f8421de4b" containerName="cleanup" Oct 31 05:30:36.163404 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:36.163386 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="6abe0c3e-cbad-49d1-89d5-a75f8421de4b" containerName="cleanup" Oct 31 05:30:36.163546 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:36.163532 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4fddbfb-2cf9-411a-a461-7a8e8d3d5f7c" containerName="revoke-keys" Oct 31 05:30:36.168928 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:36.168881 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fb8lv" Oct 31 05:30:36.186276 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:36.186177 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fb8lv"] Oct 31 05:30:36.195153 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:36.195106 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfhzn\"" Oct 31 05:30:36.195374 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:36.195192 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:30:36.277359 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:36.260475 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:30:36.277359 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:36.260609 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:30:36.299357 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:36.298915 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ade78fb-67c0-4dbe-8e5d-c17046d47af5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-fb8lv\" (UID: \"1ade78fb-67c0-4dbe-8e5d-c17046d47af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fb8lv" Oct 31 05:30:36.299357 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:36.299020 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ade78fb-67c0-4dbe-8e5d-c17046d47af5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-fb8lv\" (UID: \"1ade78fb-67c0-4dbe-8e5d-c17046d47af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fb8lv" Oct 31 05:30:36.299357 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:36.299104 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gd9n\" (UniqueName: \"kubernetes.io/projected/1ade78fb-67c0-4dbe-8e5d-c17046d47af5-kube-api-access-5gd9n\") pod \"maas-api-revoke-sub-79805ca9-fb8lv\" (UID: \"1ade78fb-67c0-4dbe-8e5d-c17046d47af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fb8lv" Oct 31 05:30:36.403716 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:36.402910 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ade78fb-67c0-4dbe-8e5d-c17046d47af5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-fb8lv\" (UID: \"1ade78fb-67c0-4dbe-8e5d-c17046d47af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fb8lv" Oct 31 05:30:36.403716 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:36.403031 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gd9n\" (UniqueName: \"kubernetes.io/projected/1ade78fb-67c0-4dbe-8e5d-c17046d47af5-kube-api-access-5gd9n\") pod \"maas-api-revoke-sub-79805ca9-fb8lv\" (UID: \"1ade78fb-67c0-4dbe-8e5d-c17046d47af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fb8lv" Oct 31 05:30:36.403716 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:36.403116 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ade78fb-67c0-4dbe-8e5d-c17046d47af5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-fb8lv\" (UID: \"1ade78fb-67c0-4dbe-8e5d-c17046d47af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fb8lv" Oct 31 05:30:36.405172 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:36.405146 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ade78fb-67c0-4dbe-8e5d-c17046d47af5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-fb8lv\" (UID: \"1ade78fb-67c0-4dbe-8e5d-c17046d47af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fb8lv" Oct 31 05:30:36.436412 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:36.433751 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ade78fb-67c0-4dbe-8e5d-c17046d47af5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-fb8lv\" (UID: \"1ade78fb-67c0-4dbe-8e5d-c17046d47af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fb8lv" Oct 31 05:30:36.440487 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:36.440457 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gd9n\" (UniqueName: \"kubernetes.io/projected/1ade78fb-67c0-4dbe-8e5d-c17046d47af5-kube-api-access-5gd9n\") pod \"maas-api-revoke-sub-79805ca9-fb8lv\" (UID: \"1ade78fb-67c0-4dbe-8e5d-c17046d47af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fb8lv" Oct 31 05:30:36.495552 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:36.495509 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fb8lv" Oct 31 05:30:37.361609 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:37.361571 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fb8lv"] Oct 31 05:30:37.670725 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:37.670629 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fb8lv" event={"ID":"1ade78fb-67c0-4dbe-8e5d-c17046d47af5","Type":"ContainerStarted","Data":"20c03b5a5ee1b8d3922d61a80661345ea6e1c6c2fc6fe43e7e4bf3650758a32d"} Oct 31 05:30:38.678770 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:38.678724 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fb8lv" event={"ID":"1ade78fb-67c0-4dbe-8e5d-c17046d47af5","Type":"ContainerStarted","Data":"4aef163ea5f17de4939e01ecae7d5dde9f2216043e6103078b011c4e64408d92"} Oct 31 05:30:38.721676 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:38.721618 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fb8lv" podStartSLOduration=2.721603797 podStartE2EDuration="2.721603797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:30:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:30:38.718416995 +0000 UTC m=+5193.474755035" watchObservedRunningTime="2026-10-31 05:30:38.721603797 +0000 UTC m=+5193.477941837" Oct 31 05:30:38.836776 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:30:38.836733 5019 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ade78fb_67c0_4dbe_8e5d_c17046d47af5.slice/crio-4aef163ea5f17de4939e01ecae7d5dde9f2216043e6103078b011c4e64408d92.scope\": RecentStats: unable to find data in memory cache]" Oct 31 05:30:39.686916 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:39.686875 5019 generic.go:356] "Generic (PLEG): container finished" podID="1ade78fb-67c0-4dbe-8e5d-c17046d47af5" containerID="4aef163ea5f17de4939e01ecae7d5dde9f2216043e6103078b011c4e64408d92" exitCode=0 Oct 31 05:30:39.687365 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:39.686953 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fb8lv" event={"ID":"1ade78fb-67c0-4dbe-8e5d-c17046d47af5","Type":"ContainerDied","Data":"4aef163ea5f17de4939e01ecae7d5dde9f2216043e6103078b011c4e64408d92"} Oct 31 05:30:40.838325 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:40.838300 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fb8lv" Oct 31 05:30:40.928243 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:40.928210 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ade78fb-67c0-4dbe-8e5d-c17046d47af5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ade78fb-67c0-4dbe-8e5d-c17046d47af5-maas-api-xks-ca\") pod \"1ade78fb-67c0-4dbe-8e5d-c17046d47af5\" (UID: \"1ade78fb-67c0-4dbe-8e5d-c17046d47af5\") " Oct 31 05:30:40.928416 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:40.928392 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ade78fb-67c0-4dbe-8e5d-c17046d47af5-kube-api-access-5gd9n\" (UniqueName: \"kubernetes.io/projected/1ade78fb-67c0-4dbe-8e5d-c17046d47af5-kube-api-access-5gd9n\") pod \"1ade78fb-67c0-4dbe-8e5d-c17046d47af5\" (UID: \"1ade78fb-67c0-4dbe-8e5d-c17046d47af5\") " Oct 31 05:30:40.928480 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:40.928441 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ade78fb-67c0-4dbe-8e5d-c17046d47af5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ade78fb-67c0-4dbe-8e5d-c17046d47af5-maas-api-openshift-ca\") pod \"1ade78fb-67c0-4dbe-8e5d-c17046d47af5\" (UID: \"1ade78fb-67c0-4dbe-8e5d-c17046d47af5\") " Oct 31 05:30:40.928849 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:40.928820 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ade78fb-67c0-4dbe-8e5d-c17046d47af5-maas-api-openshift-ca" pod "1ade78fb-67c0-4dbe-8e5d-c17046d47af5" (UID: "1ade78fb-67c0-4dbe-8e5d-c17046d47af5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:30:40.930511 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:40.930474 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ade78fb-67c0-4dbe-8e5d-c17046d47af5-maas-api-xks-ca" pod "1ade78fb-67c0-4dbe-8e5d-c17046d47af5" (UID: "1ade78fb-67c0-4dbe-8e5d-c17046d47af5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:30:40.930511 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:40.930499 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ade78fb-67c0-4dbe-8e5d-c17046d47af5-kube-api-access-5gd9n" pod "1ade78fb-67c0-4dbe-8e5d-c17046d47af5" (UID: "1ade78fb-67c0-4dbe-8e5d-c17046d47af5"). InnerVolumeSpecName "kube-api-access-5gd9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:30:41.029814 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:41.029780 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gd9n\" (UniqueName: \"kubernetes.io/projected/1ade78fb-67c0-4dbe-8e5d-c17046d47af5-kube-api-access-5gd9n\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:30:41.029814 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:41.029809 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ade78fb-67c0-4dbe-8e5d-c17046d47af5-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:30:41.029814 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:41.029819 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ade78fb-67c0-4dbe-8e5d-c17046d47af5-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:30:41.700071 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:41.699989 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fb8lv" Oct 31 05:30:41.700071 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:41.700021 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fb8lv" event={"ID":"1ade78fb-67c0-4dbe-8e5d-c17046d47af5","Type":"ContainerDied","Data":"20c03b5a5ee1b8d3922d61a80661345ea6e1c6c2fc6fe43e7e4bf3650758a32d"} Oct 31 05:30:41.700071 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:41.700069 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20c03b5a5ee1b8d3922d61a80661345ea6e1c6c2fc6fe43e7e4bf3650758a32d" Oct 31 05:30:49.862354 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:49.856103 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pqls6"] Oct 31 05:30:49.873368 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:49.869979 5019 cpu_manager.go:414] "containerMap: removing container" podUID="1ade78fb-67c0-4dbe-8e5d-c17046d47af5" containerName="revoke-keys" Oct 31 05:30:49.873368 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:49.870018 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ade78fb-67c0-4dbe-8e5d-c17046d47af5" containerName="revoke-keys" Oct 31 05:30:49.873368 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:49.870180 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ade78fb-67c0-4dbe-8e5d-c17046d47af5" containerName="revoke-keys" Oct 31 05:30:49.874205 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:49.874164 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pqls6" Oct 31 05:30:49.887844 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:49.887788 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pqls6"] Oct 31 05:30:49.897222 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:49.896781 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:30:49.897222 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:49.896857 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:30:49.926610 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:49.910743 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfhzn\"" Oct 31 05:30:49.926610 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:49.910815 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:30:50.053811 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:50.053766 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9mvr\" (UniqueName: \"kubernetes.io/projected/2148a95c-cdb0-4675-aa27-2ed4b7804bfc-kube-api-access-m9mvr\") pod \"maas-api-revoke-sub-0b7e8edb-pqls6\" (UID: \"2148a95c-cdb0-4675-aa27-2ed4b7804bfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pqls6" Oct 31 05:30:50.054175 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:50.054151 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2148a95c-cdb0-4675-aa27-2ed4b7804bfc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-pqls6\" (UID: \"2148a95c-cdb0-4675-aa27-2ed4b7804bfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pqls6" Oct 31 05:30:50.054420 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:50.054389 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2148a95c-cdb0-4675-aa27-2ed4b7804bfc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-pqls6\" (UID: \"2148a95c-cdb0-4675-aa27-2ed4b7804bfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pqls6" Oct 31 05:30:50.156329 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:50.156228 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2148a95c-cdb0-4675-aa27-2ed4b7804bfc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-pqls6\" (UID: \"2148a95c-cdb0-4675-aa27-2ed4b7804bfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pqls6" Oct 31 05:30:50.156745 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:50.156712 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9mvr\" (UniqueName: \"kubernetes.io/projected/2148a95c-cdb0-4675-aa27-2ed4b7804bfc-kube-api-access-m9mvr\") pod \"maas-api-revoke-sub-0b7e8edb-pqls6\" (UID: \"2148a95c-cdb0-4675-aa27-2ed4b7804bfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pqls6" Oct 31 05:30:50.157751 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:50.157729 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2148a95c-cdb0-4675-aa27-2ed4b7804bfc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-pqls6\" (UID: \"2148a95c-cdb0-4675-aa27-2ed4b7804bfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pqls6" Oct 31 05:30:50.158832 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:50.158704 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2148a95c-cdb0-4675-aa27-2ed4b7804bfc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-pqls6\" (UID: \"2148a95c-cdb0-4675-aa27-2ed4b7804bfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pqls6" Oct 31 05:30:50.191371 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:50.182834 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2148a95c-cdb0-4675-aa27-2ed4b7804bfc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-pqls6\" (UID: \"2148a95c-cdb0-4675-aa27-2ed4b7804bfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pqls6" Oct 31 05:30:50.255380 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:50.238507 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9mvr\" (UniqueName: \"kubernetes.io/projected/2148a95c-cdb0-4675-aa27-2ed4b7804bfc-kube-api-access-m9mvr\") pod \"maas-api-revoke-sub-0b7e8edb-pqls6\" (UID: \"2148a95c-cdb0-4675-aa27-2ed4b7804bfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pqls6" Oct 31 05:30:50.255380 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:50.247592 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pqls6" Oct 31 05:30:51.079627 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:51.079567 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42aa8c1d-snpcr"] Oct 31 05:30:51.258808 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:51.258389 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42aa8c1d-snpcr"] Oct 31 05:30:51.525487 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:51.525448 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1f204b0-02c0-46d1-aab4-093b54f7caca" path="/var/lib/kubelet/pods/d1f204b0-02c0-46d1-aab4-093b54f7caca/volumes" Oct 31 05:30:51.538470 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:51.538438 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pqls6"] Oct 31 05:30:51.575697 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:30:51.575650 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2148a95c_cdb0_4675_aa27_2ed4b7804bfc.slice/crio-72203b3b1d5ee1dd05697b862214d92a1f8967fe689336df3dbfcf980e7d3651 WatchSource:0}: Error finding container 72203b3b1d5ee1dd05697b862214d92a1f8967fe689336df3dbfcf980e7d3651: Status 404 returned error can't find the container with id 72203b3b1d5ee1dd05697b862214d92a1f8967fe689336df3dbfcf980e7d3651 Oct 31 05:30:52.012675 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:52.012559 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pqls6" event={"ID":"2148a95c-cdb0-4675-aa27-2ed4b7804bfc","Type":"ContainerStarted","Data":"72203b3b1d5ee1dd05697b862214d92a1f8967fe689336df3dbfcf980e7d3651"} Oct 31 05:30:52.391791 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:30:52.391698 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:30:53.020218 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:53.020174 5019 generic.go:356] "Generic (PLEG): container finished" podID="2148a95c-cdb0-4675-aa27-2ed4b7804bfc" containerID="229f36cb60ccceaa1fad23f1d6d60dcbf4336b2c76b9634c6cbb8bc8694777e4" exitCode=0 Oct 31 05:30:53.020444 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:53.020260 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pqls6" event={"ID":"2148a95c-cdb0-4675-aa27-2ed4b7804bfc","Type":"ContainerDied","Data":"229f36cb60ccceaa1fad23f1d6d60dcbf4336b2c76b9634c6cbb8bc8694777e4"} Oct 31 05:30:54.174166 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:54.174133 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pqls6" Oct 31 05:30:54.228245 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:54.228207 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2148a95c-cdb0-4675-aa27-2ed4b7804bfc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2148a95c-cdb0-4675-aa27-2ed4b7804bfc-maas-api-openshift-ca\") pod \"2148a95c-cdb0-4675-aa27-2ed4b7804bfc\" (UID: \"2148a95c-cdb0-4675-aa27-2ed4b7804bfc\") " Oct 31 05:30:54.228475 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:54.228309 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2148a95c-cdb0-4675-aa27-2ed4b7804bfc-kube-api-access-m9mvr\" (UniqueName: \"kubernetes.io/projected/2148a95c-cdb0-4675-aa27-2ed4b7804bfc-kube-api-access-m9mvr\") pod \"2148a95c-cdb0-4675-aa27-2ed4b7804bfc\" (UID: \"2148a95c-cdb0-4675-aa27-2ed4b7804bfc\") " Oct 31 05:30:54.228569 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:54.228499 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2148a95c-cdb0-4675-aa27-2ed4b7804bfc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2148a95c-cdb0-4675-aa27-2ed4b7804bfc-maas-api-xks-ca\") pod \"2148a95c-cdb0-4675-aa27-2ed4b7804bfc\" (UID: \"2148a95c-cdb0-4675-aa27-2ed4b7804bfc\") " Oct 31 05:30:54.228725 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:54.228696 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2148a95c-cdb0-4675-aa27-2ed4b7804bfc-maas-api-openshift-ca" pod "2148a95c-cdb0-4675-aa27-2ed4b7804bfc" (UID: "2148a95c-cdb0-4675-aa27-2ed4b7804bfc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:30:54.228917 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:54.228891 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2148a95c-cdb0-4675-aa27-2ed4b7804bfc-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:30:54.230664 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:54.230633 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2148a95c-cdb0-4675-aa27-2ed4b7804bfc-maas-api-xks-ca" pod "2148a95c-cdb0-4675-aa27-2ed4b7804bfc" (UID: "2148a95c-cdb0-4675-aa27-2ed4b7804bfc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:30:54.230998 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:54.230978 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2148a95c-cdb0-4675-aa27-2ed4b7804bfc-kube-api-access-m9mvr" pod "2148a95c-cdb0-4675-aa27-2ed4b7804bfc" (UID: "2148a95c-cdb0-4675-aa27-2ed4b7804bfc"). InnerVolumeSpecName "kube-api-access-m9mvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:30:54.330043 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:54.329961 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2148a95c-cdb0-4675-aa27-2ed4b7804bfc-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:30:54.330043 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:54.329993 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9mvr\" (UniqueName: \"kubernetes.io/projected/2148a95c-cdb0-4675-aa27-2ed4b7804bfc-kube-api-access-m9mvr\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:30:55.033366 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:55.033314 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pqls6" Oct 31 05:30:55.033366 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:55.033354 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pqls6" event={"ID":"2148a95c-cdb0-4675-aa27-2ed4b7804bfc","Type":"ContainerDied","Data":"72203b3b1d5ee1dd05697b862214d92a1f8967fe689336df3dbfcf980e7d3651"} Oct 31 05:30:55.033625 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:30:55.033409 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72203b3b1d5ee1dd05697b862214d92a1f8967fe689336df3dbfcf980e7d3651" Oct 31 05:31:00.590265 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:00.590228 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pf9wq"] Oct 31 05:31:00.590714 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:00.590699 5019 cpu_manager.go:414] "containerMap: removing container" podUID="2148a95c-cdb0-4675-aa27-2ed4b7804bfc" containerName="revoke-keys" Oct 31 05:31:00.590786 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:00.590715 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2148a95c-cdb0-4675-aa27-2ed4b7804bfc" containerName="revoke-keys" Oct 31 05:31:00.590826 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:00.590805 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="2148a95c-cdb0-4675-aa27-2ed4b7804bfc" containerName="revoke-keys" Oct 31 05:31:00.594477 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:00.594452 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pf9wq" Oct 31 05:31:00.614100 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:00.614061 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pf9wq"] Oct 31 05:31:00.622499 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:00.622460 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:31:00.622499 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:00.622475 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:31:00.622782 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:00.622606 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:31:00.622782 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:00.622736 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfhzn\"" Oct 31 05:31:00.701204 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:00.701157 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4kbs\" (UniqueName: \"kubernetes.io/projected/b7aa3052-2427-47c1-a9de-366b2372f111-kube-api-access-d4kbs\") pod \"maas-api-revoke-sub-3d4156f1-pf9wq\" (UID: \"b7aa3052-2427-47c1-a9de-366b2372f111\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pf9wq" Oct 31 05:31:00.701395 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:00.701265 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7aa3052-2427-47c1-a9de-366b2372f111-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-pf9wq\" (UID: \"b7aa3052-2427-47c1-a9de-366b2372f111\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pf9wq" Oct 31 05:31:00.701395 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:00.701298 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7aa3052-2427-47c1-a9de-366b2372f111-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-pf9wq\" (UID: \"b7aa3052-2427-47c1-a9de-366b2372f111\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pf9wq" Oct 31 05:31:00.802922 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:00.802865 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7aa3052-2427-47c1-a9de-366b2372f111-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-pf9wq\" (UID: \"b7aa3052-2427-47c1-a9de-366b2372f111\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pf9wq" Oct 31 05:31:00.802922 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:00.802924 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7aa3052-2427-47c1-a9de-366b2372f111-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-pf9wq\" (UID: \"b7aa3052-2427-47c1-a9de-366b2372f111\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pf9wq" Oct 31 05:31:00.803172 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:00.803027 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4kbs\" (UniqueName: \"kubernetes.io/projected/b7aa3052-2427-47c1-a9de-366b2372f111-kube-api-access-d4kbs\") pod \"maas-api-revoke-sub-3d4156f1-pf9wq\" (UID: \"b7aa3052-2427-47c1-a9de-366b2372f111\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pf9wq" Oct 31 05:31:00.803650 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:00.803622 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7aa3052-2427-47c1-a9de-366b2372f111-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-pf9wq\" (UID: \"b7aa3052-2427-47c1-a9de-366b2372f111\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pf9wq" Oct 31 05:31:00.805540 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:00.805511 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7aa3052-2427-47c1-a9de-366b2372f111-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-pf9wq\" (UID: \"b7aa3052-2427-47c1-a9de-366b2372f111\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pf9wq" Oct 31 05:31:00.826749 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:00.826715 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4kbs\" (UniqueName: \"kubernetes.io/projected/b7aa3052-2427-47c1-a9de-366b2372f111-kube-api-access-d4kbs\") pod \"maas-api-revoke-sub-3d4156f1-pf9wq\" (UID: \"b7aa3052-2427-47c1-a9de-366b2372f111\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pf9wq" Oct 31 05:31:00.908319 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:00.908233 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pf9wq" Oct 31 05:31:01.211163 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:01.211116 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pf9wq"] Oct 31 05:31:01.212758 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:31:01.212721 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7aa3052_2427_47c1_a9de_366b2372f111.slice/crio-cf40dc23be480779f412b78ba473a2da42f0349f5300c6d2cbbfb439830865f1 WatchSource:0}: Error finding container cf40dc23be480779f412b78ba473a2da42f0349f5300c6d2cbbfb439830865f1: Status 404 returned error can't find the container with id cf40dc23be480779f412b78ba473a2da42f0349f5300c6d2cbbfb439830865f1 Oct 31 05:31:02.076300 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:02.076265 5019 generic.go:356] "Generic (PLEG): container finished" podID="b7aa3052-2427-47c1-a9de-366b2372f111" containerID="18163c60e1a1b98c59fab71ebb16c666ccd20fbd710359ce16938097f92155e1" exitCode=0 Oct 31 05:31:02.076758 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:02.076377 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pf9wq" event={"ID":"b7aa3052-2427-47c1-a9de-366b2372f111","Type":"ContainerDied","Data":"18163c60e1a1b98c59fab71ebb16c666ccd20fbd710359ce16938097f92155e1"} Oct 31 05:31:02.076758 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:02.076432 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pf9wq" event={"ID":"b7aa3052-2427-47c1-a9de-366b2372f111","Type":"ContainerStarted","Data":"cf40dc23be480779f412b78ba473a2da42f0349f5300c6d2cbbfb439830865f1"} Oct 31 05:31:03.232302 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:03.232274 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pf9wq" Oct 31 05:31:03.329581 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:03.329540 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7aa3052-2427-47c1-a9de-366b2372f111-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7aa3052-2427-47c1-a9de-366b2372f111-maas-api-openshift-ca\") pod \"b7aa3052-2427-47c1-a9de-366b2372f111\" (UID: \"b7aa3052-2427-47c1-a9de-366b2372f111\") " Oct 31 05:31:03.329775 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:03.329719 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7aa3052-2427-47c1-a9de-366b2372f111-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7aa3052-2427-47c1-a9de-366b2372f111-maas-api-xks-ca\") pod \"b7aa3052-2427-47c1-a9de-366b2372f111\" (UID: \"b7aa3052-2427-47c1-a9de-366b2372f111\") " Oct 31 05:31:03.329775 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:03.329755 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7aa3052-2427-47c1-a9de-366b2372f111-kube-api-access-d4kbs\" (UniqueName: \"kubernetes.io/projected/b7aa3052-2427-47c1-a9de-366b2372f111-kube-api-access-d4kbs\") pod \"b7aa3052-2427-47c1-a9de-366b2372f111\" (UID: \"b7aa3052-2427-47c1-a9de-366b2372f111\") " Oct 31 05:31:03.330005 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:03.329982 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7aa3052-2427-47c1-a9de-366b2372f111-maas-api-openshift-ca" pod "b7aa3052-2427-47c1-a9de-366b2372f111" (UID: "b7aa3052-2427-47c1-a9de-366b2372f111"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:31:03.330122 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:03.330108 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7aa3052-2427-47c1-a9de-366b2372f111-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:31:03.331920 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:03.331896 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7aa3052-2427-47c1-a9de-366b2372f111-maas-api-xks-ca" pod "b7aa3052-2427-47c1-a9de-366b2372f111" (UID: "b7aa3052-2427-47c1-a9de-366b2372f111"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:31:03.332008 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:03.331923 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7aa3052-2427-47c1-a9de-366b2372f111-kube-api-access-d4kbs" pod "b7aa3052-2427-47c1-a9de-366b2372f111" (UID: "b7aa3052-2427-47c1-a9de-366b2372f111"). InnerVolumeSpecName "kube-api-access-d4kbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:31:03.431064 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:03.431026 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7aa3052-2427-47c1-a9de-366b2372f111-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:31:03.431064 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:03.431062 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4kbs\" (UniqueName: \"kubernetes.io/projected/b7aa3052-2427-47c1-a9de-366b2372f111-kube-api-access-d4kbs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:31:04.090223 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:04.090191 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pf9wq" Oct 31 05:31:04.090223 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:04.090205 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pf9wq" event={"ID":"b7aa3052-2427-47c1-a9de-366b2372f111","Type":"ContainerDied","Data":"cf40dc23be480779f412b78ba473a2da42f0349f5300c6d2cbbfb439830865f1"} Oct 31 05:31:04.090488 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:04.090251 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf40dc23be480779f412b78ba473a2da42f0349f5300c6d2cbbfb439830865f1" Oct 31 05:31:14.939356 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:14.938599 5019 scope.go:122] "RemoveContainer" containerID="65842166e029e69dcb5e53e86e9c8a92b30f78cc877bf5ccc2278658026dce50" Oct 31 05:31:19.065890 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:19.065849 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pf9wq"] Oct 31 05:31:19.077995 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:19.077953 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pf9wq"] Oct 31 05:31:19.117446 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:19.117412 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zjn6f"] Oct 31 05:31:19.117893 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:19.117879 5019 cpu_manager.go:414] "containerMap: removing container" podUID="b7aa3052-2427-47c1-a9de-366b2372f111" containerName="revoke-keys" Oct 31 05:31:19.117941 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:19.117897 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7aa3052-2427-47c1-a9de-366b2372f111" containerName="revoke-keys" Oct 31 05:31:19.118018 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:19.118008 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7aa3052-2427-47c1-a9de-366b2372f111" containerName="revoke-keys" Oct 31 05:31:19.122860 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:19.122834 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zjn6f" Oct 31 05:31:19.128362 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:19.128292 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72140dec-6c78-4eb6-a9a0-c8f54bd2b000-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-zjn6f\" (UID: \"72140dec-6c78-4eb6-a9a0-c8f54bd2b000\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zjn6f" Oct 31 05:31:19.128776 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:19.128744 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwbp2\" (UniqueName: \"kubernetes.io/projected/72140dec-6c78-4eb6-a9a0-c8f54bd2b000-kube-api-access-mwbp2\") pod \"maas-api-revoke-sub-3d4156f1-zjn6f\" (UID: \"72140dec-6c78-4eb6-a9a0-c8f54bd2b000\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zjn6f" Oct 31 05:31:19.128878 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:19.128864 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72140dec-6c78-4eb6-a9a0-c8f54bd2b000-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-zjn6f\" (UID: \"72140dec-6c78-4eb6-a9a0-c8f54bd2b000\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zjn6f" Oct 31 05:31:19.140625 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:19.140585 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfhzn\"" Oct 31 05:31:19.140790 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:19.140715 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:31:19.140790 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:19.140787 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:31:19.140936 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:19.140828 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:31:19.146489 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:19.146452 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zjn6f"] Oct 31 05:31:19.229754 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:19.229715 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72140dec-6c78-4eb6-a9a0-c8f54bd2b000-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-zjn6f\" (UID: \"72140dec-6c78-4eb6-a9a0-c8f54bd2b000\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zjn6f" Oct 31 05:31:19.229993 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:19.229761 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72140dec-6c78-4eb6-a9a0-c8f54bd2b000-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-zjn6f\" (UID: \"72140dec-6c78-4eb6-a9a0-c8f54bd2b000\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zjn6f" Oct 31 05:31:19.229993 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:19.229853 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwbp2\" (UniqueName: \"kubernetes.io/projected/72140dec-6c78-4eb6-a9a0-c8f54bd2b000-kube-api-access-mwbp2\") pod \"maas-api-revoke-sub-3d4156f1-zjn6f\" (UID: \"72140dec-6c78-4eb6-a9a0-c8f54bd2b000\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zjn6f" Oct 31 05:31:19.230551 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:19.230531 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72140dec-6c78-4eb6-a9a0-c8f54bd2b000-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-zjn6f\" (UID: \"72140dec-6c78-4eb6-a9a0-c8f54bd2b000\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zjn6f" Oct 31 05:31:19.232414 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:19.232396 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72140dec-6c78-4eb6-a9a0-c8f54bd2b000-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-zjn6f\" (UID: \"72140dec-6c78-4eb6-a9a0-c8f54bd2b000\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zjn6f" Oct 31 05:31:19.256274 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:19.256239 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwbp2\" (UniqueName: \"kubernetes.io/projected/72140dec-6c78-4eb6-a9a0-c8f54bd2b000-kube-api-access-mwbp2\") pod \"maas-api-revoke-sub-3d4156f1-zjn6f\" (UID: \"72140dec-6c78-4eb6-a9a0-c8f54bd2b000\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zjn6f" Oct 31 05:31:19.406975 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:19.406890 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7aa3052-2427-47c1-a9de-366b2372f111" path="/var/lib/kubelet/pods/b7aa3052-2427-47c1-a9de-366b2372f111/volumes" Oct 31 05:31:19.449027 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:19.448983 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zjn6f" Oct 31 05:31:19.748848 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:19.748798 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zjn6f"] Oct 31 05:31:19.753964 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:31:19.753926 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72140dec_6c78_4eb6_a9a0_c8f54bd2b000.slice/crio-c2b691dd307ac91499acda84fee064aefbef7a02a5764e809b971047d67ec385 WatchSource:0}: Error finding container c2b691dd307ac91499acda84fee064aefbef7a02a5764e809b971047d67ec385: Status 404 returned error can't find the container with id c2b691dd307ac91499acda84fee064aefbef7a02a5764e809b971047d67ec385 Oct 31 05:31:20.235684 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:20.235558 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zjn6f" event={"ID":"72140dec-6c78-4eb6-a9a0-c8f54bd2b000","Type":"ContainerStarted","Data":"cc2ee2fe2c7564e494fe48bd9217f29537653dcdd71aeed459749f4d24749e03"} Oct 31 05:31:20.235684 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:20.235644 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zjn6f" event={"ID":"72140dec-6c78-4eb6-a9a0-c8f54bd2b000","Type":"ContainerStarted","Data":"c2b691dd307ac91499acda84fee064aefbef7a02a5764e809b971047d67ec385"} Oct 31 05:31:20.276852 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:20.276785 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zjn6f" podStartSLOduration=1.276766908 podStartE2EDuration="1.276766908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:31:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:31:20.275074996 +0000 UTC m=+5235.031413039" watchObservedRunningTime="2026-10-31 05:31:20.276766908 +0000 UTC m=+5235.033104949" Oct 31 05:31:21.242982 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:21.242941 5019 generic.go:356] "Generic (PLEG): container finished" podID="72140dec-6c78-4eb6-a9a0-c8f54bd2b000" containerID="cc2ee2fe2c7564e494fe48bd9217f29537653dcdd71aeed459749f4d24749e03" exitCode=0 Oct 31 05:31:21.243475 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:21.243018 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zjn6f" event={"ID":"72140dec-6c78-4eb6-a9a0-c8f54bd2b000","Type":"ContainerDied","Data":"cc2ee2fe2c7564e494fe48bd9217f29537653dcdd71aeed459749f4d24749e03"} Oct 31 05:31:22.392319 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:31:22.392275 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:31:22.400865 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:22.400841 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zjn6f" Oct 31 05:31:22.464672 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:22.464636 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72140dec-6c78-4eb6-a9a0-c8f54bd2b000-kube-api-access-mwbp2\" (UniqueName: \"kubernetes.io/projected/72140dec-6c78-4eb6-a9a0-c8f54bd2b000-kube-api-access-mwbp2\") pod \"72140dec-6c78-4eb6-a9a0-c8f54bd2b000\" (UID: \"72140dec-6c78-4eb6-a9a0-c8f54bd2b000\") " Oct 31 05:31:22.464886 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:22.464733 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72140dec-6c78-4eb6-a9a0-c8f54bd2b000-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72140dec-6c78-4eb6-a9a0-c8f54bd2b000-maas-api-xks-ca\") pod \"72140dec-6c78-4eb6-a9a0-c8f54bd2b000\" (UID: \"72140dec-6c78-4eb6-a9a0-c8f54bd2b000\") " Oct 31 05:31:22.464886 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:22.464770 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72140dec-6c78-4eb6-a9a0-c8f54bd2b000-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72140dec-6c78-4eb6-a9a0-c8f54bd2b000-maas-api-openshift-ca\") pod \"72140dec-6c78-4eb6-a9a0-c8f54bd2b000\" (UID: \"72140dec-6c78-4eb6-a9a0-c8f54bd2b000\") " Oct 31 05:31:22.465221 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:22.465182 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72140dec-6c78-4eb6-a9a0-c8f54bd2b000-maas-api-openshift-ca" pod "72140dec-6c78-4eb6-a9a0-c8f54bd2b000" (UID: "72140dec-6c78-4eb6-a9a0-c8f54bd2b000"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:31:22.465596 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:22.465573 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72140dec-6c78-4eb6-a9a0-c8f54bd2b000-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:31:22.466991 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:22.466965 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72140dec-6c78-4eb6-a9a0-c8f54bd2b000-maas-api-xks-ca" pod "72140dec-6c78-4eb6-a9a0-c8f54bd2b000" (UID: "72140dec-6c78-4eb6-a9a0-c8f54bd2b000"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:31:22.467119 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:22.467034 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72140dec-6c78-4eb6-a9a0-c8f54bd2b000-kube-api-access-mwbp2" pod "72140dec-6c78-4eb6-a9a0-c8f54bd2b000" (UID: "72140dec-6c78-4eb6-a9a0-c8f54bd2b000"). InnerVolumeSpecName "kube-api-access-mwbp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:31:22.566699 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:22.566609 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwbp2\" (UniqueName: \"kubernetes.io/projected/72140dec-6c78-4eb6-a9a0-c8f54bd2b000-kube-api-access-mwbp2\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:31:22.566699 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:22.566643 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72140dec-6c78-4eb6-a9a0-c8f54bd2b000-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:31:23.256059 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:23.256026 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zjn6f" Oct 31 05:31:23.256059 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:23.256044 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zjn6f" event={"ID":"72140dec-6c78-4eb6-a9a0-c8f54bd2b000","Type":"ContainerDied","Data":"c2b691dd307ac91499acda84fee064aefbef7a02a5764e809b971047d67ec385"} Oct 31 05:31:23.256375 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:23.256086 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2b691dd307ac91499acda84fee064aefbef7a02a5764e809b971047d67ec385" Oct 31 05:31:45.706449 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:45.706313 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-hwcsq"] Oct 31 05:31:45.707079 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:45.706911 5019 cpu_manager.go:414] "containerMap: removing container" podUID="72140dec-6c78-4eb6-a9a0-c8f54bd2b000" containerName="revoke-keys" Oct 31 05:31:45.707079 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:45.706931 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72140dec-6c78-4eb6-a9a0-c8f54bd2b000" containerName="revoke-keys" Oct 31 05:31:45.707079 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:45.707023 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="72140dec-6c78-4eb6-a9a0-c8f54bd2b000" containerName="revoke-keys" Oct 31 05:31:45.710650 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:45.710628 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-hwcsq" Oct 31 05:31:45.725366 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:45.725326 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:31:45.729412 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:45.725517 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:31:45.729571 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:45.725564 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:31:45.729571 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:45.725892 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfhzn\"" Oct 31 05:31:45.729964 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:45.729939 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-hwcsq"] Oct 31 05:31:45.752618 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:45.752567 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff935065-618c-438b-aa40-4f2150df3ac5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-hwcsq\" (UID: \"ff935065-618c-438b-aa40-4f2150df3ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-hwcsq" Oct 31 05:31:45.752825 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:45.752760 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff935065-618c-438b-aa40-4f2150df3ac5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-hwcsq\" (UID: \"ff935065-618c-438b-aa40-4f2150df3ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-hwcsq" Oct 31 05:31:45.752825 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:45.752798 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd55g\" (UniqueName: \"kubernetes.io/projected/ff935065-618c-438b-aa40-4f2150df3ac5-kube-api-access-wd55g\") pod \"maas-api-revoke-sub-e2fc9c5b-hwcsq\" (UID: \"ff935065-618c-438b-aa40-4f2150df3ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-hwcsq" Oct 31 05:31:45.854304 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:45.854264 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff935065-618c-438b-aa40-4f2150df3ac5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-hwcsq\" (UID: \"ff935065-618c-438b-aa40-4f2150df3ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-hwcsq" Oct 31 05:31:45.854496 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:45.854375 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff935065-618c-438b-aa40-4f2150df3ac5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-hwcsq\" (UID: \"ff935065-618c-438b-aa40-4f2150df3ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-hwcsq" Oct 31 05:31:45.854496 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:45.854393 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd55g\" (UniqueName: \"kubernetes.io/projected/ff935065-618c-438b-aa40-4f2150df3ac5-kube-api-access-wd55g\") pod \"maas-api-revoke-sub-e2fc9c5b-hwcsq\" (UID: \"ff935065-618c-438b-aa40-4f2150df3ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-hwcsq" Oct 31 05:31:45.854949 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:45.854925 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff935065-618c-438b-aa40-4f2150df3ac5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-hwcsq\" (UID: \"ff935065-618c-438b-aa40-4f2150df3ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-hwcsq" Oct 31 05:31:45.856981 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:45.856956 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff935065-618c-438b-aa40-4f2150df3ac5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-hwcsq\" (UID: \"ff935065-618c-438b-aa40-4f2150df3ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-hwcsq" Oct 31 05:31:45.877448 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:45.877415 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd55g\" (UniqueName: \"kubernetes.io/projected/ff935065-618c-438b-aa40-4f2150df3ac5-kube-api-access-wd55g\") pod \"maas-api-revoke-sub-e2fc9c5b-hwcsq\" (UID: \"ff935065-618c-438b-aa40-4f2150df3ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-hwcsq" Oct 31 05:31:46.024707 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:46.024667 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-hwcsq" Oct 31 05:31:46.321466 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:46.321432 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-hwcsq"] Oct 31 05:31:46.322911 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:31:46.322885 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff935065_618c_438b_aa40_4f2150df3ac5.slice/crio-03c4140eacae725f5b84c1ceb2be44b91eb8e512e62c394b7a6e1825e07c4b0c WatchSource:0}: Error finding container 03c4140eacae725f5b84c1ceb2be44b91eb8e512e62c394b7a6e1825e07c4b0c: Status 404 returned error can't find the container with id 03c4140eacae725f5b84c1ceb2be44b91eb8e512e62c394b7a6e1825e07c4b0c Oct 31 05:31:46.423091 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:46.423051 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-hwcsq" event={"ID":"ff935065-618c-438b-aa40-4f2150df3ac5","Type":"ContainerStarted","Data":"03c4140eacae725f5b84c1ceb2be44b91eb8e512e62c394b7a6e1825e07c4b0c"} Oct 31 05:31:47.430036 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:47.429948 5019 generic.go:356] "Generic (PLEG): container finished" podID="ff935065-618c-438b-aa40-4f2150df3ac5" containerID="b210a73a5b2ce68e28f1322c6bb1c861aee6b766a656f44aad9f034830e7f7ff" exitCode=0 Oct 31 05:31:47.430442 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:47.430035 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-hwcsq" event={"ID":"ff935065-618c-438b-aa40-4f2150df3ac5","Type":"ContainerDied","Data":"b210a73a5b2ce68e28f1322c6bb1c861aee6b766a656f44aad9f034830e7f7ff"} Oct 31 05:31:48.587970 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:48.587945 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-hwcsq" Oct 31 05:31:48.685597 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:48.685561 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff935065-618c-438b-aa40-4f2150df3ac5-kube-api-access-wd55g\" (UniqueName: \"kubernetes.io/projected/ff935065-618c-438b-aa40-4f2150df3ac5-kube-api-access-wd55g\") pod \"ff935065-618c-438b-aa40-4f2150df3ac5\" (UID: \"ff935065-618c-438b-aa40-4f2150df3ac5\") " Oct 31 05:31:48.685754 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:48.685635 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff935065-618c-438b-aa40-4f2150df3ac5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff935065-618c-438b-aa40-4f2150df3ac5-maas-api-xks-ca\") pod \"ff935065-618c-438b-aa40-4f2150df3ac5\" (UID: \"ff935065-618c-438b-aa40-4f2150df3ac5\") " Oct 31 05:31:48.685754 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:48.685725 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff935065-618c-438b-aa40-4f2150df3ac5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff935065-618c-438b-aa40-4f2150df3ac5-maas-api-openshift-ca\") pod \"ff935065-618c-438b-aa40-4f2150df3ac5\" (UID: \"ff935065-618c-438b-aa40-4f2150df3ac5\") " Oct 31 05:31:48.686077 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:48.686054 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff935065-618c-438b-aa40-4f2150df3ac5-maas-api-openshift-ca" pod "ff935065-618c-438b-aa40-4f2150df3ac5" (UID: "ff935065-618c-438b-aa40-4f2150df3ac5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:31:48.687812 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:48.687763 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff935065-618c-438b-aa40-4f2150df3ac5-maas-api-xks-ca" pod "ff935065-618c-438b-aa40-4f2150df3ac5" (UID: "ff935065-618c-438b-aa40-4f2150df3ac5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:31:48.687893 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:48.687870 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff935065-618c-438b-aa40-4f2150df3ac5-kube-api-access-wd55g" pod "ff935065-618c-438b-aa40-4f2150df3ac5" (UID: "ff935065-618c-438b-aa40-4f2150df3ac5"). InnerVolumeSpecName "kube-api-access-wd55g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:31:48.787127 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:48.787092 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd55g\" (UniqueName: \"kubernetes.io/projected/ff935065-618c-438b-aa40-4f2150df3ac5-kube-api-access-wd55g\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:31:48.787127 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:48.787123 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff935065-618c-438b-aa40-4f2150df3ac5-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:31:48.787127 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:48.787133 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff935065-618c-438b-aa40-4f2150df3ac5-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:31:49.443491 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:49.443455 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-hwcsq" Oct 31 05:31:49.443692 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:49.443454 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-hwcsq" event={"ID":"ff935065-618c-438b-aa40-4f2150df3ac5","Type":"ContainerDied","Data":"03c4140eacae725f5b84c1ceb2be44b91eb8e512e62c394b7a6e1825e07c4b0c"} Oct 31 05:31:49.443692 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:49.443600 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03c4140eacae725f5b84c1ceb2be44b91eb8e512e62c394b7a6e1825e07c4b0c" Oct 31 05:31:52.390552 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:31:52.390509 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:31:56.056973 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:56.056930 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xc9fl"] Oct 31 05:31:56.057696 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:56.057666 5019 cpu_manager.go:414] "containerMap: removing container" podUID="ff935065-618c-438b-aa40-4f2150df3ac5" containerName="revoke-keys" Oct 31 05:31:56.057696 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:56.057695 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff935065-618c-438b-aa40-4f2150df3ac5" containerName="revoke-keys" Oct 31 05:31:56.057897 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:56.057780 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff935065-618c-438b-aa40-4f2150df3ac5" containerName="revoke-keys" Oct 31 05:31:56.061466 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:56.061439 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xc9fl" Oct 31 05:31:56.076589 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:56.076552 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:31:56.076778 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:56.076661 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:31:56.076778 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:56.076710 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfhzn\"" Oct 31 05:31:56.076778 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:56.076741 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:31:56.081412 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:56.081382 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xc9fl"] Oct 31 05:31:56.161152 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:56.161113 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h7x7\" (UniqueName: \"kubernetes.io/projected/a6a313f9-e056-49a9-9619-121fa96de322-kube-api-access-7h7x7\") pod \"maas-api-revoke-sub-5a29efb3-xc9fl\" (UID: \"a6a313f9-e056-49a9-9619-121fa96de322\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xc9fl" Oct 31 05:31:56.161152 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:56.161157 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6a313f9-e056-49a9-9619-121fa96de322-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-xc9fl\" (UID: \"a6a313f9-e056-49a9-9619-121fa96de322\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xc9fl" Oct 31 05:31:56.161420 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:56.161304 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6a313f9-e056-49a9-9619-121fa96de322-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-xc9fl\" (UID: \"a6a313f9-e056-49a9-9619-121fa96de322\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xc9fl" Oct 31 05:31:56.262665 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:56.262624 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6a313f9-e056-49a9-9619-121fa96de322-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-xc9fl\" (UID: \"a6a313f9-e056-49a9-9619-121fa96de322\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xc9fl" Oct 31 05:31:56.262885 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:56.262712 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h7x7\" (UniqueName: \"kubernetes.io/projected/a6a313f9-e056-49a9-9619-121fa96de322-kube-api-access-7h7x7\") pod \"maas-api-revoke-sub-5a29efb3-xc9fl\" (UID: \"a6a313f9-e056-49a9-9619-121fa96de322\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xc9fl" Oct 31 05:31:56.262885 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:56.262738 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6a313f9-e056-49a9-9619-121fa96de322-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-xc9fl\" (UID: \"a6a313f9-e056-49a9-9619-121fa96de322\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xc9fl" Oct 31 05:31:56.263519 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:56.263496 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6a313f9-e056-49a9-9619-121fa96de322-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-xc9fl\" (UID: \"a6a313f9-e056-49a9-9619-121fa96de322\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xc9fl" Oct 31 05:31:56.265070 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:56.265046 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6a313f9-e056-49a9-9619-121fa96de322-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-xc9fl\" (UID: \"a6a313f9-e056-49a9-9619-121fa96de322\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xc9fl" Oct 31 05:31:56.285948 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:56.285916 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h7x7\" (UniqueName: \"kubernetes.io/projected/a6a313f9-e056-49a9-9619-121fa96de322-kube-api-access-7h7x7\") pod \"maas-api-revoke-sub-5a29efb3-xc9fl\" (UID: \"a6a313f9-e056-49a9-9619-121fa96de322\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xc9fl" Oct 31 05:31:56.377603 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:56.377567 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xc9fl" Oct 31 05:31:56.674964 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:56.674933 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xc9fl"] Oct 31 05:31:56.676128 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:31:56.676098 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6a313f9_e056_49a9_9619_121fa96de322.slice/crio-3e8a65bca79fb5814928afec5fd44cfdfb37a81bb837d2dca97d16d39f0b984f WatchSource:0}: Error finding container 3e8a65bca79fb5814928afec5fd44cfdfb37a81bb837d2dca97d16d39f0b984f: Status 404 returned error can't find the container with id 3e8a65bca79fb5814928afec5fd44cfdfb37a81bb837d2dca97d16d39f0b984f Oct 31 05:31:57.498111 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:57.498071 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xc9fl" event={"ID":"a6a313f9-e056-49a9-9619-121fa96de322","Type":"ContainerStarted","Data":"26ff4e4976d329df1ccba34cdbdbf8d15fee4f013c3cf6816aa560d85969078f"} Oct 31 05:31:57.498601 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:57.498139 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xc9fl" event={"ID":"a6a313f9-e056-49a9-9619-121fa96de322","Type":"ContainerStarted","Data":"3e8a65bca79fb5814928afec5fd44cfdfb37a81bb837d2dca97d16d39f0b984f"} Oct 31 05:31:57.542818 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:57.542737 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xc9fl" podStartSLOduration=1.5427143810000001 podStartE2EDuration="1.542714381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:31:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:31:57.53883337 +0000 UTC m=+5272.295171414" watchObservedRunningTime="2026-10-31 05:31:57.542714381 +0000 UTC m=+5272.299052421" Oct 31 05:31:58.553894 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:58.553854 5019 generic.go:356] "Generic (PLEG): container finished" podID="a6a313f9-e056-49a9-9619-121fa96de322" containerID="26ff4e4976d329df1ccba34cdbdbf8d15fee4f013c3cf6816aa560d85969078f" exitCode=0 Oct 31 05:31:58.554685 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:31:58.554637 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xc9fl" event={"ID":"a6a313f9-e056-49a9-9619-121fa96de322","Type":"ContainerDied","Data":"26ff4e4976d329df1ccba34cdbdbf8d15fee4f013c3cf6816aa560d85969078f"} Oct 31 05:32:00.387559 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:00.387138 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xc9fl" Oct 31 05:32:00.400712 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:00.400675 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6a313f9-e056-49a9-9619-121fa96de322-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6a313f9-e056-49a9-9619-121fa96de322-maas-api-xks-ca\") pod \"a6a313f9-e056-49a9-9619-121fa96de322\" (UID: \"a6a313f9-e056-49a9-9619-121fa96de322\") " Oct 31 05:32:00.401052 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:00.401026 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6a313f9-e056-49a9-9619-121fa96de322-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6a313f9-e056-49a9-9619-121fa96de322-maas-api-openshift-ca\") pod \"a6a313f9-e056-49a9-9619-121fa96de322\" (UID: \"a6a313f9-e056-49a9-9619-121fa96de322\") " Oct 31 05:32:00.401274 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:00.401251 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6a313f9-e056-49a9-9619-121fa96de322-kube-api-access-7h7x7\" (UniqueName: \"kubernetes.io/projected/a6a313f9-e056-49a9-9619-121fa96de322-kube-api-access-7h7x7\") pod \"a6a313f9-e056-49a9-9619-121fa96de322\" (UID: \"a6a313f9-e056-49a9-9619-121fa96de322\") " Oct 31 05:32:00.402673 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:00.402627 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6a313f9-e056-49a9-9619-121fa96de322-maas-api-openshift-ca" pod "a6a313f9-e056-49a9-9619-121fa96de322" (UID: "a6a313f9-e056-49a9-9619-121fa96de322"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:32:00.405631 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:00.405606 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6a313f9-e056-49a9-9619-121fa96de322-maas-api-xks-ca" pod "a6a313f9-e056-49a9-9619-121fa96de322" (UID: "a6a313f9-e056-49a9-9619-121fa96de322"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:32:00.405918 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:00.405887 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6a313f9-e056-49a9-9619-121fa96de322-kube-api-access-7h7x7" pod "a6a313f9-e056-49a9-9619-121fa96de322" (UID: "a6a313f9-e056-49a9-9619-121fa96de322"). InnerVolumeSpecName "kube-api-access-7h7x7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:32:00.503381 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:00.503320 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6a313f9-e056-49a9-9619-121fa96de322-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:32:00.503381 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:00.503385 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7h7x7\" (UniqueName: \"kubernetes.io/projected/a6a313f9-e056-49a9-9619-121fa96de322-kube-api-access-7h7x7\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:32:00.503643 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:00.503405 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6a313f9-e056-49a9-9619-121fa96de322-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:32:00.590359 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:00.590313 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xc9fl" Oct 31 05:32:00.590540 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:00.590316 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xc9fl" event={"ID":"a6a313f9-e056-49a9-9619-121fa96de322","Type":"ContainerDied","Data":"3e8a65bca79fb5814928afec5fd44cfdfb37a81bb837d2dca97d16d39f0b984f"} Oct 31 05:32:00.590540 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:00.590447 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e8a65bca79fb5814928afec5fd44cfdfb37a81bb837d2dca97d16d39f0b984f" Oct 31 05:32:12.034755 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:12.034711 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zjn6f"] Oct 31 05:32:12.046233 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:12.046204 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zjn6f"] Oct 31 05:32:12.078212 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:12.078179 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fb7p4"] Oct 31 05:32:12.078695 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:12.078681 5019 cpu_manager.go:414] "containerMap: removing container" podUID="a6a313f9-e056-49a9-9619-121fa96de322" containerName="revoke-keys" Oct 31 05:32:12.078740 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:12.078697 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6a313f9-e056-49a9-9619-121fa96de322" containerName="revoke-keys" Oct 31 05:32:12.078810 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:12.078802 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6a313f9-e056-49a9-9619-121fa96de322" containerName="revoke-keys" Oct 31 05:32:12.082159 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:12.082137 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fb7p4" Oct 31 05:32:12.098221 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:12.098192 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:32:12.098400 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:12.098313 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:32:12.098924 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:12.098900 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:32:12.099135 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:12.099113 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfhzn\"" Oct 31 05:32:12.102653 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:12.102627 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fb7p4"] Oct 31 05:32:12.118995 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:12.118952 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca41b364-4695-43b5-809a-dd5152c7a71b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fb7p4\" (UID: \"ca41b364-4695-43b5-809a-dd5152c7a71b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fb7p4" Oct 31 05:32:12.119197 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:12.119134 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6t7x\" (UniqueName: \"kubernetes.io/projected/ca41b364-4695-43b5-809a-dd5152c7a71b-kube-api-access-m6t7x\") pod \"maas-api-revoke-sub-3d4156f1-fb7p4\" (UID: \"ca41b364-4695-43b5-809a-dd5152c7a71b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fb7p4" Oct 31 05:32:12.119249 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:12.119199 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca41b364-4695-43b5-809a-dd5152c7a71b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fb7p4\" (UID: \"ca41b364-4695-43b5-809a-dd5152c7a71b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fb7p4" Oct 31 05:32:12.220063 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:12.220020 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6t7x\" (UniqueName: \"kubernetes.io/projected/ca41b364-4695-43b5-809a-dd5152c7a71b-kube-api-access-m6t7x\") pod \"maas-api-revoke-sub-3d4156f1-fb7p4\" (UID: \"ca41b364-4695-43b5-809a-dd5152c7a71b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fb7p4" Oct 31 05:32:12.220316 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:12.220073 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca41b364-4695-43b5-809a-dd5152c7a71b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fb7p4\" (UID: \"ca41b364-4695-43b5-809a-dd5152c7a71b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fb7p4" Oct 31 05:32:12.220316 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:12.220135 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca41b364-4695-43b5-809a-dd5152c7a71b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fb7p4\" (UID: \"ca41b364-4695-43b5-809a-dd5152c7a71b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fb7p4" Oct 31 05:32:12.220811 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:12.220779 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca41b364-4695-43b5-809a-dd5152c7a71b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fb7p4\" (UID: \"ca41b364-4695-43b5-809a-dd5152c7a71b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fb7p4" Oct 31 05:32:12.222842 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:12.222796 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca41b364-4695-43b5-809a-dd5152c7a71b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fb7p4\" (UID: \"ca41b364-4695-43b5-809a-dd5152c7a71b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fb7p4" Oct 31 05:32:12.243850 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:12.243812 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6t7x\" (UniqueName: \"kubernetes.io/projected/ca41b364-4695-43b5-809a-dd5152c7a71b-kube-api-access-m6t7x\") pod \"maas-api-revoke-sub-3d4156f1-fb7p4\" (UID: \"ca41b364-4695-43b5-809a-dd5152c7a71b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fb7p4" Oct 31 05:32:12.397253 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:12.397158 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fb7p4" Oct 31 05:32:12.693454 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:12.693416 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fb7p4"] Oct 31 05:32:12.694435 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:32:12.694400 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca41b364_4695_43b5_809a_dd5152c7a71b.slice/crio-fe6726fe5851d75cebf2876829309f124cbbc4e604e185fa3aed46e19d30f605 WatchSource:0}: Error finding container fe6726fe5851d75cebf2876829309f124cbbc4e604e185fa3aed46e19d30f605: Status 404 returned error can't find the container with id fe6726fe5851d75cebf2876829309f124cbbc4e604e185fa3aed46e19d30f605 Oct 31 05:32:13.406835 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:13.406754 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72140dec-6c78-4eb6-a9a0-c8f54bd2b000" path="/var/lib/kubelet/pods/72140dec-6c78-4eb6-a9a0-c8f54bd2b000/volumes" Oct 31 05:32:13.671577 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:13.671481 5019 generic.go:356] "Generic (PLEG): container finished" podID="ca41b364-4695-43b5-809a-dd5152c7a71b" containerID="c3755d69419d503ef6678b7fc8acf9117f40a7937ac73acf36ba4255fb390272" exitCode=0 Oct 31 05:32:13.671750 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:13.671578 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fb7p4" event={"ID":"ca41b364-4695-43b5-809a-dd5152c7a71b","Type":"ContainerDied","Data":"c3755d69419d503ef6678b7fc8acf9117f40a7937ac73acf36ba4255fb390272"} Oct 31 05:32:13.671750 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:13.671641 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fb7p4" event={"ID":"ca41b364-4695-43b5-809a-dd5152c7a71b","Type":"ContainerStarted","Data":"fe6726fe5851d75cebf2876829309f124cbbc4e604e185fa3aed46e19d30f605"} Oct 31 05:32:14.833048 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:14.833012 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fb7p4" Oct 31 05:32:14.946957 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:14.946860 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca41b364-4695-43b5-809a-dd5152c7a71b-kube-api-access-m6t7x\" (UniqueName: \"kubernetes.io/projected/ca41b364-4695-43b5-809a-dd5152c7a71b-kube-api-access-m6t7x\") pod \"ca41b364-4695-43b5-809a-dd5152c7a71b\" (UID: \"ca41b364-4695-43b5-809a-dd5152c7a71b\") " Oct 31 05:32:14.947148 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:14.947006 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca41b364-4695-43b5-809a-dd5152c7a71b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca41b364-4695-43b5-809a-dd5152c7a71b-maas-api-xks-ca\") pod \"ca41b364-4695-43b5-809a-dd5152c7a71b\" (UID: \"ca41b364-4695-43b5-809a-dd5152c7a71b\") " Oct 31 05:32:14.947148 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:14.947072 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca41b364-4695-43b5-809a-dd5152c7a71b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca41b364-4695-43b5-809a-dd5152c7a71b-maas-api-openshift-ca\") pod \"ca41b364-4695-43b5-809a-dd5152c7a71b\" (UID: \"ca41b364-4695-43b5-809a-dd5152c7a71b\") " Oct 31 05:32:14.947556 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:14.947518 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca41b364-4695-43b5-809a-dd5152c7a71b-maas-api-openshift-ca" pod "ca41b364-4695-43b5-809a-dd5152c7a71b" (UID: "ca41b364-4695-43b5-809a-dd5152c7a71b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:32:14.949196 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:14.949170 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca41b364-4695-43b5-809a-dd5152c7a71b-maas-api-xks-ca" pod "ca41b364-4695-43b5-809a-dd5152c7a71b" (UID: "ca41b364-4695-43b5-809a-dd5152c7a71b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:32:14.949277 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:14.949223 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca41b364-4695-43b5-809a-dd5152c7a71b-kube-api-access-m6t7x" pod "ca41b364-4695-43b5-809a-dd5152c7a71b" (UID: "ca41b364-4695-43b5-809a-dd5152c7a71b"). InnerVolumeSpecName "kube-api-access-m6t7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:32:15.010073 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:15.010032 5019 scope.go:122] "RemoveContainer" containerID="e242b56caad567fcecc1a2d6ca7f3fa0174b025389e3cb01a64793ad1e6430a8" Oct 31 05:32:15.020040 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:15.020018 5019 scope.go:122] "RemoveContainer" containerID="f6d8dd3b056da28d8807fb058affcae31bf0b54c829c548fe082b425bae07055" Oct 31 05:32:15.048307 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:15.048279 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6t7x\" (UniqueName: \"kubernetes.io/projected/ca41b364-4695-43b5-809a-dd5152c7a71b-kube-api-access-m6t7x\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:32:15.048472 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:15.048309 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca41b364-4695-43b5-809a-dd5152c7a71b-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:32:15.048472 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:15.048323 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca41b364-4695-43b5-809a-dd5152c7a71b-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:32:15.685049 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:15.684959 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fb7p4" Oct 31 05:32:15.685049 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:15.685000 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fb7p4" event={"ID":"ca41b364-4695-43b5-809a-dd5152c7a71b","Type":"ContainerDied","Data":"fe6726fe5851d75cebf2876829309f124cbbc4e604e185fa3aed46e19d30f605"} Oct 31 05:32:15.685268 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:15.685058 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe6726fe5851d75cebf2876829309f124cbbc4e604e185fa3aed46e19d30f605" Oct 31 05:32:22.396689 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:32:22.396621 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:32:33.705872 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:33.705825 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-9632aa-55bd8dg9nd"] Oct 31 05:32:33.706538 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:33.706513 5019 cpu_manager.go:414] "containerMap: removing container" podUID="ca41b364-4695-43b5-809a-dd5152c7a71b" containerName="revoke-keys" Oct 31 05:32:33.706651 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:33.706540 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca41b364-4695-43b5-809a-dd5152c7a71b" containerName="revoke-keys" Oct 31 05:32:33.706714 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:33.706686 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca41b364-4695-43b5-809a-dd5152c7a71b" containerName="revoke-keys" Oct 31 05:32:33.710268 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:33.710244 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9632aa-55bd8dg9nd" Oct 31 05:32:33.730367 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:33.730321 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-9632aa-55bd8dg9nd"] Oct 31 05:32:33.739280 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:33.739216 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-9632aa\"" Oct 31 05:32:33.739280 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:33.739246 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-bf915a09-dockercfg-q5lb9\"" Oct 31 05:32:33.806309 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:33.806248 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-9632aa-687df6dccdb2v2"] Oct 31 05:32:33.811919 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:33.811889 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9632aa-687df6dccdb2v2" Oct 31 05:32:33.819042 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:33.819011 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-9632aa-687df6dccdb2v2"] Oct 31 05:32:33.851761 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:33.851712 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19a1b418-8cfc-403c-90f6-b085ef6c9e8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-9632aa-55bd8dg9nd\" (UID: \"19a1b418-8cfc-403c-90f6-b085ef6c9e8d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9632aa-55bd8dg9nd" Oct 31 05:32:33.851955 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:33.851924 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjn5j\" (UniqueName: \"kubernetes.io/projected/19a1b418-8cfc-403c-90f6-b085ef6c9e8d-kube-api-access-wjn5j\") pod \"payload-pre-processing-e2e-worker-single-main-9632aa-55bd8dg9nd\" (UID: \"19a1b418-8cfc-403c-90f6-b085ef6c9e8d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9632aa-55bd8dg9nd" Oct 31 05:32:33.953588 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:33.953538 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19a1b418-8cfc-403c-90f6-b085ef6c9e8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-9632aa-55bd8dg9nd\" (UID: \"19a1b418-8cfc-403c-90f6-b085ef6c9e8d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9632aa-55bd8dg9nd" Oct 31 05:32:33.953793 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:33.953613 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3331e5f-ac28-4beb-acb6-176075a5bee1-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-9632aa-687df6dccdb2v2\" (UID: \"f3331e5f-ac28-4beb-acb6-176075a5bee1\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-9632aa-687df6dccdb2v2" Oct 31 05:32:33.953793 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:33.953639 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksz4v\" (UniqueName: \"kubernetes.io/projected/f3331e5f-ac28-4beb-acb6-176075a5bee1-kube-api-access-ksz4v\") pod \"payload-processing-e2e-worker-single-main-9632aa-687df6dccdb2v2\" (UID: \"f3331e5f-ac28-4beb-acb6-176075a5bee1\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-9632aa-687df6dccdb2v2" Oct 31 05:32:33.953793 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:33.953770 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjn5j\" (UniqueName: \"kubernetes.io/projected/19a1b418-8cfc-403c-90f6-b085ef6c9e8d-kube-api-access-wjn5j\") pod \"payload-pre-processing-e2e-worker-single-main-9632aa-55bd8dg9nd\" (UID: \"19a1b418-8cfc-403c-90f6-b085ef6c9e8d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9632aa-55bd8dg9nd" Oct 31 05:32:33.954452 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:33.954431 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19a1b418-8cfc-403c-90f6-b085ef6c9e8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-9632aa-55bd8dg9nd\" (UID: \"19a1b418-8cfc-403c-90f6-b085ef6c9e8d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9632aa-55bd8dg9nd" Oct 31 05:32:33.977405 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:33.977354 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjn5j\" (UniqueName: \"kubernetes.io/projected/19a1b418-8cfc-403c-90f6-b085ef6c9e8d-kube-api-access-wjn5j\") pod \"payload-pre-processing-e2e-worker-single-main-9632aa-55bd8dg9nd\" (UID: \"19a1b418-8cfc-403c-90f6-b085ef6c9e8d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9632aa-55bd8dg9nd" Oct 31 05:32:34.024146 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:34.024096 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9632aa-55bd8dg9nd" Oct 31 05:32:34.054774 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:34.054727 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3331e5f-ac28-4beb-acb6-176075a5bee1-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-9632aa-687df6dccdb2v2\" (UID: \"f3331e5f-ac28-4beb-acb6-176075a5bee1\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-9632aa-687df6dccdb2v2" Oct 31 05:32:34.054774 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:34.054773 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksz4v\" (UniqueName: \"kubernetes.io/projected/f3331e5f-ac28-4beb-acb6-176075a5bee1-kube-api-access-ksz4v\") pod \"payload-processing-e2e-worker-single-main-9632aa-687df6dccdb2v2\" (UID: \"f3331e5f-ac28-4beb-acb6-176075a5bee1\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-9632aa-687df6dccdb2v2" Oct 31 05:32:34.055879 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:34.055853 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3331e5f-ac28-4beb-acb6-176075a5bee1-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-9632aa-687df6dccdb2v2\" (UID: \"f3331e5f-ac28-4beb-acb6-176075a5bee1\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-9632aa-687df6dccdb2v2" Oct 31 05:32:34.078570 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:34.078528 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksz4v\" (UniqueName: \"kubernetes.io/projected/f3331e5f-ac28-4beb-acb6-176075a5bee1-kube-api-access-ksz4v\") pod \"payload-processing-e2e-worker-single-main-9632aa-687df6dccdb2v2\" (UID: \"f3331e5f-ac28-4beb-acb6-176075a5bee1\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-9632aa-687df6dccdb2v2" Oct 31 05:32:34.130269 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:34.130224 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9632aa-687df6dccdb2v2" Oct 31 05:32:34.328133 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:34.328092 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-9632aa-55bd8dg9nd"] Oct 31 05:32:34.329324 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:32:34.329289 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19a1b418_8cfc_403c_90f6_b085ef6c9e8d.slice/crio-916d295c1350f32971c3f0b3b0304fb31333513a01f329c93bed02cf48a77060 WatchSource:0}: Error finding container 916d295c1350f32971c3f0b3b0304fb31333513a01f329c93bed02cf48a77060: Status 404 returned error can't find the container with id 916d295c1350f32971c3f0b3b0304fb31333513a01f329c93bed02cf48a77060 Oct 31 05:32:34.436621 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:34.436587 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-9632aa-687df6dccdb2v2"] Oct 31 05:32:34.438259 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:32:34.438228 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3331e5f_ac28_4beb_acb6_176075a5bee1.slice/crio-144e3283adeb234d12977649aa5291187bb418f52b5bc7053c36c5500ab30147 WatchSource:0}: Error finding container 144e3283adeb234d12977649aa5291187bb418f52b5bc7053c36c5500ab30147: Status 404 returned error can't find the container with id 144e3283adeb234d12977649aa5291187bb418f52b5bc7053c36c5500ab30147 Oct 31 05:32:34.824633 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:34.824589 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9632aa-55bd8dg9nd" event={"ID":"19a1b418-8cfc-403c-90f6-b085ef6c9e8d","Type":"ContainerStarted","Data":"fbf6d27d88e5fd42272df69e820c06165f257fcac7bdae603467ba652d82f791"} Oct 31 05:32:34.825127 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:34.824658 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9632aa-55bd8dg9nd" event={"ID":"19a1b418-8cfc-403c-90f6-b085ef6c9e8d","Type":"ContainerStarted","Data":"916d295c1350f32971c3f0b3b0304fb31333513a01f329c93bed02cf48a77060"} Oct 31 05:32:34.825127 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:34.824687 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9632aa-55bd8dg9nd" Oct 31 05:32:34.826370 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:34.826317 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9632aa-687df6dccdb2v2" event={"ID":"f3331e5f-ac28-4beb-acb6-176075a5bee1","Type":"ContainerStarted","Data":"6e8c3049f9b4ad320e3880952527e75783584628f70df35b9e92b0cba7799ff7"} Oct 31 05:32:34.826503 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:34.826397 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9632aa-687df6dccdb2v2" Oct 31 05:32:34.826503 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:34.826415 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9632aa-687df6dccdb2v2" event={"ID":"f3331e5f-ac28-4beb-acb6-176075a5bee1","Type":"ContainerStarted","Data":"144e3283adeb234d12977649aa5291187bb418f52b5bc7053c36c5500ab30147"} Oct 31 05:32:34.869990 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:34.869933 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9632aa-55bd8dg9nd" podStartSLOduration=1.869917856 podStartE2EDuration="1.869917856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:32:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:32:34.867001881 +0000 UTC m=+5309.623339921" watchObservedRunningTime="2026-10-31 05:32:34.869917856 +0000 UTC m=+5309.626255895" Oct 31 05:32:34.907433 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:34.907366 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9632aa-687df6dccdb2v2" podStartSLOduration=1.907319358 podStartE2EDuration="1.907319358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:32:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:32:34.90684092 +0000 UTC m=+5309.663178960" watchObservedRunningTime="2026-10-31 05:32:34.907319358 +0000 UTC m=+5309.663657596" Oct 31 05:32:45.835680 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:45.835643 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9632aa-687df6dccdb2v2" Oct 31 05:32:45.836114 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:45.835862 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9632aa-55bd8dg9nd" Oct 31 05:32:50.052610 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:50.052572 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-9632aa/simulator-single-main-9632aa-kserve-6b9cc66dbb-sx22m"] Oct 31 05:32:50.057948 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:50.057913 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-9632aa/simulator-single-main-9632aa-kserve-6b9cc66dbb-sx22m" Oct 31 05:32:50.077398 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:50.077318 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-9632aa/simulator-single-main-9632aa-kserve-6b9cc66dbb-sx22m"] Oct 31 05:32:50.085185 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:50.085116 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-9632aa\"/\"kube-root-ca.crt\"" Oct 31 05:32:50.085185 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:50.085137 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-9632aa\"/\"openshift-service-ca.crt\"" Oct 31 05:32:50.085473 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:50.085245 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-9632aa\"/\"default-dockercfg-s6rj6\"" Oct 31 05:32:50.085671 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:50.085650 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-9632aa\"/\"simulator-single-main-9632aa-kserve-self-signed-certs\"" Oct 31 05:32:50.130191 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:50.130142 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b-dshm\") pod \"simulator-single-main-9632aa-kserve-6b9cc66dbb-sx22m\" (UID: \"ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b\") " pod="e2e-models-e2e-worker-single-main-9632aa/simulator-single-main-9632aa-kserve-6b9cc66dbb-sx22m" Oct 31 05:32:50.130432 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:50.130274 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b-tmp-dir\") pod \"simulator-single-main-9632aa-kserve-6b9cc66dbb-sx22m\" (UID: \"ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b\") " pod="e2e-models-e2e-worker-single-main-9632aa/simulator-single-main-9632aa-kserve-6b9cc66dbb-sx22m" Oct 31 05:32:50.130432 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:50.130311 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8xqn\" (UniqueName: \"kubernetes.io/projected/ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b-kube-api-access-r8xqn\") pod \"simulator-single-main-9632aa-kserve-6b9cc66dbb-sx22m\" (UID: \"ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b\") " pod="e2e-models-e2e-worker-single-main-9632aa/simulator-single-main-9632aa-kserve-6b9cc66dbb-sx22m" Oct 31 05:32:50.130570 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:50.130435 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b-model-cache\") pod \"simulator-single-main-9632aa-kserve-6b9cc66dbb-sx22m\" (UID: \"ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b\") " pod="e2e-models-e2e-worker-single-main-9632aa/simulator-single-main-9632aa-kserve-6b9cc66dbb-sx22m" Oct 31 05:32:50.130570 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:50.130514 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b-home\") pod \"simulator-single-main-9632aa-kserve-6b9cc66dbb-sx22m\" (UID: \"ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b\") " pod="e2e-models-e2e-worker-single-main-9632aa/simulator-single-main-9632aa-kserve-6b9cc66dbb-sx22m" Oct 31 05:32:50.130685 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:50.130604 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b-tls-certs\") pod \"simulator-single-main-9632aa-kserve-6b9cc66dbb-sx22m\" (UID: \"ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b\") " pod="e2e-models-e2e-worker-single-main-9632aa/simulator-single-main-9632aa-kserve-6b9cc66dbb-sx22m" Oct 31 05:32:50.232196 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:50.232149 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b-model-cache\") pod \"simulator-single-main-9632aa-kserve-6b9cc66dbb-sx22m\" (UID: \"ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b\") " pod="e2e-models-e2e-worker-single-main-9632aa/simulator-single-main-9632aa-kserve-6b9cc66dbb-sx22m" Oct 31 05:32:50.232460 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:50.232260 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b-home\") pod \"simulator-single-main-9632aa-kserve-6b9cc66dbb-sx22m\" (UID: \"ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b\") " pod="e2e-models-e2e-worker-single-main-9632aa/simulator-single-main-9632aa-kserve-6b9cc66dbb-sx22m" Oct 31 05:32:50.232460 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:50.232297 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b-tls-certs\") pod \"simulator-single-main-9632aa-kserve-6b9cc66dbb-sx22m\" (UID: \"ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b\") " pod="e2e-models-e2e-worker-single-main-9632aa/simulator-single-main-9632aa-kserve-6b9cc66dbb-sx22m" Oct 31 05:32:50.232640 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:50.232498 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b-dshm\") pod \"simulator-single-main-9632aa-kserve-6b9cc66dbb-sx22m\" (UID: \"ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b\") " pod="e2e-models-e2e-worker-single-main-9632aa/simulator-single-main-9632aa-kserve-6b9cc66dbb-sx22m" Oct 31 05:32:50.232640 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:50.232600 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b-tmp-dir\") pod \"simulator-single-main-9632aa-kserve-6b9cc66dbb-sx22m\" (UID: \"ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b\") " pod="e2e-models-e2e-worker-single-main-9632aa/simulator-single-main-9632aa-kserve-6b9cc66dbb-sx22m" Oct 31 05:32:50.232640 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:50.232633 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8xqn\" (UniqueName: \"kubernetes.io/projected/ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b-kube-api-access-r8xqn\") pod \"simulator-single-main-9632aa-kserve-6b9cc66dbb-sx22m\" (UID: \"ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b\") " pod="e2e-models-e2e-worker-single-main-9632aa/simulator-single-main-9632aa-kserve-6b9cc66dbb-sx22m" Oct 31 05:32:50.232834 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:50.232677 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b-model-cache\") pod \"simulator-single-main-9632aa-kserve-6b9cc66dbb-sx22m\" (UID: \"ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b\") " pod="e2e-models-e2e-worker-single-main-9632aa/simulator-single-main-9632aa-kserve-6b9cc66dbb-sx22m" Oct 31 05:32:50.232896 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:50.232859 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b-home\") pod \"simulator-single-main-9632aa-kserve-6b9cc66dbb-sx22m\" (UID: \"ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b\") " pod="e2e-models-e2e-worker-single-main-9632aa/simulator-single-main-9632aa-kserve-6b9cc66dbb-sx22m" Oct 31 05:32:50.233117 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:50.233094 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b-tmp-dir\") pod \"simulator-single-main-9632aa-kserve-6b9cc66dbb-sx22m\" (UID: \"ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b\") " pod="e2e-models-e2e-worker-single-main-9632aa/simulator-single-main-9632aa-kserve-6b9cc66dbb-sx22m" Oct 31 05:32:50.234880 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:50.234855 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b-dshm\") pod \"simulator-single-main-9632aa-kserve-6b9cc66dbb-sx22m\" (UID: \"ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b\") " pod="e2e-models-e2e-worker-single-main-9632aa/simulator-single-main-9632aa-kserve-6b9cc66dbb-sx22m" Oct 31 05:32:50.235184 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:50.235162 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b-tls-certs\") pod \"simulator-single-main-9632aa-kserve-6b9cc66dbb-sx22m\" (UID: \"ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b\") " pod="e2e-models-e2e-worker-single-main-9632aa/simulator-single-main-9632aa-kserve-6b9cc66dbb-sx22m" Oct 31 05:32:50.256269 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:50.256232 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8xqn\" (UniqueName: \"kubernetes.io/projected/ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b-kube-api-access-r8xqn\") pod \"simulator-single-main-9632aa-kserve-6b9cc66dbb-sx22m\" (UID: \"ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b\") " pod="e2e-models-e2e-worker-single-main-9632aa/simulator-single-main-9632aa-kserve-6b9cc66dbb-sx22m" Oct 31 05:32:50.377035 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:50.376916 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-9632aa/simulator-single-main-9632aa-kserve-6b9cc66dbb-sx22m" Oct 31 05:32:50.688717 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:50.688678 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-9632aa/simulator-single-main-9632aa-kserve-6b9cc66dbb-sx22m"] Oct 31 05:32:50.689516 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:32:50.689477 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podecee9d31_9fd4_4d37_bfa8_8f3f80b3af1b.slice/crio-f53bacf36c5ca66ff3b99f5c18e0e9bfc75b65647ce2717eac038f0707e1e178 WatchSource:0}: Error finding container f53bacf36c5ca66ff3b99f5c18e0e9bfc75b65647ce2717eac038f0707e1e178: Status 404 returned error can't find the container with id f53bacf36c5ca66ff3b99f5c18e0e9bfc75b65647ce2717eac038f0707e1e178 Oct 31 05:32:50.932233 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:50.932116 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-9632aa/simulator-single-main-9632aa-kserve-6b9cc66dbb-sx22m" event={"ID":"ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b","Type":"ContainerStarted","Data":"a1afd7c558fa603a749ddf2edcec928cbeca144d2394a7de5f8511027cdbae1b"} Oct 31 05:32:50.932233 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:50.932180 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-9632aa/simulator-single-main-9632aa-kserve-6b9cc66dbb-sx22m" event={"ID":"ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b","Type":"ContainerStarted","Data":"f53bacf36c5ca66ff3b99f5c18e0e9bfc75b65647ce2717eac038f0707e1e178"} Oct 31 05:32:50.975203 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:50.975133 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-9632aa/simulator-single-main-9632aa-kserve-6b9cc66dbb-sx22m" podStartSLOduration=0.975116609 podStartE2EDuration="975.116609ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:32:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:32:50.972857931 +0000 UTC m=+5325.729195970" watchObservedRunningTime="2026-10-31 05:32:50.975116609 +0000 UTC m=+5325.731454648" Oct 31 05:32:51.377168 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:32:51.377124 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-9632aa/simulator-single-main-9632aa-kserve-6b9cc66dbb-sx22m" Oct 31 05:32:52.405419 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:32:52.405269 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:33:00.377312 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:00.377262 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-9632aa/simulator-single-main-9632aa-kserve-6b9cc66dbb-sx22m" Oct 31 05:33:00.389844 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:00.389801 5019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-9632aa/simulator-single-main-9632aa-kserve-6b9cc66dbb-sx22m" Oct 31 05:33:01.011313 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:01.011276 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-9632aa/simulator-single-main-9632aa-kserve-6b9cc66dbb-sx22m" Oct 31 05:33:09.030835 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:09.030753 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-zbjhj"] Oct 31 05:33:09.054264 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:09.054221 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-zbjhj"] Oct 31 05:33:09.407206 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:09.407108 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb507ac0-aec2-4f6e-af5a-83fc6a3ab9b8" path="/var/lib/kubelet/pods/eb507ac0-aec2-4f6e-af5a-83fc6a3ab9b8/volumes" Oct 31 05:33:21.146593 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:21.146554 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51342272-dlmwk"] Oct 31 05:33:21.150524 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:21.150500 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51342272-dlmwk" Oct 31 05:33:21.169508 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:21.169467 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51342272-dlmwk"] Oct 31 05:33:21.178123 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:21.178073 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:33:21.178123 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:21.178106 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:33:21.178389 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:21.178073 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:33:21.178389 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:21.178150 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfhzn\"" Oct 31 05:33:21.277188 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:21.277137 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b028b40-5fa7-4e54-aab6-af205d402646-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51342272-dlmwk\" (UID: \"9b028b40-5fa7-4e54-aab6-af205d402646\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51342272-dlmwk" Oct 31 05:33:21.277472 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:21.277202 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtf5v\" (UniqueName: \"kubernetes.io/projected/9b028b40-5fa7-4e54-aab6-af205d402646-kube-api-access-dtf5v\") pod \"maas-api-revoke-sub-51342272-dlmwk\" (UID: \"9b028b40-5fa7-4e54-aab6-af205d402646\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51342272-dlmwk" Oct 31 05:33:21.277472 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:21.277275 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b028b40-5fa7-4e54-aab6-af205d402646-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51342272-dlmwk\" (UID: \"9b028b40-5fa7-4e54-aab6-af205d402646\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51342272-dlmwk" Oct 31 05:33:21.378241 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:21.378201 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtf5v\" (UniqueName: \"kubernetes.io/projected/9b028b40-5fa7-4e54-aab6-af205d402646-kube-api-access-dtf5v\") pod \"maas-api-revoke-sub-51342272-dlmwk\" (UID: \"9b028b40-5fa7-4e54-aab6-af205d402646\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51342272-dlmwk" Oct 31 05:33:21.378241 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:21.378249 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b028b40-5fa7-4e54-aab6-af205d402646-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51342272-dlmwk\" (UID: \"9b028b40-5fa7-4e54-aab6-af205d402646\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51342272-dlmwk" Oct 31 05:33:21.378634 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:21.378447 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b028b40-5fa7-4e54-aab6-af205d402646-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51342272-dlmwk\" (UID: \"9b028b40-5fa7-4e54-aab6-af205d402646\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51342272-dlmwk" Oct 31 05:33:21.378965 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:21.378939 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b028b40-5fa7-4e54-aab6-af205d402646-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51342272-dlmwk\" (UID: \"9b028b40-5fa7-4e54-aab6-af205d402646\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51342272-dlmwk" Oct 31 05:33:21.380931 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:21.380907 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b028b40-5fa7-4e54-aab6-af205d402646-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51342272-dlmwk\" (UID: \"9b028b40-5fa7-4e54-aab6-af205d402646\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51342272-dlmwk" Oct 31 05:33:21.401989 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:21.401903 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtf5v\" (UniqueName: \"kubernetes.io/projected/9b028b40-5fa7-4e54-aab6-af205d402646-kube-api-access-dtf5v\") pod \"maas-api-revoke-sub-51342272-dlmwk\" (UID: \"9b028b40-5fa7-4e54-aab6-af205d402646\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51342272-dlmwk" Oct 31 05:33:21.464344 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:21.464300 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51342272-dlmwk" Oct 31 05:33:21.765550 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:21.765518 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51342272-dlmwk"] Oct 31 05:33:21.767505 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:33:21.767462 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b028b40_5fa7_4e54_aab6_af205d402646.slice/crio-45a81562df71a5f06b958b0a191ee48c6ca1fd4fdf2f85640ac7757a733b0652 WatchSource:0}: Error finding container 45a81562df71a5f06b958b0a191ee48c6ca1fd4fdf2f85640ac7757a733b0652: Status 404 returned error can't find the container with id 45a81562df71a5f06b958b0a191ee48c6ca1fd4fdf2f85640ac7757a733b0652 Oct 31 05:33:22.152850 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:22.152803 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51342272-dlmwk" event={"ID":"9b028b40-5fa7-4e54-aab6-af205d402646","Type":"ContainerStarted","Data":"3c9526816faf91fcddeadd9be1f1516f5c28c26e7f433997d7cc720f8d8da49d"} Oct 31 05:33:22.153284 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:22.152874 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51342272-dlmwk" event={"ID":"9b028b40-5fa7-4e54-aab6-af205d402646","Type":"ContainerStarted","Data":"45a81562df71a5f06b958b0a191ee48c6ca1fd4fdf2f85640ac7757a733b0652"} Oct 31 05:33:22.193164 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:22.193103 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51342272-dlmwk" podStartSLOduration=1.1930871650000001 podStartE2EDuration="1.193087165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:33:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:33:22.191991921 +0000 UTC m=+5356.948329959" watchObservedRunningTime="2026-10-31 05:33:22.193087165 +0000 UTC m=+5356.949425204" Oct 31 05:33:22.393259 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:33:22.393169 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:33:23.160951 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:23.160911 5019 generic.go:356] "Generic (PLEG): container finished" podID="9b028b40-5fa7-4e54-aab6-af205d402646" containerID="3c9526816faf91fcddeadd9be1f1516f5c28c26e7f433997d7cc720f8d8da49d" exitCode=0 Oct 31 05:33:23.161418 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:23.160990 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51342272-dlmwk" event={"ID":"9b028b40-5fa7-4e54-aab6-af205d402646","Type":"ContainerDied","Data":"3c9526816faf91fcddeadd9be1f1516f5c28c26e7f433997d7cc720f8d8da49d"} Oct 31 05:33:24.321474 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:24.321442 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51342272-dlmwk" Oct 31 05:33:24.410035 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:24.409991 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b028b40-5fa7-4e54-aab6-af205d402646-kube-api-access-dtf5v\" (UniqueName: \"kubernetes.io/projected/9b028b40-5fa7-4e54-aab6-af205d402646-kube-api-access-dtf5v\") pod \"9b028b40-5fa7-4e54-aab6-af205d402646\" (UID: \"9b028b40-5fa7-4e54-aab6-af205d402646\") " Oct 31 05:33:24.410248 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:24.410157 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b028b40-5fa7-4e54-aab6-af205d402646-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b028b40-5fa7-4e54-aab6-af205d402646-maas-api-xks-ca\") pod \"9b028b40-5fa7-4e54-aab6-af205d402646\" (UID: \"9b028b40-5fa7-4e54-aab6-af205d402646\") " Oct 31 05:33:24.410248 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:24.410185 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b028b40-5fa7-4e54-aab6-af205d402646-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b028b40-5fa7-4e54-aab6-af205d402646-maas-api-openshift-ca\") pod \"9b028b40-5fa7-4e54-aab6-af205d402646\" (UID: \"9b028b40-5fa7-4e54-aab6-af205d402646\") " Oct 31 05:33:24.410664 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:24.410636 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b028b40-5fa7-4e54-aab6-af205d402646-maas-api-openshift-ca" pod "9b028b40-5fa7-4e54-aab6-af205d402646" (UID: "9b028b40-5fa7-4e54-aab6-af205d402646"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:33:24.412324 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:24.412290 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b028b40-5fa7-4e54-aab6-af205d402646-maas-api-xks-ca" pod "9b028b40-5fa7-4e54-aab6-af205d402646" (UID: "9b028b40-5fa7-4e54-aab6-af205d402646"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:33:24.412324 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:24.412316 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b028b40-5fa7-4e54-aab6-af205d402646-kube-api-access-dtf5v" pod "9b028b40-5fa7-4e54-aab6-af205d402646" (UID: "9b028b40-5fa7-4e54-aab6-af205d402646"). InnerVolumeSpecName "kube-api-access-dtf5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:33:24.511022 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:24.510974 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b028b40-5fa7-4e54-aab6-af205d402646-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:33:24.511022 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:24.511014 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b028b40-5fa7-4e54-aab6-af205d402646-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:33:24.511022 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:24.511024 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtf5v\" (UniqueName: \"kubernetes.io/projected/9b028b40-5fa7-4e54-aab6-af205d402646-kube-api-access-dtf5v\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:33:25.175024 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:25.174991 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51342272-dlmwk" Oct 31 05:33:25.175232 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:25.175026 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51342272-dlmwk" event={"ID":"9b028b40-5fa7-4e54-aab6-af205d402646","Type":"ContainerDied","Data":"45a81562df71a5f06b958b0a191ee48c6ca1fd4fdf2f85640ac7757a733b0652"} Oct 31 05:33:25.175232 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:25.175075 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45a81562df71a5f06b958b0a191ee48c6ca1fd4fdf2f85640ac7757a733b0652" Oct 31 05:33:33.029709 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:33.029663 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rlthw"] Oct 31 05:33:33.041463 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:33.041419 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rlthw"] Oct 31 05:33:33.408513 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:33.408413 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5c796ae-478c-4b61-886c-63ff4532433e" path="/var/lib/kubelet/pods/a5c796ae-478c-4b61-886c-63ff4532433e/volumes" Oct 31 05:33:37.328858 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:37.328806 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-9632aa/simulator-single-main-9632aa-kserve-6b9cc66dbb-sx22m"] Oct 31 05:33:37.908646 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:37.908607 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-9632aa-687df6dccdb2v2"] Oct 31 05:33:37.940245 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:37.940205 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-9632aa-55bd8dg9nd"] Oct 31 05:33:42.909978 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:42.909895 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9632aa-687df6dccdb2v2" podUID="f3331e5f-ac28-4beb-acb6-176075a5bee1" containerName="payload-processing" containerID="cri-o://6e8c3049f9b4ad320e3880952527e75783584628f70df35b9e92b0cba7799ff7" gracePeriod=25 Oct 31 05:33:42.940612 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:42.940542 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9632aa-55bd8dg9nd" podUID="19a1b418-8cfc-403c-90f6-b085ef6c9e8d" containerName="payload-pre-processing" containerID="cri-o://fbf6d27d88e5fd42272df69e820c06165f257fcac7bdae603467ba652d82f791" gracePeriod=25 Oct 31 05:33:45.834876 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:45.834839 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9632aa-687df6dccdb2v2" podUID="f3331e5f-ac28-4beb-acb6-176075a5bee1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:33:45.835283 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:45.834886 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9632aa-55bd8dg9nd" podUID="19a1b418-8cfc-403c-90f6-b085ef6c9e8d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:33:52.350476 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:52.350423 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-9632aa/simulator-single-main-9632aa-kserve-6b9cc66dbb-sx22m" podUID="ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b" containerName="main" containerID="cri-o://a1afd7c558fa603a749ddf2edcec928cbeca144d2394a7de5f8511027cdbae1b" gracePeriod=45 Oct 31 05:33:52.391808 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:33:52.391769 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:33:52.622396 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:52.622364 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-9632aa/simulator-single-main-9632aa-kserve-6b9cc66dbb-sx22m" Oct 31 05:33:52.711430 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:52.711394 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b-dshm\") pod \"ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b\" (UID: \"ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b\") " Oct 31 05:33:52.711656 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:52.711468 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b-kube-api-access-r8xqn\" (UniqueName: \"kubernetes.io/projected/ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b-kube-api-access-r8xqn\") pod \"ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b\" (UID: \"ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b\") " Oct 31 05:33:52.711656 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:52.711498 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b-tls-certs\" (UniqueName: \"kubernetes.io/secret/ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b-tls-certs\") pod \"ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b\" (UID: \"ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b\") " Oct 31 05:33:52.714266 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:52.714200 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b-dshm" pod "ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b" (UID: "ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:33:52.714474 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:52.714387 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b-kube-api-access-r8xqn" pod "ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b" (UID: "ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b"). InnerVolumeSpecName "kube-api-access-r8xqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:33:52.714554 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:52.714510 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b-tls-certs" pod "ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b" (UID: "ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:33:52.812377 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:52.812285 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b-tmp-dir\") pod \"ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b\" (UID: \"ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b\") " Oct 31 05:33:52.812620 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:52.812400 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b-model-cache\") pod \"ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b\" (UID: \"ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b\") " Oct 31 05:33:52.812620 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:52.812462 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b-home\" (UniqueName: \"kubernetes.io/empty-dir/ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b-home\") pod \"ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b\" (UID: \"ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b\") " Oct 31 05:33:52.812745 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:52.812617 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b-tmp-dir" pod "ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b" (UID: "ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:33:52.812745 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:52.812654 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b-model-cache" pod "ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b" (UID: "ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:33:52.812745 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:52.812723 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b-home" pod "ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b" (UID: "ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:33:52.812889 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:52.812872 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8xqn\" (UniqueName: \"kubernetes.io/projected/ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b-kube-api-access-r8xqn\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:33:52.812929 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:52.812892 5019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b-tls-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:33:52.812929 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:52.812904 5019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b-tmp-dir\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:33:52.812929 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:52.812912 5019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b-model-cache\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:33:52.812929 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:52.812920 5019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b-home\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:33:52.812929 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:52.812929 5019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b-dshm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:33:53.027633 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:53.027593 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hrlmq"] Oct 31 05:33:53.030574 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:53.030542 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hrlmq"] Oct 31 05:33:53.365610 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:53.365500 5019 generic.go:356] "Generic (PLEG): container finished" podID="ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b" containerID="a1afd7c558fa603a749ddf2edcec928cbeca144d2394a7de5f8511027cdbae1b" exitCode=0 Oct 31 05:33:53.365610 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:53.365547 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-9632aa/simulator-single-main-9632aa-kserve-6b9cc66dbb-sx22m" Oct 31 05:33:53.366123 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:53.365549 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-9632aa/simulator-single-main-9632aa-kserve-6b9cc66dbb-sx22m" event={"ID":"ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b","Type":"ContainerDied","Data":"a1afd7c558fa603a749ddf2edcec928cbeca144d2394a7de5f8511027cdbae1b"} Oct 31 05:33:53.366123 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:53.365678 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-9632aa/simulator-single-main-9632aa-kserve-6b9cc66dbb-sx22m" event={"ID":"ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b","Type":"ContainerDied","Data":"f53bacf36c5ca66ff3b99f5c18e0e9bfc75b65647ce2717eac038f0707e1e178"} Oct 31 05:33:53.366123 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:53.365702 5019 scope.go:122] "RemoveContainer" containerID="a1afd7c558fa603a749ddf2edcec928cbeca144d2394a7de5f8511027cdbae1b" Oct 31 05:33:53.377012 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:53.376985 5019 scope.go:122] "RemoveContainer" containerID="a1afd7c558fa603a749ddf2edcec928cbeca144d2394a7de5f8511027cdbae1b" Oct 31 05:33:53.377433 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:33:53.377407 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1afd7c558fa603a749ddf2edcec928cbeca144d2394a7de5f8511027cdbae1b\": container with ID starting with a1afd7c558fa603a749ddf2edcec928cbeca144d2394a7de5f8511027cdbae1b not found: ID does not exist" containerID="a1afd7c558fa603a749ddf2edcec928cbeca144d2394a7de5f8511027cdbae1b" Oct 31 05:33:53.377503 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:53.377443 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1afd7c558fa603a749ddf2edcec928cbeca144d2394a7de5f8511027cdbae1b"} err="failed to get container status \"a1afd7c558fa603a749ddf2edcec928cbeca144d2394a7de5f8511027cdbae1b\": rpc error: code = NotFound desc = could not find container \"a1afd7c558fa603a749ddf2edcec928cbeca144d2394a7de5f8511027cdbae1b\": container with ID starting with a1afd7c558fa603a749ddf2edcec928cbeca144d2394a7de5f8511027cdbae1b not found: ID does not exist" Oct 31 05:33:53.406447 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:53.406400 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ec58d93-ce49-4c53-a165-ba08c5439b41" path="/var/lib/kubelet/pods/2ec58d93-ce49-4c53-a165-ba08c5439b41/volumes" Oct 31 05:33:53.418812 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:53.418774 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-9632aa/simulator-single-main-9632aa-kserve-6b9cc66dbb-sx22m"] Oct 31 05:33:53.422525 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:53.422486 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-9632aa/simulator-single-main-9632aa-kserve-6b9cc66dbb-sx22m"] Oct 31 05:33:55.407765 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:55.407726 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b" path="/var/lib/kubelet/pods/ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b/volumes" Oct 31 05:33:55.834043 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:55.834005 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9632aa-687df6dccdb2v2" podUID="f3331e5f-ac28-4beb-acb6-176075a5bee1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:33:55.834043 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:33:55.834021 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9632aa-55bd8dg9nd" podUID="19a1b418-8cfc-403c-90f6-b085ef6c9e8d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 05:34:03.208768 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:03.208735 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9632aa-687df6dccdb2v2" Oct 31 05:34:03.242947 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:03.242912 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9632aa-55bd8dg9nd" Oct 31 05:34:03.322170 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:03.322108 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19a1b418-8cfc-403c-90f6-b085ef6c9e8d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19a1b418-8cfc-403c-90f6-b085ef6c9e8d-plugins-config-volume\") pod \"19a1b418-8cfc-403c-90f6-b085ef6c9e8d\" (UID: \"19a1b418-8cfc-403c-90f6-b085ef6c9e8d\") " Oct 31 05:34:03.322170 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:03.322174 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3331e5f-ac28-4beb-acb6-176075a5bee1-kube-api-access-ksz4v\" (UniqueName: \"kubernetes.io/projected/f3331e5f-ac28-4beb-acb6-176075a5bee1-kube-api-access-ksz4v\") pod \"f3331e5f-ac28-4beb-acb6-176075a5bee1\" (UID: \"f3331e5f-ac28-4beb-acb6-176075a5bee1\") " Oct 31 05:34:03.322468 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:03.322379 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19a1b418-8cfc-403c-90f6-b085ef6c9e8d-kube-api-access-wjn5j\" (UniqueName: \"kubernetes.io/projected/19a1b418-8cfc-403c-90f6-b085ef6c9e8d-kube-api-access-wjn5j\") pod \"19a1b418-8cfc-403c-90f6-b085ef6c9e8d\" (UID: \"19a1b418-8cfc-403c-90f6-b085ef6c9e8d\") " Oct 31 05:34:03.322468 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:03.322425 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3331e5f-ac28-4beb-acb6-176075a5bee1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3331e5f-ac28-4beb-acb6-176075a5bee1-plugins-config-volume\") pod \"f3331e5f-ac28-4beb-acb6-176075a5bee1\" (UID: \"f3331e5f-ac28-4beb-acb6-176075a5bee1\") " Oct 31 05:34:03.322670 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:03.322642 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19a1b418-8cfc-403c-90f6-b085ef6c9e8d-plugins-config-volume" pod "19a1b418-8cfc-403c-90f6-b085ef6c9e8d" (UID: "19a1b418-8cfc-403c-90f6-b085ef6c9e8d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:34:03.322908 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:03.322875 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3331e5f-ac28-4beb-acb6-176075a5bee1-plugins-config-volume" pod "f3331e5f-ac28-4beb-acb6-176075a5bee1" (UID: "f3331e5f-ac28-4beb-acb6-176075a5bee1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:34:03.323045 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:03.322894 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19a1b418-8cfc-403c-90f6-b085ef6c9e8d-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:34:03.324691 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:03.324665 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3331e5f-ac28-4beb-acb6-176075a5bee1-kube-api-access-ksz4v" pod "f3331e5f-ac28-4beb-acb6-176075a5bee1" (UID: "f3331e5f-ac28-4beb-acb6-176075a5bee1"). InnerVolumeSpecName "kube-api-access-ksz4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:34:03.324803 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:03.324705 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19a1b418-8cfc-403c-90f6-b085ef6c9e8d-kube-api-access-wjn5j" pod "19a1b418-8cfc-403c-90f6-b085ef6c9e8d" (UID: "19a1b418-8cfc-403c-90f6-b085ef6c9e8d"). InnerVolumeSpecName "kube-api-access-wjn5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:34:03.423907 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:03.423873 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjn5j\" (UniqueName: \"kubernetes.io/projected/19a1b418-8cfc-403c-90f6-b085ef6c9e8d-kube-api-access-wjn5j\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:34:03.423907 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:03.423904 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3331e5f-ac28-4beb-acb6-176075a5bee1-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:34:03.424129 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:03.423914 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksz4v\" (UniqueName: \"kubernetes.io/projected/f3331e5f-ac28-4beb-acb6-176075a5bee1-kube-api-access-ksz4v\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:34:03.430728 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:03.430645 5019 generic.go:356] "Generic (PLEG): container finished" podID="f3331e5f-ac28-4beb-acb6-176075a5bee1" containerID="6e8c3049f9b4ad320e3880952527e75783584628f70df35b9e92b0cba7799ff7" exitCode=0 Oct 31 05:34:03.430728 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:03.430700 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9632aa-687df6dccdb2v2" Oct 31 05:34:03.430935 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:03.430735 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9632aa-687df6dccdb2v2" event={"ID":"f3331e5f-ac28-4beb-acb6-176075a5bee1","Type":"ContainerDied","Data":"6e8c3049f9b4ad320e3880952527e75783584628f70df35b9e92b0cba7799ff7"} Oct 31 05:34:03.430935 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:03.430793 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9632aa-687df6dccdb2v2" event={"ID":"f3331e5f-ac28-4beb-acb6-176075a5bee1","Type":"ContainerDied","Data":"144e3283adeb234d12977649aa5291187bb418f52b5bc7053c36c5500ab30147"} Oct 31 05:34:03.430935 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:03.430823 5019 scope.go:122] "RemoveContainer" containerID="6e8c3049f9b4ad320e3880952527e75783584628f70df35b9e92b0cba7799ff7" Oct 31 05:34:03.432539 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:03.432509 5019 generic.go:356] "Generic (PLEG): container finished" podID="19a1b418-8cfc-403c-90f6-b085ef6c9e8d" containerID="fbf6d27d88e5fd42272df69e820c06165f257fcac7bdae603467ba652d82f791" exitCode=0 Oct 31 05:34:03.432690 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:03.432570 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9632aa-55bd8dg9nd" Oct 31 05:34:03.432690 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:03.432613 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9632aa-55bd8dg9nd" event={"ID":"19a1b418-8cfc-403c-90f6-b085ef6c9e8d","Type":"ContainerDied","Data":"fbf6d27d88e5fd42272df69e820c06165f257fcac7bdae603467ba652d82f791"} Oct 31 05:34:03.432690 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:03.432658 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9632aa-55bd8dg9nd" event={"ID":"19a1b418-8cfc-403c-90f6-b085ef6c9e8d","Type":"ContainerDied","Data":"916d295c1350f32971c3f0b3b0304fb31333513a01f329c93bed02cf48a77060"} Oct 31 05:34:03.441159 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:03.441134 5019 scope.go:122] "RemoveContainer" containerID="6e8c3049f9b4ad320e3880952527e75783584628f70df35b9e92b0cba7799ff7" Oct 31 05:34:03.441529 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:34:03.441503 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e8c3049f9b4ad320e3880952527e75783584628f70df35b9e92b0cba7799ff7\": container with ID starting with 6e8c3049f9b4ad320e3880952527e75783584628f70df35b9e92b0cba7799ff7 not found: ID does not exist" containerID="6e8c3049f9b4ad320e3880952527e75783584628f70df35b9e92b0cba7799ff7" Oct 31 05:34:03.441594 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:03.441539 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e8c3049f9b4ad320e3880952527e75783584628f70df35b9e92b0cba7799ff7"} err="failed to get container status \"6e8c3049f9b4ad320e3880952527e75783584628f70df35b9e92b0cba7799ff7\": rpc error: code = NotFound desc = could not find container \"6e8c3049f9b4ad320e3880952527e75783584628f70df35b9e92b0cba7799ff7\": container with ID starting with 6e8c3049f9b4ad320e3880952527e75783584628f70df35b9e92b0cba7799ff7 not found: ID does not exist" Oct 31 05:34:03.441594 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:03.441554 5019 scope.go:122] "RemoveContainer" containerID="fbf6d27d88e5fd42272df69e820c06165f257fcac7bdae603467ba652d82f791" Oct 31 05:34:03.451197 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:03.451175 5019 scope.go:122] "RemoveContainer" containerID="fbf6d27d88e5fd42272df69e820c06165f257fcac7bdae603467ba652d82f791" Oct 31 05:34:03.451549 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:34:03.451517 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbf6d27d88e5fd42272df69e820c06165f257fcac7bdae603467ba652d82f791\": container with ID starting with fbf6d27d88e5fd42272df69e820c06165f257fcac7bdae603467ba652d82f791 not found: ID does not exist" containerID="fbf6d27d88e5fd42272df69e820c06165f257fcac7bdae603467ba652d82f791" Oct 31 05:34:03.451696 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:03.451560 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbf6d27d88e5fd42272df69e820c06165f257fcac7bdae603467ba652d82f791"} err="failed to get container status \"fbf6d27d88e5fd42272df69e820c06165f257fcac7bdae603467ba652d82f791\": rpc error: code = NotFound desc = could not find container \"fbf6d27d88e5fd42272df69e820c06165f257fcac7bdae603467ba652d82f791\": container with ID starting with fbf6d27d88e5fd42272df69e820c06165f257fcac7bdae603467ba652d82f791 not found: ID does not exist" Oct 31 05:34:03.485060 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:03.485012 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-9632aa-687df6dccdb2v2"] Oct 31 05:34:03.491632 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:03.491590 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-9632aa-687df6dccdb2v2"] Oct 31 05:34:03.522480 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:03.522436 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-9632aa-55bd8dg9nd"] Oct 31 05:34:03.526310 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:03.526276 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-9632aa-55bd8dg9nd"] Oct 31 05:34:04.028451 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:04.028413 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xqj88"] Oct 31 05:34:04.040223 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:04.040188 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xqj88"] Oct 31 05:34:05.407398 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:05.407363 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19a1b418-8cfc-403c-90f6-b085ef6c9e8d" path="/var/lib/kubelet/pods/19a1b418-8cfc-403c-90f6-b085ef6c9e8d/volumes" Oct 31 05:34:05.408062 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:05.408011 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88b92fcd-a647-48a5-a06f-1767d73a51c3" path="/var/lib/kubelet/pods/88b92fcd-a647-48a5-a06f-1767d73a51c3/volumes" Oct 31 05:34:05.408469 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:05.408453 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3331e5f-ac28-4beb-acb6-176075a5bee1" path="/var/lib/kubelet/pods/f3331e5f-ac28-4beb-acb6-176075a5bee1/volumes" Oct 31 05:34:09.830992 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:09.830961 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s88kp_3945ff65-e877-4372-9334-3968c68b7295/console-operator/2.log" Oct 31 05:34:09.836654 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:09.836623 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s88kp_3945ff65-e877-4372-9334-3968c68b7295/console-operator/2.log" Oct 31 05:34:12.478170 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:12.478133 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-9632aa-687df6dccwv6lt"] Oct 31 05:34:12.478613 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:12.478598 5019 cpu_manager.go:414] "containerMap: removing container" podUID="f3331e5f-ac28-4beb-acb6-176075a5bee1" containerName="payload-processing" Oct 31 05:34:12.478659 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:12.478612 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3331e5f-ac28-4beb-acb6-176075a5bee1" containerName="payload-processing" Oct 31 05:34:12.478659 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:12.478638 5019 cpu_manager.go:414] "containerMap: removing container" podUID="9b028b40-5fa7-4e54-aab6-af205d402646" containerName="revoke-keys" Oct 31 05:34:12.478659 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:12.478643 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b028b40-5fa7-4e54-aab6-af205d402646" containerName="revoke-keys" Oct 31 05:34:12.478659 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:12.478652 5019 cpu_manager.go:414] "containerMap: removing container" podUID="19a1b418-8cfc-403c-90f6-b085ef6c9e8d" containerName="payload-pre-processing" Oct 31 05:34:12.478659 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:12.478658 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19a1b418-8cfc-403c-90f6-b085ef6c9e8d" containerName="payload-pre-processing" Oct 31 05:34:12.478838 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:12.478669 5019 cpu_manager.go:414] "containerMap: removing container" podUID="ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b" containerName="main" Oct 31 05:34:12.478838 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:12.478676 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b" containerName="main" Oct 31 05:34:12.478838 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:12.478753 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="19a1b418-8cfc-403c-90f6-b085ef6c9e8d" containerName="payload-pre-processing" Oct 31 05:34:12.478838 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:12.478762 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecee9d31-9fd4-4d37-bfa8-8f3f80b3af1b" containerName="main" Oct 31 05:34:12.478838 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:12.478777 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3331e5f-ac28-4beb-acb6-176075a5bee1" containerName="payload-processing" Oct 31 05:34:12.478838 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:12.478785 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b028b40-5fa7-4e54-aab6-af205d402646" containerName="revoke-keys" Oct 31 05:34:12.483448 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:12.483424 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9632aa-687df6dccwv6lt" Oct 31 05:34:12.501651 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:12.501599 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-9632aa\"" Oct 31 05:34:12.504864 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:12.504829 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-9632aa-687df6dccwv6lt"] Oct 31 05:34:12.513674 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:12.513635 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-bf915a09-dockercfg-r2hhl\"" Oct 31 05:34:12.537503 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:12.537468 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-9632aa-55bd8w5ksw"] Oct 31 05:34:12.541623 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:12.541594 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9632aa-55bd8w5ksw" Oct 31 05:34:12.561789 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:12.561752 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-9632aa-55bd8w5ksw"] Oct 31 05:34:12.621807 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:12.620590 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa27c921-f840-4866-aaff-135de970c7f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-9632aa-55bd8w5ksw\" (UID: \"aa27c921-f840-4866-aaff-135de970c7f4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9632aa-55bd8w5ksw" Oct 31 05:34:12.621807 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:12.620665 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94b5ffe6-1b77-42e3-8607-1d7158861b72-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-9632aa-687df6dccwv6lt\" (UID: \"94b5ffe6-1b77-42e3-8607-1d7158861b72\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-9632aa-687df6dccwv6lt" Oct 31 05:34:12.621807 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:12.620720 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csctp\" (UniqueName: \"kubernetes.io/projected/aa27c921-f840-4866-aaff-135de970c7f4-kube-api-access-csctp\") pod \"payload-pre-processing-e2e-worker-single-main-9632aa-55bd8w5ksw\" (UID: \"aa27c921-f840-4866-aaff-135de970c7f4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9632aa-55bd8w5ksw" Oct 31 05:34:12.621807 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:12.620836 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zmcv\" (UniqueName: \"kubernetes.io/projected/94b5ffe6-1b77-42e3-8607-1d7158861b72-kube-api-access-2zmcv\") pod \"payload-processing-e2e-worker-single-main-9632aa-687df6dccwv6lt\" (UID: \"94b5ffe6-1b77-42e3-8607-1d7158861b72\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-9632aa-687df6dccwv6lt" Oct 31 05:34:12.721793 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:12.721750 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zmcv\" (UniqueName: \"kubernetes.io/projected/94b5ffe6-1b77-42e3-8607-1d7158861b72-kube-api-access-2zmcv\") pod \"payload-processing-e2e-worker-single-main-9632aa-687df6dccwv6lt\" (UID: \"94b5ffe6-1b77-42e3-8607-1d7158861b72\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-9632aa-687df6dccwv6lt" Oct 31 05:34:12.721984 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:12.721810 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa27c921-f840-4866-aaff-135de970c7f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-9632aa-55bd8w5ksw\" (UID: \"aa27c921-f840-4866-aaff-135de970c7f4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9632aa-55bd8w5ksw" Oct 31 05:34:12.721984 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:12.721842 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94b5ffe6-1b77-42e3-8607-1d7158861b72-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-9632aa-687df6dccwv6lt\" (UID: \"94b5ffe6-1b77-42e3-8607-1d7158861b72\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-9632aa-687df6dccwv6lt" Oct 31 05:34:12.721984 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:12.721870 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csctp\" (UniqueName: \"kubernetes.io/projected/aa27c921-f840-4866-aaff-135de970c7f4-kube-api-access-csctp\") pod \"payload-pre-processing-e2e-worker-single-main-9632aa-55bd8w5ksw\" (UID: \"aa27c921-f840-4866-aaff-135de970c7f4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9632aa-55bd8w5ksw" Oct 31 05:34:12.722838 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:12.722812 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94b5ffe6-1b77-42e3-8607-1d7158861b72-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-9632aa-687df6dccwv6lt\" (UID: \"94b5ffe6-1b77-42e3-8607-1d7158861b72\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-9632aa-687df6dccwv6lt" Oct 31 05:34:12.722894 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:12.722812 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa27c921-f840-4866-aaff-135de970c7f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-9632aa-55bd8w5ksw\" (UID: \"aa27c921-f840-4866-aaff-135de970c7f4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9632aa-55bd8w5ksw" Oct 31 05:34:12.745699 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:12.745616 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zmcv\" (UniqueName: \"kubernetes.io/projected/94b5ffe6-1b77-42e3-8607-1d7158861b72-kube-api-access-2zmcv\") pod \"payload-processing-e2e-worker-single-main-9632aa-687df6dccwv6lt\" (UID: \"94b5ffe6-1b77-42e3-8607-1d7158861b72\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-9632aa-687df6dccwv6lt" Oct 31 05:34:12.745699 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:12.745652 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csctp\" (UniqueName: \"kubernetes.io/projected/aa27c921-f840-4866-aaff-135de970c7f4-kube-api-access-csctp\") pod \"payload-pre-processing-e2e-worker-single-main-9632aa-55bd8w5ksw\" (UID: \"aa27c921-f840-4866-aaff-135de970c7f4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9632aa-55bd8w5ksw" Oct 31 05:34:12.797666 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:12.797625 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9632aa-687df6dccwv6lt" Oct 31 05:34:12.856056 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:12.855774 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9632aa-55bd8w5ksw" Oct 31 05:34:13.105894 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:13.105857 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-9632aa-687df6dccwv6lt"] Oct 31 05:34:13.106227 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:34:13.106202 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94b5ffe6_1b77_42e3_8607_1d7158861b72.slice/crio-d4113f5ba20722df177afbac36107318e808f1f3e8509d1b8947b2b1efe04cab WatchSource:0}: Error finding container d4113f5ba20722df177afbac36107318e808f1f3e8509d1b8947b2b1efe04cab: Status 404 returned error can't find the container with id d4113f5ba20722df177afbac36107318e808f1f3e8509d1b8947b2b1efe04cab Oct 31 05:34:13.200887 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:13.200078 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-9632aa-55bd8w5ksw"] Oct 31 05:34:13.202029 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:34:13.202000 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa27c921_f840_4866_aaff_135de970c7f4.slice/crio-8c8ae56a7e8cd2165cc2551ee3bb7b5bc3eab8c979303571996eefc4d8d7714c WatchSource:0}: Error finding container 8c8ae56a7e8cd2165cc2551ee3bb7b5bc3eab8c979303571996eefc4d8d7714c: Status 404 returned error can't find the container with id 8c8ae56a7e8cd2165cc2551ee3bb7b5bc3eab8c979303571996eefc4d8d7714c Oct 31 05:34:13.497312 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:13.497259 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9632aa-687df6dccwv6lt" event={"ID":"94b5ffe6-1b77-42e3-8607-1d7158861b72","Type":"ContainerStarted","Data":"7cb534b5e33ecfed1147e32e364f677d93a19a7f29401ebe0e55038201296f9b"} Oct 31 05:34:13.497815 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:13.497342 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9632aa-687df6dccwv6lt" event={"ID":"94b5ffe6-1b77-42e3-8607-1d7158861b72","Type":"ContainerStarted","Data":"d4113f5ba20722df177afbac36107318e808f1f3e8509d1b8947b2b1efe04cab"} Oct 31 05:34:13.497815 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:13.497375 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9632aa-687df6dccwv6lt" Oct 31 05:34:13.499746 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:13.499712 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9632aa-55bd8w5ksw" event={"ID":"aa27c921-f840-4866-aaff-135de970c7f4","Type":"ContainerStarted","Data":"c3de04f7a087ebc7351e983655750a118a9fc35c52768fe46e9ee1ca6c5dcc8b"} Oct 31 05:34:13.499919 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:13.499776 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9632aa-55bd8w5ksw" Oct 31 05:34:13.499919 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:13.499802 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9632aa-55bd8w5ksw" event={"ID":"aa27c921-f840-4866-aaff-135de970c7f4","Type":"ContainerStarted","Data":"8c8ae56a7e8cd2165cc2551ee3bb7b5bc3eab8c979303571996eefc4d8d7714c"} Oct 31 05:34:13.538658 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:13.538593 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9632aa-687df6dccwv6lt" podStartSLOduration=1.5385805179999998 podStartE2EDuration="1.538580518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:34:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:34:13.53748859 +0000 UTC m=+5408.293826630" watchObservedRunningTime="2026-10-31 05:34:13.538580518 +0000 UTC m=+5408.294918558" Oct 31 05:34:13.580204 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:13.580128 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9632aa-55bd8w5ksw" podStartSLOduration=1.580109861 podStartE2EDuration="1.580109861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:34:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:34:13.576341938 +0000 UTC m=+5408.332679970" watchObservedRunningTime="2026-10-31 05:34:13.580109861 +0000 UTC m=+5408.336447901" Oct 31 05:34:15.062446 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:15.062410 5019 scope.go:122] "RemoveContainer" containerID="717b25ad4b234b253c42da2aa531420624165056ba52dc1dfc03c564df713465" Oct 31 05:34:20.427096 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:20.427049 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9632aa-krmx4"] Oct 31 05:34:20.435305 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:20.435271 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9632aa-krmx4" Oct 31 05:34:20.454143 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:20.454104 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9632aa-krmx4"] Oct 31 05:34:20.466925 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:20.466883 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:34:20.467138 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:20.466954 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:34:20.467138 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:20.467045 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfhzn\"" Oct 31 05:34:20.467138 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:20.467133 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:34:20.549765 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:20.549727 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5n7p\" (UniqueName: \"kubernetes.io/projected/039b73ba-2235-4da6-9434-6e69a938e8a1-kube-api-access-w5n7p\") pod \"maas-api-revoke-keys-e2e-worker-single-main-9632aa-krmx4\" (UID: \"039b73ba-2235-4da6-9434-6e69a938e8a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9632aa-krmx4" Oct 31 05:34:20.549993 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:20.549819 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/039b73ba-2235-4da6-9434-6e69a938e8a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-9632aa-krmx4\" (UID: \"039b73ba-2235-4da6-9434-6e69a938e8a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9632aa-krmx4" Oct 31 05:34:20.550144 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:20.550118 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/039b73ba-2235-4da6-9434-6e69a938e8a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-9632aa-krmx4\" (UID: \"039b73ba-2235-4da6-9434-6e69a938e8a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9632aa-krmx4" Oct 31 05:34:20.651780 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:20.651732 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5n7p\" (UniqueName: \"kubernetes.io/projected/039b73ba-2235-4da6-9434-6e69a938e8a1-kube-api-access-w5n7p\") pod \"maas-api-revoke-keys-e2e-worker-single-main-9632aa-krmx4\" (UID: \"039b73ba-2235-4da6-9434-6e69a938e8a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9632aa-krmx4" Oct 31 05:34:20.651999 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:20.651863 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/039b73ba-2235-4da6-9434-6e69a938e8a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-9632aa-krmx4\" (UID: \"039b73ba-2235-4da6-9434-6e69a938e8a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9632aa-krmx4" Oct 31 05:34:20.651999 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:20.651930 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/039b73ba-2235-4da6-9434-6e69a938e8a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-9632aa-krmx4\" (UID: \"039b73ba-2235-4da6-9434-6e69a938e8a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9632aa-krmx4" Oct 31 05:34:20.652678 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:20.652648 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/039b73ba-2235-4da6-9434-6e69a938e8a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-9632aa-krmx4\" (UID: \"039b73ba-2235-4da6-9434-6e69a938e8a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9632aa-krmx4" Oct 31 05:34:20.654572 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:20.654535 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/039b73ba-2235-4da6-9434-6e69a938e8a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-9632aa-krmx4\" (UID: \"039b73ba-2235-4da6-9434-6e69a938e8a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9632aa-krmx4" Oct 31 05:34:20.675220 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:20.675177 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5n7p\" (UniqueName: \"kubernetes.io/projected/039b73ba-2235-4da6-9434-6e69a938e8a1-kube-api-access-w5n7p\") pod \"maas-api-revoke-keys-e2e-worker-single-main-9632aa-krmx4\" (UID: \"039b73ba-2235-4da6-9434-6e69a938e8a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9632aa-krmx4" Oct 31 05:34:20.783521 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:20.783475 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9632aa-krmx4" Oct 31 05:34:21.085983 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:21.085946 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9632aa-krmx4"] Oct 31 05:34:21.088020 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:34:21.087975 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod039b73ba_2235_4da6_9434_6e69a938e8a1.slice/crio-05e550d284535ae65df59f54d21ef12839fdcdc1a357be8e15164fb0460b5dd2 WatchSource:0}: Error finding container 05e550d284535ae65df59f54d21ef12839fdcdc1a357be8e15164fb0460b5dd2: Status 404 returned error can't find the container with id 05e550d284535ae65df59f54d21ef12839fdcdc1a357be8e15164fb0460b5dd2 Oct 31 05:34:21.560615 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:21.560572 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9632aa-krmx4" event={"ID":"039b73ba-2235-4da6-9434-6e69a938e8a1","Type":"ContainerStarted","Data":"58bdc6a2cd3e8290e7d9e55f90207281c3b097788683075d2b602081bb497d46"} Oct 31 05:34:21.561049 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:21.560634 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9632aa-krmx4" event={"ID":"039b73ba-2235-4da6-9434-6e69a938e8a1","Type":"ContainerStarted","Data":"05e550d284535ae65df59f54d21ef12839fdcdc1a357be8e15164fb0460b5dd2"} Oct 31 05:34:21.601993 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:21.601927 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9632aa-krmx4" podStartSLOduration=1.601913232 podStartE2EDuration="1.601913232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:34:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:34:21.600005392 +0000 UTC m=+5416.356343431" watchObservedRunningTime="2026-10-31 05:34:21.601913232 +0000 UTC m=+5416.358251272" Oct 31 05:34:22.392079 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:34:22.391990 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:34:22.569152 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:22.569117 5019 generic.go:356] "Generic (PLEG): container finished" podID="039b73ba-2235-4da6-9434-6e69a938e8a1" containerID="58bdc6a2cd3e8290e7d9e55f90207281c3b097788683075d2b602081bb497d46" exitCode=0 Oct 31 05:34:22.569598 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:22.569201 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9632aa-krmx4" event={"ID":"039b73ba-2235-4da6-9434-6e69a938e8a1","Type":"ContainerDied","Data":"58bdc6a2cd3e8290e7d9e55f90207281c3b097788683075d2b602081bb497d46"} Oct 31 05:34:23.725927 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:23.725899 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9632aa-krmx4" Oct 31 05:34:23.784036 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:23.783999 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/039b73ba-2235-4da6-9434-6e69a938e8a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/039b73ba-2235-4da6-9434-6e69a938e8a1-maas-api-openshift-ca\") pod \"039b73ba-2235-4da6-9434-6e69a938e8a1\" (UID: \"039b73ba-2235-4da6-9434-6e69a938e8a1\") " Oct 31 05:34:23.784228 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:23.784131 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/039b73ba-2235-4da6-9434-6e69a938e8a1-kube-api-access-w5n7p\" (UniqueName: \"kubernetes.io/projected/039b73ba-2235-4da6-9434-6e69a938e8a1-kube-api-access-w5n7p\") pod \"039b73ba-2235-4da6-9434-6e69a938e8a1\" (UID: \"039b73ba-2235-4da6-9434-6e69a938e8a1\") " Oct 31 05:34:23.784228 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:23.784161 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/039b73ba-2235-4da6-9434-6e69a938e8a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/039b73ba-2235-4da6-9434-6e69a938e8a1-maas-api-xks-ca\") pod \"039b73ba-2235-4da6-9434-6e69a938e8a1\" (UID: \"039b73ba-2235-4da6-9434-6e69a938e8a1\") " Oct 31 05:34:23.784511 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:23.784487 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/039b73ba-2235-4da6-9434-6e69a938e8a1-maas-api-openshift-ca" pod "039b73ba-2235-4da6-9434-6e69a938e8a1" (UID: "039b73ba-2235-4da6-9434-6e69a938e8a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:34:23.786458 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:23.786421 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/039b73ba-2235-4da6-9434-6e69a938e8a1-kube-api-access-w5n7p" pod "039b73ba-2235-4da6-9434-6e69a938e8a1" (UID: "039b73ba-2235-4da6-9434-6e69a938e8a1"). InnerVolumeSpecName "kube-api-access-w5n7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:34:23.786599 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:23.786489 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/039b73ba-2235-4da6-9434-6e69a938e8a1-maas-api-xks-ca" pod "039b73ba-2235-4da6-9434-6e69a938e8a1" (UID: "039b73ba-2235-4da6-9434-6e69a938e8a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:34:23.885906 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:23.885799 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/039b73ba-2235-4da6-9434-6e69a938e8a1-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:34:23.885906 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:23.885840 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5n7p\" (UniqueName: \"kubernetes.io/projected/039b73ba-2235-4da6-9434-6e69a938e8a1-kube-api-access-w5n7p\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:34:23.885906 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:23.885853 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/039b73ba-2235-4da6-9434-6e69a938e8a1-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:34:24.509916 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:24.509886 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9632aa-55bd8w5ksw" Oct 31 05:34:24.510118 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:24.509964 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9632aa-687df6dccwv6lt" Oct 31 05:34:24.586393 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:24.586360 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9632aa-krmx4" event={"ID":"039b73ba-2235-4da6-9434-6e69a938e8a1","Type":"ContainerDied","Data":"05e550d284535ae65df59f54d21ef12839fdcdc1a357be8e15164fb0460b5dd2"} Oct 31 05:34:24.586393 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:24.586384 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9632aa-krmx4" Oct 31 05:34:24.586674 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:24.586413 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05e550d284535ae65df59f54d21ef12839fdcdc1a357be8e15164fb0460b5dd2" Oct 31 05:34:30.898013 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:30.897966 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-9632aa-687df6dccwv6lt"] Oct 31 05:34:30.929213 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:30.929161 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-9632aa-55bd8w5ksw"] Oct 31 05:34:32.644368 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:32.644317 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9632aa-krmx4"] Oct 31 05:34:32.655907 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:32.655875 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9632aa-krmx4"] Oct 31 05:34:33.029893 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:33.029859 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xl58n"] Oct 31 05:34:33.041622 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:33.041592 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xl58n"] Oct 31 05:34:33.407494 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:33.407409 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="039b73ba-2235-4da6-9434-6e69a938e8a1" path="/var/lib/kubelet/pods/039b73ba-2235-4da6-9434-6e69a938e8a1/volumes" Oct 31 05:34:33.407892 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:33.407877 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf427484-81ab-4eaa-8714-8d800e8c11d0" path="/var/lib/kubelet/pods/bf427484-81ab-4eaa-8714-8d800e8c11d0/volumes" Oct 31 05:34:34.295084 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:34.295045 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fb7p4"] Oct 31 05:34:34.307186 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:34.307153 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fb7p4"] Oct 31 05:34:34.341204 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:34.341162 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bgqjk"] Oct 31 05:34:34.341663 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:34.341650 5019 cpu_manager.go:414] "containerMap: removing container" podUID="039b73ba-2235-4da6-9434-6e69a938e8a1" containerName="revoke-keys" Oct 31 05:34:34.341727 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:34.341665 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="039b73ba-2235-4da6-9434-6e69a938e8a1" containerName="revoke-keys" Oct 31 05:34:34.341766 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:34.341757 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="039b73ba-2235-4da6-9434-6e69a938e8a1" containerName="revoke-keys" Oct 31 05:34:34.346123 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:34.346102 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bgqjk" Oct 31 05:34:34.362493 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:34.362384 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfhzn\"" Oct 31 05:34:34.362493 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:34.362434 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:34:34.362493 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:34.362468 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:34:34.362493 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:34.362477 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:34:34.366142 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:34.366111 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bgqjk"] Oct 31 05:34:34.403416 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:34.403377 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc9x8\" (UniqueName: \"kubernetes.io/projected/32fd0aeb-9f3b-4add-bb99-f6cac6cf910d-kube-api-access-pc9x8\") pod \"maas-api-revoke-sub-3d4156f1-bgqjk\" (UID: \"32fd0aeb-9f3b-4add-bb99-f6cac6cf910d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bgqjk" Oct 31 05:34:34.403623 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:34.403440 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32fd0aeb-9f3b-4add-bb99-f6cac6cf910d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-bgqjk\" (UID: \"32fd0aeb-9f3b-4add-bb99-f6cac6cf910d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bgqjk" Oct 31 05:34:34.403623 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:34.403586 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32fd0aeb-9f3b-4add-bb99-f6cac6cf910d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-bgqjk\" (UID: \"32fd0aeb-9f3b-4add-bb99-f6cac6cf910d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bgqjk" Oct 31 05:34:34.505020 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:34.504962 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc9x8\" (UniqueName: \"kubernetes.io/projected/32fd0aeb-9f3b-4add-bb99-f6cac6cf910d-kube-api-access-pc9x8\") pod \"maas-api-revoke-sub-3d4156f1-bgqjk\" (UID: \"32fd0aeb-9f3b-4add-bb99-f6cac6cf910d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bgqjk" Oct 31 05:34:34.505246 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:34.505039 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32fd0aeb-9f3b-4add-bb99-f6cac6cf910d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-bgqjk\" (UID: \"32fd0aeb-9f3b-4add-bb99-f6cac6cf910d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bgqjk" Oct 31 05:34:34.505246 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:34.505226 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32fd0aeb-9f3b-4add-bb99-f6cac6cf910d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-bgqjk\" (UID: \"32fd0aeb-9f3b-4add-bb99-f6cac6cf910d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bgqjk" Oct 31 05:34:34.505829 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:34.505804 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32fd0aeb-9f3b-4add-bb99-f6cac6cf910d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-bgqjk\" (UID: \"32fd0aeb-9f3b-4add-bb99-f6cac6cf910d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bgqjk" Oct 31 05:34:34.507767 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:34.507739 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32fd0aeb-9f3b-4add-bb99-f6cac6cf910d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-bgqjk\" (UID: \"32fd0aeb-9f3b-4add-bb99-f6cac6cf910d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bgqjk" Oct 31 05:34:34.528117 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:34.528083 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc9x8\" (UniqueName: \"kubernetes.io/projected/32fd0aeb-9f3b-4add-bb99-f6cac6cf910d-kube-api-access-pc9x8\") pod \"maas-api-revoke-sub-3d4156f1-bgqjk\" (UID: \"32fd0aeb-9f3b-4add-bb99-f6cac6cf910d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bgqjk" Oct 31 05:34:34.660510 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:34.660425 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bgqjk" Oct 31 05:34:34.967475 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:34.967440 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bgqjk"] Oct 31 05:34:34.968966 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:34:34.968940 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32fd0aeb_9f3b_4add_bb99_f6cac6cf910d.slice/crio-3727623e0cbd776c443b8450310e98d13e4a894b3a3b5c1246c314500f4dafbc WatchSource:0}: Error finding container 3727623e0cbd776c443b8450310e98d13e4a894b3a3b5c1246c314500f4dafbc: Status 404 returned error can't find the container with id 3727623e0cbd776c443b8450310e98d13e4a894b3a3b5c1246c314500f4dafbc Oct 31 05:34:35.409378 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:35.409319 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca41b364-4695-43b5-809a-dd5152c7a71b" path="/var/lib/kubelet/pods/ca41b364-4695-43b5-809a-dd5152c7a71b/volumes" Oct 31 05:34:35.657038 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:35.656942 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bgqjk" event={"ID":"32fd0aeb-9f3b-4add-bb99-f6cac6cf910d","Type":"ContainerStarted","Data":"318c9de62da05f30fbac220c9adc3071881304dd95a95884bc9825d7cc87080e"} Oct 31 05:34:35.657038 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:35.657011 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bgqjk" event={"ID":"32fd0aeb-9f3b-4add-bb99-f6cac6cf910d","Type":"ContainerStarted","Data":"3727623e0cbd776c443b8450310e98d13e4a894b3a3b5c1246c314500f4dafbc"} Oct 31 05:34:35.699553 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:35.699482 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bgqjk" podStartSLOduration=1.6994666170000001 podStartE2EDuration="1.699466617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:34:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:34:35.698758307 +0000 UTC m=+5430.455096348" watchObservedRunningTime="2026-10-31 05:34:35.699466617 +0000 UTC m=+5430.455804657" Oct 31 05:34:35.899233 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:35.899184 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9632aa-687df6dccwv6lt" podUID="94b5ffe6-1b77-42e3-8607-1d7158861b72" containerName="payload-processing" containerID="cri-o://7cb534b5e33ecfed1147e32e364f677d93a19a7f29401ebe0e55038201296f9b" gracePeriod=25 Oct 31 05:34:35.929898 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:35.929792 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9632aa-55bd8w5ksw" podUID="aa27c921-f840-4866-aaff-135de970c7f4" containerName="payload-pre-processing" containerID="cri-o://c3de04f7a087ebc7351e983655750a118a9fc35c52768fe46e9ee1ca6c5dcc8b" gracePeriod=25 Oct 31 05:34:36.191467 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:36.191389 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9632aa-687df6dccwv6lt" Oct 31 05:34:36.214576 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:36.214549 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9632aa-55bd8w5ksw" Oct 31 05:34:36.326069 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:36.326030 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa27c921-f840-4866-aaff-135de970c7f4-kube-api-access-csctp\" (UniqueName: \"kubernetes.io/projected/aa27c921-f840-4866-aaff-135de970c7f4-kube-api-access-csctp\") pod \"aa27c921-f840-4866-aaff-135de970c7f4\" (UID: \"aa27c921-f840-4866-aaff-135de970c7f4\") " Oct 31 05:34:36.326254 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:36.326115 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94b5ffe6-1b77-42e3-8607-1d7158861b72-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94b5ffe6-1b77-42e3-8607-1d7158861b72-plugins-config-volume\") pod \"94b5ffe6-1b77-42e3-8607-1d7158861b72\" (UID: \"94b5ffe6-1b77-42e3-8607-1d7158861b72\") " Oct 31 05:34:36.326254 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:36.326143 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94b5ffe6-1b77-42e3-8607-1d7158861b72-kube-api-access-2zmcv\" (UniqueName: \"kubernetes.io/projected/94b5ffe6-1b77-42e3-8607-1d7158861b72-kube-api-access-2zmcv\") pod \"94b5ffe6-1b77-42e3-8607-1d7158861b72\" (UID: \"94b5ffe6-1b77-42e3-8607-1d7158861b72\") " Oct 31 05:34:36.326254 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:36.326188 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa27c921-f840-4866-aaff-135de970c7f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa27c921-f840-4866-aaff-135de970c7f4-plugins-config-volume\") pod \"aa27c921-f840-4866-aaff-135de970c7f4\" (UID: \"aa27c921-f840-4866-aaff-135de970c7f4\") " Oct 31 05:34:36.326665 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:36.326637 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94b5ffe6-1b77-42e3-8607-1d7158861b72-plugins-config-volume" pod "94b5ffe6-1b77-42e3-8607-1d7158861b72" (UID: "94b5ffe6-1b77-42e3-8607-1d7158861b72"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:34:36.326755 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:36.326641 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa27c921-f840-4866-aaff-135de970c7f4-plugins-config-volume" pod "aa27c921-f840-4866-aaff-135de970c7f4" (UID: "aa27c921-f840-4866-aaff-135de970c7f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:34:36.328418 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:36.328396 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa27c921-f840-4866-aaff-135de970c7f4-kube-api-access-csctp" pod "aa27c921-f840-4866-aaff-135de970c7f4" (UID: "aa27c921-f840-4866-aaff-135de970c7f4"). InnerVolumeSpecName "kube-api-access-csctp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:34:36.328495 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:36.328429 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94b5ffe6-1b77-42e3-8607-1d7158861b72-kube-api-access-2zmcv" pod "94b5ffe6-1b77-42e3-8607-1d7158861b72" (UID: "94b5ffe6-1b77-42e3-8607-1d7158861b72"). InnerVolumeSpecName "kube-api-access-2zmcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:34:36.427543 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:36.427502 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csctp\" (UniqueName: \"kubernetes.io/projected/aa27c921-f840-4866-aaff-135de970c7f4-kube-api-access-csctp\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:34:36.427543 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:36.427534 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94b5ffe6-1b77-42e3-8607-1d7158861b72-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:34:36.427992 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:36.427554 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zmcv\" (UniqueName: \"kubernetes.io/projected/94b5ffe6-1b77-42e3-8607-1d7158861b72-kube-api-access-2zmcv\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:34:36.427992 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:36.427567 5019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa27c921-f840-4866-aaff-135de970c7f4-plugins-config-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:34:36.664079 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:36.664042 5019 generic.go:356] "Generic (PLEG): container finished" podID="aa27c921-f840-4866-aaff-135de970c7f4" containerID="c3de04f7a087ebc7351e983655750a118a9fc35c52768fe46e9ee1ca6c5dcc8b" exitCode=0 Oct 31 05:34:36.664283 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:36.664100 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9632aa-55bd8w5ksw" Oct 31 05:34:36.664283 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:36.664131 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9632aa-55bd8w5ksw" event={"ID":"aa27c921-f840-4866-aaff-135de970c7f4","Type":"ContainerDied","Data":"c3de04f7a087ebc7351e983655750a118a9fc35c52768fe46e9ee1ca6c5dcc8b"} Oct 31 05:34:36.664283 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:36.664205 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9632aa-55bd8w5ksw" event={"ID":"aa27c921-f840-4866-aaff-135de970c7f4","Type":"ContainerDied","Data":"8c8ae56a7e8cd2165cc2551ee3bb7b5bc3eab8c979303571996eefc4d8d7714c"} Oct 31 05:34:36.664283 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:36.664228 5019 scope.go:122] "RemoveContainer" containerID="c3de04f7a087ebc7351e983655750a118a9fc35c52768fe46e9ee1ca6c5dcc8b" Oct 31 05:34:36.665972 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:36.665951 5019 generic.go:356] "Generic (PLEG): container finished" podID="32fd0aeb-9f3b-4add-bb99-f6cac6cf910d" containerID="318c9de62da05f30fbac220c9adc3071881304dd95a95884bc9825d7cc87080e" exitCode=0 Oct 31 05:34:36.666102 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:36.665996 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bgqjk" event={"ID":"32fd0aeb-9f3b-4add-bb99-f6cac6cf910d","Type":"ContainerDied","Data":"318c9de62da05f30fbac220c9adc3071881304dd95a95884bc9825d7cc87080e"} Oct 31 05:34:36.667487 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:36.667465 5019 generic.go:356] "Generic (PLEG): container finished" podID="94b5ffe6-1b77-42e3-8607-1d7158861b72" containerID="7cb534b5e33ecfed1147e32e364f677d93a19a7f29401ebe0e55038201296f9b" exitCode=0 Oct 31 05:34:36.667612 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:36.667519 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9632aa-687df6dccwv6lt" event={"ID":"94b5ffe6-1b77-42e3-8607-1d7158861b72","Type":"ContainerDied","Data":"7cb534b5e33ecfed1147e32e364f677d93a19a7f29401ebe0e55038201296f9b"} Oct 31 05:34:36.667612 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:36.667522 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9632aa-687df6dccwv6lt" Oct 31 05:34:36.667612 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:36.667555 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9632aa-687df6dccwv6lt" event={"ID":"94b5ffe6-1b77-42e3-8607-1d7158861b72","Type":"ContainerDied","Data":"d4113f5ba20722df177afbac36107318e808f1f3e8509d1b8947b2b1efe04cab"} Oct 31 05:34:36.675301 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:36.675266 5019 scope.go:122] "RemoveContainer" containerID="c3de04f7a087ebc7351e983655750a118a9fc35c52768fe46e9ee1ca6c5dcc8b" Oct 31 05:34:36.675617 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:34:36.675596 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3de04f7a087ebc7351e983655750a118a9fc35c52768fe46e9ee1ca6c5dcc8b\": container with ID starting with c3de04f7a087ebc7351e983655750a118a9fc35c52768fe46e9ee1ca6c5dcc8b not found: ID does not exist" containerID="c3de04f7a087ebc7351e983655750a118a9fc35c52768fe46e9ee1ca6c5dcc8b" Oct 31 05:34:36.675746 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:36.675623 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3de04f7a087ebc7351e983655750a118a9fc35c52768fe46e9ee1ca6c5dcc8b"} err="failed to get container status \"c3de04f7a087ebc7351e983655750a118a9fc35c52768fe46e9ee1ca6c5dcc8b\": rpc error: code = NotFound desc = could not find container \"c3de04f7a087ebc7351e983655750a118a9fc35c52768fe46e9ee1ca6c5dcc8b\": container with ID starting with c3de04f7a087ebc7351e983655750a118a9fc35c52768fe46e9ee1ca6c5dcc8b not found: ID does not exist" Oct 31 05:34:36.675746 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:36.675639 5019 scope.go:122] "RemoveContainer" containerID="7cb534b5e33ecfed1147e32e364f677d93a19a7f29401ebe0e55038201296f9b" Oct 31 05:34:36.685467 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:36.685444 5019 scope.go:122] "RemoveContainer" containerID="7cb534b5e33ecfed1147e32e364f677d93a19a7f29401ebe0e55038201296f9b" Oct 31 05:34:36.685786 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:34:36.685764 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cb534b5e33ecfed1147e32e364f677d93a19a7f29401ebe0e55038201296f9b\": container with ID starting with 7cb534b5e33ecfed1147e32e364f677d93a19a7f29401ebe0e55038201296f9b not found: ID does not exist" containerID="7cb534b5e33ecfed1147e32e364f677d93a19a7f29401ebe0e55038201296f9b" Oct 31 05:34:36.685844 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:36.685797 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cb534b5e33ecfed1147e32e364f677d93a19a7f29401ebe0e55038201296f9b"} err="failed to get container status \"7cb534b5e33ecfed1147e32e364f677d93a19a7f29401ebe0e55038201296f9b\": rpc error: code = NotFound desc = could not find container \"7cb534b5e33ecfed1147e32e364f677d93a19a7f29401ebe0e55038201296f9b\": container with ID starting with 7cb534b5e33ecfed1147e32e364f677d93a19a7f29401ebe0e55038201296f9b not found: ID does not exist" Oct 31 05:34:36.714955 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:36.714909 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-9632aa-55bd8w5ksw"] Oct 31 05:34:36.720281 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:36.720248 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-9632aa-55bd8w5ksw"] Oct 31 05:34:36.753366 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:36.753312 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-9632aa-687df6dccwv6lt"] Oct 31 05:34:36.759005 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:36.758968 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-9632aa-687df6dccwv6lt"] Oct 31 05:34:37.406458 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:37.406417 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94b5ffe6-1b77-42e3-8607-1d7158861b72" path="/var/lib/kubelet/pods/94b5ffe6-1b77-42e3-8607-1d7158861b72/volumes" Oct 31 05:34:37.406853 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:37.406838 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa27c921-f840-4866-aaff-135de970c7f4" path="/var/lib/kubelet/pods/aa27c921-f840-4866-aaff-135de970c7f4/volumes" Oct 31 05:34:37.834294 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:37.834270 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bgqjk" Oct 31 05:34:37.943220 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:37.943133 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32fd0aeb-9f3b-4add-bb99-f6cac6cf910d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32fd0aeb-9f3b-4add-bb99-f6cac6cf910d-maas-api-openshift-ca\") pod \"32fd0aeb-9f3b-4add-bb99-f6cac6cf910d\" (UID: \"32fd0aeb-9f3b-4add-bb99-f6cac6cf910d\") " Oct 31 05:34:37.943220 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:37.943185 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32fd0aeb-9f3b-4add-bb99-f6cac6cf910d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32fd0aeb-9f3b-4add-bb99-f6cac6cf910d-maas-api-xks-ca\") pod \"32fd0aeb-9f3b-4add-bb99-f6cac6cf910d\" (UID: \"32fd0aeb-9f3b-4add-bb99-f6cac6cf910d\") " Oct 31 05:34:37.943220 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:37.943209 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32fd0aeb-9f3b-4add-bb99-f6cac6cf910d-kube-api-access-pc9x8\" (UniqueName: \"kubernetes.io/projected/32fd0aeb-9f3b-4add-bb99-f6cac6cf910d-kube-api-access-pc9x8\") pod \"32fd0aeb-9f3b-4add-bb99-f6cac6cf910d\" (UID: \"32fd0aeb-9f3b-4add-bb99-f6cac6cf910d\") " Oct 31 05:34:37.943654 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:37.943631 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32fd0aeb-9f3b-4add-bb99-f6cac6cf910d-maas-api-openshift-ca" pod "32fd0aeb-9f3b-4add-bb99-f6cac6cf910d" (UID: "32fd0aeb-9f3b-4add-bb99-f6cac6cf910d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:34:37.945485 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:37.945457 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32fd0aeb-9f3b-4add-bb99-f6cac6cf910d-maas-api-xks-ca" pod "32fd0aeb-9f3b-4add-bb99-f6cac6cf910d" (UID: "32fd0aeb-9f3b-4add-bb99-f6cac6cf910d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:34:37.945611 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:37.945481 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32fd0aeb-9f3b-4add-bb99-f6cac6cf910d-kube-api-access-pc9x8" pod "32fd0aeb-9f3b-4add-bb99-f6cac6cf910d" (UID: "32fd0aeb-9f3b-4add-bb99-f6cac6cf910d"). InnerVolumeSpecName "kube-api-access-pc9x8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:34:38.044201 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:38.044170 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32fd0aeb-9f3b-4add-bb99-f6cac6cf910d-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:34:38.044201 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:38.044197 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32fd0aeb-9f3b-4add-bb99-f6cac6cf910d-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:34:38.044201 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:38.044207 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pc9x8\" (UniqueName: \"kubernetes.io/projected/32fd0aeb-9f3b-4add-bb99-f6cac6cf910d-kube-api-access-pc9x8\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:34:38.682774 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:38.682743 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bgqjk" Oct 31 05:34:38.682933 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:38.682781 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bgqjk" event={"ID":"32fd0aeb-9f3b-4add-bb99-f6cac6cf910d","Type":"ContainerDied","Data":"3727623e0cbd776c443b8450310e98d13e4a894b3a3b5c1246c314500f4dafbc"} Oct 31 05:34:38.682933 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:38.682829 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3727623e0cbd776c443b8450310e98d13e4a894b3a3b5c1246c314500f4dafbc" Oct 31 05:34:43.027862 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:43.027831 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kwklx"] Oct 31 05:34:43.039846 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:43.039818 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kwklx"] Oct 31 05:34:43.406041 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:43.405968 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f76dead-50bf-4d0e-94e0-1152a518abe3" path="/var/lib/kubelet/pods/9f76dead-50bf-4d0e-94e0-1152a518abe3/volumes" Oct 31 05:34:52.331759 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:52.331682 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vwcwp"] Oct 31 05:34:52.332173 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:52.331877 5019 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vwcwp" podUID="1f9f7baf-a850-453c-8e16-849942ba754a" containerName="manager" containerID="cri-o://2a9928a702fcca2791e0857443fcbcff01dc994a018a93a1a2f29af6c1cdfeaf" gracePeriod=10 Oct 31 05:34:52.390400 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:34:52.390370 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:34:52.590207 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:52.590152 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vwcwp" Oct 31 05:34:52.765486 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:52.765452 5019 generic.go:356] "Generic (PLEG): container finished" podID="1f9f7baf-a850-453c-8e16-849942ba754a" containerID="2a9928a702fcca2791e0857443fcbcff01dc994a018a93a1a2f29af6c1cdfeaf" exitCode=0 Oct 31 05:34:52.765682 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:52.765498 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vwcwp" Oct 31 05:34:52.765682 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:52.765540 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vwcwp" event={"ID":"1f9f7baf-a850-453c-8e16-849942ba754a","Type":"ContainerDied","Data":"2a9928a702fcca2791e0857443fcbcff01dc994a018a93a1a2f29af6c1cdfeaf"} Oct 31 05:34:52.765682 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:52.765591 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vwcwp" event={"ID":"1f9f7baf-a850-453c-8e16-849942ba754a","Type":"ContainerDied","Data":"a67cae05638d07ae6cdd048cb13748c9a85745fd483e414ab659d354bfe2805d"} Oct 31 05:34:52.765682 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:52.765614 5019 scope.go:122] "RemoveContainer" containerID="2a9928a702fcca2791e0857443fcbcff01dc994a018a93a1a2f29af6c1cdfeaf" Oct 31 05:34:52.771571 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:52.771547 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f9f7baf-a850-453c-8e16-849942ba754a-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1f9f7baf-a850-453c-8e16-849942ba754a-extensions-socket-volume\") pod \"1f9f7baf-a850-453c-8e16-849942ba754a\" (UID: \"1f9f7baf-a850-453c-8e16-849942ba754a\") " Oct 31 05:34:52.771713 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:52.771611 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f9f7baf-a850-453c-8e16-849942ba754a-kube-api-access-wffw4\" (UniqueName: \"kubernetes.io/projected/1f9f7baf-a850-453c-8e16-849942ba754a-kube-api-access-wffw4\") pod \"1f9f7baf-a850-453c-8e16-849942ba754a\" (UID: \"1f9f7baf-a850-453c-8e16-849942ba754a\") " Oct 31 05:34:52.771958 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:52.771937 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f9f7baf-a850-453c-8e16-849942ba754a-extensions-socket-volume" pod "1f9f7baf-a850-453c-8e16-849942ba754a" (UID: "1f9f7baf-a850-453c-8e16-849942ba754a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 05:34:52.773663 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:52.773644 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f9f7baf-a850-453c-8e16-849942ba754a-kube-api-access-wffw4" pod "1f9f7baf-a850-453c-8e16-849942ba754a" (UID: "1f9f7baf-a850-453c-8e16-849942ba754a"). InnerVolumeSpecName "kube-api-access-wffw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:34:52.779450 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:52.779415 5019 scope.go:122] "RemoveContainer" containerID="2a9928a702fcca2791e0857443fcbcff01dc994a018a93a1a2f29af6c1cdfeaf" Oct 31 05:34:52.779739 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:34:52.779715 5019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a9928a702fcca2791e0857443fcbcff01dc994a018a93a1a2f29af6c1cdfeaf\": container with ID starting with 2a9928a702fcca2791e0857443fcbcff01dc994a018a93a1a2f29af6c1cdfeaf not found: ID does not exist" containerID="2a9928a702fcca2791e0857443fcbcff01dc994a018a93a1a2f29af6c1cdfeaf" Oct 31 05:34:52.779847 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:52.779750 5019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a9928a702fcca2791e0857443fcbcff01dc994a018a93a1a2f29af6c1cdfeaf"} err="failed to get container status \"2a9928a702fcca2791e0857443fcbcff01dc994a018a93a1a2f29af6c1cdfeaf\": rpc error: code = NotFound desc = could not find container \"2a9928a702fcca2791e0857443fcbcff01dc994a018a93a1a2f29af6c1cdfeaf\": container with ID starting with 2a9928a702fcca2791e0857443fcbcff01dc994a018a93a1a2f29af6c1cdfeaf not found: ID does not exist" Oct 31 05:34:52.872949 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:52.872896 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wffw4\" (UniqueName: \"kubernetes.io/projected/1f9f7baf-a850-453c-8e16-849942ba754a-kube-api-access-wffw4\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:34:52.872949 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:52.872919 5019 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1f9f7baf-a850-453c-8e16-849942ba754a-extensions-socket-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:34:53.112130 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:53.112101 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vwcwp"] Oct 31 05:34:53.123856 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:53.123775 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vwcwp"] Oct 31 05:34:53.406627 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:34:53.406570 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f9f7baf-a850-453c-8e16-849942ba754a" path="/var/lib/kubelet/pods/1f9f7baf-a850-453c-8e16-849942ba754a/volumes" Oct 31 05:35:15.104371 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:35:15.104317 5019 scope.go:122] "RemoveContainer" containerID="3c2f98146ce8425b145e8db54e5f32823a454ec1e0838fb1ad8dc53c6b0e0c36" Oct 31 05:35:15.113662 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:35:15.113636 5019 scope.go:122] "RemoveContainer" containerID="5a6853750dfe00a9ccc5305010e19eb65da299f4e6445a3a6acfc0429ea05a6a" Oct 31 05:35:15.123002 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:35:15.122982 5019 scope.go:122] "RemoveContainer" containerID="360702bfefd49e63d9b99ae2c8921f3f35e229e636392c9dec12c071715f13b6" Oct 31 05:35:22.390721 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:35:22.390682 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:35:23.028543 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:35:23.028504 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lswwt"] Oct 31 05:35:23.040321 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:35:23.040288 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-lswwt"] Oct 31 05:35:23.406369 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:35:23.406274 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4fddbfb-2cf9-411a-a461-7a8e8d3d5f7c" path="/var/lib/kubelet/pods/c4fddbfb-2cf9-411a-a461-7a8e8d3d5f7c/volumes" Oct 31 05:35:41.030503 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:35:41.030461 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fb8lv"] Oct 31 05:35:41.041993 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:35:41.041962 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fb8lv"] Oct 31 05:35:41.406818 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:35:41.406738 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ade78fb-67c0-4dbe-8e5d-c17046d47af5" path="/var/lib/kubelet/pods/1ade78fb-67c0-4dbe-8e5d-c17046d47af5/volumes" Oct 31 05:35:52.391327 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:35:52.391279 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:35:55.027699 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:35:55.027654 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pqls6"] Oct 31 05:35:55.039646 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:35:55.039607 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pqls6"] Oct 31 05:35:55.405941 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:35:55.405844 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2148a95c-cdb0-4675-aa27-2ed4b7804bfc" path="/var/lib/kubelet/pods/2148a95c-cdb0-4675-aa27-2ed4b7804bfc/volumes" Oct 31 05:35:55.466977 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:35:55.466936 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cljqx"] Oct 31 05:35:55.467417 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:35:55.467403 5019 cpu_manager.go:414] "containerMap: removing container" podUID="1f9f7baf-a850-453c-8e16-849942ba754a" containerName="manager" Oct 31 05:35:55.467473 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:35:55.467419 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f9f7baf-a850-453c-8e16-849942ba754a" containerName="manager" Oct 31 05:35:55.467473 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:35:55.467430 5019 cpu_manager.go:414] "containerMap: removing container" podUID="aa27c921-f840-4866-aaff-135de970c7f4" containerName="payload-pre-processing" Oct 31 05:35:55.467473 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:35:55.467436 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa27c921-f840-4866-aaff-135de970c7f4" containerName="payload-pre-processing" Oct 31 05:35:55.467592 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:35:55.467473 5019 cpu_manager.go:414] "containerMap: removing container" podUID="32fd0aeb-9f3b-4add-bb99-f6cac6cf910d" containerName="revoke-keys" Oct 31 05:35:55.467592 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:35:55.467480 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32fd0aeb-9f3b-4add-bb99-f6cac6cf910d" containerName="revoke-keys" Oct 31 05:35:55.467592 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:35:55.467494 5019 cpu_manager.go:414] "containerMap: removing container" podUID="94b5ffe6-1b77-42e3-8607-1d7158861b72" containerName="payload-processing" Oct 31 05:35:55.467592 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:35:55.467499 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94b5ffe6-1b77-42e3-8607-1d7158861b72" containerName="payload-processing" Oct 31 05:35:55.467592 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:35:55.467576 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="32fd0aeb-9f3b-4add-bb99-f6cac6cf910d" containerName="revoke-keys" Oct 31 05:35:55.467592 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:35:55.467587 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="94b5ffe6-1b77-42e3-8607-1d7158861b72" containerName="payload-processing" Oct 31 05:35:55.467800 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:35:55.467597 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f9f7baf-a850-453c-8e16-849942ba754a" containerName="manager" Oct 31 05:35:55.467800 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:35:55.467603 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa27c921-f840-4866-aaff-135de970c7f4" containerName="payload-pre-processing" Oct 31 05:35:55.471515 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:35:55.471487 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cljqx" Oct 31 05:35:55.486687 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:35:55.486647 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-5jq5s\"" Oct 31 05:35:55.501681 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:35:55.501644 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cljqx"] Oct 31 05:35:55.600029 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:35:55.599968 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/26d2360a-6a02-44ea-a750-6f634b2f90a6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-cljqx\" (UID: \"26d2360a-6a02-44ea-a750-6f634b2f90a6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cljqx" Oct 31 05:35:55.600313 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:35:55.600039 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqpkl\" (UniqueName: \"kubernetes.io/projected/26d2360a-6a02-44ea-a750-6f634b2f90a6-kube-api-access-qqpkl\") pod \"kuadrant-operator-controller-manager-678d87766f-cljqx\" (UID: \"26d2360a-6a02-44ea-a750-6f634b2f90a6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cljqx" Oct 31 05:35:55.700914 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:35:55.700817 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/26d2360a-6a02-44ea-a750-6f634b2f90a6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-cljqx\" (UID: \"26d2360a-6a02-44ea-a750-6f634b2f90a6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cljqx" Oct 31 05:35:55.700914 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:35:55.700873 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqpkl\" (UniqueName: \"kubernetes.io/projected/26d2360a-6a02-44ea-a750-6f634b2f90a6-kube-api-access-qqpkl\") pod \"kuadrant-operator-controller-manager-678d87766f-cljqx\" (UID: \"26d2360a-6a02-44ea-a750-6f634b2f90a6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cljqx" Oct 31 05:35:55.701401 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:35:55.701370 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/26d2360a-6a02-44ea-a750-6f634b2f90a6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-cljqx\" (UID: \"26d2360a-6a02-44ea-a750-6f634b2f90a6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cljqx" Oct 31 05:35:55.725474 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:35:55.725440 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqpkl\" (UniqueName: \"kubernetes.io/projected/26d2360a-6a02-44ea-a750-6f634b2f90a6-kube-api-access-qqpkl\") pod \"kuadrant-operator-controller-manager-678d87766f-cljqx\" (UID: \"26d2360a-6a02-44ea-a750-6f634b2f90a6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cljqx" Oct 31 05:35:55.785470 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:35:55.785425 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cljqx" Oct 31 05:35:56.109570 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:35:56.109534 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cljqx"] Oct 31 05:35:56.111292 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:35:56.111252 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26d2360a_6a02_44ea_a750_6f634b2f90a6.slice/crio-2522b5f98b6552e4f599e1365485c2e7904d593926a64eddff4a38c806dd857c WatchSource:0}: Error finding container 2522b5f98b6552e4f599e1365485c2e7904d593926a64eddff4a38c806dd857c: Status 404 returned error can't find the container with id 2522b5f98b6552e4f599e1365485c2e7904d593926a64eddff4a38c806dd857c Oct 31 05:35:56.152132 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:35:56.152074 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cljqx" event={"ID":"26d2360a-6a02-44ea-a750-6f634b2f90a6","Type":"ContainerStarted","Data":"2522b5f98b6552e4f599e1365485c2e7904d593926a64eddff4a38c806dd857c"} Oct 31 05:35:57.161614 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:35:57.161567 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cljqx" event={"ID":"26d2360a-6a02-44ea-a750-6f634b2f90a6","Type":"ContainerStarted","Data":"84726086404f5b3ca3c145fe2a07f1f78849c56ec65fd3733dedc4779a7db97a"} Oct 31 05:35:57.162020 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:35:57.161633 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cljqx" Oct 31 05:35:57.230288 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:35:57.230231 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cljqx" podStartSLOduration=2.230214539 podStartE2EDuration="2.230214539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:35:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:35:57.216823494 +0000 UTC m=+5511.973161567" watchObservedRunningTime="2026-10-31 05:35:57.230214539 +0000 UTC m=+5511.986552582" Oct 31 05:36:08.170248 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:08.170207 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cljqx" Oct 31 05:36:15.170755 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:15.170698 5019 scope.go:122] "RemoveContainer" containerID="b4195590cb12e8c2d72eebbb69c002d9787166a28c49d5c62b9a74a6fc96de1c" Oct 31 05:36:15.186313 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:15.186280 5019 scope.go:122] "RemoveContainer" containerID="37d8791d56ca4f417147d97b85c04fc2231dec515b2d8d90d5491f372835e49c" Oct 31 05:36:16.539580 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:16.539479 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-shld4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 05:36:16.540987 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:16.540313 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-shld4" podUID="a69fd138-9906-4127-ac26-38a21a5b7590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 05:36:16.613577 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:16.613530 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.221:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 05:36:16.613943 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:16.613903 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5" podUID="6d68d19b-5392-4308-9e2c-4b5e7593d17b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.221:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 05:36:18.612297 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:18.612259 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-q6tnf"] Oct 31 05:36:18.617682 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:18.617659 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-q6tnf" Oct 31 05:36:18.644375 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:18.644342 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:36:18.644563 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:18.644448 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:36:18.644563 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:18.644532 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfhzn\"" Oct 31 05:36:18.644694 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:18.644599 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:36:18.647027 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:18.646999 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-q6tnf"] Oct 31 05:36:18.698101 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:18.698066 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2356a24f-9ff0-477e-86c3-cad6ec2ab2b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-q6tnf\" (UID: \"2356a24f-9ff0-477e-86c3-cad6ec2ab2b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-q6tnf" Oct 31 05:36:18.698299 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:18.698106 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2356a24f-9ff0-477e-86c3-cad6ec2ab2b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-q6tnf\" (UID: \"2356a24f-9ff0-477e-86c3-cad6ec2ab2b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-q6tnf" Oct 31 05:36:18.698404 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:18.698310 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6m64\" (UniqueName: \"kubernetes.io/projected/2356a24f-9ff0-477e-86c3-cad6ec2ab2b5-kube-api-access-z6m64\") pod \"maas-api-revoke-sub-d576e68b-q6tnf\" (UID: \"2356a24f-9ff0-477e-86c3-cad6ec2ab2b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-q6tnf" Oct 31 05:36:18.799228 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:18.799181 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2356a24f-9ff0-477e-86c3-cad6ec2ab2b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-q6tnf\" (UID: \"2356a24f-9ff0-477e-86c3-cad6ec2ab2b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-q6tnf" Oct 31 05:36:18.799455 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:18.799246 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2356a24f-9ff0-477e-86c3-cad6ec2ab2b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-q6tnf\" (UID: \"2356a24f-9ff0-477e-86c3-cad6ec2ab2b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-q6tnf" Oct 31 05:36:18.799455 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:18.799409 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6m64\" (UniqueName: \"kubernetes.io/projected/2356a24f-9ff0-477e-86c3-cad6ec2ab2b5-kube-api-access-z6m64\") pod \"maas-api-revoke-sub-d576e68b-q6tnf\" (UID: \"2356a24f-9ff0-477e-86c3-cad6ec2ab2b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-q6tnf" Oct 31 05:36:18.800124 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:18.800091 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2356a24f-9ff0-477e-86c3-cad6ec2ab2b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-q6tnf\" (UID: \"2356a24f-9ff0-477e-86c3-cad6ec2ab2b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-q6tnf" Oct 31 05:36:18.801641 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:18.801613 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2356a24f-9ff0-477e-86c3-cad6ec2ab2b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-q6tnf\" (UID: \"2356a24f-9ff0-477e-86c3-cad6ec2ab2b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-q6tnf" Oct 31 05:36:18.822466 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:18.822439 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6m64\" (UniqueName: \"kubernetes.io/projected/2356a24f-9ff0-477e-86c3-cad6ec2ab2b5-kube-api-access-z6m64\") pod \"maas-api-revoke-sub-d576e68b-q6tnf\" (UID: \"2356a24f-9ff0-477e-86c3-cad6ec2ab2b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-q6tnf" Oct 31 05:36:18.932537 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:18.932455 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-q6tnf" Oct 31 05:36:19.228495 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:19.228453 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-q6tnf"] Oct 31 05:36:19.230178 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:36:19.230152 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2356a24f_9ff0_477e_86c3_cad6ec2ab2b5.slice/crio-a7bd81a6986579a5a5ee152a718a1e434be492db87b26fb39e776643901d302b WatchSource:0}: Error finding container a7bd81a6986579a5a5ee152a718a1e434be492db87b26fb39e776643901d302b: Status 404 returned error can't find the container with id a7bd81a6986579a5a5ee152a718a1e434be492db87b26fb39e776643901d302b Oct 31 05:36:19.326685 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:19.326636 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-q6tnf" event={"ID":"2356a24f-9ff0-477e-86c3-cad6ec2ab2b5","Type":"ContainerStarted","Data":"a7bd81a6986579a5a5ee152a718a1e434be492db87b26fb39e776643901d302b"} Oct 31 05:36:20.333438 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:20.333353 5019 generic.go:356] "Generic (PLEG): container finished" podID="2356a24f-9ff0-477e-86c3-cad6ec2ab2b5" containerID="d5d5dce65be3628fcdcba7d1a4643843336439abda3852a899eec67177eb8209" exitCode=0 Oct 31 05:36:20.333825 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:20.333428 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-q6tnf" event={"ID":"2356a24f-9ff0-477e-86c3-cad6ec2ab2b5","Type":"ContainerDied","Data":"d5d5dce65be3628fcdcba7d1a4643843336439abda3852a899eec67177eb8209"} Oct 31 05:36:21.494689 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:21.494660 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-q6tnf" Oct 31 05:36:21.520084 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:21.520047 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2356a24f-9ff0-477e-86c3-cad6ec2ab2b5-kube-api-access-z6m64\" (UniqueName: \"kubernetes.io/projected/2356a24f-9ff0-477e-86c3-cad6ec2ab2b5-kube-api-access-z6m64\") pod \"2356a24f-9ff0-477e-86c3-cad6ec2ab2b5\" (UID: \"2356a24f-9ff0-477e-86c3-cad6ec2ab2b5\") " Oct 31 05:36:21.520284 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:21.520127 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2356a24f-9ff0-477e-86c3-cad6ec2ab2b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2356a24f-9ff0-477e-86c3-cad6ec2ab2b5-maas-api-xks-ca\") pod \"2356a24f-9ff0-477e-86c3-cad6ec2ab2b5\" (UID: \"2356a24f-9ff0-477e-86c3-cad6ec2ab2b5\") " Oct 31 05:36:21.520284 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:21.520190 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2356a24f-9ff0-477e-86c3-cad6ec2ab2b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2356a24f-9ff0-477e-86c3-cad6ec2ab2b5-maas-api-openshift-ca\") pod \"2356a24f-9ff0-477e-86c3-cad6ec2ab2b5\" (UID: \"2356a24f-9ff0-477e-86c3-cad6ec2ab2b5\") " Oct 31 05:36:21.520643 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:21.520616 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2356a24f-9ff0-477e-86c3-cad6ec2ab2b5-maas-api-openshift-ca" pod "2356a24f-9ff0-477e-86c3-cad6ec2ab2b5" (UID: "2356a24f-9ff0-477e-86c3-cad6ec2ab2b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:36:21.522370 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:21.522313 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2356a24f-9ff0-477e-86c3-cad6ec2ab2b5-maas-api-xks-ca" pod "2356a24f-9ff0-477e-86c3-cad6ec2ab2b5" (UID: "2356a24f-9ff0-477e-86c3-cad6ec2ab2b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:36:21.522492 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:21.522384 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2356a24f-9ff0-477e-86c3-cad6ec2ab2b5-kube-api-access-z6m64" pod "2356a24f-9ff0-477e-86c3-cad6ec2ab2b5" (UID: "2356a24f-9ff0-477e-86c3-cad6ec2ab2b5"). InnerVolumeSpecName "kube-api-access-z6m64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:36:21.624147 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:21.621702 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2356a24f-9ff0-477e-86c3-cad6ec2ab2b5-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:36:21.624147 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:21.621747 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2356a24f-9ff0-477e-86c3-cad6ec2ab2b5-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:36:21.624147 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:21.621770 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6m64\" (UniqueName: \"kubernetes.io/projected/2356a24f-9ff0-477e-86c3-cad6ec2ab2b5-kube-api-access-z6m64\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:36:22.347142 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:22.347105 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-q6tnf" Oct 31 05:36:22.347355 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:22.347117 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-q6tnf" event={"ID":"2356a24f-9ff0-477e-86c3-cad6ec2ab2b5","Type":"ContainerDied","Data":"a7bd81a6986579a5a5ee152a718a1e434be492db87b26fb39e776643901d302b"} Oct 31 05:36:22.347355 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:22.347259 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7bd81a6986579a5a5ee152a718a1e434be492db87b26fb39e776643901d302b" Oct 31 05:36:22.390385 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:36:22.390349 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:36:35.966630 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:35.966579 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9554q"] Oct 31 05:36:35.967366 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:35.967128 5019 cpu_manager.go:414] "containerMap: removing container" podUID="2356a24f-9ff0-477e-86c3-cad6ec2ab2b5" containerName="revoke-keys" Oct 31 05:36:35.967366 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:35.967150 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2356a24f-9ff0-477e-86c3-cad6ec2ab2b5" containerName="revoke-keys" Oct 31 05:36:35.967366 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:35.967267 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="2356a24f-9ff0-477e-86c3-cad6ec2ab2b5" containerName="revoke-keys" Oct 31 05:36:35.970789 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:35.970765 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9554q" Oct 31 05:36:35.986273 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:35.986236 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:36:35.998456 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:35.998417 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:36:35.998456 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:35.998450 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:36:35.999011 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:35.998419 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfhzn\"" Oct 31 05:36:36.001650 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:36.001623 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9554q"] Oct 31 05:36:36.030651 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:36.030598 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5k2s\" (UniqueName: \"kubernetes.io/projected/e5650965-4ba0-4fe8-a2fd-ff42413710f9-kube-api-access-h5k2s\") pod \"maas-api-revoke-sub-d2bc93bc-9554q\" (UID: \"e5650965-4ba0-4fe8-a2fd-ff42413710f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9554q" Oct 31 05:36:36.030856 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:36.030782 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5650965-4ba0-4fe8-a2fd-ff42413710f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-9554q\" (UID: \"e5650965-4ba0-4fe8-a2fd-ff42413710f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9554q" Oct 31 05:36:36.030856 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:36.030826 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5650965-4ba0-4fe8-a2fd-ff42413710f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-9554q\" (UID: \"e5650965-4ba0-4fe8-a2fd-ff42413710f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9554q" Oct 31 05:36:36.132177 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:36.132128 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5650965-4ba0-4fe8-a2fd-ff42413710f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-9554q\" (UID: \"e5650965-4ba0-4fe8-a2fd-ff42413710f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9554q" Oct 31 05:36:36.132453 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:36.132252 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5k2s\" (UniqueName: \"kubernetes.io/projected/e5650965-4ba0-4fe8-a2fd-ff42413710f9-kube-api-access-h5k2s\") pod \"maas-api-revoke-sub-d2bc93bc-9554q\" (UID: \"e5650965-4ba0-4fe8-a2fd-ff42413710f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9554q" Oct 31 05:36:36.132453 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:36.132303 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5650965-4ba0-4fe8-a2fd-ff42413710f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-9554q\" (UID: \"e5650965-4ba0-4fe8-a2fd-ff42413710f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9554q" Oct 31 05:36:36.132948 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:36.132918 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5650965-4ba0-4fe8-a2fd-ff42413710f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-9554q\" (UID: \"e5650965-4ba0-4fe8-a2fd-ff42413710f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9554q" Oct 31 05:36:36.134752 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:36.134727 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5650965-4ba0-4fe8-a2fd-ff42413710f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-9554q\" (UID: \"e5650965-4ba0-4fe8-a2fd-ff42413710f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9554q" Oct 31 05:36:36.156195 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:36.156156 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5k2s\" (UniqueName: \"kubernetes.io/projected/e5650965-4ba0-4fe8-a2fd-ff42413710f9-kube-api-access-h5k2s\") pod \"maas-api-revoke-sub-d2bc93bc-9554q\" (UID: \"e5650965-4ba0-4fe8-a2fd-ff42413710f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9554q" Oct 31 05:36:36.285010 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:36.284917 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9554q" Oct 31 05:36:36.587475 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:36.587442 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9554q"] Oct 31 05:36:36.589848 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:36:36.589801 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5650965_4ba0_4fe8_a2fd_ff42413710f9.slice/crio-190c66ec4efc407d6b4c4089cadef550902f597cb02553936d6b4e0f60a67092 WatchSource:0}: Error finding container 190c66ec4efc407d6b4c4089cadef550902f597cb02553936d6b4e0f60a67092: Status 404 returned error can't find the container with id 190c66ec4efc407d6b4c4089cadef550902f597cb02553936d6b4e0f60a67092 Oct 31 05:36:37.501021 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:37.500984 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9554q" event={"ID":"e5650965-4ba0-4fe8-a2fd-ff42413710f9","Type":"ContainerStarted","Data":"43268018ec530e0d169848f12e67cdcb7c2a88e407f5cf75abf117764a9982a0"} Oct 31 05:36:37.501499 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:37.501053 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9554q" event={"ID":"e5650965-4ba0-4fe8-a2fd-ff42413710f9","Type":"ContainerStarted","Data":"190c66ec4efc407d6b4c4089cadef550902f597cb02553936d6b4e0f60a67092"} Oct 31 05:36:38.508011 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:38.507971 5019 generic.go:356] "Generic (PLEG): container finished" podID="e5650965-4ba0-4fe8-a2fd-ff42413710f9" containerID="43268018ec530e0d169848f12e67cdcb7c2a88e407f5cf75abf117764a9982a0" exitCode=0 Oct 31 05:36:38.508476 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:38.508055 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9554q" event={"ID":"e5650965-4ba0-4fe8-a2fd-ff42413710f9","Type":"ContainerDied","Data":"43268018ec530e0d169848f12e67cdcb7c2a88e407f5cf75abf117764a9982a0"} Oct 31 05:36:38.652735 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:38.652706 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9554q" Oct 31 05:36:38.762708 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:38.762602 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5650965-4ba0-4fe8-a2fd-ff42413710f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5650965-4ba0-4fe8-a2fd-ff42413710f9-maas-api-xks-ca\") pod \"e5650965-4ba0-4fe8-a2fd-ff42413710f9\" (UID: \"e5650965-4ba0-4fe8-a2fd-ff42413710f9\") " Oct 31 05:36:38.762708 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:38.762697 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5650965-4ba0-4fe8-a2fd-ff42413710f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5650965-4ba0-4fe8-a2fd-ff42413710f9-maas-api-openshift-ca\") pod \"e5650965-4ba0-4fe8-a2fd-ff42413710f9\" (UID: \"e5650965-4ba0-4fe8-a2fd-ff42413710f9\") " Oct 31 05:36:38.762992 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:38.762802 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5650965-4ba0-4fe8-a2fd-ff42413710f9-kube-api-access-h5k2s\" (UniqueName: \"kubernetes.io/projected/e5650965-4ba0-4fe8-a2fd-ff42413710f9-kube-api-access-h5k2s\") pod \"e5650965-4ba0-4fe8-a2fd-ff42413710f9\" (UID: \"e5650965-4ba0-4fe8-a2fd-ff42413710f9\") " Oct 31 05:36:38.763124 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:38.763090 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5650965-4ba0-4fe8-a2fd-ff42413710f9-maas-api-openshift-ca" pod "e5650965-4ba0-4fe8-a2fd-ff42413710f9" (UID: "e5650965-4ba0-4fe8-a2fd-ff42413710f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:36:38.763279 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:38.763262 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5650965-4ba0-4fe8-a2fd-ff42413710f9-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:36:38.764998 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:38.764966 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5650965-4ba0-4fe8-a2fd-ff42413710f9-kube-api-access-h5k2s" pod "e5650965-4ba0-4fe8-a2fd-ff42413710f9" (UID: "e5650965-4ba0-4fe8-a2fd-ff42413710f9"). InnerVolumeSpecName "kube-api-access-h5k2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:36:38.765139 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:38.764973 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5650965-4ba0-4fe8-a2fd-ff42413710f9-maas-api-xks-ca" pod "e5650965-4ba0-4fe8-a2fd-ff42413710f9" (UID: "e5650965-4ba0-4fe8-a2fd-ff42413710f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:36:38.864724 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:38.864675 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5650965-4ba0-4fe8-a2fd-ff42413710f9-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:36:38.864724 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:38.864715 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5k2s\" (UniqueName: \"kubernetes.io/projected/e5650965-4ba0-4fe8-a2fd-ff42413710f9-kube-api-access-h5k2s\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:36:39.516092 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:39.516052 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9554q" Oct 31 05:36:39.516619 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:39.516089 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9554q" event={"ID":"e5650965-4ba0-4fe8-a2fd-ff42413710f9","Type":"ContainerDied","Data":"190c66ec4efc407d6b4c4089cadef550902f597cb02553936d6b4e0f60a67092"} Oct 31 05:36:39.516619 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:39.516138 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="190c66ec4efc407d6b4c4089cadef550902f597cb02553936d6b4e0f60a67092" Oct 31 05:36:49.026171 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:49.026127 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-hwcsq"] Oct 31 05:36:49.037909 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:49.037865 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-hwcsq"] Oct 31 05:36:49.406406 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:36:49.406296 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff935065-618c-438b-aa40-4f2150df3ac5" path="/var/lib/kubelet/pods/ff935065-618c-438b-aa40-4f2150df3ac5/volumes" Oct 31 05:36:52.392112 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:36:52.392057 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:37:00.029358 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:00.029286 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xc9fl"] Oct 31 05:37:00.041006 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:00.040960 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xc9fl"] Oct 31 05:37:01.412630 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:01.412574 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-rgj85 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.218:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 05:37:01.413380 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:01.413318 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rgj85" podUID="8430b919-a9dd-4c7e-9266-ef6872cc0b9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.218:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 05:37:01.447367 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:01.444306 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6a313f9-e056-49a9-9619-121fa96de322" path="/var/lib/kubelet/pods/a6a313f9-e056-49a9-9619-121fa96de322/volumes" Oct 31 05:37:01.545375 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:01.543461 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-shld4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 05:37:01.545375 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:01.543567 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-shld4" podUID="a69fd138-9906-4127-ac26-38a21a5b7590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 05:37:01.615353 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:01.614544 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.221:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 05:37:01.615353 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:01.614622 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5" podUID="6d68d19b-5392-4308-9e2c-4b5e7593d17b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.221:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 05:37:06.893035 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:06.892990 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-440fe870-mltxk"] Oct 31 05:37:06.893552 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:06.893512 5019 cpu_manager.go:414] "containerMap: removing container" podUID="e5650965-4ba0-4fe8-a2fd-ff42413710f9" containerName="revoke-keys" Oct 31 05:37:06.893552 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:06.893525 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5650965-4ba0-4fe8-a2fd-ff42413710f9" containerName="revoke-keys" Oct 31 05:37:06.893640 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:06.893618 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5650965-4ba0-4fe8-a2fd-ff42413710f9" containerName="revoke-keys" Oct 31 05:37:06.897120 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:06.897094 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-440fe870-mltxk" Oct 31 05:37:06.914786 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:06.913947 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56505ab3-8320-474e-84a0-a8470cbe6453-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-440fe870-mltxk\" (UID: \"56505ab3-8320-474e-84a0-a8470cbe6453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-440fe870-mltxk" Oct 31 05:37:06.914786 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:06.914009 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87pvg\" (UniqueName: \"kubernetes.io/projected/56505ab3-8320-474e-84a0-a8470cbe6453-kube-api-access-87pvg\") pod \"maas-api-revoke-sub-440fe870-mltxk\" (UID: \"56505ab3-8320-474e-84a0-a8470cbe6453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-440fe870-mltxk" Oct 31 05:37:06.914786 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:06.914107 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56505ab3-8320-474e-84a0-a8470cbe6453-maas-api-xks-ca\") pod \"maas-api-revoke-sub-440fe870-mltxk\" (UID: \"56505ab3-8320-474e-84a0-a8470cbe6453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-440fe870-mltxk" Oct 31 05:37:06.917903 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:06.917867 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-440fe870-mltxk"] Oct 31 05:37:06.924934 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:06.924903 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:37:06.924934 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:06.924936 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:37:06.966808 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:06.966764 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:37:06.966808 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:06.966804 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfhzn\"" Oct 31 05:37:07.014790 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:07.014747 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56505ab3-8320-474e-84a0-a8470cbe6453-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-440fe870-mltxk\" (UID: \"56505ab3-8320-474e-84a0-a8470cbe6453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-440fe870-mltxk" Oct 31 05:37:07.014790 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:07.014798 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87pvg\" (UniqueName: \"kubernetes.io/projected/56505ab3-8320-474e-84a0-a8470cbe6453-kube-api-access-87pvg\") pod \"maas-api-revoke-sub-440fe870-mltxk\" (UID: \"56505ab3-8320-474e-84a0-a8470cbe6453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-440fe870-mltxk" Oct 31 05:37:07.015111 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:07.014826 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56505ab3-8320-474e-84a0-a8470cbe6453-maas-api-xks-ca\") pod \"maas-api-revoke-sub-440fe870-mltxk\" (UID: \"56505ab3-8320-474e-84a0-a8470cbe6453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-440fe870-mltxk" Oct 31 05:37:07.015536 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:07.015502 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56505ab3-8320-474e-84a0-a8470cbe6453-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-440fe870-mltxk\" (UID: \"56505ab3-8320-474e-84a0-a8470cbe6453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-440fe870-mltxk" Oct 31 05:37:07.017456 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:07.017413 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56505ab3-8320-474e-84a0-a8470cbe6453-maas-api-xks-ca\") pod \"maas-api-revoke-sub-440fe870-mltxk\" (UID: \"56505ab3-8320-474e-84a0-a8470cbe6453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-440fe870-mltxk" Oct 31 05:37:07.038865 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:07.038830 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87pvg\" (UniqueName: \"kubernetes.io/projected/56505ab3-8320-474e-84a0-a8470cbe6453-kube-api-access-87pvg\") pod \"maas-api-revoke-sub-440fe870-mltxk\" (UID: \"56505ab3-8320-474e-84a0-a8470cbe6453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-440fe870-mltxk" Oct 31 05:37:07.212257 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:07.212161 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-440fe870-mltxk" Oct 31 05:37:07.513844 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:07.513811 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-440fe870-mltxk"] Oct 31 05:37:07.515610 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:37:07.515581 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56505ab3_8320_474e_84a0_a8470cbe6453.slice/crio-4c093d9dc0c7873af8efe7a3f627905583d86ef17fc471821568ee20e10931f7 WatchSource:0}: Error finding container 4c093d9dc0c7873af8efe7a3f627905583d86ef17fc471821568ee20e10931f7: Status 404 returned error can't find the container with id 4c093d9dc0c7873af8efe7a3f627905583d86ef17fc471821568ee20e10931f7 Oct 31 05:37:07.785977 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:07.785939 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-440fe870-mltxk" event={"ID":"56505ab3-8320-474e-84a0-a8470cbe6453","Type":"ContainerStarted","Data":"4c093d9dc0c7873af8efe7a3f627905583d86ef17fc471821568ee20e10931f7"} Oct 31 05:37:08.793693 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:08.793600 5019 generic.go:356] "Generic (PLEG): container finished" podID="56505ab3-8320-474e-84a0-a8470cbe6453" containerID="ecaa99cf578d398cd717938083e8caec28ae1a96c1d28bf768410086d0a365ab" exitCode=0 Oct 31 05:37:08.793693 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:08.793642 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-440fe870-mltxk" event={"ID":"56505ab3-8320-474e-84a0-a8470cbe6453","Type":"ContainerDied","Data":"ecaa99cf578d398cd717938083e8caec28ae1a96c1d28bf768410086d0a365ab"} Oct 31 05:37:09.946193 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:09.946158 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-440fe870-mltxk" Oct 31 05:37:10.046668 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:10.046612 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56505ab3-8320-474e-84a0-a8470cbe6453-kube-api-access-87pvg\" (UniqueName: \"kubernetes.io/projected/56505ab3-8320-474e-84a0-a8470cbe6453-kube-api-access-87pvg\") pod \"56505ab3-8320-474e-84a0-a8470cbe6453\" (UID: \"56505ab3-8320-474e-84a0-a8470cbe6453\") " Oct 31 05:37:10.046877 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:10.046689 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56505ab3-8320-474e-84a0-a8470cbe6453-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56505ab3-8320-474e-84a0-a8470cbe6453-maas-api-openshift-ca\") pod \"56505ab3-8320-474e-84a0-a8470cbe6453\" (UID: \"56505ab3-8320-474e-84a0-a8470cbe6453\") " Oct 31 05:37:10.046877 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:10.046739 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56505ab3-8320-474e-84a0-a8470cbe6453-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56505ab3-8320-474e-84a0-a8470cbe6453-maas-api-xks-ca\") pod \"56505ab3-8320-474e-84a0-a8470cbe6453\" (UID: \"56505ab3-8320-474e-84a0-a8470cbe6453\") " Oct 31 05:37:10.047168 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:10.047142 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56505ab3-8320-474e-84a0-a8470cbe6453-maas-api-openshift-ca" pod "56505ab3-8320-474e-84a0-a8470cbe6453" (UID: "56505ab3-8320-474e-84a0-a8470cbe6453"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:37:10.049012 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:10.048990 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56505ab3-8320-474e-84a0-a8470cbe6453-maas-api-xks-ca" pod "56505ab3-8320-474e-84a0-a8470cbe6453" (UID: "56505ab3-8320-474e-84a0-a8470cbe6453"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:37:10.049115 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:10.049102 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56505ab3-8320-474e-84a0-a8470cbe6453-kube-api-access-87pvg" pod "56505ab3-8320-474e-84a0-a8470cbe6453" (UID: "56505ab3-8320-474e-84a0-a8470cbe6453"). InnerVolumeSpecName "kube-api-access-87pvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:37:10.147896 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:10.147847 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56505ab3-8320-474e-84a0-a8470cbe6453-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:37:10.147896 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:10.147888 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87pvg\" (UniqueName: \"kubernetes.io/projected/56505ab3-8320-474e-84a0-a8470cbe6453-kube-api-access-87pvg\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:37:10.147896 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:10.147901 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56505ab3-8320-474e-84a0-a8470cbe6453-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:37:10.806044 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:10.806005 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-440fe870-mltxk" event={"ID":"56505ab3-8320-474e-84a0-a8470cbe6453","Type":"ContainerDied","Data":"4c093d9dc0c7873af8efe7a3f627905583d86ef17fc471821568ee20e10931f7"} Oct 31 05:37:10.806240 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:10.806059 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c093d9dc0c7873af8efe7a3f627905583d86ef17fc471821568ee20e10931f7" Oct 31 05:37:10.806240 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:10.806060 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-440fe870-mltxk" Oct 31 05:37:15.239640 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:15.239604 5019 scope.go:122] "RemoveContainer" containerID="229f36cb60ccceaa1fad23f1d6d60dcbf4336b2c76b9634c6cbb8bc8694777e4" Oct 31 05:37:15.250047 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:15.250022 5019 scope.go:122] "RemoveContainer" containerID="cf0db329332f5f3e88338007a8a3400aed58eb0c818a31203cdb0b1667be5707" Oct 31 05:37:15.259737 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:15.259707 5019 scope.go:122] "RemoveContainer" containerID="4aef163ea5f17de4939e01ecae7d5dde9f2216043e6103078b011c4e64408d92" Oct 31 05:37:15.272395 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:15.272354 5019 scope.go:122] "RemoveContainer" containerID="18163c60e1a1b98c59fab71ebb16c666ccd20fbd710359ce16938097f92155e1" Oct 31 05:37:17.261798 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:17.261749 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e69dc44-26cqc"] Oct 31 05:37:17.262345 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:17.262316 5019 cpu_manager.go:414] "containerMap: removing container" podUID="56505ab3-8320-474e-84a0-a8470cbe6453" containerName="revoke-keys" Oct 31 05:37:17.262426 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:17.262348 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56505ab3-8320-474e-84a0-a8470cbe6453" containerName="revoke-keys" Oct 31 05:37:17.262470 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:17.262428 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="56505ab3-8320-474e-84a0-a8470cbe6453" containerName="revoke-keys" Oct 31 05:37:17.267425 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:17.267398 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e69dc44-26cqc" Oct 31 05:37:17.295391 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:17.293986 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:37:17.295391 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:17.294020 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:37:17.295391 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:17.294124 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:37:17.295391 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:17.294159 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfhzn\"" Oct 31 05:37:17.297181 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:17.297148 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e69dc44-26cqc"] Oct 31 05:37:17.330088 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:17.330045 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvf7f\" (UniqueName: \"kubernetes.io/projected/ddd66c15-bf68-4b42-a5f5-9666d6e3dc20-kube-api-access-dvf7f\") pod \"maas-api-revoke-sub-9e69dc44-26cqc\" (UID: \"ddd66c15-bf68-4b42-a5f5-9666d6e3dc20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e69dc44-26cqc" Oct 31 05:37:17.330282 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:17.330187 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddd66c15-bf68-4b42-a5f5-9666d6e3dc20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e69dc44-26cqc\" (UID: \"ddd66c15-bf68-4b42-a5f5-9666d6e3dc20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e69dc44-26cqc" Oct 31 05:37:17.330282 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:17.330220 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddd66c15-bf68-4b42-a5f5-9666d6e3dc20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e69dc44-26cqc\" (UID: \"ddd66c15-bf68-4b42-a5f5-9666d6e3dc20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e69dc44-26cqc" Oct 31 05:37:17.431823 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:17.431776 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvf7f\" (UniqueName: \"kubernetes.io/projected/ddd66c15-bf68-4b42-a5f5-9666d6e3dc20-kube-api-access-dvf7f\") pod \"maas-api-revoke-sub-9e69dc44-26cqc\" (UID: \"ddd66c15-bf68-4b42-a5f5-9666d6e3dc20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e69dc44-26cqc" Oct 31 05:37:17.432030 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:17.431869 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddd66c15-bf68-4b42-a5f5-9666d6e3dc20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e69dc44-26cqc\" (UID: \"ddd66c15-bf68-4b42-a5f5-9666d6e3dc20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e69dc44-26cqc" Oct 31 05:37:17.432030 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:17.431895 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddd66c15-bf68-4b42-a5f5-9666d6e3dc20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e69dc44-26cqc\" (UID: \"ddd66c15-bf68-4b42-a5f5-9666d6e3dc20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e69dc44-26cqc" Oct 31 05:37:17.432753 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:17.432718 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddd66c15-bf68-4b42-a5f5-9666d6e3dc20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e69dc44-26cqc\" (UID: \"ddd66c15-bf68-4b42-a5f5-9666d6e3dc20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e69dc44-26cqc" Oct 31 05:37:17.434779 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:17.434756 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddd66c15-bf68-4b42-a5f5-9666d6e3dc20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e69dc44-26cqc\" (UID: \"ddd66c15-bf68-4b42-a5f5-9666d6e3dc20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e69dc44-26cqc" Oct 31 05:37:17.455591 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:17.455557 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvf7f\" (UniqueName: \"kubernetes.io/projected/ddd66c15-bf68-4b42-a5f5-9666d6e3dc20-kube-api-access-dvf7f\") pod \"maas-api-revoke-sub-9e69dc44-26cqc\" (UID: \"ddd66c15-bf68-4b42-a5f5-9666d6e3dc20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e69dc44-26cqc" Oct 31 05:37:17.582648 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:17.582543 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e69dc44-26cqc" Oct 31 05:37:17.890965 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:17.890921 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e69dc44-26cqc"] Oct 31 05:37:17.891808 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:37:17.891764 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddd66c15_bf68_4b42_a5f5_9666d6e3dc20.slice/crio-b720b9d46d8e02efed45828840b1059375d462015cbff3f3b0e49ef37a33b79d WatchSource:0}: Error finding container b720b9d46d8e02efed45828840b1059375d462015cbff3f3b0e49ef37a33b79d: Status 404 returned error can't find the container with id b720b9d46d8e02efed45828840b1059375d462015cbff3f3b0e49ef37a33b79d Oct 31 05:37:18.857282 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:18.857236 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e69dc44-26cqc" event={"ID":"ddd66c15-bf68-4b42-a5f5-9666d6e3dc20","Type":"ContainerStarted","Data":"8092a3d2278763541bfb2abe6341b5be891eed74cc58248b21610ee4d7d1f19b"} Oct 31 05:37:18.857785 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:18.857312 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e69dc44-26cqc" event={"ID":"ddd66c15-bf68-4b42-a5f5-9666d6e3dc20","Type":"ContainerStarted","Data":"b720b9d46d8e02efed45828840b1059375d462015cbff3f3b0e49ef37a33b79d"} Oct 31 05:37:18.899724 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:18.899661 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e69dc44-26cqc" podStartSLOduration=1.899643632 podStartE2EDuration="1.899643632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:37:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:37:18.896634694 +0000 UTC m=+5593.652972734" watchObservedRunningTime="2026-10-31 05:37:18.899643632 +0000 UTC m=+5593.655981672" Oct 31 05:37:19.866459 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:19.866416 5019 generic.go:356] "Generic (PLEG): container finished" podID="ddd66c15-bf68-4b42-a5f5-9666d6e3dc20" containerID="8092a3d2278763541bfb2abe6341b5be891eed74cc58248b21610ee4d7d1f19b" exitCode=0 Oct 31 05:37:19.866932 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:19.866492 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e69dc44-26cqc" event={"ID":"ddd66c15-bf68-4b42-a5f5-9666d6e3dc20","Type":"ContainerDied","Data":"8092a3d2278763541bfb2abe6341b5be891eed74cc58248b21610ee4d7d1f19b"} Oct 31 05:37:21.020674 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:21.020643 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e69dc44-26cqc" Oct 31 05:37:21.172153 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:21.172046 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddd66c15-bf68-4b42-a5f5-9666d6e3dc20-kube-api-access-dvf7f\" (UniqueName: \"kubernetes.io/projected/ddd66c15-bf68-4b42-a5f5-9666d6e3dc20-kube-api-access-dvf7f\") pod \"ddd66c15-bf68-4b42-a5f5-9666d6e3dc20\" (UID: \"ddd66c15-bf68-4b42-a5f5-9666d6e3dc20\") " Oct 31 05:37:21.172153 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:21.172100 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddd66c15-bf68-4b42-a5f5-9666d6e3dc20-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddd66c15-bf68-4b42-a5f5-9666d6e3dc20-maas-api-xks-ca\") pod \"ddd66c15-bf68-4b42-a5f5-9666d6e3dc20\" (UID: \"ddd66c15-bf68-4b42-a5f5-9666d6e3dc20\") " Oct 31 05:37:21.172153 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:21.172124 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddd66c15-bf68-4b42-a5f5-9666d6e3dc20-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddd66c15-bf68-4b42-a5f5-9666d6e3dc20-maas-api-openshift-ca\") pod \"ddd66c15-bf68-4b42-a5f5-9666d6e3dc20\" (UID: \"ddd66c15-bf68-4b42-a5f5-9666d6e3dc20\") " Oct 31 05:37:21.172600 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:21.172575 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddd66c15-bf68-4b42-a5f5-9666d6e3dc20-maas-api-openshift-ca" pod "ddd66c15-bf68-4b42-a5f5-9666d6e3dc20" (UID: "ddd66c15-bf68-4b42-a5f5-9666d6e3dc20"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:37:21.174426 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:21.174396 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddd66c15-bf68-4b42-a5f5-9666d6e3dc20-maas-api-xks-ca" pod "ddd66c15-bf68-4b42-a5f5-9666d6e3dc20" (UID: "ddd66c15-bf68-4b42-a5f5-9666d6e3dc20"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:37:21.174507 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:21.174436 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddd66c15-bf68-4b42-a5f5-9666d6e3dc20-kube-api-access-dvf7f" pod "ddd66c15-bf68-4b42-a5f5-9666d6e3dc20" (UID: "ddd66c15-bf68-4b42-a5f5-9666d6e3dc20"). InnerVolumeSpecName "kube-api-access-dvf7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:37:21.273630 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:21.273581 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvf7f\" (UniqueName: \"kubernetes.io/projected/ddd66c15-bf68-4b42-a5f5-9666d6e3dc20-kube-api-access-dvf7f\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:37:21.273630 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:21.273621 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddd66c15-bf68-4b42-a5f5-9666d6e3dc20-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:37:21.273630 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:21.273631 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddd66c15-bf68-4b42-a5f5-9666d6e3dc20-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:37:21.879654 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:21.879621 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e69dc44-26cqc" Oct 31 05:37:21.879852 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:21.879609 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e69dc44-26cqc" event={"ID":"ddd66c15-bf68-4b42-a5f5-9666d6e3dc20","Type":"ContainerDied","Data":"b720b9d46d8e02efed45828840b1059375d462015cbff3f3b0e49ef37a33b79d"} Oct 31 05:37:21.879852 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:21.879782 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b720b9d46d8e02efed45828840b1059375d462015cbff3f3b0e49ef37a33b79d" Oct 31 05:37:22.392426 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:37:22.392386 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:37:39.246932 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:39.246893 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e0d2d60-vkgxl"] Oct 31 05:37:39.247505 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:39.247391 5019 cpu_manager.go:414] "containerMap: removing container" podUID="ddd66c15-bf68-4b42-a5f5-9666d6e3dc20" containerName="revoke-keys" Oct 31 05:37:39.247505 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:39.247405 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddd66c15-bf68-4b42-a5f5-9666d6e3dc20" containerName="revoke-keys" Oct 31 05:37:39.247613 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:39.247525 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddd66c15-bf68-4b42-a5f5-9666d6e3dc20" containerName="revoke-keys" Oct 31 05:37:39.250999 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:39.250978 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e0d2d60-vkgxl" Oct 31 05:37:39.270273 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:39.270233 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e0d2d60-vkgxl"] Oct 31 05:37:39.278962 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:39.278925 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:37:39.279213 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:39.278927 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:37:39.279325 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:39.279019 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfhzn\"" Oct 31 05:37:39.279445 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:39.279040 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:37:39.326667 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:39.326629 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d30a1985-72b2-4162-a797-5b55a2272fd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e0d2d60-vkgxl\" (UID: \"d30a1985-72b2-4162-a797-5b55a2272fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e0d2d60-vkgxl" Oct 31 05:37:39.326880 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:39.326687 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c8n5\" (UniqueName: \"kubernetes.io/projected/d30a1985-72b2-4162-a797-5b55a2272fd9-kube-api-access-4c8n5\") pod \"maas-api-revoke-sub-1e0d2d60-vkgxl\" (UID: \"d30a1985-72b2-4162-a797-5b55a2272fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e0d2d60-vkgxl" Oct 31 05:37:39.326880 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:39.326781 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d30a1985-72b2-4162-a797-5b55a2272fd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e0d2d60-vkgxl\" (UID: \"d30a1985-72b2-4162-a797-5b55a2272fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e0d2d60-vkgxl" Oct 31 05:37:39.428298 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:39.428249 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d30a1985-72b2-4162-a797-5b55a2272fd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e0d2d60-vkgxl\" (UID: \"d30a1985-72b2-4162-a797-5b55a2272fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e0d2d60-vkgxl" Oct 31 05:37:39.428527 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:39.428312 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c8n5\" (UniqueName: \"kubernetes.io/projected/d30a1985-72b2-4162-a797-5b55a2272fd9-kube-api-access-4c8n5\") pod \"maas-api-revoke-sub-1e0d2d60-vkgxl\" (UID: \"d30a1985-72b2-4162-a797-5b55a2272fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e0d2d60-vkgxl" Oct 31 05:37:39.428527 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:39.428374 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d30a1985-72b2-4162-a797-5b55a2272fd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e0d2d60-vkgxl\" (UID: \"d30a1985-72b2-4162-a797-5b55a2272fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e0d2d60-vkgxl" Oct 31 05:37:39.429049 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:39.429023 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d30a1985-72b2-4162-a797-5b55a2272fd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e0d2d60-vkgxl\" (UID: \"d30a1985-72b2-4162-a797-5b55a2272fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e0d2d60-vkgxl" Oct 31 05:37:39.430846 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:39.430824 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d30a1985-72b2-4162-a797-5b55a2272fd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e0d2d60-vkgxl\" (UID: \"d30a1985-72b2-4162-a797-5b55a2272fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e0d2d60-vkgxl" Oct 31 05:37:39.452148 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:39.452109 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c8n5\" (UniqueName: \"kubernetes.io/projected/d30a1985-72b2-4162-a797-5b55a2272fd9-kube-api-access-4c8n5\") pod \"maas-api-revoke-sub-1e0d2d60-vkgxl\" (UID: \"d30a1985-72b2-4162-a797-5b55a2272fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e0d2d60-vkgxl" Oct 31 05:37:39.565839 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:39.565739 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e0d2d60-vkgxl" Oct 31 05:37:39.865627 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:39.865593 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e0d2d60-vkgxl"] Oct 31 05:37:39.867213 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:37:39.867179 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd30a1985_72b2_4162_a797_5b55a2272fd9.slice/crio-a898d6fe09d0df42bd541669f8686363eaff635de8e59d7444bba451a5888ef7 WatchSource:0}: Error finding container a898d6fe09d0df42bd541669f8686363eaff635de8e59d7444bba451a5888ef7: Status 404 returned error can't find the container with id a898d6fe09d0df42bd541669f8686363eaff635de8e59d7444bba451a5888ef7 Oct 31 05:37:40.010066 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:40.010026 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e0d2d60-vkgxl" event={"ID":"d30a1985-72b2-4162-a797-5b55a2272fd9","Type":"ContainerStarted","Data":"a898d6fe09d0df42bd541669f8686363eaff635de8e59d7444bba451a5888ef7"} Oct 31 05:37:41.017011 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:41.016973 5019 generic.go:356] "Generic (PLEG): container finished" podID="d30a1985-72b2-4162-a797-5b55a2272fd9" containerID="501e23e408d3038797f14801a5c9533fd58704d0ceb3de3073ed3daa57f1e849" exitCode=0 Oct 31 05:37:41.017469 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:41.017057 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e0d2d60-vkgxl" event={"ID":"d30a1985-72b2-4162-a797-5b55a2272fd9","Type":"ContainerDied","Data":"501e23e408d3038797f14801a5c9533fd58704d0ceb3de3073ed3daa57f1e849"} Oct 31 05:37:42.196880 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:42.196849 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e0d2d60-vkgxl" Oct 31 05:37:42.258702 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:42.258652 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d30a1985-72b2-4162-a797-5b55a2272fd9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d30a1985-72b2-4162-a797-5b55a2272fd9-maas-api-openshift-ca\") pod \"d30a1985-72b2-4162-a797-5b55a2272fd9\" (UID: \"d30a1985-72b2-4162-a797-5b55a2272fd9\") " Oct 31 05:37:42.258920 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:42.258749 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d30a1985-72b2-4162-a797-5b55a2272fd9-kube-api-access-4c8n5\" (UniqueName: \"kubernetes.io/projected/d30a1985-72b2-4162-a797-5b55a2272fd9-kube-api-access-4c8n5\") pod \"d30a1985-72b2-4162-a797-5b55a2272fd9\" (UID: \"d30a1985-72b2-4162-a797-5b55a2272fd9\") " Oct 31 05:37:42.258920 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:42.258793 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d30a1985-72b2-4162-a797-5b55a2272fd9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d30a1985-72b2-4162-a797-5b55a2272fd9-maas-api-xks-ca\") pod \"d30a1985-72b2-4162-a797-5b55a2272fd9\" (UID: \"d30a1985-72b2-4162-a797-5b55a2272fd9\") " Oct 31 05:37:42.259096 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:42.259069 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d30a1985-72b2-4162-a797-5b55a2272fd9-maas-api-openshift-ca" pod "d30a1985-72b2-4162-a797-5b55a2272fd9" (UID: "d30a1985-72b2-4162-a797-5b55a2272fd9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:37:42.259315 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:42.259291 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d30a1985-72b2-4162-a797-5b55a2272fd9-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:37:42.261651 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:42.261622 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d30a1985-72b2-4162-a797-5b55a2272fd9-maas-api-xks-ca" pod "d30a1985-72b2-4162-a797-5b55a2272fd9" (UID: "d30a1985-72b2-4162-a797-5b55a2272fd9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:37:42.261764 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:42.261659 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d30a1985-72b2-4162-a797-5b55a2272fd9-kube-api-access-4c8n5" pod "d30a1985-72b2-4162-a797-5b55a2272fd9" (UID: "d30a1985-72b2-4162-a797-5b55a2272fd9"). InnerVolumeSpecName "kube-api-access-4c8n5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:37:42.360372 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:42.360225 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c8n5\" (UniqueName: \"kubernetes.io/projected/d30a1985-72b2-4162-a797-5b55a2272fd9-kube-api-access-4c8n5\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:37:42.360372 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:42.360268 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d30a1985-72b2-4162-a797-5b55a2272fd9-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:37:43.032910 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:43.032871 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e0d2d60-vkgxl" Oct 31 05:37:43.033113 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:43.032901 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e0d2d60-vkgxl" event={"ID":"d30a1985-72b2-4162-a797-5b55a2272fd9","Type":"ContainerDied","Data":"a898d6fe09d0df42bd541669f8686363eaff635de8e59d7444bba451a5888ef7"} Oct 31 05:37:43.033113 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:37:43.032960 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a898d6fe09d0df42bd541669f8686363eaff635de8e59d7444bba451a5888ef7" Oct 31 05:37:52.392566 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:37:52.392515 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:38:01.411357 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:01.410448 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-rgj85 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.218:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 05:38:01.411357 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:01.410532 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rgj85" podUID="8430b919-a9dd-4c7e-9266-ef6872cc0b9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.218:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 05:38:01.480555 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:01.480469 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 05:38:01.480968 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:01.480926 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r" podUID="68e465ba-2b5d-4e3c-8b63-7bdb9b695b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 05:38:01.535001 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:01.534462 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-shld4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 05:38:01.535001 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:01.534543 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-shld4" podUID="a69fd138-9906-4127-ac26-38a21a5b7590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 05:38:01.622502 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:01.622447 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.221:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 05:38:01.622839 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:01.622789 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5" podUID="6d68d19b-5392-4308-9e2c-4b5e7593d17b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.221:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 05:38:10.125841 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:10.125789 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d08cea9-2d75z"] Oct 31 05:38:10.126348 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:10.126320 5019 cpu_manager.go:414] "containerMap: removing container" podUID="d30a1985-72b2-4162-a797-5b55a2272fd9" containerName="revoke-keys" Oct 31 05:38:10.126348 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:10.126348 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d30a1985-72b2-4162-a797-5b55a2272fd9" containerName="revoke-keys" Oct 31 05:38:10.126462 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:10.126423 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="d30a1985-72b2-4162-a797-5b55a2272fd9" containerName="revoke-keys" Oct 31 05:38:10.133973 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:10.133929 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d08cea9-2d75z" Oct 31 05:38:10.149364 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:10.149308 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d08cea9-2d75z"] Oct 31 05:38:10.157654 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:10.157621 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:38:10.157654 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:10.157621 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:38:10.157914 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:10.157663 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:38:10.160794 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:10.160761 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfhzn\"" Oct 31 05:38:10.190055 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:10.189995 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd5ff\" (UniqueName: \"kubernetes.io/projected/9de5e560-922e-4c50-9150-c07c09feb42b-kube-api-access-jd5ff\") pod \"maas-api-revoke-sub-3d08cea9-2d75z\" (UID: \"9de5e560-922e-4c50-9150-c07c09feb42b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d08cea9-2d75z" Oct 31 05:38:10.190276 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:10.190095 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9de5e560-922e-4c50-9150-c07c09feb42b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d08cea9-2d75z\" (UID: \"9de5e560-922e-4c50-9150-c07c09feb42b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d08cea9-2d75z" Oct 31 05:38:10.190276 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:10.190219 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9de5e560-922e-4c50-9150-c07c09feb42b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d08cea9-2d75z\" (UID: \"9de5e560-922e-4c50-9150-c07c09feb42b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d08cea9-2d75z" Oct 31 05:38:10.291883 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:10.291839 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd5ff\" (UniqueName: \"kubernetes.io/projected/9de5e560-922e-4c50-9150-c07c09feb42b-kube-api-access-jd5ff\") pod \"maas-api-revoke-sub-3d08cea9-2d75z\" (UID: \"9de5e560-922e-4c50-9150-c07c09feb42b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d08cea9-2d75z" Oct 31 05:38:10.292110 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:10.291923 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9de5e560-922e-4c50-9150-c07c09feb42b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d08cea9-2d75z\" (UID: \"9de5e560-922e-4c50-9150-c07c09feb42b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d08cea9-2d75z" Oct 31 05:38:10.292110 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:10.291984 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9de5e560-922e-4c50-9150-c07c09feb42b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d08cea9-2d75z\" (UID: \"9de5e560-922e-4c50-9150-c07c09feb42b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d08cea9-2d75z" Oct 31 05:38:10.292718 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:10.292680 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9de5e560-922e-4c50-9150-c07c09feb42b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d08cea9-2d75z\" (UID: \"9de5e560-922e-4c50-9150-c07c09feb42b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d08cea9-2d75z" Oct 31 05:38:10.294575 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:10.294556 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9de5e560-922e-4c50-9150-c07c09feb42b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d08cea9-2d75z\" (UID: \"9de5e560-922e-4c50-9150-c07c09feb42b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d08cea9-2d75z" Oct 31 05:38:10.316250 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:10.316216 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd5ff\" (UniqueName: \"kubernetes.io/projected/9de5e560-922e-4c50-9150-c07c09feb42b-kube-api-access-jd5ff\") pod \"maas-api-revoke-sub-3d08cea9-2d75z\" (UID: \"9de5e560-922e-4c50-9150-c07c09feb42b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d08cea9-2d75z" Oct 31 05:38:10.449209 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:10.449120 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d08cea9-2d75z" Oct 31 05:38:10.751389 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:10.751360 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d08cea9-2d75z"] Oct 31 05:38:10.753203 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:38:10.753171 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9de5e560_922e_4c50_9150_c07c09feb42b.slice/crio-fef58b52942020aa1275ac60efb716f30b19227246c22bbb5df1283c34cbb5f3 WatchSource:0}: Error finding container fef58b52942020aa1275ac60efb716f30b19227246c22bbb5df1283c34cbb5f3: Status 404 returned error can't find the container with id fef58b52942020aa1275ac60efb716f30b19227246c22bbb5df1283c34cbb5f3 Oct 31 05:38:11.464649 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:11.464544 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d08cea9-2d75z" event={"ID":"9de5e560-922e-4c50-9150-c07c09feb42b","Type":"ContainerStarted","Data":"353b1ec6c7bc9595370ff8b713c27ec04d85f8aa089b9f351b3b16f2bee019b6"} Oct 31 05:38:11.464649 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:11.464602 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d08cea9-2d75z" event={"ID":"9de5e560-922e-4c50-9150-c07c09feb42b","Type":"ContainerStarted","Data":"fef58b52942020aa1275ac60efb716f30b19227246c22bbb5df1283c34cbb5f3"} Oct 31 05:38:11.507768 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:11.507702 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d08cea9-2d75z" podStartSLOduration=1.507686891 podStartE2EDuration="1.507686891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:38:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:38:11.504654328 +0000 UTC m=+5646.260992367" watchObservedRunningTime="2026-10-31 05:38:11.507686891 +0000 UTC m=+5646.264024931" Oct 31 05:38:12.471824 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:12.471785 5019 generic.go:356] "Generic (PLEG): container finished" podID="9de5e560-922e-4c50-9150-c07c09feb42b" containerID="353b1ec6c7bc9595370ff8b713c27ec04d85f8aa089b9f351b3b16f2bee019b6" exitCode=0 Oct 31 05:38:12.472283 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:12.471868 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d08cea9-2d75z" event={"ID":"9de5e560-922e-4c50-9150-c07c09feb42b","Type":"ContainerDied","Data":"353b1ec6c7bc9595370ff8b713c27ec04d85f8aa089b9f351b3b16f2bee019b6"} Oct 31 05:38:13.638803 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:13.638770 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d08cea9-2d75z" Oct 31 05:38:13.730369 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:13.730225 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9de5e560-922e-4c50-9150-c07c09feb42b-kube-api-access-jd5ff\" (UniqueName: \"kubernetes.io/projected/9de5e560-922e-4c50-9150-c07c09feb42b-kube-api-access-jd5ff\") pod \"9de5e560-922e-4c50-9150-c07c09feb42b\" (UID: \"9de5e560-922e-4c50-9150-c07c09feb42b\") " Oct 31 05:38:13.730369 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:13.730322 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9de5e560-922e-4c50-9150-c07c09feb42b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9de5e560-922e-4c50-9150-c07c09feb42b-maas-api-openshift-ca\") pod \"9de5e560-922e-4c50-9150-c07c09feb42b\" (UID: \"9de5e560-922e-4c50-9150-c07c09feb42b\") " Oct 31 05:38:13.730643 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:13.730441 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9de5e560-922e-4c50-9150-c07c09feb42b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9de5e560-922e-4c50-9150-c07c09feb42b-maas-api-xks-ca\") pod \"9de5e560-922e-4c50-9150-c07c09feb42b\" (UID: \"9de5e560-922e-4c50-9150-c07c09feb42b\") " Oct 31 05:38:13.730777 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:13.730749 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9de5e560-922e-4c50-9150-c07c09feb42b-maas-api-openshift-ca" pod "9de5e560-922e-4c50-9150-c07c09feb42b" (UID: "9de5e560-922e-4c50-9150-c07c09feb42b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:38:13.732756 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:13.732723 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9de5e560-922e-4c50-9150-c07c09feb42b-maas-api-xks-ca" pod "9de5e560-922e-4c50-9150-c07c09feb42b" (UID: "9de5e560-922e-4c50-9150-c07c09feb42b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:38:13.732882 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:13.732768 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9de5e560-922e-4c50-9150-c07c09feb42b-kube-api-access-jd5ff" pod "9de5e560-922e-4c50-9150-c07c09feb42b" (UID: "9de5e560-922e-4c50-9150-c07c09feb42b"). InnerVolumeSpecName "kube-api-access-jd5ff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:38:13.831220 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:13.831170 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9de5e560-922e-4c50-9150-c07c09feb42b-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:38:13.831220 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:13.831210 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jd5ff\" (UniqueName: \"kubernetes.io/projected/9de5e560-922e-4c50-9150-c07c09feb42b-kube-api-access-jd5ff\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:38:13.831220 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:13.831220 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9de5e560-922e-4c50-9150-c07c09feb42b-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:38:14.485746 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:14.485709 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d08cea9-2d75z" event={"ID":"9de5e560-922e-4c50-9150-c07c09feb42b","Type":"ContainerDied","Data":"fef58b52942020aa1275ac60efb716f30b19227246c22bbb5df1283c34cbb5f3"} Oct 31 05:38:14.485952 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:14.485752 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d08cea9-2d75z" Oct 31 05:38:14.485952 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:14.485768 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fef58b52942020aa1275ac60efb716f30b19227246c22bbb5df1283c34cbb5f3" Oct 31 05:38:15.321956 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:15.321919 5019 scope.go:122] "RemoveContainer" containerID="26ff4e4976d329df1ccba34cdbdbf8d15fee4f013c3cf6816aa560d85969078f" Oct 31 05:38:15.331448 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:15.331424 5019 scope.go:122] "RemoveContainer" containerID="cc2ee2fe2c7564e494fe48bd9217f29537653dcdd71aeed459749f4d24749e03" Oct 31 05:38:15.340762 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:15.340737 5019 scope.go:122] "RemoveContainer" containerID="b210a73a5b2ce68e28f1322c6bb1c861aee6b766a656f44aad9f034830e7f7ff" Oct 31 05:38:15.350615 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:15.350580 5019 scope.go:122] "RemoveContainer" containerID="c3755d69419d503ef6678b7fc8acf9117f40a7937ac73acf36ba4255fb390272" Oct 31 05:38:20.613032 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:20.612992 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cd035ac-vlcq6"] Oct 31 05:38:20.613550 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:20.613533 5019 cpu_manager.go:414] "containerMap: removing container" podUID="9de5e560-922e-4c50-9150-c07c09feb42b" containerName="revoke-keys" Oct 31 05:38:20.613606 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:20.613552 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9de5e560-922e-4c50-9150-c07c09feb42b" containerName="revoke-keys" Oct 31 05:38:20.613660 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:20.613648 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="9de5e560-922e-4c50-9150-c07c09feb42b" containerName="revoke-keys" Oct 31 05:38:20.618878 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:20.618848 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cd035ac-vlcq6" Oct 31 05:38:20.634548 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:20.634512 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:38:20.636493 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:20.636451 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cd035ac-vlcq6"] Oct 31 05:38:20.645267 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:20.645216 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfhzn\"" Oct 31 05:38:20.645267 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:20.645252 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:38:20.647298 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:20.647266 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:38:20.701907 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:20.701863 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9483d81c-c122-4194-a1ec-016ab7fdfbf7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cd035ac-vlcq6\" (UID: \"9483d81c-c122-4194-a1ec-016ab7fdfbf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cd035ac-vlcq6" Oct 31 05:38:20.702128 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:20.702028 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfk8d\" (UniqueName: \"kubernetes.io/projected/9483d81c-c122-4194-a1ec-016ab7fdfbf7-kube-api-access-nfk8d\") pod \"maas-api-revoke-sub-6cd035ac-vlcq6\" (UID: \"9483d81c-c122-4194-a1ec-016ab7fdfbf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cd035ac-vlcq6" Oct 31 05:38:20.702128 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:20.702065 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9483d81c-c122-4194-a1ec-016ab7fdfbf7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cd035ac-vlcq6\" (UID: \"9483d81c-c122-4194-a1ec-016ab7fdfbf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cd035ac-vlcq6" Oct 31 05:38:20.803635 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:20.803574 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfk8d\" (UniqueName: \"kubernetes.io/projected/9483d81c-c122-4194-a1ec-016ab7fdfbf7-kube-api-access-nfk8d\") pod \"maas-api-revoke-sub-6cd035ac-vlcq6\" (UID: \"9483d81c-c122-4194-a1ec-016ab7fdfbf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cd035ac-vlcq6" Oct 31 05:38:20.803635 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:20.803638 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9483d81c-c122-4194-a1ec-016ab7fdfbf7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cd035ac-vlcq6\" (UID: \"9483d81c-c122-4194-a1ec-016ab7fdfbf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cd035ac-vlcq6" Oct 31 05:38:20.803952 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:20.803741 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9483d81c-c122-4194-a1ec-016ab7fdfbf7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cd035ac-vlcq6\" (UID: \"9483d81c-c122-4194-a1ec-016ab7fdfbf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cd035ac-vlcq6" Oct 31 05:38:20.804501 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:20.804474 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9483d81c-c122-4194-a1ec-016ab7fdfbf7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cd035ac-vlcq6\" (UID: \"9483d81c-c122-4194-a1ec-016ab7fdfbf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cd035ac-vlcq6" Oct 31 05:38:20.806343 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:20.806300 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9483d81c-c122-4194-a1ec-016ab7fdfbf7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cd035ac-vlcq6\" (UID: \"9483d81c-c122-4194-a1ec-016ab7fdfbf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cd035ac-vlcq6" Oct 31 05:38:20.829069 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:20.829028 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfk8d\" (UniqueName: \"kubernetes.io/projected/9483d81c-c122-4194-a1ec-016ab7fdfbf7-kube-api-access-nfk8d\") pod \"maas-api-revoke-sub-6cd035ac-vlcq6\" (UID: \"9483d81c-c122-4194-a1ec-016ab7fdfbf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cd035ac-vlcq6" Oct 31 05:38:20.940464 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:20.940360 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cd035ac-vlcq6" Oct 31 05:38:21.243829 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:21.243707 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cd035ac-vlcq6"] Oct 31 05:38:21.248688 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:38:21.248648 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9483d81c_c122_4194_a1ec_016ab7fdfbf7.slice/crio-bcab0e02e9d41b70db3ef8a9a134c3939773af8b538dd9563bb0761fa8297b55 WatchSource:0}: Error finding container bcab0e02e9d41b70db3ef8a9a134c3939773af8b538dd9563bb0761fa8297b55: Status 404 returned error can't find the container with id bcab0e02e9d41b70db3ef8a9a134c3939773af8b538dd9563bb0761fa8297b55 Oct 31 05:38:21.528426 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:21.528312 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cd035ac-vlcq6" event={"ID":"9483d81c-c122-4194-a1ec-016ab7fdfbf7","Type":"ContainerStarted","Data":"bcab0e02e9d41b70db3ef8a9a134c3939773af8b538dd9563bb0761fa8297b55"} Oct 31 05:38:22.389913 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:38:22.389868 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:38:22.536522 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:22.536485 5019 generic.go:356] "Generic (PLEG): container finished" podID="9483d81c-c122-4194-a1ec-016ab7fdfbf7" containerID="256b519f8c75b13dd276d53bda879272028cd1fbc3cd8f6a8460224d9f921fe9" exitCode=0 Oct 31 05:38:22.536772 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:22.536570 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cd035ac-vlcq6" event={"ID":"9483d81c-c122-4194-a1ec-016ab7fdfbf7","Type":"ContainerDied","Data":"256b519f8c75b13dd276d53bda879272028cd1fbc3cd8f6a8460224d9f921fe9"} Oct 31 05:38:23.689398 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:23.689368 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cd035ac-vlcq6" Oct 31 05:38:23.836649 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:23.836542 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9483d81c-c122-4194-a1ec-016ab7fdfbf7-kube-api-access-nfk8d\" (UniqueName: \"kubernetes.io/projected/9483d81c-c122-4194-a1ec-016ab7fdfbf7-kube-api-access-nfk8d\") pod \"9483d81c-c122-4194-a1ec-016ab7fdfbf7\" (UID: \"9483d81c-c122-4194-a1ec-016ab7fdfbf7\") " Oct 31 05:38:23.836859 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:23.836680 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9483d81c-c122-4194-a1ec-016ab7fdfbf7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9483d81c-c122-4194-a1ec-016ab7fdfbf7-maas-api-openshift-ca\") pod \"9483d81c-c122-4194-a1ec-016ab7fdfbf7\" (UID: \"9483d81c-c122-4194-a1ec-016ab7fdfbf7\") " Oct 31 05:38:23.836859 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:23.836821 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9483d81c-c122-4194-a1ec-016ab7fdfbf7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9483d81c-c122-4194-a1ec-016ab7fdfbf7-maas-api-xks-ca\") pod \"9483d81c-c122-4194-a1ec-016ab7fdfbf7\" (UID: \"9483d81c-c122-4194-a1ec-016ab7fdfbf7\") " Oct 31 05:38:23.837066 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:23.837041 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9483d81c-c122-4194-a1ec-016ab7fdfbf7-maas-api-openshift-ca" pod "9483d81c-c122-4194-a1ec-016ab7fdfbf7" (UID: "9483d81c-c122-4194-a1ec-016ab7fdfbf7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:38:23.837184 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:23.837169 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9483d81c-c122-4194-a1ec-016ab7fdfbf7-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:38:23.838903 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:23.838876 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9483d81c-c122-4194-a1ec-016ab7fdfbf7-maas-api-xks-ca" pod "9483d81c-c122-4194-a1ec-016ab7fdfbf7" (UID: "9483d81c-c122-4194-a1ec-016ab7fdfbf7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:38:23.839037 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:23.839007 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9483d81c-c122-4194-a1ec-016ab7fdfbf7-kube-api-access-nfk8d" pod "9483d81c-c122-4194-a1ec-016ab7fdfbf7" (UID: "9483d81c-c122-4194-a1ec-016ab7fdfbf7"). InnerVolumeSpecName "kube-api-access-nfk8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:38:23.938118 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:23.938077 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9483d81c-c122-4194-a1ec-016ab7fdfbf7-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:38:23.938118 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:23.938112 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfk8d\" (UniqueName: \"kubernetes.io/projected/9483d81c-c122-4194-a1ec-016ab7fdfbf7-kube-api-access-nfk8d\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:38:24.550022 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:24.549983 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cd035ac-vlcq6" event={"ID":"9483d81c-c122-4194-a1ec-016ab7fdfbf7","Type":"ContainerDied","Data":"bcab0e02e9d41b70db3ef8a9a134c3939773af8b538dd9563bb0761fa8297b55"} Oct 31 05:38:24.550022 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:24.550005 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cd035ac-vlcq6" Oct 31 05:38:24.550324 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:24.550038 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcab0e02e9d41b70db3ef8a9a134c3939773af8b538dd9563bb0761fa8297b55" Oct 31 05:38:25.029469 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:25.029423 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51342272-dlmwk"] Oct 31 05:38:25.032589 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:25.032551 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51342272-dlmwk"] Oct 31 05:38:25.406669 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:25.406581 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b028b40-5fa7-4e54-aab6-af205d402646" path="/var/lib/kubelet/pods/9b028b40-5fa7-4e54-aab6-af205d402646/volumes" Oct 31 05:38:52.390995 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:38:52.390946 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:38:52.515097 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:52.515049 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6212d0dc-qkq4g"] Oct 31 05:38:52.515668 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:52.515646 5019 cpu_manager.go:414] "containerMap: removing container" podUID="9483d81c-c122-4194-a1ec-016ab7fdfbf7" containerName="revoke-keys" Oct 31 05:38:52.515668 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:52.515668 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9483d81c-c122-4194-a1ec-016ab7fdfbf7" containerName="revoke-keys" Oct 31 05:38:52.515864 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:52.515830 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="9483d81c-c122-4194-a1ec-016ab7fdfbf7" containerName="revoke-keys" Oct 31 05:38:52.520449 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:52.520419 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6212d0dc-qkq4g" Oct 31 05:38:52.538097 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:52.538060 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6212d0dc-qkq4g"] Oct 31 05:38:52.547016 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:52.546977 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:38:52.547016 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:52.547014 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:38:52.547316 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:52.546990 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:38:52.547316 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:52.547068 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfhzn\"" Oct 31 05:38:52.628348 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:52.628296 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d6b3a3f-67cf-4dc2-9489-f3e134b76560-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6212d0dc-qkq4g\" (UID: \"9d6b3a3f-67cf-4dc2-9489-f3e134b76560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6212d0dc-qkq4g" Oct 31 05:38:52.628591 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:52.628557 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc7wr\" (UniqueName: \"kubernetes.io/projected/9d6b3a3f-67cf-4dc2-9489-f3e134b76560-kube-api-access-nc7wr\") pod \"maas-api-revoke-sub-6212d0dc-qkq4g\" (UID: \"9d6b3a3f-67cf-4dc2-9489-f3e134b76560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6212d0dc-qkq4g" Oct 31 05:38:52.628682 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:52.628665 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d6b3a3f-67cf-4dc2-9489-f3e134b76560-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6212d0dc-qkq4g\" (UID: \"9d6b3a3f-67cf-4dc2-9489-f3e134b76560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6212d0dc-qkq4g" Oct 31 05:38:52.729410 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:52.729328 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc7wr\" (UniqueName: \"kubernetes.io/projected/9d6b3a3f-67cf-4dc2-9489-f3e134b76560-kube-api-access-nc7wr\") pod \"maas-api-revoke-sub-6212d0dc-qkq4g\" (UID: \"9d6b3a3f-67cf-4dc2-9489-f3e134b76560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6212d0dc-qkq4g" Oct 31 05:38:52.729612 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:52.729449 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d6b3a3f-67cf-4dc2-9489-f3e134b76560-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6212d0dc-qkq4g\" (UID: \"9d6b3a3f-67cf-4dc2-9489-f3e134b76560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6212d0dc-qkq4g" Oct 31 05:38:52.729612 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:52.729491 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d6b3a3f-67cf-4dc2-9489-f3e134b76560-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6212d0dc-qkq4g\" (UID: \"9d6b3a3f-67cf-4dc2-9489-f3e134b76560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6212d0dc-qkq4g" Oct 31 05:38:52.730185 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:52.730148 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d6b3a3f-67cf-4dc2-9489-f3e134b76560-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6212d0dc-qkq4g\" (UID: \"9d6b3a3f-67cf-4dc2-9489-f3e134b76560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6212d0dc-qkq4g" Oct 31 05:38:52.732063 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:52.732033 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d6b3a3f-67cf-4dc2-9489-f3e134b76560-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6212d0dc-qkq4g\" (UID: \"9d6b3a3f-67cf-4dc2-9489-f3e134b76560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6212d0dc-qkq4g" Oct 31 05:38:52.753269 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:52.753227 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc7wr\" (UniqueName: \"kubernetes.io/projected/9d6b3a3f-67cf-4dc2-9489-f3e134b76560-kube-api-access-nc7wr\") pod \"maas-api-revoke-sub-6212d0dc-qkq4g\" (UID: \"9d6b3a3f-67cf-4dc2-9489-f3e134b76560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6212d0dc-qkq4g" Oct 31 05:38:52.834432 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:52.834384 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6212d0dc-qkq4g" Oct 31 05:38:53.135292 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:53.135260 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6212d0dc-qkq4g"] Oct 31 05:38:53.136649 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:38:53.136623 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d6b3a3f_67cf_4dc2_9489_f3e134b76560.slice/crio-2c279503d7a24e465ddd406cc2c7ef0c5dd92309cc3a5ef32d570b8a4fece092 WatchSource:0}: Error finding container 2c279503d7a24e465ddd406cc2c7ef0c5dd92309cc3a5ef32d570b8a4fece092: Status 404 returned error can't find the container with id 2c279503d7a24e465ddd406cc2c7ef0c5dd92309cc3a5ef32d570b8a4fece092 Oct 31 05:38:53.880595 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:53.880554 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6212d0dc-qkq4g" event={"ID":"9d6b3a3f-67cf-4dc2-9489-f3e134b76560","Type":"ContainerStarted","Data":"3cb751a9fa33f50f2ba24cdab338b6c6e0f58f043e5d025270a45c58a0463179"} Oct 31 05:38:53.881118 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:53.880615 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6212d0dc-qkq4g" event={"ID":"9d6b3a3f-67cf-4dc2-9489-f3e134b76560","Type":"ContainerStarted","Data":"2c279503d7a24e465ddd406cc2c7ef0c5dd92309cc3a5ef32d570b8a4fece092"} Oct 31 05:38:53.921859 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:53.921786 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6212d0dc-qkq4g" podStartSLOduration=1.9217148160000002 podStartE2EDuration="1.921714816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:38:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:38:53.920680772 +0000 UTC m=+5688.677018812" watchObservedRunningTime="2026-10-31 05:38:53.921714816 +0000 UTC m=+5688.678052856" Oct 31 05:38:54.887290 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:54.887254 5019 generic.go:356] "Generic (PLEG): container finished" podID="9d6b3a3f-67cf-4dc2-9489-f3e134b76560" containerID="3cb751a9fa33f50f2ba24cdab338b6c6e0f58f043e5d025270a45c58a0463179" exitCode=0 Oct 31 05:38:54.887733 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:54.887367 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6212d0dc-qkq4g" event={"ID":"9d6b3a3f-67cf-4dc2-9489-f3e134b76560","Type":"ContainerDied","Data":"3cb751a9fa33f50f2ba24cdab338b6c6e0f58f043e5d025270a45c58a0463179"} Oct 31 05:38:56.045931 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:56.045899 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6212d0dc-qkq4g" Oct 31 05:38:56.161462 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:56.161324 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d6b3a3f-67cf-4dc2-9489-f3e134b76560-kube-api-access-nc7wr\" (UniqueName: \"kubernetes.io/projected/9d6b3a3f-67cf-4dc2-9489-f3e134b76560-kube-api-access-nc7wr\") pod \"9d6b3a3f-67cf-4dc2-9489-f3e134b76560\" (UID: \"9d6b3a3f-67cf-4dc2-9489-f3e134b76560\") " Oct 31 05:38:56.161642 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:56.161486 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d6b3a3f-67cf-4dc2-9489-f3e134b76560-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d6b3a3f-67cf-4dc2-9489-f3e134b76560-maas-api-xks-ca\") pod \"9d6b3a3f-67cf-4dc2-9489-f3e134b76560\" (UID: \"9d6b3a3f-67cf-4dc2-9489-f3e134b76560\") " Oct 31 05:38:56.161642 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:56.161550 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d6b3a3f-67cf-4dc2-9489-f3e134b76560-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d6b3a3f-67cf-4dc2-9489-f3e134b76560-maas-api-openshift-ca\") pod \"9d6b3a3f-67cf-4dc2-9489-f3e134b76560\" (UID: \"9d6b3a3f-67cf-4dc2-9489-f3e134b76560\") " Oct 31 05:38:56.161979 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:56.161954 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d6b3a3f-67cf-4dc2-9489-f3e134b76560-maas-api-openshift-ca" pod "9d6b3a3f-67cf-4dc2-9489-f3e134b76560" (UID: "9d6b3a3f-67cf-4dc2-9489-f3e134b76560"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:38:56.163727 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:56.163701 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d6b3a3f-67cf-4dc2-9489-f3e134b76560-kube-api-access-nc7wr" pod "9d6b3a3f-67cf-4dc2-9489-f3e134b76560" (UID: "9d6b3a3f-67cf-4dc2-9489-f3e134b76560"). InnerVolumeSpecName "kube-api-access-nc7wr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:38:56.163727 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:56.163716 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d6b3a3f-67cf-4dc2-9489-f3e134b76560-maas-api-xks-ca" pod "9d6b3a3f-67cf-4dc2-9489-f3e134b76560" (UID: "9d6b3a3f-67cf-4dc2-9489-f3e134b76560"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:38:56.263247 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:56.263202 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d6b3a3f-67cf-4dc2-9489-f3e134b76560-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:38:56.263247 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:56.263240 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d6b3a3f-67cf-4dc2-9489-f3e134b76560-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:38:56.263247 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:56.263250 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nc7wr\" (UniqueName: \"kubernetes.io/projected/9d6b3a3f-67cf-4dc2-9489-f3e134b76560-kube-api-access-nc7wr\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:38:56.899883 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:56.899848 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6212d0dc-qkq4g" Oct 31 05:38:56.899883 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:56.899862 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6212d0dc-qkq4g" event={"ID":"9d6b3a3f-67cf-4dc2-9489-f3e134b76560","Type":"ContainerDied","Data":"2c279503d7a24e465ddd406cc2c7ef0c5dd92309cc3a5ef32d570b8a4fece092"} Oct 31 05:38:56.900132 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:38:56.899908 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c279503d7a24e465ddd406cc2c7ef0c5dd92309cc3a5ef32d570b8a4fece092" Oct 31 05:39:02.859387 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:02.859344 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c16ae582-kmwx6"] Oct 31 05:39:02.859921 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:02.859841 5019 cpu_manager.go:414] "containerMap: removing container" podUID="9d6b3a3f-67cf-4dc2-9489-f3e134b76560" containerName="revoke-keys" Oct 31 05:39:02.859921 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:02.859855 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d6b3a3f-67cf-4dc2-9489-f3e134b76560" containerName="revoke-keys" Oct 31 05:39:02.860007 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:02.859965 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d6b3a3f-67cf-4dc2-9489-f3e134b76560" containerName="revoke-keys" Oct 31 05:39:02.863634 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:02.863609 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c16ae582-kmwx6" Oct 31 05:39:02.879248 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:02.879193 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:39:02.879248 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:02.879209 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfhzn\"" Oct 31 05:39:02.893854 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:02.893817 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:39:02.894116 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:02.893943 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:39:02.895533 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:02.895500 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c16ae582-kmwx6"] Oct 31 05:39:02.933061 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:02.933026 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c89fa3eb-3b89-4047-a233-e428240273d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c16ae582-kmwx6\" (UID: \"c89fa3eb-3b89-4047-a233-e428240273d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c16ae582-kmwx6" Oct 31 05:39:02.933249 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:02.933094 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkhtb\" (UniqueName: \"kubernetes.io/projected/c89fa3eb-3b89-4047-a233-e428240273d7-kube-api-access-wkhtb\") pod \"maas-api-revoke-sub-c16ae582-kmwx6\" (UID: \"c89fa3eb-3b89-4047-a233-e428240273d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c16ae582-kmwx6" Oct 31 05:39:02.933249 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:02.933165 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c89fa3eb-3b89-4047-a233-e428240273d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c16ae582-kmwx6\" (UID: \"c89fa3eb-3b89-4047-a233-e428240273d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c16ae582-kmwx6" Oct 31 05:39:03.034462 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:03.034420 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c89fa3eb-3b89-4047-a233-e428240273d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c16ae582-kmwx6\" (UID: \"c89fa3eb-3b89-4047-a233-e428240273d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c16ae582-kmwx6" Oct 31 05:39:03.034658 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:03.034484 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkhtb\" (UniqueName: \"kubernetes.io/projected/c89fa3eb-3b89-4047-a233-e428240273d7-kube-api-access-wkhtb\") pod \"maas-api-revoke-sub-c16ae582-kmwx6\" (UID: \"c89fa3eb-3b89-4047-a233-e428240273d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c16ae582-kmwx6" Oct 31 05:39:03.034658 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:03.034515 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c89fa3eb-3b89-4047-a233-e428240273d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c16ae582-kmwx6\" (UID: \"c89fa3eb-3b89-4047-a233-e428240273d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c16ae582-kmwx6" Oct 31 05:39:03.035137 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:03.035111 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c89fa3eb-3b89-4047-a233-e428240273d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c16ae582-kmwx6\" (UID: \"c89fa3eb-3b89-4047-a233-e428240273d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c16ae582-kmwx6" Oct 31 05:39:03.037030 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:03.037008 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c89fa3eb-3b89-4047-a233-e428240273d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c16ae582-kmwx6\" (UID: \"c89fa3eb-3b89-4047-a233-e428240273d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c16ae582-kmwx6" Oct 31 05:39:03.058082 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:03.058039 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkhtb\" (UniqueName: \"kubernetes.io/projected/c89fa3eb-3b89-4047-a233-e428240273d7-kube-api-access-wkhtb\") pod \"maas-api-revoke-sub-c16ae582-kmwx6\" (UID: \"c89fa3eb-3b89-4047-a233-e428240273d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c16ae582-kmwx6" Oct 31 05:39:03.178275 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:03.178179 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c16ae582-kmwx6" Oct 31 05:39:03.485077 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:03.485041 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c16ae582-kmwx6"] Oct 31 05:39:03.486834 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:39:03.486806 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc89fa3eb_3b89_4047_a233_e428240273d7.slice/crio-e135fb2b491e0fb29be7bddb6a51d501bc10198bb86b4e63754622d235d5fc8a WatchSource:0}: Error finding container e135fb2b491e0fb29be7bddb6a51d501bc10198bb86b4e63754622d235d5fc8a: Status 404 returned error can't find the container with id e135fb2b491e0fb29be7bddb6a51d501bc10198bb86b4e63754622d235d5fc8a Oct 31 05:39:03.945134 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:03.945096 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c16ae582-kmwx6" event={"ID":"c89fa3eb-3b89-4047-a233-e428240273d7","Type":"ContainerStarted","Data":"23dd896ff2f3381acab53693e4d2e2b5fff59696b9a8dd852b192d442b457792"} Oct 31 05:39:03.945612 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:03.945188 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c16ae582-kmwx6" event={"ID":"c89fa3eb-3b89-4047-a233-e428240273d7","Type":"ContainerStarted","Data":"e135fb2b491e0fb29be7bddb6a51d501bc10198bb86b4e63754622d235d5fc8a"} Oct 31 05:39:03.984736 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:03.984677 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c16ae582-kmwx6" podStartSLOduration=1.984662267 podStartE2EDuration="1.984662267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:39:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:39:03.984325072 +0000 UTC m=+5698.740663111" watchObservedRunningTime="2026-10-31 05:39:03.984662267 +0000 UTC m=+5698.741000355" Oct 31 05:39:04.958521 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:04.958461 5019 generic.go:356] "Generic (PLEG): container finished" podID="c89fa3eb-3b89-4047-a233-e428240273d7" containerID="23dd896ff2f3381acab53693e4d2e2b5fff59696b9a8dd852b192d442b457792" exitCode=0 Oct 31 05:39:04.959293 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:04.958553 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c16ae582-kmwx6" event={"ID":"c89fa3eb-3b89-4047-a233-e428240273d7","Type":"ContainerDied","Data":"23dd896ff2f3381acab53693e4d2e2b5fff59696b9a8dd852b192d442b457792"} Oct 31 05:39:06.916361 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:06.913184 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c16ae582-kmwx6" Oct 31 05:39:06.987358 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:06.986556 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c89fa3eb-3b89-4047-a233-e428240273d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c89fa3eb-3b89-4047-a233-e428240273d7-maas-api-xks-ca\") pod \"c89fa3eb-3b89-4047-a233-e428240273d7\" (UID: \"c89fa3eb-3b89-4047-a233-e428240273d7\") " Oct 31 05:39:06.987358 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:06.986619 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c89fa3eb-3b89-4047-a233-e428240273d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c89fa3eb-3b89-4047-a233-e428240273d7-maas-api-openshift-ca\") pod \"c89fa3eb-3b89-4047-a233-e428240273d7\" (UID: \"c89fa3eb-3b89-4047-a233-e428240273d7\") " Oct 31 05:39:06.987358 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:06.986701 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c89fa3eb-3b89-4047-a233-e428240273d7-kube-api-access-wkhtb\" (UniqueName: \"kubernetes.io/projected/c89fa3eb-3b89-4047-a233-e428240273d7-kube-api-access-wkhtb\") pod \"c89fa3eb-3b89-4047-a233-e428240273d7\" (UID: \"c89fa3eb-3b89-4047-a233-e428240273d7\") " Oct 31 05:39:06.988024 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:06.987992 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c89fa3eb-3b89-4047-a233-e428240273d7-maas-api-openshift-ca" pod "c89fa3eb-3b89-4047-a233-e428240273d7" (UID: "c89fa3eb-3b89-4047-a233-e428240273d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:39:07.099452 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:07.082185 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c89fa3eb-3b89-4047-a233-e428240273d7-kube-api-access-wkhtb" pod "c89fa3eb-3b89-4047-a233-e428240273d7" (UID: "c89fa3eb-3b89-4047-a233-e428240273d7"). InnerVolumeSpecName "kube-api-access-wkhtb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:39:07.099772 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:07.096516 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c89fa3eb-3b89-4047-a233-e428240273d7-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:39:07.193533 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:07.193461 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c89fa3eb-3b89-4047-a233-e428240273d7-maas-api-xks-ca" pod "c89fa3eb-3b89-4047-a233-e428240273d7" (UID: "c89fa3eb-3b89-4047-a233-e428240273d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:39:07.252359 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:07.225466 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c16ae582-kmwx6" event={"ID":"c89fa3eb-3b89-4047-a233-e428240273d7","Type":"ContainerDied","Data":"e135fb2b491e0fb29be7bddb6a51d501bc10198bb86b4e63754622d235d5fc8a"} Oct 31 05:39:07.252359 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:07.225541 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e135fb2b491e0fb29be7bddb6a51d501bc10198bb86b4e63754622d235d5fc8a" Oct 31 05:39:07.252359 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:07.225625 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c16ae582-kmwx6" Oct 31 05:39:07.280369 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:07.279512 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c89fa3eb-3b89-4047-a233-e428240273d7-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:39:07.280369 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:07.279556 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkhtb\" (UniqueName: \"kubernetes.io/projected/c89fa3eb-3b89-4047-a233-e428240273d7-kube-api-access-wkhtb\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:39:09.879028 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:09.878998 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s88kp_3945ff65-e877-4372-9334-3968c68b7295/console-operator/2.log" Oct 31 05:39:09.885023 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:09.884996 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s88kp_3945ff65-e877-4372-9334-3968c68b7295/console-operator/2.log" Oct 31 05:39:22.391164 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:39:22.391123 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:39:35.982653 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:35.982612 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c908acb8-p7fzj"] Oct 31 05:39:35.983232 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:35.983101 5019 cpu_manager.go:414] "containerMap: removing container" podUID="c89fa3eb-3b89-4047-a233-e428240273d7" containerName="revoke-keys" Oct 31 05:39:35.983232 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:35.983116 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c89fa3eb-3b89-4047-a233-e428240273d7" containerName="revoke-keys" Oct 31 05:39:35.983232 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:35.983223 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="c89fa3eb-3b89-4047-a233-e428240273d7" containerName="revoke-keys" Oct 31 05:39:35.986826 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:35.986803 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c908acb8-p7fzj" Oct 31 05:39:36.014576 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:36.013999 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:39:36.014576 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:36.014010 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:39:36.014576 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:36.014035 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:39:36.014576 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:36.014074 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfhzn\"" Oct 31 05:39:36.015326 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:36.015292 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0e91733-757e-4549-ac4f-b8b43db81288-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c908acb8-p7fzj\" (UID: \"b0e91733-757e-4549-ac4f-b8b43db81288\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c908acb8-p7fzj" Oct 31 05:39:36.015486 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:36.015367 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6vnp\" (UniqueName: \"kubernetes.io/projected/b0e91733-757e-4549-ac4f-b8b43db81288-kube-api-access-f6vnp\") pod \"maas-api-revoke-sub-c908acb8-p7fzj\" (UID: \"b0e91733-757e-4549-ac4f-b8b43db81288\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c908acb8-p7fzj" Oct 31 05:39:36.015486 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:36.015418 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0e91733-757e-4549-ac4f-b8b43db81288-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c908acb8-p7fzj\" (UID: \"b0e91733-757e-4549-ac4f-b8b43db81288\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c908acb8-p7fzj" Oct 31 05:39:36.018639 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:36.018599 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c908acb8-p7fzj"] Oct 31 05:39:36.116140 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:36.116112 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0e91733-757e-4549-ac4f-b8b43db81288-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c908acb8-p7fzj\" (UID: \"b0e91733-757e-4549-ac4f-b8b43db81288\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c908acb8-p7fzj" Oct 31 05:39:36.116297 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:36.116162 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6vnp\" (UniqueName: \"kubernetes.io/projected/b0e91733-757e-4549-ac4f-b8b43db81288-kube-api-access-f6vnp\") pod \"maas-api-revoke-sub-c908acb8-p7fzj\" (UID: \"b0e91733-757e-4549-ac4f-b8b43db81288\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c908acb8-p7fzj" Oct 31 05:39:36.116297 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:36.116207 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0e91733-757e-4549-ac4f-b8b43db81288-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c908acb8-p7fzj\" (UID: \"b0e91733-757e-4549-ac4f-b8b43db81288\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c908acb8-p7fzj" Oct 31 05:39:36.116957 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:36.116934 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0e91733-757e-4549-ac4f-b8b43db81288-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c908acb8-p7fzj\" (UID: \"b0e91733-757e-4549-ac4f-b8b43db81288\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c908acb8-p7fzj" Oct 31 05:39:36.118561 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:36.118541 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0e91733-757e-4549-ac4f-b8b43db81288-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c908acb8-p7fzj\" (UID: \"b0e91733-757e-4549-ac4f-b8b43db81288\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c908acb8-p7fzj" Oct 31 05:39:36.139376 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:36.139350 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6vnp\" (UniqueName: \"kubernetes.io/projected/b0e91733-757e-4549-ac4f-b8b43db81288-kube-api-access-f6vnp\") pod \"maas-api-revoke-sub-c908acb8-p7fzj\" (UID: \"b0e91733-757e-4549-ac4f-b8b43db81288\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c908acb8-p7fzj" Oct 31 05:39:36.301050 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:36.301001 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c908acb8-p7fzj" Oct 31 05:39:36.600915 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:36.600885 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c908acb8-p7fzj"] Oct 31 05:39:36.602164 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:39:36.602124 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0e91733_757e_4549_ac4f_b8b43db81288.slice/crio-e6bc871b711dfbe6a2c39bdf1b77824cf2272cc75b75f4ea120be2e092aca9e3 WatchSource:0}: Error finding container e6bc871b711dfbe6a2c39bdf1b77824cf2272cc75b75f4ea120be2e092aca9e3: Status 404 returned error can't find the container with id e6bc871b711dfbe6a2c39bdf1b77824cf2272cc75b75f4ea120be2e092aca9e3 Oct 31 05:39:37.542636 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:37.542606 5019 generic.go:356] "Generic (PLEG): container finished" podID="b0e91733-757e-4549-ac4f-b8b43db81288" containerID="41c96d28c9d278f02751f0831ddeb65d7846ada26973aa892d2bd7804e616900" exitCode=0 Oct 31 05:39:37.542971 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:37.542689 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c908acb8-p7fzj" event={"ID":"b0e91733-757e-4549-ac4f-b8b43db81288","Type":"ContainerDied","Data":"41c96d28c9d278f02751f0831ddeb65d7846ada26973aa892d2bd7804e616900"} Oct 31 05:39:37.542971 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:37.542735 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c908acb8-p7fzj" event={"ID":"b0e91733-757e-4549-ac4f-b8b43db81288","Type":"ContainerStarted","Data":"e6bc871b711dfbe6a2c39bdf1b77824cf2272cc75b75f4ea120be2e092aca9e3"} Oct 31 05:39:38.030716 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:38.030681 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bgqjk"] Oct 31 05:39:38.033550 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:38.033517 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bgqjk"] Oct 31 05:39:38.697498 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:38.697471 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c908acb8-p7fzj" Oct 31 05:39:38.741672 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:38.741646 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0e91733-757e-4549-ac4f-b8b43db81288-kube-api-access-f6vnp\" (UniqueName: \"kubernetes.io/projected/b0e91733-757e-4549-ac4f-b8b43db81288-kube-api-access-f6vnp\") pod \"b0e91733-757e-4549-ac4f-b8b43db81288\" (UID: \"b0e91733-757e-4549-ac4f-b8b43db81288\") " Oct 31 05:39:38.741806 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:38.741730 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0e91733-757e-4549-ac4f-b8b43db81288-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0e91733-757e-4549-ac4f-b8b43db81288-maas-api-xks-ca\") pod \"b0e91733-757e-4549-ac4f-b8b43db81288\" (UID: \"b0e91733-757e-4549-ac4f-b8b43db81288\") " Oct 31 05:39:38.741806 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:38.741783 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0e91733-757e-4549-ac4f-b8b43db81288-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0e91733-757e-4549-ac4f-b8b43db81288-maas-api-openshift-ca\") pod \"b0e91733-757e-4549-ac4f-b8b43db81288\" (UID: \"b0e91733-757e-4549-ac4f-b8b43db81288\") " Oct 31 05:39:38.742305 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:38.742282 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0e91733-757e-4549-ac4f-b8b43db81288-maas-api-openshift-ca" pod "b0e91733-757e-4549-ac4f-b8b43db81288" (UID: "b0e91733-757e-4549-ac4f-b8b43db81288"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:39:38.744296 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:38.744267 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0e91733-757e-4549-ac4f-b8b43db81288-kube-api-access-f6vnp" pod "b0e91733-757e-4549-ac4f-b8b43db81288" (UID: "b0e91733-757e-4549-ac4f-b8b43db81288"). InnerVolumeSpecName "kube-api-access-f6vnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:39:38.744454 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:38.744275 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0e91733-757e-4549-ac4f-b8b43db81288-maas-api-xks-ca" pod "b0e91733-757e-4549-ac4f-b8b43db81288" (UID: "b0e91733-757e-4549-ac4f-b8b43db81288"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:39:38.843587 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:38.843510 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6vnp\" (UniqueName: \"kubernetes.io/projected/b0e91733-757e-4549-ac4f-b8b43db81288-kube-api-access-f6vnp\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:39:38.843587 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:38.843554 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0e91733-757e-4549-ac4f-b8b43db81288-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:39:38.843587 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:38.843564 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0e91733-757e-4549-ac4f-b8b43db81288-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:39:39.406565 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:39.406527 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32fd0aeb-9f3b-4add-bb99-f6cac6cf910d" path="/var/lib/kubelet/pods/32fd0aeb-9f3b-4add-bb99-f6cac6cf910d/volumes" Oct 31 05:39:39.555893 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:39.555866 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c908acb8-p7fzj" Oct 31 05:39:39.556041 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:39.555905 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c908acb8-p7fzj" event={"ID":"b0e91733-757e-4549-ac4f-b8b43db81288","Type":"ContainerDied","Data":"e6bc871b711dfbe6a2c39bdf1b77824cf2272cc75b75f4ea120be2e092aca9e3"} Oct 31 05:39:39.556041 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:39.555967 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6bc871b711dfbe6a2c39bdf1b77824cf2272cc75b75f4ea120be2e092aca9e3" Oct 31 05:39:46.323606 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:46.323573 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f2e7163-59wnt"] Oct 31 05:39:46.324031 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:46.324018 5019 cpu_manager.go:414] "containerMap: removing container" podUID="b0e91733-757e-4549-ac4f-b8b43db81288" containerName="revoke-keys" Oct 31 05:39:46.324092 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:46.324033 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0e91733-757e-4549-ac4f-b8b43db81288" containerName="revoke-keys" Oct 31 05:39:46.324140 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:46.324131 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0e91733-757e-4549-ac4f-b8b43db81288" containerName="revoke-keys" Oct 31 05:39:46.327775 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:46.327753 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2e7163-59wnt" Oct 31 05:39:46.344078 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:46.344053 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:39:46.356396 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:46.356358 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfhzn\"" Oct 31 05:39:46.359321 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:46.359294 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f2e7163-59wnt"] Oct 31 05:39:46.371886 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:46.371866 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:39:46.372000 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:46.371913 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:39:46.404286 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:46.404261 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4l2m\" (UniqueName: \"kubernetes.io/projected/06ca025b-42d9-4d04-9fbe-69d99d932f7e-kube-api-access-p4l2m\") pod \"maas-api-revoke-sub-3f2e7163-59wnt\" (UID: \"06ca025b-42d9-4d04-9fbe-69d99d932f7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2e7163-59wnt" Oct 31 05:39:46.404416 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:46.404310 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06ca025b-42d9-4d04-9fbe-69d99d932f7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f2e7163-59wnt\" (UID: \"06ca025b-42d9-4d04-9fbe-69d99d932f7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2e7163-59wnt" Oct 31 05:39:46.404489 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:46.404474 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06ca025b-42d9-4d04-9fbe-69d99d932f7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f2e7163-59wnt\" (UID: \"06ca025b-42d9-4d04-9fbe-69d99d932f7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2e7163-59wnt" Oct 31 05:39:46.505164 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:46.505133 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06ca025b-42d9-4d04-9fbe-69d99d932f7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f2e7163-59wnt\" (UID: \"06ca025b-42d9-4d04-9fbe-69d99d932f7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2e7163-59wnt" Oct 31 05:39:46.505286 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:46.505187 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4l2m\" (UniqueName: \"kubernetes.io/projected/06ca025b-42d9-4d04-9fbe-69d99d932f7e-kube-api-access-p4l2m\") pod \"maas-api-revoke-sub-3f2e7163-59wnt\" (UID: \"06ca025b-42d9-4d04-9fbe-69d99d932f7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2e7163-59wnt" Oct 31 05:39:46.505286 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:46.505256 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06ca025b-42d9-4d04-9fbe-69d99d932f7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f2e7163-59wnt\" (UID: \"06ca025b-42d9-4d04-9fbe-69d99d932f7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2e7163-59wnt" Oct 31 05:39:46.505946 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:46.505922 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06ca025b-42d9-4d04-9fbe-69d99d932f7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f2e7163-59wnt\" (UID: \"06ca025b-42d9-4d04-9fbe-69d99d932f7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2e7163-59wnt" Oct 31 05:39:46.507658 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:46.507636 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06ca025b-42d9-4d04-9fbe-69d99d932f7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f2e7163-59wnt\" (UID: \"06ca025b-42d9-4d04-9fbe-69d99d932f7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2e7163-59wnt" Oct 31 05:39:46.528630 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:46.528609 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4l2m\" (UniqueName: \"kubernetes.io/projected/06ca025b-42d9-4d04-9fbe-69d99d932f7e-kube-api-access-p4l2m\") pod \"maas-api-revoke-sub-3f2e7163-59wnt\" (UID: \"06ca025b-42d9-4d04-9fbe-69d99d932f7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2e7163-59wnt" Oct 31 05:39:46.645621 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:46.645564 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2e7163-59wnt" Oct 31 05:39:46.943147 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:46.943115 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f2e7163-59wnt"] Oct 31 05:39:46.944962 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:39:46.944925 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06ca025b_42d9_4d04_9fbe_69d99d932f7e.slice/crio-71f18fa3b6bb128aff7c2683ce82b3ac8b7aeedc431206a2d001f737d82f5034 WatchSource:0}: Error finding container 71f18fa3b6bb128aff7c2683ce82b3ac8b7aeedc431206a2d001f737d82f5034: Status 404 returned error can't find the container with id 71f18fa3b6bb128aff7c2683ce82b3ac8b7aeedc431206a2d001f737d82f5034 Oct 31 05:39:47.604582 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:47.604495 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2e7163-59wnt" event={"ID":"06ca025b-42d9-4d04-9fbe-69d99d932f7e","Type":"ContainerStarted","Data":"03f79c23505f369212d63f3e015814fbf06af971cdf8c5bde3f94f8c3ac3d9ea"} Oct 31 05:39:47.604582 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:47.604553 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2e7163-59wnt" event={"ID":"06ca025b-42d9-4d04-9fbe-69d99d932f7e","Type":"ContainerStarted","Data":"71f18fa3b6bb128aff7c2683ce82b3ac8b7aeedc431206a2d001f737d82f5034"} Oct 31 05:39:47.645010 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:47.644961 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2e7163-59wnt" podStartSLOduration=1.644948318 podStartE2EDuration="1.644948318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:39:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:39:47.643574513 +0000 UTC m=+5742.399912553" watchObservedRunningTime="2026-10-31 05:39:47.644948318 +0000 UTC m=+5742.401286357" Oct 31 05:39:48.612582 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:48.612549 5019 generic.go:356] "Generic (PLEG): container finished" podID="06ca025b-42d9-4d04-9fbe-69d99d932f7e" containerID="03f79c23505f369212d63f3e015814fbf06af971cdf8c5bde3f94f8c3ac3d9ea" exitCode=0 Oct 31 05:39:48.613050 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:48.612634 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2e7163-59wnt" event={"ID":"06ca025b-42d9-4d04-9fbe-69d99d932f7e","Type":"ContainerDied","Data":"03f79c23505f369212d63f3e015814fbf06af971cdf8c5bde3f94f8c3ac3d9ea"} Oct 31 05:39:49.753473 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:49.753443 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2e7163-59wnt" Oct 31 05:39:49.838733 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:49.838691 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06ca025b-42d9-4d04-9fbe-69d99d932f7e-kube-api-access-p4l2m\" (UniqueName: \"kubernetes.io/projected/06ca025b-42d9-4d04-9fbe-69d99d932f7e-kube-api-access-p4l2m\") pod \"06ca025b-42d9-4d04-9fbe-69d99d932f7e\" (UID: \"06ca025b-42d9-4d04-9fbe-69d99d932f7e\") " Oct 31 05:39:49.838914 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:49.838790 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06ca025b-42d9-4d04-9fbe-69d99d932f7e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06ca025b-42d9-4d04-9fbe-69d99d932f7e-maas-api-openshift-ca\") pod \"06ca025b-42d9-4d04-9fbe-69d99d932f7e\" (UID: \"06ca025b-42d9-4d04-9fbe-69d99d932f7e\") " Oct 31 05:39:49.838914 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:49.838902 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06ca025b-42d9-4d04-9fbe-69d99d932f7e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06ca025b-42d9-4d04-9fbe-69d99d932f7e-maas-api-xks-ca\") pod \"06ca025b-42d9-4d04-9fbe-69d99d932f7e\" (UID: \"06ca025b-42d9-4d04-9fbe-69d99d932f7e\") " Oct 31 05:39:49.839178 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:49.839155 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06ca025b-42d9-4d04-9fbe-69d99d932f7e-maas-api-openshift-ca" pod "06ca025b-42d9-4d04-9fbe-69d99d932f7e" (UID: "06ca025b-42d9-4d04-9fbe-69d99d932f7e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:39:49.839285 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:49.839242 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06ca025b-42d9-4d04-9fbe-69d99d932f7e-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:39:49.841038 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:49.841007 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06ca025b-42d9-4d04-9fbe-69d99d932f7e-maas-api-xks-ca" pod "06ca025b-42d9-4d04-9fbe-69d99d932f7e" (UID: "06ca025b-42d9-4d04-9fbe-69d99d932f7e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:39:49.841158 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:49.841113 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06ca025b-42d9-4d04-9fbe-69d99d932f7e-kube-api-access-p4l2m" pod "06ca025b-42d9-4d04-9fbe-69d99d932f7e" (UID: "06ca025b-42d9-4d04-9fbe-69d99d932f7e"). InnerVolumeSpecName "kube-api-access-p4l2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:39:49.940101 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:49.940044 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06ca025b-42d9-4d04-9fbe-69d99d932f7e-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:39:49.940101 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:49.940067 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4l2m\" (UniqueName: \"kubernetes.io/projected/06ca025b-42d9-4d04-9fbe-69d99d932f7e-kube-api-access-p4l2m\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:39:50.629315 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:50.629283 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2e7163-59wnt" Oct 31 05:39:50.629674 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:50.629303 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2e7163-59wnt" event={"ID":"06ca025b-42d9-4d04-9fbe-69d99d932f7e","Type":"ContainerDied","Data":"71f18fa3b6bb128aff7c2683ce82b3ac8b7aeedc431206a2d001f737d82f5034"} Oct 31 05:39:50.629787 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:50.629708 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71f18fa3b6bb128aff7c2683ce82b3ac8b7aeedc431206a2d001f737d82f5034" Oct 31 05:39:52.570357 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:39:52.558559 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:39:56.654783 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:56.654746 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd6cc76f-q7drd"] Oct 31 05:39:56.655275 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:56.655260 5019 cpu_manager.go:414] "containerMap: removing container" podUID="06ca025b-42d9-4d04-9fbe-69d99d932f7e" containerName="revoke-keys" Oct 31 05:39:56.655322 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:56.655277 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06ca025b-42d9-4d04-9fbe-69d99d932f7e" containerName="revoke-keys" Oct 31 05:39:56.655416 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:56.655404 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="06ca025b-42d9-4d04-9fbe-69d99d932f7e" containerName="revoke-keys" Oct 31 05:39:56.658831 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:56.658808 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6cc76f-q7drd" Oct 31 05:39:56.681090 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:56.681052 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:39:56.681090 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:56.681072 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:39:56.720447 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:56.720389 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:39:56.720763 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:56.720736 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfhzn\"" Oct 31 05:39:56.724756 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:56.724726 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd6cc76f-q7drd"] Oct 31 05:39:56.756318 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:56.756285 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2330a863-0b04-4d46-9768-28332b366152-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd6cc76f-q7drd\" (UID: \"2330a863-0b04-4d46-9768-28332b366152\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6cc76f-q7drd" Oct 31 05:39:56.756492 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:56.756340 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5sjh\" (UniqueName: \"kubernetes.io/projected/2330a863-0b04-4d46-9768-28332b366152-kube-api-access-w5sjh\") pod \"maas-api-revoke-sub-cd6cc76f-q7drd\" (UID: \"2330a863-0b04-4d46-9768-28332b366152\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6cc76f-q7drd" Oct 31 05:39:56.756558 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:56.756508 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2330a863-0b04-4d46-9768-28332b366152-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd6cc76f-q7drd\" (UID: \"2330a863-0b04-4d46-9768-28332b366152\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6cc76f-q7drd" Oct 31 05:39:56.858033 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:56.857988 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2330a863-0b04-4d46-9768-28332b366152-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd6cc76f-q7drd\" (UID: \"2330a863-0b04-4d46-9768-28332b366152\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6cc76f-q7drd" Oct 31 05:39:56.858273 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:56.858044 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5sjh\" (UniqueName: \"kubernetes.io/projected/2330a863-0b04-4d46-9768-28332b366152-kube-api-access-w5sjh\") pod \"maas-api-revoke-sub-cd6cc76f-q7drd\" (UID: \"2330a863-0b04-4d46-9768-28332b366152\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6cc76f-q7drd" Oct 31 05:39:56.858273 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:56.858145 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2330a863-0b04-4d46-9768-28332b366152-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd6cc76f-q7drd\" (UID: \"2330a863-0b04-4d46-9768-28332b366152\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6cc76f-q7drd" Oct 31 05:39:56.858829 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:56.858798 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2330a863-0b04-4d46-9768-28332b366152-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd6cc76f-q7drd\" (UID: \"2330a863-0b04-4d46-9768-28332b366152\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6cc76f-q7drd" Oct 31 05:39:56.860618 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:56.860597 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2330a863-0b04-4d46-9768-28332b366152-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd6cc76f-q7drd\" (UID: \"2330a863-0b04-4d46-9768-28332b366152\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6cc76f-q7drd" Oct 31 05:39:56.881887 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:56.881857 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5sjh\" (UniqueName: \"kubernetes.io/projected/2330a863-0b04-4d46-9768-28332b366152-kube-api-access-w5sjh\") pod \"maas-api-revoke-sub-cd6cc76f-q7drd\" (UID: \"2330a863-0b04-4d46-9768-28332b366152\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6cc76f-q7drd" Oct 31 05:39:56.972489 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:56.972453 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6cc76f-q7drd" Oct 31 05:39:57.271761 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:57.271733 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd6cc76f-q7drd"] Oct 31 05:39:57.274450 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:39:57.274421 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2330a863_0b04_4d46_9768_28332b366152.slice/crio-2dc01092af9e702b2e6deb87353c93cef37723db230eaa2740ccfe085ed65eae WatchSource:0}: Error finding container 2dc01092af9e702b2e6deb87353c93cef37723db230eaa2740ccfe085ed65eae: Status 404 returned error can't find the container with id 2dc01092af9e702b2e6deb87353c93cef37723db230eaa2740ccfe085ed65eae Oct 31 05:39:57.694678 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:57.694634 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6cc76f-q7drd" event={"ID":"2330a863-0b04-4d46-9768-28332b366152","Type":"ContainerStarted","Data":"f53bcf16c026b258e8acdfa0340e6968e4ad9260b1c3115d315039a0bf2bf847"} Oct 31 05:39:57.695096 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:57.694698 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6cc76f-q7drd" event={"ID":"2330a863-0b04-4d46-9768-28332b366152","Type":"ContainerStarted","Data":"2dc01092af9e702b2e6deb87353c93cef37723db230eaa2740ccfe085ed65eae"} Oct 31 05:39:57.735311 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:57.735238 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6cc76f-q7drd" podStartSLOduration=1.7352232810000001 podStartE2EDuration="1.735223281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:39:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:39:57.734205213 +0000 UTC m=+5752.490543253" watchObservedRunningTime="2026-10-31 05:39:57.735223281 +0000 UTC m=+5752.491561314" Oct 31 05:39:58.702317 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:58.702273 5019 generic.go:356] "Generic (PLEG): container finished" podID="2330a863-0b04-4d46-9768-28332b366152" containerID="f53bcf16c026b258e8acdfa0340e6968e4ad9260b1c3115d315039a0bf2bf847" exitCode=0 Oct 31 05:39:58.702766 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:58.702367 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6cc76f-q7drd" event={"ID":"2330a863-0b04-4d46-9768-28332b366152","Type":"ContainerDied","Data":"f53bcf16c026b258e8acdfa0340e6968e4ad9260b1c3115d315039a0bf2bf847"} Oct 31 05:39:59.860878 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:59.860847 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6cc76f-q7drd" Oct 31 05:39:59.993303 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:59.993263 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2330a863-0b04-4d46-9768-28332b366152-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2330a863-0b04-4d46-9768-28332b366152-maas-api-openshift-ca\") pod \"2330a863-0b04-4d46-9768-28332b366152\" (UID: \"2330a863-0b04-4d46-9768-28332b366152\") " Oct 31 05:39:59.993527 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:59.993383 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2330a863-0b04-4d46-9768-28332b366152-kube-api-access-w5sjh\" (UniqueName: \"kubernetes.io/projected/2330a863-0b04-4d46-9768-28332b366152-kube-api-access-w5sjh\") pod \"2330a863-0b04-4d46-9768-28332b366152\" (UID: \"2330a863-0b04-4d46-9768-28332b366152\") " Oct 31 05:39:59.993527 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:59.993452 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2330a863-0b04-4d46-9768-28332b366152-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2330a863-0b04-4d46-9768-28332b366152-maas-api-xks-ca\") pod \"2330a863-0b04-4d46-9768-28332b366152\" (UID: \"2330a863-0b04-4d46-9768-28332b366152\") " Oct 31 05:39:59.993693 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:59.993668 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2330a863-0b04-4d46-9768-28332b366152-maas-api-openshift-ca" pod "2330a863-0b04-4d46-9768-28332b366152" (UID: "2330a863-0b04-4d46-9768-28332b366152"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:39:59.993885 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:59.993863 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2330a863-0b04-4d46-9768-28332b366152-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:39:59.995754 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:59.995726 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2330a863-0b04-4d46-9768-28332b366152-kube-api-access-w5sjh" pod "2330a863-0b04-4d46-9768-28332b366152" (UID: "2330a863-0b04-4d46-9768-28332b366152"). InnerVolumeSpecName "kube-api-access-w5sjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:39:59.995754 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:39:59.995745 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2330a863-0b04-4d46-9768-28332b366152-maas-api-xks-ca" pod "2330a863-0b04-4d46-9768-28332b366152" (UID: "2330a863-0b04-4d46-9768-28332b366152"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:40:00.094978 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:00.094933 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5sjh\" (UniqueName: \"kubernetes.io/projected/2330a863-0b04-4d46-9768-28332b366152-kube-api-access-w5sjh\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:40:00.094978 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:00.094973 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2330a863-0b04-4d46-9768-28332b366152-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:40:00.716587 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:00.716552 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6cc76f-q7drd" Oct 31 05:40:00.716775 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:00.716590 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6cc76f-q7drd" event={"ID":"2330a863-0b04-4d46-9768-28332b366152","Type":"ContainerDied","Data":"2dc01092af9e702b2e6deb87353c93cef37723db230eaa2740ccfe085ed65eae"} Oct 31 05:40:00.716775 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:00.716644 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2dc01092af9e702b2e6deb87353c93cef37723db230eaa2740ccfe085ed65eae" Oct 31 05:40:07.108432 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:07.108387 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36d1b076-tr4fl"] Oct 31 05:40:07.108998 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:07.108861 5019 cpu_manager.go:414] "containerMap: removing container" podUID="2330a863-0b04-4d46-9768-28332b366152" containerName="revoke-keys" Oct 31 05:40:07.108998 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:07.108874 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2330a863-0b04-4d46-9768-28332b366152" containerName="revoke-keys" Oct 31 05:40:07.108998 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:07.108988 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="2330a863-0b04-4d46-9768-28332b366152" containerName="revoke-keys" Oct 31 05:40:07.112809 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:07.112788 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d1b076-tr4fl" Oct 31 05:40:07.128016 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:07.127964 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:40:07.140267 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:07.140161 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:40:07.140498 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:07.140471 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfhzn\"" Oct 31 05:40:07.140593 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:07.140572 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:40:07.145165 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:07.145134 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36d1b076-tr4fl"] Oct 31 05:40:07.266611 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:07.266578 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng42d\" (UniqueName: \"kubernetes.io/projected/e10387ea-7e1c-49f8-9f45-95e7073a3b55-kube-api-access-ng42d\") pod \"maas-api-revoke-sub-36d1b076-tr4fl\" (UID: \"e10387ea-7e1c-49f8-9f45-95e7073a3b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d1b076-tr4fl" Oct 31 05:40:07.266898 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:07.266615 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e10387ea-7e1c-49f8-9f45-95e7073a3b55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36d1b076-tr4fl\" (UID: \"e10387ea-7e1c-49f8-9f45-95e7073a3b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d1b076-tr4fl" Oct 31 05:40:07.266898 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:07.266636 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e10387ea-7e1c-49f8-9f45-95e7073a3b55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36d1b076-tr4fl\" (UID: \"e10387ea-7e1c-49f8-9f45-95e7073a3b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d1b076-tr4fl" Oct 31 05:40:07.368085 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:07.367992 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng42d\" (UniqueName: \"kubernetes.io/projected/e10387ea-7e1c-49f8-9f45-95e7073a3b55-kube-api-access-ng42d\") pod \"maas-api-revoke-sub-36d1b076-tr4fl\" (UID: \"e10387ea-7e1c-49f8-9f45-95e7073a3b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d1b076-tr4fl" Oct 31 05:40:07.368085 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:07.368053 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e10387ea-7e1c-49f8-9f45-95e7073a3b55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36d1b076-tr4fl\" (UID: \"e10387ea-7e1c-49f8-9f45-95e7073a3b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d1b076-tr4fl" Oct 31 05:40:07.368394 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:07.368084 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e10387ea-7e1c-49f8-9f45-95e7073a3b55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36d1b076-tr4fl\" (UID: \"e10387ea-7e1c-49f8-9f45-95e7073a3b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d1b076-tr4fl" Oct 31 05:40:07.368798 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:07.368771 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e10387ea-7e1c-49f8-9f45-95e7073a3b55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36d1b076-tr4fl\" (UID: \"e10387ea-7e1c-49f8-9f45-95e7073a3b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d1b076-tr4fl" Oct 31 05:40:07.370601 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:07.370571 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e10387ea-7e1c-49f8-9f45-95e7073a3b55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36d1b076-tr4fl\" (UID: \"e10387ea-7e1c-49f8-9f45-95e7073a3b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d1b076-tr4fl" Oct 31 05:40:07.391315 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:07.391284 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng42d\" (UniqueName: \"kubernetes.io/projected/e10387ea-7e1c-49f8-9f45-95e7073a3b55-kube-api-access-ng42d\") pod \"maas-api-revoke-sub-36d1b076-tr4fl\" (UID: \"e10387ea-7e1c-49f8-9f45-95e7073a3b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d1b076-tr4fl" Oct 31 05:40:07.429018 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:07.428989 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d1b076-tr4fl" Oct 31 05:40:07.735018 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:07.734980 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36d1b076-tr4fl"] Oct 31 05:40:07.735519 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:40:07.735490 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode10387ea_7e1c_49f8_9f45_95e7073a3b55.slice/crio-2d15848641bb0130137a862a8570421474a6f1e7e2e301650bb37c6d76a1e608 WatchSource:0}: Error finding container 2d15848641bb0130137a862a8570421474a6f1e7e2e301650bb37c6d76a1e608: Status 404 returned error can't find the container with id 2d15848641bb0130137a862a8570421474a6f1e7e2e301650bb37c6d76a1e608 Oct 31 05:40:07.761139 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:07.761100 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d1b076-tr4fl" event={"ID":"e10387ea-7e1c-49f8-9f45-95e7073a3b55","Type":"ContainerStarted","Data":"2d15848641bb0130137a862a8570421474a6f1e7e2e301650bb37c6d76a1e608"} Oct 31 05:40:08.773164 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:08.773118 5019 generic.go:356] "Generic (PLEG): container finished" podID="e10387ea-7e1c-49f8-9f45-95e7073a3b55" containerID="c58504779015d88f8057c0a62202b2d439457043fa06ee313b45c7fb2dc6b334" exitCode=0 Oct 31 05:40:08.773743 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:08.773317 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d1b076-tr4fl" event={"ID":"e10387ea-7e1c-49f8-9f45-95e7073a3b55","Type":"ContainerDied","Data":"c58504779015d88f8057c0a62202b2d439457043fa06ee313b45c7fb2dc6b334"} Oct 31 05:40:10.815726 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:10.815665 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d1b076-tr4fl" Oct 31 05:40:10.865812 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:10.865724 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e10387ea-7e1c-49f8-9f45-95e7073a3b55-kube-api-access-ng42d\" (UniqueName: \"kubernetes.io/projected/e10387ea-7e1c-49f8-9f45-95e7073a3b55-kube-api-access-ng42d\") pod \"e10387ea-7e1c-49f8-9f45-95e7073a3b55\" (UID: \"e10387ea-7e1c-49f8-9f45-95e7073a3b55\") " Oct 31 05:40:10.877501 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:10.877460 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e10387ea-7e1c-49f8-9f45-95e7073a3b55-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e10387ea-7e1c-49f8-9f45-95e7073a3b55-maas-api-xks-ca\") pod \"e10387ea-7e1c-49f8-9f45-95e7073a3b55\" (UID: \"e10387ea-7e1c-49f8-9f45-95e7073a3b55\") " Oct 31 05:40:10.877982 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:10.877940 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e10387ea-7e1c-49f8-9f45-95e7073a3b55-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e10387ea-7e1c-49f8-9f45-95e7073a3b55-maas-api-openshift-ca\") pod \"e10387ea-7e1c-49f8-9f45-95e7073a3b55\" (UID: \"e10387ea-7e1c-49f8-9f45-95e7073a3b55\") " Oct 31 05:40:10.880709 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:10.880641 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e10387ea-7e1c-49f8-9f45-95e7073a3b55-maas-api-openshift-ca" pod "e10387ea-7e1c-49f8-9f45-95e7073a3b55" (UID: "e10387ea-7e1c-49f8-9f45-95e7073a3b55"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:40:10.881730 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:10.881700 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e10387ea-7e1c-49f8-9f45-95e7073a3b55-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:40:10.887973 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:10.887583 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d1b076-tr4fl" event={"ID":"e10387ea-7e1c-49f8-9f45-95e7073a3b55","Type":"ContainerDied","Data":"2d15848641bb0130137a862a8570421474a6f1e7e2e301650bb37c6d76a1e608"} Oct 31 05:40:10.887973 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:10.887657 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d15848641bb0130137a862a8570421474a6f1e7e2e301650bb37c6d76a1e608" Oct 31 05:40:10.887973 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:10.887734 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d1b076-tr4fl" Oct 31 05:40:11.022016 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:11.021941 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e10387ea-7e1c-49f8-9f45-95e7073a3b55-maas-api-xks-ca" pod "e10387ea-7e1c-49f8-9f45-95e7073a3b55" (UID: "e10387ea-7e1c-49f8-9f45-95e7073a3b55"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:40:11.022516 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:11.022481 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e10387ea-7e1c-49f8-9f45-95e7073a3b55-kube-api-access-ng42d" pod "e10387ea-7e1c-49f8-9f45-95e7073a3b55" (UID: "e10387ea-7e1c-49f8-9f45-95e7073a3b55"). InnerVolumeSpecName "kube-api-access-ng42d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:40:11.143423 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:11.125029 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng42d\" (UniqueName: \"kubernetes.io/projected/e10387ea-7e1c-49f8-9f45-95e7073a3b55-kube-api-access-ng42d\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:40:11.143423 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:11.125079 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e10387ea-7e1c-49f8-9f45-95e7073a3b55-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:40:15.404674 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:15.404641 5019 scope.go:122] "RemoveContainer" containerID="3c9526816faf91fcddeadd9be1f1516f5c28c26e7f433997d7cc720f8d8da49d" Oct 31 05:40:17.468657 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:17.468615 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68729206-bgxmz"] Oct 31 05:40:17.469166 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:17.469064 5019 cpu_manager.go:414] "containerMap: removing container" podUID="e10387ea-7e1c-49f8-9f45-95e7073a3b55" containerName="revoke-keys" Oct 31 05:40:17.469166 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:17.469077 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e10387ea-7e1c-49f8-9f45-95e7073a3b55" containerName="revoke-keys" Oct 31 05:40:17.469362 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:17.469179 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e10387ea-7e1c-49f8-9f45-95e7073a3b55" containerName="revoke-keys" Oct 31 05:40:17.473646 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:17.473627 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68729206-bgxmz" Oct 31 05:40:17.492893 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:17.492867 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68729206-bgxmz"] Oct 31 05:40:17.500074 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:17.500045 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfhzn\"" Oct 31 05:40:17.500300 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:17.500045 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:40:17.500413 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:17.500078 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:40:17.500413 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:17.500240 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:40:17.591893 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:17.591846 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ea5bbc9-21fc-4dfb-9f14-1ad3ab9b852b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68729206-bgxmz\" (UID: \"0ea5bbc9-21fc-4dfb-9f14-1ad3ab9b852b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68729206-bgxmz" Oct 31 05:40:17.592104 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:17.591920 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ea5bbc9-21fc-4dfb-9f14-1ad3ab9b852b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68729206-bgxmz\" (UID: \"0ea5bbc9-21fc-4dfb-9f14-1ad3ab9b852b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68729206-bgxmz" Oct 31 05:40:17.592104 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:17.592014 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d767x\" (UniqueName: \"kubernetes.io/projected/0ea5bbc9-21fc-4dfb-9f14-1ad3ab9b852b-kube-api-access-d767x\") pod \"maas-api-revoke-sub-68729206-bgxmz\" (UID: \"0ea5bbc9-21fc-4dfb-9f14-1ad3ab9b852b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68729206-bgxmz" Oct 31 05:40:17.693270 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:17.693222 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ea5bbc9-21fc-4dfb-9f14-1ad3ab9b852b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68729206-bgxmz\" (UID: \"0ea5bbc9-21fc-4dfb-9f14-1ad3ab9b852b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68729206-bgxmz" Oct 31 05:40:17.693523 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:17.693301 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ea5bbc9-21fc-4dfb-9f14-1ad3ab9b852b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68729206-bgxmz\" (UID: \"0ea5bbc9-21fc-4dfb-9f14-1ad3ab9b852b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68729206-bgxmz" Oct 31 05:40:17.693523 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:17.693384 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d767x\" (UniqueName: \"kubernetes.io/projected/0ea5bbc9-21fc-4dfb-9f14-1ad3ab9b852b-kube-api-access-d767x\") pod \"maas-api-revoke-sub-68729206-bgxmz\" (UID: \"0ea5bbc9-21fc-4dfb-9f14-1ad3ab9b852b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68729206-bgxmz" Oct 31 05:40:17.694066 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:17.694040 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ea5bbc9-21fc-4dfb-9f14-1ad3ab9b852b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68729206-bgxmz\" (UID: \"0ea5bbc9-21fc-4dfb-9f14-1ad3ab9b852b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68729206-bgxmz" Oct 31 05:40:17.695936 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:17.695906 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ea5bbc9-21fc-4dfb-9f14-1ad3ab9b852b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68729206-bgxmz\" (UID: \"0ea5bbc9-21fc-4dfb-9f14-1ad3ab9b852b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68729206-bgxmz" Oct 31 05:40:17.717557 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:17.717515 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d767x\" (UniqueName: \"kubernetes.io/projected/0ea5bbc9-21fc-4dfb-9f14-1ad3ab9b852b-kube-api-access-d767x\") pod \"maas-api-revoke-sub-68729206-bgxmz\" (UID: \"0ea5bbc9-21fc-4dfb-9f14-1ad3ab9b852b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68729206-bgxmz" Oct 31 05:40:17.787310 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:17.787208 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68729206-bgxmz" Oct 31 05:40:18.095035 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:18.095000 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68729206-bgxmz"] Oct 31 05:40:18.095960 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:40:18.095925 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ea5bbc9_21fc_4dfb_9f14_1ad3ab9b852b.slice/crio-8a19acc26bbab3690102d7393ad58df5354627a81fd474bb0905715283882acd WatchSource:0}: Error finding container 8a19acc26bbab3690102d7393ad58df5354627a81fd474bb0905715283882acd: Status 404 returned error can't find the container with id 8a19acc26bbab3690102d7393ad58df5354627a81fd474bb0905715283882acd Oct 31 05:40:18.938874 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:18.938829 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68729206-bgxmz" event={"ID":"0ea5bbc9-21fc-4dfb-9f14-1ad3ab9b852b","Type":"ContainerStarted","Data":"a3b795deba7aa23588095f7dac28ff723db3e4654b0d648b72f36d42bd32a5de"} Oct 31 05:40:18.939449 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:18.938903 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68729206-bgxmz" event={"ID":"0ea5bbc9-21fc-4dfb-9f14-1ad3ab9b852b","Type":"ContainerStarted","Data":"8a19acc26bbab3690102d7393ad58df5354627a81fd474bb0905715283882acd"} Oct 31 05:40:18.983927 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:18.983832 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68729206-bgxmz" podStartSLOduration=1.983808828 podStartE2EDuration="1.983808828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:40:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:40:18.983525618 +0000 UTC m=+5773.739863659" watchObservedRunningTime="2026-10-31 05:40:18.983808828 +0000 UTC m=+5773.740146870" Oct 31 05:40:19.975357 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:19.971226 5019 generic.go:356] "Generic (PLEG): container finished" podID="0ea5bbc9-21fc-4dfb-9f14-1ad3ab9b852b" containerID="a3b795deba7aa23588095f7dac28ff723db3e4654b0d648b72f36d42bd32a5de" exitCode=0 Oct 31 05:40:19.975357 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:19.971372 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68729206-bgxmz" event={"ID":"0ea5bbc9-21fc-4dfb-9f14-1ad3ab9b852b","Type":"ContainerDied","Data":"a3b795deba7aa23588095f7dac28ff723db3e4654b0d648b72f36d42bd32a5de"} Oct 31 05:40:21.685881 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:21.685469 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68729206-bgxmz" Oct 31 05:40:21.768726 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:21.740751 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ea5bbc9-21fc-4dfb-9f14-1ad3ab9b852b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ea5bbc9-21fc-4dfb-9f14-1ad3ab9b852b-maas-api-openshift-ca\") pod \"0ea5bbc9-21fc-4dfb-9f14-1ad3ab9b852b\" (UID: \"0ea5bbc9-21fc-4dfb-9f14-1ad3ab9b852b\") " Oct 31 05:40:21.768726 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:21.740865 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ea5bbc9-21fc-4dfb-9f14-1ad3ab9b852b-kube-api-access-d767x\" (UniqueName: \"kubernetes.io/projected/0ea5bbc9-21fc-4dfb-9f14-1ad3ab9b852b-kube-api-access-d767x\") pod \"0ea5bbc9-21fc-4dfb-9f14-1ad3ab9b852b\" (UID: \"0ea5bbc9-21fc-4dfb-9f14-1ad3ab9b852b\") " Oct 31 05:40:21.768726 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:21.740917 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ea5bbc9-21fc-4dfb-9f14-1ad3ab9b852b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ea5bbc9-21fc-4dfb-9f14-1ad3ab9b852b-maas-api-xks-ca\") pod \"0ea5bbc9-21fc-4dfb-9f14-1ad3ab9b852b\" (UID: \"0ea5bbc9-21fc-4dfb-9f14-1ad3ab9b852b\") " Oct 31 05:40:21.768726 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:21.743674 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ea5bbc9-21fc-4dfb-9f14-1ad3ab9b852b-maas-api-openshift-ca" pod "0ea5bbc9-21fc-4dfb-9f14-1ad3ab9b852b" (UID: "0ea5bbc9-21fc-4dfb-9f14-1ad3ab9b852b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:40:21.768726 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:21.768566 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ea5bbc9-21fc-4dfb-9f14-1ad3ab9b852b-kube-api-access-d767x" pod "0ea5bbc9-21fc-4dfb-9f14-1ad3ab9b852b" (UID: "0ea5bbc9-21fc-4dfb-9f14-1ad3ab9b852b"). InnerVolumeSpecName "kube-api-access-d767x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:40:21.768726 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:21.768669 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ea5bbc9-21fc-4dfb-9f14-1ad3ab9b852b-maas-api-xks-ca" pod "0ea5bbc9-21fc-4dfb-9f14-1ad3ab9b852b" (UID: "0ea5bbc9-21fc-4dfb-9f14-1ad3ab9b852b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:40:21.842316 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:21.842252 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ea5bbc9-21fc-4dfb-9f14-1ad3ab9b852b-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:40:21.842638 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:21.842613 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d767x\" (UniqueName: \"kubernetes.io/projected/0ea5bbc9-21fc-4dfb-9f14-1ad3ab9b852b-kube-api-access-d767x\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:40:21.842895 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:21.842872 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ea5bbc9-21fc-4dfb-9f14-1ad3ab9b852b-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:40:22.003848 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:22.003816 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68729206-bgxmz" Oct 31 05:40:22.003848 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:22.003831 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68729206-bgxmz" event={"ID":"0ea5bbc9-21fc-4dfb-9f14-1ad3ab9b852b","Type":"ContainerDied","Data":"8a19acc26bbab3690102d7393ad58df5354627a81fd474bb0905715283882acd"} Oct 31 05:40:22.004088 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:22.003903 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a19acc26bbab3690102d7393ad58df5354627a81fd474bb0905715283882acd" Oct 31 05:40:22.390347 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:40:22.390243 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:40:27.838576 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:27.838533 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbc70dfe-w7wzz"] Oct 31 05:40:27.839032 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:27.839018 5019 cpu_manager.go:414] "containerMap: removing container" podUID="0ea5bbc9-21fc-4dfb-9f14-1ad3ab9b852b" containerName="revoke-keys" Oct 31 05:40:27.839081 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:27.839034 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ea5bbc9-21fc-4dfb-9f14-1ad3ab9b852b" containerName="revoke-keys" Oct 31 05:40:27.839116 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:27.839106 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ea5bbc9-21fc-4dfb-9f14-1ad3ab9b852b" containerName="revoke-keys" Oct 31 05:40:27.842380 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:27.842360 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbc70dfe-w7wzz" Oct 31 05:40:27.871764 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:27.871720 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbc70dfe-w7wzz"] Oct 31 05:40:27.877565 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:27.877524 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:40:27.877784 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:27.877565 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:40:27.877784 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:27.877582 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfhzn\"" Oct 31 05:40:27.877784 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:27.877613 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:40:27.903003 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:27.902953 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2dqr\" (UniqueName: \"kubernetes.io/projected/ce1175fc-39e4-44e7-a47a-0cc40c570f8e-kube-api-access-z2dqr\") pod \"maas-api-revoke-sub-bbc70dfe-w7wzz\" (UID: \"ce1175fc-39e4-44e7-a47a-0cc40c570f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbc70dfe-w7wzz" Oct 31 05:40:27.903237 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:27.903111 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce1175fc-39e4-44e7-a47a-0cc40c570f8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbc70dfe-w7wzz\" (UID: \"ce1175fc-39e4-44e7-a47a-0cc40c570f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbc70dfe-w7wzz" Oct 31 05:40:27.903237 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:27.903172 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce1175fc-39e4-44e7-a47a-0cc40c570f8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbc70dfe-w7wzz\" (UID: \"ce1175fc-39e4-44e7-a47a-0cc40c570f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbc70dfe-w7wzz" Oct 31 05:40:28.004452 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:28.004411 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce1175fc-39e4-44e7-a47a-0cc40c570f8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbc70dfe-w7wzz\" (UID: \"ce1175fc-39e4-44e7-a47a-0cc40c570f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbc70dfe-w7wzz" Oct 31 05:40:28.004668 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:28.004482 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce1175fc-39e4-44e7-a47a-0cc40c570f8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbc70dfe-w7wzz\" (UID: \"ce1175fc-39e4-44e7-a47a-0cc40c570f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbc70dfe-w7wzz" Oct 31 05:40:28.004668 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:28.004515 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2dqr\" (UniqueName: \"kubernetes.io/projected/ce1175fc-39e4-44e7-a47a-0cc40c570f8e-kube-api-access-z2dqr\") pod \"maas-api-revoke-sub-bbc70dfe-w7wzz\" (UID: \"ce1175fc-39e4-44e7-a47a-0cc40c570f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbc70dfe-w7wzz" Oct 31 05:40:28.005298 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:28.005267 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce1175fc-39e4-44e7-a47a-0cc40c570f8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbc70dfe-w7wzz\" (UID: \"ce1175fc-39e4-44e7-a47a-0cc40c570f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbc70dfe-w7wzz" Oct 31 05:40:28.006976 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:28.006948 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce1175fc-39e4-44e7-a47a-0cc40c570f8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbc70dfe-w7wzz\" (UID: \"ce1175fc-39e4-44e7-a47a-0cc40c570f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbc70dfe-w7wzz" Oct 31 05:40:28.028030 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:28.027995 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2dqr\" (UniqueName: \"kubernetes.io/projected/ce1175fc-39e4-44e7-a47a-0cc40c570f8e-kube-api-access-z2dqr\") pod \"maas-api-revoke-sub-bbc70dfe-w7wzz\" (UID: \"ce1175fc-39e4-44e7-a47a-0cc40c570f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbc70dfe-w7wzz" Oct 31 05:40:28.158910 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:28.158811 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbc70dfe-w7wzz" Oct 31 05:40:28.464929 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:28.464892 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbc70dfe-w7wzz"] Oct 31 05:40:28.465247 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:40:28.465218 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce1175fc_39e4_44e7_a47a_0cc40c570f8e.slice/crio-6bec3f112604a378e2d6f52a202d95ed396bdf0788fafdf3490688c7d90b1e7e WatchSource:0}: Error finding container 6bec3f112604a378e2d6f52a202d95ed396bdf0788fafdf3490688c7d90b1e7e: Status 404 returned error can't find the container with id 6bec3f112604a378e2d6f52a202d95ed396bdf0788fafdf3490688c7d90b1e7e Oct 31 05:40:29.046695 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:29.046599 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbc70dfe-w7wzz" event={"ID":"ce1175fc-39e4-44e7-a47a-0cc40c570f8e","Type":"ContainerStarted","Data":"1ed10b14ac775415a74d507b7451da6f0c17ef278c1fa567cc0904204517c5ed"} Oct 31 05:40:29.046695 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:29.046665 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbc70dfe-w7wzz" event={"ID":"ce1175fc-39e4-44e7-a47a-0cc40c570f8e","Type":"ContainerStarted","Data":"6bec3f112604a378e2d6f52a202d95ed396bdf0788fafdf3490688c7d90b1e7e"} Oct 31 05:40:29.086904 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:29.086846 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbc70dfe-w7wzz" podStartSLOduration=2.086827204 podStartE2EDuration="2.086827204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:40:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:40:29.085779089 +0000 UTC m=+5783.842117130" watchObservedRunningTime="2026-10-31 05:40:29.086827204 +0000 UTC m=+5783.843165245" Oct 31 05:40:30.058797 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:30.058745 5019 generic.go:356] "Generic (PLEG): container finished" podID="ce1175fc-39e4-44e7-a47a-0cc40c570f8e" containerID="1ed10b14ac775415a74d507b7451da6f0c17ef278c1fa567cc0904204517c5ed" exitCode=0 Oct 31 05:40:30.059396 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:30.058866 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbc70dfe-w7wzz" event={"ID":"ce1175fc-39e4-44e7-a47a-0cc40c570f8e","Type":"ContainerDied","Data":"1ed10b14ac775415a74d507b7451da6f0c17ef278c1fa567cc0904204517c5ed"} Oct 31 05:40:31.554575 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:31.445893 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 05:40:31.554575 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:31.445956 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hgv9r" podUID="68e465ba-2b5d-4e3c-8b63-7bdb9b695b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 05:40:31.554575 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:31.446507 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-rgj85 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.218:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 05:40:31.554575 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:31.446569 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rgj85" podUID="8430b919-a9dd-4c7e-9266-ef6872cc0b9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.218:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 05:40:31.788616 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:31.788516 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-shld4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 05:40:31.788905 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:31.788868 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-shld4" podUID="a69fd138-9906-4127-ac26-38a21a5b7590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 05:40:31.939420 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:31.937462 5019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.221:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 05:40:31.939420 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:31.937538 5019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lqrr5" podUID="6d68d19b-5392-4308-9e2c-4b5e7593d17b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.221:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 05:40:31.952364 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:31.951650 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbc70dfe-w7wzz" Oct 31 05:40:31.994357 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:31.994001 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce1175fc-39e4-44e7-a47a-0cc40c570f8e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce1175fc-39e4-44e7-a47a-0cc40c570f8e-maas-api-openshift-ca\") pod \"ce1175fc-39e4-44e7-a47a-0cc40c570f8e\" (UID: \"ce1175fc-39e4-44e7-a47a-0cc40c570f8e\") " Oct 31 05:40:31.994357 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:31.994089 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce1175fc-39e4-44e7-a47a-0cc40c570f8e-kube-api-access-z2dqr\" (UniqueName: \"kubernetes.io/projected/ce1175fc-39e4-44e7-a47a-0cc40c570f8e-kube-api-access-z2dqr\") pod \"ce1175fc-39e4-44e7-a47a-0cc40c570f8e\" (UID: \"ce1175fc-39e4-44e7-a47a-0cc40c570f8e\") " Oct 31 05:40:31.994357 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:31.994151 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce1175fc-39e4-44e7-a47a-0cc40c570f8e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce1175fc-39e4-44e7-a47a-0cc40c570f8e-maas-api-xks-ca\") pod \"ce1175fc-39e4-44e7-a47a-0cc40c570f8e\" (UID: \"ce1175fc-39e4-44e7-a47a-0cc40c570f8e\") " Oct 31 05:40:31.998303 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:31.995442 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce1175fc-39e4-44e7-a47a-0cc40c570f8e-maas-api-openshift-ca" pod "ce1175fc-39e4-44e7-a47a-0cc40c570f8e" (UID: "ce1175fc-39e4-44e7-a47a-0cc40c570f8e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:40:32.013542 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:32.013500 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce1175fc-39e4-44e7-a47a-0cc40c570f8e-maas-api-xks-ca" pod "ce1175fc-39e4-44e7-a47a-0cc40c570f8e" (UID: "ce1175fc-39e4-44e7-a47a-0cc40c570f8e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:40:32.127642 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:32.127062 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce1175fc-39e4-44e7-a47a-0cc40c570f8e-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:40:32.130365 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:32.130326 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce1175fc-39e4-44e7-a47a-0cc40c570f8e-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:40:32.150357 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:32.149310 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce1175fc-39e4-44e7-a47a-0cc40c570f8e-kube-api-access-z2dqr" pod "ce1175fc-39e4-44e7-a47a-0cc40c570f8e" (UID: "ce1175fc-39e4-44e7-a47a-0cc40c570f8e"). InnerVolumeSpecName "kube-api-access-z2dqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:40:32.168356 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:32.164200 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbc70dfe-w7wzz" event={"ID":"ce1175fc-39e4-44e7-a47a-0cc40c570f8e","Type":"ContainerDied","Data":"6bec3f112604a378e2d6f52a202d95ed396bdf0788fafdf3490688c7d90b1e7e"} Oct 31 05:40:32.168356 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:32.164270 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bec3f112604a378e2d6f52a202d95ed396bdf0788fafdf3490688c7d90b1e7e" Oct 31 05:40:32.168356 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:32.164369 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbc70dfe-w7wzz" Oct 31 05:40:32.231858 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:32.231811 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2dqr\" (UniqueName: \"kubernetes.io/projected/ce1175fc-39e4-44e7-a47a-0cc40c570f8e-kube-api-access-z2dqr\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:40:38.204863 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:38.204821 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adbbff71-jwc6x"] Oct 31 05:40:38.205457 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:38.205378 5019 cpu_manager.go:414] "containerMap: removing container" podUID="ce1175fc-39e4-44e7-a47a-0cc40c570f8e" containerName="revoke-keys" Oct 31 05:40:38.205457 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:38.205394 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce1175fc-39e4-44e7-a47a-0cc40c570f8e" containerName="revoke-keys" Oct 31 05:40:38.205620 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:38.205488 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce1175fc-39e4-44e7-a47a-0cc40c570f8e" containerName="revoke-keys" Oct 31 05:40:38.208768 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:38.208737 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbbff71-jwc6x" Oct 31 05:40:38.239367 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:38.237284 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:40:38.239367 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:38.237374 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:40:38.239367 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:38.237298 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfhzn\"" Oct 31 05:40:38.239367 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:38.237393 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:40:38.241625 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:38.241602 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adbbff71-jwc6x"] Oct 31 05:40:38.280036 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:38.280002 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtjzd\" (UniqueName: \"kubernetes.io/projected/c7bee25a-4a74-4b09-bcec-8304ad6b180b-kube-api-access-dtjzd\") pod \"maas-api-revoke-sub-adbbff71-jwc6x\" (UID: \"c7bee25a-4a74-4b09-bcec-8304ad6b180b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbbff71-jwc6x" Oct 31 05:40:38.280036 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:38.280042 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7bee25a-4a74-4b09-bcec-8304ad6b180b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-adbbff71-jwc6x\" (UID: \"c7bee25a-4a74-4b09-bcec-8304ad6b180b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbbff71-jwc6x" Oct 31 05:40:38.280269 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:38.280116 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7bee25a-4a74-4b09-bcec-8304ad6b180b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-adbbff71-jwc6x\" (UID: \"c7bee25a-4a74-4b09-bcec-8304ad6b180b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbbff71-jwc6x" Oct 31 05:40:38.380723 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:38.380674 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtjzd\" (UniqueName: \"kubernetes.io/projected/c7bee25a-4a74-4b09-bcec-8304ad6b180b-kube-api-access-dtjzd\") pod \"maas-api-revoke-sub-adbbff71-jwc6x\" (UID: \"c7bee25a-4a74-4b09-bcec-8304ad6b180b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbbff71-jwc6x" Oct 31 05:40:38.380723 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:38.380729 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7bee25a-4a74-4b09-bcec-8304ad6b180b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-adbbff71-jwc6x\" (UID: \"c7bee25a-4a74-4b09-bcec-8304ad6b180b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbbff71-jwc6x" Oct 31 05:40:38.381083 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:38.380772 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7bee25a-4a74-4b09-bcec-8304ad6b180b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-adbbff71-jwc6x\" (UID: \"c7bee25a-4a74-4b09-bcec-8304ad6b180b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbbff71-jwc6x" Oct 31 05:40:38.381435 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:38.381405 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7bee25a-4a74-4b09-bcec-8304ad6b180b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-adbbff71-jwc6x\" (UID: \"c7bee25a-4a74-4b09-bcec-8304ad6b180b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbbff71-jwc6x" Oct 31 05:40:38.383292 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:38.383272 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7bee25a-4a74-4b09-bcec-8304ad6b180b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-adbbff71-jwc6x\" (UID: \"c7bee25a-4a74-4b09-bcec-8304ad6b180b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbbff71-jwc6x" Oct 31 05:40:38.403926 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:38.403901 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtjzd\" (UniqueName: \"kubernetes.io/projected/c7bee25a-4a74-4b09-bcec-8304ad6b180b-kube-api-access-dtjzd\") pod \"maas-api-revoke-sub-adbbff71-jwc6x\" (UID: \"c7bee25a-4a74-4b09-bcec-8304ad6b180b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbbff71-jwc6x" Oct 31 05:40:38.522945 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:38.522905 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbbff71-jwc6x" Oct 31 05:40:38.828069 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:38.828039 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adbbff71-jwc6x"] Oct 31 05:40:38.830138 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:40:38.830107 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7bee25a_4a74_4b09_bcec_8304ad6b180b.slice/crio-7c7993bc2332a68a6062f5512b6b6ee9365ff91f8ba7a419472d2fef22084129 WatchSource:0}: Error finding container 7c7993bc2332a68a6062f5512b6b6ee9365ff91f8ba7a419472d2fef22084129: Status 404 returned error can't find the container with id 7c7993bc2332a68a6062f5512b6b6ee9365ff91f8ba7a419472d2fef22084129 Oct 31 05:40:39.212063 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:39.212028 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbbff71-jwc6x" event={"ID":"c7bee25a-4a74-4b09-bcec-8304ad6b180b","Type":"ContainerStarted","Data":"bd67d231f580fffe69c321a332a0fa5729e2d5f267ba7b436ab6fc7d9e73ed55"} Oct 31 05:40:39.212510 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:39.212081 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbbff71-jwc6x" event={"ID":"c7bee25a-4a74-4b09-bcec-8304ad6b180b","Type":"ContainerStarted","Data":"7c7993bc2332a68a6062f5512b6b6ee9365ff91f8ba7a419472d2fef22084129"} Oct 31 05:40:39.264534 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:39.264477 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbbff71-jwc6x" podStartSLOduration=1.264463383 podStartE2EDuration="1.264463383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:40:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:40:39.251713271 +0000 UTC m=+5794.008051314" watchObservedRunningTime="2026-10-31 05:40:39.264463383 +0000 UTC m=+5794.020801470" Oct 31 05:40:40.224559 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:40.224508 5019 generic.go:356] "Generic (PLEG): container finished" podID="c7bee25a-4a74-4b09-bcec-8304ad6b180b" containerID="bd67d231f580fffe69c321a332a0fa5729e2d5f267ba7b436ab6fc7d9e73ed55" exitCode=0 Oct 31 05:40:40.225096 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:40.224643 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbbff71-jwc6x" event={"ID":"c7bee25a-4a74-4b09-bcec-8304ad6b180b","Type":"ContainerDied","Data":"bd67d231f580fffe69c321a332a0fa5729e2d5f267ba7b436ab6fc7d9e73ed55"} Oct 31 05:40:42.257232 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:42.256899 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbbff71-jwc6x" Oct 31 05:40:42.340357 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:42.340240 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbbff71-jwc6x" event={"ID":"c7bee25a-4a74-4b09-bcec-8304ad6b180b","Type":"ContainerDied","Data":"7c7993bc2332a68a6062f5512b6b6ee9365ff91f8ba7a419472d2fef22084129"} Oct 31 05:40:42.340625 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:42.340602 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c7993bc2332a68a6062f5512b6b6ee9365ff91f8ba7a419472d2fef22084129" Oct 31 05:40:42.340824 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:42.340808 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbbff71-jwc6x" Oct 31 05:40:42.445013 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:42.444955 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7bee25a-4a74-4b09-bcec-8304ad6b180b-kube-api-access-dtjzd\" (UniqueName: \"kubernetes.io/projected/c7bee25a-4a74-4b09-bcec-8304ad6b180b-kube-api-access-dtjzd\") pod \"c7bee25a-4a74-4b09-bcec-8304ad6b180b\" (UID: \"c7bee25a-4a74-4b09-bcec-8304ad6b180b\") " Oct 31 05:40:42.450550 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:42.450520 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7bee25a-4a74-4b09-bcec-8304ad6b180b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7bee25a-4a74-4b09-bcec-8304ad6b180b-maas-api-xks-ca\") pod \"c7bee25a-4a74-4b09-bcec-8304ad6b180b\" (UID: \"c7bee25a-4a74-4b09-bcec-8304ad6b180b\") " Oct 31 05:40:42.451113 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:42.451092 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7bee25a-4a74-4b09-bcec-8304ad6b180b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7bee25a-4a74-4b09-bcec-8304ad6b180b-maas-api-openshift-ca\") pod \"c7bee25a-4a74-4b09-bcec-8304ad6b180b\" (UID: \"c7bee25a-4a74-4b09-bcec-8304ad6b180b\") " Oct 31 05:40:42.457206 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:42.456602 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7bee25a-4a74-4b09-bcec-8304ad6b180b-maas-api-xks-ca" pod "c7bee25a-4a74-4b09-bcec-8304ad6b180b" (UID: "c7bee25a-4a74-4b09-bcec-8304ad6b180b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:40:42.458261 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:42.456742 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7bee25a-4a74-4b09-bcec-8304ad6b180b-maas-api-openshift-ca" pod "c7bee25a-4a74-4b09-bcec-8304ad6b180b" (UID: "c7bee25a-4a74-4b09-bcec-8304ad6b180b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:40:42.458402 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:42.458358 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7bee25a-4a74-4b09-bcec-8304ad6b180b-kube-api-access-dtjzd" pod "c7bee25a-4a74-4b09-bcec-8304ad6b180b" (UID: "c7bee25a-4a74-4b09-bcec-8304ad6b180b"). InnerVolumeSpecName "kube-api-access-dtjzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:40:42.558178 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:42.558133 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7bee25a-4a74-4b09-bcec-8304ad6b180b-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:40:42.558178 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:42.558171 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7bee25a-4a74-4b09-bcec-8304ad6b180b-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:40:42.558178 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:42.558182 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtjzd\" (UniqueName: \"kubernetes.io/projected/c7bee25a-4a74-4b09-bcec-8304ad6b180b-kube-api-access-dtjzd\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:40:48.552462 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:48.552380 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c545349-srpxt"] Oct 31 05:40:48.553021 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:48.553001 5019 cpu_manager.go:414] "containerMap: removing container" podUID="c7bee25a-4a74-4b09-bcec-8304ad6b180b" containerName="revoke-keys" Oct 31 05:40:48.553079 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:48.553023 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7bee25a-4a74-4b09-bcec-8304ad6b180b" containerName="revoke-keys" Oct 31 05:40:48.553150 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:48.553139 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7bee25a-4a74-4b09-bcec-8304ad6b180b" containerName="revoke-keys" Oct 31 05:40:48.557011 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:48.556981 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c545349-srpxt" Oct 31 05:40:48.576200 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:48.576168 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c545349-srpxt"] Oct 31 05:40:48.583661 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:48.583630 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:40:48.583990 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:48.583949 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:40:48.584121 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:48.584092 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:40:48.584594 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:48.584566 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfhzn\"" Oct 31 05:40:48.619368 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:48.619311 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e43c3df-d577-4a9a-bfd4-64df51d97a5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c545349-srpxt\" (UID: \"1e43c3df-d577-4a9a-bfd4-64df51d97a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c545349-srpxt" Oct 31 05:40:48.619882 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:48.619409 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e43c3df-d577-4a9a-bfd4-64df51d97a5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c545349-srpxt\" (UID: \"1e43c3df-d577-4a9a-bfd4-64df51d97a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c545349-srpxt" Oct 31 05:40:48.619882 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:48.619494 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdd5v\" (UniqueName: \"kubernetes.io/projected/1e43c3df-d577-4a9a-bfd4-64df51d97a5c-kube-api-access-mdd5v\") pod \"maas-api-revoke-sub-9c545349-srpxt\" (UID: \"1e43c3df-d577-4a9a-bfd4-64df51d97a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c545349-srpxt" Oct 31 05:40:48.720650 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:48.720610 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e43c3df-d577-4a9a-bfd4-64df51d97a5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c545349-srpxt\" (UID: \"1e43c3df-d577-4a9a-bfd4-64df51d97a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c545349-srpxt" Oct 31 05:40:48.720854 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:48.720660 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e43c3df-d577-4a9a-bfd4-64df51d97a5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c545349-srpxt\" (UID: \"1e43c3df-d577-4a9a-bfd4-64df51d97a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c545349-srpxt" Oct 31 05:40:48.720854 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:48.720750 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdd5v\" (UniqueName: \"kubernetes.io/projected/1e43c3df-d577-4a9a-bfd4-64df51d97a5c-kube-api-access-mdd5v\") pod \"maas-api-revoke-sub-9c545349-srpxt\" (UID: \"1e43c3df-d577-4a9a-bfd4-64df51d97a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c545349-srpxt" Oct 31 05:40:48.721503 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:48.721480 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e43c3df-d577-4a9a-bfd4-64df51d97a5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c545349-srpxt\" (UID: \"1e43c3df-d577-4a9a-bfd4-64df51d97a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c545349-srpxt" Oct 31 05:40:48.723078 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:48.723058 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e43c3df-d577-4a9a-bfd4-64df51d97a5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c545349-srpxt\" (UID: \"1e43c3df-d577-4a9a-bfd4-64df51d97a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c545349-srpxt" Oct 31 05:40:48.744000 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:48.743969 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdd5v\" (UniqueName: \"kubernetes.io/projected/1e43c3df-d577-4a9a-bfd4-64df51d97a5c-kube-api-access-mdd5v\") pod \"maas-api-revoke-sub-9c545349-srpxt\" (UID: \"1e43c3df-d577-4a9a-bfd4-64df51d97a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c545349-srpxt" Oct 31 05:40:48.871099 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:48.871004 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c545349-srpxt" Oct 31 05:40:49.173352 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:49.173305 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c545349-srpxt"] Oct 31 05:40:49.175398 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:40:49.175367 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e43c3df_d577_4a9a_bfd4_64df51d97a5c.slice/crio-09995b367a06965cfd8e2a3e36433224ebb7a5c3097ec5bde216f898ab79c379 WatchSource:0}: Error finding container 09995b367a06965cfd8e2a3e36433224ebb7a5c3097ec5bde216f898ab79c379: Status 404 returned error can't find the container with id 09995b367a06965cfd8e2a3e36433224ebb7a5c3097ec5bde216f898ab79c379 Oct 31 05:40:49.384388 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:49.384349 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c545349-srpxt" event={"ID":"1e43c3df-d577-4a9a-bfd4-64df51d97a5c","Type":"ContainerStarted","Data":"09995b367a06965cfd8e2a3e36433224ebb7a5c3097ec5bde216f898ab79c379"} Oct 31 05:40:50.414947 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:50.414851 5019 generic.go:356] "Generic (PLEG): container finished" podID="1e43c3df-d577-4a9a-bfd4-64df51d97a5c" containerID="12b81e2b4fc5105be05bbda05ce913c398f0202e59dc566620a3fb83b1ab69eb" exitCode=0 Oct 31 05:40:50.415606 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:50.414988 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c545349-srpxt" event={"ID":"1e43c3df-d577-4a9a-bfd4-64df51d97a5c","Type":"ContainerDied","Data":"12b81e2b4fc5105be05bbda05ce913c398f0202e59dc566620a3fb83b1ab69eb"} Oct 31 05:40:52.270357 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:52.266736 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c545349-srpxt" Oct 31 05:40:52.434270 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:52.434165 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e43c3df-d577-4a9a-bfd4-64df51d97a5c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e43c3df-d577-4a9a-bfd4-64df51d97a5c-maas-api-openshift-ca\") pod \"1e43c3df-d577-4a9a-bfd4-64df51d97a5c\" (UID: \"1e43c3df-d577-4a9a-bfd4-64df51d97a5c\") " Oct 31 05:40:52.434640 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:52.434610 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e43c3df-d577-4a9a-bfd4-64df51d97a5c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e43c3df-d577-4a9a-bfd4-64df51d97a5c-maas-api-xks-ca\") pod \"1e43c3df-d577-4a9a-bfd4-64df51d97a5c\" (UID: \"1e43c3df-d577-4a9a-bfd4-64df51d97a5c\") " Oct 31 05:40:52.434831 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:52.434814 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e43c3df-d577-4a9a-bfd4-64df51d97a5c-kube-api-access-mdd5v\" (UniqueName: \"kubernetes.io/projected/1e43c3df-d577-4a9a-bfd4-64df51d97a5c-kube-api-access-mdd5v\") pod \"1e43c3df-d577-4a9a-bfd4-64df51d97a5c\" (UID: \"1e43c3df-d577-4a9a-bfd4-64df51d97a5c\") " Oct 31 05:40:52.503348 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:52.503289 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e43c3df-d577-4a9a-bfd4-64df51d97a5c-maas-api-openshift-ca" pod "1e43c3df-d577-4a9a-bfd4-64df51d97a5c" (UID: "1e43c3df-d577-4a9a-bfd4-64df51d97a5c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:40:52.530378 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:52.530320 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c545349-srpxt" event={"ID":"1e43c3df-d577-4a9a-bfd4-64df51d97a5c","Type":"ContainerDied","Data":"09995b367a06965cfd8e2a3e36433224ebb7a5c3097ec5bde216f898ab79c379"} Oct 31 05:40:52.530665 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:52.530644 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09995b367a06965cfd8e2a3e36433224ebb7a5c3097ec5bde216f898ab79c379" Oct 31 05:40:52.530862 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:52.530844 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c545349-srpxt" Oct 31 05:40:52.536634 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:52.536577 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e43c3df-d577-4a9a-bfd4-64df51d97a5c-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:40:52.536798 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:52.536770 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e43c3df-d577-4a9a-bfd4-64df51d97a5c-kube-api-access-mdd5v" pod "1e43c3df-d577-4a9a-bfd4-64df51d97a5c" (UID: "1e43c3df-d577-4a9a-bfd4-64df51d97a5c"). InnerVolumeSpecName "kube-api-access-mdd5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:40:52.568585 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:52.568532 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e43c3df-d577-4a9a-bfd4-64df51d97a5c-maas-api-xks-ca" pod "1e43c3df-d577-4a9a-bfd4-64df51d97a5c" (UID: "1e43c3df-d577-4a9a-bfd4-64df51d97a5c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:40:52.617701 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:40:52.617626 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:40:52.639269 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:52.639230 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e43c3df-d577-4a9a-bfd4-64df51d97a5c-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:40:52.639576 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:52.639556 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdd5v\" (UniqueName: \"kubernetes.io/projected/1e43c3df-d577-4a9a-bfd4-64df51d97a5c-kube-api-access-mdd5v\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:40:58.903148 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:58.903108 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c9a2c83-7jf8w"] Oct 31 05:40:58.903673 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:58.903593 5019 cpu_manager.go:414] "containerMap: removing container" podUID="1e43c3df-d577-4a9a-bfd4-64df51d97a5c" containerName="revoke-keys" Oct 31 05:40:58.903673 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:58.903606 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e43c3df-d577-4a9a-bfd4-64df51d97a5c" containerName="revoke-keys" Oct 31 05:40:58.903756 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:58.903717 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e43c3df-d577-4a9a-bfd4-64df51d97a5c" containerName="revoke-keys" Oct 31 05:40:58.906825 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:58.906801 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9a2c83-7jf8w" Oct 31 05:40:58.922787 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:58.922694 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:40:58.926940 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:58.926896 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c9a2c83-7jf8w"] Oct 31 05:40:58.934321 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:58.934288 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfhzn\"" Oct 31 05:40:58.934797 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:58.934772 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:40:58.934954 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:58.934841 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:40:59.001634 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:59.001594 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/038cc95f-9b08-4a57-9229-5b74acb9e968-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c9a2c83-7jf8w\" (UID: \"038cc95f-9b08-4a57-9229-5b74acb9e968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9a2c83-7jf8w" Oct 31 05:40:59.001857 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:59.001723 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/038cc95f-9b08-4a57-9229-5b74acb9e968-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c9a2c83-7jf8w\" (UID: \"038cc95f-9b08-4a57-9229-5b74acb9e968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9a2c83-7jf8w" Oct 31 05:40:59.001857 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:59.001785 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2v7g\" (UniqueName: \"kubernetes.io/projected/038cc95f-9b08-4a57-9229-5b74acb9e968-kube-api-access-q2v7g\") pod \"maas-api-revoke-sub-1c9a2c83-7jf8w\" (UID: \"038cc95f-9b08-4a57-9229-5b74acb9e968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9a2c83-7jf8w" Oct 31 05:40:59.102976 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:59.102935 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/038cc95f-9b08-4a57-9229-5b74acb9e968-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c9a2c83-7jf8w\" (UID: \"038cc95f-9b08-4a57-9229-5b74acb9e968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9a2c83-7jf8w" Oct 31 05:40:59.103209 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:59.103004 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/038cc95f-9b08-4a57-9229-5b74acb9e968-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c9a2c83-7jf8w\" (UID: \"038cc95f-9b08-4a57-9229-5b74acb9e968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9a2c83-7jf8w" Oct 31 05:40:59.103209 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:59.103046 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2v7g\" (UniqueName: \"kubernetes.io/projected/038cc95f-9b08-4a57-9229-5b74acb9e968-kube-api-access-q2v7g\") pod \"maas-api-revoke-sub-1c9a2c83-7jf8w\" (UID: \"038cc95f-9b08-4a57-9229-5b74acb9e968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9a2c83-7jf8w" Oct 31 05:40:59.103825 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:59.103801 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/038cc95f-9b08-4a57-9229-5b74acb9e968-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c9a2c83-7jf8w\" (UID: \"038cc95f-9b08-4a57-9229-5b74acb9e968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9a2c83-7jf8w" Oct 31 05:40:59.105510 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:59.105488 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/038cc95f-9b08-4a57-9229-5b74acb9e968-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c9a2c83-7jf8w\" (UID: \"038cc95f-9b08-4a57-9229-5b74acb9e968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9a2c83-7jf8w" Oct 31 05:40:59.126979 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:59.126944 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2v7g\" (UniqueName: \"kubernetes.io/projected/038cc95f-9b08-4a57-9229-5b74acb9e968-kube-api-access-q2v7g\") pod \"maas-api-revoke-sub-1c9a2c83-7jf8w\" (UID: \"038cc95f-9b08-4a57-9229-5b74acb9e968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9a2c83-7jf8w" Oct 31 05:40:59.226973 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:59.226934 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9a2c83-7jf8w" Oct 31 05:40:59.528225 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:59.528194 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c9a2c83-7jf8w"] Oct 31 05:40:59.529449 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:40:59.529421 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod038cc95f_9b08_4a57_9229_5b74acb9e968.slice/crio-446e0356ecd7079e119b8fbacf56b050f9dc86336ae3d83edbeb265903afe328 WatchSource:0}: Error finding container 446e0356ecd7079e119b8fbacf56b050f9dc86336ae3d83edbeb265903afe328: Status 404 returned error can't find the container with id 446e0356ecd7079e119b8fbacf56b050f9dc86336ae3d83edbeb265903afe328 Oct 31 05:40:59.577867 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:40:59.577827 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9a2c83-7jf8w" event={"ID":"038cc95f-9b08-4a57-9229-5b74acb9e968","Type":"ContainerStarted","Data":"446e0356ecd7079e119b8fbacf56b050f9dc86336ae3d83edbeb265903afe328"} Oct 31 05:41:00.591145 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:00.591039 5019 generic.go:356] "Generic (PLEG): container finished" podID="038cc95f-9b08-4a57-9229-5b74acb9e968" containerID="4ccdd4ac6048a71791bde9a8244ba3957b6e4f4503cc60ce1c26285fb75637f3" exitCode=0 Oct 31 05:41:00.591145 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:00.591109 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9a2c83-7jf8w" event={"ID":"038cc95f-9b08-4a57-9229-5b74acb9e968","Type":"ContainerDied","Data":"4ccdd4ac6048a71791bde9a8244ba3957b6e4f4503cc60ce1c26285fb75637f3"} Oct 31 05:41:02.393619 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:02.393575 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9a2c83-7jf8w" Oct 31 05:41:02.519361 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:02.495966 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/038cc95f-9b08-4a57-9229-5b74acb9e968-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/038cc95f-9b08-4a57-9229-5b74acb9e968-maas-api-xks-ca\") pod \"038cc95f-9b08-4a57-9229-5b74acb9e968\" (UID: \"038cc95f-9b08-4a57-9229-5b74acb9e968\") " Oct 31 05:41:02.519774 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:02.519737 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/038cc95f-9b08-4a57-9229-5b74acb9e968-kube-api-access-q2v7g\" (UniqueName: \"kubernetes.io/projected/038cc95f-9b08-4a57-9229-5b74acb9e968-kube-api-access-q2v7g\") pod \"038cc95f-9b08-4a57-9229-5b74acb9e968\" (UID: \"038cc95f-9b08-4a57-9229-5b74acb9e968\") " Oct 31 05:41:02.519997 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:02.519978 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/038cc95f-9b08-4a57-9229-5b74acb9e968-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/038cc95f-9b08-4a57-9229-5b74acb9e968-maas-api-openshift-ca\") pod \"038cc95f-9b08-4a57-9229-5b74acb9e968\" (UID: \"038cc95f-9b08-4a57-9229-5b74acb9e968\") " Oct 31 05:41:02.543677 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:02.543625 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/038cc95f-9b08-4a57-9229-5b74acb9e968-maas-api-openshift-ca" pod "038cc95f-9b08-4a57-9229-5b74acb9e968" (UID: "038cc95f-9b08-4a57-9229-5b74acb9e968"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:41:02.560931 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:02.560863 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/038cc95f-9b08-4a57-9229-5b74acb9e968-maas-api-xks-ca" pod "038cc95f-9b08-4a57-9229-5b74acb9e968" (UID: "038cc95f-9b08-4a57-9229-5b74acb9e968"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:41:02.619768 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:02.619706 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/038cc95f-9b08-4a57-9229-5b74acb9e968-kube-api-access-q2v7g" pod "038cc95f-9b08-4a57-9229-5b74acb9e968" (UID: "038cc95f-9b08-4a57-9229-5b74acb9e968"). InnerVolumeSpecName "kube-api-access-q2v7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:41:02.626386 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:02.621252 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/038cc95f-9b08-4a57-9229-5b74acb9e968-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:41:02.626386 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:02.621292 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2v7g\" (UniqueName: \"kubernetes.io/projected/038cc95f-9b08-4a57-9229-5b74acb9e968-kube-api-access-q2v7g\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:41:02.626386 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:02.621308 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/038cc95f-9b08-4a57-9229-5b74acb9e968-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:41:02.678363 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:02.674776 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9a2c83-7jf8w" Oct 31 05:41:02.679043 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:02.678654 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9a2c83-7jf8w" event={"ID":"038cc95f-9b08-4a57-9229-5b74acb9e968","Type":"ContainerDied","Data":"446e0356ecd7079e119b8fbacf56b050f9dc86336ae3d83edbeb265903afe328"} Oct 31 05:41:02.679043 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:02.678753 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="446e0356ecd7079e119b8fbacf56b050f9dc86336ae3d83edbeb265903afe328" Oct 31 05:41:09.262138 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:09.262096 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba23861f-fkglb"] Oct 31 05:41:09.262794 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:09.262772 5019 cpu_manager.go:414] "containerMap: removing container" podUID="038cc95f-9b08-4a57-9229-5b74acb9e968" containerName="revoke-keys" Oct 31 05:41:09.262851 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:09.262796 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="038cc95f-9b08-4a57-9229-5b74acb9e968" containerName="revoke-keys" Oct 31 05:41:09.262887 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:09.262878 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="038cc95f-9b08-4a57-9229-5b74acb9e968" containerName="revoke-keys" Oct 31 05:41:09.266420 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:09.266398 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba23861f-fkglb" Oct 31 05:41:09.286156 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:09.286119 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba23861f-fkglb"] Oct 31 05:41:09.293384 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:09.293344 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:41:09.293606 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:09.293393 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:41:09.293606 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:09.293361 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:41:09.293606 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:09.293474 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfhzn\"" Oct 31 05:41:09.293937 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:09.293918 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79721c0c-c0d9-478d-b2cd-64f1efaf2cf9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba23861f-fkglb\" (UID: \"79721c0c-c0d9-478d-b2cd-64f1efaf2cf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba23861f-fkglb" Oct 31 05:41:09.294038 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:09.293981 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79721c0c-c0d9-478d-b2cd-64f1efaf2cf9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba23861f-fkglb\" (UID: \"79721c0c-c0d9-478d-b2cd-64f1efaf2cf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba23861f-fkglb" Oct 31 05:41:09.294109 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:09.294088 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68m5j\" (UniqueName: \"kubernetes.io/projected/79721c0c-c0d9-478d-b2cd-64f1efaf2cf9-kube-api-access-68m5j\") pod \"maas-api-revoke-sub-ba23861f-fkglb\" (UID: \"79721c0c-c0d9-478d-b2cd-64f1efaf2cf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba23861f-fkglb" Oct 31 05:41:09.395734 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:09.395685 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79721c0c-c0d9-478d-b2cd-64f1efaf2cf9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba23861f-fkglb\" (UID: \"79721c0c-c0d9-478d-b2cd-64f1efaf2cf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba23861f-fkglb" Oct 31 05:41:09.395943 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:09.395748 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79721c0c-c0d9-478d-b2cd-64f1efaf2cf9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba23861f-fkglb\" (UID: \"79721c0c-c0d9-478d-b2cd-64f1efaf2cf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba23861f-fkglb" Oct 31 05:41:09.395943 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:09.395786 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68m5j\" (UniqueName: \"kubernetes.io/projected/79721c0c-c0d9-478d-b2cd-64f1efaf2cf9-kube-api-access-68m5j\") pod \"maas-api-revoke-sub-ba23861f-fkglb\" (UID: \"79721c0c-c0d9-478d-b2cd-64f1efaf2cf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba23861f-fkglb" Oct 31 05:41:09.396605 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:09.396567 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79721c0c-c0d9-478d-b2cd-64f1efaf2cf9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba23861f-fkglb\" (UID: \"79721c0c-c0d9-478d-b2cd-64f1efaf2cf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba23861f-fkglb" Oct 31 05:41:09.398470 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:09.398444 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79721c0c-c0d9-478d-b2cd-64f1efaf2cf9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba23861f-fkglb\" (UID: \"79721c0c-c0d9-478d-b2cd-64f1efaf2cf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba23861f-fkglb" Oct 31 05:41:09.419890 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:09.419853 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68m5j\" (UniqueName: \"kubernetes.io/projected/79721c0c-c0d9-478d-b2cd-64f1efaf2cf9-kube-api-access-68m5j\") pod \"maas-api-revoke-sub-ba23861f-fkglb\" (UID: \"79721c0c-c0d9-478d-b2cd-64f1efaf2cf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba23861f-fkglb" Oct 31 05:41:09.580844 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:09.580747 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba23861f-fkglb" Oct 31 05:41:09.886029 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:09.885992 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba23861f-fkglb"] Oct 31 05:41:09.887224 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:41:09.887179 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79721c0c_c0d9_478d_b2cd_64f1efaf2cf9.slice/crio-7f9bd5b74b4a47634d55af17628a4968579d20119ce2822baa4ce030a1bc0796 WatchSource:0}: Error finding container 7f9bd5b74b4a47634d55af17628a4968579d20119ce2822baa4ce030a1bc0796: Status 404 returned error can't find the container with id 7f9bd5b74b4a47634d55af17628a4968579d20119ce2822baa4ce030a1bc0796 Oct 31 05:41:10.724407 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:10.724356 5019 generic.go:356] "Generic (PLEG): container finished" podID="79721c0c-c0d9-478d-b2cd-64f1efaf2cf9" containerID="d5606d68924af07f03e6c5fd8ff8719e83a99af2849123103b108c56470175b8" exitCode=0 Oct 31 05:41:10.724938 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:10.724554 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba23861f-fkglb" event={"ID":"79721c0c-c0d9-478d-b2cd-64f1efaf2cf9","Type":"ContainerDied","Data":"d5606d68924af07f03e6c5fd8ff8719e83a99af2849123103b108c56470175b8"} Oct 31 05:41:10.724938 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:10.724606 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba23861f-fkglb" event={"ID":"79721c0c-c0d9-478d-b2cd-64f1efaf2cf9","Type":"ContainerStarted","Data":"7f9bd5b74b4a47634d55af17628a4968579d20119ce2822baa4ce030a1bc0796"} Oct 31 05:41:12.600588 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:12.600495 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba23861f-fkglb" Oct 31 05:41:12.619165 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:12.619099 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79721c0c-c0d9-478d-b2cd-64f1efaf2cf9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79721c0c-c0d9-478d-b2cd-64f1efaf2cf9-maas-api-openshift-ca\") pod \"79721c0c-c0d9-478d-b2cd-64f1efaf2cf9\" (UID: \"79721c0c-c0d9-478d-b2cd-64f1efaf2cf9\") " Oct 31 05:41:12.626811 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:12.626678 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79721c0c-c0d9-478d-b2cd-64f1efaf2cf9-maas-api-openshift-ca" pod "79721c0c-c0d9-478d-b2cd-64f1efaf2cf9" (UID: "79721c0c-c0d9-478d-b2cd-64f1efaf2cf9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:41:12.627642 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:12.627112 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79721c0c-c0d9-478d-b2cd-64f1efaf2cf9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79721c0c-c0d9-478d-b2cd-64f1efaf2cf9-maas-api-xks-ca\") pod \"79721c0c-c0d9-478d-b2cd-64f1efaf2cf9\" (UID: \"79721c0c-c0d9-478d-b2cd-64f1efaf2cf9\") " Oct 31 05:41:12.627986 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:12.627961 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79721c0c-c0d9-478d-b2cd-64f1efaf2cf9-kube-api-access-68m5j\" (UniqueName: \"kubernetes.io/projected/79721c0c-c0d9-478d-b2cd-64f1efaf2cf9-kube-api-access-68m5j\") pod \"79721c0c-c0d9-478d-b2cd-64f1efaf2cf9\" (UID: \"79721c0c-c0d9-478d-b2cd-64f1efaf2cf9\") " Oct 31 05:41:12.629120 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:12.629098 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79721c0c-c0d9-478d-b2cd-64f1efaf2cf9-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:41:12.717364 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:12.713526 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79721c0c-c0d9-478d-b2cd-64f1efaf2cf9-maas-api-xks-ca" pod "79721c0c-c0d9-478d-b2cd-64f1efaf2cf9" (UID: "79721c0c-c0d9-478d-b2cd-64f1efaf2cf9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:41:12.717364 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:12.713672 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79721c0c-c0d9-478d-b2cd-64f1efaf2cf9-kube-api-access-68m5j" pod "79721c0c-c0d9-478d-b2cd-64f1efaf2cf9" (UID: "79721c0c-c0d9-478d-b2cd-64f1efaf2cf9"). InnerVolumeSpecName "kube-api-access-68m5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:41:12.865416 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:12.864667 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79721c0c-c0d9-478d-b2cd-64f1efaf2cf9-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:41:12.865416 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:12.864714 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68m5j\" (UniqueName: \"kubernetes.io/projected/79721c0c-c0d9-478d-b2cd-64f1efaf2cf9-kube-api-access-68m5j\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:41:12.937843 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:12.937793 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba23861f-fkglb" event={"ID":"79721c0c-c0d9-478d-b2cd-64f1efaf2cf9","Type":"ContainerDied","Data":"7f9bd5b74b4a47634d55af17628a4968579d20119ce2822baa4ce030a1bc0796"} Oct 31 05:41:12.938280 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:12.938250 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f9bd5b74b4a47634d55af17628a4968579d20119ce2822baa4ce030a1bc0796" Oct 31 05:41:12.938501 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:12.938190 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba23861f-fkglb" Oct 31 05:41:15.432654 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:15.432614 5019 scope.go:122] "RemoveContainer" containerID="58bdc6a2cd3e8290e7d9e55f90207281c3b097788683075d2b602081bb497d46" Oct 31 05:41:15.442682 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:15.442648 5019 scope.go:122] "RemoveContainer" containerID="318c9de62da05f30fbac220c9adc3071881304dd95a95884bc9825d7cc87080e" Oct 31 05:41:19.638074 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:19.638030 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94d5ea09-dvgs6"] Oct 31 05:41:19.638549 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:19.638540 5019 cpu_manager.go:414] "containerMap: removing container" podUID="79721c0c-c0d9-478d-b2cd-64f1efaf2cf9" containerName="revoke-keys" Oct 31 05:41:19.638597 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:19.638554 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79721c0c-c0d9-478d-b2cd-64f1efaf2cf9" containerName="revoke-keys" Oct 31 05:41:19.638691 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:19.638679 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="79721c0c-c0d9-478d-b2cd-64f1efaf2cf9" containerName="revoke-keys" Oct 31 05:41:19.643914 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:19.643875 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94d5ea09-dvgs6" Oct 31 05:41:19.662101 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:19.662065 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94d5ea09-dvgs6"] Oct 31 05:41:19.669933 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:19.669900 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:41:19.670168 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:19.670075 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:41:19.670168 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:19.670160 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfhzn\"" Oct 31 05:41:19.670430 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:19.670410 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:41:19.738249 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:19.738194 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb9eedd7-27f4-4b59-8760-082faaf67650-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94d5ea09-dvgs6\" (UID: \"bb9eedd7-27f4-4b59-8760-082faaf67650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94d5ea09-dvgs6" Oct 31 05:41:19.738494 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:19.738367 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t8wg\" (UniqueName: \"kubernetes.io/projected/bb9eedd7-27f4-4b59-8760-082faaf67650-kube-api-access-4t8wg\") pod \"maas-api-revoke-sub-94d5ea09-dvgs6\" (UID: \"bb9eedd7-27f4-4b59-8760-082faaf67650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94d5ea09-dvgs6" Oct 31 05:41:19.738494 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:19.738416 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb9eedd7-27f4-4b59-8760-082faaf67650-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94d5ea09-dvgs6\" (UID: \"bb9eedd7-27f4-4b59-8760-082faaf67650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94d5ea09-dvgs6" Oct 31 05:41:19.839320 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:19.839278 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t8wg\" (UniqueName: \"kubernetes.io/projected/bb9eedd7-27f4-4b59-8760-082faaf67650-kube-api-access-4t8wg\") pod \"maas-api-revoke-sub-94d5ea09-dvgs6\" (UID: \"bb9eedd7-27f4-4b59-8760-082faaf67650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94d5ea09-dvgs6" Oct 31 05:41:19.839320 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:19.839322 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb9eedd7-27f4-4b59-8760-082faaf67650-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94d5ea09-dvgs6\" (UID: \"bb9eedd7-27f4-4b59-8760-082faaf67650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94d5ea09-dvgs6" Oct 31 05:41:19.839587 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:19.839388 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb9eedd7-27f4-4b59-8760-082faaf67650-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94d5ea09-dvgs6\" (UID: \"bb9eedd7-27f4-4b59-8760-082faaf67650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94d5ea09-dvgs6" Oct 31 05:41:19.840151 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:19.840119 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb9eedd7-27f4-4b59-8760-082faaf67650-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94d5ea09-dvgs6\" (UID: \"bb9eedd7-27f4-4b59-8760-082faaf67650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94d5ea09-dvgs6" Oct 31 05:41:19.842211 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:19.842179 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb9eedd7-27f4-4b59-8760-082faaf67650-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94d5ea09-dvgs6\" (UID: \"bb9eedd7-27f4-4b59-8760-082faaf67650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94d5ea09-dvgs6" Oct 31 05:41:19.863540 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:19.863506 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t8wg\" (UniqueName: \"kubernetes.io/projected/bb9eedd7-27f4-4b59-8760-082faaf67650-kube-api-access-4t8wg\") pod \"maas-api-revoke-sub-94d5ea09-dvgs6\" (UID: \"bb9eedd7-27f4-4b59-8760-082faaf67650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94d5ea09-dvgs6" Oct 31 05:41:19.961791 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:19.961687 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94d5ea09-dvgs6" Oct 31 05:41:20.293794 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:20.293747 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94d5ea09-dvgs6"] Oct 31 05:41:20.293968 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:41:20.293740 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb9eedd7_27f4_4b59_8760_082faaf67650.slice/crio-ab839b1e7f125f50dfcb5a75f816477ebfbf5287d6ace9c91020ed12b43293de WatchSource:0}: Error finding container ab839b1e7f125f50dfcb5a75f816477ebfbf5287d6ace9c91020ed12b43293de: Status 404 returned error can't find the container with id ab839b1e7f125f50dfcb5a75f816477ebfbf5287d6ace9c91020ed12b43293de Oct 31 05:41:20.992610 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:20.992560 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94d5ea09-dvgs6" event={"ID":"bb9eedd7-27f4-4b59-8760-082faaf67650","Type":"ContainerStarted","Data":"5390df24c92251bfacbb60662e6a0014f6bfdbec8bef6bedbd38bbfa1acd68e0"} Oct 31 05:41:20.993054 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:20.992630 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94d5ea09-dvgs6" event={"ID":"bb9eedd7-27f4-4b59-8760-082faaf67650","Type":"ContainerStarted","Data":"ab839b1e7f125f50dfcb5a75f816477ebfbf5287d6ace9c91020ed12b43293de"} Oct 31 05:41:21.055530 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:21.055444 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94d5ea09-dvgs6" podStartSLOduration=2.055408876 podStartE2EDuration="2.055408876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:41:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:41:21.052639261 +0000 UTC m=+5835.808977301" watchObservedRunningTime="2026-10-31 05:41:21.055408876 +0000 UTC m=+5835.811746916" Oct 31 05:41:22.011712 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:22.011659 5019 generic.go:356] "Generic (PLEG): container finished" podID="bb9eedd7-27f4-4b59-8760-082faaf67650" containerID="5390df24c92251bfacbb60662e6a0014f6bfdbec8bef6bedbd38bbfa1acd68e0" exitCode=0 Oct 31 05:41:22.012289 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:22.011754 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94d5ea09-dvgs6" event={"ID":"bb9eedd7-27f4-4b59-8760-082faaf67650","Type":"ContainerDied","Data":"5390df24c92251bfacbb60662e6a0014f6bfdbec8bef6bedbd38bbfa1acd68e0"} Oct 31 05:41:22.067358 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:22.055463 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-q6tnf"] Oct 31 05:41:22.067358 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:22.061119 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-q6tnf"] Oct 31 05:41:22.665873 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:41:22.665822 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:41:23.810080 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:23.810040 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2356a24f-9ff0-477e-86c3-cad6ec2ab2b5" path="/var/lib/kubelet/pods/2356a24f-9ff0-477e-86c3-cad6ec2ab2b5/volumes" Oct 31 05:41:23.902039 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:23.901642 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94d5ea09-dvgs6" Oct 31 05:41:24.021377 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:24.015734 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb9eedd7-27f4-4b59-8760-082faaf67650-kube-api-access-4t8wg\" (UniqueName: \"kubernetes.io/projected/bb9eedd7-27f4-4b59-8760-082faaf67650-kube-api-access-4t8wg\") pod \"bb9eedd7-27f4-4b59-8760-082faaf67650\" (UID: \"bb9eedd7-27f4-4b59-8760-082faaf67650\") " Oct 31 05:41:24.021377 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:24.015907 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb9eedd7-27f4-4b59-8760-082faaf67650-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb9eedd7-27f4-4b59-8760-082faaf67650-maas-api-xks-ca\") pod \"bb9eedd7-27f4-4b59-8760-082faaf67650\" (UID: \"bb9eedd7-27f4-4b59-8760-082faaf67650\") " Oct 31 05:41:24.021377 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:24.016002 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb9eedd7-27f4-4b59-8760-082faaf67650-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb9eedd7-27f4-4b59-8760-082faaf67650-maas-api-openshift-ca\") pod \"bb9eedd7-27f4-4b59-8760-082faaf67650\" (UID: \"bb9eedd7-27f4-4b59-8760-082faaf67650\") " Oct 31 05:41:24.022589 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:24.022480 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb9eedd7-27f4-4b59-8760-082faaf67650-maas-api-openshift-ca" pod "bb9eedd7-27f4-4b59-8760-082faaf67650" (UID: "bb9eedd7-27f4-4b59-8760-082faaf67650"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:41:24.066576 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:24.065477 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb9eedd7-27f4-4b59-8760-082faaf67650-kube-api-access-4t8wg" pod "bb9eedd7-27f4-4b59-8760-082faaf67650" (UID: "bb9eedd7-27f4-4b59-8760-082faaf67650"). InnerVolumeSpecName "kube-api-access-4t8wg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:41:24.079752 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:24.079708 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94d5ea09-dvgs6" event={"ID":"bb9eedd7-27f4-4b59-8760-082faaf67650","Type":"ContainerDied","Data":"ab839b1e7f125f50dfcb5a75f816477ebfbf5287d6ace9c91020ed12b43293de"} Oct 31 05:41:24.080010 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:24.079986 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab839b1e7f125f50dfcb5a75f816477ebfbf5287d6ace9c91020ed12b43293de" Oct 31 05:41:24.080230 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:24.080215 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94d5ea09-dvgs6" Oct 31 05:41:24.080747 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:24.080711 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb9eedd7-27f4-4b59-8760-082faaf67650-maas-api-xks-ca" pod "bb9eedd7-27f4-4b59-8760-082faaf67650" (UID: "bb9eedd7-27f4-4b59-8760-082faaf67650"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:41:24.117453 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:24.117397 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb9eedd7-27f4-4b59-8760-082faaf67650-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:41:24.117453 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:24.117455 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t8wg\" (UniqueName: \"kubernetes.io/projected/bb9eedd7-27f4-4b59-8760-082faaf67650-kube-api-access-4t8wg\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:41:24.117765 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:24.117473 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb9eedd7-27f4-4b59-8760-082faaf67650-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:41:29.992835 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:29.992794 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8032244-t5mv4"] Oct 31 05:41:29.993304 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:29.993288 5019 cpu_manager.go:414] "containerMap: removing container" podUID="bb9eedd7-27f4-4b59-8760-082faaf67650" containerName="revoke-keys" Oct 31 05:41:29.993304 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:29.993304 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb9eedd7-27f4-4b59-8760-082faaf67650" containerName="revoke-keys" Oct 31 05:41:29.993446 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:29.993402 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb9eedd7-27f4-4b59-8760-082faaf67650" containerName="revoke-keys" Oct 31 05:41:29.996953 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:29.996923 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8032244-t5mv4" Oct 31 05:41:30.016685 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:30.016645 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8032244-t5mv4"] Oct 31 05:41:30.024553 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:30.023894 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 05:41:30.024553 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:30.024085 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfhzn\"" Oct 31 05:41:30.024553 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:30.024134 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 05:41:30.024553 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:30.024140 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 05:41:30.080751 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:30.080702 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtspk\" (UniqueName: \"kubernetes.io/projected/4acf3436-cf82-4d5a-b15b-b3a2d2d2385c-kube-api-access-mtspk\") pod \"maas-api-revoke-sub-a8032244-t5mv4\" (UID: \"4acf3436-cf82-4d5a-b15b-b3a2d2d2385c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8032244-t5mv4" Oct 31 05:41:30.080751 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:30.080759 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4acf3436-cf82-4d5a-b15b-b3a2d2d2385c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8032244-t5mv4\" (UID: \"4acf3436-cf82-4d5a-b15b-b3a2d2d2385c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8032244-t5mv4" Oct 31 05:41:30.081077 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:30.080957 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4acf3436-cf82-4d5a-b15b-b3a2d2d2385c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8032244-t5mv4\" (UID: \"4acf3436-cf82-4d5a-b15b-b3a2d2d2385c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8032244-t5mv4" Oct 31 05:41:30.181692 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:30.181646 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4acf3436-cf82-4d5a-b15b-b3a2d2d2385c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8032244-t5mv4\" (UID: \"4acf3436-cf82-4d5a-b15b-b3a2d2d2385c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8032244-t5mv4" Oct 31 05:41:30.181876 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:30.181749 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtspk\" (UniqueName: \"kubernetes.io/projected/4acf3436-cf82-4d5a-b15b-b3a2d2d2385c-kube-api-access-mtspk\") pod \"maas-api-revoke-sub-a8032244-t5mv4\" (UID: \"4acf3436-cf82-4d5a-b15b-b3a2d2d2385c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8032244-t5mv4" Oct 31 05:41:30.181876 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:30.181783 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4acf3436-cf82-4d5a-b15b-b3a2d2d2385c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8032244-t5mv4\" (UID: \"4acf3436-cf82-4d5a-b15b-b3a2d2d2385c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8032244-t5mv4" Oct 31 05:41:30.182437 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:30.182407 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4acf3436-cf82-4d5a-b15b-b3a2d2d2385c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8032244-t5mv4\" (UID: \"4acf3436-cf82-4d5a-b15b-b3a2d2d2385c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8032244-t5mv4" Oct 31 05:41:30.184429 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:30.184405 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4acf3436-cf82-4d5a-b15b-b3a2d2d2385c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8032244-t5mv4\" (UID: \"4acf3436-cf82-4d5a-b15b-b3a2d2d2385c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8032244-t5mv4" Oct 31 05:41:30.206127 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:30.206091 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtspk\" (UniqueName: \"kubernetes.io/projected/4acf3436-cf82-4d5a-b15b-b3a2d2d2385c-kube-api-access-mtspk\") pod \"maas-api-revoke-sub-a8032244-t5mv4\" (UID: \"4acf3436-cf82-4d5a-b15b-b3a2d2d2385c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8032244-t5mv4" Oct 31 05:41:30.312941 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:30.312837 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8032244-t5mv4" Oct 31 05:41:30.615964 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:30.615928 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8032244-t5mv4"] Oct 31 05:41:30.616934 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:41:30.616906 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4acf3436_cf82_4d5a_b15b_b3a2d2d2385c.slice/crio-37a759a8136d7e9ff86820055bd25221f5f10b8801facc56070cfed85bb9f422 WatchSource:0}: Error finding container 37a759a8136d7e9ff86820055bd25221f5f10b8801facc56070cfed85bb9f422: Status 404 returned error can't find the container with id 37a759a8136d7e9ff86820055bd25221f5f10b8801facc56070cfed85bb9f422 Oct 31 05:41:31.128042 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:31.128005 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8032244-t5mv4" event={"ID":"4acf3436-cf82-4d5a-b15b-b3a2d2d2385c","Type":"ContainerStarted","Data":"a18903a10ea297efab71395bb3cd6ad22c28c113787e0feaae3b9ae14692768a"} Oct 31 05:41:31.128571 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:31.128063 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8032244-t5mv4" event={"ID":"4acf3436-cf82-4d5a-b15b-b3a2d2d2385c","Type":"ContainerStarted","Data":"37a759a8136d7e9ff86820055bd25221f5f10b8801facc56070cfed85bb9f422"} Oct 31 05:41:31.173771 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:31.173642 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8032244-t5mv4" podStartSLOduration=2.173622628 podStartE2EDuration="2.173622628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:41:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:41:31.171881289 +0000 UTC m=+5845.928219329" watchObservedRunningTime="2026-10-31 05:41:31.173622628 +0000 UTC m=+5845.929960672" Oct 31 05:41:32.171681 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:32.171621 5019 generic.go:356] "Generic (PLEG): container finished" podID="4acf3436-cf82-4d5a-b15b-b3a2d2d2385c" containerID="a18903a10ea297efab71395bb3cd6ad22c28c113787e0feaae3b9ae14692768a" exitCode=0 Oct 31 05:41:32.172429 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:32.172399 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8032244-t5mv4" event={"ID":"4acf3436-cf82-4d5a-b15b-b3a2d2d2385c","Type":"ContainerDied","Data":"a18903a10ea297efab71395bb3cd6ad22c28c113787e0feaae3b9ae14692768a"} Oct 31 05:41:33.618855 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:33.618816 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8032244-t5mv4" Oct 31 05:41:33.704798 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:33.704752 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4acf3436-cf82-4d5a-b15b-b3a2d2d2385c-kube-api-access-mtspk\" (UniqueName: \"kubernetes.io/projected/4acf3436-cf82-4d5a-b15b-b3a2d2d2385c-kube-api-access-mtspk\") pod \"4acf3436-cf82-4d5a-b15b-b3a2d2d2385c\" (UID: \"4acf3436-cf82-4d5a-b15b-b3a2d2d2385c\") " Oct 31 05:41:33.705029 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:33.704851 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4acf3436-cf82-4d5a-b15b-b3a2d2d2385c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4acf3436-cf82-4d5a-b15b-b3a2d2d2385c-maas-api-xks-ca\") pod \"4acf3436-cf82-4d5a-b15b-b3a2d2d2385c\" (UID: \"4acf3436-cf82-4d5a-b15b-b3a2d2d2385c\") " Oct 31 05:41:33.705029 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:33.704917 5019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4acf3436-cf82-4d5a-b15b-b3a2d2d2385c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4acf3436-cf82-4d5a-b15b-b3a2d2d2385c-maas-api-openshift-ca\") pod \"4acf3436-cf82-4d5a-b15b-b3a2d2d2385c\" (UID: \"4acf3436-cf82-4d5a-b15b-b3a2d2d2385c\") " Oct 31 05:41:33.705306 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:33.705277 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4acf3436-cf82-4d5a-b15b-b3a2d2d2385c-maas-api-openshift-ca" pod "4acf3436-cf82-4d5a-b15b-b3a2d2d2385c" (UID: "4acf3436-cf82-4d5a-b15b-b3a2d2d2385c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 05:41:33.707415 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:33.707369 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4acf3436-cf82-4d5a-b15b-b3a2d2d2385c-maas-api-xks-ca" pod "4acf3436-cf82-4d5a-b15b-b3a2d2d2385c" (UID: "4acf3436-cf82-4d5a-b15b-b3a2d2d2385c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 05:41:33.707670 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:33.707641 5019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4acf3436-cf82-4d5a-b15b-b3a2d2d2385c-kube-api-access-mtspk" pod "4acf3436-cf82-4d5a-b15b-b3a2d2d2385c" (UID: "4acf3436-cf82-4d5a-b15b-b3a2d2d2385c"). InnerVolumeSpecName "kube-api-access-mtspk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 05:41:33.806588 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:33.806486 5019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtspk\" (UniqueName: \"kubernetes.io/projected/4acf3436-cf82-4d5a-b15b-b3a2d2d2385c-kube-api-access-mtspk\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:41:33.806588 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:33.806527 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4acf3436-cf82-4d5a-b15b-b3a2d2d2385c-maas-api-xks-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:41:33.806588 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:33.806537 5019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4acf3436-cf82-4d5a-b15b-b3a2d2d2385c-maas-api-openshift-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Oct 31 05:41:34.206433 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:34.206312 5019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8032244-t5mv4" Oct 31 05:41:34.206433 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:34.206350 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8032244-t5mv4" event={"ID":"4acf3436-cf82-4d5a-b15b-b3a2d2d2385c","Type":"ContainerDied","Data":"37a759a8136d7e9ff86820055bd25221f5f10b8801facc56070cfed85bb9f422"} Oct 31 05:41:34.206433 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:34.206405 5019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37a759a8136d7e9ff86820055bd25221f5f10b8801facc56070cfed85bb9f422" Oct 31 05:41:39.035796 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:39.035745 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9554q"] Oct 31 05:41:39.045259 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:39.045214 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9554q"] Oct 31 05:41:39.407428 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:39.407325 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5650965-4ba0-4fe8-a2fd-ff42413710f9" path="/var/lib/kubelet/pods/e5650965-4ba0-4fe8-a2fd-ff42413710f9/volumes" Oct 31 05:41:50.819563 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:41:50.819506 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-77ddb887cc-6csbj_720f62c6-de2c-4269-bc48-b785891f8fec/authorino/0.log" Oct 31 05:41:52.395866 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:41:52.395813 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:42:03.362975 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:03.362931 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6dc65d95b6-gpnbc_d194e31f-b57e-490d-a5c8-9f7b12de4b2c/manager/0.log" Oct 31 05:42:06.465964 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:06.465921 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-1c9a2c83-7jf8w_038cc95f-9b08-4a57-9229-5b74acb9e968/revoke-keys/0.log" Oct 31 05:42:06.675936 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:06.675901 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-1e0d2d60-vkgxl_d30a1985-72b2-4162-a797-5b55a2272fd9/revoke-keys/0.log" Oct 31 05:42:06.883279 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:06.883187 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-36d1b076-tr4fl_e10387ea-7e1c-49f8-9f45-95e7073a3b55/revoke-keys/0.log" Oct 31 05:42:07.077506 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:07.077474 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3d08cea9-2d75z_9de5e560-922e-4c50-9150-c07c09feb42b/revoke-keys/0.log" Oct 31 05:42:07.271030 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:07.270997 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3f2e7163-59wnt_06ca025b-42d9-4d04-9fbe-69d99d932f7e/revoke-keys/0.log" Oct 31 05:42:07.471373 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:07.471319 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-440fe870-mltxk_56505ab3-8320-474e-84a0-a8470cbe6453/revoke-keys/0.log" Oct 31 05:42:07.680118 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:07.680045 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6212d0dc-qkq4g_9d6b3a3f-67cf-4dc2-9489-f3e134b76560/revoke-keys/0.log" Oct 31 05:42:07.884078 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:07.884043 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-68729206-bgxmz_0ea5bbc9-21fc-4dfb-9f14-1ad3ab9b852b/revoke-keys/0.log" Oct 31 05:42:08.076488 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:08.076456 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6cd035ac-vlcq6_9483d81c-c122-4194-a1ec-016ab7fdfbf7/revoke-keys/0.log" Oct 31 05:42:08.274389 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:08.274358 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-94d5ea09-dvgs6_bb9eedd7-27f4-4b59-8760-082faaf67650/revoke-keys/0.log" Oct 31 05:42:08.477402 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:08.477369 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-9c545349-srpxt_1e43c3df-d577-4a9a-bfd4-64df51d97a5c/revoke-keys/0.log" Oct 31 05:42:08.685583 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:08.685553 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-9e69dc44-26cqc_ddd66c15-bf68-4b42-a5f5-9666d6e3dc20/revoke-keys/0.log" Oct 31 05:42:08.860847 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:08.860768 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-a8032244-t5mv4_4acf3436-cf82-4d5a-b15b-b3a2d2d2385c/revoke-keys/0.log" Oct 31 05:42:09.067707 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:09.067676 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-adbbff71-jwc6x_c7bee25a-4a74-4b09-bcec-8304ad6b180b/revoke-keys/0.log" Oct 31 05:42:09.271580 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:09.271537 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-ba23861f-fkglb_79721c0c-c0d9-478d-b2cd-64f1efaf2cf9/revoke-keys/0.log" Oct 31 05:42:09.462998 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:09.462959 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-bbc70dfe-w7wzz_ce1175fc-39e4-44e7-a47a-0cc40c570f8e/revoke-keys/0.log" Oct 31 05:42:09.654819 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:09.654739 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c16ae582-kmwx6_c89fa3eb-3b89-4047-a233-e428240273d7/revoke-keys/0.log" Oct 31 05:42:09.864521 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:09.864487 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c908acb8-p7fzj_b0e91733-757e-4549-ac4f-b8b43db81288/revoke-keys/0.log" Oct 31 05:42:10.063051 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:10.063024 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-cd6cc76f-q7drd_2330a863-0b04-4d46-9768-28332b366152/revoke-keys/0.log" Oct 31 05:42:11.029364 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:11.029312 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-440fe870-mltxk"] Oct 31 05:42:11.040844 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:11.040795 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-440fe870-mltxk"] Oct 31 05:42:11.406789 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:11.406701 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56505ab3-8320-474e-84a0-a8470cbe6453" path="/var/lib/kubelet/pods/56505ab3-8320-474e-84a0-a8470cbe6453/volumes" Oct 31 05:42:12.292723 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:12.292684 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-77ddb887cc-6csbj_720f62c6-de2c-4269-bc48-b785891f8fec/authorino/0.log" Oct 31 05:42:12.530792 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:12.530757 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-r22dt_48e6e3f6-9d22-42a3-bc22-a27e516aeda1/manager/0.log" Oct 31 05:42:12.754008 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:12.753959 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-ggcnf_cd96ff5f-fb42-4491-aae0-bcdd6e4002aa/manager/0.log" Oct 31 05:42:13.195287 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:13.195190 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-cljqx_26d2360a-6a02-44ea-a750-6f634b2f90a6/manager/0.log" Oct 31 05:42:13.660862 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:13.660819 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-gsxbf_176b5a4d-06b5-484e-b6ba-e5727c90afba/manager/0.log" Oct 31 05:42:14.756564 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:14.756527 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68dd7b898flnvv2_1e142c20-377f-4aa7-a507-780e8efc906d/istio-proxy/0.log" Oct 31 05:42:15.828383 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:15.828274 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-hgv9r_68e465ba-2b5d-4e3c-8b63-7bdb9b695b75/istio-proxy/0.log" Oct 31 05:42:16.521362 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:16.521191 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-lqrr5_6d68d19b-5392-4308-9e2c-4b5e7593d17b/istio-proxy/0.log" Oct 31 05:42:17.189210 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:17.189165 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-rgj85_8430b919-a9dd-4c7e-9266-ef6872cc0b9e/istio-proxy/0.log" Oct 31 05:42:17.402787 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:17.402752 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-shld4_a69fd138-9906-4127-ac26-38a21a5b7590/istio-proxy/0.log" Oct 31 05:42:17.805140 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:17.805082 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq_17297059-70a0-480f-ad3c-4427585cf15c/istio-proxy/0.log" Oct 31 05:42:17.992795 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:17.992755 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-5bd7d6d88-dth2t_db04abaa-c7e6-46c2-86d1-edb4e4b71862/payload-pre-processing/0.log" Oct 31 05:42:18.394978 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:18.394936 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-678d747c5f-bgj92_d22707c7-21fa-4511-b0b1-869b41fb2351/router/0.log" Oct 31 05:42:19.678203 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:19.678166 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-mprn9_ffacd465-1055-444a-9f85-f46a57ac3724/main/0.log" Oct 31 05:42:19.867597 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:19.867562 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-9mgfd_705ea68d-0f37-4d66-a90a-5f089d437b09/main/0.log" Oct 31 05:42:20.266540 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:20.266506 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c44pm_0513559e-98f2-4889-b6cd-5b2fff981f3e/main/0.log" Oct 31 05:42:20.666515 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:20.666420 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-cv556_63726a00-18c8-4056-b970-ba1809e2cfc4/main/0.log" Oct 31 05:42:21.028672 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:21.028631 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e69dc44-26cqc"] Oct 31 05:42:21.040155 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:21.040109 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e69dc44-26cqc"] Oct 31 05:42:21.406365 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:21.406242 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddd66c15-bf68-4b42-a5f5-9666d6e3dc20" path="/var/lib/kubelet/pods/ddd66c15-bf68-4b42-a5f5-9666d6e3dc20/volumes" Oct 31 05:42:22.392054 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:42:22.392013 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:42:32.310590 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:32.310547 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nnndq/must-gather-qpsqb"] Oct 31 05:42:32.311053 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:32.311038 5019 cpu_manager.go:414] "containerMap: removing container" podUID="4acf3436-cf82-4d5a-b15b-b3a2d2d2385c" containerName="revoke-keys" Oct 31 05:42:32.311117 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:32.311054 5019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4acf3436-cf82-4d5a-b15b-b3a2d2d2385c" containerName="revoke-keys" Oct 31 05:42:32.311163 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:32.311127 5019 memory_manager.go:365] "RemoveStaleState removing state" podUID="4acf3436-cf82-4d5a-b15b-b3a2d2d2385c" containerName="revoke-keys" Oct 31 05:42:32.314705 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:32.314680 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nnndq/must-gather-qpsqb" Oct 31 05:42:32.342409 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:32.342368 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nnndq\"/\"kube-root-ca.crt\"" Oct 31 05:42:32.342409 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:32.342361 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-nnndq\"/\"default-dockercfg-vxffb\"" Oct 31 05:42:32.342409 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:32.342366 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nnndq\"/\"openshift-service-ca.crt\"" Oct 31 05:42:32.345988 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:32.345943 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nnndq/must-gather-qpsqb"] Oct 31 05:42:32.403738 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:32.403696 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-p559h/must-gather-6mk92"] Oct 31 05:42:32.407619 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:32.407579 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26v9q\" (UniqueName: \"kubernetes.io/projected/c8ce28a9-d4f2-453f-93fb-e954d9bccb1b-kube-api-access-26v9q\") pod \"must-gather-qpsqb\" (UID: \"c8ce28a9-d4f2-453f-93fb-e954d9bccb1b\") " pod="openshift-must-gather-nnndq/must-gather-qpsqb" Oct 31 05:42:32.407794 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:32.407591 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p559h/must-gather-6mk92" Oct 31 05:42:32.407794 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:32.407694 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c8ce28a9-d4f2-453f-93fb-e954d9bccb1b-must-gather-output\") pod \"must-gather-qpsqb\" (UID: \"c8ce28a9-d4f2-453f-93fb-e954d9bccb1b\") " pod="openshift-must-gather-nnndq/must-gather-qpsqb" Oct 31 05:42:32.425821 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:32.425785 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p559h/must-gather-6mk92"] Oct 31 05:42:32.430346 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:32.430302 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-p559h\"/\"default-dockercfg-t27v9\"" Oct 31 05:42:32.430346 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:32.430303 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-p559h\"/\"kube-root-ca.crt\"" Oct 31 05:42:32.430624 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:32.430302 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-p559h\"/\"openshift-service-ca.crt\"" Oct 31 05:42:32.508723 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:32.508679 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c992f7a1-3f0c-451f-888c-65cf42777f95-must-gather-output\") pod \"must-gather-6mk92\" (UID: \"c992f7a1-3f0c-451f-888c-65cf42777f95\") " pod="openshift-must-gather-p559h/must-gather-6mk92" Oct 31 05:42:32.508723 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:32.508721 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s79sn\" (UniqueName: \"kubernetes.io/projected/c992f7a1-3f0c-451f-888c-65cf42777f95-kube-api-access-s79sn\") pod \"must-gather-6mk92\" (UID: \"c992f7a1-3f0c-451f-888c-65cf42777f95\") " pod="openshift-must-gather-p559h/must-gather-6mk92" Oct 31 05:42:32.509041 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:32.508838 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c8ce28a9-d4f2-453f-93fb-e954d9bccb1b-must-gather-output\") pod \"must-gather-qpsqb\" (UID: \"c8ce28a9-d4f2-453f-93fb-e954d9bccb1b\") " pod="openshift-must-gather-nnndq/must-gather-qpsqb" Oct 31 05:42:32.509041 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:32.508984 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26v9q\" (UniqueName: \"kubernetes.io/projected/c8ce28a9-d4f2-453f-93fb-e954d9bccb1b-kube-api-access-26v9q\") pod \"must-gather-qpsqb\" (UID: \"c8ce28a9-d4f2-453f-93fb-e954d9bccb1b\") " pod="openshift-must-gather-nnndq/must-gather-qpsqb" Oct 31 05:42:32.509229 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:32.509211 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c8ce28a9-d4f2-453f-93fb-e954d9bccb1b-must-gather-output\") pod \"must-gather-qpsqb\" (UID: \"c8ce28a9-d4f2-453f-93fb-e954d9bccb1b\") " pod="openshift-must-gather-nnndq/must-gather-qpsqb" Oct 31 05:42:32.537958 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:32.537925 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26v9q\" (UniqueName: \"kubernetes.io/projected/c8ce28a9-d4f2-453f-93fb-e954d9bccb1b-kube-api-access-26v9q\") pod \"must-gather-qpsqb\" (UID: \"c8ce28a9-d4f2-453f-93fb-e954d9bccb1b\") " pod="openshift-must-gather-nnndq/must-gather-qpsqb" Oct 31 05:42:32.609880 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:32.609776 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c992f7a1-3f0c-451f-888c-65cf42777f95-must-gather-output\") pod \"must-gather-6mk92\" (UID: \"c992f7a1-3f0c-451f-888c-65cf42777f95\") " pod="openshift-must-gather-p559h/must-gather-6mk92" Oct 31 05:42:32.609880 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:32.609819 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s79sn\" (UniqueName: \"kubernetes.io/projected/c992f7a1-3f0c-451f-888c-65cf42777f95-kube-api-access-s79sn\") pod \"must-gather-6mk92\" (UID: \"c992f7a1-3f0c-451f-888c-65cf42777f95\") " pod="openshift-must-gather-p559h/must-gather-6mk92" Oct 31 05:42:32.610136 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:32.610111 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c992f7a1-3f0c-451f-888c-65cf42777f95-must-gather-output\") pod \"must-gather-6mk92\" (UID: \"c992f7a1-3f0c-451f-888c-65cf42777f95\") " pod="openshift-must-gather-p559h/must-gather-6mk92" Oct 31 05:42:32.630719 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:32.630674 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nnndq/must-gather-qpsqb" Oct 31 05:42:32.648062 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:32.648031 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s79sn\" (UniqueName: \"kubernetes.io/projected/c992f7a1-3f0c-451f-888c-65cf42777f95-kube-api-access-s79sn\") pod \"must-gather-6mk92\" (UID: \"c992f7a1-3f0c-451f-888c-65cf42777f95\") " pod="openshift-must-gather-p559h/must-gather-6mk92" Oct 31 05:42:32.723484 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:32.723433 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p559h/must-gather-6mk92" Oct 31 05:42:32.937394 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:32.937361 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nnndq/must-gather-qpsqb"] Oct 31 05:42:32.938822 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:42:32.938793 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc8ce28a9_d4f2_453f_93fb_e954d9bccb1b.slice/crio-721a7270a6074a7e421df4dfd60163b47ba5046ef1c8a9eac76a02be35070a91 WatchSource:0}: Error finding container 721a7270a6074a7e421df4dfd60163b47ba5046ef1c8a9eac76a02be35070a91: Status 404 returned error can't find the container with id 721a7270a6074a7e421df4dfd60163b47ba5046ef1c8a9eac76a02be35070a91 Oct 31 05:42:32.940606 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:32.940582 5019 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 05:42:33.026373 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:33.026321 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p559h/must-gather-6mk92"] Oct 31 05:42:33.028161 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:42:33.028131 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc992f7a1_3f0c_451f_888c_65cf42777f95.slice/crio-47da710c3c43ef716bf9e9d4581a3610f0245b6147b2186661392090bf645d81 WatchSource:0}: Error finding container 47da710c3c43ef716bf9e9d4581a3610f0245b6147b2186661392090bf645d81: Status 404 returned error can't find the container with id 47da710c3c43ef716bf9e9d4581a3610f0245b6147b2186661392090bf645d81 Oct 31 05:42:33.658090 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:33.658049 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nnndq/must-gather-qpsqb" event={"ID":"c8ce28a9-d4f2-453f-93fb-e954d9bccb1b","Type":"ContainerStarted","Data":"721a7270a6074a7e421df4dfd60163b47ba5046ef1c8a9eac76a02be35070a91"} Oct 31 05:42:33.659392 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:33.659359 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p559h/must-gather-6mk92" event={"ID":"c992f7a1-3f0c-451f-888c-65cf42777f95","Type":"ContainerStarted","Data":"47da710c3c43ef716bf9e9d4581a3610f0245b6147b2186661392090bf645d81"} Oct 31 05:42:36.622098 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:36.622055 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hnbll_c1560abd-3afc-4f94-b0bb-ac7ef8b2464a/global-pull-secret-syncer/0.log" Oct 31 05:42:37.159197 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:37.159167 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-xgjnm_b541554a-0460-4716-9f14-f7a46567465c/connection-checker/0.log" Oct 31 05:42:37.275582 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:37.275474 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-xqd9k_57b47c7b-9c84-4888-be69-64afca28a8c0/connection-checker/0.log" Oct 31 05:42:37.391972 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:37.391862 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-z75pq_2050cf2e-0f83-43b3-bb85-7bd4ac42a473/connection-checker/0.log" Oct 31 05:42:37.693074 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:37.693036 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-hm5cr_89ea384b-705f-47a9-8df6-d3c266ed6761/konnectivity-agent/0.log" Oct 31 05:42:37.972679 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:37.972643 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-57.ec2.internal_6536fdc09c4b535fc805fe98673b8597/haproxy/0.log" Oct 31 05:42:39.733660 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:39.733446 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nnndq/must-gather-qpsqb" event={"ID":"c8ce28a9-d4f2-453f-93fb-e954d9bccb1b","Type":"ContainerStarted","Data":"cbbe20d8c3cdca8b9abaf397f0c8bc69907383f04e776ef8e0ace4bf7a452c0f"} Oct 31 05:42:40.748835 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:40.748781 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nnndq/must-gather-qpsqb" event={"ID":"c8ce28a9-d4f2-453f-93fb-e954d9bccb1b","Type":"ContainerStarted","Data":"83945c6ee54cdaa6c58edf8805a36573953204f1d8e13ef9c14b52b2ee620253"} Oct 31 05:42:40.752409 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:40.752368 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p559h/must-gather-6mk92" event={"ID":"c992f7a1-3f0c-451f-888c-65cf42777f95","Type":"ContainerStarted","Data":"2a5427ea8e5eb29fb9f85c790b42ce6d1d6a11b6ec27be81b17c366d3bcd0118"} Oct 31 05:42:40.752616 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:40.752439 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p559h/must-gather-6mk92" event={"ID":"c992f7a1-3f0c-451f-888c-65cf42777f95","Type":"ContainerStarted","Data":"78eb3168e29c81b524785d22c3ae7f2e851cb7b78940ba9348e6a648c6f3459d"} Oct 31 05:42:40.790808 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:40.790751 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-nnndq/must-gather-qpsqb" podStartSLOduration=2.208800206 podStartE2EDuration="8.790735589s" totalImagesPullingTime="6.581935383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:42:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:42:40.789711364 +0000 UTC m=+5915.546049423" watchObservedRunningTime="2026-10-31 05:42:40.790735589 +0000 UTC m=+5915.547073629" Oct 31 05:42:40.830658 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:40.830598 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-p559h/must-gather-6mk92" podStartSLOduration=2.265284849 podStartE2EDuration="8.830575767s" totalImagesPullingTime="6.565290918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:42:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:42:40.828227662 +0000 UTC m=+5915.584565701" watchObservedRunningTime="2026-10-31 05:42:40.830575767 +0000 UTC m=+5915.586913821" Oct 31 05:42:41.623132 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:41.623089 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6dc65d95b6-gpnbc_d194e31f-b57e-490d-a5c8-9f7b12de4b2c/manager/0.log" Oct 31 05:42:41.765196 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:41.765159 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6dc65d95b6-gpnbc_d194e31f-b57e-490d-a5c8-9f7b12de4b2c/manager/0.log" Oct 31 05:42:43.029217 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:43.029174 5019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e0d2d60-vkgxl"] Oct 31 05:42:43.041140 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:43.041096 5019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e0d2d60-vkgxl"] Oct 31 05:42:43.407727 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:43.407647 5019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d30a1985-72b2-4162-a797-5b55a2272fd9" path="/var/lib/kubelet/pods/d30a1985-72b2-4162-a797-5b55a2272fd9/volumes" Oct 31 05:42:43.732127 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:43.732091 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-77ddb887cc-6csbj_720f62c6-de2c-4269-bc48-b785891f8fec/authorino/0.log" Oct 31 05:42:44.235292 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:44.235248 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-r22dt_48e6e3f6-9d22-42a3-bc22-a27e516aeda1/manager/0.log" Oct 31 05:42:44.361085 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:44.361052 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-ggcnf_cd96ff5f-fb42-4491-aae0-bcdd6e4002aa/manager/0.log" Oct 31 05:42:44.657473 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:44.657323 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-cljqx_26d2360a-6a02-44ea-a750-6f634b2f90a6/manager/0.log" Oct 31 05:42:44.981638 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:44.981596 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-gsxbf_176b5a4d-06b5-484e-b6ba-e5727c90afba/manager/0.log" Oct 31 05:42:47.323831 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:47.323788 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-zqvkx_a173790b-6c13-45a1-a361-3fe51844b4c2/cluster-monitoring-operator/0.log" Oct 31 05:42:47.398143 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:47.398096 5019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-n76t8/perf-node-gather-daemonset-shv95"] Oct 31 05:42:47.406545 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:47.406510 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-shv95" Oct 31 05:42:47.425807 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:47.425770 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n76t8/perf-node-gather-daemonset-shv95"] Oct 31 05:42:47.433824 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:47.433784 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-n76t8\"/\"kube-root-ca.crt\"" Oct 31 05:42:47.433824 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:47.433803 5019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-n76t8\"/\"openshift-service-ca.crt\"" Oct 31 05:42:47.434108 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:47.433875 5019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-n76t8\"/\"default-dockercfg-f2kxb\"" Oct 31 05:42:47.479171 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:47.479120 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e7aeaf19-380d-4fba-8e09-d6d4bef76d65-lib-modules\") pod \"perf-node-gather-daemonset-shv95\" (UID: \"e7aeaf19-380d-4fba-8e09-d6d4bef76d65\") " pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-shv95" Oct 31 05:42:47.479422 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:47.479250 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e7aeaf19-380d-4fba-8e09-d6d4bef76d65-podres\") pod \"perf-node-gather-daemonset-shv95\" (UID: \"e7aeaf19-380d-4fba-8e09-d6d4bef76d65\") " pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-shv95" Oct 31 05:42:47.479422 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:47.479318 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e7aeaf19-380d-4fba-8e09-d6d4bef76d65-proc\") pod \"perf-node-gather-daemonset-shv95\" (UID: \"e7aeaf19-380d-4fba-8e09-d6d4bef76d65\") " pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-shv95" Oct 31 05:42:47.479583 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:47.479426 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e7aeaf19-380d-4fba-8e09-d6d4bef76d65-sys\") pod \"perf-node-gather-daemonset-shv95\" (UID: \"e7aeaf19-380d-4fba-8e09-d6d4bef76d65\") " pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-shv95" Oct 31 05:42:47.479583 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:47.479455 5019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xlf5\" (UniqueName: \"kubernetes.io/projected/e7aeaf19-380d-4fba-8e09-d6d4bef76d65-kube-api-access-9xlf5\") pod \"perf-node-gather-daemonset-shv95\" (UID: \"e7aeaf19-380d-4fba-8e09-d6d4bef76d65\") " pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-shv95" Oct 31 05:42:47.580357 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:47.580246 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e7aeaf19-380d-4fba-8e09-d6d4bef76d65-sys\") pod \"perf-node-gather-daemonset-shv95\" (UID: \"e7aeaf19-380d-4fba-8e09-d6d4bef76d65\") " pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-shv95" Oct 31 05:42:47.580357 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:47.580295 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xlf5\" (UniqueName: \"kubernetes.io/projected/e7aeaf19-380d-4fba-8e09-d6d4bef76d65-kube-api-access-9xlf5\") pod \"perf-node-gather-daemonset-shv95\" (UID: \"e7aeaf19-380d-4fba-8e09-d6d4bef76d65\") " pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-shv95" Oct 31 05:42:47.580641 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:47.580364 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e7aeaf19-380d-4fba-8e09-d6d4bef76d65-lib-modules\") pod \"perf-node-gather-daemonset-shv95\" (UID: \"e7aeaf19-380d-4fba-8e09-d6d4bef76d65\") " pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-shv95" Oct 31 05:42:47.580641 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:47.580412 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e7aeaf19-380d-4fba-8e09-d6d4bef76d65-podres\") pod \"perf-node-gather-daemonset-shv95\" (UID: \"e7aeaf19-380d-4fba-8e09-d6d4bef76d65\") " pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-shv95" Oct 31 05:42:47.580641 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:47.580435 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e7aeaf19-380d-4fba-8e09-d6d4bef76d65-sys\") pod \"perf-node-gather-daemonset-shv95\" (UID: \"e7aeaf19-380d-4fba-8e09-d6d4bef76d65\") " pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-shv95" Oct 31 05:42:47.580641 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:47.580505 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e7aeaf19-380d-4fba-8e09-d6d4bef76d65-lib-modules\") pod \"perf-node-gather-daemonset-shv95\" (UID: \"e7aeaf19-380d-4fba-8e09-d6d4bef76d65\") " pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-shv95" Oct 31 05:42:47.580641 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:47.580524 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e7aeaf19-380d-4fba-8e09-d6d4bef76d65-podres\") pod \"perf-node-gather-daemonset-shv95\" (UID: \"e7aeaf19-380d-4fba-8e09-d6d4bef76d65\") " pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-shv95" Oct 31 05:42:47.580641 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:47.580574 5019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e7aeaf19-380d-4fba-8e09-d6d4bef76d65-proc\") pod \"perf-node-gather-daemonset-shv95\" (UID: \"e7aeaf19-380d-4fba-8e09-d6d4bef76d65\") " pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-shv95" Oct 31 05:42:47.580908 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:47.580712 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e7aeaf19-380d-4fba-8e09-d6d4bef76d65-proc\") pod \"perf-node-gather-daemonset-shv95\" (UID: \"e7aeaf19-380d-4fba-8e09-d6d4bef76d65\") " pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-shv95" Oct 31 05:42:47.605641 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:47.605605 5019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xlf5\" (UniqueName: \"kubernetes.io/projected/e7aeaf19-380d-4fba-8e09-d6d4bef76d65-kube-api-access-9xlf5\") pod \"perf-node-gather-daemonset-shv95\" (UID: \"e7aeaf19-380d-4fba-8e09-d6d4bef76d65\") " pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-shv95" Oct 31 05:42:47.730937 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:47.730890 5019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-shv95" Oct 31 05:42:48.053709 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:48.053673 5019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n76t8/perf-node-gather-daemonset-shv95"] Oct 31 05:42:48.054995 ip-10-0-130-57 kubenswrapper[5019]: W1031 05:42:48.054961 5019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode7aeaf19_380d_4fba_8e09_d6d4bef76d65.slice/crio-53edd73e745632a7407fbe3d50fa6e0541722e9adbb0fa19e6464a6a3f8a23ee WatchSource:0}: Error finding container 53edd73e745632a7407fbe3d50fa6e0541722e9adbb0fa19e6464a6a3f8a23ee: Status 404 returned error can't find the container with id 53edd73e745632a7407fbe3d50fa6e0541722e9adbb0fa19e6464a6a3f8a23ee Oct 31 05:42:48.212495 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:48.212459 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6dc65d95b6-gpnbc_d194e31f-b57e-490d-a5c8-9f7b12de4b2c/manager/0.log" Oct 31 05:42:48.571595 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:48.571555 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6dc65d95b6-gpnbc_d194e31f-b57e-490d-a5c8-9f7b12de4b2c/manager/0.log" Oct 31 05:42:48.797275 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:48.797220 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x7gql_39fa44fb-3af0-46c2-82a6-d3b21c3e5383/node-exporter/0.log" Oct 31 05:42:48.819473 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:48.819432 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-shv95" event={"ID":"e7aeaf19-380d-4fba-8e09-d6d4bef76d65","Type":"ContainerStarted","Data":"b40f66e66f16be190d1646a72b345e4f5a42d3bb6afd2b2f3b716b59de79c0b3"} Oct 31 05:42:48.819689 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:48.819530 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-shv95" Oct 31 05:42:48.819689 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:48.819563 5019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-shv95" event={"ID":"e7aeaf19-380d-4fba-8e09-d6d4bef76d65","Type":"ContainerStarted","Data":"53edd73e745632a7407fbe3d50fa6e0541722e9adbb0fa19e6464a6a3f8a23ee"} Oct 31 05:42:48.861878 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:48.861760 5019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-shv95" podStartSLOduration=1.8617412340000001 podStartE2EDuration="1.861741234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 05:42:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:42:48.859981844 +0000 UTC m=+5923.616319885" watchObservedRunningTime="2026-10-31 05:42:48.861741234 +0000 UTC m=+5923.618079278" Oct 31 05:42:48.914169 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:48.914136 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x7gql_39fa44fb-3af0-46c2-82a6-d3b21c3e5383/kube-rbac-proxy/0.log" Oct 31 05:42:49.035297 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:49.035266 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x7gql_39fa44fb-3af0-46c2-82a6-d3b21c3e5383/init-textfile/0.log" Oct 31 05:42:52.396056 ip-10-0-130-57 kubenswrapper[5019]: E1031 05:42:52.396018 5019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c6gcg" containerName="extractor" Oct 31 05:42:53.138325 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:53.138283 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-pnwds_183f4178-99c0-4945-b490-c71cbd2780f3/networking-console-plugin/0.log" Oct 31 05:42:54.211072 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:54.211040 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s88kp_3945ff65-e877-4372-9334-3968c68b7295/console-operator/2.log" Oct 31 05:42:54.217917 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:54.217885 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s88kp_3945ff65-e877-4372-9334-3968c68b7295/console-operator/3.log" Oct 31 05:42:54.840493 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:54.840459 5019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-shv95" Oct 31 05:42:54.986053 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:54.986016 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6dc65d95b6-gpnbc_d194e31f-b57e-490d-a5c8-9f7b12de4b2c/manager/0.log" Oct 31 05:42:55.338845 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:55.338789 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6dc65d95b6-gpnbc_d194e31f-b57e-490d-a5c8-9f7b12de4b2c/manager/0.log" Oct 31 05:42:56.411501 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:56.411462 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-hntxx_c4a091a2-58d2-4b05-a968-d16fe1bd9d40/volume-data-source-validator/0.log" Oct 31 05:42:59.301627 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:59.301592 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zjpdv_48309c9d-5f34-49bf-b55b-f74959984e0a/dns/0.log" Oct 31 05:42:59.414441 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:59.414407 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zjpdv_48309c9d-5f34-49bf-b55b-f74959984e0a/kube-rbac-proxy/0.log" Oct 31 05:42:59.656948 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:42:59.656871 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nbl9z_3a05c43f-e2db-44c0-b9bd-4bdabca92a98/dns-node-resolver/0.log" Oct 31 05:43:00.374919 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:43:00.374885 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68dd7b898flnvv2_1e142c20-377f-4aa7-a507-780e8efc906d/istio-proxy/0.log" Oct 31 05:43:01.147635 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:43:01.147591 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-h9sww_31221235-f863-445b-a5ca-375aae905d38/node-ca/0.log" Oct 31 05:43:01.700993 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:43:01.700962 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-hgv9r_68e465ba-2b5d-4e3c-8b63-7bdb9b695b75/istio-proxy/0.log" Oct 31 05:43:01.893043 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:43:01.892975 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-lqrr5_6d68d19b-5392-4308-9e2c-4b5e7593d17b/istio-proxy/0.log" Oct 31 05:43:02.097071 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:43:02.097022 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-rgj85_8430b919-a9dd-4c7e-9266-ef6872cc0b9e/istio-proxy/0.log" Oct 31 05:43:02.161073 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:43:02.161033 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-shld4_a69fd138-9906-4127-ac26-38a21a5b7590/istio-proxy/0.log" Oct 31 05:43:02.289496 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:43:02.289465 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq_17297059-70a0-480f-ad3c-4427585cf15c/istio-proxy/0.log" Oct 31 05:43:02.360324 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:43:02.360241 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-5bd7d6d88-dth2t_db04abaa-c7e6-46c2-86d1-edb4e4b71862/payload-pre-processing/0.log" Oct 31 05:43:02.408864 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:43:02.408834 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68dd7b898flnvv2_1e142c20-377f-4aa7-a507-780e8efc906d/istio-proxy/0.log" Oct 31 05:43:02.485264 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:43:02.485232 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68dd7b898flnvv2_1e142c20-377f-4aa7-a507-780e8efc906d/istio-proxy/0.log" Oct 31 05:43:02.493665 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:43:02.493630 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-678d747c5f-bgj92_d22707c7-21fa-4511-b0b1-869b41fb2351/router/0.log" Oct 31 05:43:03.243942 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:43:03.243896 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68dd7b898flnvv2_1e142c20-377f-4aa7-a507-780e8efc906d/istio-proxy/0.log" Oct 31 05:43:04.332522 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:43:04.332450 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-hgv9r_68e465ba-2b5d-4e3c-8b63-7bdb9b695b75/istio-proxy/0.log" Oct 31 05:43:04.460686 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:43:04.460656 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-hgv9r_68e465ba-2b5d-4e3c-8b63-7bdb9b695b75/istio-proxy/0.log" Oct 31 05:43:04.533999 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:43:04.533966 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-lqrr5_6d68d19b-5392-4308-9e2c-4b5e7593d17b/istio-proxy/0.log" Oct 31 05:43:04.656569 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:43:04.656473 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-lqrr5_6d68d19b-5392-4308-9e2c-4b5e7593d17b/istio-proxy/0.log" Oct 31 05:43:04.738560 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:43:04.738513 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-rgj85_8430b919-a9dd-4c7e-9266-ef6872cc0b9e/istio-proxy/0.log" Oct 31 05:43:04.803049 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:43:04.803012 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-shld4_a69fd138-9906-4127-ac26-38a21a5b7590/istio-proxy/0.log" Oct 31 05:43:04.853715 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:43:04.853673 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-rgj85_8430b919-a9dd-4c7e-9266-ef6872cc0b9e/istio-proxy/0.log" Oct 31 05:43:04.917346 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:43:04.917270 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-shld4_a69fd138-9906-4127-ac26-38a21a5b7590/istio-proxy/0.log" Oct 31 05:43:04.935952 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:43:04.935929 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq_17297059-70a0-480f-ad3c-4427585cf15c/istio-proxy/0.log" Oct 31 05:43:04.998590 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:43:04.998556 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-5bd7d6d88-dth2t_db04abaa-c7e6-46c2-86d1-edb4e4b71862/payload-pre-processing/0.log" Oct 31 05:43:05.077729 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:43:05.077692 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6bd8768855-k2ztq_17297059-70a0-480f-ad3c-4427585cf15c/istio-proxy/0.log" Oct 31 05:43:05.159755 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:43:05.159718 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-678d747c5f-bgj92_d22707c7-21fa-4511-b0b1-869b41fb2351/router/0.log" Oct 31 05:43:05.161534 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:43:05.161514 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-5bd7d6d88-dth2t_db04abaa-c7e6-46c2-86d1-edb4e4b71862/payload-pre-processing/0.log" Oct 31 05:43:05.332646 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:43:05.332613 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-678d747c5f-bgj92_d22707c7-21fa-4511-b0b1-869b41fb2351/router/0.log" Oct 31 05:43:05.732085 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:43:05.732058 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-hgv9r_68e465ba-2b5d-4e3c-8b63-7bdb9b695b75/istio-proxy/0.log" Oct 31 05:43:06.100217 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:43:06.100118 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-lqrr5_6d68d19b-5392-4308-9e2c-4b5e7593d17b/istio-proxy/0.log" Oct 31 05:43:06.473866 ip-10-0-130-57 kubenswrapper[5019]: I1031 05:43:06.473837 5019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-rgj85_8430b919-a9dd-4c7e-9266-ef6872cc0b9e/istio-proxy/0.log"